Hi Kasun, I have tested Kafka producer side of Kafka ESB connector and shared the document[1]. Kafka inbound throughput average is around 1/s.Current Kafka inbound implementation has a limitation that we cannot reduce the time interval less than 1000 ms.
[1] - https://docs.google.com/a/wso2.com/spreadsheets/d/1LrzxDh98V8PBfXctQ6v6MWZitCKg9Bm41ju5gHZTNWE/edit?usp=sharing Thanks, Kathees On Thu, Oct 16, 2014 at 10:02 AM, Kasun Indrasiri <[email protected]> wrote: > Hi Kathees, > > Can you please share the performance stats for consumer and producer side > of the Kafka inbound and connector please. > > On Mon, Oct 6, 2014 at 10:29 AM, Kathees Rajendram <[email protected]> > wrote: > >> Hi Dushan, >> >> Yes I set it. Reason for this issue is that producer connection was not >> closed properly. Now number of open file limit is enough. >> >> Thanks, >> Kathees >> >> On Sun, Oct 5, 2014 at 7:43 PM, Dushan Abeyruwan <[email protected]> wrote: >> >>> ulimit ? >>> >>> have you set them in your OS >>> >>> On Sat, Oct 4, 2014 at 8:48 PM, Kathees Rajendram <[email protected]> >>> wrote: >>> >>>> >>>> Hi, >>>> >>>> I tried to send 5000 messages to Kafka broker using Jmeter ( 10 thread >>>> and 500 messages per a thread,one message size is 105 byes). After 2000 >>>> messages I am getting the following exception. >>>> Exception shows "Too many open files" error.So I changed to increase >>>> no. of open files limit on my machine. >>>> >>>> >>>> http://askubuntu.com/questions/162229/how-do-i-increase-the-open-files-limit-for-a-non-root-user >>>> >>>> and changed buffer size (socket.request.max.bytes) value in >>>> server.properties file but still I am getting same exception. >>>> >>>> What is the cause for this issue? >>>> >>>> >>>> [2014-10-03 12:31:07,051] ERROR - Utils$ fetching topic metadata for >>>> topics [Set(test1)] from broker >>>> [ArrayBuffer(id:0,host:localhost,port:9092)] failed >>>> kafka.common.KafkaException: fetching topic metadata for topics >>>> [Set(test1)] from broker [ArrayBuffer(id:0,host:localhost,port:9092)] >>>> failed >>>> at >>>> kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:67) >>>> at >>>> kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82) >>>> at >>>> kafka.producer.async.DefaultEventHandler$$anonfun$handle$2.apply$mcV$sp(DefaultEventHandler.scala:78) >>>> at kafka.utils.Utils$.swallow(Utils.scala:167) >>>> at kafka.utils.Logging$class.swallowError(Logging.scala:106) >>>> at kafka.utils.Utils$.swallowError(Utils.scala:46) >>>> at >>>> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:78) >>>> at kafka.producer.Producer.send(Producer.scala:76) >>>> at kafka.javaapi.producer.Producer.send(Producer.scala:33) >>>> at org.wso2.carbon.connector.KafkaProduce.send(KafkaProduce.java:71) >>>> at >>>> org.wso2.carbon.connector.KafkaProduce.connect(KafkaProduce.java:28) >>>> at >>>> org.wso2.carbon.connector.core.AbstractConnector.mediate(AbstractConnector.java:32) >>>> at >>>> org.apache.synapse.mediators.ext.ClassMediator.mediate(ClassMediator.java:78) >>>> at >>>> org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:77) >>>> at >>>> org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:47) >>>> at >>>> org.apache.synapse.mediators.template.TemplateMediator.mediate(TemplateMediator.java:77) >>>> at >>>> org.apache.synapse.mediators.template.InvokeMediator.mediate(InvokeMediator.java:129) >>>> at >>>> org.apache.synapse.mediators.template.InvokeMediator.mediate(InvokeMediator.java:78) >>>> at >>>> org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:77) >>>> at >>>> org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:47) >>>> at >>>> org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:131) >>>> at >>>> org.apache.synapse.core.axis2.ProxyServiceMessageReceiver.receive(ProxyServiceMessageReceiver.java:166) >>>> at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180) >>>> at >>>> org.apache.synapse.transport.passthru.ServerWorker.processNonEntityEnclosingRESTHandler(ServerWorker.java:344) >>>> at >>>> org.apache.synapse.transport.passthru.ServerWorker.processEntityEnclosingRequest(ServerWorker.java:385) >>>> at >>>> org.apache.synapse.transport.passthru.ServerWorker.run(ServerWorker.java:183) >>>> at >>>> org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>> at java.lang.Thread.run(Thread.java:745) >>>> Caused by: java.net.SocketException: Too many open files >>>> at sun.nio.ch.Net.socket0(Native Method) >>>> at sun.nio.ch.Net.socket(Net.java:423) >>>> at sun.nio.ch.Net.socket(Net.java:416) >>>> at sun.nio.ch.SocketChannelImpl.<init>(SocketChannelImpl.java:104) >>>> at >>>> sun.nio.ch.SelectorProviderImpl.openSocketChannel(SelectorProviderImpl.java:60) >>>> at java.nio.channels.SocketChannel.open(SocketChannel.java:142) >>>> at kafka.network.BlockingChannel.connect(BlockingChannel.scala:48) >>>> at kafka.producer.SyncProducer.connect(SyncProducer.scala:141) >>>> at >>>> kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:156) >>>> at >>>> kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:68) >>>> at kafka.producer.SyncProducer.send(SyncProducer.scala:112) >>>> at >>>> kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:53) >>>> ... 29 more >>>> >>>> >>>> >>>> Thanks, >>>> Kathees >>>> >>>> >>>> -- >>>> Kathees >>>> Software Engineer, >>>> email: [email protected] >>>> mobile: +94772596173 >>>> >>>> _______________________________________________ >>>> Dev mailing list >>>> [email protected] >>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>> >>>> >>> >>> >>> -- >>> Dushan Abeyruwan | Associate Tech Lead >>> Integration Technologies Team >>> PMC Member Apache Synpase >>> WSO2 Inc. http://wso2.com/ >>> Blog:http://dushansview.blogspot.com/ >>> Mobile:(0094)713942042 >>> >>> >> >> >> -- >> Kathees >> Software Engineer, >> email: [email protected] >> mobile: +94772596173 >> >> _______________________________________________ >> Dev mailing list >> [email protected] >> http://wso2.org/cgi-bin/mailman/listinfo/dev >> >> > > > -- > Kasun Indrasiri > Software Architect > WSO2, Inc.; http://wso2.com > lean.enterprise.middleware > > cell: +94 77 556 5206 > Blog : http://kasunpanorama.blogspot.com/ > -- Kathees Software Engineer, email: [email protected] mobile: +94772596173
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
