When you add more processors into the route, camel need to do lots of addition work to handle the message. If you want a good performance, you should use the Java code to do the loop in a single method. But it could be good method for us find the hot spots the performance those camel introduces.
Willem Jiang Red Hat, Inc. FuseSource is now part of Red Hat Web: http://www.fusesource.com | http://www.redhat.com Blog: http://willemjiang.blogspot.com (http://willemjiang.blogspot.com/) (English) http://jnn.iteye.com (http://jnn.javaeye.com/) (Chinese) Twitter: willemjiang Weibo: 姜宁willem On Mon, Jul 22, 2013 at 4:29 AM, Claus Ibsen <claus.ib...@gmail.com> wrote: > Hi > > You cant really compare the 2 approaches. > In the pure java code you just have a for loop in a single method > which is as fast as you can go. > When using Camel routes and the EIPs then a lot more goes on under the > hood. > > > On Sun, Jul 21, 2013 at 10:12 PM, Viktor Kubinec <vikto...@seznam.cz> > wrote: > > Thanks Willem. > > > > I've created custom splitter, which is using BufferedReader, just as you > > suggested. It has improved the performance a bit but it is still much > slower > > (10-20x) than the java code that I posted. I have spent bit more time > > investigating this problem. The code (with documentation on how to use > it) > > is available here : > > > > https://github.com/Viktor-Kubinec/Camel > > > > I am using Apache Camel verison 2.11.1. My test file is ~1Gb large, > contains > > 100M rows with message "Test Line" in each row. I have two testing > scenarios > > : > > > > 1. SplitTest - I just read big file containing short lines, split it > line by > > line with my custom splitter and logging the process (every 50k-th > message) > > > > 2. Same as 1. but I've added 10 processors that do nothing (because I > > observed that adding processors reduces performance) > > > > Results : > > > > java code : 1.5-2M lines per second - no profiling done > > > > 1. 150k lines per second and here are some hot spots (from VisualVM) : > > > > java.lang.Class.getSimpleName() (57.3%) > > org.apache.camel.processor.MulticastProcessor.doAggregate() (39.2%) > > java.io.BufferedReader.readLine() (3.1%) > > ... > > > > 2. 72k lines per second : > > > > org.apache.camel.processor.MulticastProcessor.doAggregate() (70.5%) > > java.lang.Class.getSimpleName() (28.8%) > > java.io.BufferedReader.readLine() (0.3%) > > ... > > > > I have saved snapshots from VisualVM - they are available in the github > > repository (link above) : SplitTest.nps, SplitTestWithProcessors.nps. > More > > details are available there. > > > > This was done on my laptop. I've read that VisualVM has some effect on > > performance (~5%). java.lang.Class.getSimpleName() is being called in > > constructor of org.apache.camel.impl.DefaultUnitOfWork. In first scenario > > only 3.1% of thread time was spent in the readLine() method. In second > > scenario it was only 0.3%. That is very low. > > > > Is there a way how I can optimize this? > > > > Thanks, > > > > Viktor > > > > > > > > > > > > -- > > View this message in context: > http://camel.465427.n5.nabble.com/Message-Processing-Performance-while-splitting-tp5735824p5735979.html > > Sent from the Camel - Users mailing list archive at Nabble.com. > > > > -- > Claus Ibsen > ----------------- > Red Hat, Inc. > Email: cib...@redhat.com > Twitter: davsclaus > Blog: http://davsclaus.com > Author of Camel in Action: http://www.manning.com/ibsen >
