thanks Ashwin for the reply. I'll try to implement the ActiveMQ strategy to distribute the load to several servers.
And I also found out the main cause of this 'slow' throughput is the scanStreamDelay in stream component. I stupidly set the value to 50. After I set the value to 0, the processing speed is much faster. The drawback is that will drain your CPU a lot. On Mon, Apr 11, 2011 at 6:17 PM, Muhammad Rayhan <[email protected]> wrote: > Hi All > > I have a case in which I need to stream 3 transaction log files in > real-time, process each line and do some filtering, transformation etc > and send the output to certain server. The current route I use: > > <camelContext id="camel" xmlns="http://camel.apache.org/schema/spring";> > <route id="route1"> > <from > uri="stream:file?fileName=/export/home/apps/smpp/sys/gw/tdr/MtTdr-1&scanStream=true&scanStreamDelay=50"/> > <to > uri="seda:input?timeout=0&concurrentConsumers=16"/> > </route> > > <route id="route2"> > <from > uri="stream:file?fileName=/export/home/apps/smpp/sys/gw2/tdr/MtTdr-1&scanStream=true&scanStreamDelay=50"/> > <to > uri="seda:input?timeout=0&concurrentConsumers=16"/> > </route> > > <route id="route3"> > <from > uri="stream:file?fileName=/export/home/apps/smpp/sys/gw3/tdr/MtTdr-1&scanStream=true&scanStreamDelay=50"/> > <to > uri="seda:input?timeout=0&concurrentConsumers=16"/> > </route> > > <route id="route-x"> > <from > uri="seda:input?timeout=0&concurrentConsumers=16" /> > <log message="streaming TDR line = ${body}"/> > <!-- Some filtering, transformation and aggregation > here --> > </route> > </camelContext> > > The problem is, those 3 input files are produced by high throughput > application. And the SEDA consumer processing speed is always lower, > so I ended up with late processing response time. For example, after > several minutes, let say at 16:15, the module is still processing the > log that is produced at 16:05. And the gap become bigger and bigger. > > I try to remove the filtering, transforming part but still face the > same issue.I try to increase the concurrentConsumers but also no luck > > Is there anyway to tune or improve the SEDA consumer throughput? Do I > miss something, or I design the route in incorrect way? > > > > -- > Twitter: @md_ray > Email/IM: [email protected] > -- Sent using Telkomsel network Twitter: @md_ray Email/IM: [email protected] and [email protected]
