Re: Best Strategy to process a large number of rows in File

[Brad Johnson]

"When I log a message that contains 1000 lines I see
line\rline2\rline3\r...line999
The split works when token is \\r. Why?"

I'm not positive but it appears that something in the marshaling is
escaping the backslash on the \r so when it writes it out you get \\r but
I'm not sure.  You could try this as a test:

<tokenize token="\r" group="1000" regex="true"/>
<transform>
  <simple>${body}.split("\r")</simple>
</transform>
 <to uri="activemq:queue:Cbikit.Key" />

I'm using the Java String split mechanics there and I'm not sure that
syntax works in that place but there is an equivalent if it doesn't.  That
should send a String[] across that then you can just use.

<split streaming="true">
  <to uri="sedaQueue:queue.ReadyToProcess?blockWhenFull=true" />
</split>

Which should simply understand it is splitting an array at that point.



On Tue, Apr 26, 2016 at 1:32 PM, Michele <michele.mazzi...@finconsgroup.com>
wrote:

> Hi Brad,
>
> as you suggested and I split big files into chunks of 1000 lines changing
> my
> route like this:
>
> <route id="cr-cbikit-1">
>                         <from
>
> uri="{{uri.inbound}}?scheduler=quartz2&amp;scheduler.cron={{poll.consumer.scheduler}}&amp;scheduler.triggerId=FileRetriever&amp;scheduler.triggerGroup=IF_CBIKIT{{uri.inbound.options}}"
> />
>                         <split streaming="true" parallelProcessing="false"
> >
>                                 <tokenize token="\r" group="1000"
> regex="true"/>
>                                 <to uri="activemq:queue:Cbikit.Key" />
>                         </split>
>                 </route>
>
>                 <route id="cr-cbikit-2">
>                         <from
> uri="activemq:queue:Cbikit.Key?destination.consumer.prefetchSize=1"
> />
>                         <split streaming="true">
>                                 <tokenize token="\r" />
>                                 <to
> uri="sedaQueue:queue.ReadyToProcess?blockWhenFull=true" />
>                         </split>
>                 </route>
>
>                 <route id="cr-cbikit-3">
>                         <from
> uri="sedaQueue:queue.ReadyToProcess?concurrentConsumers=3" />
>                         process....
>                 </route>
>
> This works fine! Memory Usage and Processing Time are very good.
> But, I found a problem: Second route doesn't work when token is equal to
> \r.
> When I log a message that contains 1000 lines I see
> line\rline2\rline3\r...line999
> The split works when token is \\r. Why? The file affected by this issue is
> a
> csv.
>
> Thanks a lot again for your support.
>
> Best regards
>
> Michele
>
>
>
>
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Best-Strategy-to-process-a-large-number-of-rows-in-File-tp5779856p5781750.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>