Hi Gonzales. Take a look at this:
http://www.catify.com/2012/07/09/parsing-large-files-with-apache-camel/
perhaps it solves your issue.
Best regards - Claus
I'm running a route that basically adds a character per line to a
plain text file, but it's taking to long, and it seems that it's due
to some kind of buffering issue when reading/writing from disk.
I'm processing a 5MB file (attached as DC_FACCL132_0000
MORA_1075_16-10-2012_19-09-47_15.txt.zip), with the corresponding XSL
template (also attached).
It's taking for ever to process such a file, I understand I'm
tokenizing on line breaks, which could be the source of the problem as
there are many lines in the file (48198 exactly), but when running
jvisualvm (see attached images/snapshot)I can see the writing op is
invoked 20386 times, which seem not related to the line count. Is
there an output buffer size that I can configure? Or something like that?
This is the route:
<camel:routeid="pager"autoStartup="true">
<camel:from
uri="file:///tmp/in?charset=Windows-1252&move=${file:parent}/../paged/${file:name.noext}.paged.ack&preMove=${file:name.noext}-${date:now:yyyyMMddHHmmssSSS}.${file:ext}
<file:///%5C%5Ctmp%5Cin?charset=Windows-1252&move=$%7bfile:parent%7d/../paged/$%7bfile:name.noext%7d.paged.ack&preMove=$%7bfile:name.noext%7d-$%7bdate:now:yyyyMMddHHmmssSSS%7d.$%7bfile:ext%7d>"/>
<camel:splitstreaming="true"parallelProcessing="false">
<camel:tokenizetoken="\n"/>
<camel:touri="bean:pager"/>
<camel:to
uri="file:///tmp/paged?charset=utf8&fileName=${file:name.noext}.paged&fileExist=Append
<file:///%5C%5Ctmp%5Cpaged?charset=utf8&fileName=$%7bfile:name.noext%7d.paged&fileExist=Append>"/>
</camel:split>
</camel:route>
This is the referenced bean:
<beanid="pager"class="cl.altiuz.reports.etl.TextProcessor">
<propertyname="xsltPath"
value="/Users/gonzalovasquez/Documents/workspace/altiuz-reports/reports-etl/xsl/pager.xsl"/>
<propertyname="param"value="C.*PAG.* 1"/>
</bean>
Camel versión is 2,10.1, and happens both on OSX & MS Windows, so I
think isn't a platform dependent problem, but a configuration one.
Any ideas? Any thing else that I should send?
Thanks!
*Gonzalo Vásquez Sáez*
*Gerente Investigación y Desarrollo (R&D)*
*Altiuz* Soluciones Tecnológicas de Negocios Ltda.
Av. Nueva Tajamar 555 Of. 802, Las Condes
(56-2) 335 2461
[email protected] <mailto:[email protected]>l_
_http://www.altiuz.cl <http://www.altiuz.cl/>_
This e-mail and any files transmitted with it are for the sole use of
the intended recipient(s) and may contain confidential and privileged
information. If you are not the intended recipient(s), please reply to
the sender and destroy all copies of the original message. Any
unauthorized review, use, disclosure, dissemination, forwarding,
printing or copying of this email, and/or any action taken in reliance
on the contents of this e-mail is strictly prohibited and may be
unlawful.
