We're mainly using SOAP as our incoming protocol. In typical cases we get the actual payload data as attachments which we need to rip of from the message and send them to further processing in the bus. These attachments can be hundreds of megs in size.
We're thinking of some kind of a streaming solution 'cause typically we do not need to transform the actual payload data. The transformation and "real" processing happens only for the incoming message and not for the attachments. BR, - Jussi - On Mon, Sep 22, 2008 at 5:24 PM, Guillaume Nodet <[EMAIL PROTECTED]> wrote: > It mostly depends on the flow and components used. > Could you give more informations ? > > On Mon, Sep 22, 2008 at 3:06 PM, Jussi Nummelin > <[EMAIL PROTECTED]> wrote: >> Hi, >> >> We're facing the exact same issue. And no, turning DEBUG logging off >> does not solve it. :-) >> >> How to configure the flow(s) to handle huge input messages as streams? >> Or is this something that should be configured in the binding >> components in use? >> >> BR, >> - Jussi - >> >> On Wed, Sep 3, 2008 at 1:31 PM, Guillaume Nodet <[EMAIL PROTECTED]> wrote: >>> You need to turn off DEBUG logging, else the exchanges are displayed, >>> thus the whole xml document is loaded in memory and parsed. >>> Alternatively, you can set the system property named >>> "org.apache.servicemix.preserveContent" to "true" if you want to keep >>> DEBUG logging. >>> >>> On Wed, Sep 3, 2008 at 10:53 AM, steff aka sid >>> <[EMAIL PROTECTED]> wrote: >>>> >>>> Hi, >>>> its me again :) I'am trying to handle a huge amount of data with servicemix >>>> but always get an error message (I'am not so familar with tuning smx etc.). >>>> So what I want to do is the following: >>>> >>>> 1. Get an xml file with ftp (see my other posts x) ). This xml files are >>>> about 100 to 200 of MByte. >>>> 2. This files are send as a message to a bean component and this bean >>>> component has to do some transformations by using Smooks. This sending >>>> fails. >>>> 3. After the transformation the transformed files which are shrinked at >>>> about 50Mbyte has to be send to another ftp or folder. >>>> >>>> My questions: >>>> - What is the best flow to use in such a use case, where can I define the >>>> flow to use? >>>> - What parameters affect this issue? >>>> - How can I receive the message as Stream? >>>> >>>> Regards >>>> /Steffen >>>> >>>> P.s. Here is the Exception I got: >>>> >>>> Exception in thread "pool-flow.seda.servicemix-bean-thread-1" >>>> java.lang.OutOfMemoryError: Java heap space >>>> at java.util.Arrays.copyOf(Arrays.java:2882) >>>> at >>>> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100) >>>> at >>>> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:572) >>>> at java.lang.StringBuffer.append(StringBuffer.java:320) >>>> at java.io.StringWriter.write(StringWriter.java:60) >>>> at >>>> org.apache.xml.serializer.ToStream.writeAttrString(ToStream.java:1935) >>>> at >>>> org.apache.xml.serializer.ToStream.processAttributes(ToStream.java:1902) >>>> at org.apache.xml.serializer.ToStream.endElement(ToStream.java:1989) >>>> at >>>> org.apache.xalan.transformer.TransformerIdentityImpl.endElement(TransformerIdentityImpl.java:1101) >>>> at org.apache.xml.serializer.TreeWalker.endNode(TreeWalker.java:458) >>>> at >>>> org.apache.xml.serializer.TreeWalker.traverse(TreeWalker.java:149) >>>> at >>>> org.apache.xalan.transformer.TransformerIdentityImpl.transform(TransformerIdentityImpl.java:389) >>>> at >>>> org.apache.servicemix.jbi.jaxp.SourceTransformer.toResult(SourceTransformer.java:125) >>>> at >>>> org.apache.servicemix.jbi.jaxp.SourceTransformer.toResult(SourceTransformer.java:103) >>>> at >>>> org.apache.servicemix.jbi.jaxp.SourceTransformer.toString(SourceTransformer.java:140) >>>> at >>>> org.apache.servicemix.jbi.jaxp.SourceTransformer.toString(SourceTransformer.java:149) >>>> at >>>> org.apache.servicemix.jbi.messaging.MessageExchangeImpl.display(MessageExchangeImpl.java:769) >>>> at >>>> org.apache.servicemix.jbi.messaging.MessageExchangeImpl.toString(MessageExchangeImpl.java:743) >>>> at java.lang.String.valueOf(String.java:2827) >>>> at java.lang.StringBuilder.append(StringBuilder.java:115) >>>> at >>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:136) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) >>>> at java.lang.Thread.run(Thread.java:619) >>>> >>>> I am starting my servicemix with an max Heap of 1024Mbyte. >>>> >>>> >>>> ----- >>>> Brockhaus GmbH >>>> COMPETITIVE THROUGH KNOWLEDGE >>>> >>>> Web: http://www.brockhaus-gruppe.de www.brockhaus-gruppe.de / >>>> http://www.brockhaus-group.com www.brockhaus-group.com >>>> -- >>>> View this message in context: >>>> http://www.nabble.com/Huge-amount-of-data-%28about-100-MByte%29-tp19285331p19285331.html >>>> Sent from the ServiceMix - User mailing list archive at Nabble.com. >>>> >>>> >>> >>> >>> >>> -- >>> Cheers, >>> Guillaume Nodet >>> ------------------------ >>> Blog: http://gnodet.blogspot.com/ >>> >> > > > > -- > Cheers, > Guillaume Nodet > ------------------------ > Blog: http://gnodet.blogspot.com/ > ------------------------ > Open Source SOA > http://open.iona.com >
