Yes, you should avoid JMS and JCA flows as they will serialize the JBI exchanges, thus loading the whole streams in memory.
On Tue, Sep 23, 2008 at 12:38 PM, Jussi Nummelin <[EMAIL PROTECTED]> wrote: > Thanks. > > But how about configurations for different flows, is there something > that we should configure to enable streaming for messages within the > NMR? > > - Jussi - > > On Tue, Sep 23, 2008 at 12:59 PM, Guillaume Nodet <[EMAIL PROTECTED]> wrote: >> I think the CXF binding component does support attachments caching to >> disk to avoid loading them entirely in memory. >> >> On Tue, Sep 23, 2008 at 10:37 AM, Jussi Nummelin >> <[EMAIL PROTECTED]> wrote: >>> We're mainly using SOAP as our incoming protocol. In typical cases we >>> get the actual payload data as attachments which we need to rip of >>> from the message and send them to further processing in the bus. These >>> attachments can be hundreds of megs in size. >>> >>> We're thinking of some kind of a streaming solution 'cause typically >>> we do not need to transform the actual payload data. The >>> transformation and "real" processing happens only for the incoming >>> message and not for the attachments. >>> >>> BR, >>> - Jussi - >>> >>> On Mon, Sep 22, 2008 at 5:24 PM, Guillaume Nodet <[EMAIL PROTECTED]> wrote: >>>> It mostly depends on the flow and components used. >>>> Could you give more informations ? >>>> >>>> On Mon, Sep 22, 2008 at 3:06 PM, Jussi Nummelin >>>> <[EMAIL PROTECTED]> wrote: >>>>> Hi, >>>>> >>>>> We're facing the exact same issue. And no, turning DEBUG logging off >>>>> does not solve it. :-) >>>>> >>>>> How to configure the flow(s) to handle huge input messages as streams? >>>>> Or is this something that should be configured in the binding >>>>> components in use? >>>>> >>>>> BR, >>>>> - Jussi - >>>>> >>>>> On Wed, Sep 3, 2008 at 1:31 PM, Guillaume Nodet <[EMAIL PROTECTED]> wrote: >>>>>> You need to turn off DEBUG logging, else the exchanges are displayed, >>>>>> thus the whole xml document is loaded in memory and parsed. >>>>>> Alternatively, you can set the system property named >>>>>> "org.apache.servicemix.preserveContent" to "true" if you want to keep >>>>>> DEBUG logging. >>>>>> >>>>>> On Wed, Sep 3, 2008 at 10:53 AM, steff aka sid >>>>>> <[EMAIL PROTECTED]> wrote: >>>>>>> >>>>>>> Hi, >>>>>>> its me again :) I'am trying to handle a huge amount of data with >>>>>>> servicemix >>>>>>> but always get an error message (I'am not so familar with tuning smx >>>>>>> etc.). >>>>>>> So what I want to do is the following: >>>>>>> >>>>>>> 1. Get an xml file with ftp (see my other posts x) ). This xml files are >>>>>>> about 100 to 200 of MByte. >>>>>>> 2. This files are send as a message to a bean component and this bean >>>>>>> component has to do some transformations by using Smooks. This sending >>>>>>> fails. >>>>>>> 3. After the transformation the transformed files which are shrinked at >>>>>>> about 50Mbyte has to be send to another ftp or folder. >>>>>>> >>>>>>> My questions: >>>>>>> - What is the best flow to use in such a use case, where can I define >>>>>>> the >>>>>>> flow to use? >>>>>>> - What parameters affect this issue? >>>>>>> - How can I receive the message as Stream? >>>>>>> >>>>>>> Regards >>>>>>> /Steffen >>>>>>> >>>>>>> P.s. Here is the Exception I got: >>>>>>> >>>>>>> Exception in thread "pool-flow.seda.servicemix-bean-thread-1" >>>>>>> java.lang.OutOfMemoryError: Java heap space >>>>>>> at java.util.Arrays.copyOf(Arrays.java:2882) >>>>>>> at >>>>>>> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100) >>>>>>> at >>>>>>> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:572) >>>>>>> at java.lang.StringBuffer.append(StringBuffer.java:320) >>>>>>> at java.io.StringWriter.write(StringWriter.java:60) >>>>>>> at >>>>>>> org.apache.xml.serializer.ToStream.writeAttrString(ToStream.java:1935) >>>>>>> at >>>>>>> org.apache.xml.serializer.ToStream.processAttributes(ToStream.java:1902) >>>>>>> at >>>>>>> org.apache.xml.serializer.ToStream.endElement(ToStream.java:1989) >>>>>>> at >>>>>>> org.apache.xalan.transformer.TransformerIdentityImpl.endElement(TransformerIdentityImpl.java:1101) >>>>>>> at >>>>>>> org.apache.xml.serializer.TreeWalker.endNode(TreeWalker.java:458) >>>>>>> at >>>>>>> org.apache.xml.serializer.TreeWalker.traverse(TreeWalker.java:149) >>>>>>> at >>>>>>> org.apache.xalan.transformer.TransformerIdentityImpl.transform(TransformerIdentityImpl.java:389) >>>>>>> at >>>>>>> org.apache.servicemix.jbi.jaxp.SourceTransformer.toResult(SourceTransformer.java:125) >>>>>>> at >>>>>>> org.apache.servicemix.jbi.jaxp.SourceTransformer.toResult(SourceTransformer.java:103) >>>>>>> at >>>>>>> org.apache.servicemix.jbi.jaxp.SourceTransformer.toString(SourceTransformer.java:140) >>>>>>> at >>>>>>> org.apache.servicemix.jbi.jaxp.SourceTransformer.toString(SourceTransformer.java:149) >>>>>>> at >>>>>>> org.apache.servicemix.jbi.messaging.MessageExchangeImpl.display(MessageExchangeImpl.java:769) >>>>>>> at >>>>>>> org.apache.servicemix.jbi.messaging.MessageExchangeImpl.toString(MessageExchangeImpl.java:743) >>>>>>> at java.lang.String.valueOf(String.java:2827) >>>>>>> at java.lang.StringBuilder.append(StringBuilder.java:115) >>>>>>> at >>>>>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:136) >>>>>>> at >>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885) >>>>>>> at >>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) >>>>>>> at java.lang.Thread.run(Thread.java:619) >>>>>>> >>>>>>> I am starting my servicemix with an max Heap of 1024Mbyte. >>>>>>> >>>>>>> >>>>>>> ----- >>>>>>> Brockhaus GmbH >>>>>>> COMPETITIVE THROUGH KNOWLEDGE >>>>>>> >>>>>>> Web: http://www.brockhaus-gruppe.de www.brockhaus-gruppe.de / >>>>>>> http://www.brockhaus-group.com www.brockhaus-group.com >>>>>>> -- >>>>>>> View this message in context: >>>>>>> http://www.nabble.com/Huge-amount-of-data-%28about-100-MByte%29-tp19285331p19285331.html >>>>>>> Sent from the ServiceMix - User mailing list archive at Nabble.com. >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Cheers, >>>>>> Guillaume Nodet >>>>>> ------------------------ >>>>>> Blog: http://gnodet.blogspot.com/ >>>>>> >>>>> >>>> >>>> >>>> >>>> -- >>>> Cheers, >>>> Guillaume Nodet >>>> ------------------------ >>>> Blog: http://gnodet.blogspot.com/ >>>> ------------------------ >>>> Open Source SOA >>>> http://open.iona.com >>>> >>> >> >> >> >> -- >> Cheers, >> Guillaume Nodet >> ------------------------ >> Blog: http://gnodet.blogspot.com/ >> ------------------------ >> Open Source SOA >> http://open.iona.com >> > -- Cheers, Guillaume Nodet ------------------------ Blog: http://gnodet.blogspot.com/ ------------------------ Open Source SOA http://open.iona.com
