Hi Amit, When you do toString in the class mediator it build the complete DOM into the memory. In this case streaming does not make any sense.
ESB itself need memory for some additional tasks that maybe the reason your java app worked without any issues. To get more details on this better if you could profile the memory usage when the file is processing and afterwards. On Thu, Jan 12, 2017 at 3:06 PM, Amit Rajvanshi < [email protected]> wrote: > Hello WSO2 dev , > > > > Thanks Malaka for your reply but streaming in Inbound endpoint did not > work for my case. > > If I set JVM to *-Xms1024m -Xmx4096m -XX:MaxPermSize=2048m *under eclipse > configuration then only it start processing of the file otherwise it > failed to read message from Message context in Custom mediator. > > > > *Code in custom mediator :* > > > > public boolean mediate(MessageContext context) { > > SOAPEnvelope envelope = context.getEnvelope(); > > SOAPBody soapBody = envelope.getBody(); > > OMElement omElement = soapBody.getFirstElement(); > > String xmlString = omElement.toString(); > > formEnrichedXml(xmlString, context); > > return true; > > } > > > > > > My concern is that if we are following correct approach to process 140 MB > XML file or increasing memory is wrong approach and something else is wrong > with the code. > > I also tried with a sample application with iterate mediator to split this > file on each instance of the element in the xml with default configuration > but also it failed to execute and given me same error > *(*java.lang.OutOfMemoryError: > Java heap space*) *and works fine if I increase memory limit to 4 GB. > > I have also tried with on simple java class to read this file and it > worked fine without increasing any memory : > > DocumentBuilderFactory docFactory = DocumentBuilderFactory.newInstance(); > > DocumentBuilder docBuilder; > > try > > { > > docBuilder = docFactory.newDocumentBuilder(); > > Document doc = docBuilder.parse("G:// > 20170105_114800_PRODUCT.XML"); > > > > NodeList rootChilds = doc.getDocumentElement(). > getChildNodes(); > > System.out.println(rootChilds.getLength()); > > > > for (int i = 0; i < rootChilds.getLength(); i++) > > { > > Node rootChild = rootChilds.item(i); > > System.out.println(rootChild.getNodeName()); > > } > > > > System.out.println(111); > > } > > catch(Exception e) > > { > > // TODO Auto-generated catch block > > e.printStackTrace(); > > } > > > > XML file contains 57000 instance of an object which size is around 140 MB > which does not seems big file to me. > > > > Please suggest if memory increase is the only way to process this file. > > > > Best regards, > *Amit Rajvanshi* > Eperium Business Solutions (I) Pvt Ltd, > B-13, Sector 57, > Noida (U.P.) - 201301 > Phone: +91-120-4737401 <+91%20120%20473%207401> > Web: www.eperiumindia.com > > *Shaping future commerce together* > > This message may contain confidential or legally privileged information. > In the event of any error in transmission, unauthorized recipients are > requested to contact the sender immediately and not to disclose or make use > of this information. No warranties or assurance are made or given as to the > accuracy of the information given or in relation to the safety of this > e-mail and any attachments. No liability whatsoever is accepted for any > consequences arising from this e-mail. > > > > *From:* Malaka Silva [mailto:[email protected]] > *Sent:* 11 January 2017 09:12 > *To:* WSO2 Developers' List > *Cc:* Amit Rajvanshi; Vivekananthan Sivanayagam > *Subject:* Fwd: Help required > > > > Hi Amit, > > > > Have you tried the streaming with inbound endpoints? > > > > Also please send your queries to [email protected]. You can cc me. That way > more people from community can help you. :) > > > > [1] http://mrmalakasilva.blogspot.com/2016/03/streaming-large-files-using- > wso2-esb.html > > [2] https://docs.wso2.com/display/ESB500/Tuning+the+VFS+Transport > > [3] https://docs.wso2.com/display/ESB500/File+Inbound+Protocol > > > > ---------- Forwarded message ---------- > From: *Amit Rajvanshi* <[email protected]> > Date: Tue, Jan 10, 2017 at 7:05 PM > Subject: Help required > To: Malaka Silva <[email protected]> > > Hi Malaka, > > > > We again need your help : > > > > We are trying to process one xml which contains around 60000 instance of > an object and the size of XML is 140 MB. > > We are receiving this file through Inbound endpoint file location and > doing some changes in custom mediator java code using DOM object. > > While we are processing small files around 30 MB it get processed > successfully but for large files it is giving out of memory error: > > > > java.lang.OutOfMemoryError: Java heap space > > > > if we set our JVM to *-Xms2048m -Xmx2048m -XX:MaxPermSize=1024m* which is > suggested by WSO2 it is giving error and if we set it to *-Xms1024m > -Xmx4096m -XX:MaxPermSize=2048m* > > then it is working fine while if we process same file in java code then > it is working fine with less memory as well. While debugging we have also > found that it is increasing heap memory when we are taking > > xml object from context. > > > > This scenario is also reproducible for iterate mediator . > > Could you please help us or suggest if there are any memory tuning > required for xml at message context level or somewhere else. > > > > Best regards, > *Amit Rajvanshi* > Eperium Business Solutions (I) Pvt Ltd, > B-13, Sector 57, > Noida (U.P.) - 201301 > Phone: +91-120-4737401 <+91%20120%20473%207401> > Web: www.eperiumindia.com > > *Shaping future commerce together* > > This message may contain confidential or legally privileged information. > In the event of any error in transmission, unauthorized recipients are > requested to contact the sender immediately and not to disclose or make use > of this information. No warranties or assurance are made or given as to the > accuracy of the information given or in relation to the safety of this > e-mail and any attachments. No liability whatsoever is accepted for any > consequences arising from this e-mail. > > > > > > > > -- > > > > Best Regards, > > > Malaka Silva > Senior Technical Lead > M: +94 777 219 791 <077%20721%209791> > > Tel : 94 11 214 5345 > > Fax :94 11 2145300 <011%202%20145300> > > Skype : malaka.sampath.silva > > LinkedIn : http://www.linkedin.com/pub/malaka-silva/6/33/77 > > Blog : http://mrmalakasilva.blogspot.com/ > > > > WSO2, Inc. > > lean . enterprise . middleware > https://wso2.com/signature > > http://www.wso2.com/about/team/malaka-silva/ > <http://wso2.com/about/team/malaka-silva/> > > https://store.wso2.com/store/ > > Don't make Trees rare, we should keep them with care > -- Best Regards, Malaka Silva Senior Technical Lead M: +94 777 219 791 Tel : 94 11 214 5345 Fax :94 11 2145300 Skype : malaka.sampath.silva LinkedIn : http://www.linkedin.com/pub/malaka-silva/6/33/77 Blog : http://mrmalakasilva.blogspot.com/ WSO2, Inc. lean . enterprise . middleware https://wso2.com/signature http://www.wso2.com/about/team/malaka-silva/ <http://wso2.com/about/team/malaka-silva/> https://store.wso2.com/store/ Don't make Trees rare, we should keep them with care
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
