RE: Passing large data in SOAP Message

[Vinod Patil]

Title: RE: Passing large data in SOAP Message

Hi Ken and All,

    Thanks Ken for the reply and the code...

I also have a similar requirement...to fetch from db and send the data as an xml...

 

but the issue is that we have to use DOM for message style webservice. The messaging service method's return signature uses the DOM Api objects...and because of this and the fact that we want to pass large xml the object size increases...

 

One thing i want to know is wether we can construct the SOAPBodyElement(or any Response type of Message type service) with it treating the xml as a string and not Element/Document obj....or any other way which will not be memory intensive...i may be missing something very important and end up coding a memory intensive application....pls if anyone has any pointers reply back... 

 

Can one solution be to generate the xml file from the data, compress the physical file and pass it back as an attachment...will that be less memory intensive...?

3 Mb file is not a huge file....but when it is interprerted as any DOM object the size of the object increases more than its original size...

 

Have people implemented/seen systems that use webservices to pass back and forth large xml data ?

 

Also the question that Ken asked

Are webservices appropriate to send large amounts of data?  

 

Please whenever ppl have time think about this issue and reply back...

 

Regards,

Vinod.

 

 -----Original Message-----
From: Lee, Ken [mailto:[EMAIL PROTECTED]
Sent: Wednesday, May 19, 2004 8:10 PM
To: [EMAIL PROTECTED]; [EMAIL PROTECTED]
Subject: RE: Passing large data in SOAP Message

I have been working on solutions to this problem for the last month.  In my case, I convert a resultset to a document using a simple method:  I don't have the person I received this code from referenced (oops!).  If you need the imports for this, just let me know.

        private static Document toDocument(ResultSet rs) throws ParserConfigurationException, SQLException {
                DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
                DocumentBuilder builder        = factory.newDocumentBuilder();
                Document doc                   = builder.newDocument();

                Element results = doc.createElement("Results");
                doc.appendChild(results);

                ResultSetMetaData rsmd = rs.getMetaData();
                int colCount           = rsmd.getColumnCount();

                /* System.out.println("Starting to build Document"); */
                while (rs.next()) {
                        Element row = doc.createElement("Row");
                        results.appendChild(row);
                        for (int i = 1; i <= colCount; i++) {
                                String columnName = rsmd.getColumnName(i);
                                String value      = rs.getString(i);
                                Element node      = doc.createElement(columnName);
                                node.appendChild(doc.createTextNode(value.trim()));
                                row.appendChild(node);
                        }
                }
                /* System.out.println("Document created"); */
                return doc;
        }

To get around memory problems, I did two things, added memory to the web server, and changed the memory settings to use quite a bit of memory on the jvm.  In tomcat, I use the following JAVA_OPTS setting: "-Xms256m -Xmx256m -DentityExpansionLimit=1000000"  The -DentityExpansionLimit change came from a comment by Nelson Minar on this forum.

I still don't have an answer that would send the document back compressed, and using DIME or some other chunking mechanism.  I just haven't figured it out yet.

I hope some of this may help.  I am not sure Web services is the right answer for sending large amounts of data.  I have also been looking at using SSH tunnels to access the data directly in a secure manner.  The speed has actually been better than all of the hoops that SOAP has be jumping through.

For smaller datasets, SOAP is working great!

Hope this helps.  I also hope that someone else with knowledge working with large output objects will provide more enlightenment!

Ken

>  -----Original Message-----
> From:         Vinod Patil [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, May 18, 2004 10:40 PM
> To:   [EMAIL PROTECTED]
> Subject:      Passing large data in SOAP Message
>
> Hi All,
>
>  A type java:MSG service should have the signatures as
> --public Element [] method(Element [] bodies)
> --public SOAPBodyElement [] method (SOAPBodyElement [] bodies)
> --public Document method(Document body)
> --public void method(SOAPEnvelope req, SOAPEnvelope resp)
>
> Now if the XML that i am going to embed in the body of the
> message is very large (say abt 3+ Mb) then the output objects
> of the service methods will be also be large in size.
> And this will lead to memory problems.
>
> What i want to ask is that how do we pass very large XML's in
> the SOAP messages...What approach should be taken if one
> wants to pass very large XML data in the SOAP message body?
>
> I am currently using the messaging style service ( with
> signature public SOAPBodyElement [] method (SOAPBodyElement
> [] bodies)  ) but i am running into OutOfMemory problems...
> This is because the Document object that we use to construct
> the SOAPBodyElement is very large.
>
> Can anyone please guide me regarding the approcah that needs
> to be taken in case of passing back huge XML data from a
> webservice using Axis or any SOAP Implementation?
>
> Regards,
> Vinod