By Parameter I meant <property>. So the answer is really NO for now ? Not in a memory efficient way. Any time frames ?
Andreas Veithen-2 wrote: > > There are two things that need to be done to satisfy your requirement: > > 1. The message builders for text/plain (PlainTextBuilder) and > application/octet-stream (BinaryBuilder) need to be optimized using > techniques that we developed some time ago to efficiently handle large > text output from XSL transformations (TemporaryData, > WrappedTextNodeStreamReader and TextFileDataSource). This is > technically not difficult to do but it is a bit tricky at this moment > because we will have to move some code from Synapse to WS-Commons > transport (and maybe even to Axiom and Axis2) and we are in the middle > of a release process. Note that there was a discussion some time ago > to do a general overhaul and optimization of the message builders and > formatters after the Axis2 1.5 release and the > PlainTextBuilder/BinaryBuilder optimization issue is part of this. > > 2. We need to be able to store Axiom elements in properties during > mediation. I think this is supported by Synapse's core but not by the > <property> mediator. There was a discussion around this a few days > ago. We also need a mediator that takes an element from a property and > adds it to the current message (in a location specified by an XPath > expression). This is required because using XSLT or a scripting > mediator would indeed cause the content of the file to be read into > memory. > > By extending the message builder interface so that messages can be > build from DataSource objects (instead of InputStreams), we could even > go as far as allowing to stream the data from the FTP directly to the > target Web service without even using a temporary file. > > Regards, > > Andreas > > On Wed, Feb 25, 2009 at 23:58, kimhorn <[email protected]> wrote: >> >> >> An X12 EDI file is FTped to us. We want to submit that data to a Web >> Service.The file is essentially a single string of text data up to about >> 2MB >> in size.We want to submit that data to a 'Submit' Web Service that takes >> that data in a tag. e.g. <data>ISA....ISE<data/>.To use the 'Submit' Web >> Service, a transaction ID is required. A call to another Web Service >> prior >> to the submit is required to get a transaction ID. Hence thechain. >> >> If VFS reads a text file then the XML payload has all the data in one >> field. >> E.G. >> >> <?xml version="1.0" encoding="utf-8"?> >> <soapenv:Envelope >> xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/";> >> <soapenv:Body> >> <axis2ns1:text >> xmlns:axis2ns1="http://ws.apache.org/commons/ns/payload";> >> "All the X12 Text data is here. One big blob...." >> </axis2ns1:text> >> </soapenv:Body> >> </soapenv:Envelope> >> >> So AXIOM does not help here; at all. Its all or none for that VFS <text> >> tag.The text could be up to 2 MB, once a day. Many smaller files, 500KB >> across the day. >> >> The final submit web service call will need to send this data. The data >> is >> provided by the VFS trigger. A web service call is required in between to >> get the transaction ID. SO the data has to be held temporarily in a >> parameter OR some alternative between the calls. SO: >> >> VFS Trigger - provides <data> - store temporarily somewhere. >> Web Servie Call-1 - get <Transaction ID> >> Web Service Call-2 - send <data> and <Transaction ID> >> >> However this backgorund in not necessary to answer the Technical Question >> above; that focuses on the specific technical risk. >> >> A Standalone Java application can sove this problem easily; Given most >> ESB's >> and B2B, integration products e.g. BEA Aqualogic, provide ways to process >> and temporarily store large data blobs. Given Synapse does not document >> its >> architecture or provide any information on Quality of Service limits. >> E.G. >> What is the 'implementation' of a Parameter; can it handle large blobs of >> data. If not what is the alternate Synapse Mechanism to do this ? >> >> FTP is traditionally used to move large amounts of data and integration >> products that include FTP adapters, provide mechanisms to process large >> amounts of data. Synapse includes an FTP VFS adapter but no documented >> mechanism to process large blobs of data. Is VFS just a 'TOY' add hoc >> addon >> or can it do real work ? Given Synapse does not have a tailored script >> mechanism for handling B2B data, and only uses other approaches ie, >> Javascript, etc that are not designed for integration tasks. >> >> Is the answer that Synapse should not be used to solve this type of >> integration problem, as it does not have mechanisms to process large >> data >> blobs ? If it can then what are the best Practises and Patterns that >> should >> be used for SYnapse to do this ? >> >> >> >> Asankha C. Perera wrote: >>> >>> Hi Kim >>>> The following example script is used for illustrative purposes. In two >>>> places >>>> the VFS payload is copied. Into a property, and in the JSScript. In >>>> full >>>> scenario the property value is used later in a chain of Web service but >>>> needs to be stored temporily, as the request will be replaced by >>>> request/responce of next web service. The file drop will be replaced by >>>> FTP. >>>> Issue is we get 'big' files by FTP that we need to submit to a Web >>>> Service. >>>> The Web Service is OK with the large data, but will Synapse cope. >>>> >>>> Issues: >>>> >>>> 1) What if the payload gets large; >>>> What is large (5MB, 20 MB ?) and >>>> What problems will this mean for this code e.g. Memory etc. >>>> >>> Synapse, especially the NIO transport, is written to never exhaust with >>> OOM with large payloads - i.e. it does not bite more than it can chew. >>> But this does not mean your configuration is safe - for example, reading >>> any payload into a String is bad.. even if its 1MB.. and I see your >>> script mediator doing that - which is worse.. You could maybe use a >>> simple class/POJO mediator to save or process large payload etc. >>>> 2) The JSScript could be replaced with a Java Mediator and use a >>>> stream to better copy the VFS text element {data}. However still >>>> means >>>> it would >>>> copy whole string into memory. >>>> >>> No, in Java, you could get the payload written safely into a file >>> stream, without reading into memory. >>>> 3) Storing the payload temporarily, e.g. alternative to property. How >>>> big >>>> can Property be ? >>>> >>> This will depend on how much memory you allocate and how many of these >>> messages are processed concurrently. Ideally, properties should not take >>> much space >>>> 4) using the XPath to get the payload into property. >>>> >>> Not sure I understand.. again, if you are dealing with large payloads, >>> you would need to be careful. Can you state how large your files are, >>> whats the frequency of processing, and the number of files per each >>> batch etc, and the type of file - XML, Text, CSV etc and what type of >>> operations you want to take on this data.. >>> >>> One thing I always ask users is to state their business problem or the >>> requirement, and ask us for the best technical solution. Once we >>> understand your problem, we could look into alternatives and weigh pros >>> and cons to help you decide >>> >>> cheers >>> asankha >>> >>> -- >>> Asankha C. Perera >>> http://adroitlogic.org >>> >>> http://esbmagic.blogspot.com >>> >>> >>> >> >> -- >> View this message in context: >> http://www.nabble.com/Large-Payload%3A-Issues-tp22196278p22213925.html >> Sent from the Synapse - User mailing list archive at Nabble.com. >> >> > > -- View this message in context: http://www.nabble.com/Large-Payload%3A-Issues-tp22196278p22215650.html Sent from the Synapse - User mailing list archive at Nabble.com.
