On Tue, Oct 13, 2009 at 10:22, Yoshinobu Kano <[email protected]> wrote:
> One thing to confirm, does the streaming input you described make things > efficient? Yes, if your service is able to start processing before the complete list is received, this means that you can process data as they are produced upstream. In the same way if your service can output data by pipelining (which is the term we prefer for this itemized streaming), then future iterations or pipeline-receiving services can start before you are finished. Iterations as done by Taverna are pipelining themselves, the iteration don't need to wait for the complete lists to arrive, as it can use the iteration strategies and the indexes to assign new iterated indexes. > My purpose is not to load everything on the memory at once for the > efficiency, > but to recieve elements of a input list one by one, > output corresponding data one by one. > This is due to the assumed input/output could be a very large amount of > data, but can be divided into smaller chunks. I agree that this would be an advisable strategy, as long as you don't need some kind of assembly-phase that needs the full data in the end anyway. (If you do need this, perhaps the assembly phase can choose to access the data as InputStreams and read it byte by byte) > Seems like the last input you showed, a list of all of the elements, will be > loaded to the memory at the same time? Yes, but only if you declare that you need to complete values. If you say that you accept T2Reference as a type on an input port (from configurePorts() I believe), you will get the references instead, and can choose to dereference them yourself using the reference service (which is available through the context). For the last case you just have to avoid dereferencing when it's the complete list. -- Stian Soiland-Reyes, myGrid team School of Computer Science The University of Manchester ------------------------------------------------------------------------------ Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference _______________________________________________ taverna-hackers mailing list [email protected] Web site: http://www.taverna.org.uk Mailing lists: http://www.taverna.org.uk/taverna-mailing-lists/ Developers Guide: http://www.mygrid.org.uk/tools/developer-information
