Hi, Suresh,
On Tue, Jul 2, 2013 at 11:47 AM, Suresh Marru <[email protected]> wrote: > On Jun 29, 2013, at 3:38 AM, Heejoon Chae <[email protected]> wrote: > > > Hi, Suresh, > > > > Any updates on this? > > > > By the way, can you also let me know how I cloud run for-each component > on files? Actually, what I want to do is to run my script on several input > files(let's say 100 inputs), but actual input files' informations are > listed on a single file instead of getting all the input one by one. > > > > So, the input is single file containing all the input file informations > looks like, > > > > ... > > http://xxx,xxx,xxx/input1.txt > > http://xxx,xxx,xxx/input2.txt > > http://xxx,xxx,xxx/input3.txt > > … > > Hi Heejoon, > > While Chathura and Saminda help you get pass the for-each issues, let me > answer this specific question. I have seen a similar requirement previously > in a different project [1] as well. It should be trivial to add these kind > of "convenience" nodes in workflow which can read a file and split the > content and pass to for-each as an array. We can develop these nodes, but > can you please spare some time and formalize this requirement. > > * Other than the ones you are scripting, did you come across similar > bioinformatics applications or otherwise which produce such listing? Did > you see and ad-hoc standards for these formats? Like what is the delimiter > for each file, newline? it can be comma or space separated as well right? > do you think if Airavata establishes some norm, will application writers > follow them? > What I want to handle here with for each component is input data and I guess one of the key difference between bioinformatics and others are number of data and their size. Let's say I have 30 biological data inputs which is normall in this area. I guess it could be better way to handle them, but simple solution that I imagine is listing & parsing. Without the listing and parsing the inputs, I need to drag and drop 30 input components which will mess up the workflow by only inputs. As you mentioned space separated, tab or comma separated, I guess they are totally fine with couple of inputs, but you could simply imagine long long 30 inputs with containing delimiters. > * How do we determine the type of the parsed content? For instance, > distinguishing form a URI, or a String or integer? > In the example, the reason I used URI is that I assumes the system will be used by users with their own data. As I mentioned the data size in bioinformatics is normally large(for example 1 cell line per 5GB) and they are deposited as web accessible locations. My script download them using parsed URIs and start analysis. So, It might be useful if there is a system component which will handle several inputs and large data at once. In general, bioinformatics applications generate text file as output, but the type(or format) is varies which is one of the issues in this area as well. Thanks, Heejoon. > > I am sure there will be other questions, but once we agree upon such > semantic issues of the content, we can add this as a feature request for > next release. > > Suresh > > [1] - https://cwiki.apache.org/confluence/display/AIRAVATA/ODI > > > > > Thanks! > > Heejoon. > > > > > > On Tue, Jun 11, 2013 at 10:26 PM, Suresh Marru <[email protected]> > wrote: > > Hi Heejoon, > > > > Normally we use for-each on files, so the way to handle that is let all > the output files go into a outputData directory and iterate over them. But > I see what you are doing here. Let me try this and get back to you within > the next day. > > > > Thanks for your patience, > > Suresh > > > > On Jun 10, 2013, at 11:46 PM, Heejoon Chae <[email protected]> > wrote: > > > >> Hi, Airavata users, > >> > >> Do you have any advises or experiences for my previous mail about > ForEach component? > >> I am still struggling with generating input for ForEach component in my > script :/ > >> Any comment will be helpful to me to use the component :) > >> > >> Thank you! > >> > >> ---------- Forwarded message ---------- > >> From: Heejoon Chae <[email protected]> > >> Date: Fri, Jun 7, 2013 at 5:44 PM > >> Subject: Re: Question for Airavata For-Each component > >> To: "[email protected]" <[email protected]> > >> > >> > >> Hi, Lahir, > >> > >> I succeeded to use ForEach component with attaching 'Input' component > in front of ForEach componet with comma separated value(i.e "a,b,c") as > input like in attached figure. ForEach component works well with Input > component with comma separate value. > >> > >> Now I want to mimic what Input component does within my script as in > another figure. This is required to run ForEach on generated data not on > data inputed from user. > >> > >> I tired several ways to generate a input for ForEach component in my > script including what you suggested in previous mail, but I couldn't. > Please help me to solve this problem. > >> > >> Thank you, > >> Heejoon. > >> > >> > >> On Wed, Jun 5, 2013 at 11:44 PM, Lahiru Gunathilake <[email protected]> > wrote: > >> Following script will be useful for you.. > >> > >> echo $@ > >> echo $* > >> echo $# > >> for arg; do > >> printf '%s\n' "$arg" > >> done > >> > >> > >> Lahir > >> > >> > >> On Wed, Jun 5, 2013 at 9:53 AM, 채희준 <[email protected]> wrote: > >> Hi Lahiru, > >> > >> Thank you for your answer. > >> > >> By the way, how to create string array in shell script, so that I can > pass it to ForEach component? > >> > >> Thank you, > >> Heejoon. > >> > >> 2013. 6. 5. 오후 10:44 Lahiru Gunathilake <[email protected]> 작성: > >> > >>> Hi Heejoon, > >>> > >>> We take string array as input but we invoke the script with each > element of the string array multiple times, in Foreach we don't pass whole > string array to the application script at once. So you really don't have to > handle a string array in your script. > >>> > >>> Lahiru > >>> > >>> > >>> On Tue, Jun 4, 2013 at 10:06 PM, Heejoon Chae <[email protected]> > wrote: > >>> Hi, > >>> > >>> I have quick question about Airavata For-Each component. > >>> > >>> It seems that ForEach component takes String array as an input and > pass them to next component one by one. > >>> > >>> My question is how to create string array in shell script to pass it > to ForEach component? > >>> > >>> I normally generate string result like > >>> > >>> echo "output=xxx" > >>> > >>> at the end of script and connect it as an input of next component in > Airavata workflow to pass the result and let next component know the input > is arrived. > >>> > >>> Thanks, > >>> Heejoon. > >>> > >>> > >>> > >>> -- > >>> System Analyst Programmer > >>> PTI Lab > >>> Indiana University > >> > >> > >> > >> -- > >> System Analyst Programmer > >> PTI Lab > >> Indiana University > >> > >> > >> <Screen Shot 2013-06-07 at 3.12.05 PM.png><Screen Shot 2013-06-07 at > 3.11.31 PM.png> > > > > > >
