Re: [galaxy-dev] . Output file to history (Sebastian Luna Valero)
I have opened a pull request that I believe could significantly ease the operations like this (for instance grabbing a directory of similar output files all of the same type based on a pattern specified by the tool instead of by Galaxy). There are lots of details of in the pull request body and some examples in the form of both functional and unit tests: https://bitbucket.org/galaxy/galaxy-central/pull-request/356/enhancements-for-runtime-discovered Let me know (either on the pull request or here) if you would like to see changes or if this is "good enough". These kinds of dynamic outputs are not quite as feature-full as I would like - but hopefully these limitations are addressed by forthcoming dataset collection work which I believe will be able to leverage the work in this pull request. -John On Sat, Mar 22, 2014 at 6:19 PM, Ross wrote: > Hi, Jun and Sebastian, > > This is not trivial, but it is already being done with some tools using the > method described at > https://wiki.galaxyproject.org/Admin/Tools/Multiple%20Output%20Files#Single_history_output_Html_file_with_links_to_any_number_of_output_files_and_images > if that can solve your problem. The Galaxy Html composite datatype can be > used to expose arbitrary outputs without creating a separate history item > for each one if that's what you want. The tool executable or the wrapper > must write valid and complete Html content to the path Galaxy provides as > the Html file for the tool and the sanitize_all_html switch must be turned > off in universe_wsgi.ini (which is a potential security problem for public > sites!) for the page to be pretty, but since the script itself is generating > all the output it is possible to write code to create a nicely laid out page > of links and images for the user. Composite datatypes (like Html) are > documented in the wiki and contributions to the wiki are always welcomed - > but at present, it's an advanced and not well documented possibility. > > If there really is demand for this functionality to be exposed, I have been > thinking about an autoHtml datatype that would do what the tool factory does > - look inside the job working directory at the end of the job and arrange > every file it finds there into a simple Html page of links. If > https://trello.com/c/vNQLZnSk gets enough upvotes, we'll put it into the > development pipeline? > > > > On Sun, Mar 23, 2014 at 4:43 AM, Jun Fan wrote: >> >> Hi Sebastian, >> >> I am not an expert in galaxy developing. So the following content is >> just my personal opinion. I do not think it is possible to meet you >> requirement. I reckon Galaxy searches the certain folder, defined in the >> universal_wsgi.ini by a parameter called #collect_outputs_from = >> new_file_path,job_working_directory. On top of that, a job id is required, >> which is the reason in the dynamic multiple output job id is part of the >> file name. If your script.py cannot follow the rules, Galaxy for sure cannot >> recognize the outputs. Normally in this kind of case, I write an additional >> script to execute the script.py with its original parameter and do the file >> manipulation afterwards. Of course this is not ideal, but I do nothing about >> python. It may be possible to write some complicated python code within the >> command tag of your wrapper file to do this. I am here also waiting for >> experts' opinions/comments. >> >> Best regards! >> Jun >> >> On 3/21/14 9:28 AM, Sebastian Luna Valero wrote: >> > >> > Dear All, >> > >> > I am trying to add a new tool in Galaxy and I have the following >> > problem. >> > >> > Let me explain a simplified example. Let's imagine that my script >> > works from CLI as follows: >> > >> > python script.py --input "input-file" --pattern "output-pattern" >> > >> > After processing "input-file", the script writes several output files >> > and their name is given according to "output-pattern". >> > >> > The number of output files depends on the content of "input-file". The >> > output files are written to the current working directory where >> > script.py is located. >> > >> > In the simplest scenario, I get only one output file in the working >> > directory. >> > >> > My problem is that I would like to see the output files in Galaxy's >> > history without modifying the CLI interface. >> > >> > To solve this problem, I have looked at the wiki page: >> > >> > https://wiki.galaxyproject.org/Admin/Tools/Multiple%20Output%20Files#N >> > umber_of_Output_datasets_cannot_be_determined_until_tool_run >> > >> > and the email here: >> > >> > http://dev.list.galaxyproject.org/Multiple-output-files-do-not-appear- >> > in-history-td4660470.html >> > >> > However, I think that mine is a different scenario. I do not want to >> > add new output parameters in the CLI. I would like Galaxy to bring >> > these output files to my history without modifying the CLI options, is >> > that possible? >> > >> > Many thanks in advance for your help! >> > >> > Best regards, >> > Seba
Re: [galaxy-dev] . Output file to history (Sebastian Luna Valero)
Hi, Jun and Sebastian, This is not trivial, but it is already being done with some tools using the method described at https://wiki.galaxyproject.org/Admin/Tools/Multiple%20Output%20Files#Single_history_output_Html_file_with_links_to_any_number_of_output_files_and_images if that can solve your problem. The Galaxy Html composite datatype can be used to expose arbitrary outputs without creating a separate history item for each one if that's what you want. The tool executable or the wrapper must write valid and complete Html content to the path Galaxy provides as the Html file for the tool and the sanitize_all_html switch must be turned off in universe_wsgi.ini (which is a potential security problem for public sites!) for the page to be pretty, but since the script itself is generating all the output it is possible to write code to create a nicely laid out page of links and images for the user. Composite datatypes (like Html) are documented in the wiki and contributions to the wiki are always welcomed - but at present, it's an advanced and not well documented possibility. If there really is demand for this functionality to be exposed, I have been thinking about an autoHtml datatype that would do what the tool factory does - look inside the job working directory at the end of the job and arrange every file it finds there into a simple Html page of links. If https://trello.com/c/vNQLZnSk gets enough upvotes, we'll put it into the development pipeline? On Sun, Mar 23, 2014 at 4:43 AM, Jun Fan wrote: > Hi Sebastian, > > I am not an expert in galaxy developing. So the following content is > just my personal opinion. I do not think it is possible to meet you > requirement. I reckon Galaxy searches the certain folder, defined in the > universal_wsgi.ini by a parameter called #collect_outputs_from = > new_file_path,job_working_directory. On top of that, a job id is required, > which is the reason in the dynamic multiple output job id is part of the > file name. If your script.py cannot follow the rules, Galaxy for sure > cannot recognize the outputs. Normally in this kind of case, I write an > additional script to execute the script.py with its original parameter and > do the file manipulation afterwards. Of course this is not ideal, but I do > nothing about python. It may be possible to write some complicated python > code within the command tag of your wrapper file to do this. I am here also > waiting for experts' opinions/comments. > > Best regards! > Jun > > On 3/21/14 9:28 AM, Sebastian Luna Valero wrote: > > > > Dear All, > > > > I am trying to add a new tool in Galaxy and I have the following problem. > > > > Let me explain a simplified example. Let's imagine that my script > > works from CLI as follows: > > > > python script.py --input "input-file" --pattern "output-pattern" > > > > After processing "input-file", the script writes several output files > > and their name is given according to "output-pattern". > > > > The number of output files depends on the content of "input-file". The > > output files are written to the current working directory where > > script.py is located. > > > > In the simplest scenario, I get only one output file in the working > > directory. > > > > My problem is that I would like to see the output files in Galaxy's > > history without modifying the CLI interface. > > > > To solve this problem, I have looked at the wiki page: > > > > https://wiki.galaxyproject.org/Admin/Tools/Multiple%20Output%20Files#N > > umber_of_Output_datasets_cannot_be_determined_until_tool_run > > > > and the email here: > > > > http://dev.list.galaxyproject.org/Multiple-output-files-do-not-appear- > > in-history-td4660470.html > > > > However, I think that mine is a different scenario. I do not want to > > add new output parameters in the CLI. I would like Galaxy to bring > > these output files to my history without modifying the CLI options, is > > that possible? > > > > Many thanks in advance for your help! > > > > Best regards, > > Sebastian. > > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > http://lists.bx.psu.edu/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/ > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] . Output file to history (Sebastian Luna Valero)
Hi Sebastian, I am not an expert in galaxy developing. So the following content is just my personal opinion. I do not think it is possible to meet you requirement. I reckon Galaxy searches the certain folder, defined in the universal_wsgi.ini by a parameter called #collect_outputs_from = new_file_path,job_working_directory. On top of that, a job id is required, which is the reason in the dynamic multiple output job id is part of the file name. If your script.py cannot follow the rules, Galaxy for sure cannot recognize the outputs. Normally in this kind of case, I write an additional script to execute the script.py with its original parameter and do the file manipulation afterwards. Of course this is not ideal, but I do nothing about python. It may be possible to write some complicated python code within the command tag of your wrapper file to do this. I am here also waiting for experts' opinions/comments. Best regards! Jun On 3/21/14 9:28 AM, Sebastian Luna Valero wrote: > > Dear All, > > I am trying to add a new tool in Galaxy and I have the following problem. > > Let me explain a simplified example. Let's imagine that my script > works from CLI as follows: > > python script.py --input "input-file" --pattern "output-pattern" > > After processing "input-file", the script writes several output files > and their name is given according to "output-pattern". > > The number of output files depends on the content of "input-file". The > output files are written to the current working directory where > script.py is located. > > In the simplest scenario, I get only one output file in the working > directory. > > My problem is that I would like to see the output files in Galaxy's > history without modifying the CLI interface. > > To solve this problem, I have looked at the wiki page: > > https://wiki.galaxyproject.org/Admin/Tools/Multiple%20Output%20Files#N > umber_of_Output_datasets_cannot_be_determined_until_tool_run > > and the email here: > > http://dev.list.galaxyproject.org/Multiple-output-files-do-not-appear- > in-history-td4660470.html > > However, I think that mine is a different scenario. I do not want to > add new output parameters in the CLI. I would like Galaxy to bring > these output files to my history without modifying the CLI options, is > that possible? > > Many thanks in advance for your help! > > Best regards, > Sebastian. ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Output file to history
Hi Sebastian, I'm going to move this over to the galaxy-...@bx.psu.edu mailing list so that it reaches the development community effectively. I personally do not know the answer, but I am sure we can get you some help. Best, Jen Galaxy team https://wiki.galaxyproject.org/MailingLists On 3/21/14 9:28 AM, Sebastian Luna Valero wrote: Dear All, I am trying to add a new tool in Galaxy and I have the following problem. Let me explain a simplified example. Let's imagine that my script works from CLI as follows: python script.py --input "input-file" --pattern "output-pattern" After processing "input-file", the script writes several output files and their name is given according to "output-pattern". The number of output files depends on the content of "input-file". The output files are written to the current working directory where script.py is located. In the simplest scenario, I get only one output file in the working directory. My problem is that I would like to see the output files in Galaxy's history without modifying the CLI interface. To solve this problem, I have looked at the wiki page: https://wiki.galaxyproject.org/Admin/Tools/Multiple%20Output%20Files#Number_of_Output_datasets_cannot_be_determined_until_tool_run and the email here: http://dev.list.galaxyproject.org/Multiple-output-files-do-not-appear-in-history-td4660470.html However, I think that mine is a different scenario. I do not want to add new output parameters in the CLI. I would like Galaxy to bring these output files to my history without modifying the CLI options, is that possible? Many thanks in advance for your help! Best regards, Sebastian. ___ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/listinfo/galaxy-dev To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ -- Jennifer Hillman-Jackson http://galaxyproject.org ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
