Re: [galaxy-dev] card 79: Split large jobs over multiple nodes for processing
On Wed, Feb 6, 2013 at 11:43 PM, alex.khassa...@csiro.au wrote: Hi All, Can anybody please add a few words on how can we use the “initial implementation” which “ exists in the tasks framework”? -Alex To enable this, set use_tasked_jobs = True in your universe_wsgi.ini file. The tools must also be configured to allow this via the parallelism tag. Many of my tools do this, for example see the NCBI BLAST+ wrappers in the tool shed. Additionally the data file formats must support being split, or being merged - which is done via Python code in the Galaxy datatype definition (see the split and merge methods in lib/galaxy/datatypes/*.py). Some other relevant Python code is in lib/galaxy/jobs/splitters/*.py Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] card 79: Split large jobs over multiple nodes for processing
Thanks Peter. I see, parallelism works on a single large file by splitting it and using multiple instances to process the bits in parallel. In our case we use 'composite' data type, simply an array of input files and we would like to process them in parallel, instead of having a 'foreach' loop in the tool wrapper. Is it possible? We are looking at CloudMan for creating a cluster in Galaxy now. -Alex -Original Message- From: Peter Cock [mailto:p.j.a.c...@googlemail.com] Sent: Thursday, 7 February 2013 9:09 PM To: Khassapov, Alex (CSIRO IMT, Clayton) Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] card 79: Split large jobs over multiple nodes for processing On Wed, Feb 6, 2013 at 11:43 PM, alex.khassa...@csiro.au wrote: Hi All, Can anybody please add a few words on how can we use the initial implementation which exists in the tasks framework? -Alex To enable this, set use_tasked_jobs = True in your universe_wsgi.ini file. The tools must also be configured to allow this via the parallelism tag. Many of my tools do this, for example see the NCBI BLAST+ wrappers in the tool shed. Additionally the data file formats must support being split, or being merged - which is done via Python code in the Galaxy datatype definition (see the split and merge methods in lib/galaxy/datatypes/*.py). Some other relevant Python code is in lib/galaxy/jobs/splitters/*.py Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] card 79: Split large jobs over multiple nodes for processing
Hi All, Can anybody please add a few words on how can we use the “initial implementation” which “ exists in the tasks framework”? -Alex From: Trello [mailto:do-not-re...@trello.com] Sent: Wednesday, 6 February 2013 10:58 AM To: Khassapov, Alex (CSIRO IMT, Clayton) Subject: 4 new notifications on the board Galaxy: Development since 5:56 PM (Tuesday) [https://trello.com/images/logo-s.png] Notifications On Galaxy: Developmenthttps://trello.com/board/galaxy-development/506338ce32ae458f6d15e4b3 [https://trello-avatars.s3.amazonaws.com/a6e93a63989ab71cd87ade0165a04b08/30.png]James Taylor added [https://trello-avatars.s3.amazonaws.com/d0f1bba8eb293d305140421271c383a9/30.png] Dannon Baker to the card 79: Split large jobs over multiple nodes for processinghttps://trello.com/card/79-split-large-jobs-over-multiple-nodes-for-processing/506338ce32ae458f6d15e4b3/411 on Galaxy: Developmenthttps://trello.com/board/galaxy-development/506338ce32ae458f6d15e4b3 [https://trello-avatars.s3.amazonaws.com/a6e93a63989ab71cd87ade0165a04b08/30.png]James Taylor commented on the card 79: Split large jobs over multiple nodes for processinghttps://trello.com/card/79-split-large-jobs-over-multiple-nodes-for-processing/506338ce32ae458f6d15e4b3/411 on Galaxy: Developmenthttps://trello.com/board/galaxy-development/506338ce32ae458f6d15e4b3 An initial implementation exists in the tasks framework. [https://trello-avatars.s3.amazonaws.com/a6e93a63989ab71cd87ade0165a04b08/30.png]James Taylor moved the card 79: Split large jobs over multiple nodes for processinghttps://trello.com/card/79-split-large-jobs-over-multiple-nodes-for-processing/506338ce32ae458f6d15e4b3/411 to Complete on Galaxy: Developmenthttps://trello.com/board/galaxy-development/506338ce32ae458f6d15e4b3 [https://trello-avatars.s3.amazonaws.com/a6e93a63989ab71cd87ade0165a04b08/30.png]James Taylor moved the card 137: allow multiple=true in input param fields of type datahttps://trello.com/card/137-allow-multiple-true-in-input-param-fields-of-type-data/506338ce32ae458f6d15e4b3/292 to Pull Requests / Patches on Galaxy: Developmenthttps://trello.com/board/galaxy-development/506338ce32ae458f6d15e4b3 Change how often you get email on your account pagehttps://trello.com/my/account. Follow Trello on Twitterhttps://twitter.com/intent/follow?user_id=360831528 and Facebookhttps://www.facebook.com/TrelloApp. Get the Trello app for iPhonehttp://itunes.com/apps/trello or Androidhttps://play.google.com/store/apps/details?id=com.trello. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
