Re: [galaxy-dev] RFC: remove trailing semicolons from command line -> Broken bowtie2_wrapper on some SGE systems

2013-10-17 Thread Peter Cock
On Thu, Oct 17, 2013 at 5:04 PM, John Chilton wrote: > On Thu, Oct 17, 2013 at 5:49 AM, Peter Cock wrote: >> On Thu, Oct 17, 2013 at 11:36 AM, John Chilton wrote: >> >> (And with the job splitting not being enabled by default, I am >> aware that I am in a relatively small group of Galaxy admins

Re: [galaxy-dev] RFC: remove trailing semicolons from command line -> Broken bowtie2_wrapper on some SGE systems

2013-10-17 Thread John Chilton
On Thu, Oct 17, 2013 at 5:49 AM, Peter Cock wrote: > On Thu, Oct 17, 2013 at 11:36 AM, John Chilton wrote: >> I broke the TaskWrapper last week with my exit code handling "fix", >> the double semi-colon thing you are seeing there. Your fix would break >> non-task split jobs so that is probably th

Re: [galaxy-dev] RFC: remove trailing semicolons from command line -> Broken bowtie2_wrapper on some SGE systems

2013-10-17 Thread Peter Cock
On Thu, Oct 17, 2013 at 11:36 AM, John Chilton wrote: > I broke the TaskWrapper last week with my exit code handling "fix", > the double semi-colon thing you are seeing there. Your fix would break > non-task split jobs so that is probably the problem(?) Hopefully? Want > to revert 8e001dc9675c and

Re: [galaxy-dev] RFC: remove trailing semicolons from command line -> Broken bowtie2_wrapper on some SGE systems

2013-10-17 Thread John Chilton
I broke the TaskWrapper last week with my exit code handling "fix", the double semi-colon thing you are seeing there. Your fix would break non-task split jobs so that is probably the problem(?) Hopefully? Want to revert 8e001dc9675c and pull in the changeset I just pushed out. Otherwise, I will te

Re: [galaxy-dev] RFC: remove trailing semicolons from command line -> Broken bowtie2_wrapper on some SGE systems

2013-10-17 Thread Peter Cock
On Thu, Oct 17, 2013 at 10:53 AM, Peter Cock wrote: > Hi John, > > Is all your semi-colon fixing on the trunk? I've found another bug in > this area (patch below), which is showing up with task splitting under > SGE. > > ... > > Note the repeated semi-colon, which causes the child jobs to fail (pa

Re: [galaxy-dev] RFC: remove trailing semicolons from command line -> Broken bowtie2_wrapper on some SGE systems

2013-10-17 Thread Peter Cock
Hi John, Is all your semi-colon fixing on the trunk? I've found another bug in this area (patch below), which is showing up with task splitting under SGE. e.g. this job is meant to be running and merging 5 BLAST XML files, stderr: nothing to merge for /mnt/galaxy/galaxy-central/database/files/00

Re: [galaxy-dev] RFC: remove trailing semicolons from command line -> Broken bowtie2_wrapper on some SGE systems

2013-10-11 Thread John Chilton
Hey All, Patch available with test case, let me know what I broke :). https://bitbucket.org/jmchilton/galaxy-central-multi-input-tool-fixes-2/commits/e0016057e1648d7f4d29fbaf14252cbefff6d0bd -John On Fri, Oct 11, 2013 at 10:37 AM, Björn Grüning wrote: > Am Freitag, den 11.10.2013, 09:18 -0500

Re: [galaxy-dev] RFC: remove trailing semicolons from command line -> Broken bowtie2_wrapper on some SGE systems

2013-10-11 Thread Björn Grüning
Am Freitag, den 11.10.2013, 09:18 -0500 schrieb John Chilton: > My plan is to refactor build_command_line out of > lib/galaxy/jobs/runners/__init__.py, into its own module and add unit > tests for this functionality. I'll issue a pull request and let people > test it. I am also testing Nicola's PR,

Re: [galaxy-dev] RFC: remove trailing semicolons from command line -> Broken bowtie2_wrapper on some SGE systems

2013-10-11 Thread Nicola Soranzo
I already fixed it one week ago: https://bitbucket.org/nsoranzo/galaxy-central-tools/commits/74b5fd2318d33b76382b052eae6a28b035dc075f but then Ross committed a similar fix to galaxy-central, I merged it in my pull request and then Ross reverted the fix! This is the reason for rebasing pull req

Re: [galaxy-dev] RFC: remove trailing semicolons from command line -> Broken bowtie2_wrapper on some SGE systems

2013-10-11 Thread Björn Grüning
Hi Nate, > Hi Björn, > > Are you certain that SGE is responsible? I ran in to the same thing with > bowtie2 with other runners. > The double semicolon is invalid syntax in Bourne-compatible shells and so the > tool ought to be fixed not to produce them (it happens under certain > combinatio

Re: [galaxy-dev] RFC: remove trailing semicolons from command line -> Broken bowtie2_wrapper on some SGE systems

2013-10-11 Thread John Chilton
My plan is to refactor build_command_line out of lib/galaxy/jobs/runners/__init__.py, into its own module and add unit tests for this functionality. I'll issue a pull request and let people test it. I am also testing Nicola's PR, I will approve it shortly. Nate yelled at me for committing code yes

Re: [galaxy-dev] RFC: remove trailing semicolons from command line -> Broken bowtie2_wrapper on some SGE systems

2013-10-11 Thread Björn Grüning
Ah now I got it ... there are two build_command_line functions. lib/galaxy/tools/__init__.py and lib/galaxy/jobs/runners/__init__.py. And one is adding a trailing ';' So either stripping the trailing semicolon, right after creating the command line from cheetah (patch was attached) or adding more

Re: [galaxy-dev] RFC: remove trailing semicolons from command line -> Broken bowtie2_wrapper on some SGE systems

2013-10-11 Thread John Chilton
If the tool produces a trailing semi-colon (and without Nicola's patch the bowtie 2 tool definitely could), then I bet build_command_line() is going to be what is adding the second semi-colon. It has not check that there isn't a semi-colon at the end of the command before adding one and extending t

Re: [galaxy-dev] RFC: remove trailing semicolons from command line -> Broken bowtie2_wrapper on some SGE systems

2013-10-11 Thread Björn Grüning
Hi Nicola, interesting you fixed bowtie in the same way than we did. But somehow I would like to strip a trailing ';' by default in Galaxy, when it does not break anything else. Thanks for sharing! Bjoern > Il giorno ven, 11/10/2013 alle 01.10 +0200, Björn Grüning ha scritto: > > Dear all, > >

Re: [galaxy-dev] RFC: remove trailing semicolons from command line -> Broken bowtie2_wrapper on some SGE systems

2013-10-11 Thread Nicola Soranzo
Il giorno ven, 11/10/2013 alle 01.10 +0200, Björn Grüning ha scritto: > Dear all, > > some SGE instances will add automatically an semicolon add the end of > each command, resulting in a disrupted job, because ';;' are not > allowed. > > The latest changes to the Bowtie2 wrapper resulting in suc

Re: [galaxy-dev] RFC: remove trailing semicolons from command line -> Broken bowtie2_wrapper on some SGE systems

2013-10-11 Thread John Chilton
Opps, sent that last e-mail before I meant to, gmail is acting very buggy for me this morning. I just meant to say I would modify build_command_line in lib/galaxy/jobs/runners/__init__.py instead. It seems like there is a lot of logic in there for this kind of thing already. I will create a Trello

Re: [galaxy-dev] RFC: remove trailing semicolons from command line -> Broken bowtie2_wrapper on some SGE systems

2013-10-11 Thread Nate Coraor
Hi Björn, Are you certain that SGE is responsible? I ran in to the same thing with bowtie2 with other runners. The double semicolon is invalid syntax in Bourne-compatible shells and so the tool ought to be fixed not to produce them (it happens under certain combinations of options). Have you

Re: [galaxy-dev] RFC: remove trailing semicolons from command line -> Broken bowtie2_wrapper on some SGE systems

2013-10-11 Thread John Chilton
This sounds like a good way to solve the problem, I guess I would modify On Thu, Oct 10, 2013 at 6:10 PM, Björn Grüning wrote: > > Dear all, > > some SGE instances will add automatically an semicolon add the end of > each command, resulting in a disrupted job, because ';;' are not > allowed. > >

[galaxy-dev] RFC: remove trailing semicolons from command line -> Broken bowtie2_wrapper on some SGE systems

2013-10-10 Thread Björn Grüning
Dear all, some SGE instances will add automatically an semicolon add the end of each command, resulting in a disrupted job, because ';;' are not allowed. The latest changes to the Bowtie2 wrapper resulting in such a case and a crash in our instance. An easy solution would be to fix the wrapper a