Re: [galaxy-dev] Looks like actual user breaks splitting

2011-11-08 Thread Nate Coraor
This converts line endings to UNIX line endings, so in most cases it wouldn't 
break anything to not happen, however, if the uploaded file had DOS line 
endings, the uploaded file content would be wrong.

The best way to handle this is probably to change the ownership of the uploaded 
temporary file.  The code needs to be very careful not to change ownership of 
anything not uploaded by an actual user (i.e. files for linking or even 
copied in from the library side).

--nate

On Nov 7, 2011, at 11:12 AM, Chorny, Ilya wrote:

 I am not sure what the copy is for. I did not add the move/copy I just 
 changed shutil.mv to os.system(mv) because it has less baggage.  In fact if I 
 comment it out the upload still works.
 
 Ilya
 
 -Original Message-
 From: Nate Coraor [mailto:n...@bx.psu.edu] 
 Sent: Monday, November 07, 2011 7:57 AM
 To: Chorny, Ilya
 Cc: galaxy-dev@lists.bx.psu.edu
 Subject: Re: [galaxy-dev] Looks like actual user breaks splitting
 
 On Nov 4, 2011, at 8:03 PM, Chorny, Ilya wrote:
 
 Looks like the mv is something I added and forgot about. I changed it to 
 shutil.copyfile and it works fine. This also has something to do with the 
 isilon. I pushed it to the actual user repository  so you can check it out.
 
 Okay, I'll have a look.  What's the copy for?  We try to avoid moving bits as 
 much as possible during upload since it can slow down the entire upload tool 
 run pretty significantly if there are multiple copies taking place.
 
 --nate
 
 
 Have a good weekend. 
 
 Thanks,
 
 Ilya
 
 
 -Original Message-
 From: galaxy-dev-boun...@lists.bx.psu.edu 
 [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya
 Sent: Friday, November 04, 2011 4:50 PM
 To: Nate Coraor (n...@bx.psu.edu)
 Cc: galaxy-dev@lists.bx.psu.edu
 Subject: Re: [galaxy-dev] Looks like actual user breaks splitting
 
 I am getting a strange error when I am uploading a file
 
 mv: inter-device move failed: 
 `/illumina/scratch/Swim/galaxy/tmpk2sbZl' to 
 `/home/galaxy/ichorny/galaxy-central/database/tmp/upload_file_data_isA
 nwO'; unable to remove target: Permission denied
 
 I figured out that the mv is coming from
 
 lib/galaxy/datatypes/sniff.py
 
 if in_place:
   fnamemode = stat.S_IMODE( os.stat(fname).st_mode )
   os.system('mv %s %s' %(temp_name,fname))
   # Return number of lines in file.
   return ( i + 1, None )
   else:
   return ( i + 1, temp_name )
 
 Any thoughts on what might be going on? If I comment out the move the upload 
 works just fine.
 
 Thanks,
 
 Ilya
 
 -Original Message-
 From: Chorny, Ilya
 Sent: Friday, November 04, 2011 11:11 AM
 To: Chorny, Ilya; Nate Coraor (n...@bx.psu.edu); Duddy, John
 Cc: galaxy-dev@lists.bx.psu.edu
 Subject: RE: [galaxy-dev] Looks like actual user breaks splitting
 
 Well I think I figured out why we get OSError: [Errno 39] Directory not 
 empty: './database/job_working_directory/451'. I changed it to an rm -rf and 
 got the following error:
 
 `./database/job_working_directory/455/.nfs000161e986fc0068' ': 
 Device or resource busy
 
 It might have something to do with our mounts being Isilon. Any thought on 
 how to deal with this?
 
 
 -Original Message-
 From: galaxy-dev-boun...@lists.bx.psu.edu 
 [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya
 Sent: Friday, November 04, 2011 10:50 AM
 To: Nate Coraor (n...@bx.psu.edu)
 Cc: galaxy-dev@lists.bx.psu.edu
 Subject: Re: [galaxy-dev] Looks like actual user breaks splitting
 
 I came up with a fix for User killed running job, but error encountered 
 removing from DRM queue: 'DRMAAJobRunner' object has no attribute 
 'job_user_uid' and pushed it. 
 
 -Original Message-
 From: Chorny, Ilya
 Sent: Friday, November 04, 2011 10:24 AM
 To: Chorny, Ilya; Nate Coraor (n...@bx.psu.edu)
 Cc: galaxy-dev@lists.bx.psu.edu
 Subject: RE: [galaxy-dev] Looks like actual user breaks splitting
 
 I am also getting the following error in the log file which makes no sense 
 because the directory is empty. I will try to track it down. Thoughts?
 
 Ilya
 Traceback (most recent call last):
 File /home/galaxy/ichorny/galaxy-central/lib/galaxy/jobs/__init__.py, line 
 732, in cleanup
   shutil.rmtree( self.working_directory )  File 
 /usr/lib64/python2.6/shutil.py, line 225, in rmtree
   onerror(os.rmdir, path, sys.exc_info())  File 
 /usr/lib64/python2.6/shutil.py, line 223, in rmtree
   os.rmdir(path)
 OSError: [Errno 39] Directory not empty: 
 './database/job_working_directory/451'
 
 
 -Original Message-
 From: galaxy-dev-boun...@lists.bx.psu.edu 
 [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya
 Sent: Friday, November 04, 2011 9:07 AM
 To: Duddy, John; Nate Coraor (n...@bx.psu.edu)
 Cc: galaxy-dev@lists.bx.psu.edu
 Subject: Re: [galaxy-dev] Looks like actual user breaks splitting
 
 Nate,
 
 I get the following error when I try to kill a job.
 
 User killed running job, but error encountered removing from DRM

Re: [galaxy-dev] Looks like actual user breaks splitting

2011-11-07 Thread Nate Coraor
On Nov 4, 2011, at 8:03 PM, Chorny, Ilya wrote:

 Looks like the mv is something I added and forgot about. I changed it to 
 shutil.copyfile and it works fine. This also has something to do with the 
 isilon. I pushed it to the actual user repository  so you can check it out.

Okay, I'll have a look.  What's the copy for?  We try to avoid moving bits as 
much as possible during upload since it can slow down the entire upload tool 
run pretty significantly if there are multiple copies taking place.

--nate

 
 Have a good weekend. 
 
 Thanks,
 
 Ilya
 
 
 -Original Message-
 From: galaxy-dev-boun...@lists.bx.psu.edu 
 [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya
 Sent: Friday, November 04, 2011 4:50 PM
 To: Nate Coraor (n...@bx.psu.edu)
 Cc: galaxy-dev@lists.bx.psu.edu
 Subject: Re: [galaxy-dev] Looks like actual user breaks splitting
 
 I am getting a strange error when I am uploading a file
 
 mv: inter-device move failed: `/illumina/scratch/Swim/galaxy/tmpk2sbZl' to 
 `/home/galaxy/ichorny/galaxy-central/database/tmp/upload_file_data_isAnwO'; 
 unable to remove target: Permission denied
 
 I figured out that the mv is coming from 
 
 lib/galaxy/datatypes/sniff.py
 
 if in_place:
fnamemode = stat.S_IMODE( os.stat(fname).st_mode )
os.system('mv %s %s' %(temp_name,fname))
# Return number of lines in file.
return ( i + 1, None )
else:
return ( i + 1, temp_name )
 
 Any thoughts on what might be going on? If I comment out the move the upload 
 works just fine.
 
 Thanks,
 
 Ilya
 
 -Original Message-
 From: Chorny, Ilya
 Sent: Friday, November 04, 2011 11:11 AM
 To: Chorny, Ilya; Nate Coraor (n...@bx.psu.edu); Duddy, John
 Cc: galaxy-dev@lists.bx.psu.edu
 Subject: RE: [galaxy-dev] Looks like actual user breaks splitting
 
 Well I think I figured out why we get OSError: [Errno 39] Directory not 
 empty: './database/job_working_directory/451'. I changed it to an rm -rf and 
 got the following error:
 
 `./database/job_working_directory/455/.nfs000161e986fc0068' ': Device 
 or resource busy
 
 It might have something to do with our mounts being Isilon. Any thought on 
 how to deal with this?
 
 
 -Original Message-
 From: galaxy-dev-boun...@lists.bx.psu.edu 
 [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya
 Sent: Friday, November 04, 2011 10:50 AM
 To: Nate Coraor (n...@bx.psu.edu)
 Cc: galaxy-dev@lists.bx.psu.edu
 Subject: Re: [galaxy-dev] Looks like actual user breaks splitting
 
 I came up with a fix for User killed running job, but error encountered 
 removing from DRM queue: 'DRMAAJobRunner' object has no attribute 
 'job_user_uid' and pushed it. 
 
 -Original Message-
 From: Chorny, Ilya
 Sent: Friday, November 04, 2011 10:24 AM
 To: Chorny, Ilya; Nate Coraor (n...@bx.psu.edu)
 Cc: galaxy-dev@lists.bx.psu.edu
 Subject: RE: [galaxy-dev] Looks like actual user breaks splitting
 
 I am also getting the following error in the log file which makes no sense 
 because the directory is empty. I will try to track it down. Thoughts?
 
 Ilya
 Traceback (most recent call last):
  File /home/galaxy/ichorny/galaxy-central/lib/galaxy/jobs/__init__.py, line 
 732, in cleanup
shutil.rmtree( self.working_directory )
  File /usr/lib64/python2.6/shutil.py, line 225, in rmtree
onerror(os.rmdir, path, sys.exc_info())
  File /usr/lib64/python2.6/shutil.py, line 223, in rmtree
os.rmdir(path)
 OSError: [Errno 39] Directory not empty: 
 './database/job_working_directory/451'
 
 
 -Original Message-
 From: galaxy-dev-boun...@lists.bx.psu.edu 
 [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya
 Sent: Friday, November 04, 2011 9:07 AM
 To: Duddy, John; Nate Coraor (n...@bx.psu.edu)
 Cc: galaxy-dev@lists.bx.psu.edu
 Subject: Re: [galaxy-dev] Looks like actual user breaks splitting
 
 Nate,
 
 I get the following error when I try to kill a job.
 
 User killed running job, but error encountered removing from DRM queue: 
 'DRMAAJobRunner' object has no attribute 'job_user_uid'.
 
 Ilya
 
 -Original Message-
 From: galaxy-dev-boun...@lists.bx.psu.edu 
 [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Duddy, John
 Sent: Thursday, November 03, 2011 3:55 PM
 To: Nate Coraor (n...@bx.psu.edu)
 Cc: galaxy-dev@lists.bx.psu.edu
 Subject: Re: [galaxy-dev] Looks like actual user breaks splitting
 
 I just came to that conclusion myself. If you need me to do anything, let me 
 know - but it sounds like you have a handle on it.
 
 Muchas gracias.
 
 John Duddy
 Sr. Staff Software Engineer
 Illumina, Inc.
 9885 Towne Centre Drive
 San Diego, CA 92121
 Tel: 858-736-3584
 E-mail: jdu...@illumina.com
 
 
 -Original Message-
 From: Nate Coraor (n...@bx.psu.edu) [mailto:n...@bx.psu.edu]
 Sent: Thursday, November 03, 2011 3:53 PM
 To: Duddy, John
 Cc: galaxy-dev@lists.bx.psu.edu
 Subject: Re: [galaxy-dev] Looks like actual user breaks splitting
 
 Nate Coraor

Re: [galaxy-dev] Looks like actual user breaks splitting

2011-11-07 Thread Chorny, Ilya
I am not sure what the copy is for. I did not add the move/copy I just changed 
shutil.mv to os.system(mv) because it has less baggage.  In fact if I comment 
it out the upload still works.

Ilya

-Original Message-
From: Nate Coraor [mailto:n...@bx.psu.edu] 
Sent: Monday, November 07, 2011 7:57 AM
To: Chorny, Ilya
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Looks like actual user breaks splitting

On Nov 4, 2011, at 8:03 PM, Chorny, Ilya wrote:

 Looks like the mv is something I added and forgot about. I changed it to 
 shutil.copyfile and it works fine. This also has something to do with the 
 isilon. I pushed it to the actual user repository  so you can check it out.

Okay, I'll have a look.  What's the copy for?  We try to avoid moving bits as 
much as possible during upload since it can slow down the entire upload tool 
run pretty significantly if there are multiple copies taking place.

--nate

 
 Have a good weekend. 
 
 Thanks,
 
 Ilya
 
 
 -Original Message-
 From: galaxy-dev-boun...@lists.bx.psu.edu 
 [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya
 Sent: Friday, November 04, 2011 4:50 PM
 To: Nate Coraor (n...@bx.psu.edu)
 Cc: galaxy-dev@lists.bx.psu.edu
 Subject: Re: [galaxy-dev] Looks like actual user breaks splitting
 
 I am getting a strange error when I am uploading a file
 
 mv: inter-device move failed: 
 `/illumina/scratch/Swim/galaxy/tmpk2sbZl' to 
 `/home/galaxy/ichorny/galaxy-central/database/tmp/upload_file_data_isA
 nwO'; unable to remove target: Permission denied
 
 I figured out that the mv is coming from
 
 lib/galaxy/datatypes/sniff.py
 
 if in_place:
fnamemode = stat.S_IMODE( os.stat(fname).st_mode )
os.system('mv %s %s' %(temp_name,fname))
# Return number of lines in file.
return ( i + 1, None )
else:
return ( i + 1, temp_name )
 
 Any thoughts on what might be going on? If I comment out the move the upload 
 works just fine.
 
 Thanks,
 
 Ilya
 
 -Original Message-
 From: Chorny, Ilya
 Sent: Friday, November 04, 2011 11:11 AM
 To: Chorny, Ilya; Nate Coraor (n...@bx.psu.edu); Duddy, John
 Cc: galaxy-dev@lists.bx.psu.edu
 Subject: RE: [galaxy-dev] Looks like actual user breaks splitting
 
 Well I think I figured out why we get OSError: [Errno 39] Directory not 
 empty: './database/job_working_directory/451'. I changed it to an rm -rf and 
 got the following error:
 
 `./database/job_working_directory/455/.nfs000161e986fc0068' ': 
 Device or resource busy
 
 It might have something to do with our mounts being Isilon. Any thought on 
 how to deal with this?
 
 
 -Original Message-
 From: galaxy-dev-boun...@lists.bx.psu.edu 
 [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya
 Sent: Friday, November 04, 2011 10:50 AM
 To: Nate Coraor (n...@bx.psu.edu)
 Cc: galaxy-dev@lists.bx.psu.edu
 Subject: Re: [galaxy-dev] Looks like actual user breaks splitting
 
 I came up with a fix for User killed running job, but error encountered 
 removing from DRM queue: 'DRMAAJobRunner' object has no attribute 
 'job_user_uid' and pushed it. 
 
 -Original Message-
 From: Chorny, Ilya
 Sent: Friday, November 04, 2011 10:24 AM
 To: Chorny, Ilya; Nate Coraor (n...@bx.psu.edu)
 Cc: galaxy-dev@lists.bx.psu.edu
 Subject: RE: [galaxy-dev] Looks like actual user breaks splitting
 
 I am also getting the following error in the log file which makes no sense 
 because the directory is empty. I will try to track it down. Thoughts?
 
 Ilya
 Traceback (most recent call last):
  File /home/galaxy/ichorny/galaxy-central/lib/galaxy/jobs/__init__.py, line 
 732, in cleanup
shutil.rmtree( self.working_directory )  File 
 /usr/lib64/python2.6/shutil.py, line 225, in rmtree
onerror(os.rmdir, path, sys.exc_info())  File 
 /usr/lib64/python2.6/shutil.py, line 223, in rmtree
os.rmdir(path)
 OSError: [Errno 39] Directory not empty: 
 './database/job_working_directory/451'
 
 
 -Original Message-
 From: galaxy-dev-boun...@lists.bx.psu.edu 
 [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya
 Sent: Friday, November 04, 2011 9:07 AM
 To: Duddy, John; Nate Coraor (n...@bx.psu.edu)
 Cc: galaxy-dev@lists.bx.psu.edu
 Subject: Re: [galaxy-dev] Looks like actual user breaks splitting
 
 Nate,
 
 I get the following error when I try to kill a job.
 
 User killed running job, but error encountered removing from DRM queue: 
 'DRMAAJobRunner' object has no attribute 'job_user_uid'.
 
 Ilya
 
 -Original Message-
 From: galaxy-dev-boun...@lists.bx.psu.edu 
 [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Duddy, John
 Sent: Thursday, November 03, 2011 3:55 PM
 To: Nate Coraor (n...@bx.psu.edu)
 Cc: galaxy-dev@lists.bx.psu.edu
 Subject: Re: [galaxy-dev] Looks like actual user breaks splitting
 
 I just came to that conclusion myself. If you need me to do anything, let me 
 know - but it sounds like you have a handle on it.
 
 Muchas

Re: [galaxy-dev] Looks like actual user breaks splitting

2011-11-04 Thread Chorny, Ilya
I am getting a strange error when I am uploading a file

mv: inter-device move failed: `/illumina/scratch/Swim/galaxy/tmpk2sbZl' to 
`/home/galaxy/ichorny/galaxy-central/database/tmp/upload_file_data_isAnwO'; 
unable to remove target: Permission denied

I figured out that the mv is coming from 
 
lib/galaxy/datatypes/sniff.py

if in_place:
fnamemode = stat.S_IMODE( os.stat(fname).st_mode )
os.system('mv %s %s' %(temp_name,fname))
# Return number of lines in file.
return ( i + 1, None )
else:
return ( i + 1, temp_name )

Any thoughts on what might be going on? If I comment out the move the upload 
works just fine.

Thanks,

Ilya

-Original Message-
From: Chorny, Ilya 
Sent: Friday, November 04, 2011 11:11 AM
To: Chorny, Ilya; Nate Coraor (n...@bx.psu.edu); Duddy, John
Cc: galaxy-dev@lists.bx.psu.edu
Subject: RE: [galaxy-dev] Looks like actual user breaks splitting

Well I think I figured out why we get OSError: [Errno 39] Directory not empty: 
'./database/job_working_directory/451'. I changed it to an rm -rf and got the 
following error:

`./database/job_working_directory/455/.nfs000161e986fc0068' ': Device 
or resource busy

It might have something to do with our mounts being Isilon. Any thought on how 
to deal with this?


-Original Message-
From: galaxy-dev-boun...@lists.bx.psu.edu 
[mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya
Sent: Friday, November 04, 2011 10:50 AM
To: Nate Coraor (n...@bx.psu.edu)
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Looks like actual user breaks splitting

I came up with a fix for User killed running job, but error encountered 
removing from DRM queue: 'DRMAAJobRunner' object has no attribute 
'job_user_uid' and pushed it. 

-Original Message-
From: Chorny, Ilya
Sent: Friday, November 04, 2011 10:24 AM
To: Chorny, Ilya; Nate Coraor (n...@bx.psu.edu)
Cc: galaxy-dev@lists.bx.psu.edu
Subject: RE: [galaxy-dev] Looks like actual user breaks splitting

I am also getting the following error in the log file which makes no sense 
because the directory is empty. I will try to track it down. Thoughts?

Ilya
Traceback (most recent call last):
  File /home/galaxy/ichorny/galaxy-central/lib/galaxy/jobs/__init__.py, line 
732, in cleanup
shutil.rmtree( self.working_directory )
  File /usr/lib64/python2.6/shutil.py, line 225, in rmtree
onerror(os.rmdir, path, sys.exc_info())
  File /usr/lib64/python2.6/shutil.py, line 223, in rmtree
os.rmdir(path)
OSError: [Errno 39] Directory not empty: './database/job_working_directory/451'


-Original Message-
From: galaxy-dev-boun...@lists.bx.psu.edu 
[mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya
Sent: Friday, November 04, 2011 9:07 AM
To: Duddy, John; Nate Coraor (n...@bx.psu.edu)
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Looks like actual user breaks splitting

Nate,

I get the following error when I try to kill a job.

User killed running job, but error encountered removing from DRM queue: 
'DRMAAJobRunner' object has no attribute 'job_user_uid'.

Ilya

-Original Message-
From: galaxy-dev-boun...@lists.bx.psu.edu 
[mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Duddy, John
Sent: Thursday, November 03, 2011 3:55 PM
To: Nate Coraor (n...@bx.psu.edu)
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Looks like actual user breaks splitting

I just came to that conclusion myself. If you need me to do anything, let me 
know - but it sounds like you have a handle on it.

Muchas gracias.

John Duddy
Sr. Staff Software Engineer
Illumina, Inc.
9885 Towne Centre Drive
San Diego, CA 92121
Tel: 858-736-3584
E-mail: jdu...@illumina.com


-Original Message-
From: Nate Coraor (n...@bx.psu.edu) [mailto:n...@bx.psu.edu]
Sent: Thursday, November 03, 2011 3:53 PM
To: Duddy, John
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Looks like actual user breaks splitting

Nate Coraor (n...@bx.psu.edu) wrote:
 Duddy, John wrote:
  I'm not following you - it's been 6 months since I wrote that code 
  ;-}
 
 I know the feeling!
 
  IT looks to me like a DatasetPath() object is always placed in that array, 
  and with one exception near then, it looks like the change I made generates 
  those objects the same way.
 
 It's creating a dict in self.output_dataset_paths, and that dict looks 
 like this when outputs_to_working_directory = False:
 
 { output_param_name : [ HDA, DatasetPath ], ... }
 
 And this when True:
 
 { output_param_name : [ Dataset, DatasetPath ], ... }
 
  Do you have a stack trace for the merge problem I can look at?
 
 If you put this in do_merge()'s except block:
 
 log.exception( stdout )
 
 You'll get:
 
 Traceback (most recent call last):
   File 
 /space/nate/galaxy-central-ichorny/lib/galaxy/jobs/splitters/multi.py, line 
 128, in do_merge
 output_type = outputs[output][0].datatype
 AttributeError

Re: [galaxy-dev] Looks like actual user breaks splitting

2011-11-04 Thread Chorny, Ilya
Looks like the mv is something I added and forgot about. I changed it to 
shutil.copyfile and it works fine. This also has something to do with the 
isilon. I pushed it to the actual user repository  so you can check it out.

Have a good weekend. 

Thanks,

Ilya


-Original Message-
From: galaxy-dev-boun...@lists.bx.psu.edu 
[mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya
Sent: Friday, November 04, 2011 4:50 PM
To: Nate Coraor (n...@bx.psu.edu)
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Looks like actual user breaks splitting

I am getting a strange error when I am uploading a file

mv: inter-device move failed: `/illumina/scratch/Swim/galaxy/tmpk2sbZl' to 
`/home/galaxy/ichorny/galaxy-central/database/tmp/upload_file_data_isAnwO'; 
unable to remove target: Permission denied

I figured out that the mv is coming from 
 
lib/galaxy/datatypes/sniff.py

if in_place:
fnamemode = stat.S_IMODE( os.stat(fname).st_mode )
os.system('mv %s %s' %(temp_name,fname))
# Return number of lines in file.
return ( i + 1, None )
else:
return ( i + 1, temp_name )

Any thoughts on what might be going on? If I comment out the move the upload 
works just fine.

Thanks,

Ilya

-Original Message-
From: Chorny, Ilya
Sent: Friday, November 04, 2011 11:11 AM
To: Chorny, Ilya; Nate Coraor (n...@bx.psu.edu); Duddy, John
Cc: galaxy-dev@lists.bx.psu.edu
Subject: RE: [galaxy-dev] Looks like actual user breaks splitting

Well I think I figured out why we get OSError: [Errno 39] Directory not empty: 
'./database/job_working_directory/451'. I changed it to an rm -rf and got the 
following error:

`./database/job_working_directory/455/.nfs000161e986fc0068' ': Device 
or resource busy

It might have something to do with our mounts being Isilon. Any thought on how 
to deal with this?


-Original Message-
From: galaxy-dev-boun...@lists.bx.psu.edu 
[mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya
Sent: Friday, November 04, 2011 10:50 AM
To: Nate Coraor (n...@bx.psu.edu)
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Looks like actual user breaks splitting

I came up with a fix for User killed running job, but error encountered 
removing from DRM queue: 'DRMAAJobRunner' object has no attribute 
'job_user_uid' and pushed it. 

-Original Message-
From: Chorny, Ilya
Sent: Friday, November 04, 2011 10:24 AM
To: Chorny, Ilya; Nate Coraor (n...@bx.psu.edu)
Cc: galaxy-dev@lists.bx.psu.edu
Subject: RE: [galaxy-dev] Looks like actual user breaks splitting

I am also getting the following error in the log file which makes no sense 
because the directory is empty. I will try to track it down. Thoughts?

Ilya
Traceback (most recent call last):
  File /home/galaxy/ichorny/galaxy-central/lib/galaxy/jobs/__init__.py, line 
732, in cleanup
shutil.rmtree( self.working_directory )
  File /usr/lib64/python2.6/shutil.py, line 225, in rmtree
onerror(os.rmdir, path, sys.exc_info())
  File /usr/lib64/python2.6/shutil.py, line 223, in rmtree
os.rmdir(path)
OSError: [Errno 39] Directory not empty: './database/job_working_directory/451'


-Original Message-
From: galaxy-dev-boun...@lists.bx.psu.edu 
[mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya
Sent: Friday, November 04, 2011 9:07 AM
To: Duddy, John; Nate Coraor (n...@bx.psu.edu)
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Looks like actual user breaks splitting

Nate,

I get the following error when I try to kill a job.

User killed running job, but error encountered removing from DRM queue: 
'DRMAAJobRunner' object has no attribute 'job_user_uid'.

Ilya

-Original Message-
From: galaxy-dev-boun...@lists.bx.psu.edu 
[mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Duddy, John
Sent: Thursday, November 03, 2011 3:55 PM
To: Nate Coraor (n...@bx.psu.edu)
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Looks like actual user breaks splitting

I just came to that conclusion myself. If you need me to do anything, let me 
know - but it sounds like you have a handle on it.

Muchas gracias.

John Duddy
Sr. Staff Software Engineer
Illumina, Inc.
9885 Towne Centre Drive
San Diego, CA 92121
Tel: 858-736-3584
E-mail: jdu...@illumina.com


-Original Message-
From: Nate Coraor (n...@bx.psu.edu) [mailto:n...@bx.psu.edu]
Sent: Thursday, November 03, 2011 3:53 PM
To: Duddy, John
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Looks like actual user breaks splitting

Nate Coraor (n...@bx.psu.edu) wrote:
 Duddy, John wrote:
  I'm not following you - it's been 6 months since I wrote that code 
  ;-}
 
 I know the feeling!
 
  IT looks to me like a DatasetPath() object is always placed in that array, 
  and with one exception near then, it looks like the change I made generates 
  those objects the same way.
 
 It's creating a dict in self.output_dataset_paths, and that dict looks 
 like

Re: [galaxy-dev] Looks like actual user breaks splitting

2011-11-03 Thread Nate Coraor (n...@bx.psu.edu)
Hi John,

It looks like the first issue is related to the change from
get_output_fnames() - compute_outputs().  When
outputs_to_working_directory = False (default) this method
stores/returns a HistoryDatasetAssociation, but when True,
stores/returns a Dataset (the original method's behavior).  Thus,
accessing the object's .datatype attribute in the splitter's do_merge()
fails.

Thanks,
--nate

Duddy, John wrote:
 I'll submit a pull request shortly...
 
 John Duddy
 Sr. Staff Software Engineer
 Illumina, Inc.
 9885 Towne Centre Drive
 San Diego, CA 92121
 Tel: 858-736-3584
 E-mail: jdu...@illumina.com
 
 
 -Original Message-
 From: Nate Coraor (n...@bx.psu.edu) [mailto:n...@bx.psu.edu] 
 Sent: Wednesday, November 02, 2011 12:24 PM
 To: Duddy, John
 Cc: Chorny, Ilya; galaxy-dev@lists.bx.psu.edu
 Subject: Re: Looks like actual user breaks splitting
 
 John, Ilya,
 
 I get further with sequence type inputs but it looks like
 JobWrapper.get_output_datasets_and_fnames() is not returning the right
 thing when outputs_to_working_directory = True
 
 BTW, the base Data.split() method is broken after the updates to
 Sequence.split() since it wasn't updated to expect
 HistoryDatasetAssociations rather than filenames.  Could you take a look
 at that when you get a chance?
 
 --nate
 
 Duddy, John wrote:
  The datatype you are using does not define a split method. Are you working 
  with our in-progress gz type or fastqillumina?
  
  John Duddy
  Sr. Staff Software Engineer
  Illumina, Inc.
  9885 Towne Centre Drive
  San Diego, CA 92121
  Tel: 858-736-3584
  E-mail: jdu...@illumina.commailto:jdu...@illumina.com
  
  From: Chorny, Ilya
  Sent: Wednesday, November 02, 2011 11:50 AM
  To: Duddy, John
  Cc: Nate Coraor (n...@bx.psu.edu); galaxy-dev@lists.bx.psu.edu
  Subject: Looks like actual user breaks splitting
  
  Hey John,
  
  Any thoughts?
  
  Ilya
  
  Traceback (most recent call last):
File 
  /home/galaxy/ichorny/galaxy-central/lib/galaxy/jobs/runners/tasks.py, 
  line 73, in run_job
  tasks = splitter.do_split(job_wrapper)
File 
  /home/galaxy/ichorny/galaxy-central/lib/galaxy/jobs/splitters/multi.py, 
  line 73, in do_split
  input_type.split(input_datasets, get_new_working_directory_name, 
  parallel_settings)
File /home/galaxy/ichorny/galaxy-central/lib/galaxy/datatypes/data.py, 
  line 473, in split
  raise Exception(Text file splitting does not support multiple files)
  Exception: Text file splitting does not support multiple files
  
  Ilya Chorny Ph.D.
  Bioinformatics Scientist I
  Illumina, Inc.
  9885 Towne Centre Drive
  San Diego, CA 92121
  Work: 858.202.4582
  Email: icho...@illumina.commailto:icho...@illumina.com
  Website: www.illumina.comhttp://www.illumina.com
  
  
 
___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/


Re: [galaxy-dev] Looks like actual user breaks splitting

2011-11-03 Thread Duddy, John
I'm not following you - it's been 6 months since I wrote that code ;-}

IT looks to me like a DatasetPath() object is always placed in that array, and 
with one exception near then, it looks like the change I made generates those 
objects the same way.

Do you have a stack trace for the merge problem I can look at?

John Duddy
Sr. Staff Software Engineer
Illumina, Inc.
9885 Towne Centre Drive
San Diego, CA 92121
Tel: 858-736-3584
E-mail: jdu...@illumina.com


-Original Message-
From: Nate Coraor (n...@bx.psu.edu) [mailto:n...@bx.psu.edu] 
Sent: Thursday, November 03, 2011 2:22 PM
To: Duddy, John
Cc: Chorny, Ilya; galaxy-dev@lists.bx.psu.edu
Subject: Re: Looks like actual user breaks splitting

Hi John,

It looks like the first issue is related to the change from
get_output_fnames() - compute_outputs().  When
outputs_to_working_directory = False (default) this method
stores/returns a HistoryDatasetAssociation, but when True,
stores/returns a Dataset (the original method's behavior).  Thus,
accessing the object's .datatype attribute in the splitter's do_merge()
fails.

Thanks,
--nate

Duddy, John wrote:
 I'll submit a pull request shortly...
 
 John Duddy
 Sr. Staff Software Engineer
 Illumina, Inc.
 9885 Towne Centre Drive
 San Diego, CA 92121
 Tel: 858-736-3584
 E-mail: jdu...@illumina.com
 
 
 -Original Message-
 From: Nate Coraor (n...@bx.psu.edu) [mailto:n...@bx.psu.edu] 
 Sent: Wednesday, November 02, 2011 12:24 PM
 To: Duddy, John
 Cc: Chorny, Ilya; galaxy-dev@lists.bx.psu.edu
 Subject: Re: Looks like actual user breaks splitting
 
 John, Ilya,
 
 I get further with sequence type inputs but it looks like
 JobWrapper.get_output_datasets_and_fnames() is not returning the right
 thing when outputs_to_working_directory = True
 
 BTW, the base Data.split() method is broken after the updates to
 Sequence.split() since it wasn't updated to expect
 HistoryDatasetAssociations rather than filenames.  Could you take a look
 at that when you get a chance?
 
 --nate
 
 Duddy, John wrote:
  The datatype you are using does not define a split method. Are you working 
  with our in-progress gz type or fastqillumina?
  
  John Duddy
  Sr. Staff Software Engineer
  Illumina, Inc.
  9885 Towne Centre Drive
  San Diego, CA 92121
  Tel: 858-736-3584
  E-mail: jdu...@illumina.commailto:jdu...@illumina.com
  
  From: Chorny, Ilya
  Sent: Wednesday, November 02, 2011 11:50 AM
  To: Duddy, John
  Cc: Nate Coraor (n...@bx.psu.edu); galaxy-dev@lists.bx.psu.edu
  Subject: Looks like actual user breaks splitting
  
  Hey John,
  
  Any thoughts?
  
  Ilya
  
  Traceback (most recent call last):
File 
  /home/galaxy/ichorny/galaxy-central/lib/galaxy/jobs/runners/tasks.py, 
  line 73, in run_job
  tasks = splitter.do_split(job_wrapper)
File 
  /home/galaxy/ichorny/galaxy-central/lib/galaxy/jobs/splitters/multi.py, 
  line 73, in do_split
  input_type.split(input_datasets, get_new_working_directory_name, 
  parallel_settings)
File /home/galaxy/ichorny/galaxy-central/lib/galaxy/datatypes/data.py, 
  line 473, in split
  raise Exception(Text file splitting does not support multiple files)
  Exception: Text file splitting does not support multiple files
  
  Ilya Chorny Ph.D.
  Bioinformatics Scientist I
  Illumina, Inc.
  9885 Towne Centre Drive
  San Diego, CA 92121
  Work: 858.202.4582
  Email: icho...@illumina.commailto:icho...@illumina.com
  Website: www.illumina.comhttp://www.illumina.com
  
  
 

___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/


Re: [galaxy-dev] Looks like actual user breaks splitting

2011-11-03 Thread Nate Coraor (n...@bx.psu.edu)
Duddy, John wrote:
 I'm not following you - it's been 6 months since I wrote that code ;-}

I know the feeling!

 IT looks to me like a DatasetPath() object is always placed in that array, 
 and with one exception near then, it looks like the change I made generates 
 those objects the same way.

It's creating a dict in self.output_dataset_paths, and that dict looks
like this when outputs_to_working_directory = False:

{ output_param_name : [ HDA, DatasetPath ], ... }

And this when True:

{ output_param_name : [ Dataset, DatasetPath ], ... }

 Do you have a stack trace for the merge problem I can look at?

If you put this in do_merge()'s except block:

log.exception( stdout )

You'll get:

Traceback (most recent call last):
  File 
/space/nate/galaxy-central-ichorny/lib/galaxy/jobs/splitters/multi.py, line 
128, in do_merge
output_type = outputs[output][0].datatype
AttributeError: 'Dataset' object has no attribute 'datatype'

I could just change both methods to put an HDA in the list inside the
dict there, but I haven't looked much into what output_dataset_paths is
used for, so I wasn't sure what that might break.

Thanks,
--nate

 John Duddy
 Sr. Staff Software Engineer
 Illumina, Inc.
 9885 Towne Centre Drive
 San Diego, CA 92121
 Tel: 858-736-3584
 E-mail: jdu...@illumina.com
 
 
 -Original Message-
 From: Nate Coraor (n...@bx.psu.edu) [mailto:n...@bx.psu.edu] 
 Sent: Thursday, November 03, 2011 2:22 PM
 To: Duddy, John
 Cc: Chorny, Ilya; galaxy-dev@lists.bx.psu.edu
 Subject: Re: Looks like actual user breaks splitting
 
 Hi John,
 
 It looks like the first issue is related to the change from
 get_output_fnames() - compute_outputs().  When
 outputs_to_working_directory = False (default) this method
 stores/returns a HistoryDatasetAssociation, but when True,
 stores/returns a Dataset (the original method's behavior).  Thus,
 accessing the object's .datatype attribute in the splitter's do_merge()
 fails.
 
 Thanks,
 --nate
 
 Duddy, John wrote:
  I'll submit a pull request shortly...
  
  John Duddy
  Sr. Staff Software Engineer
  Illumina, Inc.
  9885 Towne Centre Drive
  San Diego, CA 92121
  Tel: 858-736-3584
  E-mail: jdu...@illumina.com
  
  
  -Original Message-
  From: Nate Coraor (n...@bx.psu.edu) [mailto:n...@bx.psu.edu] 
  Sent: Wednesday, November 02, 2011 12:24 PM
  To: Duddy, John
  Cc: Chorny, Ilya; galaxy-dev@lists.bx.psu.edu
  Subject: Re: Looks like actual user breaks splitting
  
  John, Ilya,
  
  I get further with sequence type inputs but it looks like
  JobWrapper.get_output_datasets_and_fnames() is not returning the right
  thing when outputs_to_working_directory = True
  
  BTW, the base Data.split() method is broken after the updates to
  Sequence.split() since it wasn't updated to expect
  HistoryDatasetAssociations rather than filenames.  Could you take a look
  at that when you get a chance?
  
  --nate
  
  Duddy, John wrote:
   The datatype you are using does not define a split method. Are you 
   working with our in-progress gz type or fastqillumina?
   
   John Duddy
   Sr. Staff Software Engineer
   Illumina, Inc.
   9885 Towne Centre Drive
   San Diego, CA 92121
   Tel: 858-736-3584
   E-mail: jdu...@illumina.commailto:jdu...@illumina.com
   
   From: Chorny, Ilya
   Sent: Wednesday, November 02, 2011 11:50 AM
   To: Duddy, John
   Cc: Nate Coraor (n...@bx.psu.edu); galaxy-dev@lists.bx.psu.edu
   Subject: Looks like actual user breaks splitting
   
   Hey John,
   
   Any thoughts?
   
   Ilya
   
   Traceback (most recent call last):
 File 
   /home/galaxy/ichorny/galaxy-central/lib/galaxy/jobs/runners/tasks.py, 
   line 73, in run_job
   tasks = splitter.do_split(job_wrapper)
 File 
   /home/galaxy/ichorny/galaxy-central/lib/galaxy/jobs/splitters/multi.py, 
   line 73, in do_split
   input_type.split(input_datasets, get_new_working_directory_name, 
   parallel_settings)
 File 
   /home/galaxy/ichorny/galaxy-central/lib/galaxy/datatypes/data.py, line 
   473, in split
   raise Exception(Text file splitting does not support multiple files)
   Exception: Text file splitting does not support multiple files
   
   Ilya Chorny Ph.D.
   Bioinformatics Scientist I
   Illumina, Inc.
   9885 Towne Centre Drive
   San Diego, CA 92121
   Work: 858.202.4582
   Email: icho...@illumina.commailto:icho...@illumina.com
   Website: www.illumina.comhttp://www.illumina.com
   
   
  
 
___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/


Re: [galaxy-dev] Looks like actual user breaks splitting

2011-11-03 Thread Nate Coraor (n...@bx.psu.edu)
Nate Coraor (n...@bx.psu.edu) wrote:
 Duddy, John wrote:
  I'm not following you - it's been 6 months since I wrote that code ;-}
 
 I know the feeling!
 
  IT looks to me like a DatasetPath() object is always placed in that array, 
  and with one exception near then, it looks like the change I made generates 
  those objects the same way.
 
 It's creating a dict in self.output_dataset_paths, and that dict looks
 like this when outputs_to_working_directory = False:
 
 { output_param_name : [ HDA, DatasetPath ], ... }
 
 And this when True:
 
 { output_param_name : [ Dataset, DatasetPath ], ... }
 
  Do you have a stack trace for the merge problem I can look at?
 
 If you put this in do_merge()'s except block:
 
 log.exception( stdout )
 
 You'll get:
 
 Traceback (most recent call last):
   File 
 /space/nate/galaxy-central-ichorny/lib/galaxy/jobs/splitters/multi.py, line 
 128, in do_merge
 output_type = outputs[output][0].datatype
 AttributeError: 'Dataset' object has no attribute 'datatype'
 
 I could just change both methods to put an HDA in the list inside the
 dict there, but I haven't looked much into what output_dataset_paths is
 used for, so I wasn't sure what that might break.

Sorta answered it myself, it looks like you created this precisely for
do_merge(), so changing it to contain the HDA fixes the problem (and
shouldn't break anything else).

--nate

 
 Thanks,
 --nate
 
  John Duddy
  Sr. Staff Software Engineer
  Illumina, Inc.
  9885 Towne Centre Drive
  San Diego, CA 92121
  Tel: 858-736-3584
  E-mail: jdu...@illumina.com
  
  
  -Original Message-
  From: Nate Coraor (n...@bx.psu.edu) [mailto:n...@bx.psu.edu] 
  Sent: Thursday, November 03, 2011 2:22 PM
  To: Duddy, John
  Cc: Chorny, Ilya; galaxy-dev@lists.bx.psu.edu
  Subject: Re: Looks like actual user breaks splitting
  
  Hi John,
  
  It looks like the first issue is related to the change from
  get_output_fnames() - compute_outputs().  When
  outputs_to_working_directory = False (default) this method
  stores/returns a HistoryDatasetAssociation, but when True,
  stores/returns a Dataset (the original method's behavior).  Thus,
  accessing the object's .datatype attribute in the splitter's do_merge()
  fails.
  
  Thanks,
  --nate
  
  Duddy, John wrote:
   I'll submit a pull request shortly...
   
   John Duddy
   Sr. Staff Software Engineer
   Illumina, Inc.
   9885 Towne Centre Drive
   San Diego, CA 92121
   Tel: 858-736-3584
   E-mail: jdu...@illumina.com
   
   
   -Original Message-
   From: Nate Coraor (n...@bx.psu.edu) [mailto:n...@bx.psu.edu] 
   Sent: Wednesday, November 02, 2011 12:24 PM
   To: Duddy, John
   Cc: Chorny, Ilya; galaxy-dev@lists.bx.psu.edu
   Subject: Re: Looks like actual user breaks splitting
   
   John, Ilya,
   
   I get further with sequence type inputs but it looks like
   JobWrapper.get_output_datasets_and_fnames() is not returning the right
   thing when outputs_to_working_directory = True
   
   BTW, the base Data.split() method is broken after the updates to
   Sequence.split() since it wasn't updated to expect
   HistoryDatasetAssociations rather than filenames.  Could you take a look
   at that when you get a chance?
   
   --nate
   
   Duddy, John wrote:
The datatype you are using does not define a split method. Are you 
working with our in-progress gz type or fastqillumina?

John Duddy
Sr. Staff Software Engineer
Illumina, Inc.
9885 Towne Centre Drive
San Diego, CA 92121
Tel: 858-736-3584
E-mail: jdu...@illumina.commailto:jdu...@illumina.com

From: Chorny, Ilya
Sent: Wednesday, November 02, 2011 11:50 AM
To: Duddy, John
Cc: Nate Coraor (n...@bx.psu.edu); galaxy-dev@lists.bx.psu.edu
Subject: Looks like actual user breaks splitting

Hey John,

Any thoughts?

Ilya

Traceback (most recent call last):
  File 
/home/galaxy/ichorny/galaxy-central/lib/galaxy/jobs/runners/tasks.py, 
line 73, in run_job
tasks = splitter.do_split(job_wrapper)
  File 
/home/galaxy/ichorny/galaxy-central/lib/galaxy/jobs/splitters/multi.py,
 line 73, in do_split
input_type.split(input_datasets, get_new_working_directory_name, 
parallel_settings)
  File 
/home/galaxy/ichorny/galaxy-central/lib/galaxy/datatypes/data.py, 
line 473, in split
raise Exception(Text file splitting does not support multiple 
files)
Exception: Text file splitting does not support multiple files

Ilya Chorny Ph.D.
Bioinformatics Scientist I
Illumina, Inc.
9885 Towne Centre Drive
San Diego, CA 92121
Work: 858.202.4582
Email: icho...@illumina.commailto:icho...@illumina.com
Website: www.illumina.comhttp://www.illumina.com


   
  
 ___
 Please keep all replies on the list by using reply all
 in your mail client.  To manage your 

Re: [galaxy-dev] Looks like actual user breaks splitting

2011-11-02 Thread Duddy, John
The datatype you are using does not define a split method. Are you working with 
our in-progress gz type or fastqillumina?

John Duddy
Sr. Staff Software Engineer
Illumina, Inc.
9885 Towne Centre Drive
San Diego, CA 92121
Tel: 858-736-3584
E-mail: jdu...@illumina.commailto:jdu...@illumina.com

From: Chorny, Ilya
Sent: Wednesday, November 02, 2011 11:50 AM
To: Duddy, John
Cc: Nate Coraor (n...@bx.psu.edu); galaxy-dev@lists.bx.psu.edu
Subject: Looks like actual user breaks splitting

Hey John,

Any thoughts?

Ilya

Traceback (most recent call last):
  File /home/galaxy/ichorny/galaxy-central/lib/galaxy/jobs/runners/tasks.py, 
line 73, in run_job
tasks = splitter.do_split(job_wrapper)
  File 
/home/galaxy/ichorny/galaxy-central/lib/galaxy/jobs/splitters/multi.py, line 
73, in do_split
input_type.split(input_datasets, get_new_working_directory_name, 
parallel_settings)
  File /home/galaxy/ichorny/galaxy-central/lib/galaxy/datatypes/data.py, line 
473, in split
raise Exception(Text file splitting does not support multiple files)
Exception: Text file splitting does not support multiple files

Ilya Chorny Ph.D.
Bioinformatics Scientist I
Illumina, Inc.
9885 Towne Centre Drive
San Diego, CA 92121
Work: 858.202.4582
Email: icho...@illumina.commailto:icho...@illumina.com
Website: www.illumina.comhttp://www.illumina.com


___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] Looks like actual user breaks splitting

2011-11-02 Thread Nate Coraor (n...@bx.psu.edu)
John, Ilya,

I get further with sequence type inputs but it looks like
JobWrapper.get_output_datasets_and_fnames() is not returning the right
thing when outputs_to_working_directory = True

BTW, the base Data.split() method is broken after the updates to
Sequence.split() since it wasn't updated to expect
HistoryDatasetAssociations rather than filenames.  Could you take a look
at that when you get a chance?

--nate

Duddy, John wrote:
 The datatype you are using does not define a split method. Are you working 
 with our in-progress gz type or fastqillumina?
 
 John Duddy
 Sr. Staff Software Engineer
 Illumina, Inc.
 9885 Towne Centre Drive
 San Diego, CA 92121
 Tel: 858-736-3584
 E-mail: jdu...@illumina.commailto:jdu...@illumina.com
 
 From: Chorny, Ilya
 Sent: Wednesday, November 02, 2011 11:50 AM
 To: Duddy, John
 Cc: Nate Coraor (n...@bx.psu.edu); galaxy-dev@lists.bx.psu.edu
 Subject: Looks like actual user breaks splitting
 
 Hey John,
 
 Any thoughts?
 
 Ilya
 
 Traceback (most recent call last):
   File 
 /home/galaxy/ichorny/galaxy-central/lib/galaxy/jobs/runners/tasks.py, line 
 73, in run_job
 tasks = splitter.do_split(job_wrapper)
   File 
 /home/galaxy/ichorny/galaxy-central/lib/galaxy/jobs/splitters/multi.py, 
 line 73, in do_split
 input_type.split(input_datasets, get_new_working_directory_name, 
 parallel_settings)
   File /home/galaxy/ichorny/galaxy-central/lib/galaxy/datatypes/data.py, 
 line 473, in split
 raise Exception(Text file splitting does not support multiple files)
 Exception: Text file splitting does not support multiple files
 
 Ilya Chorny Ph.D.
 Bioinformatics Scientist I
 Illumina, Inc.
 9885 Towne Centre Drive
 San Diego, CA 92121
 Work: 858.202.4582
 Email: icho...@illumina.commailto:icho...@illumina.com
 Website: www.illumina.comhttp://www.illumina.com
 
 
___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/


Re: [galaxy-dev] Looks like actual user breaks splitting

2011-11-02 Thread Duddy, John
I'll submit a pull request shortly...

John Duddy
Sr. Staff Software Engineer
Illumina, Inc.
9885 Towne Centre Drive
San Diego, CA 92121
Tel: 858-736-3584
E-mail: jdu...@illumina.com


-Original Message-
From: Nate Coraor (n...@bx.psu.edu) [mailto:n...@bx.psu.edu] 
Sent: Wednesday, November 02, 2011 12:24 PM
To: Duddy, John
Cc: Chorny, Ilya; galaxy-dev@lists.bx.psu.edu
Subject: Re: Looks like actual user breaks splitting

John, Ilya,

I get further with sequence type inputs but it looks like
JobWrapper.get_output_datasets_and_fnames() is not returning the right
thing when outputs_to_working_directory = True

BTW, the base Data.split() method is broken after the updates to
Sequence.split() since it wasn't updated to expect
HistoryDatasetAssociations rather than filenames.  Could you take a look
at that when you get a chance?

--nate

Duddy, John wrote:
 The datatype you are using does not define a split method. Are you working 
 with our in-progress gz type or fastqillumina?
 
 John Duddy
 Sr. Staff Software Engineer
 Illumina, Inc.
 9885 Towne Centre Drive
 San Diego, CA 92121
 Tel: 858-736-3584
 E-mail: jdu...@illumina.commailto:jdu...@illumina.com
 
 From: Chorny, Ilya
 Sent: Wednesday, November 02, 2011 11:50 AM
 To: Duddy, John
 Cc: Nate Coraor (n...@bx.psu.edu); galaxy-dev@lists.bx.psu.edu
 Subject: Looks like actual user breaks splitting
 
 Hey John,
 
 Any thoughts?
 
 Ilya
 
 Traceback (most recent call last):
   File 
 /home/galaxy/ichorny/galaxy-central/lib/galaxy/jobs/runners/tasks.py, line 
 73, in run_job
 tasks = splitter.do_split(job_wrapper)
   File 
 /home/galaxy/ichorny/galaxy-central/lib/galaxy/jobs/splitters/multi.py, 
 line 73, in do_split
 input_type.split(input_datasets, get_new_working_directory_name, 
 parallel_settings)
   File /home/galaxy/ichorny/galaxy-central/lib/galaxy/datatypes/data.py, 
 line 473, in split
 raise Exception(Text file splitting does not support multiple files)
 Exception: Text file splitting does not support multiple files
 
 Ilya Chorny Ph.D.
 Bioinformatics Scientist I
 Illumina, Inc.
 9885 Towne Centre Drive
 San Diego, CA 92121
 Work: 858.202.4582
 Email: icho...@illumina.commailto:icho...@illumina.com
 Website: www.illumina.comhttp://www.illumina.com
 
 

___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/