Re: [galaxy-dev] Galaxy pbs scripts and job_working_directory files

2012-04-24 Thread Dannon Baker
Josh,

Check out the cleanup_job setting in universe_wsgi.ini(and included below).  It 
sounds like 'cleanup_job = onsuccess' is exactly what you're looking for.

-Dannon

# Clean up various bits of jobs left on the filesystem after completion.  These
# bits include the job working directory, external metadata temporary files,
# and DRM stdout and stderr files (if using a DRM).  Possible values are:
# always, onsuccess, never
#cleanup_job = always

On Apr 24, 2012, at 5:00 PM, Josh Nielsen wrote:

 Hello again,
 
 As I mentioned in a recent post I often find need to debug jobs running from 
 our local Galaxy mirror and I often have the need to look at the script  
 data files that the job is trying to use in order to figure out what is 
 causing a problem. The directories containing those file are in 
 '[galaxy_dist]/database/pbs/' and 
 '[galaxy_dist]/database/job_working_directory/' for me. Each job that is run 
 gets a corresponding .sh file in the pbs/ directory (like 344.sh) which will 
 have the entire sequence of bash commands to execute the job with and also a 
 call to a wrapper script somewhere in the middle normally. That script 
 information is very useful, but the problem is that when a job fails (often 
 within the first 30 seconds of running it) the script is deleted and there is 
 no trace of it left in the directory. The same with the output or job data 
 files in job_working_directory/. 
 
 I have had to suffice with using the technique of coordinating with the user 
 when to (re)run their failed job and then quickly within the 30 second window 
 do a cp -R script_I_care_about.sh copy_of_script.sh command, so that when 
 the script is deleted I have a copy that I can examine. The same goes with 
 copying the job_working_directory/ files. I know that it would get very 
 cluttered in those directories if they were not automatically cleaned/deleted 
 but I find those files essential for debugging. Is there a way to force 
 Galaxy to retain those files (optionally) for debugging purposes? Maybe make 
 a new option in the universe.ini file for that purpose that can be set for 
 people who want it?
 
 Thanks,
 Josh Nielsen
 ___
 Please keep all replies on the list by using reply all
 in your mail client.  To manage your subscriptions to this
 and other Galaxy lists, please use the interface at:
 
  http://lists.bx.psu.edu/

___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/


Re: [galaxy-dev] Galaxy pbs scripts and job_working_directory files

2012-04-24 Thread Josh Nielsen
Ah, so this would just betray my ignorance of current features. :-) I'll
give that a try!

Thanks,
Josh

On Tue, Apr 24, 2012 at 4:09 PM, Dannon Baker dannonba...@me.com wrote:

 Josh,

 Check out the cleanup_job setting in universe_wsgi.ini(and included
 below).  It sounds like 'cleanup_job = onsuccess' is exactly what you're
 looking for.

 -Dannon

 # Clean up various bits of jobs left on the filesystem after completion.
  These
 # bits include the job working directory, external metadata temporary
 files,
 # and DRM stdout and stderr files (if using a DRM).  Possible values are:
 # always, onsuccess, never
 #cleanup_job = always

 On Apr 24, 2012, at 5:00 PM, Josh Nielsen wrote:

  Hello again,
 
  As I mentioned in a recent post I often find need to debug jobs running
 from our local Galaxy mirror and I often have the need to look at the
 script  data files that the job is trying to use in order to figure out
 what is causing a problem. The directories containing those file are in
 '[galaxy_dist]/database/pbs/' and
 '[galaxy_dist]/database/job_working_directory/' for me. Each job that is
 run gets a corresponding .sh file in the pbs/ directory (like 344.sh) which
 will have the entire sequence of bash commands to execute the job with and
 also a call to a wrapper script somewhere in the middle normally. That
 script information is very useful, but the problem is that when a job fails
 (often within the first 30 seconds of running it) the script is deleted and
 there is no trace of it left in the directory. The same with the output or
 job data files in job_working_directory/.
 
  I have had to suffice with using the technique of coordinating with the
 user when to (re)run their failed job and then quickly within the 30 second
 window do a cp -R script_I_care_about.sh copy_of_script.sh command, so
 that when the script is deleted I have a copy that I can examine. The same
 goes with copying the job_working_directory/ files. I know that it would
 get very cluttered in those directories if they were not automatically
 cleaned/deleted but I find those files essential for debugging. Is there a
 way to force Galaxy to retain those files (optionally) for debugging
 purposes? Maybe make a new option in the universe.ini file for that purpose
 that can be set for people who want it?
 
  Thanks,
  Josh Nielsen
  ___
  Please keep all replies on the list by using reply all
  in your mail client.  To manage your subscriptions to this
  and other Galaxy lists, please use the interface at:
 
   http://lists.bx.psu.edu/


___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/