Re: [galaxy-dev] Unable to finish job - metadata settings
Hey Jens, I have tried a few different things and I have been unable to replicate the behavior locally. Is this tool specific or configuration specific - i.e. do you see this behavior only with a specific tool or do the concatenate datasets tool experience this as well say? If it is tool specific - you (or the underlying application) may be deleting metadata_results_XXX files in the working directory as part of the job? If you are convinced the tool is not deleting these files but it is a tool-specific problem can you pass along the tool you are using (or better a minimal version of the tool that produces this behavior.) If it is configuration specific and you can get it to happen with many different tools - can you try to do it against a clean copy of galaxy-dist or galaxy-central and pass along the exact updates (universe_wsgi.ini, job_conf.xml, etc...) that you used to configure Galaxy to cause it to produce this error. Couple more things - it hasn't gone into the if in the jobs/__init__.py statement - it is doing that first check external_metadata_set_successfully when the error occurs - so I don't think it is a problem with the retry_metadata_internally configuration option. Additionally, you are using the local job runner so Galaxy will always "retry_metadata_internally" - the local job runner doesn't try to embed the metadata calculation into the job like the cluster job runners (so retry_metadata_internally doesn't really matter with the local job runner... right now anyway). If you don't want the metadata calculation to be embedded into the local job runner job the way cluster job runners do it (so retry_metadata_internally does in fact matter) there was an option added to the local job runner last release that I realized I hadn't documented - you can add True to the local job destination in job_conf.xml. More information about this new option here: https://bitbucket.org/galaxy/galaxy-central/commits/75c63b579ccdd63e0558dd9aefce7786677dbacd -John On Wed, Jun 25, 2014 at 7:41 AM, Preussner, Jens wrote: > Dear all, > > > > I noticed a strange behaviour in our local galaxy installation. First of > all, my universe_wsgi.ini contains “retry_metadata_internally = False” and > “cleanup_job = always”. The tool writes its output simply into the > job_working_directory and we move it via && mv static_filename.txt $output > in the -tag. This works fine. > > > > When restarting the galaxy server and executing the tool after a fresh > restart, everything is ok, there are no errors. > > When executing the same tool a second time, galaxy brings a “tool error” > stating that it was unable to finish the job. Nevertheless, the output files > are all correct (but marked as red or failed). > > > > The error report states: > > Traceback (most recent call last): > > File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/runners/local.py", line > 129, in queue_job > > job_wrapper.finish( stdout, stderr, exit_code ) > > File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 997, in > finish > > if ( not > self.external_output_metadata.external_metadata_set_successfully( dataset, > self.sa_session ) and self.app.config.retry_metadata_internally ): > > File "/home/galaxy/galaxy-dist/lib/galaxy/datatypes/metadata.py", line > 731, in external_metadata_set_successfully > > rval, rstring = json.load( open( metadata_files.filename_results_code ) > ) > > IOError: [Errno 2] No such file or directory: > u'/home/galaxy/galaxy-dist/database/job_working_directory/000/59/metadata_results_HistoryDatasetAssociation_281_oHFjx0' > > > > And in the logfile you can find multiple entries like that: > > galaxy.datatypes.metadata DEBUG 2014-06-25 14:29:35,466 Failed to cleanup > external metadata file (filename_results_code) for > HistoryDatasetAssociation_281: [Errno 2] No such file or directory: > '/home/galaxy/galaxy-dist/database/job_working_directory/000/59/metadata_results_HistoryDatasetAssociation_281_oHFjx0' > > > > The if-statement in /home/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", > line 997 should evaluate to False, since > self.app.config.retry_metadata_internally is set to False in the > universe_wsgi.ini but it seems it doesn’t in this case? > > Anyone having experienced such a behavior? Any suggestions how to go on and > solve the issue? > > Many thanks! > > > > Jens > > > > > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > http://lists.bx.psu.edu/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: htt
[galaxy-dev] Unable to finish job - metadata settings
Dear all, I noticed a strange behaviour in our local galaxy installation. First of all, my universe_wsgi.ini contains "retry_metadata_internally = False" and "cleanup_job = always". The tool writes its output simply into the job_working_directory and we move it via && mv static_filename.txt $output in the -tag. This works fine. When restarting the galaxy server and executing the tool after a fresh restart, everything is ok, there are no errors. When executing the same tool a second time, galaxy brings a "tool error" stating that it was unable to finish the job. Nevertheless, the output files are all correct (but marked as red or failed). The error report states: Traceback (most recent call last): File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/runners/local.py", line 129, in queue_job job_wrapper.finish( stdout, stderr, exit_code ) File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 997, in finish if ( not self.external_output_metadata.external_metadata_set_successfully( dataset, self.sa_session ) and self.app.config.retry_metadata_internally ): File "/home/galaxy/galaxy-dist/lib/galaxy/datatypes/metadata.py", line 731, in external_metadata_set_successfully rval, rstring = json.load( open( metadata_files.filename_results_code ) ) IOError: [Errno 2] No such file or directory: u'/home/galaxy/galaxy-dist/database/job_working_directory/000/59/metadata_results_HistoryDatasetAssociation_281_oHFjx0' And in the logfile you can find multiple entries like that: galaxy.datatypes.metadata DEBUG 2014-06-25 14:29:35,466 Failed to cleanup external metadata file (filename_results_code) for HistoryDatasetAssociation_281: [Errno 2] No such file or directory: '/home/galaxy/galaxy-dist/database/job_working_directory/000/59/metadata_results_HistoryDatasetAssociation_281_oHFjx0' The if-statement in /home/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 997 should evaluate to False, since self.app.config.retry_metadata_internally is set to False in the universe_wsgi.ini but it seems it doesn't in this case? Anyone having experienced such a behavior? Any suggestions how to go on and solve the issue? Many thanks! Jens ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Unable to finish job
On Apr 17, 2013, at 9:11 AM, Ramon Tiburski wrote:
> Hi Nate, thank you for the help, I'm posting more details about the error,
> Did you ever see something like this?
Hi Ramon,
Without being familiar with the tool, I am not sure what is happening here.
Either the process of sourcing the env.sh file or calling cdo is failing, for
unknown reasons. It could be that the command is not being executed through a
shell, in which case the source (',') and the join (';') specials would not be
recognized.
--nate
>
> Thank you so much,
>
> WARNING:galaxy.datatypes.registry:Error loading datatype with extension 'ps':
> 'module' object has no attribute 'scripts'
> WARNING:galaxy.datatypes.registry:Error appending sniffer for datatype
> 'galaxy.datatypes.scripts:PostScript' to sniff_order: No module named scripts
> INFO:NetCDFwithTimeAxis:NetCDFwithTimeAxis set_meta
> INFO:NetCDFwithTimeAxis:NetCDF set_meta
> INFO:NetCDFwithTimeAxis:Climate set_meta
> [localhost] local: .
> /home/ramao/shed_tools/web.uniparthenope.it/repos/montella/galaxy_es_core/a8f8a904173c/galaxy_es_core/env.sh;cdo
> infon /home/ramao/galaxy/database/files/000/dataset_15.dat
>
> Fatal error: local() encountered an error (return code 127) while executing
> '.
> /home/ramao/shed_tools/web.uniparthenope.it/repos/montella/galaxy_es_core/a8f8a904173c/galaxy_es_core/env.sh;cdo
> infon /home/ramao/galaxy/database/files/000/dataset_15.dat'
>
> Aborting.
> [localhost] local: .
> /home/ramao/shed_tools/web.uniparthenope.it/repos/montella/galaxy_es_core/a8f8a904173c/galaxy_es_core/env.sh;cdo
> sinfo /home/ramao/galaxy/database/files/000/dataset_15.dat
>
> Fatal error: local() encountered an error (return code 127) while executing
> '.
> /home/ramao/shed_tools/web.uniparthenope.it/repos/montella/galaxy_es_core/a8f8a904173c/galaxy_es_core/env.sh;cdo
> sinfo /home/ramao/galaxy/database/files/000/dataset_15.dat'
>
> Aborting.
> [localhost] local: .
> /home/ramao/shed_tools/web.uniparthenope.it/repos/montella/galaxy_es_core/a8f8a904173c/galaxy_es_core/env.sh;ncdump
> -x /home/ramao/galaxy/database/files/000/dataset_15.dat
>
> Fatal error: local() encountered an error (return code 127) while executing
> '.
> /home/ramao/shed_tools/web.uniparthenope.it/repos/montella/galaxy_es_core/a8f8a904173c/galaxy_es_core/env.sh;ncdump
> -x /home/ramao/galaxy/database/files/000/dataset_15.dat'
>
> Aborting.
> galaxy.jobs.runners.local DEBUG 2013-04-17 09:51:59,955 execution of external
> set_meta for job 15 finished
> galaxy.jobs DEBUG 2013-04-17 09:51:59,979 Tool did not define exit code or
> stdio handling; checking stderr for success
> galaxy.datatypes.metadata DEBUG 2013-04-17 09:52:00,029 setting metadata
> externally failed for HistoryDatasetAssociation 15: External set_meta() not
> called
> NetCDFwithTimeAxis INFO 2013-04-17 09:52:00,029 NetCDFwithTimeAxis set_meta
> NetCDFwithTimeAxis INFO 2013-04-17 09:52:00,029 NetCDF set_meta
> NetCDFwithTimeAxis INFO 2013-04-17 09:52:00,029 Climate set_meta
> [localhost] local: .
> /home/ramao/shed_tools/web.uniparthenope.it/repos/montella/galaxy_es_core/a8f8a904173c/galaxy_es_core/env.sh;cdo
> infon /home/ramao/galaxy/database/files/000/dataset_15.dat
>
> Fatal error: local() encountered an error (return code 127) while executing
> '.
> /home/ramao/shed_tools/web.uniparthenope.it/repos/montella/galaxy_es_core/a8f8a904173c/galaxy_es_core/env.sh;cdo
> infon /home/ramao/galaxy/database/files/000/dataset_15.dat'
>
> Aborting.
> [localhost] local: .
> /home/ramao/shed_tools/web.uniparthenope.it/repos/montella/galaxy_es_core/a8f8a904173c/galaxy_es_core/env.sh;cdo
> sinfo /home/ramao/galaxy/database/files/000/dataset_15.dat
>
> Fatal error: local() encountered an error (return code 127) while executing
> '.
> /home/ramao/shed_tools/web.uniparthenope.it/repos/montella/galaxy_es_core/a8f8a904173c/galaxy_es_core/env.sh;cdo
> sinfo /home/ramao/galaxy/database/files/000/dataset_15.dat'
>
> Aborting.
> [localhost] local: .
> /home/ramao/shed_tools/web.uniparthenope.it/repos/montella/galaxy_es_core/a8f8a904173c/galaxy_es_core/env.sh;ncdump
> -x /home/ramao/galaxy/database/files/000/dataset_15.dat
>
> Fatal error: local() encountered an error (return code 127) while executing
> '.
> /home/ramao/shed_tools/web.uniparthenope.it/repos/montella/galaxy_es_core/a8f8a904173c/galaxy_es_core/env.sh;ncdump
> -x /home/ramao/galaxy/database/files/000/dataset_15.dat'
>
>
> > Subject: Re: [galaxy-dev] Unable to finish job
> > From: n...@bx.psu.edu
> > Date: Tue, 16 Apr 2013 11:45:37 -0400
> > CC: galaxy-dev@
Re: [galaxy-dev] Unable to finish job
Hi Nate, thank you for the help, I'm posting more details about the error, Did you ever see something like this? Thank you so much, WARNING:galaxy.datatypes.registry:Error loading datatype with extension 'ps': 'module' object has no attribute 'scripts' WARNING:galaxy.datatypes.registry:Error appending sniffer for datatype 'galaxy.datatypes.scripts:PostScript' to sniff_order: No module named scripts INFO:NetCDFwithTimeAxis:NetCDFwithTimeAxis set_meta INFO:NetCDFwithTimeAxis:NetCDF set_meta INFO:NetCDFwithTimeAxis:Climate set_meta [localhost] local: . /home/ramao/shed_tools/web.uniparthenope.it/repos/montella/galaxy_es_core/a8f8a904173c/galaxy_es_core/env.sh;cdo infon /home/ramao/galaxy/database/files/000/dataset_15.dat Fatal error: local() encountered an error (return code 127) while executing '. /home/ramao/shed_tools/web.uniparthenope.it/repos/montella/galaxy_es_core/a8f8a904173c/galaxy_es_core/env.sh;cdo infon /home/ramao/galaxy/database/files/000/dataset_15.dat' Aborting. [localhost] local: . /home/ramao/shed_tools/web.uniparthenope.it/repos/montella/galaxy_es_core/a8f8a904173c/galaxy_es_core/env.sh;cdo sinfo /home/ramao/galaxy/database/files/000/dataset_15.dat Fatal error: local() encountered an error (return code 127) while executing '. /home/ramao/shed_tools/web.uniparthenope.it/repos/montella/galaxy_es_core/a8f8a904173c/galaxy_es_core/env.sh;cdo sinfo /home/ramao/galaxy/database/files/000/dataset_15.dat' Aborting. [localhost] local: . /home/ramao/shed_tools/web.uniparthenope.it/repos/montella/galaxy_es_core/a8f8a904173c/galaxy_es_core/env.sh;ncdump -x /home/ramao/galaxy/database/files/000/dataset_15.dat Fatal error: local() encountered an error (return code 127) while executing '. /home/ramao/shed_tools/web.uniparthenope.it/repos/montella/galaxy_es_core/a8f8a904173c/galaxy_es_core/env.sh;ncdump -x /home/ramao/galaxy/database/files/000/dataset_15.dat' Aborting. galaxy.jobs.runners.local DEBUG 2013-04-17 09:51:59,955 execution of external set_meta for job 15 finished galaxy.jobs DEBUG 2013-04-17 09:51:59,979 Tool did not define exit code or stdio handling; checking stderr for success galaxy.datatypes.metadata DEBUG 2013-04-17 09:52:00,029 setting metadata externally failed for HistoryDatasetAssociation 15: External set_meta() not called NetCDFwithTimeAxis INFO 2013-04-17 09:52:00,029 NetCDFwithTimeAxis set_meta NetCDFwithTimeAxis INFO 2013-04-17 09:52:00,029 NetCDF set_meta NetCDFwithTimeAxis INFO 2013-04-17 09:52:00,029 Climate set_meta [localhost] local: . /home/ramao/shed_tools/web.uniparthenope.it/repos/montella/galaxy_es_core/a8f8a904173c/galaxy_es_core/env.sh;cdo infon /home/ramao/galaxy/database/files/000/dataset_15.dat Fatal error: local() encountered an error (return code 127) while executing '. /home/ramao/shed_tools/web.uniparthenope.it/repos/montella/galaxy_es_core/a8f8a904173c/galaxy_es_core/env.sh;cdo infon /home/ramao/galaxy/database/files/000/dataset_15.dat' Aborting. [localhost] local: . /home/ramao/shed_tools/web.uniparthenope.it/repos/montella/galaxy_es_core/a8f8a904173c/galaxy_es_core/env.sh;cdo sinfo /home/ramao/galaxy/database/files/000/dataset_15.dat Fatal error: local() encountered an error (return code 127) while executing '. /home/ramao/shed_tools/web.uniparthenope.it/repos/montella/galaxy_es_core/a8f8a904173c/galaxy_es_core/env.sh;cdo sinfo /home/ramao/galaxy/database/files/000/dataset_15.dat' Aborting. [localhost] local: . /home/ramao/shed_tools/web.uniparthenope.it/repos/montella/galaxy_es_core/a8f8a904173c/galaxy_es_core/env.sh;ncdump -x /home/ramao/galaxy/database/files/000/dataset_15.dat Fatal error: local() encountered an error (return code 127) while executing '. /home/ramao/shed_tools/web.uniparthenope.it/repos/montella/galaxy_es_core/a8f8a904173c/galaxy_es_core/env.sh;ncdump -x /home/ramao/galaxy/database/files/000/dataset_15.dat' > Subject: Re: [galaxy-dev] Unable to finish job > From: n...@bx.psu.edu > Date: Tue, 16 Apr 2013 11:45:37 -0400 > CC: galaxy-dev@lists.bx.psu.edu > To: ramao_tiago_tibur...@hotmail.com > > On Apr 16, 2013, at 9:37 AM, Ramon Tiburski wrote: > > > I,m getting this error when I try to convert some data, can someone help me? > > > > Traceback (most recent call last): > > File "/home/ramao/galaxy/lib/galaxy/jobs/runners/local.py", line 156, in > > run_job > > job_wrapper.finish( stdout, stderr, exit_code ) > > File "/home/ramao/galaxy/lib/galaxy/jobs/__init__.py", line 412, in finish > > dataset.datatype.set_meta( dataset, overwrite = False ) #call > > datatype.set_meta directly for the initial set_meta call during dataset > > creation > > File > > "/home/ramao/shed_tools/web.uniparthenope.it/repos/montella/galaxy_es_core/a8f8a904173c/galaxy_es_core/earth
Re: [galaxy-dev] Unable to finish job
On Apr 16, 2013, at 9:37 AM, Ramon Tiburski wrote:
> I,m getting this error when I try to convert some data, can someone help me?
>
> Traceback (most recent call last):
> File "/home/ramao/galaxy/lib/galaxy/jobs/runners/local.py", line 156, in
> run_job
> job_wrapper.finish( stdout, stderr, exit_code )
> File "/home/ramao/galaxy/lib/galaxy/jobs/__init__.py", line 412, in finish
> dataset.datatype.set_meta( dataset, overwrite = False ) #call
> datatype.set_meta directly for the initial set_meta call during dataset
> creation
> File
> "/home/ramao/shed_tools/web.uniparthenope.it/repos/montella/galaxy_es_core/a8f8a904173c/galaxy_es_core/earthsystem.py",
> line 318, in set_meta
> NetCDF.set_meta(self,dataset)
> File
> "/home/ramao/shed_tools/web.uniparthenope.it/repos/montella/galaxy_es_core/a8f8a904173c/galaxy_es_core/earthsystem.py",
> line 261, in set_meta
> ncml= execAndGetStdout("ncdump -x "+dataset.file_name)
> File
> "/home/ramao/shed_tools/web.uniparthenope.it/repos/montella/galaxy_es_core/a8f8a904173c/galaxy_es_core/earthsystem.py",
> line 48, in execAndGetStdout
> output=local(cmd, capture=True)
> File "/home/ramao/galaxy/eggs/Fabric-1.4.2-py2.7.egg/fabric/operations.py",
> line 1012, in local
> error(message=msg, stdout=out, stderr=err)
> File "/home/ramao/galaxy/eggs/Fabric-1.4.2-py2.7.egg/fabric/utils.py", line
> 305, in error
> return func(message)
> File "/home/ramao/galaxy/eggs/Fabric-1.4.2-py2.7.egg/fabric/utils.py", line
> 26, in abort
> sys.exit(1)
> SystemExit: 1
Hi Ramon,
For a definite answer, you'll want to contact the tool/datatype author.
However, I would guess that the `ncdump` command is failing for some reason,
and that would be the place to look.
--nate
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client. To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
> http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
> http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Unable to finish job
I,m getting this error when I try to convert some data, can someone help me?
Traceback (most recent call last):
File "/home/ramao/galaxy/lib/galaxy/jobs/runners/local.py", line 156, in
run_job
job_wrapper.finish( stdout, stderr, exit_code )
File "/home/ramao/galaxy/lib/galaxy/jobs/__init__.py", line 412, in finish
dataset.datatype.set_meta( dataset, overwrite = False ) #call
datatype.set_meta directly for the initial set_meta call during dataset creation
File
"/home/ramao/shed_tools/web.uniparthenope.it/repos/montella/galaxy_es_core/a8f8a904173c/galaxy_es_core/earthsystem.py",
line 318, in set_meta
NetCDF.set_meta(self,dataset)
File
"/home/ramao/shed_tools/web.uniparthenope.it/repos/montella/galaxy_es_core/a8f8a904173c/galaxy_es_core/earthsystem.py",
line 261, in set_meta
ncml= execAndGetStdout("ncdump -x "+dataset.file_name)
File
"/home/ramao/shed_tools/web.uniparthenope.it/repos/montella/galaxy_es_core/a8f8a904173c/galaxy_es_core/earthsystem.py",
line 48, in execAndGetStdout
output=local(cmd, capture=True)
File "/home/ramao/galaxy/eggs/Fabric-1.4.2-py2.7.egg/fabric/operations.py",
line 1012, in local
error(message=msg, stdout=out, stderr=err)
File "/home/ramao/galaxy/eggs/Fabric-1.4.2-py2.7.egg/fabric/utils.py", line
305, in error
return func(message)
File "/home/ramao/galaxy/eggs/Fabric-1.4.2-py2.7.egg/fabric/utils.py", line
26, in abort
sys.exit(1)
SystemExit: 1 ___
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
