Re: [galaxy-dev] only showing first lines of large output file

2011-11-28 Thread Edward Kirton 
hi matthias,

i've also been getting metadata errors (below) for several weeks, but it
works with the set_metadata_externally option (in the universe config file)
set to False.  for large files, it could take many minutes for your job
server to check the metadata, and no new jobs will be run or completed jobs
picked up during this delay, but at least it works reliably.

fyi, the time these errors appeared roughly coincided with when i moved the
data files (database/) to a different disk than the galaxy code.  i don't
know for sure if that's the issue or not, but may be a clue; i will
experiment when i get a chance.

galaxy.datatypes.metadata DEBUG 2011-11-28 12:39:01,129 setting metadata
externally failed for Hist
oryDatasetAssociation 449: External set_meta() not called
galaxy.jobs DEBUG 2011-11-28 12:39:01,221 job 355 ended
galaxy.datatypes.metadata DEBUG 2011-11-28 12:39:01,357 Cleaning up
external metadata files
galaxy.datatypes.metadata DEBUG 2011-11-28 12:39:01,496 Failed to cleanup
MetadataTempFile temp fil
es from
../../../groupdirs/Galaxy/dev/database/tmp/metadata_out_HistoryDatasetAssociation_449_sB_cT
B: No JSON object could be decoded: line 1 column 0 (char 0)


On Wed, Nov 16, 2011 at 3:00 AM, Mattias de Hollander 
m.dehollan...@nioo.knaw.nl wrote:

 **
 Hello,

 I just noticed I get metadata errors in the paster.log file. Can someone
 tell me what they are and if they are related to above problem?

 galaxy.jobs.runners.drmaa DEBUG 2011-11-16 11:55:28,037 (840/588) state 
 change: job finished normally
 galaxy.datatypes.metadata DEBUG 2011-11-16 11:55:28,231 setting metadata 
 externally failed for HistoryDatasetAssociation 1680: [Errno 2] No such file 
 or directory: 
 '../../galaxyData/tmp/metadata_in_HistoryDatasetAssociation_1680_N7JVtq'
 galaxy.datatypes.metadata DEBUG 2011-11-16 11:55:28,325 setting metadata 
 externally failed for HistoryDatasetAssociation 1679: [Errno 2] No such file 
 or directory: 
 '../../galaxyData/tmp/metadata_in_HistoryDatasetAssociation_1679_oTzTG_'
 galaxy.datatypes.metadata DEBUG 2011-11-16 11:55:28,634 setting metadata 
 externally failed for HistoryDatasetAssociation 1678: [Errno 2] No such file 
 or directory: 
 '../../galaxyData/tmp/metadata_in_HistoryDatasetAssociation_1678_MgW15e'galaxy.jobs
  DEBUG 2011-11-16 11:55:29,103 job 840 ended
 galaxy.datatypes.metadata DEBUG 2011-11-16 11:55:29,103 Cleaning up external 
 metadata files
 galaxy.datatypes.metadata DEBUG 2011-11-16 11:55:29,152 Failed to cleanup 
 MetadataTempFile temp files from 
 ../../galaxyData/tmp/metadata_out_HistoryDatasetAssociation_1678_UTGZz8: 
 [Errno 2] No such file or directory: 
 '../../galaxyData/tmp/metadata_out_HistoryDatasetAssociation_1678_UTGZz8'
 galaxy.datatypes.metadata DEBUG 2011-11-16 11:55:29,152 Failed to cleanup 
 external metadata file (filename_in) for HistoryDatasetAssociation_1678: 
 [Errno 2] No such file or directory: 
 '../../galaxyData/tmp/metadata_in_HistoryDatasetAssociation_1678_MgW15e'
 galaxy.datatypes.metadata DEBUG 2011-11-16 11:55:29,152 Failed to cleanup 
 external metadata file (filename_out) for HistoryDatasetAssociation_1678: 
 [Errno 2] No such file or directory: 
 '../../galaxyData/tmp/metadata_out_HistoryDatasetAssociation_1678_UTGZz8'
 galaxy.datatypes.metadata DEBUG 2011-11-16 11:55:29,153 Failed to cleanup 
 external metadata file (filename_kwds) for HistoryDatasetAssociation_1678: 
 [Errno 2] No such file or directory: 
 '../../galaxyData/tmp/metadata_kwds_HistoryDatasetAssociation_1678_3SDMLo'


 And of course, how to solve it :)

 Any help is appreciated.


 Mattias

 --
 Bioinformatician
 Netherlands Institute of Ecology (NIOO-KNAW)
 Wageningen, the Netherlands


 ___
 Please keep all replies on the list by using reply all
 in your mail client.  To manage your subscriptions to this
 and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] only showing first lines of large output file

2011-11-16 Thread Mattias de Hollander 
Hello,

I just noticed I get metadata errors in the paster.log file. Can someone
tell me what they are and if they are related to above problem?


galaxy.jobs.runners.drmaa DEBUG 2011-11-16 11:55:28,037 (840/588) state change: 
job finished normally
galaxy.datatypes.metadata DEBUG 2011-11-16 11:55:28,231 setting metadata 
externally failed for HistoryDatasetAssociation 1680: [Errno 2] No such file or 
directory: 
'../../galaxyData/tmp/metadata_in_HistoryDatasetAssociation_1680_N7JVtq'
galaxy.datatypes.metadata DEBUG 2011-11-16 11:55:28,325 setting metadata 
externally failed for HistoryDatasetAssociation 1679: [Errno 2] No such file or 
directory: 
'../../galaxyData/tmp/metadata_in_HistoryDatasetAssociation_1679_oTzTG_'
galaxy.datatypes.metadata DEBUG 2011-11-16 11:55:28,634 setting metadata 
externally failed for HistoryDatasetAssociation 1678: [Errno 2] No such file or 
directory: 
'../../galaxyData/tmp/metadata_in_HistoryDatasetAssociation_1678_MgW15e'
galaxy.jobs DEBUG 2011-11-16 11:55:29,103 job 840 ended
galaxy.datatypes.metadata DEBUG 2011-11-16 11:55:29,103 Cleaning up external 
metadata files
galaxy.datatypes.metadata DEBUG 2011-11-16 11:55:29,152 Failed to cleanup 
MetadataTempFile temp files from 
../../galaxyData/tmp/metadata_out_HistoryDatasetAssociation_1678_UTGZz8: [Errno 
2] No such file or directory: 
'../../galaxyData/tmp/metadata_out_HistoryDatasetAssociation_1678_UTGZz8'
galaxy.datatypes.metadata DEBUG 2011-11-16 11:55:29,152 Failed to cleanup 
external metadata file (filename_in) for HistoryDatasetAssociation_1678: [Errno 
2] No such file or directory: 
'../../galaxyData/tmp/metadata_in_HistoryDatasetAssociation_1678_MgW15e'
galaxy.datatypes.metadata DEBUG 2011-11-16 11:55:29,152 Failed to cleanup 
external metadata file (filename_out) for HistoryDatasetAssociation_1678: 
[Errno 2] No such file or directory: 
'../../galaxyData/tmp/metadata_out_HistoryDatasetAssociation_1678_UTGZz8'
galaxy.datatypes.metadata DEBUG 2011-11-16 11:55:29,153 Failed to cleanup 
external metadata file (filename_kwds) for HistoryDatasetAssociation_1678: 
[Errno 2] No such file or directory: 
'../../galaxyData/tmp/metadata_kwds_HistoryDatasetAssociation_1678_3SDMLo'


And of course, how to solve it :)

Any help is appreciated.

Mattias


-- 
Bioinformatician
Netherlands Institute of Ecology (NIOO-KNAW)
Wageningen, the Netherlands
___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

[galaxy-dev] only showing first lines of large output file

2011-11-14 Thread Mattias de Hollander 
Hello,

I am observing some difficulties running a custom tool (processing an
sff file). It shows up in Galaxy, runs and its three output files are
given back. Only one of them is only showing the first 12 lines of many
more... 
More symptoms:

  * The  corrupted file is written successfully to disk as temp file
in the galaxy tmp directory
  * Something goes wrong while transfering the file to the galaxy
data dir (files/001), because while the tool is running the
dataset_XX file is shown (as empty file), but after the other
outputs are moved this file is gone
  * in galaxy only a snippet is shown, while the actual datafile is
not on disk


I would be happy to receive feedback on how to start debugging and
solving this issue.

Thanks!

Mattias


-- 
Bioinformatician
Netherlands Institute of Ecology (NIOO-KNAW)
Wageningen, the Netherlands
___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/