On Dec 31, 2011, at 6:38 PM, Ted Goldstein wrote:
> This bug irritated me, so I fixed it. Essentially add_file() in upload.py
> is not in on the joke that local dirs are relative paths and need the
> absolute path tacked onto it.
> Is there a written process on how to submit the fix? I could not find it.
Is this the case when library_import_dir in the config file is relative? I've
always used an absolute path there. I suppose it couldn't hurt to make it
> diff -r 21b645303c02 tools/data_source/upload.py
> --- a/tools/data_source/upload.py Thu Dec 22 13:54:33 2011 -0500
> +++ b/tools/data_source/upload.py Sat Dec 31 15:29:45 2011 -0800
> @@ -74,7 +74,10 @@
> id, files_path, path = arg.split( ':', 2 )
> rval[int( id )] = ( path, files_path )
> return rval
> -def add_file( dataset, registry, json_file, output_path ):
> +import pdb
> +def add_file( dataset, registry, json_file, output_path, root_dir):
> data_type = None
> line_count = None
> converted_path = None
> @@ -94,7 +97,10 @@
> file_err( 'Unable to fetch %s\n%s' % ( dataset.path, str( e ) ),
> dataset, json_file )
> dataset.path = temp_name
> - # See if we have an empty file
> + if dataset.type == 'server_dir' and not os.path.isabs( dataset.path):
> + dataset.path = os.path.join( root_dir, dataset.path )
> if not os.path.exists( dataset.path ):
> file_err( 'Uploaded temporary file (%s) does not exist.' %
> dataset.path, dataset, json_file )
> @@ -384,7 +390,7 @@
> files_path = output_paths[int( dataset.dataset_id )]
> add_composite_file( dataset, registry, json_file, output_path,
> files_path )
> - add_file( dataset, registry, json_file, output_path )
> + add_file( dataset, registry, json_file, output_path ,
> # clean up paramfile
> os.remove( sys.argv )
> [ted@tap galaxy-central]$ !v
> On Dec 29, 2011, at 1:22 AM, Ted Goldstein wrote:
>> Hi there,
>> Here are three interrelated issues.
>> I am trying to use Galaxy with some large cancer genomic datasets here at
>> UCSC and do some systems biology. I have petabyte size dataset data
>> libraries which will constantly be in flux at the edges. I would prefer to
>> just have the Galaxy read the metadata from the file system for large
>> datasets without using the database. Is there a convenient api boundary to
>> write an adapter to the dataset object interface?
>> In the meantime, I am going to try to just import day using the link. Its
>> great that this feature is in already When I import into a couple of a
>> modest megabyte size dataset using "Link to files without copying to
>> Galaxy" option, the status never changes from "queued". Is this a bug? Is
>> there a known work around? I have many large datasets.
>> Also, it takes a long time to expand the dataset name link. (My experiment
>> on import is a data tree of about a thousand files). Is this a known bug?
>> Please keep all replies on the list by using "reply all"
>> in your mail client. To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
> Please keep all replies on the list by using "reply all"
> in your mail client. To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at: