Re: [galaxy-dev] Importing existing data into Galaxy

[Greg Von Kuster]

The files are not touched - metadata is generated for the files, but the 
content of the files themselves is not changed.

On Oct 28, 2011, at 3:21 PM, Oren Livne wrote:

> Dear Greg,
> Great, thanks! Just one more question: if I uncheck the "copy data files" 
> option, does Galaxy change the original day files in any way when it loads 
> them into a data library (e.g., prepend them with some metadata header), or 
> does it leave them intact and stores the metadata separately?
> Thanks,
> Oren
> 
> On 10/28/2011 1:44 PM, Greg Von Kuster wrote:
>> Hello Oren,
>> 
>> We use a sort of informal approach to implementing new Galaxy API features 
>> (they're implemented when the need arises), so unfortunately it doesn't look 
>> like the features you are looking for exist yet. If you'd like to contribute 
>> them, the community would certainly appreciate it!
>> 
>> Greg Von Kuster
>> 
>> On Oct 28, 2011, at 12:45 PM, Oren Livne wrote:
>> 
>>> Dear Greg,
>>> 
>>> Thank you so much, this is exactly what I need. I will use the API access 
>>> (scripts/api/*.py) to create the new data library. I have two more 
>>> questions:
>>> - Is there an API function for deleting a library?
>>> - Is there an API function to make a library accessible only to the user 
>>> whose API key is passed in?
>>> 
>>> Oren
>>> 
>>> On 10/27/2011 7:55 PM, Greg Von Kuster wrote:
 Oren,
 
 The best way to do this would be to use galaxy data libraries, uploading 
 files from file system paths and not copying the files into the Galaxy 
 file location.
 
 The following provides all of the details about data libraries:
 
 http://wiki.g2.bx.psu.edu/Admin/Data%20Libraries/Libraries
 
 
 The following provides specific details about the various options for 
 uploading files to data libraries.
 
 http://wiki.g2.bx.psu.edu/Admin/Data%20Libraries/Uploading%20Library%20Files
 
 Let us know if you encounter any issues.
 
 Greg Von Kuster
 
 
 On Oct 27, 2011, at 4:31 PM, Oren Livne wrote:
 
> Dear All,
> 
> We have a large collection of large data files on our shared file system. 
> We would like to make a subset of them available to a galaxy user session 
> (based on user privileges; different subsets for different users). We 
> want to leave files in their original locations and point galaxy to their 
> paths without copying them.  What is the best option for implementing 
> this bridge? Remote data source/data library uploaded on the fly/other?
> 
> Thanks,
> Oren
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
> 
> http://lists.bx.psu.edu/
 Greg Von Kuster
 Galaxy Development Team
 g...@bx.psu.edu
 
 
 
>> Greg Von Kuster
>> Galaxy Development Team
>> g...@bx.psu.edu
>> 
>> 
>> 
> 

Greg Von Kuster
Galaxy Development Team
g...@bx.psu.edu




___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] Importing existing data into Galaxy

[Greg Von Kuster]

Hello Oren,

We use a sort of informal approach to implementing new Galaxy API features 
(they're implemented when the need arises), so unfortunately it doesn't look 
like the features you are looking for exist yet. If you'd like to contribute 
them, the community would certainly appreciate it!

Greg Von Kuster

On Oct 28, 2011, at 12:45 PM, Oren Livne wrote:

> Dear Greg,
> 
> Thank you so much, this is exactly what I need. I will use the API access 
> (scripts/api/*.py) to create the new data library. I have two more questions:
> - Is there an API function for deleting a library?
> - Is there an API function to make a library accessible only to the user 
> whose API key is passed in?
> 
> Oren
> 
> On 10/27/2011 7:55 PM, Greg Von Kuster wrote:
>> Oren,
>> 
>> The best way to do this would be to use galaxy data libraries, uploading 
>> files from file system paths and not copying the files into the Galaxy file 
>> location.
>> 
>> The following provides all of the details about data libraries:
>> 
>> http://wiki.g2.bx.psu.edu/Admin/Data%20Libraries/Libraries
>> 
>> 
>> The following provides specific details about the various options for 
>> uploading files to data libraries.
>> 
>> http://wiki.g2.bx.psu.edu/Admin/Data%20Libraries/Uploading%20Library%20Files
>> 
>> Let us know if you encounter any issues.
>> 
>> Greg Von Kuster
>> 
>> 
>> On Oct 27, 2011, at 4:31 PM, Oren Livne wrote:
>> 
>>> Dear All,
>>> 
>>> We have a large collection of large data files on our shared file system. 
>>> We would like to make a subset of them available to a galaxy user session 
>>> (based on user privileges; different subsets for different users). We want 
>>> to leave files in their original locations and point galaxy to their paths 
>>> without copying them.  What is the best option for implementing this 
>>> bridge? Remote data source/data library uploaded on the fly/other?
>>> 
>>> Thanks,
>>> Oren
>>> ___
>>> Please keep all replies on the list by using "reply all"
>>> in your mail client.  To manage your subscriptions to this
>>> and other Galaxy lists, please use the interface at:
>>> 
>>> http://lists.bx.psu.edu/
>> Greg Von Kuster
>> Galaxy Development Team
>> g...@bx.psu.edu
>> 
>> 
>> 
> 

Greg Von Kuster
Galaxy Development Team
g...@bx.psu.edu




___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] Some questions about cloudman

[Enis Afgan]

On Fri, Oct 28, 2011 at 5:50 PM, Cittaro Davide wrote:

>
> On Oct 28, 2011, at 5:35 PM, James Taylor wrote:
>
> It currently uses a tiny amount of S3 storage just to save configuration
> information about your instance.
>
>
> Ok.. never used AWS, actually, I didn't know S3 holds the information. I
> guess I will have to read some how-to
>

This configuration is something cloudman does behind the scenes so nothing
there to worry about much.


> Long term though we plan to move dataset storage over to S3 as well.
>
>
> Mmm... I've just had a chat with an AWS engineer, he told me that every
> operation on S3-stored data goes through a download/upload process... isn't
> that a PITA for data analysis? I guess S3 is for "static"  data
>

> EBS has limits, S3 is more durable and scalable.
>
>
> Which limits (in addition to the 1 Tb size)? I know these are not
> Galaxy-related questions but you are the best people I can ask :-)
>

The 1TB size is the primary issue, especially with NGS data. That's why
we're looking into S3 as a way to offload some of the data size issues while
handling it all behind the scenes. Other than that, the only other comment
is that these instances are independent and self standing so if any
customizations are required, manual effort will be required (but this
applies to any local instance).

Let us know if you have any more questions,
Enis


>
> d
>
>/*
> Davide Cittaro, PhD
>
> Head of Bioinformatics Core
> Center for Translational Genomics and Bioinformatics
> San Raffaele Scientific Institute
> Via Olgettina 58
> 20132 Milano
> Italy
>
> Office: +39 02 26439140
> Mail: cittaro.dav...@hsr.it
> Skype: daweonline
> */
>
>
>
>
>
>
>
>
>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>
>  http://lists.bx.psu.edu/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] Some questions about cloudman

[Cittaro Davide]

On Oct 28, 2011, at 5:35 PM, James Taylor wrote:

It currently uses a tiny amount of S3 storage just to save configuration
information about your instance.


Ok.. never used AWS, actually, I didn't know S3 holds the information. I guess 
I will have to read some how-to

Long term though we plan to move dataset storage over to S3 as well.

Mmm... I've just had a chat with an AWS engineer, he told me that every 
operation on S3-stored data goes through a download/upload process... isn't 
that a PITA for data analysis? I guess S3 is for "static"  data

EBS has limits, S3 is more durable and scalable.

Which limits (in addition to the 1 Tb size)? I know these are not 
Galaxy-related questions but you are the best people I can ask :-)

d

/*
Davide Cittaro, PhD

Head of Bioinformatics Core
Center for Translational Genomics and Bioinformatics
San Raffaele Scientific Institute
Via Olgettina 58
20132 Milano
Italy

Office: +39 02 26439140
Mail: cittaro.dav...@hsr.it
Skype: daweonline
*/










___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] Some questions about cloudman

[James Taylor]

It currently uses a tiny amount of S3 storage just to save configuration
information about your instance.

Long term though we plan to move dataset storage over to S3 as well. EBS
has limits, S3 is more durable and scalable.

Excerpts from Cittaro Davide's message of 2011-10-28 14:32:42 +:
> Hi there, I'm in the middle of a decision: should I go into the cloud or not? 
> I'm reading the docs on galaxy wiki, and I see that besides EC2, EBS I need 
> S3 storage. What is that for (meaning: why galaxy needs S3)?
> @people already using it: how to you send NGS data? I need some feedback! :-)
> 
> Thanks
> 
> d
> /*
> Davide Cittaro, PhD
> 
> Head of Bioinformatics Core
> Center for Translational Genomics and Bioinformatics
> San Raffaele Scientific Institute
> Via Olgettina 58
> 20132 Milano
> Italy
> 
> Office: +39 02 26439140
> Mail: cittaro.dav...@hsr.it
> Skype: daweonline
> */
-- 
James Taylor, Assistant Professor, Biology / Computer Science, Emory University
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

[galaxy-dev] Some questions about cloudman

[Cittaro Davide]

Hi there, I'm in the middle of a decision: should I go into the cloud or not? 
I'm reading the docs on galaxy wiki, and I see that besides EC2, EBS I need S3 
storage. What is that for (meaning: why galaxy needs S3)?
@people already using it: how to you send NGS data? I need some feedback! :-)

Thanks

d
/*
Davide Cittaro, PhD

Head of Bioinformatics Core
Center for Translational Genomics and Bioinformatics
San Raffaele Scientific Institute
Via Olgettina 58
20132 Milano
Italy

Office: +39 02 26439140
Mail: cittaro.dav...@hsr.it
Skype: daweonline
*/










___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] Importing Historys in other Galaxy instances

[Stefanie Ververs]

On 26.10.2011 16:13, Jeremy Goecks wrote:

Hi Ross,

I checked the following:

wget  works, I can download the file. It's not saved as.tar.gz but as 
"export_archive?id=a69ee3e00cb4d02c". Anyway, I can unpack the archive with tar xfvz.

hg head:

Änderung:5955:949e4f5fa03a
Marke:   tip
Nutzer:  Nate Coraor
Datum:   Mon Aug 29 14:42:04 2011 -0400
Zusammenfassung: Missing import in library_contents.py

Where can I find the file (I assume?) paster.log?

Best place to look for the error is in the job table. Try this query for your 
database:

--
select id, state, command_line, stdout, stderr from job where 
tool_id='__EXPORT_HISTORY__' order by id desc;
--

and look at the stderr column for some information about the problem. Please 
send us the command line and the stderr for the problematic jobs; this should 
help us figure out what's going on.

Thanks,
J.


Hi Jeremy,

76 | ok| python 
/home/cweinberg/Software/galaxy-dist/lib/galaxy/tools/imp_exp/unpack_tar_gz_archive.py 
http://main.g2.bx.psu.edu/history/export_archive?id=a69ee3e00cb4d02c 
./database/tmp/tmpbxuXL2 --url  | Error unpacking tar/gz archive: [Errno 
2] No such file or directory: './database/tmp/tmpbxuXL2/tmpBn0UbN' file '', mode 'w' at 0x7f1c2cc991e0> | Error cleaning up history 
import job: [Errno 2] No such file or directory: 
'./database/tmp/tmpbxuXL2/history_attrs.txt'


This is one of the db entries, the other ones are similar (different 
tmp-directories).


Looks like a problem with the temporary directories..


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

 http://lists.bx.psu.edu/