Re: [galaxy-dev] Problems uploading files to galaxy

2016-03-15 Thread Lemley, Rob 
Thanks Peter, that's really helpful. I'm glad we're not the only ones running 
into this.

-Rob

Rob Lemley
Systems Administrator – CIRC
University of Rochester

-Original Message-
From: Peter Cock [mailto:p.j.a.c...@googlemail.com] 
Sent: Tuesday, March 15, 2016 10:34 AM
To: John Chilton <jmchil...@gmail.com>
Cc: Lemley, Rob <rob.lem...@rochester.edu>; galaxy-dev@lists.galaxyproject.org
Subject: Re: [galaxy-dev] Problems uploading files to galaxy

On Tue, Mar 15, 2016 at 2:21 PM, John Chilton <jmchil...@gmail.com> wrote:
> If you are running jobs as "the real user" with drmaa (using the chown
> script)  - you will probably want to setup a separate job destination
> using the local runner for upload jobs. The upload tool uses files
> that are created outside of Galaxy and these aren't modeled well.
>
> Does this help?
>
> -John

We found the opposite - if using "the real user", the Galaxy v16.01
would do a chown on the datafiles for DRMAA jobs (good) but also
for local jobs (bad):

https://urldefense.proofpoint.com/v2/url?u=http-3A__dev.list.galaxyproject.org_external-2Dchown-2Dscript-2Dpy-2Dand-2Dthe-2Dlocal-2Djob-2Drunner-2Dtt4668738.html=BQIBaQ=kbmfwr1Yojg42sGEpaQh5ofMHBeTl9EI2eaqQZhHbOU=d-Zn2Sbc-XheRlOYkSFXPupZ8ZmuYq52isQBiI3MBWA=rMI9zRSmgepuzJAMtXaa8aFFhpcuYbKx2r_h3aC6wIg=IZ8BQgQLJhvOnBc_xzGzMYMVVETGgFUhcBM_tNlMXRE=
 

I will reply on that thread now with an update...

Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Problems uploading files to galaxy

2016-03-15 Thread Lemley, Rob 
Hi John,

Thanks for getting back to me.

If I'm understanding the XML files correctly, the upload tool is already 
configured to run as a local job. From job_conf.xml:



where "local" is our local job runner and "upload1" is defined in 
data_source/upload.xml.

-Rob

Rob Lemley
Systems Administrator – CIRC
University of Rochester

-Original Message-
From: John Chilton [mailto:jmchil...@gmail.com] 
Sent: Tuesday, March 15, 2016 10:21 AM
To: Lemley, Rob <rob.lem...@rochester.edu>
Cc: galaxy-dev@lists.galaxyproject.org
Subject: Re: [galaxy-dev] Problems uploading files to galaxy

If you are running jobs as "the real user" with drmaa (using the chown
script)  - you will probably want to setup a separate job destination
using the local runner for upload jobs. The upload tool uses files
that are created outside of Galaxy and these aren't modeled well.

Does this help?

-John


On Fri, Mar 11, 2016 at 6:45 PM, Lemley, Rob <rob.lem...@rochester.edu> wrote:
> Hi everyone,
>
> I didn’t see any response to my email below. Does anyone have any thoughts? 
> I'm pretty stuck.
>
> Thanks again,
>
> -Rob
>
> Rob Lemley
> Systems Administrator – CIRC
> University of Rochester
>
> -Original Message-
> From: galaxy-dev [mailto:galaxy-dev-boun...@lists.galaxyproject.org] On 
> Behalf Of Lemley, Rob
> Sent: Monday, March 7, 2016 2:03 PM
> To: galaxy-dev@lists.galaxyproject.org
> Subject: [galaxy-dev] Problems uploading files to galaxy
>
> Hi,
>
> I'm hoping you can help out with a problem we are experiencing uploading 
> files to our own Galaxy instance.
>
> When a user tries to upload a file to Galaxy through their web browser, the 
> job errors out:
>
> Traceback (most recent call last):
>   File "/gpfs/fs1/sfw/cibem_galaxy/app/galaxy/tools/data_source/upload.py", 
> line 430, in 
> __main__()
>   File "/gpfs/fs1/sfw/cibem_galaxy/app/galaxy/tools/data_source/upload.py", 
> line 407, in __main__
> for line in open( sys.argv[3], 'r' ):
> IOError: [Errno 13] Permission denied: 
> '/scratch/XX/XX/galaxy/tmp/tmpSlG2pC'
>
> The permissions on the referenced file:
> -rw---+ 1 rlemley  rlemley  317 Mar  7 13:42 tmpSlG2pC
>
> As the Galaxy daemons run as XX_galaxy, it does not have permission to 
> read the file.
>
> I tried modifying external_chown_script.py so that XX_galaxy has 
> permission to read that file, but that caused problems elsewhere so I backed 
> out the changes.
>
> Any suggestions?
>
> Many thanks,
>
> -Rob
>
>
> Rob Lemley
> Systems Administrator - CIRC
> University of Rochester
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.galaxyproject.org_=BQIGaQ=kbmfwr1Yojg42sGEpaQh5ofMHBeTl9EI2eaqQZhHbOU=d-Zn2Sbc-XheRlOYkSFXPupZ8ZmuYq52isQBiI3MBWA=xTXPxT_d1oq2TJoB3W7P0xfhV52TZzeUjoGhvts8xNU=yW1eNcfXnR8Rut3-T62JBQMRJXblVpOZEA0w7DxiJ04=
>
> To search Galaxy mailing lists use the unified search at:
>   
> https://urldefense.proofpoint.com/v2/url?u=http-3A__galaxyproject.org_search_mailinglists_=BQIGaQ=kbmfwr1Yojg42sGEpaQh5ofMHBeTl9EI2eaqQZhHbOU=d-Zn2Sbc-XheRlOYkSFXPupZ8ZmuYq52isQBiI3MBWA=xTXPxT_d1oq2TJoB3W7P0xfhV52TZzeUjoGhvts8xNU=sQNOB73SDBa-fat4oHkeYucMCsvheL3eXDp3qoUndLU=
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.galaxyproject.org_=BQIFaQ=kbmfwr1Yojg42sGEpaQh5ofMHBeTl9EI2eaqQZhHbOU=d-Zn2Sbc-XheRlOYkSFXPupZ8ZmuYq52isQBiI3MBWA=CEkTHfFOPDidaSf14okOW1C-PgZzhlhbJVrEhanyil0=1Eql06w3_X1XFsEnMGdKCDlZus4pqc2vqwICDnPPNNs=
>  
>
> To search Galaxy mailing lists use the unified search at:
>   
> https://urldefense.proofpoint.com/v2/url?u=http-3A__galaxyproject.org_search_mailinglists_=BQIFaQ=kbmfwr1Yojg42sGEpaQh5ofMHBeTl9EI2eaqQZhHbOU=d-Zn2Sbc-XheRlOYkSFXPupZ8ZmuYq52isQBiI3MBWA=CEkTHfFOPDidaSf14okOW1C-PgZzhlhbJVrEhanyil0=XiocBa1yVec6l9Im_dv0CZ93S_3tEfxU9tDAH5iHWnk=
>  
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Problems uploading files to galaxy

2016-03-15 Thread Peter Cock 
On Tue, Mar 15, 2016 at 2:21 PM, John Chilton  wrote:
> If you are running jobs as "the real user" with drmaa (using the chown
> script)  - you will probably want to setup a separate job destination
> using the local runner for upload jobs. The upload tool uses files
> that are created outside of Galaxy and these aren't modeled well.
>
> Does this help?
>
> -John

We found the opposite - if using "the real user", the Galaxy v16.01
would do a chown on the datafiles for DRMAA jobs (good) but also
for local jobs (bad):

http://dev.list.galaxyproject.org/external-chown-script-py-and-the-local-job-runner-tt4668738.html

I will reply on that thread now with an update...

Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Problems uploading files to galaxy

2016-03-15 Thread John Chilton 
If you are running jobs as "the real user" with drmaa (using the chown
script)  - you will probably want to setup a separate job destination
using the local runner for upload jobs. The upload tool uses files
that are created outside of Galaxy and these aren't modeled well.

Does this help?

-John


On Fri, Mar 11, 2016 at 6:45 PM, Lemley, Rob <rob.lem...@rochester.edu> wrote:
> Hi everyone,
>
> I didn’t see any response to my email below. Does anyone have any thoughts? 
> I'm pretty stuck.
>
> Thanks again,
>
> -Rob
>
> Rob Lemley
> Systems Administrator – CIRC
> University of Rochester
>
> -Original Message-
> From: galaxy-dev [mailto:galaxy-dev-boun...@lists.galaxyproject.org] On 
> Behalf Of Lemley, Rob
> Sent: Monday, March 7, 2016 2:03 PM
> To: galaxy-dev@lists.galaxyproject.org
> Subject: [galaxy-dev] Problems uploading files to galaxy
>
> Hi,
>
> I'm hoping you can help out with a problem we are experiencing uploading 
> files to our own Galaxy instance.
>
> When a user tries to upload a file to Galaxy through their web browser, the 
> job errors out:
>
> Traceback (most recent call last):
>   File "/gpfs/fs1/sfw/cibem_galaxy/app/galaxy/tools/data_source/upload.py", 
> line 430, in 
> __main__()
>   File "/gpfs/fs1/sfw/cibem_galaxy/app/galaxy/tools/data_source/upload.py", 
> line 407, in __main__
> for line in open( sys.argv[3], 'r' ):
> IOError: [Errno 13] Permission denied: 
> '/scratch/XX/XX/galaxy/tmp/tmpSlG2pC'
>
> The permissions on the referenced file:
> -rw---+ 1 rlemley  rlemley  317 Mar  7 13:42 tmpSlG2pC
>
> As the Galaxy daemons run as XX_galaxy, it does not have permission to 
> read the file.
>
> I tried modifying external_chown_script.py so that XX_galaxy has 
> permission to read that file, but that caused problems elsewhere so I backed 
> out the changes.
>
> Any suggestions?
>
> Many thanks,
>
> -Rob
>
>
> Rob Lemley
> Systems Administrator - CIRC
> University of Rochester
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.galaxyproject.org_=BQIGaQ=kbmfwr1Yojg42sGEpaQh5ofMHBeTl9EI2eaqQZhHbOU=d-Zn2Sbc-XheRlOYkSFXPupZ8ZmuYq52isQBiI3MBWA=xTXPxT_d1oq2TJoB3W7P0xfhV52TZzeUjoGhvts8xNU=yW1eNcfXnR8Rut3-T62JBQMRJXblVpOZEA0w7DxiJ04=
>
> To search Galaxy mailing lists use the unified search at:
>   
> https://urldefense.proofpoint.com/v2/url?u=http-3A__galaxyproject.org_search_mailinglists_=BQIGaQ=kbmfwr1Yojg42sGEpaQh5ofMHBeTl9EI2eaqQZhHbOU=d-Zn2Sbc-XheRlOYkSFXPupZ8ZmuYq52isQBiI3MBWA=xTXPxT_d1oq2TJoB3W7P0xfhV52TZzeUjoGhvts8xNU=sQNOB73SDBa-fat4oHkeYucMCsvheL3eXDp3qoUndLU=
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Problems uploading files to galaxy

2016-03-11 Thread Lemley, Rob 
Hi everyone,

I didn’t see any response to my email below. Does anyone have any thoughts? I'm 
pretty stuck.

Thanks again,

-Rob

Rob Lemley
Systems Administrator – CIRC
University of Rochester

-Original Message-
From: galaxy-dev [mailto:galaxy-dev-boun...@lists.galaxyproject.org] On Behalf 
Of Lemley, Rob
Sent: Monday, March 7, 2016 2:03 PM
To: galaxy-dev@lists.galaxyproject.org
Subject: [galaxy-dev] Problems uploading files to galaxy

Hi,

I'm hoping you can help out with a problem we are experiencing uploading files 
to our own Galaxy instance.

When a user tries to upload a file to Galaxy through their web browser, the job 
errors out:

Traceback (most recent call last):
  File "/gpfs/fs1/sfw/cibem_galaxy/app/galaxy/tools/data_source/upload.py", 
line 430, in 
__main__()
  File "/gpfs/fs1/sfw/cibem_galaxy/app/galaxy/tools/data_source/upload.py", 
line 407, in __main__
for line in open( sys.argv[3], 'r' ):
IOError: [Errno 13] Permission denied: 
'/scratch/XX/XX/galaxy/tmp/tmpSlG2pC'

The permissions on the referenced file:
-rw---+ 1 rlemley  rlemley  317 Mar  7 13:42 tmpSlG2pC

As the Galaxy daemons run as XX_galaxy, it does not have permission to read 
the file.

I tried modifying external_chown_script.py so that XX_galaxy has permission 
to read that file, but that caused problems elsewhere so I backed out the 
changes.

Any suggestions?

Many thanks,

-Rob


Rob Lemley
Systems Administrator - CIRC
University of Rochester

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.galaxyproject.org_=BQIGaQ=kbmfwr1Yojg42sGEpaQh5ofMHBeTl9EI2eaqQZhHbOU=d-Zn2Sbc-XheRlOYkSFXPupZ8ZmuYq52isQBiI3MBWA=xTXPxT_d1oq2TJoB3W7P0xfhV52TZzeUjoGhvts8xNU=yW1eNcfXnR8Rut3-T62JBQMRJXblVpOZEA0w7DxiJ04=
 

To search Galaxy mailing lists use the unified search at:
  
https://urldefense.proofpoint.com/v2/url?u=http-3A__galaxyproject.org_search_mailinglists_=BQIGaQ=kbmfwr1Yojg42sGEpaQh5ofMHBeTl9EI2eaqQZhHbOU=d-Zn2Sbc-XheRlOYkSFXPupZ8ZmuYq52isQBiI3MBWA=xTXPxT_d1oq2TJoB3W7P0xfhV52TZzeUjoGhvts8xNU=sQNOB73SDBa-fat4oHkeYucMCsvheL3eXDp3qoUndLU=
 
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/