Re: [galaxy-dev] download dataset from the proxy-ed Galaxy failed with broken link
If I downloaded with curl -i, the page it returned was: pwbcad@gamma01:tmp$ curl -i http://galaxy-qa.garvan.unsw.edu.au/datasets/dc097fa4e7ad53f6/display?to_ext=fastqsanger HTTP/1.1 302 Found Date: Tue, 30 Oct 2012 04:25:21 GMT Server: PasteWSGIServer/0.5 Python/2.6.6 content-type: text/html location: /root?dataset_id=dc097fa4e7ad53f6filename=None Set-Cookie: galaxysession=c6ca0ddb55be603a4107d5e4793b9ce28e93ce1e646977c209d1af46a242147b57953de37776fcc2; expires=Mon, 28-Jan-2013 15:25:21 GMT; httponly; Max-Age=7776000; Path=/; Version=1 Vary: Accept-Encoding Transfer-Encoding: chunked html headtitleFound/title/head body h1Found/h1 pThe resource was found at a href=/root?dataset_id=dc097fa4e7ad53f6amp;filename=None/root?dataset_id=dc097fa4e7ad53f6amp;filename=None/a; you should be redirected automatically. !-- --/p hr noshade div align=rightWSGI Server/div /body /html Don't know why the apache returned 302 only when access via wget or curl. Derrick On Sat, Oct 27, 2012 at 7:44 AM, Derrick Lin klin...@gmail.com wrote: Hi Nate, I checked and what downloaded is a webpage, I opened it on browser, and screenshot attached. Basically it displays error saying File not found. But dataset download works fine via browser. Cheers, Derrick On Fri, Oct 26, 2012 at 11:44 PM, Nate Coraor n...@bx.psu.edu wrote: On Oct 24, 2012, at 11:15 PM, Derrick Lin wrote: Hi guys, On our Galaxy is proxy-ed via Apache according to the wiki guide, the normal download dataset via browser is working fine. However, when I tried download from command line via wget or curl -O (I copied the dataset URL by right clicking the floppy disk icon). And here is the wget output: pwbcad@gamma01:tmp$ wget http://galaxy.demo.edu.au/galaxy/datasets/ccc5aff28d2c8616/display?to_ext=fastq --2012-10-25 10:57:42-- http://galaxy.demo.edu.au/galaxy/datasets/ccc5aff28d2c8616/display?to_ext=fastq Resolving galaxy.demo.edu.au... 1xx.x4.1xx.xx5 Connecting to galaxy.demo.edu.au|1xx.x4.1xx.xx5|:80... connected. HTTP request sent, awaiting response... 302 Found Location: /galaxy/root?dataset_id=ccc5aff28d2c8616filename=None [following] --2012-10-25 10:57:42-- http://galaxy.demo.edu.au/galaxy/root?dataset_id=ccc5aff28d2c8616filename=None Connecting to galaxy.demo.edu.au|129.94.136.205|:80... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/html] Saving to: root?dataset_id=ccc5aff28d2c8616filename=None.1 [ = ] 28,476 --.-K/s in 0.001s 2012-10-25 10:57:42 (31.4 MB/s) - root?dataset_id=ccc5aff28d2c8616filename=None.1 saved [28476] It seems like the Apache returned another URL (returned 302, probably due to proxy) and the new URL is broken. Hi Derrick, What are the contents of the file that wget saved? This should give you a clue as to what's going on. --nate Thanks in advance. Cheers, Derrick ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] download dataset from the proxy-ed Galaxy failed with broken link
Hi Ross, Good point and it makes sense in term of security. But I just tried it, and it didn't help. I captured these two from the Paste log (not apache log): xx.xx.1xx.xx - - [30/Oct/2012:15:42:17 +1100] GET /galaxy_dev/datasets/10b866a1b258d1f1/display?to_ext=fastqsanger HTTP/1.1 302 - - Wget/1.12 (linux-gnu) galaxy.web.framework DEBUG 2012-10-30 15:42:17,676 Error: this request returned None from get_history(): http://pwbc.garvan.unsw.edu.au/galaxy_dev/root xx.xx.1xx.xx - - [30/Oct/2012:15:42:17 +1100] GET /galaxy_dev/root?dataset_id=10b866a1b258d1f1filename=None HTTP/1.1 200 - - Wget/1.12 (linux-gnu) Another observation is, I can wget my own dataset from the main Galaxy straight away without providing any credential. Maybe something wrong with my universe ini file. Derrick On Tue, Oct 30, 2012 at 3:32 PM, Ross ross.laza...@gmail.com wrote: That url requires a login from here - have you tried supplying it - eg curl -i -u user:password target On Tue, Oct 30, 2012 at 3:27 PM, Derrick Lin klin...@gmail.com wrote: If I downloaded with curl -i, the page it returned was: pwbcad@gamma01:tmp$ curl -i http://galaxy-qa.garvan.unsw.edu.au/datasets/dc097fa4e7ad53f6/display?to_ext=fastqsanger HTTP/1.1 302 Found Date: Tue, 30 Oct 2012 04:25:21 GMT Server: PasteWSGIServer/0.5 Python/2.6.6 content-type: text/html location: /root?dataset_id=dc097fa4e7ad53f6filename=None Set-Cookie: galaxysession=c6ca0ddb55be603a4107d5e4793b9ce28e93ce1e646977c209d1af46a242147b57953de37776fcc2; expires=Mon, 28-Jan-2013 15:25:21 GMT; httponly; Max-Age=7776000; Path=/; Version=1 Vary: Accept-Encoding Transfer-Encoding: chunked html headtitleFound/title/head body h1Found/h1 pThe resource was found at a href=/root?dataset_id=dc097fa4e7ad53f6amp;filename=None/root?dataset_id=dc097fa4e7ad53f6amp;filename=None/a; you should be redirected automatically. !-- --/p hr noshade div align=rightWSGI Server/div /body /html Don't know why the apache returned 302 only when access via wget or curl. Derrick On Sat, Oct 27, 2012 at 7:44 AM, Derrick Lin klin...@gmail.com wrote: Hi Nate, I checked and what downloaded is a webpage, I opened it on browser, and screenshot attached. Basically it displays error saying File not found. But dataset download works fine via browser. Cheers, Derrick On Fri, Oct 26, 2012 at 11:44 PM, Nate Coraor n...@bx.psu.edu wrote: On Oct 24, 2012, at 11:15 PM, Derrick Lin wrote: Hi guys, On our Galaxy is proxy-ed via Apache according to the wiki guide, the normal download dataset via browser is working fine. However, when I tried download from command line via wget or curl -O (I copied the dataset URL by right clicking the floppy disk icon). And here is the wget output: pwbcad@gamma01:tmp$ wget http://galaxy.demo.edu.au/galaxy/datasets/ccc5aff28d2c8616/display?to_ext=fastq --2012-10-25 10:57:42-- http://galaxy.demo.edu.au/galaxy/datasets/ccc5aff28d2c8616/display?to_ext=fastq Resolving galaxy.demo.edu.au... 1xx.x4.1xx.xx5 Connecting to galaxy.demo.edu.au|1xx.x4.1xx.xx5|:80... connected. HTTP request sent, awaiting response... 302 Found Location: /galaxy/root?dataset_id=ccc5aff28d2c8616filename=None [following] --2012-10-25 10:57:42-- http://galaxy.demo.edu.au/galaxy/root?dataset_id=ccc5aff28d2c8616filename=None Connecting to galaxy.demo.edu.au|129.94.136.205|:80... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/html] Saving to: root?dataset_id=ccc5aff28d2c8616filename=None.1 [ = ] 28,476 --.-K/s in 0.001s 2012-10-25 10:57:42 (31.4 MB/s) - root?dataset_id=ccc5aff28d2c8616filename=None.1 saved [28476] It seems like the Apache returned another URL (returned 302, probably due to proxy) and the new URL is broken. Hi Derrick, What are the contents of the file that wget saved? This should give you a clue as to what's going on. --nate Thanks in advance. Cheers, Derrick ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Ross Lazarus MBBS MPH; Head, Medical Bioinformatics, BakerIDI; Tel: +61 385321444 http://scholar.google.com/citations?hl=enuser=UCUuEM4J ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this
Re: [galaxy-dev] download dataset from the proxy-ed Galaxy failed with broken link
On Tue, Oct 30, 2012 at 4:03 PM, Ross ross.laza...@gmail.com wrote: Doesn't make sense that you can wget directly from the paste process if you have security properly configured! That's what you can do with the main Galaxy site. Not sure if that's intentional though. You can try wget --no-check-certificate https://main.g2.bx.psu.edu/datasets/851fbe5b7ff576a6/display?to_ext=tabular See if you can download my own dataset without providing any credential. Do you have Apache authenticating and passing headers through to the paste process - this section in universe_wsgi.ini # User authentication can be delegated to an upstream proxy server (usually # Apache). The upstream proxy should set a REMOTE_USER header in the request. # Enabling remote user disables regular logins. For more information, see: # http://wiki.g2.bx.psu.edu/Admin/Config/Apache%20Proxy use_remote_user = True If so, you should not be able to access anything via the paste process directly without adding authentication headers. Once that's fixed (you do NOT want anyone to be able to do what you can do - it bypasses all security!) the apache configuration will probably need tweaking. It's hard to advise - it's mostly voodoo IMHO - do you have an apache fluent sysadmin? We don't use Apache for authentication. Thus use_remote_user = False. Will read more about this remote user thing. Cheers Derrick ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] download dataset from the proxy-ed Galaxy failed with broken link
On Oct 24, 2012, at 11:15 PM, Derrick Lin wrote: Hi guys, On our Galaxy is proxy-ed via Apache according to the wiki guide, the normal download dataset via browser is working fine. However, when I tried download from command line via wget or curl -O (I copied the dataset URL by right clicking the floppy disk icon). And here is the wget output: pwbcad@gamma01:tmp$ wget http://galaxy.demo.edu.au/galaxy/datasets/ccc5aff28d2c8616/display?to_ext=fastq --2012-10-25 10:57:42-- http://galaxy.demo.edu.au/galaxy/datasets/ccc5aff28d2c8616/display?to_ext=fastq Resolving galaxy.demo.edu.au... 1xx.x4.1xx.xx5 Connecting to galaxy.demo.edu.au|1xx.x4.1xx.xx5|:80... connected. HTTP request sent, awaiting response... 302 Found Location: /galaxy/root?dataset_id=ccc5aff28d2c8616filename=None [following] --2012-10-25 10:57:42-- http://galaxy.demo.edu.au/galaxy/root?dataset_id=ccc5aff28d2c8616filename=None Connecting to galaxy.demo.edu.au|129.94.136.205|:80... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/html] Saving to: root?dataset_id=ccc5aff28d2c8616filename=None.1 [ = ] 28,476 --.-K/s in 0.001s 2012-10-25 10:57:42 (31.4 MB/s) - root?dataset_id=ccc5aff28d2c8616filename=None.1 saved [28476] It seems like the Apache returned another URL (returned 302, probably due to proxy) and the new URL is broken. Hi Derrick, What are the contents of the file that wget saved? This should give you a clue as to what's going on. --nate Thanks in advance. Cheers, Derrick ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
