[galaxy-user] Problem uploading files gt 1 Mbyte to Galaxy Cloud
Hi, I'm building a Galaxy Cloud implementation, and running into a problem uploading data files 1 Mbyte, or 1,048,576 bytes. I've started Galaxy Cloudman and a single server, all according to to the Wiki. I can upload fasta format files 1048756 bytes, and general CSV format peptide files, and indeed junk data files built with variants of jot -r -c 103 a z | rs -g 0 100 junk . If I upload files (via the built-in Get Data / Upload Files From Your Computer) that are than approx 2^20 bytes, the upload just hangs in the History and never completes. These files upload fine to a local Galaxy server, and to Galaxy Main. On the Galaxy local server, they run fine with my custom proteomics code. Ditto on the Cloud galaxy server when the (smaller than 1 Mbtye) files successfully upoad. I can't see anthing useful in the logs, but I'm not sure which logs I should be looking at. I'm just looking at tyhe logs presented on the Cloudman admin page. [image: logs.png] My environment is Mac, OS X 10.6.8. My data files are terminated with /n, but the problem manifests the same with Windows style /r/n files. I'll leave it at that in the hope this is a ridiculously simple probelm and needs no more description ! If not, please let me know the logs I should look at and whatever else I should check. Many thanks, Greg E -- Greg Edwards, Port Jackson Bioinformatics gedwar...@gmail.com logs.png___ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using reply all in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/listinfo/galaxy-dev To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-user] Problem uploading files gt 1 Mbyte to Galaxy Cloud
Dannon, Thanks. Glad it's not my error ! Could ypu let me know where nginx.conf is, and how to restart the nginx process ? This is a prototype proof-of-concept implementation, and I think a once-per-instance custom fix which preserves the simple ability to use Get Data / Upload Files From Your Computer will be best for now. Thanks (and also for being awake at 4am in Pennsylvania, if I have my time zones right) Greg E On Mon, Jan 23, 2012 at 8:18 PM, Dannon Baker dannonba...@me.com wrote: Hi Greg, This is a problem with the default client_max_body_size option in nginx being set far too small in the nginx.conf on the cloud AMI. It'll be fixed with our next AMI update, but to fix it you have to edit the nginx.conf on your cloud node to change the client_max_body_size to something more appropriate for Galaxy uploads and then restart the nginx process. You'd have to do it for every instance, unfortunately, since that section of the filesystem is not persisted after shutdown. As a workaround the URL upload will work correctly with any size file if you're able to host the file you want to upload somewhere local, or you could use FTP upload and that should also function correctly. -Dannon On Jan 23, 2012, at 4:09 AM, Greg Edwards wrote: Hi, I'm building a Galaxy Cloud implementation, and running into a problem uploading data files 1 Mbyte, or 1,048,576 bytes. I've started Galaxy Cloudman and a single server, all according to to the Wiki. I can upload fasta format files 1048756 bytes, and general CSV format peptide files, and indeed junk data files built with variants of jot -r -c 103 a z | rs -g 0 100 junk . If I upload files (via the built-in Get Data / Upload Files From Your Computer) that are than approx 2^20 bytes, the upload just hangs in the History and never completes. These files upload fine to a local Galaxy server, and to Galaxy Main. On the Galaxy local server, they run fine with my custom proteomics code. Ditto on the Cloud galaxy server when the (smaller than 1 Mbtye) files successfully upoad. I can't see anthing useful in the logs, but I'm not sure which logs I should be looking at. I'm just looking at tyhe logs presented on the Cloudman admin page. logs.png My environment is Mac, OS X 10.6.8. My data files are terminated with /n, but the problem manifests the same with Windows style /r/n files. I'll leave it at that in the hope this is a ridiculously simple probelm and needs no more description ! If not, please let me know the logs I should look at and whatever else I should check. Many thanks, Greg E -- Greg Edwards, Port Jackson Bioinformatics gedwar...@gmail.com ___ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using reply all in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/listinfo/galaxy-dev To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Greg Edwards, Port Jackson Bioinformatics gedwar...@gmail.com ___ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using reply all in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/listinfo/galaxy-dev To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-user] Problem uploading files gt 1 Mbyte to Galaxy Cloud
It's the other nginx.conf at /opt/galaxy/pkg/nginx/conf/nginx.conf, I don't think that one you found is used. And it isn't explicitly set at all currently, so the default 1m gets applied. And, to reload with the modified config, I think this should do the trick (as the ubuntu user)- sudo kill -HUP `cat /opt/galaxy/pkg/nginx/logs/nginx.pid` -Dannon On Jan 23, 2012, at 4:43 AM, Greg Edwards wrote: Hi Dannon, I found nginx.conf at /opt/galaxy/pkg/nginx_upload_module-2.0.12/nginx.conf It contains .. client_max_body_size 100m; That looks like 100 Megabyte ?? but seems to result in 1 Megabyte. Can I set it to 1m, or 1g ? Can't see how to restart nginx yet. Thanks, Greg E On Mon, Jan 23, 2012 at 8:18 PM, Dannon Baker dannonba...@me.com wrote: Hi Greg, This is a problem with the default client_max_body_size option in nginx being set far too small in the nginx.conf on the cloud AMI. It'll be fixed with our next AMI update, but to fix it you have to edit the nginx.conf on your cloud node to change the client_max_body_size to something more appropriate for Galaxy uploads and then restart the nginx process. You'd have to do it for every instance, unfortunately, since that section of the filesystem is not persisted after shutdown. As a workaround the URL upload will work correctly with any size file if you're able to host the file you want to upload somewhere local, or you could use FTP upload and that should also function correctly. -Dannon On Jan 23, 2012, at 4:09 AM, Greg Edwards wrote: Hi, I'm building a Galaxy Cloud implementation, and running into a problem uploading data files 1 Mbyte, or 1,048,576 bytes. I've started Galaxy Cloudman and a single server, all according to to the Wiki. I can upload fasta format files 1048756 bytes, and general CSV format peptide files, and indeed junk data files built with variants of jot -r -c 103 a z | rs -g 0 100 junk . If I upload files (via the built-in Get Data / Upload Files From Your Computer) that are than approx 2^20 bytes, the upload just hangs in the History and never completes. These files upload fine to a local Galaxy server, and to Galaxy Main. On the Galaxy local server, they run fine with my custom proteomics code. Ditto on the Cloud galaxy server when the (smaller than 1 Mbtye) files successfully upoad. I can't see anthing useful in the logs, but I'm not sure which logs I should be looking at. I'm just looking at tyhe logs presented on the Cloudman admin page. logs.png My environment is Mac, OS X 10.6.8. My data files are terminated with /n, but the problem manifests the same with Windows style /r/n files. I'll leave it at that in the hope this is a ridiculously simple probelm and needs no more description ! If not, please let me know the logs I should look at and whatever else I should check. Many thanks, Greg E -- Greg Edwards, Port Jackson Bioinformatics gedwar...@gmail.com ___ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using reply all in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/listinfo/galaxy-dev To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Greg Edwards, Port Jackson Bioinformatics gedwar...@gmail.com ___ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using reply all in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/listinfo/galaxy-dev To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-user] Problem uploading files gt 1 Mbyte to Galaxy Cloud
Dannon,
Cool, that seems to have worked. I set
server {
listen 80;
client_max_body_size 500m;
in /opt/galaxy/pkg/nginx/conf/nginx.conf
and issued sudo kill -HUP `cat /opt/galaxy/pkg/nginx/logs/nginx.pid`
and it's uploading my test files of 2mb to 25mb ok.
I think it will be easier to do that in my Cloud setup than instruct my
(not very computer keen) wet-lab Proteomist clients how to use a subsidiary
FTP package, easy though that is. But I'll try the FTP approach too for
compariosn.
Thanks,
Greg E
On Mon, Jan 23, 2012 at 8:58 PM, Dannon Baker dannonba...@me.com wrote:
It's the other nginx.conf at /opt/galaxy/pkg/nginx/conf/nginx.conf, I
don't think that one you found is used. And it isn't explicitly set at all
currently, so the default 1m gets applied.
And, to reload with the modified config, I think this should do the trick
(as the ubuntu user)-
sudo kill -HUP `cat /opt/galaxy/pkg/nginx/logs/nginx.pid`
-Dannon
On Jan 23, 2012, at 4:43 AM, Greg Edwards wrote:
Hi Dannon,
I found nginx.conf at
/opt/galaxy/pkg/nginx_upload_module-2.0.12/nginx.conf
It contains ..
client_max_body_size 100m;
That looks like 100 Megabyte ?? but seems to result in 1 Megabyte. Can I
set it to 1m, or 1g ?
Can't see how to restart nginx yet.
Thanks,
Greg E
On Mon, Jan 23, 2012 at 8:18 PM, Dannon Baker dannonba...@me.com
wrote:
Hi Greg,
This is a problem with the default client_max_body_size option in nginx
being set far too small in the nginx.conf on the cloud AMI. It'll be fixed
with our next AMI update, but to fix it you have to edit the nginx.conf on
your cloud node to change the client_max_body_size to something more
appropriate for Galaxy uploads and then restart the nginx process.
You'd have to do it for every instance, unfortunately, since that
section of the filesystem is not persisted after shutdown. As a workaround
the URL upload will work correctly with any size file if you're able to
host the file you want to upload somewhere local, or you could use FTP
upload and that should also function correctly.
-Dannon
On Jan 23, 2012, at 4:09 AM, Greg Edwards wrote:
Hi,
I'm building a Galaxy Cloud implementation, and running into a problem
uploading data files 1 Mbyte, or 1,048,576 bytes.
I've started Galaxy Cloudman and a single server, all according to to
the Wiki. I can upload fasta format files 1048756 bytes, and general CSV
format peptide files, and indeed junk data files built with variants of
jot -r -c 103 a z | rs -g 0 100 junk .
If I upload files (via the built-in Get Data / Upload Files From Your
Computer) that are than approx 2^20 bytes, the upload just hangs in the
History and never completes.
These files upload fine to a local Galaxy server, and to Galaxy Main.
On the Galaxy local server, they run fine with my custom proteomics code.
Ditto on the Cloud galaxy server when the (smaller than 1 Mbtye) files
successfully upoad.
I can't see anthing useful in the logs, but I'm not sure which logs I
should be looking at. I'm just looking at tyhe logs presented on the
Cloudman admin page.
logs.png
My environment is Mac, OS X 10.6.8. My data files are terminated with
/n, but the problem manifests the same with Windows style /r/n files.
I'll leave it at that in the hope this is a ridiculously simple
probelm and needs no more description ! If not, please let me know the logs
I should look at and whatever else I should check.
Many thanks,
Greg E
--
Greg Edwards,
Port Jackson Bioinformatics
gedwar...@gmail.com
___
The Galaxy User list should be used for the discussion of
Galaxy analysis and other features on the public server
at usegalaxy.org. Please keep all replies on the list by
using reply all in your mail client. For discussion of
local Galaxy instances and the Galaxy source code, please
use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists,
please use the interface at:
http://lists.bx.psu.edu/
--
Greg Edwards,
Port Jackson Bioinformatics
gedwar...@gmail.com
--
Greg Edwards,
Port Jackson Bioinformatics
gedwar...@gmail.com
___
The Galaxy User list should be used for the discussion of
Galaxy analysis and other features on the public server
at usegalaxy.org. Please keep all replies on the list by
using reply all in your mail client. For discussion of
local Galaxy instances and the Galaxy source code, please
use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists,
please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-user] Problem uploading files gt 1 Mbyte to Galaxy Cloud
Great, glad it worked for you!
And, once we release a new AMI, it'll be fixed retroactively. That is, you'll
be able to use your existing cloud setup with the new AMI and it should just
work.
-Dannon
On Jan 23, 2012, at 5:35 AM, Greg Edwards wrote:
Dannon,
Cool, that seems to have worked. I set
server {
listen 80;
client_max_body_size 500m;
in /opt/galaxy/pkg/nginx/conf/nginx.conf
and issued sudo kill -HUP `cat /opt/galaxy/pkg/nginx/logs/nginx.pid`
and it's uploading my test files of 2mb to 25mb ok.
I think it will be easier to do that in my Cloud setup than instruct my (not
very computer keen) wet-lab Proteomist clients how to use a subsidiary FTP
package, easy though that is. But I'll try the FTP approach too for
compariosn.
Thanks,
Greg E
On Mon, Jan 23, 2012 at 8:58 PM, Dannon Baker dannonba...@me.com wrote:
It's the other nginx.conf at /opt/galaxy/pkg/nginx/conf/nginx.conf, I don't
think that one you found is used. And it isn't explicitly set at all
currently, so the default 1m gets applied.
And, to reload with the modified config, I think this should do the trick (as
the ubuntu user)-
sudo kill -HUP `cat /opt/galaxy/pkg/nginx/logs/nginx.pid`
-Dannon
On Jan 23, 2012, at 4:43 AM, Greg Edwards wrote:
Hi Dannon,
I found nginx.conf at /opt/galaxy/pkg/nginx_upload_module-2.0.12/nginx.conf
It contains ..
client_max_body_size 100m;
That looks like 100 Megabyte ?? but seems to result in 1 Megabyte. Can I
set it to 1m, or 1g ?
Can't see how to restart nginx yet.
Thanks,
Greg E
On Mon, Jan 23, 2012 at 8:18 PM, Dannon Baker dannonba...@me.com wrote:
Hi Greg,
This is a problem with the default client_max_body_size option in nginx
being set far too small in the nginx.conf on the cloud AMI. It'll be fixed
with our next AMI update, but to fix it you have to edit the nginx.conf on
your cloud node to change the client_max_body_size to something more
appropriate for Galaxy uploads and then restart the nginx process.
You'd have to do it for every instance, unfortunately, since that section
of the filesystem is not persisted after shutdown. As a workaround the URL
upload will work correctly with any size file if you're able to host the
file you want to upload somewhere local, or you could use FTP upload and
that should also function correctly.
-Dannon
On Jan 23, 2012, at 4:09 AM, Greg Edwards wrote:
Hi,
I'm building a Galaxy Cloud implementation, and running into a problem
uploading data files 1 Mbyte, or 1,048,576 bytes.
I've started Galaxy Cloudman and a single server, all according to to the
Wiki. I can upload fasta format files 1048756 bytes, and general CSV
format peptide files, and indeed junk data files built with variants of
jot -r -c 103 a z | rs -g 0 100 junk .
If I upload files (via the built-in Get Data / Upload Files From Your
Computer) that are than approx 2^20 bytes, the upload just hangs in the
History and never completes.
These files upload fine to a local Galaxy server, and to Galaxy Main. On
the Galaxy local server, they run fine with my custom proteomics code.
Ditto on the Cloud galaxy server when the (smaller than 1 Mbtye) files
successfully upoad.
I can't see anthing useful in the logs, but I'm not sure which logs I
should be looking at. I'm just looking at tyhe logs presented on the
Cloudman admin page.
logs.png
My environment is Mac, OS X 10.6.8. My data files are terminated with /n,
but the problem manifests the same with Windows style /r/n files.
I'll leave it at that in the hope this is a ridiculously simple probelm
and needs no more description ! If not, please let me know the logs I
should look at and whatever else I should check.
Many thanks,
Greg E
--
Greg Edwards,
Port Jackson Bioinformatics
gedwar...@gmail.com
___
The Galaxy User list should be used for the discussion of
Galaxy analysis and other features on the public server
at usegalaxy.org. Please keep all replies on the list by
using reply all in your mail client. For discussion of
local Galaxy instances and the Galaxy source code, please
use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists,
please use the interface at:
http://lists.bx.psu.edu/
--
Greg Edwards,
Port Jackson Bioinformatics
gedwar...@gmail.com
--
Greg Edwards,
Port Jackson Bioinformatics
gedwar...@gmail.com
___
The Galaxy User list should be used for the discussion of
Galaxy analysis and other features on the public server
at usegalaxy.org. Please keep all
