Re: [galaxy-user] Problem uploading files gt 1 Mbyte to Galaxy Cloud

2012-01-23 Thread Greg Edwards
Dannon,

Thanks. Glad it's not my error !

Could ypu let me know where nginx.conf is, and how to restart the nginx
process ? This is a prototype proof-of-concept implementation, and I think
a once-per-instance custom fix which preserves the simple ability to use
Get Data / Upload Files From Your Computer will be best for now.

Thanks (and also for being awake at 4am in Pennsylvania, if I have my time
zones right)

Greg E


On Mon, Jan 23, 2012 at 8:18 PM, Dannon Baker dannonba...@me.com wrote:

 Hi Greg,

 This is a problem with the default client_max_body_size option in nginx
 being set far too small in the nginx.conf on the cloud AMI.  It'll be fixed
 with our next AMI update, but to fix it you have to edit the nginx.conf on
 your cloud node to change the client_max_body_size to something more
 appropriate for Galaxy uploads and then restart the nginx process.

 You'd have to do it for every instance, unfortunately, since that section
 of the filesystem is not persisted after shutdown.  As a workaround the URL
 upload will work correctly with any size file if you're able to host the
 file you want to upload somewhere local, or you could use FTP upload and
 that should also function correctly.

 -Dannon


 On Jan 23, 2012, at 4:09 AM, Greg Edwards wrote:

  Hi,
 
  I'm building a Galaxy Cloud implementation, and running into a problem
 uploading data files  1 Mbyte, or 1,048,576 bytes.
 
  I've started Galaxy Cloudman and a single server, all according to to
 the Wiki. I can upload fasta format files  1048756 bytes, and general CSV
 format peptide files, and indeed junk data files built with variants of 
 jot -r -c 103 a z | rs -g 0 100 junk .
 
  If I upload files (via the built-in Get Data / Upload Files From Your
 Computer) that are  than approx 2^20 bytes, the upload just hangs in the
 History and never completes.
 
  These files upload fine to a local Galaxy server, and to Galaxy Main. On
 the Galaxy local server, they run fine with my custom proteomics code.
 Ditto on the Cloud galaxy server when the (smaller than 1 Mbtye) files
 successfully upoad.
 
  I can't see anthing useful in the logs, but I'm not sure which logs I
 should be looking at. I'm just looking at tyhe logs presented on the
 Cloudman admin page.
 
  logs.png
 
  My environment is Mac, OS X 10.6.8. My data files are terminated with
 /n, but the problem manifests the same with Windows style /r/n files.
 
  I'll leave it at that in the hope this is a ridiculously simple probelm
 and needs no more description ! If not, please let me know the logs I
 should look at and whatever else I should check.
 
  Many thanks,
  Greg E
 
  --
  Greg Edwards,
  Port Jackson Bioinformatics
  gedwar...@gmail.com
 
  ___
  The Galaxy User list should be used for the discussion of
  Galaxy analysis and other features on the public server
  at usegalaxy.org.  Please keep all replies on the list by
  using reply all in your mail client.  For discussion of
  local Galaxy instances and the Galaxy source code, please
  use the Galaxy Development list:
 
   http://lists.bx.psu.edu/listinfo/galaxy-dev
 
  To manage your subscriptions to this and other Galaxy lists,
  please use the interface at:
 
   http://lists.bx.psu.edu/




-- 
Greg Edwards,
Port Jackson Bioinformatics
gedwar...@gmail.com
___
The Galaxy User list should be used for the discussion of
Galaxy analysis and other features on the public server
at usegalaxy.org.  Please keep all replies on the list by
using reply all in your mail client.  For discussion of
local Galaxy instances and the Galaxy source code, please
use the Galaxy Development list:

  http://lists.bx.psu.edu/listinfo/galaxy-dev

To manage your subscriptions to this and other Galaxy lists,
please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-user] Problem uploading files gt 1 Mbyte to Galaxy Cloud

2012-01-23 Thread Dannon Baker
It's the other nginx.conf at /opt/galaxy/pkg/nginx/conf/nginx.conf, I don't 
think that one you found is used.  And it isn't explicitly set at all 
currently, so the default 1m gets applied.

And, to reload with the modified config, I think this should do the trick (as 
the ubuntu user)-

sudo kill -HUP `cat /opt/galaxy/pkg/nginx/logs/nginx.pid`

-Dannon


On Jan 23, 2012, at 4:43 AM, Greg Edwards wrote:

 Hi Dannon,
 
 I found nginx.conf at  /opt/galaxy/pkg/nginx_upload_module-2.0.12/nginx.conf
 
 It contains ..
 
  client_max_body_size 100m;
 
 That looks like 100 Megabyte ?? but seems to result in 1 Megabyte. Can I set 
 it to 1m, or 1g ?
 
 Can't see how to restart nginx yet.
 
 Thanks,
 Greg E
 
 
 
 
 
 On Mon, Jan 23, 2012 at 8:18 PM, Dannon Baker dannonba...@me.com wrote:
 Hi Greg,
 
 This is a problem with the default client_max_body_size option in nginx being 
 set far too small in the nginx.conf on the cloud AMI.  It'll be fixed with 
 our next AMI update, but to fix it you have to edit the nginx.conf on your 
 cloud node to change the client_max_body_size to something more appropriate 
 for Galaxy uploads and then restart the nginx process.
 
 You'd have to do it for every instance, unfortunately, since that section of 
 the filesystem is not persisted after shutdown.  As a workaround the URL 
 upload will work correctly with any size file if you're able to host the file 
 you want to upload somewhere local, or you could use FTP upload and that 
 should also function correctly.
 
 -Dannon
 
 
 On Jan 23, 2012, at 4:09 AM, Greg Edwards wrote:
 
  Hi,
 
  I'm building a Galaxy Cloud implementation, and running into a problem 
  uploading data files  1 Mbyte, or 1,048,576 bytes.
 
  I've started Galaxy Cloudman and a single server, all according to to the 
  Wiki. I can upload fasta format files  1048756 bytes, and general CSV 
  format peptide files, and indeed junk data files built with variants of  
  jot -r -c 103 a z | rs -g 0 100 junk .
 
  If I upload files (via the built-in Get Data / Upload Files From Your 
  Computer) that are  than approx 2^20 bytes, the upload just hangs in the 
  History and never completes.
 
  These files upload fine to a local Galaxy server, and to Galaxy Main. On 
  the Galaxy local server, they run fine with my custom proteomics code. 
  Ditto on the Cloud galaxy server when the (smaller than 1 Mbtye) files 
  successfully upoad.
 
  I can't see anthing useful in the logs, but I'm not sure which logs I 
  should be looking at. I'm just looking at tyhe logs presented on the 
  Cloudman admin page.
 
  logs.png
 
  My environment is Mac, OS X 10.6.8. My data files are terminated with /n, 
  but the problem manifests the same with Windows style /r/n files.
 
  I'll leave it at that in the hope this is a ridiculously simple probelm and 
  needs no more description ! If not, please let me know the logs I should 
  look at and whatever else I should check.
 
  Many thanks,
  Greg E
 
  --
  Greg Edwards,
  Port Jackson Bioinformatics
  gedwar...@gmail.com
 
  ___
  The Galaxy User list should be used for the discussion of
  Galaxy analysis and other features on the public server
  at usegalaxy.org.  Please keep all replies on the list by
  using reply all in your mail client.  For discussion of
  local Galaxy instances and the Galaxy source code, please
  use the Galaxy Development list:
 
   http://lists.bx.psu.edu/listinfo/galaxy-dev
 
  To manage your subscriptions to this and other Galaxy lists,
  please use the interface at:
 
   http://lists.bx.psu.edu/
 
 
 
 
 -- 
 Greg Edwards,
 Port Jackson Bioinformatics
 gedwar...@gmail.com
 

___
The Galaxy User list should be used for the discussion of
Galaxy analysis and other features on the public server
at usegalaxy.org.  Please keep all replies on the list by
using reply all in your mail client.  For discussion of
local Galaxy instances and the Galaxy source code, please
use the Galaxy Development list:

  http://lists.bx.psu.edu/listinfo/galaxy-dev

To manage your subscriptions to this and other Galaxy lists,
please use the interface at:

  http://lists.bx.psu.edu/


Re: [galaxy-user] Problem uploading files gt 1 Mbyte to Galaxy Cloud

2012-01-23 Thread Greg Edwards
Dannon,

Cool, that seems to have worked. I set

server {
listen   80;
client_max_body_size 500m;

in  /opt/galaxy/pkg/nginx/conf/nginx.conf

and issued   sudo kill -HUP `cat /opt/galaxy/pkg/nginx/logs/nginx.pid`

and it's uploading my test files of 2mb to 25mb ok.

I think it will be easier to do that in my Cloud setup than instruct my
(not very computer keen) wet-lab Proteomist clients how to use a subsidiary
FTP package, easy though that is. But I'll try the FTP approach too for
compariosn.

Thanks,
Greg E


On Mon, Jan 23, 2012 at 8:58 PM, Dannon Baker dannonba...@me.com wrote:

 It's the other nginx.conf at /opt/galaxy/pkg/nginx/conf/nginx.conf, I
 don't think that one you found is used.  And it isn't explicitly set at all
 currently, so the default 1m gets applied.

 And, to reload with the modified config, I think this should do the trick
 (as the ubuntu user)-

 sudo kill -HUP `cat /opt/galaxy/pkg/nginx/logs/nginx.pid`

 -Dannon


 On Jan 23, 2012, at 4:43 AM, Greg Edwards wrote:

  Hi Dannon,
 
  I found nginx.conf at
  /opt/galaxy/pkg/nginx_upload_module-2.0.12/nginx.conf
 
  It contains ..
 
   client_max_body_size 100m;
 
  That looks like 100 Megabyte ?? but seems to result in 1 Megabyte. Can I
 set it to 1m, or 1g ?
 
  Can't see how to restart nginx yet.
 
  Thanks,
  Greg E
 
 
 
 
 
  On Mon, Jan 23, 2012 at 8:18 PM, Dannon Baker dannonba...@me.com
 wrote:
  Hi Greg,
 
  This is a problem with the default client_max_body_size option in nginx
 being set far too small in the nginx.conf on the cloud AMI.  It'll be fixed
 with our next AMI update, but to fix it you have to edit the nginx.conf on
 your cloud node to change the client_max_body_size to something more
 appropriate for Galaxy uploads and then restart the nginx process.
 
  You'd have to do it for every instance, unfortunately, since that
 section of the filesystem is not persisted after shutdown.  As a workaround
 the URL upload will work correctly with any size file if you're able to
 host the file you want to upload somewhere local, or you could use FTP
 upload and that should also function correctly.
 
  -Dannon
 
 
  On Jan 23, 2012, at 4:09 AM, Greg Edwards wrote:
 
   Hi,
  
   I'm building a Galaxy Cloud implementation, and running into a problem
 uploading data files  1 Mbyte, or 1,048,576 bytes.
  
   I've started Galaxy Cloudman and a single server, all according to to
 the Wiki. I can upload fasta format files  1048756 bytes, and general CSV
 format peptide files, and indeed junk data files built with variants of 
 jot -r -c 103 a z | rs -g 0 100 junk .
  
   If I upload files (via the built-in Get Data / Upload Files From Your
 Computer) that are  than approx 2^20 bytes, the upload just hangs in the
 History and never completes.
  
   These files upload fine to a local Galaxy server, and to Galaxy Main.
 On the Galaxy local server, they run fine with my custom proteomics code.
 Ditto on the Cloud galaxy server when the (smaller than 1 Mbtye) files
 successfully upoad.
  
   I can't see anthing useful in the logs, but I'm not sure which logs I
 should be looking at. I'm just looking at tyhe logs presented on the
 Cloudman admin page.
  
   logs.png
  
   My environment is Mac, OS X 10.6.8. My data files are terminated with
 /n, but the problem manifests the same with Windows style /r/n files.
  
   I'll leave it at that in the hope this is a ridiculously simple
 probelm and needs no more description ! If not, please let me know the logs
 I should look at and whatever else I should check.
  
   Many thanks,
   Greg E
  
   --
   Greg Edwards,
   Port Jackson Bioinformatics
   gedwar...@gmail.com
  
   ___
   The Galaxy User list should be used for the discussion of
   Galaxy analysis and other features on the public server
   at usegalaxy.org.  Please keep all replies on the list by
   using reply all in your mail client.  For discussion of
   local Galaxy instances and the Galaxy source code, please
   use the Galaxy Development list:
  
http://lists.bx.psu.edu/listinfo/galaxy-dev
  
   To manage your subscriptions to this and other Galaxy lists,
   please use the interface at:
  
http://lists.bx.psu.edu/
 
 
 
 
  --
  Greg Edwards,
  Port Jackson Bioinformatics
  gedwar...@gmail.com
 




-- 
Greg Edwards,
Port Jackson Bioinformatics
gedwar...@gmail.com
___
The Galaxy User list should be used for the discussion of
Galaxy analysis and other features on the public server
at usegalaxy.org.  Please keep all replies on the list by
using reply all in your mail client.  For discussion of
local Galaxy instances and the Galaxy source code, please
use the Galaxy Development list:

  http://lists.bx.psu.edu/listinfo/galaxy-dev

To manage your subscriptions to this and other Galaxy lists,
please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-user] Problem uploading files gt 1 Mbyte to Galaxy Cloud

2012-01-23 Thread Dannon Baker
Great, glad it worked for you!

And, once we release a new AMI, it'll be fixed retroactively.  That is, you'll 
be able to use your existing cloud setup with the new AMI and it should just 
work.

-Dannon


On Jan 23, 2012, at 5:35 AM, Greg Edwards wrote:

 Dannon,
 
 Cool, that seems to have worked. I set 
 
 server {
 listen   80;
 client_max_body_size 500m;
 
 in  /opt/galaxy/pkg/nginx/conf/nginx.conf
 
 and issued   sudo kill -HUP `cat /opt/galaxy/pkg/nginx/logs/nginx.pid`
 
 and it's uploading my test files of 2mb to 25mb ok.
 
 I think it will be easier to do that in my Cloud setup than instruct my (not 
 very computer keen) wet-lab Proteomist clients how to use a subsidiary FTP 
 package, easy though that is. But I'll try the FTP approach too for 
 compariosn.
 
 Thanks,
 Greg E
 
 
 On Mon, Jan 23, 2012 at 8:58 PM, Dannon Baker dannonba...@me.com wrote:
 It's the other nginx.conf at /opt/galaxy/pkg/nginx/conf/nginx.conf, I don't 
 think that one you found is used.  And it isn't explicitly set at all 
 currently, so the default 1m gets applied.
 
 And, to reload with the modified config, I think this should do the trick (as 
 the ubuntu user)-
 
 sudo kill -HUP `cat /opt/galaxy/pkg/nginx/logs/nginx.pid`
 
 -Dannon
 
 
 On Jan 23, 2012, at 4:43 AM, Greg Edwards wrote:
 
  Hi Dannon,
 
  I found nginx.conf at  /opt/galaxy/pkg/nginx_upload_module-2.0.12/nginx.conf
 
  It contains ..
 
   client_max_body_size 100m;
 
  That looks like 100 Megabyte ?? but seems to result in 1 Megabyte. Can I 
  set it to 1m, or 1g ?
 
  Can't see how to restart nginx yet.
 
  Thanks,
  Greg E
 
 
 
 
 
  On Mon, Jan 23, 2012 at 8:18 PM, Dannon Baker dannonba...@me.com wrote:
  Hi Greg,
 
  This is a problem with the default client_max_body_size option in nginx 
  being set far too small in the nginx.conf on the cloud AMI.  It'll be fixed 
  with our next AMI update, but to fix it you have to edit the nginx.conf on 
  your cloud node to change the client_max_body_size to something more 
  appropriate for Galaxy uploads and then restart the nginx process.
 
  You'd have to do it for every instance, unfortunately, since that section 
  of the filesystem is not persisted after shutdown.  As a workaround the URL 
  upload will work correctly with any size file if you're able to host the 
  file you want to upload somewhere local, or you could use FTP upload and 
  that should also function correctly.
 
  -Dannon
 
 
  On Jan 23, 2012, at 4:09 AM, Greg Edwards wrote:
 
   Hi,
  
   I'm building a Galaxy Cloud implementation, and running into a problem 
   uploading data files  1 Mbyte, or 1,048,576 bytes.
  
   I've started Galaxy Cloudman and a single server, all according to to the 
   Wiki. I can upload fasta format files  1048756 bytes, and general CSV 
   format peptide files, and indeed junk data files built with variants of  
   jot -r -c 103 a z | rs -g 0 100 junk .
  
   If I upload files (via the built-in Get Data / Upload Files From Your 
   Computer) that are  than approx 2^20 bytes, the upload just hangs in the 
   History and never completes.
  
   These files upload fine to a local Galaxy server, and to Galaxy Main. On 
   the Galaxy local server, they run fine with my custom proteomics code. 
   Ditto on the Cloud galaxy server when the (smaller than 1 Mbtye) files 
   successfully upoad.
  
   I can't see anthing useful in the logs, but I'm not sure which logs I 
   should be looking at. I'm just looking at tyhe logs presented on the 
   Cloudman admin page.
  
   logs.png
  
   My environment is Mac, OS X 10.6.8. My data files are terminated with /n, 
   but the problem manifests the same with Windows style /r/n files.
  
   I'll leave it at that in the hope this is a ridiculously simple probelm 
   and needs no more description ! If not, please let me know the logs I 
   should look at and whatever else I should check.
  
   Many thanks,
   Greg E
  
   --
   Greg Edwards,
   Port Jackson Bioinformatics
   gedwar...@gmail.com
  
   ___
   The Galaxy User list should be used for the discussion of
   Galaxy analysis and other features on the public server
   at usegalaxy.org.  Please keep all replies on the list by
   using reply all in your mail client.  For discussion of
   local Galaxy instances and the Galaxy source code, please
   use the Galaxy Development list:
  
http://lists.bx.psu.edu/listinfo/galaxy-dev
  
   To manage your subscriptions to this and other Galaxy lists,
   please use the interface at:
  
http://lists.bx.psu.edu/
 
 
 
 
  --
  Greg Edwards,
  Port Jackson Bioinformatics
  gedwar...@gmail.com
 
 
 
 
 
 -- 
 Greg Edwards,
 Port Jackson Bioinformatics
 gedwar...@gmail.com
 

___
The Galaxy User list should be used for the discussion of
Galaxy analysis and other features on the public server
at usegalaxy.org.  Please keep all