Re: [galaxy-user] Local Install - Understanding Upload File Concept
On Fri, Nov 16, 2012 at 12:28 PM, Nate Coraor wrote: > If you follow the log, you should see messages from galaxy.jobs, and > specifically, galaxy.jobs.runners.drmaa. > No, I don't see anything like that in the log after I started the upload. Any ideas on how to see what might be going wrong there? -Greg ___ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/listinfo/galaxy-dev To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-user] Local Install - Understanding Upload File Concept
On Nov 16, 2012, at 12:24 PM, greg wrote: > Also what would I look for in the log file to know if it's issuing jobs? > > I started a new upload, but I don't see anything from the galaxy user > when I run qstat. If you follow the log, you should see messages from galaxy.jobs, and specifically, galaxy.jobs.runners.drmaa. --nate > > Thanks again. > > Greg > > On Fri, Nov 16, 2012 at 12:22 PM, greg wrote: >> I guess I'm still confused about new_file_path. >> >> The upload tool uses that directory to store files temporariliy until >> the upload completes? And then moves it? (Will it clean up anything >> placed in the new_file_path directory?) >> >> Thanks, >> >> Greg >> >> On Fri, Nov 16, 2012 at 10:56 AM, Nate Coraor wrote: >>> On Nov 16, 2012, at 10:41 AM, greg wrote: >>> On Fri, Nov 16, 2012 at 10:00 AM, Nate Coraor wrote: > On Nov 16, 2012, at 9:14 AM, greg wrote: > > Are you running the upload tool on a cluster? > > --nate > Well I have Galaxy set up to send it's jobs to SGE/qsub but I haven't verified that it's doing that yet. I wanted to upload a file to test on first. Here are the changes I made to universe_wsgi.ini after copying the sample: > #upload1 = local:/// >>> >>> I'm guessing that it's working, you can verify by having a look in your >>> Galaxy server output/log file. new_file_path will need to be set to >>> something that is accessible on the cluster. >>> >>> --nate >>> -Greg >>> ___ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/listinfo/galaxy-dev To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-user] Local Install - Understanding Upload File Concept
On Nov 16, 2012, at 12:22 PM, greg wrote: > I guess I'm still confused about new_file_path. > > The upload tool uses that directory to store files temporariliy until > the upload completes? And then moves it? (Will it clean up anything > placed in the new_file_path directory?) Yes, the parameters of the upload tool are written to a temp file in this directory. It is cleaned up upon completion. A lot of other things use new_file_path, and not all of them clean themselves up, so it's a good idea to use an automated process to clean the directory. --nate > > Thanks, > > Greg > > On Fri, Nov 16, 2012 at 10:56 AM, Nate Coraor wrote: >> On Nov 16, 2012, at 10:41 AM, greg wrote: >> >>> On Fri, Nov 16, 2012 at 10:00 AM, Nate Coraor wrote: On Nov 16, 2012, at 9:14 AM, greg wrote: Are you running the upload tool on a cluster? --nate >>> >>> Well I have Galaxy set up to send it's jobs to SGE/qsub but I haven't >>> verified that it's doing that yet. I wanted to upload a file to test >>> on first. >>> >>> Here are the changes I made to universe_wsgi.ini after copying the sample: >>> >>> #upload1 = local:/// >>> >> >> I'm guessing that it's working, you can verify by having a look in your >> Galaxy server output/log file. new_file_path will need to be set to >> something that is accessible on the cluster. >> >> --nate >> >>> >>> -Greg >> ___ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/listinfo/galaxy-dev To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-user] Local Install - Understanding Upload File Concept
Also what would I look for in the log file to know if it's issuing jobs? I started a new upload, but I don't see anything from the galaxy user when I run qstat. Thanks again. Greg On Fri, Nov 16, 2012 at 12:22 PM, greg wrote: > I guess I'm still confused about new_file_path. > > The upload tool uses that directory to store files temporariliy until > the upload completes? And then moves it? (Will it clean up anything > placed in the new_file_path directory?) > > Thanks, > > Greg > > On Fri, Nov 16, 2012 at 10:56 AM, Nate Coraor wrote: >> On Nov 16, 2012, at 10:41 AM, greg wrote: >> >>> On Fri, Nov 16, 2012 at 10:00 AM, Nate Coraor wrote: On Nov 16, 2012, at 9:14 AM, greg wrote: Are you running the upload tool on a cluster? --nate >>> >>> Well I have Galaxy set up to send it's jobs to SGE/qsub but I haven't >>> verified that it's doing that yet. I wanted to upload a file to test >>> on first. >>> >>> Here are the changes I made to universe_wsgi.ini after copying the sample: >>> >>> #upload1 = local:/// >>> >> >> I'm guessing that it's working, you can verify by having a look in your >> Galaxy server output/log file. new_file_path will need to be set to >> something that is accessible on the cluster. >> >> --nate >> >>> >>> -Greg >> ___ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/listinfo/galaxy-dev To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-user] Local Install - Understanding Upload File Concept
I guess I'm still confused about new_file_path. The upload tool uses that directory to store files temporariliy until the upload completes? And then moves it? (Will it clean up anything placed in the new_file_path directory?) Thanks, Greg On Fri, Nov 16, 2012 at 10:56 AM, Nate Coraor wrote: > On Nov 16, 2012, at 10:41 AM, greg wrote: > >> On Fri, Nov 16, 2012 at 10:00 AM, Nate Coraor wrote: >>> On Nov 16, 2012, at 9:14 AM, greg wrote: >>> >>> Are you running the upload tool on a cluster? >>> >>> --nate >>> >> >> Well I have Galaxy set up to send it's jobs to SGE/qsub but I haven't >> verified that it's doing that yet. I wanted to upload a file to test >> on first. >> >> Here are the changes I made to universe_wsgi.ini after copying the sample: >> >> >>> #upload1 = local:/// >> > > I'm guessing that it's working, you can verify by having a look in your > Galaxy server output/log file. new_file_path will need to be set to > something that is accessible on the cluster. > > --nate > >> >> -Greg > ___ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/listinfo/galaxy-dev To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-user] Local Install - Understanding Upload File Concept
On Nov 16, 2012, at 10:41 AM, greg wrote: > On Fri, Nov 16, 2012 at 10:00 AM, Nate Coraor wrote: >> On Nov 16, 2012, at 9:14 AM, greg wrote: >> >> Are you running the upload tool on a cluster? >> >> --nate >> > > Well I have Galaxy set up to send it's jobs to SGE/qsub but I haven't > verified that it's doing that yet. I wanted to upload a file to test > on first. > > Here are the changes I made to universe_wsgi.ini after copying the sample: > > >> #upload1 = local:/// > I'm guessing that it's working, you can verify by having a look in your Galaxy server output/log file. new_file_path will need to be set to something that is accessible on the cluster. --nate > > -Greg ___ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/listinfo/galaxy-dev To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-user] Local Install - Understanding Upload File Concept
On Fri, Nov 16, 2012 at 10:00 AM, Nate Coraor wrote: > On Nov 16, 2012, at 9:14 AM, greg wrote: > > Are you running the upload tool on a cluster? > > --nate > Well I have Galaxy set up to send it's jobs to SGE/qsub but I haven't verified that it's doing that yet. I wanted to upload a file to test on first. Here are the changes I made to universe_wsgi.ini after copying the sample: > port = 8080 > host = 0.0.0.0 > database_connection = mysql://user:password@devel10-db/galaxy > database_engine_option_pool_size = 10 > database_engine_option_max_overflow = 15 > database_engine_option_pool_recycle = 7200 > database_engine_option_strategy = threadlocal > file_path = /home/galaxy > new_file_path = /scratch/galaxy > job_working_directory = /home/galaxy/job_working_directory > cluster_files_directory = /home/galaxy/pbs > log_level = INFO > log_events = True > log_actions = True > use_remote_user = True > remote_user_maildomain = mydomain.org > require_login = True > allow_user_creation = True > allow_user_deletion = False > allow_user_impersonation = True > allow_user_dataset_purge = True > new_user_dataset_access_role_default_private = True > job_manager = main > job_handlers = main > default_job_handlers = main > track_jobs_in_database = True > enable_job_recovery = True > set_metadata_externally = True > cleanup_job = always > start_job_runners = drmaa > environment_setup_file = /usr/local/galaxy/job_environment_setup_file > default_cluster_job_runner = drmaa://-V/ > cluster_job_queue_workers = 3 > #biomart = local:/// > #encode_db1 = local:/// > #hbvar = local:/// > #microbial_import1 = local:/// > #ucsc_table_direct1 = local:/// > #ucsc_table_direct_archaea1 = local:/// > #ucsc_table_direct_test1 = local:/// > #upload1 = local:/// -Greg ___ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/listinfo/galaxy-dev To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-user] Local Install - Understanding Upload File Concept
On Nov 16, 2012, at 9:14 AM, greg wrote: > I'm confused if files are uploaded through the Galaxy server or through > Apache? > > I already set up the Apache proxy explained here: > http://wiki.galaxyproject.org/Admin/Config/Apache%20Proxy > (Does that affect uploads?) > > My goal is to let users upload as large of files as possible through HTTP. The file is uploaded "through" Apache, but Apache doesn't do anything special with it other than stream the bits through to Galaxy. With nginx it's possible to have the proxy server actually handle the upload itself. > So far I just tested and upload of a 140MB file and I got this error > after a couple of minutes: > > 1: dsim-all-chromosome-r1.3.fasta.msg.updated.fasta > 0 bytes > An error occurred running this job:Traceback (most recent call last): > File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", > line 384, in > __main__() > File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", > line 359, in __main__ > registry.load_datat > > > I see this information in the bug report: > > Traceback (most recent call last): > File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", > line 384, in >__main__() > File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", > line 359, in __main__ >registry.load_datatypes( root_dir=sys.argv[1], config=sys.argv[2] ) > File "/misc/local/galaxy/galaxy-dist/lib/galaxy/datatypes/registry.py", > line 65, in load_datatypes >tree = galaxy.util.parse_xml( config ) > File "/misc/local/galaxy/galaxy-dist/lib/galaxy/util/__init__.py", > line 143, in parse_xml >tree = ElementTree.parse(fname) > File "build/bdist.linux-x86_64/egg/elementtree/ElementTree.py", line > 859, in parse > File "build/bdist.linux-x86_64/egg/elementtree/ElementTree.py", line > 576, in parse > IOError: [Errno 2] No such file or directory: '/scratch/galaxy/tmptzuSYJ' > Traceback (most recent call last): > File "./scripts/set_metadata.py", line 118, in >__main__() > File "./scripts/set_metadata.py", line 68, in __main__ >datatypes_registry.load_datatypes( root_dir=config_root, > config=datatypes_config ) > File "/misc/local/galaxy/galaxy-dist/lib/galaxy/datatypes/registry.py", > line 65, in load_datatypes >tree = galaxy.util.parse_xml( config ) > File "/misc/local/galaxy/galaxy-dist/lib/galaxy/util/__init__.py", > line 143, in parse_xml >tree = ElementTree.parse(fname) > File "build/bdist.linux-x86_64/egg/elementtree/ElementTree.py", line > 859, in parse > File "build/bdist.linux-x86_64/egg/elementtree/ElementTree.py", line > 576, in parse > IOError: [Errno 2] No such file or directory: '/scratch/galaxy/tmptzuSYJ' > > > Also here is what /scratch/galaxy looks like. It seems like the file > it mentions does exist?? > >> ls -lh /scratch/galaxy/ > total 134M > -rw-r--r-- 1 galaxy scicomp 20K Nov 13 13:35 tmpfCaiGX > -rw-r--r-- 1 galaxy scicomp 20K Nov 14 13:36 tmpM2Wclq > -rw-r--r-- 1 galaxy scicomp 20K Nov 14 13:35 tmpsIyPur > -rw-r--r-- 1 galaxy scicomp 20K Nov 16 08:48 tmptY5RaM > -rw-r--r-- 1 galaxy scicomp 20K Nov 16 08:54 tmptzuSYJ > -rw-r--r-- 1 galaxy scicomp 20K Nov 15 15:47 tmpu77PEK > -rw--- 1 galaxy scicomp 286 Nov 16 08:58 tmpykoBIq > -rw--- 1 galaxy scicomp 134M Nov 16 08:58 upload_file_data_G3eZEg Are you running the upload tool on a cluster? --nate > > Thanks, > > Greg > ___ > The Galaxy User list should be used for the discussion of > Galaxy analysis and other features on the public server > at usegalaxy.org. Please keep all replies on the list by > using "reply all" in your mail client. For discussion of > local Galaxy instances and the Galaxy source code, please > use the Galaxy Development list: > > http://lists.bx.psu.edu/listinfo/galaxy-dev > > To manage your subscriptions to this and other Galaxy lists, > please use the interface at: > > http://lists.bx.psu.edu/ ___ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/listinfo/galaxy-dev To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-user] Local Install - Understanding Upload File Concept
I'm confused if files are uploaded through the Galaxy server or through Apache? I already set up the Apache proxy explained here: http://wiki.galaxyproject.org/Admin/Config/Apache%20Proxy (Does that affect uploads?) My goal is to let users upload as large of files as possible through HTTP. So far I just tested and upload of a 140MB file and I got this error after a couple of minutes: 1: dsim-all-chromosome-r1.3.fasta.msg.updated.fasta 0 bytes An error occurred running this job:Traceback (most recent call last): File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", line 384, in __main__() File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", line 359, in __main__ registry.load_datat I see this information in the bug report: Traceback (most recent call last): File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", line 384, in __main__() File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", line 359, in __main__ registry.load_datatypes( root_dir=sys.argv[1], config=sys.argv[2] ) File "/misc/local/galaxy/galaxy-dist/lib/galaxy/datatypes/registry.py", line 65, in load_datatypes tree = galaxy.util.parse_xml( config ) File "/misc/local/galaxy/galaxy-dist/lib/galaxy/util/__init__.py", line 143, in parse_xml tree = ElementTree.parse(fname) File "build/bdist.linux-x86_64/egg/elementtree/ElementTree.py", line 859, in parse File "build/bdist.linux-x86_64/egg/elementtree/ElementTree.py", line 576, in parse IOError: [Errno 2] No such file or directory: '/scratch/galaxy/tmptzuSYJ' Traceback (most recent call last): File "./scripts/set_metadata.py", line 118, in __main__() File "./scripts/set_metadata.py", line 68, in __main__ datatypes_registry.load_datatypes( root_dir=config_root, config=datatypes_config ) File "/misc/local/galaxy/galaxy-dist/lib/galaxy/datatypes/registry.py", line 65, in load_datatypes tree = galaxy.util.parse_xml( config ) File "/misc/local/galaxy/galaxy-dist/lib/galaxy/util/__init__.py", line 143, in parse_xml tree = ElementTree.parse(fname) File "build/bdist.linux-x86_64/egg/elementtree/ElementTree.py", line 859, in parse File "build/bdist.linux-x86_64/egg/elementtree/ElementTree.py", line 576, in parse IOError: [Errno 2] No such file or directory: '/scratch/galaxy/tmptzuSYJ' Also here is what /scratch/galaxy looks like. It seems like the file it mentions does exist?? >ls -lh /scratch/galaxy/ total 134M -rw-r--r-- 1 galaxy scicomp 20K Nov 13 13:35 tmpfCaiGX -rw-r--r-- 1 galaxy scicomp 20K Nov 14 13:36 tmpM2Wclq -rw-r--r-- 1 galaxy scicomp 20K Nov 14 13:35 tmpsIyPur -rw-r--r-- 1 galaxy scicomp 20K Nov 16 08:48 tmptY5RaM -rw-r--r-- 1 galaxy scicomp 20K Nov 16 08:54 tmptzuSYJ -rw-r--r-- 1 galaxy scicomp 20K Nov 15 15:47 tmpu77PEK -rw--- 1 galaxy scicomp 286 Nov 16 08:58 tmpykoBIq -rw--- 1 galaxy scicomp 134M Nov 16 08:58 upload_file_data_G3eZEg Thanks, Greg ___ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/listinfo/galaxy-dev To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
