Re: [galaxy-dev] how to use projects for fair-share on compute-cluster
I really want this for Torque/Moab where the native spec flag is -A
Sent from my iPhone
On Jan 20, 2012, at 7:34 PM, "Edward Kirton" wrote:
> Great idea, Nate (hint! hint!).
>
> On Thu, Jan 19, 2012 at 10:27 AM, Nate Coraor wrote:
>> Hey Ed,
>>
>> This is a neat approach. You could possibly also do this in the Galaxy
>> database by associating users and groups with roles that match project
>> names. A select list or history default that allowed users to select their
>> "current" project/role would remove the single-project-per-user limitation.
>>
>> --nate
>>
>> On Jan 13, 2012, at 3:17 PM, Edward Kirton wrote:
>>
>>> correction: i didn't adequately test what happens if the user_proj_map_db
>>> was not defined in the universe file; here's the changes:
>>>
>>> 157 # BEGIN ADD USER'S PROJ
>>> 158 try:
>>> 159 conn = sqlite3.connect(self.app.config.user_proj_map_db)
>>> 160 c = conn.cursor()
>>> 161 c.execute('SELECT PROJ FROM USER_PROJ WHERE GID=?',
>>> [job_wrapper.user_id])
>>> 162 row = c.fetchone()
>>> 163 c.close
>>> 164 native_spec += ' -P ' + row[0]
>>> 165 except:
>>> 166 log.debug("Cannot look up proj of user %s" %
>>> job_wrapper.user_id)
>>> 167 # /END ADD USER PROJ
>>>
>>> also, in the config, define a default instead of using None:
>>> self.user_proj_map_db = resolve_path( kwargs.get(
>>> "user_proj_map_db", "database/user_proj_map.sqlite" ), self.root )
>>>
>>> one last note: there doesn't seem to be any error displayed to the user if
>>> the job cannot be scheduled because the galaxy user doesn't have
>>> permissions to use the user's project (although there is a log entry), but
>>> the job will never be scheduled. so be sure the galaxy user has
>>> permissions to submit to all possible projects.
>>> ___
>>> Please keep all replies on the list by using "reply all"
>>> in your mail client. To manage your subscriptions to this
>>> and other Galaxy lists, please use the interface at:
>>>
>>> http://lists.bx.psu.edu/
>>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client. To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>
> http://lists.bx.psu.edu/
___
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] how to use projects for fair-share on compute-cluster
Great idea, Nate (hint! hint!).
On Thu, Jan 19, 2012 at 10:27 AM, Nate Coraor wrote:
> Hey Ed,
>
> This is a neat approach. You could possibly also do this in the Galaxy
> database by associating users and groups with roles that match project names.
> A select list or history default that allowed users to select their
> "current" project/role would remove the single-project-per-user limitation.
>
> --nate
>
> On Jan 13, 2012, at 3:17 PM, Edward Kirton wrote:
>
>> correction: i didn't adequately test what happens if the user_proj_map_db
>> was not defined in the universe file; here's the changes:
>>
>> 157 # BEGIN ADD USER'S PROJ
>> 158 try:
>> 159 conn = sqlite3.connect(self.app.config.user_proj_map_db)
>> 160 c = conn.cursor()
>> 161 c.execute('SELECT PROJ FROM USER_PROJ WHERE GID=?',
>> [job_wrapper.user_id])
>> 162 row = c.fetchone()
>> 163 c.close
>> 164 native_spec += ' -P ' + row[0]
>> 165 except:
>> 166 log.debug("Cannot look up proj of user %s" %
>> job_wrapper.user_id)
>> 167 # /END ADD USER PROJ
>>
>> also, in the config, define a default instead of using None:
>> self.user_proj_map_db = resolve_path( kwargs.get(
>> "user_proj_map_db", "database/user_proj_map.sqlite" ), self.root )
>>
>> one last note: there doesn't seem to be any error displayed to the user if
>> the job cannot be scheduled because the galaxy user doesn't have permissions
>> to use the user's project (although there is a log entry), but the job will
>> never be scheduled. so be sure the galaxy user has permissions to submit to
>> all possible projects.
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client. To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>
>> http://lists.bx.psu.edu/
>
___
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] Galaxy Hang after DrmCommunicationException
yes, nate but that fails the job but it is, in fact, still running and
the error should be ignored
except Exception, e:
# so we don't kill the monitor thread
log.exception("(%s/%s) Unable to check job status" % (
galaxy_job_id, job_id ) )
log.warning("(%s/%s) job will now be errored" % (
galaxy_job_id, job_id ) )
drm_job_state.fail_message = "Cluster could not complete job"
self.work_queue.put( ( 'fail', drm_job_state ) )
continue
On Fri, Jan 20, 2012 at 9:40 AM, Nate Coraor wrote:
>
> Hi Ann,
>
> The cause of the exception aside, this should be caught by the except block
> below it in drmaa.py (in check_watched_items()):
>
> except Exception, e:
> # so we don't kill the monitor thread
> log.exception("(%s/%s) Unable to check job status" % (
> galaxy_job_id, job_id ) )
>
> What changeset are you running?
>
> --nate
___
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] samtools version detection issue
On Jan 20, 2012, at 12:12 AM, ambarish biswas wrote: > Hi, > I'm not sure if this is related or not but I'm also noticed a > similar problem where the program is not found in the path, but it > exists. I have it submitted as an issue , but it could be related here > so mentioned here. The problem started when I recently upgraded my > Galaxy installation(It was running in cluster with SGE, in Red hat > linux). Currently, I'm running Galaxy with drmaa in a cluster in Red > Hat linux, where the web server is running in one Server, and the > execution is being performed in another. I noticed that programs like > ClustalW, dialign-tx etc, which uses a wrapper, is giving error as > follows: > sh: dialign-tx: command not found > The original command line in the wrapper is as follows: > command = "dialign-tx %s/dialign-tx %s %s -g%s -n%s -D %s %s" % > (GALAXY_DATA_INDEX_DIR, options, sensitivity, max_frag_length, > length_low_score_pos, input_file_name, output_file) > > Even though, the programs are present in the Execution node's path, > and if I add, explicit path, like "/usr/local/bin" in the command as > bellow: > command = "/usr/local/bin/dialign-tx %s/dialign-tx %s %s -g%s -n%s -D > %s %s" % (GALAXY_DATA_INDEX_DIR, options, sensitivity, > max_frag_length, length_low_score_pos, input_file_name, output_file) > > Then the programs works fine without any error. > > I compared, the wrapper files with my old installation and the current > installation, and both are 100% same. So, I guess the path to the > executable is missing from the environment. I searched a lot, but > couldn't find any solution to rectify this issue. > Can someone please point out where I need to change to avoid adding > explicit path to each programs. > Thanks in advance. I replied to your bitbucket ticket first, just in case you don't see it, here it is: https://bitbucket.org/galaxy/galaxy-central/issue/702/program-command-not-found-in-path --nate > > > With Regards, > > Ambarish Biswas, > University of Otago > Department of Biochemistry, > Dunedin, New Zealand, > > > On Fri, Jan 20, 2012 at 11:07 AM, Nate Coraor wrote: >> On Jan 18, 2012, at 5:28 PM, Matt Shirley wrote: >> >>> Hi all. I'm sorry if this has been answered before, but I've searched and >>> cannot find a solution other than "add SAMtools to your PATH", which I >>> already have done. I can invoke samtools from BASH while logged in as the >>> "galaxy" user I set up according to the production installation guide. When >>> I try to add .bam files to a shared data library, I get the following >>> message: >>> >>> Traceback (most recent call last): >>> File "/home/galaxy/galaxy-dist/tools/data_source/upload.py", line 394, in >>> __main__() >>> File "/home/galaxy/galaxy-dist/tools/data_source/upload.py", line 386, in >>> __main__ >>> add_file( dataset, registry, json_file, output_path ) >>> File "/home/galaxy/galaxy-dist/tools/data_source/upload.py", line 327, in >>> add_file >>> if link_data_only == 'copy_files' and >>> datatype.dataset_content_needs_grooming( output_path ): >>> File "/home/galaxy/galaxy-dist/lib/galaxy/datatypes/binary.py", line 79, in >>> dataset_content_needs_grooming >>> version = self._get_samtools_version() >>> File "/home/galaxy/galaxy-dist/lib/galaxy/datatypes/binary.py", line 63, in >>> _get_samtools_version >>> output = subprocess.Popen( [ 'samtools' ], stderr=subprocess.PIPE, >>> stdout=subprocess.PIPE ).communicate()[1] >>> File "/usr/lib64/python2.6/subprocess.py", line 639, in __init__ >>> errread, errwrite) >>> File "/usr/lib64/python2.6/subprocess.py", line 1220, in _execute_child >>> raise child_exception >>> OSError: [Errno 2] No such file or directory >>> >>> I've taken a peek at the "dataset_content_needs_grooming" function and it >>> looks like there is an OS call that executes "$ samtools" and splits the >>> resulting string to access the version number of samtools (why couldn't the >>> developers add a --version flag?). It seems to me like samtools cannot be >>> executed, but I can't think of any reason why. Invoking samtools as user >>> "galaxy" (with the user's associated PATH) gives: >>> >>> Program: samtools (Tools for alignments in the SAM format) >>> Version: 0.1.12a (r862) >>> Usage: samtools [options] >>> Command: view SAM<->BAM conversion >>> sort sort alignment file >>> pileup generate pileup output >>> mpileup multi-way pileup >>> faidx index/extract FASTA >>> tview text alignment viewer >>> index index alignment >>> idxstats BAM index stats (r595 or later) >>> fixmate fix mate information >>> glfview print GLFv3 file >>> flagstat simple stats >>> calmd recalculate MD/NM tags and '=' bases >>> merge merge sorted alignments >>> rmdup remove PCR duplicates >>> reheader replace BAM header >>> >>> Does anyone know where I've gone wrong? >> >> Hi Matt, >> >> Are you running tools on a cluster? If running locally, are you starting >> Galaxy by hand (i.e. run.sh) from that same she
Re: [galaxy-dev] Galaxy Hang after DrmCommunicationException
On Jan 11, 2012, at 12:18 PM, Ann Black wrote:
> Good Morning galaxy group!
>
> I was hoping that someone might have some ideas on a problem we have
> experienced a handful of times running galaxy on our local cluster.
>
> Occasionally we experience some communication timeouts between out cluster
> head node and a compute node which will self heal. However, this in turn will
> hang galaxy. Below you will see output from our galaxy log file. When the
> ERROR happens (which is not often) it consistently seems to hang galaxy. We
> have to kill it off and restart it. We are running galaxy as a single PID at
> this time (we are still just testing it out, etc) and it is running on our
> head node (which we plan to move off of in the future).
>
> galaxy.jobs.runners.drmaa DEBUG 2012-01-10 19:19:58,800 (1654/698075) state
> change: job is running
> galaxy.jobs.runners.drmaa ERROR 2012-01-10 20:57:47,021 (1654/698075) Unable
> to check job status
> Traceback (most recent call last):
> File "/data/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 236, in
> check_watched_items
> state = self.ds.jobStatus( job_id )
> File "/data/galaxy-dist/eggs/drmaa-0.4b3-py2.7.egg/drmaa/__init__.py", line
> 522, in jobStatus
> _h.c(_w.drmaa_job_ps, jobName, _ct.byref(status))
> File "/data/galaxy-dist/eggs/drmaa-0.4b3-py2.7.egg/drmaa/helpers.py", line
> 213, in c
> return f(*(args + (error_buffer, sizeof(error_buffer
> File "/data/galaxy-dist/eggs/drmaa-0.4b3-py2.7.egg/drmaa/errors.py", line
> 90, in error_check
> raise _ERRORS[code-1]("code %s: %s" % (code, error_buffer.value))
> DrmCommunicationException: code 2: failed receiving gdi request response for
> mid=24442 (got syncron message receive timeout error).
> galaxy.jobs.runners.drmaa WARNING 2012-01-10 20:58:05,090 (1654/698075) job
> will now be errored
> galaxy.jobs.runners.drmaa DEBUG 2012-01-10 20:59:06,396 (1654/698075) User
> killed running job, but error encountered removing from DRM queue: code 2:
> failed receiving gdi request response for mid=2 (got syncron message
> receive timeout error).
> galaxy.datatypes.metadata DEBUG 2012-01-10 20:59:06,896 Cleaning up external
> metadata files
> galaxy.datatypes.metadata DEBUG 2012-01-10 20:59:06,947 Failed to cleanup
> MetadataTempFile temp files from
> database/tmp/metadata_out_HistoryDatasetAssociation_2913_ZUTgBy: No JSON
> object could be decoded: line 1 column 0 (char 0)
> galaxy.datatypes.metadata DEBUG 2012-01-10 20:59:09,640 Cleaning up external
> metadata files
> galaxy.jobs INFO 2012-01-10 20:59:09,697 job 1656 unable to run: one or more
> inputs in error state
> galaxy.datatypes.metadata DEBUG 2012-01-10 20:59:10,121 Cleaning up external
> metadata files
> galaxy.jobs INFO 2012-01-10 20:59:10,159 job 1655 unable to run: one or more
> inputs in error state
> galaxy.datatypes.metadata DEBUG 2012-01-10 20:59:12,076 Cleaning up external
> metadata files
> galaxy.jobs INFO 2012-01-10 20:59:12,126 job 1657 unable to run: one or more
> inputs in error state
> galaxy.datatypes.metadata DEBUG 2012-01-10 20:59:13,601 Cleaning up external
> metadata files
> galaxy.jobs INFO 2012-01-10 20:59:13,650 job 1658 unable to run: one or more
> inputs in error state
>
>
> Has anyone else experienced this or have some ideas on how we can further
> debug to figure out why galaxy hangs?
Hi Ann,
The cause of the exception aside, this should be caught by the except block
below it in drmaa.py (in check_watched_items()):
except Exception, e:
# so we don't kill the monitor thread
log.exception("(%s/%s) Unable to check job status" % (
galaxy_job_id, job_id ) )
What changeset are you running?
--nate
>
> Thanks much!
>
> Ann Black-Ziegelbein
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client. To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>
> http://lists.bx.psu.edu/
___
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] Status on importing BAM file into Library does not update
Just wanted to add that we have consistently seen this issue of 'samtools index' running locally on our install. We are using SGE scheduler. Thanks for pointing out details in the code Nate. -- Shantanu. On Jan 20, 2012, at 9:35 AM, Nate Coraor wrote: > On Jan 18, 2012, at 11:54 AM, Ryan Golhar wrote: > >> Nate - Is there a specific place in the Galaxy code that forks the samtools >> index on bam files on the cluster or the head node? I really need to track >> this down. > > Hey Ryan, > > Sorry it's taken so long, I've been pretty busy. The relevant code is in > galaxy-dist/lib/galaxy/datatypes/binary.py, in the Bam class. When Galaxy > runs a tool, it creates a Job, which is placed inside a JobWrapper in > lib/galaxy/jobs/__init__.py. After the job execution is complete, the > JobWrapper.finish() method is called, which contains: > >if not self.app.config.set_metadata_externally or \ > ( not > self.external_output_metadata.external_metadata_set_successfully( dataset, > self.sa_session ) \ > and self.app.config.retry_metadata_internally ): >dataset.set_meta( overwrite = False ) > > Somehow, this conditional is being entered. Since set_metadata_externally is > set to True, presumably the problem is external_metadata_set_successfully() > is returning False and retry_metadata_internally is set to True. If you > leave behind the relevant job files (cleanup_job = never) and have a look at > the PBS and metadata outputs you may be able to see what's happening. Also, > you'll want to set retry_metadata_internally = False. > > --nate > >> >> On Fri, Jan 13, 2012 at 12:54 PM, Ryan Golhar >> wrote: >> I re-uploaded 3 BAM files using the "Upload system file paths. runner0.log >> shows: >> >> galaxy.jobs DEBUG 2012-01-13 12:50:08,442 dispatching job 76 to pbs runner >> galaxy.jobs INFO 2012-01-13 12:50:08,555 job 76 dispatched >> galaxy.jobs.runners.pbs DEBUG 2012-01-13 12:50:08,697 (76) submitting file >> /home/galaxy/galaxy-dist-9/database/pbs/76.sh >> galaxy.jobs.runners.pbs DEBUG 2012-01-13 12:50:08,697 (76) command is: >> python /home/galaxy/galaxy-dist-9/tools/data_source/upload.py >> /home/galaxy/galaxy-dist-9 /home/galaxy/galaxy-dist-9/datatypes_conf.xml >> /home/galaxy/galaxy-dist-9/database/tmp/tmpqrVYY7 >> 208:/home/galaxy/galaxy-dist-9/database/job_working_directory/76/dataset_208_files:None >> >> 209:/home/galaxy/galaxy-dist-9/database/job_working_directory/76/dataset_209_files:None >> >> 210:/home/galaxy/galaxy-dist-9/database/job_working_directory/76/dataset_210_files:None; >> cd /home/galaxy/galaxy-dist-9; /home/galaxy/galaxy-dist-9/set_metadata.sh >> ./database/files ./database/tmp . datatypes_conf.xml >> ./database/job_working_directory/76/galaxy.json >> galaxy.jobs.runners.pbs DEBUG 2012-01-13 12:50:08,699 (76) queued in default >> queue as 114.localhost.localdomain >> galaxy.jobs.runners.pbs DEBUG 2012-01-13 12:50:09,037 >> (76/114.localhost.localdomain) PBS job state changed from N to R >> galaxy.jobs.runners.pbs DEBUG 2012-01-13 12:51:09,205 >> (76/114.localhost.localdomain) PBS job state changed from R to E >> galaxy.jobs.runners.pbs DEBUG 2012-01-13 12:51:10,206 >> (76/114.localhost.localdomain) PBS job state changed from E to C >> galaxy.jobs.runners.pbs DEBUG 2012-01-13 12:51:10,206 >> (76/114.localhost.localdomain) PBS job has completed successfully >> >> 76.sh shows: >> [galaxy@bic pbs]$ more 76.sh >> #!/bin/sh >> GALAXY_LIB="/home/galaxy/galaxy-dist-9/lib" >> if [ "$GALAXY_LIB" != "None" ]; then >>if [ -n "$PYTHONPATH" ]; then >>export PYTHONPATH="$GALAXY_LIB:$PYTHONPATH" >>else >>export PYTHONPATH="$GALAXY_LIB" >>fi >> fi >> cd /home/galaxy/galaxy-dist-9/database/job_working_directory/76 >> python /home/galaxy/galaxy-dist-9/tools/data_source/upload.py >> /home/galaxy/galaxy-dist-9 /home/galaxy/galaxy-dist-9/datatypes_conf.xml >> /home/galaxy >> /galaxy-dist-9/database/tmp/tmpqrVYY7 >> 208:/home/galaxy/galaxy-dist-9/database/job_working_directory/76/dataset_208_files:None >> 209:/ >> home/galaxy/galaxy-dist-9/database/job_working_directory/76/dataset_209_files:None >> 210:/home/galaxy/galaxy-dist-9/database/job_working_dire >> ctory/76/dataset_210_files:None; cd /home/galaxy/galaxy-dist-9; >> /home/galaxy/galaxy-dist-9/set_metadata.sh ./database/files ./database/tmp . >> dataty >> pes_conf.xml ./database/job_working_directory/76/galaxy.json >> >> Right as the job ended I check the job output files: >> >> [galaxy@bic pbs]$ ll >> total 4 >> -rw-rw-r-- 1 galaxy galaxy 950 Jan 13 12:50 76.sh >> [galaxy@bic pbs]$ ll >> total 4 >> -rw--- 1 galaxy galaxy 0 Jan 13 12:50 76.e >> -rw--- 1 galaxy galaxy 0 Jan 13 12:50 76.o >> -rw-rw-r-- 1 galaxy galaxy 950 Jan 13 12:50 76.sh >> >> samtools is now running on the head node. >> >> >> Where does Galaxy decide
Re: [galaxy-dev] running functional tests on a galaxy server
Thanks Nate.
I have tried that but it seems to run into problems with the expected output.
This may be simply the aforementioned python dependency issues, but I'll have
to investigate more in the coming week when I have time.
Just to get an idea of what I would like, I mainly want the capability to run
tests on (for instance) a new Galaxy deployment on an already-established
cluster or other similar conditions where tools may not be configured correctly
(e.g. missing secondary dependencies) or subtle versioning issues exist (e.g.
wrong version of cufflinks).
I'm thinking this could eventually lead to a mechanism where one optionally
generates a tools_conf file that comments out or removes tool configurations
not working properly, as well as generates a report of the issues found (as
run_functional_tests already does).
chris
On Jan 20, 2012, at 8:54 AM, Nate Coraor wrote:
> On Jan 19, 2012, at 12:57 AM, Fields, Christopher J wrote:
>
>> Just to add for the list, nate mentioned on IRC that running tests on an
>> external server likely doesn't work at the moment but that
>> ./scripts/functional_tests.py may have something to work with via
>> environment variables.
>
> The tool config can be specified in:
>
>$GALAXY_TEST_TOOL_CONF
>
> The config file is a bit trickier since many Galaxy config options can be
> overridden by environment variables or are set to something other than the
> normal default right in functional_tests.py. You could change this line
> (147):
>
>global_conf = { '__file__' : 'universe_wsgi.ini.sample' }
>
> To change the config default, however.
>
> --nate
>
>> chris
>>
>> On Jan 18, 2012, at 8:03 PM, Fields, Christopher J wrote:
>>
>>> Greg,
>>>
>>> That seems to work mainly for running tests from the sample universe/tool
>>> config files, so it uses the local job runner. What I would like to do is
>>> run tests using the actual Galaxy server configuration (which might use
>>> PBS/DRMAA and a cluster). The method I mention from that page seemed to be
>>> for that purpose, but again it seems possibly out-of-date...
>>>
>>> chris
>>>
>>> On Jan 18, 2012, at 4:20 PM, Greg Von Kuster wrote:
>>>
Hello Chris,
You should be able to run functional tests from the Galaxy install
directory using the following from the command line.
%sh run_functional_tests.sh
There are several options you can use as well.
Greg
On Jan 18, 2012, at 4:38 PM, Fields, Christopher J wrote:
> I have been following the basic protocol for running remote tests on a
> Galaxy server (in this case, our local one) using:
>
> http://wiki.g2.bx.psu.edu/Admin/Running%20Tests
>
>> From reading this, I assume the tests are to be run from the galaxy root
>> directory, like so:
>
> GALAXY_TEST_EXTERNAL=1 GALAXY_TEST_HOST=my.local.galaxy.url
> GALAXY_TEST_PORT=8080 \
> ./nosetests.sh
>
> However, no 'nosetests.sh' script exists there. Should we be using
> ./scripts/nosetests.py? ./scripts/functional_tests.py?
>
> Any clarification on this would be very helpful, we're just searching for
> a way to run tests in as automated a way as possible using the server
> configuration.
>
> chris
>
>
> Chris Fields
> Senior Research Scientist
> National Center for Supercomputing Applications
> Institute for Genomic Biology
> University of Illinois at Urbana-Champaign
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client. To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>
> http://lists.bx.psu.edu/
Greg Von Kuster
Galaxy Development Team
g...@bx.psu.edu
>>>
>>>
>>> ___
>>> Please keep all replies on the list by using "reply all"
>>> in your mail client. To manage your subscriptions to this
>>> and other Galaxy lists, please use the interface at:
>>>
>>> http://lists.bx.psu.edu/
>>
>>
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client. To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>
>> http://lists.bx.psu.edu/
>
___
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
[galaxy-dev] problem with "Input dataset" workflow control feature and custom non-subclass datatypes
Hi, There seems to be a weird bug with the "Input dataset" workflow control feature, hard to explain clearly but I'll try my best. If you define a custom datatype that is a simple subclass of an existing galaxy datatype, e.g.: And if this datatype will be the input to a workflow where you want to use the multiple input files feature you must put into your workflow editor an "Input dataset" box at the beginning and connect it. If you define a custom datatype that's it's own custom class, e.g.: with a simple class in lib/galaxy/datatypes/data.py e.g.: class MyExt( Data ): file_ext = "myext" And if this datatype will be the input data to a workflow if you have an "Input dataset" box at the beginning for some reason the drop-down menu (or multi-select) won't not have files of this type from your history it just ignores them. Now what is strange is if I edit the workflow and remove the beginning "Input dataset" box and start the workflow with just the first tool which has this custom datatype as an input parameter then when I try to run the workflow everything shows up properly :-/ Hope I explained this ok, seems like something is broken with the "Input dataset" workflow control feature. best, Leandro ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Status on importing BAM file into Library does not update
Galaxy shouldn't be trying to do that, but it also shouldn't cause metadata to fail. On Jan 20, 2012, at 10:52 AM, Ryan Golhar wrote: > Thanks Nate. I'll play with that. Could it be that Galaxy is trying to > reset the permissions or ownership of the imported BAM files. I'm not > copying them into Galaxy, rather I am linking to them. That is the only > error I see in runner0.log that indicates any type of failure. > > On Fri, Jan 20, 2012 at 10:35 AM, Nate Coraor wrote: > On Jan 18, 2012, at 11:54 AM, Ryan Golhar wrote: > > > Nate - Is there a specific place in the Galaxy code that forks the samtools > > index on bam files on the cluster or the head node? I really need to track > > this down. > > Hey Ryan, > > Sorry it's taken so long, I've been pretty busy. The relevant code is in > galaxy-dist/lib/galaxy/datatypes/binary.py, in the Bam class. When Galaxy > runs a tool, it creates a Job, which is placed inside a JobWrapper in > lib/galaxy/jobs/__init__.py. After the job execution is complete, the > JobWrapper.finish() method is called, which contains: > >if not self.app.config.set_metadata_externally or \ > ( not > self.external_output_metadata.external_metadata_set_successfully( dataset, > self.sa_session ) \ > and self.app.config.retry_metadata_internally ): >dataset.set_meta( overwrite = False ) > > Somehow, this conditional is being entered. Since set_metadata_externally is > set to True, presumably the problem is external_metadata_set_successfully() > is returning False and retry_metadata_internally is set to True. If you > leave behind the relevant job files (cleanup_job = never) and have a look at > the PBS and metadata outputs you may be able to see what's happening. Also, > you'll want to set retry_metadata_internally = False. > > --nate > > > > > On Fri, Jan 13, 2012 at 12:54 PM, Ryan Golhar > > wrote: > > I re-uploaded 3 BAM files using the "Upload system file paths. runner0.log > > shows: > > > > galaxy.jobs DEBUG 2012-01-13 12:50:08,442 dispatching job 76 to pbs runner > > galaxy.jobs INFO 2012-01-13 12:50:08,555 job 76 dispatched > > galaxy.jobs.runners.pbs DEBUG 2012-01-13 12:50:08,697 (76) submitting file > > /home/galaxy/galaxy-dist-9/database/pbs/76.sh > > galaxy.jobs.runners.pbs DEBUG 2012-01-13 12:50:08,697 (76) command is: > > python /home/galaxy/galaxy-dist-9/tools/data_source/upload.py > > /home/galaxy/galaxy-dist-9 /home/galaxy/galaxy-dist-9/datatypes_conf.xml > > /home/galaxy/galaxy-dist-9/database/tmp/tmpqrVYY7 > > 208:/home/galaxy/galaxy-dist-9/database/job_working_directory/76/dataset_208_files:None > > > > 209:/home/galaxy/galaxy-dist-9/database/job_working_directory/76/dataset_209_files:None > > > > 210:/home/galaxy/galaxy-dist-9/database/job_working_directory/76/dataset_210_files:None; > > cd /home/galaxy/galaxy-dist-9; /home/galaxy/galaxy-dist-9/set_metadata.sh > > ./database/files ./database/tmp . datatypes_conf.xml > > ./database/job_working_directory/76/galaxy.json > > galaxy.jobs.runners.pbs DEBUG 2012-01-13 12:50:08,699 (76) queued in > > default queue as 114.localhost.localdomain > > galaxy.jobs.runners.pbs DEBUG 2012-01-13 12:50:09,037 > > (76/114.localhost.localdomain) PBS job state changed from N to R > > galaxy.jobs.runners.pbs DEBUG 2012-01-13 12:51:09,205 > > (76/114.localhost.localdomain) PBS job state changed from R to E > > galaxy.jobs.runners.pbs DEBUG 2012-01-13 12:51:10,206 > > (76/114.localhost.localdomain) PBS job state changed from E to C > > galaxy.jobs.runners.pbs DEBUG 2012-01-13 12:51:10,206 > > (76/114.localhost.localdomain) PBS job has completed successfully > > > > 76.sh shows: > > [galaxy@bic pbs]$ more 76.sh > > #!/bin/sh > > GALAXY_LIB="/home/galaxy/galaxy-dist-9/lib" > > if [ "$GALAXY_LIB" != "None" ]; then > > if [ -n "$PYTHONPATH" ]; then > > export PYTHONPATH="$GALAXY_LIB:$PYTHONPATH" > > else > > export PYTHONPATH="$GALAXY_LIB" > > fi > > fi > > cd /home/galaxy/galaxy-dist-9/database/job_working_directory/76 > > python /home/galaxy/galaxy-dist-9/tools/data_source/upload.py > > /home/galaxy/galaxy-dist-9 /home/galaxy/galaxy-dist-9/datatypes_conf.xml > > /home/galaxy > > /galaxy-dist-9/database/tmp/tmpqrVYY7 > > 208:/home/galaxy/galaxy-dist-9/database/job_working_directory/76/dataset_208_files:None > > 209:/ > > home/galaxy/galaxy-dist-9/database/job_working_directory/76/dataset_209_files:None > > 210:/home/galaxy/galaxy-dist-9/database/job_working_dire > > ctory/76/dataset_210_files:None; cd /home/galaxy/galaxy-dist-9; > > /home/galaxy/galaxy-dist-9/set_metadata.sh ./database/files ./database/tmp > > . dataty > > pes_conf.xml ./database/job_working_directory/76/galaxy.json > > > > Right as the job ended I check the job output files: > > > > [galaxy@bic pbs]$ ll > > total 4 > > -rw-rw-r-- 1 galaxy galaxy 950 Jan 13
Re: [galaxy-dev] Status on importing BAM file into Library does not update
Thanks Nate. I'll play with that. Could it be that Galaxy is trying to reset the permissions or ownership of the imported BAM files. I'm not copying them into Galaxy, rather I am linking to them. That is the only error I see in runner0.log that indicates any type of failure. On Fri, Jan 20, 2012 at 10:35 AM, Nate Coraor wrote: > On Jan 18, 2012, at 11:54 AM, Ryan Golhar wrote: > > > Nate - Is there a specific place in the Galaxy code that forks the > samtools index on bam files on the cluster or the head node? I really need > to track this down. > > Hey Ryan, > > Sorry it's taken so long, I've been pretty busy. The relevant code is in > galaxy-dist/lib/galaxy/datatypes/binary.py, in the Bam class. When Galaxy > runs a tool, it creates a Job, which is placed inside a JobWrapper in > lib/galaxy/jobs/__init__.py. After the job execution is complete, the > JobWrapper.finish() method is called, which contains: > >if not self.app.config.set_metadata_externally or \ > ( not > self.external_output_metadata.external_metadata_set_successfully( dataset, > self.sa_session ) \ > and self.app.config.retry_metadata_internally ): > dataset.set_meta( overwrite = False ) > > Somehow, this conditional is being entered. Since set_metadata_externally > is set to True, presumably the problem is > external_metadata_set_successfully() is returning False and > retry_metadata_internally is set to True. If you leave behind the relevant > job files (cleanup_job = never) and have a look at the PBS and metadata > outputs you may be able to see what's happening. Also, you'll want to set > retry_metadata_internally = False. > > --nate > > > > > On Fri, Jan 13, 2012 at 12:54 PM, Ryan Golhar < > ngsbioinformat...@gmail.com> wrote: > > I re-uploaded 3 BAM files using the "Upload system file paths. > runner0.log shows: > > > > galaxy.jobs DEBUG 2012-01-13 12:50:08,442 dispatching job 76 to pbs > runner > > galaxy.jobs INFO 2012-01-13 12:50:08,555 job 76 dispatched > > galaxy.jobs.runners.pbs DEBUG 2012-01-13 12:50:08,697 (76) submitting > file /home/galaxy/galaxy-dist-9/database/pbs/76.sh > > galaxy.jobs.runners.pbs DEBUG 2012-01-13 12:50:08,697 (76) command is: > python /home/galaxy/galaxy-dist-9/tools/data_source/upload.py > /home/galaxy/galaxy-dist-9 /home/galaxy/galaxy-dist-9/datatypes_conf.xml > /home/galaxy/galaxy-dist-9/database/tmp/tmpqrVYY7 > 208:/home/galaxy/galaxy-dist-9/database/job_working_directory/76/dataset_208_files:None > > 209:/home/galaxy/galaxy-dist-9/database/job_working_directory/76/dataset_209_files:None > > 210:/home/galaxy/galaxy-dist-9/database/job_working_directory/76/dataset_210_files:None; > cd /home/galaxy/galaxy-dist-9; /home/galaxy/galaxy-dist-9/set_metadata.sh > ./database/files ./database/tmp . datatypes_conf.xml > ./database/job_working_directory/76/galaxy.json > > galaxy.jobs.runners.pbs DEBUG 2012-01-13 12:50:08,699 (76) queued in > default queue as 114.localhost.localdomain > > galaxy.jobs.runners.pbs DEBUG 2012-01-13 12:50:09,037 > (76/114.localhost.localdomain) PBS job state changed from N to R > > galaxy.jobs.runners.pbs DEBUG 2012-01-13 12:51:09,205 > (76/114.localhost.localdomain) PBS job state changed from R to E > > galaxy.jobs.runners.pbs DEBUG 2012-01-13 12:51:10,206 > (76/114.localhost.localdomain) PBS job state changed from E to C > > galaxy.jobs.runners.pbs DEBUG 2012-01-13 12:51:10,206 > (76/114.localhost.localdomain) PBS job has completed successfully > > > > 76.sh shows: > > [galaxy@bic pbs]$ more 76.sh > > #!/bin/sh > > GALAXY_LIB="/home/galaxy/galaxy-dist-9/lib" > > if [ "$GALAXY_LIB" != "None" ]; then > > if [ -n "$PYTHONPATH" ]; then > > export PYTHONPATH="$GALAXY_LIB:$PYTHONPATH" > > else > > export PYTHONPATH="$GALAXY_LIB" > > fi > > fi > > cd /home/galaxy/galaxy-dist-9/database/job_working_directory/76 > > python /home/galaxy/galaxy-dist-9/tools/data_source/upload.py > /home/galaxy/galaxy-dist-9 /home/galaxy/galaxy-dist-9/datatypes_conf.xml > /home/galaxy > > /galaxy-dist-9/database/tmp/tmpqrVYY7 > 208:/home/galaxy/galaxy-dist-9/database/job_working_directory/76/dataset_208_files:None > 209:/ > > > home/galaxy/galaxy-dist-9/database/job_working_directory/76/dataset_209_files:None > 210:/home/galaxy/galaxy-dist-9/database/job_working_dire > > ctory/76/dataset_210_files:None; cd /home/galaxy/galaxy-dist-9; > /home/galaxy/galaxy-dist-9/set_metadata.sh ./database/files ./database/tmp > . dataty > > pes_conf.xml ./database/job_working_directory/76/galaxy.json > > > > Right as the job ended I check the job output files: > > > > [galaxy@bic pbs]$ ll > > total 4 > > -rw-rw-r-- 1 galaxy galaxy 950 Jan 13 12:50 76.sh > > [galaxy@bic pbs]$ ll > > total 4 > > -rw--- 1 galaxy galaxy 0 Jan 13 12:50 76.e > > -rw--- 1 galaxy galaxy 0 Jan 13 12:50 76.o > > -rw-rw-r-- 1 galaxy galaxy 950 Jan 13 12:50 76.sh > > > > samtools is now running on the
[galaxy-dev] January 20, 2012 Galaxy Development News Brief
January 20, 2012 Galaxy Development News Brief Highlights: * New Object Store data integration layer introduced * RNA-seq updates: TopHat to 1.4.0 and Cufflinks, CuffDiff, CuffCompare to 1.3.0. * Tool Shed installation features and community tool additions * Trackster performance upgrades and new drag & drop composite track creation Mercurial pull: hg pull -u -r 63bc46cc73b7 Complete News Brief http://wiki.g2.bx.psu.edu/DevNewsBriefs/2012_01_20 Release history http://wiki.g2.bx.psu.edu/DevNewsBriefs/ Install/Administration Help http://getgalaxy.org Support http://galaxyproject.org/wiki/Support Thanks for using Galaxy! ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Status on importing BAM file into Library does not update
On Jan 18, 2012, at 11:54 AM, Ryan Golhar wrote: > Nate - Is there a specific place in the Galaxy code that forks the samtools > index on bam files on the cluster or the head node? I really need to track > this down. Hey Ryan, Sorry it's taken so long, I've been pretty busy. The relevant code is in galaxy-dist/lib/galaxy/datatypes/binary.py, in the Bam class. When Galaxy runs a tool, it creates a Job, which is placed inside a JobWrapper in lib/galaxy/jobs/__init__.py. After the job execution is complete, the JobWrapper.finish() method is called, which contains: if not self.app.config.set_metadata_externally or \ ( not self.external_output_metadata.external_metadata_set_successfully( dataset, self.sa_session ) \ and self.app.config.retry_metadata_internally ): dataset.set_meta( overwrite = False ) Somehow, this conditional is being entered. Since set_metadata_externally is set to True, presumably the problem is external_metadata_set_successfully() is returning False and retry_metadata_internally is set to True. If you leave behind the relevant job files (cleanup_job = never) and have a look at the PBS and metadata outputs you may be able to see what's happening. Also, you'll want to set retry_metadata_internally = False. --nate > > On Fri, Jan 13, 2012 at 12:54 PM, Ryan Golhar > wrote: > I re-uploaded 3 BAM files using the "Upload system file paths. runner0.log > shows: > > galaxy.jobs DEBUG 2012-01-13 12:50:08,442 dispatching job 76 to pbs runner > galaxy.jobs INFO 2012-01-13 12:50:08,555 job 76 dispatched > galaxy.jobs.runners.pbs DEBUG 2012-01-13 12:50:08,697 (76) submitting file > /home/galaxy/galaxy-dist-9/database/pbs/76.sh > galaxy.jobs.runners.pbs DEBUG 2012-01-13 12:50:08,697 (76) command is: python > /home/galaxy/galaxy-dist-9/tools/data_source/upload.py > /home/galaxy/galaxy-dist-9 /home/galaxy/galaxy-dist-9/datatypes_conf.xml > /home/galaxy/galaxy-dist-9/database/tmp/tmpqrVYY7 > 208:/home/galaxy/galaxy-dist-9/database/job_working_directory/76/dataset_208_files:None > > 209:/home/galaxy/galaxy-dist-9/database/job_working_directory/76/dataset_209_files:None > > 210:/home/galaxy/galaxy-dist-9/database/job_working_directory/76/dataset_210_files:None; > cd /home/galaxy/galaxy-dist-9; /home/galaxy/galaxy-dist-9/set_metadata.sh > ./database/files ./database/tmp . datatypes_conf.xml > ./database/job_working_directory/76/galaxy.json > galaxy.jobs.runners.pbs DEBUG 2012-01-13 12:50:08,699 (76) queued in default > queue as 114.localhost.localdomain > galaxy.jobs.runners.pbs DEBUG 2012-01-13 12:50:09,037 > (76/114.localhost.localdomain) PBS job state changed from N to R > galaxy.jobs.runners.pbs DEBUG 2012-01-13 12:51:09,205 > (76/114.localhost.localdomain) PBS job state changed from R to E > galaxy.jobs.runners.pbs DEBUG 2012-01-13 12:51:10,206 > (76/114.localhost.localdomain) PBS job state changed from E to C > galaxy.jobs.runners.pbs DEBUG 2012-01-13 12:51:10,206 > (76/114.localhost.localdomain) PBS job has completed successfully > > 76.sh shows: > [galaxy@bic pbs]$ more 76.sh > #!/bin/sh > GALAXY_LIB="/home/galaxy/galaxy-dist-9/lib" > if [ "$GALAXY_LIB" != "None" ]; then > if [ -n "$PYTHONPATH" ]; then > export PYTHONPATH="$GALAXY_LIB:$PYTHONPATH" > else > export PYTHONPATH="$GALAXY_LIB" > fi > fi > cd /home/galaxy/galaxy-dist-9/database/job_working_directory/76 > python /home/galaxy/galaxy-dist-9/tools/data_source/upload.py > /home/galaxy/galaxy-dist-9 /home/galaxy/galaxy-dist-9/datatypes_conf.xml > /home/galaxy > /galaxy-dist-9/database/tmp/tmpqrVYY7 > 208:/home/galaxy/galaxy-dist-9/database/job_working_directory/76/dataset_208_files:None > 209:/ > home/galaxy/galaxy-dist-9/database/job_working_directory/76/dataset_209_files:None > 210:/home/galaxy/galaxy-dist-9/database/job_working_dire > ctory/76/dataset_210_files:None; cd /home/galaxy/galaxy-dist-9; > /home/galaxy/galaxy-dist-9/set_metadata.sh ./database/files ./database/tmp . > dataty > pes_conf.xml ./database/job_working_directory/76/galaxy.json > > Right as the job ended I check the job output files: > > [galaxy@bic pbs]$ ll > total 4 > -rw-rw-r-- 1 galaxy galaxy 950 Jan 13 12:50 76.sh > [galaxy@bic pbs]$ ll > total 4 > -rw--- 1 galaxy galaxy 0 Jan 13 12:50 76.e > -rw--- 1 galaxy galaxy 0 Jan 13 12:50 76.o > -rw-rw-r-- 1 galaxy galaxy 950 Jan 13 12:50 76.sh > > samtools is now running on the head node. > > > Where does Galaxy decide how to run samtools? Maybe I can add a check of > some sort to see what's going on? > > > On Fri, Jan 13, 2012 at 10:53 AM, Nate Coraor wrote: > On Jan 12, 2012, at 11:41 PM, Ryan Golhar wrote: > > > Any ideas as to how to fix this? We are interested in using Galaxy to host > > all our NGS data. If indexing on the head node is going to happen, then > > this is going to
Re: [galaxy-dev] job quotas
On Jan 19, 2012, at 11:27 AM, Ivan Merelli wrote: > Hi, > > is it possible to restrict the number of concurrent jobs for a single > user in a local instance of galaxy? I see that in the public site > this feature is implemented, but I don't find documentation about > how to implement this locally. I think it's not possible through > pbs because all users submit jobs as the defauls galaxy user. Hi Ivan, It's not documented because I've been intending to make it a lot more robust, but you can set a hard limit by setting: user_job_limit = In Galaxy's config file. --nate > > Cheers, > > Ivan > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] .gz extension gets stripped off when uploading via data library Upload via filesystem paths but not via Get Data -> Upload File
Hi Greg, Ok this code change to library_common.py works, now when you use the data libraries menu to bring in .fastq.gz files it doesn't cut off the .gz thank you! best, Leandro On Fri, Jan 20, 2012 at 3:32 PM, Greg Von Kuster wrote: > Hello Leandro, > > I believe this behavior is due to the make_library_uploaded_dataset() method > in the ~/lib/galaxy/web/controllers/library_common controller. The current > method looks like this: > > def make_library_uploaded_dataset( self, trans, cntrller, params, name, > path, type, library_bunch, in_folder=None ): > library_bunch.replace_dataset = None # not valid for these types of > upload > uploaded_dataset = util.bunch.Bunch() > # Remove compressed file extensions, if any > new_name = name > if new_name.endswith( '.gz' ): > new_name = new_name.rstrip( '.gz' ) > elif new_name.endswith( '.zip' ): > new_name = new_name.rstrip( '.zip' ) > uploaded_dataset.name = new_name > uploaded_dataset.path = path > uploaded_dataset.type = type > uploaded_dataset.ext = None > uploaded_dataset.file_type = params.file_type > uploaded_dataset.dbkey = params.dbkey > uploaded_dataset.space_to_tab = params.space_to_tab > if in_folder: > uploaded_dataset.in_folder = in_folder > uploaded_dataset.data = upload_common.new_upload( trans, cntrller, > uploaded_dataset, library_bunch ) > link_data_only = params.get( 'link_data_only', 'copy_files' ) > uploaded_dataset.link_data_only = link_data_only > if link_data_only == 'link_to_files': > uploaded_dataset.data.file_name = os.path.abspath( path ) > # Since we are not copying the file into Galaxy's managed > # default file location, the dataset should never be purgable. > uploaded_dataset.data.dataset.purgable = False > trans.sa_session.add_all( ( uploaded_dataset.data, > uploaded_dataset.data.dataset ) ) > trans.sa_session.flush() > return uploaded_dataset > > Here are the code changes that I believe will resolve the issue. However, I > have not tested this, so if you wouldn't mind letting me know if this works > for you, I'll commit the changes to the central repo. > > def make_library_uploaded_dataset( self, trans, cntrller, params, name, > path, type, library_bunch, in_folder=None ): > link_data_only = params.get( 'link_data_only', 'copy_files' ) > library_bunch.replace_dataset = None # not valid for these types of > upload > uploaded_dataset = util.bunch.Bunch() > new_name = name > # Remove compressed file extensions, if any, but only if > # we're copying files into Galaxy's file space. > if link_data_only == 'copy_files': > if new_name.endswith( '.gz' ): > new_name = new_name.rstrip( '.gz' ) > elif new_name.endswith( '.zip' ): > new_name = new_name.rstrip( '.zip' ) > uploaded_dataset.name = new_name > uploaded_dataset.path = path > uploaded_dataset.type = type > uploaded_dataset.ext = None > uploaded_dataset.file_type = params.file_type > uploaded_dataset.dbkey = params.dbkey > uploaded_dataset.space_to_tab = params.space_to_tab > if in_folder: > uploaded_dataset.in_folder = in_folder > uploaded_dataset.data = upload_common.new_upload( trans, cntrller, > uploaded_dataset, library_bunch ) > uploaded_dataset.link_data_only = link_data_only > if link_data_only == 'link_to_files': > uploaded_dataset.data.file_name = os.path.abspath( path ) > # Since we are not copying the file into Galaxy's managed > # default file location, the dataset should never be purgable. > uploaded_dataset.data.dataset.purgable = False > trans.sa_session.add_all( ( uploaded_dataset.data, > uploaded_dataset.data.dataset ) ) > trans.sa_session.flush() > return uploaded_dataset > > > Thanks! > > Greg > > > On Jan 20, 2012, at 7:42 AM, Leandro Hermida wrote: > > Hello, > > We've created a new binary datatype for .fastq.gz files following the > same methodology as the BAM files since we don't want our fasta.gz > files to be gunzipped. I added the appropriate code in upload.py to > make sure of this. This new datatype and extension successfully does > not gunzip our files. But when we upload it into a data library via > the data library "Upload via filesystem paths" it for some reason > automatically strips the .gz part out. When we take the same .fastq.gz > file and upload it via Get Data -> Upload File it works fine, nothing > is stripped from file name. Where is it doing this and how can prevent > from stripping the .gz via the data library menus? > > thanks, > Leandro >
Re: [galaxy-dev] running functional tests on a galaxy server
On Jan 19, 2012, at 12:57 AM, Fields, Christopher J wrote:
> Just to add for the list, nate mentioned on IRC that running tests on an
> external server likely doesn't work at the moment but that
> ./scripts/functional_tests.py may have something to work with via environment
> variables.
The tool config can be specified in:
$GALAXY_TEST_TOOL_CONF
The config file is a bit trickier since many Galaxy config options can be
overridden by environment variables or are set to something other than the
normal default right in functional_tests.py. You could change this line (147):
global_conf = { '__file__' : 'universe_wsgi.ini.sample' }
To change the config default, however.
--nate
> chris
>
> On Jan 18, 2012, at 8:03 PM, Fields, Christopher J wrote:
>
>> Greg,
>>
>> That seems to work mainly for running tests from the sample universe/tool
>> config files, so it uses the local job runner. What I would like to do is
>> run tests using the actual Galaxy server configuration (which might use
>> PBS/DRMAA and a cluster). The method I mention from that page seemed to be
>> for that purpose, but again it seems possibly out-of-date...
>>
>> chris
>>
>> On Jan 18, 2012, at 4:20 PM, Greg Von Kuster wrote:
>>
>>> Hello Chris,
>>>
>>> You should be able to run functional tests from the Galaxy install
>>> directory using the following from the command line.
>>>
>>> %sh run_functional_tests.sh
>>>
>>> There are several options you can use as well.
>>>
>>> Greg
>>>
>>> On Jan 18, 2012, at 4:38 PM, Fields, Christopher J wrote:
>>>
I have been following the basic protocol for running remote tests on a
Galaxy server (in this case, our local one) using:
http://wiki.g2.bx.psu.edu/Admin/Running%20Tests
> From reading this, I assume the tests are to be run from the galaxy root
> directory, like so:
GALAXY_TEST_EXTERNAL=1 GALAXY_TEST_HOST=my.local.galaxy.url
GALAXY_TEST_PORT=8080 \
./nosetests.sh
However, no 'nosetests.sh' script exists there. Should we be using
./scripts/nosetests.py? ./scripts/functional_tests.py?
Any clarification on this would be very helpful, we're just searching for
a way to run tests in as automated a way as possible using the server
configuration.
chris
Chris Fields
Senior Research Scientist
National Center for Supercomputing Applications
Institute for Genomic Biology
University of Illinois at Urbana-Champaign
___
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
>>>
>>> Greg Von Kuster
>>> Galaxy Development Team
>>> g...@bx.psu.edu
>>>
>>>
>>>
>>
>>
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client. To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>
>> http://lists.bx.psu.edu/
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client. To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>
> http://lists.bx.psu.edu/
___
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] .gz extension gets stripped off when uploading via data library Upload via filesystem paths but not via Get Data -> Upload File
Hi Peter, Sorry I wasn't clear, the .gz gets stripped from the name in the Galaxy UI when you upload the files into a data library via the manage data libraries form. When you upload it via Get Data -> Upload File the .gz is preserved which is what one would want since I am not having it gunzipped by specifying its own new datatype and extension and changing binary.py to make sure it doesn't fall through to the elsif where it tries to unzip stuff. On Fri, Jan 20, 2012 at 2:25 PM, Peter Cock wrote: > On Fri, Jan 20, 2012 at 12:42 PM, Leandro Hermida > wrote: >> Hello, >> >> We've created a new binary datatype for .fastq.gz files following the >> same methodology as the BAM files since we don't want our fasta.gz >> files to be gunzipped. I added the appropriate code in upload.py to >> make sure of this. This new datatype and extension successfully does >> not gunzip our files. But when we upload it into a data library via >> the data library "Upload via filesystem paths" it for some reason >> automatically strips the .gz part out. When we take the same .fastq.gz >> file and upload it via Get Data -> Upload File it works fine, nothing >> is stripped from file name. Where is it doing this and how can prevent >> from stripping the .gz via the data library menus? > > I thought Galaxy would usually try to replace the extension > with *.dat for any file type when uploaded? > > Peter > > P.S. Is anyone working on the more general solution of > supporting a gzipped version of (almost) any Galaxy datatype? > https://bitbucket.org/galaxy/galaxy-central/issue/666/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] .gz extension gets stripped off when uploading via data library Upload via filesystem paths but not via Get Data -> Upload File
Hello Leandro, I believe this behavior is due to the make_library_uploaded_dataset() method in the ~/lib/galaxy/web/controllers/library_common controller. The current method looks like this: def make_library_uploaded_dataset( self, trans, cntrller, params, name, path, type, library_bunch, in_folder=None ): library_bunch.replace_dataset = None # not valid for these types of upload uploaded_dataset = util.bunch.Bunch() # Remove compressed file extensions, if any new_name = name if new_name.endswith( '.gz' ): new_name = new_name.rstrip( '.gz' ) elif new_name.endswith( '.zip' ): new_name = new_name.rstrip( '.zip' ) uploaded_dataset.name = new_name uploaded_dataset.path = path uploaded_dataset.type = type uploaded_dataset.ext = None uploaded_dataset.file_type = params.file_type uploaded_dataset.dbkey = params.dbkey uploaded_dataset.space_to_tab = params.space_to_tab if in_folder: uploaded_dataset.in_folder = in_folder uploaded_dataset.data = upload_common.new_upload( trans, cntrller, uploaded_dataset, library_bunch ) link_data_only = params.get( 'link_data_only', 'copy_files' ) uploaded_dataset.link_data_only = link_data_only if link_data_only == 'link_to_files': uploaded_dataset.data.file_name = os.path.abspath( path ) # Since we are not copying the file into Galaxy's managed # default file location, the dataset should never be purgable. uploaded_dataset.data.dataset.purgable = False trans.sa_session.add_all( ( uploaded_dataset.data, uploaded_dataset.data.dataset ) ) trans.sa_session.flush() return uploaded_dataset Here are the code changes that I believe will resolve the issue. However, I have not tested this, so if you wouldn't mind letting me know if this works for you, I'll commit the changes to the central repo. def make_library_uploaded_dataset( self, trans, cntrller, params, name, path, type, library_bunch, in_folder=None ): link_data_only = params.get( 'link_data_only', 'copy_files' ) library_bunch.replace_dataset = None # not valid for these types of upload uploaded_dataset = util.bunch.Bunch() new_name = name # Remove compressed file extensions, if any, but only if # we're copying files into Galaxy's file space. if link_data_only == 'copy_files': if new_name.endswith( '.gz' ): new_name = new_name.rstrip( '.gz' ) elif new_name.endswith( '.zip' ): new_name = new_name.rstrip( '.zip' ) uploaded_dataset.name = new_name uploaded_dataset.path = path uploaded_dataset.type = type uploaded_dataset.ext = None uploaded_dataset.file_type = params.file_type uploaded_dataset.dbkey = params.dbkey uploaded_dataset.space_to_tab = params.space_to_tab if in_folder: uploaded_dataset.in_folder = in_folder uploaded_dataset.data = upload_common.new_upload( trans, cntrller, uploaded_dataset, library_bunch ) uploaded_dataset.link_data_only = link_data_only if link_data_only == 'link_to_files': uploaded_dataset.data.file_name = os.path.abspath( path ) # Since we are not copying the file into Galaxy's managed # default file location, the dataset should never be purgable. uploaded_dataset.data.dataset.purgable = False trans.sa_session.add_all( ( uploaded_dataset.data, uploaded_dataset.data.dataset ) ) trans.sa_session.flush() return uploaded_dataset Thanks! Greg On Jan 20, 2012, at 7:42 AM, Leandro Hermida wrote: > Hello, > > We've created a new binary datatype for .fastq.gz files following the > same methodology as the BAM files since we don't want our fasta.gz > files to be gunzipped. I added the appropriate code in upload.py to > make sure of this. This new datatype and extension successfully does > not gunzip our files. But when we upload it into a data library via > the data library "Upload via filesystem paths" it for some reason > automatically strips the .gz part out. When we take the same .fastq.gz > file and upload it via Get Data -> Upload File it works fine, nothing > is stripped from file name. Where is it doing this and how can prevent > from stripping the .gz via the data library menus? > > thanks, > Leandro > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/ Greg Von Kuster Galaxy Development Team g...@bx.psu.edu ___ Please keep all replies on the list by using "re
[galaxy-dev] postgres user change problem
Dear all,
due to some work on the user management of our servers we had to rename
the user galaxy runs as. Up to now we used ident as postgres
authentication method, meaning that here postgres expects unix username
"galaxy" to have permissions of "galaxy" postgres user.
The entry in universe_wsgi.ini is simply:
database_connection = postgres:///galaxy?host=/var/run/postgresql
After the renaming, the new user "galaxynew" didn't get access at all at
first. Now I tried two things:
- adding a user "galaxynew" to postgres with permissions for database galaxy
and
- dumping the contents of database "galaxy" into a file and re-reading
this into database "galaxynew", which is owned by the user "galaxynew"
and of course changing the config file line to
database_connection = postgres:///galaxynew?host=/var/run/postgresql
In both cases I end up with an error during startup of galaxy:
[...]
WARNING 2012-01-20 14:46:55,556 Error closing cursor: current
transaction is aborted, commands ignored until end of transaction block
[...]
ProgrammingError: (ProgrammingError) permission denied for relation
migrate_version
'SELECT migrate_version.repository_id, migrate_version.repository_path,
migrate_version.version \nFROM migrate_version \nWHERE
migrate_version.repository_id = %(repository_id_1)s' {'repository_id_1':
'Galaxy'}
(complete traceback below).
Does anyone have a hint on how I could fix this?
Cheers,
Holger
sqlalchemy.pool.QueuePool.0x...8d10 WARNING 2012-01-20 14:46:55,556
Error closing cursor: current transaction is aborted, commands ignored
until end of transaction block
Traceback (most recent call last):
File
"/local/data/home/galaxy/galaxy-dist-2011-11-23/lib/galaxy/web/buildapp.py",
line 82, in app_factory
app = UniverseApplication( global_conf = global_conf, **kwargs )
File
"/local/data/home/galaxy/galaxy-dist-2011-11-23/lib/galaxy/app.py", line
39, in __init__
create_or_verify_database( db_url, kwargs.get( 'global_conf', {}
).get( '__file__', None ), self.config.database_engine_options )
File
"/local/data/home/galaxy/galaxy-dist-2011-11-23/lib/galaxy/model/migrate/check.py",
line 99, in create_or_verify_database
db_schema = schema.ControlledSchema( engine, migrate_repository )
File
"/local/data/home/galaxy/galaxy-dist-2011-11-23/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/schema.py",
line 24, in __init__
self._load()
File
"/local/data/home/galaxy/galaxy-dist-2011-11-23/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/schema.py",
line 41, in _load
self.table.c.repository_id == str(self.repository.id)))
File
"/local/data/home/galaxy/galaxy-dist-2011-11-23/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py",
line 1202, in execute
return connection.execute(statement, *multiparams, **params)
File
"/local/data/home/galaxy/galaxy-dist-2011-11-23/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py",
line 824, in execute
return Connection.executors[c](self, object, multiparams, params)
File
"/local/data/home/galaxy/galaxy-dist-2011-11-23/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py",
line 874, in _execute_clauseelement
return self.__execute_context(context)
File
"/local/data/home/galaxy/galaxy-dist-2011-11-23/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py",
line 896, in __execute_context
self._cursor_execute(context.cursor, context.statement,
context.parameters[0], context=context)
File
"/local/data/home/galaxy/galaxy-dist-2011-11-23/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py",
line 950, in _cursor_execute
self._handle_dbapi_exception(e, statement, parameters, cursor, context)
File
"/local/data/home/galaxy/galaxy-dist-2011-11-23/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py",
line 931, in _handle_dbapi_exception
raise exc.DBAPIError.instance(statement, parameters, e,
connection_invalidated=is_disconnect)
ProgrammingError: (ProgrammingError) permission denied for relation
migrate_version
'SELECT migrate_version.repository_id, migrate_version.repository_path,
migrate_version.version \nFROM migrate_version \nWHERE
migrate_version.repository_id = %(repository_id_1)s' {'repository_id_1':
'Galaxy'}
--
Dr. Holger Klein
Core Facility Bioinformatics
Institute of Molecular Biology gGmbH (IMB)
http://www.imb-mainz.de/
Tel: +49(6131) 39 21511
___
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] .gz extension gets stripped off when uploading via data library Upload via filesystem paths but not via Get Data -> Upload File
On Fri, Jan 20, 2012 at 12:42 PM, Leandro Hermida wrote: > Hello, > > We've created a new binary datatype for .fastq.gz files following the > same methodology as the BAM files since we don't want our fasta.gz > files to be gunzipped. I added the appropriate code in upload.py to > make sure of this. This new datatype and extension successfully does > not gunzip our files. But when we upload it into a data library via > the data library "Upload via filesystem paths" it for some reason > automatically strips the .gz part out. When we take the same .fastq.gz > file and upload it via Get Data -> Upload File it works fine, nothing > is stripped from file name. Where is it doing this and how can prevent > from stripping the .gz via the data library menus? I thought Galaxy would usually try to replace the extension with *.dat for any file type when uploaded? Peter P.S. Is anyone working on the more general solution of supporting a gzipped version of (almost) any Galaxy datatype? https://bitbucket.org/galaxy/galaxy-central/issue/666/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] .gz extension gets stripped off when uploading via data library Upload via filesystem paths but not via Get Data -> Upload File
Hello, We've created a new binary datatype for .fastq.gz files following the same methodology as the BAM files since we don't want our fasta.gz files to be gunzipped. I added the appropriate code in upload.py to make sure of this. This new datatype and extension successfully does not gunzip our files. But when we upload it into a data library via the data library "Upload via filesystem paths" it for some reason automatically strips the .gz part out. When we take the same .fastq.gz file and upload it via Get Data -> Upload File it works fine, nothing is stripped from file name. Where is it doing this and how can prevent from stripping the .gz via the data library menus? thanks, Leandro ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] "changeset" inconsistency
Hi Nate
Thank you very much for your e-mail
I am still testing and working on the last distribution, hopefully going
live this Sunday. But I will look into downloading and upgrading (one of
our dev servers) to the new distribution next week.
Regards, Hans
On 01/19/2012 07:39 PM, Nate Coraor wrote:
Hi Hans,
We're releasing a distribution today. If this is still the case with that
distribution, let me know. There were branches that weren't showing up under
newer versions of Mercurial. Hopefully this is all resolved in the latest.
--nate
On Jan 17, 2012, at 4:11 AM, Hans-Rudolf Hotz wrote:
Hi
I am in the process of upgrading all our Galaxy servers to the current changeset
("b258de1e6cea", Nov. 18) and I have noticed an inconsistency:
If I upgrade one of our old servers (which was on "720455407d1c", June 23) with
'hg pull -u -r b258de1e6cea' I get the following:
haruhotz@silicon: hg heads
changeset: 6297:b258de1e6cea
tag: tip
user:jeremy goecks
date:Thu Nov 17 15:08:49 2011 -0500
summary: Pack js scripts.
haruhotz@silicon:
If I create a new installation with
'hg clone http://www.bx.psu.edu/hg/galaxy galaxy-dist' I get the following:
haruhotz@silicon: hg heads
changeset: 6298:b258de1e6cea
tag: tip
user:jeremy goecks
date:Thu Nov 17 15:08:49 2011 -0500
summary: Pack js scripts.
changeset: 5832:70b8039a2eef
branch: feature/ws
parent: 5826:6206a27bd2ae
user:Nate Coraor
date:Wed Jul 27 10:48:03 2011 -0400
summary: Closed the feature/ws branch, changes from this branch were merged
in 5827:f3a1086fac91.
haruhotz@silicon:
I am not worry about the merged branched, I am just a little bit surprised
about the differences:
"6298:b258de1e6cea" versus "6297:b258de1e6cea"
Can anybody with more mercurial experience enlighten me what went wrong? and
even better tell me that there is nothing to worry about...
Regards, Hans
___
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
___
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
