Re: [galaxy-dev] galaxy api documentation
The documentation for the API is looking real nice...Great... I do have some trouble finding how I move a history item into a data library... Could you give me a few pointers which API call to make for that? We can assume I have the HDA indentifiers for those objects, just need to know the command for moving it into a folder of the data library? Thanks! Thon -Original Message- From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Dannon Baker Sent: Saturday, February 02, 2013 7:57 PM To: mark.r...@syngenta.com Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] galaxy api documentation Mark, There's documentation currently available here: https://galaxy-dist.readthedocs.org/en/latest/lib/galaxy.webapps.galaxy.api. html Is there anything else in particular you're looking for? -Dannon On Feb 2, 2013, at 9:15 PM, mark.r...@syngenta.com wrote: Hi All When is it anticipated that the documentation for the API will be available? Thanks Mark This message may contain confidential information. If you are not the designated recipient, please notify the sender immediately, and delete the original and any copies. Any use of the message by you is prohibited. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Error in database after trying the innocently named, but devastatingly evil test in manage_db.sh
Hi,
I had a small issue with the total usage of my galaxy system showing some
negative random number so I figured my database was a little “off”.
I snooped around for some database management tools and found “manage_db.sh”
which had an option innocently called “test”.
I ran it, without reading the little fine print that states: “Performs the
upgrade and downgrade option on the given database. This is not a real test and
may leave the database in a bad state. You should therefore better run the test
on a copy of your database.”
Well…It surely did leave my database in a bad state since now I get this when I
try to access the toolshed installed tools:
Error Traceback:
View as:http://srv151/admin_toolshed/browse_repositories Interactive |
http://srv151/admin_toolshed/browse_repositories Text |
http://srv151/admin_toolshed/browse_repositories XML
http://srv151/admin_toolshed/browse_repositories (full)
⇝ ProgrammingError: (ProgrammingError) relation
repository_repository_dependency_association does not exist LINE 2: FROM
repository_repository_dependency_association ^ 'SELECT
repository_repository_dependency_association.id AS
repository_repository_dependency_association_id,
repository_repository_dependency_association.create_time AS
repository_repository_dependency_association_create_time,
repository_repository_dependency_association.update_time AS
repository_repository_dependency_association_update_time,
repository_repository_dependency_association.tool_shed_repository_id AS
repository_repository_dependency_association_tool_shed_re_1,
repository_repository_dependency_association.repository_dependency_id AS
repository_repository_dependency_association_repository_d_2 \nFROM
repository_repository_dependency_association \nWHERE %(param_1)s =
repository_repository_dependency_association.tool_shed_repository_id'
{'param_1': 5}
URL: http://srv151/admin_toolshed/browse_repositories
http://srv151/admin_toolshed/browse_repositories
Is there ANY way I can rescue this fiasco? Suffice it to say I did not do it on
a copy and did not do a backup (yes, I learned my lesson), but maybe some SQL
skullduggery can help me?
Thanks
Thon
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] Error in database after trying the innocently named, but devastatingly evil test in manage_db.sh
Hi Dan,
I tried that but it failed ofcourse since the table it tries to drop from 109
- 108 does not exisit.
sh manage_db.sh downgrade 108
109 - 108...
0109_add_repository_dependency_tables DEBUG 2013-02-08 13:04:33,464 Dropping
repository_repository_dependency_association table failed: (ProgrammingError)
table repository_repository_dependency_association does not exist
'\nDROP TABLE repository_repository_dependency_association' {}
0109_add_repository_dependency_tables DEBUG 2013-02-08 13:04:33,464 Dropping
repository_repository_dependency_association table failed: (ProgrammingError)
table repository_repository_dependency_association does not exist
'\nDROP TABLE repository_repository_dependency_association' {}
0109_add_repository_dependency_tables DEBUG 2013-02-08 13:04:33,467 Dropping
repository_dependency table failed: (ProgrammingError) table
repository_dependency does not exist
'\nDROP TABLE repository_dependency' {}
0109_add_repository_dependency_tables DEBUG 2013-02-08 13:04:33,467 Dropping
repository_dependency table failed: (ProgrammingError) table
repository_dependency does not exist
'\nDROP TABLE repository_dependency' {}
done
(galaxy_env)[svcgalaxy@srv151 g]$ sh manage_db.sh version
109
I just went into the database and create the two dummy tables and tried it
again and it seem to work..
I downgraded and upgraded to 109…
Then I restarted Galaxy, reset the metatdata (which failed for 2 of my 13
tools) and then I see this in the manage toolshed:
Error Traceback:
View as:http://srv151/admin_toolshed/browse_repositories Interactive |
http://srv151/admin_toolshed/browse_repositories Text |
http://srv151/admin_toolshed/browse_repositories XML
http://srv151/admin_toolshed/browse_repositories (full)
⇝ ProgrammingError: (ProgrammingError) column
repository_repository_dependency_association.id does not exist LINE 1:
..._7f8dcc0e81d0_de8f CURSOR WITHOUT HOLD FOR SELECT repository... ^ 'SELECT
repository_repository_dependency_association.id AS
repository_repository_dependency_association_id,
repository_repository_dependency_association.create_time AS
repository_repository_dependency_association_create_time,
repository_repository_dependency_association.update_time AS
repository_repository_dependency_association_update_time,
repository_repository_dependency_association.tool_shed_repository_id AS
repository_repository_dependency_association_tool_shed_re_1,
repository_repository_dependency_association.repository_dependency_id AS
repository_repository_dependency_association_repository_d_2 \nFROM
repository_repository_dependency_association \nWHERE %(param_1)s =
repository_repository_dependency_association.tool_shed_repository_id'
{'param_1': 5}
Slightly different error, but an error nonetheless…
Any more ideas?
I am (somewhat) OK with having to reload the toolshed tools but I can’t even
install or manage any new toolsheds…
Thanks
Thon
From: Daniel Blankenberg [mailto:d...@bx.psu.edu]
Sent: Friday, February 08, 2013 12:32 PM
To: Thon de Boer
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Error in database after trying the innocently named,
but devastatingly evil test in manage_db.sh
Hi Thon,
The test option runs the upgrade and downgrade of the latest migration script.
I'm assuming that you had already run the latest migrations available and the
server was up and running OK (other than the dataset size issue) before trying
the test command.
On a COPY of your database, you can try this:
$ sh manage_db.sh version
XYZ
$ sh manage_db.sh downgrade [XYZ -1]
..
done
sh manage_db.sh upgrade XYZ
done
If it works, then you can try it on your real database. Of course, this will
destroy whatever data existed in the table/columns of the latest migration (but
it looks like you already lost them.)
Thanks for using Galaxy,
Dan
On Feb 8, 2013, at 2:32 PM, Thon de Boer wrote:
Hi,
I had a small issue with the total usage of my galaxy system showing some
negative random number so I figured my database was a little “off”.
I snooped around for some database management tools and found “manage_db.sh”
which had an option innocently called “test”.
I ran it, without reading the little fine print that states: “Performs the
upgrade and downgrade option on the given database. This is not a real test and
may leave the database in a bad state. You should therefore better run the test
on a copy of your database.”
Well…It surely did leave my database in a bad state since now I get this when I
try to access the toolshed installed tools:
Error Traceback:
View as:http://srv151/admin_toolshed/browse_repositories Interactive |
http://srv151/admin_toolshed/browse_repositories Text |
http://srv151/admin_toolshed/browse_repositories XML
http://srv151/admin_toolshed/browse_repositories (full)
⇝ ProgrammingError: (ProgrammingError) relation
Re: [galaxy-dev] Error in database after trying the innocently named, but devastatingly evil test in manage_db.sh
OK..I tried it a few more times and this time with different results! HA! I
knew I wasn’t insane…As long as you try anything often enough in the same
manner, you WILL get a different outcome…Yea uncertainty principle!
I also had to resort to downloading the whole complete galaxy-dist from 11-Jan
since the current development version of galaxy-dist is broken, but I’m back in
business and first order is to create a backup plan
Thon
From: galaxy-dev-boun...@lists.bx.psu.edu
[mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Thon de Boer
Sent: Friday, February 08, 2013 1:23 PM
To: 'Daniel Blankenberg'
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Error in database after trying the innocently named,
but devastatingly evil test in manage_db.sh
Hi Dan,
I tried that but it failed ofcourse since the table it tries to drop from 109
- 108 does not exisit.
sh manage_db.sh downgrade 108
109 - 108...
0109_add_repository_dependency_tables DEBUG 2013-02-08 13:04:33,464 Dropping
repository_repository_dependency_association table failed: (ProgrammingError)
table repository_repository_dependency_association does not exist
'\nDROP TABLE repository_repository_dependency_association' {}
0109_add_repository_dependency_tables DEBUG 2013-02-08 13:04:33,464 Dropping
repository_repository_dependency_association table failed: (ProgrammingError)
table repository_repository_dependency_association does not exist
'\nDROP TABLE repository_repository_dependency_association' {}
0109_add_repository_dependency_tables DEBUG 2013-02-08 13:04:33,467 Dropping
repository_dependency table failed: (ProgrammingError) table
repository_dependency does not exist
'\nDROP TABLE repository_dependency' {}
0109_add_repository_dependency_tables DEBUG 2013-02-08 13:04:33,467 Dropping
repository_dependency table failed: (ProgrammingError) table
repository_dependency does not exist
'\nDROP TABLE repository_dependency' {}
done
(galaxy_env)[svcgalaxy@srv151 g]$ sh manage_db.sh version
109
I just went into the database and create the two dummy tables and tried it
again and it seem to work..
I downgraded and upgraded to 109…
Then I restarted Galaxy, reset the metatdata (which failed for 2 of my 13
tools) and then I see this in the manage toolshed:
Error Traceback:
View as:http://srv151/admin_toolshed/browse_repositories Interactive |
http://srv151/admin_toolshed/browse_repositories Text |
http://srv151/admin_toolshed/browse_repositories XML
http://srv151/admin_toolshed/browse_repositories (full)
⇝ ProgrammingError: (ProgrammingError) column
repository_repository_dependency_association.id does not exist LINE 1:
..._7f8dcc0e81d0_de8f CURSOR WITHOUT HOLD FOR SELECT repository... ^ 'SELECT
repository_repository_dependency_association.id AS
repository_repository_dependency_association_id,
repository_repository_dependency_association.create_time AS
repository_repository_dependency_association_create_time,
repository_repository_dependency_association.update_time AS
repository_repository_dependency_association_update_time,
repository_repository_dependency_association.tool_shed_repository_id AS
repository_repository_dependency_association_tool_shed_re_1,
repository_repository_dependency_association.repository_dependency_id AS
repository_repository_dependency_association_repository_d_2 \nFROM
repository_repository_dependency_association \nWHERE %(param_1)s =
repository_repository_dependency_association.tool_shed_repository_id'
{'param_1': 5}
Slightly different error, but an error nonetheless…
Any more ideas?
I am (somewhat) OK with having to reload the toolshed tools but I can’t even
install or manage any new toolsheds…
Thanks
Thon
From: Daniel Blankenberg [mailto:d...@bx.psu.edu]
Sent: Friday, February 08, 2013 12:32 PM
To: Thon de Boer
Cc: galaxy-dev@lists.bx.psu.edu mailto:galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Error in database after trying the innocently named,
but devastatingly evil test in manage_db.sh
Hi Thon,
The test option runs the upgrade and downgrade of the latest migration script.
I'm assuming that you had already run the latest migrations available and the
server was up and running OK (other than the dataset size issue) before trying
the test command.
On a COPY of your database, you can try this:
$ sh manage_db.sh version
XYZ
$ sh manage_db.sh downgrade [XYZ -1]
..
done
sh manage_db.sh upgrade XYZ
done
If it works, then you can try it on your real database. Of course, this will
destroy whatever data existed in the table/columns of the latest migration (but
it looks like you already lost them.)
Thanks for using Galaxy,
Dan
On Feb 8, 2013, at 2:32 PM, Thon de Boer wrote:
Hi,
I had a small issue with the total usage of my galaxy system showing some
negative random number so I figured my database was a little “off”.
I snooped around
[galaxy-dev] Search for toolname from toolshed in workflow editor does not work?
Hi, It seems that the workflow editor does not know how to search for tools that are in the toolshed. The regular toolsearch in the main Analysis window has no problem finding reorder Sam/BAM in picard if I search there, but in the workflow editor it seems to only be able to search for tools that are installed outside the toolshed. Do I need to turn this on somewhere or is this just a bug? Thanks Thon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] History does not automatically update anymore
I do now actually get an error message in my browser saying Error getting history updates from the server. Internal Server Error Any ideas? I suspect it is my proxy server settings but not sure. Thon From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Anthonius deBoer Sent: Thursday, February 14, 2013 2:31 PM To: Galaxy-dev Galaxy-dev Subject: [galaxy-dev] History does not automatically update anymore Hi, I have noticed that in the latest version of galaxy-dist, my history does not automatically update anymore. I have to press the refresh circle to have it update... I did change my proxy setup a little, so it would no longer require API calls to be authenticated, while access through the frontend would still require that so it may have something to do with that, but just wanted to check if others see this issue... I will send out my proxy settings in a later email, since it may help people that are struggling with the same... Thon image001.png___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] History does not automatically update anymore
Aaanndd ofcourse it was due to the proxy settings. I noticed that the history refresh is done through a call of the API and since I had re-routed that to a different web server than the main server, it broke. But once I added this rewrite rule: ReWriteRule ^(/api/histories/.*) http://localhost:8080$1 [P,L] Before the other /api/ one, the refresh of the history worked again. It seems I need to spend a little more time on this whole proxy setting thing. Thon From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Thon de Boer Sent: Thursday, February 14, 2013 10:58 PM To: 'Galaxy-dev Galaxy-dev' Subject: Re: [galaxy-dev] History does not automatically update anymore I do now actually get an error message in my browser saying Error getting history updates from the server. Internal Server Error Any ideas? I suspect it is my proxy server settings but not sure. Thon From: galaxy-dev-boun...@lists.bx.psu.edu mailto:galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Anthonius deBoer Sent: Thursday, February 14, 2013 2:31 PM To: Galaxy-dev Galaxy-dev Subject: [galaxy-dev] History does not automatically update anymore Hi, I have noticed that in the latest version of galaxy-dist, my history does not automatically update anymore. I have to press the refresh circle to have it update... I did change my proxy setup a little, so it would no longer require API calls to be authenticated, while access through the frontend would still require that so it may have something to do with that, but just wanted to check if others see this issue... I will send out my proxy settings in a later email, since it may help people that are struggling with the same... Thon image001.png___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Submitting jobs as a real user without using chown, please
Hi, I am trying to setup my galaxy system to allow jobs to be submitted as the real user, since people want to keep an eye on their job on the cluster sometimes and they have no ideas which ones are theirs. I tried the approach on the wiki here: http://wiki.galaxyproject.org/Admin/Config/Performance/Cluster?highlight=%28 submit%29%7C%28jobs%29%7C%28as%29%7C%28user%29#Submitting_Jobs_as_the_Real_U ser but unfortunately, the CHOWN command is not allowed, not even as a sudo user. Probably has to do with the fact that we run our cluster from an isilon system, which I assume is pretty typical. The job was actually successfully submitted as the intended user, so that part works, but if we can just get it to work without having to rely on chown that would be awesome. Can someone point me in the right direction? Here's the error. galaxy.jobs.runners.local DEBUG 2013-02-19 19:35:31,524 execution of external set_meta for job 148 finished galaxy.jobs DEBUG 2013-02-19 19:35:31,576 (148) Changing ownership of working directory with: /usr/bin/sudo -E scripts/external_chown_script.py /mnt/ngs/analysis/svcgalaxy/galaxy-test/database/job_working_directory/000/1 48 svcgalaxy 1 galaxy.jobs ERROR 2013-02-19 19:35:31,653 (148) Failed to change ownership of /mnt/ngs/analysis/svcgalaxy/galaxy-test/database/job_working_directory/000/1 48, failing Traceback (most recent call last): File /mnt/ngs/analysis/svcgalaxy/galaxy-test/lib/galaxy/jobs/__init__.py, line 343, in finish self.reclaim_ownership() File /mnt/ngs/analysis/svcgalaxy/galaxy-test/lib/galaxy/jobs/__init__.py, line 916, in reclaim_ownership self._change_ownership( self.galaxy_system_pwent[0], str( self.galaxy_system_pwent[3] ) ) File /mnt/ngs/analysis/svcgalaxy/galaxy-test/lib/galaxy/jobs/__init__.py, line 902, in _change_ownership assert p.returncode == 0 AssertionError galaxy.jobs DEBUG 2013-02-19 19:35:31,722 fail(): Moved /mnt/ngs/analysis/svcgalaxy/galaxy-test/database/job_working_directory/000/1 48/galaxy_dataset_332.dat to /mnt/ngs/analysis/svcgalaxy/galaxy-test/database/files/000/dataset_332.dat galaxy.datatypes.metadata DEBUG 2013-02-19 19:35:31,924 Cleaning up external metadata files Thanks Thon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Submitting jobs as a real user without using chown, please
OK...I think I can make this work, since it is not that difficult to make a directory world writeable...Or at least group writeable since all the users that will be able to run galaxy are in the same group as svcgalaxy that runs galaxy...I'll look at those scripts and see what they need to do... Thanks Thon -Original Message- From: Nate Coraor [mailto:n...@bx.psu.edu] Sent: Thursday, February 21, 2013 5:56 AM To: Anthonius deBoer Cc: 'Galaxy-dev Galaxy-dev' Subject: Re: [galaxy-dev] Submitting jobs as a real user without using chown, please On Feb 20, 2013, at 2:37 PM, Anthonius deBoer wrote: Ah...Found what root squashing is and yes, that is turned on our isilon system... So out of luck I take it? We need to chown? we cannot fake the submission name in another way ;) Galaxy must have a way to make the job working directory writable for the user that the job is running as. If this means logging in to another system via ssh that *does* have the ability to change ownership, then you can do that. The method is completely customizable because you can set the chown script to anything that works for you. However, some means of doing this that is appropriate for your environment has to exist for Galaxy to perform it. That said, you might be able to get away with having the script make the working directory world-writeable instead of owned by the real user. --nate Thanks Thon On Feb 20, 2013, at 10:32 AM, Anthonius deBoer thondeb...@me.com wrote: I cannot run chown even as a sudo command...Same error... What is root squashing? I am reading on the internet that it is very common not to allow users to change the ownership of files... Thon On Feb 20, 2013, at 05:52 AM, Nate Coraor n...@bx.psu.edu wrote: On Feb 19, 2013, at 11:02 PM, Thon de Boer wrote: Hi, I am trying to setup my galaxy system to allow jobs to be submitted as the real user, since people want to keep an eye on their job on the cluster sometimes and they have no ideas which ones are theirs. I tried the approach on the wiki here: http://wiki.galaxyproject.org/Admin/Config/Performance/Cluster?hig hlight=%28submit%29%7C%28jobs%29%7C%28as%29%7C%28user%29#Submittin g_Jobs_as_the_Real_User but unfortunately, the CHOWN command is not allowed, not even as a sudo user. Probably has to do with the fact that we run our cluster from an isilon system, which I assume is pretty typical. The job was actually successfully submitted as the intended user, so that part works, but if we can just get it to work without having to rely on chown that would be awesome. Can someone point me in the right direction? Hi Thon, If you run the command from the command line, what results do you get? /usr/bin/sudo -E scripts/external_chown_script.py /mnt/ngs/analysis/svcgalaxy/galaxy-test/database/job_working_directo ry/000/148 svcgalaxy 1 Note that external_chown_script.py can be modified as necessary to allow you to change ownership in whatever way is appropriate for your site. Since it should just be an NFS mount, as long as root squashing is not enabled and your svcgalaxy user has sudo permission to run this script, it should succeed. --nate Here's the error. galaxy.jobs.runners.local DEBUG 2013-02-19 19:35:31,524 execution of external set_meta for job 148 finished galaxy.jobs DEBUG 2013-02-19 19:35:31,576 (148) Changing ownership of working directory with: /usr/bin/sudo -E scripts/external_chown_script.py /mnt/ngs/analysis/svcgalaxy/galaxy-test/database/job_working_directory/000/1 48 svcgalaxy 1 galaxy.jobs ERROR 2013-02-19 19:35:31,653 (148) Failed to change ownership of /mnt/ngs/analysis/svcgalaxy/galaxy-test/database/job_working_directory/000/1 48, failing Traceback (most recent call last): File /mnt/ngs/analysis/svcgalaxy/galaxy-test/lib/galaxy/jobs/__init__. py, line 343, in finish self.reclaim_ownership() File /mnt/ngs/analysis/svcgalaxy/galaxy-test/lib/galaxy/jobs/__init__. py, line 916, in reclaim_ownership self._change_ownership( self.galaxy_system_pwent[0], str( self.galaxy_system_pwent[3] ) ) File /mnt/ngs/analysis/svcgalaxy/galaxy-test/lib/galaxy/jobs/__init__. py, line 902, in _change_ownership assert p.returncode == 0 AssertionError galaxy.jobs DEBUG 2013-02-19 19:35:31,722 fail(): Moved /mnt/ngs/analysis/svcgalaxy/galaxy-test/database/job_working_direc tory/000/148/galaxy_dataset_332.dat to /mnt/ngs/analysis/svcgalaxy/galaxy-test/database/files/000/dataset _332.dat galaxy.datatypes.metadata DEBUG 2013-02-19 19:35:31,924 Cleaning up external metadata files Thanks Thon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu
Re: [galaxy-dev] Submitting jobs as a real user without using chown, please
OK..I found a way to allow users to submit as a real users without using
CHOWN...
I did this by:
1) Tweaking the script external_chown_script.py by simply commenting out the
chown statements (see below)
2) Making sure the users are in the same group
3) Setting the umask to 0002 to make all files svcgalaxy produces writeable
by the same group (Dangerous, but I trust my users not to mess with the
files directly)
4) Add the script scripts/external_chown_script.py to the list of
allowable SUDO scripts, since that script will also be executed by sudo
(Sudo section now looks like this)
svcgalaxy ALL=(ALL) ALL
svcgalaxy ALL = (root) NOPASSWD: SETENV:
/mnt/ngs/analysis/svcgalaxy/galaxy-test/scripts/drmaa_external_runner.py
svcgalaxy ALL = (root) NOPASSWD: SETENV:
/mnt/ngs/analysis/svcgalaxy/galaxy-test/scripts/drmaa_external_killer.py
svcgalaxy ALL = (root) NOPASSWD: SETENV:
/mnt/ngs/analysis/svcgalaxy/galaxy-test/scripts/external_chown_script.py
5 (OPTIONAL) I even left the default setting for the
outputs_to_working_directory set to FALSE since I hate to lose the option to
be able to see the progress of a running job by clicking the eye icon to see
the log file fill up
Here's the tweak to external_chown_script.py...
def main():
path, galaxy_user_name, gid = validate_paramters()
#os.system('chown -Rh %s %s' %(galaxy_user_name, path))
#os.system('chgrp -Rh %s %s' %(gid, path))
I could have dispensed with the script completely ofcourse and simply have
main return 0, but this way I remember to someday find a better
solution.
But for now I am saved and have a nice system that informs the users the
progress of their own jobs with qstat...
This may not work for everyone, but for a small group of trustworthy users,
this is a passable way to allow jobs to be submitted as real users..
Thon
-Original Message-
From: galaxy-dev-boun...@lists.bx.psu.edu
[mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Thon de Boer
Sent: Thursday, February 21, 2013 11:08 PM
To: 'Nate Coraor'
Cc: 'Galaxy-dev Galaxy-dev'
Subject: Re: [galaxy-dev] Submitting jobs as a real user without using
chown, please
OK...I think I can make this work, since it is not that difficult to make a
directory world writeable...Or at least group writeable since all the users
that will be able to run galaxy are in the same group as svcgalaxy that runs
galaxy...I'll look at those scripts and see what they need to do...
Thanks
Thon
-Original Message-
From: Nate Coraor [mailto:n...@bx.psu.edu]
Sent: Thursday, February 21, 2013 5:56 AM
To: Anthonius deBoer
Cc: 'Galaxy-dev Galaxy-dev'
Subject: Re: [galaxy-dev] Submitting jobs as a real user without using
chown, please
On Feb 20, 2013, at 2:37 PM, Anthonius deBoer wrote:
Ah...Found what root squashing is and yes, that is turned on our
isilon
system...
So out of luck I take it? We need to chown? we cannot fake the
submission name in another way ;)
Galaxy must have a way to make the job working directory writable for the
user that the job is running as. If this means logging in to another system
via ssh that *does* have the ability to change ownership, then you can do
that. The method is completely customizable because you can set the chown
script to anything that works for you.
However, some means of doing this that is appropriate for your environment
has to exist for Galaxy to perform it. That said, you might be able to get
away with having the script make the working directory world-writeable
instead of owned by the real user.
--nate
Thanks
Thon
On Feb 20, 2013, at 10:32 AM, Anthonius deBoer thondeb...@me.com wrote:
I cannot run chown even as a sudo command...Same error...
What is root squashing? I am reading on the internet that it is very
common not to allow users to change the ownership of files...
Thon
On Feb 20, 2013, at 05:52 AM, Nate Coraor n...@bx.psu.edu wrote:
On Feb 19, 2013, at 11:02 PM, Thon de Boer wrote:
Hi,
I am trying to setup my galaxy system to allow jobs to be
submitted as the real user, since people want to keep an eye on
their job on the cluster sometimes and they have no ideas which
ones are theirs.
I tried the approach on the wiki here:
http://wiki.galaxyproject.org/Admin/Config/Performance/Cluster?hig
hlight=%28submit%29%7C%28jobs%29%7C%28as%29%7C%28user%29#Submittin
g_Jobs_as_the_Real_User
but unfortunately, the CHOWN command is not allowed, not even as a
sudo user. Probably has to do with the fact that we run our
cluster from an isilon system, which I assume is pretty typical.
The job was actually successfully submitted as the intended user,
so that part works, but if we can just get it to work without
having to rely on chown that would be awesome.
Can someone point me in the right direction?
Hi Thon,
If you run the command from the command line, what results do you get?
/usr/bin/sudo -E scripts
Re: [galaxy-dev] Is there a way through the API to know when a file has completed the upload process?
It seems that there are not a lot of people using/know about the API since
all my questions about the API here go unanswered, but I am happy to report
that I don't actually need to know if the upload has finished, since it
seems you can start a workflow with a dataset that is still being
uploaded.You can't do this in the UI but it seems you CAN do it through the
API, so that is a relief..
Now if only the API allowed us to pass parameters to a workflow in the same
way we can do this in the API, I am set and I can use Galaxy as a workflow
engine through the API..
Regards,
Thon
Thon deBoer Ph.D., Bioinformatics Guru
California, USA |p: +1 (650) 799-6839 |m: thondeb...@me.com
mailto:thondeb...@me.com
From: galaxy-dev-boun...@lists.bx.psu.edu
[mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Anthonius deBoer
Sent: Tuesday, May 21, 2013 6:24 PM
To: Galaxy-dev Galaxy-dev
Subject: [galaxy-dev] Is there a way through the API to know when a file has
completed the upload process?
Hi,
I am trying to use the API to upload a file and since the upload can take a
long time (see my previous post) I need to check if the upload has completed
so I know I can start a workflow (or can I start a workflow even when the
file is still in the queued state?)...
I have successfully used the examples from this page:
http://galaxy-dist.readthedocs.org/en/latest/lib/galaxy.webapps.galaxy.api.h
tml
and can easily upload a file, but I cannot seem to learn the state of the
file...
The upload process I used, returned a bunch of attributes, such as:
{'ldda_id': '049d723a33542847', 'misc_blurb': None, 'name':
'7_101_1_ACTTGA_R1.fastq', 'data_type': 'fastq', 'file_name':
'/mnt/ngs/analysis/tdeboer/PROJECTS/STAGING/SEQUENCER07/130207_SN7001380_010
1_AD1H7GACXX/fastq/7_101_1_ACTTGA_R1.fastq', 'uploaded_by':
'tdeb...@genomichealth.com', 'metadata_sequences': None, 'template_data':
{}, 'genome_build': 'hg19', 'model_class': 'LibraryDataset', 'misc_info':
None, 'file_size': 6410080411, 'metadata_data_lines': None, 'message': '',
'id': '049d723a33542847', 'date_uploaded': '2013-05-22T00:50:50.474318',
'metadata_dbkey': 'hg19'}
But none of those tell me if the upload has succeeded or not...
Is there a way to retrieve the state of an uploaded file once I have
uploaded a file through the API?
thanks
Thon
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] shall us specify which tool run in cluster?
HI Shenwiyn, The definition of regularjobs etc. is there to allow each job to be run under different environment on the cluster. I am actually not using most of those definitions, except for the BWA tool, which I want to run using 4 slots on our cluster so I use the destination multicorejobs4. The nativeSpecification are the options that you can give as if you were to use the QSUB command directly to submit jobs to the cluster. -V -q short.q -pe smp 1 Is what I normally use for the qsub command for a job that is fast, for instance.. You don't need to specify the destination for each job, since the tool section has a default, which is regularJobs in my case. So only if you want to do something other than submit a regular job (which only takes one slot) do you need to define something else, like I did for BWA. Hope that helps Regards, Thon Thon deBoer Ph.D., Bioinformatics Guru California, USA |p: +1 (650) 799-6839 |m: mailto:thondeb...@me.com thondeb...@me.com From: shenwiyn [mailto:shenw...@gmail.com] Sent: Tuesday, July 30, 2013 11:52 PM To: Thon Deboer Cc: galaxy-dev@lists.bx.psu.edu Subject: shall us specify which tool run in cluster? Hi Thon Deboer , I am a newer in Galaxy.I installed my Galaxy with Torque2.5.0 ,and Galaxy uses the pbs_modoule to interface with TORQUE.But I have some question of the job_conf.xml : 1.)In your job_conf.xml ,you use regularjobs,longjobs,shortjobs...to run different jobs ,how our Galaxy know which tool belongs to regularjobs or longjobs.And what is the meaning of nativeSpecification? 2.)Shall us use toolscollection of tool id=bwa_wrapper destination=multicorejobs4/to specify bwa ?Does it mean the bwa belong to multicorejobs4,and run in cluster? 3.)Does every tool need us to specify which job it belong to? I saw http://wiki.galaxyproject.org/Admin/Config/Jobs about this,but I am not sure above.Could you help me please? _ shenwiyn From: Thon Deboer mailto:thondeb...@me.com Date: 2013-07-18 14:31 To: galaxy-dev mailto:galaxy-dev@lists.bx.psu.edu Subject: [galaxy-dev] Jobs remain in queue until restart Hi, I have noticed that from time to time, the job queue seems to be stuck and can only be unstuck by restarting galaxy. The jobs seem to be in the queue state and the python job handler processes are hardly ticking over and the cluster is empty. When I restart, the startup procedure realizes all jobs are in the a new state and it then assigns a jobhandler after which the jobs start fine.. Any ideas? Torque Thon P.S I am using the june version of galaxy and I DO set limits on my users in job_conf.xml as so: (Maybe it is related? Before it went into dormant mode, this user had started lots of jobs and may have hit the limit, but I assumed this limit was the number of running jobs at one time, right?) ?xml version=1.0? job_conf plugins workers=4 !-- workers is the number of threads for the runner's work queue. The default from plugins is used if not defined for a plugin. -- plugin id=local type=runner load=galaxy.jobs.runners.local:LocalJobRunner workers=2/ plugin id=drmaa type=runner load=galaxy.jobs.runners.drmaa:DRMAAJobRunner workers=8/ plugin id=cli type=runner load=galaxy.jobs.runners.cli:ShellJobRunner workers=2/ /plugins handlers default=handlers !-- Additional job handlers - the id should match the name of a [server:id] in universe_wsgi.ini. -- handler id=handler0 tags=handlers/ handler id=handler1 tags=handlers/ handler id=handler2 tags=handlers/ handler id=handler3 tags=handlers/ !-- handler id=handler10 tags=handlers/ handler id=handler11 tags=handlers/ handler id=handler12 tags=handlers/ handler id=handler13 tags=handlers/ -- /handlers destinations default=regularjobs !-- Destinations define details about remote resources and how jobs should be executed on those remote resources. -- destination id=local runner=local/ destination id=regularjobs runner=drmaa tags=cluster !-- These are the parameters for qsub, such as queue etc. -- param id=nativeSpecification-V -q long.q -pe smp 1/param /destination destination id=longjobs runner=drmaa tags=cluster,long_jobs !-- These are the parameters for qsub, such as queue etc. -- param id=nativeSpecification-V -q long.q -pe smp 1/param /destination destination id=shortjobs runner=drmaa tags=cluster,short_jobs !-- These are the parameters for qsub, such as queue etc. -- param id=nativeSpecification-V -q short.q -pe smp 1/param /destination destination id=multicorejobs4 runner=drmaa tags=cluster,multicore_jobs !-- These are the parameters
Re: [galaxy-dev] Jobs remain in queue until restart
I did some more investigation of this issue I do notice that my 4 core, 8 slot VM machine has a load of 32, with only my 4 handler processes running (Plus my web server), but not even getting more than 10% of the CPU each. There seems to be some process in my handlers that takes an incredible amount of resources, even though TOP is not showing that (Show below) Has anyone have any idea how to figure out where the bottleneck is? Is there a way to turn on more detailed logging perhaps to see what each process is doing? My IT guy suggested there may be some context Switching going on due to the many threads that are running (I use a threadpool of 7 for each server), but not sure how to address that issue. Anyone? top - 10:00:53 up 37 days, 19:29, 8 users, load average: 32.10, 32.10, 32.09 Tasks: 181 total, 1 running, 180 sleeping, 0 stopped, 0 zombie Cpu(s): 4.8%us, 2.5%sy, 0.0%ni, 92.5%id, 0.0%wa, 0.0%hi, 0.2%si, 0.0%st Mem: 16334504k total, 16164084k used, 170420k free, 127720k buffers Swap: 4194296k total,15228k used, 4179068k free, 2460252k cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 7190 svcgalax 20 0 2721m 284m 5976 S 9.9 1.8 142:53.84 python ./scripts/paster.py serve universe_wsgi.ini --server-name=handler3 --pid-file=handler3.pid --log-file=handler3.log --daemon 7183 svcgalax 20 0 2720m 286m 5984 S 6.4 1.8 135:52.63 python ./scripts/paster.py serve universe_wsgi.ini --server-name=handler2 --pid-file=handler2.pid --log-file=handler2.log --daemon 7175 svcgalax 20 0 2720m 287m 5976 S 5.6 1.8 117:59.40 python ./scripts/paster.py serve universe_wsgi.ini --server-name=handler1 --pid-file=handler1.pid --log-file=handler1.log --daemon 7166 svcgalax 20 0 3442m 2.7g 4884 S 4.6 17.5 74:31.66 python ./scripts/paster.py serve universe_wsgi.ini --server-name=web0 --pid-file=web0.pid --log-file=web0.log --daemon 7172 svcgalax 20 0 2720m 294m 5984 S 4.0 1.8 133:17.19 python ./scripts/paster.py serve universe_wsgi.ini --server-name=handler0 --pid-file=handler0.pid --log-file=handler0.log --daemon 1564 root 20 0 291m 13m 7552 S 0.3 0.1 1:49.65 /usr/sbin/httpd 7890 svcgalax 20 0 17216 1456 1036 S 0.3 0.0 2:15.73 top 10682 apache20 0 297m 11m 3516 S 0.3 0.1 0:02.23 /usr/sbin/httpd 11224 apache20 0 295m 11m 3236 S 0.3 0.1 0:00.29 /usr/sbin/httpd 11263 svcgalax 20 0 17248 1460 1036 R 0.3 0.0 0:00.06 top 1 root 20 0 21320 1040 784 S 0.0 0.0 0:00.95 /sbin/init 2 root 20 0 000 S 0.0 0.0 0:00.01 [kthreadd] 3 root RT 0 000 S 0.0 0.0 0:06.35 [migration/0] Regards, Thon Thon deBoer Ph.D., Bioinformatics Guru California, USA |p: +1 (650) 799-6839 |m: mailto:thondeb...@me.com thondeb...@me.com From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Thon Deboer Sent: Wednesday, July 17, 2013 11:31 PM To: galaxy-dev@lists.bx.psu.edu Subject: [galaxy-dev] Jobs remain in queue until restart Hi, I have noticed that from time to time, the job queue seems to be stuck and can only be unstuck by restarting galaxy. The jobs seem to be in the queue state and the python job handler processes are hardly ticking over and the cluster is empty. When I restart, the startup procedure realizes all jobs are in the a new state and it then assigns a jobhandler after which the jobs start fine.. Any ideas? Thon P.S I am using the june version of galaxy and I DO set limits on my users in job_conf.xml as so: (Maybe it is related? Before it went into dormant mode, this user had started lots of jobs and may have hit the limit, but I assumed this limit was the number of running jobs at one time, right?) ?xml version=1.0? job_conf plugins workers=4 !-- workers is the number of threads for the runner's work queue. The default from plugins is used if not defined for a plugin. -- plugin id=local type=runner load=galaxy.jobs.runners.local:LocalJobRunner workers=2/ plugin id=drmaa type=runner load=galaxy.jobs.runners.drmaa:DRMAAJobRunner workers=8/ plugin id=cli type=runner load=galaxy.jobs.runners.cli:ShellJobRunner workers=2/ /plugins handlers default=handlers !-- Additional job handlers - the id should match the name of a [server:id] in universe_wsgi.ini. -- handler id=handler0 tags=handlers/ handler id=handler1 tags=handlers/ handler id=handler2 tags=handlers/ handler id=handler3 tags=handlers/ !-- handler id=handler10 tags=handlers/ handler id=handler11 tags=handlers/ handler id=handler12 tags=handlers/ handler id=handler13 tags=handlers/ -- /handlers destinations default=regularjobs !-- Destinations define details
Re: [galaxy-dev] Why does my trackster complain about not being able to display BED files?
I now have a different problem with a different (production) version of galaxy trickster. I always get this message Could not load chroms for this dbkey I have checked that I have files with the [key].len in tool-data/shared/ucsc/chrom and I have twobit files etc. The weird thing is, that the Create visualization shows the genome keys I have but this window shows nothing for the dbkey it things it should be getting. So somewhere it looses the key. Any ideas? (I'm running the latest update) Regards, Thon Thon deBoer Ph.D., Bioinformatics Guru California, USA |p: +1 (650) 799-6839 |m: mailto:thondeb...@me.com thondeb...@me.com Public profile on http://www.linkedin.com/pub/thon-de-boer/1/1ba/a5b/ LinkedIn From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Thon de Boer Sent: Friday, August 23, 2013 11:02 AM To: 'Jeremy Goecks' Cc: 'Galaxy-dev Galaxy-dev' Subject: Re: [galaxy-dev] Why does my trackster complain about not being able to display BED files? Thanks Jeremy, I did what you suggested and now it no longer complains! Not sure if I had edited datatypes_conf.xml but just copying the sample worked and let's just hope I did not break any of my changes Regards, Thon Thon deBoer Ph.D., Bioinformatics Guru California, USA |p: +1 (650) 799-6839 |m: mailto:thondeb...@me.com thondeb...@me.com Public profile on http://www.linkedin.com/pub/thon-de-boer/1/1ba/a5b/ LinkedIn From: Jeremy Goecks [mailto:jeremy.goe...@emory.edu] Sent: Thursday, August 22, 2013 11:11 AM To: Anthonius deBoer Cc: sam guerler; Galaxy-dev Galaxy-dev Subject: Re: [galaxy-dev] Why does my trackster complain about not being able to display BED files? Am I missing a converter? Yes, you are missing the bed_to_bigwig converter. If you haven't made changes to your datatypes_conf.xml file, you can just copy datatypes_conf.xml.sample to datatypes_conf.xml to get the needed converters. If you've made changes to datatypes_conf.xml, you'll need to manually add the needed converters. We recently transitioned all the *_to_summary_tree converters to *_to_bigwig, so you'll want to remove the summary_tree converters and replace them with the bigwig converters. And why would a BED file even needed to be converted? Converter = indexer for visualizations. Datasets are indexed so that (a) aggregate (coverage) data is readily available when viewing a large region and (b) finding features in a small region (when zoomed in) is fast. J. image001.png___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Why does my trackster complain about not being able to display BED files?
I tracked it down to a problem with the API proxy configuration. I had the API calls diverted to a different server since I wanted to ensure that API calls would be handled by a different server, but that does not seem to work correctly for the API calls used in trickster. Issue resolved. Regards, Thon Thon deBoer Ph.D., Bioinformatics Guru California, USA |p: +1 (650) 799-6839 |m: mailto:thondeb...@me.com thondeb...@me.com Public profile on http://www.linkedin.com/pub/thon-de-boer/1/1ba/a5b/ LinkedIn From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Thon de Boer Sent: Friday, August 23, 2013 5:12 PM To: 'Jeremy Goecks' Cc: 'Galaxy-dev Galaxy-dev' Subject: Re: [galaxy-dev] Why does my trackster complain about not being able to display BED files? I now have a different problem with a different (production) version of galaxy trickster. I always get this message Could not load chroms for this dbkey I have checked that I have files with the [key].len in tool-data/shared/ucsc/chrom and I have twobit files etc. The weird thing is, that the Create visualization shows the genome keys I have but this window shows nothing for the dbkey it things it should be getting. So somewhere it looses the key. Any ideas? (I'm running the latest update) Regards, Thon Thon deBoer Ph.D., Bioinformatics Guru California, USA |p: +1 (650) 799-6839 |m: mailto:thondeb...@me.com thondeb...@me.com Public profile on http://www.linkedin.com/pub/thon-de-boer/1/1ba/a5b/ LinkedIn From: galaxy-dev-boun...@lists.bx.psu.edu mailto:galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Thon de Boer Sent: Friday, August 23, 2013 11:02 AM To: 'Jeremy Goecks' Cc: 'Galaxy-dev Galaxy-dev' Subject: Re: [galaxy-dev] Why does my trackster complain about not being able to display BED files? Thanks Jeremy, I did what you suggested and now it no longer complains! Not sure if I had edited datatypes_conf.xml but just copying the sample worked and let's just hope I did not break any of my changes Regards, Thon Thon deBoer Ph.D., Bioinformatics Guru California, USA |p: +1 (650) 799-6839 |m: mailto:thondeb...@me.com thondeb...@me.com Public profile on http://www.linkedin.com/pub/thon-de-boer/1/1ba/a5b/ LinkedIn From: Jeremy Goecks [mailto:jeremy.goe...@emory.edu] Sent: Thursday, August 22, 2013 11:11 AM To: Anthonius deBoer Cc: sam guerler; Galaxy-dev Galaxy-dev Subject: Re: [galaxy-dev] Why does my trackster complain about not being able to display BED files? Am I missing a converter? Yes, you are missing the bed_to_bigwig converter. If you haven't made changes to your datatypes_conf.xml file, you can just copy datatypes_conf.xml.sample to datatypes_conf.xml to get the needed converters. If you've made changes to datatypes_conf.xml, you'll need to manually add the needed converters. We recently transitioned all the *_to_summary_tree converters to *_to_bigwig, so you'll want to remove the summary_tree converters and replace them with the bigwig converters. And why would a BED file even needed to be converted? Converter = indexer for visualizations. Datasets are indexed so that (a) aggregate (coverage) data is readily available when viewing a large region and (b) finding features in a small region (when zoomed in) is fast. J. image001.png___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Why does my trackster complain about not being able to display BED files?
Ofcourse I spoke too soon.
I get a problem in that the conversion of my dataset reports that it cannot
find the chromosome length files, even though trickster itself has no
problem showing it.
I looked at the problem script and it is showing below
It seems that the script is passed the path to the chromosome files like
this tool-data/shared/ucsc/chrom/hg19-decoy.len
This will clearly never work since this file is not relative to the working
directory..
SO somewhere a script forgets to add the ${GALAXY_DATA_DIR} or whatever that
parameter is...
I could probably hardcode the location of the tool-data directory, but I
don't think that should be the solution.
Why do I only have this issue? SHOULD I have hardcoded the location of
tool-data?
The universe_wgi.ini.sample file does not hardcode this.
Am I missing some updates?
Thanks,
Thon
#!/bin/sh
GALAXY_LIB=None
if [ $GALAXY_LIB != None ]; then
if [ -n $PYTHONPATH ]; then
PYTHONPATH=$GALAXY_LIB:$PYTHONPATH
else
PYTHONPATH=$GALAXY_LIB
fi
export PYTHONPATH
fi
TMPDIR=/mnt/ngs/analysis/svcgalaxy/DATATEST/TMP
export TMPDIR
cd /mnt/ngs/analysis/svcgalaxy/DATATEST/job_working_directory/001/1916
grep -v '^#' /mnt/ngs/analysis/svcgalaxy/DATATEST/files/002/dataset_2288.dat
| sort -k1,1 | bedtools genomecov -bg -split -i stdin -g
tool-data/shared/ucsc/chrom/hg19-decoy.len temp.bg ;
bedGraphToBigWig temp.bg tool-data/shared/ucsc/chrom/hg19-decoy.len
/mnt/ngs/analysis/svcgalaxy/DATATEST/files/002/dataset_2315.dat; cd
/mnt/ngs/analysis/svcgalaxy/galaxy-test;
/mnt/ngs/analysis/svcgalaxy/galaxy-test/set_metadata.sh
/mnt/ngs/analysis/svcgalaxy/DATATEST/files
/mnt/ngs/analysis/svcgalaxy/DATATEST/job_working_directory/001/1916 .
/mnt/ngs/analysis/svcgalaxy/galaxy-test/universe_wsgi.ini
/mnt/ngs/analysis/svcgalaxy/DATA/TMP/tmp2KAn7W
/mnt/ngs/analysis/svcgalaxy/DATATEST/job_working_directory/001/1916/galaxy.j
son
/mnt/ngs/analysis/svcgalaxy/DATATEST/job_working_directory/001/1916/metadata
_in_HistoryDatasetAssociation_2551_XsCZer,/mnt/ngs/analysis/svcgalaxy/DATATE
ST/job_working_directory/001/1916/metadata_kwds_HistoryDatasetAssociation_25
51_iNUAr_,/mnt/ngs/analysis/svcgalaxy/DATATEST/job_working_directory/001/191
6/metadata_out_HistoryDatasetAssociation_2551_ZHhosg,/mnt/ngs/analysis/svcga
laxy/DATATEST/job_working_directory/001/1916/metadata_results_HistoryDataset
Association_2551_tqtiIX,,/mnt/ngs/analysis/svcgalaxy/DATATEST/job_working_di
rectory/001/1916/metadata_override_HistoryDatasetAssociation_2551_MOfd8w
echo $?
/mnt/ngs/analysis/svcgalaxy/DATATEST/job_working_directory/001/1916/galaxy_1
916.ec
Regards,
Thon
Thon deBoer Ph.D., Bioinformatics Guru
California, USA |p: +1 (650) 799-6839 |m: mailto:thondeb...@me.com
thondeb...@me.com
Public profile on http://www.linkedin.com/pub/thon-de-boer/1/1ba/a5b/
LinkedIn
From: galaxy-dev-boun...@lists.bx.psu.edu
[mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Thon de Boer
Sent: Friday, August 23, 2013 5:47 PM
To: 'Jeremy Goecks'
Cc: 'Galaxy-dev Galaxy-dev'
Subject: Re: [galaxy-dev] Why does my trackster complain about not being
able to display BED files?
I tracked it down to a problem with the API proxy configuration.
I had the API calls diverted to a different server since I wanted to ensure
that API calls would be handled by a different server, but that does not
seem to work correctly for the API calls used in trickster.
Issue resolved.
Regards,
Thon
Thon deBoer Ph.D., Bioinformatics Guru
California, USA |p: +1 (650) 799-6839 |m: mailto:thondeb...@me.com
thondeb...@me.com
Public profile on http://www.linkedin.com/pub/thon-de-boer/1/1ba/a5b/
LinkedIn
From: galaxy-dev-boun...@lists.bx.psu.edu
mailto:galaxy-dev-boun...@lists.bx.psu.edu
[mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Thon de Boer
Sent: Friday, August 23, 2013 5:12 PM
To: 'Jeremy Goecks'
Cc: 'Galaxy-dev Galaxy-dev'
Subject: Re: [galaxy-dev] Why does my trackster complain about not being
able to display BED files?
I now have a different problem with a different (production) version of
galaxy trickster.
I always get this message Could not load chroms for this dbkey
I have checked that I have files with the [key].len in
tool-data/shared/ucsc/chrom and I have twobit files etc.
The weird thing is, that the Create visualization shows the genome keys I
have but this window shows nothing for the dbkey it things it should be
getting.
So somewhere it looses the key.
Any ideas?
(I'm running the latest update)
Regards,
Thon
Thon deBoer Ph.D., Bioinformatics Guru
California, USA |p: +1 (650) 799-6839 |m: mailto:thondeb...@me.com
thondeb...@me.com
Public profile on http://www.linkedin.com/pub/thon-de-boer/1/1ba/a5b/
LinkedIn
From: galaxy-dev-boun...@lists.bx.psu.edu
mailto:galaxy-dev-boun...@lists.bx.psu.edu
[mailto:galaxy-dev-boun
Re: [galaxy-dev] Why does my trackster complain about not being able to display BED files?
Yeah, pretty sure I'm up to date.
$ hg branches
default10411:c42567f43aa7
stable 10408:6822f41bc9bb (inactive)
[svcbioinfo@srv160 gs]$ hg branch
stable
[svcbioinfo@srv160 gs]$ hg tip
changeset: 10411:c42567f43aa7
tag: tip
user:greg
date:Mon Aug 19 13:19:56 2013 -0400
summary: Filter invalid objects when generating the list of
repository_dependencies objects that are associated with a tool shed
repository installed into Galaxy.
[svcbioinfo@srv160 gs]$ hg incoming
warning: bitbucket.org certificate with fingerprint
24:9c:45:8b:9c:aa:ba:55:4e:01:6d:58:ff:e4:28:7d:2a:14:ae:3b not verified
(check hostfingerprints or web.cacerts config setting)
comparing with https://bitbucket.org/galaxy/galaxy-dist/
searching for changes
no changes found
$ hg update release_2013.08.12
0 files updated, 0 files merged, 0 files removed, 0 files unresolved
Should I just take lib/galaxy/tools/actions/__init__.py from a fresh
install?
Regards,
Thon
Thon deBoer Ph.D., Bioinformatics Guru
California, USA |p: +1 (650) 799-6839 |m: thondeb...@me.com
mailto:thondeb...@me.com
Public profile on LinkedIn
http://www.linkedin.com/pub/thon-de-boer/1/1ba/a5b/
From: Jeremy Goecks [mailto:jeremy.goe...@emory.edu]
Sent: Saturday, August 24, 2013 7:39 AM
To: Thon de Boer
Cc: 'Galaxy-dev Galaxy-dev'
Subject: Re: [galaxy-dev] Why does my trackster complain about not being
able to display BED files?
It appears you are missing some updates. This issue was fixed in this
changeset 9992:
--
changeset: 9992:7105c53139d4
branch: stable
parent: 9990:f35c0441e374
user:jeremy goecks jeremy.goe...@emory.edu
mailto:jeremy.goe...@emory.edu
date:Tue Jun 11 17:34:26 2013 -0400
files: lib/galaxy/tools/actions/__init__.py
description:
Use len_file_path config option rather than hardcoded path for chromInfo
tool parameter.
--
You might have missed this changeset if you updated to the late June release
soon after it came out. However, it is definitely included in the August
release. Can you confirm that you're running the August release?
Thanks,
J.
On Aug 23, 2013, at 9:08 PM, Thon de Boer wrote:
Ofcourse I spoke too soon.
I get a problem in that the conversion of my dataset reports that it cannot
find the chromosome length files, even though trickster itself has no
problem showing it.
I looked at the problem script and it is showing below
It seems that the script is passed the path to the chromosome files like
this tool-data/shared/ucsc/chrom/hg19-decoy.len
This will clearly never work since this file is not relative to the working
directory..
SO somewhere a script forgets to add the ${GALAXY_DATA_DIR} or whatever that
parameter is...
I could probably hardcode the location of the tool-data directory, but I
don't think that should be the solution.
Why do I only have this issue? SHOULD I have hardcoded the location of
tool-data?
The universe_wgi.ini.sample file does not hardcode this.
Am I missing some updates?
Thanks,
Thon
#!/bin/sh
GALAXY_LIB=None
if [ $GALAXY_LIB != None ]; then
if [ -n $PYTHONPATH ]; then
PYTHONPATH=$GALAXY_LIB:$PYTHONPATH
else
PYTHONPATH=$GALAXY_LIB
fi
export PYTHONPATH
fi
TMPDIR=/mnt/ngs/analysis/svcgalaxy/DATATEST/TMP
export TMPDIR
cd /mnt/ngs/analysis/svcgalaxy/DATATEST/job_working_directory/001/1916
grep -v '^#' /mnt/ngs/analysis/svcgalaxy/DATATEST/files/002/dataset_2288.dat
| sort -k1,1 | bedtools genomecov -bg -split -i stdin
-gtool-data/shared/ucsc/chrom/hg19-decoy.len temp.bg ;
bedGraphToBigWig temp.bg
tool-data/shared/ucsc/chrom/hg19-decoy.len/mnt/ngs/analysis/svcgalaxy/DATATE
ST/files/002/dataset_2315.dat; cd /mnt/ngs/analysis/svcgalaxy/galaxy-test;
/mnt/ngs/analysis/svcgalaxy/galaxy-test/set_metadata.sh
/mnt/ngs/analysis/svcgalaxy/DATATEST/files
/mnt/ngs/analysis/svcgalaxy/DATATEST/job_working_directory/001/1916 .
/mnt/ngs/analysis/svcgalaxy/galaxy-test/universe_wsgi.ini
/mnt/ngs/analysis/svcgalaxy/DATA/TMP/tmp2KAn7W
/mnt/ngs/analysis/svcgalaxy/DATATEST/job_working_directory/001/1916/galaxy.j
son
/mnt/ngs/analysis/svcgalaxy/DATATEST/job_working_directory/001/1916/metadata
_in_HistoryDatasetAssociation_2551_XsCZer,/mnt/ngs/analysis/svcgalaxy/DATATE
ST/job_working_directory/001/1916/metadata_kwds_HistoryDatasetAssociation_25
51_iNUAr_,/mnt/ngs/analysis/svcgalaxy/DATATEST/job_working_directory/001/191
6/metadata_out_HistoryDatasetAssociation_2551_ZHhosg,/mnt/ngs/analysis/svcga
laxy/DATATEST/job_working_directory/001/1916/metadata_results_HistoryDataset
Association_2551_tqtiIX,,/mnt/ngs/analysis/svcgalaxy/DATATEST/job_working_di
rectory/001/1916/metadata_override_HistoryDatasetAssociation_2551_MOfd8w
echo $?
/mnt/ngs/analysis/svcgalaxy/DATATEST/job_working_directory/001/1916/galaxy_1
916.ec
Regards,
Thon
Thon deBoer Ph.D
