Re: [galaxy-dev] MergeSamFiles.jar and TMPDIR
On Sep 27, 2011, at 1:14 PM, Nate Coraor wrote: Glen Beane wrote: We recently updated to the latest galaxy-dist, and learned that the sam_merge.xml tool now uses picard MergeSamFiles.jar to merge the files instead of the samtools merge wrapper sam_merge.py. this is a problem for us because MergeSamFiles.jar does not honor $TMPDIR when creating temporary file names (the jvm developers inexplicably hard code the value of java.io.tmpdir to /tmp in Unix/Linux rather than doing the Right Thing) . On our cluster, TMPDIR is set to something like /scratch/batch_job_id/. This location has plenty of free space, however /tmp does not and now we can't successfully merge largeish bam files. In case anyone else is bit by this, I think there are two options the Picard tools take an optional TMP_DIR= argument that lets us specify the location we want to use for a temporary directory. Initially we ended up modifying the .xml to add TMP_DIR=\$TMPDIR to the arguments to MergeSamFiles.jar. This works, but we could potentially need to do this with multiple Picard tools and not just MergeSamFiles. Now I am probably going to go with the following solution: add something like export _JAVA_OPTIONS=-Djava.io.tmpdir=$TMPDIR to the .bashrc file for my Galaxy user. This is what I've just done on our local cluster as well. I was also confounded by the lack of a proper environment variable to do this. It looks like the export _JAVA_OPTIONS=-Djava.io.tmpdir=$TMPDIR solution breaks some tools (like snpEFF, which is a 3rd party tool we use). The Jvm prints a diagnostic message to stderr that looks something like this: Picked up _JAVA_OPTIONS: -Djava.io.tmpdir=/scratch/32095.scyld.localdomain so in this case the tool fails, since it does not have a wrapper. The sam_merge.xml tool redirects stderr, so it doesn't have this problem. I am thinking about putting a wrapper script for java in my galaxy user's path that adds -Djava.io.tmpdir=$TMPDIR to the arguments. -- Glen L. Beane Senior Software Engineer The Jackson Laboratory (207) 288-6153 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] manage_db.sh upgrade error
When I run sh manage_db.sh upgrade I get the following error: 2 - 3... Traceback (most recent call last): File ./scripts/manage_db.py, line 55, in module main( repository=repo, url=db_url ) File /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/shell.py, line 150, in main ret = command_func(**kwargs) File /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/api.py, line 221, in upgrade return _migrate(url, repository, version, upgrade=True, err=err, **opts) File /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/api.py, line 349, in _migrate schema.runchange(ver, change, changeset.step) File /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/schema.py, line 184, in runchange change.run(self.engine, step) File /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/script/py.py, line 101, in run func() File lib/galaxy/model/migrate/versions/0003_security_and_libraries.py, line 423, in upgrade metadata.create_all() File /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/schema.py, line 1796, in create_all bind.create(self, checkfirst=checkfirst, tables=tables) File /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py, line 1129, in create self._run_visitor(self.dialect.schemagenerator, entity, connection=connection, **kwargs) File /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py, line 1158, in _run_visitor visitorcallable(self.dialect, conn, **kwargs).traverse(element) File /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/sql/visitors.py, line 89, in traverse return traverse(obj, self.__traverse_options__, self._visitor_dict) File /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/sql/visitors.py, line 200, in traverse return traverse_using(iterate(obj, opts), obj, visitors) File /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/sql/visitors.py, line 194, in traverse_using meth(target) File /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/sql/compiler.py, line 829, in visit_metadata collection = [t for t in sql_util.sort_tables(tables) if self._can_create(t)] File /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/sql/util.py, line 22, in sort_tables return topological.sort(tuples, tables) File /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/topological.py, line 32, in sort return [n.item for n in _sort(tuples, allitems, allow_cycles=False, ignore_self_cycles=True)] File /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/topological.py, line 209, in _sort raise CircularDependencyError(Circular dependency detected + repr(edges) + repr(queue)) sqlalchemy.exc.CircularDependencyError: Circular dependency detected [(form_definition_current, form_definition), (library_item_info, library_folder_info_association), (library_item_info, library_item_info_element), (library_item_info, library_item_info_permissions), (library_item_info, library_dataset_dataset_info_association), (library_item_info, library_info_association), (library_item_info, library_dataset_info_association), (visualization_revision, visualization), (form_definition, external_service), (form_definition, form_values), (form_definition, request_type), (form_definition, run), (form_definition, form_definition_current), (history_dataset_association, job_to_input_dataset), (history_dataset_association, job_to_output_dataset), (history_dataset_association, implicitly_converted_dataset_association), (history_dataset_association, history_dataset_association_tag_association), (history_dataset_association, validation_error), (history_dataset_association, history_dataset_association_rating_association), (history_dataset_association, job_external_output_metadata), (history_dataset_association, history_dataset_association_annotation_association), (history_dataset_association, history_dataset_association_display_at_authorization), (history_dataset_association, metadata_file), (galaxy_user, stored_workflow_rating_association), (galaxy_user, stored_workflow_menu_entry), (galaxy_user, library_item_info), (galaxy_user, visualization_user_share_association), (galaxy_user, default_user_permissions), (galaxy_user,
Re: [galaxy-dev] Setting http_proxy for Python
Hi Nate Thanks for your help. The issue was resolved I put detail here for reference to others. My RHEL username was not the owner of the folder where I was directed to install the software i.e. /opt/. As I am super user so I was running all commands using sudo. This method worked for Mercurial however I did not realize that running galaxy as sudo sh run.sh could read env from root. So I made myself the full owner of the folder and ran galaxy by issuing simple sh run.sh problem solved. Thanks. Regards, Matloob -Original Message- From: Nate Coraor [mailto:n...@bx.psu.edu] Sent: Friday, 30 September 2011 5:39 AM To: Matloob Khushi Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Setting http_proxy for Python Hi Matloob, It looks like urllib, which Galaxy uses to fetch eggs, can't use authenticated proxies. urllib2 would do it: http://stackoverflow.com/questions/34079/how-to-specify-an-authenticated -proxy-for-a-python-http-connection But this would require some changes to the egg handling code. In the meantime, you could use the egg packaging scripts on a system with an internet connection to fetch the eggs for you. See the instructions at: http://wiki.g2.bx.psu.edu/Admin/Config/Eggs Under If your Galaxy server does not have Internet access --nate Matloob Khushi wrote: Hi Guys I am trying to install Galaxy on Red Hat Enterprise Linux Server release 5.7 (Tikanga). I have installed Python 2.6. Our server is behind a proxy that needs to be authenticated. I have set up the proxy for Mercurial as described in http://www.selenic.com/mercurial/hgrc.5#http-proxy Now Mercurial is working and I have cloned the repository, however, when I issue sh run.sh command I get the following error. Though I have set the environment variable http_proxy by issuing export command on command line: export HTTP_PROXY=UserName:myunip...@servername.edu.au:8080 I have also tried various export syntax such as: export http_proxy=http://UserName:MyUniPass@ servername.edu.au:8080 I suspect Python is not reading the proxy setting to access internet. Help please. Some eggs are out of date, attempting to fetch... Traceback (most recent call last): File ./scripts/fetch_eggs.py, line 30, in ? c.resolve() # Only fetch eggs required by the config File /opt/galaxy-dist/lib/galaxy/eggs/__init__.py, line 345, in resolve egg.resolve() File /opt/galaxy-dist/lib/galaxy/eggs/__init__.py, line 195, in resolve return self.version_conflict( e.args[0], e.args[1] ) File /opt/galaxy-dist/lib/galaxy/eggs/__init__.py, line 226, in version_conflict r = pkg_resources.working_set.resolve( ( dist.as_requirement(), ), env, egg.fetch ) File /opt/galaxy-dist/lib/pkg_resources.py, line 565, in resolve raise DistributionNotFound(req) # XXX put more info here pkg_resources.DistributionNotFound: simplejson==2.1.1 Fetch failed. Regards Matloob ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Python2.7 SAM to BAM Egg Not Fetchable
Haden, Kirt wrote: Hi, When I run Galaxy under Python 2.7, I am unable to run SAM to BAM tools without getting the following error (below). The same configuration with Python 2.6 does not have an issue. The error seems to indicate bx_python eggs are not fetchable. What am I doing wrong? Hey Kirt, Are you running it on a machine with no Internet access? If not, can you run: python ./scripts/fetch_eggs.py With python set to your 2.7 version? Thanks, --nate Error report below: GALAXY TOOL ERROR REPORT This error report was sent from the Galaxy instance hosted on the server ussd-dev-lnx01.illumina.com - This is in reference to dataset id 301 from history id 68 - You should be able to view the history containing the related history item 7: SAM-to-BAM on data 6: converted BAM by logging in as a Galaxy admin user to the Galaxy instance referenced above and pointing your browser to the following link. ussd-dev-lnx01.illumina.com/history/view?id=42e99d98e0ee2844 - The user 'kha...@illumina.com' provided the following information: - job id: 200 tool id: sam_to_bam - job command line: python /home/galaxy/eval/sept09/galaxy-central/tools/samtools/sam_to_bam.py --input1=/home/galaxy/production/Sept06/galaxy-central/database/files/000/dataset_300.dat --dbkey=PhiX --output1=/home/galaxy/production/Sept06/galaxy-central/database/files/000/dataset_301.dat --index_dir=/home/galaxy/eval/sept09/galaxy-central/tool-data - job stderr: Traceback (most recent call last): File /home/galaxy/eval/sept09/galaxy-central/tools/samtools/sam_to_bam.py, line 14, in ? import pkg_resources; pkg_resources.require( bx-python ) File /home/galaxy/eval/sept09/galaxy-central/lib/galaxy/eggs/__init__.py, line 415, in require raise EggNotFetchable( str( [ egg.name for egg in e.eggs ] ) ) galaxy.eggs.EggNotFetchable: ['bx_python'] - job stdout: - job info: None - job traceback: None - (This is an automated message). Thanks, Kirt Haden Sr. Mgr. Engineering Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Tel: 858.202.4655 Fax: 858.202.4545 Email: kha...@illumina.commailto:kha...@illumina.com Website: www.illumina.comhttp://www.illumina.com/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Python2.7 SAM to BAM Egg Not Fetchable
Hey Nate, Yes, I did run this as part of the setup and it looks like I have all the eggs: total 8071 -rw-r--r-- 1 galaxy aladdin 78031 Sep 22 08:14 amqplib-0.6.1-py2.6.egg -rw-r--r-- 1 galaxy aladdin 77872 Sep 27 12:10 amqplib-0.6.1-py2.7.egg drwxr-xr-x 4 galaxy aladdin 33 Sep 22 08:14 Babel-0.9.4-py2.6.egg drwxr-xr-x 4 galaxy aladdin 33 Sep 27 12:09 Babel-0.9.4-py2.7.egg drwxr-xr-x 4 galaxy aladdin 34 Sep 22 08:14 Beaker-1.4-py2.6.egg drwxr-xr-x 4 galaxy aladdin 34 Sep 27 12:10 Beaker-1.4-py2.7.egg drwxr-xr-x 5 galaxy aladdin 96 Sep 22 08:14 bx_python-0.7.0_494c2d1d68b3_rebuild1-py2.6-linux-x86_64-ucs4.egg drwxr-xr-x 5 galaxy aladdin 96 Sep 27 12:09 bx_python-0.7.0_494c2d1d68b3_rebuild1-py2.7-linux-x86_64-ucs4.egg drwxr-xr-x 4 galaxy aladdin 35 Sep 22 08:14 Cheetah-2.2.2-py2.6-linux-x86_64-ucs4.egg drwxr-xr-x 4 galaxy aladdin 35 Sep 27 12:09 Cheetah-2.2.2-py2.7-linux-x86_64-ucs4.egg drwxr-xr-x 3 galaxy aladdin 63 Sep 22 11:36 decorator-3.1.2-py2.6.egg drwxr-xr-x 3 galaxy aladdin 63 Sep 27 12:11 decorator-3.1.2-py2.7.egg drwxr-xr-x 4 galaxy aladdin 73 Sep 23 11:09 docutils-0.7-py2.6.egg drwxr-xr-x 4 galaxy aladdin 73 Sep 27 12:11 docutils-0.7-py2.7.egg -rw-r--r-- 1 galaxy aladdin 34016 Sep 22 08:14 drmaa-0.4b3-py2.6.egg -rw-r--r-- 1 galaxy aladdin 33952 Sep 27 12:09 drmaa-0.4b3-py2.7.egg -rw-r--r-- 1 galaxy aladdin 51610 Sep 22 08:14 elementtree-1.2.6_20050316-py2.6.egg -rw-r--r-- 1 galaxy aladdin 51457 Sep 27 12:10 elementtree-1.2.6_20050316-py2.7.egg drwxr-xr-x 5 galaxy aladdin 52 Sep 22 08:14 GeneTrack-2.0.0_beta_1_dev_48da9e998f0caf01c5be731e926f4b0481f658f0-py2.6.egg drwxr-xr-x 5 galaxy aladdin 52 Sep 27 12:10 GeneTrack-2.0.0_beta_1_dev_48da9e998f0caf01c5be731e926f4b0481f658f0-py2.7.egg -rw-r--r-- 1 galaxy aladdin 7107 Sep 22 08:14 lrucache-0.2-py2.6.egg -rw-r--r-- 1 galaxy aladdin 7083 Sep 27 12:09 lrucache-0.2-py2.7.egg drwxr-xr-x 4 galaxy aladdin 32 Sep 22 08:13 Mako-0.4.1-py2.6.egg drwxr-xr-x 4 galaxy aladdin 32 Sep 27 12:09 Mako-0.4.1-py2.7.egg drwxr-xr-x 4 galaxy aladdin 38 Sep 22 08:14 MarkupSafe-0.12-py2.6-linux-x86_64-ucs4.egg drwxr-xr-x 4 galaxy aladdin 38 Sep 27 12:10 MarkupSafe-0.12-py2.7-linux-x86_64-ucs4.egg -rw-r--r-- 1 galaxy aladdin 425930 Sep 22 08:14 MySQL_python-1.2.3c1_5.1.41_static-py2.6-linux-x86_64-ucs4.egg -rw-r--r-- 1 galaxy aladdin 425988 Sep 27 12:10 MySQL_python-1.2.3c1_5.1.41_static-py2.7-linux-x86_64-ucs4.egg drwxr-xr-x 5 galaxy aladdin 45 Sep 22 08:15 nose-0.11.1-py2.6.egg drwxr-xr-x 5 galaxy aladdin 45 Sep 27 12:10 nose-0.11.1-py2.7.egg -rw-r--r-- 1 galaxy aladdin 5724 Sep 22 08:14 NoseHTML-0.4.1-py2.6.egg -rw-r--r-- 1 galaxy aladdin 5718 Sep 27 12:09 NoseHTML-0.4.1-py2.7.egg -rw-r--r-- 1 galaxy aladdin 3744 Sep 22 08:14 NoseTestDiff-0.1-py2.6.egg -rw-r--r-- 1 galaxy aladdin 3740 Sep 27 12:10 NoseTestDiff-0.1-py2.7.egg drwxr-xr-x 4 galaxy aladdin 33 Sep 22 08:14 numpy-1.6.0-py2.6-linux-x86_64-ucs4.egg drwxr-xr-x 4 galaxy aladdin 33 Sep 27 12:10 numpy-1.6.0-py2.7-linux-x86_64-ucs4.egg drwxr-xr-x 4 galaxy aladdin 33 Sep 22 08:15 Paste-1.6-py2.6.egg drwxr-xr-x 4 galaxy aladdin 33 Sep 27 12:10 Paste-1.6-py2.7.egg drwxr-xr-x 4 galaxy aladdin 33 Sep 22 08:14 PasteDeploy-1.3.3-py2.6.egg drwxr-xr-x 4 galaxy aladdin 33 Sep 27 12:09 PasteDeploy-1.3.3-py2.7.egg drwxr-xr-x 4 galaxy aladdin 33 Sep 22 08:15 PasteScript-1.7.3-py2.6.egg drwxr-xr-x 4 galaxy aladdin 33 Sep 27 12:10 PasteScript-1.7.3-py2.7.egg -rw-r--r-- 1 galaxy aladdin 78539 Sep 22 08:14 pexpect-2.4-py2.6.egg -rw-r--r-- 1 galaxy aladdin 78460 Sep 27 12:09 pexpect-2.4-py2.7.egg drwxr-xr-x 4 galaxy aladdin 34 Sep 22 08:14 pycrypto-2.0.1-py2.6-linux-x86_64-ucs4.egg -rw-r--r-- 1 galaxy aladdin 381997 Sep 27 12:10 pycrypto-2.0.1-py2.7-linux-x86_64-ucs4.egg drwxr-xr-x 4 galaxy aladdin158 Sep 22 08:13 pysam-0.4.2_kanwei_b10f6e722e9a-py2.6-linux-x86_64-ucs4.egg drwxr-xr-x 4 galaxy aladdin158 Sep 27 12:09 pysam-0.4.2_kanwei_b10f6e722e9a-py2.7-linux-x86_64-ucs4.egg drwxr-xr-x 4 galaxy aladdin 34 Sep 22 08:15 python_daemon-1.5.5-py2.6.egg drwxr-xr-x 4 galaxy aladdin 34 Sep 27 12:10 python_daemon-1.5.5-py2.7.egg -rw-r--r-- 1 galaxy aladdin 145984 Sep 22 08:14 python_lzo-1.08_2.03_static-py2.6-linux-x86_64-ucs4.egg -rw-r--r-- 1 galaxy aladdin 146031 Sep 27 12:10 python_lzo-1.08_2.03_static-py2.7-linux-x86_64-ucs4.egg drwxr-xr-x 4 galaxy aladdin 34 Sep 22 08:14 Routes-1.12.3-py2.6.egg drwxr-xr-x 4 galaxy aladdin 34 Sep 27 12:10 Routes-1.12.3-py2.7.egg -rw-r--r-- 1 galaxy aladdin 113910 Sep 22 08:14 simplejson-2.1.1-py2.6-linux-x86_64-ucs4.egg -rw-r--r-- 1 galaxy aladdin 113704 Sep 27 12:10 simplejson-2.1.1-py2.7-linux-x86_64-ucs4.egg drwxr-xr-x 4 galaxy aladdin 38 Sep 22 08:14 SQLAlchemy-0.5.6_dev_r6498-py2.6.egg drwxr-xr-x 4 galaxy aladdin 38 Sep 27 12:10 SQLAlchemy-0.5.6_dev_r6498-py2.7.egg drwxr-xr-x 4 galaxy aladdin 35 Sep 22 08:15
Re: [galaxy-dev] (Composite) Dataset Upload not Setting Metadata
Paniagua, Eric wrote:
Hi Nate,
Thank you for your response! I am glad that it was you in particular who did
respond, because I also have some questions about the way the upload tool
handles compressed files and saw that you have opened several Issues related
to this on the Galaxy bitbucket site. First though, I'll fill you in on my
further progress on the composite file issue.
As I mentioned in my original email, the trouble is that JobWrapper.finish()
calls dataset.set_meta() before it calls collect_associated_files(),
resulting dataset.extra_files_path being inaccurate because the files haven't
been moved yet from the job working directory. This is all with
set_metadata_externally=False. (I haven't worked with setting metadata
externally yet, but I think it is worth verifying whether everything works
correctly for the case I pointed out when set_metadata_externally=True.)
Since my last email, I poked around a bit more and found that my suggested
short patch was not correct but incomplete. The core problem is that
component files are not moved with the primary file, so I changed that (patch
attached, relative to { https://bitbucket.org/galaxy/galaxy-dist
5955:949e4f5fa03a }. Early in JobWrapper.finish() the primary file is moved
from the working directory to the appropriate directory under
config.file_path. This patch uses the structure of the path naming
convention to build the accurate path to the component files, and then moves
them along with the primary file. It's the least invasive (in terms of
modifying Galaxy core code) potential fix I came up with, but since it relies
explicitly on path structure and naming conventions I still think it's a bit
of a hack. However, it does seem to work, at least to me. I don't have the
resources or experience yet to attempt to test whether it negatively (or
positively!) impacts others' use cases!
. I do hope that someone more intimately familiar with the project would be
able to vet it and make improvements / provide feedback / incorporate a fix for
this issue to the Galaxy mainline.
However, since the group I'm working for operates downstream of the main
Galaxy Development Team (primarily to add tools and maintain a local Galaxy
server for our institution), we really try to avoid complications in merges
when pulling down updates by avoiding changing Galaxy core code as much as
possible. Consequently, I had to shift focus from finding the best correct
fix to simply finding a workaround, and I found a very simple one. However,
it relies on the deprecated code tool config tag, so it will only work
until support for the code tag is removed. It's self-explanatory and also
attached.
Hi Eric,
I haven't tried out the patch, but would this have an impact on
processing that occurs when collect_associated_files() runs since they
have been moved back already by your code?
I'd suggest enabling set_metadata_externally = True as I suspect this
will solve the problems since metadata will be set while the primary
file is still in the temporary location. set_metadata_externally = True
should really be the default setting and we may remove internal metadata
setting entirely at some point.
Regarding potential fixes: Is there any reason why component files of a
composite dataset should not always follow the primary file? I don't know of
one, but maybe there is some case in running Galaxy over a cluster where
there is reason to do otherwise. If the answer is no, it seems the best
approach is to add an abstraction layer between the datasets and the file
system to avoid various paths that might be associated to a dataset from
falling out of sync. One (obviously motivational) function of this layer
would be to move or copy datasets on the file system atomically. This seems
pretty important especially for enabling even more flexible support of
distributed or clustered installations.
We're going to be working on just such an abstraction layer shortly.
Regarding the upload tool: I just saw your response to my other email on
Leandro's thread. Thanks for pointing me to the code responsible. I've
already forked a version of the upload tool for our Galaxy instance. There
are a lot of issues with the way the way the default upload tool works, and
it sounds like you're aware of that and perhaps it's been a headache for you
too. If it might help make progress more swift in the Galaxy mainline, I'd
be glad to send you patches for updates and changes I make in the upload
tool. I do have at least one big question about it though: It currently
attempts to sniff and uncompress files regardless of whether Auto-detect
was selected for File format; is that by design or accident? It seems that
(particularly the bits try uncompressing files) causes headaches for a number
of developers/users. I'm likely to remove that behavior from my version of
the upload
Re: [galaxy-dev] Python2.7 SAM to BAM Egg Not Fetchable
The egg is there, but the error still shows up. -Original Message- From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Nate Coraor Sent: Friday, September 30, 2011 6:40 AM To: Haden, Kirt Cc: 'galaxy-dev@lists.bx.psu.edu' Subject: Re: [galaxy-dev] Python2.7 SAM to BAM Egg Not Fetchable Haden, Kirt wrote: Hi, When I run Galaxy under Python 2.7, I am unable to run SAM to BAM tools without getting the following error (below). The same configuration with Python 2.6 does not have an issue. The error seems to indicate bx_python eggs are not fetchable. What am I doing wrong? Hey Kirt, Are you running it on a machine with no Internet access? If not, can you run: python ./scripts/fetch_eggs.py With python set to your 2.7 version? Thanks, --nate Error report below: GALAXY TOOL ERROR REPORT This error report was sent from the Galaxy instance hosted on the server ussd-dev-lnx01.illumina.com -- --- This is in reference to dataset id 301 from history id 68 -- --- You should be able to view the history containing the related history item 7: SAM-to-BAM on data 6: converted BAM by logging in as a Galaxy admin user to the Galaxy instance referenced above and pointing your browser to the following link. ussd-dev-lnx01.illumina.com/history/view?id=42e99d98e0ee2844 -- --- The user 'kha...@illumina.com' provided the following information: -- --- job id: 200 tool id: sam_to_bam -- --- job command line: python /home/galaxy/eval/sept09/galaxy-central/tools/samtools/sam_to_bam.py --input1=/home/galaxy/production/Sept06/galaxy-central/database/files/000/dataset_300.dat --dbkey=PhiX --output1=/home/galaxy/production/Sept06/galaxy-central/database/files/000/dataset_301.dat --index_dir=/home/galaxy/eval/sept09/galaxy-central/tool-data -- --- job stderr: Traceback (most recent call last): File /home/galaxy/eval/sept09/galaxy-central/tools/samtools/sam_to_bam.py, line 14, in ? import pkg_resources; pkg_resources.require( bx-python ) File /home/galaxy/eval/sept09/galaxy-central/lib/galaxy/eggs/__init__.py, line 415, in require raise EggNotFetchable( str( [ egg.name for egg in e.eggs ] ) ) galaxy.eggs.EggNotFetchable: ['bx_python'] -- --- job stdout: -- --- job info: None -- --- job traceback: None -- --- (This is an automated message). Thanks, Kirt Haden Sr. Mgr. Engineering Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Tel: 858.202.4655 Fax: 858.202.4545 Email: kha...@illumina.commailto:kha...@illumina.com Website: www.illumina.comhttp://www.illumina.com/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Python2.7 SAM to BAM Egg Not Fetchable
Is this instance using the run job as user patch, and if so, does the user running the job have permission to access the egg? Thanks for using Galaxy, Dan On Sep 30, 2011, at 11:36 AM, Chorny, Ilya wrote: The egg is there, but the error still shows up. -Original Message- From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Nate Coraor Sent: Friday, September 30, 2011 6:40 AM To: Haden, Kirt Cc: 'galaxy-dev@lists.bx.psu.edu' Subject: Re: [galaxy-dev] Python2.7 SAM to BAM Egg Not Fetchable Haden, Kirt wrote: Hi, When I run Galaxy under Python 2.7, I am unable to run SAM to BAM tools without getting the following error (below). The same configuration with Python 2.6 does not have an issue. The error seems to indicate bx_python eggs are not fetchable. What am I doing wrong? Hey Kirt, Are you running it on a machine with no Internet access? If not, can you run: python ./scripts/fetch_eggs.py With python set to your 2.7 version? Thanks, --nate Error report below: GALAXY TOOL ERROR REPORT This error report was sent from the Galaxy instance hosted on the server ussd-dev-lnx01.illumina.com -- --- This is in reference to dataset id 301 from history id 68 -- --- You should be able to view the history containing the related history item 7: SAM-to-BAM on data 6: converted BAM by logging in as a Galaxy admin user to the Galaxy instance referenced above and pointing your browser to the following link. ussd-dev-lnx01.illumina.com/history/view?id=42e99d98e0ee2844 -- --- The user 'kha...@illumina.com' provided the following information: -- --- job id: 200 tool id: sam_to_bam -- --- job command line: python /home/galaxy/eval/sept09/galaxy-central/tools/samtools/sam_to_bam.py --input1=/home/galaxy/production/Sept06/galaxy-central/database/files/000/dataset_300.dat --dbkey=PhiX --output1=/home/galaxy/production/Sept06/galaxy-central/database/files/000/dataset_301.dat --index_dir=/home/galaxy/eval/sept09/galaxy-central/tool-data -- --- job stderr: Traceback (most recent call last): File /home/galaxy/eval/sept09/galaxy-central/tools/samtools/sam_to_bam.py, line 14, in ? import pkg_resources; pkg_resources.require( bx-python ) File /home/galaxy/eval/sept09/galaxy-central/lib/galaxy/eggs/__init__.py, line 415, in require raise EggNotFetchable( str( [ egg.name for egg in e.eggs ] ) ) galaxy.eggs.EggNotFetchable: ['bx_python'] -- --- job stdout: -- --- job info: None -- --- job traceback: None -- --- (This is an automated message). Thanks, Kirt Haden Sr. Mgr. Engineering Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Tel: 858.202.4655 Fax: 858.202.4545 Email: kha...@illumina.commailto:kha...@illumina.com Website: www.illumina.comhttp://www.illumina.com/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Python2.7 SAM to BAM Egg Not Fetchable
Same permissions as the 2.6 egg. -Original Message- From: Daniel Blankenberg [mailto:d...@bx.psu.edu] Sent: Friday, September 30, 2011 8:59 AM To: Chorny, Ilya Cc: Nate Coraor; Haden, Kirt; 'galaxy-dev@lists.bx.psu.edu' Subject: Re: [galaxy-dev] Python2.7 SAM to BAM Egg Not Fetchable Is this instance using the run job as user patch, and if so, does the user running the job have permission to access the egg? Thanks for using Galaxy, Dan On Sep 30, 2011, at 11:36 AM, Chorny, Ilya wrote: The egg is there, but the error still shows up. -Original Message- From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Nate Coraor Sent: Friday, September 30, 2011 6:40 AM To: Haden, Kirt Cc: 'galaxy-dev@lists.bx.psu.edu' Subject: Re: [galaxy-dev] Python2.7 SAM to BAM Egg Not Fetchable Haden, Kirt wrote: Hi, When I run Galaxy under Python 2.7, I am unable to run SAM to BAM tools without getting the following error (below). The same configuration with Python 2.6 does not have an issue. The error seems to indicate bx_python eggs are not fetchable. What am I doing wrong? Hey Kirt, Are you running it on a machine with no Internet access? If not, can you run: python ./scripts/fetch_eggs.py With python set to your 2.7 version? Thanks, --nate Error report below: GALAXY TOOL ERROR REPORT This error report was sent from the Galaxy instance hosted on the server ussd-dev-lnx01.illumina.com - - --- This is in reference to dataset id 301 from history id 68 - - --- You should be able to view the history containing the related history item 7: SAM-to-BAM on data 6: converted BAM by logging in as a Galaxy admin user to the Galaxy instance referenced above and pointing your browser to the following link. ussd-dev-lnx01.illumina.com/history/view?id=42e99d98e0ee2844 - - --- The user 'kha...@illumina.com' provided the following information: - - --- job id: 200 tool id: sam_to_bam - - --- job command line: python /home/galaxy/eval/sept09/galaxy-central/tools/samtools/sam_to_bam.py --input1=/home/galaxy/production/Sept06/galaxy-central/database/files/000/dataset_300.dat --dbkey=PhiX --output1=/home/galaxy/production/Sept06/galaxy-central/database/files/000/dataset_301.dat --index_dir=/home/galaxy/eval/sept09/galaxy-central/tool-data - - --- job stderr: Traceback (most recent call last): File /home/galaxy/eval/sept09/galaxy-central/tools/samtools/sam_to_bam.py, line 14, in ? import pkg_resources; pkg_resources.require( bx-python ) File /home/galaxy/eval/sept09/galaxy-central/lib/galaxy/eggs/__init__.py , line 415, in require raise EggNotFetchable( str( [ egg.name for egg in e.eggs ] ) ) galaxy.eggs.EggNotFetchable: ['bx_python'] - - --- job stdout: - - --- job info: None - - --- job traceback: None - - --- (This is an automated message). Thanks, Kirt Haden Sr. Mgr. Engineering Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Tel: 858.202.4655 Fax: 858.202.4545 Email: kha...@illumina.commailto:kha...@illumina.com Website: www.illumina.comhttp://www.illumina.com/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using
Re: [galaxy-dev] Python2.7 SAM to BAM Egg Not Fetchable
If this is running on a cluster, is there any chance the version of Python used on your cluster nodes was compiled with UCS-2 Unicode? You can check by running scripts/get_platforms.py on the host in question using the desired python binary. The eggs you have only work with UCS-4 Unicode. --nate Chorny, Ilya wrote: Same permissions as the 2.6 egg. -Original Message- From: Daniel Blankenberg [mailto:d...@bx.psu.edu] Sent: Friday, September 30, 2011 8:59 AM To: Chorny, Ilya Cc: Nate Coraor; Haden, Kirt; 'galaxy-dev@lists.bx.psu.edu' Subject: Re: [galaxy-dev] Python2.7 SAM to BAM Egg Not Fetchable Is this instance using the run job as user patch, and if so, does the user running the job have permission to access the egg? Thanks for using Galaxy, Dan On Sep 30, 2011, at 11:36 AM, Chorny, Ilya wrote: The egg is there, but the error still shows up. -Original Message- From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Nate Coraor Sent: Friday, September 30, 2011 6:40 AM To: Haden, Kirt Cc: 'galaxy-dev@lists.bx.psu.edu' Subject: Re: [galaxy-dev] Python2.7 SAM to BAM Egg Not Fetchable Haden, Kirt wrote: Hi, When I run Galaxy under Python 2.7, I am unable to run SAM to BAM tools without getting the following error (below). The same configuration with Python 2.6 does not have an issue. The error seems to indicate bx_python eggs are not fetchable. What am I doing wrong? Hey Kirt, Are you running it on a machine with no Internet access? If not, can you run: python ./scripts/fetch_eggs.py With python set to your 2.7 version? Thanks, --nate Error report below: GALAXY TOOL ERROR REPORT This error report was sent from the Galaxy instance hosted on the server ussd-dev-lnx01.illumina.com - - --- This is in reference to dataset id 301 from history id 68 - - --- You should be able to view the history containing the related history item 7: SAM-to-BAM on data 6: converted BAM by logging in as a Galaxy admin user to the Galaxy instance referenced above and pointing your browser to the following link. ussd-dev-lnx01.illumina.com/history/view?id=42e99d98e0ee2844 - - --- The user 'kha...@illumina.com' provided the following information: - - --- job id: 200 tool id: sam_to_bam - - --- job command line: python /home/galaxy/eval/sept09/galaxy-central/tools/samtools/sam_to_bam.py --input1=/home/galaxy/production/Sept06/galaxy-central/database/files/000/dataset_300.dat --dbkey=PhiX --output1=/home/galaxy/production/Sept06/galaxy-central/database/files/000/dataset_301.dat --index_dir=/home/galaxy/eval/sept09/galaxy-central/tool-data - - --- job stderr: Traceback (most recent call last): File /home/galaxy/eval/sept09/galaxy-central/tools/samtools/sam_to_bam.py, line 14, in ? import pkg_resources; pkg_resources.require( bx-python ) File /home/galaxy/eval/sept09/galaxy-central/lib/galaxy/eggs/__init__.py , line 415, in require raise EggNotFetchable( str( [ egg.name for egg in e.eggs ] ) ) galaxy.eggs.EggNotFetchable: ['bx_python'] - - --- job stdout: - - --- job info: None - - --- job traceback: None - - --- (This is an automated message). Thanks, Kirt Haden Sr. Mgr. Engineering Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Tel: 858.202.4655 Fax: 858.202.4545 Email: kha...@illumina.commailto:kha...@illumina.com Website: www.illumina.comhttp://www.illumina.com/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by
Re: [galaxy-dev] Python2.7 SAM to BAM Egg Not Fetchable
Yes, this seem to be the issue. The Python on the compute nodes is not configured correctly. Thanks. Kirt -Original Message- From: Nate Coraor [mailto:n...@bx.psu.edu] Sent: Friday, September 30, 2011 10:04 AM To: Chorny, Ilya Cc: Daniel Blankenberg; Haden, Kirt; 'galaxy-dev@lists.bx.psu.edu' Subject: Re: [galaxy-dev] Python2.7 SAM to BAM Egg Not Fetchable If this is running on a cluster, is there any chance the version of Python used on your cluster nodes was compiled with UCS-2 Unicode? You can check by running scripts/get_platforms.py on the host in question using the desired python binary. The eggs you have only work with UCS-4 Unicode. --nate Chorny, Ilya wrote: Same permissions as the 2.6 egg. -Original Message- From: Daniel Blankenberg [mailto:d...@bx.psu.edu] Sent: Friday, September 30, 2011 8:59 AM To: Chorny, Ilya Cc: Nate Coraor; Haden, Kirt; 'galaxy-dev@lists.bx.psu.edu' Subject: Re: [galaxy-dev] Python2.7 SAM to BAM Egg Not Fetchable Is this instance using the run job as user patch, and if so, does the user running the job have permission to access the egg? Thanks for using Galaxy, Dan On Sep 30, 2011, at 11:36 AM, Chorny, Ilya wrote: The egg is there, but the error still shows up. -Original Message- From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Nate Coraor Sent: Friday, September 30, 2011 6:40 AM To: Haden, Kirt Cc: 'galaxy-dev@lists.bx.psu.edu' Subject: Re: [galaxy-dev] Python2.7 SAM to BAM Egg Not Fetchable Haden, Kirt wrote: Hi, When I run Galaxy under Python 2.7, I am unable to run SAM to BAM tools without getting the following error (below). The same configuration with Python 2.6 does not have an issue. The error seems to indicate bx_python eggs are not fetchable. What am I doing wrong? Hey Kirt, Are you running it on a machine with no Internet access? If not, can you run: python ./scripts/fetch_eggs.py With python set to your 2.7 version? Thanks, --nate Error report below: GALAXY TOOL ERROR REPORT This error report was sent from the Galaxy instance hosted on the server ussd-dev-lnx01.illumina.com --- -- - --- This is in reference to dataset id 301 from history id 68 --- -- - --- You should be able to view the history containing the related history item 7: SAM-to-BAM on data 6: converted BAM by logging in as a Galaxy admin user to the Galaxy instance referenced above and pointing your browser to the following link. ussd-dev-lnx01.illumina.com/history/view?id=42e99d98e0ee2844 --- -- - --- The user 'kha...@illumina.com' provided the following information: --- -- - --- job id: 200 tool id: sam_to_bam --- -- - --- job command line: python /home/galaxy/eval/sept09/galaxy-central/tools/samtools/sam_to_bam.py --input1=/home/galaxy/production/Sept06/galaxy-central/database/files/000/dataset_300.dat --dbkey=PhiX --output1=/home/galaxy/production/Sept06/galaxy-central/database/files/000/dataset_301.dat --index_dir=/home/galaxy/eval/sept09/galaxy-central/tool-data --- -- - --- job stderr: Traceback (most recent call last): File /home/galaxy/eval/sept09/galaxy-central/tools/samtools/sam_to_bam.py, line 14, in ? import pkg_resources; pkg_resources.require( bx-python ) File /home/galaxy/eval/sept09/galaxy-central/lib/galaxy/eggs/__init__.py , line 415, in require raise EggNotFetchable( str( [ egg.name for egg in e.eggs ] ) ) galaxy.eggs.EggNotFetchable: ['bx_python'] --- -- - --- job stdout: --- -- - --- job info: None --- -- - --- job traceback: None --- -- - --- (This is an automated message). Thanks, Kirt Haden Sr. Mgr. Engineering Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Tel: 858.202.4655 Fax: 858.202.4545 Email: kha...@illumina.commailto:kha...@illumina.com Website:
Re: [galaxy-dev] Some changes to tophat and cufflinks wrappers for indexing loc files
I made some changes to the cufflinks and tophat wrapper to pull gtf files from an indexed loc file as opposed to from history. The diff's are attached. I made it an optional feature. Please let me know if this makes its way into galaxy-central. Best, Ilya Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: icho...@illumina.commailto:icho...@illumina.com Website: www.illumina.comhttp://www.illumina.com cufflinks_wrapper.xml.diff Description: cufflinks_wrapper.xml.diff tophat_wrapper.xml.diff Description: tophat_wrapper.xml.diff ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
