Re: [galaxy-dev] Galaxy dropping jobs?
Hi again, The loop (as explained below), did the job :) Nikolay Thank you very much, Nate, 1. I have put a fix : a loop running the JobStatus check 5 times every 60 secs and only then throwing an exception as the one below. It happened that all connect failures happen at the same time - at slurm log rotation time at 3 am. Hopefully it helps :) 2. Our slurm conf keeps the info about each job for 5 min. But looking at the code, it seems that in in the case you describe below, there will be an Invalid job exception leading to Job finished state. Am I wrong? Anyway, I'll let you know if the loop does the job. Thanks again, Nikolay On 2013-11-04 15:57, Nate Coraor wrote: Hi Nikolay, With slurm, the following change that I backed out should fix the problem: https://bitbucket.org/galaxy/galaxy-central/diff/lib/galaxy/jobs/runners/drmaa.py?diff2=d46b64f12c52at=default Although I do believe that if Galaxy doesn't read the completion state before slurm forgets about the job (MinJobAge in slurm.conf), this change could result in the job becoming permanently stuck in the running state. I should have some enhancements to the DRMAA runner for slurm coming soon that would prevent this. --nate On Oct 31, 2013, at 5:27 AM, Nikolai Vazov wrote: Hi, I discovered a weird issue in the job behaviour : Galaxy is running a long job on a cluster (more than 24h), about 15 hours later it misses the connection with SLURM on the cluster and throws the following message : [root@galaxy-prod01 galaxy-dist]# grep 3715200 paster.log galaxy.jobs.runners.drmaa INFO 2013-10-30 10:51:54,149 (555) queued as 3715200 galaxy.jobs.runners.drmaa DEBUG 2013-10-30 10:51:55,149 (555/3715200) state change: job is queued and active galaxy.jobs.runners.drmaa DEBUG 2013-10-30 10:52:13,516 (555/3715200) state change: job is running galaxy.jobs.runners.drmaa INFO 2013-10-31 03:29:33,090 (555/3715200) job left DRM queue with following message: code 1: slurm_load_jobs error: Unable to contact slurm controller (connect failure),job_id: 3715200 Is there a timeout in Galaxy for contacting slurm? Yet, the job is still running properly on the cluster ... Thanks for help, it's really urgent :) Nikolay -- Nikolay Vazov, PhD Research Computing Centre - http://hpc.uio.no USIT, University of Oslo ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ -- Nikolay Vazov, PhD Research Computing Centre - http://hpc.uio.no USIT, University of Oslo ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] bug in handling sessions
Hi, I have discovered the following weird Galaxy behaviour. If one sets log_actions = True in universe_wsgi.ini and tries to display the Saved Histories, then the Saved histories don't show up. The problem arises from ../lib/galaxy/web/framework/helpers/grids.py line (around 230) trans.log_action( trans.get_user(), unicode( grid.view ), context, params) Until this line, the Query object 'query' has queried the DB and fetched the necessary data. Yet, in the file ../lib/galaxy/web/framework/__init__.py in the function log_action self.sa_session.flush() seems to flush everything, including the data fetched by the 'query' object in grids.py Thus, the Saved Histories get No Items message. Commenting out the line log_actions = True does the job, but ... Any suggestions? Best regards Nikolay -- Nikolay Vazov, PhD Research Computing Centre - http://hpc.uio.no USIT, University of Oslo ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] POST and WSGI
Hi, I am trying to set up Galaxy with an IdP. Yet, after the autentication, the SAML data coming via POST method to the WSGI (Galaxy) dissappears. I can only see it in a Firefox add-on, scrolling, but there is no way I can get hold of it in order to validate the users. Any idea how one can get environ[wsgi.input].body or similar in controllers/user.py finctions? Some global var with this? Thank you in advance Nikolay -- Nikolay Vazov, PhD Research Computing Centre - http://hpc.uio.no USIT, University of Oslo ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Interested in speaking with other institutions deploying Galaxy locally?
Hi, all, We are in the process of setting up Galaxy as a big university framework for the entire university and world-wide free academic access. We currently have a portal (www.bioportal.uio.no) which is bioinformatics oriented. The Galaxy we are setting up will give access to more than 100 applications and will include the following features: - Apache proxy enabled - implemented - PostgreSQL located in another host - - implemented - sending jobs to a cluster (using SLURM, drmaa) - implemented - identification via SAML2 - (not OpenID) - implementaiton phase now, problems with reading the response from IdP If any of you has experience in pysaml2 + WSGI, please let me know. In turn, I am willing to share our experience so far with the issues above. Best regards Nikolay On Sun, 29 Apr 2012 17:11:05 +1200, Edward Hills ehills...@gmail.com wrote: Hi Ann, We are in the process of doing the same thing here at the University of Otago (New Zealand). If we are able to sort something via the timezone difference (we are GMT+12, first to see the sun!) we would very much be interested. Cheers, Ed Hi everyone - Here at the University of Iowa we are working on deploying Galaxy locally for campus wide access. I am interested in forming a community of other institutions trying to deploy Galaxy locally and mange/operate it on a broad level. Is anyone else? If there is enough interest, possibly we could have a community conference call every other month to have an open discussion on how we are all deploying galaxy, customizations we are making, problems we are encountering, bugs, and any add-on operations management for galaxy being developed, etc. Would love to hear from others operating Galaxy or in process of standing up a local deployment. Thanks! Ann Black-Ziegelbein ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Nikolay Vazov, PhD Research Computing Centre - http://hpc.uio.no USIT, University of Oslo ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] ID providers other than OpenID
Hi, we work at the University of Oslo, USIT, The Research Computing Service group (Norway). We are preparing a bioinformatics portal using Galaxy and one of the requirements for the University of Oslo production is to implement an authentication called FEIDE. Feide is local (for Norway) IDp service based on saml2 which is yet different from OpenID. It supposes the existence of metadata files on the sp server side containing blocks with, e.g. (for idp) 'SingleSignOnService' = 'https://idp.feide.no/simplesaml/saml2/idp/SSOService.php', 'SingleLogoutService' = 'https://idp.feide.no/simplesaml/saml2/idp/SingleLogoutServiceiFrame.php', 'SingleLogoutServiceResponse' = 'https://idp.feide.no/simplesaml/saml2/idp/SingleLogoutServiceiFrameResponse.php', 'certFingerprint' = 'cde69e332fa7dd0eaa99ee0ddf06916e8942ac53', 'hint.cidr' = '158.38.0.0/16' These blocks are read during the authentication process. Galaxy seems to be only supporting OpenID and new idp-s are added simply by adding a new url to OPENID_PROVIDER variable. Is there a solution if we have to communicate metadata between idp and sp? And ... is your egg python_openid-2.2.5-py2.6.egg using pysaml? Can it be rescrambled such that it can read some pysaml metadata files. Thank you in advance Nikolay Vazov -- Nikolay Vazov, PhD Research Computing Centre - http://hpc.uio.no USIT, University of Oslo ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Adding an OpenID provider to Galaxy
Hi, I have been trying to add a new OpenID provider to the Galaxy-hardcoded list of providers. I type in my credentials in the new provider's window, but I got the following error (I am giving here only the last five lines) Module openid.consumer.consumer:413 in complete Module openid.message:167 in fromPostArgs Module openid.message:200 in _fromOpenIDArgs Module openid.message:240 in setOpenIDNamespace InvalidOpenIDNamespace: Invalid OpenID Namespace u'http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0' Does anybody have an idea what in my server goes wrong with the OpenID Namspace and how I can fix it? It might be useful to say that my Galaxy database is located on a separate machine with an SSL connection form the Galaxy app server. Thank you in advance Nikolai -- Nikolay Vazov, PhD Research Computing Centre - http://hpc.uio.no USIT, University of Oslo ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] connecting to a DB on a different host
Hi, Nate, I am trying to follow the designed steps below. 1. I downloaded psycopg2 2.0.13. Does it matter where in galaxy-dist directory I place it? 2. I modified setup.py - OK 3. pg_config --version gives : PostgreSQL 8.1.23 I don't really get what you mean by make sure the version you want to link against is the version found first on your $PATH Is this the Python version you mean? I am using python26 (64bit for CentosOS) and I use a link I created to it in my /home/galaxy/galaxy-python directory. The galaxy dist is located at /home/galaxy/galaxy-dist 4. Given the above details, can I just execute 4? Thank you Nikolay On Mon, 2 May 2011 10:27:41 -0400, Nate Coraor n...@bx.psu.edu wrote: Nikolai Vazov wrote: Hi, Nate, Thanks a lot for the hint : I also had suspicions about the SSL and it seems to be working, well, I mean, failing, as you supposed because of my psycopg2 which was built without SSL. OperationalError: (OperationalError) sslmode value require invalid when SSL support is not compiled in At least I know what I shall do :) We provide the psycopg2 egg, so rebuilding it may not be so simple. I would suggest the following, if you want to do it by hand and link it against your system libpq: 1. Download psycopg2 2.0.13. The version is going to matter since Galaxy expects it. 2. In setup.py, find: from distutils.core import setup, Extension Prior to that import, add the following: from scramble_lib import * 3. Find the pg_config executable and make sure the version you want to link against is the version found first on your $PATH. 4. Execute: % PYTHONPATH=/path/to/galaxy-dist/scripts/scramble/lib python setup.py egg_info --tag-build=_8.4.2_static bdist_egg Let me know if you have any troubles with this process. --nate -- Nikolay Vazov, PhD Research Computing Centre - http://hpc.uio.no USIT, University of Oslo ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] connecting to a DB on a different host
Hi again, The version of the PostgreSQL server I am trying to connect to is actually 8.3. The DB department say they will soon move to 9.0. The psycopg2 version in my eggs.ini for Galaxy is 8.4.2: psycopg2-2.0.13_8.4.2_static-py2.6-linux-x86_64-ucs4.egg Do you think it will work? 3. Find the pg_config executable and make sure the version you want to link against is the version found first on your $PATH. -- Nikolay Vazov, PhD Research Computing Centre - http://hpc.uio.no USIT, University of Oslo ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] connecting to a DB on a different host
Hi, Nate, When trying to execute PYTHONPATH=/home/galaxy/galaxy-dist/scripts/scramble/lib python26 setup.py egg_info --tag-build=_8.4.2_static bdist_egg I get Traceback (most recent call last): File setup.py, line 52, in module from scramble_lib import * File /home/galaxy/galaxy-dist/scripts/scramble/lib/scramble_lib.py, line 149, in module from ez_setup import use_setuptools ImportError: No module named ez_setup I guess I shall install both setuptools and ez_setup before I continue ... Thank you again for your time Nikolay -- Nikolay Vazov, PhD Research Computing Centre - http://hpc.uio.no USIT, University of Oslo ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] connecting to a DB on a different host
Hi, I am trying to connect my galaxy instance to a DB on a different host. This DB uses an SSL encryption and Auth method md5. I have no problems in connecting to the DB via the command line, but the line (in universe_wsqi.ini) : database_connection = postgres://USERNAME:PASSWORD@HOST:5432/DB_NAME gives the following error when I run the script 'run.sh': OperationalError: (OperationalError) FATAL: no pg_hba.conf entry for host HOST_IP, user USERNAME, database DB_NAME, SSL off I managed to connect to a local instance of PostgreSQL server (without SSL, though) Can you give me some hints? Thanks in advance Nikolay -- Nikolay Vazov, PhD Research Computing Centre - http://hpc.uio.no USIT, University of Oslo ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] connecting to a DB on a different host
Hi, Nate, Thanks a lot for the hint : I also had suspicions about the SSL and it seems to be working, well, I mean, failing, as you supposed because of my psycopg2 which was built without SSL. OperationalError: (OperationalError) sslmode value require invalid when SSL support is not compiled in At least I know what I shall do :) Best regards Nikolay Hi Nikolay, You may be able to connect with: postgres://USERNAME:PASSWORD@HOST:5432/DB_NAME?sslmode=SSLMODE Where SSLMODE is one of the valid values from libpq: http://www.postgresql.org/docs/9.0/static/libpq-ssl.html#LIBPQ-SSL-SSLMODE-STATEMENTS If psycopg2 was built without SSL this will probably fail, in which case I can rebuild the psycopg2 eggs to link against SSL, but this may take a while. --nate -- Nikolay Vazov, PhD Research Computing Centre - http://hpc.uio.no USIT, University of Oslo ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
