Re: [galaxy-dev] Error: Job output not returned from cluster
Hi, Thanks a lot it actually helped. It is not exactly as straightforward in drmaa.py but somehow I could manage. However, it was not the problem. For some reason, the user needs to write files from the node to job_working_directory/00X// and the latter is not world-writable. I had to make everyone chmod everything to 777 to make it work. Did I miss something? Best, L-A Le 24/04/2012 15:17, Alban Lermine a écrit : Hi L-A, I run Galaxy as real user on our cluster with pbs (free version). We first configure LDAP authentification for having email account related to unix account (just cut the @curie.fr) Then I have modify pbs.py (in GALAXY_DIRgalaxy-dist/lib/galaxy/jobs/runners) I have just disconnected the pbs submission through python library and replace it by a system call (just like I send jobs to the cluster with command line), here is the code used: galaxy_job_id = job_wrapper.job_id log.debug((%s) submitting file %s % ( galaxy_job_id, job_file ) ) log.debug((%s) command is: %s % ( galaxy_job_id, command_line ) ) # Submit job with system call instead of using python PBS library - Permit to run jobs as .. with sudo -u cmd prefix galaxy_job_idSTR = str(job_wrapper.job_id) galaxy_tool_idSTR = str(job_wrapper.tool.id) galaxy_job_name = galaxy_job_idSTR+_+galaxy_tool_idSTR+_+job_wrapper.user torque_options = runner_url.split(/) queue = torque_options[3] ressources = torque_options[4] user_mail = job_wrapper.user.split(@) username = user_mail[0] torque_cmd = sudo -u username echo +\+command_line+\ | qsub -o +ofile+ -e +efile+ -M +job_wrapper.user+ -N +galaxy_job_name+ -q +queue+ +ressources submit_pbs_job = os.popen(torque_cmd) job_id = submit_pbs_job.read().rstrip(\n) #Original job launcher #job_id = pbs.pbs_submit(c, job_attrs, job_file, pbs_queue_name, None) pbs.pbs_disconnect(c) Second thing I have done is to wait error and output file from torque in the finish_job function (if not, I never receive the output, seems to be your problem..), here is the code used: def finish_job( self, pbs_job_state ): Get the output/error for a finished job, pass to `job_wrapper.finish` and cleanup all the PBS temporary files. ofile = pbs_job_state.ofile efile = pbs_job_state.efile job_file = pbs_job_state.job_file # collect the output try: # With qsub system call, need to wait efile and ofile creation at the end of the job execution before reading them efileExists = os.path.isfile(efile) ofileExists = os.path.isfile(ofile) efileExistsSTR = str(efileExists) ofileExistsSTR = str(ofileExists) while efileExistsSTR == False: time.sleep( 1 ) efileExists = os.path.isfile(efile) efileExistsSTR = str(efileExists) while ofileExistsSTR == False: time.sleep( 1 ) ofileExists = os.path.isfile(ofile) ofileExistsSTR = str(ofileExists) # Back to original code ofh = file(ofile, r) efh = file(efile, r) stdout = ofh.read( 32768 ) stderr = efh.read( 32768 ) except: stdout = '' stderr = 'Job output not returned by PBS: the output datasets were deleted while the job was running, the job was manually dequeued or there was a cluster error.' log.debug(stderr) * Last step is to allow galaxy user to run sudo Hope it can help you finding your problem.. See you, Alban ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Error running job as real user: looking et the wrong dir?
Hi everyone, I'm (still) having issues with running jobs as the real user on our PBS Pro cluster. When I try running a job, it ends up in error state and displays the following error message: touch: cannot touch `/home/galaxy/.drmaa/9167860.pbs-master2.embl.de.started': Permission denied /g/funcgen/galaxy-dev/database/pbs/galaxy_1516.sh: line 15: syntax error: unexpected end of file /var/spool/PBS/mom_priv/jobs/9167860.pbs-master2.embl.de.SC: line 1: /home/galaxy/.drmaa/9167860.pbs-master2.embl.de.exitcode: Permission denied I suspect this is due to some code executed as the real user (since the permission is denied) but then why looking at /home/galaxy instead of /home/realuser ? If you have any idea about what's going on, please share, I'll be forever grateful! Thanks, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Running jobs as real user and extra_file_path
Hi everyone, I just wanted to ask how the extra_file_path is handled in case of job running as the real user since the file_path is only writable by the galaxy user. Any clue? Thanks, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Error: Job output not returned from cluster
At first we thought it could be an ssh issue but submitting jobs and getting the output back isn't a problem when I do it from my personal user manually, so it's really related to Galaxy. We're using PBS Pro btw. And I'm still at loss... :( L-A Le 23/04/2012 15:42, zhengqiu cai a écrit : I am having the same problem when I use condor as the scheduler instead of sge. Cai --- 12年4月23日，周一, Louise-Amélie Schmittlouise-amelie.schm...@embl.de 写道： 发件人: Louise-Amélie Schmittlouise-amelie.schm...@embl.de 主题: [galaxy-dev] Error: Job output not returned from cluster 收件人: galaxy-dev@lists.bx.psu.edu 日期: 2012年4月23日,周一,下午5:09 Hello everyone, I'm still trying to set up the job submission as the real user, and I get a mysterious error. The job obviously runs somewhere and when it ends it is in error state and displays the following message: Job output not returned from cluster In the Galaxy log I have the following lines when the job finishes running: galaxy.jobs.runners.drmaa DEBUG 2012-04-23 10:36:41,509 (1455/9161620.pbs-master2.embl.de) state change: job finished, but failed galaxy.jobs.runners.drmaa DEBUG 2012-04-23 10:36:41,511 Job output not returned from cluster galaxy.jobs DEBUG 2012-04-23 10:36:41,547 finish(): Moved /g/funcgen/galaxy-dev/database/job_working_directory/001/1455/galaxy_dataset_2441.dat to /g/funcgen/galaxy-dev/database/files/002/dataset_2441.dat galaxy.jobs DEBUG 2012-04-23 10:36:41,755 job 1455 ended galaxy.datatypes.metadata DEBUG 2012-04-23 10:36:41,755 Cleaning up external metadata files galaxy.datatypes.metadata DEBUG 2012-04-23 10:36:41,768 Failed to cleanup MetadataTempFile temp files from /g/funcgen/galaxy-dev/database/job_working_directory/001/1455/metadata_out_HistoryDatasetAssociation_1606_npFIJM: No JSON object could be decoded: line 1 column 0 (char 0) The /g/funcgen/galaxy-dev/database/job_working_directory/001/1455/ directory is empty and /g/funcgen/galaxy-dev/database/files/002/dataset_2441.dat exists but is empty. Any ideas about what can go wrong there? Any lead would be immensely appreciated! Thanks, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Error: Job output not returned from cluster
Hello everyone, I'm still trying to set up the job submission as the real user, and I get a mysterious error. The job obviously runs somewhere and when it ends it is in error state and displays the following message: Job output not returned from cluster In the Galaxy log I have the following lines when the job finishes running: galaxy.jobs.runners.drmaa DEBUG 2012-04-23 10:36:41,509 (1455/9161620.pbs-master2.embl.de) state change: job finished, but failed galaxy.jobs.runners.drmaa DEBUG 2012-04-23 10:36:41,511 Job output not returned from cluster galaxy.jobs DEBUG 2012-04-23 10:36:41,547 finish(): Moved /g/funcgen/galaxy-dev/database/job_working_directory/001/1455/galaxy_dataset_2441.dat to /g/funcgen/galaxy-dev/database/files/002/dataset_2441.dat galaxy.jobs DEBUG 2012-04-23 10:36:41,755 job 1455 ended galaxy.datatypes.metadata DEBUG 2012-04-23 10:36:41,755 Cleaning up external metadata files galaxy.datatypes.metadata DEBUG 2012-04-23 10:36:41,768 Failed to cleanup MetadataTempFile temp files from /g/funcgen/galaxy-dev/database/job_working_directory/001/1455/metadata_out_HistoryDatasetAssociation_1606_npFIJM: No JSON object could be decoded: line 1 column 0 (char 0) The /g/funcgen/galaxy-dev/database/job_working_directory/001/1455/ directory is empty and /g/funcgen/galaxy-dev/database/files/002/dataset_2441.dat exists but is empty. Any ideas about what can go wrong there? Any lead would be immensely appreciated! Thanks, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] run job as real user error: environment variables issue?
Mmm... for some reason LD_LIBRARY_PATH was ignored but it looks like it's working fine when I set the lib path in a .conf file in /etc/ld.so.conf.d/ Hopefully it'll not break again :) Best, L-A Le 19/04/2012 17:19, Louise-Amélie Schmitt a écrit : Hi everyone, I'm currently trying to set up our local Galaxy so it can run jobs as the real user. I followed the documentation and set the galaxy user as a sudoer. However, I get an error message whenever I'm trying to run a job: galaxy.jobs.runners.drmaa ERROR 2012-04-19 14:57:48,376 Uncaught exception queueing job Traceback (most recent call last): File /g/funcgen/galaxy-dev/lib/galaxy/jobs/runners/drmaa.py, line 133, in run_next self.queue_job( obj ) File /g/funcgen/galaxy-dev/lib/galaxy/jobs/runners/drmaa.py, line 219, in queue_job job_id = self.external_runjob(filename, job_wrapper.user_system_pwent[2]).strip() File /g/funcgen/galaxy-dev/lib/galaxy/jobs/runners/drmaa.py, line 427, in external_runjob raise RuntimeError(External_runjob failed (exit code %s)\nCalled from %s:%d\nChild process reported error:\n%s % (str(exitcode), __filename__(), __lineno__(), stderrdata)) RuntimeError: External_runjob failed (exit code 127) Called from /g/funcgen/galaxy-dev/lib/galaxy/jobs/runners/drmaa.py:427 Child process reported error: python: error while loading shared libraries: libpython2.6.so.1.0: cannot open shared object file: No such file or directory Looking closely, it's the non-root user it tries to switch to that doesn't have the LD_LIBRARY_PATH properly set, so there should be an environment inheritance issue. However, I tried to print stuff from the scripts/drmaa_external_runner.py script in EVERY WAY I could think of, to no avail. As if it doesn't even run. Which is surprising since root can run python properly, so it really looks like it's really changing users. I really fail to see where the problem could come from, so if you have leads to suggest, I'll be forever grateful. Best, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] run job as real user error: environment variables issue?
Hi everyone, I'm currently trying to set up our local Galaxy so it can run jobs as the real user. I followed the documentation and set the galaxy user as a sudoer. However, I get an error message whenever I'm trying to run a job: galaxy.jobs.runners.drmaa ERROR 2012-04-19 14:57:48,376 Uncaught exception queueing job Traceback (most recent call last): File /g/funcgen/galaxy-dev/lib/galaxy/jobs/runners/drmaa.py, line 133, in run_next self.queue_job( obj ) File /g/funcgen/galaxy-dev/lib/galaxy/jobs/runners/drmaa.py, line 219, in queue_job job_id = self.external_runjob(filename, job_wrapper.user_system_pwent[2]).strip() File /g/funcgen/galaxy-dev/lib/galaxy/jobs/runners/drmaa.py, line 427, in external_runjob raise RuntimeError(External_runjob failed (exit code %s)\nCalled from %s:%d\nChild process reported error:\n%s % (str(exitcode), __filename__(), __lineno__(), stderrdata)) RuntimeError: External_runjob failed (exit code 127) Called from /g/funcgen/galaxy-dev/lib/galaxy/jobs/runners/drmaa.py:427 Child process reported error: python: error while loading shared libraries: libpython2.6.so.1.0: cannot open shared object file: No such file or directory Looking closely, it's the non-root user it tries to switch to that doesn't have the LD_LIBRARY_PATH properly set, so there should be an environment inheritance issue. However, I tried to print stuff from the scripts/drmaa_external_runner.py script in EVERY WAY I could think of, to no avail. As if it doesn't even run. Which is surprising since root can run python properly, so it really looks like it's really changing users. I really fail to see where the problem could come from, so if you have leads to suggest, I'll be forever grateful. Best, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Tool syntax: How to get history name?
Hi everyone, I as wondering if there was a way to get the history name of an input dataset within a tool (i.e. in the cheetah between the command tags), is there? Thanks, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Tool syntax: How to get history name?
Ok, after a lot of testing and searching in the libs, I found the solution: $inputdataset.dataset.history.name Le 18/04/2012 10:22, Louise-Amélie Schmitt a écrit : Hi everyone, I as wondering if there was a way to get the history name of an input dataset within a tool (i.e. in the cheetah between the command tags), is there? Thanks, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Just wanted to say...
I love the improvements you recently made to the web interface layout in galaxy-central, it's really neat! Collapsing the margins and making everything smoother makes the interface really easier to read. Thanks for the hard work, as always, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Error on local galaxy using SAM-to-BAM tool on a cluster
Hi, I'm having the same issue, has it been fixed since then? Thanks, L-A Le 07/11/2011 21:43, Nate Coraor a écrit : On Nov 4, 2011, at 1:11 PM, Carlos Borroto wrote: Hi, Reading a little more about this problem, I see Galaxy uses python tempfile library (http://docs.python.org/library/tempfile.html), specifically at line 70 in tools/samtools/sam_to_bam.py: tmp_dir = tempfile.mkdtemp() mkdtemp should honor TMPDIR, TEMP or TMP environment variables, I setup all tree of them in ~/.bashrc with no results. I'm using: default_cluster_job_runner = drmaa://-q all.q -V/ With -V I was hoping to be able to export all my environment variables, which seems to work for everything else, but not for the TMP. I ended hardcoding the dir argument, which is not good workaround, as I'm guessing this is not the only tool that will run into this problem: tmp_dir = tempfile.mkdtemp( dir='/home/cborroto/galaxy_dist/database/tmp') Any advice? In a more SGE related question, is there a way for me to debug what environment I'm getting when running Galaxy jobs? Hi Carlos, Try submitting an SGE job from the command line and having a look at the environment variables set on the execution host. Most likely, SGE is setting its own TMPDIR variable which would be overriding the value set with -V. --nate Thanks, Carlos On Fri, Nov 4, 2011 at 11:22 AM, Carlos Borroto carlos.borr...@gmail.com wrote: Hi Jen, Thanks for the quick response. The workaround you describe could work, but I might run into trouble later on. My interest is to develop a workflow for GATK, which have very strict requirements on the input BAM file. One of which is that the sorting have to be exactly the same as the reference. My reference is not sorted lexicographically chr1, chr10, chr11, , but instead is sorted karyotypically chr1, chr2, I don't think I'll be able to do this with Filter and Sort - Sort. Also GATK needs the header for the @RG tags, which I could resolve by just reintroducing the header later on, but still it will be cumbersome. I'll work on my galaxy/cluster configuration and see if I can find why the SAM-to-BAM tool is failing. Thanks again, Carlos On Thu, Nov 3, 2011 at 6:35 PM, Jennifer Jacksonj...@bx.psu.edu wrote: Hello Carlos, If what you want is a sorted SAM file, then the tool Filter and Sort - Sort may be a better choice. A SAM file is a tabular file. If there is header data at the beginning of the SAM file, it can be removed before running Sort with the tool Filter and Sort - Select (with a not matching regex). Although, you can choose to not include header output as a BWA option. Perhaps this will solve the immediate problem? Best, Jen Galaxy team On 11/3/11 12:43 PM, Carlos Borroto wrote: Hi, I'm running into this error: Error sorting alignments from (/tmp/5800600.1.all.q/tmpXOc5mD/tmpAZCzt_), When using SAM-to-BAM tool on a locally install Galaxy using a SGE cluster. I'm using the last version of galaxy-dist. I'm guessing I have a problem with the configuration for the tmp folder. I have this on universe_wsgi.ini: # Temporary files are stored in this directory. new_file_path = /home/cborroto/galaxy_dist/database/tmp But I don't see this directory being used and from the error looks like /tmp in the node is used. I wonder if this is the problem, as I don't know if there is enough space in the local /tmp directory at the nodes? I ran the same tool in a subset of the same SAM file and it ran fine. Also, I see this in the description of the tool: This tool uses the SAMTools toolkit to produce an indexed BAM file based on a sorted input SAM file. But what I actually need is to sort a SAM file output from bwa, I haven't found any other way than to converting it to BAM. Looking at sam_to_bam.py I see the BAM file will also be sorted. Would it be wrong to feed an unsorted SAM file into this tool? Finally, just to be sure there is nothing wrong with the initial SAM file, I ran samtools view ... and samtools sort ... on this file manually outside of Galaxy and it ran fine. Thanks in advance, Carlos ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Jennifer Jackson http://usegalaxy.org http://galaxyproject.org/wiki/Support ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___
Re: [galaxy-dev] add a tool to galaxy
Hi, Try: --no_overlap $singleOrPair.no_overlap Best, L-A Le 28/03/2012 18:59, cai cai a écrit : Hi All, I am trying to add a tool to Galaxy, here is my xml configuration: ## Set params based on whether reads are single-end or paired. #if $singleOrPair.readType == paired: --no_overlap $singleOrPair.readType.no_overlap #end if conditional name=singleOrPair param name=readType type=select label=single end or paired end reads option value=singlesingle end/option option value=pairedpaired end/option /param when value=paired param name=no_overlap type=select label= option value=nono/option option value=yesyes/option /param /when when value=single / /conditional I met the following error. Could some let me know what is wrong? Thank you very much. Cai Traceback (most recent call last): File /mnt/galaxyTools/galaxy-central/lib/galaxy/jobs/runners/drmaa.py, line 124, in queue_job job_wrapper.prepare() File /mnt/galaxyTools/galaxy-central/lib/galaxy/jobs/__init__.py, line 372, in prepare self.command_line = self.tool.build_command_line( param_dict ) File /mnt/galaxyTools/galaxy-central/lib/galaxy/tools/__init__.py, line 1585, in build_command_line command_line = fill_template( self.command, context=param_dict ) File /mnt/galaxyTools/galaxy-central/lib/galaxy/util/template.py, line 9, in fill_template return str( Template( source=template_text, searchList=[context] ) ) File /mnt/galaxyTools/galaxy-central/eggs/Cheetah-2.2.2-py2.6-linux-x86_64-ucs4.egg/Cheetah/Template.py, line 1004, in __str__ return getattr(self, mainMethName)() File cheetah_DynamicallyCompiledCheetahTemplate_1332953248_1_41760.py, line 96, in respond *NotFound: cannot find 'no_overlap' while searching for 'singleOrPair.readType.no_overlap'* ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] drmaa module does not load
Le 27/03/2012 11:03, Louise-Amélie Schmitt a écrit : Le 26/03/2012 16:13, Nate Coraor a écrit : On Mar 26, 2012, at 5:11 AM, Louise-Amélie Schmitt wrote: Hello everyone, I wanted to start the drmaa job runner and followed the instructions in the wiki, but I have this error message when I start Galaxy: galaxy.jobs ERROR 2012-03-23 15:28:49,845 Job runner is not loadable: galaxy.jobs.runners. drmaa Traceback (most recent call last): File /g/funcgen/galaxy/lib/galaxy/jobs/__init__.py, line 1195, in _load_plugin module = __import__( module_name ) ImportError: No module named drmaa I checked /g/funcgen/galaxy/lib/galaxy/jobs/runners and it contains the drmaa.py file There was no drmaa egg so I made a copy of it from our other Galaxy install but it didn't solve the problem. I don't really know where to start looking, any idea? Thanks, L-A Hi L-A, There's an errant space in the runner name: ' drmaa'. I am going to guess that your start_job_runners looks like: start_job_runners = pbs, drmaa Only the whitespace at the beginning and end of that parameter is stripped. I've committed a fix for this that'll be in the next galaxy-dist, but in the meantime, remove the space after the comma and the drmaa runner should load. --nate Hi Nate, After a removing the space (and a facepalm) it now fetches the drmaa egg and starts the module properly, thanks a lot! I still have an issue though: When I use the run shell script Galaxy crashes right before loading the drmaa runner (right after the pbs runner is loaded). The weird thing is that when I launch the command manually it works fine: python ./scripts/paster.py serve universe_wsgi.runner.ini --server-name=runner0 --pid-file=runner0.pid --log-file=runner0.log --daemon The other weird thing is that I get no error message at all. I'll try looking into it but if you have any idea about what's going wrong, it would help greatly :) Thanks again, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ Ok it looks like I didn't notice the changes in the start scripts and since we use two custom copies of run.sh the code was outdated. I corrected that and it now seems to work properly. Best, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] [request] drmaa way too chatty
Hi everyone, The drmaa runner literally floods the Galaxy output with its own output (dozens of lines every second) which makes the Galaxy log impossible to read. Would there be a way to separate the two of them? I tried to look into the code but I'm not fully sure about what exactly produces this humongous output so I'm walking in the dark. Thanks, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] drmaa module does not load
Hello everyone, I wanted to start the drmaa job runner and followed the instructions in the wiki, but I have this error message when I start Galaxy: galaxy.jobs ERROR 2012-03-23 15:28:49,845 Job runner is not loadable: galaxy.jobs.runners. drmaa Traceback (most recent call last): File /g/funcgen/galaxy/lib/galaxy/jobs/__init__.py, line 1195, in _load_plugin module = __import__( module_name ) ImportError: No module named drmaa I checked /g/funcgen/galaxy/lib/galaxy/jobs/runners and it contains the drmaa.py file There was no drmaa egg so I made a copy of it from our other Galaxy install but it didn't solve the problem. I don't really know where to start looking, any idea? Thanks, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] NFS, Cluster and working directories questions
Hi Nate, and thanks for the reply - We saw the outputs_to_working_directory option in the .ini file, but it only concerns output files, is there a way to make a local copy of all the input files in the job working directory? (including indices) Hi L-A, Unfortunately no. This option was only created so that you could mount the file_path directory read-only on your cluster nodes for security purposes. In the end it shouldn't be necessary. The guy in charge of the cluster said so. - Can we set the job working directory as an absolute path so it's local to the nodes, like /tmp ? No, things like the external metadata inputs and outputs are written to the working directory and are expected to be available on both the cluster and the application server at the same path. Ok, I see. How do you manage that then? Is it actually not an issue I/O-wise? - If it's possible, will the job working directories created for each job be cleaned properly at the end? Yes, depending on the value of cleanup_job in the config file. Enhancements that remove this shared directory limitation are on our roadmap for this year. Ok, thanks a lot for all this information! Best, L-A --nate Thanks, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] NFS, Cluster and working directories questions
Hello, We're currently trying to switch to a big cluster but we have a lot of doubts and questions, especially since the I/O is a serious issue for our NFS. - We saw the outputs_to_working_directory option in the .ini file, but it only concerns output files, is there a way to make a local copy of all the input files in the job working directory? (including indices) - Can we set the job working directory as an absolute path so it's local to the nodes, like /tmp ? - If it's possible, will the job working directories created for each job be cleaned properly at the end? Thanks, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] [API] allow_user_impersonation
Hello everyone! I just wanted to know if the user impersonation will be available through the API someday :) It could be veeery useful for triggering automatic QA on the data without having to share the resulting histories afterwards. Best regards, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] universe_wsgi.ini
Hi Daniel, I have to admit it took me some time and a lot of trial and error to figure out how the .loc files work :) I'm glad it works for you now. I'm not fully sure about what exactly you want to do but here is what I think could be of help: If your Galaxy runs on the same machine your files are stored in, or if the place they are stored is mounted on your Galaxy's machine, don't bother with the upload tool in the analysis panel nor the Shared Data section, use the Admin section Manage data libraries so you can load files from giving Galaxy a filesystem path. This way you can ask it to load the data without moving/copying it. (it's also superfast) Hope that helps, L-A Le 24/02/2012 19:25, Brami, Daniel a écrit : Hi L-A, Thank you for the reply. Another kind member (thanks Dan Dorset!) also took the time to explain me these over the phone this morning. The missing piece of information, which you have included in your instructions, and which was omitted in the documentation, is the link between the bowtie_indices.loc and the builds.txt file, which is never mentioned anywhere as far as I know. As for the file size limit, it's more of a question regarding the understanding of how Galaxy works. In my local copy of Galaxy running on Unix, given an input file using the complete path, does Galaxy need to copy the input data through the browser to the 'tmp' and output directories or does it use a system call. Can this be made into a link to save space and time? I really do appreciate the help -- thanks! Daniel *From:*galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] *On Behalf Of *Louise-Amélie Schmitt *Sent:* Friday, February 24, 2012 12:23 AM *To:* galaxy-dev@lists.bx.psu.edu *Subject:* Re: [galaxy-dev] universe_wsgi.ini Hello Daniel, I had similar issues when setting up our own. The POST size limit is not set by Galaxy. If I remember correctly it's in your Apache config, though I don't remember exactly where it was. For the .loc files, here is an example: dm3-btdm3D.melanogaster 3 /path/to/indexes/bowtie/d_melanogaster_fb5_22 The items are as follows: - dm3-bt is a unique ID for this index. It can be whatever you want but it needs to be unique in this file. - dm3 is the build ID that you set in the tool-data/shared/ucsc/builds.txt file. It's the first item there. It's there to tell what genome build your index was generated from. Make sure that the two of them correspond. - D.melanogaster 3 is the display name that you want to have in the widgets (can have spaces but no tabs) - /path/to/indexes/bowtie/d_melanogaster_fb5_22 is the full path of the index files without the final suffixes (the first part before the first dot) Each line's items must be separated by an actual tab character. Depending on your editor, you might want to make sure the tab key doesn't print like 4 spaces instead of \t. Hope that helps Best, L-A Le 23/02/2012 19:47, Brami, Daniel a écrit : I have been struggling with setting up my local Galaxy instance for the past two days. Can anyone post a copy of their universe_wsgi.ini file , with the sensitive info commented / removed? I need an instance that can handle local files greater than 2 GB (ie: does not go through POST method of browser to get data) and has permissions to use pre-built bowtie indices. Also, a FUNCTIONNAL copy of the 'bowtie_indices.loc' would be very helpful. (none of my prebuilt databases are appearing in drop down menu). Please assist. Daniel Brami Synthetic Genomics, Inc. Senior Research Associate, Bioinformatics 11149 North Torrey Pines Road La Jolla, California 92037 Phone: 858.433.2230 Fax: 858.754.2988 dbr...@syntheticgenomics.com mailto:dbr...@syntheticgenomics.com www.SyntheticGenomics.com http://www.syntheticgenomics.com/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] universe_wsgi.ini
Hello Daniel, I had similar issues when setting up our own. The POST size limit is not set by Galaxy. If I remember correctly it's in your Apache config, though I don't remember exactly where it was. For the .loc files, here is an example: dm3-btdm3D.melanogaster 3 /path/to/indexes/bowtie/d_melanogaster_fb5_22 The items are as follows: - dm3-bt is a unique ID for this index. It can be whatever you want but it needs to be unique in this file. - dm3 is the build ID that you set in the tool-data/shared/ucsc/builds.txt file. It's the first item there. It's there to tell what genome build your index was generated from. Make sure that the two of them correspond. - D.melanogaster 3 is the display name that you want to have in the widgets (can have spaces but no tabs) - /path/to/indexes/bowtie/d_melanogaster_fb5_22 is the full path of the index files without the final suffixes (the first part before the first dot) Each line's items must be separated by an actual tab character. Depending on your editor, you might want to make sure the tab key doesn't print like 4 spaces instead of \t. Hope that helps Best, L-A Le 23/02/2012 19:47, Brami, Daniel a écrit : I have been struggling with setting up my local Galaxy instance for the past two days. Can anyone post a copy of their universe_wsgi.ini file , with the sensitive info commented / removed? I need an instance that can handle local files greater than 2 GB (ie: does not go through POST method of browser to get data) and has permissions to use pre-built bowtie indices. Also, a FUNCTIONNAL copy of the 'bowtie_indices.loc' would be very helpful. (none of my prebuilt databases are appearing in drop down menu). Please assist. Daniel Brami Synthetic Genomics, Inc. Senior Research Associate, Bioinformatics 11149 North Torrey Pines Road La Jolla, California 92037 Phone: 858.433.2230 Fax: 858.754.2988 dbr...@syntheticgenomics.com mailto:dbr...@syntheticgenomics.com www.SyntheticGenomics.com http://www.syntheticgenomics.com/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Switching TorquePBSpro: qsub error 111 (cannot connect to server)
You don't need a full PBS server on that machine, just the client libraries and configuration. i.e. if you can run 'qstat' from machine A and have it return the queue on machine B, that should be all you need. We asked the IT guys they did just that :) now qsub works fine from machine A! But I still have a problem: They have no libdrmaa.so anywhere! What should I do? If I understood correctly it's part of the Fedstage DRMAA service provider, will it be enough installing it on our Galaxy server? Torque's syntax allows you to specify the server name right in the runner URL, but pbs_python links against libtorque, which is part of the torque client, which must be installed somewhere on the local system. Ok, thanks a lot! Best, L-A Best, L-A Le 30/01/2012 18:20, Nate Coraor a écrit : On Jan 30, 2012, at 12:07 PM, Louise-Amélie Schmitt wrote: Hi Nate, Thanks for the leads! But setting DRMAA_LIBRARY_PATH means I'm in trouble since the libraries are on machine B which is maintained by our IT dept. I cannot access them from machine A. Is it a desperate situation? Will it work if I have a copy of those libs somewhere? :/ Hi L-A, The Galaxy server will need to be a submission host, so I believe it will have to have PBS Pro installed. If it has this, then the FedStage DRMAA library should be installable on the same host. It may be possible copy the libraries, although I don't know whether you'd be able to configure the server address without access to the directories in which the library will look for its configuration. --nate Best, L-A Le 30/01/2012 17:18, Nate Coraor a écrit : On Jan 30, 2012, at 7:37 AM, Louise-Amélie Schmitt wrote: Hi Nate, Thanks for the info! I'm trying to understand how the URL for DRMAA works but I don't understand how we can set it so it uses a different machine. Our Galaxy runner runs on machine A and the cluster is on machine B, where do I put B in the URL? In the wiki there is this example: drmaa://[native_options]/ I'm a bit confused, I would have expected something like: drmaa://[machine]/[native_options]/ like for TORQUE. Did I miss something? Hi L-A, Hrm, I've only used it with SGE, which uses an environment variable to define the cell location, and LSF, which I don't remember, but I assume it used the default. I think if you configure the PBS Pro client on the submission host and point DRMAA_LIBRARY_PATH at the correct libdrmaa, it will use your client configuration. There are other PBS Pro users on the list who can hopefully chime in with more details. --nate Best, L-A Le 19/01/2012 19:43, Nate Coraor a écrit : On Jan 16, 2012, at 5:22 AM, Louise-Amélie Schmitt wrote: Hello, We want to move Galaxy's jobs from our small TORQUE local install to a big cluster running PBS Pro. In the universe_wsgi.ini, I changed the cluster address as follows: default_cluster_job_runner = pbs:/// to: default_cluster_job_runner = pbs://sub-master/clng_new/ where sub-master is the name of the machine and clng_new is the queue. However, I get an error when trying to run any job: galaxy.jobs.runners.pbs ERROR 2012-01-16 11:10:00,894 Connection to PBS server for submit failed: 111: Could not find a text for this error, uhhh This corresponds to the qsub error 111 (Cannot connect to specified server host) which is, for some reason, caught by pbs_python as an error of its own (111 not corresponding to any pbs_python error code, hence the face-plant-message). Our guess is that we might need to re-scramble the pbs_python egg with PBS pro's libraries, is that correct? If it's the case, what do we have to set as LIBTORQUE_DIR? Hi L-A, pbs_python is only designed for TORQUE, I don't think it is compatible with the PBS Pro API. For that, you need to use the drmaa runner, which uses the FedStage libdrmaa for PBS Pro. --nate Thanks, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Galaxy and Cluster options
Hi Tanguy, You can set that at the very bottom of your universe_wsgi.ini file. I did it myself with Torque to set a different behavior for a couple of tools, it works fine. The related Wiki page is here: http://wiki.g2.bx.psu.edu/Admin/Config/Performance/Cluster :) Best, L-A Le 17/02/2012 09:43, Tanguy LE CARROUR a écrit : Hi! Is there a way to specify parameters for the job submission on the cluster from the tool configuration file? Let's say I want tool A to be submitted with -pe smp 8 but tool B submitted with -l mem_free=500. Is it already supported? I could not find it in the wiki. ... or do I have to start digging in the code and try to implement it?! ^_^' Best regards, Tanguy ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Switching TorquePBSpro: qsub error 111 (cannot connect to server)
Thanks a lot Chris, actually we found another place where we found rpms for PBS Pro: http://apps.man.poznan.pl/trac/pbs-drmaa And it works now! :D Have a great weekend, L-A Le 17/02/2012 17:07, Fields, Christopher J a écrit : If they used a package install, I think the DRMAA libs for torque are normally separate. For instance: http://pkgs.org/download/torque-drmaa If going from source, you can enable DRMAA compilation with --enable-drmaa; I don't recall if that is on by default (I don't think it is). chris On Feb 17, 2012, at 3:49 AM, Louise-Amélie Schmitt wrote: You don't need a full PBS server on that machine, just the client libraries and configuration. i.e. if you can run 'qstat' from machine A and have it return the queue on machine B, that should be all you need. We asked the IT guys they did just that :) now qsub works fine from machine A! But I still have a problem: They have no libdrmaa.so anywhere! What should I do? If I understood correctly it's part of the Fedstage DRMAA service provider, will it be enough installing it on our Galaxy server? Torque's syntax allows you to specify the server name right in the runner URL, but pbs_python links against libtorque, which is part of the torque client, which must be installed somewhere on the local system. Ok, thanks a lot! Best, L-A Best, L-A Le 30/01/2012 18:20, Nate Coraor a écrit : On Jan 30, 2012, at 12:07 PM, Louise-Amélie Schmitt wrote: Hi Nate, Thanks for the leads! But setting DRMAA_LIBRARY_PATH means I'm in trouble since the libraries are on machine B which is maintained by our IT dept. I cannot access them from machine A. Is it a desperate situation? Will it work if I have a copy of those libs somewhere? :/ Hi L-A, The Galaxy server will need to be a submission host, so I believe it will have to have PBS Pro installed. If it has this, then the FedStage DRMAA library should be installable on the same host. It may be possible copy the libraries, although I don't know whether you'd be able to configure the server address without access to the directories in which the library will look for its configuration. --nate Best, L-A Le 30/01/2012 17:18, Nate Coraor a écrit : On Jan 30, 2012, at 7:37 AM, Louise-Amélie Schmitt wrote: Hi Nate, Thanks for the info! I'm trying to understand how the URL for DRMAA works but I don't understand how we can set it so it uses a different machine. Our Galaxy runner runs on machine A and the cluster is on machine B, where do I put B in the URL? In the wiki there is this example: drmaa://[native_options]/ I'm a bit confused, I would have expected something like: drmaa://[machine]/[native_options]/ like for TORQUE. Did I miss something? Hi L-A, Hrm, I've only used it with SGE, which uses an environment variable to define the cell location, and LSF, which I don't remember, but I assume it used the default. I think if you configure the PBS Pro client on the submission host and point DRMAA_LIBRARY_PATH at the correct libdrmaa, it will use your client configuration. There are other PBS Pro users on the list who can hopefully chime in with more details. --nate Best, L-A Le 19/01/2012 19:43, Nate Coraor a écrit : On Jan 16, 2012, at 5:22 AM, Louise-Amélie Schmitt wrote: Hello, We want to move Galaxy's jobs from our small TORQUE local install to a big cluster running PBS Pro. In the universe_wsgi.ini, I changed the cluster address as follows: default_cluster_job_runner = pbs:/// to: default_cluster_job_runner = pbs://sub-master/clng_new/ where sub-master is the name of the machine and clng_new is the queue. However, I get an error when trying to run any job: galaxy.jobs.runners.pbs ERROR 2012-01-16 11:10:00,894 Connection to PBS server for submit failed: 111: Could not find a text for this error, uhhh This corresponds to the qsub error 111 (Cannot connect to specified server host) which is, for some reason, caught by pbs_python as an error of its own (111 not corresponding to any pbs_python error code, hence the face-plant-message). Our guess is that we might need to re-scramble the pbs_python egg with PBS pro's libraries, is that correct? If it's the case, what do we have to set as LIBTORQUE_DIR? Hi L-A, pbs_python is only designed for TORQUE, I don't think it is compatible with the PBS Pro API. For that, you need to use the drmaa runner, which uses the FedStage libdrmaa for PBS Pro. --nate Thanks, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu
Re: [galaxy-dev] Syntax issue for adding new tool
Hello,
I'm not sure this will fix the issue but you might have to use
-p $param_file.params
instead of
-p $params
in your command line.
Best,
L-A
Le 06/02/2012 22:15, Jeffrey Long a écrit :
Hi all,
I am having trouble with what looked to me at first to be a simple
syntactical issue with writing the XML wrapper for a new Galaxy tool.
I had thought this would be easy, but perhaps I am missing something
incredibly simple, as I am having no success at all. The relevant
parts look something like this:
command interpreter=sh
mycommand.sh
-i $input
#if $param_file.use_params == true#
-p $params
#end if
/command
inputs
param format=fasta name=input type=data label=Source File/
conditional name=param_file
param name=use_params type=select label=Use custom parameter file?
option value=trueYes/option
option value=false selected=trueNo/option
/param
when value=true
param format=text name=params type=data label=Parameter File
value=/
/when
/conditional
/inputs
It's very simple, I just want the line -p $params to appear in the
command line when the parameter use_params parameter is set to true.
However, when building the command line statement to run, I get this
error:
--
Traceback (most recent call last):
File
/Users/jrl909/Galaxy/galaxy-dist/lib/galaxy/jobs/runners/local.py,
line 59, in run_job
job_wrapper.prepare()
File /Users/jrl909/Galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py,
line 424, in prepare
self.command_line = self.tool.build_command_line( param_dict )
File
/Users/jrl909/Galaxy/galaxy-dist/lib/galaxy/tools/__init__.py, line
1827, in build_command_line
command_line = fill_template( self.command, context=param_dict )
File /Users/jrl909/Galaxy/galaxy-dist/lib/galaxy/util/template.py,
line 9, in fill_template
return str( Template( source=template_text, searchList=[context] ) )
File
/Users/jrl909/Galaxy/galaxy-dist/eggs/Cheetah-2.2.2-py2.7-macosx-10.6-intel-ucs2.egg/Cheetah/Template.py,
line 1004, in __str__
return getattr(self, mainMethName)()
File
cheetah_DynamicallyCompiledCheetahTemplate_1328561167_73_97196.py,
line 84, in respond
NotFound: cannot find 'use_params' while searching for
'param_file.use_params'
michael
---
As far as I can tell, the syntax I've used here is the same as that
found in the example file suggested by the Galaxy documentation:
interval2maf.xml (what I see as the relevant sections being
reproduced below):
---
command interpreter=python
#if $maf_source_type.maf_source == user #interval2maf.py
--dbkey=${input1.dbkey} --chromCol=${input1.metadata.chromCol}
--startCol=${input1.metadata.startCol}
--endCol=${input1.metadata.endCol}
--strandCol=${input1.metadata.strandCol}
--mafFile=$maf_source_type.mafFile
--mafIndex=$maf_source_type.mafFile.metadata.maf_index
--interval_file=$input1 --output_file=$out_file1
--mafIndexFile=${GALAXY_DATA_INDEX_DIR}/maf_index.loc
--species=$maf_source_type.species
#else #interval2maf.py
--dbkey=${input1.dbkey} --chromCol=${input1.metadata.chromCol}
--startCol=${input1.metadata.startCol}
--endCol=${input1.metadata.endCol}
--strandCol=${input1.metadata.strandCol}
--mafType=$maf_source_type.mafType --interval_file=$input1
--output_file=$out_file1
--mafIndexFile=${GALAXY_DATA_INDEX_DIR}/maf_index.loc
--species=$maf_source_type.species
#end if#
--split_blocks_by_species=$split_blocks_by_species_selector.split_blocks_by_species
#if $split_blocks_by_species_selector.split_blocks_by_species ==
split_blocks_by_species#
--remove_all_gap_columns=$split_blocks_by_species_selector.remove_all_gap_columns
#end if
/command
inputs
...
blahblahblah
...
conditional name=split_blocks_by_species_selector
param name=split_blocks_by_speciestype=selectlabel=Split blocks
by specieshelp=Not usually applicable. See help below for more
information.
option value=split_blocks_by_speciesSplit by species/option
option value=dont_split_blocks_by_species selected=trueDo not
split/option
/param
when value=dont_split_blocks_by_species
!-- do nothing here --
/when
when value=split_blocks_by_species
param name=remove_all_gap_columnstype=selectlabel=Collapse empty
alignment columns
option value=remove_all_gap_columns selected=trueCollapse empty
columns/option
option value=do_not_remove_all_gap_columnsDo not collapse/option
/param
/when
/conditional
/inputs
---
As you can see, the field 'split_blocks_by_species' is referenced in
an #if-block in the command specification the same way my 'use_params'
field is.
I would like very much to know the apparently crucial difference that
I'm missing!
Thanks,
-Jeff
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please
Re: [galaxy-dev] Looking for recommendations: How to run galaxy workflows in batch
Hello Dannon Could it be possible to have the input dataset's display name appended to the new history's name instead of plain numbers when the Send results in a new history option is checked? This new feature is indeed very useful (thanks a million for it) but the numbered suffixes make it hard to track what new history belongs to which dataset. Thanks, L-A Le 06/02/2012 23:00, Dannon Baker a écrit : This method only works for single inputs at the moment, though eventually it'd be nice to allow pairing. Another option for you would be to use the workflows API, with which you can definitely specify multiple inputs. See workflow_execute.py in the scripts/api folder of your galaxy installation for one method of doing this. -Dannon On Feb 6, 2012, at 4:53 PM, Dave Lin wrote: Thank you Dannon. That is helpful. What if I need to specify multiple inputs per run (i.e. .csfasta + .qual file)? -Dave On Mon, Feb 6, 2012 at 1:27 PM, Dannon Bakerdannonba...@me.com wrote: Hi Dave, Yes, galaxy's standard run-workflow dialog has a feature where you can select multiple datasets as input for a single Input Dataset step. To do this, click the icon referenced by the tooltip in the screenshot below to select multiple files. All parameters remain static between executions except for the single input dataset that gets modified for each run, and that only one input dataset can be set to multiple files in this fashion. -Dannon PastedGraphic-2.png On Feb 6, 2012, at 4:18 PM, Dave Lin wrote: Hi All, I'm looking to batch process 40 large data sets with the same galaxy workflow. This obviously can be done in a brute-force manual manner. However, is there a better way to schedule/invoke these jobs in batch 1) from the UI with a plugin 2) command-line 3) web-service Thanks in advance for any pointers. Dave ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Switching TorquePBSpro: qsub error 111 (cannot connect to server)
Hi Nate, Yeah, looks like I have no choice... I'll try to negociate with the IT guys to run the runner on machine B. Thanks for all the information! BTW, out of curiosity, is submitting jobs across machines possible with Torque? Best, L-A Le 30/01/2012 18:20, Nate Coraor a écrit : On Jan 30, 2012, at 12:07 PM, Louise-Amélie Schmitt wrote: Hi Nate, Thanks for the leads! But setting DRMAA_LIBRARY_PATH means I'm in trouble since the libraries are on machine B which is maintained by our IT dept. I cannot access them from machine A. Is it a desperate situation? Will it work if I have a copy of those libs somewhere? :/ Hi L-A, The Galaxy server will need to be a submission host, so I believe it will have to have PBS Pro installed. If it has this, then the FedStage DRMAA library should be installable on the same host. It may be possible copy the libraries, although I don't know whether you'd be able to configure the server address without access to the directories in which the library will look for its configuration. --nate Best, L-A Le 30/01/2012 17:18, Nate Coraor a écrit : On Jan 30, 2012, at 7:37 AM, Louise-Amélie Schmitt wrote: Hi Nate, Thanks for the info! I'm trying to understand how the URL for DRMAA works but I don't understand how we can set it so it uses a different machine. Our Galaxy runner runs on machine A and the cluster is on machine B, where do I put B in the URL? In the wiki there is this example: drmaa://[native_options]/ I'm a bit confused, I would have expected something like: drmaa://[machine]/[native_options]/ like for TORQUE. Did I miss something? Hi L-A, Hrm, I've only used it with SGE, which uses an environment variable to define the cell location, and LSF, which I don't remember, but I assume it used the default. I think if you configure the PBS Pro client on the submission host and point DRMAA_LIBRARY_PATH at the correct libdrmaa, it will use your client configuration. There are other PBS Pro users on the list who can hopefully chime in with more details. --nate Best, L-A Le 19/01/2012 19:43, Nate Coraor a écrit : On Jan 16, 2012, at 5:22 AM, Louise-Amélie Schmitt wrote: Hello, We want to move Galaxy's jobs from our small TORQUE local install to a big cluster running PBS Pro. In the universe_wsgi.ini, I changed the cluster address as follows: default_cluster_job_runner = pbs:/// to: default_cluster_job_runner = pbs://sub-master/clng_new/ where sub-master is the name of the machine and clng_new is the queue. However, I get an error when trying to run any job: galaxy.jobs.runners.pbs ERROR 2012-01-16 11:10:00,894 Connection to PBS server for submit failed: 111: Could not find a text for this error, uhhh This corresponds to the qsub error 111 (Cannot connect to specified server host) which is, for some reason, caught by pbs_python as an error of its own (111 not corresponding to any pbs_python error code, hence the face-plant-message). Our guess is that we might need to re-scramble the pbs_python egg with PBS pro's libraries, is that correct? If it's the case, what do we have to set as LIBTORQUE_DIR? Hi L-A, pbs_python is only designed for TORQUE, I don't think it is compatible with the PBS Pro API. For that, you need to use the drmaa runner, which uses the FedStage libdrmaa for PBS Pro. --nate Thanks, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Switching TorquePBSpro: qsub error 111 (cannot connect to server)
Hi Nate, Thanks for the info! I'm trying to understand how the URL for DRMAA works but I don't understand how we can set it so it uses a different machine. Our Galaxy runner runs on machine A and the cluster is on machine B, where do I put B in the URL? In the wiki there is this example: drmaa://[native_options]/ I'm a bit confused, I would have expected something like: drmaa://[machine]/[native_options]/ like for TORQUE. Did I miss something? Best, L-A Le 19/01/2012 19:43, Nate Coraor a écrit : On Jan 16, 2012, at 5:22 AM, Louise-Amélie Schmitt wrote: Hello, We want to move Galaxy's jobs from our small TORQUE local install to a big cluster running PBS Pro. In the universe_wsgi.ini, I changed the cluster address as follows: default_cluster_job_runner = pbs:/// to: default_cluster_job_runner = pbs://sub-master/clng_new/ where sub-master is the name of the machine and clng_new is the queue. However, I get an error when trying to run any job: galaxy.jobs.runners.pbs ERROR 2012-01-16 11:10:00,894 Connection to PBS server for submit failed: 111: Could not find a text for this error, uhhh This corresponds to the qsub error 111 (Cannot connect to specified server host) which is, for some reason, caught by pbs_python as an error of its own (111 not corresponding to any pbs_python error code, hence the face-plant-message). Our guess is that we might need to re-scramble the pbs_python egg with PBS pro's libraries, is that correct? If it's the case, what do we have to set as LIBTORQUE_DIR? Hi L-A, pbs_python is only designed for TORQUE, I don't think it is compatible with the PBS Pro API. For that, you need to use the drmaa runner, which uses the FedStage libdrmaa for PBS Pro. --nate Thanks, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Error download composite dataset from history
Hi Ross Thanks a million, I hg-pull'ed it solved the problem!! Please push it to galaxy-dist? ;) I ran in another little bug though, but nothing serious, you might even have already spotted it by now: In lib/galaxy/web/framework/__init__.py the line 873 made the whole thing crash due to a wrong arguments number, so I changed it from: self._ensure_logged_in_user( environ ) to: self._ensure_logged_in_user( environ, session_cookie ) and it works fine now! Best, L-A Le 19/01/2012 02:13, Ross a écrit : Hi, L-A Thanks for reporting this - I just pushed a fix in galaxy-central 6556:ab1a5ab063ce Please let us know if that resolves the problem? 2012/1/19 Louise-Amélie Schmittlouise-amelie.schm...@embl.de: Hello! I'm running into an error when trying to download a composite dataset from the history with the floppy disk icon. Here is the error message in the logs: galaxy.web.controllers.dataset ERROR 2012-01-18 16:32:10,324 Unable to remove temporary library download archive and directory Traceback (most recent call last): File /g/funcgen/galaxy-dev/lib/galaxy/web/controllers/dataset.py, line 285, in archive_composite_dataset os.rmdir( tmpd ) OSError: [Errno 39] Directory not empty: './database/tmp/tmpkRv28I' Then I tried replacing the os.rmdir by a shutil.rmtree to see if it would solve the issue, and here is what I got: galaxy.web.controllers.dataset ERROR 2012-01-18 16:34:30,239 Unable to remove temporary library download archive and directory Traceback (most recent call last): File /g/funcgen/galaxy-dev/lib/galaxy/web/controllers/dataset.py, line 286, in archive_composite_dataset shutil.rmtree( tmpd ) File /g/steinmetz/collaboration/software/CentOS5/opt/Python-2.6.5/lib/python2.6/shutil.py, line 221, in rmtree onerror(os.remove, fullname, sys.exc_info()) File /g/steinmetz/collaboration/software/CentOS5/opt/Python-2.6.5/lib/python2.6/shutil.py, line 219, in rmtree os.remove(fullname) OSError: [Errno 16] Device or resource busy: './database/tmp/tmpwpsShD/.nfs0289ccea0138' But when I check the contents of the direcory, there is no hidden .nfs-something file. Does anyone have an idea why this could occur? Both our testing instance and our production instance have the same issue, the first being on galaxy-central and the second on galaxy-dist. Thanks, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Error download composite dataset from history
Hello! I'm running into an error when trying to download a composite dataset from the history with the floppy disk icon. Here is the error message in the logs: galaxy.web.controllers.dataset ERROR 2012-01-18 16:32:10,324 Unable to remove temporary library download archive and directory Traceback (most recent call last): File /g/funcgen/galaxy-dev/lib/galaxy/web/controllers/dataset.py, line 285, in archive_composite_dataset os.rmdir( tmpd ) OSError: [Errno 39] Directory not empty: './database/tmp/tmpkRv28I' Then I tried replacing the os.rmdir by a shutil.rmtree to see if it would solve the issue, and here is what I got: galaxy.web.controllers.dataset ERROR 2012-01-18 16:34:30,239 Unable to remove temporary library download archive and directory Traceback (most recent call last): File /g/funcgen/galaxy-dev/lib/galaxy/web/controllers/dataset.py, line 286, in archive_composite_dataset shutil.rmtree( tmpd ) File /g/steinmetz/collaboration/software/CentOS5/opt/Python-2.6.5/lib/python2.6/shutil.py, line 221, in rmtree onerror(os.remove, fullname, sys.exc_info()) File /g/steinmetz/collaboration/software/CentOS5/opt/Python-2.6.5/lib/python2.6/shutil.py, line 219, in rmtree os.remove(fullname) OSError: [Errno 16] Device or resource busy: './database/tmp/tmpwpsShD/.nfs0289ccea0138' But when I check the contents of the direcory, there is no hidden .nfs-something file. Does anyone have an idea why this could occur? Both our testing instance and our production instance have the same issue, the first being on galaxy-central and the second on galaxy-dist. Thanks, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Switching TorquePBSpro: qsub error 111 (cannot connect to server)
Hello, We want to move Galaxy's jobs from our small TORQUE local install to a big cluster running PBS Pro. In the universe_wsgi.ini, I changed the cluster address as follows: default_cluster_job_runner = pbs:/// to: default_cluster_job_runner = pbs://sub-master/clng_new/ where sub-master is the name of the machine and clng_new is the queue. However, I get an error when trying to run any job: galaxy.jobs.runners.pbs ERROR 2012-01-16 11:10:00,894 Connection to PBS server for submit failed: 111: Could not find a text for this error, uhhh This corresponds to the qsub error 111 (Cannot connect to specified server host) which is, for some reason, caught by pbs_python as an error of its own (111 not corresponding to any pbs_python error code, hence the face-plant-message). Our guess is that we might need to re-scramble the pbs_python egg with PBS pro's libraries, is that correct? If it's the case, what do we have to set as LIBTORQUE_DIR? Thanks, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Tool creation: how to restrict input dataset selector to appropriate file types?
Hello, I'm running in a strange issue which I cannot seem to find a solution for. In every tool I created from scratch, the input dataset selector does not only include the datasets corresponding to the file types I set in the format= attribute in the param tag in my XML tool file, but instead it includes the whole history contents. I searched in the wiki's tool syntax section to no avail. I believe I stupidly missed something huge there but I'm really confused. What should I do to have the selector's contents appropriate? Here is one of the tools for which I have this issue: http://pastebin.com/XqvSwtMH Thanks, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Tool creation: how to restrict input dataset selector to appropriate file types?
Hi Dannon, and thanks for the answer. Problem solved: The Export class had somehow disappeared from the tabular.py file, most likely during a failed merge. Yeah, I indeed stupidly missed something huge... Sorry for wasting your time! Best, L-A Le 07/12/2011 12:35, Dannon Baker a écrit : The only thing that sticks out to me is the 'export' format listed in inputs. What are the children datatypes of export, or how is that set up? The filter logic automatically includes all children datatypes of the specified formats. -Dannon On Dec 7, 2011, at 5:10 AM, Louise-Amélie Schmitt wrote: Hello, I'm running in a strange issue which I cannot seem to find a solution for. In every tool I created from scratch, the input dataset selector does not only include the datasets corresponding to the file types I set in the format= attribute in theparam tag in my XML tool file, but instead it includes the whole history contents. I searched in the wiki's tool syntax section to no avail. I believe I stupidly missed something huge there but I'm really confused. What should I do to have the selector's contents appropriate? Here is one of the tools for which I have this issue: http://pastebin.com/XqvSwtMH Thanks, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] About database indexes
Hello, Since our local Galaxy is a little sluggish, we were wondering if it could be related to broken or missing database indexes (we're using PostgreSQL), so we would like to know how they are managed in Galaxy, and if there is a way to restore them automatically. Thanks, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Web links not leading to appropriate location anymore.
Hello, We're experiencing a major issue in the web application and don't know where to look for a potential solution: Whichever link we click on it always brigs us back to the welcome page (galaxy/root). There are a few exceptions to that: - saved histories, saved datasets and api key in the User menu that lead to the appropriate pages - the tools on the left: even better, they open an new full set of Galaxy's frames in the middle frame. We tried reverting to the last pull, restarting Apache, checking the Apache config file, checking the rights in the static directory... and we still do not have the least idea of where to look. Does anyone have an idea, a lead to explore? Thanks, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Web links not leading to appropriate location anymore.
Thanks a lot Ross, it works fine now. Best, L-A Le 30/11/2011 11:34, Ross a écrit : Louise, I had the same problem. After hg revert -r 3ee9430186fb everything was back to normal for me. I hope this helps.. 2011/11/30 Louise-Amélie Schmitt louise-amelie.schm...@embl.de mailto:louise-amelie.schm...@embl.de Hello, We're experiencing a major issue in the web application and don't know where to look for a potential solution: Whichever link we click on it always brigs us back to the welcome page (galaxy/root). There are a few exceptions to that: - saved histories, saved datasets and api key in the User menu that lead to the appropriate pages - the tools on the left: even better, they open an new full set of Galaxy's frames in the middle frame. We tried reverting to the last pull, restarting Apache, checking the Apache config file, checking the rights in the static directory... and we still do not have the least idea of where to look. Does anyone have an idea, a lead to explore? Thanks, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Ross Lazarus MBBS MPH; Associate Professor, Harvard Medical School; Head, Medical Bioinformatics, BakerIDI; Tel: +61 385321444; ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Web links not leading to appropriate location anymore.
It works fine now, thanks for the fix! :) Best, L-A Le 30/11/2011 15:20, Greg Von Kuster a écrit : Hello Ross and Louise-Amelie, I've just committed what I hope is a fix for this in change set 834de698004e, so when it is convenient can you check things out and let me know? If things are not yet fixed, I'll revert most of what was included in change set 5db0da0007fc, just leaving what is necessary to not throw server errors when running a tool shed that requires users to login. Thanks for your help on this, and sorry for the inconvenience. Greg Von Kuster On Nov 30, 2011, at 6:40 AM, Greg Von Kuster wrote: This is undoubtedly a result of something I broke in change set Commit 5db0da0007fc, so I'll get things functional again. Sorry for the problems this caused you. Ross, does your Galaxy instance have the following config setting in universe_wsgi.ini? require_login = True On Nov 30, 2011, at 5:34 AM, Ross wrote: Louise, I had the same problem. After hg revert -r 3ee9430186fb everything was back to normal for me. I hope this helps.. 2011/11/30 Louise-Amélie Schmitt louise-amelie.schm...@embl.de mailto:louise-amelie.schm...@embl.de Hello, We're experiencing a major issue in the web application and don't know where to look for a potential solution: Whichever link we click on it always brigs us back to the welcome page (galaxy/root). There are a few exceptions to that: - saved histories, saved datasets and api key in the User menu that lead to the appropriate pages - the tools on the left: even better, they open an new full set of Galaxy's frames in the middle frame. We tried reverting to the last pull, restarting Apache, checking the Apache config file, checking the rights in the static directory... and we still do not have the least idea of where to look. Does anyone have an idea, a lead to explore? Thanks, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Ross Lazarus MBBS MPH; Associate Professor, Harvard Medical School; Head, Medical Bioinformatics, BakerIDI; Tel: +61 385321444; ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ Greg Von Kuster Galaxy Development Team g...@bx.psu.edu mailto:g...@bx.psu.edu ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ Greg Von Kuster Galaxy Development Team g...@bx.psu.edu mailto:g...@bx.psu.edu ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Local installation of the tool shed
Hello Greg Please find attached the file you asked for. Just in case, I also sent you the run_community.sh script we use to start the toolshed. Thanks a lot, Louise-Amélie Le 25/11/2011 00:36, Greg Von Kuster a écrit : Hello Louise, Can you send me your community_wsgi.ini file? I assume your copied it from community_wsgi.ini.sample, but somehow it looks like your configuration settings are attempting to start the Galaxy server instead of the tool shed server. I should be able to sort it out for you by looking at your configuration file. Thanks, Greg Von Kuster On Nov 24, 2011, at 10:34 AM, Louise-Amélie Schmitt wrote: Hello Greg We tried to run the toolshed like you explained (thanks a lot for the quick answer btw), it starts fine, but when we try to access it on the web, we get this error in the browser: Server Error An error occurred. See the error logs for more information. (Turn debug on to display exception reports here) In the logs, here is what we get: Error -type 'exceptions.AttributeError': 'Configuration' object has no attribute 'ucsc_display_sites' URL: http://taiji/ File '/home/galaxy/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/exceptions/errormiddleware.py', line 143 in __call__ app_iter = self.application(environ, start_response) File '/home/galaxy/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/home/galaxy/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File '/home/galaxy/galaxy-dev/lib/galaxy/web/framework/base.py', line 134 in __call__ trans = self.transaction_factory( environ ) File '/home/galaxy/galaxy-dev/lib/galaxy/web/framework/__init__.py', line 187 inlambda self.set_transaction_factory( lambda e: self.transaction_chooser( e, galaxy_app, session_cookie ) ) File '/home/galaxy/galaxy-dev/lib/galaxy/web/framework/__init__.py', line 207 in transaction_chooser return GalaxyWebUITransaction( environ, galaxy_app, self, session_cookie ) File '/home/galaxy/galaxy-dev/lib/galaxy/web/framework/__init__.py', line 860 in __init__ self._ensure_logged_in_user( environ ) File '/home/galaxy/galaxy-dev/lib/galaxy/web/framework/__init__.py', line 420 in _ensure_logged_in_user if self.app.config.ucsc_display_sites and self.request.path == display_as: AttributeError: 'Configuration' object has no attribute 'ucsc_display_sites' Since this name ringed a bell, I checked the universe_wsgi.ini and found this Display sites section. I therefore copied it and pasted it in the community.wsgi.ini file and uncommented the variables in this section. But when we tried again, we still get the same error. What is confusing, is that after the error message, all the CGI, congifuration and WSGI variables are displayed, and in the configuration section we have this line: ucsc_display_sites: 'main,test,archaea,ucla' Did we miss anything? Where should we start searching for a possible error source? Thanks, L-A Le 23/11/2011 16:11, Greg Von Kuster a écrit : Hello Nicolas, On Nov 22, 2011, at 11:04 AM, Nicolas Delhomme wrote: Hi Greg, I've been scanning the mailing list, but wasn't lucky enough to find the answer I was looking for. Basically, we would be interested to have our own Galaxy Tool Shed in-house, to develop and test tools within our local Galaxy instance. What I'm thinking of, is to use the power of the tool shed to develop new algorithm, new tools, etc. that could be published afterwards. This is great, but keep in mind that the intent of the tool shed (whether it is a local, proprietary tool shed or the public Galaxy tool shed) is to provide a vehicle for sharing tools (and tool-related objects like workflows, data types, etc) that are determined to be functional within a Galaxy instance. So the primary use of a tool shed should be to enable sharing. The tools themselves should be implemented within a development environment that includes a Galaxy instance, and when a tool is deemed functional, it can then be uploaded to the tool shed for sharing. You can, however, tweak the primary intent of a tool shed to meet your needs. In your case, it seems that you may be interested in using a local tool shed instance to share tools between developers during the development process. If this is the case, then your approach can be one where a developer creates a repository on the local tool shed multiple developers can clone it. The mercurial command line process for committing, pushing, pulling and updating can be used to share updates to the tool code by multiple developers throughout the process. If a single developer is implementing the tool however, it may make more sense to not use the tool shed as part of the development process - just upload the tool when it is functional. Because these would be sensitive material, we would not want to put them right away
Re: [galaxy-dev] Local installation of the tool shed
Hello Greg, This is precisely because we got the error that I added that section, like I mentioned earlier, but it didn't change anything. I will remove it again then. Do you have any other idea that we could explore? Do you need other files? Ah, I forgot to mention that at first start, it would complain about missing files so we created a symlink: ln -s galaxy_home/templates/webapps/community galaxy_home/lib/galaxy/webapps/community/templates Thanks, L-A Le 25/11/2011 14:07, Greg Von Kuster a écrit : Hello Louise-Amelie, I believe the problem is caused by the inclusion of the following settings in your community_webapp.ini file. This sections should not be included in the tool shed config, so if you remove it things should work. Let me know if you bump into other issues. I apologize for the difficulty you've had in getting your own local tool shed up and running. The tool shed wiki currently focuses on using the Galaxy public tool sheds, so I'll next add a section providing details for setting up your own. # -- Display sites # Galaxy can display data at various external browsers. These options specify # which browsers should be available. URLs and builds available at these # browsers are defined in the specifield files. # UCSC browsers: tool-data/shared/ucsc/ucsc_build_sites.txt ucsc_display_sites = main,test,archaea,ucla # GBrowse servers: tool-data/shared/gbrowse/gbrowse_build_sites.txt gbrowse_display_sites = wormbase,tair,modencode_worm,modencode_fly,yeast_sgd # GeneTrack servers: tool-data/shared/genetrack/genetrack_sites.txt genetrack_display_sites = main,test On Nov 25, 2011, at 3:01 AM, Louise-Amélie Schmitt wrote: Hello Greg Please find attached the file you asked for. Just in case, I also sent you the run_community.sh script we use to start the toolshed. Thanks a lot, Louise-Amélie Le 25/11/2011 00:36, Greg Von Kuster a écrit : Hello Louise, Can you send me your community_wsgi.ini file? I assume your copied it from community_wsgi.ini.sample, but somehow it looks like your configuration settings are attempting to start the Galaxy server instead of the tool shed server. I should be able to sort it out for you by looking at your configuration file. Thanks, Greg Von Kuster On Nov 24, 2011, at 10:34 AM, Louise-Amélie Schmitt wrote: Hello Greg We tried to run the toolshed like you explained (thanks a lot for the quick answer btw), it starts fine, but when we try to access it on the web, we get this error in the browser: Server Error An error occurred. See the error logs for more information. (Turn debug on to display exception reports here) In the logs, here is what we get: Error -type 'exceptions.AttributeError': 'Configuration' object has no attribute 'ucsc_display_sites' URL: http://taiji/ File '/home/galaxy/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/exceptions/errormiddleware.py', line 143 in __call__ app_iter = self.application(environ, start_response) File '/home/galaxy/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/home/galaxy/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File '/home/galaxy/galaxy-dev/lib/galaxy/web/framework/base.py', line 134 in __call__ trans = self.transaction_factory( environ ) File '/home/galaxy/galaxy-dev/lib/galaxy/web/framework/__init__.py', line 187 inlambda self.set_transaction_factory( lambda e: self.transaction_chooser( e, galaxy_app, session_cookie ) ) File '/home/galaxy/galaxy-dev/lib/galaxy/web/framework/__init__.py', line 207 in transaction_chooser return GalaxyWebUITransaction( environ, galaxy_app, self, session_cookie ) File '/home/galaxy/galaxy-dev/lib/galaxy/web/framework/__init__.py', line 860 in __init__ self._ensure_logged_in_user( environ ) File '/home/galaxy/galaxy-dev/lib/galaxy/web/framework/__init__.py', line 420 in _ensure_logged_in_user if self.app.config.ucsc_display_sites and self.request.path == display_as: AttributeError: 'Configuration' object has no attribute 'ucsc_display_sites' Since this name ringed a bell, I checked the universe_wsgi.ini and found this Display sites section. I therefore copied it and pasted it in the community.wsgi.ini file and uncommented the variables in this section. But when we tried again, we still get the same error. What is confusing, is that after the error message, all the CGI, congifuration and WSGI variables are displayed, and in the configuration section we have this line: ucsc_display_sites: 'main,test,archaea,ucla' Did we miss anything? Where should we start searching for a possible error source? Thanks, L-A Le 23/11/2011 16:11, Greg Von Kuster a écrit : Hello Nicolas, On Nov 22, 2011, at 11:04 AM, Nicolas Delhomme wrote: Hi Greg, I've been scanning the mailing list, but wasn't lucky enough to find the answer I
Re: [galaxy-dev] Local installation of the tool shed
Hello Greg We tried to run the toolshed like you explained (thanks a lot for the quick answer btw), it starts fine, but when we try to access it on the web, we get this error in the browser: Server Error An error occurred. See the error logs for more information. (Turn debug on to display exception reports here) In the logs, here is what we get: Error - type 'exceptions.AttributeError': 'Configuration' object has no attribute 'ucsc_display_sites' URL: http://taiji/ File '/home/galaxy/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/exceptions/errormiddleware.py', line 143 in __call__ app_iter = self.application(environ, start_response) File '/home/galaxy/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/home/galaxy/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File '/home/galaxy/galaxy-dev/lib/galaxy/web/framework/base.py', line 134 in __call__ trans = self.transaction_factory( environ ) File '/home/galaxy/galaxy-dev/lib/galaxy/web/framework/__init__.py', line 187 in lambda self.set_transaction_factory( lambda e: self.transaction_chooser( e, galaxy_app, session_cookie ) ) File '/home/galaxy/galaxy-dev/lib/galaxy/web/framework/__init__.py', line 207 in transaction_chooser return GalaxyWebUITransaction( environ, galaxy_app, self, session_cookie ) File '/home/galaxy/galaxy-dev/lib/galaxy/web/framework/__init__.py', line 860 in __init__ self._ensure_logged_in_user( environ ) File '/home/galaxy/galaxy-dev/lib/galaxy/web/framework/__init__.py', line 420 in _ensure_logged_in_user if self.app.config.ucsc_display_sites and self.request.path == display_as: AttributeError: 'Configuration' object has no attribute 'ucsc_display_sites' Since this name ringed a bell, I checked the universe_wsgi.ini and found this Display sites section. I therefore copied it and pasted it in the community.wsgi.ini file and uncommented the variables in this section. But when we tried again, we still get the same error. What is confusing, is that after the error message, all the CGI, congifuration and WSGI variables are displayed, and in the configuration section we have this line: ucsc_display_sites: 'main,test,archaea,ucla' Did we miss anything? Where should we start searching for a possible error source? Thanks, L-A Le 23/11/2011 16:11, Greg Von Kuster a écrit : Hello Nicolas, On Nov 22, 2011, at 11:04 AM, Nicolas Delhomme wrote: Hi Greg, I've been scanning the mailing list, but wasn't lucky enough to find the answer I was looking for. Basically, we would be interested to have our own Galaxy Tool Shed in-house, to develop and test tools within our local Galaxy instance. What I'm thinking of, is to use the power of the tool shed to develop new algorithm, new tools, etc. that could be published afterwards. This is great, but keep in mind that the intent of the tool shed (whether it is a local, proprietary tool shed or the public Galaxy tool shed) is to provide a vehicle for sharing tools (and tool-related objects like workflows, data types, etc) that are determined to be functional within a Galaxy instance. So the primary use of a tool shed should be to enable sharing. The tools themselves should be implemented within a development environment that includes a Galaxy instance, and when a tool is deemed functional, it can then be uploaded to the tool shed for sharing. You can, however, tweak the primary intent of a tool shed to meet your needs. In your case, it seems that you may be interested in using a local tool shed instance to share tools between developers during the development process. If this is the case, then your approach can be one where a developer creates a repository on the local tool shed multiple developers can clone it. The mercurial command line process for committing, pushing, pulling and updating can be used to share updates to the tool code by multiple developers throughout the process. If a single developer is implementing the tool however, it may make more sense to not use the tool shed as part of the development process - just upload the tool when it is functional. Because these would be sensitive material, we would not want to put them right away on the public test Tool Shed. Having a local Tool Shed instance would still be very helpful in releasing these tools to the community afterwards, as they would have been integrated and developed within that framework from the start. This is a good approach. Any pointers on how to achieve this are welcome as I'm not so familiar with the Tool Shed yet, e.g. would making a local clone of the tool shed repository be enough? Make sure to read the tool shed wiki at http://wiki.g2.bx.psu.edu/Tool%20Shed. I make sure to keep this up-to-date with the latest features of the tool shed. You'll ffind, however,
Re: [galaxy-dev] API keys and id encryption (silly questions)
Le 29/08/2011 18:54, Nate Coraor a écrit : Louise-Amélie Schmitt wrote: Le 29/08/2011 15:52, Nate Coraor a écrit : Louise-Amélie Schmitt wrote: Hello everyone, These questions are a bit silly but I'm really ignorant when it comes to security. Sorry about that. Why use API keys instead of user names? Is it to to prevent anyone from figuring out who is behind an URL? Or did I miss the point? Hi L-A, To provide a username password, we'd either need to implement HTTP Authentication in Galaxy for these resources, or encode it in the URL. If in the URL, the password have to be non-plaintext which would require encoding on the user's end. The key model seemed to be simplest since it doesn't require you to handle HTTP Authentication in your client-side code. Ok, I actually missed the point, thanks! :D Also, why encrypt the dataset/library/folder ids when a simple display is enough to get them? Anywhere that the IDs are visible are remnants of old code and should eventually be removed. Sorry I meant the encrypted ids. Why encrypt them? is it to prevent any direct use of the database? There are a couple of reasons - the first is that since by default, data is public, we wanted to make it non-trivial to just run sequentially through IDs to view related data. The other is that some people may prefer that it not be obvious how many datasets/jobs/libraries/etc. there are on their server. Ok, thanks a lot for all this information! :) Best, L-A --nate Thanks, L-A --nate Thanks L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] [API] Get the hashed dataset id within Galaxy
I just changed it and ran into an error, so I modified the line and it now works fine: $__app__.security.encode_id( '%s' % $input1.id ) What was the 'file.' originally for? Le 24/08/2011 15:00, Louise-Amélie Schmitt a écrit : Thanks a lot! I found another way to do it but it is awfully more complicated so I'll change as soon as I have some time. Best, L-A Le 23/08/2011 19:49, Nate Coraor a écrit : Louise-Amélie Schmitt wrote: Hi, I would need to make a tool that can get the API dataset id out of the input dataset of the tool, how can I do that? Hi L-A, Pass in as an argument: $__app__.security.encode_id( 'file.%s' % $input1.id ) Where 'input1' is the param name of your input. --nate ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] [API] Get the hashed dataset id within Galaxy
Le 30/08/2011 16:51, Nate Coraor a écrit : Louise-Amélie Schmitt wrote: I just changed it and ran into an error, so I modified the line and it now works fine: $__app__.security.encode_id( '%s' % $input1.id ) What was the 'file.' originally for? On the API side, a library or history's contents are all identified by ID, so: http://server/api/libraries/library_id/contents/content_id That content ID can refer to a dataset (file) or folder, but if this was not 'file.id' or 'folder.id', when the framework decoded it, it would just get a numeric ID and wouldn't know to which type it refered. So instead of using just the ID, it uses the encoded form of 'file.id' and 'folder.id'. This was extended to histories even though there isn't a concept of folders in histories. So just encoding the ID is going to give you the encoded HistoryDatasetAssociation id, but that ID won't be the dataset's API id. For example, using the default id_secret in universe_wsgi.ini, if I upload a dataset to a fresh instance of Galaxy, its encoded ID (the encoded value of '1') is: f2db41e1fa331b3e As you can see in the 'display' URL for the dataset: http://server/datasets/f2db41e1fa331b3e/display/ But that ID won't work for the history API: % ./display.pykey http://server/api/histories/f597429621d6eb2b/contents Collection Members -- #1: /api/histories/f597429621d6eb2b/contents/cc1c6c1afcb4fd4e name: 1.bed type: file id: cc1c6c1afcb4fd4e f597429621d6eb2b is the encoding of '2' with the default id_secret since this dataset is in history ID 2. cc1c6c1afcb4fd4e is the encoding of 'file.1'. --nate This really confuses me. I'll try to explain why: I have a script that runs two workflows one after the other. The ids in the results of workflow 1 are stored in a database, along with some parameters for workflow 2. Then, I run workflow 2 on workflow 1's results. In this workflow I have a tool that queries the database to get the parameters it needs, so it compares the input ids with those stored previously. If I use this line: $__app__.security.encode_id( 'file.%s' % $input1.id ) It does not match the output ids of workflow 1, so I cannot access the corresponding parameters. But if I use this: $__app__.security.encode_id( '%s' % $input1.id ) the ids match and I get the appropriate database entry. What I understood of it: There must be something wrong somewhere, since I'm clearly manipulating file. free stuff and it works just fine. Those same ids are passed to workflow 2 and it runs properly, on the appropriate datasets. Did I miss anything? Did I do something wrong? L-A Le 24/08/2011 15:00, Louise-Amélie Schmitt a écrit : Thanks a lot! I found another way to do it but it is awfully more complicated so I'll change as soon as I have some time. Best, L-A Le 23/08/2011 19:49, Nate Coraor a écrit : Louise-Amélie Schmitt wrote: Hi, I would need to make a tool that can get the API dataset id out of the input dataset of the tool, how can I do that? Hi L-A, Pass in as an argument: $__app__.security.encode_id( 'file.%s' % $input1.id ) Where 'input1' is the param name of your input. --nate ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] [API] Get the hashed dataset id within Galaxy
Le 30/08/2011 18:00, Nate Coraor a écrit : Louise-Amélie Schmitt wrote: Le 30/08/2011 16:51, Nate Coraor a écrit : Louise-Amélie Schmitt wrote: I just changed it and ran into an error, so I modified the line and it now works fine: $__app__.security.encode_id( '%s' % $input1.id ) What was the 'file.' originally for? On the API side, a library or history's contents are all identified by ID, so: http://server/api/libraries/library_id/contents/content_id That content ID can refer to a dataset (file) or folder, but if this was not 'file.id' or 'folder.id', when the framework decoded it, it would just get a numeric ID and wouldn't know to which type it refered. So instead of using just the ID, it uses the encoded form of 'file.id' and 'folder.id'. This was extended to histories even though there isn't a concept of folders in histories. So just encoding the ID is going to give you the encoded HistoryDatasetAssociation id, but that ID won't be the dataset's API id. For example, using the default id_secret in universe_wsgi.ini, if I upload a dataset to a fresh instance of Galaxy, its encoded ID (the encoded value of '1') is: f2db41e1fa331b3e As you can see in the 'display' URL for the dataset: http://server/datasets/f2db41e1fa331b3e/display/ But that ID won't work for the history API: % ./display.pykey http://server/api/histories/f597429621d6eb2b/contents Collection Members -- #1: /api/histories/f597429621d6eb2b/contents/cc1c6c1afcb4fd4e name: 1.bed type: file id: cc1c6c1afcb4fd4e f597429621d6eb2b is the encoding of '2' with the default id_secret since this dataset is in history ID 2. cc1c6c1afcb4fd4e is the encoding of 'file.1'. --nate This really confuses me. I'll try to explain why: I have a script that runs two workflows one after the other. The ids in the results of workflow 1 are stored in a database, along with some parameters for workflow 2. Then, I run workflow 2 on workflow 1's results. In this workflow I have a tool that queries the database to get the parameters it needs, so it compares the input ids with those stored previously. If I use this line: $__app__.security.encode_id( 'file.%s' % $input1.id ) It does not match the output ids of workflow 1, so I cannot access the corresponding parameters. But if I use this: $__app__.security.encode_id( '%s' % $input1.id ) the ids match and I get the appropriate database entry. What I understood of it: There must be something wrong somewhere, since I'm clearly manipulating file. free stuff and it works just fine. Those same ids are passed to workflow 2 and it runs properly, on the appropriate datasets. Did I miss anything? Did I do something wrong? Ah, no, the workflow API is using the encoded id without 'file.', and specifies the source by using the format: workflow_step_id=type=encoded_id Since the type is provided as a parameter it isn't included in the encoded id. This is more ideal since the encoded id will match what's in the UI, so let me think about how we can do this in the history and library APIs without having ugly URLs. --nate Ah ok, I get it :) Thanks for all the information! Best, L-A L-A Le 24/08/2011 15:00, Louise-Amélie Schmitt a écrit : Thanks a lot! I found another way to do it but it is awfully more complicated so I'll change as soon as I have some time. Best, L-A Le 23/08/2011 19:49, Nate Coraor a écrit : Louise-Amélie Schmitt wrote: Hi, I would need to make a tool that can get the API dataset id out of the input dataset of the tool, how can I do that? Hi L-A, Pass in as an argument: $__app__.security.encode_id( 'file.%s' % $input1.id ) Where 'input1' is the param name of your input. --nate ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] API keys and id encryption (silly questions)
Le 29/08/2011 15:52, Nate Coraor a écrit : Louise-Amélie Schmitt wrote: Hello everyone, These questions are a bit silly but I'm really ignorant when it comes to security. Sorry about that. Why use API keys instead of user names? Is it to to prevent anyone from figuring out who is behind an URL? Or did I miss the point? Hi L-A, To provide a username password, we'd either need to implement HTTP Authentication in Galaxy for these resources, or encode it in the URL. If in the URL, the password have to be non-plaintext which would require encoding on the user's end. The key model seemed to be simplest since it doesn't require you to handle HTTP Authentication in your client-side code. Ok, I actually missed the point, thanks! :D Also, why encrypt the dataset/library/folder ids when a simple display is enough to get them? Anywhere that the IDs are visible are remnants of old code and should eventually be removed. Sorry I meant the encrypted ids. Why encrypt them? is it to prevent any direct use of the database? Thanks, L-A --nate Thanks L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] checkboxes bug fix
Hi Kanwei, and thanks for your reply. I might have missed something really big but I'm kinda confused. I just double checked galaxy-dist and galaxy-central on bitbucket and I would like to know if I understood the code properly. Please correct me if I'm wrong. Let's look at the is_checked() method of the CheckboxField class in the lib/galaxy/web/form_builder.py file: If I'm not mistaken, it relies on the fact that if the checkbox is checked, the request will get two values (the checkbox and the hidden field) and if it is not, it will only get one value (the hidden field) since they bear the same name. The first part is the conditional that checks if the value is True: That means that there is only one value, so the checkbox must be unchecked. Therefore it tests the hidden field's value. It will return True if the hidden field's value is true. The second part is only executed if the above condition is false: this means either the checkbox is actually checked, or it is unchecked and the hidden field is false. It returns true if there are two values, i.e. if the checkbox is checked. If the hidden field is false, it will obviously return False. So, let's see both cases: - 1st case: the hidden field is true: If the checkbox is unchecked, the method will return True in the first part. If it is checked, it will also return True in the second part. - 2d case: the hidden field is false: If the checkbox is checked, like in the 1st case, the second part will return True. But if it is unchecked, the first part won't return anything since the received value is not True. So it will return the result of the second part which is false since False is not an instance of list. Now my questions are: Did I miss anything? If not, how can the first case work for you? Wouldn't there be some cases where a checkbox actually gives a value even if it is unchecked, and a field is ignored of there is already one with the same name? Thanks, L-A Le 24/08/2011 19:36, Kanwei Li a écrit : Hi L-A, I tried this on today's tip but could not reproduce the bug. I added your param param name=SQA type=boolean label=Run SolexaQA checked=true / to a tool and the correct value was produced on the command line and in the configfiles block as well. Re-running also gave the correct result. Are you still seeing this problem? Thanks, K 2011/8/15 Louise-Amélie Schmitt louise-amelie.schm...@embl.de mailto:louise-amelie.schm...@embl.de Hello, Sorry for being late, again, but please find the tool attached. Here is my last pull: 5743:720455407d1c Stuff changed in the file but in the end I had to make the same modification again (change true to false) since it didn't work anymore. Best, L-A Le 15/06/2011 09:56, Louise-Amélie Schmitt a écrit : Hi, and sorry for the late reply Here is the last pull: 5355:50e249442c5a I'll try to be as concise as I can but I can send you the whole file if you need it. It's not completely done yet though. This is a tool that produces a HTML file containing the results of various quality assessment programs. To trigger or block the execution of a specific program, I use checkboxes in the form, like this: param name=SQA type=boolean label=Run SolexaQA checked=true / Then, I use the configfile name=script tags to define a shell script that will be modified with cheetah tests on the checkboxes. Then the final script is sent to another shell script that will check its existence and launch it: command interpreter='sh'all_QA.sh $script /command As you can see, the checkboxes are checked by default. But before the change I mentioned is my last email, un-checking them didn't have any effect whatsoever. The following test, for instance, was always true: #if $SQA.value == True Moreover, even if I un-checked the checkboxes and ran the tool, then clicked the re-run button in the result in my history, all of them were checked in the form that appears, as if I never un-checked them. Now, the state of the checkboxes is respected and I can properly choose what to run in my tool. Hope that helps. L-A Le 08/06/2011 20:54, Jennifer Jackson a écrit : Hi L-A, Could you describe a use case (which tool/from, what options/input)? If you know the last pull # for your local install, that would also be helpful for us to know. Best, Jen Galaxy team On 6/6/11 2:15 AM, Louise-Amélie Schmitt wrote: Hi, Since I haven't updated Galaxy for a while now I don't know if it was actually fixed but I had issues with my default-selected
Re: [galaxy-dev] [API] Get the hashed dataset id within Galaxy
Thanks a lot! I found another way to do it but it is awfully more complicated so I'll change as soon as I have some time. Best, L-A Le 23/08/2011 19:49, Nate Coraor a écrit : Louise-Amélie Schmitt wrote: Hi, I would need to make a tool that can get the API dataset id out of the input dataset of the tool, how can I do that? Hi L-A, Pass in as an argument: $__app__.security.encode_id( 'file.%s' % $input1.id ) Where 'input1' is the param name of your input. --nate ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] checkboxes bug fix
Hello, Sorry for being late, again, but please find the tool attached. Here is my last pull: 5743:720455407d1c Stuff changed in the file but in the end I had to make the same modification again (change true to false) since it didn't work anymore. Best, L-A Le 15/06/2011 09:56, Louise-Amélie Schmitt a écrit : Hi, and sorry for the late reply Here is the last pull: 5355:50e249442c5a I'll try to be as concise as I can but I can send you the whole file if you need it. It's not completely done yet though. This is a tool that produces a HTML file containing the results of various quality assessment programs. To trigger or block the execution of a specific program, I use checkboxes in the form, like this: param name=SQA type=boolean label=Run SolexaQA checked=true / Then, I use the configfile name=script tags to define a shell script that will be modified with cheetah tests on the checkboxes. Then the final script is sent to another shell script that will check its existence and launch it: command interpreter='sh'all_QA.sh $script /command As you can see, the checkboxes are checked by default. But before the change I mentioned is my last email, un-checking them didn't have any effect whatsoever. The following test, for instance, was always true: #if $SQA.value == True Moreover, even if I un-checked the checkboxes and ran the tool, then clicked the re-run button in the result in my history, all of them were checked in the form that appears, as if I never un-checked them. Now, the state of the checkboxes is respected and I can properly choose what to run in my tool. Hope that helps. L-A Le 08/06/2011 20:54, Jennifer Jackson a écrit : Hi L-A, Could you describe a use case (which tool/from, what options/input)? If you know the last pull # for your local install, that would also be helpful for us to know. Best, Jen Galaxy team On 6/6/11 2:15 AM, Louise-Amélie Schmitt wrote: Hi, Since I haven't updated Galaxy for a while now I don't know if it was actually fixed but I had issues with my default-selected checkboxes: When I deselected them, the value sent in the query remained as if they were still selected. Even when I re-ran the job, all the checkboxes were selected as if I never deselected them. Therefore, in lib/galaxy/web/form_builder.py I changed: 105 return 'input type=checkbox id=%s name=%s value=true%s%sinput type=hidden name=%s%s value=true%s' \ for: 105 return 'input type=checkbox id=%s name=%s value=true%s%sinput type=hidden name=%s%s value=false%s' \ And it works fine now :) Cheers, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ tool id=all_QA name=QA Bundle description=performs multiple QA command interpreter='sh'all_QA.sh $script /command inputs param name=input type=data label=Input file format=fastq,fastqillumina,sam,export,bam checked=true / param name=R type=boolean label=Run R QA (runs only on export files) checked=true / param name=HTSeq type=boolean label=Run HTSeq QA checked=true / param name=C type=boolean label=Run baseQual (work only with fastq files) checked=true / param name=SQA type=boolean label=Run SolexaQA checked=true / param type=select name=SQA_cutoff label=#60;span style=#34;font-weight:normal;#34;#62;Cutoff (phred value at which base-calling error is considered too high)#60;/span#62; help= option value=1010 (90%)/option option value=13 selected=true13 (95%)/option option value=2020 (99%)/option option value=3030 (99.9%)/option option value=4040 (99.99%)/option /param param type=select name=SQA_variance label=#60;span style=#34;font-weight:normal;#34;#62;Calculate variance statistics#60;/span#62; display=radio help= option value= selected=trueNo thanks/option option value=-vYes please/option /param param type=select name=SQA_minmax label=#60;span style=#34;font-weight:normal;#34;#62;Calculate minimum and maximum error probabilities for each read position of each tile#60;/span#62; display=radio help= option value= selected=trueNo thanks/option option value=-mYes please/option /param param name=SQA_sample size=6 type=integer value=1 label=#60;span style=#34;font-weight:normal;#34;#62;Number of sequences to be sampled per tile for satistic estimates#60;/span#62; help= / param name=CovE type=boolean label=Run Coverage Estimation checked=true / param type=select name=window label=#60;span style=#34;font-weight:normal;#34;#62;Windows size for #60;b
Re: [galaxy-dev] [API] About workflow parameters :)
Hi Jen, Hum.. what is a bitbucket ticket? How do you get there? There's no real hurry anyway, as far as I'm concerned. I made API-versions of the tools I need where the python wrapper gets the needed parameters from a database. But yeah, I'm still very interested in how the API evolves. By the way, is there (or will there be) a way to share and delete histories through the API? Thanks, L-A Le 09/08/2011 02:58, Jennifer Jackson a écrit : Hi L-A, Should we make a bitbucket ticket to keep track of this enhancement request? If you would like to make one, that would be great since you could put in all the details, or just write back and I can create a simple ticket. Thanks! Jen Galaxy team On 8/5/11 6:59 AM, Louise-Amélie Schmitt wrote: Hello, Just a quick message to ask if you had the time to work on passing parameters to workflows through the API. Just to know, since I'm using workflows with the API. Best, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] [API] About workflow parameters :)
Hello, Just a quick message to ask if you had the time to work on passing parameters to workflows through the API. Just to know, since I'm using workflows with the API. Best, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] [API] Apache error?
Ok, I found some tape I can put on it. Here:
http://misc.feedfury.com/content/45712644-anton-bobrov-bad-servers-chunked-encoding-and-incompleteread.html
In scripts/api/common.py I put this piece of code at the beginning of
the file, after the imports:
import httplib
def patch_http_response_read(func):
def inner(*args):
try:
return func(*args)
except httplib.IncompleteRead, e:
return e.partial
return inner
httplib.HTTPResponse.read =
patch_http_response_read(httplib.HTTPResponse.read)
And it works fine now.
Le 29/07/2011 17:06, Louise-Amélie Schmitt a écrit :
Hello everyone,
I'm working on a script that uploads files and launchs workflows on
them, but I keep getting errors that appear more or less randomly when
the display() and submit() functions are called. In a nutshell, there
is a 1/3 chance the calls fail this way.
Nevertheless, the actions are properly triggered in Galaxy, surprisingly.
Here is an example:
When I launch a workflow, I get the following traceback even though
the workflow is properly executed:
http://localhost/galaxy-dev/api/workflows?key=273c7b4e3aaffd3884ef715aaf780d9a
File automated_preprocessing.py, line 61, in expandFile
result = submit( api_key, api_url + 'workflows', wf_data,
return_formatted=False)
File common.py, line 100, in submit
r = post( api_key, url, data )
File common.py, line 44, in post
return simplejson.loads( urllib2.urlopen( req ).read() )
File
/g/steinmetz/collaboration/software/CentOS5/opt/Python-2.6.5/lib/python2.6/socket.py,
line 329, in read
data = self._sock.recv(rbufsize)
File
/g/steinmetz/collaboration/software/CentOS5/opt/Python-2.6.5/lib/python2.6/httplib.py,
line 518, in read
return self._read_chunked(amt)
File
/g/steinmetz/collaboration/software/CentOS5/opt/Python-2.6.5/lib/python2.6/httplib.py,
line 561, in _read_chunked
raise IncompleteRead(''.join(value))
Failed to expand file A:
Type: class 'httplib.IncompleteRead'
IncompleteRead(118 bytes read)
Therefore, I cannot get the results within the script.
Any idea?
Best,
L-A
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
[galaxy-dev] [API] Apache error?
Hello everyone,
I'm working on a script that uploads files and launchs workflows on
them, but I keep getting errors that appear more or less randomly when
the display() and submit() functions are called. In a nutshell, there is
a 1/3 chance the calls fail this way.
Nevertheless, the actions are properly triggered in Galaxy, surprisingly.
Here is an example:
When I launch a workflow, I get the following traceback even though the
workflow is properly executed:
http://localhost/galaxy-dev/api/workflows?key=273c7b4e3aaffd3884ef715aaf780d9a
File automated_preprocessing.py, line 61, in expandFile
result = submit( api_key, api_url + 'workflows', wf_data,
return_formatted=False)
File common.py, line 100, in submit
r = post( api_key, url, data )
File common.py, line 44, in post
return simplejson.loads( urllib2.urlopen( req ).read() )
File
/g/steinmetz/collaboration/software/CentOS5/opt/Python-2.6.5/lib/python2.6/socket.py,
line 329, in read
data = self._sock.recv(rbufsize)
File
/g/steinmetz/collaboration/software/CentOS5/opt/Python-2.6.5/lib/python2.6/httplib.py,
line 518, in read
return self._read_chunked(amt)
File
/g/steinmetz/collaboration/software/CentOS5/opt/Python-2.6.5/lib/python2.6/httplib.py,
line 561, in _read_chunked
raise IncompleteRead(''.join(value))
Failed to expand file A:
Type: class 'httplib.IncompleteRead'
IncompleteRead(118 bytes read)
Therefore, I cannot get the results within the script.
Any idea?
Best,
L-A
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
[galaxy-dev] [API] managing histories
Hello everyone, I'm currently trying to automate data loading and pre-processing through Galaxy's API, and I will need to delete and share histories at some point. I know the API is still very new but is there a way to do that by any chance? Thanks, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] postgresql connection error: could not connect to server: No such file or directory...
Hi,
Could you please show us the contents of your pg_hba.conf file? This is
where you configure which machines can access the database through the
network.
Best,
L-A
Le 27/06/2011 20:33, Marco Moretto a écrit :
Hi Gus,
it seems that your postgres is not configured to accept connection
except localhost (or the deamon is not running). Did you try to
connect to postgres from another PC in the network (no the localhost)?
Did you install postgres using apt-get install? Do you run postgres
using /etc/init.d/postgresql-8.4 start?
Greets.
---
Marco
On 24 June 2011 21:52, W. Augustine Dunn III wadun...@gmail.com
mailto:wadun...@gmail.com wrote:
Hello all:
Please forgive me if this is a matter of RTFM but I can assure you
that I have in fact TRIED that for a day or so now...
I am trying to set up a local GALAXY site for my lab so that I can
start to offload some of the work that comes my way as the sole
biologist in our lab not afraid of the command line. My system is
Ubuntu 10.10 64b, i7 quad, 12 GB RAM. Since Multiple people will
be using the site I decided to go with the slightly more fancy
postgresql db back-end rather than the sqlite default. I had
things running fine with the sqlite but can NOT seem to get psql
and galaxy to play nice.
I have created a galaxy role with rights to make tables on a
created galaxy database:
galaxy@vlad 12:40:20 ~/galaxy-dist:
psql
psql (8.4.8)
Type help for help.
galaxy= \l
List of databases
Name | Owner | Encoding | Collation | Ctype|
Access privileges
--+--+--+++---
galaxy | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
=Tc/postgres
:
postgres=CTc/postgres
:
galaxy=CTc/postgres
postgres | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
template0| postgres | UTF8 | en_US.utf8 | en_US.utf8 |
=c/postgres
:
postgres=CTc/postgres
template1| postgres | UTF8 | en_US.utf8 | en_US.utf8 |
=c/postgres
:
postgres=CTc/postgres
zabbix_proxy | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
(5 rows)
I created a PGDATA file at /var/lib/postgresql/8.4/main/PGDATA and
export a PGDATA envar upon loading of the galaxy user's .bashrc
pointing to this place. I am able to start and stop the server
with the postgres user using pg_ctl. Once its running, I change
users back to galaxy and try to run run.sh and get the following
traceback:
Traceback (most recent call last):
File /home/galaxy/galaxy-dist/lib/galaxy/web/buildapp.py, line
82, in app_factory
app = UniverseApplication( global_conf = global_conf, **kwargs )
File /home/galaxy/galaxy-dist/lib/galaxy/app.py, line 32, in
__init__
create_or_verify_database( db_url, kwargs.get( 'global_conf',
{} ).get( '__file__', None ), self.config.database_engine_options )
File
/home/galaxy/galaxy-dist/lib/galaxy/model/migrate/check.py, line
54, in create_or_verify_database
dataset_table = Table( dataset, meta, autoload=True )
File
/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/schema.py,
line 108, in __call__
return type.__call__(self, name, metadata, *args, **kwargs)
File
/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/schema.py,
line 236, in __init__
_bind_or_error(metadata).reflecttable(self,
include_columns=include_columns)
File
/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py,
line 1261, in reflecttable
conn = self.contextual_connect()
File
/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/threadlocal.py,
line 194, in contextual_connect
return self.session.get_connection(**kwargs)
File
/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/threadlocal.py,
line 20, in get_connection
return self.engine.TLConnection(self,
self.engine.pool.connect(), close_with_result=close_with_result)
File
/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/pool.py,
line 151, in connect
agent = _ConnectionFairy(self)
File
/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/pool.py,
line 304, in __init__
rec = self._connection_record = pool.get()
File
[galaxy-dev] [API] Error message forwarding
Hi, I ran into a little something that is a bit annoying for debug when trying to upload files through the API with library_upload_from_import_dir.py. When the specified folder is wrong, python tries to process the error tuple like a dict, so the original error is hard to find. I modified a little the code to avoid that. It might not work in all cases but at least it solved my problem. Here is the original code: lib/galaxy/web/api/contents.py l.145-end else: rval = [] for k, v in output.items(): if type( v ) == trans.app.model.LibraryDatasetDatasetAssociation: v = v.library_dataset encoded_id = trans.security.encode_id( create_type + '.' + str( v.id ) ) rval.append( dict( id = encoded_id, name = v.name, url = url_for( 'content', library_id=library_id, id=encoded_id ) ) ) return rval Here is how I modified it: else: rval = [] try: for k, v in output.items(): if type( v ) == trans.app.model.LibraryDatasetDatasetAssociation: v = v.library_dataset encoded_id = trans.security.encode_id( create_type + '.' + str( v.id ) ) rval.append( dict( id = encoded_id, name = v.name, url = url_for( 'content', library_id=library_id, id=encoded_id ) ) ) return rval except: log.debug( 'HTTP Error %s: %s' % ( output[0], output[1] ) ) return None Since it saves me some time energy I just wanted to share it, so I hope it can help. Best L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] LDAP and API issue?
Ok, again sorry for being an idiot. I just had to modify the Apache .conf file related to Galaxy. For the time being I allowed everyone on the API (since it's not accessible via the Internet it's allright) so the all will eventually be changed but here is what I added in the file: Location /galaxy-dev AuthName Galaxy (development) AuthType Basic AuthBasicProvider ldap AuthLDAPURL ldap://ocs.embl.org/cn=Users,dc=embl,dc=org?uid; AuthzLDAPAuthoritative off Require valid-user /Location Location /galaxy-dev/api AuthName Galaxy (API) AuthType Basic AuthBasicProvider ldap AuthLDAPURL ldap://ocs.embl.org/cn=Users,dc=embl,dc=org?uid; AuthzLDAPAuthoritative off Satisfy any Allow from all /Location L-A Le 17/06/2011 14:14, Louise-Amélie Schmitt a écrit : Hi everyone I'm currently trying to use the API, but I think that our LDAP logging doesn't really help. When I call the script specifying the port in the url, here's what I get: urlopen error [Errno 111] Connection refused Then when I don't specify it Here's what I get: HTTP Error 401: Authorization Required !DOCTYPE HTML PUBLIC -//IETF//DTD HTML 2.0//EN htmlhead title401 Authorization Required/title /headbody h1Authorization Required/h1 pThis server could not verify that you are authorized to access the document requested. Either you supplied the wrong credentials (e.g., bad password), or your browser doesn't understand how to supply the credentials required./p hr addressApache/2.2.3 (CentOS) Server at localhost Port 80/address /body/html Did I miss anything? Is there a way to avoid that? Thanks LA ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] checkboxes bug fix
Hi, Since I haven't updated Galaxy for a while now I don't know if it was actually fixed but I had issues with my default-selected checkboxes: When I deselected them, the value sent in the query remained as if they were still selected. Even when I re-ran the job, all the checkboxes were selected as if I never deselected them. Therefore, in lib/galaxy/web/form_builder.py I changed: 105return 'input type=checkbox id=%s name=%s value=true%s%sinput type=hidden name=%s%s value=true%s' \ for: 105return 'input type=checkbox id=%s name=%s value=true%s%sinput type=hidden name=%s%s value=false%s' \ And it works fine now :) Cheers, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Error when adding datasets
Le 02/06/2011 21:23, Nate Coraor a écrit : Louise-Amélie Schmitt wrote: On Thu, 2 Jun 2011 11:48:55 -0400, Nate Coraorn...@bx.psu.edu wrote: Greg Von Kuster wrote: Hello Louise, I've CC'd Nate on this as he may be able to help - although no guarantees. I'm not expert enough in this area to know where to look for the cause. Perhaps someone in the community can help as well. It is likely that LDAP is playing a role in this behavior. On Apr 14, 2011, at 1:00 PM, Louise-Amélie Schmitt wrote: The thing is, we use LDAP logging so we can't even access the website without logging in. Moreover, when I logged in, I arrived on the data analysis page where the automatic unnamed history was obviously created in the history panel. I forgot to mention we have issues creating and deleting histories, like, we can't access some histories and everytime we delete histories, two extra unnamed histories are created. As I mentioned before, it is also impossible to load a dataset in any history, existing or not. Okay, somehow I missed all the followup on this question when I committed a fix a few minutes ago. Could you provide some details about the inaccessible histories (what happens when you try to access them, traceback, etc?) and loading data - you mean the upload tool fails? Is there also a traceback here? ooh it's been a while I haven't seen the issue since I not-so-elegantly avoided the problem making our two instances run on separate machines. When these were running on the same machine, were they both served as seperate directories under the same hostname? If so, did you set cookie_path in the config? If not, your cookies for one instance were being used with another instance, with unpredictable results. Although from recent discussion here on -dev, it looks like there might be a bug with cookie_path. --nate Yes I did, and it was different for both instances. I just checked. L-A But I'll try to answer your question as accurately as I can. I experienced a similar error (pretty much the same traceback as the one I pasted in a previous message, just yelling at me saying that the history is nonexistent) whenever I tried to: - upload data in the manage libraries admin panel and in the analyse data panel (upload tool) - load an already uploaded dataset in any kind of history (saved or new) - probably do something else, I can't really remember I also had issues dealing with histories but not getting any traceback, like I mentioned a little lower: Everytime I tried to delete any number of histories, the histories were deleted but two were created. I hope this will help, sorry I can't really be more precise. L-A Thanks, --nate Do you think it could be linked to our using LDAP? Thanks L-A On Thu, 14 Apr 2011 12:06:55 -0400, Greg Von Kusterg...@bx.psu.edu wrote: On Apr 14, 2011, at 11:49 AM, Louise-Amélie Schmitt wrote: Here is the result I got from the debug statements: galaxy.web.controllers.library_common DEBUG 2011-04-14 17:46:02,286 ### history: None This is the problem - when you registered normally, a history would have been automatically created for you. But somehow you don't have a history - no idea why / how this happened, but it is very unlikely this is a result of a bug within Galaxy. Try logging out and logging in again and a new history should be created for you. galaxy.web.controllers.library_common DEBUG 2011-04-14 17:46:02,286 ### trans:galaxy.web.framework.GalaxyWebUITransaction object at 0x29f40950 galaxy.web.controllers.library_common DEBUG 2011-04-14 17:46:02,286 ### trans.sa_session:sqlalchemy.orm.scoping.ScopedSession object at 0x19ab21d0 Thanks again L-A Le jeudi 14 avril 2011 à 11:05 -0400, Greg Von Kuster a écrit : I assume when you dropped the old database and recreated the new, empty database, you made sure the database connection string in universe_wsgi.ini was correctly set. if so, when you started up Galaxy, it would have created all of the required tables in the new database, and they would all be empty. When you first registered as the admin user, it would have automatically populated several tables with data, including the history table. One or more of these things must not have been successful. To attempt to narrow down the problem, I'll need you to do the following. Here are lines # 905 - 907 in ~/lib/galaxy/web/controllers/library-common.py # Send the current history to the form to enable importing datasets from history to library history = trans.get_history() trans.sa_session.refresh( history ) Please add the following debug statements: # Send the current history to the form to enable importing datasets from history to library history = trans.get_history() log.debug(### history: %s % str( history )) log.debug(### trans: %s % str( trans )) log.debug(### trans.sa_session: %s % str( trans.sa_session )) trans.sa_session.refresh( history ) Stop
Re: [galaxy-dev] Error when adding datasets
On Thu, 2 Jun 2011 11:48:55 -0400, Nate Coraor n...@bx.psu.edu wrote: Greg Von Kuster wrote: Hello Louise, I've CC'd Nate on this as he may be able to help - although no guarantees. I'm not expert enough in this area to know where to look for the cause. Perhaps someone in the community can help as well. It is likely that LDAP is playing a role in this behavior. On Apr 14, 2011, at 1:00 PM, Louise-Amélie Schmitt wrote: The thing is, we use LDAP logging so we can't even access the website without logging in. Moreover, when I logged in, I arrived on the data analysis page where the automatic unnamed history was obviously created in the history panel. I forgot to mention we have issues creating and deleting histories, like, we can't access some histories and everytime we delete histories, two extra unnamed histories are created. As I mentioned before, it is also impossible to load a dataset in any history, existing or not. Okay, somehow I missed all the followup on this question when I committed a fix a few minutes ago. Could you provide some details about the inaccessible histories (what happens when you try to access them, traceback, etc?) and loading data - you mean the upload tool fails? Is there also a traceback here? ooh it's been a while I haven't seen the issue since I not-so-elegantly avoided the problem making our two instances run on separate machines. But I'll try to answer your question as accurately as I can. I experienced a similar error (pretty much the same traceback as the one I pasted in a previous message, just yelling at me saying that the history is nonexistent) whenever I tried to: - upload data in the manage libraries admin panel and in the analyse data panel (upload tool) - load an already uploaded dataset in any kind of history (saved or new) - probably do something else, I can't really remember I also had issues dealing with histories but not getting any traceback, like I mentioned a little lower: Everytime I tried to delete any number of histories, the histories were deleted but two were created. I hope this will help, sorry I can't really be more precise. L-A Thanks, --nate Do you think it could be linked to our using LDAP? Thanks L-A On Thu, 14 Apr 2011 12:06:55 -0400, Greg Von Kuster g...@bx.psu.edu wrote: On Apr 14, 2011, at 11:49 AM, Louise-Amélie Schmitt wrote: Here is the result I got from the debug statements: galaxy.web.controllers.library_common DEBUG 2011-04-14 17:46:02,286 ### history: None This is the problem - when you registered normally, a history would have been automatically created for you. But somehow you don't have a history - no idea why / how this happened, but it is very unlikely this is a result of a bug within Galaxy. Try logging out and logging in again and a new history should be created for you. galaxy.web.controllers.library_common DEBUG 2011-04-14 17:46:02,286 ### trans: galaxy.web.framework.GalaxyWebUITransaction object at 0x29f40950 galaxy.web.controllers.library_common DEBUG 2011-04-14 17:46:02,286 ### trans.sa_session: sqlalchemy.orm.scoping.ScopedSession object at 0x19ab21d0 Thanks again L-A Le jeudi 14 avril 2011 à 11:05 -0400, Greg Von Kuster a écrit : I assume when you dropped the old database and recreated the new, empty database, you made sure the database connection string in universe_wsgi.ini was correctly set. if so, when you started up Galaxy, it would have created all of the required tables in the new database, and they would all be empty. When you first registered as the admin user, it would have automatically populated several tables with data, including the history table. One or more of these things must not have been successful. To attempt to narrow down the problem, I'll need you to do the following. Here are lines # 905 - 907 in ~/lib/galaxy/web/controllers/library-common.py # Send the current history to the form to enable importing datasets from history to library history = trans.get_history() trans.sa_session.refresh( history ) Please add the following debug statements: # Send the current history to the form to enable importing datasets from history to library history = trans.get_history() log.debug(### history: %s % str( history )) log.debug(### trans: %s % str( trans )) log.debug(### trans.sa_session: %s % str( trans.sa_session )) trans.sa_session.refresh( history ) Stop and restart your Galaxy server after making the above changes, and reply back with the output of the debug statements. Assuming you start your Galaxy instance with: sh run.sh you'll see the results of the debug statements in the log scrolling in the window in which you started Galaxy. Thanks On Apr 14, 2011, at 10:46 AM
[galaxy-dev] Workflows and unknown number of output datasets
Hi everyone First let me thank all the Team Galaxy, the conference was really great. Kanwei asked me to send the tool I told him about. It's one of these tools for which you can't know the exact number of output datasets before tool run. Here are the files. It's really simple actually, but it would be nice if I could integrate it in workflows without it having to be the final step. Would it even be possible? And please ignore the idiotic comments in the files, I was really tired that day. Cheers, L-A # -*- coding: UTF-8 -*- import os, sys, string mpxdata = sys.argv[1] barcodes= sys.argv[2] output1 = sys.argv[3] output1id = sys.argv[4] newfilepath = sys.argv[5] # Building the command line cmd = java -cp /g/steinmetz/projects/solexa/java/solexaJ/bin/:/g/steinmetz/projects/solexa/software/picard/trunk/dist/picard-1.18.jar:/g/steinmetz/projects/solexa/software/picard/trunk/dist/sam-1.18.jar cmd+= deBarcoding F1= cmd+= mpxdata cmd+= BL= cmd+= barcodes cmd+= DR=\ cmd+= newfilepath cmd+= \ # Executing deBarcoding status = os.system(cmd) # In the unlikely event of a fire, please use the nearest emergency exit if status != 0: print Demultiplexing failed. sys.exit(status) oldnames=[] # Reconstructing the output file names as deBarcoding writes them bc = open(barcodes, r) for l in bc.readlines(): l = l.split() if l[0] != : oldnames.append(l[0]) for i in range(len(oldnames)): oldnames[i] = oldnames[i] + .txt newnames=[] # Creating the required paths for multiple outputs if os.path.isdir(newfilepath): for f in oldnames: if os.path.isfile(newfilepath+/+f): name = os.path.splitext(f)[0] s = primary_ s+= output1id s+= _ s+= string.replace(name, _, -) s+= _visible_fastq newnames.append(newfilepath+/+s) # Adding the appropriate prefixes to the old filenames for i in range(len(oldnames)): oldnames[i] = newfilepath+/+oldnames[i] # Setting the first file as the mandatory output file defined in the xml newnames[0] = output1 # Moving everything where it will be seen properly by Galaxy for i in range(len(oldnames)): os.rename(oldnames[i],newnames[i]) # Ta-da! tool id=debarcoding name=Demultiplexer descriptionDemultiplexes multiplexed data (who would have guessed?)/description command interpreter=pythondebarcoding.py $mpxdata $barcodes $output1 $output1.id $__new_file_path__/command inputs param type=data format=gz name=mpxdata label=Compressed Sequence/ param type=data format=bc name=barcodes label=Barcode Set/ /inputs outputs data format=fastq name=output1 metadata_source=mpxdata / /outputs help **Program:** debarcoding.py (v1.0.0) **Author:** This is a wrapper for Wave's deBarcoding java tool **Summary:** This tool demutiplexes data according to a list of barcodes containing a column withe the sample name and a second column with the barcode sequence. **Usage:** Here is an example of the java tool's usage:: Two failed PE runs (multiplex 2 and multiplex8) yielded SE data that can be used for assembly together with the corrected PE ones. To this end, the deBarcoding script from wave was used: javasol deBarcoding where javasol is an alias to: java -cp /g/steinmetz/projects/solexa/java/solexaJ/bin/:/g/steinmetz/projects/solexa/software/picard/trunk/dist/picard-1.18.jar:/g/steinmetz/projects/solexa/software/picard/trunk/dist/sam-1.18.jar The command lines were (thanks to Wave for the clarifications): For the mplex num. 3, some pre-processing was necessary R src/getBarcodeSequencing.R javasol deBarcoding F1=s_1_LESAFFRE_sequence.txt.gz BL=barcodeList.txt DR=seq_lane1 javasol deBarcoding F1=s_2_LESAFFRE_sequence.txt.gz BL=barcodeList.txt DR=seq_lane2 javasol deBarcoding F1=s_3_LESAFFRE_sequence.txt.gz BL=barcodeList.txt DR=seq_lane3 cat seq_lane1/111.txt seq_lane2/111.txt seq_lane3/111.txt sequences/111.txt cat seq_lane1/112.txt seq_lane2/112.txt seq_lane3/112.txt sequences/112.txt cat seq_lane1/1251.txt seq_lane2/1251.txt seq_lane3/1251.txt sequences/1251.txt cat seq_lane1/1303.txt seq_lane2/1303.txt seq_lane3/1303.txt sequences/1303.txt cat seq_lane1/93ep.txt seq_lane2/93ep.txt seq_lane3/93ep.txt sequences/93ep.txt rename .txt _s_sequence.txt *.txt gzip *.txt /help /tool ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Uploading Binary Files into Galaxy
Hi, Have you tried grep'ing the error message in the lib files to see where Galaxy goes to during the upload of these files? like: grep /The uploaded binary file contains inappropriate content/ `find *` I had a similar problem when I modified Galaxy to manage .gz files without uncompressing them automatically (I had to add a new file type too), and diving into the code to see what is actually executed really helped. Good luck, L-A Le 24/05/2011 04:40, darren.culle...@csiro.au a écrit : Hi All, Sorry for the delay, I have had a couple of other projects on the go. Still no good. I have made the data type unsniffable and added what I thought was the correct information. I have also added the extension (sfx) to the list of unsniffable binary formats in binary.py but it does not make a difference. Code is as follows: ./datatypes_conf.xml datatype extension=sfx type=galaxy.datatypes.binary:Sfx display_in_upload=true/ ./lib/galaxy/datatypes/binary.py class Sfx( Binary ): Kanga and uAssembler Suffix Array file_ext = sfx def __init__( self, **kwd ): Binary.__init__( self, **kwd ) def set_peek( self, dataset, is_multi_byte=False ): if not dataset.dataset.purged: dataset.peek = 'Suffix Array sfx' dataset.blurb = 'binary data' else: dataset.peek = 'file does not exist' dataset.blurb = 'file purged from disk' My gut feeling is that this is related to the fact that Galaxy wants to look in the file, but as soon as its opened, it can't recognise the characters and it reports /The uploaded binary file contains inappropriate content/. I thought the set_peekmethod I have above would override the one described in the Class Binary. I have tried adding a sniffer tag into datatypes_conf.xml, but I get the same result whether the sniffer tag is there or not. I am sure I have missed a step or piece of content but I have no idea what I am missing... Thanks again for your time, Darren Cullerne Computational Biology CSIRO Plant Industry x5042 +61 2 62465042 darren.culle...@csiro.au mailto:darren.culle...@csiro.au ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Tools: dynamic check of file type?
Well I'm still missing something... Is there a way to use the conditional tag sets in the tool's xml with it? To suppress the unnecessary options for a given file type? or more than one? regards, L-A Le 20/05/2011 13:15, Louise-Amélie Schmitt a écrit : It's exactly what I needed!!! Thanks a lot! L-A Le 20/05/2011 12:29, Peter Cock a écrit : 2011/5/20 Louise-Amélie Schmittlouise-amelie.schm...@embl.de: Hello Is there a way to check the file type with a conditional tag set within a tool? For example, if the tool can accept several file types but I have to make cheetah shut off some stuff for a couple of them? You can do conditional code in the cheetah template using $input.ext or $input.extension which both give you the Galaxy file type of an input file parameter called $input. For example, look at tools/fastx_toolkit/fastx_clipper.xml which does: #if $input.ext == fastqsanger: -Q 33 #end if Alternatively, if you want to do this in the script being called by Galaxy, just pass $input.ext as part of the command line. Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Specifying number of requested cores to Galaxy DRMAA
Hi again Leandro
Well I might not have been really clear, perhaps I should have re-read
the mail before posting it :)
The thing is, it was not an issue of Torque starting jobs when there
were not enough resources available, but rather it believing the needed
resources for each job being fewer that they were (e.g. always assuming
the jobs were single-threaded even if the actual tools needed more tan
one core). if Torque is properly notified of the needed resources, it
will dispatch them or make them wait accordingly (since it knows the
nodes' limits and load), like your LSF does.
This hack is not very sexy but it just notifies Torque of the cores
needed by every multithreaded tool, so it doesn't run a multithreaded
job when there's only one core available in the chosen node.
Hope that helps :)
Regards,
L-A
On 05/19/2011 03:05 PM, Leandro Hermida wrote:
Hi Louise-Amelie,
Thank you for the post reference, this is exactly what I was looking
for. For us for for example when I want to execute a tool that is a
Java command the JVM typically will typically use multiple cores as
it's running. You said with TORQUE it will crash when there aren't
enough resources when the job is submitted. I wonder if you can do
the same thing we have done here with LSF? With LSF you can configure
a maximum server load for each node and if the submitted jobs push the
node load above this threshold (e.g. more cores requested than
available) LSF will temporarily suspend jobs (using some kind of
heuristics) so that the load stays below the threshold and unsuspend
as resources become available. So for us things just will run slower
when we cannot pass the requested number of cores to LSF.
I would think maybe there is a way with TORQUE to have it achieve the
same thing so jobs don't crash when resources requested are more than
available?
regards,
Leandro
2011/5/19 Louise-Amélie Schmitt louise-amelie.schm...@embl.de
mailto:louise-amelie.schm...@embl.de
Hi,
In a previous message, I explained how I did to multithreads
certain jobs, perhaps you can modify the corresponding files for
drmaa in a similar way:
On 04/26/2011 11:26 AM, Louise-Amélie Schmitt wrote:
Just one little fix on line 261:
261 if ( len(l) 1 and l[0] ==job_wrapper.tool.id
http://job_wrapper.tool.id ):
Otherwise it pathetically crashes when non-multithreaded jobs are
submitted. Sorry about that.
Regards,
L-A
Le mardi 19 avril 2011 à 14:33 +0200, Louise-Amélie Schmitt a écrit :
Hello everyone,
I'm using TORQUE with Galaxy, and we noticed that if a tool is
multithreaded, the number of needed cores is not communicated to pbs,
leading to job crashes if the required resources are not available when
the job is submitted.
Therefore I modified a little the code as follows in
lib/galaxy/jobs/runners/pbs.py
256 # define PBS job options
257 attrs.append( dict( name = pbs.ATTR_N, value = str( %s_%s_%
s % ( job_wrapper.job_id,job_wrapper.tool.id
http://job_wrapper.tool.id, job_wrapper.user ) ) ) )
258 mt_file = open('tool-data/multithreading.csv', 'r')
259 for l in mt_file:
260 l = string.split(l)
261 if ( l[0] ==job_wrapper.tool.id
http://job_wrapper.tool.id ):
262 attrs.append( dict( name = pbs.ATTR_l,
resource = 'nodes', value = '1:ppn='+str(l[1]) ) )
263 attrs.append( dict( name = pbs.ATTR_l,
resource = 'mem', value = str(l[2]) ) )
264 break
265 mt_file.close()
266 job_attrs = pbs.new_attropl( len( attrs ) +
len( pbs_options ) )
(sorry it didn't come out very well due to line breaking)
The csv file contains a list of the multithreaded tools, each line
containing:
tool id\tnumber of threads\tmemory needed\n
And it works fine, the jobs wait for their turn properly, but
information is duplicated. Perhaps there would be a way to include
something similar in galaxy's original code (if it is not already the
case, I may not be up-to-date) without duplicating data.
I hope that helps :)
Best regards,
L-A
___
The Galaxy User list should be used for the discussion of
Galaxy analysis and other features on the public server
atusegalaxy.org http://usegalaxy.org. Please keep all replies on the
list by
using reply all in your mail client. For discussion of
local Galaxy instances and the Galaxy source code, please
use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists,
please use the interface at:
http://lists.bx.psu.edu/
On 05/19/2011 12:03 PM, Leandro Hermida wrote:
Hi,
When
Re: [galaxy-dev] Specifying number of requested cores to Galaxy DRMAA
Hi again Oh I see... Not the best way to deal with the resources indeed but better than nothing though. I do run Java tools via Galaxy but I haven't paid attention to this issue, so I can't really help. It's a homemade tool that only has one class so I guess it's not worth the effort in my case. But if you find the answer, I'd be interested too. Good luck, L-A On 05/19/2011 04:28 PM, Leandro Hermida wrote: Hi Louise, I see thank you for the response, maybe there was some confusion, the feature I was trying to explain with LSF is that you *don't* need to tell it the required resources for a job and it will still be able to run all the submitted jobs on a node without crashing even if the jobs submitted need e.g. 10 more cores are available (that is 10 more cores than LSF thought it needed). LSF will just temporarily suspend jobs in mid-run on a node to keep the load down, but nothing will ever crash even if you are running jobs that require 20 CPUs and you only have 2. Thought maybe there was a way to do this with TORQUE. If LSF or TORQUE are explcitly passed the resources needed then they will never need to temporarily suspend because they will pick a node with those resources free. That being said, your method is more efficient for this reason as it will pick the right node with the cores available instead of picking a node with maybe just one core available and then running the multithreaded job slower because it has to periodically suspend it as it is running. Also I wonder do you run any Java command-line tools via Galaxy? I can't seem to find with the JVM exactly how many cores it needs during execution or how to limit it to a certain max, it just jumps around in CPU usage from 50% to over 400% regards, Leandro 2011/5/19 Louise-Amélie Schmitt louise-amelie.schm...@embl.de mailto:louise-amelie.schm...@embl.de Hi again Leandro Well I might not have been really clear, perhaps I should have re-read the mail before posting it :) The thing is, it was not an issue of Torque starting jobs when there were not enough resources available, but rather it believing the needed resources for each job being fewer that they were (e.g. always assuming the jobs were single-threaded even if the actual tools needed more tan one core). if Torque is properly notified of the needed resources, it will dispatch them or make them wait accordingly (since it knows the nodes' limits and load), like your LSF does. This hack is not very sexy but it just notifies Torque of the cores needed by every multithreaded tool, so it doesn't run a multithreaded job when there's only one core available in the chosen node. Hope that helps :) Regards, L-A On 05/19/2011 03:05 PM, Leandro Hermida wrote: Hi Louise-Amelie, Thank you for the post reference, this is exactly what I was looking for. For us for for example when I want to execute a tool that is a Java command the JVM typically will typically use multiple cores as it's running. You said with TORQUE it will crash when there aren't enough resources when the job is submitted. I wonder if you can do the same thing we have done here with LSF? With LSF you can configure a maximum server load for each node and if the submitted jobs push the node load above this threshold (e.g. more cores requested than available) LSF will temporarily suspend jobs (using some kind of heuristics) so that the load stays below the threshold and unsuspend as resources become available. So for us things just will run slower when we cannot pass the requested number of cores to LSF. I would think maybe there is a way with TORQUE to have it achieve the same thing so jobs don't crash when resources requested are more than available? regards, Leandro 2011/5/19 Louise-Amélie Schmitt louise-amelie.schm...@embl.de mailto:louise-amelie.schm...@embl.de Hi, In a previous message, I explained how I did to multithreads certain jobs, perhaps you can modify the corresponding files for drmaa in a similar way: On 04/26/2011 11:26 AM, Louise-Amélie Schmitt wrote: Just one little fix on line 261: 261 if ( len(l) 1 and l[0] ==job_wrapper.tool.id http://job_wrapper.tool.id ): Otherwise it pathetically crashes when non-multithreaded jobs are submitted. Sorry about that. Regards, L-A Le mardi 19 avril 2011 à 14:33 +0200, Louise-Amélie Schmitt a écrit : Hello everyone, I'm using TORQUE with Galaxy, and we noticed that if a tool is multithreaded, the number of needed cores is not communicated to pbs, leading to job crashes if the required resources are not available when the job is submitted. Therefore I modified a little the code as follows in lib
Re: [galaxy-dev] user and dataset management with LDAP: some questions
Hello everyone I'm updating the remaining questions: 1) We use LDAP for user registration and logging, would it be possible to retrieve automatically the LDAP users' groups and create the groups in Galaxy accordigly? (and of course put the user in their respective groups) 2) Is it possible, still using LDAP, to delete a user and all his datasets? 3) Is it possible to automatically suppress, for instance, any dataset that was added more than a couple of months ago? 4) Is there a not-so-intrusive way to add a column in the user list showing the disk space they use respectively? Thanks in advance for your help Regards, L-A Le mardi 12 avril 2011 à 16:41 +0200, Louise-Amélie Schmitt a écrit : Hello everyone I have a couple of questions regarding user and dataset management. 1) We use LDAP for user registration and logging, would it be possible to retrieve automatically the LDAP users' groups and create the groups in Galaxy accordigly? (and of course put the user in their respective groups) 2) Is it possible, still using LDAP, to delete a user and all his datasets? 3) Is it possible to automatically suppress, for instance, any dataset that was added more than a couple of months ago? 4) Is there a not-so-intrusive way to add a column in the user list showing the disk space they use respectively? 5) I tried to see how the API works but i have to admit I didn't get a thing. I read the scripts/api/README file and there I saw one needs to access the user preferences to generate an API key. What is its purpose? Is there a way to do it when you use LDAP (therefore no access to that key generator) ? Sorry, this is a bit random but I'm kinda drowning here, since I'm not used to manipulating apps this huge. Thanks for your help and patience. Cheers, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Error when adding datasets
Hello again I tried deleting the compiled templates like Sarah Diehl did for her user registration issue: first thank you for your help. I tried reloading everything, cleared the cache of my browser and tried it on a different computer whose browser never before was on the Galaxy website. All of this did not work. What worked in the end was deleting the compiled_templates directory. But it didn't work either. I'm back to square one, with no option in sight. I keep getting the same error message when trying to add datasets. Any idea? Cheers, L-A Le vendredi 15 avril 2011 à 16:52 +0200, Louise-Amélie Schmitt a écrit : Ooops, this is very right, I totally forgot about that. It woud have become problematic at some point I guess. Thank you for pointing this out! I changed it so the new database is associated with brand new appropriate directories. (and dropped and re-created the db again) But I keep getting the same error message with supposedly nonexistent histories. Thanks again L-A Le vendredi 15 avril 2011 à 13:47 +0200, Hans-Rudolf Hotz a écrit : On 04/15/2011 10:56 AM, Louise-Amélie Schmitt wrote: Hi Hans, thanks for your reply. - is your PostgreSQL database in sync with the database folder? I'm sorry I'm not sure I get what you mean, what folder is it? the folder where you keep the datasets, eg: your local path/galaxy_dist/database/ and it particular your local path/galaxy_dist/database/files/000/dataset_*.dat if you have an empty PostgreSQL database, then there should also be no datasets. Hans - are you sure there is no second galaxy installation accessing the same PostgreSQL database? Never been this sure of something before. :) I installed both galaxy and postgresql myself so I can guarantee there's ony one galaxy install on it. - you mention, that you have started with an empty PostgreSQL database, so this last question does probably not apply: make sure you have the right user names (you even run into trouble with case sensitivity) in your PostgreSQL database. We had troubles when we switched to external authentication: some users just couldn't work anymore, ie could not create new history items anymore. The problem was then solved by fixing their user names in the (MySQL) database. Yes, the username is properly created in the database when I log in on an empty database. We never used anything besides LDAP anyway. Regards, Hans Cheers, L-A On Apr 14, 2011, at 1:00 PM, Louise-Amélie Schmitt wrote: The thing is, we use LDAP logging so we can't even access the website without logging in. Moreover, when I logged in, I arrived on the data analysis page where the automatic unnamed history was obviously created in the history panel. I forgot to mention we have issues creating and deleting histories, like, we can't access some histories and everytime we delete histories, two extra unnamed histories are created. As I mentioned before, it is also impossible to load a dataset in any history, existing or not. Do you think it could be linked to our using LDAP? Thanks L-A On Thu, 14 Apr 2011 12:06:55 -0400, Greg Von Kusterg...@bx.psu.edu wrote: On Apr 14, 2011, at 11:49 AM, Louise-Amélie Schmitt wrote: Here is the result I got from the debug statements: galaxy.web.controllers.library_common DEBUG 2011-04-14 17:46:02,286 ### history: None This is the problem - when you registered normally, a history would have been automatically created for you. But somehow you don't have a history - no idea why / how this happened, but it is very unlikely this is a result of a bug within Galaxy. Try logging out and logging in again and a new history should be created for you. galaxy.web.controllers.library_common DEBUG 2011-04-14 17:46:02,286 ### trans:galaxy.web.framework.GalaxyWebUITransaction object at 0x29f40950 galaxy.web.controllers.library_common DEBUG 2011-04-14 17:46:02,286 ### trans.sa_session:sqlalchemy.orm.scoping.ScopedSession object at 0x19ab21d0 Thanks again L-A Le jeudi 14 avril 2011 à 11:05 -0400, Greg Von Kuster a écrit : I assume when you dropped the old database and recreated the new, empty database, you made sure the database connection string in universe_wsgi.ini was correctly set. if so, when you started up Galaxy, it would have created all of the required tables in the new database, and they would all be empty. When you first registered as the admin user, it would have automatically populated several tables with data, including the history table. One or more of these things must not have been successful. To attempt to narrow down the problem, I'll need you to do
Re: [galaxy-dev] Error when adding datasets
Ooops, this is very right, I totally forgot about that. It woud have become problematic at some point I guess. Thank you for pointing this out! I changed it so the new database is associated with brand new appropriate directories. (and dropped and re-created the db again) But I keep getting the same error message with supposedly nonexistent histories. Thanks again L-A Le vendredi 15 avril 2011 à 13:47 +0200, Hans-Rudolf Hotz a écrit : On 04/15/2011 10:56 AM, Louise-Amélie Schmitt wrote: Hi Hans, thanks for your reply. - is your PostgreSQL database in sync with the database folder? I'm sorry I'm not sure I get what you mean, what folder is it? the folder where you keep the datasets, eg: your local path/galaxy_dist/database/ and it particular your local path/galaxy_dist/database/files/000/dataset_*.dat if you have an empty PostgreSQL database, then there should also be no datasets. Hans - are you sure there is no second galaxy installation accessing the same PostgreSQL database? Never been this sure of something before. :) I installed both galaxy and postgresql myself so I can guarantee there's ony one galaxy install on it. - you mention, that you have started with an empty PostgreSQL database, so this last question does probably not apply: make sure you have the right user names (you even run into trouble with case sensitivity) in your PostgreSQL database. We had troubles when we switched to external authentication: some users just couldn't work anymore, ie could not create new history items anymore. The problem was then solved by fixing their user names in the (MySQL) database. Yes, the username is properly created in the database when I log in on an empty database. We never used anything besides LDAP anyway. Regards, Hans Cheers, L-A On Apr 14, 2011, at 1:00 PM, Louise-Amélie Schmitt wrote: The thing is, we use LDAP logging so we can't even access the website without logging in. Moreover, when I logged in, I arrived on the data analysis page where the automatic unnamed history was obviously created in the history panel. I forgot to mention we have issues creating and deleting histories, like, we can't access some histories and everytime we delete histories, two extra unnamed histories are created. As I mentioned before, it is also impossible to load a dataset in any history, existing or not. Do you think it could be linked to our using LDAP? Thanks L-A On Thu, 14 Apr 2011 12:06:55 -0400, Greg Von Kusterg...@bx.psu.edu wrote: On Apr 14, 2011, at 11:49 AM, Louise-Amélie Schmitt wrote: Here is the result I got from the debug statements: galaxy.web.controllers.library_common DEBUG 2011-04-14 17:46:02,286 ### history: None This is the problem - when you registered normally, a history would have been automatically created for you. But somehow you don't have a history - no idea why / how this happened, but it is very unlikely this is a result of a bug within Galaxy. Try logging out and logging in again and a new history should be created for you. galaxy.web.controllers.library_common DEBUG 2011-04-14 17:46:02,286 ### trans:galaxy.web.framework.GalaxyWebUITransaction object at 0x29f40950 galaxy.web.controllers.library_common DEBUG 2011-04-14 17:46:02,286 ### trans.sa_session:sqlalchemy.orm.scoping.ScopedSession object at 0x19ab21d0 Thanks again L-A Le jeudi 14 avril 2011 à 11:05 -0400, Greg Von Kuster a écrit : I assume when you dropped the old database and recreated the new, empty database, you made sure the database connection string in universe_wsgi.ini was correctly set. if so, when you started up Galaxy, it would have created all of the required tables in the new database, and they would all be empty. When you first registered as the admin user, it would have automatically populated several tables with data, including the history table. One or more of these things must not have been successful. To attempt to narrow down the problem, I'll need you to do the following. Here are lines # 905 - 907 in ~/lib/galaxy/web/controllers/library-common.py # Send the current history to the form to enable importing datasets from history to library history = trans.get_history() trans.sa_session.refresh( history ) Please add the following debug statements: # Send the current history to the form to enable importing datasets from history to library history = trans.get_history() log.debug(### history: %s % str( history )) log.debug(### trans: %s % str( trans )) log.debug(### trans.sa_session: %s % str( trans.sa_session )) trans.sa_session.refresh( history ) Stop and restart your Galaxy server after making the above changes
Re: [galaxy-dev] Error when adding datasets
The thing is, we use LDAP logging so we can't even access the website without logging in. Moreover, when I logged in, I arrived on the data analysis page where the automatic unnamed history was obviously created in the history panel. I forgot to mention we have issues creating and deleting histories, like, we can't access some histories and everytime we delete histories, two extra unnamed histories are created. As I mentioned before, it is also impossible to load a dataset in any history, existing or not. Do you think it could be linked to our using LDAP? Thanks L-A On Thu, 14 Apr 2011 12:06:55 -0400, Greg Von Kuster g...@bx.psu.edu wrote: On Apr 14, 2011, at 11:49 AM, Louise-Amélie Schmitt wrote: Here is the result I got from the debug statements: galaxy.web.controllers.library_common DEBUG 2011-04-14 17:46:02,286 ### history: None This is the problem - when you registered normally, a history would have been automatically created for you. But somehow you don't have a history - no idea why / how this happened, but it is very unlikely this is a result of a bug within Galaxy. Try logging out and logging in again and a new history should be created for you. galaxy.web.controllers.library_common DEBUG 2011-04-14 17:46:02,286 ### trans: galaxy.web.framework.GalaxyWebUITransaction object at 0x29f40950 galaxy.web.controllers.library_common DEBUG 2011-04-14 17:46:02,286 ### trans.sa_session: sqlalchemy.orm.scoping.ScopedSession object at 0x19ab21d0 Thanks again L-A Le jeudi 14 avril 2011 à 11:05 -0400, Greg Von Kuster a écrit : I assume when you dropped the old database and recreated the new, empty database, you made sure the database connection string in universe_wsgi.ini was correctly set. if so, when you started up Galaxy, it would have created all of the required tables in the new database, and they would all be empty. When you first registered as the admin user, it would have automatically populated several tables with data, including the history table. One or more of these things must not have been successful. To attempt to narrow down the problem, I'll need you to do the following. Here are lines # 905 - 907 in ~/lib/galaxy/web/controllers/library-common.py # Send the current history to the form to enable importing datasets from history to library history = trans.get_history() trans.sa_session.refresh( history ) Please add the following debug statements: # Send the current history to the form to enable importing datasets from history to library history = trans.get_history() log.debug(### history: %s % str( history )) log.debug(### trans: %s % str( trans )) log.debug(### trans.sa_session: %s % str( trans.sa_session )) trans.sa_session.refresh( history ) Stop and restart your Galaxy server after making the above changes, and reply back with the output of the debug statements. Assuming you start your Galaxy instance with: sh run.sh you'll see the results of the debug statements in the log scrolling in the window in which you started Galaxy. Thanks On Apr 14, 2011, at 10:46 AM, Louise-Amélie Schmitt wrote: Hello Greg Thank you for answering. Please find the answers after each question. Le jeudi 14 avril 2011 à 10:19 -0400, Greg Von Kuster a écrit : Hello Louise, I do not think this issue is related to the Galaxy eggs, but instead looks like a data issue of some kind. Please replly back with answers to the following questions. How did you create your database? Couldn't have done it more simply ^^: CREATE DATABASE galaxy_db; GRANT ALL ON DATABASE galaxy_db TO galaxy; executed in psql. The very same way I did for the one that's still working fine. Did you populate it with any data exported from another database? In the beginning yes but when I saw that error I dropped the database and re-created it again, to start anew and test on an empty database. I even created a brand new test database to see if the issue wasn't related to stuff remaining after dropping the database, but it turned out badly too. What version of Galaxy are you using? The latest dist, cause when I saw how things were turning out I hg pulled and updated hoping it would fix the issue. I did that this morning. What database are you using? PostgreSQL, as recommended in the doc. Greg Von Kuster Cheers, L-A On Apr 14, 2011, at 5:38 AM, Louise-Amélie Schmitt wrote: Hello everyone I have an issue when trying to import new datasets or when putting a dataset into a history. I saw Edward Kirton had the same problem but he got no answer: http://lists.bx.psu.edu/pipermail/galaxy-dev/2010-May/002732.html Here is the error message I get when clicking the Add datasets button in a library, in the admin's Manage data libraries panel: UnmappedInstanceError: Class '__builtin__.NoneType
Re: [galaxy-dev] PostgreSQL issue: reserved keyword used as identifier
On Fri, 8 Apr 2011 16:33:58 -0400, Kanwei Li kan...@gmail.com wrote: Hi Louise, Have you considered doing a SQL dump and import? Sounds easier to me than writing a perl script ;) -K Oh yes I did... Yeah, that solution sounded so sweet that is was the first thing I tried. But don't be fooled by the apparent easyness, it's another whole ordeal to go through. If it was MySQL to MySQL or PSQL to PSQL, yeah that would be the easiest and fastest way. But the syntax/structure used by both is very different (MySQL is awfully non-standard), and there's no batteries-included way to do the translation, just a few scripts here and there. You have to edit stuff manually anyway. An example: everywhere I see people giving the --compatible=postgresql option of mysqldump as the obvious solution of all problems, but when I tried it and searched about it, I really wondered why they provide this option at all. If you want more information about it you should check this website out: http://en.wikibooks.org/wiki/Converting_MySQL_to_PostgreSQL When I saw the mess it was, I told myself that since the tables are automatically generated, I might as well make the most of it and only transfer the data. I tried using Kettle: http://pinoytech.org/question/5417386/import-mysql-dump-to-postgresql-database but in the end I got job crashes and error message like something crashed, period so since I didn't want to waste too much time on trying to make this steam machine work, I'd rather make my own lil' script. Yeah, in the end, it was the easiest way to me, believe me or not. ^^ Cheers ;) L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] PostgreSQL issue: reserved keyword used as identifier
Hello everyone I just met a huge problem concerning the database. I'm currently trying to transfer my data from MySQL to PostgreSQL by writing a Perl script that would do the job. Here is the issue: In the form_definition table, one of the field identifiers is desc, which is a reserved SQL keyword used for ordering values. Therefore, There's currently no way of making any query of the type INSERT INTO table_name (identifiers list) VALUES (values list); which is a big handicap in this context, since the order of the identifiers list we dynamically retrieve is not necessarily (and seldom) the right order. Is there a way to fix this issue without blowing everything up? Cheers, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] PostgreSQL issue: reserved keyword used as identifier
Le vendredi 08 avril 2011 à 11:12 -0400, Sean Davis a écrit : 2011/4/8 Louise-Amélie Schmitt louise-amelie.schm...@embl.de: Hello everyone I just met a huge problem concerning the database. I'm currently trying to transfer my data from MySQL to PostgreSQL by writing a Perl script that would do the job. Here is the issue: In the form_definition table, one of the field identifiers is desc, which is a reserved SQL keyword used for ordering values. Therefore, There's currently no way of making any query of the type INSERT INTO table_name (identifiers list) VALUES (values list); which is a big handicap in this context, since the order of the identifiers list we dynamically retrieve is not necessarily (and seldom) the right order. Is there a way to fix this issue without blowing everything up? You need to quote the identifiers. A simple example using desc as a column name: sdavis=# create table test_table( id int, desc varchar); ERROR: syntax error at or near desc LINE 3: desc varchar); ^ sdavis=# create table test_table( id int, desc varchar); CREATE TABLE Hope that helps. Sean Thanks, but I cheated by duplicating a little the information and it works fine. I use insert into table values (...);. I still have problems transfering the data though. The script runs without growling but nothing is actually written in the target tables... I'll have to see that on monday now :/ If I succeed, I'll let you know. Cheers, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] new_file_path vs $TMP
Hello We managed to install Galaxy according to the unified method, with the runner and the web application running on separate machines sharing by NFS the same storage space where the Galaxy files are. The thing is, the data must be saved in another NFS storage space so we had to define the file paths accordingly, including modifying (and exporting) the TMP environment variables for both machines, so that the temporary directory is in the shared NFS volume meant to host the data. My question is: What exactly is the new_file_path configuration value for since it doesn't seem to have any effect if the TMP environment variable doesn't do the job? Wouldn't there be a solution for using the new_file_path value instead of having to rely on the shell variables for temporary files? Best regards L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
