Re: [galaxy-dev] ProFTPd in CloudMan
Great! Thanks! 2015-09-02 8:47 GMT-05:00 Enis Afgan <enis.af...@irb.hr>: > Thanks for reporting this (and figuring out the solution). It's been fixed > with this commit > <https://github.com/galaxyproject/cloudman/commit/550d90cbde49f37d6d7bdbea3170e6f6085ca613> > so any new cluster will work fine. > > Cheers. > > On Tue, Sep 1, 2015 at 7:42 PM, Alexander Vowinkel < > vowinkel.alexan...@gmail.com> wrote: > >> Update: >> >> ProFTP gives as an answer to passive mode the INTERNAL IP to connect >> which makes FileZilla break the connection. >> >> Indeed adding MasqueradeAddressin the config file, it >> works. >> >> Where is the proftp config file being created in cloudman? How can this >> be added dynamically? >> >> Best, >> Alexander >> >> 2015-09-01 18:19 GMT-05:00 Alexander Vowinkel < >> vowinkel.alexan...@gmail.com>: >> >>> Hi, >>> >>> I get the error "Could not read from socket: ECONNRESET - Connection >>> reset by peer" when connecting with FileZilla to the Cloud Instance. >>> Using launch.usegalaxy.org >>> >>> FileZilla Log: http://pastebin.com/zfACC6wH >>> ProFTP Log: http://pastebin.com/T9JxKVCb >>> >>> With console FTP in linux I can connect, get a listing and upload files. >>> I tried to find out what the reason is, but no solution found. >>> bareFTP is also not able to connect. >>> >>> The ports for PassivePorts are open (checked in AWS) >>> >>> Any idea on this? >>> >>> Thanks, >>> Alexander >>> >> >> > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] BUG: Collections and History export
Hi, when downloading/exporting a history via the interface, visible collections are not being exported. The contents of the collection(s) are hidden, the Colleciton itself is visible. Best, Alexander ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] ProFTPd in CloudMan
Update: ProFTP gives as an answer to passive mode the INTERNAL IP to connect which makes FileZilla break the connection. Indeed adding MasqueradeAddressin the config file, it works. Where is the proftp config file being created in cloudman? How can this be added dynamically? Best, Alexander 2015-09-01 18:19 GMT-05:00 Alexander Vowinkel <vowinkel.alexan...@gmail.com> : > Hi, > > I get the error "Could not read from socket: ECONNRESET - Connection reset > by peer" when connecting with FileZilla to the Cloud Instance. > Using launch.usegalaxy.org > > FileZilla Log: http://pastebin.com/zfACC6wH > ProFTP Log: http://pastebin.com/T9JxKVCb > > With console FTP in linux I can connect, get a listing and upload files. > I tried to find out what the reason is, but no solution found. > bareFTP is also not able to connect. > > The ports for PassivePorts are open (checked in AWS) > > Any idea on this? > > Thanks, > Alexander > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] cancelled by slurm - job is fine
Hi, I have a big problem here. Jobs that are cancelled by slurm appear to galaxy as finished normally. For me this is especially bad because all following workflow steps go on working with corrupted/empty/whatever data. in the job stderr I can find: slurmd[w4]: *** JOB 194 CANCELLED AT 2015-08-06T04:04:38 *** slurmd[w4]: Unable to unlink domain socket: No such file or directory slurmd[w4]: unlink(/tmp/slurm/slurmd_spool/job00194/slurm_script): No such file or directory slurmd[w4]: rmdir(/tmp/slurm/slurmd_spool/job00194): No such file or directory In the galaxy log for that job: galaxy.jobs.runners.drmaa DEBUG 2015-08-06 04:02:10,050 (4278) submitting file /mnt/galaxy/tmp/job_working_directory/004/4278/galaxy_4278.sh galaxy.jobs.runners.drmaa INFO 2015-08-06 04:02:10,056 (4278) queued as 192galaxy.jobs DEBUG 2015-08-06 04:02:10,185 (4278) Persisting job destination (destination id: slurm_cluster) [...] galaxy.jobs.runners.drmaa DEBUG 2015-08-06 04:04:39,525 (4278/192) state change: job finished normallygalaxy.jobs DEBUG 2015-08-06 04:04:45,806 job 4278 ended (finish() executed in (5290.522 ms)) galaxy.datatypes.metadata DEBUG 2015-08-06 04:04:45,837 Cleaning up external metadata files galaxy.datatypes.metadata DEBUG 2015-08-06 04:04:46,100 Failed to cleanup MetadataTempFile temp files from /mnt/galaxy/tmp/job_working_directory/004/4278/metadata_out_HistoryDatasetAssociation_9426_Ib1Niz: No JSON object could be decoded galaxy.datatypes.metadata DEBUG 2015-08-06 04:04:46,397 Failed to cleanup MetadataTempFile temp files from /mnt/galaxy/tmp/job_working_directory/004/4278/metadata_out_HistoryDatasetAssociation_9427_8X77j4: No JSON object could be decoded Can someone please check that? Best, Alexander ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] configuring job runner slurm on cloudman
Hi, can someone tell me something about this? Pointing at the runner stated in the log vs. expected runner. Thanks, Alexander 2015-07-06 18:08 GMT-05:00 Alexander Vowinkel vowinkel.alexan...@gmail.com : Hi, I have adopted /mnt/galaxy/galaxy-app/config/job_conf.xml: I added a destination with following param: param id=nativeSpecification-N1 --cpus-per-task 1 --mem-per-cpu=4G/param Galaxy service was restarted after chaning this. Now I get the error Unable to run job due to a misconfiguration of the Galaxy job running system. Please contact a site administrator. when I try to run a job. Looking in the log gives: galaxy.jobs.runners.drmaa DEBUG 2015-07-06 22:50:40,187 (493) submitting file /mnt/galaxy/tmp/job_working_directory/000/493/galaxy_493.sh galaxy.jobs.runners.drmaa DEBUG 2015-07-06 22:50:40,187 (493) native specification is: -N1 --cpus-per-task 1 --mem-per-cpu=4G galaxy.jobs.runners.drmaa ERROR 2015-07-06 22:50:40,187 (493) drmaa.Session.runJob() failed unconditionally Traceback (most recent call last): [...] InvalidAttributeValueException: code 14: Invalid native specification: -N1 --cpus-per-task 1 --mem-per-cpu=4G Well. running this as galaxy user in the console with $ srun -N1 --cpus-per-task 1 --mem-per-cpu=4G /mnt/galaxy/tmp/job_working_directory/000/494/galaxy_494.sh , it actually runs good. I was wondering if this is connected with the fact that the log states galaxy.jobs.runners.drmaa and not something like galaxy.jobs.runners.slurm, like defined in the job_conf.xml: plugin id=slurm type=runner load=galaxy.jobs.runners.slurm:SlurmJobRunner / So - What is going wrong here? Am I working on the wrong job_conf.xml? What does galaxy do here? Thanks for help!Alexander ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] configuring job runner slurm on cloudman
I finally found out that this native specification is NOT about the native parameters of slurm. It comes from the DRMA-wrapper of slurm: http://apps.man.poznan.pl/trac/slurm-drmaa/wiki/WikiStart#Nativespecification And this does not support mem-per-cpu. Which is a huge restriction. The logger seems to print drmaa because it is the base class of the SlurmRunner. Anybody stumbled upon this before? A way would be to have a slurm runner that uses native calls instead of drmaa. Pros/Cons? Any opinions on this topic? Or solutions to get the mem-per-cpu option? I read, that this is criticised by some people from the slurm-drmaa community too. Patches are available, but not included in a release. Best, Alexander 2015-07-31 11:38 GMT-05:00 Alexander Vowinkel vowinkel.alexan...@gmail.com : Hi, can someone tell me something about this? Pointing at the runner stated in the log vs. expected runner. Thanks, Alexander 2015-07-06 18:08 GMT-05:00 Alexander Vowinkel vowinkel.alexan...@gmail.com: Hi, I have adopted /mnt/galaxy/galaxy-app/config/job_conf.xml: I added a destination with following param: param id=nativeSpecification-N1 --cpus-per-task 1 --mem-per-cpu=4G/param Galaxy service was restarted after chaning this. Now I get the error Unable to run job due to a misconfiguration of the Galaxy job running system. Please contact a site administrator. when I try to run a job. Looking in the log gives: galaxy.jobs.runners.drmaa DEBUG 2015-07-06 22:50:40,187 (493) submitting file /mnt/galaxy/tmp/job_working_directory/000/493/galaxy_493.sh galaxy.jobs.runners.drmaa DEBUG 2015-07-06 22:50:40,187 (493) native specification is: -N1 --cpus-per-task 1 --mem-per-cpu=4G galaxy.jobs.runners.drmaa ERROR 2015-07-06 22:50:40,187 (493) drmaa.Session.runJob() failed unconditionally Traceback (most recent call last): [...] InvalidAttributeValueException: code 14: Invalid native specification: -N1 --cpus-per-task 1 --mem-per-cpu=4G Well. running this as galaxy user in the console with $ srun -N1 --cpus-per-task 1 --mem-per-cpu=4G /mnt/galaxy/tmp/job_working_directory/000/494/galaxy_494.sh , it actually runs good. I was wondering if this is connected with the fact that the log states galaxy.jobs.runners.drmaa and not something like galaxy.jobs.runners.slurm, like defined in the job_conf.xml: plugin id=slurm type=runner load=galaxy.jobs.runners.slurm:SlurmJobRunner / So - What is going wrong here? Am I working on the wrong job_conf.xml? What does galaxy do here? Thanks for help!Alexander ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] CloudMan auto-scaling - Use spot instance for worker node?
Hi, I am referring to this: http://dev.list.galaxyproject.org/CloudMan-auto-scaling-Use-spot-instance-for-worker-node-tp4665813p4665835.html I would give an upvote for spot instances for auto-scaling. Have there been more ideas about this since? Best, Alexander ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] collection problem
I did the runs now just in a single history. the collections are empty (I checked also in the API) 2015-07-15 18:49 GMT-05:00 Alexander Vowinkel vowinkel.alexan...@gmail.com : Hi, I have this tool that creates a collection by discovery datasets. It always worked fine, but this time I had 3 Histories, where I ran the tool on three different datasets. When it finished, the collections were empty. The datasets, that should be in the collections however, were there. Is that known? Is that fixed in 15.07? Thanks, Alexander ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] cloudlaunch not working
Hi guys, cloudlounch at launch.usegalaxy.org doesn't work for me anymore. Some hours ago I still was able to start a cluster, but now it seems that the launcher can't connect to my amazon anymore? No machnie get's started. But I'm still able to fetch key pair and placement. Any suggestions what to try? Already: Reloaded page, Restarted PC. Thanks, Alexander ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] cloudman galaxy - too large volume
Great! Especially also witht he IO performance. Is it possible to define the volume type manually by putting 'volume_type' to the user data? So I can also have smaller volumes with SSD? Best, Alexander 2015-07-14 13:57 GMT-05:00 Enis Afgan enis.af...@irb.hr: I just fixed this here https://github.com/galaxyproject/cloudman/commit/0f6ab132958ad57a338c66387112407edbc7632d#diff-9eba488c9175de87945a13d7b7228a6dR621 This required switching to SSD-based volumes on AWS and, besides actually being able to create volumes up to 16TB now, it should give us better IO performance now. On Mon, Jul 13, 2015 at 5:06 PM, Alexander Vowinkel vowinkel.alexan...@gmail.com wrote: Hi, I get this error on cluster creation: Error creating volume: EC2ResponseError: 400 Bad Request InvalidParameterValueVolume of 2048GiB is too large for volume type standard; maximum is 1024GiB Is there a way to get a volume that big for galaxy? Best, Alexander ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] configuring job runner slurm on cloudman
Hi, I have adopted /mnt/galaxy/galaxy-app/config/job_conf.xml: I added a destination with following param: param id=nativeSpecification-N1 --cpus-per-task 1 --mem-per-cpu=4G/param Galaxy service was restarted after chaning this. Now I get the error Unable to run job due to a misconfiguration of the Galaxy job running system. Please contact a site administrator. when I try to run a job. Looking in the log gives: galaxy.jobs.runners.drmaa DEBUG 2015-07-06 22:50:40,187 (493) submitting file /mnt/galaxy/tmp/job_working_directory/000/493/galaxy_493.sh galaxy.jobs.runners.drmaa DEBUG 2015-07-06 22:50:40,187 (493) native specification is: -N1 --cpus-per-task 1 --mem-per-cpu=4G galaxy.jobs.runners.drmaa ERROR 2015-07-06 22:50:40,187 (493) drmaa.Session.runJob() failed unconditionally Traceback (most recent call last): [...] InvalidAttributeValueException: code 14: Invalid native specification: -N1 --cpus-per-task 1 --mem-per-cpu=4G Well. running this as galaxy user in the console with $ srun -N1 --cpus-per-task 1 --mem-per-cpu=4G /mnt/galaxy/tmp/job_working_directory/000/494/galaxy_494.sh , it actually runs good. I was wondering if this is connected with the fact that the log states galaxy.jobs.runners.drmaa and not something like galaxy.jobs.runners.slurm, like defined in the job_conf.xml: plugin id=slurm type=runner load=galaxy.jobs.runners.slurm:SlurmJobRunner / So - What is going wrong here? Am I working on the wrong job_conf.xml? What does galaxy do here? Thanks for help!Alexander ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] cloudman launch
Hi Enis, thank you. I will try it next time I recreate the cluster. I have some more things: A) I wanted to tell you that I did the update of the galaxyFS with the playbook, by using the role cm.filesystem only. It worked - thanks! B) How could I mount another volume to the upstarting cluster? - name: stuff snap_id: snap-xyz Would that work? E.g when I want this snapshot be mounted as /mnt/stuff C) How can I use the dev branch of galaxy with cloudman? There are some things only in dev I will need. How would that be possible? D) There is an email from me to the list from two days ago. Could you look at that? It's not critical for me, but might lead to a bug. I think that's it so far. If I have more, I'll come back to you ;) 2015-07-02 11:32 GMT-05:00 Enis Afgan enis.af...@irb.hr: Hi Alexander, I traced the code and tried it out and you can override what is defined in the flavor by providing the details in the 'Extra user data' field on the launch form. For example: filesystem_templates: - name: galaxy roles: galaxyTools,galaxyData archive_url: http://s3.amazonaws.com/cloudman/fs-archives/galaxyFS-dev-latest.tar.gz type: archive size: 10 - name: galaxyIndices roles: galaxyIndices snap_id: snap-4b20f451 If you want to use your custom bucket (i.e., if you have modified CloudMan code in some way), you can then specify it in the 'default bucket' form field and place cm.tar.gz and cm_boot.py in that bucket but no need to provide snaps.yaml. If you do not want to use a custom bucket, you can use the current 'Standard' flavor, which defaults to cloudman-test bucket with CloudMan from June 19th (I will not be updating anything about it until after GCC). Hope this helps and let me know if you have more questions, Enis On Thu, Jul 2, 2015 at 9:29 AM, Enis Afgan enis.af...@irb.hr wrote: Hi Alexander, Let me investigate this a bit more and get back to you because I was under the impression I had fixed this. In the mean time, I updated your flavor to specify snap-6f0b211d. On Wed, Jul 1, 2015 at 4:44 PM, Alexander Vowinkel vowinkel.alexan...@gmail.com wrote: Hi Enis, I am trying to launch my cloudman instance with this launcher again: https://launch.usegalaxy.org/launch You created the flavor for me because it wouldn't load my galaxyIndices from the snaps.yaml in my default bucket. I thought, this problem was fixed, but it still creates a galaxyIndices from the default snapshot (600GB) when using the Standard flavor. When I use my flavor, it loads the snapshot, but it is a little outdated ;) So my question: Is this behaviour expected? If no, are you going to fix it soon? If not, could you update my flavor in the way that galaxyIndices gets created from snap-6f0b211d ? Further: Should I update CM? I am still on the version from Jun 22. I am still not sure what I need to change (my snaps.yaml) in order to have my clean cluster creation still functioning. Thanks, Alexander ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] cloudman persist filesystem error
No, I don't have more anymore. If this happens again, I will tell you. 2015-07-02 13:46 GMT-05:00 Enis Afgan enis.af...@irb.hr: Hi Alexander, I tested the process of persisting file system changes but it all worked as expected. Looking at the log snippet you sent, the only thing that seems off is the very first line: Completed creation of a snapshot for the volume 'None', snap id: 'snap-6f0b211d' - volume should not be None there. Actually, it's incredible that it would be None because a volume must exist in order to create a snapshot... Do you have more content from that log? In the mean time, I added more safeguards if this happens again: https://github.com/galaxyproject/cloudman/commit/337ecb95a04788e540d60be37b4f4f817257b24f On Tue, Jun 30, 2015 at 7:32 PM, Alexander Vowinkel vowinkel.alexan...@gmail.com wrote: Hi, from IRC this question/bug report: (18:06:13) avowinkel: some cloudman guy here? enis? (18:06:30) avowinkel: AttributeError: 'NoneType' object has no attribute 'id' (18:06:50) avowinkel: when persisting galaxyIndices (18:07:43) avowinkel: snapshot is created, but it crashes when ... somewhen well after that (18:11:08) avowinkel: http://pastebin.com/SDp1p96K (18:11:10) mrscribe: Title: paster.log - Pastebin.com (at pastebin.com) Best, Alexander ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] cloudman launch
Hi Enis, I am trying to launch my cloudman instance with this launcher again: https://launch.usegalaxy.org/launch You created the flavor for me because it wouldn't load my galaxyIndices from the snaps.yaml in my default bucket. I thought, this problem was fixed, but it still creates a galaxyIndices from the default snapshot (600GB) when using the Standard flavor. When I use my flavor, it loads the snapshot, but it is a little outdated ;) So my question: Is this behaviour expected? If no, are you going to fix it soon? If not, could you update my flavor in the way that galaxyIndices gets created from snap-6f0b211d ? Further: Should I update CM? I am still on the version from Jun 22. I am still not sure what I need to change (my snaps.yaml) in order to have my clean cluster creation still functioning. Thanks, Alexander ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Job still waiting - Command Line None - SGE
At what galaxy version are you? 2015-06-30 7:36 GMT-05:00 Gildas Le Corguille lecorgui...@sb-roscoff.fr: Hi, We have a problem on our production instances since one or two years (never found the time to write it and we still hope to find by ourself). Some jobs stay in waiting status (grey) indefinitely. Job ID User Last Update Tool State Inputs Command Line Job Runner PID/Cluster ID 63094 f...@bar.fr 10 hours ago toolshed.g2.bx.psu.edu/repos/devteam/fastq_groomer/fastq_groomer/1.0.4 new 123250 setting_metadata None None None 63093 f...@bar.fr 10 hours ago toolshed.g2.bx.psu.edu/repos/devteam/fastq_groomer/fastq_groomer/1.0.4 new 123249 setting_metadata None None None 63088 f...@bar.fr 14 hours ago testtoolshed.g2.bx.psu.edu/repos/jjohnson/trinityrnaseq/trinityrnaseq_norm/0.0.2 new 123250 setting_metadata, 123249 setting_metadata None None None Our scheduler is SGE. We tried to found the origin of this issue : - there is not relevant clues in the log files - sometimes, a restart make start those jobs - sometimes, kill a job make the others start - sometimes, there is no choice than resubmit Is anyone can propose an idea to fix this behaviour ? Thanks Gildas ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] output filter bugs
2015-06-30 11:04 GMT-05:00 John Chilton jmchil...@gmail.com: On Mon, Jun 29, 2015 at 2:48 PM, Alexander Vowinkel vowinkel.alexan...@gmail.com wrote: I have a tool with filtered outputs (data [..]filter[...]/data). A) As I understood the filters are pure python. But if I put it in ![CDATA[, it doesn't work. I think this should work even better ;) This sounds wrong - the standard library is handling the CDATA parsing for us and so Galaxy should be looking at an identical representation of the data. Are you sure you are using the CDATA tags correctly? I modified the test tool demonstrating output filters with the following patch (https://gist.github.com/jmchilton/1c8d13c01815a9b0fc8e) and ran the tool tests using the command: ./run_tests.sh -framework -id output_filter and everything seemed fine. Yeah. Weird. Now I don't have problems anymore too. Must have been something else. B) The workflow generator doesn't interpret the filters. When I put the tool in a workflow, all possible outputs are shown. This doesn't change when I change an option that would change the output choice. Can someone fix this? ;) This would be a nice enhancement - but it would take a lot of effort to implement. I would suspect we are going to be stuck with the display of invalid outputs for a while yet. I've created a Trello card for this issue: https://trello.com/c/Hcm2MlQg What a pitty :( ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] cloudman persist filesystem error
Hi, from IRC this question/bug report: (18:06:13) avowinkel: some cloudman guy here? enis? (18:06:30) avowinkel: AttributeError: 'NoneType' object has no attribute 'id' (18:06:50) avowinkel: when persisting galaxyIndices (18:07:43) avowinkel: snapshot is created, but it crashes when ... somewhen well after that (18:11:08) avowinkel: http://pastebin.com/SDp1p96K (18:11:10) mrscribe: Title: paster.log - Pastebin.com (at pastebin.com) Best, Alexander ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] planemo shed_update - error
Thanks, now I get the following directly after the diff output: Could not update stacks object of type 'NoneType' has no len() /usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html InsecureRequestWarning) /usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html InsecureRequestWarning) Repository metadata updated. Failed to update a repository. What does that mean? Best, Alexander 2015-06-29 10:41 GMT-05:00 John Chilton jmchil...@gmail.com: Thanks for the bug report. I have fixed this problem here ( https://github.com/galaxyproject/planemo/commit/a32d0820a6b856d8d15c9619c97549230a17 ) and released a new version of planemo (0.13.0) with fix. Hopefully you will get a better error message after the upgrade if there is still a configuration issue. -John On Sat, Jun 27, 2015 at 6:32 AM, Björn Grüning bjoern.gruen...@gmail.com wrote: Hi Alexander, I have opend an issue: https://github.com/galaxyproject/planemo/issues/240 Thanks for reporting, Bjoern Am 27.06.2015 um 07:11 schrieb Alexander Vowinkel: Hi, I'm running: planemo shed_update --check_diff --shed_target toolshed and get a python error message (see below). shed_diff and shed_lint are working fine. planemo, version 0.12.2 Best, Alexander Traceback (most recent call last): File /usr/local/bin/planemo, line 11, in module sys.exit(planemo()) File /usr/local/lib/python2.7/dist-packages/click/core.py, line 664, in __call__ return self.main(*args, **kwargs) File /usr/local/lib/python2.7/dist-packages/click/core.py, line 644, in main rv = self.invoke(ctx) File /usr/local/lib/python2.7/dist-packages/click/core.py, line 991, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File /usr/local/lib/python2.7/dist-packages/click/core.py, line 837, in invoke return ctx.invoke(self.callback, **ctx.params) File /usr/local/lib/python2.7/dist-packages/click/core.py, line 464, in invoke return callback(*args, **kwargs) File /usr/local/lib/python2.7/dist-packages/click/decorators.py, line 64, in new_func return ctx.invoke(f, obj, *args[1:], **kwargs) File /usr/local/lib/python2.7/dist-packages/click/core.py, line 464, in invoke return callback(*args, **kwargs) File /usr/local/lib/python2.7/dist-packages/planemo/commands/cmd_shed_update.py, line 52, in cli tsi = shed.tool_shed_client(ctx, **kwds) File /usr/local/lib/python2.7/dist-packages/planemo/shed/__init__.py, line 331, in tool_shed_client key = prop(key) File /usr/local/lib/python2.7/dist-packages/planemo/shed/__init__.py, line 323, in prop return kwds.get(shed_%s % key, None) or shed_config.get(key, None) AttributeError: 'NoneType' object has no attribute 'get' ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] planemo shed_update - error
Ay shit. Just realized during looking at it the third time: Someone forgot to uncomment the key line o_o' 2015-06-29 16:21 GMT-05:00 Alexander Vowinkel vowinkel.alexan...@gmail.com : Here they are: ~/.planemo.yml: http://pastebin.com/nWdT6ybx Command-Log: http://pastebin.com/baBmrD8d 2015-06-29 15:23 GMT-05:00 John Chilton jmchil...@gmail.com: Do you have a ~/.planemo.yml file? I think probably yes based on the previous exception. Can you replace passwords and keys with and send me a copy of that as well as the command-line that produced the above output? -John On Mon, Jun 29, 2015 at 12:05 PM, Alexander Vowinkel vowinkel.alexan...@gmail.com wrote: Thanks, now I get the following directly after the diff output: Could not update stacks object of type 'NoneType' has no len() /usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html InsecureRequestWarning) /usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html InsecureRequestWarning) Repository metadata updated. Failed to update a repository. What does that mean? Best, Alexander 2015-06-29 10:41 GMT-05:00 John Chilton jmchil...@gmail.com: Thanks for the bug report. I have fixed this problem here ( https://github.com/galaxyproject/planemo/commit/a32d0820a6b856d8d15c9619c97549230a17 ) and released a new version of planemo (0.13.0) with fix. Hopefully you will get a better error message after the upgrade if there is still a configuration issue. -John On Sat, Jun 27, 2015 at 6:32 AM, Björn Grüning bjoern.gruen...@gmail.com wrote: Hi Alexander, I have opend an issue: https://github.com/galaxyproject/planemo/issues/240 Thanks for reporting, Bjoern Am 27.06.2015 um 07:11 schrieb Alexander Vowinkel: Hi, I'm running: planemo shed_update --check_diff --shed_target toolshed and get a python error message (see below). shed_diff and shed_lint are working fine. planemo, version 0.12.2 Best, Alexander Traceback (most recent call last): File /usr/local/bin/planemo, line 11, in module sys.exit(planemo()) File /usr/local/lib/python2.7/dist-packages/click/core.py, line 664, in __call__ return self.main(*args, **kwargs) File /usr/local/lib/python2.7/dist-packages/click/core.py, line 644, in main rv = self.invoke(ctx) File /usr/local/lib/python2.7/dist-packages/click/core.py, line 991, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File /usr/local/lib/python2.7/dist-packages/click/core.py, line 837, in invoke return ctx.invoke(self.callback, **ctx.params) File /usr/local/lib/python2.7/dist-packages/click/core.py, line 464, in invoke return callback(*args, **kwargs) File /usr/local/lib/python2.7/dist-packages/click/decorators.py, line 64, in new_func return ctx.invoke(f, obj, *args[1:], **kwargs) File /usr/local/lib/python2.7/dist-packages/click/core.py, line 464, in invoke return callback(*args, **kwargs) File /usr/local/lib/python2.7/dist-packages/planemo/commands/cmd_shed_update.py, line 52, in cli tsi = shed.tool_shed_client(ctx, **kwds) File /usr/local/lib/python2.7/dist-packages/planemo/shed/__init__.py, line 331, in tool_shed_client key = prop(key) File /usr/local/lib/python2.7/dist-packages/planemo/shed/__init__.py, line 323, in prop return kwds.get(shed_%s % key, None) or shed_config.get(key, None) AttributeError: 'NoneType' object has no attribute 'get' ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply
Re: [galaxy-dev] planemo shed_update - error
Here they are: ~/.planemo.yml: http://pastebin.com/nWdT6ybx Command-Log: http://pastebin.com/baBmrD8d 2015-06-29 15:23 GMT-05:00 John Chilton jmchil...@gmail.com: Do you have a ~/.planemo.yml file? I think probably yes based on the previous exception. Can you replace passwords and keys with and send me a copy of that as well as the command-line that produced the above output? -John On Mon, Jun 29, 2015 at 12:05 PM, Alexander Vowinkel vowinkel.alexan...@gmail.com wrote: Thanks, now I get the following directly after the diff output: Could not update stacks object of type 'NoneType' has no len() /usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html InsecureRequestWarning) /usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html InsecureRequestWarning) Repository metadata updated. Failed to update a repository. What does that mean? Best, Alexander 2015-06-29 10:41 GMT-05:00 John Chilton jmchil...@gmail.com: Thanks for the bug report. I have fixed this problem here ( https://github.com/galaxyproject/planemo/commit/a32d0820a6b856d8d15c9619c97549230a17 ) and released a new version of planemo (0.13.0) with fix. Hopefully you will get a better error message after the upgrade if there is still a configuration issue. -John On Sat, Jun 27, 2015 at 6:32 AM, Björn Grüning bjoern.gruen...@gmail.com wrote: Hi Alexander, I have opend an issue: https://github.com/galaxyproject/planemo/issues/240 Thanks for reporting, Bjoern Am 27.06.2015 um 07:11 schrieb Alexander Vowinkel: Hi, I'm running: planemo shed_update --check_diff --shed_target toolshed and get a python error message (see below). shed_diff and shed_lint are working fine. planemo, version 0.12.2 Best, Alexander Traceback (most recent call last): File /usr/local/bin/planemo, line 11, in module sys.exit(planemo()) File /usr/local/lib/python2.7/dist-packages/click/core.py, line 664, in __call__ return self.main(*args, **kwargs) File /usr/local/lib/python2.7/dist-packages/click/core.py, line 644, in main rv = self.invoke(ctx) File /usr/local/lib/python2.7/dist-packages/click/core.py, line 991, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File /usr/local/lib/python2.7/dist-packages/click/core.py, line 837, in invoke return ctx.invoke(self.callback, **ctx.params) File /usr/local/lib/python2.7/dist-packages/click/core.py, line 464, in invoke return callback(*args, **kwargs) File /usr/local/lib/python2.7/dist-packages/click/decorators.py, line 64, in new_func return ctx.invoke(f, obj, *args[1:], **kwargs) File /usr/local/lib/python2.7/dist-packages/click/core.py, line 464, in invoke return callback(*args, **kwargs) File /usr/local/lib/python2.7/dist-packages/planemo/commands/cmd_shed_update.py, line 52, in cli tsi = shed.tool_shed_client(ctx, **kwds) File /usr/local/lib/python2.7/dist-packages/planemo/shed/__init__.py, line 331, in tool_shed_client key = prop(key) File /usr/local/lib/python2.7/dist-packages/planemo/shed/__init__.py, line 323, in prop return kwds.get(shed_%s % key, None) or shed_config.get(key, None) AttributeError: 'NoneType' object has no attribute 'get' ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use
[galaxy-dev] Set build when discovering output in collection
Hi, as many of you probably know by now, I have a tool that creates an output collection by discovering the datasets in this collection like propsed there: [1] Specifically I have a fastq input with set build information (a ref genome). The output is a collection of fastq files. After I managed to give them the same format like the input (hacky version), I now want them to adopt the build ID. How can I do that? Thanks, Alexander [1] https://bitbucket.org/galaxy/galaxy-central/pull-request/634/allow-tools-to-explicitly-produce-dataset/diff ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Set build when discovering output in collection
FOR GOD'S SAKE SEARCH THE ARCHIVE !!! http://dev.list.galaxyproject.org/Database-Build-td4140140.html https://github.com/galaxyproject/tools-devteam/blob/master/tools/tophat2/tophat2_wrapper.xml#L313 2015-06-26 21:58 GMT-05:00 Alexander Vowinkel vowinkel.alexan...@gmail.com : Hi, as many of you probably know by now, I have a tool that creates an output collection by discovering the datasets in this collection like propsed there: [1] Specifically I have a fastq input with set build information (a ref genome). The output is a collection of fastq files. After I managed to give them the same format like the input (hacky version), I now want them to adopt the build ID. How can I do that? Thanks, Alexander [1] https://bitbucket.org/galaxy/galaxy-central/pull-request/634/allow-tools-to-explicitly-produce-dataset/diff ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Set build when discovering output in collection
And the corresponding pull request: https://github.com/galaxyproject/tools-devteam/pull/173 The BWA github-repo is newer than what is on the toolshed, right? When do you update the toolshed? Best, Alexander 2015-06-26 22:17 GMT-05:00 Alexander Vowinkel vowinkel.alexan...@gmail.com : FOR GOD'S SAKE SEARCH THE ARCHIVE !!! http://dev.list.galaxyproject.org/Database-Build-td4140140.html https://github.com/galaxyproject/tools-devteam/blob/master/tools/tophat2/tophat2_wrapper.xml#L313 2015-06-26 21:58 GMT-05:00 Alexander Vowinkel vowinkel.alexan...@gmail.com: Hi, as many of you probably know by now, I have a tool that creates an output collection by discovering the datasets in this collection like propsed there: [1] Specifically I have a fastq input with set build information (a ref genome). The output is a collection of fastq files. After I managed to give them the same format like the input (hacky version), I now want them to adopt the build ID. How can I do that? Thanks, Alexander [1] https://bitbucket.org/galaxy/galaxy-central/pull-request/634/allow-tools-to-explicitly-produce-dataset/diff ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] planemo shed_update - error
Hi, I'm running: planemo shed_update --check_diff --shed_target toolshed and get a python error message (see below). shed_diff and shed_lint are working fine. planemo, version 0.12.2 Best, Alexander Traceback (most recent call last): File /usr/local/bin/planemo, line 11, in module sys.exit(planemo()) File /usr/local/lib/python2.7/dist-packages/click/core.py, line 664, in __call__ return self.main(*args, **kwargs) File /usr/local/lib/python2.7/dist-packages/click/core.py, line 644, in main rv = self.invoke(ctx) File /usr/local/lib/python2.7/dist-packages/click/core.py, line 991, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File /usr/local/lib/python2.7/dist-packages/click/core.py, line 837, in invoke return ctx.invoke(self.callback, **ctx.params) File /usr/local/lib/python2.7/dist-packages/click/core.py, line 464, in invoke return callback(*args, **kwargs) File /usr/local/lib/python2.7/dist-packages/click/decorators.py, line 64, in new_func return ctx.invoke(f, obj, *args[1:], **kwargs) File /usr/local/lib/python2.7/dist-packages/click/core.py, line 464, in invoke return callback(*args, **kwargs) File /usr/local/lib/python2.7/dist-packages/planemo/commands/cmd_shed_update.py, line 52, in cli tsi = shed.tool_shed_client(ctx, **kwds) File /usr/local/lib/python2.7/dist-packages/planemo/shed/__init__.py, line 331, in tool_shed_client key = prop(key) File /usr/local/lib/python2.7/dist-packages/planemo/shed/__init__.py, line 323, in prop return kwds.get(shed_%s % key, None) or shed_config.get(key, None) AttributeError: 'NoneType' object has no attribute 'get' ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] problem with view of list collection in history view
Hi Team, my tool creates dynamically 96 datasets bundled into a list. In the history I can see the number 96 in the top as hidden datasets (6 shown, 96 hidden) When I open the list, I just can see 64 items. Now I run the job again and I have 96 more hidden items. I open the new list and can see 66 items in that new list. What is going on here? Is that just a visual bug? Or are my datasets affected? Thanks, Alexander PS: I use postgres ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] list collection output - format set from input
no - not a typo. The tool can process both.
It's just my naming, because I use it for fastq.
I'll change the help tag.
2015-06-25 3:44 GMT-05:00 Peter Cock p.j.a.c...@googlemail.com:
Hi Alexander,
If this wasn't a collection, I would expect format_source to work
(possibly also using metadata_source=fastq_input1), so perhaps
this is a bug - John?
Peter
P.S. Your help caption and output label both say FASTQ, but the
input also allows FASTA input. Typo?
On Thu, Jun 25, 2015 at 2:39 AM, Alexander Vowinkel
vowinkel.alexan...@gmail.com wrote:
Hi,
I have an input, that can be fasta,fastqsanger,fastqillumina:
param name=fastq_input1 type=data
format=fasta,fastqsanger,fastqillumina label=Select the fastq file
help=Specify fastq file with reads/
I have multiple outputfiles - bundled in a list collection:
collection name=split_output type=list label=@OUTPUT_NAME_PREFIX@
on
${on_string} (Fastq Collection) format_source=fastq_input1
discover_datasets pattern=__name_and_ext__ directory=splits /
/collection
The format_source parameter doesn't work - the files in the list
(extension
fq) are of format fq
How can I make it possible that they are fasta,fastqsanger,fastqillumina
depending on fastq_input1?
Thanks,
Alexander
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] list collection output - format set from input
Hi John,
yes - I created this hacky solution and it works.
I now tried what you said, but no success.
Code:
collection name=split_output type=list label=@OUTPUT_NAME_PREFIX@
on ${on_string} (Fastq Collection) format_source=fastq_input1
discover_datasets pattern=sample_(?Plt;namegt;.+)\.fq
directory=output /
/collection
It puts the samples into the collection correctly, but doesn't set a data
type.
Weird enough: In the collection, the files even don't show any format
(in the small history view on the right). Is that normal?
Addon: How can I let the files in the collection have the same database
build
attached? The other file (log file) gets it.
Best,
Alexander
2015-06-25 9:12 GMT-05:00 John Chilton jmchil...@gmail.com:
You are giving Galaxy mixed signals :). format_source will say to use
the data type specified by the corresponding input - but
discover_datasets pattern=__name_and_ext__ directory=splits /
Is saying (with the pattern __name_and_ext__) read files of the form
out1.fastq and assign the collection identifier to out1 and the
extension/format to fastq.
Two things should work - one of them will and the other might but should.
One thing you can do is not override the extension in your pattern:
discover_datasets pattern=__name__ directory=splits /
or if you don't want to include the .fq in the output identifier (and
you probably don't want to)
discover_datasets pattern=(?Plt;namegt;.+)\.fq /
If that doesn't work - it is a bug and please let me know and I will
attempt to fix it.
The hackier way to get this to work that I am more confident will work
- is to drop the format_source - use the same pattern:
discover_datasets pattern=__name_and_ext__ directory=splits /
But just use a shell command or something to rename all files of the
form *.fq to *.${fastq_input1.ext}.
Hope this helps.
-John
On Thu, Jun 25, 2015 at 9:56 AM, Alexander Vowinkel
vowinkel.alexan...@gmail.com wrote:
no - not a typo. The tool can process both.
It's just my naming, because I use it for fastq.
I'll change the help tag.
2015-06-25 3:44 GMT-05:00 Peter Cock p.j.a.c...@googlemail.com:
Hi Alexander,
If this wasn't a collection, I would expect format_source to work
(possibly also using metadata_source=fastq_input1), so perhaps
this is a bug - John?
Peter
P.S. Your help caption and output label both say FASTQ, but the
input also allows FASTA input. Typo?
On Thu, Jun 25, 2015 at 2:39 AM, Alexander Vowinkel
vowinkel.alexan...@gmail.com wrote:
Hi,
I have an input, that can be fasta,fastqsanger,fastqillumina:
param name=fastq_input1 type=data
format=fasta,fastqsanger,fastqillumina label=Select the fastq file
help=Specify fastq file with reads/
I have multiple outputfiles - bundled in a list collection:
collection name=split_output type=list
label=@OUTPUT_NAME_PREFIX@
on
${on_string} (Fastq Collection) format_source=fastq_input1
discover_datasets pattern=__name_and_ext__ directory=splits
/
/collection
The format_source parameter doesn't work - the files in the list
(extension
fq) are of format fq
How can I make it possible that they are
fasta,fastqsanger,fastqillumina
depending on fastq_input1?
Thanks,
Alexander
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] BAM datatype - bam bai
Hi, thanks! Let me clarify myself on the multiple generation question. When I do command line, I do this: Bam,Bai - Prog1 - Bam,Bai(v2) - Prog2 - Bam,Bai(v3) What I understand, what galaxy is doing (always overwrite [1]): Bam,Bai - Prog1 - Bam,Bai(v2) - Bam(v2) - Bam,Bai(v2) - Prog2 - Bam,Bai(v3) - Bam(v3) - Bam,Bai(v3) Best, Alexander [1] https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/datatypes/binary.py#L314 2015-06-19 1:26 GMT-05:00 Björn Grüning bjoern.gruen...@gmail.com: Hi Alexander, Am 19.06.2015 um 01:44 schrieb Alexander Vowinkel: Hi, when I 'produce' a bam file in my output, I can download the bam file and a corresponding bai file. I am now wondering where does the bai file come from? From Galaxy! :) Galaxy has a concept of datatypes and metadata. In the case of BAM files, Galaxy is producing the index file to every position sorted BAM file. When is it being generated, what code is used for that? As soon as the BAM file is created. https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/datatypes/binary.py#L312 When I do multiple actions with my bam file - is the index generated every time? (overhead) No why should it? The BAM files stays the same right? So if you are using the BAM multiple time the one index is used multiple times. Where is the bai saved on disk (if it is)? You can access it with this snipped: $your_bam_file.metadata.bam_index I found that in the IUC gatk, a new bai is always generated when a gatk tool is started. This creates quite an overhead. Isn't there a better solution? Do you have an idea? Without Galaxy you also create an index once and reuse it, this is was Galaxy is doing. This concept is also used for other datatypes and improves the usability dramatically. If this was helpful, please consider to ask this question on Biostar again. I think this might be useful for others as well. Cheers, Bjoern @ref https://github.com/galaxyproject/tools-iuc/issues/194 Best, Alexander ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] BAM datatype - bam bai
The response speed is awesome! On github Eric answered: https://github.com/galaxyproject/tools-iuc/blob/0887009a23d176b21536c9fd8a18c4fecc417d4f/tools/bedtools/multiCov.xml see L12 of that file for where the bai file is stored. That explains a lot. Very precise example. Thank you! What I'm still wondering is: When is it being generated, what code is used for that? More specific: When I use Picard SortSam in galaxy, the output is a bam file with an accompanying bai file. But when I use Picard on command line, the output is just a bam file. In the Picard SortSam galaxy tool, it is not being done. (If I didn't miss something) What leads me to When I do multiple actions with my bam file - is the index generated every time? (overhead) GATK processes BAM files and at the same time the bai files. It would be a waste to just use the bam files and re-generate the bai files, that already exist. How could that be done? Thanks, Alexander 2015-06-18 18:44 GMT-05:00 Alexander Vowinkel vowinkel.alexan...@gmail.com : Hi, when I 'produce' a bam file in my output, I can download the bam file and a corresponding bai file. I am now wondering where does the bai file come from? When is it being generated, what code is used for that? When I do multiple actions with my bam file - is the index generated every time? (overhead) Where is the bai saved on disk (if it is)? I found that in the IUC gatk, a new bai is always generated when a gatk tool is started. This creates quite an overhead. Isn't there a better solution? @ref https://github.com/galaxyproject/tools-iuc/issues/194 Best, Alexander ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Repository installation error - missing
I fixed the error. It have been the directory permissions. Found in galaxy log: tool_shed.util.hg_util DEBUG 2015-06-16 16:17:04,045 Error cloning repository: [Errno 13] Permission denied: '/mnt/galaxy/galaxy-app/gbs_tools/testtoolshed.g2.bx.psu.edu' But the question with the error message is still active, I guess. 2015-06-15 15:53 GMT-05:00 Alexander Vowinkel vowinkel.alexan...@gmail.com : Hi, I have a custom tool_conf.xml in gbs_tools/gbs_tool_conf.xml It is added to the config and loaded without problems. toolbox tool_path=./gbs_tools But when I want to install tools from a toolshed (test and main) to it (Shed tool configuration file), it reports Error during installation. Clicking on the tool, it says This repository is not installed correctly (see the Repository installation error below). [...] But there is no Repository installation error. So, how can I find out, what the problem is? Installing in the default tool_shed_conf.xml works. I also checked if it's a problem with the tool_path (I assume it points to the galaxy root). So I created a directory gbs_tools in gbs_tools, in case it is relative to the xml file directory. No difference. Thanks for help! Alexander ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] cloudman error somewhere
Hi, I have the following problem: First, the job scheduling didn't work. I tried to upload a file, but the job was waiting. Then I checked squeue, which timed out. The slurm service was running. Then I restarted the cluster. Now I even have problems access the cloudman admin page. Meaning, the page is loading endless. The machine is not busy (CPU and mem @ htop) In the paster.log it tries this every 4 seconds: 2015-06-16 18:35:56,547 DEBUG master:2579 Trying to setup AMQP connection; conn = 'cm.util.comm.CMMasterComm object at 0x7f0aa26d72d0' 2015-06-16 18:35:56,547 DEBUG comm:30 Setting up a new AMQP connection 2015-06-16 18:35:56,548 DEBUG comm:49 AMQP Connection Failure: [Errno 111] Connection refused Any idea what this might be? Something seems to be really weird. I am using the next version of CloudMan (dev). I created an own instance with the Cloud Man playbook. Best, Alexander ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Data Collections
Thank you for this detailed descriptions! I already have a followup question. I'm working on Galaxy Cloudman: Galaxy is at revision: 93cda3eb81 (master branch) from 11 Jun 2015) But I just can find Build dataset pair|list, not List of Dataset Pairs like in the video. At what version is that implemented? Best, Alexander 2015-06-15 10:17 GMT-05:00 John Chilton jmchil...@gmail.com: On Wed, Jun 10, 2015 at 4:04 PM, Alexander Vowinkel vowinkel.alexan...@gmail.com wrote: Hi Folks, thank you so far for the previous help. I got much further. Now I'm stuck with data collections. Because this is quite a list, I appreciate also answers to parts of my questions ;) I have two issues: A) manual definition of data collections (any type) by user and/or admin B) definition of data collections as input/output of a tool and inside a workflow A) manual Basically I would like to create i) a list of fastq files (unpaired) ii) a paired set of two fastq files iii) a list of each two paired fastq files How can I do that? By using the web app? As user? As admin? By working via ssh on the server? So each of these got much easier/more robust with the most recent release. For the user perspective - for any of these options you will want to load the fastq files into a history, open the manage multiple datasets option ( https://wiki.galaxyproject.org/Histories#Managing_Multiple_Datasets_Easily ), select the datasets, and then choose the list type from the menu. Each will cause a widget to pop up allowing you to group the datasets (into a list, a pair, or a list of pairs depending on your selection). The most complicated option is the list of pairs - this option is demonstrated in a the first video in Anton's recent NGS 101 - Reference-based RNA-seq series (https://vimeo.com/channels/884356/128265983). More information at https://wiki.galaxyproject.org/Learn/GalaxyNGS101. For all user-centric scenarios - you will need to get the plain datasets into a history first. FTP upload for instance doesn't support creating collections directly - you can import datasets and then create them. Likewise - data libraries do not currently support dataset collections. I believe there are Trello cards for both of these issues. For admins - there is a dataset collection API - I can point you at examples if you want - but this doesn't seem to be your interest. B) in tool/workflow Here I also have different approaches I would like to realize: i) use a collection as input for a tool ii) create a collection as output of a tool ii.1) from known # of output parameters ii.2) from unknown # of output parameters For these things I was trying to find some tools in toolshed to see how they do it, but I couldn't quite adopt it. I would look in the following directory instead of the tool shed - https://github.com/galaxyproject/galaxy/tree/dev/test/functional/tools. These are the tools used to drive the testing of the collections implementation and contain some very stripped down examples of what is possible. i) use a collection as input for a tool this is good documented - realizable by type=data_collection and the collection_type. Unfortunately I can't test this because I can't create a collection so far ;) - see A Indeed :). Here some good examples are like the tools in the RNA-seq pipeline - Tophat, Bowtie2, etc ii) create a collection as output of a tool Here it gets blurry for me. So one can get very far without ever creating an output from a tool explicitly. I contend most of the time - if you have a list of bam files and you want to create another list of bam files - you just want to map some operation over them. This is demonstrated in that RNA-seq outline - and talked about in a more theoretical way in my GCC talk from last year http://bit.ly/gcc2014workflows. There are definitely cases when you want to explicitly create collections though - the current best documentation on this is going to be the pull request that added them - not the implementation but the description which actually lays out these same categories and how to handle them with explicit complete examples. https://bitbucket.org/galaxy/galaxy-central/pull-request/634/allow-tools-to-explicitly-produce-dataset Hopefully this helps - please follow up with additional questions as you have them. I am keen to see more developers leveraging dataset collections. Thanks a bunch. -John ii.1) from known # of output parameters Here I didn't find a tool. I just thought, it might be a simpler case than ii.2 and good to understand the concept. I would be glad if someone could explain the way(s) to do this. ii.2) from unknown # of output parameters For this I found barcode splitter tools (also from devteam) that have different approaches. But. Their output (defined in xml) is only some report file. The output files seem
[galaxy-dev] Repository installation error - missing
Hi, I have a custom tool_conf.xml in gbs_tools/gbs_tool_conf.xml It is added to the config and loaded without problems. toolbox tool_path=./gbs_tools But when I want to install tools from a toolshed (test and main) to it (Shed tool configuration file), it reports Error during installation. Clicking on the tool, it says This repository is not installed correctly (see the Repository installation error below). [...] But there is no Repository installation error. So, how can I find out, what the problem is? Installing in the default tool_shed_conf.xml works. I also checked if it's a problem with the tool_path (I assume it points to the galaxy root). So I created a directory gbs_tools in gbs_tools, in case it is relative to the xml file directory. No difference. Thanks for help! Alexander ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] config files in cloudman
Hi, the config/galaxy.ini is being overwritten when restarting Galaxy via the CloudMan Console. Unfortunately I can't find the file, that is used as template. I searched on the whole machine for *.cloud, galaxy.ini* and universe_wsgi.ini*. The only file I found was /mnt/cm/installed_files/universe_wsgi.ini.cloud. Changes to this file doesn't have an effect. The corresponding bucket also doesn't contain this file. Where is this blueprint coming from? And where can I change it? I am using the next version of CloudMan (dev). I created an own instance with the Cloud Man playbook. Thanks a lot! Alexander ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] job queue problem with cloud man
Hi list-members, I have a Cloud Man instance on amazon. I did minimal adaption (installing some tools, adding an index) I'm currently running only a master, which is allowed to run jobs. I have now 3 jobs saying: This job is waiting to run But in the CloudMan console it says: Worker status: Idle: 0 Available: 0 Requested: 0 Also condor_q gives 0 jobs. Where is the problem? I already restarted the galaxy service via the admin console... Can someone help me to find whats going wrong? Thanks! Alexander PS: I'm running different job types (one is just Add column) PPS: Upload File jobs are working ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] job queue problem with cloud man
I figured out that only jobs that used one of the files could not run. They wanted a tabular file, which I uploaded before as txt and changed then to tabular. It gave the message that some metadata could not be determined. Auto-detection somehow just worked now when I tried it the 3rd time. It would be good to be able to see some information about a reason why a job can not be executed and is waiting. Is that somewhere implemented? Best, Alexander 2015-06-10 16:19 GMT-05:00 Alexander Vowinkel vowinkel.alexan...@gmail.com : Hi list-members, I have a Cloud Man instance on amazon. I did minimal adaption (installing some tools, adding an index) I'm currently running only a master, which is allowed to run jobs. I have now 3 jobs saying: This job is waiting to run But in the CloudMan console it says: Worker status: Idle: 0 Available: 0 Requested: 0 Also condor_q gives 0 jobs. Where is the problem? I already restarted the galaxy service via the admin console... Can someone help me to find whats going wrong? Thanks! Alexander PS: I'm running different job types (one is just Add column) PPS: Upload File jobs are working ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] install packagess without toolshed
Hi Björn, thank you for your fast answer! I already walked through these pages. Thus I understood the basics of a dependency-file. The problem is: I don't know how to load it into my local galaxy installation. The only way I can see is by uploading it to ToolShed. But I would like to avoid that while I'm still learning to create these things ;) Best, Alexander 2015-06-09 17:16 GMT-05:00 Björn Grüning bjoern.gruen...@gmail.com: Hi Alex, here are some informations: https://wiki.galaxyproject.org/ToolDependencyRecipes https://wiki.galaxyproject.org/ToolDependenciesTagSets https://wiki.galaxyproject.org/PackageRecipes https://wiki.galaxyproject.org/SimpleRepositoryDependencies But the best way probably is to learn from examples: https://github.com/galaxyproject/tools-iuc/tree/master/packages Feel free to ask if you have a concrete example, we will help you to get this in! Cheers, Bjoern Am 10.06.2015 um 00:10 schrieb Alexander Vowinkel: Hi, I would like to create a wrapper for a program. For that I would like to test it, but I just found how to create a tool in galaxy. I couldn't find anything about how to create a package as a dependency. Can someone give me some help here? Best, Alexander ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] install packagess without toolshed
Hi, I would like to create a wrapper for a program. For that I would like to test it, but I just found how to create a tool in galaxy. I couldn't find anything about how to create a package as a dependency. Can someone give me some help here? Best, Alexander ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Sending tool output to input parameters of downstream tool(s)
Hi, I'd like to refer to an old post I found in the archive [1] and ask if there has been some progress in this? Best, Alexander [1] http://dev.list.galaxyproject.org/Sending-tool-output-to-input-parameters-of-downstream-tool-s-tt4663398.html Hi all, I was recently wondering how it might be possible for an output from tool A to be used as a parameter when calling tool B (without modifying tool B to take a data configuration file or anything like that). To clarify, I'm thinking about simple parameters like a single integer/float/string like median insert size or average coverage which map to a single input parameter in a downstream tool. Workflow parameters basic $variable support is similar to what I'm thinking about here, see: https://wiki.galaxyproject.org/DevNewsBriefs/2011_01_31 Perhaps one day in addition to tool outputs being files, they can include setting $variables (local to the history), which could then be used within downstream tools? I can see this has downsides with tracking the providence of each variable though... thus: Another idea would be extra filetypes for each parameter type (Unix philosophy - make everything into a file), e.g. integer_parameter, float_parameter, ... which Tool A could define as an output, and write to. Galaxy would then need to offer these history entries to the user when a later tool (e.g. Tool B) asked for a parameter of that type (e.g. an integer, or a float). (This idea could be extended further, with data files using JSON or something to hold a structured set of parameter values which Galaxy could then offer to fill later tool's inputs.) Crazy, or is there some mileage in this idea? Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
