Re: [galaxy-dev] good practice for Galaxy admin and updates ?
Hi Thanks for your answer, so I will test this solution. Julie Le lun. 3 sept. 2018 à 13:53, Hans-Rudolf Hotz a écrit : > > Hi Julie > > We do something similar, I have described it here: > > https://galaxyproject.org/admin/ten-simple-steps-galaxy-as-a-service/#rule-10-keep-the-galaxy-software-and-yourself-up-to-date > > > Regards, Hans-Rudolf > > > On 09/03/2018 01:19 PM, julie dubois wrote: > > Hi Galaxy Community, > > > > I manage a Galaxy local instance on a standalone station since 6 years. > > Regarding the numerous troubleshoots I have encountered (and the very > > long time I spent) during these years with Galaxy updates trying to > > minimize impact for my users, trying in the same time to keep safe > > our customization of this instance. I'm wondering about the best solution > > to manage my instance and to test my own dev and also Galaxy updates > > before pushing them on > > my production server. > > > > So I want to test a new way and I'd appreciate community feedback on this > > idea > > or let me know if other practices could be more appropriate in my situation. > > So my basic idea is: > > > > First install a new Galaxy from scratch, not by cloning the stable branch of > > Galaxy but by forking it. And use the master branch of this fork as > > production server. > > Then to manage an update of the Galaxy stable, my idea is to make a > > test branch on my > > fork, then pull Galaxy update on this test branch and run this test > > instance as > > a dev server in another station to be sure that update will not override our > > customization or any crucial tools/functionality for my users. > > Finally when I'll be sure this update is safe, I'll merge the test branch > > with > > the master branch of my forked Galaxy. > > > > > > Is anyone already acting in this way? Is it a good (the best) way to manage > > local instance of Galaxy? Have you other good practices? > > > > Thanks a lot for your help. > > > > Julie > > ___ > > Please keep all replies on the list by using "reply all" > > in your mail client. To manage your subscriptions to this > > and other Galaxy lists, please use the interface at: > >https://lists.galaxyproject.org/ > > > > To search Galaxy mailing lists use the unified search at: > >http://galaxyproject.org/search/ > > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
[galaxy-dev] good practice for Galaxy admin and updates ?
Hi Galaxy Community, I manage a Galaxy local instance on a standalone station since 6 years. Regarding the numerous troubleshoots I have encountered (and the very long time I spent) during these years with Galaxy updates trying to minimize impact for my users, trying in the same time to keep safe our customization of this instance. I'm wondering about the best solution to manage my instance and to test my own dev and also Galaxy updates before pushing them on my production server. So I want to test a new way and I'd appreciate community feedback on this idea or let me know if other practices could be more appropriate in my situation. So my basic idea is: First install a new Galaxy from scratch, not by cloning the stable branch of Galaxy but by forking it. And use the master branch of this fork as production server. Then to manage an update of the Galaxy stable, my idea is to make a test branch on my fork, then pull Galaxy update on this test branch and run this test instance as a dev server in another station to be sure that update will not override our customization or any crucial tools/functionality for my users. Finally when I'll be sure this update is safe, I'll merge the test branch with the master branch of my forked Galaxy. Is anyone already acting in this way? Is it a good (the best) way to manage local instance of Galaxy? Have you other good practices? Thanks a lot for your help. Julie ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
Re: [galaxy-dev] Toolshed partial installation ? deeptools error
_local dependency_resolvers_config_file = config/dependency_resolvers_conf.xml 2017-10-03 9:53 GMT+02:00 Marius van den Beek <m.vandenb...@gmail.com>: > Also upon re-reading I see that your dependency is being picked up from > /usr/local/bin, > which indicates that this isn't using conda at all. > A small excerpt from the logs when you try to run the tool may be helpful > to understand what is going on there. > > Best, > Marius > > On 3 October 2017 at 09:29, Marius van den Beek <m.vandenb...@gmail.com> > wrote: >> >> Hi Julie, >> >> any chance you are setting PYTHONPATH before starting galaxy's job >> handlers (a common place would the supervisor config, in case you're using >> that)? >> This will interfere with all python modules that are installed via conda. >> >> Best, >> Marius >> >> On 3 October 2017 at 09:13, julie dubois <dubju...@gmail.com> wrote: >>> >>> Dear Greg, >>> I read your discussion with @bgruening and I think it's not exactly >>> the same. Because in my case, The executable IS present in my >>> galaxy-dist/tool_dependency/_conda/envs/__deeptools@2.5.1/bin . >>> The problem is far away in the execution when the tool (computeMatrix >>> in my example) try to load the python module deeptools.computeMatrix. >>> So is there a solution to discover where is the problem ? >>> >>> Thanks >>> >>> 2017-10-02 15:16 GMT+02:00 Von Kuster, Greg <g...@psu.edu>: >>> > This looks like the issue I’ve recently seen as well - I just discussed >>> > this >>> > with @bgruening on gitter - https://gitter.im/galaxy-iuc/iuc. I’m >>> > using a >>> > work-around of creating the __deeptools conda env manually. In my >>> > case, I’m >>> > not installing the whole suite, just the bamCoverage tool from the TS - >>> > >>> > https://toolshed.g2.bx.psu.edu/view/bgruening/deeptools_bam_coverage/5d11599b8a7d. >>> > My work-around is to create the conda env manually: >>> > >>> > $conda create -n __deeptools@2.5.1 deep tools=2.5.1 >>> > >>> > This env is created in the conda environment outside of Galaxy, so I >>> > just >>> > copy it to the conda environment I’ve configured for Galaxy. This >>> > works, >>> > but it seems there is some issue with the installation from the TS, so >>> > perhaps a more ling term fix is needed. >>> > >>> > >>> > On Oct 2, 2017, at 8:59 AM, julie dubois <dubju...@gmail.com> wrote: >>> > >>> > Hi dev team, >>> > >>> > I have a very big problem since I updated to galaxy 17.05 with >>> > deeptools from toolshed. >>> > >>> > Due to a bug in galaxy 17.05 with toolshed installed tools (reported >>> > in this issue https://github.com/galaxyproject/galaxy/issues/4591), I >>> > re-installed deeptools to try to resolve it. >>> > Not resolved really but it's another problem. I temporarly bypassed it. >>> > >>> > Here is my problem : I've installed deeptools from toolshed and when >>> > we try to use "compute matrix" for example, we have this error message >>> > : >>> > >>> > Fatal error: Exit code 1 () >>> > Traceback (most recent call last): >>> > File "/usr/local/bin/computeMatrix", line 4, in >>> >from deeptools.computeMatrix import main >>> > ImportError: No module named deeptools.computeMatrix >>> > >>> > It seems that wrapper are installed but not the tool "deeptools". >>> > >>> > For more information, here it's the way for the installation : >>> > search and install via toolshed >>> > Install "suite_deeptools" (version 2.5.1.1.0 : 3d68b716965a) >>> > Info in Manage tool dependencies : version=2.5.1, resolver=Conda, >>> > green check symbol in the line with a list of all deeptools. >>> > >>> > Galaxy 17.05, commit cfabe37 (21 sept 2017) >>> > >>> > I've tried to uninstall/reinstall dependencies with conda but without >>> > success. >>> > >>> > Have you any idea to install the deeptools please ? >>> > >>> > Thanks. >>> > Julie >>> > ___ >>> > Please keep all replies on the list by using "reply all" >>> > in your mail client. To manage your subscriptions to this >>> > and other Galaxy lists, please use the interface at: >>> > https://lists.galaxyproject.org/ >>> > >>> > To search Galaxy mailing lists use the unified search at: >>> > http://galaxyproject.org/search/ >>> > >>> > >>> ___ >>> Please keep all replies on the list by using "reply all" >>> in your mail client. To manage your subscriptions to this >>> and other Galaxy lists, please use the interface at: >>> https://lists.galaxyproject.org/ >>> >>> To search Galaxy mailing lists use the unified search at: >>> http://galaxyproject.org/search/ >> >> > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
Re: [galaxy-dev] Toolshed partial installation ? deeptools error
Dear Greg, I read your discussion with @bgruening and I think it's not exactly the same. Because in my case, The executable IS present in my galaxy-dist/tool_dependency/_conda/envs/__deeptools@2.5.1/bin . The problem is far away in the execution when the tool (computeMatrix in my example) try to load the python module deeptools.computeMatrix. So is there a solution to discover where is the problem ? Thanks 2017-10-02 15:16 GMT+02:00 Von Kuster, Greg <g...@psu.edu>: > This looks like the issue I’ve recently seen as well - I just discussed this > with @bgruening on gitter - https://gitter.im/galaxy-iuc/iuc. I’m using a > work-around of creating the __deeptools conda env manually. In my case, I’m > not installing the whole suite, just the bamCoverage tool from the TS - > https://toolshed.g2.bx.psu.edu/view/bgruening/deeptools_bam_coverage/5d11599b8a7d. > My work-around is to create the conda env manually: > > $conda create -n __deeptools@2.5.1 deep tools=2.5.1 > > This env is created in the conda environment outside of Galaxy, so I just > copy it to the conda environment I’ve configured for Galaxy. This works, > but it seems there is some issue with the installation from the TS, so > perhaps a more ling term fix is needed. > > > On Oct 2, 2017, at 8:59 AM, julie dubois <dubju...@gmail.com> wrote: > > Hi dev team, > > I have a very big problem since I updated to galaxy 17.05 with > deeptools from toolshed. > > Due to a bug in galaxy 17.05 with toolshed installed tools (reported > in this issue https://github.com/galaxyproject/galaxy/issues/4591), I > re-installed deeptools to try to resolve it. > Not resolved really but it's another problem. I temporarly bypassed it. > > Here is my problem : I've installed deeptools from toolshed and when > we try to use "compute matrix" for example, we have this error message > : > > Fatal error: Exit code 1 () > Traceback (most recent call last): > File "/usr/local/bin/computeMatrix", line 4, in >from deeptools.computeMatrix import main > ImportError: No module named deeptools.computeMatrix > > It seems that wrapper are installed but not the tool "deeptools". > > For more information, here it's the way for the installation : > search and install via toolshed > Install "suite_deeptools" (version 2.5.1.1.0 : 3d68b716965a) > Info in Manage tool dependencies : version=2.5.1, resolver=Conda, > green check symbol in the line with a list of all deeptools. > > Galaxy 17.05, commit cfabe37 (21 sept 2017) > > I've tried to uninstall/reinstall dependencies with conda but without > success. > > Have you any idea to install the deeptools please ? > > Thanks. > Julie > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > https://lists.galaxyproject.org/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/ > > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
[galaxy-dev] CleanUp Datastets ERROR since update from bitbucket to git
Hi all, I've recently updated my local instance of Galaxy from bitbucket to git. It seems to work fine. No error when we run galaxy. But I've been tried to cleanup datasets and when I ran this command : python scripts/cleanup_datasets/cleanup_datasets_Redirection.py universe_wsgi.ini -d 0 -1 -r $@ I have error in python module like this : Traceback (most recent call last): File "scripts/cleanup_datasets/cleanup_datasets_Redirection.py", line 23, in import galaxy.model.mapping File "/data/galaxy-dist/lib/galaxy/model/mapping.py", line 19, in from galaxy.model.custom_types import JSONType, MetadataType, TrimmedString, UUIDType File "/data/galaxy-dist/lib/galaxy/model/custom_types.py", line 13, in from galaxy import app File "/data/galaxy-dist/lib/galaxy/app.py", line 16, in from galaxy.visualization.data_providers.registry import DataProviderRegistry File "/data/galaxy-dist/lib/galaxy/visualization/data_providers/registry.py", line 4, in from galaxy.visualization.data_providers import genome File "/data/galaxy-dist/lib/galaxy/visualization/data_providers/genome.py", line 15, in from bx.bbi.bigbed_file import BigBedFile ImportError: No module named bigbed_file I've already reinstall in galaxy virtual env lot of modules required as bx-python , pyyaml, pysam, webhelpers... But it's infinite and I don't understand why all python modules are not retrieve while the galaxy instance works fine and I've had no problem when I've updated the galaxy version. How can I manage this problem and eventually re-install python, python modules in virtual env ? Thanks a lot. Julie ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] UTF8 problem with database "An error occured when getting updates from the server"
Dear all, I have just changed my galaxy database encoding in utf8 to avoid error of loading histories. It works fine. But when I run a new job after this change, the bad characters re-appear in my output and the error "An error occured when getting updates from the server" re-appear too. When I've changed my galaxy database envoding, I've modified the /etc/mysql/my.cnf file and replace all of the bad charcater in the previous database before re-install it. Any idea of a change I omit ? Thanks. Julie ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Order of element in datasetcollection "List"
Hi all, I've created a tool which takes a collection List as input. This tool iterate on this collection with a "for" loop. I'd like to know the order of my datasets in this collection for the next treatment in my tools. For example, when I take dataset number 1 and number 2 and number 3 to create my collection. When I iterate on it : what is the first element? the number 1 or the number 3. I ask it because I've seen that when a tool works on a dataset collection, the first job in the history (number of dataset in history is the lowest) ran on dataset number3 and not on dataset number 1 as I expected ! Thanks in advance. Julie Chevalier ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Error sorting or uploading Bam File
Hi, Thanks for your answers. I understand that permit to load or create unsorted bam (or sorted by name) can create bug with lot of tools and it's difficult to restrict this manipulation just for few tools. But I don't understand one thing. Why can I download from the toolshed a tool which permit me to sort bam file by name (from devteam) while it's not supported by galaxy ?! Other question. Anyone can help me to write a new entry for an ‘unsorted bam’ (or namesorted) file type in the file which describe the format file uploaded ? Or do you think that I can run this sorting during the processing of the tool which require this type of sorting : so run the tool of interest and internally to the process run the sorting by name before treating the bam file. In this way do you think that Galaxy will not create an index during the process ? Thanks. Julie 2015-07-17 21:51 GMT+02:00 Frederik Coppens fr...@psb.vib-ugent.be: We also encountered this issue. As I understood it: if Galaxy gets a BAM file (either through upload or generated by a tool), it will generate metadata and indexing is part of that. This only works if the BAM file is coordinate sorted, otherwise it fails (although there is nothing wrong with the BAM file). As a lot of tools (and visualisation) require sorted BAM files, this avoids a lot of problems for users (but does create this issue). An ‘unsorted bam’ (or namesorted) file type could be a solution although this would cause the illogic situation where a ‘bam' is hierarchically not above ‘unsorted bam’ as is the case for e.g. the types of fastq. I guess changing what is now ‘bam’ to ‘sorted bam’ will cause problems for a lot of users... Frederik On 17 Jul 2015, at 21:29, Alexander Vowinkel vowinkel.alexan...@gmail.com wrote: Indeed, this would be the right approach. But so far the current strategy is quite imputed also in tools. E.g. BWA from devteam automatically sorts the bam afterwards - wanted or not. 2015-07-17 11:59 GMT-05:00 Ryan G ngsbioinformat...@gmail.com: I would think the correct approach would be to allow any bam file and if a tool requires a coordinate sorted bam and one is not provided it would make sense for the tool to fail with an error. Does galaxy itself require a coordinate sorted bam or this this so the majority of tools can function properly? Sent from my iPhone On Jul 17, 2015, at 11:08 AM, John Chilton jmchil...@gmail.com wrote: I'm not the right person to respond to this but since no one else has I will explain my limited (probably incorrect) understanding of the problem and what needs to be done. I believe Galaxy assumes all bam files are coordinated sorted by default - so Galaxy's bam datatype would be better thought of as a coordinate-sorted bam. Certain people do not want to allow non-coordinate sorted bam files because these would be allowed by Galaxy as valid as inputs to tools that expect sorted bams - and things would break. For the same reason I imagine these people would argue the bai file needs to be generated because certain tools will depend on its existence. I believe the proposed solution is to add an unsorted bam datatype to Galaxy. I don't know if there is any work in progress on this but it seems to be needed by many researchers. Hope this clarifies things somewhat, -John On Mon, Jun 29, 2015 at 11:06 AM, julie dubois dubju...@gmail.com wrote: Hi all, My galaxy is the last update and I work with samtools 1.2. I have a bug with the sorting and the upload of bam file. This bug appears only when I try to upload bam sorted by name or when I try to sort by name. When I work with bam sorted by coordinate, there is no problem. So, when I try to sort a bam file by name, this error appears under the bug icon : Tool execution generated the following error message: [bam_sort_core] merging from 20 files... *** Error in `python': double free or corruption (!prev): 0x02897990 *** Aborted (core dumped) The tool produced the following additional output: [E::hts_idx_push] chromosome blocks not continuous Galaxy attempted to build the BAM index with samtools 1.0+ but failed: [Errno 2] No such file or directory: '/data/galaxy-dist/database/job_working_directory/024/24109/__dataset_48619_metadata_temp_file_MqhjBN.bai' And the same type of error appears when I try to upload bam sorted by name. What I understand is that the bam can be sorted properly (The command line to sort is ok in a terminal with no bug and I can follow this step in the job_working_directory during the run) but Galaxy seems to fail to create a bai. For our analysis pipeline, I think the bai is not crucial, is it possible to avoid this creation of bai during uploading and sorting of bam file by name ? Is there another problem than this creation of bai ? Thanks. Julie
[galaxy-dev] Error sorting or uploading Bam File
Hi all, My galaxy is the last update and I work with samtools 1.2. I have a bug with the sorting and the upload of bam file. This bug appears only when I try to upload bam sorted by name or when I try to sort by name. When I work with bam sorted by coordinate, there is no problem. So, when I try to sort a bam file by name, this error appears under the bug icon : Tool execution generated the following error message: [bam_sort_core] merging from 20 files... *** Error in `python': double free or corruption (!prev): 0x02897990 *** Aborted (core dumped) The tool produced the following additional output: [E::hts_idx_push] chromosome blocks not continuous Galaxy attempted to build the BAM index with samtools 1.0+ but failed: [Errno 2] No such file or directory: '/data/galaxy-dist/database/job_working_directory/024/24109/__dataset_48619_metadata_temp_file_MqhjBN.bai' And the same type of error appears when I try to upload bam sorted by name. What I understand is that the bam can be sorted properly (The command line to sort is ok in a terminal with no bug and I can follow this step in the job_working_directory during the run) but Galaxy seems to fail to create a bai. For our analysis pipeline, I think the bai is not crucial, is it possible to avoid this creation of bai during uploading and sorting of bam file by name ? Is there another problem than this creation of bai ? Thanks. Julie ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
