So Ihave finally determined the cause and found a resolution for the snpEff/java problem. The issue was the fact that the vrsion of Java on the Ubuntu linux VM from AWS is 1.6.x snpEff 4.0e requires version 1.7.0 or higher. This is fine, but you willneed to manually upgrade the java version. To do so you will need to complete the following:
SSH to the VM sdo su - root CD to /mnt/galaxy/ Complete the following steps: 1. sudo apt-get purge openjdk* (This removes java 1.6 completely) 2. Modify this file: vi /etc/apt/sources.list.d/cloudbiolinu.list (Remove the leading section of line 11) 3. sudo add-apt-repository ppa:webupd8team/java 4. sudo apt-get update 5. sudo apt-get installoracle-java7-installer 6. java -version Now if you plan to add nodes via the Cloudman Console you will need to perform these tasks for each node you install. I worked with AWS Support to setup an "Auto Scaling Group" to accommodate this process. This required getting my Master instance upgraded and creating an AMI from it. >From that point you can build the group based on the following: http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/creating-your- auto-scaling-groups.html This was, in theory, a great idea. However, it did not work for me. Each of the node that were generated by this tool had Java version 1.6.x and this caused snpEff t fail. My recommendation is that if you have time you should paly around with this more, but I did not have that luxury for this project. Iry On 10/16/14 12:00 PM, "galaxy-dev-requ...@lists.bx.psu.edu" <galaxy-dev-requ...@lists.bx.psu.edu> wrote: >Send galaxy-dev mailing list submissions to > galaxy-dev@lists.bx.psu.edu > >To subscribe or unsubscribe via the Worl Wide Web, visit > http://lists.bx.psu.edu/listinfo/galaxy-dev >or, via email, send a message with subject or body 'help' to > galaxy-ev-requ...@lists.bx.psu.edu > >You can reach the person managing the list at > galaxy-dev-ow...@lists.bx.psu.edu > >When replying, please edit your Subjct line so it is more specific >than "Re: Contents of galaxy-dev digest..." > > >HEY! This is important! If you reply to a thread in a igest, please >1. Change the subject of your response from "Galaxy-dev Digest Vol ..." >to the original subject for the thread. >2. Strip out everything else in the dgest that is not part of the thread >you are responding to. > >Why? >1. This will keep the subject meaningful. People will have some idea>from the >subject line if they should read it or not. >2. Not doing this greatly increases the number of emails that match >search ueries, but that aren't actually informative. > >Today's Topics: > > 1. Re: Set output dbkey from parameter value (Daniel Blankenberg)> 2. Re: > Set output dbkey from parameter value (Nikos Sidiropoulos) > 3. Re: snpeff tool for Galay extra_files_path (John Chilton) > 4. Re: snpeff tool for Galaxy extra_files_path (Bj?rn Gr?nng) > 5. Re: snpeff tool for Galaxy extra_files_path (John Chilton) > 6. Re: snpeff tool for Galaxy extra_files_path (Jim Johnson) > 7. snpEff and java issue(Iry Witham) > 8. Re: HOWTO share tool parameter settings? (Lukasse, Pieter) > 9.Help with Galaxy server migration (Sarah Diehl) > 10. Re: Help with Galaxy server migrtion (John Chilton) > 11. Login issue with a nginx proxy (Alexandre Loywick) > 12. Re: Help with Galaxy server migration (Sarah Diehl) > > >--------------------------------------------------------------------- > >Message: 1 >Date: Wed, 15 Oct 2014 12:14:28 -0400 >From: Daniel Blankenberg <d...@bx.psu.edu> >To: Nikos Sidiropoulos <nikos.sid...@gmail.com> >Cc: "<galaxy-...@bx.psu.edu>" <galaxy-...@bx.psu.edu> >Subject: Re: [galaxy-dev] Set output dbkey from parameter value >Message-ID: <e77c8a92-617a-455e-b5eb-209f4505b...@bx.psu.edu> >Content-Type: text/plain; charset=windows-1252 > >Does removing the ?param_attribute=?value"' attribute help? > > >On Oct 15, 2014, at 11:23 AM, Nikos Sidiropoulos <nikos.sid...@gmail.com> >wrote: > >> Hi Daniel, >> >> Thanks for the response. >> >> I've edited the output to: >> >> <data format="bedgraph" name="bedgraph_slograt" >> label="${tool.name} on ${on_string}: Smoot Log2ratio (bedGraph)" >> from_work_dir="output_dir/slograt.bedgraph"> >> <filter> bedgraph['check'] == 'yes' and slograt['check'] >> == 'yes' </filter> >> <actions> >> <conditional name="bedgraph.check"> >> <when value="yes"> >> <action type="metadata" name="dbkey"> >> <option type="from_param" >> name="bedgraph.genome" param_attribute="value" /> >> </action> >> </when> >> </conditional> >> </actions> >> </data> >> >> Now I'm getting a tool execution error. >> >> Error executing tool: 'unicode' object has no attribute 'value' >> >> I've tried to change the param_attribute to "ext", "dbkey" (ones that >> I know that exist) and got a similar error. >> >> Bests, >> Nikos >> >> On 15 October 2014 16:58, Daniel Blankenberg <d...@bx.psu.edu> wrote: >>> Hi Nikos, >>> >>> In the very least, you?ll want to make sure that you have a bounding >>><actions></actions> tag set around your actions. It is probably also >>>advisable to add a set of conditional/whens around the action, since >>>you?re only setting the dbkey under certain circumstances. >>> >>> >>> Thanks for using Galaxy, >>> >>> Dan >>> >>> >>> On Oct 15, 2014, at 6:37 AM, Nikos Sidiropoulos >>><nikos.sid...@gmail.com> wote: >>> >>>> Hi all, >>>> >>>> I'm trying to set the dbkey of an output file from the value (text) of >>>> a parameter. >>>> >>>> The parameter I want to use is genome. >>>> >>>> <conditional name="bedgraph"> >>>> <param name="check" type="select" label="Produce BedGraph >>>> output" help="Can be displayed directly on UCSC browser. One file per >>>> normalisation method." > >>>> <option value="no" selected="True">No</option> >>>> <option value="yes">Yes</option> >>>> </param> >>>> <when value="yes"> >>>> <param name="bed_file" type="data" format="bed" >>>> label="Transcripts ins BED format" help="12 column BED file containing >>>> trancript definitions." /> >>>> <param name="genome" type="text" label="Genome Build" >>>> help="E.g. hg19" /> >>>> <param name="track_name" type="text" label="Track >>>> Name" size="20" value="Track Name" /> >>>> </when> >>>> <when value="no" /> >>>> </conditional> >>>> >>>> and this is how I've set the output: >>>> >>>> <data format="bedgraph" name="bedgraph_slograt" >>>> label="${tool.name} on ${on_string}: Smoot Log2ratio (bedGraph)" >>>> from_work_dir="output_dir/slograt.bedgraph"> >>>> <filter> bedgraph['check'] == 'yes' and slograt['check'] >>>> == 'yes' </filter> >>>> <action type="metadata" name="dbkey"> >>>> <option type="from_param" name="bedgraph.genome" >>>> param_attribute="value" /> >>>> </action> >>>> </data> >>>> >>>> >>>> When I run the tool the dbkey isn't set to the output file. Does >>>> anyone know a workaround? >>>> >>>> Bests, >>>> Nikos >>>> ___________________________________________________________ >>>> Please keep all replies on the list by using "reply all" >>>> in your mail client. To manage your subscriptions to this >>>> and other Galaxy lists, please use the interface at: >>>> http://lists.bx.psu.edu/ >>>> >>>> To search Galaxy mailing lists use the unified search at: >>>> http://galaxyproject.org/search/mailinglists/ >>> >> > > > > >------------------------------ > >Message: 2 >Date: Wed, 15 Oct 2014 18:43:17 +0200 >From: Nikos Sidiropoulos <nikos.sid...@gmail.com> >To: Daniel Blankenberg <d...@bx.psu.edu> >Cc: "<galaxy-...@bx.psu.edu>" <galaxy-...@bx.psu.edu> >Subject: Re: [galaxy-dev] Set output dbkey from parameter value >Message-ID: > <camjao2o1iw6zhbrmuj3vmh8kye+7dvp2wk7sdomb32oodfn...@mail.gmail.com> >Content-Type: text/plain; charset=UTF-8 > >Yes! > >Thank you very much Daniel. > >On 15 October 2014 18:14, Daniel Blankenberg <d...@bx.psu.edu> wrote: >> Does removing the ?param_attribute=?value"' attribute help? >> >> >> On Oct 15, 2014, at 11:23 AM, Nikos Sidiropoulos >><nikos.sid...@gmail.com> wrote: >> >>> Hi Daniel, >>> >>> Thanks for the response. >>> >>> I've edited the output to: >>> >>> <data format="bedgraph" name="bedgraph_slograt" >>> label="${tool.name} on ${on_string}: Smoot Log2ratio (bedGraph)" >>> from_work_dir="output_dir/slograt.bedgraph"> >>> <filter> bedgraph['check'] == 'yes' and slograt['check'] >>> == 'yes' </filter> >>> <actions> >>> <conditional name="bedgraph.check"> >>> <when value="yes"> >>> <action type="metadata" name="dbkey"> >>> <option type="from_param" >>> name="bedgraph.genome" param_attribute="value" /> >>> </action> >>> </when> >>> </conditional> >>> </actions> >>> </data> >>> >>> Now I'm getting a tool execution error. >>> >>> Error executing tool: 'unicode' object has no attribute 'value' >>> >>> I've tried to change the param_attribute to "ext", "dbkey" (ones that >>> I know that exist) and got a similar error. >>> >>> Bests, >>> Nikos >>> >>> On 15 October 2014 16:58, Daniel Blankenberg <d...@bx.psu.edu> wrote: >>>> Hi Nikos, >>>> >>>> In the very least, you?ll want to make sure that you have a bounding >>>><actions></actions> tag set around your actions. It is probably also >>>>advisable to add a set of conditional/whens around the action, since >>>>you?re only setting the dbkey under certain circumstances. >>>> >>>> >>>> Thanks for using Galaxy, >>>> >>>> Dan >>>> >>>> >>>> On Oct 15, 2014, at 6:37 AM, Nikos Sidiropoulos >>>><nikos.sid...@gmail.com> wrote: >>>> >>>>> Hi all, >>>>> >>>>> I'm tryingto set the dbkey of an output file from the value (text) >>>>>of >>>>>a parameter. >>>>> >>>>> The parameter I want to use is genome. >>>>> >>>> <conditional name="bedgraph"> >>>>> <param name="ceck" type="select" label="Produce BedGraph >>>>> output" help="Can be displayed directly on UCSC browser. One file per >>>>> normalisation method." > >>>>> <option value="no" selected="True">No</option> >>>>> <option value="yes">Yes</option> >>>>> </param> >>>>> <when value="yes"> >>>>> <param name="bed_file" type="data" format="bed" >>>>> label="Transcripts ins BED format" help="12 column BED file >>>>>containing >>>>> trancript definitions." /> >>>>> <param name="genome" type="text" label="Genome Build" >>>>> help="E.g. hg19" /> >>>>> <param name="track_name" type="text" label="Track >>>>> Name" size="20" value="Track Name" /> >>>>> </when> >>>>> <when value="no" /> >>>>> </conditional> >>>>> >>>>> and this is how I've set the output: >>>>> >>>>> <data format="bedgraph" name="bedgraph_slograt" >>>>> label="${tool.name} on ${on_string}: Smoot Log2ratio (bedGraph)" >>>>> from_work_dir="output_dir/slograt.bedgraph"> >>>>> <filter> bedgraph['check'] == 'yes' and slograt['check'] >>>>> == 'yes' </filter> >>>>> <action type="metadata" name="dbkey"> >>>>> <option type="from_param" name="bedgraph.genome" >>>>> param_attribute="value" /> >>>>> </action> >>>>> </data> >>>>> >>>>> >>>>> When I run the tool the dbkey isn't set to the output file. Does >>>>> anyone know a workaround? >>>>> >>>>> Bests, >>>>> Nikos >>>>> ___________________________________________________________ >>>>> Please keep all replies on the list by using "reply all" >>>>> in your mail client. To manage your subscriptions to this >>>>> and other Galaxy lists, please use the interface at: >>>>> http://lists.bx.psu.edu/ >>>>> >>>>> To search Galaxy mailing lists use the unified search at: >>>>> http://galaxyproject.org/search/mailinglists/ >>>> >>> >> > > > >------------------------------ > >Message: 3 >Date: Wed, 15 Oct 2014 13:05:55 -0400 >From: John Chilton <jmchil...@gmail.com> >To: Jim Johnson <j...@umn.edu> >Cc: "galaxy-dev@lists.bx.psu.edu" <galaxy-dev@lists.bx.psu.edu>, Nuwan > Goonasekera <nuwan.goonasek...@unimelb.edu.au>, Andrew Lonie > <alo...@unimelb.edu.au> >Subject: Re: [galaxy-dev] snpeff tool for Galaxy extra_files_path >Message-ID: > <CANwbokevqaN=g_cdtet7e-s25crz2no1at-xaiajmi_t_ts...@mail.gmail.com> >Content-Type: text/plain; charset=UTF-8 > >Hey JJ, > >Opened a pull request to stable with my best guess at the right to >proceed and hopefully a best practice recommendation we can all get >behind. Do you want to try it out and let me know if it fixes snpeff? >(It does fix the velvet datatypes you contributed to Galaxy.) > >https://bitbucket.org/galaxy/galaxy-central/pull-request/532/fix-for-datat >ypes-consuming-output-extra/diff > >Dan Bjoern - does this make sense - can we move forward with this >approach ($input.extra_files_path or inputs and $output.files_path >for outputs) as the best practices forhow to reference these >directories. > >-John > >On Wed, Oct 15, 2014 at 1144 AM, Jim Johnson <johns...@umn.edu> wrote: >> I agree with you about theinadvisable use of: input.dataset.*. >> >> I'm looking at: >> >> lib/galay/model/__init__.py >> class Dataset( object ): >> ... >> def __iit__( self, id=None, state=None, external_filename=None, >> extra_files_path=None, file_size=None, purgable=True, uid=None ): >> ... >> self._extra_files_path = extra_fils_path >> ... >> @property >> def extra_files_path( self ):>> return >> self.object_store.get_filename( self, dir_only=True, >> extra_dir=self._extrafiles_path or "dataset_%d_files" % self.id ) >> >> I'm trying to see whenself._extra_files_path gets set. Otherwise, would >> this return the path rlative to the current file location of dataset? >> >> >> >> >> On 0/15/14, 9:36 AM, John Chilton wrote: >>> >>> Okay - so this is what boke things: >>> >>> >>> >>>https://bitbucket.org/galaxy/galaxy-central/commits/d781366bc120787e201b >>>73a4dd99b56282169d86 >> >>> My feeling with the commit was that wrappers and tools should nevr be >>> explicitly accessing paths explicitly through input.dataset.*. I tink >>> this would circumvent options like outputs_to_working_directory and>>> >>> break remote job execution through Pulsar. It also breaks the object >>> store abstraction I think - which is why I made the change for Bjoer >>> I guess. >>> >>> I did not (and this was stupid on my part) realize hat datatype code >>> would be running on the remote host and accessng these model >>> properties directly outside the abstractions setup by the wrappers >>> supplied to cheetah code and s they have become out of sync as of >>> that commit. >> >>> I am thinking somehow changing what the datatype code gets is the >>>right approach and not fixing things by circumvent the wrapper and >>> accessing properties directly on the datset. Since you will find that >>> doing this breaks things for Bjoern object store and could probably >>> never run on usegalaxy.org say for the same reason. >>> >>> Too many different competng deployment options all being incompatible >>> with each other :(. >>> >>> Will keep thinking about this and respond again. >>> >>> -John >> >>> On Wed, Oct 15, 2014 at 9:39 AM, John Chilton <jmchil...@gmail.com> >>>wrote: >>>> >>>> JJ, >>>> >>>> Arg this is a mess. I am very sorry about this - Istill don't >>>> understand extra_files_path versus files_path myself. There are open >>>> questions on Peter's blast repo and no one ever followed up on my >>>> object store questions about this with Bjoern's issues a couple >>>> release cycles ago. We need to get these to work - write documetation >>>> explicitly declaring best practices we can all agree on and then write >>>> some tests to ensure things don't break in the future. >>>> >>>> When you say your tools broke recently - can you say fo certain which >>>> release broke these - the August14, October14, something older? >>>> >>>> I'll try to do some more research and get back to you. >>>> >>>> -John >>>> >>>> On Tue, Oct 14, 2014 at 6:04 AM, Jim Johnson <johns...@umn.edu> wrote: >>>>> >>>>> Andrew, >>>>> >>>>> Thanks for investigating this. I changed the subject and sent to the >>>>> galaxy >>>>> dev lst. >>>>> >>>>> I've had a number of tools quit working recently. Particularly >>>>>tools >>>>> that >>>>> inspect the extra_files_pth when setting metadata, Defuse, Rsem, >>>>> SnpEff. >>>>> >>>>> I think there was a change in the galaxy framework: >>>>> The extra_files_path when reerenced from an input or output in the >>>>> cheetah >>>>> template sections of the tool config xml will be relative to the job >>>>> woring >>>>> directly rather than the files location. >>>>> I've just changed a few of my tools on my server yesterday >>>>> from: <param_name>.extra_files_path >>>>> o: <param_name>.dataset.extra_files_path >>>>> and they now work again. >>>>> >>>>> Dan or John, is that the right way to handle this? >>>> Thanks, >>>>> >>>>> JJ >>>>> >>>>> >>>>> >>>>> On 10/13/14, 9:29 PM, Andrew Lonie wrote: >>>>>> >>>>>> Hi Jim. I am probably oing about this the wrong way, but I am not >>>>>> clear on how to report tool errors (if in fact this is a tool >>>>>>error!) >>>>>> >>>>> I've been trialling your snpeff wrapper from the test toolshed and >>>>>> getting a consistent eror with the SnpEff Download and SnpEff sub >>>>>> tools (the SnpSift dbNSFP works fine). The prolem seems to be with >>>>>>an >>>>>> attribute declaration and manifests during database download as: >>>>>> >>>>>> Traceback (most recent call last): >>>>>> Fle >>>>>>"/mnt/galaxy/galaxy-app/lib/galaxy/jobs/runners/__init__.py", >>>>>> line 54, in finish_job >>>>>> job_state.job_wrapper.finish( stdout, stderr, exit_code )>>>>>> >>>>>> File "/mnt/galaxy/galaxy-app/lib/galaxy/jobs/__init__.py", line >>>>>> 1107, in finish >>>>>> dataset.datatype.set_meta( dataset, overwrite=False ) # call >>>>> datatype.set_meta directly for the initial set_meta call during >>>>>> dataset creation >>>>>> File >>>>>> >>>>>> >>>>>>"/mnt/galaxy/shed_tools/testtoolshed.g2.bx.psu.edu/repos/iuc/snpeff/1 >>>>>>938721334b3/snpeff/lib/galaxy/datatypes/snpeff.py", >>>>>> line 21, in set_meta >>>>>> data_dir = dataset.files_path >>>>>> AttributeError: 'HistoryDatasetAssociation' object has no attribute >>>>>> 'files_path' >>>>>> >>>>>> >>>>>> We fiddled around with the wrapper, eventually replacing >>>>>> 'dataset.files_path' with 'dataset.extra_files_path' in snpeff.py, >>>>>> which fixed the download bug, but then SnpEff subtool itself threw a >>>>>> similar error when I tried to use that database from the history. >>>>>> >>>>>> I chased up a bit more but cannot understand the various posts on >>>>>> files_path vs extra_files_path >>>>>> >>>>>> I've shared a history with both of these errors here: >>>>>> http://130.56.251.62/galaxy/u/alonie/h/unnamed-history >>>>>> >>>>>> Maybe this is a problem with our Galaxy image? >>>>>> >>>>>> Any help appreciated! >>>>>> >>>>>> Andrew >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> A/Prof Andrew Lonie >>>>>> University of Melbourne >>>>> >>>>> >>>>> >>>>> -- >>>>> James E. Jhnson Minnesota Supercomputing Institute University of >>>>> Minnesota >>>>> ___________________________________________________________ >>>>> Please keep all replies on the list by using "reply all" >>>>> in your mail client. To manage your subscriptions to this >>>>> and other Galaxy lists, please use the interface at: >>>>> http://lists.bx.psu.edu/ >>>>> >>>>> To search Galaxy mailing lists use the unified search at: >>>>> http://galaxyproject.org/search/mailinglists/ >> >> >> >> -- >> James E. Johnson Minnesota Supercomputing Institute University of >>Minnesota > > >------------------------------ > >Message: 4 >Date: Wed, 15 Oct 2014 19:47:40 +0200 >From: Bj?rn Gr?ning <bjoern.gruen...@gmail.com> >To: galaxy-dev@lists.bx.psu.edu, John Chilton <jmchil...@gmail.com> >Subject: Re: [galaxy-dev] snpeff tool for Galaxy extra_files_path >Message-ID: <543eb33c.3070...@gmail.com> >Content-Type: text/plain; charset=windows-1252 > >Hi John, > >glad to see this gets some attention! > >Am 15.10.2014 um 19:05 schrieb John Chilton: >> Hey JJ, >> >> Opened a pull request to stable with my best guess at the right to >> proceed and hopefully a best practice recommendation we can all get >> behind. Do you want to try it out and let me know if it fixes snpeff? >> (It does fix the velvet datatypes you contributed to Galaxy.) >> >> >>https://bitbucket.org/galaxy/galaxy-central/pull-request/532/fix-for-data >>types-consuming-output-extra/diff >> >> Dan, Bjoern - does this make sense - can we move forward with this >> approach ($input.extra_files_path for inputs and $output.files_path >> for outputs) as the best practices for how to reference these >> directories. > >I'm not sure why we need this distinction? Can we not simply choose one >for both, inputs and outputs? >Otherwise we need to explain it very well, why this is needed and I >would vote to rename it to reflect that files_path can be only used by >$outputs ... > >Salve, >Bjoern > >> -John >> >> On Wed, Oct 15, 2014 at 11:44 AM, Jim Johnson <johns...@umn.edu> wrote: >>> I agree with you about the inadvisable use of: input.dataset.*. >>> >>> I'm looking at: >>> >>> lib/galaxy/model/__init__.py >>> class Dataset( object ): >>> ... >>> def __init__( self, id=None, state=None, external_filename=None, >>> extra_files_path=None, file_size=None, purgable=True, uuid=None ): >>> ... >>> self._extra_files_path = extra_files_path >>> ... >>> @property >>> def extra_files_path( self ): >>> return self.object_store.get_filename( self, dir_only=True, >>> extra_dir=self._extra_files_path or "dataset_%d_files" % self.id ) >>> >>> I'm trying to see when self._extra_files_path gets set. Otherwise, >>>would >>> this return the path relative to the current file location of dataset? >>> >>> >>> >>> >>> On 10/15/14, 9:36 AM, John Chilton wrote: >>>> >>>> Okay - so this is what broke things: >>>> >>>> >>>> >>>>https://bitbucket.org/galaxy/galaxy-central/commits/d781366bc120787e201 >>>>b73a4dd99b56282169d86 >>>> >>>> My feeling with the commit was that wrappers and tools should never be >>>> explicitly accessing paths explicitly through input.dataset.*. I think >>>> this would circumvent options like outputs_to_working_directory and >>>> break remote job execution through Pulsar. It also breaks the object >>>> store abstraction I think - which is why I made the change for Bjoern >>>> I guess. >>>> >>>> I did not (and this was stupid on my part) realize that datatype code >>>> would be running on the remote host and accessing these model >>>> properties directly outside the abstractions setup by the wrappers >>>> supplied to cheetah code and so they have become out of sync as of >>>> that commit. >>>> >>>> I am thinking somehow changing what the datatype code gets is the >>>> right approach and not fixing things by circumvent the wrapper and >>>> accessing properties directly on the dataset. Since you will find that >>>> doing this breaks things for Bjoern object store and could probably >>>> never run on usegalaxy.org say for the same reason. >>>> >>>> Too many different competing deployment options all being incompatible >>>> with each other :(. >>>> >>>> Will keep thinking about this and respond again. >>>> >>>> -John >>>> >>>> On Wed, Oct 15, 2014 at 9:39 AM, John Chilton <jmchil...@gmail.com> >>>>wrote: >>>>> >>>>> JJ, >>>>> >>>>> Arg this is a mess. I am very sorry about this - I still don't >>>>> understand extra_files_path versus files_path myself. There are open >>>>> questions on Peter's blast repo and no one ever followed up on my >>>>> object store questions about this with Bjoern's issues a couple >>>>> release cycles ago. We need to get these to work - write documetation >>>>> explicitly declaring best practices we can all agree on and then >>>>>write >>>>> some tests to ensure things don't break in the future. >>>>> >>>>> When you say your tools broke recently - can you say for certain >>>>>which >>>>> release broke these - the August14, October14, something older? >>>>> >>>>> I'll try to do some more research and get back to you. >>>>> >>>>> -John >>>>> >>>>> On Tue, Oct 14, 2014 at 6:04 AM, Jim Johnson <johns...@umn.edu> >>>>>wrote: >>>>>> >>>>>> Andrew, >>>>>> >>>>>> Thanks for investigating this. I changed the subject and sent to the >>>>>> galaxy >>>>>> dev list. >>>>>> >>>>>> I've had a number of tools quit working recently. Particularly >>>>>>tools >>>>>> that >>>>>> inspect the extra_files_path when setting metadata, Defuse, Rsem, >>>>>> SnpEff. >>>>>> >>>>>> I think there was a change in the galaxy framework: >>>>>> The extra_files_path when referenced from an input or output in the >>>>>> cheetah >>>>>> template sections of the tool config xml will be relative to the job >>>>>> working >>>>>> directly rather than the files location. >>>>>> I've just changed a few of my tools on my server yesterday >>>>>> from: <param_name>.extra_files_path >>>>>> to: <param_name>.dataset.extra_files_path >>>>>> and they now work again. >>>>>> >>>>>> Dan or John, is that the right way to handle this? >>>>>> Thanks, >>>>>> >>>>>> JJ >>>>>> >>>>>> >>>>>> >>>>>> On 10/13/14, 9:29 PM, Andrew Lonie wrote: >>>>>>> >>>>>>> Hi Jim. I am probably going about this the wrong way, but I am not >>>>>>> clear on how to report tool errors (if in fact this is a tool >>>>>>>error!) >>>>>>> >>>>>>> I've been trialling your snpeff wrapper from the test toolshed and >>>>>>> getting a consistent error with the SnpEff Download and SnpEff sub >>>>>>> tools (the SnpSift dbNSFP works fine). The problem seems to be >>>>>>>with an >>>>>>> attribute declaration and manifests during database download as: >>>>>>> >>>>>>> Traceback (most recent call last): >>>>>>> File >>>>>>>"/mnt/galaxy/galaxy-app/lib/galaxy/jobs/runners/__init__.py", >>>>>>> line 564, in finish_job >>>>>>> job_state.job_wrapper.finish( stdout, stderr, exit_code ) >>>>>>> File "/mnt/galaxy/galaxy-app/lib/galaxy/jobs/__init__.py", line >>>>>>> 1107, in finish >>>>>>> dataset.datatype.set_meta( dataset, overwrite=False ) # call >>>>>>> datatype.set_meta directly for the initial set_meta call during >>>>>>> dataset creation >>>>>>> File >>>>>>> >>>>>>> >>>>>>>"/mnt/galaxy/shed_tools/testtoolshed.g2.bx.psu.edu/repos/iuc/snpeff/ >>>>>>>1938721334b3/snpeff/lib/galaxy/datatypes/snpeff.py", >>>>>>> line 21, in set_meta >>>>>>> data_dir = dataset.files_path >>>>>>> AttributeError: 'HistoryDatasetAssociation' object has no attribute >>>>>>> 'files_path' >>>>>>> >>>>>>> >>>>>>> We fiddled around with the wrapper, eventually replacing >>>>>>> 'dataset.files_path' with 'dataset.extra_files_path' in snpeff.py, >>>>>>> which fixed the download bug, but then SnpEff subtool itself threw >>>>>>>a >>>>>>> similar error when I tried to use that database from the history. >>>>>>> >>>>>>> I chased up a bit more but cannot understand the various posts on >>>>>>> files_path vs extra_files_path >>>>>>> >>>>>>> I've shared a history with both of these errors here: >>>>>>> http://130.56.251.62/galaxy/u/alonie/h/unnamed-history >>>>>>> >>>>>>> Maybe this is a problem with our Galaxy image? >>>>>>> >>>>>>> Any help appreciated! >>>>>>> >>>>>>> Andrew >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> A/Prof Andrew Lonie >>>>>>> University of Melbourne >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> James E. Johnson Minnesota Supercomputing Institute University of >>>>>> Minnesota >>>>>> ___________________________________________________________ >>>>>> Please keep all replies on the list by using "reply all" >>>>>> in your mail client. To manage your subscriptions to this >>>>>> and other Galaxy lists, please use the interface at: >>>>>> http://lists.bx.psu.edu/ >>>>>> >>>>>> To search Galaxy mailing lists use the unified search at: >>>>>> http://galaxyproject.org/search/mailinglists/ >>> >>> >>> >>> -- >>> James E. Johnson Minnesota Supercomputing Institute University of >>>Minnesota >> ___________________________________________________________ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> http://lists.bx.psu.edu/ >> >> To search Galaxy mailing lists use the unified search at: >> http://galaxyproject.org/search/mailinglists/ >> > > >------------------------------ > >Message: 5 >Date: Wed, 15 Oct 2014 14:06:14 -0400 >From: John Chilton <jmchil...@gmail.com> >To: Bj?rn Gr?ning <bjoern.gruen...@gmail.com> >Cc: "galaxy-dev@lists.bx.psu.edu" <galaxy-dev@lists.bx.psu.edu> >Subject: Re: [galaxy-dev] snpeff tool for Galaxy extra_files_path >Message-ID: > <canwbokfud83xlibr9ewsklapd+kecq6ssavm6yyednqjqqm...@mail.gmail.com> >Content-Type: text/plain; charset=UTF-8 > >On Wed, Oct 15, 2014 at 1:47 PM, Bj?rn Gr?ning ><bjoern.gruen...@gmail.com> wrote: >> Hi John, >> >> glad to see this gets some attention! >> >> Am 15.10.2014 um 19:05 schrieb John Chilton: >>> Hey JJ, >>> >>> Opened a pull request to stable with my best guess at the right to >>> proceed and hopefully a best practice recommendation we can all get >>> behind. Do you want to try it out and let me know if it fixes snpeff? >>> (It does fix the velvet datatypes you contributed to Galaxy.) >>> >>> >>>https://bitbucket.org/galaxy/galaxy-central/pull-request/532/fix-for-dat >>>atypes-consuming-output-extra/diff >>> >>> Dan, Bjoern - does this make sense - can we move forward with this >>> approach ($input.extra_files_path for inputs and $output.files_path >>> for outputs) as the best practices for how to reference these >>> directories. >> >> I'm not sure why we need this distinction? Can we not simply choose one >> for both, inputs and outputs? >> Otherwise we need to explain it very well, why this is needed and I >> would vote to rename it to reflect that files_path can be only used by >> $outputs ... > >I sympathize with you that this adds complexity - I really do. But if >we do anything else we restrict the range of Galaxy versions these >tools can target even further - and we still have to maintain backward >compatibility on all of this junk anyway which is really weighing down >the wrapper and now metadata code as well. > >If you want input.files_path to work - that is fine - I wouldn't be >eager for the change given the complexity it would add to the >implementation but I would probably accept a pull request for that. If >you want $input.input_files_path and $output.output_files_path to work >- I would probably accept pull requests for those as well but I would >not be excited. Finally, I don't personally really want to put the >time in given my reservations and the benefits would not be so great I >don't think because I would think it would be awhile before we could >really recommend those as best practices anyway - given the range of >Galaxy versions people run. > >How about we reach an agreement that with a fictitious Tool 2.0 spec >(https://trello.com/c/AWVobyv1) where we fix all the problems we will >not grant access to $input.dataset directly and we will uniformly only >allow $input.files_path and $output.files_path. > >-John > >> >> Salve, >> Bjoern >> >>> -John >>> >>> On Wed, Oct 15, 2014 at 11:44 AM, Jim Johnson <johns...@umn.edu> wrote: >>>> I agree with you about the inadvisable use of: input.dataset.*. >>>> >>>> I'm looking at: >>>> >>>> lib/galaxy/model/__init__.py >>>> class Dataset( object ): >>>> ... >>>> def __init__( self, id=None, state=None, external_filename=None, >>>> extra_files_path=None, file_size=None, purgable=True, uuid=None ): >>>> ... >>>> self._extra_files_path = extra_files_path >>>> ... >>>> @property >>>> def extra_files_path( self ): >>>> return self.object_store.get_filename( self, dir_only=True, >>>> extra_dir=self._extra_files_path or "dataset_%d_files" % self.id ) >>>> >>>> I'm trying to see when self._extra_files_path gets set. Otherwise, >>>>would >>>> this return the path relative to the current file location of dataset? >>>> >>>> >>>> >>>> >>>> On 10/15/14, 9:36 AM, John Chilton wrote: >>>>> >>>>> Okay - so this is what broke things: >>>>> >>>>> >>>>> >>>>>https://bitbucket.org/galaxy/galaxy-central/commits/d781366bc120787e20 >>>>>1b73a4dd99b56282169d86 >>>>> >>>>> My feeling with the commit was that wrappers and tools should never >>>>>be >>>>> explicitly accessing paths explicitly through input.dataset.*. I >>>>>think >>>>> this would circumvent options like outputs_to_working_directory and >>>>> break remote job execution through Pulsar. It also breaks the object >>>>> store abstraction I think - which is why I made the change for Bjoern >>>>> I guess. >>>>> >>>>> I did not (and this was stupid on my part) realize that datatype code >>>>> would be running on the remote host and accessing these model >>>>> properties directly outside the abstractions setup by the wrappers >>>>> supplied to cheetah code and so they have become out of sync as of >>>>> that commit. >>>>> >>>>> I am thinking somehow changing what the datatype code gets is the >>>>> right approach and not fixing things by circumvent the wrapper and >>>>> accessing properties directly on the dataset. Since you will find >>>>>that >>>>> doing this breaks things for Bjoern object store and could probably >>>>> never run on usegalaxy.org say for the same reason. >>>>> >>>>> Too many different competing deployment options all being >>>>>incompatible >>>>> with each other :(. >>>>> >>>>> Will keep thinking about this and respond again. >>>>> >>>>> -John >>>>> >>>>> On Wed, Oct 15, 2014 at 9:39 AM, John Chilton <jmchil...@gmail.com> >>>>>wrote: >>>>>> >>>>>> JJ, >>>>>> >>>>>> Arg this is a mess. I am very sorry about this - I still don't >>>>>> understand extra_files_path versus files_path myself. There are open >>>>>> questions on Peter's blast repo and no one ever followed up on my >>>>>> object store questions about this with Bjoern's issues a couple >>>>>> release cycles ago. We need to get these to work - write >>>>>>documetation >>>>>> explicitly declaring best practices we can all agree on and then >>>>>>write >>>>>> some tests to ensure things don't break in the future. >>>>>> >>>>>> When you say your tools broke recently - can you say for certain >>>>>>which >>>>>> release broke these - the August14, October14, something older? >>>>>> >>>>>> I'll try to do some more research and get back to you. >>>>>> >>>>>> -John >>>>>> >>>>>> On Tue, Oct 14, 2014 at 6:04 AM, Jim Johnson <johns...@umn.edu> >>>>>>wrote: >>>>>>> >>>>>>> Andrew, >>>>>>> >>>>>>> Thanks for investigating this. I changed the subject and sent to >>>>>>>the >>>>>>> galaxy >>>>>>> dev list. >>>>>>> >>>>>>> I've had a number of tools quit working recently. Particularly >>>>>>>tools >>>>>>> that >>>>>>> inspect the extra_files_path when setting metadata, Defuse, Rsem, >>>>>>> SnpEff. >>>>>>> >>>>>>> I think there was a change in the galaxy framework: >>>>>>> The extra_files_path when referenced from an input or output in the >>>>>>> cheetah >>>>>>> template sections of the tool config xml will be relative to the >>>>>>>job >>>>>>> working >>>>>>> directly rather than the files location. >>>>>>> I've just changed a few of my tools on my server yesterday >>>>>>> from: <param_name>.extra_files_path >>>>>>> to: <param_name>.dataset.extra_files_path >>>>>>> and they now work again. >>>>>>> >>>>>>> Dan or John, is that the right way to handle this? >>>>>>> Thanks, >>>>>>> >>>>>>> JJ >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 10/13/14, 9:29 PM, Andrew Lonie wrote: >>>>>>>> >>>>>>>> Hi Jim. I am probably going about this the wrong way, but I am not >>>>>>>> clear on how to report tool errors (if in fact this is a tool >>>>>>>>error!) >>>>>>>> >>>>>>>> I've been trialling your snpeff wrapper from the test toolshed and >>>>>>>> getting a consistent error with the SnpEff Download and SnpEff sub >>>>>>>> tools (the SnpSift dbNSFP works fine). The problem seems to be >>>>>>>>with an >>>>>>>> attribute declaration and manifests during database download as: >>>>>>>> >>>>>>>> Traceback (most recent call last): >>>>>>>> File >>>>>>>>"/mnt/galaxy/galaxy-app/lib/galaxy/jobs/runners/__init__.py", >>>>>>>> line 564, in finish_job >>>>>>>> job_state.job_wrapper.finish( stdout, stderr, exit_code ) >>>>>>>> File "/mnt/galaxy/galaxy-app/lib/galaxy/jobs/__init__.py", >>>>>>>>line >>>>>>>> 1107, in finish >>>>>>>> dataset.datatype.set_meta( dataset, overwrite=False ) # >>>>>>>>call >>>>>>>> datatype.set_meta directly for the initial set_meta call during >>>>>>>> dataset creation >>>>>>>> File >>>>>>>> >>>>>>>> >>>>>>>>"/mnt/galaxy/shed_tools/testtoolshed.g2.bx.psu.edu/repos/iuc/snpeff >>>>>>>>/1938721334b3/snpeff/lib/galaxy/datatypes/snpeff.py", >>>>>>>> line 21, in set_meta >>>>>>>> data_dir = dataset.files_path >>>>>>>> AttributeError: 'HistoryDatasetAssociation' object has no >>>>>>>>attribute >>>>>>>> 'files_path' >>>>>>>> >>>>>>>> >>>>>>>> We fiddled around with the wrapper, eventually replacing >>>>>>>> 'dataset.files_path' with 'dataset.extra_files_path' in snpeff.py, >>>>>>>> which fixed the download bug, but then SnpEff subtool itself >>>>>>>>threw a >>>>>>>> similar error when I tried to use that database from the history. >>>>>>>> >>>>>>>> I chased up a bit more but cannot understand the various posts on >>>>>>>> files_path vs extra_files_path >>>>>>>> >>>>>>>> I've shared a history with both of these errors here: >>>>>>>> http://130.56.251.62/galaxy/u/alonie/h/unnamed-history >>>>>>>> >>>>>>>> Maybe this is a problem with our Galaxy image? >>>>>>>> >>>>>>>> Any help appreciated! >>>>>>>> >>>>>>>> Andrew >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> A/Prof Andrew Lonie >>>>>>>> University of Melbourne >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> James E. Johnson Minnesota Supercomputing Institute University of >>>>>>> Minnesota >>>>>>> ___________________________________________________________ >>>>>>> Please keep all replies on the list by using "reply all" >>>>>>> in your mail client. To manage your subscriptions to this >>>>>>> and other Galaxy lists, please use the interface at: >>>>>>> http://lists.bx.psu.edu/ >>>>>>> >>>>>>> To search Galaxy mailing lists use the unified search at: >>>>>>> http://galaxyproject.org/search/mailinglists/ >>>> >>>> >>>> >>>> -- >>>> James E. Johnson Minnesota Supercomputing Institute University of >>>>Minnesota >>> ___________________________________________________________ >>> Please keep all replies on the list by using "reply all" >>> in your mail client. To manage your subscriptions to this >>> and other Galaxy lists, please use the interface at: >>> http://lists.bx.psu.edu/ >>> >>> To search Galaxy mailing lists use the unified search at: >>> http://galaxyproject.org/search/mailinglists/ >>> > > > >------------------------------ > >Message: 6 >Date: Wed, 15 Oct 2014 13:07:40 -0500 >From: Jim Johnson <johns...@umn.edu> >To: John Chilton <jmchil...@gmail.com> >Cc: Jim Johnson <j...@umn.edu>, "galaxy-dev@lists.bx.psu.edu" > <galaxy-dev@lists.bx.psu.edu>, Nuwan Goonasekera > <nuwan.goonasek...@unimelb.edu.au>, Andrew Lonie > <alo...@unimelb.edu.au> >Subject: Re: [galaxy-dev] snpeff tool for Galaxy extra_files_path >Message-ID: <543eb7ec.2070...@umn.edu> >Content-Type: text/plain; charset=UTF-8; format=flowed > >Looks good, John. > >I tested with: >https://testtoolshed.g2.bx.psu.edu/view/jjohnson/snpsift_dbnsfp_datatypes > >lib/galaxy/datatypes/converters/tabular_to_dbnsfp.xml > >reverting from hack: > <command interpreter="python">tabular_to_dbnsfp.py $input >$dbnsfp.dataset.extra_files_path/dbNSFP.gz</command> >back to: > <command interpreter="python">tabular_to_dbnsfp.py $input >$dbnsfp.files_path/dbNSFP.gz</command> > >On 10/15/14, 12:05 PM, John Chilton wrote: >> Hey JJ, >> >> Opened a pull request to stable with my best guess at the right to >> proceed and hopefully a best practice recommendation we can all get >> behind. Do you want to try it out and let me know if it fixes snpeff? >> (It does fix the velvet datatypes you contributed to Galaxy.) >> >> >>https://bitbucket.org/galaxy/galaxy-central/pull-request/532/fix-for-data >>types-consuming-output-extra/diff >> >> Dan, Bjoern - does this make sense - can we move forward with this >> approach ($input.extra_files_path for inputs and $output.files_path >> for outputs) as the best practices for how to reference these >> directories. >> >> -John >> >> On Wed, Oct 15, 2014 at 11:44 AM, Jim Johnson <johns...@umn.edu> wrote: >>> I agree with you about the inadvisable use of: input.dataset.*. >>> >>> I'm looking at: >>> >>> lib/galaxy/model/__init__.py >>> class Dataset( object ): >>> ... >>> def __init__( self, id=None, state=None, external_filename=None, >>> extra_files_path=None, file_size=None, purgable=True, uuid=None ): >>> ... >>> self._extra_files_path = extra_files_path >>> ... >>> @property >>> def extra_files_path( self ): >>> return self.object_store.get_filename( self, dir_only=True, >>> extra_dir=self._extra_files_path or "dataset_%d_files" % self.id ) >>> >>> I'm trying to see when self._extra_files_path gets set. Otherwise, >>>would >>> this return the path relative to the current file location of dataset? >>> >>> >>> >>> >>> On 10/15/14, 9:36 AM, John Chilton wrote: >>>> Okay - so this is what broke things: >>>> >>>> >>>> >>>>https://bitbucket.org/galaxy/galaxy-central/commits/d781366bc120787e201 >>>>b73a4dd99b56282169d86 >>>> >>>> My feeling with the commit was that wrappers and tools should never be >>>> explicitly accessing paths explicitly through input.dataset.*. I think >>>> this would circumvent options like outputs_to_working_directory and >>>> break remote job execution through Pulsar. It also breaks the object >>>> store abstraction I think - which is why I made the change for Bjoern >>>> I guess. >>>> >>>> I did not (and this was stupid on my part) realize that datatype code >>>> would be running on the remote host and accessing these model >>>> properties directly outside the abstractions setup by the wrappers >>>> supplied to cheetah code and so they have become out of sync as of >>>> that commit. >>>> >>>> I am thinking somehow changing what the datatype code gets is the >>>> right approach and not fixing things by circumvent the wrapper and >>>> accessing properties directly on the dataset. Since you will find that >>>> doing this breaks things for Bjoern object store and could probably >>>> never run on usegalaxy.org say for the same reason. >>>> >>>> Too many different competing deployment options all being incompatible >>>> with each other :(. >>>> >>>> Will keep thinking about this and respond again. >>>> >>>> -John >>>> >>>> On Wed, Oct 15, 2014 at 9:39 AM, John Chilton <jmchil...@gmail.com> >>>>wrote: >>>>> JJ, >>>>> >>>>> Arg this is a mess. I am very sorry about this - I still don't >>>>> understand extra_files_path versus files_path myself. There are open >>>>> questions on Peter's blast repo and no one ever followed up on my >>>>> object store questions about this with Bjoern's issues a couple >>>>> release cycles ago. We need to get these to work - write documetation >>>>> explicitly declaring best practices we can all agree on and then >>>>>write >>>>> some tests to ensure things don't break in the future. >>>>> >>>>> When you say your tools broke recently - can you say for certain >>>>>which >>>>> release broke these - the August14, October14, something older? >>>>> >>>>> I'll try to do some more research and get back to you. >>>>> >>>>> -John >>>>> >>>>> On Tue, Oct 14, 2014 at 6:04 AM, Jim Johnson <johns...@umn.edu> >>>>>wrote: >>>>>> Andrew, >>>>>> >>>>>> Thanks for investigating this. I changed the subject and sent to the >>>>>> galaxy >>>>>> dev list. >>>>>> >>>>>> I've had a number of tools quit working recently. Particularly >>>>>>tools >>>>>> that >>>>>> inspect the extra_files_path when setting metadata, Defuse, Rsem, >>>>>> SnpEff. >>>>>> >>>>>> I think there was a change in the galaxy framework: >>>>>> The extra_files_path when referenced from an input or output in the >>>>>> cheetah >>>>>> template sections of the tool config xml will be relative to the job >>>>>> working >>>>>> directly rather than the files location. >>>>>> I've just changed a few of my tools on my server yesterday >>>>>> from: <param_name>.extra_files_path >>>>>> to: <param_name>.dataset.extra_files_path >>>>>> and they now work again. >>>>>> >>>>>> Dan or John, is that the right way to handle this? >>>>>> Thanks, >>>>>> >>>>>> JJ >>>>>> >>>>>> >>>>>> >>>>>> On 10/13/14, 9:29 PM, Andrew Lonie wrote: >>>>>>> Hi Jim. I am probably going about this the wrong way, but I am not >>>>>>> clear on how to report tool errors (if in fact this is a tool >>>>>>>error!) >>>>>>> >>>>>>> I've been trialling your snpeff wrapper from the test toolshed and >>>>>>> getting a consistent error with the SnpEff Download and SnpEff sub >>>>>>> tools (the SnpSift dbNSFP works fine). The problem seems to be >>>>>>>with an >>>>>>> attribute declaration and manifests during database download as: >>>>>>> >>>>>>> Traceback (most recent call last): >>>>>>> File >>>>>>>"/mnt/galaxy/galaxy-app/lib/galaxy/jobs/runners/__init__.py", >>>>>>> line 564, in finish_job >>>>>>> job_state.job_wrapper.finish( stdout, stderr, exit_code ) >>>>>>> File "/mnt/galaxy/galaxy-app/lib/galaxy/jobs/__init__.py", >>>>>>>line >>>>>>> 1107, in finish >>>>>>> dataset.datatype.set_meta( dataset, overwrite=False ) # >>>>>>>call >>>>>>> datatype.set_meta directly for the initial set_meta call during >>>>>>> dataset creation >>>>>>> File >>>>>>> >>>>>>> >>>>>>>"/mnt/galaxy/shed_tools/testtoolshed.g2.bx.psu.edu/repos/iuc/snpeff/ >>>>>>>1938721334b3/snpeff/lib/galaxy/datatypes/snpeff.py", >>>>>>> line 21, in set_meta >>>>>>> data_dir = dataset.files_path >>>>>>> AttributeError: 'HistoryDatasetAssociation' object has no attribute >>>>>>> 'files_path' >>>>>>> >>>>>>> >>>>>>> We fiddled around with the wrapper, eventually replacing >>>>>>> 'dataset.files_path' with 'dataset.extra_files_path' in snpeff.py, >>>>>>> which fixed the download bug, but then SnpEff subtool itself threw >>>>>>>a >>>>>>> similar error when I tried to use that database from the history. >>>>>>> >>>>>>> I chased up a bit more but cannot understand the various posts on >>>>>>> files_path vs extra_files_path >>>>>>> >>>>>>> I've shared a history with both of these errors here: >>>>>>> http://130.56.251.62/galaxy/u/alonie/h/unnamed-history >>>>>>> >>>>>>> Maybe this is a problem with our Galaxy image? >>>>>>> >>>>>>> Any help appreciated! >>>>>>> >>>>>>> Andrew >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> A/Prof Andrew Lonie >>>>>>> University of Melbourne >>>>>> >>>>>> >>>>>> -- >>>>>> James E. Johnson Minnesota Supercomputing Institute University of >>>>>> Minnesota >>>>>> ___________________________________________________________ >>>>>> Please keep all replies on the list by using "reply all" >>>>>> in your mail client. To manage your subscriptions to this >>>>>> and other Galaxy lists, please use the interface at: >>>>>> http://lists.bx.psu.edu/ >>>>>> >>>>>> To search Galaxy mailing lists use the unified search at: >>>>>> http://galaxyproject.org/search/mailinglists/ >>> >>> >>> -- >>> James E. Johnson Minnesota Supercomputing Institute University of >>>Minnesota > > >-- >James E. Johnson Minnesota Supercomputing Institute University of >Minnesota > > >------------------------------ > >Message: 7 >Date: Wed, 15 Oct 2014 20:50:56 +0000 >From: Iry Witham <iry.wit...@jax.org> >To: "galaxy-dev@lists.bx.psu.edu" <galaxy-dev@lists.bx.psu.edu> >Subject: [galaxy-dev] snpEff and java issue >Message-ID: <d064566e.39080%iry.wit...@jax.org> >Content-Type: text/plain; charset="us-ascii" > >Hi Team, > >I had manually installed the latest version of snpEff based on the >recommendation of Pablo and after modifying the XML files I had a working >version of snpEff that produced a vcf. However, this morning I reran my >workflow and it is failing again, but this time I am getting the >following error: > >Fatal error: Exit code 1 (Error) >Exception in thread "main" java.lang.UnsupportedClassVersionError: >ca/mcgill/mcb/pcingola/snpEffect/commandLine/SnpEff : Unsupported >major.minor version 51.0 >at java.lang.ClassLoader.defineClass1(Native Method) >at java.lang.ClassLoader.defineClass(ClassLoader.java:643) >at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) >at java.net.URLClassLoader.defineClass(URLClassLoader.java:277) >at java.net.URLClassLoader.access$000(URLClassLoader.java:73) >at java.net.URLClassLoader$1.run(URLClassLoader.java:212) >at java.security.AccessController.doPrivileged(Native Method) >at java.net.URLClassLoader.findClass(URLClassLoader.java:205) >at java.lang.ClassLoader.loadClass(ClassLoader.java:323) >at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294) >at java.lang.ClassLoader.loadClass(ClassLoader.java:268) >Could not find the main class: >ca.mcgill.mcb.pcingola.snpEffect.commandLine.SnpEff. Program will exit. > >Nothing has been changed since I had success. > >Regards, >Iry > > >The information in this email, including attachments, may be confidential >and is intended solely for the addressee(s). If you believe you received >this email by mistake, please notify the sender by return email as soon >as possible. >-------------- next part -------------- >An HTML attachment was scrubbed... >URL: ><http://lists.bx.psu.edu/pipermail/galaxy-dev/attachments/20141015/b39b942 >0/attachment-0001.html> > >------------------------------ > >Message: 8 >Date: Thu, 16 Oct 2014 08:15:50 +0000 >From: "Lukasse, Pieter" <pieter.luka...@wur.nl> >To: "'John Chilton'" <jmchil...@gmail.com> >Cc: "galaxy-dev@lists.bx.psu.edu" <galaxy-dev@lists.bx.psu.edu> >Subject: Re: [galaxy-dev] HOWTO share tool parameter settings? >Message-ID: > <2b5462aefd6a454c810ffd8b0d1e7da130275...@scomp0933.wurnet.nl> >Content-Type: text/plain; charset="utf-8" > >Hi John, > >I see that in my last email I replied to you with the fourth (and not the >third) option in mind :(. > >Just to come back to this related to the third option (i.e. " Have a >dummy tool to produce a settings file and allow the users to choose this >file when running the real tool"): I don't like this option so much >because the file is never as clear as the tool form that produced it. If >users are used to looking at the tool form for seeing details about the >set parameters and now they will have to look at a text file, this will >be confusing. It is not a huge problem, but it is one that I am trying to >avoid. I.e. it would be best if users are presented with only one way of >checking parameters of executed processes. > >In the scenario I am proposing, the settings files are disseminated just >as they are (e.g. a text file) either by sharing them in the normal ways >or by placing a number of them in pre-defined folders in Galaxy. The tool >form would allow the user to select one and would then fill in the values >of the parameters accordingly. I'm guessing similar binding logic is also >happening right now when a user clicks "rerun" on a step in his history. > >Best regards, > >Pieter. > >-----Original Message----- >From: John Chilton [mailto:jmchil...@gmail.com] >Sent: donderdag 9 oktober 2014 15:45 >To: Lukasse, Pieter >Cc: galaxy-dev@lists.bx.psu.edu >Subject: Re: [galaxy-dev] HOWTO share tool parameter settings? > >How would this be different then the third option you listed? > >You want it to work with all tools and you as the developer want to be >able to construct these files without needing a dummy tool to produce the >values? > >How would imagine these setting files would be disseminated to users and >then selected by users? > >-John > > >On Thu, Oct 9, 2014 at 9:34 AM, Lukasse, Pieter <pieter.luka...@wur.nl> >wrote: >> Hi, >> >> >> >> Do we have a way (or plans) for sharing tool parameter settings in >>Galaxy? >> >> >> >> I know the following workarounds : >> >> >> >> ? Share a history with all users: so users can import your step >>and >> do ?rerun? to run on their own file with your settings >> >> ? Wrap the step in a workflow with all parameters set and >>publish >> this workflow: users can run this ?workflow? >> >> ? Have a dummy tool to produce a settings file and allow the >>users >> to choose this file when running the real tool >> >> ? Use a conditional and many macros that are basically a copy of >> each other, only differing in the parameter values >> >> >> >> >> >> But what I would like to have is a way to define bindings between a >> settings file and the parameters in the tool form. Any plans, ideas? >> >> >> >> Thanks, >> >> >> >> Pieter Lukasse >> >> Wageningen UR, Plant Research International >> >> Department of Bioinformatics (Bioscience) >> >> Wageningen Campus, Building 107, Droevendaalsesteeg 1, 6708 PB, >> Wageningen, the Netherlands >> >> T: +31-317481122; >> M: +31-628189540; >> skype: pieter.lukasse.wur >> >> http://www.pri.wur.nl >> >> >> >> >> ___________________________________________________________ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this and other >> Galaxy lists, please use the interface at: >> http://lists.bx.psu.edu/ >> >> To search Galaxy mailing lists use the unified search at: >> http://galaxyproject.org/search/mailinglists/ > > > >------------------------------ > >Message: 9 >Date: Thu, 16 Oct 2014 14:36:14 +0200 (CEST) >From: Sarah Diehl <di...@ie-freiburg.mpg.de> >To: "galaxy-dev@lists.bx.psu.edu List" <galaxy-dev@lists.bx.psu.edu> >Subject: [galaxy-dev] Help with Galaxy server migration >Message-ID: > <9eac34f1-a953-400c-be48-526f8f1ab...@solsys2.immunbio.mpg.de> >Content-Type: text/plain; charset=utf-8 > >Dear all, > >I'm trying to migrate our local Galaxy server from a standalone server >running on CentOS 5 to a setup that runs Galaxy in a Docker container >(bgruening/galaxy-stable). Due to Docker and the setup of the container, >almost all data, tools, configs etc. have to move to a different place. >Since our old server is pretty cluttered up anyway, I would like to move >just user data (logins, histories, datasets ...), but none of the tool >installations, tool data or configs. > >My plan was to start with a fresh and clean new Galaxy installation, >transfer the datasets and the database and then reinstall all tools >through the toolshed (of course there are many more small odds and ends >that need fixing). > >The problem is that after transfering the database, I always get an error >when going to "Manage installed tool shed repositories" (see end of the >mail). I tried just overwriting the pg_data directory (PostgreSQL >versions are identical) and also exporting to sql (pg_dump) and importing >in the new DB. Both times I made sure to run manage_db.sh upgrade >afterwards. The error stays the same. > >I can't make much sense of the error message, my only guess is that it >looks for something that's not there. However, I really don't want to >transfer the tool directories and tool configurations, because that will >get messy and difficult, since many paths need to be adjusted. There's a >high chance for error and this way I also carry over the clutter. > >So is there a way to remove all tool installation information from the >database, before I transfer it? Can this even be disentangled from the >histories and datasets? Any help or other ideas on how to do this >migration are highly appreciated! > >Thanks, >Sarah > > > >10.1.5.190 - - [16/Oct/2014:09:54:35 +0000] "GET >/admin_toolshed/browse_repositories HTTP/1.1" 500 - >"http://deep2:8080/admin/index"; "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; >rv:32.0) Gecko/20100101 Firefox/32.0" >Error - <type 'exceptions.UnicodeDecodeError'>: 'ascii' codec can't >decode byte 0xe2 in position 393: ordinal not in range(128) >URL: http://deep2:8080/admin_toolshed/browse_repositories >File 'lib/galaxy/web/framework/middleware/error.py', line 149 in __call__ > app_iter = self.application(environ, sr_checker) >File '/galaxy-central/eggs/Paste-1.7.5.1-py2.7.egg/paste/recursive.py', >line 84 in __call__ > return self.application(environ, start_response) >File >'/galaxy-central/eggs/Paste-1.7.5.1-py2.7.egg/paste/httpexceptions.py', >line 633 in __call__ > return self.application(environ, start_response) >File 'lib/galaxy/web/framework/base.py', line 132 in __call__ > return self.handle_request( environ, start_response ) >File 'lib/galaxy/web/framework/base.py', line 190 in handle_request > body = method( trans, **kwargs ) >File 'lib/galaxy/web/framework/decorators.py', line 87 in decorator > return func( self, trans, *args, **kwargs ) >File 'lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py', line 167 >in browse_repositories > return self.installed_repository_grid( trans, **kwd ) >File 'lib/galaxy/web/framework/helpers/grids.py', line 306 in __call__ > kwargs=kwargs ) >File 'lib/galaxy/web/framework/webapp.py', line 771 in fill_template > return self.fill_template_mako( filename, **kwargs ) >File 'lib/galaxy/web/framework/webapp.py', line 786 in fill_template_mako > return template.render( **data ) >File '/galaxy-central/eggs/Mako-0.4.1-py2.7.egg/mako/template.py', line >296 in render > return runtime._render(self, self.callable_, args, data) >File '/galaxy-central/eggs/Mako-0.4.1-py2.7.egg/mako/runtime.py', line >660 in _render > **_kwargs_for_callable(callable_, data)) >File '/galaxy-central/eggs/Mako-0.4.1-py2.7.egg/mako/runtime.py', line >692 in _render_context > _exec_template(inherit, lclcontext, args=args, kwargs=kwargs) >File '/galaxy-central/eggs/Mako-0.4.1-py2.7.egg/mako/runtime.py', line >718 in _exec_template > callable_(context, *args, **kwargs) >File '/galaxy-central/database/compiled_templates/base.mako.py', line 66 >in render_body > __M_writer(unicode(next.body())) >File '/galaxy-central/database/compiled_templates/grid_base.mako.py', >line 91 in render_body > __M_writer(unicode(self.load())) >File '/galaxy-central/database/compiled_templates/grid_base.mako.py', >line 120 in render_load > __M_writer(unicode( h.dumps( self.get_grid_config( embedded=embedded, >insert=insert ) ) )) >File '/galaxy-central/database/compiled_templates/grid_base.mako.py', >line 303 in render_get_grid_config > value = column.get_value( trans, grid, item ) >File 'lib/tool_shed/galaxy_install/grids/admin_toolshed_grids.py', line >106 in get_value > return suc.get_tool_shed_repository_status_label( trans.app, >tool_shed_repository ) >File 'lib/tool_shed/util/shed_util_common.py', line 829 in >get_tool_shed_repository_status_label > elif tool_shed_repository.tool_dependencies_being_installed: >File 'lib/galaxy/model/tool_shed_install/__init__.py', line 408 in >tool_dependencies_being_installed > for tool_dependency in self.tool_dependencies: >File >'/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalch >emy/orm/attributes.py', line 168 in __get__ >File >'/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalch >emy/orm/attributes.py', line 453 in get >File >'/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalch >emy/orm/strategies.py', line 508 in _load_for_state >File >'/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalch >emy/orm/strategies.py', line 574 in _emit_lazyload >File >'/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalch >emy/orm/query.py', line 2115 in all >File >'/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalch >emy/orm/query.py', line 2341 in instances >File >'/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalch >emy/engine/base.py', line 3204 in fetchall >File >'/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalch >emy/engine/base.py', line 3171 in _fetchall_impl >UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 393: >ordinal not in range(128) > > >------------------------------ > >Message: 10 >Date: Thu, 16 Oct 2014 09:17:39 -0400 >From: John Chilton <jmchil...@gmail.com> >To: Sarah Diehl <di...@ie-freiburg.mpg.de> >Cc: "galaxy-dev@lists.bx.psu.edu List" <galaxy-dev@lists.bx.psu.edu> >Subject: Re: [galaxy-dev] Help with Galaxy server migration >Message-ID: > <CANwbokfO1ZBxqWk9YLkqNiVm5qkN_LD5yNr5xrtanX=2SuzM=w...@mail.gmail.com> >Content-Type: text/plain; charset=UTF-8 > >Hmm.... hopefully someone more knowledgeable about the tool shed than >me responds also but I had a couple quick thoughts. > >The first is a warning - workflows may break. At very least workflows >that depend on the previous instance having gone through tool >migrations instead of tool install. My understanding is tool >migrations cause links to the older versions of the tools to be >created so workflows continue to operate - just installing the same >version of the tool again doesn't set this up. Also - in theory if you >resinstall the same versions of tools tool re-runs and stuff should >still work - but I am not confident and I have not tested it myself - >so if that functionality is important to you, you test it out before >throwing out the old configuration. > >Next - I am a little bit concerned about this error. It would seem to >point to some sort of encoding problem with the way the database was >transferred - different table encodings or something? I don't really >know anything about encoding and postgres though. > >That said I am not surprised there are problems with the tool shed. >There are all sorts of implicit dependencies between what is on disk >and what is in the database. If you really do want to start fresh I >would clear out all of the tool shed tables before continuing. These >tables including - >tool_shed_repository,repository_repository_dependency_association, >repository_dependency, tool_dependency, tool_version, >tool_version_association, migrate_tools. > >Rather than truncating the existing tables - you could also just have >Galaxy target a completely separate database for tool shed install >stuff by creating a new postgres database, setting >install_database_connection in your galaxy ini file (either >universe_wsgi.ini or galaxy.ini depending on what version of Bjoern's >docker image you are targetting) - or in the newest version of that >Dockerfile you can just make sure your docker run includes a -e >"GALAXY_CONFIG_INSTALL_DATABASE_CONNECTION=<postgresconnectionstring>". > >You will also want to make sure that database gets populated - I think >either of the following would work >./create_db.sh install >./migrate_db.sh install upgrade > >Sorry I have not definitive answers - but hopefully this is helpful. >Good luck with the migration. You should let the list know how the >Docker container thing goes for a production server. > >-John > >On Thu, Oct 16, 2014 at 8:36 AM, Sarah Diehl <di...@ie-freiburg.mpg.de> >wrote: >> Dear all, >> >> I'm trying to migrate our local Galaxy server from a standalone server >>running on CentOS 5 to a setup that runs Galaxy in a Docker container >>(bgruening/galaxy-stable). Due to Docker and the setup of the container, >>almost all data, tools, configs etc. have to move to a different place. >>Since our old server is pretty cluttered up anyway, I would like to move >>just user data (logins, histories, datasets ...), but none of the tool >>installations, tool data or configs. >> >> My plan was to start with a fresh and clean new Galaxy installation, >>transfer the datasets and the database and then reinstall all tools >>through the toolshed (of course there are many more small odds and ends >>that need fixing). >> >> The problem is that after transfering the database, I always get an >>error when going to "Manage installed tool shed repositories" (see end >>of the mail). I tried just overwriting the pg_data directory (PostgreSQL >>versions are identical) and also exporting to sql (pg_dump) and >>importing in the new DB. Both times I made sure to run manage_db.sh >>upgrade afterwards. The error stays the same. >> >> I can't make much sense of the error message, my only guess is that it >>looks for something that's not there. However, I really don't want to >>transfer the tool directories and tool configurations, because that will >>get messy and difficult, since many paths need to be adjusted. There's a >>high chance for error and this way I also carry over the clutter. >> >> So is there a way to remove all tool installation information from the >>database, before I transfer it? Can this even be disentangled from the >>histories and datasets? Any help or other ideas on how to do this >>migration are highly appreciated! >> >> Thanks, >> Sarah >> >> >> >> 10.1.5.190 - - [16/Oct/2014:09:54:35 +0000] "GET >>/admin_toolshed/browse_repositories HTTP/1.1" 500 - >>"http://deep2:8080/admin/index"; "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; >>rv:32.0) Gecko/20100101 Firefox/32.0" >> Error - <type 'exceptions.UnicodeDecodeError'>: 'ascii' codec can't >>decode byte 0xe2 in position 393: ordinal not in range(128) >> URL: http://deep2:8080/admin_toolshed/browse_repositories >> File 'lib/galaxy/web/framework/middleware/error.py', line 149 in >>__call__ >> app_iter = self.application(environ, sr_checker) >> File '/galaxy-central/eggs/Paste-1.7.5.1-py2.7.egg/paste/recursive.py', >>line 84 in __call__ >> return self.application(environ, start_response) >> File >>'/galaxy-central/eggs/Paste-1.7.5.1-py2.7.egg/paste/httpexceptions.py', >>line 633 in __call__ >> return self.application(environ, start_response) >> File 'lib/galaxy/web/framework/base.py', line 132 in __call__ >> return self.handle_request( environ, start_response ) >> File 'lib/galaxy/web/framework/base.py', line 190 in handle_request >> body = method( trans, **kwargs ) >> File 'lib/galaxy/web/framework/decorators.py', line 87 in decorator >> return func( self, trans, *args, **kwargs ) >> File 'lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py', line >>167 in browse_repositories >> return self.installed_repository_grid( trans, **kwd ) >> File 'lib/galaxy/web/framework/helpers/grids.py', line 306 in __call__ >> kwargs=kwargs ) >> File 'lib/galaxy/web/framework/webapp.py', line 771 in fill_template >> return self.fill_template_mako( filename, **kwargs ) >> File 'lib/galaxy/web/framework/webapp.py', line 786 in >>fill_template_mako >> return template.render( **data ) >> File '/galaxy-central/eggs/Mako-0.4.1-py2.7.egg/mako/template.py', line >>296 in render >> return runtime._render(self, self.callable_, args, data) >> File '/galaxy-central/eggs/Mako-0.4.1-py2.7.egg/mako/runtime.py', line >>660 in _render >> **_kwargs_for_callable(callable_, data)) >> File '/galaxy-central/eggs/Mako-0.4.1-py2.7.egg/mako/runtime.py', line >>692 in _render_context >> _exec_template(inherit, lclcontext, args=args, kwargs=kwargs) >> File '/galaxy-central/eggs/Mako-0.4.1-py2.7.egg/mako/runtime.py', line >>718 in _exec_template >> callable_(context, *args, **kwargs) >> File '/galaxy-central/database/compiled_templates/base.mako.py', line >>66 in render_body >> __M_writer(unicode(next.body())) >> File '/galaxy-central/database/compiled_templates/grid_base.mako.py', >>line 91 in render_body >> __M_writer(unicode(self.load())) >> File '/galaxy-central/database/compiled_templates/grid_base.mako.py', >>line 120 in render_load >> __M_writer(unicode( h.dumps( self.get_grid_config( embedded=embedded, >>insert=insert ) ) )) >> File '/galaxy-central/database/compiled_templates/grid_base.mako.py', >>line 303 in render_get_grid_config >> value = column.get_value( trans, grid, item ) >> File 'lib/tool_shed/galaxy_install/grids/admin_toolshed_grids.py', line >>106 in get_value >> return suc.get_tool_shed_repository_status_label( trans.app, >>tool_shed_repository ) >> File 'lib/tool_shed/util/shed_util_common.py', line 829 in >>get_tool_shed_repository_status_label >> elif tool_shed_repository.tool_dependencies_being_installed: >> File 'lib/galaxy/model/tool_shed_install/__init__.py', line 408 in >>tool_dependencies_being_installed >> for tool_dependency in self.tool_dependencies: >> File >>'/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalc >>hemy/orm/attributes.py', line 168 in __get__ >> File >>'/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalc >>hemy/orm/attributes.py', line 453 in get >> File >>'/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalc >>hemy/orm/strategies.py', line 508 in _load_for_state >> File >>'/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalc >>hemy/orm/strategies.py', line 574 in _emit_lazyload >> File >>'/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalc >>hemy/orm/query.py', line 2115 in all >> File >>'/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalc >>hemy/orm/query.py', line 2341 in instances >> File >>'/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalc >>hemy/engine/base.py', line 3204 in fetchall >> File >>'/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalc >>hemy/engine/base.py', line 3171 in _fetchall_impl >> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position >>393: ordinal not in range(128) >> ___________________________________________________________ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> http://lists.bx.psu.edu/ >> >> To search Galaxy mailing lists use the unified search at: >> http://galaxyproject.org/search/mailinglists/ > > > >------------------------------ > >Message: 11 >Date: Thu, 16 Oct 2014 14:29:26 +0200 >From: Alexandre Loywick <a.loyw...@genesdiffusion.com> >To: "galaxy-dev@lists.bx.psu.edu" <galaxy-dev@lists.bx.psu.edu> >Subject: [galaxy-dev] Login issue with a nginx proxy >Message-ID: > <cakkmhgkush6ybqo6rgtagcn4jv2okm0ttg2eeskbfnfsx-d...@mail.gmail.com> >Content-Type: text/plain; charset="utf-8" > >Hello galaxians, > >I'm currently trying to set up a production galaxy and make it available >to >the public but I face a weird issue, probably because of my nginx proxy >configuration. > >Galaxy is on a kvm virtual machine running on a host using nginx as proxy. > >to make it a bit clearer: > >172.30.24.35 (client) ---> http://172.30.24.63/machine1 (Host & nginx >proxy) --- > http://192.168.122.254:8080/ (virtual machine) > > >I can connect to galaxy but when logging in, the virtual machine redirects >me on 192.168.122.254:8080, which is unreachable from the 172.30.24.0 >network. If I log in from the Host, it's all fine. > > > > >in the universe_wsgi.ini, i've set: > ># Define the proxy-prefix filter. >[filter:proxy-prefix] >use = egg:PasteDeploy#prefix >prefix = /machine1 > >... > ># [filter:proxy-prefix] section above. >filter-with = proxy-prefix > >as described in the documentation. > > > > >My virtual machine nginx conf file: > >user galaxy; >worker_processes 2; > >#error_log logs/error.log; >#error_log logs/error.log notice; >#error_log logs/error.log info; > >#pid logs/nginx.pid; > > >events { > worker_connections 1024; >} > > >http { > include mime.types; > default_type application/octet-stream; > > #log_format main '$remote_addr - $remote_user [$time_local] >"$request" ' > # '$status $body_bytes_sent "$http_referer" ' > # '"$http_user_agent" "$http_x_forwarded_for"'; > > #access_log logs/access.log main; > > sendfile on; > #tcp_nopush on; > > #keepalive_timeout 0; > keepalive_timeout 65; > > gzip on; > gzip_http_version 1.1; > gzip_vary on; > gzip_comp_level 4; > gzip_proxied any; > gzip_types text/plain text/css application/x-javascript text/xml >application/xml text/javascript application/json; > gzip_buffers 16 8k; > gzip_disable "MSIE [1-6].(?!.*SV1)"; > > upstream galaxy_app{ > server localhost:8080; > } > server { > client_max_body_size 10G; > listen 80; > server_name localhost; > > #charset koi8-r; > #access_log logs/host.access.log main; > > location /machine1 { > proxy_pass http://galaxy_app; > proxy_set_header Host $host; > proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; > } > location /machine1/static { > alias /home/galaxy/galaxy-dist/static; > } > location /machine1/static/style { > alias /home/galaxy/galaxy-dist/static/june_2007_style/blue; > } > location /machine1/static/scripts { > alias /home/galaxy/galaxy-dist/static/scripts/packed; > } > location /machine1/favicon.ico { > alias /home/galaxy/galaxy-dist/static/favicon.ico; > } > location /machine1/robots.txt { > alias /home/galaxy/galaxy-dist/static/robots.txt; > } > location /_x_accel_redirect/ { > internal; > alias /machine1; > } > > #error_page 404 /404.html; > #redirect server error pages to the static page /50x.html > # > error_page 500 502 503 504 /50x.html; > location = /50x.html { > root html; > } > } >} > > > > >And my host configuration : > > >server { > listen 80 default_server; > listen [::]:80 default_server ipv6only=on; > > root /usr/share/nginx/html; > index index.html index.htm; > > # Make site accessible from http://localhost/ > server_name localhost; > > location / { > try_files $uri $uri/ =404; > } > > location /machine1 { > proxy_pass http://192.168.122.254:8080; > } >} >-------------- next part -------------- >An HTML attachment was scrubbed... >URL: ><http://lists.bx.psu.edu/pipermail/galaxy-dev/attachments/20141016/9fab6b9 >7/attachment-0001.html> > >------------------------------ > >Message: 12 >Date: Thu, 16 Oct 2014 17:50:03 +0200 (CEST) >From: Sarah Diehl <di...@ie-freiburg.mpg.de> >To: John Chilton <jmchil...@gmail.com> >Cc: "galaxy-dev@lists.bx.psu.edu List" <galaxy-dev@lists.bx.psu.edu> >Subject: Re: [galaxy-dev] Help with Galaxy server migration >Message-ID: > <4847b400-4eed-4ea1-a014-218945ffd...@solsys2.immunbio.mpg.de> >Content-Type: text/plain; charset=utf-8 > >Dear John, > >thank you very much your help and the warnings! > >I'm testing it with a second database and so far it seems to work :-). > >I was also suspecting the encoding to be the issue, that's why I did it >through pg_dump and specified the encoding of the new server for the >dump. The error stayed the same and so far I have just seen it in the >context of toolshed operations. I don't know what else I could do to >ensure proper encoding. > >Best regards, >Sarah > > > >----- Original Message ----- >From: "John Chilton" <jmchil...@gmail.com> >To: "Sarah Diehl" <di...@ie-freiburg.mpg.de> >Cc: "galaxy-dev@lists.bx.psu.edu List" <galaxy-dev@lists.bx.psu.edu> >Sent: Thursday, October 16, 2014 3:17:39 PM >Subject: Re: [galaxy-dev] Help with Galaxy server migration > >Hmm.... hopefully someone more knowledgeable about the tool shed than >me responds also but I had a couple quick thoughts. > >The first is a warning - workflows may break. At very least workflows >that depend on the previous instance having gone through tool >migrations instead of tool install. My understanding is tool >migrations cause links to the older versions of the tools to be >created so workflows continue to operate - just installing the same >version of the tool again doesn't set this up. Also - in theory if you >resinstall the same versions of tools tool re-runs and stuff should >still work - but I am not confident and I have not tested it myself - >so if that functionality is important to you, you test it out before >throwing out the old configuration. > >Next - I am a little bit concerned about this error. It would seem to >point to some sort of encoding problem with the way the database was >transferred - different table encodings or something? I don't really >know anything about encoding and postgres though. > >That said I am not surprised there are problems with the tool shed. >There are all sorts of implicit dependencies between what is on disk >and what is in the database. If you really do want to start fresh I >would clear out all of the tool shed tables before continuing. These >tables including - >tool_shed_repository,repository_repository_dependency_association, >repository_dependency, tool_dependency, tool_version, >tool_version_association, migrate_tools. > >Rather than truncating the existing tables - you could also just have >Galaxy target a completely separate database for tool shed install >stuff by creating a new postgres database, setting >install_database_connection in your galaxy ini file (either >universe_wsgi.ini or galaxy.ini depending on what version of Bjoern's >docker image you are targetting) - or in the newest version of that >Dockerfile you can just make sure your docker run includes a -e >"GALAXY_CONFIG_INSTALL_DATABASE_CONNECTION=<postgresconnectionstring>". > >You will also want to make sure that database gets populated - I think >either of the following would work >./create_db.sh install >./migrate_db.sh install upgrade > >Sorry I have not definitive answers - but hopefully this is helpful. >Good luck with the migration. You should let the list know how the >Docker container thing goes for a production server. > >-John > >On Thu, Oct 16, 2014 at 8:36 AM, Sarah Diehl <di...@ie-freiburg.mpg.de> >wrote: >> Dear all, >> >> I'm trying to migrate our local Galaxy server from a standalone server >>running on CentOS 5 to a setup that runs Galaxy in a Docker container >>(bgruening/galaxy-stable). Due to Docker and the setup of the container, >>almost all data, tools, configs etc. have to move to a different place. >>Since our old server is pretty cluttered up anyway, I would like to move >>just user data (logins, histories, datasets ...), but none of the tool >>installations, tool data or configs. >> >> My plan was to start with a fresh and clean new Galaxy installation, >>transfer the datasets and the database and then reinstall all tools >>through the toolshed (of course there are many more small odds and ends >>that need fixing). >> >> The problem is that after transfering the database, I always get an >>error when going to "Manage installed tool shed repositories" (see end >>of the mail). I tried just overwriting the pg_data directory (PostgreSQL >>versions are identical) and also exporting to sql (pg_dump) and >>importing in the new DB. Both times I made sure to run manage_db.sh >>upgrade afterwards. The error stays the same. >> >> I can't make much sense of the error message, my only guess is that it >>looks for something that's not there. However, I really don't want to >>transfer the tool directories and tool configurations, because that will >>get messy and difficult, since many paths need to be adjusted. There's a >>high chance for error and this way I also carry over the clutter. >> >> So is there a way to remove all tool installation information from the >>database, before I transfer it? Can this even be disentangled from the >>histories and datasets? Any help or other ideas on how to do this >>migration are highly appreciated! >> >> Thanks, >> Sarah >> >> >> >> 10.1.5.190 - - [16/Oct/2014:09:54:35 +0000] "GET >>/admin_toolshed/browse_repositories HTTP/1.1" 500 - >>"http://deep2:8080/admin/index"; "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; >>rv:32.0) Gecko/20100101 Firefox/32.0" >> Error - <type 'exceptions.UnicodeDecodeError'>: 'ascii' codec can't >>decode byte 0xe2 in position 393: ordinal not in range(128) >> URL: http://deep2:8080/admin_toolshed/browse_repositories >> File 'lib/galaxy/web/framework/middleware/error.py', line 149 in >>__call__ >> app_iter = self.application(environ, sr_checker) >> File '/galaxy-central/eggs/Paste-1.7.5.1-py2.7.egg/paste/recursive.py', >>line 84 in __call__ >> return self.application(environ, start_response) >> File >>'/galaxy-central/eggs/Paste-1.7.5.1-py2.7.egg/paste/httpexceptions.py', >>line 633 in __call__ >> return self.application(environ, start_response) >> File 'lib/galaxy/web/framework/base.py', line 132 in __call__ >> return self.handle_request( environ, start_response ) >> File 'lib/galaxy/web/framework/base.py', line 190 in handle_request >> body = method( trans, **kwargs ) >> File 'lib/galaxy/web/framework/decorators.py', line 87 in decorator >> return func( self, trans, *args, **kwargs ) >> File 'lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py', line >>167 in browse_repositories >> return self.installed_repository_grid( trans, **kwd ) >> File 'lib/galaxy/web/framework/helpers/grids.py', line 306 in __call__ >> kwargs=kwargs ) >> File 'lib/galaxy/web/framework/webapp.py', line 771 in fill_template >> return self.fill_template_mako( filename, **kwargs ) >> File 'lib/galaxy/web/framework/webapp.py', line 786 in >>fill_template_mako >> return template.render( **data ) >> File '/galaxy-central/eggs/Mako-0.4.1-py2.7.egg/mako/template.py', line >>296 in render >> return runtime._render(self, self.callable_, args, data) >> File '/galaxy-central/eggs/Mako-0.4.1-py2.7.egg/mako/runtime.py', line >>660 in _render >> **_kwargs_for_callable(callable_, data)) >> File '/galaxy-central/eggs/Mako-0.4.1-py2.7.egg/mako/runtime.py', line >>692 in _render_context >> _exec_template(inherit, lclcontext, args=args, kwargs=kwargs) >> File '/galaxy-central/eggs/Mako-0.4.1-py2.7.egg/mako/runtime.py', line >>718 in _exec_template >> callable_(context, *args, **kwargs) >> File '/galaxy-central/database/compiled_templates/base.mako.py', line >>66 in render_body >> __M_writer(unicode(next.body())) >> File '/galaxy-central/database/compiled_templates/grid_base.mako.py', >>line 91 in render_body >> __M_writer(unicode(self.load())) >> File '/galaxy-central/database/compiled_templates/grid_base.mako.py', >>line 120 in render_load >> __M_writer(unicode( h.dumps( self.get_grid_config( embedded=embedded, >>insert=insert ) ) )) >> File '/galaxy-central/database/compiled_templates/grid_base.mako.py', >>line 303 in render_get_grid_config >> value = column.get_value( trans, grid, item ) >> File 'lib/tool_shed/galaxy_install/grids/admin_toolshed_grids.py', line >>106 in get_value >> return suc.get_tool_shed_repository_status_label( trans.app, >>tool_shed_repository ) >> File 'lib/tool_shed/util/shed_util_common.py', line 829 in >>get_tool_shed_repository_status_label >> elif tool_shed_repository.tool_dependencies_being_installed: >> File 'lib/galaxy/model/tool_shed_install/__init__.py', line 408 in >>tool_dependencies_being_installed >> for tool_dependency in self.tool_dependencies: >> File >>'/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalc >>hemy/orm/attributes.py', line 168 in __get__ >> File >>'/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalc >>hemy/orm/attributes.py', line 453 in get >> File >>'/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalc >>hemy/orm/strategies.py', line 508 in _load_for_state >> File >>'/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalc >>hemy/orm/strategies.py', line 574 in _emit_lazyload >> File >>'/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalc >>hemy/orm/query.py', line 2115 in all >> File >>'/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalc >>hemy/orm/query.py', line 2341 in instances >> File >>'/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalc >>hemy/engine/base.py', line 3204 in fetchall >> File >>'/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalc >>hemy/engine/base.py', line 3171 in _fetchall_impl >> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position >>393: ordinal not in range(128) >> ___________________________________________________________ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> http://lists.bx.psu.edu/ >> >> To search Galaxy mailing lists use the unified search at: >> http://galaxyproject.org/search/mailinglists/ > > >------------------------------ > >_______________________________________________ >galaxy-dev mailing list >galaxy-dev@lists.bx.psu.edu >http://lists.bx.psu.edu/listinfo/galaxy-dev > >To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/ > >End of galaxy-dev Digest, Vol 100, Issue 15 >******************************************* The information in this email, including attachments, may be confidential and is intended solely for the addressee(s). If you believe you received this email by mistake, please notify the sender by return email as soon as possible. ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
