Re: [galaxy-dev] [galaxy-user] Galaxy does not find my executables
On 21/02/2011 16:20, Nate Coraor wrote: Marina Gourtovaia wrote: Hi In a bash shell, I define the path (needed both by Galaxy to find the right version of python and by tools that run on a cluster to find the executables) and some other global variables on the command line. The cluster jobs (LSF) inherit all these values. This is my line DRMAA_LIBRARY_PATH=/usr/local/lsf/7.0/linux2.6-glibc2.3-x86_64/lib/libdrmaa.so.1.0.4 PATH=/usr/bin:/software/solexa/bin:$PATH http_proxy="http://somecache:3128"; sh run.sh Is there anything inherently wrong with this simple solution? No, this is effectively what we do to start the server. The only difference is that when using PBS the environment is not passed through to the cluster. Is that a feature specific to LSF? --nate This is a feature of LSF, a very useful one. An LSF job inherits the environment of the user who started it and knows where the home directory of this user is. We run our pipelines as a special user, and I am going to run the production Galaxy server as another special user. I do not know whether LSF is configured like this at Sanger or this is is generic feature Marina -- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. ___ To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] [galaxy-user] Galaxy does not find my executables
Marina Gourtovaia wrote: > Hi > > In a bash shell, I define the path (needed both by Galaxy to find > the right version of python and by tools that run on a cluster to > find the executables) and some other global variables on the > command line. The cluster jobs (LSF) inherit all these values. This > is my line > > > DRMAA_LIBRARY_PATH=/usr/local/lsf/7.0/linux2.6-glibc2.3-x86_64/lib/libdrmaa.so.1.0.4 > PATH=/usr/bin:/software/solexa/bin:$PATH http_proxy="http://somecache:3128"; > sh run.sh > > Is there anything inherently wrong with this simple solution? No, this is effectively what we do to start the server. The only difference is that when using PBS the environment is not passed through to the cluster. Is that a feature specific to LSF? --nate > > Marina > > > > > -- > The Wellcome Trust Sanger Institute is operated by Genome Research > Limited, a charity registered in England with number 1021457 and a > company registered in England with number 2742969, whose registered > office is 215 Euston Road, London, NW1 2BE. > ___ > To manage your subscriptions to this and other Galaxy lists, please use the > interface at: > > http://lists.bx.psu.edu/ ___ To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] [galaxy-user] Galaxy does not find my executables
Hi In a bash shell, I define the path (needed both by Galaxy to find the right version of python and by tools that run on a cluster to find the executables) and some other global variables on the command line. The cluster jobs (LSF) inherit all these values. This is my line DRMAA_LIBRARY_PATH=/usr/local/lsf/7.0/linux2.6-glibc2.3-x86_64/lib/libdrmaa.so.1.0.4 PATH=/usr/bin:/software/solexa/bin:$PATH http_proxy="http://somecache:3128"; sh run.sh Is there anything inherently wrong with this simple solution? Marina -- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. ___ To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] [galaxy-user] Galaxy does not find my executables
Florent Angly wrote: > I understand that it is best to flow with the logic of the > underlying OS and not too make too many platform-dependent > assumptions about what files should be read. > > Perhaps, it seems like the best way to go would be to have a setting > in universe_wsgi.ini to specify a file to read additional > environment settings from? For example, in my case, I would add a > line like: > source_environment_from = '/Users/galaxy/.profile' I'd rather not read from the config file over on the cluster when we're just running basic tools, but how about a default of '~/.galaxy_env' or '/path/to/galaxy-dist/env.sh', and can be overridden with the file pointed to by '$GALAXY_ENV'? I haven't fully thought this out so there may be reasons why this won't work in cluster environments, but it should be something we can add to the job template. --nate > > Best, > > Florent > > > On 16/02/11 05:58, Nate Coraor wrote: > >Hi Florent, et. al., > > > >I've moved the discussion over to galaxy-dev since it pertains to local > >instances. > > > >Florent Angly wrote: > >>Indeed, this workaround works! Thank you. > >>I added these lines to the beginning of my run.sh: > >> PATH="/opt/local/bin:/opt/local/sbin:$PATH" > >> export $PATH > >> > >>Now, Galaxy can find Grinder, Velvet, etc > >> > >>Could Galaxy detect and use ~/.profile automatically if it exists? > >>Alternatively, maybe there should be a place in the configuration > >>file universe_wsgi.ini to specify which file to read environment > >>variables from? Is it worth filing an enhancement request in the bug > >>tracker? > >Unfortunately, this is really more of an issue with the way shells are > >designed. Galaxy can't make too many assumptions about how you want the > >environment to be set up, especially since it's run under a variety of > >shells. What's going to happen is dependent entirely upon how the > >parent shell to Galaxy is started. > > > >By default if invoked in non-login, non-interactive mode (which would > >likely be the case for most managed startup methods like launchd, rc > >scripts, etc.), bash will not read *any* startup files. This behavior > >can be changed by specifying the path to a file to be sourced in the > >$BASH_ENV environment variable in the calling environment. > > > >However, when bash is called as 'sh', even $BASH_ENV is not available. > > > >For more details, see the "INVOCATION" section of the bash(1) man page. > > > >Since the public site runs on Solaris, we solve this by setting $PATH > >and any other relevant variables within SMF. Similar methods should > >work (setting the $PATH in an init script or in the LaunchDaemon plist) > >for other startup methods. > > > >Although modifying run.sh as mentioned does work, modifications could > >conflict with future changes from the upstream source repository. > > > >--nate > > > >P.S. I'll add a plug for zsh - its invocation is much less confusing > > and ~/.zshenv is *always* read regardless of invocation. > > > >>Again, thanks for the help. > >> > >>Florent > ___ To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] [galaxy-user] Galaxy does not find my executables
I understand that it is best to flow with the logic of the underlying OS and not too make too many platform-dependent assumptions about what files should be read. Perhaps, it seems like the best way to go would be to have a setting in universe_wsgi.ini to specify a file to read additional environment settings from? For example, in my case, I would add a line like: source_environment_from = '/Users/galaxy/.profile' Best, Florent On 16/02/11 05:58, Nate Coraor wrote: Hi Florent, et. al., I've moved the discussion over to galaxy-dev since it pertains to local instances. Florent Angly wrote: Indeed, this workaround works! Thank you. I added these lines to the beginning of my run.sh: PATH="/opt/local/bin:/opt/local/sbin:$PATH" export $PATH Now, Galaxy can find Grinder, Velvet, etc Could Galaxy detect and use ~/.profile automatically if it exists? Alternatively, maybe there should be a place in the configuration file universe_wsgi.ini to specify which file to read environment variables from? Is it worth filing an enhancement request in the bug tracker? Unfortunately, this is really more of an issue with the way shells are designed. Galaxy can't make too many assumptions about how you want the environment to be set up, especially since it's run under a variety of shells. What's going to happen is dependent entirely upon how the parent shell to Galaxy is started. By default if invoked in non-login, non-interactive mode (which would likely be the case for most managed startup methods like launchd, rc scripts, etc.), bash will not read *any* startup files. This behavior can be changed by specifying the path to a file to be sourced in the $BASH_ENV environment variable in the calling environment. However, when bash is called as 'sh', even $BASH_ENV is not available. For more details, see the "INVOCATION" section of the bash(1) man page. Since the public site runs on Solaris, we solve this by setting $PATH and any other relevant variables within SMF. Similar methods should work (setting the $PATH in an init script or in the LaunchDaemon plist) for other startup methods. Although modifying run.sh as mentioned does work, modifications could conflict with future changes from the upstream source repository. --nate P.S. I'll add a plug for zsh - its invocation is much less confusing and ~/.zshenv is *always* read regardless of invocation. Again, thanks for the help. Florent ___ To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] [galaxy-user] Galaxy does not find my executables
Hi Florent, et. al., I've moved the discussion over to galaxy-dev since it pertains to local instances. Florent Angly wrote: > > Indeed, this workaround works! Thank you. > I added these lines to the beginning of my run.sh: > PATH="/opt/local/bin:/opt/local/sbin:$PATH" > export $PATH > > Now, Galaxy can find Grinder, Velvet, etc > > Could Galaxy detect and use ~/.profile automatically if it exists? > Alternatively, maybe there should be a place in the configuration > file universe_wsgi.ini to specify which file to read environment > variables from? Is it worth filing an enhancement request in the bug > tracker? Unfortunately, this is really more of an issue with the way shells are designed. Galaxy can't make too many assumptions about how you want the environment to be set up, especially since it's run under a variety of shells. What's going to happen is dependent entirely upon how the parent shell to Galaxy is started. By default if invoked in non-login, non-interactive mode (which would likely be the case for most managed startup methods like launchd, rc scripts, etc.), bash will not read *any* startup files. This behavior can be changed by specifying the path to a file to be sourced in the $BASH_ENV environment variable in the calling environment. However, when bash is called as 'sh', even $BASH_ENV is not available. For more details, see the "INVOCATION" section of the bash(1) man page. Since the public site runs on Solaris, we solve this by setting $PATH and any other relevant variables within SMF. Similar methods should work (setting the $PATH in an init script or in the LaunchDaemon plist) for other startup methods. Although modifying run.sh as mentioned does work, modifications could conflict with future changes from the upstream source repository. --nate P.S. I'll add a plug for zsh - its invocation is much less confusing and ~/.zshenv is *always* read regardless of invocation. > > Again, thanks for the help. > > Florent ___ To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
