I believe I've fixed this in 16583:5b2f1eab2819, if you want to give it
another shot.  External version of pkg_resources (and not ours in lib/) is
what was causing the weird egg fetching errors.  Newer versions of
pkg_resources create a mangled distribution string for some eggs with
nonstandard version identifiers.

On Thu Feb 12 2015 at 6:44:29 AM Nicola Soranzo <nsora...@tiscali.it> wrote:

>  Hi Stephen and Nate,
> it seems that the problem with the WebError egg is now hitting also
> BioBlend TravisCI testing:
> https://travis-ci.org/galaxyproject/bioblend/builds/50391748
> Both for Python 2.6 and 2.7, scripts/common_startup.sh fails with:
> "WebError 0.8a couldn't be downloaded automatically. You can try
> building it by hand with:
> python scripts/scramble.py -c config/galaxy.ini.sample -e WebError
> Fetch failed."
> Please note that this has been working fine for 6 months (last TravisCI
> build 19 days ago) and is still working on my laptops.
> Nicola
> Il 08.02.2015 04:35 Stephen Rosen ha scritto:
>  It took me a couple of extra days to get the time to test this, sorry
> about the delay.
> The initial run:
> WebError 0.8a couldn't be downloaded automatically.  You can try
> building it by hand with:
>   python scripts/scramble.py -e WebError
> Fetch failed.
> When I patch a traceback.print_exc() into fetch_eggs.py , this is what I
> see:
> Some eggs are out of date, attempting to fetch...
> Warning: sqlalchemy (a dependent egg of sqlalchemy-migrate) cannot be
> fetched
> Traceback (most recent call last):
>   File "./scripts/fetch_eggs.py", line 36, in
>     c.resolve() # Only fetch eggs required by the config
>   File "/opt/galaxy/lib/galaxy/eggs/__init__.py", line 349, in resolve
>     raise EggNotFetchable( missing )
> EggNotFetchable:
> WebError 0.8a couldn't be downloaded automatically.  You can try
> building it by hand with:
>   python scripts/scramble.py -e WebError
> Fetch failed.
> If I look into lib/galaxy/eggs , that line is here:
> https://bitbucket.org/galaxycloud/galaxy/src/e7f887cdf229aaf3b44977ecbf24130ddc8467e3/lib/galaxy/eggs/__init__.py?at=default#cl-349
> In the current galaxy source, that is here:
> https://bitbucket.org/galaxy/galaxy-central/src/1b09e60f84db859ae49bed71c6f3fd838af612f5/lib/galaxy/eggs/__init__.py?at=default#cl-351
> As a bit of an aside, I really do have to install this other package in
> the same python that galaxy uses, since it's one of the major parts of our
> fork -- not part of our custom tools.
> I wouldn't consider this a normal use case of course, but I'll bet that
> many people will install galaxy with system python, which can lead to
> similar circumstances.
> Please let me know if I can provide any other useful information on what
> I'm seeing.
> Again, thanks for looking into this with me,
> -Stephen
> On Thu, Feb 5, 2015 at 8:19 AM, Nate Coraor <n...@bx.psu.edu> wrote:
>> Hi Stephen,
>> On Wed, Feb 4, 2015 at 8:30 PM, Stephen Rosen <siro...@uchicago.edu>
>> wrote:
>>>    Hi Nate,
>>> Thanks so much for helping me out with this.
>>> It seems that I've miscommunicated what I'm doing a little bit.
>>> I'm installing a package from that git repo using pip separately from
>>> running the galaxy setup script.
>>> That is, I want to install a python package and I want to setup galaxy
>>> as two separate (and ideally independent) steps in my script.
>>> I have no desire to do anything particularly esoteric or clever with
>>> Galaxy's eggs or to install them myself.
>>> Somehow, doing a pip install of my desired package is breaking the later
>>> run of fetch_eggs.py
>>> If I understand you correctly, fetch_eggs.py can hit conflicts with
>>> site-packages, but is misreporting them as EggNotFetchable.
>>> I am aware that we're using a fork of the Galaxy source (
>>> https://bitbucket.org/galaxycloud/galaxy/src ), so it may be that we
>>> have an outdated of fetch_eggs.py which carries a bug that has since been
>>> patched.
>>> Sorry for not mentioning it earlier -- it slipped my mind that we're
>>> probably trailing behind the modern Galaxy default head.
>>> If I put in the extra legwork now to wrap Galaxy in a virtualenv, this
>>> issue will presumably disappear, but that assumes that I don't want to add
>>> any packages to the python used for the uwsgi process.
>> You can add additional packages to this virtualenv, although there could
>> be problems if you install different versions of the same packages that
>> Galaxy has as dependencies. For what it's worth, you should not need to
>> install additional packages in Galaxy's virtualenv, and tools can be
>> configured to use a different python or virtualenv than the one in which
>> Galaxy is started.
>>>  I'll have to ask folks on the Globus Genomics team whether or not that
>>> is needed -- I certainly hope not, since it means that there is some
>>> packaging conflict we need to resolve.
>>> The only reason I haven't done so already is time constraint -- I've
>>> been trying to get this install scripting done as quickly as possible
>>> without compromising anything truly needful.
>>> All the same, I'll take some time tomorrow to provision a fresh server
>>> and generate the stacktrace for you.
>>> I'll also take some time to test with the latest Galaxy source, to see
>>> if I get different behavior.
>>> In the best case, this bug no longer exists for the modern source, but
>>> in the worst case it bears the attention.
>> Thanks for the help with debugging.
>>>  I don't have any sense of the reliability of fetch_eggs.py and company
>>> -- my experience has been of a particular bug that presented on my first
>>> contact with these scripts, so naturally I have a bias against them.
>>> That said, I think the Python community may have finally settled on a
>>> package manager, even if we can't seem to agree on a package format or
>>> tooling surrounding it.
>>> That's just an opinion though -- I have no blog post from Guido to back
>>> it up.
>> I hope we'll see the necessary improvements within the next year, but I
>> think anything we do now will essentially reinvent the wheel (no pun
>> intended) with regards to what will (hopefully) be officially implemented
>> to fix the problem.
>> --nate
>>>  I was not aware of the UCS2 vs. UCS4 issue -- thanks very much for the
>>> citations, very helpful in understanding the problem space.
>>> Thanks,
>>> -Stephen
>>> On Wed, Feb 4, 2015 at 5:41 PM, Nate Coraor <n...@bx.psu.edu> wrote:
>>>> Hi Stephen,
>>>> I'll try to reply as in-depth as I can.
>>>> On Wed, Feb 4, 2015 at 1:41 PM, Stephen Rosen <siro...@uchicago.edu>
>>>> wrote:
>>>>>       Hi Galaxy Dev,
>>>>> I've been looking at the setup scripts for Galaxy to try to understand
>>>>> a problem I recently had provisioning a Galaxy server.
>>>>> I will readily admit that I have not read all of the relevant code
>>>>> top-to-bottom, but I have at least skimmed all of it and read much of it.
>>>>> Sorry if these questions are answered somewhere in Trello, the Wiki,
>>>>> or somewhere else, but I was not able to find answers in any public
>>>>> locations.
>>>>> As a small bit of probably irrelevant context:
>>>>> I'm working with the Globus Genomics group on the DevOps side of
>>>>> things.
>>>>> We're using Chef.
>>>>> I've only just started working with the group in the past couple of
>>>>> weeks (so my expertise with Galaxy itself is limited to nonexistent).
>>>>> First, to describe the problem:
>>>>> We want to provision a server running Galaxy without explicitly
>>>>> wrapping it in a virtualenv.
>>>>> Unless I missed something, that means that it's using system python.
>>>>> When we use pip to install a package from a git repository before
>>>>> running the setup scripts, fetch_eggs fails saying it failed to fetch
>>>>> WebError 0.8a
>>>>> If we install the same package from git with `pip install --egg ...`
>>>>> we get a hunky-dory system where everything seems to work.
>>>> A virtualenv is itself just a wrapper around whichever python binary
>>>> was used to create it. I'd still suggest using a virtualenv created with
>>>> the system python unless you have a really strong reason not to. In fact,
>>>> I'm working on Galaxy process management and a command line tool for that
>>>> management that will automatically create and use a virtualenv going
>>>> forward.
>>>> I'm a bit confused at what's happening here - you mention installing a
>>>> package from a git repository with pip, but then refer to Galaxy's
>>>> fetch_eggs(.py) script, which doesn't use pip or git.
>>>>>    As far as I can tell, there is no reason that this should be the
>>>>> case.
>>>>>  Sure, putting the git source directly into site-packages might cause
>>>>> issues upon installation, but EggNotFetchable exceptions should only be
>>>>> thrown if the egg actually can't be pulled down from
>>>>> eggs.g2.bx.psu.edu , right?
>>>> EggNotFetchable can be thrown if you happen to be using a platform for
>>>> which we do not provide eggs, although those are fairly uncommon. Right now
>>>> we should cover x86/x86_64 Linux and any flavor of Intel OS X after 10.5.
>>>>>   I don't feel comfortable trying to make further progress on my
>>>>> provisioning scripts without knowing why this is happening.
>>>>> I'd hate to be bitten by this later on in the process.
>>>>> Yes, the package in question may have poor behavior (likely it does),
>>>>> but that doesn't change the fact that the error is totally misleading.
>>>>> Furthermore, it doesn't appear that this poor behavior impedes me from
>>>>> doing a pip install of the WebError package or any other packages from 
>>>>> PyPI.
>>>>> In case someone else wants to test to replicate, this is the command
>>>>> being used:
>>>>>   pip install --egg git+
>>>>> https://github.com/globusonline/python-nexus-client@599f04edef6b72569b7a5b272b0b847dcda3ea99#egg=nexus-client
>>>>> problems occur if you omit `--egg`.
>>>> None of this process is using Galaxy's egg handling, so I am not sure
>>>> where the EggNotFetchable is coming from. What command are you running to
>>>> get an EggNotFetchable error.
>>>>>   Second, a question about the rationale for Galaxy's egg handling:
>>>>> Why is all of this wrapped up in these scripts in the first place?
>>>>> I understand that pip might not be present on every platform, and I
>>>>> don't mean to question a decision to support systems without it.
>>>>> However, as detailed below, Galaxy does not support any platforms
>>>>> which are incapable of running pip.
>>>> This isn't the case - Galaxy does not use pip to install the framework
>>>> dependencies at all. Some tool dependencies installed from the Tool Shed do
>>>> use pip, but that's entirely separate from the dependencies of the Galaxy
>>>> application.
>>>> The `scripts/scramble.py` script can be used to automatically build
>>>> eggs on platforms which we do not prebuild eggs for. If this is necessary,
>>>> `scripts/fetch_eggs.py` should tell you.
>>>>>  Furthermore, pip is being pushed by the Python maintainers over
>>>>> easy_install, so it's not like there isn't a clear choice in terms of 
>>>>> which
>>>>> one to support.
>>>>> Perhaps most importantly, there don't appear to be any clear-cut
>>>>> options to do the following, which I would consider a more ordinary
>>>>> workflow:
>>>>> - Run a galaxy script (like check_eggs) to generate a list of packages
>>>>> from eggs.g2.bx.psu.edu for platform (redirect output to
>>>>> requirements.txt or similar)
>>>>> - `pip install -r requirements.txt`
>>>> This is exactly what `scripts/check_eggs.py` and
>>>> `scripts/fetch_eggs.py` do.
>>>> There are 3 reasons for the way we handle eggs in Galaxy:
>>>> 1. Galaxy has a huge (and ever-growing) list of dependent python
>>>> modules with C extensions. If we did not prebuild and distribute eggs for
>>>> these, the initial setup to get Galaxy running would be long and
>>>> problematic. Some people who download Galaxy to develop tools may not even
>>>> have compilers installed, let alone the multitude of -dev or -devel
>>>> packages that aren't part of a default Debian or RHEL installation that
>>>> would be required to build all of these packages from source. One of the
>>>> things that I feel makes Galaxy so accessible is that you can start using
>>>> it immediately after you clone the source. So that ability to clone and
>>>> start and have it work as reliably (and quickly) as possible is a high
>>>> priority.
>>>> 2. Galaxy started using eggs in 2005 or 2006. At this time, everything
>>>> used distutils. pkg_resources came around, which soon brought setuptools
>>>> and easy_install. After this came distutils2, pip and finally, these days,
>>>> wheels. Our need for binary dependency packaging predated almost all of
>>>> these (in fact, most packages in these days didn't even install .egg-info,
>>>> which was the only reliable way to know what version of a module you were
>>>> using) and as each new iteration of packaging/management came along it was
>>>> never clear that any of them had "won" (and in fact, most of them lost). On
>>>> top of this, the Python packaging folks have known for years that Python's
>>>> platform detection for binary compatibility is broken[1]. While I was
>>>> assured it'd be fixed soon, even with a complete reimplementation of Python
>>>> packaging (wheels), they still haven't even made an effort to fix this
>>>> problem[2]. In fact, binary wheels for Linux are explicitly not allowed on
>>>> PyPI because of this.
>>>> 3. We tightly control the versions of all of our dependencies, which is
>>>> not always possible with pip if you aren't also controlling the source of
>>>> your packages.
>>>> [1]
>>>> https://mail.python.org/pipermail/distutils-sig/2010-January/015345.html
>>>> [2] http://lucumr.pocoo.org/2014/1/27/python-on-wheels/
>>>>>  The above could be part of a galaxy provisioning script, rather than
>>>>> exposed to the administrator.
>>>>> That also makes it significantly easier to control and manage the
>>>>> virtualenv in which we run Galaxy, since we don't have to worry about
>>>>> egg-related logic that we don't control and we know that the virtualenv's
>>>>> bin dir will be earlier in the PATH than the system pip's dir.
>>>>> Yes, I said above that our setup is presently using system python --
>>>>> switching to a virtualenv is one of the many items on my to-do list.
>>>>> In fact, I would expect that the default, desired setup for Galaxy
>>>>> would be to put it in a virtualenv, rather than using system python, and 
>>>>> to
>>>>> use pip, rather than fetch_eggs.py and company.
>>>> A virtualenv is indeed the strongly preferred setup as I mentioned
>>>> above. However, Galaxy does not install its eggs to the virtualenv. The
>>>> virtualenv is there to avoid conflicts with things in the default python's
>>>> site-packages/dist-packages. Galaxy's eggs are installed to (by default)
>>>> the `eggs/` directory in the Galaxy source.
>>>> However, the problem, as I now see it, is that you are trying to
>>>> install all of Galaxy's dependencies, even at their correct versions, using
>>>> pip, rather than letting Galaxy handle its eggs as it does. This is not
>>>> going to work, Galaxy is going to insist on using its eggs.
>>>>>  When I look at the logic being used here, especially at
>>>>> https://bitbucket.org/galaxy/galaxy-central/src/f0ae870b22e9/lib/galaxy/eggs/?at=default
>>>>> , it looks like a solution built exclusively for platforms on which pip is
>>>>> not installed.
>>>>> According to the wiki, Galaxy support only goes back as far as 2.6,
>>>>> and get-pip.py supports 2.6, so there is no way of building a Galaxy 
>>>>> server
>>>>> on a platform that can't also have pip installed.
>>>>> Adding pip to the requirements for Galaxy would not be particularly
>>>>> onerous, and may simplify things significantly (no need to bundle
>>>>> get-pip.py or similar).
>>>> As mentioned above, we don't use or depend on pip, so it's not
>>>> required. That said, a lot of our egg fetching logic could likely be
>>>> replaced with pip (this code predates pip by a few years). And the eggs
>>>> could probably be replaced with prebuilt wheels. However, even if we did
>>>> use pip/wheels, we'd need to install them from our own repository, and it'd
>>>> still require modifications for binary platform incompatibilities. The egg
>>>> handling code we have now works pretty reliably, so I am not sure there is
>>>> a whole lot to be gained by changing it until Python finally figures out
>>>> how to handle binary compatibility properly.
>>>>>  As a last note about the misleading error from the fetch_eggs
>>>>> script, telling me that WebError is "NotFetchable".
>>>>> I probably wouldn't have much of a complaint about this if the error
>>>>> had been more on target and I hadn't felt the need to do things like patch
>>>>> lib/galaxy/eggs to print a stacktrace.
>>>>> For example, if the script detected that there was a source installed
>>>>> package which was getting underfoot, it should have alerted me or even
>>>>> suggested installing my various packages in egg format.
>>>> This is a bug - it is supposed to explain that there is a version
>>>> conflict. If you have a stack trace, please send it along.
>>>>>  That kind of error detection is hard to maintain and hard to keep
>>>>> accurate since the Galaxy team's priority is to build Galaxy, not a 
>>>>> package
>>>>> manager.
>>>>> This, I suppose, circles back to an earlier question: why isn't Galaxy
>>>>> using a python package manager to... manage its packages?
>>>>> At the very least something along the lines of
>>>>> `./scripts/common_setup.sh --use-pip` should be added.
>>>>> It doesn't seem like it would be that hard to implement -- but I feel
>>>>> like my lack of knowledge of Galaxy disqualifies me from building the
>>>>> changeset reliably.
>>>>> Of course, if no one objects, I will readily do so anyway (when I have
>>>>> the time) as a proof of concept.
>>>> There are modifications to a few of the dependencies, such as psycopg2.
>>>> When psycopg2 is scrambled, the scrambling process fetches and compiles a
>>>> bit of PostgreSQL's libpq and then statically links to it to provide a
>>>> standalone egg that does not depend on the user having installed libpq on
>>>> their system. So a pip-based system that allowed building from source would
>>>> need to account for this.
>>>> --nate
>>>>>  Barring even that improvement, the Wiki page at
>>>>> https://wiki.galaxyproject.org/Admin/Config/Eggs should definitely be
>>>>> updated to include some note on why Galaxy has this complex logic for
>>>>> fetching eggs instead of using pip.
>>>>> A quick tl;dr and summary:
>>>>> - I don't have extensive experience with Galaxy, so I may not know
>>>>> what I'm talking about.
>>>>> - fetch_eggs.py can be made to raise EggNotFetchable by doing a pip
>>>>> install from a VCS without using `--egg`. This is a bug in fetch_eggs.py
>>>>> - All supported platforms for Galaxy support pip
>>>>> - Galaxy should have an option to use pip to download its packages
>>>>> over https from eggs.g2.bx.psu.edu
>>>>> - Galaxy should probably default to using pip when it's available,
>>>>> since its failure modes are significantly better than a home-brewed 
>>>>> package
>>>>> manager -- this also leads to good behavior in virtualenvs.
>>>>> Best regards, and many thanks for your time and attention,
>>>>> -Stephen
>>>>>  ___________________________________________________________
>>>>> Please keep all replies on the list by using "reply all"
>>>>> in your mail client.  To manage your subscriptions to this
>>>>> and other Galaxy lists, please use the interface at:
>>>>>   https://lists.galaxyproject.org/
>>>>> To search Galaxy mailing lists use the unified search at:
>>>>>   http://galaxyproject.org/search/mailinglists/
> Connetti gratis il mondo con la nuova indoona: hai la chat, le chiamate,
> le video chiamate e persino le chiamate di gruppo.
> E chiami gratis anche i numeri fissi e mobili nel mondo!
> Scarica subito l’app Vai su https://www.indoona.com/
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

To search Galaxy mailing lists use the unified search at:

Reply via email to