Well, I don't seem to be able to easy_install pycurl in the first place, so I'm no longer convinced that just mirroring urlgrabber is sufficient.
Maybe this is a Fedora-specific issue, but does anyone who's been able to manually install pycurl so far have some pointers on troubleshooting? $ easy_install pycurl Runs without error, but afterwards trying to install urlgrabber complains that no module named pycurl can be found. Interestingly, pycurl only produces a curl/ module in my site-packages; it looks like pycurl is the name of the relevant C module. There's a pypi index with just pycurl and urlgrabber at http://dev.geonode.org/pypi/simple/ if you would like to try it out yourself. -- David Winslow OpenGeo - http://opengeo.org/ On Mon, Nov 1, 2010 at 3:28 PM, Ariel Nunez <[email protected]>wrote: > > On Nov 1, 2010, at 14:16, David Winslow <[email protected]> wrote: > > I was really asking if you knew a better way to handle the problem of > getting stuff into the repository in the first place. > > > Gotcha, in that case i think using the --fetch-site-packages option in > compoze looks like the right place to start. I know there are many projects > to create PyPi mirrors but have used none of them, Compoze looks good enough > to me. > > btw, for whatever reason the place where urlgrabber is failing is using > easy_install to get stuff, so I don't think pip's -f flag is an option > there. > > -d > > On Mon, Nov 1, 2010 at 3:12 PM, Ariel Nunez < <[email protected]> > [email protected]> wrote: > >> Quoting Simon Willinson: >> >> At work, the first line in our requirements.txt file is "-f >> <http://pypi.internal/%22><http://pypi.internal/> >> http://pypi.internal/"; - which causes pip to use our own internal PyPI >> mirror (just an Apache directory listing which pip/setuptools is smart >> enough to scrape). >> I dig that idea of just using apache and throwing .tar.gz files there. >> >> Sent from my i >> On Nov 1, 2010, at 13:52, David Winslow < <[email protected]> >> [email protected]> wrote: >> >> Looking into the "mirror pypi" idea, it seems the >> Compoze<http://docs.repoze.org/compoze/>project from Repoze is a pretty >> simple way to set up a Pypi-compatible >> package index with packages from Pypi or elsewhere. I don't think it will >> automatically read our pip requirements files, but for now I'll just it to >> just set up a repo with these problem packages. >> >> Does anyone know a better tool for this? (Ariel??) Otherwise I'll take a >> look later this week into making a complete mirror with all the Python >> packages we use to prevent this sort of problem from cropping up again. >> >> -- >> David Winslow >> OpenGeo - <http://opengeo.org/> <http://opengeo.org/>http://opengeo.org/ >> >> On Fri, Oct 29, 2010 at 9:20 AM, Ariel Nunez < >> <[email protected]><[email protected]> >> [email protected]> wrote: >> >>> I had to do the same for my install yesterday, in my case it was >>> because the curl version I had was 1.15 and the latest pycurl required >>> 1.17. I had to first install a new curl and then run (inside the >>> venv): >>> >>> pip install pycurl >>> pip install >>> <http://urlgrabber.baseurl.org/download/urlgrabber-3.9.1.tar.gz><http://urlgrabber.baseurl.org/download/urlgrabber-3.9.1.tar.gz> >>> http://urlgrabber.baseurl.org/download/urlgrabber-3.9.1.tar.gz >>> >>> I fully support the idea of setting up our own PyPi mirror, hopefully >>> avoiding the rewrite of the build script. >>> >>> Best, >>> Ariel. >>> >>> >>> On Fri, Oct 29, 2010 at 8:02 AM, David Winslow < >>> <[email protected]><[email protected]> >>> [email protected]> wrote: >>> > I haven't looked very hard but I can't find any information about the >>> > removal of urlgrabber from the python package repository. I think it >>> would >>> > probably be a good idea for us to start mirroring our dependencies in a >>> > repository of our own to avoid this sort of thing. >>> > In the shorter term I would be open to rewriting the build script to >>> avoid >>> > requiring urlgrabber. >>> > -- >>> > David Winslow >>> > OpenGeo - <http://opengeo.org/> <http://opengeo.org/> >>> http://opengeo.org/ >>> > On Fri, Oct 29, 2010 at 7:56 AM, Christian Spanring >>> > <<[email protected]><[email protected]> >>> [email protected]> >>> > wrote: >>> >> >>> >> Ole, >>> >> >>> >> the problem is that easy_install can't find urlgrabber at >>> <http://pypi.python.org> <http://pypi.python.org>pypi.python.org >>> >> anymore, so you have to install urlgrabber manually as workaround: >>> >> >>> >> download and unpack it from >>> >> <http://urlgrabber.baseurl.org/><http://urlgrabber.baseurl.org/> >>> http://urlgrabber.baseurl.org/ >>> >> >>> >> $ wget >>> <http://urlgrabber.baseurl.org/download/urlgrabber-3.9.1.tar.gz><http://urlgrabber.baseurl.org/download/urlgrabber-3.9.1.tar.gz> >>> http://urlgrabber.baseurl.org/download/urlgrabber-3.9.1.tar.gz >>> >> $ tar xvzf urlgrabber-3.9.1.tar.gz >>> >> >>> >> install urlgrabber in GeoNode's Virtual Environment: >>> >> >>> >> $ . bin/activate >>> >> (geonode) $ cd urlgrabber-3.9.1 >>> >> (geonode) $ python setup.py install >>> >> >>> >> in my case it asked for pycurl, which I had to install first >>> >> (geonode) $ easy_install pycurl >>> >> >>> >> That should to it. >>> >> >>> >> Christian >>> >> >>> >> On Oct 29, 2010, at 1:54 AM, Ole Nielsen wrote: >>> >> >>> >> > Dear all >>> >> > I am trying to install GeoNode according to the instructions at >>> >> > <http://github.com/GeoNode/geonode/blob/1.0-RC2/README.rst><http://github.com/GeoNode/geonode/blob/1.0-RC2/README.rst> >>> http://github.com/GeoNode/geonode/blob/1.0-RC2/README.rst and have >>> gotten as >>> >> > far as paver build. >>> >> > >>> >> > However, paver build fails with the following error: ImportError: No >>> >> > module named urlgrabber.grabber >>> >> > However, if I start Python normally (i.e. without virtual >>> environment) >>> >> > and try it works just fine: >>> >> > >>> >> > Python 2.6.2 (release26-maint, Apr 19 2009, 01:58:18) >>> >> > [GCC 4.3.3] on linux2 >>> >> > Type "help", "copyright", "credits" or "license" for more >>> information. >>> >> > >>> from urlgrabber.grabber import urlgrab, URLGrabError >>> >> > >>> >>> >> > >>> >> > I am running Ubuntu 9.04 - 64 bit. >>> >> > >>> >> > The tail of the paver output follows below. >>> >> > Cheers and thanks for any help >>> >> > Ole Nielsen >>> >> > >>> >> > -- output from paver build -- >>> >> > >>> >> > ########## SUMMARY INFO ######### >>> >> > ################################################ >>> >> > #Attempting install of _rl_accel, sgmlop & pyHnj >>> >> > #extensions from '/tmp/pip-0Buy2p-build/src/rl_addons/rl_accel' >>> >> > ################################################ >>> >> > ################################################ >>> >> > #Attempting install of _renderPM >>> >> > #extensions from '/tmp/pip-0Buy2p-build/src/rl_addons/renderPM' >>> >> > # installing with freetype version 21 >>> >> > ################################################ >>> >> > Successfully installed avatar gsconfig.py OWSLib reportlab >>> >> > ---> pavement.setup_webapps >>> >> > ---> pavement.setup_geoserver >>> >> > ---> pavement.setup_gs_data >>> >> > Traceback (most recent call last): >>> >> > File "/home/nielso/sandpit/geonode/bin/paver", line 8, in <module> >>> >> > load_entry_point('Paver==1.0.1', 'console_scripts', 'paver')() >>> >> > File >>> >> > >>> "/home/nielso/sandpit/geonode/lib/python2.6/site-packages/Paver-1.0.1-py2.6.egg/paver/tasks.py", >>> >> > line 615, in main >>> >> > _launch_pavement(args) >>> >> > File >>> >> > >>> "/home/nielso/sandpit/geonode/lib/python2.6/site-packages/Paver-1.0.1-py2.6.egg/paver/tasks.py", >>> >> > line 598, in _launch_pavement >>> >> > _process_commands(args, auto_pending=auto_pending) >>> >> > File >>> >> > >>> "/home/nielso/sandpit/geonode/lib/python2.6/site-packages/Paver-1.0.1-py2.6.egg/paver/tasks.py", >>> >> > line 563, in _process_commands >>> >> > task() >>> >> > File >>> >> > >>> "/home/nielso/sandpit/geonode/lib/python2.6/site-packages/Paver-1.0.1-py2.6.egg/paver/tasks.py", >>> >> > line 241, in __call__ >>> >> > retval = environment._run_task( >>> >> > <http://self.name><http://self.name> >>> self.name, self.needs, self.func) >>> >> > File >>> >> > >>> "/home/nielso/sandpit/geonode/lib/python2.6/site-packages/Paver-1.0.1-py2.6.egg/paver/tasks.py", >>> >> > line 181, in _run_task >>> >> > return do_task() >>> >> > File >>> >> > >>> "/home/nielso/sandpit/geonode/lib/python2.6/site-packages/Paver-1.0.1-py2.6.egg/paver/tasks.py", >>> >> > line 160, in do_task >>> >> > task() >>> >> > File >>> >> > >>> "/home/nielso/sandpit/geonode/lib/python2.6/site-packages/Paver-1.0.1-py2.6.egg/paver/tasks.py", >>> >> > line 241, in __call__ >>> >> > retval = environment._run_task( >>> >> > <http://self.name><http://self.name> >>> self.name, self.needs, self.func) >>> >> > File >>> >> > >>> "/home/nielso/sandpit/geonode/lib/python2.6/site-packages/Paver-1.0.1-py2.6.egg/paver/tasks.py", >>> >> > line 181, in _run_task >>> >> > return do_task() >>> >> > File >>> >> > >>> "/home/nielso/sandpit/geonode/lib/python2.6/site-packages/Paver-1.0.1-py2.6.egg/paver/tasks.py", >>> >> > line 160, in do_task >>> >> > task() >>> >> > File >>> >> > >>> "/home/nielso/sandpit/geonode/lib/python2.6/site-packages/Paver-1.0.1-py2.6.egg/paver/tasks.py", >>> >> > line 241, in __call__ >>> >> > retval = environment._run_task( >>> >> > <http://self.name><http://self.name> >>> self.name, self.needs, self.func) >>> >> > File >>> >> > >>> "/home/nielso/sandpit/geonode/lib/python2.6/site-packages/Paver-1.0.1-py2.6.egg/paver/tasks.py", >>> >> > line 181, in _run_task >>> >> > return do_task() >>> >> > File >>> >> > >>> "/home/nielso/sandpit/geonode/lib/python2.6/site-packages/Paver-1.0.1-py2.6.egg/paver/tasks.py", >>> >> > line 160, in do_task >>> >> > task() >>> >> > File >>> >> > >>> "/home/nielso/sandpit/geonode/lib/python2.6/site-packages/Paver-1.0.1-py2.6.egg/paver/tasks.py", >>> >> > line 241, in __call__ >>> >> > retval = environment._run_task( >>> >> > <http://self.name><http://self.name> >>> self.name, self.needs, self.func) >>> >> > File >>> >> > >>> "/home/nielso/sandpit/geonode/lib/python2.6/site-packages/Paver-1.0.1-py2.6.egg/paver/tasks.py", >>> >> > line 181, in _run_task >>> >> > return do_task() >>> >> > File >>> >> > >>> "/home/nielso/sandpit/geonode/lib/python2.6/site-packages/Paver-1.0.1-py2.6.egg/paver/tasks.py", >>> >> > line 161, in do_task >>> >> > return func(**kw) >>> >> > File "pavement.py", line 186, in setup_gs_data >>> >> > grab(src_url, dst_url) >>> >> > File "pavement.py", line 159, in grab >>> >> > from urlgrabber.grabber import urlgrab, URLGrabError >>> >> > ImportError: No module named urlgrabber.grabber >>> >> > (geonode)nie...@shiva:~/sandpit/geonode$ >>> >> > >>> >> > >>> >> > >>> >> > >>> >> >>> > >>> > >>> >> >> >
