On Jul 22, 2015 5:12 PM, "Brett Cannon" <bcan...@gmail.com> wrote: > > > > On Wed, Jul 22, 2015 at 2:19 PM Wes Turner <wes.tur...@gmail.com> wrote: >> >> https://github.com/dstufft/pypi-stats >> >> https://github.com/dstufft/pypi-external-stats > > > I'm not quite sure what I'm supposed to get from those links, Wes, as that code still scrapes every project individually and downloads them while all I'm trying to avoid having to scrape PyPI and instead just download a single file (plus I don't want the files but just the metadata already returned by the JSON API).
An online query or an offline dump? > > -Brett > >> >> - [ ] a flat bigquery w/ pandas.io.gbq ala GitHub Archive would be great http://pandas.pydata.org/pandas-docs/version/0.16.2/io.html#io-bigquery >> - [ ] it's probably worth it to add RDFa to PyPi and warehouse pages (in addition to the auxiliary executed/extracted JSON) for #search https://github.com/pypa/warehouse/blob/master/warehouse/packaging/models.py https://github.com/pypa/warehouse/blob/master/tests/unit/packaging/test_models.py https://github.com/pypa/warehouse/blob/master/warehouse/packaging/views.py https://github.com/pypa/warehouse/blob/master/warehouse/templates/packaging/detail.html https://github.com/pypa/warehouse/blob/master/warehouse/routes.py https://github.com/pypa/warehouse/blob/master/tests/unit/legacy/api/test_json.py https://github.com/pypa/warehouse/blob/master/warehouse/legacy/api/json.py >> >> On Jul 22, 2015 4:08 PM, "Brett Cannon" <bcan...@gmail.com> wrote: >>> >>> When I wrote https://nothingbutsnark.svbtle.com/python-3-support-on-pypi I wrote a script to download every project's JSON metadata by scraping the simple index and then making the appropriate GET request for the JSON metadata. It worked, but somewhat of a hassle. >>> >>> Is there some dump somewhere that is built daily, weekly, or monthly of all the metadata on PyPI for offline analysis? >>> >>> _______________________________________________ >>> Distutils-SIG maillist - Distutils-SIG@python.org >>> https://mail.python.org/mailman/listinfo/distutils-sig >>>
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
