What command are you using to result in the output? The root page lists all available packages on that index, and with 11MB of data there must be literally millions of entries to parse, and would definitely take a long time. But it definitely don’t need to be done under normal circumstances, and pip/setuptools definitely does not do it under normal circumstances.
Sent from Mail for Windows 10 From: chris.cra...@rsa.com Sent: 02 August 2019 05:06 To: distutils-sig@python.org Subject: [Distutils] setuptools reading simple index takes 30sec Hello, I have a python dev team that I support in our local jenkins/artifactory environment. Recently this team switched over from an old local pypi legacy repository to a current pypi remote/virtual of pypi.org. So we could get them updated libraries and such. Anyway, I am banging my head to explain the time taken for reading the simple index. This index is about 11MB, takes 1 second to download itself, but then it takes ~30 seconds for setuptools to scan this and find the package it's looking for. Multiple this by the ~170 libraries they have in requirements, and the build has gone from 30minutes to almost 3 hours. And 90% comes from reading the simple index for each dependency. Here's a sample output with timestamps 00:57:00.036 Searching for tenacity==4.8.0 00:57:00.037 Reading https://repo1.local/artifactory/api/pypi/py-pypi-virt/simple/ 00:57:27.650 Reading https://repo1.local/artifactory/api/pypi/py-pypi-virt/simple/tenacity/ 00:57:27.881 Downloading https://repo1.local/artifactory/api/pypi/py-pypi-virt/packages/packages/fc/e9/5499018e0d420f8d03a215c310ee7bc6e1a7e84adaa63f6ea208e864bdb6/tenacity-4.8.0-py2.py3-none-any.whl#sha256=efcf0672547f52fd49f96c2c1912e0f0e77d78a6630823aad54f99227a3c332d 00:57:28.037 Best match: tenacity 4.8.0 00:57:28.037 Processing tenacity-4.8.0-py2.py3-none-any.whl 00:57:28.038 Installing tenacity-4.8.0-py2.py3-none-any.whl to /home/build/workspace/workflows-pr/workflows-extension/.eggs 00:57:28.069 writing requirements to /home/build/workspace/workflows-pr/workflows-extension/.eggs/tenacity-4.8.0-py2.7.egg/EGG-INFO/requires.txt 00:57:28.114 Installed /home/build/workspace/workflows-pr/workflows-extension/.eggs/tenacity-4.8.0-py2.7.egg Note the time on the first Reading. I'm hoping there is some obvious simple solution to make this faster. Any ideas or help would be greatly appreciated? Thanks -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/O7M274ZYQMMU3WT52PY2YIVJMCNOQJIR/
-- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/M3R6JQ5X6MN4TRYAM5JYVL2OLO52BJW7/
