Re: [Distutils] Deviation between distutils and setuptools over package_data
On Thu, Sep 19, 2013 at 12:13 AM, Benjamin Root ben.v.r...@gmail.comwrote: On Sep 18, 2013 11:37 PM, PJ Eby p...@telecommunity.com wrote: On Wed, Sep 18, 2013 at 5:07 PM, Benjamin Root ben.v.r...@gmail.com wrote: In creating a source distribution, I have found a disparity between the behaviors of distutils and setuptools with respect to package_data. As of python Issue 2279: http://bugs.python.org/issue2279, entries listed in package_data are used when building an sdist. I have verified that this works in a simple example when setup() is imported from distutils.core. However, if I import setup() from setuptools, it does not pull in the data as listed in package_data (and presumedly data_files). You need to use the include_package_data = True flag. I did and it didn't make a difference. Plus, shouldn't it have picked up everything that I had version controlled, anyway? Does setuptools recognize the new svn 1.7 format? Even svn 1.6 clients will refuse to do an 'svn status' on a 1.7 repo checkout. So, I think there might be multiple bugs in play. I also don't see why I should have to use that keyword argument if distutils does fine without it. sdist isnt -- from my perspective -- a setuptools specific feature, so I had no expectation of there being such an egregious difference. P.S. - on a related note, for a package foo, foo.egg-info directory is created with a SOURCES.txt file. I have found that under certain situations, it seems that a successful install would result in a fully listed SOURCES.txt, and then subsequent calls to sdist seems to use that information to build seemingly correct archives. A co-working double-checking a deployment process I made did an sdist and created a source distribution without the package_data when he did a fresh checkout, but whenever I did it from my development branch, the source distribution worked fine. I haven't figured out exactly how this came about, but it seems to be tied to the SOURCES.txt file. SOURCES.txt mostly exists so that you can safely build an sdist from an sdist, as is required by e.g. bdist_rpm, without having any revision control data on hand to guide the process. Setuptools also can insert a possibly-modified setup.cfg into an sdist for the same reason, so that if you used revision control tags to specify the version when building the sdist, any sdists rebuilt from that sdist will have the same version tags. Yes, I understand what sources.txt is for. The issue is that it seems that it is possible for different build commands to produce different source.txt results. Since I haven't figured out how I managed to do that, it isn't the focus of my bug report. Cheers! Ben Root Stranger and stranger still... So, on a hunch, I tried making sure I had the latest version of setuptools installed (mine existing install was a little over a year old). On one system, easy_install found setuptools 1.1.6 and installed it. After that, by setup.py script behaved exactly as expected on my demo package I mocked up to test out the bug. No need for include_package_data, or MANIFEST.in or version control, just package_data. Great! To double-check, I tried it on another system (same version of python), and easy_install found setuptools 1.1.6 as well and installed it. But the bug demo I made didn't work. Using pydoc setuptools, it seems to have been pointing to distribute 0.6.28 for some reason. I retried easy_install -U setuptools, and this time it found distribute 0.7.3. After installing, pydoc setuptools now says that it is setuptools 1.1.6. Why it didn't say that before, I haven't a clue. But the bug example package I made then worked. Don't know what to make of all that, but in the end, package_data did work as expected. I will point out that even with setuptools 1.1.6, sdist isn't picking up all the files in version control, as I have a few other files under version control in my package that I didn't list for package_data. So, I still think there is an issue with crawling an SVN 1.7 repository. Cheers! Ben Root ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Deviation between distutils and setuptools over package_data
On 20 September 2013 00:02, Benjamin Root ben.v.r...@gmail.com wrote: So, on a hunch, I tried making sure I had the latest version of setuptools installed (mine existing install was a little over a year old). On one system, easy_install found setuptools 1.1.6 and installed it. After that, by setup.py script behaved exactly as expected on my demo package I mocked up to test out the bug. No need for include_package_data, or MANIFEST.in or version control, just package_data. Great! To double-check, I tried it on another system (same version of python), and easy_install found setuptools 1.1.6 as well and installed it. But the bug demo I made didn't work. Using pydoc setuptools, it seems to have been pointing to distribute 0.6.28 for some reason. I retried easy_install -U setuptools, and this time it found distribute 0.7.3. After installing, pydoc setuptools now says that it is setuptools 1.1.6. Why it didn't say that before, I haven't a clue. But the bug example package I made then worked. Don't know what to make of all that, but in the end, package_data did work as expected. For a variety of reasons, versions of distribute prior to 0.7 actively interfered with the usage of regular setuptools. With distribute 0.7+ (which was released after the two projects merged back together under the original setuptools name), it's just an empty shell that depends on the underlying setuptools. So that's probably what you're seeing - the distribute version was old, and preventing use of the updated setuptools. Once distribute was updated as well, the new setuptools started to be used as expected. I will point out that even with setuptools 1.1.6, sdist isn't picking up all the files in version control, as I have a few other files under version control in my package that I didn't list for package_data. So, I still think there is an issue with crawling an SVN 1.7 repository. That part I have no idea about :) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Deviation between distutils and setuptools over package_data
On Fri, Sep 20, 2013 at 04:43:23PM +1000, Nick Coghlan wrote: On 20 September 2013 00:02, Benjamin Root ben.v.r...@gmail.com wrote: I will point out that even with setuptools 1.1.6, sdist isn't picking up all the files in version control, as I have a few other files under version control in my package that I didn't list for package_data. So, I still think there is an issue with crawling an SVN 1.7 repository. That part I have no idea about :) I would recommend not relying on setuptools version control magic. Instead write a MANIFEST.in. shameless_plug There are tools that make this process somewhat less painful: https://pypi.python.org/pypi/check-manifest /shameless_plug Regards, Marius Gedminas -- Nobody will ever need more than 640k RAM! -- Bill Gates, 1981 Windows 95 needs at least 8 MB RAM. -- Bill Gates, 1996 Nobody will ever need Windows 95. -- logical conclusion signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Deviation between distutils and setuptools over package_data
On Fri, Sep 20, 2013 at 6:41 AM, Marius Gedminas mar...@pov.lt wrote: On Fri, Sep 20, 2013 at 04:43:23PM +1000, Nick Coghlan wrote: On 20 September 2013 00:02, Benjamin Root ben.v.r...@gmail.com wrote: I will point out that even with setuptools 1.1.6, sdist isn't picking up all the files in version control, as I have a few other files under version control in my package that I didn't list for package_data. So, I still think there is an issue with crawling an SVN 1.7 repository. That part I have no idea about :) I would recommend not relying on setuptools version control magic. Instead write a MANIFEST.in. It appears that MANIFEST.in is ignored if setuptools recognizes your VCS. Although there's so much magic in this particular facet of setuptools that I never have much confidence in my understanding. Did setuptools recently learn about git? For a while, an advantage of git was that setuptools ignored it. I would love an option to ignore VCS and let me specify what I want *explicitly*. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Deviation between distutils and setuptools over package_data
On Fri, Sep 20, 2013 at 08:15:43AM -0400, Jim Fulton wrote: On Fri, Sep 20, 2013 at 6:41 AM, Marius Gedminas mar...@pov.lt wrote: On Fri, Sep 20, 2013 at 04:43:23PM +1000, Nick Coghlan wrote: On 20 September 2013 00:02, Benjamin Root ben.v.r...@gmail.com wrote: I will point out that even with setuptools 1.1.6, sdist isn't picking up all the files in version control, as I have a few other files under version control in my package that I didn't list for package_data. So, I still think there is an issue with crawling an SVN 1.7 repository. That part I have no idea about :) I would recommend not relying on setuptools version control magic. Instead write a MANIFEST.in. It appears that MANIFEST.in is ignored if setuptools recognizes your VCS. Although there's so much magic in this particular facet of setuptools that I never have much confidence in my understanding. This is why check-manifest runs 'python setup.py sdist' in a clean exported directory with no VCS metadata for verification. Did setuptools recently learn about git? For a while, an advantage of git was that setuptools ignored it. I don't believe the setuptools-git plugin was ever merged into the core. I would love an option to ignore VCS and let me specify what I want *explicitly*. Marius Gedminas -- If it ain't broke, don't fix it. - Bert Lantz signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Deviation between distutils and setuptools over package_data
On Fri, Sep 20, 2013 at 2:15 PM, Jim Fulton j...@zope.com wrote: Did setuptools recently learn about git? Nop. I would love an option to ignore VCS and let me specify what I want *explicitly*. Or remove the handling of VCS? -- Sebastien Douche sdou...@gmail.com Twitter: @sdouche / G+: +sdouche ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Deviation between distutils and setuptools over package_data
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 09/20/2013 09:11 AM, Sebastien Douche wrote: On Fri, Sep 20, 2013 at 2:15 PM, Jim Fulton j...@zope.com wrote: Did setuptools recently learn about git? Nop. I use setuptools-git regularly without problems. I would love an option to ignore VCS and let me specify what I want *explicitly*. Or remove the handling of VCS? - -1. I haven't yet tried Marius' tool, but manually maintaining MANIFEST.in is an unaceptable bug magnet. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlI8WLwACgkQ+gerLs4ltQ4SIgCgh2dRMexciWxnaJBFrU700PDl dOkAoNUNELNFj9Ee0T+mD6ajC3FvQ8xV =FAe6 -END PGP SIGNATURE- ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Deviation between distutils and setuptools over package_data
On Fri, Sep 20, 2013 at 4:16 PM, Tres Seaver tsea...@palladion.com wrote: I use setuptools-git regularly without problems. Good to know. Or remove the handling of VCS? - -1. I haven't yet tried Marius' tool, but manually maintaining MANIFEST.in is an unaceptable bug magnet. $ cat MANIFEST.in recursive-include docs/source/ * recursive-include src/ * prune docs/build/ Not very complicated. -- Sebastien Douche sdou...@gmail.com Twitter: @sdouche / G+: +sdouche ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Deviation between distutils and setuptools over package_data
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 9/20/2013 10:16 AM, Tres Seaver wrote: On 09/20/2013 09:11 AM, Sebastien Douche wrote: On Fri, Sep 20, 2013 at 2:15 PM, Jim Fulton j...@zope.com wrote: Did setuptools recently learn about git? Nop. I use setuptools-git regularly without problems. I would love an option to ignore VCS and let me specify what I want *explicitly*. Or remove the handling of VCS? -1. I haven't yet tried Marius' tool, but manually maintaining MANIFEST.in is an unaceptable bug magnet. I like having MANIFEST.in correct in case I need to build in a system that's not a checkout. But I agree it's a bug magnet. What I'd really like is a mode that throws an error if MANIFEST.in doesn't agree with the VCS detection. - -- Eric. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.13 (Cygwin) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJSPF/qAAoJENxauZFcKtNx0KAIAKNRzH3yPw13TIAPBmsIs/Vt sJT0ilbxwY8DV9fEJ5stC5P2QiZVmAxP2UNvYqZPTjC9qwvMS6UdMNznJAZBgRd7 cRESzpF1Sf7PZKKNvrIlbRfYZMZP89lGjD21UT/qbFclSZGkormxLVk3kDnDzpSe eOAM7gfWWBi/69eAuT1G5pJbIt+2KVZd4AYh27NXmu8F5RtB/EDgSiMuPvIwPP6r SrTa+1maqs6Ik5o768ED+GoXkGmGMvaUIr1GWZLasA7Q36oM1m9pX9xZ87RIA/aw n+ZcqlWM6tjnevP3SXAjOKNi+fIY1t2I2ECfn6BtY+JXbrlAUAWnbO/MsIoR+NA= =Lorj -END PGP SIGNATURE- ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Deviation between distutils and setuptools over package_data
On Fri, Sep 20, 2013 at 8:15 AM, Jim Fulton j...@zope.com wrote: It appears that MANIFEST.in is ignored if setuptools recognizes your VCS. Not in setuptools 0.6 it isn't. However, it's really only useful for *adding* files not picked up by revision control; it can't *remove* files found by revision control. Manifest and revision control support are semi-independent, as it's quite possible to have files you don't put in revision control but which nonetheless need to be included in an sdist. (For example, you might want to have Cython/Pyrex-generated C code, or compiled documentation files.) ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Deviation between distutils and setuptools over package_data
On Fri, Sep 20, 2013 at 10:16:28AM -0400, Tres Seaver wrote: On 09/20/2013 09:11 AM, Sebastien Douche wrote: On Fri, Sep 20, 2013 at 2:15 PM, Jim Fulton j...@zope.com wrote: I would love an option to ignore VCS and let me specify what I want *explicitly*. Or remove the handling of VCS? -1. I haven't yet tried Marius' tool, but manually maintaining MANIFEST.in is an unaceptable bug magnet. Heh. I wrote check-manifest because relying on setuptools's handling of VCS was an unacceptable bug magnet to me. (New svn working tree format _or_ you forget to install setuptools-git on this one machine? No error, just a broken sdist.) BTW if you use zest.releaser, check-manifest plugs into it and stops the release (well, asks if you REALLY want to continue, defaulting to No) if your MANIFEST.in doesn't match your VCS. Marius Gedminas -- It was the kind of night on which bad novels began. -- 1634: The Galileo Affair by Eric Flint and Andrew Dennis signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] what is the difference in process between python setup.py install and pip install
Thus I expected naively that pip install . and python setup.py install should have the same results. they process install_requires with different code, so things could turn out differently. I have a reproducible case when the dependency handling differs if you use pip install -r requirements.txt or when you use python setup.py install https://github.com/zeromq/pyzmq/issues/414#issuecomment-24679800 I looked at this case, but atleast for me, the results were the same in all 3 cases (note that this project splits requirements.txt to generate 'install_requires') 1) python setup.py install 2) pip install . 3) pip install -r requirements.txt from pip list after the installs: anyjson (0.3.3) configparser (3.3.0r2) coverage (3.6) docutils (0.11) Jinja2 (2.7.1) MarkupSafe (0.18) picomongo (0.6) Pygments (1.6) pymongo (2.6.2) pyzmq (13.1.0) satlive (0.8.0) Sphinx (1.2b2) btw, pip1.4 has a bug where a stale build directory can lead to unexpected results. check for the offending project in venv/build (if using a virtualenv), or if using a global python /tmp/pip-build-username ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
