Re: [Distutils] People want CPAN
On Sun, Nov 15, 2009 at 3:44 AM, Jeremy Kloth jeremy.kl...@gmail.com wrote: [..] So did you end up changing the way options are passed to the commands, or do you just have a specific config command that looks over other options passed to the other commands ? It is done with the 'config' command having all the options used by the other commands. The other commands would then look up their options' values from 'config' (if not supplied on the command-line). If, for example, `--prefix` was supplied to 'install', the 'install' command would then cause the 'config' to redo its stored options. If at all possible, I would eliminate the redundant options on build_*/install_* and leave them solely on 'config' as it greatly simplifies the interaction of the commands. So, pratically speaking, if: $ python setup.py install is called, the install command will instanciate the configure command, that will return options that were stored in some file, in a previous call ? But, for the option redundancy problem, the simplest way I can see to have the configure options in install, or to let the end user pass them along, would be to make configure the base class for all commands that are part of the configure-make-install story, so when they run they can read and write options in the stored file and use if needed the options passed in the command line. Tarek ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Sun, Nov 15, 2009 at 7:24 AM, Greg Ewing greg.ew...@canterbury.ac.nz wrote: Tarek Ziadé wrote: And have the community create new ExtensionBuilder subclasses that could be registered like command. I don't see a need for registering anything. You should just be able to explicitly say what tool to use for each stage of the process. I envisage something like this: from distutils import Extension, CCompile from pyrex.distutils import PyrexCompile foo_ext = Extension(foo, CCompile( PyrexCompile(foo.pyx), somelib.c)) Here Extension, CCompile and PyrexCompile are constructors for dependency graph nodes. Their responsibilities are: Extension -- takes compiled object files and libraries and links them into a Python extension. CCompile -- takes C source files and turns them into object files. PyrexCompile -- takes Pyrex source files and turns them into C source files. They would of course also take other relevant arguments such as compiler flags, search paths, etc. The advantage of the registery is that a project can provide a compiler type, let's say Pyrex. Then you can use in your own project setup.py this compiler without explicitely importing something. But the result is similar, and explicit imports should work too, so maybe registeries are just sugar on the top of something we first need to make work. Tarek ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
Tarek Ziadé wrote: But the result is similar, and explicit imports should work too, so maybe registeries are just sugar on the top of something we first need to make work. It's completely unnecessary sugar, if you ask me. I don't see what's bad about importing functionality you want to use. Where and how do you intend the registration to happen, anyway? Would it be done by the setup.py script? In that case I don't see how it saves you anything, since you would have to first import the thing you want to register anyway. Or are you envisaging that Pyrex or whatever tool is involved would somehow patch itself into distutils when you install it? I don't like that idea much, since it smacks of the kind of monkeypatching that setuptools is reviled for. -- Greg ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
At 03:13 PM 11/11/2009 +0100, Tarek Ziadé wrote: But you call it with install in your example, meaning that is is called at install time, right ? Or it is just that you want to get the --prefix value finalized and computed by the install command. If it's the later, I guess you will be able to use the upcoming sysconfig module, that gives you the install schemes, depending on sys.prefix/sys.exec_prefix. The issue is that setup.py can accept multiple commands on the command line, and in principle build_clib might be being called as a subcommand of build (and thus of install). So, he needs the *active* --prefix, either from the command line, config file(s), or defaults. Simply having an API to get the defaults won't help this. Really, getting a finalized install command object is the only way to do this correctly in distutils at the moment, and the sysconfig API won't change that. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Fri, Nov 13, 2009 at 3:46 PM, Jeremy Kloth jeremy.kl...@gmail.com wrote: [..] The exact thing being described has been done in 4Suite for 6 years (along with many other distutils improvements). Feel free to take or discuss or request help with any of the features/additions (like FHS layout of files) in the 4Suite distutils extensions. The code is available for browsing: http://cvs.4suite.org/viewcvs/4Suite/Ft/Lib/DistExt/ That's great, thanks for the pointer ! I'll look into it asap The way we've impl'd 'config' was as a prerequisite for 'build', just as 'build' is for 'install'. If any of the options stored by the 'config' command are overridden via 'build' or 'install' options, the 'config' command would be re-run to store the new choices. Any questions or just simple help with integrating a similar system, just let me know. Been there, done that. So did you end up changing the way options are passed to the commands, or do you just have a specific config command that looks over other options passed to the other commands ? Tarek ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Wed, Nov 11, 2009 at 11:49 PM, Pauli Virtanen p...@iki.fi wrote: [..] Just to throw some wild, perhaps obvious, and definitely unasked-for ideas in the air (especially as I can't promise I can give any sustained help here :/ ): I suppose one option would be to factor *everything* related to extension module building into build_ext (and abolish build_clib): the rest of distutils would just say to build_ext Here's some info about the environment: dict(python_lib_path=/usr/lib/..., optimize=yes/no, python_lib_name=..., python_includes=/usr/include/..., install_prefix=/usr/lib/python2.6, ... ..., python_extension=..., build_temp_dir=...) Look the rest up from sysconfig. Please build any extensions and data files you like, and tell the file and directory names where you placed them and where (relative paths) they should go. I ought to be something like that, but what is unclear to me is how to describe which compiler to use for what files. I had this one extension == one compiler type pattern in my head, but it seems more complex than that. IOW an extension can invoke several compilers and tools to be built. So one extension == one extension builder might best describe it. and I am wondering if we can't define a simple interface for these extension builders, from the simplest case (one tool uses one compiler) to the weirdest one (one tool uses a complex toolchain to create the extension) So at the end we would have: - Extension (the existing extension class, that takes a extension_builder_type) - ExtensionBuilder (class in charge of creating an extension) - a registery for ExtensionBuilder subclasses And have the community create new ExtensionBuilder subclasses that could be registered like command. build_ext would then become an empty shell, just in charge of looping through the extensions, so each extension invokes its builder. [..] I think this idea quickly boils down more or less to David's idea about a pluggable build system -- implementing a good one takes a lot of work, so it might make sense to refactor distutils so that it would be possible [1] to use some of the existing ones (scons, waf, whatever, heck, even autoconf+make). The *default* build system could be a simple one, and backwards compatible. Especially so, since it seems to me that building extension modules is orthogonal to what distutils and setuptools want to do -- package and install Python modules in a Python-specific way, not really worry about how to properly call different compilers on obscure platforms. Anyway, even in the case pluggability is a bad idea, refactoring the build system out from the rest of distutils might make sense. Agreed. it seems that the addition of the configure command, and the refactoring of the build_ext one, are the right things to do, together with the addition of the sysconfig stdlib module (which allows configure to get more info that what distutils.sysconfig provides) Now, if we can take back the work done in 4suite as suggested, or in scons, etc, that's even better. Tarek ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Saturday 14 November 2009 05:14:05 pm Tarek Ziadé wrote: On Fri, Nov 13, 2009 at 3:46 PM, Jeremy Kloth jeremy.kl...@gmail.com The way we've impl'd 'config' was as a prerequisite for 'build', just as 'build' is for 'install'. If any of the options stored by the 'config' command are overridden via 'build' or 'install' options, the 'config' command would be re-run to store the new choices. Any questions or just simple help with integrating a similar system, just let me know. Been there, done that. So did you end up changing the way options are passed to the commands, or do you just have a specific config command that looks over other options passed to the other commands ? It is done with the 'config' command having all the options used by the other commands. The other commands would then look up their options' values from 'config' (if not supplied on the command-line). If, for example, `--prefix` was supplied to 'install', the 'install' command would then cause the 'config' to redo its stored options. If at all possible, I would eliminate the redundant options on build_*/install_* and leave them solely on 'config' as it greatly simplifies the interaction of the commands. Jeremy -- Jeremy Kloth http://4suite.org/ ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
Tarek Ziadé wrote: And have the community create new ExtensionBuilder subclasses that could be registered like command. I don't see a need for registering anything. You should just be able to explicitly say what tool to use for each stage of the process. I envisage something like this: from distutils import Extension, CCompile from pyrex.distutils import PyrexCompile foo_ext = Extension(foo, CCompile( PyrexCompile(foo.pyx), somelib.c)) Here Extension, CCompile and PyrexCompile are constructors for dependency graph nodes. Their responsibilities are: Extension -- takes compiled object files and libraries and links them into a Python extension. CCompile -- takes C source files and turns them into object files. PyrexCompile -- takes Pyrex source files and turns them into C source files. They would of course also take other relevant arguments such as compiler flags, search paths, etc. -- Greg ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Fri, Nov 13, 2009 at 6:22 AM, David Cournapeau da...@ar.media.kyoto-u.ac.jp wrote: Tarek Ziadé wrote: A deprecation warning would be added in install, if it finds a local option, rather than a global. Meaning both would work in 2.7/3.2. If changing the command line in incompatible ways is acceptable, what do you think of scrapping the commands (at the UI level only) altogether ? This would be more consistent and easier to deal for the user, and easier to implement as well: python setup.py configure --option1 --option2=value2 python setup.py build python setup.py install We could then make this work: python setup.py install (would run both build and configure implicitly). Making all finalized options available at the build stage would then be easier. Is that scraping, or just preparing finalized options using configure ? Meaning other command would just have to get them when they run, if present ? How would that work ? configure would create a file ? It seems that you are pushing all the work in the configure option, wich is fine to me, but it also looks like you can already achieve this with the existing system, by changing the subcommands that are in the install command and their order. That is: install configure build all the install_* But if we want to see this working with build alone, configure has to be a subcommand of build: install build configure all the install_* IOW this would just require: 1/ adding a configure command 2/ inserting it as the first sub command of build 3/ make it possible, to share in the whole chain of commands, the passed arguments Tarek ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Fri, Nov 13, 2009 at 1:18 PM, Floris Bruynooghe floris.bruynoo...@gmail.com wrote: [..] Is that scraping, or just preparing finalized options using configure ? Meaning other command would just have to get them when they run, if present ? How would that work ? configure would create a file ? That would be the obvious solution. Both autoconf and CPAN do this by my understanding so it's a pretty logical thing to do. I find it logical too now. [..] Would it not be harder to add new command (or build tasks) that require information from the configure step in you structure it like this? I was thinking about an API that would allow any command to read/write the configuration data, and using it from the configure command to write it, and from the others to read it. That would allow including this new behaviour in existing commands with a deprecation step (in today's distutils, a build is triggered when you call install in any case, when there are some extensions) Tarek ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Fri, Nov 13, 2009 at 11:26:23AM +0100, Tarek Ziadé wrote: On Fri, Nov 13, 2009 at 6:22 AM, David Cournapeau da...@ar.media.kyoto-u.ac.jp wrote: Tarek Ziadé wrote: A deprecation warning would be added in install, if it finds a local option, rather than a global. Meaning both would work in 2.7/3.2. If changing the command line in incompatible ways is acceptable, what do you think of scrapping the commands (at the UI level only) altogether ? This would be more consistent and easier to deal for the user, and easier to implement as well: python setup.py configure --option1 --option2=value2 python setup.py build python setup.py install We could then make this work: python setup.py install (would run both build and configure implicitly). Making all finalized options available at the build stage would then be easier. Is that scraping, or just preparing finalized options using configure ? Meaning other command would just have to get them when they run, if present ? How would that work ? configure would create a file ? That would be the obvious solution. Both autoconf and CPAN do this by my understanding so it's a pretty logical thing to do. It seems that you are pushing all the work in the configure option, wich is fine to me, but it also looks like you can already achieve this with the existing system, by changing the subcommands that are in the install command and their order. That is: install configure build all the install_* But if we want to see this working with build alone, configure has to be a subcommand of build: install build configure all the install_* Would it not be harder to add new command (or build tasks) that require information from the configure step in you structure it like this? Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Fri, Nov 13, 2009 at 9:46 PM, Tarek Ziadé ziade.ta...@gmail.com wrote: On Fri, Nov 13, 2009 at 1:18 PM, Floris Bruynooghe floris.bruynoo...@gmail.com wrote: [..] Is that scraping, or just preparing finalized options using configure ? Meaning other command would just have to get them when they run, if present ? How would that work ? configure would create a file ? That would be the obvious solution. Both autoconf and CPAN do this by my understanding so it's a pretty logical thing to do. I find it logical too now. Scons and waf do it through a db kind of file, but that's because they are much more fancy (scons for example keeps so called signature of every command and nodes to know what has already been built or not). A first step as a plain file, which should clearly be marked as implementation-defined and only guaranteed to work through an API is the way to go I think. David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Fri, Nov 13, 2009 at 1:52 PM, David Cournapeau courn...@gmail.com wrote: On Fri, Nov 13, 2009 at 9:46 PM, Tarek Ziadé ziade.ta...@gmail.com wrote: On Fri, Nov 13, 2009 at 1:18 PM, Floris Bruynooghe floris.bruynoo...@gmail.com wrote: [..] Is that scraping, or just preparing finalized options using configure ? Meaning other command would just have to get them when they run, if present ? How would that work ? configure would create a file ? That would be the obvious solution. Both autoconf and CPAN do this by my understanding so it's a pretty logical thing to do. I find it logical too now. Scons and waf do it through a db kind of file, but that's because they are much more fancy (scons for example keeps so called signature of every command and nodes to know what has already been built or not). A first step as a plain file, which should clearly be marked as implementation-defined and only guaranteed to work through an API is the way to go I think. Here's my proposal for this to happen, if you (and others) want to contribute: Let's build this new configure command in Distribute 0.7, together with the APIs to read/write the data. Then let's change the other commands consequently (without thinking about backward compat first) so we can try out this new behaviour. Once it's proven to work good, we could publish Distribute 0.7 with it, and depending on the community feedback, we could integrate it to Distutils and work on the backward compat part. this two phase step wouldn't be a problem imho, for early adopters that would use and test it. Tarek -- Tarek Ziadé | http://ziade.org | オープンソースはすごい! | 开源传万世，因有你参与 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Friday 13 November 2009 06:44:37 am Tarek Ziadé wrote: Here's my proposal for this to happen, if you (and others) want to contribute: Let's build this new configure command in Distribute 0.7, together with the APIs to read/write the data. Then let's change the other commands consequently (without thinking about backward compat first) so we can try out this new behaviour. Fourthought's 4Suite distutils extensions already provide this behavior. You may consider looking at how the 'config' command is handled there as a guide in implementing this. Or just copy it wholesale, for that matter, as it is my doing. Once it's proven to work good, we could publish Distribute 0.7 with it, and depending on the community feedback, we could integrate it to Distutils and work on the backward compat part. this two phase step wouldn't be a problem imho, for early adopters that would use and test it. The exact thing being described has been done in 4Suite for 6 years (along with many other distutils improvements). Feel free to take or discuss or request help with any of the features/additions (like FHS layout of files) in the 4Suite distutils extensions. The code is available for browsing: http://cvs.4suite.org/viewcvs/4Suite/Ft/Lib/DistExt/ The way we've impl'd 'config' was as a prerequisite for 'build', just as 'build' is for 'install'. If any of the options stored by the 'config' command are overridden via 'build' or 'install' options, the 'config' command would be re-run to store the new choices. Any questions or just simple help with integrating a similar system, just let me know. Been there, done that. Jeremy -- Jeremy Kloth http://4suite.org/ ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
ke, 2009-11-11 kello 21:42 +0100, Tarek Ziadé kirjoitti: [clip] Do you mean, an Extension that would require several compilers ? I was thinking of a one-to-one relation between an Extension and a compiler type, even if there are are multiple source files (in different languages) for this extension. Meaning that *one* compiler would have to handle of those files. Does that fit the cython use case? It might. The distutils Cython compiler would however need to be smart enough to handle all languages used in the other source files by itself, or somehow delegate this back to distutils (which still sounds doable). Also, the f2py compiler does similar things. Ditto for Pyrex. How much code duplication would be needed between these? Source file preprocessing / generation is one thing that also comes useful. For example, Numpy has its own templated C code generator file format, and also generates some of the source files automatically from scratch during the build. These are compiled and linked as a part of ordinary C extension module. Currently, as I understand it, numpy.distutils does source generation in a separate build_src command. Also, the option of determining the necessary compiler and compiler options from file extensions does not always work. For example, Fortran 90 files come in two flavors, fixed-form and free-form, which often are not differentiated by the file extension. However, they may require different compiler flags to compile. (Hopefully, however, nobody uses the fixed-form for F90 any more...) How do they do then? Is file extensions, inline options in setup.py, and the environ, are enough? numpy.distutils's Fortran compiler class reads the file and tries to analyze whether it's fixed-form or not. And chooses appropriate compiler flags. This is sort of a special worst case scenario for compiler selection, of course. *** Just to throw some wild, perhaps obvious, and definitely unasked-for ideas in the air (especially as I can't promise I can give any sustained help here :/ ): I suppose one option would be to factor *everything* related to extension module building into build_ext (and abolish build_clib): the rest of distutils would just say to build_ext Here's some info about the environment: dict(python_lib_path=/usr/lib/..., optimize=yes/no, python_lib_name=..., python_includes=/usr/include/..., install_prefix=/usr/lib/python2.6, ... ..., python_extension=..., build_temp_dir=...) Look the rest up from sysconfig. Please build any extensions and data files you like, and tell the file and directory names where you placed them and where (relative paths) they should go. Information needed how to build each component would be passed to build_ext subcomponent directly from setup.py or from a config file. Now, perhaps it is already like this --- I don't really know the internals of distutils --- and the build subcomponent is insulated from the others. In any case, something like this could make refactoring the build system easier. I think this idea quickly boils down more or less to David's idea about a pluggable build system -- implementing a good one takes a lot of work, so it might make sense to refactor distutils so that it would be possible [1] to use some of the existing ones (scons, waf, whatever, heck, even autoconf+make). The *default* build system could be a simple one, and backwards compatible. Especially so, since it seems to me that building extension modules is orthogonal to what distutils and setuptools want to do -- package and install Python modules in a Python-specific way, not really worry about how to properly call different compilers on obscure platforms. Anyway, even in the case pluggability is a bad idea, refactoring the build system out from the rest of distutils might make sense. .. [1] possible and easy -- I understand numpyscons is a stab at the possible, but it sounds like it was not easy to do. -- Pauli Virtanen ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Thu, Nov 12, 2009 at 5:30 AM, David Cournapeau
da...@ar.media.kyoto-u.ac.jp wrote:
Greg Ewing wrote:
As far as I can tell David seems to be saying that
instantiating a class for every file in the system
would be too much overhead.
I'm not convinced about that -- make builds a dependency
graph with a node for every target before it goes
to work, and I think scons does something similar
Yes, and scons has scalability problems because of this (both from CPU
and memory POV).
But you were saying in the discussion that Disutils has a *design*
problem because it uses OOP...
Anyways, let's not drop the other part of this thread:
get_install_paths('FOO')
And that's the API we want to add in sysconfig, roughly.
That does not solve the problem about getting FOO in the first place
when calling get_install_path.
Why that ? where FOO comes from ? if it's an option you provide at
build time like you said,
earlier, you just pass it to the API to get the paths.
FOO is not coming from nowhere...
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
Thu, 12 Nov 2009 10:54:41 +0100, Tarek Ziadé wrote:
[clip]
get_install_paths('FOO')
And that's the API we want to add in sysconfig, roughly.
That does not solve the problem about getting FOO in the first place
when calling get_install_path.
Why that ? where FOO comes from ? if it's an option you provide at
build time like you said, earlier, you just pass it to the API to get
the paths.
FOO is not coming from nowhere...
If this is not painfully clear already, the user passes FOO to the command
python setup.py install --prefix=FOO
Now, clearly the distutils install subcommand knows what FOO is.
But does build_ext? Does sysconfig?
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
Tarek Ziadé schrieb: For the documentation part I am afraid it will be messy for the end users trying to package apps in Python *until* all PEPs have made it into Python. Although, as Ian Bicking says: we could write today some kind of all-in-one tutorial so end-users can work out without having to run after the documenation in several places. Big +1, and I would be more than willing to include it in the standard Python documentation, *even* if it mostly describes setuptools/distribute/pip. When people want to package a library, they *will* look for docs in Python, and at the moment they only find the distutils reference. While the latter is necessary, a more howto-like standard document is much needed. Georg I have the start of such a document, derived from my own recent experience pulling together all the information needed to release a new Python project. It's aimed at people new to packaging and possibly open source in general, so it's broader than just packaging. It includes things like basic file layout, unit testing, Pylint, getting started with Sphinx, a sample (distutils) setup.py, and how to register with PyPI. I try to pull together all the info a novice would need to have a reasonable shot at sharing their code. It's by no means all-encompassing (e.g. no mention of native extensions), but perhaps someone would be willing to write a complimentary advanced packaging howto. You can find it at: http://infinitemonkeycorps.net/docs/pph/ Let me know what you think. -John ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Thu, Nov 12, 2009 at 12:31 PM, Pauli Virtanen pav...@iki.fi wrote:
Thu, 12 Nov 2009 10:54:41 +0100, Tarek Ziadé wrote:
[clip]
get_install_paths('FOO')
And that's the API we want to add in sysconfig, roughly.
That does not solve the problem about getting FOO in the first place
when calling get_install_path.
Why that ? where FOO comes from ? if it's an option you provide at
build time like you said, earlier, you just pass it to the API to get
the paths.
FOO is not coming from nowhere...
If this is not painfully clear already, the user passes FOO to the command
python setup.py install --prefix=FOO
Now, clearly the distutils install subcommand knows what FOO is.
But does build_ext? Does sysconfig?
The install command takes FOO, and build several paths with it:
/FOO/lib/site-package/
/FOO/xxx
The other commands that needs to get the same path can rebuild it using an API,
that does:
get_paths(scheme, vars={'prefix': 'FOO')
Instead of doing what David did:
$ python setup.py build_ext install --prefix=FOO
They can do:
$ python setup.py build_ext --prefix=FOO
and don't require to use the install command anymore to get these paths cooked.
Remember that the build step has nothing to do with the install step,
and David in his example was doing a build that was not relocatable,
e.g. receiving the install paths and applying them in the binaries.
So instead of having this paths created by code in the install command,
they are moved in an API than can be used by install or by any other command.
Is that clearer ?
Tarek
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Thu, Nov 12, 2009 at 9:42 PM, John Kleint jkle...@gmail.com wrote: Tarek Ziadé schrieb: For the documentation part I am afraid it will be messy for the end users trying to package apps in Python *until* all PEPs have made it into Python. Although, as Ian Bicking says: we could write today some kind of all-in-one tutorial so end-users can work out without having to run after the documenation in several places. Big +1, and I would be more than willing to include it in the standard Python documentation, *even* if it mostly describes setuptools/distribute/pip. When people want to package a library, they *will* look for docs in Python, and at the moment they only find the distutils reference. While the latter is necessary, a more howto-like standard document is much needed. Georg I have the start of such a document, derived from my own recent experience pulling together all the information needed to release a new Python project. It's aimed at people new to packaging and possibly open source in general, so it's broader than just packaging. It includes things like basic file layout, unit testing, Pylint, getting started with Sphinx, a sample (distutils) setup.py, and how to register with PyPI. I try to pull together all the info a novice would need to have a reasonable shot at sharing their code. It's by no means all-encompassing (e.g. no mention of native extensions), but perhaps someone would be willing to write a complimentary advanced packaging howto. You can find it at: http://infinitemonkeycorps.net/docs/pph/ Let me know what you think. That's a great start indeed. If we can expand on the packaging thing we could take it from here and see who wants to help on this. I could try to convince some developers from pip and virtualenv to help on this. Are you willing to share it in a DVCS or something for that work ? How would you like to manage contributions in it ? Tarek ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Fri, Nov 13, 2009 at 7:12 AM, Tarek Ziadé ziade.ta...@gmail.com wrote:
On Thu, Nov 12, 2009 at 12:31 PM, Pauli Virtanen pav...@iki.fi wrote:
Thu, 12 Nov 2009 10:54:41 +0100, Tarek Ziadé wrote:
[clip]
get_install_paths('FOO')
And that's the API we want to add in sysconfig, roughly.
That does not solve the problem about getting FOO in the first place
when calling get_install_path.
Why that ? where FOO comes from ? if it's an option you provide at
build time like you said, earlier, you just pass it to the API to get
the paths.
FOO is not coming from nowhere...
If this is not painfully clear already, the user passes FOO to the command
python setup.py install --prefix=FOO
Now, clearly the distutils install subcommand knows what FOO is.
But does build_ext? Does sysconfig?
The install command takes FOO, and build several paths with it:
/FOO/lib/site-package/
/FOO/xxx
The other commands that needs to get the same path can rebuild it using an
API,
that does:
get_paths(scheme, vars={'prefix': 'FOO')
Instead of doing what David did:
$ python setup.py build_ext install --prefix=FOO
They can do:
$ python setup.py build_ext --prefix=FOO
and don't require to use the install command anymore to get these paths
cooked.
I think I was confusing with my rpath example, which may be the source
of our misunderstanding. I don't want to add a --prefix option to
build_ext. I want to support the following user cases:
python setup.py build # here, prefix would be whatever default, as
available from sysconfig
python setup.py install --prefix=foo # here, prefix would be the one
as computer by install command
python setup.py build_ext -i # here prefix is the current directory
Requiring users to handle new options of commands is impractical IMHO,
and a prefix option to build has a strange feeling to it.
Remember that the build step has nothing to do with the install step
Ideally, what we want here is like autoconf does it. You have a
configure step, and then at build/install stages, you have access to
*all* options. Those options can be customized at build through usual
make mechanism, but this is taken into account as well without the
makefile writer having to care.
IOW, we need to know *all* the finalized options *before* the first
build command is even run.
David
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Fri, Nov 13, 2009 at 1:08 AM, David Cournapeau courn...@gmail.com wrote:
[..]
$ python setup.py --prefix=foo cmd1 cmd1 etc
and the result would be in Distribution.options = {'path1': xxx, 'path2': xx}
This is a major change in distutils behavior, so we need to solve the
following issues:
- every user will have to change how to call distutils
- what happens if people still use python setup.py install
--option1=foo instead of python setup.py --prefix=option1 install
A deprecation warning would be added in install, if it finds a local
option, rather
than a global. Meaning both would work in 2.7/3.2.
That would give them 18 to 24 months * 2 to go with the new style.
Tarek
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Thu, Nov 12, 2009 at 11:45 PM, David Cournapeau courn...@gmail.com wrote:
[..]
I think I was confusing with my rpath example, which may be the source
of our misunderstanding. I don't want to add a --prefix option to
build_ext. I want to support the following user cases:
python setup.py build # here, prefix would be whatever default, as
available from sysconfig
python setup.py install --prefix=foo # here, prefix would be the one
as computer by install command
python setup.py build_ext -i # here prefix is the current directory
Requiring users to handle new options of commands is impractical IMHO,
and a prefix option to build has a strange feeling to it.
I am not sure to follow here: let's forget about your example where
you call build_src and install together.
- in the real world, how a --prefix passed to install is going to
impact a build command and vice-versa ?
install just copy files, where it's told to copy them, and build does
some work with some options,
and as a matter of fact you seem to use installation prefix at this stage.
So, if you are not *installing*, it doesn't make sense to call the
*install* command, and build could
have its --prefix option in your case.
Remember that the build step has nothing to do with the install step
Ideally, what we want here is like autoconf does it. You have a
configure step, and then at build/install stages, you have access to
*all* options. Those options can be customized at build through usual
make mechanism, but this is taken into account as well without the
makefile writer having to care.
IOW, we need to know *all* the finalized options *before* the first
build command is even run.
So, IOW, these options cannot be finalized by *any* command. So the
proposal I've made
to have global options that are common to all commands, and that can
be used to create
an execution environment through public APIs would help in there.
So if the install command transforms prefix into path1 path2 and
path3, we need this transformation
to occur outside install, so it can be done before the commands are run.
that is:
$ python setup.py --prefix=foo cmd1 cmd1 etc
and the result would be in Distribution.options = {'path1': xxx, 'path2': xx}
Tarek
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Nov 12, 2009, at 12:42 PM, John Kleint wrote:
I have the start of such a document, derived from my own recent
experience pulling together all the information needed to release a
new Python project. It's aimed at people new to packaging and
possibly open source in general, so it's broader than just packaging.
It includes things like basic file layout, unit testing, Pylint,
getting started with Sphinx, a sample (distutils) setup.py, and how to
register with PyPI. I try to pull together all the info a novice
would need to have a reasonable shot at sharing their code. It's by
no means all-encompassing (e.g. no mention of native extensions), but
perhaps someone would be willing to write a complimentary advanced
packaging howto. You can find it at:
http://infinitemonkeycorps.net/docs/pph/
Let me know what you think.
It's a good start, but a few critques on the packaging section:
* Avoid importing your package before you've installed it.
Importing a package works fine for packages which have no
dependencies, but once your package depends upon other packages you
can't use this technique. It's better to teach people how to write
setup.py scripts which don't import the package so that their
setup.py's can be consistent once they do write packages with
dependencies. There are lots of cases where the assumptions of the
configuration of a developer's environment where dependencies are
installed that won't work for other people trying to install such a
package, which leads to setup.py files which don't work.
One example for handling the long_description field can be done
with (there are lots of variants on this, and you can just call open()
for a single text file:
def read(*rnames):
return open(os.path.join(os.path.dirname(__file__),
*rnames)).read()
long_description=read(
read('README.txt')
+ '\n' +
'Detailed Documentation\n'
'**\n'
+ '\n' +
'src/some-other-file.txt',
)
* download_url field is not necessary.
Libraries are often installed automatically by tools (buildout,
pip, easy_install). This work best if the downloads are in a
consistent structure (e.g. on PyPI). If releases are made to PyPI,
then you don't need a download_url to get to another place where the
file can be downloaded. The field is also prone to mistakes, since
easy_install will follow the download URL and screen-scrape that page
looking for a download link. Which can lead to older or wrong versions
of the package being downloaded. Basically it just causes room for
errors, and is just one extra step that you need to do to create a
package, so why do it?
* package_dir location.
This is a very minor point, but I find putting the source in a
directory named after the module or package which isn't the module or
package confusing. A common alternative is to use 'src' as the name
for the source directory. Then that location can be consistent from
package to package.
* provides (and requires) are scheduled for deprecation
These fields don't make any sense. If you are listing the imports
that your package needs, you need to list the name of the distribution
which provides those packages, not the names of the imports that your
code is using. setuptools (or now distribute) lets you use an
'install_requires' field, and this fields is also likely to be in
Distutils in the next Python release.
* Use a tool to help automate releases
Creating each release requires a lot of manual changes here and
there (bump the version, update the changelog, run sdist, upload to
pypi). In the spirit of release early, release often, I like to use
a tool to automate releases, and also helps establish a few more
common conventions between Python projects. I'm using zest.releaser
for this ATM (shameless plug, since I'm also a contributor to the
project):
http://pypi.python.org/pypi/zest.releaser
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
Tarek Ziadé wrote: A deprecation warning would be added in install, if it finds a local option, rather than a global. Meaning both would work in 2.7/3.2. If changing the command line in incompatible ways is acceptable, what do you think of scrapping the commands (at the UI level only) altogether ? This would be more consistent and easier to deal for the user, and easier to implement as well: python setup.py configure --option1 --option2=value2 python setup.py build python setup.py install We could then make this work: python setup.py install (would run both build and configure implicitly). Making all finalized options available at the build stage would then be easier. David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Wed, Nov 11, 2009 at 7:14 AM, Greg Ewing greg.ew...@canterbury.ac.nz wrote: David Cournapeau wrote: One of the main argument to avoid rewrite is that you will end up doing the same mistakes, and old code is more complicated because it has been tested. But here, we know what a good design is like as other languages have vastly superior solutions to this problem. Also, it seems to me that in this case, the basic architecture of distutils is already so full of mistakes that there just isn't an incremental way of getting to a better place, especially given the requirement of not breaking any existing setup.py scripts together with the fact that the API of distutils effectively consists of its *entire implementation*. So while complete rewrites are best avoided *if possible*, I don't think we have a choice in this case. While build_ext is not handy, I don't buy the fact that Distutils is full of mistakes. We have to work with use cases. David gave a use case: being able to compile cython or assembly files. I proposed a solution based on being able to define a compiler at the Extension level, rather that for the entire build_ext command. If the answer to that solution is just: Distutils sucks anyways.., it is not really helpfull imho.. I don't see the point to write Distutils from scratch, instead of making it evolve. Tarek ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Wed, Nov 11, 2009 at 7:05 AM, David Cournapeau da...@ar.media.kyoto-u.ac.jp wrote: [..] Exactly. The fact that we in numpy consider distutils backward compatibility not worth the cost, even though we are most likely the most tied up with distutils, is quite telling about the state of affairs IMHO. That doesn't prove Distutils can't evolve. That just shows that numpy worked on its side. If the Numpy projects made some refactoring/improvement, why the project didn't try to push it back in Distutils itself ? ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
To the people who wants a rewrite, two things need to be asked: 1. Do you think the new PEPs in development should be followed? In that case, what is the benefit of rewriting, instead of fixing? 2. When are you done? :-) I'm not being rude, but this is open source. There is no Someone that can rewrite distutils from scratch, it must be done by those who thinks it should be done. Those who think distutils should be rewritten from scratch need to sit down and do it. If nobody is willing to write the code, the code is not needed. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
Tarek Ziadé wrote: On Wed, Nov 11, 2009 at 7:05 AM, David Cournapeau da...@ar.media.kyoto-u.ac.jp wrote: [..] Exactly. The fact that we in numpy consider distutils backward compatibility not worth the cost, even though we are most likely the most tied up with distutils, is quite telling about the state of affairs IMHO. That doesn't prove Distutils can't evolve. No, but that's not the point I was trying to make: I meant that we consider the distutils API not to be a significant asset, and would be happy to throw away all numpy.distutils for a significantly better system. We now have two build systems in numpy (one based on scons): I think it takes me 5 to 10 times more effort on average to add a feature to the distutils-based compared to the scons system. There are some features I can not implement because I have not find a solution to it in distutils. There are arbitrary limitations like the inability to call commands directly from setup.py, retrieve global informations from setup.py, classes which behave differently on different platforms. Example: how to retrieve the install prefix in setup.py. You need a good understanding of distutils to understand why it is so complicated, and the example shows almost everything that's wrong in distutils design. Many expectations are undocumented, like what method/class can be called where and when. All this implicit behavior is part of the API, and that's not documented anywhere that I know of. Also, what happened in python 2.6.3/2.6.4 w.r.t. distribute has happened quite often for numpy.distutils, and I consider it inherent to distutils way of working. If the Numpy projects made some refactoring/improvement, why the project didn't try to push it back in Distutils itself ? They are not improvements or refactoring for most part. They are things quite specific to our needs: fortran support, support for our internal code generator, swig, f2py, etc... A few things could be useful to distutils, like manifest support for mingw for 2.6 and later, as well as basic autoconf-like tests (checking for functions, type, type size, etc...). We would be happy to contribute patches to Distribute if this is considered useful. cheers, David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Wed, Nov 11, 2009 at 10:46 AM, David Cournapeau da...@ar.media.kyoto-u.ac.jp wrote: [..] Example: how to retrieve the install prefix in setup.py. You need a good understanding of distutils to understand why it is so complicated, and the example shows almost everything that's wrong in distutils design. Many expectations are undocumented, like what method/class can be called where and when. All this implicit behavior is part of the API, and that's not documented anywhere that I know of. Ok, so I read this example as a lack of documentation and the lack of clear APIs to get the installation paths. Also, notice that we are in the process of adding a new python module in the stdlib, called sysconfig, that will contains all installation paths scheme for all supported platforms. I have a branch in the svn going on for that. Knowing where to install things and what is up with a given platform is a wider problem that Distutils in fact, it concerns site.py as well, and having a sysconfig module that handles this will help. I also expect it to help the work people from Jython, PyPy, and IronPython will do in the future. Also, what happened in python 2.6.3/2.6.4 w.r.t. distribute has happened quite often for numpy.distutils, and I consider it inherent to distutils way of working. I fully agree that this particular example is demonstrating that build_ext is suffering from a lack of API documentation. I am still waiting for you comment on the solution to remove the compile code from build_ext and have it at the Extension level, with an option to add new compilers in a more easier way. If the Numpy projects made some refactoring/improvement, why the project didn't try to push it back in Distutils itself ? They are not improvements or refactoring for most part. They are things quite specific to our needs: fortran support, support for our internal code generator, swig, f2py, etc... A few things could be useful to distutils, like manifest support for mingw for 2.6 and later, as well as basic autoconf-like tests (checking for functions, type, type size, etc...). We would be happy to contribute patches to Distribute if this is considered useful. If there's someone from the Numpy project that can help in isolating patches againts Distutils trunk in our issue tracker, I'd be more than happy to reduce the gap between the two projects. Notice that this may have been done in the past, I didn't manage to review all the distutils bugs yet (some are 5 years old) Tarek ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Wed, 11 Nov 2009 19:14:42 +1300, Greg Ewing greg.ew...@canterbury.ac.nz wrote: Also, it seems to me that in this case, the basic architecture of distutils is already so full of mistakes that there just isn't an incremental way of getting to a better place, especially given the requirement of not breaking any existing setup.py scripts together with the fact that the API of distutils effectively consists of its *entire implementation*. So while complete rewrites are best avoided *if possible*, I don't think we have a choice in this case. Fwiw, I can't see any way to make that work. Even if I said that I didn't like distutils, which btw isn't true because my knowledge of it is so limited, it's just not practical to rewrite such a complex tool within any reasonable timeframe. Anyway, moving towards CPAN isn't about rewriting all of distutils, only a few key parts. Or restructuring. The parts that shouldn't be rewritten (because the people that wrote it were too clever - and the degree of difficulty to high) is any C interface. That's an important part of distutils that is likely to end up worse rather than better. If there are candidates for 'evolution', it has to be in the areas such as package creation (collecting all the files and putting them in a .tar.gz or .egg or .zip or .exe) for upload to pypi. In CPAN, one wouldn't dream of there being so many possibilities. As I've said on python-dev, we should just revise and call a package an egg. In science, it wouldn't be tolerated to call a hydrogen atom using 6 different synonyms. But in python we do. Then we wonder why people get confused... There's a big difference between a rewrite and an evolution. Rewrite isn't viable, but an evolution is. From science, we even know that it only takes two generations to become nearly immune from strong radiation as shown in chernoble. So when we talk about evolution lets use it in the modern sense, not in the darwinian. Evolution is better than revolution is better than nothing happening at all. David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
Tarek Ziadé ziade.tarek at gmail.com writes: Also, notice that we are in the process of adding a new python module in the stdlib, called sysconfig, that will contains all installation paths scheme for all supported platforms. I don't think we are talking about the same thing. If the pb was just missing API, we could have just added it to numpy.distutils. We override pretty much every command of distutils and setuptools anyway. I needed to obtain the install prefix, be it the default one, or the one set through --prefix, or the current directory for in place build/develop install. I need to know it when building C libraries, in build_clib. AFAIK, there is only one way to get the information: since the --prefix is only known once install.finalize_options() has run, you need to call the method from build_clib. Naive code could be something like class MyBuildClib(build_clib): def run(self): install_cmd = self.distribution.get_command_obj(install) if not install_cmd.finalized == 1: install_cmd.finalize_options() if self.inplace == 1: top = . else: top = install_cmd.install_libbase # top is the top directory where libraries should be installed build_clib.run(self) It is already quite ugly, but it looks like it could work. It actually does in the simple case, like python setup.py install. But if you call python setup.py build_clib install --prefix=something, it does not. The error message is a typical example of distutils style: error: must supply either prefix/exec- prefix/home or install-base/install-platbase -- not both. The only way I managed to make this work is to replace: install_cmd = self.distribution.get_command_obj(install)) by install_cmd = copy.copy(self.distribution.get_command_obj(install)) That's not an API problem, or a documentation problem. That's a fundamental issue because --prefix is only known from install, and when you start running commands in an order different than expected by distutils, you get weird errors (the above error is actually almost sensical if you really know distutils code). I passed over the fact that in some conditions that elude me ATM and are platform specific, install_libbase does not exist and raises an attribute error. If you have a configure / build / install like the vast majority of build systems out there, this problem disappears. I don't see how the problem can be fixed without touching how commands work. Moreover, that's a typical example where the only way to be robust is to check that every attribute you are using actually exist before. At that point, you really want to run back to m4, perl, autogenerated makefiles and shell programming :) I am still waiting for you comment on the solution to remove the compile code from build_ext and have it at the Extension level, with an option to add new compilers in a more easier way. I will try to document how scons does it. I think the basic idea could be reused in distutils. If there's someone from the Numpy project that can help in isolating patches againts Distutils trunk in our issue tracker, I'd be more than happy to reduce the gap between the two projects. If that was not clear, I am that guy. I have been the main maintainer of numpy.distutils and of numpy/scipy build infrastructure for some time now, David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Nov 11, 2009, at 4:39 AM, Lennart Regebro wrote: If nobody is willing to write the code, the code is not needed. I think it would be more accurate to say that nobody is deciding that the need is sufficient to invest the resources to fill it. There are lots of things that many people would be willing and able to write, that would fill a great need, and that many people would benefit from, but that aren't getting written because there's no one willing to invest the time or money. In come cases people are both willing and able, but unable to invest the time due to the pressing need to make a living. I'd love to work on open source all day, but my wife and kids get cranky if they don't eat for a couple of days. People don't always agree on what is the rational need to fill, and sometimes what we, as programmers need, and know would make us more productive, is not what the people controlling the pursestrings are willing to finance, even if it would ultimately benefit them financially. S ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
2009/11/11 sstein...@gmail.com sstein...@gmail.com: On Nov 11, 2009, at 4:39 AM, Lennart Regebro wrote: If nobody is willing to write the code, the code is not needed. I think it would be more accurate to say that nobody is deciding that the need is sufficient to invest the resources to fill it. More accurate, but longer and redundant. :) -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Wed, Nov 11, 2009 at 1:31 PM, David Cournapeau courn...@gmail.com wrote: [..] AFAIK, there is only one way to get the information: since the --prefix is only known once install.finalize_options() has run, you need to call the method from build_clib. Naive code could be something like class MyBuildClib(build_clib): def run(self): install_cmd = self.distribution.get_command_obj(install) if not install_cmd.finalized == 1: install_cmd.finalize_options() if self.inplace == 1: top = . else: top = install_cmd.install_libbase # top is the top directory where libraries should be installed build_clib.run(self) It is already quite ugly, but it looks like it could work. It actually does in the simple case, like python setup.py install. But if you call python setup.py build_clib install --prefix=something, it does not. The error message is a typical example of distutils style: error: must supply either prefix/exec- prefix/home or install-base/install-platbase -- not both. The only way I managed to make this work is to replace: Ouch, That's not to be done. Something is wrong with your build_clib design here. You are roughly calling install as a sub command. If you want to have install options for your command, and if your command is about installing, it means that your command has to be a subcommand of install. Those get called once the options passed to install have been finalized. IOW, build_clib is not a subcommand of install, so you have troubles. subcommands are: install_lib, install_headers, install_scripts, install_data, install_egg_info (and I agree that it's not simple to extend this list) But, why in the first place, do you need the install --prefix in a build command ? For build_clib, if you need to build something, it goes in the build_dir, or in place., and this is not impacted by the install command. If you have a configure / build / install like the vast majority of build systems out there, this problem disappears. I don't see how the problem can be fixed without touching how commands work. I fail to understand your demonstration. Commands that are in charge of the *build* have nothing to do with commands that are in charge of *installing* file in various places in the target system. So I fail to understand why build_clib interacts with install, and why it has to impact it howsoever (or vice-versa) Now, if we take the generic use case (even if I don't think it should be used in your case): a simple way to share options amongst commands As a matter of fact, there's a hackish way to perform this, by using the distribution instance as a placeholder for these common options, so several command can read/write it. (as opposed to local options and global options) without having to get the command that manage the option. But, at the end, since an option is either global, either specific to a command, I guess a simple API in the command class should be enough to avoid this hack: get_option(command_name, option_name) This is similar to getting the command, (instanciate it + finalize it if it doesn't exists yet) and return a finalized option. Moreover, that's a typical example where the only way to be robust is to check that every attribute you are using actually exist before. At that point, you really want to run back to m4, perl, autogenerated makefiles and shell programming :) I am still waiting for you comment on the solution to remove the compile code from build_ext and have it at the Extension level, with an option to add new compilers in a more easier way. I will try to document how scons does it. I think the basic idea could be reused in distutils. But, you didn't answer: if we add the ability to define a different compiler for each Extension, will it solve your use case ? If there's someone from the Numpy project that can help in isolating patches againts Distutils trunk in our issue tracker, I'd be more than happy to reduce the gap between the two projects. If that was not clear, I am that guy. I have been the main maintainer of numpy.distutils and of numpy/scipy build infrastructure for some time now, If you are willing to spend some time in that, I am the guy who can commit your patches in Python. Regards Tarek ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Nov 11, 2009, at 7:38 AM, Lennart Regebro wrote: 2009/11/11 sstein...@gmail.com sstein...@gmail.com: On Nov 11, 2009, at 4:39 AM, Lennart Regebro wrote: If nobody is willing to write the code, the code is not needed. I think it would be more accurate to say that nobody is deciding that the need is sufficient to invest the resources to fill it. More accurate, but longer and redundant. :) I don't think it's redundant because a lot of needs in Python go unmet, not due to people's inability or unwillingness, but due to a lack of time/funding which amount to the same thing. S ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Wed, Nov 11, 2009 at 10:16 PM, Tarek Ziadé ziade.ta...@gmail.com wrote: Ouch, That's not to be done. Something is wrong with your build_clib design here. You are roughly calling install as a sub command. Yes, it is wrong from distutils POV, but there is no other solution to get the prefix. The problem is that prefix is available in install, not that I want to know the prefix when building. If you want to have install options for your command, and if your command is about installing, I did not say my command was about installing - it does not install anything. To be complete, we do not use this in build_clib, but in build_src , to generate pkg-config-like files (build_src is a numpy.distutils-specific command to build source files, for example automatically generated wrappers around fortran libraries, and is typically run before any other build_ command) For build_clib, if you need to build something, it goes in the build_dir, or in place., and this is not impacted by the install command. That's exactly the problem. build + install description is too simplistic for complex builds: you often need to know install options at build time. You need to know some build options when building configuration files, you need to pass configuration options to other subsequent steps, etc... This is similar to getting the command, (instanciate it + finalize it if it doesn't exists yet) and return a finalized option. This does not solve the issue IMHO. Since we both seem to like thinking about use-cases, consider this use-case: you have a python package with a complex extension built with make (say you have a build_make command which calls a makefile). How do you do it ? How to communicate path informations, compiler options between make and distutils ? How to handle relinking (changing rpath at install time) ? I will try to document how scons does it. I think the basic idea could be reused in distutils. But, you didn't answer: if we add the ability to define a different compiler for each Extension, will it solve your use case ? I did not answer because my answer will take time to accurately summarize tools problems in build tools. We have discussed quite a bit about those issues in scons as well and I want to be sure that my answer takes this into account as well. If you are willing to spend some time in that, I am the guy who can commit your patches in Python. I will try to prepare a couple of patches against the hg repo this WE, David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Wed, Nov 11, 2009 at 2:48 PM, David Cournapeau courn...@gmail.com wrote: If you want to have install options for your command, and if your command is about installing, I did not say my command was about installing - it does not install anything. To be complete, we do not use this in build_clib, but in build_src , to generate pkg-config-like files (build_src is a numpy.distutils-specific command to build source files, for example automatically generated wrappers around fortran libraries, and is typically run before any other build_ command) But you call it with install in your example, meaning that is is called at install time, right ? Or it is just that you want to get the --prefix value finalized and computed by the install command. If it's the later, I guess you will be able to use the upcoming sysconfig module, that gives you the install schemes, depending on sys.prefix/sys.exec_prefix. And I will probably add a way to override those prefix, meaning that you will be able to get from your command all install paths depending on a root prefix. If not, it means that you are doing a post or pre install step during installation. e.g. like RPM's pre/post commits hooks. For build_clib, if you need to build something, it goes in the build_dir, or in place., and this is not impacted by the install command. That's exactly the problem. build + install description is too simplistic for complex builds: you often need to know install options at build time. You need to know some build options when building configuration files, you need to pass configuration options to other subsequent steps, etc... Sorry, I can see it yet, it still fuzzy. Does that mean your binary distribution will not be relocatable ? e.g. meaning that once you have done your build for a given --prefix, it won't be installable anywhere else ? In that case you would need to remove the --prefix option from install, and have it on your build command. Sounds like a pre/post install hook but I am not sure. This is similar to getting the command, (instanciate it + finalize it if it doesn't exists yet) and return a finalized option. This does not solve the issue IMHO. Since we both seem to like thinking about use-cases, consider this use-case: you have a python package with a complex extension built with make (say you have a build_make command which calls a makefile). How do you do it ? How to communicate path informations, compiler options between make and distutils ? How to handle relinking (changing rpath at install time) ? I don't know for the first part. I have to try it out. Can you provide me such an extension ? For the relinking at installation time problem, it is obvious that something can be done. we could have a pre/post install option where an arbitrary command could be launch, as an install subcommand. pre: works on the build_dir built by install or a previous build. post: works on the installed file list [..] If you are willing to spend some time in that, I am the guy who can commit your patches in Python. I will try to prepare a couple of patches against the hg repo this WE, yeah, thanks ! \o/ Tarek ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Wed, Nov 11, 2009 at 14:16 +0100, Tarek Ziadé wrote: On Wed, Nov 11, 2009 at 1:31 PM, David Cournapeau courn...@gmail.com wrote: But, at the end, since an option is either global, either specific to a command, I guess a simple API in the command class should be enough to avoid this hack: get_option(command_name, option_name) +1 -- .''`. Wolodja Wentlandwentl...@cl.uni-heidelberg.de : :' : `. `'` 4096R/CAF14EFC `- 081C B7CD FF04 2BA9 94EA 36B2 8B7F 7D30 CAF1 4EFC signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
2009/11/11 sstein...@gmail.com sstein...@gmail.com: I don't think it's redundant because a lot of needs in Python go unmet, not due to people's inability or unwillingness, but due to a lack of time/funding which amount to the same thing. People *want* a lot of things. But if they truly *need* it, they'll do it. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
In line with this discussion, I found a document that details the aspects of CPAN that can be used for writing packaging systems in other languages. The author says: over the years people from at least Python, Ruby, and Java communities have approached me or other core CPAN people to ask basically How did we do it? http://www.cpan.org/misc/ZCAN.html -srid On Fri, 06 Nov 2009 09:53:44 -0800, Guido van Rossum gu...@python.org wrote: I just found this comment on my blog. People have told me this in person too, so I believe it is real pain (even if the solution may be elusive and the suggested solutions may not work). But I don't know how to improve the world. Is the work on distutils-sig going to be enough? Or do we need some other kind of work in addition? Do we need more than PyPI? --Guido -- Forwarded message -- From: dalloliogm noreply-comm...@blogger.com Date: Fri, Nov 6, 2009 at 8:01 AM Subject: [Neopythonic] New comment on Python in the Scientific World. To: gvanros...@gmail.com dalloliogm has left a new comment on your post Python in the Scientific World: Python is suffering a lot in the scientific word, because it has not a CPAN-like repository. PyPI is fine, but it is still far from the level of CPAN, CRAN, Bioconductor, etc.. Scientists who use programming usually have a lot of different interests and approaches, therefore it is really difficult to write a package that can be useful to everyone. Other programming language like Perl and R have repository-like structure which enable people to download packages easily, and to upload new ones and organize them withouth having to worry about having to integrate them into existing packages. This is what is happening to biopython now: it is a monolitic package that it is supposed to work for any bioinformatic problem; but this is so general that to accomplish that you would need to add a lot of dependencies, to numpy, networkx, suds, any kind of library. However, since easy_install is not as ready yet as the counterparts in other languages, if the biopython developers add too many dependencies, nobody will be able to install it properly, and nobody will use it. Posted by dalloliogm to Neopythonic at November 6, 2009 8:01 AM ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
ke, 2009-11-11 kello 16:47 +0100, Tarek Ziadé kirjoitti:
[clip]
If it's the later, I guess you will be able to use the
upcoming sysconfig module,
that gives you the install schemes, depending on
sys.prefix/sys.exec_prefix.
Where is the sysconfig sources ? I don't see it in bitbucket.
That's in python's svn, in a tarek_sysconfig branch. It's a revamp of
distutils/sysconfig.py with the schemes from distutils/command/install.py
(work in progress)
What if the user passes a different install prefix via
python setup.py install --prefix=FOO
I believe David would like to know FOO here. Since sysconf is not a part
of distutils, will it know what FOO is?
[clip]
For me, one of the core idea of an improved distutils would be to make
this much easier. All compilers options form distutils would be in
simple data files with simple API, no objects, no class with countless
methods and complex protocol. Distutils v2 would have a default dumb
build tool, but you could use whatever tool instead if desired.
The default compiler class exists for that, it's CCompiler, and is
mostly a placeholder for options.
But it's C oriented. In my mind, implementing a new compiler for
Distutils means overriding it, and implementing, mainly:
- preprocess()
- compile()
- create_static_lib()
- link()
Now that's quite complex, and we could probably have a single method
(compile) that would do the required work of compiling an extension
the way it wants to.
I think one question here is that how do the different compilers speak
to each other?
Consider the following chain needed to compile a Cython file linking to
some Fortran and C files to a Python module:
- cython foo.pyx - foo.c
- compile foo.c - foo.o
- compile bar.c - bar.o
- compile quux.f90 - quux.o
- link foo.o bar.o quux.o - foo.so
This is a completely possible use-case, so it would be good if distutils
could handle it. Also, dependency handling would be nice here. Changing
bar.c or foo.pyx should trigger relinking foo.so etc.
So, yes, being able to register an arbitrary compiler class, with
arbitrary options passed through the Extension
could make it simpler:
setup(
..
ext_modules=[Extension('Foo', files=['foo.d'], compiler='pyd')],
..)
Does this work easily out when there are multiple source files (in
different languages) for a single extension module?
Also, the option of determining the necessary compiler and compiler
options from file extensions does not always work. For example, Fortran
90 files come in two flavors, fixed-form and free-form, which often are
not differentiated by the file extension. However, they may require
different compiler flags to compile. (Hopefully, however, nobody uses
the fixed-form for F90 any more...)
Cheers,
--
Pauli Virtanen
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Wed, Nov 11, 2009 at 9:22 PM, Pauli Virtanen p...@iki.fi wrote:
ke, 2009-11-11 kello 16:47 +0100, Tarek Ziadé kirjoitti:
[clip]
If it's the later, I guess you will be able to use the
upcoming sysconfig module,
that gives you the install schemes, depending on
sys.prefix/sys.exec_prefix.
Where is the sysconfig sources ? I don't see it in bitbucket.
That's in python's svn, in a tarek_sysconfig branch. It's a revamp of
distutils/sysconfig.py with the schemes from distutils/command/install.py
(work in progress)
What if the user passes a different install prefix via
python setup.py install --prefix=FOO
I believe David would like to know FOO here. Since sysconf is not a part
of distutils, will it know what FOO is?
I am not sure he wants FOO, I think he wants all installation paths,
that gets built
by the install command with the provided FOO root prefix.
that could be in pseudo code:
get_install_paths('FOO')
And that's the API we want to add in sysconfig, roughly.
[..]
I think one question here is that how do the different compilers speak
to each other?
Consider the following chain needed to compile a Cython file linking to
some Fortran and C files to a Python module:
- cython foo.pyx - foo.c
- compile foo.c - foo.o
- compile bar.c - bar.o
- compile quux.f90 - quux.o
- link foo.o bar.o quux.o - foo.so
This is a completely possible use-case, so it would be good if distutils
could handle it. Also, dependency handling would be nice here. Changing
bar.c or foo.pyx should trigger relinking foo.so etc.
So, yes, being able to register an arbitrary compiler class, with
arbitrary options passed through the Extension
could make it simpler:
setup(
..
ext_modules=[Extension('Foo', files=['foo.d'], compiler='pyd')],
..)
Does this work easily out when there are multiple source files (in
different languages) for a single extension module?
Do you mean, an Extension that would require several compilers ?
I was thinking of a one-to-one relation between an Extension and a
compiler type,
even if there are are multiple source files (in different languages)
for this extension.
Meaning that *one* compiler would have to handle of those files. Does that fit
the cython use case ?
Also, the option of determining the necessary compiler and compiler
options from file extensions does not always work. For example, Fortran
90 files come in two flavors, fixed-form and free-form, which often are
not differentiated by the file extension. However, they may require
different compiler flags to compile. (Hopefully, however, nobody uses
the fixed-form for F90 any more...)
How do they do then ? Is file extensions, inline options in setup.py,
and the environ, are enough ?
regards
Tarek
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
Tarek Ziadé wrote: If the answer to that solution is just: Distutils sucks anyways.., it is not really helpfull imho.. I don't see the point to write Distutils from scratch, instead of making it evolve. If you can see a way to get from the current distutils code to something with a well-designed and well-documented API and a clean implementation, that would be fine by me. When I say it's fundamentally broken, I'm really talking about the API. My idea of what an API for a build system should be like is more like make or scons, which slice the functionality up in a completely orthogonal direction to the way distutils does. Maybe it would be possible to plug such a system in under the existing build_ext class. I don't know. I think I would like the same philosophy applied to other areas of distutils, not just compiling extensions. Otherwise it would feel like two incompatible systems bolted together. What we need right now, I think, is some discussion about a new API, unconstrained by any considerations of backwards compatibility or reuse of existing distutils code. Once we know where we're going, then we can think about the best way to get there. -- Greg ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On 2009-11-11 15:02 PM, Greg Ewing wrote: Tarek Ziadé wrote: If the answer to that solution is just: Distutils sucks anyways.., it is not really helpfull imho.. I don't see the point to write Distutils from scratch, instead of making it evolve. If you can see a way to get from the current distutils code to something with a well-designed and well-documented API and a clean implementation, that would be fine by me. When I say it's fundamentally broken, I'm really talking about the API. My idea of what an API for a build system should be like is more like make or scons, which slice the functionality up in a completely orthogonal direction to the way distutils does. Maybe it would be possible to plug such a system in under the existing build_ext class. I don't know. In fact, David has. http://pypi.python.org/pypi/numscons/ I think I would like the same philosophy applied to other areas of distutils, not just compiling extensions. Otherwise it would feel like two incompatible systems bolted together. It does feel something like that. The build system is just one of the problems with distutils' internals, in my experience. You can think of the rest of distutils as a little application framework for command line utilities. I think this framework simply fails to provide in very fundamental ways, like the extend commands by subclassing design pattern. That choice makes it fundamentally difficult to combine extensions together. I really don't see a way to evolve away from that (and believe me, over the last decade, I've tried). You just need to redesign the internals if you want to get away from that. You can't get from any point A to any point B by evolving in small steps that are functional (not to mention backwards compatible!) all the way through. With all respect to Greg Ward and the rest of the original distutils authors, it was a fantastic improvement over the state of affairs ten years ago, but we've learned a lot about application frameworks and about building software since then. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Wed, Nov 11, 2009 at 10:36 PM, Robert Kern robert.k...@gmail.com wrote: [..] It does feel something like that. The build system is just one of the problems with distutils' internals, in my experience. You can think of the rest of distutils as a little application framework for command line utilities. I think this framework simply fails to provide in very fundamental ways, like the extend commands by subclassing design pattern. That choice makes it fundamentally difficult to combine extensions together. I really don't see a way to evolve away from that (and believe me, over the last decade, I've tried). You just need to redesign the internals if you want to get away from that. You can't get from any point A to any point B by evolving in small steps that are functional (not to mention backwards compatible!) all the way through. I am very surprised about this statement. What did you tried for the paste decade and failed to do ? I hear some complaints since a week, but beside's David examples I didn't read any other precise use cases. We're looking through the build_ext use case, and we are making some improvement on the other thread. So why not doing this in other issues ? Let's discuss your use case. And if it means adding new options to run arbitrary commands like post/pre hooks to a given command, to avoid subclassing an existing command, let's do it. And let's drop the backward compat issues in these discussions, so we don't burn out in details. Tarek ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On 2009-11-11 15:59 PM, Tarek Ziadé wrote: On Wed, Nov 11, 2009 at 10:36 PM, Robert Kernrobert.k...@gmail.com wrote: [..] It does feel something like that. The build system is just one of the problems with distutils' internals, in my experience. You can think of the rest of distutils as a little application framework for command line utilities. I think this framework simply fails to provide in very fundamental ways, like the extend commands by subclassing design pattern. That choice makes it fundamentally difficult to combine extensions together. I really don't see a way to evolve away from that (and believe me, over the last decade, I've tried). You just need to redesign the internals if you want to get away from that. You can't get from any point A to any point B by evolving in small steps that are functional (not to mention backwards compatible!) all the way through. I am very surprised about this statement. What did you tried for the paste decade and failed to do ? I hear some complaints since a week, but beside's David examples I didn't read any other precise use cases. We're looking through the build_ext use case, and we are making some improvement on the other thread. So why not doing this in other issues ? Let's discuss your use case. And if it means adding new options to run arbitrary commands like post/pre hooks to a given command, to avoid subclassing an existing command, let's do it. http://svn.scipy.org/svn/numpy/trunk/numpy/distutils/command/ All of it. Now consider that here we are also trying to play nicely with the setuptools extensions, and Pyrex, and David is working on integrating Cython support. To get to one real specific problem, let's consider build_src. build_src is a new subcommand in numpy.distutils that builds C extension sources from other files. We use this to hook in f2py's wrapper generator and other more ad hoc forms of generating wrappers. When build_ext uses --inplace, we need build_src to use --inplace, too, because it will often output some final products in addition to intermediate wrapper sources. In order to integrate this with setuptools' develop command (which invokes build_ext --inplace but not build_src --inplace because setuptools knows nothing about numpy.distutils), we need to create a subclass of setuptool's develop command that will reinitialize build_src with the appropriate option. Then we need to conditionally place the develop command into the set of command classes so as not to introduce a setuptools dependency on those people who don't want to use it. This is nuts. numpy.distutils shouldn't have to know anything about setuptools to accomplish this if the framework were properly designed. And this doesn't even get into the fact that many of the numpy.distutils command classes that are shared with setuptools are conditional subclasses and probably still buggily cobbled together. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Thu, Nov 12, 2009 at 5:42 AM, Tarek Ziadé ziade.ta...@gmail.com wrote:
I am not sure he wants FOO, I think he wants all installation paths,
that gets built
by the install command with the provided FOO root prefix.
that could be in pseudo code:
get_install_paths('FOO')
And that's the API we want to add in sysconfig, roughly.
That does not solve the problem about getting FOO in the first place
when calling get_install_path.
Having a function which gives you all the build directories as well as
the install paths would already be great, though. Right now, for
numscons, I needed to reimplement all the logic, with the different
paths in different modes (inplace vs standard, Release vs Debug on
windows, etc...). Some of them are even python version dependent IIRC.
Do you mean, an Extension that would require several compilers ?
I was thinking of a one-to-one relation between an Extension and a
compiler type,
even if there are are multiple source files (in different languages)
for this extension.
This cannot work - you need different tools at different stages. I
don't think any imperative method can work here. Note also that
extension alone is not enough to trigger the right tool: it should be
overridable on a per extension-basis (actually, it should be
overridable on a per source basis if needed).
I think compiler class and the likes should simply be removed of the
picture here. You need tasks to transform a source into a target, and
tasks would use compiler configuration. There should not be any
objects/classes for compilers, it it not flexible enough. Although
the details differ in significant ways, both waf and scons use strings
for command lines (waf compile them for performance reason),
consisting of different parts which can be altered at will. In scons,
the task to compile C code is something like
$CC $CFLAGS $CPPDEFINES $CPPPATH -c $SOURCE $TARGET
You need to be able to control those variable content in a very fine
grained manner: prepending and appending may lead to different
compiler behavior, for example. This is especially important when
linking, where the wrong order may be the different between a working
extension and a crashing extension.
You cannot obtain this with classes and objects (especially when you
start talking about performance: thousand of source files for one
extension is not a crazy usercase).
David
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
Hi Greg, On Thu, 12 Nov 2009 10:02:18 +1300, Greg Ewing greg.ew...@canterbury.ac.nz wrote: What we need right now, I think, is some discussion about a new API, unconstrained by any considerations of backwards compatibility A new API isn't so hard, but like anything it takes time and effort to do. Already this year, *a lot* of discussion went into such a thing. For example, doing metadata based installs as an option to traditional installs. Backwards compatibility is going to be an issue that *does* need to be addressed but even that I don't believe is so hard. There's two halves to backwards compatability. At the build side and at the install side. It's confusing I know. Where distutils/setup is confusing is at the build side in the way that all the files have to be specified for being picked up. On the install side, setup.py is much simpler and hardly* does anything at all except copy files in from the sources.txt. So compatability there is easier. It's my opinion that a boilerplate setup.py could completely replace 9/10 user written setup.py files. When I say it's fundamentally broken, I'm really talking about the API. My idea of what an API for a build system should be like is more like make or scons, which slice the functionality up in a completely orthogonal direction to the way distutils does. yes. Broken on the build side. Maybe it would be possible to plug such a system in under the existing build_ext class. For even that to happen, there needs to be some work done. As in fleshing out some code and examples and documentation. Here's a configuration file for an alternate build system that I'm working on at: http://bitbucket.org/djlyon/pypi-package-builder Any help thrown at getting this working would be welcomed. Regards David setup.cfg [setup] name = artistflair version = 1.2 description = artistflair is a package written in spare time to show colour variations. [dependencies] packages = pyopengl objectrelationalmapper = 1.6 [dependencies python=2.4] packages = lxml = 2.5 [dependencies windows] packages = win32com [dependencies linux] packages = pyodbc [Application] scripts = artisticflairgui.py [configuration] files = artisticflair.conf [datafiles] files = artisticdb.db directory = db [Documentation] directory = docs url = index.html [preinstall] script = checksystem.py [postinstall linux] script = builddocumentation.py [uninstall] script = uninstallcleanup.py - ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Thu, Nov 12, 2009 at 12:03 AM, David Cournapeau courn...@gmail.com wrote:
On Thu, Nov 12, 2009 at 5:42 AM, Tarek Ziadé ziade.ta...@gmail.com wrote:
I am not sure he wants FOO, I think he wants all installation paths,
that gets built
by the install command with the provided FOO root prefix.
that could be in pseudo code:
get_install_paths('FOO')
And that's the API we want to add in sysconfig, roughly.
That does not solve the problem about getting FOO in the first place
when calling get_install_path.
Why that ? where FOO comes from ? if it's an option you provide at
build time like you said,
earlier, you just pass it to the API to get the paths.
FOO is not coming from nowhere...
[..]
Do you mean, an Extension that would require several compilers ?
I was thinking of a one-to-one relation between an Extension and a
compiler type,
even if there are are multiple source files (in different languages)
for this extension.
This cannot work - you need different tools at different stages. I
don't think any imperative method can work here. Note also that
extension alone is not enough to trigger the right tool: it should be
overridable on a per extension-basis (actually, it should be
overridable on a per source basis if needed).
I think compiler class and the likes should simply be removed of the
picture here. You need tasks to transform a source into a target, and
tasks would use compiler configuration. There should not be any
objects/classes for compilers, it it not flexible enough.
Although
the details differ in significant ways, both waf and scons use strings
for command lines (waf compile them for performance reason),
consisting of different parts which can be altered at will. In scons,
the task to compile C code is something like
$CC $CFLAGS $CPPDEFINES $CPPPATH -c $SOURCE $TARGET
You need to be able to control those variable content in a very fine
grained manner: prepending and appending may lead to different
compiler behavior, for example. This is especially important when
linking, where the wrong order may be the different between a working
extension and a crashing extension.
You cannot obtain this with classes and objects (especially when you
start talking about performance: thousand of source files for one
extension is not a crazy usercase).
Sorry, I am getting confused here. This is getting all mixed up.
What OOP has to do with performance in the first place ?
OOP is useful to describe patterns and resuability, and that's what
Distutils does.
OOP is not a bottleneck here for speed. (everything is an object in
Python anyways)
Now we are saying that the compiler pattern in Distutils doesn't fit
some requirement,
fine, let's see what other design we can have.
But that will be based on OOP with classes, and objects, and there
ought to be an object that
orchestrates the building of one given extension at the end.
That might be the existing Extension class. This class is instanciated
with a name for the
extension, and a list of files.
Here's a proposal to restrict the scope: let's drop the concept of
compiler class and let's work only with
the Extension class.
This class is used by a distutils command to build a given extension.
Let's say it is standalone, and it has a unique build()
method. Could we take it from here and try to prototype this Extension class ?
Tarek
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Wed, Nov 11, 2009 at 11:22 PM, Robert Kern robert.k...@gmail.com wrote: [..] To get to one real specific problem, let's consider build_src. build_src is a new subcommand in numpy.distutils that builds C extension sources from other files. We use this to hook in f2py's wrapper generator and other more ad hoc forms of generating wrappers. When build_ext uses --inplace, we need build_src to use --inplace, too, because it will often output some final products in addition to intermediate wrapper sources. In order to integrate this with setuptools' develop command (which invokes build_ext --inplace but not build_src --inplace because setuptools knows nothing about numpy.distutils), we need to create a subclass of setuptool's develop command that will reinitialize build_src with the appropriate option. Then we need to conditionally place the develop command into the set of command classes so as not to introduce a setuptools dependency on those people who don't want to use it. This is nuts. This clearly indicates that we should be able to extend build_ext behaviour without subclassing it. And having the ability to drive a specific compiling from within an Extension subclass can solve this issue. numpy.distutils shouldn't have to know anything about setuptools to accomplish this if the framework were properly designed. And this doesn't even get into the fact that many of the numpy.distutils command classes that are shared with setuptools are conditional subclasses and probably still buggily cobbled together. Same goes with install, and I've proposed in the past the ability to run arbitrary commands as pre/post hooks for it. So we can configure this command instead of replacing it. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
Robert Kern wrote: With all respect to Greg Ward and the rest of the original distutils authors, it was a fantastic improvement over the state of affairs ten years ago, but we've learned a lot about application frameworks and about building software since then. I think we knew more about it *before* then as well. Make has been around for a lot longer, but distutils ignores everything we've learned from it. -- Greg ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
Tarek Ziadé wrote: On Thu, Nov 12, 2009 at 12:03 AM, David Cournapeau courn...@gmail.com wrote: You cannot obtain this with classes and objects (especially when you start talking about performance: thousand of source files for one extension is not a crazy usercase). Sorry, I am getting confused here. This is getting all mixed up. What OOP has to do with performance in the first place ? As far as I can tell David seems to be saying that instantiating a class for every file in the system would be too much overhead. I'm not convinced about that -- make builds a dependency graph with a node for every target before it goes to work, and I think scons does something similar (although I might be wrong, I haven't looked at scons very closely yet). $CC $CFLAGS $CPPDEFINES $CPPPATH -c $SOURCE $TARGET You need to be able to control those variable content in a very fine grained manner: prepending and appending may lead to different compiler behavior It's true that the current distutils compiler classes are lacking in flexibility here -- there's an option for extra flags, for example, but it sticks them all at the end of the command. Sometimes that's wrong -- I've had trouble trying to use it for MacOSX -F arguments, for example. But that's not to say that a class couldn't be devised that allowed the required flexibility without degenerating into just a string with textual substitutions. -- Greg ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
Greg Ewing wrote: As far as I can tell David seems to be saying that instantiating a class for every file in the system would be too much overhead. I'm not convinced about that -- make builds a dependency graph with a node for every target before it goes to work, and I think scons does something similar Yes, and scons has scalability problems because of this (both from CPU and memory POV). Waf has also an object per node, but it is aggressively optimized. If you are interested in the details, I can point you to the corresponding discussions in both scons and waf, where the main designers were involved. But that's not to say that a class couldn't be devised that allowed the required flexibility without degenerating into just a string with textual substitutions. I am not saying that's impossible, but both waf an scons use string susbtitution to do it (they do it differently though). So you would have to find a good reason not to do it IMHO. cheers, David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
Tarek Ziadé wrote: On Tue, Nov 10, 2009 at 2:31 AM, David Cournapeau da...@ar.media.kyoto-u.ac.jp wrote: [..] If you want to support a new source/target (say assembler), you need to deal at this stage (or use hacks to deal with the new file extension you want to deal with, and remove it later...). So you need to copy build_ext, you cannot extend it. If you look at msvccompiler vs msvc9compiler, both compile methods are almost the same and copied from each other. So here's a refactoring proposal : - build_ext doesn't handle a compiler instance anymore (therefore it doesn't have a compile_type anymore) - instead, the Extension class handles this compile type - right now we have a dict in ccompiler (compiler_class) that has all the compiler classes. let's make it extensible through a registery using distutils.cfg (like the commands) so: when build_ext is run, it compiles the Extensions by using this registery of compilers, and the type of compiler contained in each extension. It keeps instances in a pool to reuse them if needed/ What is important here is how to add new tools, without touching or impacting distutils in many places. In particular, what distutils expects from a tool should be clearly defined (right now it is implementation defined). Adding cython or assembly support are good use cases to refactor this I think. You just don't replace a stdlib package with its v2 like that, there is a deprecation process to have. Distutils is not living only through setup.py. It has some public APIs imported and used by people. I am aware about the usage of distutils: I don't think it has public API, though. It has functions that people use, but it is far from clear what is public from what is private. Many things we do numpy.distutils are clearly dependent on implementation details of distutils. The problem with moving forward distutils is not deprecation, the problem is when you break/remove the old API. At this point, you will break people's code who rely on the old API. If the changes are big enough (as they should be to warrant breaking the old API in the first place), it will require significant effort from distutils API users. So people will see this as a trade-off: does it worth my time to spend time to use the new version of distutils, what will it bring to me ? Without significant new features, this will be difficult. -1 I'm hearing this sometimes. That's what I thought too when I started to work with Distutils a year and a half ago. And I couldn't understand Guido when he was saying that redoing it from scratch would be a mistake. Now I agree with him. IIRC, the main argument was that a new system would mean giving up existing setup.py, but that can be worked around. I don't know whether Guido's opinion has changed since, but he wondered last year wether dropping backward compatibility was an option (http://www.opensubscriber.com/message/distutils-sig@python.org/10291496.html). Otherwise, a new system would look nothing like distutils. One of the main argument to avoid rewrite is that you will end up doing the same mistakes, and old code is more complicated because it has been tested. But here, we know what a good design is like as other languages have vastly superior solutions to this problem. As far as compilation is concerned at least, the distutils knowledge is vastly overblown. First, most of it comes from autoconf on unices. You have the MSVC tools knowledge, but that can be easily reused (in numscons, I added msvc support from the python 2.6 msvc9compiler, this was not difficult). Most other tools are rather obsolete - and if you break any API in distribute there, you will most likely lose them as well anyway (I am thinking about OS/2, Metrowerk kind of tools). Again, I don't mean to say that working on distribute is a mistake, or criticize what you do in any way. I just don't think it will solve any significant issue for the scientific python community. But enough stop-energy: at this point, I will just shut up, and continue working on my own idea - if they make sense, the scientific community is big enough so that the solution could be used there only, at least for some time. cheers, David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
David Cournapeau wrote: One of the main argument to avoid rewrite is that you will end up doing the same mistakes, and old code is more complicated because it has been tested. But here, we know what a good design is like as other languages have vastly superior solutions to this problem. Also, it seems to me that in this case, the basic architecture of distutils is already so full of mistakes that there just isn't an incremental way of getting to a better place, especially given the requirement of not breaking any existing setup.py scripts together with the fact that the API of distutils effectively consists of its *entire implementation*. So while complete rewrites are best avoided *if possible*, I don't think we have a choice in this case. -- Greg ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
Greg Ewing wrote: Also, it seems to me that in this case, the basic architecture of distutils is already so full of mistakes that there just isn't an incremental way of getting to a better place, especially given the requirement of not breaking any existing setup.py scripts together with the fact that the API of distutils effectively consists of its *entire implementation*. Exactly. The fact that we in numpy consider distutils backward compatibility not worth the cost, even though we are most likely the most tied up with distutils, is quite telling about the state of affairs IMHO. David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
Ian Bicking wrote: I don't think changing distutils to improve error output would be hard at all. It looks like there's a line in distutils.core that catches these exceptions (and doesn't look like it actually catches all exceptions?), and that can just be fixed. I agree this is one of the thing which can be improved without unintended consequences. Another topic that has come up: I do agree subclassing makes it really hard to have multiple lines of development (specifically something like Setuptools or Distribute, along with ad hoc development in setup.py files). But I think that can be resolved. Perhaps, for instance, Distribute can have implementations of commands like build, that can easily be customized or extended without subclassing (e.g., by pre-build or post-build functions). In numpy's case, we subclass the Distribution class mainly to add new data which are shared within commands (for example, to build pure C libraries made available to 3rd parties, or to keep track of the scons scripts). I have myself tried the pre/post hooks for commands (for numpy.distutils), but I did not go very far: the problem was almost always linked to commands which need to share knowledge from each other. OTOH, I tried this when I started pocking into numpy.distutils 2 years ago, maybe I could go further today. Some things are fixable in distutils: for example, to build things, you should be able to get rid of the imperative operations, and have instead of registry of extension - action (ala scons/waf). This would make adding new tools (cython, assembly, etc...) easier, as you would add tools through a registry instead of overriding build_ext as currently done by e.g. numpy.distutils or cython.distutils. Doing so while keeping backward compatibility would be hard, though. I'd really be shocked if a rewrite of distutils was necessary, or even necessary to simplify things. That's the opposite of my own experience. I think I have given several reasonable examples of shortcomings of distutils: I would be glad to hear that each of them has a simple solution which is backward-compatible in distutils. But another way to look at it is to ask: What is there in distutils that you would consider important to keep ? It is important that people who maintain packages do not have to rewrite their setup to a new hypothetical system: asking thousand of developers to rewrite their setup would be insane. But this can be done without being tied to distutils API. David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Sat, 07 Nov 2009 07:37:37 -0800, Tarek Ziadé ziade.ta...@gmail.com wrote: The solution for a better PyPI: - more checks, more restrictions - every package maintainer uploading something to PyPI should have a certain attitude that PyPI is a public resource where the content should met certain quality criteria and where each package has a certain responsibility to Python community. More checks would be nice, so we can provide QA rates or something similar. I don't think we should enforce any policy whatsoever though at PyPI. We can't force people that upload distributions to comply with some strict QA rules imho (no binary distro allowed if no sdist is present for example). I suggest that we check for valid metadata on the uploaded sdists at the least. If you visit http://pypm.activestate.com/ - most failed packages are due to the fact the sdist uploaded by the author misses certain files such as README.txt (that is read by setup.py) or setup.py/PKG-INFO itself. Without such quality policing, I can't see how tools like pip/easy_install could even install the package (let alone doing it in an user-friendly way). -srid ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
I suggest that we check for valid metadata on the uploaded sdists at the least. If you visit http://pypm.activestate.com/ - most failed packages are due to the fact the sdist uploaded by the author misses certain files such as README.txt (that is read by setup.py) or setup.py/PKG-INFO itself. Unfortunately we can't run arbitrary code on PyPI. So if someone ships a broken setup.py, there's nothing much we can do unless we are able to run it in some kind of jail. Some work was started with Steve Steiner on that topic, but we're using a buildbot. It's still experimental because running an arbitrary setup.py can fail for many reasons. Another thing: once PEP 345 has the required changes (having metadata fields with platform conditions) we will be able to do some checks without having to run any code for any field located in PKG-INFO In any case, I am still not convinced that these checks should be forced on PyPI side when the sdist is uploaded. I see this as a QA rating, because even if a project's setup.py is great, other things can be wrong in the project's code itself. Tarek On 11/9/09, Sridhar Ratnakumar sridh...@activestate.com wrote: On Sat, 07 Nov 2009 07:37:37 -0800, Tarek Ziadé ziade.ta...@gmail.com wrote: The solution for a better PyPI: - more checks, more restrictions - every package maintainer uploading something to PyPI should have a certain attitude that PyPI is a public resource where the content should met certain quality criteria and where each package has a certain responsibility to Python community. More checks would be nice, so we can provide QA rates or something similar. I don't think we should enforce any policy whatsoever though at PyPI. We can't force people that upload distributions to comply with some strict QA rules imho (no binary distro allowed if no sdist is present for example). I suggest that we check for valid metadata on the uploaded sdists at the least. If you visit http://pypm.activestate.com/ - most failed packages are due to the fact the sdist uploaded by the author misses certain files such as README.txt (that is read by setup.py) or setup.py/PKG-INFO itself. Without such quality policing, I can't see how tools like pip/easy_install could even install the package (let alone doing it in an user-friendly way). -srid -- Tarek Ziadé | http://ziade.org | オープンソースはすごい! | 开源传万世，因有你参与 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On 11/9/09, David Cournapeau da...@ar.media.kyoto-u.ac.jp wrote: [...] Some things are fixable in distutils: for example, to build things, you should be able to get rid of the imperative operations, and have instead of registry of extension - action (ala scons/waf). What is a registery of extension exactly ? Distutils let you register your own commands, you can use through the CLI. Can you provide more details ? I'd really be shocked if a rewrite of distutils was necessary, or even necessary to simplify things. That's the opposite of my own experience. I think I have given several reasonable examples of shortcomings of distutils: I would be glad to hear that each of them has a simple solution which is backward-compatible in distutils. I am in for making Distutils evolve, but I need very precise real world use cases not saying that Distutils shouldn't do imperative operations). Last, I am not sure why you want only backward-compatible changes in distutils. There's no plan to keep backward-compatibility if breaking it makes DIstutils better. We will have pending deprecation warnings, that's all. Tarek ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
Guido van Rossum guido at python.org writes: I just found this comment on my blog. People have told me this in person too, so I believe it is real pain (even if the solution may be elusive and the suggested solutions may not work). But I don't know how to improve the world. Is the work on distutils-sig going to be enough? Or do we need some other kind of work in addition? Do we need more than PyPI? --Guido I didn't see this mentioned in the other comments: Sometimes easy_install just gets stuck with an IncompleteRead error or something like that. (I may be butchering the error's name, sorry.) Then some part of the html of the page gets printed up to a seemingly arbitrary point, and the install fails. I'm not sure what causes that, but I've seen it happening at random for at least a couple of months. Sometimes it works, sometimes it fails with the incomplete read thing, and then if you try again it will fail at the same point of the html page. Try it a few hours later, it might work then. Not nice. (I've seen it happen on both Linux and Windows.) Ram. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
Tarek Ziadé wrote:
What is a registery of extension exactly ? Distutils let you register your own
commands, you can use through the CLI.
Can you provide more details ?
Sure. Right now, if you want to deal with a new source or a new target,
you need to create a new command or override one. For example, cython
has distutils extension which subclass build_ext, we in numpy do the
same. This causes several issues:
- cython build_ext subclass distutils command, we do the same. How
to use both at the same time ? That's one typical example of the
subclassing issue Ian Bicking mentioned,
- in the case of compiled extensions, the basic structure is to
build all object files for each extension from the compiler class
(compile method). The compile method structure is:
for obj in objects:
if src == bla:
do_bla()
elif src == blabla:
do_blabla()
...
else:
raise some exception for unrecognied extension.
If you want to support a new source/target (say assembler), you need to
deal at this stage (or use hacks to deal with the new file extension
you want to deal with, and remove it later...). So you need to copy
build_ext, you cannot extend it.
If you look at msvccompiler vs msvc9compiler, both compile methods are
almost the same and copied from each other.
Now, if instead you have a dictionary {src_extension: callable}, you
would be able to add a new tool and extend existing code without
touching or copying anything. You could have syntactic sugar to have
something like:
@extension(.asm)
def assemble():
do_assembler_stuff()
All the assembler-related thing would be in a separate module, and you
would only need to register for the tool explicitly when you need it
(every tool could be the same, and current distutils tools would then be
registered automatically).
I am in for making Distutils evolve, but I need very precise real
world use cases
not saying that Distutils shouldn't do imperative operations).
I have given very explicit examples in this discussion - I have written
them on the wiki last time it was discussed as requested:
http://wiki.python.org/moin/Distutils/PluginSystem
I don't think it is accurate to summarize my critic as a vague do
imperative operations.
Last, I am not sure why you want only backward-compatible changes in
distutils.
There's no plan to keep backward-compatibility if breaking it makes
DIstutils better. We will have pending deprecation warnings, that's
all.
What is the intended policy: deprecate something in v1 and break it in
v1+1 ? I am not sure this works well when you need to correct deep
design issues which impact a lot of code (and this will be required).
For example, I would prefer not changing numpy.distutils several times
just to make a few API cleaner.
I guess I don't see the point of breaking things while keeping the
current distutils codebase. I would rather do the exact contrary: throw
away the codebase, but keep current setup.py working through a
conversion script. Things like sandboxing pypi and other things become
much easier that way.
cheers,
David
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
Georg Brandl wrote: One thing about CPAN (and Haskell's libraries on hackage) that I think many people see favorably, even though it's only superficial, is the more-or-less consistent hierarchical naming for the individual packages (or the contained modules in Haskell). Compared with that, the Python package namespace looks untidy. That's true, but there is not much we can do on this one, so I did not mention it. Note that the downloadable distutils manual has 94 pages and *should* be enough to explain the basics of packaging. It has to be updated, of course, once the more advanced mechanisms are part of the core. The manual is too complicated for simple tasks, and not very useful for complex ones. Mostly because distutils does not follow the only one way to do things mantra. I can help to improve the distutils documentation for the build part, which is mostly undocumented (things like how to create a new command to build ctypes extensions, etc...). Me too. Though it would be Snakebite + serious sandboxing. Sandboxing is of course needed, but that's a known problem, and people have already thought hard about it. The open suse build system, albeit linux specific, works quite well, for example. For environment sandboxing, chroot works on all unix I know (including mac os x) - security is more challenging, I don't have any expertise there. Windows is more difficult to handle, though (maybe windows people know good sandboxing solutions outside full-blown vm). What you're saying there about Cabal is exactly my experience. It is very nice to work with, and I've not yet seen a conceptual failure. But we're not far from that, with a static metadata file. Several people have claimed this so far, but I don't understand why - could you expand on this ? My impression is that the focus is mostly on version specification and install/build requirements in the static data, but to me that's a tiny detail. I want something like .cabal files, where you can specify documentation, data, source files, etc... Something like what I started to prototype there: http://github.com/cournape/toydist/ To take an example you are well familiar with, you can fully describe sphinx with it, and the conversion is mostly automatic. This is not even 500 LOC. With this kind of design, you can use different build systems on top of it (there is for example unpublished code in toydist to use a scons-based build system instead of distutils as currently done). I won't rehearse it here, but basically: - distutils is too complex for simple packages, and too inflexible for complex ones. Adding new features to distutils is a painful experience. Even autotools with its mix of 100 000 lines autogenerated shell code, perl, m4 is more pleasant. Really? Sure, the perl/shell/awk/m4 mix is painful, but at least the result is reasonably robust, and can be extended. I would have assumed that even writing a whole new distutils command/build step can't be more painful than adding the equivalent to an autotools-style build system, being Python after all. However, I've never done such a thing, so I have to believe you. I expand on that, because I think few people understand the problem here, and that's maybe the main source of frustration for core numpy developers as far as distutils is concerned. True, writing your own command is easy. But it has many failure modes: - if you extend an existing command, you have to take care whether you run under setuptools or distutils (and soon distribute will make this worse). Those commands will not work the same when you run them under paver either. - the division in subcommands is painful, and the abstraction does not make much sense IMHO. Recently, I needed to access simple things like library filename (foo -libfoo.a/foo.lib/etc..), install prefix. But those are not accessible to each command. The install prefix was particularly painful, it took me several hours to get it work right with distutils inplace, develop mode on all platforms. All this is trvially easy to get with autotools, scons or waf. Every new feature I needed to add to numpy.distutils was an unpleasant experience. I had to read the distutils sources (for every supported python version), run it on several platforms, and got it working by trial an error. - if you want to add a new source file extension, you have to rewrite the build_ext or build_src command and you often cannot reuse the base class methods. - etc... Also, the distutils code is horrible: you don't really know what's public and what's not, most attributes are added at runtime (and sometimes differ depending on the platform). Often, you get strange errors with the exception swallowed, and that happens only on some platforms for some users; in that case, the only way to debug it is to be able to run their platform. When you write extensions to distutils, this contributes to the whole unpleasant
Re: [Distutils] People want CPAN :-)
On Sun, Nov 8, 2009 at 2:22 PM, David Cournapeau da...@ar.media.kyoto-u.ac.jp [..] - if you extend an existing command, you have to take care whether you run under setuptools or distutils (and soon distribute will make this worse). No, because unlike setuptools, we want to remove the patching that is done on the Command class and on the DIstribution class, and make Distribute a good Distutils citizen in that matter. IOW you won't suffer anymore from what you've described. Those commands will not work the same when you run them under paver either. - the division in subcommands is painful, and the abstraction does not make much sense IMHO. Recently, I needed to access simple things like library filename (foo -libfoo.a/foo.lib/etc..), install prefix. But those are not accessible to each command. [...] - if you want to add a new source file extension, you have to rewrite the build_ext or build_src command and you often cannot reuse the base class methods. - etc... This is going to be changed because I am currently refactoring the build_ext command so such methods are in a base compiler class and eventually in util.py You are welcome to help in this refactoring, Also, the distutils code is horrible: you don't really know what's public and what's not, most attributes are added at runtime (and sometimes differ depending on the platform). Often, you get strange errors with the exception swallowed, and that happens only on some platforms for some users; in that case, the only way to debug it is to be able to run their platform. When you write extensions to distutils, this contributes to the whole unpleasant experience. I agree the code is not modern, but things are getting changed through small steps. Although, I don't buy the strange errors part and things getting swallowed :) [..] Guido wanted to know how scientific python people feel about the whole situation, and my own impression is that we are going further from what we need. I don't think anything based on distutils can help us. This is not to criticize Tarek, PJE and other people's work: I understand that distutils and setuptools solve a lot of problems for many people, and I may just be a minority. My opinion is that you've build something else when Distutils was not evolving anymore. This is not true anymore. It's moving again. And I think that the work that is going on is heading in the right direction, even for your use cases imho. If projects that maintain distutils patched versions push those patches now in the Python issue tracker, diffing against the current trunk, and if those patches make sense and are with tests, that's by far the *easiest* way to help improving Distutils. And that's the easiest work for me : I'll just review them and commit them if they do improve Distutils. Tarek ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Sat, Nov 07, 2009 at 10:56:24AM -0500, David Lyon wrote: On Sat, 7 Nov 2009 16:08:46 +0100, Tarek Ziadé ziade.ta...@gmail.com wrote: Gosh. I am not your boss, and I am not telling you what to do. otoh you're the boss of distutils. So you can direct people to work on certain things to help you along. That would have kept me much quieter with work. If you have time to burn and need to be told how to spend it: on top of my head a useful contribution would be to improve the PEP 386 reference implementation for example (I pick that one as I know most about it from all the PEP proposals currently). This is a PEP that should maybe be finished first of the bunch, so that's definitely useful. Useful things: * Read up on the PEP and all the documentation in the reference implementation * Check out previous discussions and make sure nothing is missed. * Add more test! The core class could be tested better, and suggest_rational_version is massively undertested. For this you need to read up on the old distutils version scheme as well as setuptools scheme and build lots of test cases for it. Also looking at all versions on the cheesshop to figure out tricky version numbers and test the function. All the reasonable version numbers used on PyPI should be tested really. * Check the history of the implementation. Maybe there have been checkins that correct things in one area but left similar bugs in other areas (e.g. a bug fixed in the class but not in suggest_rational_version). If so add tests. * If you managed to create tests that fail, see if you can figure out ways to make them pass. * Improve the documentation of the reference implementation, after you've done many of the above things you'll definitely have found some places where it could be improved. * Create patches out of the above work and submit it to the refrence implementation, if they're useful and good they'll get accepted. But don't be discouraged if a re-work is asked for initially. I'm not telling you the above things because I want to be your (or anyone's) boss and tell other people what to do. I'm telling you them as an example of how to contribute to the current work. The important thing is that I don't know anything more then you do, I haven't had secret conversations with a cabal or so (I'm not even on IRC usually). The only thing I do is read the mailing list and look at proposals I'm interested in. Finding and carrying out work like this is contributing. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Sun, 8 Nov 2009 19:19:44 +, Floris Bruynooghe floris.bruynoo...@gmail.com wrote: .. improve the PEP 386 reference implementation for example (I pick that one as I know most about it from all the PEP proposals currently). I'm +1 on PEP-386. It makes sense to me. This is a PEP that should maybe be finished first of the bunch PEP-345 is perphaps more important. More depends on that. I think/hope Guido's post has changed things a little. I agree with those who say everybody should work together to some PEPs closed off and I also agree with the push to offer something that is of a more comparable standard to CPAN. To get to where Guido is asking for, I think there are some gaps in the PEP coverage. So we need to cover those bases also. David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
Tarek Ziadé wrote: On Sun, Nov 8, 2009 at 2:22 PM, David Cournapeau da...@ar.media.kyoto-u.ac.jp [..] - if you extend an existing command, you have to take care whether you run under setuptools or distutils (and soon distribute will make this worse). No, because unlike setuptools, we want to remove the patching that is done on the Command class and on the DIstribution class, and make Distribute a good Distutils citizen in that matter. I don't think that's possible, not without major changes of the distutils design at least. Right now, in numpy.distutils, we extend the Distribution class to hold a few additional data. We inherit from setuptools or distutils depending on whether setuptools has been imported. The fundamental problem is not so much that setuptools does monkey-patching, but that you need to inherit to extend. Once you have several possible packages which inherit independently from those classes, you have to special case each of them if you want to support all of them. This is going to be changed because I am currently refactoring the build_ext command so such methods are in a base compiler class and eventually in util.py Refactoring cannot help, unless you fundamentally change how it works. I gave an example of a simple feature which is hard to do with current distutils: obtaining the install prefix early in the build scheme. When you do: python setup.py install --prefix=foo the build_* commands don't know prefix, only install knows it. In numpy, I added various hack each more ugly than the other to fake running the install command before running the build commands to obtain this information. I would be surprised if subsequent refactor of distutils will not break it. I agree the code is not modern, but things are getting changed through small steps. Although, I don't buy the strange errors part and things getting swallowed :) I don't understand what's there to buy. Several people reported distutils errors without any backtrace, though a fair shared of those were caused by our own extensions. Concerning changing into small steps: I understand that changing gradually is often better than starting from scratch, but my own experience with numpy.distutils and distutils convinced me that there is not much to save from distutils code. I consider the API, the UI and the implementation deeply flawed. For me, an improved backward compatible distutils is an oxymoron. More practically, I think the recent distutils-related changes in 2.6.3 and 2.6.4 will keep happening. In numpy.distutils, we depend a lot on internal details of distutils, because we need things not available in the API, and I don't think we are alone. I would mention that numpy is one of the package which has the most invested in distutils in terms of code (sloccount tells me that numpy.distutils is 10315 LOC, whereas python 2.6.2 distutils is 10648 LOC), so I am certainly aware of the cost of breaking backward compatibility. My opinion is that you've build something else when Distutils was not evolving anymore. It has nothing to do with distutils evolving. We have had our own extensions for years (I am a relatively newcommer, but some numpy.distutils code goes back to 2001), so we could have done pretty much what we wanted. We now have two build systems: one based on distutils, one based on scons. Every-time I added a build feature to numpy, it took me much more time to add it to distutils than to the scons build. The code needed to support scons build is ~ 2000 lines of code, is more robust, handle dependencies automatically, support reliable parallel builds, and is more maintainable. cheers, David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
David Cournapeau wrote: I don't understand what's there to buy. Several people reported distutils errors without any backtrace, though a fair shared of those were caused by our own extensions. distutils specifically swallows exceptions and formats them for users by default. After all, it is trying to behave like a regular command line program that interacts with users, not developers. This is easily overridable for developers who are trying debug problems by setting the environment variable DISTUTILS_DEBUG=1. This will make distutils just give the traceback. Is this what you are referring to? -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
Robert Kern wrote: After all, it is trying to behave like a regular command line program that interacts with users, not developers. This is easily overridable for developers who are trying debug problems by setting the environment variable DISTUTILS_DEBUG=1. This will make distutils just give the traceback. I did not know about that option, thanks. The last example I remember was python 2.6 breaking our mingw support: the raised exception was an empty message, so you got a error: as your error message when the exception was caught in setup. I am not sure it is better than a traceback: maybe distutils could separate between 'expected' and 'unexpected' exceptions to make this clearer. David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Sun, Nov 8, 2009 at 10:28 PM, Robert Kern robert.k...@gmail.com wrote: David Cournapeau wrote: I don't understand what's there to buy. Several people reported distutils errors without any backtrace, though a fair shared of those were caused by our own extensions. distutils specifically swallows exceptions and formats them for users by default. After all, it is trying to behave like a regular command line program that interacts with users, not developers. This is easily overridable for developers who are trying debug problems by setting the environment variable DISTUTILS_DEBUG=1. This will make distutils just give the traceback. In the tools I've written, I generally give the traceback if the verbosity is turned up, and in a case like this (an unexpected exception -- for distutils that's any exception except a few that distutils defines) I include use -v for more. I think distutils (or Distribute) could do the same. -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Sun, 8 Nov 2009 23:28:42 -0600, Ian Bicking i...@colorstudy.com wrote: In the tools I've written, I generally give the traceback if the verbosity is turned up, and in a case like this (an unexpected exception -- for distutils that's any exception except a few that distutils defines) I include use -v for more. I think distutils (or Distribute) could do the same. Hi Ian, I think PIP is quite an accomplishment. But don't you think that its a big ask to refactor distutils/distribute to redo their error messages for package building? imo the basic problem in setup.py and package building comes from ambiguity in the file specification section. Everything else is pretty much ok. With the declarative format David C is hinting at (from Haskell) it should be a lot easier to collect those files up and put them in a .tar.gz archive. I'm just wondering if you've ever thought about the tool chain on the other side from pip. Like how hard is it to create archive files of source with all the right stuff. So I guess what's your take on how close to being right is David C? Best Regards David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
11/06/2009 08:53 PM, Guido van Rossum: Python is suffering a lot in the scientific word, because it has not a CPAN-like repository. One thing I think is important too: Perl has helpers that help to distribution-package easily. Python must have too. For example, debian packaging what is on pypi must be straighforward. (same for RPM based distribution and other systems) -- Architecte Informatique chez Blueline/Gulfsat: Administration Systeme, Recherche Developpement +261 33 11 207 36 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
Rakotomandimby Mihamina miham...@gulfsat.mg writes: For example, debian packaging what is on pypi must be straighforward. (same for RPM based distribution and other systems) Packaging for Debian is much more about following the policy, which deliberately involves human intervention and judgement, and can't very well be automated. RPM, as I understand it, is more lax and a simple RPM can be generated quite automatically. Can you give an example of what you mean: * How straightforward do you find performing the Debian packaging for a Perl package, and what tools do you use to do it? * Would you consider it sufficient for the same (or equivalent) process to apply for Debian packaging of a Python package? -- \ “If consumers even know there's a DRM, what it is, and how it | `\ works, we've already failed.” —Peter Lee, Disney corporation, | _o__) 2005 | Ben Finney ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
David Lyon david.l...@preisshare.net writes: On Sun, 8 Nov 2009 23:28:42 -0600, Ian Bicking i...@colorstudy.com wrote: In the tools I've written, I generally give the traceback if the verbosity is turned up, and in a case like this (an unexpected exception -- for distutils that's any exception except a few that distutils defines) I include use -v for more. I think distutils (or Distribute) could do the same. Hi Ian, I think PIP is quite an accomplishment. But don't you think that its a big ask to refactor distutils/distribute to redo their error messages for package building? I've just had a read through the code for ‘pip’; AFAICT, the “redo the error messages for package building” essentially amounts to using the ‘logging’ module. Is that a big ask? -- \“We have to go forth and crush every world view that doesn't | `\believe in tolerance and free speech.” —David Brin | _o__) | Ben Finney ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
Ben Finney wrote: Rakotomandimby Mihamina miham...@gulfsat.mg writes: For example, debian packaging what is on pypi must be straighforward. (same for RPM based distribution and other systems) Packaging for Debian is much more about following the policy, which deliberately involves human intervention and judgement, and can't very well be automated. RPM, as I understand it, is more lax and a simple RPM can be generated quite automatically. A .deb can also be automated up to a certain level. Even for packages like numpy, there is not much needed. Also, the official debian policy is mandatory for adoption into official repositories - but if I need to push a simple .deb to colleagues for a new package of mine, I don't want/need to spend that much time. For example, if distutils were to support the --*dir options of autoconf plus all the related metadata to tag files accordingly, it would make the task quite simple. Even for complex package, this would make packaging and package update simpler for package maintainer. If I look at the debian dir for the numpy package (in Ubuntu), there is really not much needed: the .install files to tag which files to install for which package, adding debian specific files (README.debian and co). Most of it if not all could be removed with the corresponding distutils support (which would not be .deb specific in any way, and would help rpm, .pkg and even windows packaging as well though .e.g. nsis files). David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
11/09/2009 09:57 AM, Ben Finney: For example, debian packaging what is on pypi must be straighforward. (same for RPM based distribution and other systems) Packaging for Debian is much more about following the policy, which deliberately involves human intervention and judgement, and can't very well be automated. As far as I know (I might know nothing;)), and as far as I follow the debian-mentors ML, most of the problems is about filesystem hierarchy Python packages are pretty clear on their FS hierarchy, normalizing it should not be that hard. Once it's clear about that, Debian packaging straighforwardness is 80% done. Can you give an example of what you mean: * How straightforward do you find performing the Debian packaging for a Perl package, and what tools do you use to do it? * Would you consider it sufficient for the same (or equivalent) process to apply for Debian packaging of a Python package? Give me just some time to give a try on an examples. My tries might invalidate my assumptions, but I need to test that on a bunch of Perl modules to confirm. Not just on 2-3. -- Architecte Informatique chez Blueline/Gulfsat: Administration Systeme, Recherche Developpement +261 33 11 207 36 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Mon, Nov 9, 2009 at 1:09 AM, Ben Finney ben+pyt...@benfinney.id.au wrote: David Lyon david.l...@preisshare.net writes: On Sun, 8 Nov 2009 23:28:42 -0600, Ian Bicking i...@colorstudy.com wrote: In the tools I've written, I generally give the traceback if the verbosity is turned up, and in a case like this (an unexpected exception -- for distutils that's any exception except a few that distutils defines) I include use -v for more. I think distutils (or Distribute) could do the same. Hi Ian, I think PIP is quite an accomplishment. But don't you think that its a big ask to refactor distutils/distribute to redo their error messages for package building? I've just had a read through the code for ‘pip’; AFAICT, the “redo the error messages for package building” essentially amounts to using the ‘logging’ module. Is that a big ask? pip doesn't use the logging module, it uses its own logger, which is intended more for managing the output of a command-line program and not just post-mortem debugging. I don't think changing distutils to improve error output would be hard at all. It looks like there's a line in distutils.core that catches these exceptions (and doesn't look like it actually catches all exceptions?), and that can just be fixed. Another topic that has come up: I do agree subclassing makes it really hard to have multiple lines of development (specifically something like Setuptools or Distribute, along with ad hoc development in setup.py files). But I think that can be resolved. Perhaps, for instance, Distribute can have implementations of commands like build, that can easily be customized or extended without subclassing (e.g., by pre-build or post-build functions). I'd really be shocked if a rewrite of distutils was necessary, or even necessary to simplify things. -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
Hi Guido, Guido van Rossum wrote: On Fri, Nov 6, 2009 at 2:52 PM, David Lyon david.l...@preisshare.net wrote: So the packages on CPAN are typically of a higher quality, simply because they've been machine checked. I like that. Speaking purely on hearsay, I don't believe that. In fact, I've heard plenty of laments about the complete lack of quality control on CPAN. I cannot speak for CPAN, as I have never used it. But CRAN (which is CPAN for R) works much better that PyPI today in practice. I am not sure what exactly makes it work better, but it has the following properties, both technical and more 'social': - R is a niche language, and targets mostly scientists. It is a smaller community, more focused. They can push solutions more easily. - There is an extensive doc on how to develop R extensions (you can download a 130 pages pdf). - R packages are much more constraints: there is a standard source organization, which makes for a more consistant experience - There are regular checks of the packages (all the packages are daily checked on a build farm on fedora and debian). It also has a machine to check windows. http://cran.r-project.org/web/checks/check_summary.html http://cran.r-project.org/bin/windows/contrib/checkSummaryWin.html I am obviously quite excited by Snakebite potential here. Concerning distutils, I think it is important to improve it, but I think it is inherently flawed for serious and repeatable packaging. I have written a quite extensive article on it from my point of view as a numpy/scipy core developer and release manager (http://cournape.wordpress.com/2009/04/01/python-packaging-a-few-observations-cabal-for-a-solution/), I won't rehearse it here, but basically: - distutils is too complex for simple packages, and too inflexible for complex ones. Adding new features to distutils is a painful experience. Even autotools with its mix of 100 000 lines autogenerated shell code, perl, m4 is more pleasant. - Most simple packages could be buildable from purely declarative description. This is important IMHO because it means they are simple to package by OS vendors, and you can more easily automate building and testing. - it is hard to interact with other build/distribution tools, which is sometimes needed. - distutils is too flexible for some things where it should not be(like specifying the content of sdist tarballs), and that makes it very difficult to repeat things in different environments. Contrary to other people, I don't think a successor to distutils is that hard to develop, especially since good designs in other languages already exist. It would take time, especially to get a transition story right, but I think it would be worthwhile. cheers, David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
Guido van Rossum gu...@python.org writes: On Fri, Nov 6, 2009 at 2:52 PM, David Lyon david.l...@preisshare.net wrote: So the packages on CPAN are typically of a higher quality, simply because they've been machine checked. I like that. Speaking purely on hearsay, I don't believe that. In fact, I've heard plenty of laments about the complete lack of quality control on CPAN. That's not inconsistent with CPAN having higher average quality than the average quality of PyPI packages. It could merely mean that, as a result of the awareness that CPAN packages are tested on upload, CPAN users *expect* higher quality, and complain louder than PyPI users when they find a low-quality package :-) I think buildbot-style test runs for PyPI packages would raise average package quality on PyPI. -- \ “Everything you read in newspapers is absolutely true, except | `\for that rare story of which you happen to have first-hand | _o__) knowledge.” —Erwin Knoll | Ben Finney ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
I love how this created a flurry of speculation on what people who say Python doens't have a CPAN mean. Wouldn't it be easer to *ask* them? :-) Just-wondering-ly -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
2009/11/7 Kaelin Colclasure kae...@acm.org: Since I bootstrapped the environment I used to learn Python with easy_install, I naturally went straight to the easy_install docs to learn how to give back. I wasn't even aware of this separate thing called distutils until I read about it in the easy_install (err setuptools) documentation (sic). Yes, this seems to be a reasonably way to realize that packaging is done with distutils. What was then complicated with the distutils docs? It's so long ago I did this the first time I don't remember if I found it difficult. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Nov 6, 2009, at 12:53 PM, Guido van Rossum wrote: I just found this comment on my blog. People have told me this in person too, so I believe it is real pain (even if the solution may be elusive and the suggested solutions may not work). But I don't know how to improve the world. Is the work on distutils-sig going to be enough? Or do we need some other kind of work in addition? Do we need more than PyPI? In my experience, when users say this, they just mean I tried easy_install and it broke. PyPI doesn't have some deep, fundamental, architectural issue that prevents it from working. The user experience of it is just buggy. Consider the difference between these two pages: http://docs.webfaction.com/software/perl.html http://docs.webfaction.com/software/python.html Note that the 'python' page is more than twice as long, lists a ton of different installation options, and includes a gigantic troubleshooting section that apparently isn't necessary for perl. Note also that the Perl page is just a series of steps describing how to invoke the one installation mechanism, but the Python page is a hodgepodge of qualified instructions describing different possible mechanisms you can try. It also appears that webfaction has modified the default environment's configuration to make their troubleshooting section *shorter* than it would have to be for more general Python software installation instructions. The default behavior of most Python installation mechanisms - to my knowledge, 'python setup.py install', 'easy_install', and 'pip', will all raise exceptions by default on UNIX-y platforms, unless you're root. On Windows (since a higher percentage of the user population walks around with admin rights all the time), the default invocations described by many project web pages will work if the installation is pure-python or if the author remembered to provide a Windows binary egg, but a common failure mode is you don't have a compiler. Similarly, on a Mac, you have to have Xcode installed, although Python itself works fine if you don't, so it seems like you don't. Many of these tools *would* work by default with a small amount of configuration, a couple of environment variables, and clearer error messages that indicate (A) *that* you need to install a C compiler and (B) *where* you need to go to get a C compiler. One project that would help a lot is just a easy python setup documentation project that describes, as simply as possible, in large fonts, how to get a working Python setup that adheres to a few conventions. Just include the 2 lines of .bashrc and explain how to add them; don't debate the merits of ~/bin vs. ~/.local/bin vs. ~/opt/ bin (although come on, ~/.local/bin/ is _clearly_ the right name for it), just pick one for each platform and provide clear step-by-step instructions for getting it to work. put this in your ~/.bashrc: really big PRE tag with shell setup in it. restart your shell. Anybody who has selected an alternate shell or done some other prior configuration will need to adjust their expectations, but we can worry about supporting unusual configurations when the community has a good answer for the default configuration. (Although this is a good reason to do this as documentation and not attempt to write an autoconfigurating tool: an autoconfigurating tool needs to understand every possible nuance of the environment, but advanced users can notice which parts of the short document might not apply to them.) I feel like I might be repeating this a bit much, but it's a really important point: many of the things I'm talking here are *not* about getting the code right as part of a Python tool, but in providing an easy, manageable way to integrate with _other_ tools that are outside of our collective control as Python package authors: the dynamic linker, the shell, the file manager and the C compiler (or lack thereof). By providing a default user-home-directory installation location, Python itself is already doing almost as much as it can; if easy_install started installing things into that location by default *without* any of this bootstrapping documentation (or a very, very carefully written tool to do the bootstrapping for you) then importing pure Python packages might work great but scripts would be broken and any external shared libraries required by Python modules (even if they built correctly) would just give you an ImportError. Once we have some kind of working consensus on this setup, the tools can change to support it: easy_install can default to installing things in the user's home directory in the case that (A) the environment is set up for it and (B) the user isn't an administrator. If the environment *isn't* set up, instead of spitting out twelve paragraphs explaining how really you should have read-write access to the location where
Re: [Distutils] People want CPAN :-)
On Sat, Nov 7, 2009 at 5:07 AM, David Lyon david.l...@preisshare.net wrote: On Sat, 7 Nov 2009 02:19:02 +0100, Tarek Ziadé ziade.ta...@gmail.com wrote: I unfortunately don't have all the time I wish I had to work on this more. We all know that. It needs somebody new to take on the challenge. David, You don't understand at all what is going on I think. You have been keeping sending passive-agressive emails in python-dev and in distutils-sig everytime we mention the work that's going on in Distutils. This is happening since we said we didn't want to integrate your GUI system in Python. When I say I unfortunately don't have all the time I wish I had to work on this more, it doesn't mean that we need to find somebody new to take on the challenge, it means that we can take more help from more people (and that includes you of course). Wether you like it or not, I am maintaining Distutils, so I am trying to coordinate the work that is going on for that matter. And this work is currently organized in several PEPs we are changing everytime someone provides some new feedback and insights. You have started some kind of counter-PEP to push another build system, and I really doubt this is useful in any ways. Discussing new ideas is useful of course, but changing Distutils for another build system in the stdlib will not happen, unless its proven to be sucessfully used by the community for some time. I will do at this point what other have been doing at python-dev and just ignore your mails for now on. Tarek ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
2009/11/7 Glyph Lefkowitz gl...@twistedmatrix.com: One project that would help a lot is just a easy python setup documentation project that describes, as simply as possible, in large fonts, how to get a working Python setup that adheres to a few conventions. [...] +1 Strong conventions would help a lot here, IMHO. At least as useful would be a similar document explaining how to write a Python package (contents of setup.py, doc/test subdirectories, registering on PyPI, what to upload - bdist_wininst, egg, source, ...) This would make the build/install experience much more consistent for new packages. Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
David Lyon wrote: Even if you implement PyPI + PEP 381 + PEP 381 tomorrow I promise you that you won't be anywhere close to CPAN. It's a much more serious challenge than perphaps you realise.. I keep reading and I keep hearing you and others saying this, but as someone who has never used CPAN, I'm not seeing the large number of specific implementable tasks that CPAN clearly has and PyPI clearly does not. Without handwaving please, in a technical sense what does CPAN have that is so wonderful other than the items mentioned so far of: - buildbot on the server - mirrored. hierarchical servers - one widely known and accepted way of doing packaging If we implement those, if we implement PEP 381, in what specific way will we still fall *far short* as you suggest? I can't fix the social aspects so those are not of use to me. Only architecture. BTW at the recent pyTexas regional conference we had a good group discussion about packaging, with people offering analysis from the Perl, Java, Ruby and Haskell communities. It seems each language still only covers part of the solution, albeit different parts. -Jeff ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Sat, Nov 7, 2009 at 6:04 AM, Jeff Rush j...@taupro.com wrote: BTW at the recent pyTexas regional conference we had a good group discussion about packaging, with people offering analysis from the Perl, Java, Ruby and Haskell communities. It seems each language still only covers part of the solution, albeit different parts. During that conversation Walker mentioned that he thought that Java's Maven packaging system had been really successful with the idea of having a metadata file stored outside each package. That made it easier for the metadata format to evolve; there is no need change every package in a repository every time the metadata format is improved. It also allows convenient inspection of dependencies between packages without having to download/decompress those packages. (Maven also stores a copy of the metadata inside the package, but the authoritative/complete copy is considered the one outside the package). I've cc'd Walker hoping he will comment more about the other advantages of externally stored metadata. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
David Cournapeau schrieb: Hi Guido, Guido van Rossum wrote: On Fri, Nov 6, 2009 at 2:52 PM, David Lyon david.l...@preisshare.net wrote: So the packages on CPAN are typically of a higher quality, simply because they've been machine checked. I like that. Speaking purely on hearsay, I don't believe that. In fact, I've heard plenty of laments about the complete lack of quality control on CPAN. One thing about CPAN (and Haskell's libraries on hackage) that I think many people see favorably, even though it's only superficial, is the more-or-less consistent hierarchical naming for the individual packages (or the contained modules in Haskell). Compared with that, the Python package namespace looks untidy. I cannot speak for CPAN, as I have never used it. But CRAN (which is CPAN for R) works much better that PyPI today in practice. I am not sure what exactly makes it work better, but it has the following properties, both technical and more 'social': - R is a niche language, and targets mostly scientists. It is a smaller community, more focused. They can push solutions more easily. - There is an extensive doc on how to develop R extensions (you can download a 130 pages pdf). Note that the downloadable distutils manual has 94 pages and *should* be enough to explain the basics of packaging. It has to be updated, of course, once the more advanced mechanisms are part of the core. - R packages are much more constraints: there is a standard source organization, which makes for a more consistant experience - There are regular checks of the packages (all the packages are daily checked on a build farm on fedora and debian). It also has a machine to check windows. http://cran.r-project.org/web/checks/check_summary.html http://cran.r-project.org/bin/windows/contrib/checkSummaryWin.html I am obviously quite excited by Snakebite potential here. Me too. Though it would be Snakebite + serious sandboxing. Concerning distutils, I think it is important to improve it, but I think it is inherently flawed for serious and repeatable packaging. I have written a quite extensive article on it from my point of view as a numpy/scipy core developer and release manager (http://cournape.wordpress.com/2009/04/01/python-packaging-a-few-observations-cabal-for-a-solution/), What you're saying there about Cabal is exactly my experience. It is very nice to work with, and I've not yet seen a conceptual failure. But we're not far from that, with a static metadata file. I won't rehearse it here, but basically: - distutils is too complex for simple packages, and too inflexible for complex ones. Adding new features to distutils is a painful experience. Even autotools with its mix of 100 000 lines autogenerated shell code, perl, m4 is more pleasant. Really? I would have assumed that even writing a whole new distutils command/build step can't be more painful than adding the equivalent to an autotools-style build system, being Python after all. However, I've never done such a thing, so I have to believe you. Coming back to Cabal, do you know how easy it is to customize its build steps? - Most simple packages could be buildable from purely declarative description. This is important IMHO because it means they are simple to package by OS vendors, and you can more easily automate building and testing. That's what we're heading towards, I think. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
Tarek Ziadé schrieb: For the documentation part I am afraid it will be messy for the end users trying to package apps in Python *until* all PEPs have made it into Python. Although, as Ian Bicking says: we could write today some kind of all-in-one tutorial so end-users can work out without having to run after the documenation in several places. Big +1, and I would be more than willing to include it in the standard Python documentation, *even* if it mostly describes setuptools/distribute/pip. When people want to package a library, they *will* look for docs in Python, and at the moment they only find the distutils reference. While the latter is necessary, a more howto-like standard document is much needed. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
la, 2009-11-07 kello 00:14 +0200, Alex Grönholm kirjoitti: [clip: problems in distributing scientific Python packages] I for one did not understand the problem. What does CPAN have that PyPI doesn't? It is natural for packages (distributions, in distutils terms) to have dependencies on each other. Why is this a problem? Personally, I've not so much had trouble with PyPi, but with the rest of the toolchain. What's special with scientific software is that - They're usually not pure-Python - Need support for not only for C, but e.g. Fortran compilers - It may be necessary to build them on platforms where libraries etc. are in non-standard places - It may be useful to be able to build them with non-gcc compilers - They may need to ship more data files etc. than plain Python modules - Python is a newcomer on the scientific scene. Not all people want to spend time to spend on installation problems. Not all people are experienced Python users. So it may be more likely that the following things hurt in distributing these Python modules: 1. Incomplete documentation for distutils. For example, where can you find out what `package_data` option of setup() wants as the input? What if you have your package in src/packagename and data files under data/? What are the paths given to it relative to? The Distribute documentation is starting to look quite reasonable -- so documentation is becoming less of a problem. But it seems still to assume that the reader is familiar with distutils. 2. Magic. For example, what decides which files are included by sdist? It appears this depends on (i) what's in the autogenerated *.egg-info/SOURCES.txt (ii) whether you are using SVN and are using setuptools (iii) possible package_data etc. options, (iv) MANIFEST or maybe MANIFEST.in. IMHO, the system is too byzantine in ordinary matters, which increases the number of things you need to learn. 3. Many layers: distutils, setuptools, numpy.distutils. Numpy has its own distutils extensions, primarily for Fortran support. 4. Inflexibility. The toolchain is a bit inflexible: suppose you need to do something custom during the build, say, detect sizeof(long double) and add a #define to build options according to it. Finding out how to do this properly again takes time. 5. Distutils, and tools derived from it have bad failure modes. This hurts most with building extension modules. Given the many layers, and the fact that the build is driven by software that few really understand, it's difficult to understand and fix even simple errors encountered. Suppose a build fails, because your C or Fortran compiler gets passed a flag it doesn't like. How do you work around this? Suppose you have a library installed in a non-standard location. How do you tell distutils to look for it in the correct place? (The answer is to use build_ext command separately and pass it -L, but this is difficult to find out, as build does not accept -L.) The last one is in practice quite annoying, as given the heterogenous environments, it's not easy to make your package buildable on all possible platforms where people might want to use it. When people run into problems, they are stumped by the complexity of distutils. The above concerns only building packages -- perhaps there is more to say also about other parts. Also, I don't really have much experience with CPAN or CRAN, so I can't say how much Python is better or worse off here. -- Pauli Virtanen ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
Hi Jeff, On Sat, 07 Nov 2009 06:04:15 -0600, Jeff Rush j...@taupro.com wrote: I keep reading and I keep hearing you and others saying this, but as someone who has never used CPAN, I'm not seeing the large number of specific implementable tasks that CPAN clearly has and PyPI clearly does not. ok. Fair enough. Let me clear up the terminology. In perl, CPAN means packages + local perl + repository In python, pypi means repository. If people complain about pypi versus cpan, the complaint actually has nothing to do with pypi. What they are actually complaining about is the package management support (or lack thereof) that is built into python. Let me say it a different way... Where can I find the pypi module in python? Of course there is no pypi module in python. That's why this whole thing is just so confusing. If there was such a thing as a pypi module, then users would automatically use that to access pypi and download and install packages. But what python offers is: - setuptools - distribute - pip - distutils In perl, there's only one word for everything, cpan. Even though it's components exist as libraries and command interfaces on the local system. Without handwaving please, in a technical sense what does CPAN have that is so wonderful other than the items mentioned so far of: - buildbot on the server - mirrored. hierarchical servers - one widely known and accepted way of doing packaging + Easy to find Modules and command interfaces to interface to the repository that exist on the local machine. All carrying the matching cpan name. If we implement those, if we implement PEP 381, in what specific way will we still fall *far short* as you suggest? Pypi itself isn't a problem. Buildbots for packages shouldn't go on pypi itself in any case. They should go on an entirely seperate server. So, let me sum up. Pypi itself has no major external problem. It's short on a package buildbot and local python has no pypi module. Add a package buildbot and add a pypi package support and we're pretty much caught up. David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Sat, 7 Nov 2009 06:53:17 -0600, Brad Allen bradallen...@gmail.com wrote: During that conversation Walker mentioned that he thought that Java's Maven packaging system had been really successful with the idea of having a metadata file stored outside each package. That made it easier for the metadata format to evolve; there is no need change every package in a repository every time the metadata format is improved. It also allows convenient inspection of dependencies between packages without having to download/decompress those packages. (Maven also stores a copy of the metadata inside the package, but the authoritative/complete copy is considered the one outside the package). So too in python and pypi. I have used it and it works fine. http://wiki.python.org/moin/PyPiXmlRpc?action=showredirect=CheeseShopXmlRpc The problem is not that the functionality doesn't exist, just that users are expected to do an xmlrpc call to get it... (more logical would be to have a pypi module) David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
Hi Tarek, On Sat, 7 Nov 2009 12:12:44 +0100, Tarek Ziadé ziade.ta...@gmail.com wrote: You don't understand at all what is going on I think. I guess so. .. it means that we can take more help from more people (and that includes you of course). I really do accept that. What's hard for me to understand exactly is what you will actually allow me to do. Wether you like it or not, I am maintaining Distutils, so I am trying to coordinate the work that is going on for that matter. And this work is currently organized in several PEPs we are changing everytime someone provides some new feedback and insights. Why would I mind that you're maintaining distutils ? I don't pretend to know it as well as you do. I've been working in software for over twenty years and I've seen many things but nothing really like distutils. I certainly couldn't maintain it the way you are doing. I have real trouble understanding it. You have started some kind of counter-PEP to push another build system, and I really doubt this is useful in any ways. You shouldn't take it personally that another person tried to submit a PEP. So many people helped me with my proposal on the mailing list - and I owe it to them to at least get their efforts taken into consideration as a PEP. In any case, if you feel strongly about this then I can revise my proposal so that it doesn't look like a proposal for a build system (it was for a metadata setup - not a build system). Discussing new ideas is useful of course, .. The implication there is you don't want me to submit PEPs because they somehow counter your work. Well if that's the case I can be more careful in the future. There's plenty of things that I can write PEPs for now... A metadata install was just the first.. And to my knowledge, there still isn't a PEP about that. Right? David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Sat, Nov 7, 2009 at 3:46 PM, David Lyon david.l...@preisshare.net wrote: [..] .. it means that we can take more help from more people (and that includes you of course). I really do accept that. What's hard for me to understand exactly is what you will actually allow me to do. Gosh. I am not your boss, and I am not telling you what to do. This is open source, but this is also a community... You can't just come around and ignore some parts of what is being done. I am just telling you that if you want to help in the static metadata field, and if your goal is to help *Distutils* improve, you have to work with what has been started! [..] You have started some kind of counter-PEP to push another build system, and I really doubt this is useful in any ways. You shouldn't take it personally that another person tried to submit a PEP. So many people helped me with my proposal on the mailing list - and I owe it to them to at least get their efforts taken into consideration as a PEP. In any case, if you feel strongly about this then I can revise my proposal so that it doesn't look like a proposal for a build system (it was for a metadata setup - not a build system). [..] The implication there is you don't want me to submit PEPs because they somehow counter your work. Well if that's the case I can be more careful in the future. There's plenty of things that I can write PEPs for now... Neither Guido, neither Brett neither I are taking this personally. But at some point, if your goal is to help improving Distutils, you have to work with what has been started. A metadata install was just the first.. And to my knowledge, there still isn't a PEP about that. Right? If you work on a PEP that is related to any PEP started in the same area, I will strongly oppose against adding your PEP because it is a non-sense. Take and read the existing PEPs and help us improving them. We don't add new PEPs for the heck of it. If it's a credit issue, your name will be added in any PEP you will provide a valuable help on. Tarek ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
Andreas Jung kirjoitti: Am 06.11.09 18:53, schrieb Guido van Rossum: I just found this comment on my blog. People have told me this in person too, so I believe it is real pain (even if the solution may be elusive and the suggested solutions may not work). But I don't know how to improve the world. Is the work on distutils-sig going to be enough? Or do we need some other kind of work in addition? Do we need more than PyPI? My 2 cents after reading and ignoring the whole thread: - PyPI provides a good functionality so far What is annoying about PyPI: - some package maintainers have a certain ignorance and arrogance by misusing PyPI - for uploading packages without or broken metadata - for uploading packages of doubtful quality - for uploading packages to PyPI as a replacement for a private egg server - supports too much different versioning schemas. Both schema supported by setuptools and the one proposed by Tarek in some PEP are totally over-engineered. A simple and *enforced* versioning schema is what I want to see. - no more external hosting of packages. If people want their packages listed on Pypi, they should be required to upload their packages on PyPI (no more issues with non-available external server, no more issues with mirroring external servers, no more issues with wrong download URLs within package metadata) - better checks on uploaded packages. A source code release should be made using the 'sdist' command. We don't need source eggs of a package for Python 2.4-2.6 containing Python source code only. +5 The solution for a better PyPI: - more checks, more restrictions - every package maintainer uploading something to PyPI should have a certain attitude that PyPI is a public resource where the content should met certain quality criteria and where each package has a certain responsibility to Python community. +2 Andreas ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Sat, Nov 7, 2009 at 3:57 PM, Andreas Jung li...@zopyx.com wrote: [..] - supports too much different versioning schemas. Both schema supported by setuptools and the one proposed by Tarek in some PEP are totally over-engineered. A simple and *enforced* versioning schema is what I want to see. Unfortunately, as long as we have release candidates, and development versions, we need a more complex scheme that MAJOR.MINOR.MICRO. If you think of any ways to simplify PEP 386, please help us, Last, we can encourage people to use it, but we can't enforced it: I know people that are happily using dates for their versions, and we can't forbid them to push their work on pypi just because of that. We can try to educate then, but that's their pick at the end I think. An enterprise PyPI could do enforce it, but not our community PyPI imho [...] The solution for a better PyPI: - more checks, more restrictions - every package maintainer uploading something to PyPI should have a certain attitude that PyPI is a public resource where the content should met certain quality criteria and where each package has a certain responsibility to Python community. More checks would be nice, so we can provide QA rates or something similar. I don't think we should enforce any policy whatsoever though at PyPI. We can't force people that upload distributions to comply with some strict QA rules imho (no binary distro allowed if no sdist is present for example). Tarek ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
At 03:57 PM 11/7/2009 +0100, Andreas Jung wrote: - supports too much different versioning schemas. Both schema supported by setuptools and the one proposed by Tarek in some PEP are totally over-engineered. A simple and *enforced* versioning schema is what I want to see. - no more external hosting of packages. If people want their packages listed on Pypi, they should be required to upload their packages on PyPI (no more issues with non-available external server, no more issues with mirroring external servers, no more issues with wrong download URLs within package metadata) Do note that at least these two requirements of yours are likely to be opposed by some with at least as much force (if not more so) than you are proposing them with. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
Am 07.11.09 16:37, schrieb Tarek Ziadé: On Sat, Nov 7, 2009 at 3:57 PM, Andreas Jung li...@zopyx.com wrote: [..] - supports too much different versioning schemas. Both schema supported by setuptools and the one proposed by Tarek in some PEP are totally over-engineered. A simple and *enforced* versioning schema is what I want to see. Unfortunately, as long as we have release candidates, and development versions, we need a more complex scheme that MAJOR.MINOR.MICRO. Do we need/want development on PyPI? At least not me. MAJOR.MINOR.MICRO.PICO + |a-c]1..N should be good enough. Last, we can encourage people to use it, but we can't enforced it: Of course we can.. I know people that are happily using dates for their versions, and we can't forbid them to push their work on pypi just because of that. ..one must not accept and support a whole zoo of private numbering schemes. Agree on a common and minimal standard and enforce the standard. We can try to educate then, but that's their pick at the end I think. Teaching is a good thing... An enterprise PyPI could do enforce it, but not our community PyPI imho community does not imply that we can not agree on certain rules and standards for PyPI - otherwise PyPI remains as it sometimes appears - an unflashed package toilet. Python as a quality programming language needs a package repository with some minimum standards - I completely disagree with community as a synonym for we must make everyone happy. Andreas attachment: lists.vcf___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Sat, 7 Nov 2009 16:08:46 +0100, Tarek Ziadé ziade.ta...@gmail.com wrote: Gosh. I am not your boss, and I am not telling you what to do. otoh you're the boss of distutils. So you can direct people to work on certain things to help you along. That would have kept me much quieter with work. I am just telling you that if you want to help in the static metadata field, and if your goal is to help *Distutils* improve, you have to work with what has been started! Well talk about static metadata installation was started at pycon last year. Nothing happened till I started working on it. And I believe that I'm the only one working on it now. If you work on a PEP that is related to any PEP started in the same area, I will strongly oppose against adding your PEP because it is a non-sense. Same area? as in any packaging PEP? any distutils PEP? There are PEPs in every area. And PEPs are inter-related. Does rejection of PEPs on the above grounds apply to everyone? or just me? Other people seem to have PEPs in related areas. We don't add new PEPs for the heck of it. And people don't write them for the heck of it either. If I see something I can do to assist existing PEPs I'll try to do so. David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On 03:56 pm, li...@zopyx.com wrote: Am 07.11.09 16:37, schrieb Tarek Ziad�: On Sat, Nov 7, 2009 at 3:57 PM, Andreas Jung li...@zopyx.com wrote: [..] - supports too much different versioning schemas. Both schema supported by setuptools and the one proposed by Tarek in some PEP are totally over-engineered. A simple and *enforced* versioning schema is what I want to see. Unfortunately, as long as we have release candidates, and development versions, we need a more complex scheme that MAJOR.MINOR.MICRO. Do we need/want development on PyPI? At least not me. MAJOR.MINOR.MICRO.PICO + |a-c]1..N should be good enough. Please be considerate of the time of other people and read the previous threads on this list about versioning schemes before proposing a new one. If you have done this, then please make it evident by describing how your proposed scheme addresses the problems raised in previous discussions, or how those problems are not important/real/whatever. Thanks. Jean-Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
