Re: [matplotlib-devel] [SciPy-dev] [Numpy-discussion] Announcing toydist, improving distribution and packaging situation
On Sun, Jan 3, 2010 at 3:27 AM, Andrew Straw straw...@astraw.com wrote: Typically, the dependencies only depend on the smallest subset of what they require (if they don't need lapack, they'd only depend on python-numpy-core in your example), but yes, if there's an unsatisfiable condition, then apt-get will raise an error and abort. In practice, this system seems to work quite well, IMO. Yes, but: - debian dependency resolution is complex. I think many people don't realize how complex the problem really is (AFAIK, any correct scheme to resolve dependencies in debian requires an algorithm which is NP-complete ) - introducing a lot of variants significantly slow down the whole thing. I think it worths thinking whether our problems warrant such a complexity. Anyhow, here's the full Debian documentation: http://www.debian.org/doc/debian-policy/ch-relationships.html This is not the part I am afraid of. This is: http://people.debian.org/~dburrows/model.pdf cheers, David -- This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon's best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
Re: [matplotlib-devel] [SciPy-dev] [Numpy-discussion] Announcing toydist, improving distribution and packaging situation
On Tue, Dec 29, 2009 at 3:49 AM, Dag Sverre Seljebotn da...@student.matnat.uio.no wrote: Do you here mean automatic generation of Ubuntu debs, Debian debs, Windows MSI installer, Windows EXE installer, and so on? (If so then great!) Yes (although this is not yet implemented). In particular on windows, I want to implement a scheme so that you can convert from eggs to .exe and vice et versa, so people can still install as exe (or msi), even though the method would default to eggs. If this is the goal, I wonder if one looks outside of Python-land one might find something that already does this -- there's a lot of different package format, Linux meta-distributions, install everywhere packages and so on. Yes, there are things like 0install or autopackage. I think those are deemed to fail, as long as it is not supported thoroughly by the distribution. Instead, my goal here is much simpler: producing rpm/deb. It does not solve every issue (install by non root, multiple // versions), but one has to be realistic :) I think automatically built rpm/deb, easy integration with native method can solve a lot of issues already. - Currently I'm making a Sage SPKG for CHOLMOD. This essentially gets the job done by not bothering about the problem, not even using the OS-installed Python. Something that would spit out both Sage SPKGs, Ubuntu debs, Windows installers, both with Python code and C/Fortran code or a mix (and put both in the place preferred by the system in question), seems ideal. Of course one would still need to make sure that the code builds properly everywhere, but just solving the distribution part of this would be a huge step ahead. On windows, this issue may be solved using eggs: enstaller has a feature where dll put in a special location of an egg are installed in python such as they are found by the OS loader. One could have mechanisms based on $ORIGIN + rpath on linux to solve this issue for local installs on Linux, etc... But again, one has to be realistic on the goals. With toydist, I want to remove all the pile of magic, hacks built on top of distutils so that people can again hack their own solutions, as it should have been from the start (that's a big plus of python in general). It won't magically solve every issue out there, but it would hopefully help people to make their own. Bundling solutions like SAGE, EPD, etc... are still the most robust ways to deal with those issues in general, and I do not intended to replace those. What I'm saying is that this is a software distribution problem in general, and I'm afraid that Python-specific solutions are too narrow. Distribution is a hard problem. Instead of pushing a very narrow (and mostly ill-funded) view of how people should do things like distutils/setuptools/pip/buildout do, I want people to be able to be able to build their own solutions. No more use this magic stick v 4.0.3.3.14svn1234, trust me it work you don't have to understand which is too prevalant with those tools, which has always felt deeply unpythonic to me. Thanks, this cleared things up, and I like the direction this is heading. Thanks a lot for doing this! Dag Sverre -- This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon's best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
Re: [matplotlib-devel] [SciPy-dev] [Numpy-discussion] Announcing toydist, improving distribution and packaging situation
David Cournapeau wrote: On Sat, Jan 2, 2010 at 4:58 PM, Gael Varoquaux gael.varoqu...@normalesup.org wrote: On Sat, Jan 02, 2010 at 11:32:00AM +0900, David Cournapeau wrote: [snip] - supporting different variants of the same package in the dependency graph at install time [snip] The second issue is more challenging. It complicates the dependency handling quite a bit, and may cause difficult situations to happen at dependency resolution time. This becomes particularly messy if you mix packages you build yourself with packages grabbed from a repository. I wonder if there is a simpler solution which would give a similar feature set. AFAICT, in Debian, the same feature is given via virtual packages: you would have: I don't think virtual-packages entirely fix the issue. AFAIK, virtual packages have two uses: - handle dependencies where several packages may resolve one particular dependency in an equivalent way (one good example is LAPACK: both liblapack and libatlas provides the lapack feature) - closer to this discussion, you can build several variants of the same package, and each variant would resolve the dependency on a virtual package handling the commonalities. For example, say we have two numpy packages, one built with lapack (python-numpy-full), the other without (python-numpy-core). What happens when a package foo depends on numpy-full, but numpy-core is installed ? When you type apt-get install my_new_package, the version resolution system will tell you that apt-get will remove python-numpy-core and will install python-numpy-full in the act of installing my_new_package. AFAICS, this can only work as long as the set containing every variant can be ordered (in the conventional set ordering sense), and the dependency can be satisfied by the smallest one. Typically, the dependencies only depend on the smallest subset of what they require (if they don't need lapack, they'd only depend on python-numpy-core in your example), but yes, if there's an unsatisfiable condition, then apt-get will raise an error and abort. In practice, this system seems to work quite well, IMO. Anyhow, here's the full Debian documentation: http://www.debian.org/doc/debian-policy/ch-relationships.html -- This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon's best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
Re: [matplotlib-devel] [SciPy-dev] [Numpy-discussion] Announcing toydist, improving distribution and packaging situation
On Thu, Dec 31, 2009 at 6:06 AM, Darren Dale dsdal...@gmail.com wrote: I should defer to the description of extras in the setuptools documentation. It is only a few paragraphs long: http://peak.telecommunity.com/DevCenter/setuptools#declaring-extras-optional-features-with-their-own-dependencies Ok, so there are two issues related to this feature: - supporting variant at the build stage - supporting different variants of the same package in the dependency graph at install time The first issue is definitely supported - I fixed a bug in toydist to support this correctly, and this will be used when converting setuptools-based setup.py which use the features argument. The second issue is more challenging. It complicates the dependency handling quite a bit, and may cause difficult situations to happen at dependency resolution time. This becomes particularly messy if you mix packages you build yourself with packages grabbed from a repository. I wonder if there is a simpler solution which would give a similar feature set. cheers, David -- This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon's best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
Re: [matplotlib-devel] [SciPy-dev] [Numpy-discussion] Announcing toydist, improving distribution and packaging situation
On Wed, Dec 30, 2009 at 11:16 AM, David Cournapeau courn...@gmail.com wrote: On Wed, Dec 30, 2009 at 11:26 PM, Darren Dale dsdal...@gmail.com wrote: Hi David, On Mon, Dec 28, 2009 at 9:03 AM, David Cournapeau courn...@gmail.com wrote: Executable: grin module: grin function: grin_main Executable: grind module: grin function: grind_main Have you thought at all about operations that are currently performed by post-installation scripts? For example, it might be desirable for the ipython or MayaVi windows installers to create a folder in the Start menu that contains links the the executable and the documentation. This is probably a secondary issue at this point in toydist's development, but I think it is an important feature in the long run. Also, have you considered support for package extras (package variants in Ports, allowing you to specify features that pull in additional dependencies like traits[qt4])? Enthought makes good use of them in ETS, and I think they would be worth keeping. Does this example covers what you have in mind ? I am not so familiar with this feature of setuptools: Name: hello Version: 1.0 Library: BuildRequires: paver, sphinx, numpy if os(windows) BuildRequires: pywin32 Packages: hello Extension: hello._bar sources: src/hellomodule.c if os(linux) Extension: hello._linux_backend sources: src/linbackend.c Note that instead of os(os_name), you can use flag(flag_name), where flag are boolean variables which can be user defined: http://github.com/cournape/toydist/blob/master/examples/simples/conditional/toysetup.info http://github.com/cournape/toydist/blob/master/examples/var_example/toysetup.info I should defer to the description of extras in the setuptools documentation. It is only a few paragraphs long: http://peak.telecommunity.com/DevCenter/setuptools#declaring-extras-optional-features-with-their-own-dependencies Darren -- This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon's best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
