Re: [matplotlib-devel] [SciPy-dev] [Numpy-discussion] Announcing toydist, improving distribution and packaging situation
On Sat, Jan 2, 2010 at 4:58 PM, Gael Varoquaux wrote: > On Sat, Jan 02, 2010 at 11:32:00AM +0900, David Cournapeau wrote: >> [snip] >> - supporting different variants of the same package in the >> dependency graph at install time > >> [snip] > >> The second issue is more challenging. It complicates the dependency >> handling quite a bit, and may cause difficult situations to happen at >> dependency resolution time. This becomes particularly messy if you mix >> packages you build yourself with packages grabbed from a repository. I >> wonder if there is a simpler solution which would give a similar >> feature set. > > > AFAICT, in Debian, the same feature is given via virtual packages: you > would have: I don't think virtual-packages entirely fix the issue. AFAIK, virtual packages have two uses: - handle dependencies where several packages may resolve one particular dependency in an equivalent way (one good example is LAPACK: both liblapack and libatlas provides the lapack feature) - closer to this discussion, you can build several variants of the same package, and each variant would resolve the dependency on a virtual package handling the commonalities. For example, say we have two numpy packages, one built with lapack (python-numpy-full), the other without (python-numpy-core). What happens when a package foo depends on numpy-full, but numpy-core is installed ? AFAICS, this can only work as long as the set containing every variant can be ordered (in the conventional set ordering sense), and the dependency can be satisfied by the smallest one. cheers, David -- This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon's best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
Re: [matplotlib-devel] [SPAM] Re: [SciPy-User] [Numpy-discussion] Announcing toydist, improving distribution and packaging situation
2010/1/2 David Cournapeau : > On Fri, Jan 1, 2010 at 10:43 PM, Pierre Raybaut wrote: >> Hi David, >> >> Following your announcement for the 'toydist' module, I think that >> your project is very promising: this is certainly a great idea and it >> will be very controversial but that's because people expectactions are >> great on this matter (distutils is so disappointing indeed). >> >> Anyway, if I may be useful, I'll gladly contribute to it. >> In time, I could change the whole Python(x,y) packaging system (which >> is currently quite ugly... but easy/quick to manage/maintain) to >> use/promote this new module. > > That would be a good way to test toydist on a real, complex package. I > am not familiar at all with python(x,y) internals. Do you have some > explanation I could look at somewhere ? Honestly, let's assume that there is currently no packaging system... it would not be very far from the truth. I did it when I was young and naive regarding Python. Actually I almost did it without having writing any code in Python (approx. two months after earing about the Python language for the first time) : it's an ugly collection of AutoIt, NSIS and PHP scripts -- most of the tasks are automated like updating the generated website pages and so on. So I'm not proud at all, but it was easy and very quick to do as it is, and it's still quite easy to maintain. But, it's not satisfying in terms of code "purity" -- I've been wanting to rewrite all this in Python for a year and a half but since the features are there, there is no real motivation to do the work (in other words, Python(x,y) users would not see the difference, at least at the beginning). An other thing: Python(x,y) plugins are not built from source but from existing binaries (it's a pity I know, but it was incredibly faster to do this way). For example, eggs or distutils .exe may be converted in Python(x,y) plugins directly (same internal directory structure). So it may be different from the idea you had in mind (it's not like EPD which is entirely generated from source, AFAIK). > In the meantime, I will try to clean-up the code to have a first > experimental release. > Ok, keep up the good work! Cheers, Pierre -- This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon's best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
Re: [matplotlib-devel] [SciPy-dev] [Numpy-discussion] Announcing toydist, improving distribution and packaging situation
> On Tue, Dec 29, 2009 at 3:49 AM, Dag Sverre Seljebotn > wrote: > >> >> Do you here mean automatic generation of Ubuntu debs, Debian debs, >> Windows >> MSI installer, Windows EXE installer, and so on? (If so then great!) > > Yes (although this is not yet implemented). In particular on windows, > I want to implement a scheme so that you can convert from eggs to .exe > and vice et versa, so people can still install as exe (or msi), even > though the method would default to eggs. > >> If this is the goal, I wonder if one looks outside of Python-land one >> might find something that already does this -- there's a lot of >> different >> package format, "Linux meta-distributions", "install everywhere >> packages" >> and so on. > > Yes, there are things like 0install or autopackage. I think those are > deemed to fail, as long as it is not supported thoroughly by the > distribution. Instead, my goal here is much simpler: producing > rpm/deb. It does not solve every issue (install by non root, multiple > // versions), but one has to be realistic :) > > I think automatically built rpm/deb, easy integration with native > method can solve a lot of issues already. > >> >> - Currently I'm making a Sage SPKG for CHOLMOD. This essentially gets >> the >> job done by not bothering about the problem, not even using the >> OS-installed Python. >> >> Something that would spit out both Sage SPKGs, Ubuntu debs, Windows >> installers, both with Python code and C/Fortran code or a mix (and put >> both in the place preferred by the system in question), seems ideal. Of >> course one would still need to make sure that the code builds properly >> everywhere, but just solving the distribution part of this would be a >> huge >> step ahead. > > On windows, this issue may be solved using eggs: enstaller has a > feature where dll put in a special location of an egg are installed in > python such as they are found by the OS loader. One could have > mechanisms based on $ORIGIN + rpath on linux to solve this issue for > local installs on Linux, etc... > > But again, one has to be realistic on the goals. With toydist, I want > to remove all the pile of magic, hacks built on top of distutils so > that people can again hack their own solutions, as it should have been > from the start (that's a big plus of python in general). It won't > magically solve every issue out there, but it would hopefully help > people to make their own. > > Bundling solutions like SAGE, EPD, etc... are still the most robust > ways to deal with those issues in general, and I do not intended to > replace those. > >> What I'm saying is that this is a software distribution problem in >> general, and I'm afraid that Python-specific solutions are too narrow. > > Distribution is a hard problem. Instead of pushing a very narrow (and > mostly ill-funded) view of how people should do things like > distutils/setuptools/pip/buildout do, I want people to be able to be > able to build their own solutions. No more "use this magic stick v > 4.0.3.3.14svn1234, trust me it work you don't have to understand" > which is too prevalant with those tools, which has always felt deeply > unpythonic to me. Thanks, this cleared things up, and I like the direction this is heading. Thanks a lot for doing this! Dag Sverre -- This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon's best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
Re: [matplotlib-devel] [Numpy-discussion] Announcing toydist, improving distribution and packaging situation
David wrote: > Repository > > > The goal here is to have something like CRAN > (http://cran.r-project.org/web/views/), ideally with a build farm so > that whenever anyone submits a package to our repository, it would > automatically be checked, and built for windows/mac os x and maybe a > few major linux distributions. One could investigate the build service > from open suse to that end (http://en.opensuse.org/Build_Service), > which is based on xen VM to build installers in a reproducible way. Do you here mean automatic generation of Ubuntu debs, Debian debs, Windows MSI installer, Windows EXE installer, and so on? (If so then great!) If this is the goal, I wonder if one looks outside of Python-land one might find something that already does this -- there's a lot of different package format, "Linux meta-distributions", "install everywhere packages" and so on. Of course, toydist could have such any such tool as a backend/in a pipeline. > What's next ? > == > > At this point, I would like to ask for help and comments, in particular: > - Does all this make sense, or hopelessly intractable ? > - Besides the points I have mentioned, what else do you think is needed ? Hmm. What I miss is the discussion of other native libraries which the Python libraries need to bundle. Is it assumed that one want to continue linking C and Fortran code directly into Python .so modules, like the scipy library currently does? Let me take CHOLMOD (sparse Cholesky) as an example. - The Python package cvxopt use it, simply by linking about 20 C files directly into the Python-loadable module (.so) which goes into the Python site-packages (or wherever). This makes sure it just works. But, it doesn't feel like the right way at all. - scikits.sparse.cholmod OTOH simple specifies libraries=["cholmod"], and leave it up to the end-user to make sure it is installed. Linux users with root access can simply apt-get, but it is a pain for everybody else (Windows, Mac, non-root Linux). - Currently I'm making a Sage SPKG for CHOLMOD. This essentially gets the job done by not bothering about the problem, not even using the OS-installed Python. Something that would spit out both Sage SPKGs, Ubuntu debs, Windows installers, both with Python code and C/Fortran code or a mix (and put both in the place preferred by the system in question), seems ideal. Of course one would still need to make sure that the code builds properly everywhere, but just solving the distribution part of this would be a huge step ahead. What I'm saying is that this is a software distribution problem in general, and I'm afraid that Python-specific solutions are too narrow. Dag Sverre -- This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon's best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
Re: [matplotlib-devel] [SciPy-dev] [Numpy-discussion] Announcing toydist, improving distribution and packaging situation
David Cournapeau wrote: > On Sat, Jan 2, 2010 at 4:58 PM, Gael Varoquaux > wrote: > >> On Sat, Jan 02, 2010 at 11:32:00AM +0900, David Cournapeau wrote: >> >>> [snip] >>> - supporting different variants of the same package in the >>> dependency graph at install time >>> >>> [snip] >>> >>> The second issue is more challenging. It complicates the dependency >>> handling quite a bit, and may cause difficult situations to happen at >>> dependency resolution time. This becomes particularly messy if you mix >>> packages you build yourself with packages grabbed from a repository. I >>> wonder if there is a simpler solution which would give a similar >>> feature set. >>> >> AFAICT, in Debian, the same feature is given via virtual packages: you >> would have: >> > > I don't think virtual-packages entirely fix the issue. AFAIK, virtual > packages have two uses: > - handle dependencies where several packages may resolve one > particular dependency in an equivalent way (one good example is > LAPACK: both liblapack and libatlas provides the lapack feature) > - closer to this discussion, you can build several variants of the > same package, and each variant would resolve the dependency on a > virtual package handling the commonalities. For example, say we have > two numpy packages, one built with lapack (python-numpy-full), the > other without (python-numpy-core). What happens when a package foo > depends on numpy-full, but numpy-core is installed ? When you type "apt-get install my_new_package", the version resolution system will tell you that apt-get will remove python-numpy-core and will install python-numpy-full in the act of installing my_new_package. > AFAICS, this can > only work as long as the set containing every variant can be ordered > (in the conventional set ordering sense), and the dependency can be > satisfied by the smallest one. > Typically, the dependencies only depend on the smallest subset of what they require (if they don't need lapack, they'd only depend on python-numpy-core in your example), but yes, if there's an unsatisfiable condition, then apt-get will raise an error and abort. In practice, this system seems to work quite well, IMO. Anyhow, here's the full Debian documentation: http://www.debian.org/doc/debian-policy/ch-relationships.html -- This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon's best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
Re: [matplotlib-devel] docs are now built after every commit & sphinx latex output error
Andrew Straw wrote: > In fact, does anyone know what could be wrong? The last lines of the > LaTeX output are below. > OK, this cropped up on another buildbot for another project of mine -- it looks with a Sphinx dependency, Pygments 1.2.1 (just released), there is some issue that wasn't there with Pygments 1.1.1. I'll try to isolate the issue and report it to the appropriate tracker, but at this point I don't think it's an MPL issue. -Andrew -- This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon's best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
Re: [matplotlib-devel] docs are now built after every commit & sphinx latex output error
Andrew Straw wrote: > Andrew Straw wrote: > >> In fact, does anyone know what could be wrong? The last lines of the >> LaTeX output are below. >> >> > OK, this cropped up on another buildbot for another project of mine -- > it looks with a Sphinx dependency, Pygments 1.2.1 (just released), there > is some issue that wasn't there with Pygments 1.1.1. I'll try to isolate > the issue and report it to the appropriate tracker, but at this point I > don't think it's an MPL issue. > For the record, I reported it here: http://dev.pocoo.org/projects/pygments/ticket/463 -- This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon's best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
