Re: CPAN -- moving forward
On Sat, 30 May 2009, Daniel Ruoso wrote: 1) A package format. This is supposed to be a source format, but different from current model used in CPAN, it's pretty clear already that it can't include a "build system", like ExtUtils::MakeMaker or Module::Install. There's already some consensus that this "source package" format should describe what it contains, not how it should be built or installed. For instance, it should only say something like "in the lib directory there's a set of STD Perl 6 modules that don't require any low-level integration". Would it be accurate to describe this as specifying where things should be installed, but only at a high enough level that it's platform independent? Thanks, - | Name: Tim Nelson | Because the Creator is,| | E-mail: wayl...@wayland.id.au| I am | - BEGIN GEEK CODE BLOCK Version 3.12 GCS d+++ s+: a- C++$ U+++$ P+++$ L+++ E- W+ N+ w--- V- PE(+) Y+>++ PGP->+++ R(+) !tv b++ DI D G+ e++> h! y- -END GEEK CODE BLOCK-
Re: CPAN -- moving forward
On Mon, Jun 01, 2009 at 06:34:02PM +0200, Daniel Carrera wrote: > I am in IRC > working with Rakudo folk on how Rakudo is going to store modules on the > disk. Once that is done, one can begin talking about a package format > and an installer, and then go from there. > > So far the discussion has been productive and we have some code written > that we can experiment with. So I feel encouraged. "Rough concensus and running code." +1 Pm
Re: CPAN -- moving forward
Parrot Raiser wrote: One of the common factors that has contributed to the longevity of Unix (in the generic sense), and the Internet, is their layered architectures. +1 for layered architectures. If this discussion can be split into clear layers, (what gets stored, where, how, &c.) it might be easier to produce results. I feel similar. I attempted to do something like that with my wiki page, but that didn't work so well. Now I'm trying a new approach: I am in IRC working with Rakudo folk on how Rakudo is going to store modules on the disk. Once that is done, one can begin talking about a package format and an installer, and then go from there. So far the discussion has been productive and we have some code written that we can experiment with. So I feel encouraged. Daniel.
Re: CPAN -- moving forward
Part of the tension here may be coming from the attempt to debate too many levels of structure at once. One of the common factors that has contributed to the longevity of Unix (in the generic sense), and the Internet, is their layered architectures. The kernel does its thing, the shell sits on top, shell tools and applications attend to their matters, and as long as they abide by some standard conventions, revolutions can occur at one level without disturbing others. Similarly, co-ax, twisted pair, power lines, fibre-optics and pigeons can all carry IP packets for UDP or TCP, that might be VoIP or HTTP, and apart from some latency issues with the pigeons, the various layers don't care what's happening either side of them. If this discussion can be split into clear layers, (what gets stored, where, how, &c.) it might be easier to produce results.
Re: CPAN -- moving forward
On Sun, May 31, 2009 at 12:22:31AM +0200, Daniel Carrera wrote: > Mark Overmeer wrote: >> A pity you didn't want to read the paper. > > I have better things to do with my life than read your 30-page paper. > I'd rather participate in a consensus process where I feel I can make a > difference. What is this, a contest to see who can be snippier? >> Please clarify ... how would you specify that? And how >> would you denote ranges of conflicting packages? Well, maybe start >> with putting that in your wishlist. >> >> And conflicting/dependencies in licenses? >> How would you resolve dependency conflicts? Design human intervention >> in this process of dependency processing. >> [big snip] > > My wiki page was purposely only an initial step. I don't believe in > working in isolation for days or weeks and then handing down a massive > ready-made solution to the masses. I offer a small, incremental step on > top of Synopsis 22 and hope that it is useful. People have different work styles. Part of consensus building is not expecting everyone to think the same way you do. We try to put both the tortoises and hares to work here, but each has to work at their own pace, and some temporal displacement necessarily results. A healthy community can put up with a certain amount of tension and compression. It's called "tensegrity" when it helps hold us all together. It's probably called something else when it drives us apart. Please aim for the tensegrity, because I don't want to figure out what to call the other. Larry
Re: CPAN -- moving forward
Daniel Ruoso wrote: The leap you make from the source package to the different binary formats is overlooking a lot of details. It would be interesting if you could take a look in the previous discussions on the matter. I'll be happy to. I was just trying to make a small iterative step on Synopsis 22. I know that there is a wide gap between that and binary formats. But I wouldn't do anyone any favours if I spend a lot of time on my own trying to make a complete solution. At least now you can point me to the relevant discussion and I'll be happy to read it. I just need a URL link. 1) A package format. This is supposed to be a source format, but different from current model used in CPAN, it's pretty clear already that it can't include a "build system", like ExtUtils::MakeMaker or Module::Install. There's already some consensus that this "source package" format should describe what it contains, not how it should be built or installed. For instance, it should only say something like "in the lib directory there's a set of STD Perl 6 modules that don't require any low-level integration". Ok. Thanks for the info. Notice that I didn't assume anything one way or the other on the page I wrote. Just some additions to S22 (mainly the manifest file). I tried not to make many assumptions precisely because I don't know the previous discussion. 2.0) Build tool Before installing it, you need to create a "installable package", or "binary package" (that's what CPAN plus does today). The thing here is that the process of transforming the "source package" into an "installable package" will be specific to the Perl implementation, to the Operating System and to the vendor of the Operating system. That basically means "it's implementation specific", and each implementation should do its best to provide that support. For instance, rakudo might want to compile the modules to Parrot Bytecode, while mildew might want to compile down some things to C. Seems logical. If you're in an OS that provides a rich package management system, it means you can generate native packages, otherwise you need to implement the next step as well, which is: 2.5) Install tool In systems where we don't generate native packages, we need a package manager on our own. It should be capable of taking any "installable package" and making it available in the system, checking all dependencies and other requirements.. Yes, certainly. 3) A high-level install tool, analogous to yum or apt, that uses the CPAN network and resolves dependencies. I do think this is very much implementation-specific, for instance, in Debian, with little work you could simply use apt-build to get your packages built and available for installation with apt-get. In Debian there is both apt-get and aptitude. Two different implementations of a high-level package manager wrapped around dpkg. In the RPM world there is yum and urpmi as well. In summary, The part 1 is really the critical issue, 2.0, 2.5 and 3 are mostly implementation specific and are considerably easier to adapt than the things in the part 1. We do need to find a very solid way of describing what the package contains in order that implementation-specific build tools can make them work. Yes, indeed. Most of what I put in the wiki page is an iterative step on the package format described in Synopsis 22. Should we start talking about the package format then? Daniel.
Re: CPAN -- moving forward
Em Sáb, 2009-05-30 às 22:54 +0200, Daniel Carrera escreveu: > In the hopes of helping the CPAN discussion move forward, in the > direction of tangible work, I have made a wiki page with a proposal: > Please read the "Basics" section, which is quite short. The main point > of this section is to divide the issue into three parts: Daniel, The leap you make from the source package to the different binary formats is overlooking a lot of details. It would be interesting if you could take a look in the previous discussions on the matter. > 1) A package format. This is supposed to be a source format, but different from current model used in CPAN, it's pretty clear already that it can't include a "build system", like ExtUtils::MakeMaker or Module::Install. There's already some consensus that this "source package" format should describe what it contains, not how it should be built or installed. For instance, it should only say something like "in the lib directory there's a set of STD Perl 6 modules that don't require any low-level integration". > 2) A low-level install tool, analogous to rpm or dpkg, that converts the > package (2) into a local package format (rpm, deb, ebuild). 2.0) Build tool Before installing it, you need to create a "installable package", or "binary package" (that's what CPAN plus does today). The thing here is that the process of transforming the "source package" into an "installable package" will be specific to the Perl implementation, to the Operating System and to the vendor of the Operating system. That basically means "it's implementation specific", and each implementation should do its best to provide that support. For instance, rakudo might want to compile the modules to Parrot Bytecode, while mildew might want to compile down some things to C. If you're in an OS that provides a rich package management system, it means you can generate native packages, otherwise you need to implement the next step as well, which is: 2.5) Install tool In systems where we don't generate native packages, we need a package manager on our own. It should be capable of taking any "installable package" and making it available in the system, checking all dependencies and other requirements.. > 3) A high-level install tool, analogous to yum or apt, that uses the > CPAN network and resolves dependencies. I do think this is very much implementation-specific, for instance, in Debian, with little work you could simply use apt-build to get your packages built and available for installation with apt-get. In summary, The part 1 is really the critical issue, 2.0, 2.5 and 3 are mostly implementation specific and are considerably easier to adapt than the things in the part 1. We do need to find a very solid way of describing what the package contains in order that implementation-specific build tools can make them work. daniel
Re: CPAN -- moving forward
Mark Overmeer wrote: A pity you didn't want to read the paper. I have better things to do with my life than read your 30-page paper. I'd rather participate in a consensus process where I feel I can make a difference. Please clarify ... how would you specify that? And how would you denote ranges of conflicting packages? Well, maybe start with putting that in your wishlist. And conflicting/dependencies in licenses? How would you resolve dependency conflicts? Design human intervention in this process of dependency processing. [big snip] My wiki page was purposely only an initial step. I don't believe in working in isolation for days or weeks and then handing down a massive ready-made solution to the masses. I offer a small, incremental step on top of Synopsis 22 and hope that it is useful. Daniel.
Re: CPAN -- moving forward
* Daniel Carrera (daniel.carr...@theingots.org) [090530 20:54]: > 3) A high-level install tool, analogous to yum or apt, that uses the > CPAN network and resolves dependencies. > Mark O. is most interested in (3). These are all things which I do *not* play in the layers I want to build. Although I do have an opinion on these subjects (if you know me: I have opinions on everything) but they are all not included in my design. Wrong abstraction layer. A pity you didn't want to read the paper. My Archiver implementation defines ways how to specify the meta-data for dependencies, because that is a common need, but does not resolve dependencies. It provide sufficient means of search so you can investigate the dependency details before you start downloading releases, but in general, there are nasty application dependent interpretations in these details. Interpreting dependency information sometimes requires human intervension. For instance: do you want to upgrade to a higher version which breaks api (and triggers the upgrade of a dozen other modules on your system) or stay in the maintenance track of the older api. (example is Mail:: SpamAssassin 2 vs 3) These problems cannot be resolved server-side. Apt or yum are just like CPAN install tools. Nothing more. So outside my personal scope. (But important to Perl6) Please clarify in your spec what you mean with "package" in a dependency. Is it about a single distributed version (what I call a "release" into the archive), or a release with a certain product name, or with a certain tag. Or any of those: how would you specify that? And how would you denote ranges of conflicting packages? Well, maybe start with putting that in your wishlist. And conflicting/dependencies in licenses? How would you resolve dependency conflicts? Design human intervention in this process of dependency processing. How do you handle external dependencies to Linux distribution components like database software, which change name with each new release of each linux distribution? Will we lockin Perl to a few well known distributions, or do week keep our trandition open mind. So: please pick from http://lwn.net/Distributions/ (currently 548 distros listed) which you want to support... and wait for other people to add more. Perl is one of the last pure English open source projects. When you install a localized KDE or Gnome *everything* (>95%) gets translated. But not Perl! You must include description and keywords in various languages. Can other people contribute translations? How will they add those to CPAN? The Meta.YML specs are already further than what you propose. Go to a Perl QA Hackathon and see how hard people work to get Meta.YML extended! (last year Oslo, last March in Birmingham) Reuse that. > I propose that we work on (1) and (2) first and do (3) later. (3) is > quite independent of (1) and (2). Mark might want to check that the > package format includes enough meta-data to not conflict with is ideas. There is no way that it can conflict, as long as it is well defined. Have a look at the Kwalify module, or use the horribly (but more widely known) XML Schema's to make it well-defined. My very extended meta-data design probably gives you enough to shop for your wishlist. But I may lack a few things and can add those easily on this stage of the project. Or you can put those in the application specific meta-data block which Pause6/CPAN6 supports. (Which is a pity: unification of meta- data has many advantanges) As examples of how you could adapt your design to my plans: . you do not need to organize file signatures, because those are (far more flexible) already provided on the transport layer of CPAN6. (where it fits logically) . you do not need any packing or unpacking of trees, because that is also part of my transport layer: sender and receiver decide whether they pack temporarily or not. . I offer nice standard solutions for identity management (authors) and licenses. But these are probably automatically translatable. I will keep an eye on your growing list of meta-data needs, and see whether there appear things which are not covered in my specs yet. We need more and better meta-data, not a reinvented wheel. -- MarkOv Mark Overmeer MScMARKOV Solutions m...@overmeer.net soluti...@overmeer.net http://Mark.Overmeer.net http://solutions.overmeer.net
CPAN -- moving forward
Hello, In the hopes of helping the CPAN discussion move forward, in the direction of tangible work, I have made a wiki page with a proposal: http://wiki.github.com/perl6/misc/cpan-and-package-format Please read the "Basics" section, which is quite short. The main point of this section is to divide the issue into three parts: 1) A package format. 2) A low-level install tool, analogous to rpm or dpkg, that converts the package (2) into a local package format (rpm, deb, ebuild). 3) A high-level install tool, analogous to yum or apt, that uses the CPAN network and resolves dependencies. I have used the same numeric labels on the wiki. Tim N. is most interested in (2) and Mark O. is most interested in (3). I propose that we work on (1) and (2) first and do (3) later. (3) is quite independent of (1) and (2). Mark might want to check that the package format includes enough meta-data to not conflict with is ideas. If we can "mostly" agree on the package (1), we can move to (2). I think we can make quick progress with (2): 2.1 Start with a easy&ugly script using the alien and zip commands. Get the basic functionality down. 2.2 Gradually port Perl 5 modules (e.g. Software::Packager) and rewrite the script to remove dependency on shell commands. Think of it as a boot-strapping method. What do you think? Daniel.
