RE: [RFC] CPAN6 requirements analysis
On Fri, 29 May 2009, Jesse Vincent wrote: Making binary distribution easy is a laudable goal, but it's something the existing infrastructure already supports. I'd love to see CPAN autobuilders which build perl modules for a givven platform and architecture and make them generally available. That'd be a lot cleaner than the current ad-hoc system which requires authors to jump through hoops. But really, that's just another CPAN-related service that could easily layer on the existing infrastructure. Note that this already exists with the PPM (Perl Package Manager) repositories. ActiveState PPM build machines attempt to build all CPAN uploads for a variety of architectures: http://ppm4.activestate.com/ You can see that there are a number of other people who also maintain repositories, but typically only for Windows. Of course the main problem with binary package distributions is that there are several Configure options that will generate Perl binaries that are not binary compatible with each other. So the ActiveState PPM repositories only work for ActivePerl, or any other Perl configured in the same way. Cheers, -Jan
Re: [RFC] CPAN6 requirements analysis
I support the notion of distributing binaries because nobody's gonna want to chew up their phone's battery doing unnecessary compiles. The ecology of computing devices is different from ten years ago. And, in fact, binary distribution is something that's been done on CPAN for quite a while. A quick poke at the backpan finds things like http://backpan.perl.org/authors/id/G/GS/GSAR/DB_File-1.54-bin-x86-mswin32-bc.zip dating from over a decade a year ago. The motivations for binary distribution have changed, but the need has been there all along. The only thing I'd worry about here is making one of the big mistakes that the Java community made -- Making binary-only distribution an acceptable alternative to source distribution. Nothing should end up on the CPAN unless it's there in source form. Making binary distribution easy is a laudable goal, but it's something the existing infrastructure already supports. I'd love to see CPAN autobuilders which build perl modules for a givven platform and architecture and make them generally available. That'd be a lot cleaner than the current ad-hoc system which requires authors to jump through hoops. But really, that's just another CPAN-related service that could easily layer on the existing infrastructure. Best, Jesse pgpwO3tBWEibH.pgp Description: PGP signature
Re: [RFC] CPAN6 requirements analysis
* Timothy S. Nelson (wayl...@wayland.id.au) [090530 02:15]: * PAUSE6; this is an actual network based on the CPAN6 software (see above). It also is not documented here. Pause6 is one implementation of archive maintenance software. In the first version written in Perl5, it implements things like I'm still not sure we're on the same page. Let me give an example: - Perl6 is a specification - Rakudo is an implementation of that specification - www.example.com runs a copy of Rakudo (to serve their site) So we have: - Specification - Implementation - Installation Which of those is CPAN6? Which is Pause6? Your use of the word implementation is confusing me; I'm saying Pause6 is the installation (accessible to the world). Is that what you're saying? CPAN6 is like IP: routing and exchanging packages. Quite small and stupid. Pause6 is like TCP: knows how to get the right packages. Install tools are like HTTP: understand the content of packages. You should avoid ever changing the IP level. But next to TCP, you have UDP and so on, for different package handling needs. (On that level, I define connections to external archive-like applications, like OpenID) So: CPAN6 has a spec and an implementation Pause6 has a spec and an implementation Install tools may get a spec and implementation (part of this discussion thread, but not of my project) CPAN6 will run on all servers with this group-sharing concept. It enables playing with large number of archives and mirrors: required to move from the solo namespace of current CPAN into multiple namespaces. Pause6 may be replaced eventually, gradulary, with other implementations. One CPAN6 server can run different implementations and configurations in parallel. Install tools for Perl will be installed where people run Perl and use an archive which contains Perl. Of course, other kinds of applications can use the archive data. -- MarkOv Mark Overmeer MScMARKOV Solutions m...@overmeer.net soluti...@overmeer.net http://Mark.Overmeer.net http://solutions.overmeer.net
Re: [RFC] CPAN6 requirements analysis
* Timothy S. Nelson (wayl...@wayland.id.au) [090530 03:11]: On Fri, 29 May 2009, Alex Elsayed wrote: Instead, it would go to the distributions, who are already well-prepared to handle packaging. We'd just be providing the tools and material they need to do so. Let me reiterate that, while I'd love to have the complete CPAN available pre-packaged for my distro, it doesn't seem like that's necessarily going to happen. Certainly not if you need recent releases of modules, because someone fixed a bug for you. It can easily take a year before a distribution picks that up. Thus, what I want is a tool that will allow me to download the source, turn it into a package for my distro, and install it. This is why I want the Software::Packager solution. Empower the user :). But that module does not contain anything which is required to produce packages for Linux distributions. Well, ok, I should say: it does not handle any of the complex things. Yesterday, I drew a picture of how I see services to happen. Having an automated service which is specialized in building debs and such. Now, such tools do exist, but are usually not published by the distribution makers. Maybe, you can collect those (and unify them) Could be far more complex than you wish for. -- Regards, MarkOv Mark Overmeer MScMARKOV Solutions m...@overmeer.net soluti...@overmeer.net http://Mark.Overmeer.net http://solutions.overmeer.net
Re: [RFC] CPAN6 requirements analysis
On Sat, 30 May 2009, Mark Overmeer wrote: * Timothy S. Nelson (wayl...@wayland.id.au) [090530 03:11]: On Fri, 29 May 2009, Alex Elsayed wrote: Instead, it would go to the distributions, who are already well-prepared to handle packaging. We'd just be providing the tools and material they need to do so. Let me reiterate that, while I'd love to have the complete CPAN available pre-packaged for my distro, it doesn't seem like that's necessarily going to happen. Certainly not if you need recent releases of modules, because someone fixed a bug for you. It can easily take a year before a distribution picks that up. ...or if I need a rare package that I need but isn't commonly requested. Thus, what I want is a tool that will allow me to download the source, turn it into a package for my distro, and install it. This is why I want the Software::Packager solution. Empower the user :). But that module does not contain anything which is required to produce packages for Linux distributions. Well, ok, I should say: it does not handle any of the complex things. Well, it should have the metadata; are you saying it won't have the build-install scripts? What decides how it gets built and installed then? Yesterday, I drew a picture of how I see services to happen. Having an automated service which is specialized in building debs and such. Now, such tools do exist, but are usually not published by the distribution makers. Maybe, you can collect those (and unify them) Could be far more complex than you wish for. Well, presumably, completely new ones will need to be developed to use this new framework[*] we're talking about. If we provide an interface that does some of the work for them, they might make adaptor modules for that interface instead of completely rolling their own. That's what I'm hoping for. [*] I'm using the word framework in the vaguest general sense here, that covers everything we've been discussing. HTH, - | Name: Tim Nelson | Because the Creator is,| | E-mail: wayl...@wayland.id.au| I am | - BEGIN GEEK CODE BLOCK Version 3.12 GCS d+++ s+: a- C++$ U+++$ P+++$ L+++ E- W+ N+ w--- V- PE(+) Y+++ PGP-+++ R(+) !tv b++ DI D G+ e++ h! y- -END GEEK CODE BLOCK-
Re: [RFC] CPAN6 requirements analysis
Alex Elsayed wrote: On Thursday 28 May 2009 4:54:50 pm Daniel Carrera wrote: On the other hand, distributing Parrot bytecode (or PIR, or PASM) seems fine. But I don't know what to suggest for modules that require a C compiler. The problem with that is that Rakudo isn't the Official impelentation, and never will be. Distributing modules as Parrot bytecode would lock out other implementations, something that is very strongly discouraged. I know that Rakudo is not the official implementation. The problem is that you misunderstood my post. I did not say to distribute PIR to the exclusion of Perl source. You know that I was replying to Larry's comment that he supported the notion of distributing binaries. Surely you didn't think that Larry meant distribute binaries to the exclusion of Perl source, did you? Therefore, my comment is a reply to the binary aspect and the central part of my comment is the problem with modules that require a C compiler. * Collision detection - It becomes impossible to prevent another package from overwriting a file installed this way Nothing is impossible. The first method that crossed your mind may not do it, but that doesn't mean that it can't be done. In any case, the drawbacks are no worse than what the current CPAN shell does today. And the current CPAN shell obviously work fairly well. The current system has room for improvement, but any argument that says it can't work is flawed because it is working right this minute and it has been working for years. Daniel.
Re: [RFC] CPAN6 requirements analysis
Timothy S. Nelson wrote: I can confirm that Redhat supports multiple versions: $ rpm -q kernel kernel-2.6.27.5-117.fc10.i686 kernel-2.6.27.12-170.2.5.fc10.i686 kernel-2.6.27.5-117.local.fc10.i686 AFAIK the way RPM implements multiple versions is by making an entirely different package. Like, for example, if you have Gimp 1.x installed, the package might be called gimp. When Gimp 2.0 comes out, if you don't want to remove the old Gimp, you make an entirely separate package called gimp2 with its own set of versions. The point is that we may have trouble scaling that to cover all the stuff that S11 says. I'm not saying it can't be done. I'm saying it might be difficult, as we'd be using RPM and DEB in a way that they were not intended. Daniel.
Re: [RFC] CPAN6 requirements analysis
* Alex Elsayed (eternal...@gmail.com) [090528 22:17]: While lurking in IRC, I've seen several discussions of what CPAN 6 should look like. I would really like to see a split in terminology being used for the various seperate problems. The traditional confusion about what CPAN is: an archive or an install tool. Package manager discussions are in the process AFTER the install tool: to distribute OS changes to be made. In the messages on the list, I see people merge requirements of these very independent tasks. I think that package managers are not a CPAN related problem at all. The Perl install tool decides which files it wants to have within some file-system tree and versioned environment, and then package managers distribute those files and meta-data. Also, there are various different package managers around for Linux distributions, and they tend to be replaced every few years. If you want people to use Perl modules on their Linux systems in a convenient way, you have to distribute each perl module in all of the existing formats. Of course, a tool like alien can be used to simplify the task of creating all these flavors. IMO, that discussion should go in the direction of additional services: the CPAN archive distributes what authors publish. The install tools (CPAN.pm/CPANPLUS/successors) make that code fit in specific operating systems. As a service, other people can publish the results of their specific module installation via package-managers to the world, such that those people can use they platform native software management tools. Just like search.cpan.org is an independent additional service on the CPAN archive. I personally believe that there are a few requirements for a package format that is sufficient for Perl 6: * It must enable packaging for both binary- and source-based distros * It must enable automatic generation of packages for supported systems (although it may well not be capable of it out of the box) * It must permit (or preferably help with) attempts to support new systems * It must be simple to submit packages in the correct format * It must enable the design and building of an automatic testing system The worst flaws in software design are based on the idea that you can organize the outside world. The Perl community will never be able to push its packaging mechanism into Linux distributions. We may be able to select the ideal packaging mechanism, and then they will wrap that in their own packaging mechanism. IMO, as Perl community, we should focus on the things we are good in: have people contribute free Perl code in a simple and platform independent way. Let package manager specialists do their job! They will NEVER be satisfied by the choices we make for them. To help package builders do their job simple and correctly, we have to supply them with as good metadata as we can. Which means: . as much as possible (sufficient kwantity) . very explicitly defined structure and content (predictable quality) . added with minimal hassle for the (often lazy) authors of the code (usability) . extensible meta-data structure (future grow path by design) To get these things right is already an increadible amount of work... and no-one else does that for us. There are so many improvements in this area to be made in our current set-up already, which no-one picks up. Does anyone like to see my wishlist? See http://cpan6.org/papers/2008yapceu/ In the last example http://cpan6.org/papers/2008yapceu/cache-release.xml I show much more additional meta-data information than currently provided my Meta.YML, but all very usefull for packaging tools and other services. (I have used XML as syntax, because with XML-schemas I can precisely describe the structure and content: quality!, but you can also transmit the data in YAML, JSON, Data::Dumper,... that is just unimportant syntax!!!) One thing to mention: I have separated some meta-data into own releases as well. For instance, in stead of including all kinds of personal information (like email address) inside the distribution, I let people make a release which contains that information just like a software release. The released software only contains a reference to the personal information. That means that people can upgrade their personal information (make a new release of themselves) when the email address changes. Of course, that exact content of personal information is not an Perl6 issue, but it is solved in my CPAN6. PACKAGING IS NOT OUR BALLGAME!!! -- Regards, MarkOv Mark Overmeer MScMARKOV Solutions m...@overmeer.net soluti...@overmeer.net http://Mark.Overmeer.net http://solutions.overmeer.net
Re: [RFC] CPAN6 requirements analysis
On Fri, 29 May 2009, Daniel Carrera wrote: Timothy S. Nelson wrote: I can confirm that Redhat supports multiple versions: $ rpm -q kernel kernel-2.6.27.5-117.fc10.i686 kernel-2.6.27.12-170.2.5.fc10.i686 kernel-2.6.27.5-117.local.fc10.i686 AFAIK the way RPM implements multiple versions is by making an entirely different package. Like, for example, if you have Gimp 1.x installed, the package might be called gimp. When Gimp 2.0 comes out, if you don't want to remove the old Gimp, you make an entirely separate package called gimp2 with its own set of versions. No, this is not necessary. Let me paste what I said on IRC to Daniel after he sent the mail to which I am replying. [20:00] wayland76 Let me give the kernel example [20:01] wayland76 kernel-2.6.27.5-117.local.fc10.i686 [20:01] wayland76 package = kernel [20:01] wayland76 version = 2.6.27.5 [20:01] wayland76 release = 117-local [20:01] wayland76 distro = fc10 [20:02] wayland76 arch = i686 [20:02] wayland76 (I'm not 100% sure about distro there) The point is that we may have trouble scaling that to cover all the stuff that S11 says. I'm not saying it can't be done. I'm saying it might be difficult, as we'd be using RPM and DEB in a way that they were not intended. Well, I suggest that we do our best within their frameworks, and if necessary, allow people the option of installing outside their package manager if it can't support the features we're offering. Empower the user! Easy things easy, hard things possible. :) - | Name: Tim Nelson | Because the Creator is,| | E-mail: wayl...@wayland.id.au| I am | - BEGIN GEEK CODE BLOCK Version 3.12 GCS d+++ s+: a- C++$ U+++$ P+++$ L+++ E- W+ N+ w--- V- PE(+) Y+++ PGP-+++ R(+) !tv b++ DI D G+ e++ h! y- -END GEEK CODE BLOCK-
Re: [RFC] CPAN6 requirements analysis
On Fri, 29 May 2009, Mark Overmeer wrote: * Alex Elsayed (eternal...@gmail.com) [090528 22:17]: While lurking in IRC, I've seen several discussions of what CPAN 6 should look like. I would really like to see a split in terminology being used for the various seperate problems. The traditional confusion about what CPAN is: an archive or an install tool. Package manager discussions are in the process AFTER the install tool: to distribute OS changes to be made. In the messages on the list, I see people merge requirements of these very independent tasks. Ok, here's some terms I've picked up: These are from Mark's Paper: - CPAN6: a general infrastructure to create and distribute archives - Pause6: one implementation of archive management - CPAN6.pm one transport mechanism to install Perl modules from Pause6 archives (or presumably any CPAN6 server) I've rephrased them in S22 as: * CPAN6; this is a piece of software for managing an archive network (such as Pause6, below). This is not specified in this document; see http://cpan6.org/ * PAUSE6; this is an actual network based on the CPAN6 software (see above). It also is not documented here. * CPAN6.pm; this is a piece of software that starts with what it can get on PAUSE6, and attempts to give you an installed perl module (this is a replacement for CPANPLUS/cpan2dist) I've replicated that here so that Mark et. al. can verify or deny that we're thinking the same thing. Please keep in mind that if I got them wrong, I'll be using the S22 definitions in the rest of this e-mail. The area I'm interested in overlaps mainly with CPAN6.pm. I think that package managers are not a CPAN related problem at all. The Perl install tool decides which files it wants to have within some file-system tree and versioned environment, and then package managers distribute those files and meta-data. The terms you've used here are so general that they could describe any one of a number of things. I'd like some concretification or something. Example? Also, there are various different package managers around for Linux distributions, and they tend to be replaced every few years. If you want people to use Perl modules on their Linux systems in a convenient way, you have to distribute each perl module in all of the existing formats. Of course, a tool like alien can be used to simplify the task of creating all these flavors. Have you seen Software::Packager? It's a Perl5 module that's designed to create a source package* for any supported package manager. Unfortunately, CPANPLUS has never used it. This is the part of the whole discussion that interests me the most, because it's where my problems have lain. Just for the record, I'm interested in having a Software::Packager port/rewrite be part of the installation tool; overlapping what you're calling CPAN6.pm. I'd like to suggest a further terminology split: - 6PAN: command-line install tool - CPAN6.pm: Module relied upon by 6PAN for interactions with Pause6 - Software::Packager: Module relied upon by 6PAN for interactions with the local packaging system, and for installing files. If there are no objections, I'll pop these in S22 as well. Having clarified the terms, I'd urge people not to use the term CPAN without qualifiers in the discussion any more, as it conflates things. IMO, that discussion should go in the direction of additional services: the CPAN archive distributes what authors publish. The install tools (CPAN.pm/CPANPLUS/successors) make that code fit in specific operating systems. As a service, other people can publish the results of their specific module installation via package-managers to the world, such that those people can use they platform native software management tools. Just like search.cpan.org is an independent additional service on the CPAN archive. Are you arguing here for eg. a yum/apt/whatever repository on Pause6? I personally believe that there are a few requirements for a package format that is sufficient for Perl 6: * It must enable packaging for both binary- and source-based distros * It must enable automatic generation of packages for supported systems (although it may well not be capable of it out of the box) Just to clarify, this is what I want :). * It must permit (or preferably help with) attempts to support new systems * It must be simple to submit packages in the correct format * It must enable the design and building of an automatic testing system The worst flaws in software design are based on the idea that you can organize the outside world. The Perl community will never be able to push its packaging mechanism into Linux distributions. We may be able to select the ideal packaging mechanism, and then they will wrap that in their own packaging mechanism. Agreed. I've chosen my package manager, and I'm
Re: [RFC] CPAN6 requirements analysis
* Timothy S. Nelson (wayl...@wayland.id.au) [090529 12:02]: On Fri, 29 May 2009, Mark Overmeer wrote: * CPAN6; this is a piece of software for managing an archive network (such as Pause6, below). This is not specified in this document; see http://cpan6.org/ Yes. It facilitates organization of archives: exchanging releases (f.i. software distributions), without the knowledge about the internal organization of the archive. This layer ensures that various servers do not need to run the same version of the archiving software: alternative implementations can exist as long as they have a minimal subset of features. It contains things like . protocol negotiation . XML, YAML, Data::Dumper syntax? . packaged with tar, cpio, zip, none? . compressed with gzip, bzip2, none? . transport protocol http, ftp, rsync, .. . release addresses: archive name (URI), product name, version name Quite simple. * PAUSE6; this is an actual network based on the CPAN6 software (see above). It also is not documented here. Pause6 is one implementation of archive maintenance software. In the first version written in Perl5, it implements things like . archive maintenance rules . uploaded release inspection . actual storage on disk . trust, licenses, authors . sub-setting and merging archives . mirror synchronization . simple meta-data searches . end-user downloads These are all server-side things: the client code is very small! Client- side, it is more like ftp (downloads are done via ftp, http, etc. The simplest CPAN6 client is this: wget http://cpan6.org/perl5/get?product=XML-LibXML which gives you a copy of the seperate files in the last version of XML-LibXML without packaging or compression via http from a mirror nearby you. Then cd XML-LibXML-0.69/payload/; make install Something like that. So: Pause6 is the logic behind our current Pause upload interface plus all the maintenance tasks from CPAN maintainers. * CPAN6.pm; this is a piece of software that starts with what it can get on PAUSE6, and attempts to give you an installed perl module (this is a replacement for CPANPLUS/cpan2dist) Well, in my current implementation, CPAN6.pm is a faint layer on top of CPAN.pm. It will search a little differently (search on the web or local disk when using a CD, no 02packages.details), but downloads are still managed by LWP. I think that package managers are not a CPAN related problem at all. The Perl install tool decides which files it wants to have within some file-system tree and versioned environment, and then package managers distribute those files and meta-data. The terms you've used here are so general that they could describe any one of a number of things. I'd like some concretification or something. Example? CPAN archive and Perl install tools are fields of the software author: his/her attempt to get software from the own disk usable on someone else's system. Package managers are used by the end-user to have control over their system. This is a different problem, each OS(-version) has other solutions. Have you seen Software::Packager? It's a Perl5 module that's designed to create a source package* for any supported package manager. What I see in the module is two-fold: . some meta-data . some wrapping (tar) On the meta-data side, it is much too UNIX oriented and far too little. It totally lacks any support for platform dependent problems like . which charsets are supported by filesystems . which characters have a special meaning on various OSes, and how do install tools work around encountered problems. (for instance a ':' in a name, unicode package names, ...) . all kinds of important additional meta-data are missing, like license. The Meta.YML is much more complete. . the implementation is very UNIX centric As indicated before, I think the transport wrapper, like tar, is totally unimportant: it should be abstracted as a transport layer, hidden on the application level of installation tools. Just for the record, I'm interested in having a Software::Packager port/rewrite be part of the installation tool; overlapping what you're calling CPAN6.pm. My CPAN6.pm implementation transports a directory of files from the source (author) to the end-user, without knowledge about the content of that directory: whether it is one perl5.tar.gz or a dozen pictures. In parallel, it uses a meta-data file for such release, which contains cpan6, pause6, and release specific meta-data. For Perl5, it will capture all of the Meta.YML info and more. For instance, it will add checksums for all files, and crypto-signatures. I'd like to suggest a further terminology split: - 6PAN: command-line install tool The term 6pan has also been claimed before. I would say pause6 or cpan6 (no capitals) - CPAN6.pm: Module relied upon by 6PAN for interactions with Pause6 As my idea of CPAN6.pm is just a layer on top of
Re: [RFC] CPAN6 requirements analysis
Em Sex, 2009-05-29 às 01:54 +0200, Daniel Carrera escreveu: Larry Wall wrote: I support the notion of distributing binaries because nobody's gonna want to chew up their phone's battery doing unnecessary compiles. The ecology of computing devices is different from ten years ago. By binaries, I assume you mean native binaries, as opposed to Parrot bytecode. The only problem I see is that it may be impractical to ask CPAN mirrors to hold multiple binaries of each module for every OS and every CPU. I really don't think it will be CPAN's job to distribute the binary packages, CPAN should only distribute the source package, as it does today. Binary dependency is something that needs to be handled too closely to the underlying operating system to be made generic. So, I'd expect to have a Debian archive, in the Debian case, hosted by the Debian Perl group (which packages about ~ 500 CPAN modules to Debian today) with the binary packages targetting each of the Debian versions... The same would go for RedHat and other linux distros, while for the Win32 world, we would have something in the lines of what ActiveState already does... But, I insist, CPAN should only store the source packages and the source metadata should only describe what are the sources, and not how to compile and install them... daniel
Re: [RFC] CPAN6 requirements analysis
On Fri, May 29, 2009 at 10:02:20PM +1000, Timothy S. Nelson wrote: On Fri, 29 May 2009, Mark Overmeer wrote: IMO, that discussion should go in the direction of additional services: the CPAN archive distributes what authors publish. The install tools (CPAN.pm/CPANPLUS/successors) make that code fit in specific operating systems. As a service, other people can publish the results of their specific module installation via package-managers to the world, such that those people can use they platform native software management tools. Just like search.cpan.org is an independent additional service on the CPAN archive. Are you arguing here for eg. a yum/apt/whatever repository on Pause6? I believe he is arguing that whatever we end up doing needs to make it easy for an external package-manager to find out what files CPAN6.pm is going to install, and where, and what the dependencies were (both Perl and system libraries). So that the various distributions can make native packages from perl packages automatically (or at least as close to automatically as possible, it gets a little uglier when external libraries are involved). So if there is program foo that is written in Perl that Fedora wants to distribute, they want to be able to list the dependencies when they make the package for foo. If foo requires a few packages, then they just want to say that and then when foo is installed, pull in those packages and any that those depend on, including any non-perl ones that are needed by the perl packages. -ben
Re: [RFC] CPAN6 requirements analysis
On Friday 29 May 2009 1:51:40 am Mark Overmeer wrote: I would really like to see a split in terminology being used for the various seperate problems. The traditional confusion about what CPAN is: an archive or an install tool. Package manager discussions are in the process AFTER the install tool: to distribute OS changes to be made. In the messages on the list, I see people merge requirements of these very independent tasks. Okay, I will readily admit that there is a real need to have there be a more clear separation between these two separate things which are both called CPAN. I think that package managers are not a CPAN related problem at all. The Perl install tool decides which files it wants to have within some file-system tree and versioned environment, and then package managers distribute those files and meta-data. Here I'll have to disagree, for several reasons. * The structure of the archival system will heavily impact the structure of the install tool * If the archive does not provide sufficient information in a useful format, then it may be difficult or impossible to delegate installation to the local package manager. With the current CPAN architecture for Perl 5, it is impossible for an external package manager to calculate dependencies without first downloading the entire /source/ of all modules in the dependency list *due to a limitation of the archive architecture* Also, there are various different package managers around for Linux distributions, and they tend to be replaced every few years. If you want people to use Perl modules on their Linux systems in a convenient way, you have to distribute each perl module in all of the existing formats. Of course, a tool like alien can be used to simplify the task of creating all these flavors. This feels like a straw man to me. * Package managers are actually relatively stable. The .deb, .rpm, and .ebuild formats (just to name three _big_ ones) have been around for at least 10 years each. * We don't need to provide everything in all formats - If the archive structure makes automated conversion easy, then _each_ _distro_ can just run a cron job to * Sync with the CPAN archive * Automagically build packages for their own distro * Publish those packages (PPA for Debian and Ubuntu, overlay for Gentoo) * Alien is really not a very good solution, as it only works for binary distros, not source or hybrid ones (Gentoo, Exherbo, Arch) and even then only a few binary distros are supported IMO, that discussion should go in the direction of additional services: the CPAN archive distributes what authors publish. The install tools (CPAN.pm/CPANPLUS/successors) make that code fit in specific operating systems. As a service, other people can publish the results of their specific module installation via package-managers to the world, such that those people can use they platform native software management tools. Just like search.cpan.org is an independent additional service on the CPAN archive. And once again we run into an issue: This workflow would require that each end user, rather than using /only/ CPAN(PLUS)? or /only/ their package manager to install, they are forced to use one to generate packages, and then the other to install them! By providing a very open metadata spec, we can make it so that neither the Perl community nor the end-users need to take on that burden. Instead, it would go to the distributions, who are already well-prepared to handle packaging. We'd just be providing the tools and material they need to do so. I personally believe that there are a few requirements for a package format that is sufficient for Perl 6: * It must enable packaging for both binary- and source-based distros * It must enable automatic generation of packages for supported systems (although it may well not be capable of it out of the box) * It must permit (or preferably help with) attempts to support new systems * It must be simple to submit packages in the correct format * It must enable the design and building of an automatic testing system The worst flaws in software design are based on the idea that you can organize the outside world. The Perl community will never be able to push its packaging mechanism into Linux distributions. We may be able to select the ideal packaging mechanism, and then they will wrap that in their own packaging mechanism. As I said above, it doesn't need to. What my mail was suggesting was mainly a very information-rich metadata spec, which would contain enough information that the distros could package the modules without needing much more than that. * For my first point, it merely says that the metadata must contain enough information to run an installation process, which is usually the information needed to build a distro package * The second bulled adds the constraint that this should be doable in an automated fashion * The third bullet adds the
Re: [RFC] CPAN6 requirements analysis
I believe he is arguing that whatever we end up doing needs to make it easy for an external package-manager to find out what files CPAN6.pm is going to install, and where, and what the dependencies were (both Perl and system libraries). So that the various distributions can make native packages from perl packages automatically (or at least as close to automatically as possible, it gets a little uglier when external libraries are involved). I am completely in agreement here. So if there is program foo that is written in Perl that Fedora wants to distribute, they want to be able to list the dependencies when they make the package for foo. If foo requires a few packages, then they just want to say that and then when foo is installed, pull in those packages and any that those depend on, Agreed. including any non-perl ones that are needed by the perl packages. Here is where it gets sticky. Let's take a perl module which requires the Berkely DB - in fact, requires a specific API version. In Gentoo, you'd specify that with a slot dependency, while on Fedora you'd specify the release. The two distros may well use different names for the package and/or the slot/release specifier. This problem strikes me as intractable - I think the only thing we can do is provide a dependency specifier, clearly tagged as being external to the CPAN 6 archive, with a sensible name that allows a human to intervene and find the correct package for their distro.
Re: [RFC] CPAN6 requirements analysis
On Fri, 29 May 2009, Mark Overmeer wrote: * PAUSE6; this is an actual network based on the CPAN6 software (see above). It also is not documented here. Pause6 is one implementation of archive maintenance software. In the first version written in Perl5, it implements things like I'm still not sure we're on the same page. Let me give an example: - HTTP is a specification - Apache is an implementation of a server for that specification - www.example.com runs a copy of Apache Or another example: - Perl6 is a specification - Rakudo is an implementation of that specification - www.example.com runs a copy of Rakudo (to serve their site) So we have: - Specification - Implementation - Installation Which of those is CPAN6? Which is Pause6? Your use of the word implementation is confusing me; I'm saying Pause6 is the installation (accessible to the world). Is that what you're saying? HTH, - | Name: Tim Nelson | Because the Creator is,| | E-mail: wayl...@wayland.id.au| I am | - BEGIN GEEK CODE BLOCK Version 3.12 GCS d+++ s+: a- C++$ U+++$ P+++$ L+++ E- W+ N+ w--- V- PE(+) Y+++ PGP-+++ R(+) !tv b++ DI D G+ e++ h! y- -END GEEK CODE BLOCK-
Re: [RFC] CPAN6 requirements analysis
I know that Rakudo is not the official implementation. The problem is that you misunderstood my post. I did not say to distribute PIR to the exclusion of Perl source. You know that I was replying to Larry's comment that he supported the notion of distributing binaries. Surely you didn't think that Larry meant distribute binaries to the exclusion of Perl source, did you? Therefore, my comment is a reply to the binary aspect and the central part of my comment is the problem with modules that require a C compiler. Sorry about that then. * Collision detection - It becomes impossible to prevent another package from overwriting a file installed this way Nothing is impossible. The first method that crossed your mind may not do it, but that doesn't mean that it can't be done. In any case, the drawbacks are no worse than what the current CPAN shell does today. And the current CPAN shell obviously work fairly well. The current system has room for improvement, but any argument that says it can't work is flawed because it is working right this minute and it has been working for years. It may not be impossible, but the only way to achieve that without making ebuilds and letting the package manager handle the package directly (on Gentoo at least) seem to be limited to Support the (non-standardized) VDB database for installed packages internally. Which is a pretty shitty road to go down. Also, the Gentoo developers have flat-out declared that they will not officially support modules installed with the CPAN shell because of these issues, so the CPAN shell actually _doesn't_ work fairly well on Gentoo. It gives you an environment where your install is considered to be out of the control of the package manager and, therefore, a spaghetti mess where you have multiple package managers which overlap and interfere with each other, yet don't have any mechanism for cooperation, synchronization, or communication.
Re: [RFC] CPAN6 requirements analysis
On Fri, 29 May 2009, Daniel Ruoso wrote: So, I'd expect to have a Debian archive, in the Debian case, hosted by the Debian Perl group (which packages about ~ 500 CPAN modules to Debian today) with the binary packages targetting each of the Debian versions... The same would go for RedHat and other linux distros, while for the Win32 world, we would have something in the lines of what ActiveState already does... But, I insist, CPAN should only store the source packages and the source metadata should only describe what are the sources, and not how to compile and install them... Nononono. Some of this is what I'm trying to get away from. I semi-regularly want to use packages that have never been RPM-ed by anyone, or at least not a recent version. I want an auto-building tool. I could live without CPAN-or-whatever having the binaries stored in a public place, but I want to be able to auto-generate an RPM from and CPAN package. I hope we can work out a sensible solution that keeps us all happy. I'm following the discussions of DanielC and MarkOv with interest. - | Name: Tim Nelson | Because the Creator is,| | E-mail: wayl...@wayland.id.au| I am | - BEGIN GEEK CODE BLOCK Version 3.12 GCS d+++ s+: a- C++$ U+++$ P+++$ L+++ E- W+ N+ w--- V- PE(+) Y+++ PGP-+++ R(+) !tv b++ DI D G+ e++ h! y- -END GEEK CODE BLOCK-
Re: [RFC] CPAN6 requirements analysis
On Fri, 29 May 2009, Alex Elsayed wrote: This problem strikes me as intractable - I think the only thing we can do is provide a dependency specifier, clearly tagged as being external to the CPAN 6 archive, with a sensible name that allows a human to intervene and find the correct package for their distro. ...and a store of common mappings for various distros :). - | Name: Tim Nelson | Because the Creator is,| | E-mail: wayl...@wayland.id.au| I am | - BEGIN GEEK CODE BLOCK Version 3.12 GCS d+++ s+: a- C++$ U+++$ P+++$ L+++ E- W+ N+ w--- V- PE(+) Y+++ PGP-+++ R(+) !tv b++ DI D G+ e++ h! y- -END GEEK CODE BLOCK-
Re: [RFC] CPAN6 requirements analysis
On Fri, 29 May 2009, Alex Elsayed wrote: IMO, that discussion should go in the direction of additional services: the CPAN archive distributes what authors publish. The install tools (CPAN.pm/CPANPLUS/successors) make that code fit in specific operating systems. As a service, other people can publish the results of their specific module installation via package-managers to the world, such that those people can use they platform native software management tools. Just like search.cpan.org is an independent additional service on the CPAN archive. And once again we run into an issue: This workflow would require that each end user, rather than using /only/ CPAN(PLUS)? or /only/ their package manager to install, they are forced to use one to generate packages, and then the other to install them! By providing a very open metadata spec, we can make it so that neither the Perl community nor the end-users need to take on that burden. Instead, it would go to the distributions, who are already well-prepared to handle packaging. We'd just be providing the tools and material they need to do so. Let me reiterate that, while I'd love to have the complete CPAN available pre-packaged for my distro, it doesn't seem like that's necessarily going to happen. Thus, what I want is a tool that will allow me to download the source, turn it into a package for my distro, and install it. This is why I want the Software::Packager solution. Empower the user :). Also. TMTOWTDI. I'd like it so that the solution allows for: - People who want to install distro-packages from repos (this would require nothing on the client side except yum/apt repo config) - People who want to build their distro-packages locally and install them - People who want to build their distro-packages locally and not install them - People who want to install the software directly into the Perl directories without packaging them for their distro at all Metadata++, though. - | Name: Tim Nelson | Because the Creator is,| | E-mail: wayl...@wayland.id.au| I am | - BEGIN GEEK CODE BLOCK Version 3.12 GCS d+++ s+: a- C++$ U+++$ P+++$ L+++ E- W+ N+ w--- V- PE(+) Y+++ PGP-+++ R(+) !tv b++ DI D G+ e++ h! y- -END GEEK CODE BLOCK-
Re: [RFC] CPAN6 requirements analysis
Hi Alex, I hve comments. Alex Elsayed wrote: While lurking in IRC, I've seen several discussions of what CPAN 6 should look like. Honestly, wayland76++'s idea for packaging seems the best to me. Most of the suggestions so far, especially those based on alien, apt, yum, or other existing package managers have a few major problems: * Alien only converts between a few package formats * All of these suggestions are _heavily_ biased towards binary distributions * These suggestions make automatic packaging for new distros extremely difficult, because they require major changes to multiple projects * We were mainly looking at Alien as a source of Perl code we could borrow. * The point of wayland76's proposal was to use the local package manager. Whether the local package manager is geared toward binary distributions is a separate issue. At first I liked wayland76's proposal, but now I have a new concern: Most package managers are not designed to hold multiple versions of the same package. As indicated in S11, it is important that a computer can hold multiple versions of the same package. I fear that using the native package manager will make this difficult. * Separate the metadata from the package * If the metadata is in the source distribution, have CPAN 6 extract it, and put it in a separate tree of just metadata * This enables simple fetching of the entire /metadata/ tree without the entire /source/ tree This is something I agree with. It seems smart to be able to download the metadata before downloading the source tree. This allows dependency resolution, searches, etc. Daniel.
Re: [RFC] CPAN6 requirements analysis
On Fri, May 29, 2009 at 01:04:28AM +0200, Daniel Carrera wrote: Hi Alex, I hve comments. Alex Elsayed wrote: While lurking in IRC, I've seen several discussions of what CPAN 6 should look like. Honestly, wayland76++'s idea for packaging seems the best to me. Most of the suggestions so far, especially those based on alien, apt, yum, or other existing package managers have a few major problems: * Alien only converts between a few package formats * All of these suggestions are _heavily_ biased towards binary distributions * These suggestions make automatic packaging for new distros extremely difficult, because they require major changes to multiple projects * We were mainly looking at Alien as a source of Perl code we could borrow. * The point of wayland76's proposal was to use the local package manager. Whether the local package manager is geared toward binary distributions is a separate issue. I support the notion of distributing binaries because nobody's gonna want to chew up their phone's battery doing unnecessary compiles. The ecology of computing devices is different from ten years ago. At first I liked wayland76's proposal, but now I have a new concern: Most package managers are not designed to hold multiple versions of the same package. As indicated in S11, it is important that a computer can hold multiple versions of the same package. I fear that using the native package manager will make this difficult. Most of these package managers have ways of running an installation script at the end, so we could perhaps think of this as downloading an installer rather than the actual software, and the new version of the installer contains or has access to all the versions it knows should be installed, and interacts with the official Perl library installer to install them. * Separate the metadata from the package * If the metadata is in the source distribution, have CPAN 6 extract it, and put it in a separate tree of just metadata * This enables simple fetching of the entire /metadata/ tree without the entire /source/ tree This is something I agree with. It seems smart to be able to download the metadata before downloading the source tree. This allows dependency resolution, searches, etc. By the same token, it's smart to keep the metadata close to the thing it's describing, so if it's easy to extract up front reliably, that's probably sufficient. Larry
Re: [RFC] CPAN6 requirements analysis
On Thursday 28 May 2009 4:04:28 pm Daniel Carrera wrote: * We were mainly looking at Alien as a source of Perl code we could borrow. Ah, I was lumping it in with the previous proposals to actually use .deb as the official P6 package format. My mistake. * The point of wayland76's proposal was to use the local package manager. Whether the local package manager is geared toward binary distributions is a separate issue. Again, my point on the issue of supporting binary/source distros was that wayland76's proposal /would/ be able to handle it properly; the '.deb as official' method would not. At first I liked wayland76's proposal, but now I have a new concern: Most package managers are not designed to hold multiple versions of the same package. As indicated in S11, it is important that a computer can hold multiple versions of the same package. I fear that using the native package manager will make this difficult. This could probably be resolved, but it would probably require distro-specific code. On Gentoo, the way to do it would be with 'slots', which are specifically designed for that use; on Debian, it would seem that the names would have to be distinguished by an identifier, allowing multiple versions.
Re: [RFC] CPAN6 requirements analysis
On Thursday 28 May 2009 4:22:00 pm Larry Wall wrote: I support the notion of distributing binaries because nobody's gonna want to chew up their phone's battery doing unnecessary compiles. The ecology of computing devices is different from ten years ago. I agree. My ideal situation would be that CPAN6 offers multiple 'heads' - one might be a raw metadata store; another might be a Debian/Ubuntu style PPA provider, possibly building (.deb|.rpm)s when uploaded, or in a distributed CPANTS-like fashion. Most of these package managers have ways of running an installation script at the end, so we could perhaps think of this as downloading an installer rather than the actual software, and the new version of the installer contains or has access to all the versions it knows should be installed, and interacts with the official Perl library installer to install them. This has the potential for greatly angering users and distro packagers: a major reason people use package managers is that they maintain a database of what files were installed, when, and by what package. If you use post-install hooks, many package managers won't have that information, making uninstallation a nightmare. By the same token, it's smart to keep the metadata close to the thing it's describing, so if it's easy to extract up front reliably, that's probably sufficient. Again, agreed. That would fall under the easy to submit well-formed packages requirement in my mail. In fact, it could even still be a Meta.yml file in the source distribution - it could be up to CPAN6 to extract it for external access when the distribution is uploaded.
Re: [RFC] CPAN6 requirements analysis
Larry Wall wrote: I support the notion of distributing binaries because nobody's gonna want to chew up their phone's battery doing unnecessary compiles. The ecology of computing devices is different from ten years ago. By binaries, I assume you mean native binaries, as opposed to Parrot bytecode. The only problem I see is that it may be impractical to ask CPAN mirrors to hold multiple binaries of each module for every OS and every CPU. On the other hand, distributing Parrot bytecode (or PIR, or PASM) seems fine. But I don't know what to suggest for modules that require a C compiler. Most of these package managers have ways of running an installation script at the end, so we could perhaps think of this as downloading an installer rather than the actual software, and the new version of the installer contains or has access to all the versions it knows should be installed, and interacts with the official Perl library installer to install them. I suggested something similar to wayland76 a couple of days ago, to solve a different problem. If I remember correctly, he was concerned about the local package manager not knowing which files were installed. Perhaps we should revisit this idea. This is what I proposed: We have our own package manager (e.g. /usr/bin/cpan6 ) that takes a .tgz file with an appropriate format: /usr/bin/cpan6 install Foo-Bar.tgz The RPM Foo-Bar.rpm would contain Foo-Bar.tgz and the rpm install script would simply run cpan6 install Foo-Bar.tgz. When the Fedora user uninstall the module, rpm runs a script which just calls cpan6 uninstall Foo::Bar. By the same token, it's smart to keep the metadata close to the thing it's describing, so if it's easy to extract up front reliably, that's probably sufficient. Yes. Daniel.
Re: [RFC] CPAN6 requirements analysis
On Thursday 28 May 2009 4:54:50 pm Daniel Carrera wrote: On the other hand, distributing Parrot bytecode (or PIR, or PASM) seems fine. But I don't know what to suggest for modules that require a C compiler. The problem with that is that Rakudo isn't the Official impelentation, and never will be. Distributing modules as Parrot bytecode would lock out other implementations, something that is very strongly discouraged. Also, I think Larry may have meant the compiled form of whatever the P6 equivalent of XS is, since Perl is largely designed as a scripting (or at least scripting-ish) language and therefore might lose more than it gains by precompiling the /Perl/ code rather than just the C/C++/etc. I suggested something similar to wayland76 a couple of days ago, to solve a different problem. If I remember correctly, he was concerned about the local package manager not knowing which files were installed. Perhaps we should revisit this idea. This is what I proposed: We have our own package manager (e.g. /usr/bin/cpan6 ) that takes a .tgz file with an appropriate format: /usr/bin/cpan6 install Foo-Bar.tgz The RPM Foo-Bar.rpm would contain Foo-Bar.tgz and the rpm install script would simply run cpan6 install Foo-Bar.tgz. When the Fedora user uninstall the module, rpm runs a script which just calls cpan6 uninstall Foo::Bar. The issue with that goes somewhat farther than just the package manager doesn't know what's there. Package managers provide several important features that would be rendered near-useless by that model: * Collision detection - It becomes impossible to prevent another package from overwriting a file installed this way * Linkage checking - If a library interface breaks compatibility, then binaries installed this way will not be detected when sweeping installed packages to fix linkage * Programmatic generation of binary packages for hybrid distributions [1] * Recording the attributes (permissions, etc) of installed packages, so they can be fixed if changed inappropriately * Source-based distributions which give the user the option of enabling/disabling tests selectively lose that ability (Gentoo, Exherbo) [1]: This is a footnote because it'll take a paragraph. In Gentoo (and when their binary package format is stabilized, Exherbo) binary packages are generated by invoking the package manager with specific options. With Portage (Gentoo only), the --buildpkg[[only]?] flag determines this, while with Paludis (Gentoo and Exherbo) it is accomplished by 'installing' to a binary repository. The system you describe makes this model impossible.
Re: [RFC] CPAN6 requirements analysis
On Thu, 28 May 2009, Alex Elsayed wrote: On Thursday 28 May 2009 4:04:28 pm Daniel Carrera wrote: At first I liked wayland76's proposal, but now I have a new concern: Most package managers are not designed to hold multiple versions of the same package. As indicated in S11, it is important that a computer can hold multiple versions of the same package. I fear that using the native package manager will make this difficult. This could probably be resolved, but it would probably require distro-specific code. On Gentoo, the way to do it would be with 'slots', which are specifically designed for that use; on Debian, it would seem that the names would have to be distinguished by an identifier, allowing multiple versions. I can confirm that Redhat supports multiple versions: $ rpm -q kernel kernel-2.6.27.5-117.fc10.i686 kernel-2.6.27.12-170.2.5.fc10.i686 kernel-2.6.27.5-117.local.fc10.i686 ... Whether it supports everything that Perl 6 wants is a different question. The options would be: - Embed more metadata into the version (version can contain arbitrary text) - Tell people that if they want to use multiple versions at the same time, they have to not use RPM for those modules at least. There may be some middle way, where we can keep the extra metadata out of the version unless there are conflicts. HTH, - | Name: Tim Nelson | Because the Creator is,| | E-mail: wayl...@wayland.id.au| I am | - BEGIN GEEK CODE BLOCK Version 3.12 GCS d+++ s+: a- C++$ U+++$ P+++$ L+++ E- W+ N+ w--- V- PE(+) Y+++ PGP-+++ R(+) !tv b++ DI D G+ e++ h! y- -END GEEK CODE BLOCK-
