Bug#762638: metaconfig source control/distribution and Debian's DFSG
On Sat, Nov 01, 2014 at 02:41:58PM +, Dominic Hargreaves wrote: I will propose within Debian that we package up the modified dist package and metaconfig units, and include documentation in the perl source package about this. After discussions with Helmut at Debconf, it seems like all we can reasonably do in Debian is to make it clear that the current situation is explicitly supported by Debian, rather than spending effort with metaconfig. As such, I propose we close this bug with something like the attached documentation patch. Cheers, Dominic. From 72c6dc60df6cb46a77986f96ed29643a8ec9b55c Mon Sep 17 00:00:00 2001 From: Dominic Hargreaves d...@earth.li Date: Mon, 17 Aug 2015 18:45:25 +0200 Subject: [PATCH] Document the special case of modifying Configure in in debian/README.Source (Closes: #762638) --- debian/README.source | 12 debian/changelog | 7 +++ 2 files changed, 19 insertions(+) diff --git a/debian/README.source b/debian/README.source index f4b7dbb..7cbbba8 100644 --- a/debian/README.source +++ b/debian/README.source @@ -125,6 +125,18 @@ then prove debian/t/*.t fi +A special note on patching Configure + + +The Configure file is a special case of a file which is autogenerated +upstream by specialised tools, but which upstream explicitly declares +it as being (a) preferred form of modification. As far as Debian goes, +this means that we should generally not try to regenerate it, but patch +it directly (forwarding patches upstream when needed). + +Please see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=762638 +for a discussion about this issue. + Credits --- diff --git a/debian/changelog b/debian/changelog index 14f18df..539206c 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,3 +1,10 @@ +perl (5.22.0-3) UNRELEASED; urgency=medium + + * Document the special case of modifying Configure in +in debian/README.Source (Closes: #762638) + + -- Dominic Hargreaves d...@earth.li Mon, 17 Aug 2015 18:44:44 +0200 + perl (5.22.0-2) experimental; urgency=medium * Drop the ExtUtils::MakeMaker changes we've been carrying for much too long -- 2.1.4
Bug#762638: metaconfig source control/distribution and Debian's DFSG
Hi all, Apologies for another long delay in replying; other matters have taken up most of my recent energy. I'm replying to both H.Merijn Brand and Andy Dougherty's most recent messages on the subject here. I've snipped some quoted parts for brevity, but kept what I hope are the most pertinent bits of conversation, to help summarise. On Sat, Oct 04, 2014 at 08:23:10PM +0200, H.Merijn Brand wrote: On Sat, 4 Oct 2014 17:50:51 +0100, Dominic Hargreaves d...@earth.li On Thu, Sep 25, 2014 at 09:07:59PM -0400, Andy Dougherty wrote: If we include all the units in the source tree instead, I think end users will have an expectation of greater support. They will expect it to work, and work easily, right out of the box. Getting it to that point will require significant effort, and I really don't see the benefit. Of course, if someone else wants to do it and support it, that's fine with me. One benefit would be that we wouldn't this acknowledged bus factor issue with the metaconfig wrangling. Disagree. Areas in the perl core are picked up by people that like that area: • Threading • 64bit/32bit • Regex • COW • Speed • Debugging • Toolchain • Configuration • … It just happens that Configuration is NOT an area that appeals to developers. It works, and they are happy it works the way it does. Improving the configuration or the configuration process does NOT make the language itself more appealing or more interesting, and eventually, that is what the developers want (to achieve). Yes, the bus-factor is (too) low, but it is not caused by (un)availability of the pieces Okay, I accept this. If we have to, we can make a tarball of that repository and include it with the perl source package, but it seems like it would we should explore the possibility of fixing this discrepancy upstream, rather than working around it in Debian. In fact, it's likely we'd also have to supply the patches between the current metaconfig output and what's actually in the perl release tarball, since Configure is explicitly allowed to be patched even though it's generated. To be complete, you would have to include both perl's metaconfig repository and an appropriately patched 'dist' source tree, which would likely differ somewhat from Debian's existing 'dist' tree. If the intent is to actually use it, there are probably a bunch of installation issues to work out. If you want to check your results, you'll have to confront the reshuffling problem. Short summary: I think it would be a lot of work for very little gain. On Fri, Sep 26, 2014 at 08:34:58AM +0200, H.Merijn Brand wrote: The main problem with generating Configure from meta is not the availability of the metaunits used for perl, but the way meta/dist has to be installed to make them work the way it does. Do you mean the fact that dist is being patched, or some other reason? Just like learning a new language, the whole *process* is rather complicated. It is not like most GNU tools sh autogen.sh, configure, make check, make install, but - for people building perl - Configure, make test, make install The source code for autogen is widely available, but it works fine only on Linux. (I'd invite you to the hell of using the auto* and (even worse) libtool on OS's like AIX and HP-UX. To make the developers' life of our Perl community *less* of a hell, we are shipping Configure, which is a autoconf/autogen/configure in one, that runs on almost every *nix like platform (VMS and Windows have special cased support). For how many of the GNU tools that you ship, do you include the autogen.sh stuff that is only used *before* the configure is included in the source package? I only see the auto-stuff if I clone the source repo for the tool itself. Most of the autotools stuff in Debian ships the source files and it's generally encourages to rerun autoconf/automake to generate configure these days, so that changes needed to support new architectures, etc, can be reflected without any work on the source of the package. The fact that we patched and extended meta is NO reason for it to be more complex than using meta/dist from scratch. It is like installing a patched version of autoconf. The changes are reasonable well documented, and the source is widely and openly available. Or at least it is now, thank you :) As the current maint for this process, I communicate with the maintainer of dist/meta on a rather regular basis. I give feedback of what we changed, and I look at the changes he makes to see if those will be usable for perl. As part of the metaunits is still used from dist, part is a set of (slightly) modified units from dist, and part is the units written only for perl, keeping those in sync is a hell of a job. On simply cannot stay in sync 100% Okay, so clearly from a pragmatic view we would need to
Bug#762638: metaconfig source control/distribution and Debian's DFSG
On Sat, Oct 04, 2014 at 05:50:51PM +0100, Dominic Hargreaves wrote: On Thu, Sep 25, 2014 at 09:07:59PM -0400, Andy Dougherty wrote: On Thu, Sep 25, 2014 at 10:16:32PM +0100, Dominic Hargreaves wrote: The way it's set up now, we encourage people to simply patch Configure. If someone wants to go the metaconfig route instead, it's a lot of extra work, but presumably the person deliberately chooses that route knowing it's extra work. Again, there is an ideological point here. It *shouldn't* be a lot of extra work to do things in the way that upstream developers would. Clearly perl isn't going to be kicked out of Debian because of this, but a less important package might well be. I think we may have confused you here such that you have it exactly backwards. It *isn't* a lot of extra work for anyone to do things in the way the upstream developers would. Everyone has access to the exact same source. What a few upstream developers do have is *experience* using the package, so that they can do that work somewhat more easily. Those few upstream developers (well, really only H.Merijn these days) have volunteered to help do that work for people who would rather spend their time fixing something else rather than learning a complete configuration system. Thus if a casual hacker wants to make a simple Configure change, they can simply make it directly to Configure. Thus it is *easier* for the casual hacker to get involved. If they want to do it the way the upstream developer would, then they have to do the same hard work that the upstream deveveloper does, and they are certainly free to do so. Both the dist subversion repository and the perl metaconfig git repository are freely available, so anyone can check out the appropriate versions to recreate Configure (subject to machine-dependent ordering). I hadn't realized that the precise versions used weren't clearly labeled because I don't recall anybody ever asking before. Encouraged by this request, I will try to remember and document that more clearly in perl's documentation. If someone else wants to do it first, the patches would certainly be welcome. Okay, so clearly from a pragmatic view we would need to ship our own version of dist along with the rest of the stuff from the metaconfig repository. Depressingly this violates another part of Debian policy relating to embedding copies of code not intended to be embedded, but the freedom to modify the code using the preferred form clearly overrides that in my mind. You could instead separately package dist-3.5.20 and depend on that, if you liked. (Presumably, that package would point to the standard 'dist' package as the recommended starting point for new projects, and would not overwrite the standard 'dist' package files.) I don't see how this situation is fundamentally any different from that of any other program built with a particular version of an external tool (such as bison, for example). Rebuilding Configure would not be easy or automatic, but all the necessary files would be available. Would that satisfy the Debian guidelines? I think it would satisfy the letter of the law, if not the full spirit. I'm sorry, but I really fail to see why, and suspect that there is a lingering misunderstanding. Rebuilding Configure is not easy or automatic for *anyone*, but that's not an issue of freedom. I want to be helpful and share what we have, but we can only share what we have. Cheers, -- Andy Dougherty dough...@lafayette.edu -- To UNSUBSCRIBE, email to debian-bugs-rc-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#762638: metaconfig source control/distribution and Debian's DFSG
Dear Perl maintainers, On Sat, Oct 04, 2014 at 05:50:51PM +0100, Dominic Hargreaves wrote: I'm including Helmut, the submitter to the Debian bug report, as a CC, in case he can help provide additional context that I'm missing. Yes, I do have additional context. I don't have the exact details, but Helmut suggests that this wasn't the case for him, as the changes would have had to be made in several places. In general I think a healthy free software project should aim to minimise cases where a casual hacker would be discouraged from doing things the same way a seasoned developer would. I have to admit that I am no longer 100% sure that I do indeed want to modify metaconfig units rather than just patching Configure, but given my experience with autoconf that initially seemed like the logical way to do it. I can give background on what I would like to see changed about Perl's Configure, but it'll be a longer read and it may not help solving the question as to whether metaconfig is to be considered part of the Perl source (from a Debian pov). The one extra bit I can give here is that reading metaconfig units helps with understanding Configure. I do not understand much of the other details involved, so the rest of this mail concentrates on my particular use case: Cross building Perl is hard. When Perl is cross built, it most often is a retrospective. In other words, you built it natively and collected the Configure results to be able to reproduce them as part of a real cross build. I am aware of two cross building efforts for Perl. One is Neil William's perl-cross-debian[1] and the other is OpenWRT, which apparently provides a microperl. Neil William's effort basically gathers the Configure results an treats them as source. Since these results change with Perl releases, perl-cross-debian is hard to keep up to date and it does not work with any non-ARM architectures at the moment. Ideally, Perl would cross just like any other language, but we are very very far from that. So one way to make crossing Perl more feasible is to reduce the effort of maintaining the Configure results and one way to reduce that effort is to reduce the number of checks that need to execute host[2] arch code. Indeed, there is very much low hanging fruit here. Checks such as the *size checks or d_open3 and o_nonblock can be turned into compile-only checks (i.e. without running host[2] arch code). Since the *size checks are very much repetitive, it seemed natural to me that modifying metaconfig is the way to go. Once a good portion of these checks is turned into compile-only checks, the remaining checks may become maintainable enabling reasonable cross compilation. The documented method of cross compiling Perl is a bad joke. It presumes that one actually has a host[2] architecture machine, but for a long time arm64 and ppc64el hardware was unavailable and it is still hard to come by today. It also presumes that one has a ssh server on the host[2] architecture machine, but building openssh requires Perl. So in my view, Perl currently makes cross compilation unnecessarily hard and the place to fix that is Configure. Helmut [1] https://github.com/codehelp/perl-cross-debian [2] I am using GNU terminology here, because it is most widely accepted. Perl's documentation refers to this as target. -- To UNSUBSCRIBE, email to debian-bugs-rc-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#762638: metaconfig source control/distribution and Debian's DFSG
Hi, I'm including Helmut, the submitter to the Debian bug report, as a CC, in case he can help provide additional context that I'm missing. On Thu, Sep 25, 2014 at 09:07:59PM -0400, Andy Dougherty wrote: On Thu, Sep 25, 2014 at 10:16:32PM +0100, Dominic Hargreaves wrote: Hi, The Debian project has recently received a bug report indicating that the perl source package we distribute does not comply the the DFSG[1] since the preferred form of modification of Configure is not provided. You can see the bug report at [2]. I'd quibble with that -- though I am no longer the primary Configure maintainer, I actually did prefer most users to directly modify Configure. It ended up being much easier than supporting users using metaconfig to generate Configure. Users are of course welcome to fetch and install metaconfig and use it, but I wouldn't say that was the preferred form. I don't have the exact details, but Helmut suggests that this wasn't the case for him, as the changes would have had to be made in several places. In general I think a healthy free software project should aim to minimise cases where a casual hacker would be discouraged from doing things the same way a seasoned developer would. Generating Configure requires both the 'dist' package (which contains metaconfig) and the perl-specific metaconfig units. Historically, we have often ended up using a patched 'dist', or a version that lagged behind the main package. We definitely can't simply assume the version in the Debian archive will work. Generating Configure also sometimes involves patching or fixing the output of metaconfig in various ways. All the information to do this should be in perl's metaconfig git repository. It is also sometimes difficult to check or merge the generated output. The metaconfig units are assembled in order based on dependencies, but those dependencies don't fully constrain the order. The result is that the order of units can vary depending on who generates Configure. Thus even a tiny patch to a single metaconfig unit can result in a Configure patch with hundreds of lines that are just shuffling parts around. That's another reason why modifying the metaconfig units is not necessarily the preferred form of modification of Configure. Obviously, it's all a bit involved, and a fair amount of effort to figure out. The way it's set up now, we encourage people to simply patch Configure. If someone wants to go the metaconfig route instead, it's a lot of extra work, but presumably the person deliberately chooses that route knowing it's extra work. Again, there is an ideological point here. It *shouldn't* be a lot of extra work to do things in the way that upstream developers would. Clearly perl isn't going to be kicked out of Debian because of this, but a less important package might well be. If we include all the units in the source tree instead, I think end users will have an expectation of greater support. They will expect it to work, and work easily, right out of the box. Getting it to that point will require significant effort, and I really don't see the benefit. Of course, if someone else wants to do it and support it, that's fine with me. One benefit would be that we wouldn't this acknowledged bus factor issue with the metaconfig wrangling. If we have to, we can make a tarball of that repository and include it with the perl source package, but it seems like it would we should explore the possibility of fixing this discrepancy upstream, rather than working around it in Debian. In fact, it's likely we'd also have to supply the patches between the current metaconfig output and what's actually in the perl release tarball, since Configure is explicitly allowed to be patched even though it's generated. To be complete, you would have to include both perl's metaconfig repository and an appropriately patched 'dist' source tree, which would likely differ somewhat from Debian's existing 'dist' tree. If the intent is to actually use it, there are probably a bunch of installation issues to work out. If you want to check your results, you'll have to confront the reshuffling problem. Short summary: I think it would be a lot of work for very little gain. On Fri, Sep 26, 2014 at 08:34:58AM +0200, H.Merijn Brand wrote: The main problem with generating Configure from meta is not the availability of the metaunits used for perl, but the way meta/dist has to be installed to make them work the way it does. Do you mean the fact that dist is being patched, or some other reason? As Andy already stated, it proves to be much easier to backport the direct patches to Configure and friends to meta than it is to instruct (new) users to patch metaunits the way it should. Even seasoned users seem to have problems doing so. I had two sessions past year explaining the ways and woes of how it works, sincerely hoping to get fresh blood into the
Bug#762638: metaconfig source control/distribution and Debian's DFSG
On Sat, 4 Oct 2014 17:50:51 +0100, Dominic Hargreaves d...@earth.li wrote: Hi, I'm including Helmut, the submitter to the Debian bug report, as a CC, in case he can help provide additional context that I'm missing. On Thu, Sep 25, 2014 at 09:07:59PM -0400, Andy Dougherty wrote: On Thu, Sep 25, 2014 at 10:16:32PM +0100, Dominic Hargreaves wrote: Hi, The Debian project has recently received a bug report indicating that the perl source package we distribute does not comply the the DFSG[1] since the preferred form of modification of Configure is not provided. You can see the bug report at [2]. I'd quibble with that -- though I am no longer the primary Configure maintainer, I actually did prefer most users to directly modify Configure. It ended up being much easier than supporting users using metaconfig to generate Configure. Users are of course welcome to fetch and install metaconfig and use it, but I wouldn't say that was the preferred form. I don't have the exact details, but Helmut suggests that this wasn't the case for him, as the changes would have had to be made in several places. In general I think a healthy free software project should aim to minimise cases where a casual hacker would be discouraged from doing things the same way a seasoned developer would. Generating Configure requires both the 'dist' package (which contains metaconfig) and the perl-specific metaconfig units. Historically, we have often ended up using a patched 'dist', or a version that lagged behind the main package. We definitely can't simply assume the version in the Debian archive will work. Generating Configure also sometimes involves patching or fixing the output of metaconfig in various ways. All the information to do this should be in perl's metaconfig git repository. It is also sometimes difficult to check or merge the generated output. The metaconfig units are assembled in order based on dependencies, but those dependencies don't fully constrain the order. The result is that the order of units can vary depending on who generates Configure. Thus even a tiny patch to a single metaconfig unit can result in a Configure patch with hundreds of lines that are just shuffling parts around. That's another reason why modifying the metaconfig units is not necessarily the preferred form of modification of Configure. Obviously, it's all a bit involved, and a fair amount of effort to figure out. The way it's set up now, we encourage people to simply patch Configure. If someone wants to go the metaconfig route instead, it's a lot of extra work, but presumably the person deliberately chooses that route knowing it's extra work. Again, there is an ideological point here. It *shouldn't* be a lot of extra work to do things in the way that upstream developers would. Clearly perl isn't going to be kicked out of Debian because of this, but a less important package might well be. If we include all the units in the source tree instead, I think end users will have an expectation of greater support. They will expect it to work, and work easily, right out of the box. Getting it to that point will require significant effort, and I really don't see the benefit. Of course, if someone else wants to do it and support it, that's fine with me. One benefit would be that we wouldn't this acknowledged bus factor issue with the metaconfig wrangling. Disagree. Areas in the perl core are picked up by people that like that area: • Threading • 64bit/32bit • Regex • COW • Speed • Debugging • Toolchain • Configuration • … It just happens that Configuration is NOT an area that appeals to developers. It works, and they are happy it works the way it does. Improving the configuration or the configuration process does NOT make the language itself more appealing or more interesting, and eventually, that is what the developers want (to achieve). Yes, the bus-factor is (too) low, but it is not caused by (un)availability of the pieces If we have to, we can make a tarball of that repository and include it with the perl source package, but it seems like it would we should explore the possibility of fixing this discrepancy upstream, rather than working around it in Debian. In fact, it's likely we'd also have to supply the patches between the current metaconfig output and what's actually in the perl release tarball, since Configure is explicitly allowed to be patched even though it's generated. To be complete, you would have to include both perl's metaconfig repository and an appropriately patched 'dist' source tree, which would likely differ somewhat from Debian's existing 'dist' tree. If the intent is to actually use it, there are probably a bunch of installation issues to work out. If you want to check your results, you'll have to confront the reshuffling problem. Short summary: I think it
