On Fri, Nov 14, 2014 at 11:25:36AM +0100, Raphael Hertzog wrote: > On Fri, 14 Nov 2014, Ron wrote: > > > So I am a Kali Linux contributor. We use git repos to maintain all our > > > packages and we use git-buildpackage. > > > > I guess the first question there is what were the arguments put forward > > for deciding to 'standardise' on gbp? If there wasn't one, maybe that's > > an argument you should have (and if there was, maybe it's one to revisit :) > > 1/ It's the tool I was most familiar with > 2/ It was rather well documented > 3/ It seemed to be the most popular tool within Debian > 4/ It forces the user to follow some conventions and this is good when > you have newcomers > 5/ It did the job > > You can try to sell me gitpkg but I'm not sure it will bring this > discussion forward.
I'm not trying to "sell" you anything in particular here. I'm trying to share a little of the view from a perspective where most of the problems you seem to be worried about are in fact either already solved or not actually problems at all. If your answer to "Doctor it hurts when I bang my head against the wall" is "I know, let's change the building codes to mandate walls must be made of nerf bricks!", there's still going to be a whole world full of walls that are going to hurt your head. You can't solve problems that are inherently chaotic by pretending that they are not. If you could, we'd have all been replaced by very small shell scripts years ago. Ask ubuntu how well that worked for them ;) If you don't want to learn about how the other tools that you don't really know much about yet work, you're going to have a really hard time trying to "unify" them in any actually meaningful way. > > If you have a clearer idea now of the problems you are facing, it might > > make properly evaluating things that avoid those problems easier. > > My problems are not so much technical, my problem is having a general > recipe that work well enough for most packages so that I have to be > involved for the few special cases. Have you considered starting with a solution to something a bit simpler first? Like, say, world peace ;? What you're asking for there is not only a technical problem, it's a technically Hard problem. There are ways you might be able to make it easier, but a naming convention really isn't one of them. That might make the people who like to have their socks strictly sorted by colour feel a little more at home, but it doesn't actually address what is actually hard about this. > Changing for another tool will not improve the situation if that other > tool requires an evaluation of the layout of each repository before being > able to execute the right incantation. How can you possibly know what you want to export from it without looking at it? You can't even know if the repo is currently in a coherent state suitable to be exported from if you don't look at it and exercise some amount of judgement on what you see. > When Kali forks a package it's usually because we need a new upstream > version, or because we want to apply a pre-existing patch. It's not rocket > science and it ought to be doable rather easily, whatever helper tool > Debian uses. Well that sounds a lot less like something you'd be pushing back to Debian. And not even something you particularly need a (D)VCS for, or where you'd get a whole lot of value from one. > git-buildpackage has not been a major obstacle for us, on the contrary. > Thanks to git-import-dsc we can hack an any package by always following > the same workflow. > > But now if I want to go one step further and build on the existing > repositories while still trying to keep the same level of simplicity, > we have to go one step forward. And that's the step that I'm trying > to push for with this DEP. Again, I'm not arguing with the goal. Just trying to point out that simply formalising the actual source of many of your current problems isn't really a way to solve them. > > Even if everyone did stick to the conventions already discussed > > (which in reality, they won't) there's still far too many degrees of > > (quite necessary) freedom to really approach a "just follow these > > three easy steps" kind of uniformity. And even if you got close to > > achieving that for the "debian branches", the (again necessary) > > variability between upstream repos is going to be even greater. > > I don't want 100% uniformity, I just want as much uniformity as possible > when there's no good reason to diverge from the recommendations. And I'm not particularly arguing against the value of conventions where conventions are valuable. But they aren't going to solve the problem(s) that you seem to have convinced yourself that they will - and they especially aren't going to solve them if you go making up new conventions that are different to what everyone else in the world not using gbp (ie. 99% of it) are using. Upstream developers aren't going to read and follow your recommendations. If you don't adapt to being able to cope with the infinite variations they might have reason to use, your problems are going to remain the same problems they are for you today. > > If it really is the 'real' debian package, you have no problem > > and don't need a kali tag for it anyway. If it's a kali special, > > then the package already ought to have a x.y.x-1kali2 type version > > anyway. So this part already naturally avoids conflicts too. > > Using namespaces has other possible benefits: think of building Debian > packages out of a git push. The explicit tag prefix avoids building the > Debian version of the package when I want to build the Kali one (I might > be the maintainer on both Debian and Kali, using the same GPG signature). And what do you do when I package an upstream project named Kali, where upstream tags their releases "Kali/x.y.z" ? If you're going to trigger a build from a git push, there are so many ways that can go horribly wrong that I don't even know where to start. It's fine for c-i infrastructure where you always want to test build the HEAD of some branch, but for making actual releases?? > > > And we could also build on work in progress that has not yet been > > > released as a source package. Right now, the only packages where we > > > build on top of the Debian git repositories are some native packages > > > (like debian-installer). > > > > I can't really tell you how kali ought to work, but if you can solve > > it for this case, you should be able to do the same for non-native > > packages too. And I would recommend that you try to, even if you > > can't get the "uniformity" that you'd thing of as ideal. > > > > What is the exact difference that makes native packages work for this > > but others be harder? In the case of gitpkg, this is really just > > the difference between passing one ref and two to export it. > > In the case of debian-installer, I am the one taking care of it on > Kali and I have the required knowledge. And also we needed some unreleased > fixes for newer kernel versions. > > Still, native packages are easier because they look like any upstream repo > with a master branch (there's no need to have multiple branches around). Native packages never have branches for multiple derivatives or backports to different suites? But really, if you think non-native packages are inherently more complicated in any significant way, then you really ought to be *rethinking* your workflow, not trying to formalise it as a problem that everyone else should have too. > > > I can't afford to document all the possible ways Debian is maintaining > > > their package but if I can write a documentation that covers the common > > > case and if I can tell them "when you see those branches, you can follow > > > the instructions below", then we have made some real progress. > > > > I really do think that the names of the branches are actually going to > > be the least of your worries here, unfortunately. Even with a naming > > scheme that's widely adopted, things just aren't going to be that sort > > of uniform outside of (a fairly large number of) fairly small subsets. > > > > You're going to need a better solution than this unfortunately, if you > > want it to actually work. > > Yes, completely agreed. But git is rather robust and smart, and even if > we would all prefer that derivatives continue maintaining their repository > in the same spirit that we maintained ours, I believe that until we have > proper tools to do just that, it's not too difficult to create some > script that allows us to fixup common problems when we fetch a Debian > git repository so that the standard Kali-recommended workflow works > afterwards. > > I'm certainly eager to explore all those options (with you and anyone > else interested, the scope of the suggested sprint can include such stuff > once we have cracked the easy fixup to harmonize the helper tools) If you genuinely believe there actually is an "easy fixup to harmonize the helper tools" (and I don't mean to sound like I doubt that you do) then I really do think that the absolutely most valuable thing you can do at this point is to take a step back from what you think you know about this and *really* spend some time learning to use them and getting to know how they actually work. Like ... the kind of time that Russ put into learning about init systems. Because the differences between them aren't just "defaults to different names for various refs". It's that at some level they really do fundamentally work differently to each other. And that's even before you start to think about the significant numbers of people and groups who also manage their packages in git without using *any* of them. That's why they exist as discrete projects as they do. Bernhard didn't write git-dpm because he didn't like the naming conventions of gbp or the flexibility of gitpkg. I didn't write gitpkg because I like reinventing wheels. They both exist because we looked at the problems in fundamentally different ways, based on our own experiences with the problems of the packages we maintain. They both exist as they do because they weren't just a minor tweak to something that already existed. So I do think you're going to have a really hard time understanding what can and can't be usefully "harmonised" if you're coming at this from only understanding one tool. One that neither of us thought was up to the job of doing the things we needed to be able to do. > but I would still like to complete this DEP first because I'm convinced > that it is useful nevertheless. ... which is why I think this is approaching the real problem completely backwards. If you want to produce a "how to work with gbp" DEP, I don't have a problem with that at all. But you should be quite clear about what it really is and call it that. If you want to do something broader than that, I don't think you're going to be able to do it without a much better understanding of the other tools than you currently have. That's what I've been trying to give you a crash course on for gitpkg. I'm sorry if you saw that as a sales pitch. I'm always happy to explain it to anyone who is interested in solving the problems that it was written to solve, but I have no interest at all in shoving it down people's throats if they can't see that for themselves. It's git. There really *is* more than one way to do things. And that's an enormously good thing for software developers who want the right tool for *their* job. > > Really the only way this can possibly work is if all the tools that > > you consider viable candidates to use *don't* require some special > > structure of their own in the repo to work (or knowing the right > > magic incantation specific to each repo for them to work correctly). > > git-buildpackage doesn't have many hard requirements. git-dpm a bit > less so IIRC. > > > Otherwise, the tools that do are simply never going to work with a > > repo that doesn't have that. There really isn't any way around that, > > except to replace those tools with better ones ... > > Or improve them, or merge them. Which is the outcome that I'm looking > forward in this DEP. As I already said before, if some tool authors learn more about the fundamental mistakes they've made, that would be awesome. But they aren't going to learn that from "here's how naming conventions might work better with gbp". > Right now teams are imposing one specific tool/workflow because there > are so many differences even on basic things like the branch names. When > the differences start to fade away, people look more closely at ways > to allow everyone to use their tool of choices. I want vim modelines and fold markers. You want whatever emacs calls their thing. Someone else absolutely hates that crap in their source code. When you solve that problem, you'll understand exactly how to solve the "git workflow of choice" one too! :) > > Unlike several of them, gitpkg didn't try to be a "framework" for > > package management. It simply focussed on the job of exporting a > > source package, in any valid source package format, from an existing > > git repo. It provides hooks to let you tweak that, and to perform > > automated tasks after export (like shipping the package off to be > > built), but it explicitly doesn't provide high level functions for > > repo management tasks *prior* to export. > > Why couldn't gitpkg be an option within git-buildpackage? Again, I invite you to experiment with it and explain how you think that might work. > I believe it's important to provide a global solution (i.e. from > creating/cloning the repo) and importing the history, to building > the package and tagging the releases, etc. Again, I invite you to experiment with it and tell me exactly what of those things you think is missing. After nearly a decade of use with nobody complaining that any of those things are missing, I'll certainly be interested if you see something that is. You might want to clarify whether you want to build the package in the current workspace, or with pbuilder, or with sbuild, or with qemu, or with ... though! > > Whether the other tools can be fixed ... yeah. After all, there > > was a reason gitpkg was born as a separate thing in the first place :) > > I'm much more positive than you. At leas Guido is very welcome to enhance > git-buildpackage with support of more workflows and I'm convinced that > we could include everything that makes gitpkg/git-dpm interesting in > git-buildpackage. You still haven't convinced everyone that format 3 is the one-size fits all solution they've always been looking for yet :) You might have convinced yourself of this, but you'd have a much better chance of convincing me and maybe some others if you can show that you actually understand the *real* differences between them rather than just insisting it's "a simple matter of renaming some branches". It really isn't. It's not just a matter of "adding more workflows" to gbp either. You can't make a fundamentally inflexible thing become flexible just by special casing it even harder. It's fine if you don't want to take my word for it, but if you're not going to do that then you really do need to put in the time to learn more about the other systems and how real people really use them yourself. If you don't do that, then you really are just going to yak shave the least of the problems without getting anywhere near the real things that might be able to be improved. And again, you can take this as "negativity" if you like, but I really hope you don't. Good engineering is all about identifying problems and trying to find real solutions to them. I'm just trying to point out the problems that mean a fresh coat of paint won't fix this in the way that you seem to be hoping very hard that it will. > > ... less than 8 hours later, *all* of those patches had been cherry > > picked and merged into the mainline upstream repo. > > This is just a proof that storing the patches as real commits is useful. > And that's the point of the patch-queue tag. Instead of having the patches > only as real commits in the local repository, they get pushed to the > public repository too under that tag (instead of appearing only within > debian/patches/). How is that different to "instead of having the orig tarball only as a p-t delta in the repository, you also push the whole orig tarball under some special tag too"? Or did you just not understand that the "real commits" also get pushed to the public repository too? That is after all what makes the upstream, developer able to cherry pick them. I'm actually not quite sure what you're not understanding about this at all, no matter how many times I re-read that ... It's not about "storing the patches as real commits", the real commits *are* the patches. The only question is whether or not you are one of the people who believe that also exporting them in the package as debian/patches is a useful thing to do. (which arguably it isn't if 8 hours later you can upload a new package with them properly merged upstream :) You make the commits by "working in git like you normally would", not by going through some convoluted dance with quilt and then extracting them from that to commit them. The export tools we created are what makes it possible to have that choice without needing to change your workflow or your repo content *at all*. Using this you could even export a package with a debian/patches from an ancient version in the history that existed before there was ever a format 3, or before quilt even existed. The kind of package you want exported does *not* depend on having a magic layout or tool specific configuration files in the repository. But as I say, the best way you can really realise all this is to just go play with it yourself. I don't care if you switch to it long term or not. But if you don't understand it, and can only imagine things about how it works based on what you know about just one quite different tool, this is going to be a really hard conversation to have in any sensible way. I'm trying to explain a platypus to you, and you're saying, "yeah, yeah. ducks. we have those too!" :) Cheers, Ron -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20141114150253.gf10...@hex.shelbyville.oz