Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development
On Tue, Apr 21, 2020 at 07:50:13AM +0200, Michał Górny wrote: > On Mon, 2020-04-20 at 16:04 -0500, William Hubbs wrote: > > Your proposal seems to completely go against how the go ecosystem operates, > > but if you can come up with a proof-of-concept for how it would work > > without forcing a lot of busy work on us that would never get accepted > > upstream, I'll take a look. > > > > Define 'busy work'. Busy work, to me at least, is work that generates a large amount of overhead compared to the gain it offers the distro and will never be accepted by upstream. > Is doing things right 'busy work' vs taking shortcuts? This depends on what you think "doing things right" or "taking shortcuts" mean. For example, I have had discussions with maintainers who are fine with writing patches to fix issues without reporting the issues upstream. Personally, I disagree with this approach. > Is it 'busy > work' that I'm putting a lot of effort into fixing Python packages? > Should I just last rite them all and tell people to use virtualenv? This gets into the other issue being discussed on the thread, but I have heard people say that yes you should. All we really need in portage is the base python tools -- the language, maybe pip and maybe setuptools and portage itself, then everything else should be installed via pip. > Is testing packages 'busy work'? I suppose it's all easier when you > just silently include 240 dependencies without testing a single one of > them and call it a day. Sure, you run some tests on the final package. > Leaves 240 untested, some of them likely failing tests and requiring > 'busy work' to fix them. In general I would say no, but I have run into tests that do not tollerate portage's sandbox, tests that take hours or days to run, or tests that require root privs. We don't force those kinds of tests to be run in src_test. > Is security support 'busy work'? I suppose it's all easier when you > ignore them problem and let security team deal with it (except they > won't). Sure, you can assume that when vulnerability is discovered, all > upstreams will eventually learn of it, update their bundled dependencies > and *maybe* inform people that their code might have been indirectly > vulnerable (but would they do the 'busy work' of discovering whether it > affected them or not?). In my experience, the security team doesn't write patches to fix vulnerabilities, so what they do wouldn't be any different in this case. > I realize Go is not isolated here. It's just brought as one major > example. Rust is no better. All these shiny 'write and forget' > languages share the same problem. Pay for some work hours, get > a working product, deploy it and forget unless customers want more > features. > > Today these languages are still young and the problems can be considered > largely theoretical. But some day -- well, unless all the cool kids > manage to move on to next shiny new language before then -- this will be > a major catastrophe. We can talk about theoretical issues in languages all day, but if we are going to do that we also have to talk about how terrible and permissive c is. You can google things like "security in c" and you will find that since it allows you to do exactly what you want, it is a major catastrophe in terms of security, but we accept new applications in c without any questions. Also, what about the nightmare that is php? In short, I don't see any reason to bikeshed about theoretical issues in languages. The way I see Go and Rust specifically is, more and more things are being written in these languages, and if we don't accept them, we will become irrelivent fast. William signature.asc Description: PGP signature
Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development
On 4/21/20 6:50 AM, Michał Górny wrote: > I realize Go is not isolated here. It's just brought as one major > example. Rust is no better. All these shiny 'write and forget' > languages share the same problem. Pay for some work hours, get > a working product, deploy it and forget unless customers want more > features. > > Today these languages are still young and the problems can be considered > largely theoretical. But some day -- well, unless all the cool kids > manage to move on to next shiny new language before then -- this will be > a major catastrophe. Looking into an old language (25 years old): php So many security problems... Just a few essential ebuilds exists in ::gentoo and being very well maintained by Gentoo. I think the approach to the new languages have the same requirement. Minimizing the ebuilds to the required ones would allow to minimize the effort. Then we can use an overlay to deliver all other software that would be useful. So the goal in this cases would be to define the right sieve about those libraries and tools that are required to all and that fits in package maintenance procedures, so it could be merged into ::gentoo. But don't forget the tooling available to the overlays development, where another abstraction layer takes place, since it's focused on a specific target and not all community in main ::gentoo. That's why I consider go-module as a very cleaver solution since it gives a hybrid solution connecting immutable delivery model into the mutable reality. The PR present in this subject is related to missing eclass for JVM languages, to define the common procedures to use those languages and available builders. signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development
On Mon, 2020-04-20 at 16:04 -0500, William Hubbs wrote: > Your proposal seems to completely go against how the go ecosystem operates, > but if you can come up with a proof-of-concept for how it would work > without forcing a lot of busy work on us that would never get accepted > upstream, I'll take a look. > Define 'busy work'. Is doing things right 'busy work' vs taking shortcuts? Is it 'busy work' that I'm putting a lot of effort into fixing Python packages? Should I just last rite them all and tell people to use virtualenv? Is testing packages 'busy work'? I suppose it's all easier when you just silently include 240 dependencies without testing a single one of them and call it a day. Sure, you run some tests on the final package. Leaves 240 untested, some of them likely failing tests and requiring 'busy work' to fix them. Is security support 'busy work'? I suppose it's all easier when you ignore them problem and let security team deal with it (except they won't). Sure, you can assume that when vulnerability is discovered, all upstreams will eventually learn of it, update their bundled dependencies and *maybe* inform people that their code might have been indirectly vulnerable (but would they do the 'busy work' of discovering whether it affected them or not?). I realize Go is not isolated here. It's just brought as one major example. Rust is no better. All these shiny 'write and forget' languages share the same problem. Pay for some work hours, get a working product, deploy it and forget unless customers want more features. Today these languages are still young and the problems can be considered largely theoretical. But some day -- well, unless all the cool kids manage to move on to next shiny new language before then -- this will be a major catastrophe. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development
On 4/20/20 5:48 PM, Georg Rudoy wrote: > >> I've learned the hard way that it discourages you from doing all the >> things that I just said high-quality software should do. > > Again, ranging from one-off pseudo-scripts that I had to come back to > after a couple of years, to quite complicated pieces of software > running in prod and actually serving customers I had to update up to > the latest LTS after 3-4 years of inactivity, I tend to disagree. Even > the latter was a surprisingly smooth process boiling down to about an > hour or two of very effortless, mechanical and thus boring code > changes (mostly about Semigroup/Monoid hierarchy, if you wonder). > > This is surely offtopic, but I just don't want to silently participate > in spreading things that aren't objectively true (even if they are > subjectively true in your/mine/etc experience). Fair enough, I was speaking only from personal experience. But if you do an experiment I bet you'll agree with me. Updating your code to work with the latest GHC is pretty easy. But if you wanted to write an ebuild for that program, it would have a dependency on that specific version of GHC, leading to these same problems that we're talking about when other programs require other specific versions. I can compile my C programs with any GCC going back to the 4.x days -- or even clang. In the past three years, we've collected ghc-8.0 though ghc-8.10 in ::gentoo. Try to make your code work with all of those GHC releases at once; if you come back and tell me it was easy, I'll be impressed (and may start asking you to update my Haskell code). Working around something like gcc-10's -fno-common breakage, on the other hand, is relatively easy. We write a patch, send it upstream, add it to ::gentoo in the meantime, and everything keeps working. But that's only backwards-compatible because the C standard says how it should work, and the C standard doesn't change every six months to be "whatever this version of GCC does." To refer back to something on-topic: some projects are willing to make the extra effort, and some aren't. The ones that are, are easy to package and definitely belong in ::gentoo. The others... should be dealt with case-by-case. But lowering the standards of the main tree should not be how we get them over the bar.
Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development
On 4/20/20 5:25 PM, Patrick McLean wrote: > > Please explain how we are actively making things worse for you? We > are contributing useful packages to the tree, we are doing the work > and we are doing it in the way that makes the most effective use of > our time. We simply do not have time to be trying to convince > upstreams to make changes to their build systems to support a single, > somewhat niche, distribution. We certainly don't have time to be > patching build systems with every version bump. > You're introducing unfixable security vulnerabilities into a distribution that I recommend to people, that my name implicitly endorses, and that I use to professionally to handle peoples' sensitive personal data. Likewise with the license issues that can't be fixed. It's a waste of time having to explain to every new contributor that Go ebuilds in ::gentoo are "special" and aren't something they should be using as an example. You're setting an example that our QA and security standards (that we beat to death in the quizzes) can be ignored whenever you feel like it. This indirectly creates technical debt as other developers follow your example. You're directly creating technical debt that someone else is going to have to clean up when you move on to the next shiny and the treecleaners have to waste six months last-riting everything. And you're blocking a golang implementation that doesn't have these problems.
Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development
пн, 20 апр. 2020 г. в 17:38, Michael Orlitzky : > Diamond dependencies manifest mainly in delayed version > bumps, while slyfox does all the work to make sure that the things > already in the tree will work with the new version. This requires lots > of careful updates to neighboring packages, and unfortunately a lot of > cabal file hacking. This is precisely what I meant by "doesn't scale well". Wonder if this can be automated. > > Dunno much about Go and I don't have a single Go package locally to > > check. Do they do static or dynamic linking with the deps, for > > instance? What's the language model wrt API and linking? > > FORGET I MENTIONED IT Lol, happy to! > >> and C. > > > > More stable API (and ABI). > > > > Definitely. The "Haskell" language changes entirely every few years. Some say the same about C++ (and ABI does change even within a standard), so... > I've learned the hard way that it discourages you from doing all the > things that I just said high-quality software should do. Again, ranging from one-off pseudo-scripts that I had to come back to after a couple of years, to quite complicated pieces of software running in prod and actually serving customers I had to update up to the latest LTS after 3-4 years of inactivity, I tend to disagree. Even the latter was a surprisingly smooth process boiling down to about an hour or two of very effortless, mechanical and thus boring code changes (mostly about Semigroup/Monoid hierarchy, if you wonder). This is surely offtopic, but I just don't want to silently participate in spreading things that aren't objectively true (even if they are subjectively true in your/mine/etc experience). > There's a core of mature Haskell that's pretty easy to develop against. > (I think you just have to wait about five years with any project before > the authors realize that changing everything every month isn't fun.) Out > in the woods you can still get into a lot of trouble though. We now have > the mature core stuff in ::gentoo, and the crazies out in the ::haskell > overlay. That feels like the right mix. I've seen transifex-client last-rited the other day and was thinking of throwing up something in haskell (plus I have a couple of other projects that might be useful). Let's see what it'd take to get that into ::gentoo once I'm done. BTW having things like servant in ::haskell as opposed to ::gentoo is like having boost in some C++ overlay as opposed to ::gentoo. -- Georg Rudoy
Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development
On 4/20/20 5:05 PM, Georg Rudoy wrote: > >> If upstream absolutely insists on minor-version dependencies, then you >> either tolerate it conflicting with everything else, or leave it out of >> the tree. You probably shouldn't even be packaging a library whose API >> is distinguishable across minor releases. > > That's not a matter of just the API per se. Even the library file name > encodes its deps in its name (if I understand correctly, that's the > hash in libHSregex-base-0.94.0.0-541xQVwwNRiBGjsNKmOPoy-ghc8.6.5.so > for example). Personally I just find it hard to reason about this sort > of dependencies management. But, again, I should probably avoid trying > that and just jump head first to packaging. Haskell already requires transitive rebuilds of dependencies due to a (serious) implementation constraint. Subslots help with that, but it will take a new EAPI to truly tell an ebuild how to do what we need to do. In any case, all Haskell ebuilds depend on separate packages for their dependencies, and with haskell-updater, subslots, and @preserved-rebuild it all more or less works. I think that's largely separate from the "diamond dependency problem" you posed. Diamond dependencies manifest mainly in delayed version bumps, while slyfox does all the work to make sure that the things already in the tree will work with the new version. This requires lots of careful updates to neighboring packages, and unfortunately a lot of cabal file hacking. > Dunno much about Go and I don't have a single Go package locally to > check. Do they do static or dynamic linking with the deps, for > instance? What's the language model wrt API and linking? FORGET I MENTIONED IT >> and C. > > More stable API (and ABI). > Definitely. The "Haskell" language changes entirely every few years. I've learned the hard way that it discourages you from doing all the things that I just said high-quality software should do. There's a core of mature Haskell that's pretty easy to develop against. (I think you just have to wait about five years with any project before the authors realize that changing everything every month isn't fun.) Out in the woods you can still get into a lot of trouble though. We now have the mature core stuff in ::gentoo, and the crazies out in the ::haskell overlay. That feels like the right mix.
Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development
On Mon, 20 Apr 2020 17:03:50 -0400 Michael Orlitzky wrote: > On 4/20/20 4:19 PM, Patrick McLean wrote: > > > > My co-workers are not the only ones adding/maintaining go packages in the > > tree, please do not single out any groups, and let's all work to make > > Gentoo the best it can be. > > > > Everyone else is just using the eclass that your coworkers defended and > committed the last time we did this. We're not in this together and you > sound like a COVID commercial from a billionaire that wants me to go > back to work. You're in this for you, and everything you do for you > makes things worse for me. > We are under no obligation to contribute anything to ::gentoo, we could do all our go packaging work in a private overlay, and never push anything upstream. It is a considerable amount of extra work to push our packages upstream. We do this to contribute to the wider Gentoo community. I resent your statement that we are not doing this to contribute to the community. Please explain how we are actively making things worse for you? We are contributing useful packages to the tree, we are doing the work and we are doing it in the way that makes the most effective use of our time. We simply do not have time to be trying to convince upstreams to make changes to their build systems to support a single, somewhat niche, distribution. We certainly don't have time to be patching build systems with every version bump.
Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development
пн, 20 апр. 2020 г. в 16:51, Michael Orlitzky : > Except that Haskell as a research language tends to inspire more "fire and > forget" software. I'd say this is not the case for the last few years, but that's maybe just my own impression from interacting with > (And Haskell works better in Gentoo than anywhere else, in my opinion.) Totally agreed here. That's one of the main reasons I run Gentoo (especially before dev-haskell/stack emerged upstream a few years ago). > You package each dependency as a separate package. If there's a conflict > between two packages, then... you can't have those two packages > installed together. When that happens, you alert upstream, and try to > convince them to be more lenient with their dependencies (again, nothing > Haskell-specific here). Sometimes that involves code changes, but often > it's just an edit to the cabal file. Many people follow either semver or > the PVP (https://pvp.haskell.org/) which keeps things from getting too > far out of control. I'd say the Haskell-specific thing is the presence of stackage LTS releases that people tend to stick to. For instance, beam-postgres disappeared in recent LTSes, so some projects of mine are stuck on older LTSes (adding that to extra-deps was a pain for reasons I don't remember at the moment), while the others can happily use the new ones. Libraries incompatibilities are just a matter of time in this case. Sure, hacking on .cabal/package.yaml is one way to solve this, but I wasn't sure it's reasonably robust in the long run. Anyway, thanks for your feedback. This probably means I should stop thinking about the general case and maybe start playing around with packaging stuff, and then seeing where it breaks. > If upstream absolutely insists on minor-version dependencies, then you > either tolerate it conflicting with everything else, or leave it out of > the tree. You probably shouldn't even be packaging a library whose API > is distinguishable across minor releases. That's not a matter of just the API per se. Even the library file name encodes its deps in its name (if I understand correctly, that's the hash in libHSregex-base-0.94.0.0-541xQVwwNRiBGjsNKmOPoy-ghc8.6.5.so for example). Personally I just find it hard to reason about this sort of dependencies management. But, again, I should probably avoid trying that and just jump head first to packaging. > I don't see anything fundamentally different here > between Haskell (or Go, or...) Dunno much about Go and I don't have a single Go package locally to check. Do they do static or dynamic linking with the deps, for instance? What's the language model wrt API and linking? > and C. More stable API (and ABI). -- Georg Rudoy
Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development
On Mon, Apr 20, 2020 at 03:23:15PM -0400, Michael Orlitzky wrote: > > Are you volunteering to do the work to package go packages? The people > > doing the work generally get to decide how that work gets done, and which > > approach they would like to take. The upstream situation makes it very > > labour-intensive to do the work in a the way you are proposing (many > > packages would end up with hundreds to thousands of packages in the tree). > > Separating everything out in to separate packages will just increase the > > maintenance load exponentially, with no gains as go upstreams version lock > > all their dependencies. > > > > I'm volunteering to work on one or two small Go packages. Can I convert > the eclass to use dynamic linking? Can I start replacing your packages > with my own if I need them as dependencies? I suspect not, and that's > one of many reasons why "having things in ::gentoo does not affect > anyone who does not use them" is bullshit. As one of the maintainers of the go-module eclass, I'm all ears, but keep a couple of things in mind. First, I have yet to see an upstream go-based build system that uses dynamic linking for go libraries. No one does it. they all statically link everything. This alone probably means you'll have to patch every upstream build system you find. Another issue is the sheer number of dependencies that a single go package would pull into the tree. For example, dev-vcs/cli-0.6.3, which looks like a small enough package, has 65 dependencies. sys-apps/kubernetes-1.18.2, a bigger package for sure, has 246. So, the worst case you are looking at here is over 300 dependencies, and this is just for two packages. The only way this number would come down is if these packages happened to share exact dependencies since you can't really unlock the versions. and, yes, gccgo is out there, but no one writes for it. I have yet to see an upstream that supports using it. So, if you go that route, you are looking at patching build systems. Your proposal seems to completely go against how the go ecosystem operates, but if you can come up with a proof-of-concept for how it would work without forcing a lot of busy work on us that would never get accepted upstream, I'll take a look. William
Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development
On 4/20/20 4:19 PM, Patrick McLean wrote: > > My co-workers are not the only ones adding/maintaining go packages in the > tree, please do not single out any groups, and let's all work to make Gentoo > the best it can be. > Everyone else is just using the eclass that your coworkers defended and committed the last time we did this. We're not in this together and you sound like a COVID commercial from a billionaire that wants me to go back to work. You're in this for you, and everything you do for you makes things worse for me. [0] Sony Interactive Entertainment
Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development
On 4/20/20 8:27 PM, Rich Freeman wrote: > IMO it isn't really worth worrying about, because right now the main > limitation seems to be a lack of people working on projects, not 25 > devs who each want to re-implement go their own way... This is another reason I think is so important the overlays in Gentoo. ::gentoo overlay can be the most stricter one. It is a fact that is more feasible to deliver software in overlays as already happen (for example gitlab, science stuff, ...). But should ::gentoo be confined to GNU, Linux and other OS base stuff? Lets look into a specific case for a distro where everything should be packaged and placed in an universal repository (crazy debian): https://packages.debian.org/buster/snapd There are very radical package pattern people there and looking into snapd we can see that upstream just regarded the package integration, bundling all necessary dependencies into it. So, even there, we can't find all the required go dependencies packaged... Actually, the dependencies are only those that undoubtedly could appear on ::gentoo. IMHO, there are many software that doesn't need to be in a ebuild, but the base pieces such as go, snapd, docker-ce, docker engine, are some examples of go tools that we need to have available in ::gentoo. The same happens for other platforms such as those mentioned in the subject of this email. In conclusion, I think that all the trouble about licenses and security will be always a growing challenge and a distro should adapt their tools to allow software being available in a better way than upstream provided. I think that the point here is about: - how to bring the software to an OS - how to manage the software in a better way - share knowledge and magnetize the community so Gentoo can grow and show how is so useful helping to deliver better software I'm very pleased for Gentoo! signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development
On 4/20/20 4:21 PM, Georg Rudoy wrote: > вс, 19 апр. 2020 г. в 16:09, Michael Orlitzky : >> But please quit looking to Go as an example of anything >> anyone should be doing. > > On a somewhat related note, I'd like to take this chance to ask about > packaging haskell software, which has somewhat similar issues: > > ... > 3. What is major is the following. Let's say two executables A and B > depend on libfoo:1 and libfoo:2 respectively, and depend on the same > version of libbar. libbar itself depends on libfoo (and supports both > versions), but libbar built with libfoo:1 is incompatible with libbar > built with libfoo:2. Loosely speaking, the slot of a library is > defined by all its dependencies' _versions_. > > Of all these, (3) is quite major, as it leads to exponential explosion > of the slots count. Otherwise, one might choose to carefully curate > the library versions, ensuring that no such situation arises, but this > approach is neither feasible nor possible at all for sufficiently > large packages base. There's nothing Haskell-specific here, is there? Except that Haskell as a research language tends to inspire more "fire and forget" software. (And Haskell works better in Gentoo than anywhere else, in my opinion.) You package each dependency as a separate package. If there's a conflict between two packages, then... you can't have those two packages installed together. When that happens, you alert upstream, and try to convince them to be more lenient with their dependencies (again, nothing Haskell-specific here). Sometimes that involves code changes, but often it's just an edit to the cabal file. Many people follow either semver or the PVP (https://pvp.haskell.org/) which keeps things from getting too far out of control. If upstream absolutely insists on minor-version dependencies, then you either tolerate it conflicting with everything else, or leave it out of the tree. You probably shouldn't even be packaging a library whose API is distinguishable across minor releases. Software engineering is a fractal. Curating APIs and dependencies is as much a part of it as what sorting algorithms you choose. These are all old problems with known solutions (sonames, semver, etc.) and software isn't magically high-quality because it's written in a language that came our last week. So aside from the "What's Gentoo?" that's going to be more common among Haskell upstreams, I don't see anything fundamentally different here between Haskell (or Go, or...) and C. We should be helping to make upstream software more good, and then packaging it. (It's OK to hack the cabal file while the bug report is open.)
Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development
вс, 19 апр. 2020 г. в 16:09, Michael Orlitzky : > But please quit looking to Go as an example of anything > anyone should be doing. On a somewhat related note, I'd like to take this chance to ask about packaging haskell software, which has somewhat similar issues: 1. Two different executables might depend on different versions of a library. This is, of course, solvable via slots, so this is minor (but, looking at a random sample of `dev-haskell/`, most of libs there aren't slotted). 2. The dependency trees are pretty big and wide. A simple project of mine depends on a hundred or two of dependencies on average, and the union of all deps of all the projects I do is probably closer to a thousand. Again, this is minor, just bear in mind the corresponding ebuilds tree size increase. 3. What is major is the following. Let's say two executables A and B depend on libfoo:1 and libfoo:2 respectively, and depend on the same version of libbar. libbar itself depends on libfoo (and supports both versions), but libbar built with libfoo:1 is incompatible with libbar built with libfoo:2. Loosely speaking, the slot of a library is defined by all its dependencies' _versions_. Of all these, (3) is quite major, as it leads to exponential explosion of the slots count. Otherwise, one might choose to carefully curate the library versions, ensuring that no such situation arises, but this approach is neither feasible nor possible at all for sufficiently large packages base. In practice, outside of Gentoo/system PM issues, this does not pose a problem since each executable is built in an isolated sandbox and links statically to its dependencies, so different executables can happily be deployed alongside each other even if they have incompatible dependencies. But looks like Gentoo's approach to haskell is to use dynamic linking and expose the dependencies to the package manager (which I sure agree is the right thing to do), so what should be right the solution here? -- Georg Rudoy
Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development
On Mon, 20 Apr 2020 15:23:15 -0400 Michael Orlitzky wrote: > On 4/20/20 2:58 PM, Patrick McLean wrote: > >> > >> You keep saying that, but the fact that dev-go/* is filled with trash > >> that has your name on it prevents anyone else from doing a better job. > >> > > Ad-hominen attacks are unwarranted, please refrain from them. I challenge > > you to find *anything* in dev-go/* with my name on it. > > You quoted me one sentence prior saying that it's the Go ebuilds that > are trash and not anyone personally. But OK, I should have said "Sony > Interactive Entertainment" there instead of "you." My co-workers are not the only ones adding/maintaining go packages in the tree, please do not single out any groups, and let's all work to make Gentoo the best it can be. > > > Are you volunteering to do the work to package go packages? The people > > doing the work generally get to decide how that work gets done, and which > > approach they would like to take. The upstream situation makes it very > > labour-intensive to do the work in a the way you are proposing (many > > packages would end up with hundreds to thousands of packages in the tree). > > Separating everything out in to separate packages will just increase the > > maintenance load exponentially, with no gains as go upstreams version lock > > all their dependencies. > > > > I'm volunteering to work on one or two small Go packages. Can I convert > the eclass to use dynamic linking? Can I start replacing your packages > with my own if I need them as dependencies? I suspect not, and that's > one of many reasons why "having things in ::gentoo does not affect > anyone who does not use them" is bullshit. > Please do not make backward-incompatible changes to existing eclasses. If you would like to make a new go eclasss (say go-dynamic.eclass) then go ahead, I am certainly not going to stop you. As was said elsewhere in the thread, there is no policy against having a second version of a package in the tree. If the second version is truly better, then the other packages can be moved over to depending on it instead, and the original package can be moved or last-rited. The current method of packaging go packages tries to strike a balance between maintainability, following upstream, and something that is compatible with ebuilds. If you have a better way to do things, we are not going to block you from trying, but we are also not going to necessarily adopt it if working with it would be orders of magnitude more work. We all have limited time to work on Gentoo stuff.
Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development
On Mon, Apr 20, 2020 at 2:07 PM Michael Orlitzky wrote: > > On 4/20/20 1:31 PM, Patrick McLean wrote: > >> Simply having things in ::gentoo does not affect anyone who does not > use them. > > > > You keep saying that, but the fact that dev-go/* is filled with trash > that has your name on it prevents anyone else from doing a better job. > As far as I'm aware Gentoo policy does not prevent somebody from re-implementing a package that is already in Gentoo. Obviously it could get pretty confusing if that happened. IMO it isn't really worth worrying about, because right now the main limitation seems to be a lack of people working on projects, not 25 devs who each want to re-implement go their own way... -- Rich
Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development
On 4/20/20 2:58 PM, Patrick McLean wrote: >> >> You keep saying that, but the fact that dev-go/* is filled with trash >> that has your name on it prevents anyone else from doing a better job. >> > Ad-hominen attacks are unwarranted, please refrain from them. I challenge you > to find *anything* in dev-go/* with my name on it. You quoted me one sentence prior saying that it's the Go ebuilds that are trash and not anyone personally. But OK, I should have said "Sony Interactive Entertainment" there instead of "you." > Are you volunteering to do the work to package go packages? The people doing > the work generally get to decide how that work gets done, and which approach > they would like to take. The upstream situation makes it very > labour-intensive to do the work in a the way you are proposing (many packages > would end up with hundreds to thousands of packages in the tree). Separating > everything out in to separate packages will just increase the maintenance > load exponentially, with no gains as go upstreams version lock all their > dependencies. > I'm volunteering to work on one or two small Go packages. Can I convert the eclass to use dynamic linking? Can I start replacing your packages with my own if I need them as dependencies? I suspect not, and that's one of many reasons why "having things in ::gentoo does not affect anyone who does not use them" is bullshit.
Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development
On Mon, 20 Apr 2020 14:07:01 -0400 Michael Orlitzky wrote: > On 4/20/20 1:31 PM, Patrick McLean wrote: > >> Simply having things in ::gentoo does not affect anyone who does not > use them. > > > > You keep saying that, but the fact that dev-go/* is filled with trash > that has your name on it prevents anyone else from doing a better job. > Ad-hominen attacks are unwarranted, please refrain from them. I challenge you to find *anything* in dev-go/* with my name on it. Are you volunteering to do the work to package go packages? The people doing the work generally get to decide how that work gets done, and which approach they would like to take. The upstream situation makes it very labour-intensive to do the work in a the way you are proposing (many packages would end up with hundreds to thousands of packages in the tree). Separating everything out in to separate packages will just increase the maintenance load exponentially, with no gains as go upstreams version lock all their dependencies.
Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development
On 4/20/20 1:31 PM, Patrick McLean wrote: >> Simply having things in ::gentoo does not affect anyone who does not use them. > You keep saying that, but the fact that dev-go/* is filled with trash that has your name on it prevents anyone else from doing a better job.
Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development
On Sun, 19 Apr 2020 11:37:15 -0400 Michael Orlitzky wrote: > On 4/19/20 10:55 AM, Samuel Bernardo wrote: > > > > Taking into account the network sandbox requirement, sbt.eclass needs to > > download all dependencies with some approach like EGO_SUM implementation > > in go-module.eclass[1]. > > > > EGO_SUM is not a legitimate approach to packaging. You have to create > packages for each package. I know, it sounds crazy when you say it. > A lot of upstreams are doing things that are not very friendly to packagers. Sometimes we have to take different approaches to packaging to make it feasible to package these very popular programs. Having 300+ packages in the tree with a single version that are only used as a dependency by a single package is not the way to a well-maintained tree. Sometimes things like EGO_SUM are the only feasible way to do things given we all have limited time to work on Gentoo, and we want these packages in the tree because many people find them useful. If you object to the way these are handled, you are free to mask dev-lang/go on your system to endure that you do not get anything written in go installed. Simply having things in ::gentoo does not affect anyone who does not use them. --
Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development
Hi Michael, On 4/19/20 9:09 PM, Michael Orlitzky wrote: > You can do whatever you want in an overlay, but you can't introduce > security vulnerabilities and license issues into thousands of peoples' > homes and businesses through ::gentoo because it makes your life a tiny > bit easier. > > The job description of distribution developer is, ultimately, to fix all > the dumb things that upstream does before packaging the result so that > your users get a consistent, usable, reliable, and secure product. But > the first step isn't optional. Re-packaging garbage is easy -- that's a > service nobody needs. > > Using ebuilds this way is also simply a waste of your time. If you're > not going to package the dependencies, then you're better off using the > upstream bundling tool. At least then you can update the hundreds of > bundled dependencies afterwards. With an ebuild that's not possible. I agree, I have the same concern and that's why I prefer to have an ebuild instead of using the docker container or running blindly the provided upstream bundling tool. Avoiding to package all unnecessary dependencies and keeping away the garbage can only be possible as you mentioned using the distribution tools. At least, with a Gentoo overlay, I can have software better than upstream provided and continue my work. > I don't mean to discourage you any more than necessary. I'm glad you > want to help. But please quit looking to Go as an example of anything > anyone should be doing. Thanks for your advice, but I believe that we win more with the ebuild for the development tools and base services, even if it will only be available in the overlay. Is better to have the eyes of experienced maintainers, Gentoo QA and sharing information, instead of keeping it standalone, perpetuating the bad procedures and software usage. I really feel like scoring when reviewing what I need to do for the ebuild, also when it makes me spent some more time. The profit comes afterwards! Best, Samuel signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development
On 4/19/20 3:41 PM, Samuel Bernardo wrote: >> >> EGO_SUM is not a legitimate approach to packaging. You have to create >> packages for each package. I know, it sounds crazy when you say it. >> > I understand what you mean, but deem this dependencies as internal > project dependencies where those libraries doesn't worth the effort to > package them all. I mean, most of these projects have an immutable > deployment approach that are not feasable to be packaged. The problem > starts from upstream... You can do whatever you want in an overlay, but you can't introduce security vulnerabilities and license issues into thousands of peoples' homes and businesses through ::gentoo because it makes your life a tiny bit easier. The job description of distribution developer is, ultimately, to fix all the dumb things that upstream does before packaging the result so that your users get a consistent, usable, reliable, and secure product. But the first step isn't optional. Re-packaging garbage is easy -- that's a service nobody needs. Using ebuilds this way is also simply a waste of your time. If you're not going to package the dependencies, then you're better off using the upstream bundling tool. At least then you can update the hundreds of bundled dependencies afterwards. With an ebuild that's not possible. I don't mean to discourage you any more than necessary. I'm glad you want to help. But please quit looking to Go as an example of anything anyone should be doing.
Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development
On 2020-04-19 16:37, Michael Orlitzky wrote: > On 4/19/20 10:55 AM, Samuel Bernardo wrote: >> Taking into account the network sandbox requirement, sbt.eclass needs to >> download all dependencies with some approach like EGO_SUM implementation >> in go-module.eclass[1]. >> > EGO_SUM is not a legitimate approach to packaging. You have to create > packages for each package. I know, it sounds crazy when you say it. > I understand what you mean, but deem this dependencies as internal project dependencies where those libraries doesn't worth the effort to package them all. I mean, most of these projects have an immutable deployment approach that are not feasable to be packaged. The problem starts from upstream... Maybe that's the reason why docker is so popular in this cases... But to install the base development tools and services I prefer to have ebuilds rather than dockers. signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development
On Sun, Apr 19, 2020 at 8:37 AM Michael Orlitzky wrote: > On 4/19/20 10:55 AM, Samuel Bernardo wrote: > > > > Taking into account the network sandbox requirement, sbt.eclass needs to > > download all dependencies with some approach like EGO_SUM implementation > > in go-module.eclass[1]. > > > > EGO_SUM is not a legitimate approach to packaging. You have to create > packages for each package. I know, it sounds crazy when you say it. > > EGO_SUM is great! -A
[gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development
On 4/19/20 10:55 AM, Samuel Bernardo wrote: > > Taking into account the network sandbox requirement, sbt.eclass needs to > download all dependencies with some approach like EGO_SUM implementation > in go-module.eclass[1]. > EGO_SUM is not a legitimate approach to packaging. You have to create packages for each package. I know, it sounds crazy when you say it.
[gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development
Hi Benda, Thanks for the reference to Java Packing policy since I haven't read it before. I also forget to mention maven build system in last email, but for now I'm only focused on scala and sbt. On 4/19/20 5:31 AM, Benda Xu wrote: > That's a good idea. What's your plan to realize the eclasses? Taking into account the network sandbox requirement, sbt.eclass needs to download all dependencies with some approach like EGO_SUM implementation in go-module.eclass[1]. Looking in more detail to scala ebuild[2] as a reference, as a quick brainstorm, I think that could be defined: sbt.eclass - ESBTV = This would set the DEPEND for sbt and set the necessary steps to java-pkg-2_pkg_setup, check-reqs_pkg_setup, check-reqs_pkg_pretend, java-pkg_getjars and do the substitution of SBTV in build.properties if required - envset parameters (src_prepare): definition of parameters to set necessary environment for sbt wrapper that use defined ESBTV - ESBT_SUM = - esbt_compile, esbt_run, esbt_package functions: to run sbt using envset wrapper in compile, test and install as necessary scala.eclass - ESCALAV = - ... to be defined as required Best, Samuel [1] https://devmanual.gentoo.org/eclass-reference/go-module.eclass/index.html [2] https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-lang/scala/scala-2.12.10.ebuild signature.asc Description: OpenPGP digital signature
[gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development
Hi Samuel, Samuel Bernardo writes: > My point is about other JVM languages such as scala, groovy, kotlin, > clojure, ... And about builders such as gradle or sbt? > > I think that eclasses for those would be very useful to develop > ebuilds and evolve with the right procedures. That's a good idea. What's your plan to realize the eclasses? BTW, additional reference: https://wiki.gentoo.org/wiki/Gentoo_Java_Packing_Policy Yours, Benda signature.asc Description: PGP signature
