Re: [gentoo-dev] Ebuild Generators
Am Dienstag, 11. Februar 2020, 16:18:44 CET schrieb Tom Gillespie: > For historical curiosity there was also > https://github.com/domenkozar/g-pypi at one point (similar to > https://github.com/rafaelmartins/g-octave). Having used g-octave, the > primary issue is as Michał says, there are a lot of corner cases that > the generation doesn't handle correctly which lead to broken ebuilds > and maintenance headaches. Speaking for python, catching and > correcting the use of setup_requires and other insanity visited upon > us by the wonders of setuptools makes this a non-starter That's essentially my whole point. Generators are _not_ able to catch all corner cases but should be able to lower the initial ebuild creation cost. That could be such things like figure out the license, try to fill the source url, In principal: Try to automate what is possible and leave the rest to the maintainer. But in that state they are a tool for the Gentoo developer not for the user. Gerion signature.asc Description: This is a digitally signed message part.
Re: [gentoo-dev] Ebuild Generators
Tom Gillespie writes: > For historical curiosity there was also > https://github.com/domenkozar/g-pypi at one point (similar to > https://github.com/rafaelmartins/g-octave). Having used g-octave, the > primary issue is as Michał says, there are a lot of corner cases that > the generation doesn't handle correctly which lead to broken ebuilds > and maintenance headaches. Speaking for python, catching and > correcting the use of setup_requires and other insanity visited upon > us by the wonders of setuptools makes this a non-starter. Yes, I agree. > If you have a set a known sane packages you could in theory write an > eclass that captures the regularities of those packages, but I'm not > sure eclasses are suggested for that use case. eclass is designed for eliminating duplicated code in ebuilds. Therefore yes, it is the legitimate use of eclass. Benda
Re: [gentoo-dev] Ebuild Generators
For historical curiosity there was also https://github.com/domenkozar/g-pypi at one point (similar to https://github.com/rafaelmartins/g-octave). Having used g-octave, the primary issue is as Michał says, there are a lot of corner cases that the generation doesn't handle correctly which lead to broken ebuilds and maintenance headaches. Speaking for python, catching and correcting the use of setup_requires and other insanity visited upon us by the wonders of setuptools makes this a non-starter. If you have a set a known sane packages you could in theory write an eclass that captures the regularities of those packages, but I'm not sure eclasses are suggested for that use case. Tom On Mon, Feb 3, 2020 at 5:27 AM Michał Górny wrote: > > On Mon, 2020-02-03 at 14:20 +0100, Gerion Entrup wrote: > > Am Montag, 3. Februar 2020, 13:19:52 CET schrieb Benda Xu: > > > Hi Gerion, > > > > > > Gerion Entrup writes: > > > > > > > > Yes, that makes a lot of sense. The R overlay follows this model. > > > > > Most > > > > > of the ebuilds are automated. When an ebuild generation fails, we add > > > > > the ebuild manually, understand it and then update the generator to > > > > > cover it in the future. > > > > > > > > Is this possible in all cases? I think of adding custom patches, > > > > appropriate mapping of dependencies, check for things like desktop > > > > icon cache... > > > > > > That's too complex to handle automatically. Luckily, in R overlay, such > > > packages are less than 5%. An ebuild generator is based on the > > > observation that many language-specific packages are trivial to fetch, > > > compile and install. > > > > > > > > > I'm only "maintaining" an overlay so maybe I'm missing experience > > > > > > but I often have wished a tool that automatically parses the > > > > > > language specific > > > > > > packaging files and is able to generate a primitive ebuild out of > > > > > > that. > > > > > > Maybe it even can do this in an interactive way: > > > > > > "Hey, upstream needs the dependency 'foo'. In the Gentoo packages I > > > > > > have found > > > > > > 'dev-bar/foo' and 'dev-util/foo'. What is the correct one?" > > > > > > > > > > Yes, that's the way R overlay is working. And I have a similar plan > > > > > and > > > > > proof-of-concept solution for the Java Maven overlay. > > > > > > > > Nice to hear. I think, it is meaningful to solve all generation with one > > > > tool. Maybe it can even "recognize" the used build system and package > > > > database. Is this your plan, too? > > > > > > No, I don't think it possible as far as I can see... That would be a > > > strong AI. > > I mean only on a primitive base: > > ``` > > if link contains "pypi": > > # it's a Python package from pypi > > handle_pypi() > > elif work_tree contains "setup.py": > > # it's a Python package > > handle_generic_python() > > > > Please don't use generators for Python. You'd have to put a lot of > effort to get things right. Most of the time, you'll end up with broken > or no tests, useless deps and py2 where unnecessary. > > Creating ebuilds is not a problem. Maintaining is. Python team ended > up with lots of packages dumped by former project members or other > developers. Many of them are of very bad quality. Even those that > aren't are a maintenance burden. Removing broken and/or useless > packages is taking a lot of our time. > > -- > Best regards, > Michał Górny >
Re: [gentoo-dev] Ebuild Generators
On Mon, 2020-02-03 at 14:20 +0100, Gerion Entrup wrote: > Am Montag, 3. Februar 2020, 13:19:52 CET schrieb Benda Xu: > > Hi Gerion, > > > > Gerion Entrup writes: > > > > > > Yes, that makes a lot of sense. The R overlay follows this model. Most > > > > of the ebuilds are automated. When an ebuild generation fails, we add > > > > the ebuild manually, understand it and then update the generator to > > > > cover it in the future. > > > > > > Is this possible in all cases? I think of adding custom patches, > > > appropriate mapping of dependencies, check for things like desktop > > > icon cache... > > > > That's too complex to handle automatically. Luckily, in R overlay, such > > packages are less than 5%. An ebuild generator is based on the > > observation that many language-specific packages are trivial to fetch, > > compile and install. > > > > > > > I'm only "maintaining" an overlay so maybe I'm missing experience > > > > > but I often have wished a tool that automatically parses the language > > > > > specific > > > > > packaging files and is able to generate a primitive ebuild out of > > > > > that. > > > > > Maybe it even can do this in an interactive way: > > > > > "Hey, upstream needs the dependency 'foo'. In the Gentoo packages I > > > > > have found > > > > > 'dev-bar/foo' and 'dev-util/foo'. What is the correct one?" > > > > > > > > Yes, that's the way R overlay is working. And I have a similar plan and > > > > proof-of-concept solution for the Java Maven overlay. > > > > > > Nice to hear. I think, it is meaningful to solve all generation with one > > > tool. Maybe it can even "recognize" the used build system and package > > > database. Is this your plan, too? > > > > No, I don't think it possible as far as I can see... That would be a > > strong AI. > I mean only on a primitive base: > ``` > if link contains "pypi": > # it's a Python package from pypi > handle_pypi() > elif work_tree contains "setup.py": > # it's a Python package > handle_generic_python() > Please don't use generators for Python. You'd have to put a lot of effort to get things right. Most of the time, you'll end up with broken or no tests, useless deps and py2 where unnecessary. Creating ebuilds is not a problem. Maintaining is. Python team ended up with lots of packages dumped by former project members or other developers. Many of them are of very bad quality. Even those that aren't are a maintenance burden. Removing broken and/or useless packages is taking a lot of our time. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] Ebuild Generators
Am Montag, 3. Februar 2020, 13:19:52 CET schrieb Benda Xu: > Hi Gerion, > > Gerion Entrup writes: > > >> Yes, that makes a lot of sense. The R overlay follows this model. Most > >> of the ebuilds are automated. When an ebuild generation fails, we add > >> the ebuild manually, understand it and then update the generator to > >> cover it in the future. > > > > Is this possible in all cases? I think of adding custom patches, > > appropriate mapping of dependencies, check for things like desktop > > icon cache... > > That's too complex to handle automatically. Luckily, in R overlay, such > packages are less than 5%. An ebuild generator is based on the > observation that many language-specific packages are trivial to fetch, > compile and install. > > >> > I'm only "maintaining" an overlay so maybe I'm missing experience > >> > but I often have wished a tool that automatically parses the language > >> > specific > >> > packaging files and is able to generate a primitive ebuild out of that. > >> > Maybe it even can do this in an interactive way: > >> > "Hey, upstream needs the dependency 'foo'. In the Gentoo packages I have > >> > found > >> > 'dev-bar/foo' and 'dev-util/foo'. What is the correct one?" > >> > >> Yes, that's the way R overlay is working. And I have a similar plan and > >> proof-of-concept solution for the Java Maven overlay. > > > > Nice to hear. I think, it is meaningful to solve all generation with one > > tool. Maybe it can even "recognize" the used build system and package > > database. Is this your plan, too? > > No, I don't think it possible as far as I can see... That would be a > strong AI. I mean only on a primitive base: ``` if link contains "pypi": # it's a Python package from pypi handle_pypi() elif work_tree contains "setup.py": # it's a Python package handle_generic_python() elif work_tree contains "meson.build": handle_meson_package() ... ``` Best, Gerion signature.asc Description: This is a digitally signed message part.
Re: [gentoo-dev] Ebuild Generators
On 03/02/20 12:19, Benda Xu wrote: > Hi Gerion, > > Gerion Entrup writes: > >>> Yes, that makes a lot of sense. The R overlay follows this model. Most >>> of the ebuilds are automated. When an ebuild generation fails, we add >>> the ebuild manually, understand it and then update the generator to >>> cover it in the future. >> Is this possible in all cases? I think of adding custom patches, >> appropriate mapping of dependencies, check for things like desktop >> icon cache... > That's too complex to handle automatically. Luckily, in R overlay, such > packages are less than 5%. An ebuild generator is based on the > observation that many language-specific packages are trivial to fetch, > compile and install. > I'm only "maintaining" an overlay so maybe I'm missing experience but I often have wished a tool that automatically parses the language specific packaging files and is able to generate a primitive ebuild out of that. Maybe it even can do this in an interactive way: "Hey, upstream needs the dependency 'foo'. In the Gentoo packages I have found 'dev-bar/foo' and 'dev-util/foo'. What is the correct one?" >>> Yes, that's the way R overlay is working. And I have a similar plan and >>> proof-of-concept solution for the Java Maven overlay. >> Nice to hear. I think, it is meaningful to solve all generation with one >> tool. Maybe it can even "recognize" the used build system and package >> database. Is this your plan, too? > No, I don't think it possible as far as I can see... That would be a > strong AI. > > Yours, > Benda There was some interest in doing this for PyPI packages for Liguros linux. See https://gitlab.com/liguros/bugs/issues/75 . signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] Ebuild Generators
Hi Gerion, Gerion Entrup writes: >> Yes, that makes a lot of sense. The R overlay follows this model. Most >> of the ebuilds are automated. When an ebuild generation fails, we add >> the ebuild manually, understand it and then update the generator to >> cover it in the future. > > Is this possible in all cases? I think of adding custom patches, > appropriate mapping of dependencies, check for things like desktop > icon cache... That's too complex to handle automatically. Luckily, in R overlay, such packages are less than 5%. An ebuild generator is based on the observation that many language-specific packages are trivial to fetch, compile and install. >> > I'm only "maintaining" an overlay so maybe I'm missing experience >> > but I often have wished a tool that automatically parses the language >> > specific >> > packaging files and is able to generate a primitive ebuild out of that. >> > Maybe it even can do this in an interactive way: >> > "Hey, upstream needs the dependency 'foo'. In the Gentoo packages I have >> > found >> > 'dev-bar/foo' and 'dev-util/foo'. What is the correct one?" >> >> Yes, that's the way R overlay is working. And I have a similar plan and >> proof-of-concept solution for the Java Maven overlay. > > Nice to hear. I think, it is meaningful to solve all generation with one > tool. Maybe it can even "recognize" the used build system and package > database. Is this your plan, too? No, I don't think it possible as far as I can see... That would be a strong AI. Yours, Benda signature.asc Description: PGP signature
Re: [gentoo-dev] Ebuild Generators (Was: GSoC 2020: Call for mentors and project ideas)
Am Montag, 3. Februar 2020, 05:20:42 CET schrieb Benda Xu: > Gerion Entrup writes: > > > I saw the idea „Big Data Infrastructure by Gentoo“ and found it kind of > > interesting. However, I have a little bit the fear that a full automation > > won't be possible and the whole project becomes a little bit like g-sorcery > > (gs-pypi, gs-elpa) or g-octave: a really cool project but not used at a > > large scale. > > Yes, that's true. I share the same observation and concern with you. > > This is one exception: the CRAN ebuild generator powered R overlay has > been running well for 8 years. > > https://wiki.gentoo.org/wiki/Project:Science/Overlay/R Hmm, interesting, thank you. > > What do you think of the idea to not do this fully automated but supervised > > by a maintainer? With that I mean an ebuild generator that generates only > > the parts of the ebuild that it can easily parse and then present the ebuild > > draft to a maintainer who completes it to an full ebuild. As far a I know no > > tool like this exists. I think the focus shift helps a lot: > > Developing a tool for the Gentoo maintainer not the Gentoo user. > > Yes, that makes a lot of sense. The R overlay follows this model. Most > of the ebuilds are automated. When an ebuild generation fails, we add > the ebuild manually, understand it and then update the generator to > cover it in the future. Is this possible in all cases? I think of adding custom patches, appropriate mapping of dependencies, check for things like desktop icon cache... > > I'm only "maintaining" an overlay so maybe I'm missing experience > > but I often have wished a tool that automatically parses the language > > specific > > packaging files and is able to generate a primitive ebuild out of that. > > Maybe it even can do this in an interactive way: > > "Hey, upstream needs the dependency 'foo'. In the Gentoo packages I have > > found > > 'dev-bar/foo' and 'dev-util/foo'. What is the correct one?" > > Yes, that's the way R overlay is working. And I have a similar plan and > proof-of-concept solution for the Java Maven overlay. Nice to hear. I think, it is meaningful to solve all generation with one tool. Maybe it can even "recognize" the used build system and package database. Is this your plan, too? Best, Gerion signature.asc Description: This is a digitally signed message part.
