Re: [gentoo-dev] Reverse use of Python/Ruby versions
Dnia 9 kwietnia 2017 18:15:56 CEST, "William L. Thomson Jr." napisał(a): >Not sure if this is practical, it may be less work if the use of >Python and Ruby versions ( maybe others ) is reversed. Rather than >adding all the versions that the ebuild supports. What if it only >included versions it did not support? It is always nice when a person who: a. did not bother to do any research on the topic (such as reading previous posts or even asking the relevant teams), b. has barely any clue (if any at all) about Python ecosystem or package maintenance in Gentoo, c. is either completely ignorant of how Python packages worked in the past (which quite proves the points made above) or presumes that they were changed for no reason by incompetent developers, decides that the workflow of Python team needs to be changed and goes to discuss it on the mailing list with other people who barely do any Python work. > >Rational >When new versions are added. Ebuilds must be updated to support the new >version. This can require editing a decent amount of ebuilds. Many may >work fine with the new version. Making this extra needless work. From a >user point of view, You cannot move to the newer version without >keeping older around till all are updated to the newer version. > >Now one could say its the same work to mark versions not supported. But >I think there is less of that. Also if something supports and older >version and not newer, it may stand out more failing. Requiring someone >to reduce to the older version, and maybe filing bugs to get it updated >to the newer version. > >Python 2.7 stuff aside. I am not sure how many Python and Ruby packages >break with a newer release. In pythons case I think once they support >Python 3.x, there is little chance if it breaking with further 3.x >releases. Not sure about Ruby. > >I could be very wrong and the present way of doing things being the >only way. However if this is feasible it may lead to less work. It >would allow all packages to move to a newer version. Also allowing >punting of older sooner. This leaves the versions solely up to the >eclasses. Then ebuilds simply mark the unsupported versions. Just like >we do now with adding versions it does support. > >Which if something was say only Python 2.7. It would have a >= to >excluded any newer version of Python. That said, we could do the same >with the current way. Saying this supports Python/Ruby >=SLOT. > >Either way I do not think the current way is the best way. You have to >add every version/slot to ebuilds that work fine with any version. When >new ones are added, the ebuild has to be touched again. When it may >work fine with a new version without requiring the ebuild to be >modified. > >This came up with some stuff requiring ruby23 as I moved to 24. Looking >around some stuff has Python 3.5 some 3.6, but all 3.4. So I will stick >to 3.4 till everything is at 3.6. Otherwise no point and still have >other versions. > >The approach mentioned above, if the packages do not have issue. I >could go ahead and switch to ruby24 and pyton 3.6 across the board. >Which I cannot do now till a bunch of ebulids have their targets >increased. -- Best regards, Michał Górny (by phone) -- Best regards, Michał Górny (by phone) -- Best regards, Michał Górny (by phone) -- Best regards, Michał Górny (by phone) -- Best regards, Michał Górny (by phone) -- Best regards, Michał Górny (by phone)
Re: [gentoo-dev] Reverse use of Python/Ruby versions
On Mon, 2017-04-10 at 00:04 +0100, James Le Cuirot wrote: > People thought the sky would fall when 2.4 deprecated Fixnum > and > Bignum. It really didn't. It's still just a warning right now but I > haven't seen that warning since it was dealt with in Rails. > This change broke the rspec test suite quite significantly and to the point where it isn't clear if rspec 3.5 is actually compatible with ruby24 or not. So, no ruby24 target for now, and consequently all packages depending on it also can't get a ruby24 target. Every ruby release will have some breaking changes, and the impact on the tree will depend on which packages it will affect. Hans signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] Reverse use of Python/Ruby versions
On Sun, 9 Apr 2017 22:04:13 -0400 "William L. Thomson Jr." wrote: > This has never been the case with Java. Its not a problem with C binaries either, because you have a discrete compile time, and language level interop between compiled binary forms. Meanwhile, you cannot build two parts of a given python dependency chain with different pythons, nor different perls. > If package A requires version X, but B Y, then B builds with Y as its > pulled in as a dep. while A proceeds to build with X. > > Where this is different for Python, Ruby, and also Perl. They all > install files into a directory based on version. You may have multiple > copies in each, vs one. Perl does not have targets, nor does Java. Right, but this is impossible with Ruby, Python, and Perl. Perl *could* have targets, and some people think could do with it, but it and java are very much in different boats. Perl is in the same boat as Python and Ruby where in "new version of thing" means "everything must be compiled with the new target" Perl simply has a *moving* target instead of multiple concurrent targets. Essentially, with Perl we've done the effect of "Add X, Remove Y" for all things. We additionally have a much better precedent than python at syntax-interop between versions, so its more justifiable to only have the single target ( because you practically never need an older perl "just for compat reasons", though at the rate this garbage[1] is going, that could change one day ) If anything, Perl is only avoiding the Python problem you hate with significant amounts of heroism, and its only a matter of time before upstream force our hands in ways that make that heroism non-viable, and we have to dig deep and work out how the hell to maintain concurrent targets. > The present system is a PITA for users. Fiddling with adding/removing > targets for Python/Ruby. In addition to selecting which for the system. > All these same problems exist for Java, with the exception of > installation locations as mentioned. I honestly think you're looking at the wrong problem domain to fix this problem, in ways that will introduce yet more regressions and broken trees. We should have what I've been saying we should have for a while now: * Soft Options. We only have 2 types of option at present from the users perspective, "on" options, and "off" options. Portage doesn't even distinguish between who is setting them, user profile and the gentoo profiles simply flatten into a single logical profile, and then portage then demands that changes be made to this set, failing to discriminate at all between "ok, this was the profile specified default and it needs to be non-default for this problem" and "user doesn't actually want this, how can we avoid that" And portage then compounds this by dictating that any option changes that ebuilds need must be enshrined by the user as things the /user/ wants, when in truth, they're not things the user wants, they're things the ebuilds want, and the user begrudingly accepts. Hence why an option of "on, but only if portage really needs it" is missing, but needed. I would gladly set soft-targets of every python version under the sun, and then allow portage to turn them on *as necessary*, and this would be much preferable to having to either a) turn them on globally and pull in stuff and waste time compiling stuff that's not even getting used. b) Maintain a painful and carefully catered list of things to prevent aforementioned problems. In short, users need a way to discriminate "things I care about" from "things I don't care about" Currently, its just a big cluster of those things in one place, and the complexity is inescapably thrust into the users hands on a daily basis. 1: https://bugs.gentoo.org/showdependencytree.cgi?id=613764&hide_resolved=0 pgpF4zTjVMYjD.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] Reverse use of Python/Ruby versions
On Mon, 10 Apr 2017 13:38:58 +1200 Kent Fredric wrote: > > When you reverse this, you introduce a situation where adding a new > version across the board creates a new skeleton-tree of support. FYI, this is how it is when a new Java JDK comes out. When 1.5 came out, if 1.4 code had issues compiling, the package was addressed. The same thing can happen any time a new version comes out. The impact most times is minimal. Unless there are wide sweeping changes. Which is pretty rare in any language. > And when you find something that *doesnt* work, you may have to > recursively mark its *dependents* as "non-working" to avoid a > dependency graph breakage. This has never been the case with Java. If package A requires version X, but B Y, then B builds with Y as its pulled in as a dep. while A proceeds to build with X. Where this is different for Python, Ruby, and also Perl. They all install files into a directory based on version. You may have multiple copies in each, vs one. Perl does not have targets, nor does Java. > This is the sort of thing that makes life hell, for both developers > and users. The present system is a PITA for users. Fiddling with adding/removing targets for Python/Ruby. In addition to selecting which for the system. All these same problems exist for Java, with the exception of installation locations as mentioned. > I could be barking up the wrong tree, buy the python team could give a > better idea of what that would look like in practice than me. I have direct experience in this. I am experiencing some of this now with JDK 9. It is different regarding Python and Ruby. It would be up to those teams. But I do think the entire TARGETS aspect needs to be revisited. No one likes adding/removing TARGETS. That is a waste of anyone's time. Much less developers adding/removing targets from ebuilds. -- William L. Thomson Jr. pgpDXwfVE6cAE.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] Reverse use of Python/Ruby versions
On Sun, 9 Apr 2017 12:15:56 -0400 "William L. Thomson Jr." wrote: > > The approach mentioned above, if the packages do not have issue. I > could go ahead and switch to ruby24 and pyton 3.6 across the board. > Which I cannot do now till a bunch of ebulids have their targets > increased. > This could introduce tree breakage. Why? Because of the whole "X built with python FOO depends on a Y built with python FOO" mechanic. As it stands, when unmasking a new python, people have to go through and mark packages as working, which has the requirement that their dependencies are themselves working in order to satisfy the USE requirements. When you reverse this, you introduce a situation where adding a new version across the board creates a new skeleton-tree of support. And when you find something that *doesnt* work, you may have to recursively mark its *dependents* as "non-working" to avoid a dependency graph breakage. This is the sort of thing that makes life hell, for both developers and users. I could be barking up the wrong tree, buy the python team could give a better idea of what that would look like in practice than me. But I fear this would look like "the hell of dekeywording" made harder by the lack of tools to facilitate such a thing. pgpowcZAg5rj2.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] Reverse use of Python/Ruby versions
On Sun, 9 Apr 2017 19:59:22 -0400 Michael Orlitzky wrote: > > How do you plan to test a thousand packages against the new version > of python, and then revision/stabilize all of the broken ones > immediately? Or is the plan to just break everyone's systems, and ask > them to report bugs for the things that stopped working? If packages have tests, running those is one way. If they do not, and its say a library. Long as other packages that use the library build/run against it then it should be ok. It would get trickier for things lacking tests. Breaking already occurs now but in the form of breaking updates and causing users to fiddle with the targets. I am NOT talking about stabilization at all. Simple reducing the burden of adding targets to ebuild, and users having to fiddle with targets as they come and go. > I think what you will actually get as a result is that nobody will > ever add a new version of python to the tree, because you've just > made it a huge ordeal to do so. This is actually the opposite. To add a new version as is right now. You have to edit every Python or Ruby ebuild. Otherwise they cannot use that new version. There is tremendous work as is now. Not to mention a really bad end user experience. This would considerable reduce the workload. Not to mention making life better for end users. -- William L. Thomson Jr. pgpAJEtNcS4qL.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] Reverse use of Python/Ruby versions
On 10/04/2017 01:59, Michael Orlitzky wrote: On 04/09/2017 07:15 PM, William L. Thomson Jr. wrote: If the package failed, all that would need to be done kinda like now is a given variable modified in the ebuild. Just marking what ever it did not work with. As mentioned that could be done via my ebuild-batcher[1], though same functionality is easily replicated. How do you plan to test a thousand packages against the new version of python, and then revision/stabilize all of the broken ones immediately? Or is the plan to just break everyone's systems, and ask them to report bugs for the things that stopped working? python 3.5.0 was released on 2015-09-13 and is still marked unstable and until recently was unusable because there were too many missing packages marked ready for it, stabilization isn't that faster right now. Most of the breakage would be caught when recompiling (bytecode) of a package stable or not and even when not caught it would trigger only eselecting an unstable dev-lang/python if testing all python packages is required to stabilize dev-lang/python (which is kinda the current situation). Python is slotted, that would also help a lot at keeping the breakage unobtrusive * * Provided portage is never broken by a dev-lang/python update, but that's easy to test. I think what you will actually get as a result is that nobody will ever add a new version of python to the tree, because you've just made it a huge ordeal to do so. I do respectfully disagree with this sentence.
[gentoo-dev] Automated Package Removal and Addition Tracker, for the week ending 2017-04-09 23:59 UTC
The attached list notes all of the packages that were added or removed from the tree, for the week ending 2017-04-09 23:59 UTC. Removals: app-misc/ofl 20170404-16:43 mattst88 acf50dc1d27 dev-ruby/ruby-webkit-gtk 20170405-10:10 mgorny 9f79870f533 dev-ruby/ruby-webkit-gtk220170405-10:09 mgorny b312382fff0 media-libs/libechonest 20170406-19:36 johu 8c83524fa3b media-video/lives20170403-21:06 asturm 9710264f4bd net-misc/livestreamer20170405-10:12 mgorny 0dad151515e x11-misc/andromeda 20170403-21:18 asturm 764bd0aec14 x11-misc/oroborus-desklaunch 20170405-10:11 mgorny fc1d0e5bfb4 Additions: app-text/pdf2htmlEX 20170403-10:34 dilfridge c95992906cf dev-ada/asis 20170406-19:23 tupone 04f37879f6a dev-ada/gnatcoll 20170405-09:27 tupone 04bd2b8c8a8 dev-ml/flow_parser 20170403-09:22 aballier 454bdeedcdd dev-ml/merlin20170403-10:22 aballier 461ea5502f9 dev-ml/ocaml-websocket 20170403-09:36 aballier 7e3fa6a3010 dev-perl/Mock-Config 20170408-00:41 kentnl bd9026e1be9 dev-python/prov 20170402-17:37 gokturk0f2155f40fe dev-util/repo20170405-19:13 wizardedit 3471282a8af media-video/libva-utils 20170403-07:29 aballier 7bfca9bc067 net-im/gitter-bin20170406-17:49 gokturkd1e45d64326 net-libs/libsrsirc 20170404-01:44 gokturk1d9a1355a3c sci-chemistry/ParmEd 20170405-19:47 alexxy 560758ca9c9 -- Robin Hugh Johnson Gentoo Linux Developer E-Mail : robb...@gentoo.org GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85 Removed Packages: media-libs/libechonest,removed,johu,20170406-19:36,8c83524fa3b net-misc/livestreamer,removed,mgorny,20170405-10:12,0dad151515e x11-misc/oroborus-desklaunch,removed,mgorny,20170405-10:11,fc1d0e5bfb4 dev-ruby/ruby-webkit-gtk,removed,mgorny,20170405-10:10,9f79870f533 dev-ruby/ruby-webkit-gtk2,removed,mgorny,20170405-10:09,b312382fff0 app-misc/ofl,removed,mattst88,20170404-16:43,acf50dc1d27 x11-misc/andromeda,removed,asturm,20170403-21:18,764bd0aec14 media-video/lives,removed,asturm,20170403-21:06,9710264f4bd Added Packages: dev-python/prov,added,gokturk,20170402-17:37,0f2155f40fe net-im/gitter-bin,added,gokturk,20170406-17:49,d1e45d64326 dev-perl/Mock-Config,added,kentnl,20170408-00:41,bd9026e1be9 dev-ada/asis,added,tupone,20170406-19:23,04f37879f6a sci-chemistry/ParmEd,added,alexxy,20170405-19:47,560758ca9c9 dev-util/repo,added,wizardedit,20170405-19:13,3471282a8af dev-ada/gnatcoll,added,tupone,20170405-09:27,04bd2b8c8a8 net-libs/libsrsirc,added,gokturk,20170404-01:44,1d9a1355a3c app-text/pdf2htmlEX,added,dilfridge,20170403-10:34,c95992906cf dev-ml/merlin,added,aballier,20170403-10:22,461ea5502f9 dev-ml/ocaml-websocket,added,aballier,20170403-09:36,7e3fa6a3010 dev-ml/flow_parser,added,aballier,20170403-09:22,454bdeedcdd media-video/libva-utils,added,aballier,20170403-07:29,7bfca9bc067 Done.
Re: [gentoo-dev] Reverse use of Python/Ruby versions
On 04/09/2017 07:15 PM, William L. Thomson Jr. wrote: If the package failed, all that would need to be done kinda like now is a given variable modified in the ebuild. Just marking what ever it did not work with. As mentioned that could be done via my ebuild-batcher[1], though same functionality is easily replicated. How do you plan to test a thousand packages against the new version of python, and then revision/stabilize all of the broken ones immediately? Or is the plan to just break everyone's systems, and ask them to report bugs for the things that stopped working? I think what you will actually get as a result is that nobody will ever add a new version of python to the tree, because you've just made it a huge ordeal to do so.
Re: [gentoo-dev] Reverse use of Python/Ruby versions
On Sun, 9 Apr 2017 15:20:02 -0700 Brian Dolbec wrote: > > This could be partially automated using buildbot. The easiest would > be for pkgs containing working test suites. Those that pass could be > auto-enabled with that python with ~arch keywords. I think if a > stable pkg is to be added the new python, it should be via a revision > bump and ~arch'ing the keywords. But we could enforce the bot to > rev-bump all ebuilds just as easily. If the package failed, all that would need to be done kinda like now is a given variable modified in the ebuild. Just marking what ever it did not work with. As mentioned that could be done via my ebuild-batcher[1], though same functionality is easily replicated. I also have an ebuild-bumper[2], that could be made more advanced. I am aware of ebump, but ebuild-bumper is a bit different. It works with sets of ebuilds for bumps and cleaning. It can do individual as well but so can ebump. I plan to further automate ebuild-bumper with something that tracks upstream releases and attempts to automate bumps. Though even running it manually is not cumbersome. Just prefer zero day as much as possible. 1. https://github.com/Obsidian-StudiosInc/ebuild-batcher 2. https://github.com/Obsidian-StudiosInc/ebuild-bumper -- William L. Thomson Jr. pgp_JhiFSjrcx.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] Reverse use of Python/Ruby versions
On Sun, 9 Apr 2017 23:44:50 +0200 Kristian Fiskerstrand wrote: > On 04/09/2017 06:15 PM, William L. Thomson Jr. wrote: > > Not sure if this is practical, it may be less work if the use of > > Python and Ruby versions ( maybe others ) is reversed. Rather than > > adding all the versions that the ebuild supports. What if it only > > included versions it did not support? > > It would only work if upstream provide a strong assurance for forward > compatibility. Explicit testing and marking working seems the only > practical way to ensure stability. Even if things break, you just do the opposite of now. You would disable/mask ( or something to that effect ) any versions the package did not support. Basically what is done now but in reverse. Say what it does not build with; allowing it to build with anything it can, existing today, or coming in the future. In theory at least one would have to modify less ebuilds that break with a new version. Than modifying all adding a new target. There is also the added bonus when a version is dropped. No ebuilds need be modified. I assume if say python 3.4 is dropped. All ebuilds with that target need to be updated. This would considerably reduce work all around, with a much better experience for the end user. No targets to fool with. -- William L. Thomson Jr. pgpdzajB8wC21.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] Reverse use of Python/Ruby versions
On Sun, 9 Apr 2017 12:15:56 -0400 "William L. Thomson Jr." wrote: > Python 2.7 stuff aside. I am not sure how many Python and Ruby packages > break with a newer release. In pythons case I think once they support > Python 3.x, there is little chance if it breaking with further 3.x > releases. Not sure about Ruby. I'm not going to weigh heavily into this as I don't mind the current setup but as a professional Ruby developer, I can say that breakages between versions are usually overblown by those outside the community. Yeah, there usually are some but they tend to only affect the bigger libraries and frameworks that do some more exotic things. Even then, the changes required are generally very small, sometimes even just one line. People thought the sky would fall when 2.4 deprecated Fixnum and Bignum. It really didn't. It's still just a warning right now but I haven't seen that warning since it was dealt with in Rails. -- James Le Cuirot (chewi) Gentoo Linux Developer pgpr0oLkm0YAA.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] Reverse use of Python/Ruby versions
On 10/04/2017 00:20, Brian Dolbec wrote: On Sun, 9 Apr 2017 23:36:18 +0200 Francesco Riosa wrote: On 09/04/2017 18:15, William L. Thomson Jr. wrote: Not sure if this is practical, it may be less work if the use of Python and Ruby versions ( maybe others ) is reversed. Rather than adding all the versions that the ebuild supports. What if it only included versions it did not support? Rational When new versions are added. Ebuilds must be updated to support the new version. This can require editing a decent amount of ebuilds. Many may work fine with the new version. Making this extra needless work. From a user point of view, You cannot move to the newer version without keeping older around till all are updated to the newer version. Now one could say its the same work to mark versions not supported. But I think there is less of that. Also if something supports and older version and not newer, it may stand out more failing. Requiring someone to reduce to the older version, and maybe filing bugs to get it updated to the newer version. Python 2.7 stuff aside. I am not sure how many Python and Ruby packages break with a newer release. In pythons case I think once they support Python 3.x, there is little chance if it breaking with further 3.x releases. Not sure about Ruby. I could be very wrong and the present way of doing things being the only way. However if this is feasible it may lead to less work. It would allow all packages to move to a newer version. Also allowing punting of older sooner. This leaves the versions solely up to the eclasses. Then ebuilds simply mark the unsupported versions. Just like we do now with adding versions it does support. Which if something was say only Python 2.7. It would have a >= to excluded any newer version of Python. That said, we could do the same with the current way. Saying this supports Python/Ruby >=SLOT. Either way I do not think the current way is the best way. You have to add every version/slot to ebuilds that work fine with any version. When new ones are added, the ebuild has to be touched again. When it may work fine with a new version without requiring the ebuild to be modified. This came up with some stuff requiring ruby23 as I moved to 24. Looking around some stuff has Python 3.5 some 3.6, but all 3.4. So I will stick to 3.4 till everything is at 3.6. Otherwise no point and still have other versions. The approach mentioned above, if the packages do not have issue. I could go ahead and switch to ruby24 and pyton 3.6 across the board. Which I cannot do now till a bunch of ebulids have their targets increased. +1 to the python part, cannot speak about ruby. or totally automate the addition of python versions to ebuilds at the exact time of bumping dev-lang/python. This could be partially automated using buildbot. The easiest would be for pkgs containing working test suites. Those that pass could be auto-enabled with that python with ~arch keywords. I think if a stable pkg is to be added the new python, it should be via a revision bump and ~arch'ing the keywords. But we could enforce the bot to rev-bump all ebuilds just as easily. Pkgs without tests, those would be harder and we could do some basic tests on those, like syntax, test imports, but would require additional means of testing in order to qualify for an auto-python addition. It should also be possible for the bot to scrape setup.py for comppatible python versions. Many of the pkgs I have been working with recently have them listed. Also there is travis.yml files in many upstream pkg repos which can also be scraped for tested pythons. It could certainly reduce the manpower needed to keep things up to date. All good ideas, if William proposal is rejected, I'd like to request a mandatory step like this when a new minor 3.x python version is added to the tree. No idea if that's feasible for ruby too.
Re: [gentoo-dev] Reverse use of Python/Ruby versions
On 09/04/2017 23:52, Michael Orlitzky wrote: On 04/09/2017 12:15 PM, William L. Thomson Jr. wrote: Not sure if this is practical, it may be less work if the use of Python and Ruby versions ( maybe others ) is reversed. Rather than adding all the versions that the ebuild supports. What if it only included versions it did not support? Even if this would work better, it would require retraining all developers, completely rewriting several eclasses, tons of documentation, and a few thousand ebuilds. Let's assume the retraining will be 2 or 3 orders of magnitude easier than switching from cvs to git, than that's doable. Eclasses, ebuilds and documentation will be the real burden, but at this point it's better to discuss if the feature is wanted or not, then later wait for the volunteer(s) to actually do the work. No one's going to jump on that bandwagon without a proof-of-concept that works much better than what we have now. yep, that could be done, but since it's not trivial maybe better decide if it will be wasted or possibly accepted. by the way eclasses from gentoo repo with PYTHON string inside are the following: $ grep -c PYTHON *.eclass | grep -v :0$ distutils-r1.eclass:28 enlightenment.eclass:6 gnome-python-common-r1.eclass:2 kernel-2.eclass:2 mate.eclass:1 mozcoreconf-v4.eclass:3 python-any-r1.eclass:75 python-r1.eclass:107 python-single-r1.eclass:112 python-utils-r1.eclass:203 ros-catkin.eclass:40 twisted-r1.eclass:2 waf-utils.eclass:7
Re: [gentoo-dev] Reverse use of Python/Ruby versions
On Sun, 9 Apr 2017 17:52:44 -0400 Michael Orlitzky wrote: > On 04/09/2017 12:15 PM, William L. Thomson Jr. wrote: > > Not sure if this is practical, it may be less work if the use of > > Python and Ruby versions ( maybe others ) is reversed. Rather than > > adding all the versions that the ebuild supports. What if it only > > included versions it did not support? > > Even if this would work better, it would require retraining all > developers, completely rewriting several eclasses, tons of > documentation, and a few thousand ebuilds. There could be things done to ease any transition. Regarding python, the first of which could be to ignore any targets beyond say 2.7 and just starting enabling support. That would mostly leave just effected packages, ones that break with some 3.x version but not with another. As for modifying a few thousand ebuilds; 1, That is already the case now. If a new python or ruby version comes out. All those ebuilds have to modified to support the new target. That argument isn't really valid. 2. As for the actual changes that can be done pragmatically if just swapping out the TARGETS variable. Or even if more advanced requiring changing lines. If its the same change to all. I have a script, ebuild-batcher[1], that can safely make any change to a massive number of ebuilds. I have used it a few times on hundreds of ebuilds. It can easily handle thousands. > No one's going to jump on that bandwagon without a proof-of-concept > that works much better than what we have now. Anyone who has worked with a Gentoo system long enough that has Python and/or Ruby has experienced this. You end up having multiple TARGETS and are constantly messing with those variables adding new ones, removing old, etc. It creates a considerable amount of work. I have been fighting over ruby23 that I cannot seem to avoid. Despite having had ruby24 for I think at least a week or more. My build server stopped due to needing ruby23. I have been fighting with stuff around that. More than likely anything bound to ruby23 target works with ruby24. Its mostly how the system is designed. I could be wrong in that case and something require ruby23, that does not with ruby24. However most those packages do not have a ruby24 target, so its hard to say if its negated or just not enabled yet. Being a new version, all have to be modified for such. 1. https://github.com/Obsidian-StudiosInc/ebuild-batcher -- William L. Thomson Jr. pgp_eJ93OMeS0.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] Reverse use of Python/Ruby versions
On 09/04/2017 23:44, Kristian Fiskerstrand wrote: On 04/09/2017 06:15 PM, William L. Thomson Jr. wrote: Not sure if this is practical, it may be less work if the use of Python and Ruby versions ( maybe others ) is reversed. Rather than adding all the versions that the ebuild supports. What if it only included versions it did not support? It would only work if upstream provide a strong assurance for forward compatibility. Explicit testing and marking working seems the only practical way to ensure stability. Surely enough forward compatibility may be a problem and python upstream does deprecate and remove features #1 and things that fiddle with python bytecode will easily break. However we keep $KEYWORDS between version of the same package and that it's subject to the same exact kind of problems. Honestly just trying out python 3.6 is a pain at the moment and the situation is the same at every python bump. #1 https://docs.python.org/3/whatsnew/index.html
Re: [gentoo-dev] Reverse use of Python/Ruby versions
On Sun, 9 Apr 2017 23:36:18 +0200 Francesco Riosa wrote: > On 09/04/2017 18:15, William L. Thomson Jr. wrote: > > Not sure if this is practical, it may be less work if the use of > > Python and Ruby versions ( maybe others ) is reversed. Rather than > > adding all the versions that the ebuild supports. What if it only > > included versions it did not support? > > > > Rational > > When new versions are added. Ebuilds must be updated to support the > > new version. This can require editing a decent amount of ebuilds. > > Many may work fine with the new version. Making this extra needless > > work. From a user point of view, You cannot move to the newer > > version without keeping older around till all are updated to the > > newer version. > > > > Now one could say its the same work to mark versions not supported. > > But I think there is less of that. Also if something supports and > > older version and not newer, it may stand out more failing. > > Requiring someone to reduce to the older version, and maybe filing > > bugs to get it updated to the newer version. > > > > Python 2.7 stuff aside. I am not sure how many Python and Ruby > > packages break with a newer release. In pythons case I think once > > they support Python 3.x, there is little chance if it breaking with > > further 3.x releases. Not sure about Ruby. > > > > I could be very wrong and the present way of doing things being the > > only way. However if this is feasible it may lead to less work. It > > would allow all packages to move to a newer version. Also allowing > > punting of older sooner. This leaves the versions solely up to the > > eclasses. Then ebuilds simply mark the unsupported versions. Just > > like we do now with adding versions it does support. > > > > Which if something was say only Python 2.7. It would have a >= to > > excluded any newer version of Python. That said, we could do the > > same with the current way. Saying this supports Python/Ruby >=SLOT. > > > > Either way I do not think the current way is the best way. You have > > to add every version/slot to ebuilds that work fine with any > > version. When new ones are added, the ebuild has to be touched > > again. When it may work fine with a new version without requiring > > the ebuild to be modified. > > > > This came up with some stuff requiring ruby23 as I moved to 24. > > Looking around some stuff has Python 3.5 some 3.6, but all 3.4. So > > I will stick to 3.4 till everything is at 3.6. Otherwise no point > > and still have other versions. > > > > The approach mentioned above, if the packages do not have issue. I > > could go ahead and switch to ruby24 and pyton 3.6 across the board. > > Which I cannot do now till a bunch of ebulids have their targets > > increased. > > > +1 to the python part, cannot speak about ruby. > or totally automate the addition of python versions to ebuilds at the > exact time of bumping dev-lang/python. > > This could be partially automated using buildbot. The easiest would be for pkgs containing working test suites. Those that pass could be auto-enabled with that python with ~arch keywords. I think if a stable pkg is to be added the new python, it should be via a revision bump and ~arch'ing the keywords. But we could enforce the bot to rev-bump all ebuilds just as easily. Pkgs without tests, those would be harder and we could do some basic tests on those, like syntax, test imports, but would require additional means of testing in order to qualify for an auto-python addition. It should also be possible for the bot to scrape setup.py for comppatible python versions. Many of the pkgs I have been working with recently have them listed. Also there is travis.yml files in many upstream pkg repos which can also be scraped for tested pythons. It could certainly reduce the manpower needed to keep things up to date. -- Brian Dolbec
Re: [gentoo-dev] Reverse use of Python/Ruby versions
On 04/09/2017 12:15 PM, William L. Thomson Jr. wrote: Not sure if this is practical, it may be less work if the use of Python and Ruby versions ( maybe others ) is reversed. Rather than adding all the versions that the ebuild supports. What if it only included versions it did not support? Even if this would work better, it would require retraining all developers, completely rewriting several eclasses, tons of documentation, and a few thousand ebuilds. No one's going to jump on that bandwagon without a proof-of-concept that works much better than what we have now.
Re: [gentoo-dev] Reverse use of Python/Ruby versions
On 04/09/2017 06:15 PM, William L. Thomson Jr. wrote: > Not sure if this is practical, it may be less work if the use of > Python and Ruby versions ( maybe others ) is reversed. Rather than > adding all the versions that the ebuild supports. What if it only > included versions it did not support? It would only work if upstream provide a strong assurance for forward compatibility. Explicit testing and marking working seems the only practical way to ensure stability. -- Kristian Fiskerstrand OpenPGP keyblock reachable at hkp://pool.sks-keyservers.net fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3 signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] Reverse use of Python/Ruby versions
On 09/04/2017 18:15, William L. Thomson Jr. wrote: Not sure if this is practical, it may be less work if the use of Python and Ruby versions ( maybe others ) is reversed. Rather than adding all the versions that the ebuild supports. What if it only included versions it did not support? Rational When new versions are added. Ebuilds must be updated to support the new version. This can require editing a decent amount of ebuilds. Many may work fine with the new version. Making this extra needless work. From a user point of view, You cannot move to the newer version without keeping older around till all are updated to the newer version. Now one could say its the same work to mark versions not supported. But I think there is less of that. Also if something supports and older version and not newer, it may stand out more failing. Requiring someone to reduce to the older version, and maybe filing bugs to get it updated to the newer version. Python 2.7 stuff aside. I am not sure how many Python and Ruby packages break with a newer release. In pythons case I think once they support Python 3.x, there is little chance if it breaking with further 3.x releases. Not sure about Ruby. I could be very wrong and the present way of doing things being the only way. However if this is feasible it may lead to less work. It would allow all packages to move to a newer version. Also allowing punting of older sooner. This leaves the versions solely up to the eclasses. Then ebuilds simply mark the unsupported versions. Just like we do now with adding versions it does support. Which if something was say only Python 2.7. It would have a >= to excluded any newer version of Python. That said, we could do the same with the current way. Saying this supports Python/Ruby >=SLOT. Either way I do not think the current way is the best way. You have to add every version/slot to ebuilds that work fine with any version. When new ones are added, the ebuild has to be touched again. When it may work fine with a new version without requiring the ebuild to be modified. This came up with some stuff requiring ruby23 as I moved to 24. Looking around some stuff has Python 3.5 some 3.6, but all 3.4. So I will stick to 3.4 till everything is at 3.6. Otherwise no point and still have other versions. The approach mentioned above, if the packages do not have issue. I could go ahead and switch to ruby24 and pyton 3.6 across the board. Which I cannot do now till a bunch of ebulids have their targets increased. +1 to the python part, cannot speak about ruby. or totally automate the addition of python versions to ebuilds at the exact time of bumping dev-lang/python.
[gentoo-dev] Reverse use of Python/Ruby versions
Not sure if this is practical, it may be less work if the use of Python and Ruby versions ( maybe others ) is reversed. Rather than adding all the versions that the ebuild supports. What if it only included versions it did not support? Rational When new versions are added. Ebuilds must be updated to support the new version. This can require editing a decent amount of ebuilds. Many may work fine with the new version. Making this extra needless work. From a user point of view, You cannot move to the newer version without keeping older around till all are updated to the newer version. Now one could say its the same work to mark versions not supported. But I think there is less of that. Also if something supports and older version and not newer, it may stand out more failing. Requiring someone to reduce to the older version, and maybe filing bugs to get it updated to the newer version. Python 2.7 stuff aside. I am not sure how many Python and Ruby packages break with a newer release. In pythons case I think once they support Python 3.x, there is little chance if it breaking with further 3.x releases. Not sure about Ruby. I could be very wrong and the present way of doing things being the only way. However if this is feasible it may lead to less work. It would allow all packages to move to a newer version. Also allowing punting of older sooner. This leaves the versions solely up to the eclasses. Then ebuilds simply mark the unsupported versions. Just like we do now with adding versions it does support. Which if something was say only Python 2.7. It would have a >= to excluded any newer version of Python. That said, we could do the same with the current way. Saying this supports Python/Ruby >=SLOT. Either way I do not think the current way is the best way. You have to add every version/slot to ebuilds that work fine with any version. When new ones are added, the ebuild has to be touched again. When it may work fine with a new version without requiring the ebuild to be modified. This came up with some stuff requiring ruby23 as I moved to 24. Looking around some stuff has Python 3.5 some 3.6, but all 3.4. So I will stick to 3.4 till everything is at 3.6. Otherwise no point and still have other versions. The approach mentioned above, if the packages do not have issue. I could go ahead and switch to ruby24 and pyton 3.6 across the board. Which I cannot do now till a bunch of ebulids have their targets increased. -- William L. Thomson Jr. pgpE1goV3vWlr.pgp Description: OpenPGP digital signature
