Re: [sage-devel] 7.4 release: please don't have fpylll build-depend on Sage
On Wed, Oct 19, 2016 at 1:22 PM, Ximin Luo wrote: > Jeroen Demeyer: >> On 2016-10-18 17:52, Ximin Luo wrote: >>> (2) In the long run, one can think about splitting out sage.rings.integers >>> (and related things) into a small library "sage-types" or something like >>> that. Then sagelib and fpylll can depend on this, and there would be no >>> circular dependency, even when Debian or other distros try to build it. >> >> I do not understand how that would help you. I am proposing a solution >> >> (A) Split the Sage package in 2 packages (Sage-the-library + >> Sage-the-distribution) >> >> and you have various reasons why this doesn't work. Instead you propose >> >> (B) Split the Sage package in 2 packages (sage.rings.integers (and related >> things) + the rest) >> >> To me, (A) and (B) look very similar. Why would (B) work while (A) does not? >> In other words, why are the problems that you mention for (A) not applicable >> to (B)? >> > > It's a fair observation but there are important differences. > > In (A), the majority of the code exists in the dependency (sagelib). However > the tests for this huge body of code, exist in the dependant. This causes the > huge write-run-fix loop. And this is not really what a dependency system is > "supposed" to be used for, we don't have similar situations like this > anywhere else. > > In (B), the sage-types library is supposed to be a minimal library with a > stable API that doesn't change too often. The write-run-fix loop is greater, > but it doesn't get invoked nearly as often. This would automatically be true > just because the code base is smaller, but you could also put some extra > effort into making it very very stable. > > Furthermore, this pattern is very common. If we have a huge project and it > develops a utility that is useful to other smaller projects, it often gets > split out. For example we have the NSPR library from Mozilla which Firefox > depends on, GObject from GNOME, etc. > > Smaller projects that want to play with Sage integers, might not want to > depend on *all* of sagelib - it slows down testing. Possibly some of these > people would have sage installed already - but some of them might not want to > do this, or not want to install a whole development version of sage. I agree--thank you for your insight on this. This has been discussed in various ways before. I know it was discussed before I ever came on to the project, but a recent(-ish) discussion was here: https://groups.google.com/d/msg/sage-devel/HAHulLZkKzw/Hc4aU_GDAgAJ The fact remains that there is a core sage library that implements all the essential types (i.e. sage.rings) and several other important core packages as well. The core package is still quite large on its own, but *can* be pared down quite a bit from the current "sage" package. There is rough, but very useful overview of the dependencies between sage's sub-packages here: https://groups.google.com/d/msg/sage-devel/29ndCD8z94k/pjFTq-XlAQAJ Almost all of these packages could be split off as separate, optional packages: * games * tensor * matroids * knots * interacts * crypto * logic * lfunctions * dynamics * sat * coding * game_theory * tests * sandpiles * media * manifolds I'm not saying they all necessarily *should* be split off into separate packages--there are good arguments for and against--but there's not much technical barrier to it, and it would make building, testing, and distributing the core of sage a bit easier (still not trivial, but easier). With more work on the "level 2" dependencies to untangle some of the circular dependencies, and to make more features optional (via runtime checks for dependencies of those features) we could make even more progress on modularization. There are good reasons Sage was developed in the monolithic way it was done in the first place. But I don't think those reasons hold up well as long-term solutions, and could be replaced with better, more careful engineering and development practices. Best, Erik -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] 7.4 release: please don't have fpylll build-depend on Sage
Thanks a lot! 'Martin R. Albrecht' via sage-devel: > Hi all, > > this is now https://trac.sagemath.org/ticket/21728 > > Cheers, > Martin > -- GPG: ed25519/56034877E1F87C35 GPG: rsa4096/1318EFAC5FBBDBCE https://github.com/infinity0/pubkeys.git -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] 7.4 release: please don't have fpylll build-depend on Sage
Jeroen Demeyer: > On 2016-10-18 17:52, Ximin Luo wrote: >> (2) In the long run, one can think about splitting out sage.rings.integers >> (and related things) into a small library "sage-types" or something like >> that. Then sagelib and fpylll can depend on this, and there would be no >> circular dependency, even when Debian or other distros try to build it. > > I do not understand how that would help you. I am proposing a solution > > (A) Split the Sage package in 2 packages (Sage-the-library + > Sage-the-distribution) > > and you have various reasons why this doesn't work. Instead you propose > > (B) Split the Sage package in 2 packages (sage.rings.integers (and related > things) + the rest) > > To me, (A) and (B) look very similar. Why would (B) work while (A) does not? > In other words, why are the problems that you mention for (A) not applicable > to (B)? > It's a fair observation but there are important differences. In (A), the majority of the code exists in the dependency (sagelib). However the tests for this huge body of code, exist in the dependant. This causes the huge write-run-fix loop. And this is not really what a dependency system is "supposed" to be used for, we don't have similar situations like this anywhere else. In (B), the sage-types library is supposed to be a minimal library with a stable API that doesn't change too often. The write-run-fix loop is greater, but it doesn't get invoked nearly as often. This would automatically be true just because the code base is smaller, but you could also put some extra effort into making it very very stable. Furthermore, this pattern is very common. If we have a huge project and it develops a utility that is useful to other smaller projects, it often gets split out. For example we have the NSPR library from Mozilla which Firefox depends on, GObject from GNOME, etc. Smaller projects that want to play with Sage integers, might not want to depend on *all* of sagelib - it slows down testing. Possibly some of these people would have sage installed already - but some of them might not want to do this, or not want to install a whole development version of sage. X -- GPG: ed25519/56034877E1F87C35 GPG: rsa4096/1318EFAC5FBBDBCE https://github.com/infinity0/pubkeys.git -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] 7.4 release: please don't have fpylll build-depend on Sage
Very much appreciated Martin! Since this ticket doesn’t touch sagelib Gentoo, Debian and other distributions will be able to use fpylll-0.23 directly in sage-7.4. I’ll issue a revision of the sage-7.4 ebuild tomorrow and review the ticket if no one has done it by then. François > On 19/10/2016, at 23:02, 'Martin R. Albrecht' via sage-devel > wrote: > > Hi all, > > this is now https://trac.sagemath.org/ticket/21728 > > Cheers, > Martin > > Martin R. Albrecht writes: >> Hi there, >> >> Ximin Luo writes: >>> We can do "pre-install tests" with Sage 7.3, by doing a "dummy >>> install" using Sage's Makefiles, running the tests here, then >>> installing them to the "real location". (This requires some patching, >>> but we have achieved this already and it works.) However with Sage 7.4 >>> this is not possible, due to the fpylll situation. Packages in Debian >>> (and most other buildsystems) are built and tested as distinct units, >>> we can't build A, install A, build B, install B, then test A. >> >> I see the problem here. >> >>> So, a much better alternative of resolving the fpylll issue would be >>> to not have fpylll build-depend on Sage. >> >>> (1) I can see that it's possible to build fpylll without Sage, it will >>> just have a different API. Could we patch Sage-the-library to use >>> fpylll-without-Sage, then have Sage itself convert the non-Sage >>> integers into Sage integers? >> >> […] >> >>> So, what are the problems with (1)? If there are no problems, could we >>> patch this *before* Sage 7.4 is released? I would be happy to write >>> the patch myself, but guidance on where to start would also be much >>> appreciated. >> >> It’d be easy to make that work as far as tests are concerned, we’d lose >> convenience, though: none of the fpylll functions taking integer >> arguments would work out of the box. >> >> Alternatively, we could do the conversion on the Python level instead of >> C/C++/Cython. This way, it could be resolved at runtime. There’d be some >> overhead, but the Integer conversion functions aren’t really used all >> that much. >> >> I’ll give that a try. >> >> Cheers, >> Martin > > > -- > > _pgp: https://keybase.io/martinralbrecht > _www: https://martinralbrecht.wordpress.com > _jab: martinralbre...@jabber.ccc.de > _otr: 47F43D1A 5D68C36F 468BAEBA 640E8856 D7951CCF > > -- > You received this message because you are subscribed to the Google Groups > "sage-devel" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to sage-devel+unsubscr...@googlegroups.com. > To post to this group, send email to sage-devel@googlegroups.com. > Visit this group at https://groups.google.com/group/sage-devel. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] 7.4 release: please don't have fpylll build-depend on Sage
Hi all, this is now https://trac.sagemath.org/ticket/21728 Cheers, Martin Martin R. Albrecht writes: > Hi there, > > Ximin Luo writes: >> We can do "pre-install tests" with Sage 7.3, by doing a "dummy >> install" using Sage's Makefiles, running the tests here, then >> installing them to the "real location". (This requires some patching, >> but we have achieved this already and it works.) However with Sage 7.4 >> this is not possible, due to the fpylll situation. Packages in Debian >> (and most other buildsystems) are built and tested as distinct units, >> we can't build A, install A, build B, install B, then test A. > > I see the problem here. > >> So, a much better alternative of resolving the fpylll issue would be >> to not have fpylll build-depend on Sage. > >> (1) I can see that it's possible to build fpylll without Sage, it will >> just have a different API. Could we patch Sage-the-library to use >> fpylll-without-Sage, then have Sage itself convert the non-Sage >> integers into Sage integers? > > […] > >> So, what are the problems with (1)? If there are no problems, could we >> patch this *before* Sage 7.4 is released? I would be happy to write >> the patch myself, but guidance on where to start would also be much >> appreciated. > > It’d be easy to make that work as far as tests are concerned, we’d lose > convenience, though: none of the fpylll functions taking integer > arguments would work out of the box. > > Alternatively, we could do the conversion on the Python level instead of > C/C++/Cython. This way, it could be resolved at runtime. There’d be some > overhead, but the Integer conversion functions aren’t really used all > that much. > > I’ll give that a try. > > Cheers, > Martin -- _pgp: https://keybase.io/martinralbrecht _www: https://martinralbrecht.wordpress.com _jab: martinralbre...@jabber.ccc.de _otr: 47F43D1A 5D68C36F 468BAEBA 640E8856 D7951CCF -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] 7.4 release: please don't have fpylll build-depend on Sage
On 2016-10-18 17:52, Ximin Luo wrote: (2) In the long run, one can think about splitting out sage.rings.integers (and related things) into a small library "sage-types" or something like that. Then sagelib and fpylll can depend on this, and there would be no circular dependency, even when Debian or other distros try to build it. I do not understand how that would help you. I am proposing a solution (A) Split the Sage package in 2 packages (Sage-the-library + Sage-the-distribution) and you have various reasons why this doesn't work. Instead you propose (B) Split the Sage package in 2 packages (sage.rings.integers (and related things) + the rest) To me, (A) and (B) look very similar. Why would (B) work while (A) does not? In other words, why are the problems that you mention for (A) not applicable to (B)? -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] 7.4 release: please don't have fpylll build-depend on Sage
Hi there, Ximin Luo writes: > We can do "pre-install tests" with Sage 7.3, by doing a "dummy > install" using Sage's Makefiles, running the tests here, then > installing them to the "real location". (This requires some patching, > but we have achieved this already and it works.) However with Sage 7.4 > this is not possible, due to the fpylll situation. Packages in Debian > (and most other buildsystems) are built and tested as distinct units, > we can't build A, install A, build B, install B, then test A. I see the problem here. > So, a much better alternative of resolving the fpylll issue would be > to not have fpylll build-depend on Sage. > (1) I can see that it's possible to build fpylll without Sage, it will > just have a different API. Could we patch Sage-the-library to use > fpylll-without-Sage, then have Sage itself convert the non-Sage > integers into Sage integers? […] > So, what are the problems with (1)? If there are no problems, could we > patch this *before* Sage 7.4 is released? I would be happy to write > the patch myself, but guidance on where to start would also be much > appreciated. It’d be easy to make that work as far as tests are concerned, we’d lose convenience, though: none of the fpylll functions taking integer arguments would work out of the box. Alternatively, we could do the conversion on the Python level instead of C/C++/Cython. This way, it could be resolved at runtime. There’d be some overhead, but the Integer conversion functions aren’t really used all that much. I’ll give that a try. Cheers, Martin -- _pgp: https://keybase.io/martinralbrecht _www: https://martinralbrecht.wordpress.com _jab: martinralbre...@jabber.ccc.de _otr: 47F43D1A 5D68C36F 468BAEBA 640E8856 D7951CCF -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] 7.4 release: please don't have fpylll build-depend on Sage
Jeroen Demeyer: > On 2016-10-18 17:52, Ximin Luo wrote: >> One straw-man way to resolve this is to move the tests into a separate >> Debian package "sagemath-distribution". > > I still think that this is the real solution, also because it mimics what > Sage does: within the Sage-the-distribution build system, Sage-the-library > (sagelib) is just one of the many packages. > >> However, this makes the workflow very awkward for us, especially when it >> comes to distributing binary packages. Debian has automated build systems >> that build things on architectures that the developer doesn't have access >> to. We would have to build sage-library, upload it, wait for the automated >> systems to build it and distribute it, then if this is succesful on all >> architectures, only then can we upload sagemath-distribution to run the >> actual run-time tests. If any of these fail, we have to start the whole loop >> again. > > This really sounds like a Debian-specific problem. I don't understand why > Debian makes it so difficult to test a package on the buildbots. > >> if you guys ever convert sagelib into an spkg and have Sage-the-distribution >> download this as an external dependency (I see that #21507 goes in this >> direction), you will experience this pain yourselves. My guess is you will >> then very likely add an exception into Sage-the-distribution to use a local >> development copy of sagelib, to reduce the write-run-fix loop time. > > I don't think that Sage-the-distribution will ever treat sagelib exactly like > it does other packages. Even if it is a separate package on PyPI, I expect > development to remain essentially the same as today. > Do you see the similarity between the Debian scenario and this future potential Sage scenario? That is why I mentioned those two together. You acknowledge that in the Sage scenario, Sage-the-distribution would have to treat sagelib specially, and not exactly like the other spkgs. But no other buildsystem/distribution has the equivalent concept - Debian does not have "special" packages where the buildbots say "OK we'll let you do [this workflow with sagelib<->fpylll]"; pip doesn't have this; etc etc. Nor do they have "whole-distribution" tests - tests are per-package, and are run inside the "build" of that package. This is a fundamental difference in approach to packaging/distribution. I think it's not possible (or reasonable) to expect one approach to fit inside the other - but we can work out a shared subset that we can both agree to. This shared subset would be to avoid creating dependency paths like the current fpylll+sagelib situation. It is working well for Debian and SageMath 7.3 so far. X -- GPG: ed25519/56034877E1F87C35 GPG: rsa4096/1318EFAC5FBBDBCE https://github.com/infinity0/pubkeys.git -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] 7.4 release: please don't have fpylll build-depend on Sage
On 2016-10-18 17:52, Ximin Luo wrote: One straw-man way to resolve this is to move the tests into a separate Debian package "sagemath-distribution". I still think that this is the real solution, also because it mimics what Sage does: within the Sage-the-distribution build system, Sage-the-library (sagelib) is just one of the many packages. However, this makes the workflow very awkward for us, especially when it comes to distributing binary packages. Debian has automated build systems that build things on architectures that the developer doesn't have access to. We would have to build sage-library, upload it, wait for the automated systems to build it and distribute it, then if this is succesful on all architectures, only then can we upload sagemath-distribution to run the actual run-time tests. If any of these fail, we have to start the whole loop again. This really sounds like a Debian-specific problem. I don't understand why Debian makes it so difficult to test a package on the buildbots. if you guys ever convert sagelib into an spkg and have Sage-the-distribution download this as an external dependency (I see that #21507 goes in this direction), you will experience this pain yourselves. My guess is you will then very likely add an exception into Sage-the-distribution to use a local development copy of sagelib, to reduce the write-run-fix loop time. I don't think that Sage-the-distribution will ever treat sagelib exactly like it does other packages. Even if it is a separate package on PyPI, I expect development to remain essentially the same as today. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] 7.4 release: please don't have fpylll build-depend on Sage
Here is another solution, but it involves upstream fpylll. I'm not sure how realistic it is, because of the auto-generated config.pxi: (3) Ship the Cython-generated files with fpylll. This moves the dependency on Sage from build-time to packaging-time. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.