Re: [sage-devel] 7.4 release: please don't have fpylll build-depend on Sage

[Erik Bray]

On Wed, Oct 19, 2016 at 1:22 PM, Ximin Luo  wrote:
> Jeroen Demeyer:
>> On 2016-10-18 17:52, Ximin Luo wrote:
>>> (2) In the long run, one can think about splitting out sage.rings.integers 
>>> (and related things) into a small library "sage-types" or something like 
>>> that. Then sagelib and fpylll can depend on this, and there would be no 
>>> circular dependency, even when Debian or other distros try to build it.
>>
>> I do not understand how that would help you. I am proposing a solution
>>
>> (A) Split the Sage package in 2 packages (Sage-the-library + 
>> Sage-the-distribution)
>>
>> and you have various reasons why this doesn't work. Instead you propose
>>
>> (B) Split the Sage package in 2 packages (sage.rings.integers (and related 
>> things) + the rest)
>>
>> To me, (A) and (B) look very similar. Why would (B) work while (A) does not? 
>> In other words, why are the problems that you mention for (A) not applicable 
>> to (B)?
>>
>
> It's a fair observation but there are important differences.
>
> In (A), the majority of the code exists in the dependency (sagelib). However 
> the tests for this huge body of code, exist in the dependant. This causes the 
> huge write-run-fix loop. And this is not really what a dependency system is 
> "supposed" to be used for, we don't have similar situations like this 
> anywhere else.
>
> In (B), the sage-types library is supposed to be a minimal library with a 
> stable API that doesn't change too often. The write-run-fix loop is greater, 
> but it doesn't get invoked nearly as often. This would automatically be true 
> just because the code base is smaller, but you could also put some extra 
> effort into making it very very stable.
>
> Furthermore, this pattern is very common. If we have a huge project and it 
> develops a utility that is useful to other smaller projects, it often gets 
> split out. For example we have the NSPR library from Mozilla which Firefox 
> depends on, GObject from GNOME, etc.
>
> Smaller projects that want to play with Sage integers, might not want to 
> depend on *all* of sagelib - it slows down testing. Possibly some of these 
> people would have sage installed already - but some of them might not want to 
> do this, or not want to install a whole development version of sage.

I agree--thank you for your insight on this.  This has been discussed
in various ways before.  I know it was discussed before I ever came on
to the project, but a recent(-ish) discussion was here:

https://groups.google.com/d/msg/sage-devel/HAHulLZkKzw/Hc4aU_GDAgAJ

The fact remains that there is a core sage library that implements all
the essential types (i.e. sage.rings) and several other important core
packages as well.  The core package is still quite large on its own,
but *can* be pared down quite a bit from the current "sage" package.
There is rough, but very useful overview of the dependencies between
sage's sub-packages here:

https://groups.google.com/d/msg/sage-devel/29ndCD8z94k/pjFTq-XlAQAJ

Almost all of these packages could be split off as separate, optional packages:

* games
* tensor
* matroids
* knots
* interacts
* crypto
* logic
* lfunctions
* dynamics
* sat
* coding
* game_theory
* tests
* sandpiles
* media
* manifolds

I'm not saying they all necessarily *should* be split off into
separate packages--there are good arguments for and against--but
there's not much technical barrier to it, and it would make building,
testing, and distributing the core of sage a bit easier (still not
trivial, but easier).

With more work on the "level 2" dependencies to untangle some of the
circular dependencies, and to make more features optional (via runtime
checks for dependencies of those features) we could make even more
progress on modularization.

There are good reasons Sage was developed in the monolithic way it was
done in the first place.  But I don't think those reasons hold up well
as long-term solutions, and could be replaced with better, more
careful engineering and development practices.

Best,
Erik

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

Re: [sage-devel] 7.4 release: please don't have fpylll build-depend on Sage

[Ximin Luo]

Thanks a lot!

'Martin R. Albrecht' via sage-devel:
> Hi all,
> 
> this is now  https://trac.sagemath.org/ticket/21728
> 
> Cheers,
> Martin
> 

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

Re: [sage-devel] 7.4 release: please don't have fpylll build-depend on Sage

[Ximin Luo]

Jeroen Demeyer:
> On 2016-10-18 17:52, Ximin Luo wrote:
>> (2) In the long run, one can think about splitting out sage.rings.integers 
>> (and related things) into a small library "sage-types" or something like 
>> that. Then sagelib and fpylll can depend on this, and there would be no 
>> circular dependency, even when Debian or other distros try to build it.
> 
> I do not understand how that would help you. I am proposing a solution
> 
> (A) Split the Sage package in 2 packages (Sage-the-library + 
> Sage-the-distribution)
> 
> and you have various reasons why this doesn't work. Instead you propose
> 
> (B) Split the Sage package in 2 packages (sage.rings.integers (and related 
> things) + the rest)
> 
> To me, (A) and (B) look very similar. Why would (B) work while (A) does not? 
> In other words, why are the problems that you mention for (A) not applicable 
> to (B)?
> 

It's a fair observation but there are important differences.

In (A), the majority of the code exists in the dependency (sagelib). However 
the tests for this huge body of code, exist in the dependant. This causes the 
huge write-run-fix loop. And this is not really what a dependency system is 
"supposed" to be used for, we don't have similar situations like this anywhere 
else.

In (B), the sage-types library is supposed to be a minimal library with a 
stable API that doesn't change too often. The write-run-fix loop is greater, 
but it doesn't get invoked nearly as often. This would automatically be true 
just because the code base is smaller, but you could also put some extra effort 
into making it very very stable.

Furthermore, this pattern is very common. If we have a huge project and it 
develops a utility that is useful to other smaller projects, it often gets 
split out. For example we have the NSPR library from Mozilla which Firefox 
depends on, GObject from GNOME, etc.

Smaller projects that want to play with Sage integers, might not want to depend 
on *all* of sagelib - it slows down testing. Possibly some of these people 
would have sage installed already - but some of them might not want to do this, 
or not want to install a whole development version of sage.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

Re: [sage-devel] 7.4 release: please don't have fpylll build-depend on Sage

[Francois Bissey]

Very much appreciated Martin! Since this ticket doesn’t touch 
sagelib Gentoo, Debian and other distributions will be able to
use fpylll-0.23 directly in sage-7.4. I’ll issue a revision of
the sage-7.4 ebuild tomorrow and review the ticket if no one
has done it by then.

François

> On 19/10/2016, at 23:02, 'Martin R. Albrecht' via sage-devel 
>  wrote:
> 
> Hi all,
> 
> this is now  https://trac.sagemath.org/ticket/21728
> 
> Cheers,
> Martin
> 
> Martin R. Albrecht writes:
>> Hi there,
>> 
>> Ximin Luo writes:
>>> We can do "pre-install tests" with Sage 7.3, by doing a "dummy
>>> install" using Sage's Makefiles, running the tests here, then
>>> installing them to the "real location". (This requires some patching,
>>> but we have achieved this already and it works.) However with Sage 7.4
>>> this is not possible, due to the fpylll situation. Packages in Debian
>>> (and most other buildsystems) are built and tested as distinct units,
>>> we can't build A, install A, build B, install B, then test A.
>> 
>> I see the problem here.
>> 
>>> So, a much better alternative of resolving the fpylll issue would be
>>> to not have fpylll build-depend on Sage.
>> 
>>> (1) I can see that it's possible to build fpylll without Sage, it will
>>> just have a different API. Could we patch Sage-the-library to use
>>> fpylll-without-Sage, then have Sage itself convert the non-Sage
>>> integers into Sage integers?
>> 
>> […]
>> 
>>> So, what are the problems with (1)? If there are no problems, could we
>>> patch this *before* Sage 7.4 is released? I would be happy to write
>>> the patch myself, but guidance on where to start would also be much
>>> appreciated.
>> 
>> It’d be easy to make that work as far as tests are concerned, we’d lose
>> convenience, though: none of the fpylll functions taking integer
>> arguments would work out of the box.
>> 
>> Alternatively, we could do the conversion on the Python level instead of
>> C/C++/Cython. This way, it could be resolved at runtime. There’d be some
>> overhead, but the Integer conversion functions aren’t really used all
>> that much.
>> 
>> I’ll give that a try.
>> 
>> Cheers,
>> Martin
> 
> 
> -- 
> 
> _pgp: https://keybase.io/martinralbrecht
> _www: https://martinralbrecht.wordpress.com
> _jab: martinralbre...@jabber.ccc.de
> _otr: 47F43D1A 5D68C36F 468BAEBA 640E8856 D7951CCF
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sage-devel+unsubscr...@googlegroups.com.
> To post to this group, send email to sage-devel@googlegroups.com.
> Visit this group at https://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

Re: [sage-devel] 7.4 release: please don't have fpylll build-depend on Sage

['Martin R. Albrecht' via sage-devel]

Hi all,

this is now  https://trac.sagemath.org/ticket/21728

Cheers,
Martin

Martin R. Albrecht writes:
> Hi there,
>
> Ximin Luo writes:
>> We can do "pre-install tests" with Sage 7.3, by doing a "dummy
>> install" using Sage's Makefiles, running the tests here, then
>> installing them to the "real location". (This requires some patching,
>> but we have achieved this already and it works.) However with Sage 7.4
>> this is not possible, due to the fpylll situation. Packages in Debian
>> (and most other buildsystems) are built and tested as distinct units,
>> we can't build A, install A, build B, install B, then test A.
>
> I see the problem here.
>
>> So, a much better alternative of resolving the fpylll issue would be
>> to not have fpylll build-depend on Sage.
>
>> (1) I can see that it's possible to build fpylll without Sage, it will
>> just have a different API. Could we patch Sage-the-library to use
>> fpylll-without-Sage, then have Sage itself convert the non-Sage
>> integers into Sage integers?
>
> […]
>
>> So, what are the problems with (1)? If there are no problems, could we
>> patch this *before* Sage 7.4 is released? I would be happy to write
>> the patch myself, but guidance on where to start would also be much
>> appreciated.
>
> It’d be easy to make that work as far as tests are concerned, we’d lose
> convenience, though: none of the fpylll functions taking integer
> arguments would work out of the box.
>
> Alternatively, we could do the conversion on the Python level instead of
> C/C++/Cython. This way, it could be resolved at runtime. There’d be some
> overhead, but the Integer conversion functions aren’t really used all
> that much.
>
> I’ll give that a try.
>
> Cheers,
> Martin


-- 

_pgp: https://keybase.io/martinralbrecht
_www: https://martinralbrecht.wordpress.com
_jab: martinralbre...@jabber.ccc.de
_otr: 47F43D1A 5D68C36F 468BAEBA 640E8856 D7951CCF

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

Re: [sage-devel] 7.4 release: please don't have fpylll build-depend on Sage

[Jeroen Demeyer]

On 2016-10-18 17:52, Ximin Luo wrote:

(2) In the long run, one can think about splitting out sage.rings.integers (and related 
things) into a small library "sage-types" or something like that. Then sagelib 
and fpylll can depend on this, and there would be no circular dependency, even when 
Debian or other distros try to build it.


I do not understand how that would help you. I am proposing a solution

(A) Split the Sage package in 2 packages (Sage-the-library + 
Sage-the-distribution)


and you have various reasons why this doesn't work. Instead you propose

(B) Split the Sage package in 2 packages (sage.rings.integers (and 
related things) + the rest)


To me, (A) and (B) look very similar. Why would (B) work while (A) does 
not? In other words, why are the problems that you mention for (A) not 
applicable to (B)?


--
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

Re: [sage-devel] 7.4 release: please don't have fpylll build-depend on Sage

['Martin R. Albrecht' via sage-devel]

Hi there,

Ximin Luo writes:
> We can do "pre-install tests" with Sage 7.3, by doing a "dummy
> install" using Sage's Makefiles, running the tests here, then
> installing them to the "real location". (This requires some patching,
> but we have achieved this already and it works.) However with Sage 7.4
> this is not possible, due to the fpylll situation. Packages in Debian
> (and most other buildsystems) are built and tested as distinct units,
> we can't build A, install A, build B, install B, then test A.

I see the problem here.

> So, a much better alternative of resolving the fpylll issue would be
> to not have fpylll build-depend on Sage.

> (1) I can see that it's possible to build fpylll without Sage, it will
> just have a different API. Could we patch Sage-the-library to use
> fpylll-without-Sage, then have Sage itself convert the non-Sage
> integers into Sage integers?

[…]

> So, what are the problems with (1)? If there are no problems, could we
> patch this *before* Sage 7.4 is released? I would be happy to write
> the patch myself, but guidance on where to start would also be much
> appreciated.

It’d be easy to make that work as far as tests are concerned, we’d lose
convenience, though: none of the fpylll functions taking integer
arguments would work out of the box.

Alternatively, we could do the conversion on the Python level instead of
C/C++/Cython. This way, it could be resolved at runtime. There’d be some
overhead, but the Integer conversion functions aren’t really used all
that much.

I’ll give that a try.

Cheers,
Martin



-- 

_pgp: https://keybase.io/martinralbrecht
_www: https://martinralbrecht.wordpress.com
_jab: martinralbre...@jabber.ccc.de
_otr: 47F43D1A 5D68C36F 468BAEBA 640E8856 D7951CCF

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

Re: [sage-devel] 7.4 release: please don't have fpylll build-depend on Sage

[Ximin Luo]

Jeroen Demeyer:
> On 2016-10-18 17:52, Ximin Luo wrote:
>> One straw-man way to resolve this is to move the tests into a separate 
>> Debian package "sagemath-distribution".
> 
> I still think that this is the real solution, also because it mimics what 
> Sage does: within the Sage-the-distribution build system, Sage-the-library 
> (sagelib) is just one of the many packages.
> 
>> However, this makes the workflow very awkward for us, especially when it 
>> comes to distributing binary packages. Debian has automated build systems 
>> that build things on architectures that the developer doesn't have access 
>> to. We would have to build sage-library, upload it, wait for the automated 
>> systems to build it and distribute it, then if this is succesful on all 
>> architectures, only then can we upload sagemath-distribution to run the 
>> actual run-time tests. If any of these fail, we have to start the whole loop 
>> again.
> 
> This really sounds like a Debian-specific problem. I don't understand why 
> Debian makes it so difficult to test a package on the buildbots.
> 
>> if you guys ever convert sagelib into an spkg and have Sage-the-distribution 
>> download this as an external dependency (I see that #21507 goes in this 
>> direction), you will experience this pain yourselves. My guess is you will 
>> then very likely add an exception into Sage-the-distribution to use a local 
>> development copy of sagelib, to reduce the write-run-fix loop time.
> 
> I don't think that Sage-the-distribution will ever treat sagelib exactly like 
> it does other packages. Even if it is a separate package on PyPI, I expect 
> development to remain essentially the same as today.
> 

Do you see the similarity between the Debian scenario and this future potential 
Sage scenario? That is why I mentioned those two together.

You acknowledge that in the Sage scenario, Sage-the-distribution would have to 
treat sagelib specially, and not exactly like the other spkgs.

But no other buildsystem/distribution has the equivalent concept - Debian does 
not have "special" packages where the buildbots say "OK we'll let you do [this 
workflow with sagelib<->fpylll]"; pip doesn't have this; etc etc. Nor do they 
have "whole-distribution" tests - tests are per-package, and are run inside the 
"build" of that package.

This is a fundamental difference in approach to packaging/distribution. I think 
it's not possible (or reasonable) to expect one approach to fit inside the 
other - but we can work out a shared subset that we can both agree to. This 
shared subset would be to avoid creating dependency paths like the current 
fpylll+sagelib situation. It is working well for Debian and SageMath 7.3 so far.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

Re: [sage-devel] 7.4 release: please don't have fpylll build-depend on Sage

[Jeroen Demeyer]

On 2016-10-18 17:52, Ximin Luo wrote:

One straw-man way to resolve this is to move the tests into a separate Debian package 
"sagemath-distribution".


I still think that this is the real solution, also because it mimics 
what Sage does: within the Sage-the-distribution build system, 
Sage-the-library (sagelib) is just one of the many packages.



However, this makes the workflow very awkward for us, especially when it comes 
to distributing binary packages. Debian has automated build systems that build 
things on architectures that the developer doesn't have access to. We would 
have to build sage-library, upload it, wait for the automated systems to build 
it and distribute it, then if this is succesful on all architectures, only then 
can we upload sagemath-distribution to run the actual run-time tests. If any of 
these fail, we have to start the whole loop again.


This really sounds like a Debian-specific problem. I don't understand 
why Debian makes it so difficult to test a package on the buildbots.



if you guys ever convert sagelib into an spkg and have Sage-the-distribution 
download this as an external dependency (I see that #21507 goes in this 
direction), you will experience this pain yourselves. My guess is you will then 
very likely add an exception into Sage-the-distribution to use a local 
development copy of sagelib, to reduce the write-run-fix loop time.


I don't think that Sage-the-distribution will ever treat sagelib exactly 
like it does other packages. Even if it is a separate package on PyPI, I 
expect development to remain essentially the same as today.


--
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

Re: [sage-devel] 7.4 release: please don't have fpylll build-depend on Sage

[Jeroen Demeyer]

Here is another solution, but it involves upstream fpylll. I'm not sure 
how realistic it is, because of the auto-generated config.pxi:


(3) Ship the Cython-generated files with fpylll. This moves the 
dependency on Sage from build-time to packaging-time.


--
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.