subject:"Re\: \[sage\-devel\] Re\: Urgent and important\: Please vote on disputed PR #36964 \(next step of the modularization project\)"

The GAP project provides a native Windows installer 
.
  
So, evidently, it is possible to build GAP for Windows.  They do not seem 
to provide build instructions for Windows, however.

- Marc
 
On Thursday, April 25, 2024 at 9:17:35 AM UTC-5 Marc Culler wrote:

> Another amusing quote, this time from the sbrk man page on macOS:
>
> > The brk and sbrk functions are historical curiosities left over from
>  > earlier days before the advent of virtual memory management.
>
> That seems to be a paraphrase of the FreeBSD man page, which says:
>
> > The brk() and sbrk() functions are legacy interfaces from before the 
> advent of modern virtual
> > memory management. They are deprecated and not present on the arm64 or 
> riscv architectures.
> > The mmap(2) interface should be used to allocate pages instead.
>
> Given that GAP runs on arm64, I suspect that it doesn't use sbrk in an 
> essential way anymore.  Of course fork is a different story.  But there are 
> surely lots of examples of programs that use fork which have been ported to 
> Windows.  I would guess that emacs is among them.  (It has been ported to 
> Windows - I assume it uses fork on posix systems).
>
> - Marc
> On Thursday, April 25, 2024 at 9:03:30 AM UTC-5 Marc Culler wrote:
>
>> On Thursday, April 25, 2024 at 8:28:48 AM UTC-5 Dima Pasechnik wrote:
>>
>> Essential components of sagelib such as GAP, Singular, don't run on 
>> native Windows 
>>
>>
>> I was amused to find the following statement on the GAP forum 
>>  from 2005:
>>
>>   >  While porting GAP to use native Win32 calls is doable, basically 
>> src/system.c is the only place
>>> that needs lots of changes, it is certainly a nontrivial and 
>> time-consuming task. (and one needs
>>> to be a bit of an expert in programming to do this, IMHO)
>>
>> The author was someone from the Netherlands by the name of *Dima 
>> Pasechnik.  :^)*
>>
>> - Marc  
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/7e83fead-ed81-4536-9a64-0f7b1f391d72n%40googlegroups.com.

Re: [sage-devel] Re: Urgent and important: Please vote on disputed PR #36964 (next step of the modularization project)

Another amusing quote, this time from the sbrk man page on macOS:

> The brk and sbrk functions are historical curiosities left over from
 > earlier days before the advent of virtual memory management.

That seems to be a paraphrase of the FreeBSD man page, which says:

> The brk() and sbrk() functions are legacy interfaces from before the 
advent of modern virtual
> memory management. They are deprecated and not present on the arm64 or 
riscv architectures.
> The mmap(2) interface should be used to allocate pages instead.

Given that GAP runs on arm64, I suspect that it doesn't use sbrk in an 
essential way anymore.  Of course fork is a different story.  But there are 
surely lots of examples of programs that use fork which have been ported to 
Windows.  I would guess that emacs is among them.  (It has been ported to 
Windows - I assume it uses fork on posix systems).

- Marc
On Thursday, April 25, 2024 at 9:03:30 AM UTC-5 Marc Culler wrote:

> On Thursday, April 25, 2024 at 8:28:48 AM UTC-5 Dima Pasechnik wrote:
>
> Essential components of sagelib such as GAP, Singular, don't run on 
> native Windows 
>
>
> I was amused to find the following statement on the GAP forum 
>  from 2005:
>
>   >  While porting GAP to use native Win32 calls is doable, basically 
> src/system.c is the only place
>> that needs lots of changes, it is certainly a nontrivial and 
> time-consuming task. (and one needs
>> to be a bit of an expert in programming to do this, IMHO)
>
> The author was someone from the Netherlands by the name of *Dima 
> Pasechnik.  :^)*
>
> - Marc  
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/0fad0dee-9a92-40f3-854a-412acca934e1n%40googlegroups.com.

Re: [sage-devel] Re: Urgent and important: Please vote on disputed PR #36964 (next step of the modularization project)

On Thursday, April 25, 2024 at 8:28:48 AM UTC-5 Dima Pasechnik wrote:

Essential components of sagelib such as GAP, Singular, don't run on 
native Windows 


I was amused to find the following statement on the GAP forum 
 from 2005:

  >  While porting GAP to use native Win32 calls is doable, basically 
src/system.c is the only place
   > that needs lots of changes, it is certainly a nontrivial and 
time-consuming task. (and one needs
   > to be a bit of an expert in programming to do this, IMHO)

The author was someone from the Netherlands by the name of *Dima 
Pasechnik.  :^)*

- Marc  

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/b0784b4b-ab38-4ff7-b5f7-d9cc47472b95n%40googlegroups.com.

Re: [sage-devel] Re: Urgent and important: Please vote on disputed PR #36964 (next step of the modularization project)

2024-04-25 Thread Dima Pasechnik




On 25 April 2024 14:47:35 BST, Marc Culler  wrote:
>On Thu, Apr 25, 2024 at 8:28 AM Dima Pasechnik  wrote:
>
>> On Thu, Apr 25, 2024 at 1:28 PM Nathan Dunfield 
>> wrote:
>> >
>> > On Wednesday, April 24, 2024 at 10:14:09 PM UTC-5 Matthias Koeppe wrote:
>> >
>> > Yes, native Windows would clearly be a very important target.
>> >
>> >
>> > As a data point, downloads of our stand-alone SnapPy app, which is about
>> as high level pure math as it gets, are 60% higher for Windows than macOS.
>>
>> That's not for native Windows, that's for WSL, right?
>>
>
>Wrong!  SnapPy runs as a native Windows app.
>
>We build Pari with msys2 and mingw on a CI runner.
>
>Essential components of sagelib such as GAP, Singular, don't run on
>> native Windows (on Cygwin, yes, but
>> we know by now, Cygwin is too flaky for Sage to work) and I don't
>> think anyone is keen on
>> doing the hard work to port it.
>>
>
>Msys2 and mingw are not so flaky, and use more or less the same toolchain
>as Cygwin.  There are many ports of gnu libraries to msys2 which can be
>installed with their package manage (pacman).  I am not convinced that GAP,
>Singuler, etc. could not be built with mingw.

E.g. GAP uses sbrk, fork (which are not in msys2/mingw), you'll need to port 
their GC too.
Doable, but very, very far from easy.



>
>- Marc
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/E442AC03-90B9-4167-81F4-AB0E9E3C20AE%40gmail.com.

Re: [sage-devel] Re: Urgent and important: Please vote on disputed PR #36964 (next step of the modularization project)

On Thu, Apr 25, 2024 at 8:28 AM Dima Pasechnik  wrote:

> On Thu, Apr 25, 2024 at 1:28 PM Nathan Dunfield 
> wrote:
> >
> > On Wednesday, April 24, 2024 at 10:14:09 PM UTC-5 Matthias Koeppe wrote:
> >
> > Yes, native Windows would clearly be a very important target.
> >
> >
> > As a data point, downloads of our stand-alone SnapPy app, which is about
> as high level pure math as it gets, are 60% higher for Windows than macOS.
>
> That's not for native Windows, that's for WSL, right?
>

Wrong!  SnapPy runs as a native Windows app.

We build Pari with msys2 and mingw on a CI runner.

Essential components of sagelib such as GAP, Singular, don't run on
> native Windows (on Cygwin, yes, but
> we know by now, Cygwin is too flaky for Sage to work) and I don't
> think anyone is keen on
> doing the hard work to port it.
>

Msys2 and mingw are not so flaky, and use more or less the same toolchain
as Cygwin.  There are many ports of gnu libraries to msys2 which can be
installed with their package manage (pacman).  I am not convinced that GAP,
Singuler, etc. could not be built with mingw.

- Marc

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/CALcZXRFLMGZN4Sd6A_VwXzPRnSg%3DRtsT5Am1gEsJtELS5ez9Vg%40mail.gmail.com.

Re: [sage-devel] Re: Urgent and important: Please vote on disputed PR #36964 (next step of the modularization project)

2024-04-25 Thread 'Martin R' via sage-devel



Another example is large-scale pure math computation on clusters.  Because 
of Sage's size and the nature of distributive file systems, the time to 
startup Sage can be 30 seconds or more, which complicates things if you 
want to do 100,000 calculations that are only 10 seconds each.


I agree that this an extremely important point.

Martin

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/f89899dc-db4b-4d24-b479-df375b12cba0n%40googlegroups.com.

Re: [sage-devel] Re: Urgent and important: Please vote on disputed PR #36964 (next step of the modularization project)

2024-04-25 Thread Dima Pasechnik

On Thu, Apr 25, 2024 at 1:28 PM Nathan Dunfield  wrote:
>
> On Wednesday, April 24, 2024 at 10:14:09 PM UTC-5 Matthias Koeppe wrote:
>
> Yes, native Windows would clearly be a very important target.
>
>
> As a data point, downloads of our stand-alone SnapPy app, which is about as 
> high level pure math as it gets, are 60% higher for Windows than macOS.

That's not for native Windows, that's for WSL, right?

Essential components of sagelib such as GAP, Singular, don't run on
native Windows (on Cygwin, yes, but
we know by now, Cygwin is too flaky for Sage to work) and I don't
think anyone is keen on
doing the hard work to port it.

This puts native  Windows  support into the area of wishful thinking.

 Dima

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/CAAWYfq3P-zGuoT%2B2x96oFBNeDCh%3DEzqmn9WgTGhPmH2akPp%3DcA%40mail.gmail.com.

Re: [sage-devel] Re: Urgent and important: Please vote on disputed PR #36964 (next step of the modularization project)

2024-04-25 Thread Nathan Dunfield

On Thursday, April 25, 2024 at 2:17:31 AM UTC-5 Martin R wrote:

As mentioned upthread, CyPari is one of the few examples of something
that's been modularized out of Sage. While it's small compared to Sage, it
can still do everything Pari can, which is a lot. Marc and I broke out
CyPari so we could use it in SnapPy (https://snappy.computop.org), whose
users are, at a guess, 90% mathematicians, 9% physicists, 1% other. The
most recent version stand-alone (Sage-free) version of SnapPy has been
downloaded 1,200 times on Windows and macOS. That's a lot of
mathematicians who are already installing only part of Sage.

Another example is large-scale pure math computation on clusters. Because
of Sage's size and the nature of distributive file systems, the time to
startup Sage can be 30 seconds or more, which complicates things if you
want to do 100,000 calculations that are only 10 seconds each. I was
recently at a workshop on computational topology, and several researchers
there regarded using Sage in this context as a non-starter, in one case
they were completely changing their approach to avoid using Sage.

Best,

Nathan

--
You received this message because you are subscribed to the Google Groups
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/sage-devel/c429af78-dd69-4ed5-94d4-0c0a45df8114n%40googlegroups.com.

Re: [sage-devel] Re: Urgent and important: Please vote on disputed PR #36964 (next step of the modularization project)

2024-04-25 Thread Nathan Dunfield

On Wednesday, April 24, 2024 at 10:14:09 PM UTC-5 Matthias Koeppe wrote:

Yes, native Windows would clearly be a very important target.


As a data point, downloads of our stand-alone SnapPy app, which is about as 
high level pure math as it gets, are 60% higher for Windows than macOS.
 

In another direction: I have started a port of Sage to pyodide, the 
distribution of Python for WebAssembly (WASM), which makes Python runnable 
directly in the browser. I can already run and test the modularized 
distributions **sagemath-objects**, **sagemath-categories** there.


It would be amazing if a decent portion of Sage could be run in the browser 
this way, e.g. to have the occasional HW assignment that needs Sage without 
the overhead of using something like CoCalc. 

Best,

Nathan
 

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/db5485fe-3da5-44bf-b063-7fa7274de17dn%40googlegroups.com.

Re: [sage-devel] Re: Urgent and important: Please vote on disputed PR #36964 (next step of the modularization project)

2024-04-25 Thread 'Martin R' via sage-devel

On Thursday 25 April 2024 at 05:13:37 UTC+2 Matthias Koeppe wrote:

On Wednesday, April 24, 2024 at 1:07:44 AM UTC-7 Martin R wrote:

You mentioned several times, that discoverability is an important aspect.
Do you have any evidence to support that?

I mentioned "discoverability" in the context of how I have *named* the
distributions.

Sorry that my question was not clear enough. Do you have evidence, that
this naming enhances discoverability, and that this enhanced
discoverability would be worthwhile, since it comes with a cost (as
outlined above)?

Wouldn't people in the python world who need a serious amount of math know
of sage anyway,

What is a "serious" amount of math?

You know it when you see it. What I mean is, roughly, that it certainly
does not make sense to use sage as a package if you need a few graph
algorithms (like shortest paths or some such), because then you'd be better
served with using a specialised library or perhaps copy the code and adapt
it. You might want to use sage as a package, if you want to do serious
enumeration of arbitrary combinatorial objects. But then you will need
gap, symmetrica, nauty and maxima or fricas anyway.

and then, if they cannot rely on all of sage because that is too large,
use, for example, `citation.get_systems` to see whether they can do without
some dependencies?

What do you mean by, "whether they can do without some dependencies"?

That's exactly the point of the modularization:
- To enable people to use parts of Sage without some [actually, most!] of
the dependencies.

The only point I am critical of is the splitting of the math library into
arbitrary pseudo-mathematical parts (i.e., sage-combinat ...
sage-symbolics, as listed above), which has nothing to do with dependencies.

[...] I admit, however, that I cannot really think of any serious use of
any of the functionality of sage outside of math, where people would know
that what they need is, say combinatorics or graph theory.

What do you mean by "outside of math"?

I agree that my terminology is not good. I tried to make a distinction
between research involving math and the - to me unknown - rest. I find it
hard to imagine that any mathematician would bother installing anything
else but all of sage. So, maybe I should have written: "I cannot think of
sage in an environment, where dependencies are show-stopper, ...".

I guess it would be quite insane to install sage or parts of sage to get
an implementation of a particular algorithm in a non-math environment.

Yes, "quite insane" would be a good description of doing this (reusing what
is implemented in Sage) with the status quo, the monolithic Sage.

That's exactly the point.
- From the viewpoint of users: It just makes no sense to use a small part
of Sage: Because of space and time and friction (and limited portability).

However, it *does* make sense to copy small parts of sage. If I need an
implementation of some combinatorial algorithm, chances are extremely good
that I can simply use ?? to go to the code, and copy the relevant part.

- From the viewpoint of contributors of new code that could be reused: It
just makes no sense to trap it in Sage, where it cannot be reused sanely.

I doubt this. Do you have any evidence for that?

Modularization makes it at least "less insane", or perhaps "kind of
acceptable", or sometimes even "reasonable".
Any of these degrees of improvement of "sanity" would already be a win.

I disagree. "less insane" or "kind of acceptable" would be a loss, not a
win, since the kind of modularization discussed in this message comes with
a big cost.

Martin

--
You received this message because you are subscribed to the Google Groups
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/sage-devel/9ac29423-78cd-40ca-9d33-6fc52c404b19n%40googlegroups.com.

Re: [sage-devel] Re: Urgent and important: Please vote on disputed PR #36964 (next step of the modularization project)

2024-04-24 Thread Matthias Koeppe

On Wednesday, April 24, 2024 at 6:48:30 AM UTC-7 Oscar Benjamin wrote:

Is the benefit in this case mainly about reduced disk/network usage?

I could imagine other theoretical benefits like maybe some parts could
be installed natively on Windows or some parts might be easier to
provide binaries for etc.

Oscar, thanks for bringing up these important points.

Yes, the modularization -- namely the ability to separately build and run
and test portions of the Sage library -- also facilitates porting to new
platforms in several ways.
- If nothing else, because it enables an incremental approach to porting: I
don't have to first make the whole thing build before I can run and test a
portion.
- More importantly, it varies very much by the dependency packages how hard
it is to port them to a new platform; and how much *interest* in the
developer community of these packages there is in porting.

Yes, native Windows would clearly be a very important target.

In another direction: I have started a port of Sage to pyodide, the
distribution of Python for WebAssembly (WASM), which makes Python runnable
directly in the browser. I can already run and test the modularized
distributions **sagemath-objects**, **sagemath-categories** there.

See
https://github.com/sagemath/sage/wiki/Sage-10.3-Release-Tour#porting-to-the-pyodide-web-assembly-platform

As I wrote there: "Several upstream projects that Sage uses already have
ongoing Pyodide or WASM/Emscripten efforts: The SymPy Live Shell
and the interactive shell on the
NumPy.org front page use JupyterLite
, running Pyodide in a Web
Worker in the browser. Also PARI/GP runs in the browser
using a WASM/Emscripten
port. Giac also has an Emscripten port
."

--
You received this message because you are subscribed to the Google Groups
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/sage-devel/e0582669-4495-412e-8082-aa8b6649ca49n%40googlegroups.com.

Re: [sage-devel] Re: Urgent and important: Please vote on disputed PR #36964 (next step of the modularization project)

2024-04-24 Thread Matthias Koeppe

On Wednesday, April 24, 2024 at 1:07:44 AM UTC-7 Martin R wrote:

You mentioned several times, that discoverability is an important aspect.
Do you have any evidence to support that?

I mentioned "discoverability" in the context of how I have *named* the
distributions.

Wouldn't people in the python world who need a serious amount of math know
of sage anyway,

What is a "serious" amount of math?

and then, if they cannot rely on all of sage because that is too large,
use, for example, `citation.get_systems` to see whether they can do without
some dependencies?

What do you mean by, "whether they can do without some dependencies"?

That's exactly the point of the modularization:
- To enable people to use parts of Sage without some [actually, most!] of
the dependencies.

What do you mean by "outside of math"?

I guess it would be quite insane to install sage or parts of sage to get
an implementation of a particular algorithm in a non-math environment.

Yes, "quite insane" would be a good description of doing this (reusing what
is implemented in Sage) with the status quo, the monolithic Sage.

That's exactly the point.
- From the viewpoint of users: It just makes no sense to use a small part
of Sage: Because of space and time and friction (and limited portability).
- From the viewpoint of contributors of new code that could be reused: It
just makes no sense to trap it in Sage, where it cannot be reused sanely.

Modularization makes it at least "less insane", or perhaps "kind of
acceptable", or sometimes even "reasonable".
Any of these degrees of improvement of "sanity" would already be a win.

--
You received this message because you are subscribed to the Google Groups
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/sage-devel/d0b57ed6-a00f-40fa-8459-6f2eefd04df6n%40googlegroups.com.

Re: [sage-devel] Re: Urgent and important: Please vote on disputed PR #36964 (next step of the modularization project)

2024-04-24 Thread Nathan Dunfield

On Wednesday, April 24, 2024 at 4:25:41 PM UTC-5 Dima Pasechnik wrote:

On Wed, Apr 24, 2024 at 9:29 PM Nathan Dunfield wrote:
> On a related note, the reason that CyPari2 and CyPari are still separate 
relates to what Marc mentioned earlier about tension between two models of 
installing software: the "Linux distro way" using a system-level package 
manager (where there is typically only one version of any given library on 
the system) and the "Python pip" way (where all needed libraries are 
statically linked into the wheel). CyPari2 follows the first, CyPari the 
second. (This story is further complicated by the fact that, from a user's 
perspective, conda is like "Python pip" in that it is orthogonal to any 
system libraries, but developing packages for conda is akin to building 
them for a Linux distro.) 

There isn't a big problem to set up a GitHub wheel builder for 
CyPari2, so there is not really a sea of difference here in this 
sense.


Dima,

Good point, I had forgotten that Matthias did exactly that when he released 
the most recent CyPari2:

https://github.com/sagemath/cypari2/blob/master/.github/workflows/dist.yml

and the binary wheels for CyPari2 on PyPI are indeed statically linked and 
self-contained.  So CyPari2 is a good example of a Python package the fully 
supports both modes.  (Unlike CyPari, which only supports "Python pip", 
though unlike CyPari2 it does work on Windows.)
 

Also, probably it should be mentioned that linux distro way/pip way 
can be very nicely combined using a python venv. 
So one can use system packages, but add up more packages, and, if 
needed, override system packages with other versions.


Agreed!

Best,

Nathan


-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/c9c1ac2d-2f04-4e36-97f6-dc43ff12a0b0n%40googlegroups.com.

Re: [sage-devel] Re: Urgent and important: Please vote on disputed PR #36964 (next step of the modularization project)

2024-04-24 Thread Dima Pasechnik

On Wed, Apr 24, 2024 at 9:29 PM Nathan Dunfield  wrote:
>
> On Wednesday, April 24, 2024 at 2:26:37 PM UTC-5 Oscar Benjamin wrote:
>
> Thanks Marc. This seems like a good example of a useful part of Sage
> that can be extracted to something much smaller than Sage.
>
> Presumably though a hypothetical person who wants CyPari2 but not all
> of Sage can already just use CyPari so that person is already well
> served.
>
>
> Correct.
>
>
> Is the plan that CyPari2 would effectively replace CyPari? Then the benefit 
> is not needing to maintain CyPari separately from
> sagelib's CyPari2 dependency?
>
>
> No, rather CyPari is an example of the sort of free-standing Python package 
> that could be extracted from Sage.  Modularization would create more of 
> these, in varying sizes and levels of granularity.
>
> Some level of modularization is necessary to address what William Stein 
> described last year as:
>
> The fundamental and massive problem that I think SageMath has is that it is 
> not part of the  Python ecosystem,
> by which I mean that there is no good way to do "pip install sagemath-[foo]", 
> in sufficient generality.
>
> PROBLEM: SageMath is not part of the Python ecosystem.
>
> DEFINITION: A piece of software is part of the Python ecosystem, if you can 
> do "pip install " on
> basically the same platforms as the intersection of where you can install 
> scipy/numpy/matplotlib/pandas,
> and with somewhat comparable resource usage (i.e., installing Sagemath can't 
> use 100x of the time/space of
> the above, as that would be unfair).
>
> On a related note, the reason that CyPari2 and CyPari are still separate 
> relates to what Marc mentioned earlier about tension between two models of 
> installing software: the "Linux distro way" using a system-level package 
> manager (where there is typically only one version of any given library on 
> the system) and the "Python pip" way (where all needed libraries are 
> statically linked into the wheel).  CyPari2 follows the first, CyPari the 
> second.  (This story is further complicated by the fact that, from a user's 
> perspective, conda is like "Python pip" in that it is orthogonal to any 
> system libraries, but developing packages for conda is akin to building them 
> for a Linux distro.)

There isn't a big problem to set up a GitHub wheel builder for
CyPari2, so  there is not really a sea of difference here in this
sense.

Also, probably it should be mentioned that linux distro way/pip way
can be very nicely combined  using a python venv.
So one can use system packages, but add up more packages, and, if
needed, override system packages with other versions.


>
> Of course, most projects in the scientific Python community support both 
> models, but there is technical overhead in doing so, which I believe is the 
> root of some of the current conflict.
>
> Best,
>
> Nathan
>
> --
> You received this message because you are subscribed to the Google Groups 
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sage-devel+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sage-devel/60a60056-f8e9-417e-b03a-1dcfcbc3c6ebn%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/CAAWYfq0oqXa1J4pJM78Mk8pb0YCyKMQfmFyp-NdpXQS6Yy_dRA%40mail.gmail.com.

Re: [sage-devel] Re: Urgent and important: Please vote on disputed PR #36964 (next step of the modularization project)

2024-04-24 Thread Nathan Dunfield

On Wednesday, April 24, 2024 at 2:26:37 PM UTC-5 Oscar Benjamin wrote:

Thanks Marc. This seems like a good example of a useful part of Sage 
that can be extracted to something much smaller than Sage. 

Presumably though a hypothetical person who wants CyPari2 but not all 
of Sage can already just use CyPari so that person is already well 
served. 


Correct.
 

Is the plan that CyPari2 would effectively replace CyPari? Then the benefit 
is not needing to maintain CyPari separately from 
sagelib's CyPari2 dependency?


No, rather CyPari is an example of the sort of free-standing Python package 
that could be extracted from Sage.  Modularization would create more of 
these, in varying sizes and levels of granularity.

Some level of modularization is necessary to address what William Stein 
described last year as:

*The fundamental and massive problem that I think SageMath has is that it 
is not part of the  Python ecosystem,*
*by which I mean that there is no good way to do "pip install 
sagemath-[foo]", in sufficient generality.*

*PROBLEM: SageMath is not part of the Python ecosystem.*

*DEFINITION: A piece of software is part of the Python ecosystem, if you 
can do "pip install " on*
*basically the same platforms as the intersection of where you can install 
scipy/numpy/matplotlib/pandas,*
*and with somewhat comparable resource usage (i.e., installing Sagemath 
can't use 100x of the time/space of*
*the above, as that would be unfair).*

On a related note, the reason that CyPari2 and CyPari are still separate 
relates to what Marc mentioned earlier about tension between two models of 
installing software: the "Linux distro way" using a system-level package 
manager (where there is typically only one version of any given library on 
the system) and the "Python pip" way (where all needed libraries are 
statically linked into the wheel).  CyPari2 follows the first, CyPari the 
second.  (This story is further complicated by the fact that, from a user's 
perspective, conda is like "Python pip" in that it is orthogonal to any 
system libraries, but developing packages for conda is akin to building 
them for a Linux distro.)

Of course, most projects in the scientific Python community support both 
models, but there is technical overhead in doing so, which I believe is the 
root of some of the current conflict.

Best,

Nathan

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/60a60056-f8e9-417e-b03a-1dcfcbc3c6ebn%40googlegroups.com.

Re: [sage-devel] Re: Urgent and important: Please vote on disputed PR #36964 (next step of the modularization project)

2024-04-24 Thread Marc Culler

On Wed, Apr 24, 2024 at 2:26 PM Oscar Benjamin 
wrote:

> Presumably though a hypothetical person who wants CyPari2 but not all of
> Sage can already just use CyPari so that person is already well served.
>

That hypothetical person could also use CyPari2 if they didn't care about
memory leaks and they were not running Windows.

I am wondering how representative the CyPari case is compared to other parts
> of Sage.

That is a good question.  Dima says it is not representative.

I think there are different types of hypothetical people: users and
developers.  Users should just install all of sage.  But one copy of a
software package that size should be enough.  Developers producing smaller
more specialized packages would benefit from being able to embed or require
as dependencies some parts of Sage, without having to turn a modestly sized
package into a monster.

- Marc

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/CALcZXRHtqYAfxyBss_wR7Mzipch9EQfNWuUESip5nCuQ%2BSb3XA%40mail.gmail.com.

Re: [sage-devel] Re: Urgent and important: Please vote on disputed PR #36964 (next step of the modularization project)

2024-04-24 Thread Oscar Benjamin

On Wed, 24 Apr 2024 at 15:37, Marc Culler  wrote:
>
> I think that CyPari ;and CyPari2 provide a relevant example.
>
> Some background ... CyPari is a PyPi package with binary wheels which 
> predates and was the starting point for Sage's cypari2 (hence the 2 in the 
> name).   The basis for CyPari was Sage's pari module.  That module was 
> modified to make it independent from the rest of Sage so that Sage's Pari 
> support could be provided as a component of SnapPy. Binary wheels for CyPari 
> are available for Windows, macOS and manylinux.  The current versions of 
> those wheels are statically linked against Pari 2.15.4
>
> The binary wheel for CyPari weighs in at 7MB.  Sage's CyPari2 is available as 
> a binay wheel for macOS and manylinux and the size is of the same order of 
> magnitude.  When they are unpacked and installed the sizes of these python 
> packages are respectively 19MB and 7.5MB (cypari2 is linked against the 
> dynamic libraries libgmp and libpari, which together are about 12MB and which 
> are external to the python package).  A full Sage installation is about 5 
> gigabytes.  With some significant omissions and compression of the 
> documentation, the macOS app squeezes that down to 3.4GB.
>
> So the size of this one significant self-contained component of sage is about 
> 200 times smaller than the full sage installation and it could be made 
> installable on Windows with some additional work which has already been done 
> for a very closely related package.

Thanks Marc. This seems like a good example of a useful part of Sage
that can be extracted to something much smaller than Sage.

Presumably though a hypothetical person who wants CyPari2 but not all
of Sage can already just use CyPari so that person is already well
served. Is the plan that CyPari2 would effectively replace CyPari?
Then the benefit is not needing to maintain CyPari separately from
sagelib's CyPari2 dependency?

I am wondering how representative the CyPari case is compared to other
parts of Sage. There seems to be some disagreement about what parts of
Sage would go where but not much discussion about what it means in
terms of disk footprint, portability etc i.e. the factors that make
the modularisation useful (maybe I misunderstand what the intended
benefits are).

I know that a full Sage install takes 5GB but I have no real
understanding of what it is that takes up all that space or how that
can break down into usable pieces with a smaller footprint.

At least disk footprint is something that can be quantified though:
what is the disk usage of different virtual environments that contain
some useful collection of parts of Sage?

-- 
Oscar

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/CAHVvXxSuqEAk9pNz1P35e5CTB_-TGjO0sV-CbNgzcZvHwjhyiQ%40mail.gmail.com.

Re: [sage-devel] Re: Urgent and important: Please vote on disputed PR #36964 (next step of the modularization project)

2024-04-24 Thread Dima Pasechnik

On Wed, Apr 24, 2024 at 3:37 PM Marc Culler  wrote:
>
> I think that CyPari ;and CyPari2 provide a relevant example.
>
> Some background ... CyPari is a PyPi package with binary wheels which 
> predates and was the starting point for Sage's cypari2 (hence the 2 in the 
> name).   The basis for CyPari was Sage's pari module.  That module was 
> modified to make it independent from the rest of Sage so that Sage's Pari 
> support could be provided as a component of SnapPy. Binary wheels for CyPari 
> are available for Windows, macOS and manylinux.  The current versions of 
> those wheels are statically linked against Pari 2.15.4
>
> The binary wheel for CyPari weighs in at 7MB.  Sage's CyPari2 is available as 
> a binay wheel for macOS and manylinux and the size is of the same order of 
> magnitude.  When they are unpacked and installed the sizes of these python 
> packages are respectively 19MB and 7.5MB (cypari2 is linked against the 
> dynamic libraries libgmp and libpari, which together are about 12MB and which 
> are external to the python package).  A full Sage installation is about 5 
> gigabytes.  With some significant omissions and compression of the 
> documentation, the macOS app squeezes that down to 3.4GB.
>
> So the size of this one significant self-contained component of sage is about 
> 200 times smaller than the full sage installation and it could be made 
> installable on Windows with some additional work which has already been done 
> for a very closely related package.

cypari2 is indeed self-contained (with libpari and libgmp added to the
count), and provides a Pari-Python interface.
This full installation takes about 35 Mb. But it's not a relevant
example, cause it's a very limited functionality.

This is unlike sagemath-graphs, etc, which are, as I wrote, very far
from being self-contained. Many of them, with all of its dependencies
installed, will be blow up to a good half of the whole Sage, if not
more.
So there are no 1%  savings on bandwidth here, more like 30%-50%, or less.

Dima
>
> - Marc
>
>
> On Wed, Apr 24, 2024 at 8:48 AM Oscar Benjamin  
> wrote:
>>
>> On Tue, 23 Apr 2024 at 15:27, Marc Culler  wrote:
>> >
>> > The projects that will really benefit from modularization will be those 
>> > that provide their own limited mathematical context.  Developers of such 
>> > projects will be able to choose which parts of Sage are relevant to their 
>> > specific context.  Those parts of Sage can be provided for their users 
>> > without having to embed a huge monolithic environment into a relatively 
>> > small project.
>>
>> Is the benefit in this case mainly about reduced disk/network usage?
>>
>> I could imagine other theoretical benefits like maybe some parts could
>> be installed natively on Windows or some parts might be easier to
>> provide binaries for etc.
>>
>> Are there any indicative numbers for what the size would be when
>> installing some useful portion of Sage vs installing the whole of
>> Sage?
>>
>> --
>> Oscar
>>
>> --
>> You received this message because you are subscribed to a topic in the 
>> Google Groups "sage-devel" group.
>> To unsubscribe from this topic, visit 
>> https://groups.google.com/d/topic/sage-devel/mqgtkLr2gXY/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to 
>> sage-devel+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/sage-devel/CAHVvXxQUanPY%3D1svhf7Q8xDFD5BroD9wTLRc1-wmFC3vQhBMRg%40mail.gmail.com.
>
> --
> You received this message because you are subscribed to the Google Groups 
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sage-devel+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sage-devel/CALcZXREFctqVF8Hr1B%2Bmz9WfhV9Dspop5EmWN7Q%2BsYdJLvnh4w%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/CAAWYfq2PGYDXTUP-1uCsSefDTKr45QFNbLQOk9%3Dgsx6EBzF9QQ%40mail.gmail.com.

Re: [sage-devel] Re: Urgent and important: Please vote on disputed PR #36964 (next step of the modularization project)

2024-04-24 Thread Marc Culler

I think that CyPari ;and CyPari2 provide a relevant example.

Some background ... CyPari is a PyPi package with binary wheels which
predates and was the starting point for Sage's cypari2 (hence the 2 in the
name).   The basis for CyPari was Sage's pari module.  That module was
modified to make it independent from the rest of Sage so that Sage's Pari
support could be provided as a component of SnapPy. Binary wheels for
CyPari are available for Windows, macOS and manylinux.  The current
versions of those wheels are statically linked against Pari 2.15.4

The binary wheel for CyPari weighs in at 7MB.  Sage's CyPari2 is available
as a binay wheel for macOS and manylinux and the size is of the same order
of magnitude.  When they are unpacked and installed the sizes of these
python packages are respectively 19MB and 7.5MB (cypari2 is linked against
the dynamic libraries libgmp and libpari, which together are about 12MB and
which are external to the python package).  A full Sage installation is
about 5 gigabytes.  With some significant omissions and compression of the
documentation, the macOS app squeezes that down to 3.4GB.

So the size of this one significant self-contained component of sage is
about 200 times smaller than the full sage installation and it could be
made installable on Windows with some additional work which has already
been done for a very closely related package.

- Marc

On Wed, Apr 24, 2024 at 8:48 AM Oscar Benjamin 
wrote:

> On Tue, 23 Apr 2024 at 15:27, Marc Culler  wrote:
> >
> > The projects that will really benefit from modularization will be those
> that provide their own limited mathematical context.  Developers of such
> projects will be able to choose which parts of Sage are relevant to their
> specific context.  Those parts of Sage can be provided for their users
> without having to embed a huge monolithic environment into a relatively
> small project.
>
> Is the benefit in this case mainly about reduced disk/network usage?
>
> I could imagine other theoretical benefits like maybe some parts could
> be installed natively on Windows or some parts might be easier to
> provide binaries for etc.
>
> Are there any indicative numbers for what the size would be when
> installing some useful portion of Sage vs installing the whole of
> Sage?
>
> --
> Oscar
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "sage-devel" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/sage-devel/mqgtkLr2gXY/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> sage-devel+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/sage-devel/CAHVvXxQUanPY%3D1svhf7Q8xDFD5BroD9wTLRc1-wmFC3vQhBMRg%40mail.gmail.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/CALcZXREFctqVF8Hr1B%2Bmz9WfhV9Dspop5EmWN7Q%2BsYdJLvnh4w%40mail.gmail.com.

Re: [sage-devel] Re: Urgent and important: Please vote on disputed PR #36964 (next step of the modularization project)

2024-04-24 Thread Dima Pasechnik

On Wed, Apr 24, 2024 at 2:08 PM Kwankyu Lee  wrote:
>
> Wouldn't people in the python world who need a serious amount of math know of 
> sage anyway, and then, if they cannot rely on all of sage because that is too 
> large, use, for example, `citation.get_systems` to see whether they can do 
> without some dependencies?
>
>
>  I think they would do `pip install sagemath-graphs` if they need graphs 
> functionality.
"Too large" is relative. What was "too large" few years ago is normal
now, and RAM only gets cheaper, and 32-bit systems
get irrelevant.

And they'd be often quite unhappy, as e.g. basically nothing in
src/sage/graphs/generators/classical_geometries.py
will work without sage.rings or GAP.
Many other parts of sage.graphs use things in sage.combinat, as well
as cliquer, bliss/nauty, etc.

Dima

>
> --
> You received this message because you are subscribed to the Google Groups 
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sage-devel+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sage-devel/8dd2c1ce-7811-4ada-94cc-075d5b6ab93bn%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/CAAWYfq2sRJSntR33W5%3DQOo4rZH_AN0aOkTC%2BoKwAuHY1t6htOQ%40mail.gmail.com.

Re: [sage-devel] Re: Urgent and important: Please vote on disputed PR #36964 (next step of the modularization project)

2024-04-24 Thread Oscar Benjamin

On Tue, 23 Apr 2024 at 15:27, Marc Culler  wrote:
>
> The projects that will really benefit from modularization will be those that 
> provide their own limited mathematical context.  Developers of such projects 
> will be able to choose which parts of Sage are relevant to their specific 
> context.  Those parts of Sage can be provided for their users without having 
> to embed a huge monolithic environment into a relatively small project.

Is the benefit in this case mainly about reduced disk/network usage?

I could imagine other theoretical benefits like maybe some parts could
be installed natively on Windows or some parts might be easier to
provide binaries for etc.

Are there any indicative numbers for what the size would be when
installing some useful portion of Sage vs installing the whole of
Sage?

-- 
Oscar

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/CAHVvXxQUanPY%3D1svhf7Q8xDFD5BroD9wTLRc1-wmFC3vQhBMRg%40mail.gmail.com.

Re: [sage-devel] Re: Urgent and important: Please vote on disputed PR #36964 (next step of the modularization project)

2024-04-24 Thread Kwankyu Lee



Wouldn't people in the python world who need a serious amount of math know 
of sage anyway, and then, if they cannot rely on all of sage because that 
is too large, use, for example, `citation.get_systems` to see whether they 
can do without some dependencies?


 I think they would do `pip install sagemath-graphs` if they need graphs 
functionality.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/8dd2c1ce-7811-4ada-94cc-075d5b6ab93bn%40googlegroups.com.

Re: [sage-devel] Re: Urgent and important: Please vote on disputed PR #36964 (next step of the modularization project)

2024-04-24 Thread 'Martin R' via sage-devel

Dear Matthias!

You mentioned several times, that discoverability is an important aspect.  
Do you have any evidence to support that?

Wouldn't people in the python world who need a serious amount of math know 
of sage anyway, and then, if they cannot rely on all of sage because that 
is too large, use, for example, `citation.get_systems` to see whether they 
can do without some dependencies?

I don't see how splitting the sage math library into pieces helps 
discoverability.  I admit, however, that I cannot really think of any 
serious use of any of the functionality of sage outside of math, where 
people would know that what they need is, say combinatorics or graph 
theory.  I guess it would be quite insane to install sage or parts of sage 
to get an implementation of a particular algorithm in a non-math 
environment.

Martin
On Wednesday 24 April 2024 at 03:58:35 UTC+2 Dima Pasechnik wrote:

>
>
> On 23 April 2024 19:13:44 BST, Matthias Koeppe  
> wrote:
> >On Tuesday, April 23, 2024 at 11:06:12 AM UTC-7 Dima Pasechnik wrote:
> >
> >
> >
> >On 23 April 2024 18:41:34 BST, Matthias Koeppe  
> >wrote: 
> >>*$ git blame src/sage/combinat//designs/block_design.py* 
> >> 
> >>fdbe7f7e3348 (Matthias Koeppe 2023-07-12 10:53:08 -0700 65) 
> >>lazy_import('sage.libs.gap.libgap', 'libgap') 
> >>fdbe7f7e3348 (Matthias Koeppe 2023-07-12 10:53:08 -0700 66) 
> >>lazy_import('sage.matrix.matrix_space', 'MatrixSpace') 
> >>fdbe7f7e3348 (Matthias Koeppe 2023-07-12 10:53:08 -0700 67) 
> >>lazy_import('sage.modules.free_module', 'VectorSpace') 
> >>fdbe7f7e3348 (Matthias Koeppe 2023-07-12 10:53:08 -0700 68) 
> >>lazy_import('sage.rings.finite_rings.finite_field_constructor', 
> >>'FiniteField') 
> >> 
> >>What you see there is the result of work in the modularization project, 
> >>using one of the techniques documented 
> >>in 
> >
> https://doc.sagemath.org/html/en/developer/packaging_sage_library.html#dependencies-and-distribution-packages
>  
> >>("Reducing module-level run-time dependencies:") to reduce dependencies 
> or 
> >>to make dependencies milder. 
> >
> >The problem is that without these imports, lazy or not, this module is 
> >basically useless.
> >
> >
> >That's correct, but that's not a "problem". 
>
> It is a problem, because such a module doesn't do what is says on the 
> label.
> Basically such a module is a semi-random part of sagelib, and it would not 
> be fully functional until one installs a good chunk, probably well over 
> half, of sagelib.
>
> I am not saying it is totally useless, this sort of partition, but I don't 
> see the point of distributing such pieces, as Python wheels, as something 
> useful on their own.
>
> Application builders - who, according to Marc, are the only ones to get 
> benefits from this design - can just as well build the needed part of 
> sagelib from source, they don't need these barely useful wheels.
>
>
>
> >
> >It's part of a deliberate design, explained in detail in my 2023-06 
> >sage-devel posts. 
> >
> >From "*Modularization project: IV. The rules*" 
> >(https://groups.google.com/g/sage-devel/c/ozh-7ZZ848s):
> >*3.* Distribution packages declare build dependencies and runtime 
> >dependencies 
> ><
> https://sagemath-tobias.netlify.app/developer/packaging_sage_library.html#dependencies-and-distribution-packages>
>  
> (on 
> >other Python distribution packages). These two can be entirely unrelated 
> to 
> >each other. In addition to the runtime dependencies, it is possible to 
> >declare extra dependencies – either those necessary for running tests 
> ><
> https://sagemath-tobias.netlify.app/developer/packaging_sage_library.html#doctest-only-dependencies>
>  
> or 
> >for providing advertised 
> ><
> https://sagemath-tobias.netlify.app/developer/packaging_sage_library.html#other-runtime-dependencies
> >
> >extra 
> ><
> https://sagemath-tobias.netlify.app/developer/packaging_sage_library.html#other-runtime-dependencies
> >
> >features 
> ><
> https://sagemath-tobias.netlify.app/developer/packaging_sage_library.html#other-runtime-dependencies>.
>  
>
> >But there is no such thing as an "optional build dependency" or 
> >"conditional compilation". The set of modules that form a distribution 
> >package is static. There is no place in the name (or metadata) of a wheel 
> >package that could indicate different "configurations".
> >
> >*4. *As a result of 3, it is possible to create distributions that 
> contain 
> >some modules that cannot be imported because some of their dependencies 
> are 
> >missing. That's OK; they can become importable simply by the presence of 
> >other distributions, in particular those declared as extra dependencies. 
> >All of this is discovered at the time of importing a module; there is no 
> >ahead-of-time linking step of any sort.
> >
> >From "*Modularization project: V. The blocs*" 
> >(https://groups.google.com/g/sage-devel/c/kiB32zP3xD4):
> >*We should not attempt to define a distribution package for every 
> possible 
>

Re: [sage-devel] Re: Urgent and important: Please vote on disputed PR #36964 (next step of the modularization project)




On 23 April 2024 19:13:44 BST, Matthias Koeppe  wrote:
>On Tuesday, April 23, 2024 at 11:06:12 AM UTC-7 Dima Pasechnik wrote:
>
>
>
>On 23 April 2024 18:41:34 BST, Matthias Koeppe  
>wrote: 
>>*$ git blame src/sage/combinat//designs/block_design.py* 
>> 
>>fdbe7f7e3348 (Matthias Koeppe 2023-07-12 10:53:08 -0700 65) 
>>lazy_import('sage.libs.gap.libgap', 'libgap') 
>>fdbe7f7e3348 (Matthias Koeppe 2023-07-12 10:53:08 -0700 66) 
>>lazy_import('sage.matrix.matrix_space', 'MatrixSpace') 
>>fdbe7f7e3348 (Matthias Koeppe 2023-07-12 10:53:08 -0700 67) 
>>lazy_import('sage.modules.free_module', 'VectorSpace') 
>>fdbe7f7e3348 (Matthias Koeppe 2023-07-12 10:53:08 -0700 68) 
>>lazy_import('sage.rings.finite_rings.finite_field_constructor', 
>>'FiniteField') 
>> 
>>What you see there is the result of work in the modularization project, 
>>using one of the techniques documented 
>>in 
>https://doc.sagemath.org/html/en/developer/packaging_sage_library.html#dependencies-and-distribution-packages
> 
>>("Reducing module-level run-time dependencies:") to reduce dependencies or 
>>to make dependencies milder. 
>
>The problem is that without these imports, lazy or not, this module is 
>basically useless.
>
>
>That's correct, but that's not a "problem". 

It is a problem, because such a module doesn't do what is says on the label.
Basically such a module is a semi-random part of sagelib, and it would not be 
fully functional until one installs a good chunk, probably well over half, of 
sagelib.

I am not saying it is totally useless, this sort of partition, but I don't see 
the point of distributing such  pieces, as Python wheels, as something useful 
on their own.

Application builders - who, according to Marc, are the only ones to get 
benefits from this design - can just as well build the needed part of sagelib 
from source, they don't need these barely useful wheels.



>
>It's part of a deliberate design, explained in detail in my 2023-06 
>sage-devel posts. 
>
>From "*Modularization project: IV. The rules*" 
>(https://groups.google.com/g/sage-devel/c/ozh-7ZZ848s):
>*3.* Distribution packages declare build dependencies and runtime 
>dependencies 
>
> (on 
>other Python distribution packages). These two can be entirely unrelated to 
>each other. In addition to the runtime dependencies, it is possible to 
>declare extra dependencies – either those necessary for running tests 
>
> or 
>for providing advertised  
>
>extra  
>
>features 
>.
> 
>But there is no such thing as an "optional build dependency" or 
>"conditional compilation". The set of modules that form a distribution 
>package is static. There is no place in the name (or metadata) of a wheel 
>package that could indicate different "configurations".
>
>*4. *As a result of 3, it is possible to create distributions that contain 
>some modules that cannot be imported because some of their dependencies are 
>missing. That's OK; they can become importable simply by the presence of 
>other distributions, in particular those declared as extra dependencies. 
>All of this is discovered at the time of importing a module; there is no 
>ahead-of-time linking step of any sort.
>
>From "*Modularization project: V. The blocs*" 
>(https://groups.google.com/g/sage-devel/c/kiB32zP3xD4):
>*We should not attempt to define a distribution package for every possible 
>community or subfield of mathematics that Sage supports.*
>The proposed design instead introduces 3 types of distribution packages: 
>[...]
>Each of the distribution packages can define a list of extras (nicknames 
>for sets of other distribution packages that provide additional advertised 
>features). When using pip to install a distribution, users can use square 
>bracket notation to add (adjoin?) these extras.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/6AAFD6C9-15B8-4ED2-AE2E-C1CFAE675165%40gmail.com.

Re: [sage-devel] Re: Urgent and important: Please vote on disputed PR #36964 (next step of the modularization project)

On Tuesday, April 23, 2024 at 11:06:12 AM UTC-7 Dima Pasechnik wrote:

On 23 April 2024 18:41:34 BST, Matthias Koeppe  
wrote: 
>*$ git blame src/sage/combinat//designs/block_design.py* 
> 
>fdbe7f7e3348 (Matthias Koeppe 2023-07-12 10:53:08 -0700 65) 
>lazy_import('sage.libs.gap.libgap', 'libgap') 
>fdbe7f7e3348 (Matthias Koeppe 2023-07-12 10:53:08 -0700 66) 
>lazy_import('sage.matrix.matrix_space', 'MatrixSpace') 
>fdbe7f7e3348 (Matthias Koeppe 2023-07-12 10:53:08 -0700 67) 
>lazy_import('sage.modules.free_module', 'VectorSpace') 
>fdbe7f7e3348 (Matthias Koeppe 2023-07-12 10:53:08 -0700 68) 
>lazy_import('sage.rings.finite_rings.finite_field_constructor', 
>'FiniteField') 
> 
>What you see there is the result of work in the modularization project, 
>using one of the techniques documented 
>in 
https://doc.sagemath.org/html/en/developer/packaging_sage_library.html#dependencies-and-distribution-packages

>("Reducing module-level run-time dependencies:") to reduce dependencies or 
>to make dependencies milder. 

The problem is that without these imports, lazy or not, this module is 
basically useless.

That's correct, but that's not a "problem". 

It's part of a deliberate design, explained in detail in my 2023-06 
sage-devel posts. 

>From "*Modularization project: IV. The rules*" 
(https://groups.google.com/g/sage-devel/c/ozh-7ZZ848s):
*3.* Distribution packages declare build dependencies and runtime 
dependencies 

 (on 
other Python distribution packages). These two can be entirely unrelated to 
each other. In addition to the runtime dependencies, it is possible to 
declare extra dependencies – either those necessary for running tests 

 or 
for providing advertised  

extra  

features 
.

But there is no such thing as an "optional build dependency" or 
"conditional compilation". The set of modules that form a distribution 
package is static. There is no place in the name (or metadata) of a wheel 
package that could indicate different "configurations".

*4. *As a result of 3, it is possible to create distributions that contain 
some modules that cannot be imported because some of their dependencies are 
missing. That's OK; they can become importable simply by the presence of 
other distributions, in particular those declared as extra dependencies. 
All of this is discovered at the time of importing a module; there is no 
ahead-of-time linking step of any sort.

>From "*Modularization project: V. The blocs*" 
(https://groups.google.com/g/sage-devel/c/kiB32zP3xD4):
*We should not attempt to define a distribution package for every possible 
community or subfield of mathematics that Sage supports.*
The proposed design instead introduces 3 types of distribution packages: 
[...]
Each of the distribution packages can define a list of extras (nicknames 
for sets of other distribution packages that provide additional advertised 
features). When using pip to install a distribution, users can use square 
bracket notation to add (adjoin?) these extras.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/66e08815-08bc-439f-be60-af476b7185a7n%40googlegroups.com.

Re: [sage-devel] Re: Urgent and important: Please vote on disputed PR #36964 (next step of the modularization project)




On 23 April 2024 18:41:34 BST, Matthias Koeppe  wrote:
>On Tuesday, April 23, 2024 at 10:32:22 AM UTC-7 Dima Pasechnik wrote:
>
>in 
>src/sage/combinat//designs/block_design.py 
>
>you can see 
>
>lazy_import('sage.libs.gap.libgap', 'libgap') 
>lazy_import('sage.rings.finite_rings.finite_field_constructor', 
>'FiniteField')
>
>
>*$ git blame src/sage/combinat//designs/block_design.py*
>
>fdbe7f7e3348 (Matthias Koeppe2023-07-12 10:53:08 -0700   65) 
>lazy_import('sage.libs.gap.libgap', 'libgap')
>fdbe7f7e3348 (Matthias Koeppe2023-07-12 10:53:08 -0700   66) 
>lazy_import('sage.matrix.matrix_space', 'MatrixSpace')
>fdbe7f7e3348 (Matthias Koeppe2023-07-12 10:53:08 -0700   67) 
>lazy_import('sage.modules.free_module', 'VectorSpace')
>fdbe7f7e3348 (Matthias Koeppe2023-07-12 10:53:08 -0700   68) 
>lazy_import('sage.rings.finite_rings.finite_field_constructor', 
>'FiniteField')
>
>What you see there is the result of work in the modularization project, 
>using one of the techniques documented 
>in 
>https://doc.sagemath.org/html/en/developer/packaging_sage_library.html#dependencies-and-distribution-packages
> 
>("Reducing module-level run-time dependencies:") to reduce dependencies or 
>to make dependencies milder.


The problem is that without these imports, lazy or not, this module is 
basically useless.


>
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/1EA68E6E-0DC2-4D94-8C6A-39ABF3203436%40gmail.com.

Re: [sage-devel] Re: Urgent and important: Please vote on disputed PR #36964 (next step of the modularization project)

On Tuesday, April 23, 2024 at 10:32:22 AM UTC-7 Dima Pasechnik wrote:

in 
src/sage/combinat//designs/block_design.py 

you can see 

lazy_import('sage.libs.gap.libgap', 'libgap') 
lazy_import('sage.rings.finite_rings.finite_field_constructor', 
'FiniteField')


*$ git blame src/sage/combinat//designs/block_design.py*

fdbe7f7e3348 (Matthias Koeppe2023-07-12 10:53:08 -0700   65) 
lazy_import('sage.libs.gap.libgap', 'libgap')
fdbe7f7e3348 (Matthias Koeppe2023-07-12 10:53:08 -0700   66) 
lazy_import('sage.matrix.matrix_space', 'MatrixSpace')
fdbe7f7e3348 (Matthias Koeppe2023-07-12 10:53:08 -0700   67) 
lazy_import('sage.modules.free_module', 'VectorSpace')
fdbe7f7e3348 (Matthias Koeppe2023-07-12 10:53:08 -0700   68) 
lazy_import('sage.rings.finite_rings.finite_field_constructor', 
'FiniteField')

What you see there is the result of work in the modularization project, 
using one of the techniques documented 
in 
https://doc.sagemath.org/html/en/developer/packaging_sage_library.html#dependencies-and-distribution-packages
 
("Reducing module-level run-time dependencies:") to reduce dependencies or 
to make dependencies milder.


-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/d891ed13-9ecd-40d5-81d0-f196f41a3a76n%40googlegroups.com.

Re: [sage-devel] Re: Urgent and important: Please vote on disputed PR #36964 (next step of the modularization project)

On Tuesday, April 23, 2024 at 8:38:13 AM UTC-7 Martin R wrote:

If I understand correctly, the current proposal does not mind if some 
things don't work or could be replaced without too much effort.  For 
example, Dima might have referred to the fact that 
OrderedPartitions.cardinality uses gap, even though it is in 
sagemath-combinat.

The gap dependency in `designs.database` (which is in sagemath-graphs) and 
`matrices.latin` (which is in sagemath-combinat) might have been overlooked.


Dependencies come in different flavors, or strengths. See again 
https://doc.sagemath.org/html/en/developer/packaging_sage_library.html#dependencies-and-distribution-packages

Here's a tl;dr version:

*Build-time dependencies:* If a Cython module uses cimport to pull in 
anything from .pxd files, these files must be either part of the portion 
shipped as the distribution being built, or the distribution that provides 
these files must be declared as a build-time dependency.

*Module-level runtime dependencies:* Any import statements at the top level 
of a Python or Cython module are executed when the module is imported. 
Hence, the imported modules must be part of the distribution, or provided 
by another distribution – which then must be declared as a run-time 
dependency.

*Other runtime dependencies:* If import statements are used within a 
method, the imported module is loaded the first time that the method is 
called. Hence the module defining the method can still be imported even if 
the module needed by the method is not present. 
It is then a question whether a run-time dependency should be declared. If 
the method needing that import provides core functionality, then probably 
yes. 
But if it only provides what can be considered “optional functionality”, 
then probably not, and in this case it will be up to the user to install 
the distribution enabling this optional functionality. It is possible to 
declare such *optional runtime dependencies* as "extras", which provides a 
way to give a nickname to a distribution that can be installed as an add-on.

*Doctest-only dependencies:* Doctests often use examples constructed using 
functionality provided by other portions of the Sage library. This kind of 
integration testing is one of the strengths of Sage; but it also creates 
extra dependencies. We can deal with them using the same mechanism that we 
use for making doctests conditional on the presence of optional libraries: 
using "# optional - FEATURE" (or "# needs FEATURE") directives in the 
doctests. Adding these directives will allow developers to test the 
distribution separately, without requiring all of Sage to be present.


Looking at your examples:

*$ head -n 4 src/sage/combinat/matrices/latin.py*
# sage_setup: distribution = sagemath-combinat
# sage.doctest: needs sage.combinat sage.groups sage.modules
r"""
Latin Squares

So here you see that the module is shipped by the distribution 
*sagemath-combinat*, but the functionality in this module depends at 
runtime on the features "sage.combinat sage.groups sage.modules".



*$ head -n 3 src/sage/combinat/designs/database.py*# sage_setup: 
distribution = sagemath-graphs
r"""
Database of small combinatorial designs
*$ git grep '# needs' src/sage/combinat/designs/database.py*
src/sage/combinat/designs/database.py:sage: 
is_difference_matrix(M,G,8,1) # 
needs sage.rings.finite_rings
src/sage/combinat/designs/database.py:sage: _ = 
designs.difference_matrix(56,8)   # 
needs sage.rings.finite_rings
src/sage/combinat/designs/database.py:sage: G,M = DM_57_8_1()   
  # needs 
sage.rings.finite_rings sage.schemes
src/sage/combinat/designs/database.py:sage: 
is_difference_matrix(M,G,8,1) # 
needs sage.rings.finite_rings sage.schemes
src/sage/combinat/designs/database.py:sage: _ = 
designs.difference_matrix(57,8)   # 
needs sage.rings.finite_rings sage.schemes
src/sage/combinat/designs/database.py:sage: _ = 
designs.difference_matrix(75,8)   # 
needs sage.rings.finite_rings
src/sage/combinat/designs/database.py:sage: G,M = DM_273_17_1() 
  # needs sage.schemes
src/sage/combinat/designs/database.py:sage: 
is_difference_matrix(M,G,17,1)# 
needs sage.schemes
src/sage/combinat/designs/database.py:sage: _ = 
designs.difference_matrix(273,17) # 
needs sage.schemes
src/sage/combinat/designs/database.py:sage: G,M = DM_993_32_1() 
  # needs sage.schemes
src/sage/combinat/designs/database.py:sage: 
is_difference_matrix(M,G,32,1)

Re: [sage-devel] Re: Urgent and important: Please vote on disputed PR #36964 (next step of the modularization project)

On Tue, Apr 23, 2024 at 3:31 PM Matthias Koeppe
 wrote:
>
> On Tuesday, April 23, 2024 at 6:14:05 AM UTC-7 Dima Pasechnik wrote:
>
> On Sun, Apr 21, 2024 at 10:42 PM Matthias Koeppe  wrote:
>
> Let's just go through the list of distribution packages and their 
> dependencies for concreteness. (All depend on sagemath-categories and thus on 
> the basic arithmetic libraries gmp, mpc, mpfr, gmpy2)
>
>
> these are seemingly incomplete:
>
>
> What makes that seem to you?
>
>
> sagemath-combinat
> - non-Python dependency: symmetrica 
> (https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-combinat/pyproject.toml.m4#L65)
>
> non-python: depends on GAP, givaro, too
>
>
> No. Your source?
in

src/sage/combinat//designs/block_design.py

you can see

lazy_import('sage.libs.gap.libgap', 'libgap')
lazy_import('sage.rings.finite_rings.finite_field_constructor', 'FiniteField')


>
>
>
>
> sagemath-graphs
> - non-Python dependency: boost 
> (https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-graphs/pyproject.toml.m4#L73)
>
>
> non-python:  depends on GAP, givaro, too
>
>
> No. Source?

similar to the above

>
>
> sagemath-modules
> - non-Python dependency: gsl 
> (https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-modules/pyproject.toml.m4#L109)
> - Python build requirement: numpy
>
>
> non-python: linbox, flas_ffpac, too ?
>
>
> No. Source?

same as above, basically

These modules don't function in any meaningful way without
dependencies I listed.

Dima
>
> --
> You received this message because you are subscribed to the Google Groups 
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sage-devel+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sage-devel/f54a1d55-b7c2-47a8-9245-d470a63091fan%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/CAAWYfq2AzRpc-fvxSki%3Dg8P%3D%2B%3DExWqidb%3Dtcm7DrzLmY2W89Qw%40mail.gmail.com.

Re: [sage-devel] Re: Urgent and important: Please vote on disputed PR #36964 (next step of the modularization project)

2024-04-23 Thread 'Martin R' via sage-devel

If I understand correctly, the current proposal does not mind if some 
things don't work or could be replaced without too much effort.  For 
example, Dima might have referred to the fact that 
OrderedPartitions.cardinality uses gap, even though it is in 
sagemath-combinat.

The gap dependency in `designs.database` (which is in sagemath-graphs) and 
`matrices.latin` (which is in sagemath-combinat) might have been overlooked.

Martin

On Tuesday 23 April 2024 at 16:31:36 UTC+2 Matthias Koeppe wrote:

> On Tuesday, April 23, 2024 at 6:14:05 AM UTC-7 Dima Pasechnik wrote:
>
> On Sun, Apr 21, 2024 at 10:42 PM Matthias Koeppe  
> wrote:
>
> Let's just go through the list of distribution packages and their 
> dependencies for concreteness. (All depend on *sagemath-categories* and 
> thus on the basic arithmetic libraries gmp, mpc, mpfr, gmpy2)
>
>
> these are seemingly incomplete:
>
>
> What makes that seem to you?
>  
>
> *sagemath-combinat* 
> - non-Python dependency: *symmetrica* (
> https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-combinat/pyproject.toml.m4#L65
> )
>
> non-python: depends on GAP, givaro, too
>
>
> No. Your source?
>  
>
>  
>
> *sagemath-graphs*
> - non-Python dependency: *boost *(
> https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-graphs/pyproject.toml.m4#L73
> )
>
>
> non-python:  depends on GAP, givaro, too
>
>
> No. Source?
>  
>
> *sagemath-modules*
> - non-Python dependency: *gsl* (
> https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-modules/pyproject.toml.m4#L109
> )
> - Python build requirement: 
> *numpy*
>
>
> non-python: linbox, flas_ffpac, too ?
>
>
> No. Source?
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/cddf6e13-9bdd-40a0-9e20-fe8d9be0a233n%40googlegroups.com.

Re: [sage-devel] Re: Urgent and important: Please vote on disputed PR #36964 (next step of the modularization project)

On Tuesday, April 23, 2024 at 6:14:05 AM UTC-7 Dima Pasechnik wrote:

On Sun, Apr 21, 2024 at 10:42 PM Matthias Koeppe  
wrote:

Let's just go through the list of distribution packages and their 
dependencies for concreteness. (All depend on *sagemath-categories* and 
thus on the basic arithmetic libraries gmp, mpc, mpfr, gmpy2)


these are seemingly incomplete:


What makes that seem to you?
 

*sagemath-combinat* 
- non-Python dependency: *symmetrica* (
https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-combinat/pyproject.toml.m4#L65
)

non-python: depends on GAP, givaro, too


No. Your source?
 

 

*sagemath-graphs*
- non-Python dependency: *boost *(
https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-graphs/pyproject.toml.m4#L73
)


non-python:  depends on GAP, givaro, too


No. Source?
 

*sagemath-modules*
- non-Python dependency: *gsl* (
https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-modules/pyproject.toml.m4#L109
)
- Python build requirement: 
*numpy*


non-python: linbox, flas_ffpac, too ?


No. Source?

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/f54a1d55-b7c2-47a8-9245-d470a63091fan%40googlegroups.com.

Re: [sage-devel] Re: Urgent and important: Please vote on disputed PR #36964 (next step of the modularization project)

On Sun, Apr 21, 2024 at 10:42 PM Matthias Koeppe 
wrote:

> On Sunday, April 21, 2024 at 2:30:15 AM UTC-7 Martin R wrote:
>
> Why would you separate mathematics into packages that have no more
> external dependencies from others, which at the same time may grow internal
> dependencies over time?
>
>
> Let's just go through the list of distribution packages and their
> dependencies for concreteness. (All depend on *sagemath-categories* and
> thus on the basic arithmetic libraries gmp, mpc, mpfr, gmpy2)
>

these are seemingly incomplete:

>
> *sagemath-combinat*
> - non-Python dependency: *symmetrica* (
> https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-combinat/pyproject.toml.m4#L65
> )
>
> non-python: depends on GAP, givaro, too


> *sagemath-graphs*
> - non-Python dependency: *boost *(
> https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-graphs/pyproject.toml.m4#L73
> )
>

non-python:  depends on GAP, givaro, too


>
> *sagemath-modules*
> - non-Python dependency: *gsl* (
> https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-modules/pyproject.toml.m4#L109
> )
> - Python build requirement: *numpy*
>

non-python: linbox, flas_ffpac, too ?


> *sagemath-groups*
> - non-Python dependency (via *sagemath-gap*): *gap* (
> https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-gap/pyproject.toml.m4#L52
> *)*
> - non-Python dependency (via *sagemath-modules*): *gsl* (see above)
>
> *sagemath-polyhedra*
> - non-Python dependency (via *sagemath-glpk*): *glpk*
> - non-Python dependency (via *sagemath-modules*): *gsl* (see above)
> - Python runtime dependency: *pplpy*
>
> *sagemath-schemes*
> - non-Python dependency (via *sagemath-modules*): *gsl* (see above)
> - non-Python dependencies (via *sagemath-singular*, *sagemath-flint*,
> *sagemath-ntl*, *sagemath-pari*): singular, flint, ntl, pari (
> https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-pari/pyproject.toml.m4#L61
> )
> - Python dependency (via *sagemath-singular*, *sagemath-flint*,
> *sagemath-pari*): *cypari2*
> - Python dependency: *scipy*
>
> *sagemath-symbolics*
> - non-Python dependencies: *ecl*, *maxima*, *singular*
> https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-symbolics/pyproject.toml.m4#L89
> - non-Python dependencies (via *sagemath-flint*, *sagemath-ntl*,
> *sagemath-modules*): *flint*, *ntl*, *pari*, *gsl*
> - Python dependencies: *mpmath*, *sympy*, *cypari2, numpy*
>
> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-devel+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/sage-devel/41d11d3f-d5e5-4bf5-9629-2ef17ce4d6b1n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/CAAWYfq2NxJYaqQBfVrZTkUproapZNH7_Ci%3DZjnZVV0vXFCBdqw%40mail.gmail.com.

Re: [sage-devel] Re: Urgent and important: Please vote on disputed PR #36964 (next step of the modularization project)

2024-04-22 Thread Kwankyu Lee



On Sunday, April 21, 2024 at 7:01:04 AM UTC+9 kcrisman wrote:

By choosing to be an exception in the Python world, 
Sage obviously does something quite wrong. 


Can someone who is not Dima or Matthias explain to us how it is possible 
that they both are claiming to represent the normal Python way of doing 
things?  There have been numerous statements by both of them about this, 
which makes it sound like there are two pieces to it (modularization but 
also "de-vendoring"), ...


Sage consists of two pieces: the sage library and the *sage package*s (aka 
spkgs defined in sage_root/build/pkgs). The *sage distribution* is roughly 
the sum of the sage packages. The sage library is a big *python package* 
(sage.rings etc.). The modularization project,  led by Matthias,  is 
splitting the sage library and adding (pip-installable) *distribution 
package*s (defined in sage_root/pkgs). Michael is using the term 
"modularization" to mean several things: (1) using a system package (a *distro 
package *installed in a *linux distribution*, ubuntu for example) instead 
of a sage package (2) using a system python package instead of a sage 
package (3) splitting some portion of the sage library to a separate python 
package, which is installed as a sage package, examples are cypari, 
cysignals, etc. 

Note that *"package*"s and "*distribution*"s appearing in the above 
paragraph in *bold face* all refer to different things. 

Many sage packages (known as "pip package" and "wheel packages" ) just 
install python packages from pypi. Dima wants to reduce the number of such 
sage packages (prominently, jupyter and friends), and seems to call this 
"the normal Python way of doing things". You call this "de-vendoring".

Matthias' "normal Python way of doing things" is to follow the Python 
packaging standard in designing the distribution packages.   

Philosophically, there is no reason that Michael's "modularization", Dima's 
"de-vendoring" , Tobias' "conda support", and Gonzalo's "sagemath distro 
package" conflict with Matthias' modularization. But they do in technical 
details in disputed PRs.







 

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/6d089a77-c011-414b-b24e-9c2cbf72cefbn%40googlegroups.com.

Re: [sage-devel] Re: Urgent and important: Please vote on disputed PR #36964 (next step of the modularization project)

2024-04-21 Thread Michael Orlitzky

On 2024-04-20 15:33:51, Matthias Koeppe wrote:
> Michael, I think you may be using too much jargon to get your point across 
> to the general readership of this list. 
> 
> Let's maybe use this opportunity to make this as concrete as possible and 
> explain it in the most plain terms.
> What entanglement are you concerned about?

For example, we have several tickets that are disputed because they
will use ./bootstrap and data from the sage distribution (build/pkgs)
to generate the pyproject.toml files for the modular components. This
ensures that the components cannot truly be separated, not only from
the other components, but from the mini-distro in build/pkgs that a
large chunk of us hate.

Contrast with some other parts of sage that have been modularized:

 * pplpy
 * memory-allocator
 * cypari
 * cysignals
 * primecountpy

These have been successfully disentangled from both the sage library
and the sage distribution and I don't think anyone has complaints
about them. They've been moved to separate repositories which, while
not strictly necessary, is certainly proof that they are in fact
disentangled. We can typically depend on stable versions of them and
install/test them independently. They do not depend on nonstandard or
bleeding-edge features of the python ecosystem.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/ZiWKZlNHnBxOvBXm%40stitch.

Re: [sage-devel] Re: Urgent and important: Please vote on disputed PR #36964 (next step of the modularization project)

On Saturday, April 20, 2024 at 1:01:33 PM UTC-7 Dima Pasechnik wrote:

On 20 April 2024 19:34:49 BST, Matthias Koeppe  
wrote: 
 SageMath is already pip-installable. 
>That was one of the first deliverables of the modularization project, 
>completed in 2021. 
>See 
https://wiki.sagemath.org/ReleaseTours/sage-9.4#Alternative_installation_methods_using_pip

> 
>It's documented in our README: 
>
https://github.com/sagemath/sage/blob/develop/README.md#alternative-installation-using-pypi

> 
>However, installing Sage in this way is equivalent to installing the full 
>Sage distribution from source in the normal way. 

No, it is not equivalent, far from it. One can read on 

Building sagemath-standard has a large number of system packages as 
prerequisites. See 
https://doc.sagemath.org/html/en/installation/source.html#linux-recommended-installation

for partial lists for various systems. 

That is, many packages are not built. What's being built is not a Sage 
distribution, it is sagelib.

Dima, as you well know, I designed and implemented all of these facilities 
related to the PyPI distribution method of SageMath as part of the 
modularization project. What you quote from 
https://pypi.org/project/sagemath-standard/#description -- I wrote it. It's 
hard to understand what the purpose of your message "correcting" me on 
these things could possibly be.

The distribution *sagemath-standard* does provide the Sage library.

But it *declares a required build-time and runtime dependency 
on https://pypi.org/project/sage-conf/*, which when installed, builds the 
Sage distribution in a subdirectory of ~/.sage.

(Note also that one of the disputed PRs, 
https://github.com/sagemath/sage/pull/37138, seeks to remove this declared 
dependency that implements this mechanism.)

>If you do allow me another example from discrete math: Sage has a lot of 
>very good code in "sage.graphs" that provides functionality that is not 
>available from other Python libraries. 
> 
>But to potential projects that would need this functionality, the 
>proposition to have to depend on the monolithic project SageMath, with 
>hours of compilation time and/or dependencies on system packages that are 
>obviously unrelated to the needed functionality (boost, brial, ecl, eclib, 
>ecm, fflas_ffpack, flint, libgd, gap, giac, givaro, glpk, gmp, gsl, iml, 
>lcalc, libbraiding, libhomfly, libpng, linbox, m4ri, m4rie, mpc, mpfi, 
>mpfr, ntl, pari, rw, singular, symmetrica), is simply a showstopper. 

This is not true, either. First, a lot of these packages that you list are 
available on systems, and thus there is no need to build them. 
(To "but macOS?" I have a reply: "ask Apple to provide you a package 
manager, it's the best OS, right, you pay for it a lot of money, or learn 
to use Homebrew like Mac users usually do...")

Note that I wrote "hours of compilation time and/or dependencies on system 
packages".

So there was nothing to correct here. My point stands. 
What's the purpose of your comment?

Anyway, a normal Python pypi-installable package comes with binary wheels, 
i e. things are pre-built, and it's merely matter of downloading these to 
get a functional package. Few minutes on a fast network, not hours. 
By choosing to be an exception in the Python world, 
Sage obviously does something quite wrong.

Also the purpose of this part of your message is not clear to me, Dima.

As you well know, *preparing binary wheels is another deliverable of the 
modularization project. *What Sage may have been doing wrong --- the 
modularization project has been fixing it.
- For the modularized distributions that are already available, these 
binary wheels have already been up on PyPI for a long time, 
see https://pypi.org/project/sagemath-objects/#files 
and https://pypi.org/project/sagemath-categories/#files
- https://github.com/sagemath/sage/pull/37503 (to be merged) adds wheels 
for the macOS arm64 architecture to what it is built and deployed to PyPI 
on every release
- https://github.com/sagemath/sage/pull/37099 (needs review) and 
https://github.com/sagemath/sage/pull/36525, 
https://github.com/sagemath/sage/pull/36730 (in prep) add wheels for more 
distributions

An overview of this effort of making wheels available: 
Meta-PR https://github.com/sagemath/sage/issues/31251
I repeat my invitation for others to join me in this important effort.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/e2ce22d0-07ae-4ae2-94a0-8bb2beaa8e3en%40googlegroups.com.

Re: [sage-devel] Re: Urgent and important: Please vote on disputed PR #36964 (next step of the modularization project)

On Saturday, April 20, 2024 at 1:01:33 PM UTC-7 Dima Pasechnik wrote:

Anyway, a normal Python pypi-installable package comes with binary wheels, 
i e. things are pre-built, and it's merely matter of downloading these to 
get a functional package. Few minutes on a fast network, not hours. 
By choosing to be an exception in the Python world, 
Sage obviously does something quite wrong.


I'll note that providing binary wheels is actually one of the deliverables 
of the modularization project. 
- See for example https://pypi.org/project/sagemath-categories/#files

The meta-ticket "Distribution as wheels" 
(https://github.com/sagemath/sage/issues/31251) describes the next steps, 
including a PR that is waiting for review.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/d05dfcd8-671a-4ce9-be59-949406e062f0n%40googlegroups.com.

Re: [sage-devel] Re: Urgent and important: Please vote on disputed PR #36964 (next step of the modularization project)

2024-04-21 Thread Marc Culler


On Sunday, April 21, 2024 at 9:42:24 AM UTC-5 Nathan Dunfield wrote:

For the statements in this thread, I don't see any contradictions about the 
definition of the "normal Python way of doing things".  My understanding of 
that term is to post *self-contained* binary wheels to PyPI for all 
supported platforms that install in a minute or two with no compilation (as 
Dima wrote), as well as a source-code only package that serves as a 
rarely-used backup if no suitable binary is available.  (The self-contained 
bit is important; for example, on Linux here are the only external libraries 
 a binary wheel is 
allowed to link to.)


I  hope it will help to acknowledge what might be regarded as the elephant 
in this room.  That elephant consists of two facts:

1) *Users*, as opposed to *developers*, are very unlikely to be either 
willing to or capable of building a complex source-code only package which 
depends on external libraries, because doing that requires finding and 
installing (or building from source), all of the external libraries needed 
by the python package.

2) The use of a *self-contained *binary wheel on linux violates a core 
principle of linux packaging, namely that any library which is a runtime 
dependency of a package should be installed from the appropriate distro 
package.

Note that 2) also means that a pypi package developer must support the full 
range of different versions of these libraries which are provided by 
different linux distributions.  Sage initially used the same principle in 
its design, but avoided the maintenance burden caused by trying to support 
all of these different library versions by providing its own libraries in 
SAGE_LOCAL.  It has since gradually accepted more and more of the burden by 
allowing "system versions" of libraries wherever it seemed feasible.

In the "Python ecosystem" these issues have been worked around in a 
different way in order to make *self-contained* binary wheels a 
possibility, and thereby accommodate *users*.  This is usually done with 
delocate on macOS or with auditwheel on linux.  Those tools embed all of 
the required libraries inside the python package and adjust the rpaths in 
the python extension modules to make them use the embedded libraries 
instead of libraries which might or might not be installed on the host 
system.  In the case of linux this also means compiling the libraries 
against a version of glibc which is older than the version found on any of 
the linux distributions supported by the package.  The job of building 
libraries against old versions of glibc is done by building them on a 
manylinux docker image, which can be done on a CI runner.  (On macOS one 
can specify a minimum target version of macOS, usually 10.12 these days.)

As a typical example (which recently created issues for Sage related to the 
jpeg library) you can look at the binary wheel for Pillow.  If you unzip 
that wheel you will find a subdirectory named pillow.libs which contains: 
libbrotlicommon-3ecfe81c.so.1,  libpng16-58efbb84.so.16.43.0, 
libbrotlidec-ba690955.so.1, libsharpyuv-20f78091.so.0.0.1, 
libfreetype-1f2cdfbb.so.6.20.1, libtiff-4da6744b.so.6.0.2, 
libharfbuzz-8b0b6641.so.0.60840.0,libwebp-850e2bec.so.7.1.8, 
libjpeg-f391b078.so.62.4.0, libwebpdemux-df9b36c7.so.2.0.14, 
liblcms2-e69eef39.so.2.0.16,libwebpmux-9fe05867.so.3.0.13, 
liblzma-13fa198c.so.5.4.5,libXau-154567c4.so.6.0.0, libopenjp2-05423b53.so 
, and libxcb-80c5a837.so.1.1.0.  A linux packager would provide all of 
these by adding as dependencies all of the distro packages needed to 
provide these libraries.  In the "Python ecosystem" these libraries are all 
built on manylinux docker images and then embedded in the binary wheel.

I don't see how Sage will be able to join the Python ecosystem without 
addressing this.

- Marc


-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/e96d4c08-25d7-466b-b9fd-6e76b6b740e5n%40googlegroups.com.

Re: [sage-devel] Re: Urgent and important: Please vote on disputed PR #36964 (next step of the modularization project)

2024-04-21 Thread Nathan Dunfield

On Saturday, April 20, 2024 at 5:01:04 PM UTC-5 kcrisman wrote:

Can someone who is not Dima or Matthias explain to us how it is possible
that they both are claiming to represent the normal Python way of doing
things? There have been numerous statements by both of them about this,
which makes it sound like there are two pieces to it (modularization but
also "de-vendoring"), and I can only assume that it would be possible for
both to occur simultaneously. It would be helpful for this to be
clarified, though, so that one knows precisely what *piece* of their
proposals represent the "normal Python ecosystem.

For the statements in this thread, I don't see any contradictions about the
definition of the "normal Python way of doing things". My understanding of
that term is to post *self-contained* binary wheels to PyPI for all
supported platforms that install in a minute or two with no compilation (as
Dima wrote), as well as a source-code only package that serves as a
rarely-used backup if no suitable binary is available. (The self-contained
bit is important; for example, on Linux here are the only external libraries
a binary wheel is
allowed to link to.)

So far, we only post a source-code only package "sagemath-standard" on
PyPI, and so "pip install sagemath-standard" is basically equivalent to
downloading the tarball and running "configure/make". As Matthias says
"It takes very long. This alone makes it simply not suitable as a
dependency of other pip-installable projects."

In particular, no one is arguing that Sage currently follows the "normal
Python way", though we have made one step in that direction by posting the
source-only package.

Best,

Nathan

--
You received this message because you are subscribed to the Google Groups
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/sage-devel/680fd044-ffb6-4b86-a830-cc76b7f982afn%40googlegroups.com.

Re: [sage-devel] Re: Urgent and important: Please vote on disputed PR #36964 (next step of the modularization project)

On Saturday, April 20, 2024 at 3:23:12 PM UTC-7 Michael Orlitzky wrote:

in its current incarnation, the modularization relies 
heavily on the sage distribution vendoring. Conflict arises because the 
modularization is cited as a blocker whenever someone wants to pare 
down or disentangle some aspect of the sage distribution. It's not the 
modularization per se that we object to. Personally, I would like it to 
succeed, but not at the expense of everything else


Michael, I think you may be using too much jargon to get your point across 
to the general readership of this list. 

Let's maybe use this opportunity to make this as concrete as possible and 
explain it in the most plain terms.
What entanglement are you concerned about?

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/a7582691-f818-449e-beb0-03e702a21854n%40googlegroups.com.

Re: [sage-devel] Re: Urgent and important: Please vote on disputed PR #36964 (next step of the modularization project)