Re: [sage-devel] Re: Modularation doctests

[Matthias Koeppe]

On Thursday, June 15, 2023 at 6:38:39 PM UTC-7 William Stein wrote:

Have you worried at all about doctests/examples and the sage 
preparser/global environment that is assumed in all the doctests?


I don't have plans to make any changes to this. The doctests do assume that 
the global environment is partially populated; 
which is done by files such 
as 
https://github.com/sagemath/sage/blob/develop/src/sage/all__sagemath_categories.py
 
A modularized doctest is run as "./sage -t --environment 
sagemath-categories ..."

The preparser (and Sage repl) is available separately, as part of the 
(already existing) 
package https://doc.sagemath.org/html/en/reference/repl/index.html
 

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/9813a8e7-e2f2-45c5-be39-286f7bc49e0an%40googlegroups.com.

Re: [sage-devel] Re: Modularation doctests

[Matthias Koeppe]

On Thursday, June 15, 2023 at 7:18:25 PM UTC-7 Travis Scrimshaw wrote:

then we should make sure the doc is very clean.


Then I think you'll be happy when you notice that while going through these 
doctests in detail to add # optional, I have been making massive cleanups 
of docstrings.


 

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/7bb0a904-817b-47f1-925a-3b53060d0c3an%40googlegroups.com.

Re: [sage-devel] Re: Modularation doctests

['Travis Scrimshaw' via sage-devel]

On Friday, June 16, 2023 at 10:18:57 AM UTC+9 Matthias Koeppe wrote:

On Thursday, June 15, 2023 at 6:03:41 PM UTC-7 Travis Scrimshaw wrote:

 Perhaps this is the core question (for me): What do you expect most/casual 
users to download and install?

It may be the wrong question because it seems to assume there is one 
population of such users.


This is an important question because it determines a lot about how we 
package and present the software. If we expect users to be technically 
savvy and tacitly understand they can ignore "# module - sage.*" doctests, 
then we can just go ahead and plaster them everywhere. If we expect people 
to peck at their keyboards with two fingers and copy/paste examples, then 
we should make sure the doc is very clean.


The population of current Sage users and developers can continue to install 
the monolithic Sage. 
Or, in the modularized terminology, they install the distribution 
"sagemath-standard".


What is going to be "Sage" then? What is my product to tell people about at 
a SageDays?


But the point why I have been working so intensely on the modularization 
project is to open parts of Sage to new populations of users and developers 
who are at home in the Python world. In particular those who find that they 
cannot use the monolithic Sage for their needs and therefore go looking for 
other projects that do not impose such constraints.

As William said, it could be a magnitude more, but it easily could be 0. 
Yes Sage is a lot to download (although if we cut out the documentation and 
continue to use system packages, I think we cut this down immensely), but 
Sage just exists in its own Python (plus some other stuff) bubble. I have 
some doubts that this will move the needle much at the cost of putting up 
development barriers, or at least increasing the learning curve, for 
non-expert-programmer-mathematicians (I might even put myself in this 
category).
 
Best,
Travis

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/deeb3419-4aae-457a-b65c-377b553fd5a3n%40googlegroups.com.

Re: [sage-devel] Re: Modularation doctests

[William Stein]

On Thu, Jun 15, 2023 at 6:19 PM Matthias Koeppe 
wrote:

> On Thursday, June 15, 2023 at 6:03:41 PM UTC-7 Travis Scrimshaw wrote:
>
> [...] Some of it can be fixed by changing the nomenclature "optional" to
> something else as that suggests something a user should *add* to the
> installation.
>
>
> Fine with me to introduce something like "# module - sage.groups" with
> identical semantics as "# optional - sage.groups".
> I'd suggest to take such discussions to
> https://github.com/sagemath/sage/issues/35750
>
>
> Perhaps this is the core question (for me): What do you expect most/casual
> users to download and install?
>
>
> It may be the wrong question because it seems to assume there is one
> population of such users.
>
> The population of current Sage users and developers can continue to
> install the monolithic Sage.
> Or, in the modularized terminology, they install the distribution
> "sagemath-standard".
>
> But the point why I have been working so intensely on the modularization
> project is to open parts of Sage to new populations of users and developers
> who are at home in the Python world. In particular those who find that they
> cannot use the monolithic Sage for their needs and therefore go looking for
> other projects that do not impose such constraints.
>

That could easily be an order of magnitude more users.

Have you worried at all about doctests/examples and the sage
preparser/global environment that is assumed in all the doctests?




> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-devel+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/sage-devel/80a6ba58-17e8-4a99-950a-3bfb5e0b24fen%40googlegroups.com
> 
> .
>
-- 
-- William Stein

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/CACLE5GA0bVYur7N6uet%2BZ7pbZYZZ_je%3DT_ayV7O6hpaZnvGYeA%40mail.gmail.com.

Re: [sage-devel] Re: Modularation doctests

[Matthias Koeppe]

On Thursday, June 15, 2023 at 6:03:41 PM UTC-7 Travis Scrimshaw wrote:

[...] Some of it can be fixed by changing the nomenclature "optional" to 
something else as that suggests something a user should *add* to the 
installation.


Fine with me to introduce something like "# module - sage.groups" with 
identical semantics as "# optional - sage.groups".
I'd suggest to take such discussions 
to https://github.com/sagemath/sage/issues/35750
 

Perhaps this is the core question (for me): What do you expect most/casual 
users to download and install?


It may be the wrong question because it seems to assume there is one 
population of such users.

The population of current Sage users and developers can continue to install 
the monolithic Sage. 
Or, in the modularized terminology, they install the distribution 
"sagemath-standard".

But the point why I have been working so intensely on the modularization 
project is to open parts of Sage to new populations of users and developers 
who are at home in the Python world. In particular those who find that they 
cannot use the monolithic Sage for their needs and therefore go looking for 
other projects that do not impose such constraints.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/80a6ba58-17e8-4a99-950a-3bfb5e0b24fen%40googlegroups.com.

Re: [sage-devel] Re: Modularation doctests

['Travis Scrimshaw' via sage-devel]

On Thursday, June 15, 2023 at 12:23:55 AM UTC+9 Matthias Koeppe wrote:

On Wednesday, June 14, 2023 at 8:01:29 AM UTC-7 Tobias Diez wrote:

On Wednesday, 14 June 2023 at 05:37:15 UTC+8 Matthias Koeppe wrote:

- Some # optional annotations reduce the barrier for contributors, by 
clearly signaling to developers "it's OK and definitely not your fault if 
you don't understand this doctest". 


[...] As a (new) user, reading "# optional - sage.symbolic" makes me more 
think "wtf. what is this? how do I install this sage.symbolic thing so that 
I can run use this example?!" than "oh, yes, this test depends on another 
part of sage, that I have not yet encountered, so I don't need to try to 
understand this example". 


This is a documentation / user education issue, which is addressed by 
writing documentation and educating the educators first.


I don't think the answer should be "read this long, technical document' to 
understand."  Even if it is a short and easy to understand, how would a 
random casual user who just heard about Sage know to even look for it? It 
seems like you want to make the people who are doing Sage outreach do more 
selling and explaining to people (who probably won't even remember such a 
minor point when they first start playing around with things). Some of it 
can be fixed by changing the nomenclature "optional" to something else as 
that suggests something a user should *add* to the installation.

Perhaps this is the core question (for me): What do you expect most/casual 
users to download and install?

Let me expand on this more. Right now, us developers distinguish between 
sage-the-library and sage-the-distribution internally, but when we go to 
"sell" our product, there is but one Sage (to rule them all). There is no 
outward facing sign that a user needs to care about the technical 
implementation details, but I feel they shouldn't. They can treat Sage as a 
black-box until they have a reason not to (e.g., trying to fix a bug or 
prove some code is correct). I think we need to keep this end-user 
experience in the forefront of our minds.

Best,
Travis

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/8edaf240-b378-437d-9a61-d9f37a7cd9a0n%40googlegroups.com.

Re: [sage-devel] Re: Modularation doctests

[Matthias Koeppe]

On Wednesday, June 14, 2023 at 8:01:29 AM UTC-7 Tobias Diez wrote:

On Wednesday, 14 June 2023 at 05:37:15 UTC+8 Matthias Koeppe wrote:

- Some # optional annotations reduce the barrier for contributors, by 
clearly signaling to developers "it's OK and definitely not your fault if 
you don't understand this doctest". 


[...] As a (new) user, reading "# optional - sage.symbolic" makes me more 
think "wtf. what is this? how do I install this sage.symbolic thing so that 
I can run use this example?!" than "oh, yes, this test depends on another 
part of sage, that I have not yet encountered, so I don't need to try to 
understand this example". 


This is a documentation / user education issue, which is addressed by 
writing documentation and educating the educators first.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/3000da05-8ded-48f4-b346-58220a87fdf6n%40googlegroups.com.

Re: [sage-devel] Re: Modularation doctests

[Matthias Koeppe]

On Wednesday, June 14, 2023 at 7:53:25 AM UTC-7 Tobias Diez wrote:

On Wednesday, 14 June 2023 at 05:03:21 UTC+8 Matthias Koeppe wrote:

More specifically, when users of a modularized distribution make their 
first steps to contributing to it, that's already difficult; and we do not 
want to have to tell them "if you want to make this contribution, you first 
have to become a real SageMath user."


Judging from the current people interested in the modularization, it seems 
more likely that a few senior sage devs will work on modularized 
distribution.


By definition, the new users of modularized distributions aren't reading 
this list yet.
 

But my question was more: what kind of issues/bugs do you try to prevent by 
modularizing the tests as well?


I think I have answered this already; so I'm not sure if I understand the 
question.

1. We test separately to make sure that it works when used separately.
2. Lots of the current tests fail when testing separately.
3. Some of these fail because they illustrate the interaction with far away 
parts of Sage. These ones we mark # optional - sage
4. Others fail because there is an unresolved obstacle to the 
modularization. For these ones we fix the obstacle. 
5. After fixing the obstacle, the presence of the test protects against the 
obstacle creeping back in (modularization regression).


-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/c7945c4e-0a47-4807-96ae-09139002b655n%40googlegroups.com.

Re: [sage-devel] Re: Modularation doctests

[Tobias Diez]

On Wednesday, 14 June 2023 at 05:37:15 UTC+8 Matthias Koeppe wrote:

- Some # optional annotations reduce the barrier for contributors, by 
clearly signaling to developers "it's OK and definitely not your fault if 
you don't understand this doctest". 


To be honest, this sounds like wishful thinking. As a (new) user, reading 
"# optional - sage.symbolic" makes me more think "wtf. what is this? how do 
I install this sage.symbolic thing so that I can run use this example?!" 
then "oh, yes, this test depends on another part of sage, that I have not 
yet encountered, so I don't need to try to understand this example". This 
reading is also more in line with Travis' anecdotal evidence " Furthermore, 
the large amount of optional labels, especially with no actual optional 
packages, is starting to scare some users. I tell them they can ignore it, 
but I feel that is not giving off a good impression. It is even more 
confounding for people who are starting to develop (e.g., the GSoC 
students)."

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/4d511798-99dd-4bf4-9855-5af11f8e316en%40googlegroups.com.

Re: [sage-devel] Re: Modularation doctests

[Tobias Diez]

On Wednesday, 14 June 2023 at 05:03:21 UTC+8 Matthias Koeppe wrote:

More specifically, when users of a modularized distribution make their 
first steps to contributing to it, that's already difficult; and we do not 
want to have to tell them "if you want to make this contribution, you first 
have to become a real SageMath user."


Judging from the current people interested in the modularization, it seems 
more likely that a few senior sage devs will work on modularized 
distribution. 
But my question was more: what kind of issues/bugs do you try to prevent by 
modularizing the tests as well?

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/3860b4ed-1386-4305-b1cd-ad9e2ee46614n%40googlegroups.com.

Re: [sage-devel] Re: Modularation doctests

[Kwankyu Lee]

And as I explained elsewhere, the # optional annotations also improve the 
documentation for the benefit of the project. 


The persistent optional tag that I suggested in 

https://github.com/sagemath/sage/issues/35750#issuecomment-1590606293

would be explicit, visible in documentation, and at the same time less 
cluttering than the current practice. 
 

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/57488841-caaa-4843-b61f-58a59c785cc2n%40googlegroups.com.

Re: [sage-devel] Re: Modularation doctests

[Matthias Koeppe]

On Tuesday, June 13, 2023 at 1:40:42 PM UTC-7 Tobias Diez wrote:

But I think it would be better to reuse tools (or at least their ideas) 
that were developed exactly to manage and restrict dependencies between 
different layers/modules of an application, such as 
https://github.com/zyskarch/pytestarch, 
https://github.com/seddonym/import-linter, 
https://github.com/jwbargsten/pytest-archon, https://github.com/TNG/ArchUnit. 
All of them define external rules about admissible relationships. 


Of course there are tools like this. But I decided to go for a more 
explicit approach, reusing a mechanism (# optional) that is already 
familiar to Sage developers. 

For example, the question "why did this doctest not run" is trivial to 
answer with the explicit annotations; whereas when using stacktrace 
inspection tools, all answers will be technical.

And as I explained elsewhere, the # optional annotations also improve the 
documentation for the benefit of the project. 
- Some of the # optional annotations such as "# optional - 
sage.libs.singular" give specific attribution to the various libraries that 
we use for the actual computations. This has the potential of improving our 
project's relation to upstream developers, counteracting the impression 
that Sage tries to take credit for functionality provided by the upstream 
project. (I will not elaborate on this with examples.)
- Some # optional annotations reduce the barrier for contributors, by 
clearly signaling to developers "it's OK and definitely not your fault if 
you don't understand this doctest". 

Finally, the explicit annotations also enable an incremental 
development/documentation process regarding library dependencies -- not 
just within the modularization project but also very useful when preparing 
to replace the use of one library for a feature with another library. The 
interaction of various libraries is sometimes very complex in Sage, and 
developers can add "# optional" annotations while navigating this maze of 
dependencies.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/f3865714-a407-4855-bd82-832c0b4dc424n%40googlegroups.com.

Re: [sage-devel] Re: Modularation doctests

[Matthias Koeppe]

On Tuesday, June 13, 2023 at 1:40:42 PM UTC-7 Tobias Diez wrote:

On Wednesday, 14 June 2023 at 02:36:16 UTC+8 Matthias Koeppe wrote:

- Modularized distributions must be testable separately!


And why is this a desirable goal?


So that developers of modularized distributions can develop and test them 
locally without being burdened with the complexity of the whole of SageMath.

More specifically, when users of a modularized distribution make their 
first steps to contributing to it, that's already difficult; and we do not 
want to have to tell them "if you want to make this contribution, you first 
have to become a real SageMath user."

Integration tests almost by definition are testing multiple modules/units 
and thus don't fit into this scheme.


That's right, that's exactly the point. In the Sage doctests, tests that 
can be run within the modularized distributions and such integrations tests 
appear right next to each other. And they appear there for good reasons: 
For example, to illustrate to readers the applicability of methods to a 
wide range of mathematical objects.

Hence, the doctest annotations that help tell the story of what a 
particular example is about.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/2d574ecb-7387-4599-b14a-9886b655b5cbn%40googlegroups.com.

Re: [sage-devel] Re: Modularation doctests

[Tobias Diez]

On Wednesday, 14 June 2023 at 02:36:16 UTC+8 Matthias Koeppe wrote:

- Modularized distributions must be testable separately!


And why is this a desirable goal? 
Integration tests almost by definition are testing multiple modules/units 
and thus don't fit into this scheme. 
Admittedly, the only multi-package monorepos I know are in the javascript 
world but there people usually don't bother to modularize their tests for 
exactly this reason (e.g. https://github.com/vuejs/vue/tree/main/test). Do 
you know of examples of projects that follow the "Modularized distributions 
must be testable separately!"-mantra? 
 

My thought was that when you declare a new module, you run the doctests of 
that module. They usually break for two reasons:
a) some real dependency was not declared for that module - in this case, 
add the dependency and run the tests again.
b) some doctests require a library that is not a dependency of the original 
module - in this case, add an optional tag to the failing doctest (current 
solution)

My proposal would be in b) to add it the offending import to a whitelist 
("test dependency of module xyz") and then all doctests for module xyz are 
ignoring import errors for these whitelisted dependencies. So in short, 
instead of annotating each doctest you annotate the module.


After you whitelist an import from *M* because there is some valid doctest 
that uses *M* for integration-testing purposes (for example, by testing 
basic linear algebra functionality by running it on an element of an 
algebra defined in *M*), the modularized distribution is no longer guarded 
by its testsuite against modularization regressions that involve the 
whitelisted module.


Sure, this hides issues from libraries that are declared to be test 
dependencies but that actually are/become actual dependencies. But the same 
problem, on a slightly smaller scale, you have with file-wide "optional" 
annotations - and even with line annotations. One way out would be to 
analyze the stack trace and see if the error is thrown in a pure doctest 
code or while running code in module M. 

But I think it would be better to reuse tools (or at least their ideas) 
that were developed exactly to manage and restrict dependencies between 
different layers/modules of an application, such as 
https://github.com/zyskarch/pytestarch, 
https://github.com/seddonym/import-linter, 
https://github.com/jwbargsten/pytest-archon, 
https://github.com/TNG/ArchUnit. All of them define external rules about 
admissible relationships.  

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/c18fb2bb-2644-49f2-8ea0-ca1c25642d35n%40googlegroups.com.

Re: [sage-devel] Re: Modularation doctests

[Matthias Koeppe]

On Tuesday, June 13, 2023 at 11:12:21 AM UTC-7 Tobias Diez wrote:

Maybe I'm misunderstanding the purpose of the modularized doctests. Can 
someone explain me what they should test and what not?


As it says in 
https://sagemath-tobias.netlify.app/developer/packaging_sage_library.html#testing-distribution-packages:

- Modularized distributions must be testable separately!
- But we want to keep integration testing with other portions of Sage too!
 

My thought was that when you declare a new module, you run the doctests of 
that module. They usually break for two reasons:
a) some real dependency was not declared for that module - in this case, 
add the dependency and run the tests again.
b) some doctests require a library that is not a dependency of the original 
module - in this case, add an optional tag to the failing doctest (current 
solution)

My proposal would be in b) to add it the offending import to a whitelist 
("test dependency of module xyz") and then all doctests for module xyz are 
ignoring import errors for these whitelisted dependencies. So in short, 
instead of annotating each doctest you annotate the module.


After you whitelist an import from *M* because there is some valid doctest 
that uses *M* for integration-testing purposes (for example, by testing 
basic linear algebra functionality by running it on an element of an 
algebra defined in *M*), the modularized distribution is no longer guarded 
by its testsuite against modularization regressions that involve the 
whitelisted module.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/61dcfd5a-1034-4dfb-9fc4-c0443b0d294bn%40googlegroups.com.

Re: [sage-devel] Re: Modularation doctests

[Tobias Diez]

Maybe I'm misunderstanding the purpose of the modularized doctests. Can 
someone explain me what they should test and what not?

My thought was that when you declare a new module, you run the doctests of 
that module. They usually break for two reasons:
a) some real dependency was not declared for that module - in this case, 
add the dependency and run the tests again.
b) some doctests require a library that is not a dependency of the original 
module - in this case, add an optional tag to the failing doctest (current 
solution)

My proposal would be in b) to add it the offending import to a whitelist 
("test dependency of module xyz") and then all doctests for module xyz are 
ignoring import errors for these whitelisted dependencies. So in short, 
instead of annotating each doctest you annotate the module. 

On Tuesday, 13 June 2023 at 23:46:26 UTC+8 Dima Pasechnik wrote:

> On Tue, Jun 13, 2023 at 4:14 PM Tobias Diez  wrote:
> >
> > What about a completely different approach: Instead of annotating the 
> doctests, simply ignore test blocks that fail due to a thrown import error 
> (or missing feature error) of a package that is not installed in the 
> modularized test environment?
>
> How do you see the doctest framework distinguishing "real" import
> errors from the ones caused by modularisation?
>
>
>
> > That might hide some edge cases, but would in total be a less intrusive 
> way to handle things.
> >
> > At the very least, I would propose to hide these optional tags from the 
> generated docs.
> > On Monday, 12 June 2023 at 16:50:11 UTC+8 Dima Pasechnik wrote:
> >>
> >> On Mon, Jun 12, 2023 at 1:35 AM 'Travis Scrimshaw' via sage-devel
> >>  wrote:
> >> >
> >> > Hi Matthias,
> >> >
> >> > Happy to see that you are curious regarding the modularization 
> project, but I don't think it's a good approach to start this discussion 
> with claims that sound authoritative ("nobody will actually maintain", 
> "does not scale", "nearly all end users", etc.) and a policy proposal.
> >> >
> >> >
> >> > Yes, you're right. I do not have any hard analytic data to support 
> what users want and are doing, and that observation is based solely on my 
> years of experience with working with Sage, going to and speaking at 
> SageDays, and convincing people to start using Sage. However, there is 
> clear evidence that the current approach is not scaling by the amount of 
> work that is going in and frequent updates/fixed that is needed to be done. 
> Currently, only you are the one doing this.
> >>
> >> What exactly is not scaling? "Vendor everything" approach abandoned ~9
> >> years ago obviously didn't scale.
> >> I think we're not unvendoring things aggressively enough, and that's
> >> what we quarrel with Matthias about.
> >>
> >> I'm about to start a round of discussions here to lead to a vote to
> >> remove vendored gcc and gfortran.
> >> We also should be getting rid of Sage-nonspecific things such as
> >> vendored Python (with the needed dependencies such as openssl),
> >> and vendored Jupyter and its huge slew of its dependencies (for the
> >> latter Matthias is on board, I think).
> >>
> >> It's also not correct that modularisation is currently only done by
> >> Matthias. E.g. most recently I worked on unvendoring of Maxima.
> >> I worked on doing some abstract classes for parts of sage.coding, I
> >> worked on spinning out prime counting stuff into a separate
> >> pip-installable module (that was in 2021, though), etc.
> >>
> >> The interdependecies of parts of Sage library on each other are
> >> decreasing, and this is certainly a big deal in terms of
> >> updating/fixing
> >> things. (Not importing from sage.all in the library is a big deal).
> >>
> >> Dima
> >>
> >>
> >> Some of that is the lack of discussion (which I would like to have
> >> seen given the large scale nature of this change, which is implicitly
> >> setting policy by default). You can disagree with my conclusions and
> >> proposal, but I want to actually talk about that rather than having
> >> dismissive comments. Can you provide any specific counterpoints and
> >> your expectations? What you posted on the other thread does not
> >> address any of these.
> >> >
> >> > Best,
> >> > Travis
> >> >
> >> > --
> >> > You received this message because you are subscribed to the Google 
> Groups "sage-devel" group.
> >> > To unsubscribe from this group and stop receiving emails from it, 
> send an email to sage-devel+...@googlegroups.com.
> >> > To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sage-devel/84fbc2ed-00c8-4ca6-8aa7-d4c74e8c2455n%40googlegroups.com
> .
> >
> > --
> > You received this message because you are subscribed to the Google 
> Groups "sage-devel" group.
> > To unsubscribe from this group and stop receiving emails from it, send 
> an email to sage-devel+...@googlegroups.com.
> > To view this discussion on the web visit 
> 

Re: [sage-devel] Re: Modularation doctests

[Matthias Koeppe]

On Tuesday, June 13, 2023 at 8:14:54 AM UTC-7 Tobias Diez wrote:

At the very least, I would propose to hide these optional tags from the 
generated docs.


You *did* already propose to hide them, 
in https://github.com/sagemath/sage/pull/35719#issuecomment-1575680194, in 
response to which I explained to you the important purposes that these tags 
serve.


 

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/de56f1cb-a364-4d6e-982e-490eafd7a329n%40googlegroups.com.

Re: [sage-devel] Re: Modularation doctests

[Dima Pasechnik]

On Tue, Jun 13, 2023 at 4:14 PM Tobias Diez  wrote:
>
> What about a completely different approach: Instead of annotating the 
> doctests, simply ignore test blocks that fail due to a thrown import error 
> (or missing feature error) of a package that is not installed in the 
> modularized test environment?

How do you see the doctest framework  distinguishing "real" import
errors from the ones caused by modularisation?



> That might hide some edge cases, but would in total be a less intrusive way 
> to handle things.
>
> At the very least, I would propose to hide these optional tags from the 
> generated docs.
> On Monday, 12 June 2023 at 16:50:11 UTC+8 Dima Pasechnik wrote:
>>
>> On Mon, Jun 12, 2023 at 1:35 AM 'Travis Scrimshaw' via sage-devel
>>  wrote:
>> >
>> > Hi Matthias,
>> >
>> > Happy to see that you are curious regarding the modularization project, 
>> > but I don't think it's a good approach to start this discussion with 
>> > claims that sound authoritative ("nobody will actually maintain", "does 
>> > not scale", "nearly all end users", etc.) and a policy proposal.
>> >
>> >
>> > Yes, you're right. I do not have any hard analytic data to support what 
>> > users want and are doing, and that observation is based solely on my years 
>> > of experience with working with Sage, going to and speaking at SageDays, 
>> > and convincing people to start using Sage. However, there is clear 
>> > evidence that the current approach is not scaling by the amount of work 
>> > that is going in and frequent updates/fixed that is needed to be done. 
>> > Currently, only you are the one doing this.
>>
>> What exactly is not scaling? "Vendor everything" approach abandoned ~9
>> years ago obviously didn't scale.
>> I think we're not unvendoring things aggressively enough, and that's
>> what we quarrel with Matthias about.
>>
>> I'm about to start a round of discussions here to lead to a vote to
>> remove vendored gcc and gfortran.
>> We also should be getting rid of Sage-nonspecific things such as
>> vendored Python (with the needed dependencies such as openssl),
>> and vendored Jupyter and its huge slew of its dependencies (for the
>> latter Matthias is on board, I think).
>>
>> It's also not correct that modularisation is currently only done by
>> Matthias. E.g. most recently I worked on unvendoring of Maxima.
>> I worked on doing some abstract classes for parts of sage.coding, I
>> worked on spinning out prime counting stuff into a separate
>> pip-installable module (that was in 2021, though), etc.
>>
>> The interdependecies of parts of Sage library on each other are
>> decreasing, and this is certainly a big deal in terms of
>> updating/fixing
>> things. (Not importing from sage.all in the library is a big deal).
>>
>> Dima
>>
>>
>> Some of that is the lack of discussion (which I would like to have
>> seen given the large scale nature of this change, which is implicitly
>> setting policy by default). You can disagree with my conclusions and
>> proposal, but I want to actually talk about that rather than having
>> dismissive comments. Can you provide any specific counterpoints and
>> your expectations? What you posted on the other thread does not
>> address any of these.
>> >
>> > Best,
>> > Travis
>> >
>> > --
>> > You received this message because you are subscribed to the Google Groups 
>> > "sage-devel" group.
>> > To unsubscribe from this group and stop receiving emails from it, send an 
>> > email to sage-devel+...@googlegroups.com.
>> > To view this discussion on the web visit 
>> > https://groups.google.com/d/msgid/sage-devel/84fbc2ed-00c8-4ca6-8aa7-d4c74e8c2455n%40googlegroups.com.
>
> --
> You received this message because you are subscribed to the Google Groups 
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sage-devel+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sage-devel/cc125172-2413-4416-b871-f4fdc606a4c7n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/CAAWYfq0AQ7oVC7S%3DPwQfHZas-9jyhv3mqYoH25EmTFNLQEwK6g%40mail.gmail.com.

Re: [sage-devel] Re: Modularation doctests

[Matthias Koeppe]

On Tuesday, June 13, 2023 at 8:14:54 AM UTC-7 Tobias Diez wrote:

What about a completely different approach: Instead of annotating the 
doctests, simply ignore test blocks that fail due to a thrown import error 
(or missing feature error) of a package that is not installed in the 
modularized test environment? That might hide some edge cases, but would in 
total be a less intrusive way to handle things.


No, that does not work. It does not "hide some edge cases"; it would hide 
everything that's relevant. All errors that one sees when modularizing are 
import errors.

Modularization work exactly consists of looking at the details: Is the 
error appearing because of a poorly placed import; or is this doctest 
depending on a feature that by design is not provided as part of the tested 
distribution.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/216e6abb-ecbb-4284-aa60-f53f8a40c506n%40googlegroups.com.

Re: [sage-devel] Re: Modularation doctests

[Tobias Diez]

What about a completely different approach: Instead of annotating the 
doctests, simply ignore test blocks that fail due to a thrown import error 
(or missing feature error) of a package that is not installed in the 
modularized test environment? That might hide some edge cases, but would in 
total be a less intrusive way to handle things.

At the very least, I would propose to hide these optional tags from the 
generated docs.
On Monday, 12 June 2023 at 16:50:11 UTC+8 Dima Pasechnik wrote:

> On Mon, Jun 12, 2023 at 1:35 AM 'Travis Scrimshaw' via sage-devel
>  wrote:
> >
> > Hi Matthias,
> >
> > Happy to see that you are curious regarding the modularization project, 
> but I don't think it's a good approach to start this discussion with claims 
> that sound authoritative ("nobody will actually maintain", "does not 
> scale", "nearly all end users", etc.) and a policy proposal.
> >
> >
> > Yes, you're right. I do not have any hard analytic data to support what 
> users want and are doing, and that observation is based solely on my years 
> of experience with working with Sage, going to and speaking at SageDays, 
> and convincing people to start using Sage. However, there is clear evidence 
> that the current approach is not scaling by the amount of work that is 
> going in and frequent updates/fixed that is needed to be done. Currently, 
> only you are the one doing this.
>
> What exactly is not scaling? "Vendor everything" approach abandoned ~9
> years ago obviously didn't scale.
> I think we're not unvendoring things aggressively enough, and that's
> what we quarrel with Matthias about.
>
> I'm about to start a round of discussions here to lead to a vote to
> remove vendored gcc and gfortran.
> We also should be getting rid of Sage-nonspecific things such as
> vendored Python (with the needed dependencies such as openssl),
> and vendored Jupyter and its huge slew of its dependencies (for the
> latter Matthias is on board, I think).
>
> It's also not correct that modularisation is currently only done by
> Matthias. E.g. most recently I worked on unvendoring of Maxima.
> I worked on doing some abstract classes for parts of sage.coding, I
> worked on spinning out prime counting stuff into a separate
> pip-installable module (that was in 2021, though), etc.
>
> The interdependecies of parts of Sage library on each other are
> decreasing, and this is certainly a big deal in terms of
> updating/fixing
> things. (Not importing from sage.all in the library is a big deal).
>
> Dima
>
>
> Some of that is the lack of discussion (which I would like to have
> seen given the large scale nature of this change, which is implicitly
> setting policy by default). You can disagree with my conclusions and
> proposal, but I want to actually talk about that rather than having
> dismissive comments. Can you provide any specific counterpoints and
> your expectations? What you posted on the other thread does not
> address any of these.
> >
> > Best,
> > Travis
> >
> > --
> > You received this message because you are subscribed to the Google 
> Groups "sage-devel" group.
> > To unsubscribe from this group and stop receiving emails from it, send 
> an email to sage-devel+...@googlegroups.com.
> > To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sage-devel/84fbc2ed-00c8-4ca6-8aa7-d4c74e8c2455n%40googlegroups.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/cc125172-2413-4416-b871-f4fdc606a4c7n%40googlegroups.com.

Re: [sage-devel] Re: Modularation doctests

[Dima Pasechnik]

On Mon, Jun 12, 2023 at 1:35 AM 'Travis Scrimshaw' via sage-devel
 wrote:
>
> Hi Matthias,
>
> Happy to see that you are curious regarding the modularization project, but I 
> don't think it's a good approach to start this discussion with claims that 
> sound authoritative ("nobody will actually maintain", "does not scale", 
> "nearly all end users", etc.) and a policy proposal.
>
>
> Yes, you're right. I do not have any hard analytic data to support what users 
> want and are doing, and that observation is based solely on my years of 
> experience with working with Sage, going to and speaking at SageDays, and 
> convincing people to start using Sage. However, there is clear evidence that 
> the current approach is not scaling by the amount of work that is going in 
> and frequent updates/fixed that is needed to be done. Currently, only you are 
> the one doing this.

What exactly is not scaling? "Vendor everything" approach abandoned ~9
years ago obviously didn't scale.
I think we're not unvendoring things aggressively enough, and that's
what we quarrel with Matthias about.

I'm about to start a round of discussions here to lead to a vote to
remove vendored gcc and gfortran.
We also should be getting rid of Sage-nonspecific things such as
vendored Python (with the needed dependencies such as openssl),
and vendored Jupyter and its huge slew of its dependencies (for the
latter Matthias is on board, I think).

It's also not correct that modularisation is currently only done by
Matthias. E.g. most recently I worked on unvendoring of Maxima.
I worked on doing some abstract classes for parts of sage.coding, I
worked on spinning out prime counting stuff into a separate
pip-installable module (that was in 2021, though), etc.

The interdependecies of parts of Sage library on each other are
decreasing, and this is certainly a big deal in terms of
updating/fixing
things. (Not importing from sage.all in the library is a big deal).

Dima


Some of that is the lack of discussion (which I would like to have
seen given the large scale nature of this change, which is implicitly
setting policy by default). You can disagree with my conclusions and
proposal, but I want to actually talk about that rather than having
dismissive comments. Can you provide any specific counterpoints and
your expectations? What you posted on the other thread does not
address any of these.
>
> Best,
> Travis
>
> --
> You received this message because you are subscribed to the Google Groups 
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sage-devel+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sage-devel/84fbc2ed-00c8-4ca6-8aa7-d4c74e8c2455n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/CAAWYfq3p%2B-tGb5tW2HDfp9UydYs2y92oJP3nQLXEKdxV7juuyQ%40mail.gmail.com.

[sage-devel] Re: Modularation doctests

['Travis Scrimshaw' via sage-devel]

Hi Matthias,

Happy to see that you are curious regarding the modularization project, but 
I don't think it's a good approach to start this discussion with claims 
that sound authoritative ("nobody will actually maintain", "does not 
scale", "nearly all end users", etc.) and a policy proposal. 


Yes, you're right. I do not have any hard analytic data to support what 
users want and are doing, and that observation is based solely on my years 
of experience with working with Sage, going to and speaking at SageDays, 
and convincing people to start using Sage. However, there is clear evidence 
that the current approach is not scaling by the amount of work that is 
going in and frequent updates/fixed that is needed to be done. Currently, 
only you are the one doing this. Some of that is the lack of discussion 
(which I would like to have seen given the large scale nature of this 
change, which is implicitly setting policy by default). You can disagree 
with my conclusions and proposal, but I want to actually talk about that 
rather than having dismissive comments. Can you provide any specific 
counterpoints and your expectations? What you posted on the other thread 
does not address any of these.

Best,
Travis

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/84fbc2ed-00c8-4ca6-8aa7-d4c74e8c2455n%40googlegroups.com.

[sage-devel] Re: Modularation doctests

[Matthias Koeppe]

On Thursday, June 8, 2023 at 5:11:28 AM UTC-7 kcrisman wrote:

how do you know which modules some random doctest you want to include 
depends on? 


By testing the modularized distributions and looking at the errors and 
backtraces. (I'll expand on this in a separate post.)
This is done in #35095, which currently has about 45000 lines of changes 
compared to upstream/develop.

I also think it might be good to have a style guide 


I have added a bit (on how to reformat doctests and where to put the # 
optional comments) in 
https://deploy-preview-35749--sagemath-tobias.netlify.app/developer/coding_basics.html#writing-testable-examples
 
(this is https://github.com/sagemath/sage/pull/35749).
 

[...] to inform users/developers why a particular optional tag is needed, 
in a comment or something.  For a lot of the optional libraries, that was 
fairly obvious (e.g. if you didn't have Mma or Maple installed, don't do 
those tests, or for the various LP solvers), but maybe that isn't so clear 
now. 


I am adding such information in https://github.com/sagemath/sage/pull/35749 
as well. A first version, explaining # optional - sage.combinat" in detail, 
can be seen in 
https://deploy-preview-35749--sagemath-tobias.netlify.app/reference/misc/sage/features/sagemath.html#sage.features.sagemath.sage__combinat

Comments on the new documentation are very welcome.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/cd9be2f5-b63d-4894-8fc6-6d858e5f4bc3n%40googlegroups.com.

[sage-devel] Re: Modularation doctests

[Matthias Koeppe]

I've opened https://github.com/sagemath/sage/issues/35750 to keep track of 
the idea of block tags. I have marked it as a wishlist item.

On Thursday, June 8, 2023 at 9:50:49 AM UTC-7 Matthias Koeppe wrote:

> On Wednesday, June 7, 2023 at 9:59:28 PM UTC-7 Kwankyu Lee wrote:
>
> We may also introduce a block tag like ":: optional - sage.modules 
> sage.rings.finite_rings" that applies to a block of lines.
>
>
> Quick note that I'm all in favor of introducing a mechanism for such block 
> tags; it's of course a clear gap between the line-level tags and the 
> file-level tags that we introduced in 
> https://github.com/sagemath/sage/issues/30778 (see 
> https://wiki.sagemath.org/ReleaseTours/sage-9.5#Module-level_annotation_.22.23_sage.doctest:_optional_-_FEATURE.22
> ). 
>
> (Note though that this feature was discussed several times over the past 
> 15 years without coming to life. See 
> https://github.com/sagemath/sage/issues/30778 for pointers to some of 
> these discussions.)
>
> Any help with implementing this feature would be very welcome.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/1dafb0b3-ca80-4b0c-a87a-96e22392b59an%40googlegroups.com.

[sage-devel] Re: Modularation doctests

[Matthias Koeppe]

On Wednesday, June 7, 2023 at 9:59:28 PM UTC-7 Kwankyu Lee wrote:

We may also introduce a block tag like ":: optional - sage.modules 
sage.rings.finite_rings" that applies to a block of lines.


Quick note that I'm all in favor of introducing a mechanism for such block 
tags; it's of course a clear gap between the line-level tags and the 
file-level tags that we introduced in 
https://github.com/sagemath/sage/issues/30778 (see 
https://wiki.sagemath.org/ReleaseTours/sage-9.5#Module-level_annotation_.22.23_sage.doctest:_optional_-_FEATURE.22).
 

(Note though that this feature was discussed several times over the past 15 
years without coming to life. See 
https://github.com/sagemath/sage/issues/30778 for pointers to some of these 
discussions.)

Any help with implementing this feature would be very welcome.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/acec1cb4-9e8e-4046-bed1-ba6dd9d78590n%40googlegroups.com.

[sage-devel] Re: Modularation doctests

[Matthias Koeppe]

Hi Travis,
Happy to see that you are curious regarding the modularization project, but 
I don't think it's a good approach to start this discussion with claims 
that sound authoritative ("nobody will actually maintain", "does not 
scale", "nearly all end users", etc.) and a policy proposal. 

I'd say it's more productive if I explain a few things first that seem 
unclear. (In particular I'll note that this work is _unrelated_ to making 
"Sage better in terms of being a distribution".)

So I'll post something in the next few days when I find the time for it.

In the meantime, interested readers may want to read the existing 
documentation at 
https://doc.sagemath.org/html/en/developer/packaging_sage_library.html

Matthias

On Wednesday, June 7, 2023 at 6:15:45 PM UTC-7 Travis Scrimshaw wrote:

> Dear everyone,
>I would first like to thank the people who are working to improve the 
> Sage development and build process. However, I am starting to become 
> concerned about what is being done about the modularization of SageMath. 
> Specifically, it is involving the patchbombs (e.g., 
> https://github.com/sagemath/sage/pull/35742) with labeling doctests that 
> nobody will actually maintain beyond possibly Matthias. Furthermore, the 
> large amount of optional labels, especially with no actual optional 
> packages, is starting to scare some users. I tell them they can ignore it, 
> but I feel that is not giving off a good impression. It is even more 
> confounding for people who are starting to develop (e.g., the GSoC 
> students).
>
> In short, the current approach to modularization and doctests does not 
> scale. I also feel the cost-benefit ratio is too high.
>
> I think we need a new approach, something that is both hidden from the 
> end-user (who will essentially never care about anything that is supposed 
> to be in the "standard" distribution of Sage) and allows developers much 
> more ease to actually develop (including Matthias who will largely be the 
> one who will have to fix these).
>
> My proposal is that we *only* have top-of-the-file indicators for 
> doctests unless there is a very compelling reason not to. That is, always do
>
> # sage.doctest: optional - sage.modules sage.rings.finite_rings
>
> at the beginning of files. This will hide what should be unnecessary 
> details from nearly all end users (for example, who really would not 
> install symbolics with Sage on a general install?), make the public 
> documentation cleaner, reduce the maintenance for modularization, be an 
> easy paradigm for all developers, and have a specific location for all 
> relevant information regarding dependencies.
>
> I think there is general consensus that we should make Sage better in 
> terms of being a distribution (albeit with some personal reservations with 
> ending up at the "grass is greener on the other side" feeling). Yet, I 
> really do not see the benefit to having such optional markings localized to 
> individual doctests, which often then have to propagate to subsequent ones.
>
> Best,
> Travis
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/6cce137e-094c-4775-80ec-6aba18a6c21fn%40googlegroups.com.

[sage-devel] Re: Modularation doctests

[kcrisman]

Just ... wow.  I know I don't get much of a say given my lack of recent 
development activity, but this level of granularity does seem absurd.  It 
would certainly be a (psychological) barrier to development - how do you 
know which modules some random doctest you want to include depends on? - as 
well as to bug reporting - if it's optional, the first thought would be 
it's not really necessary to test.  Travis makes some very good points.

Anyway, though I'm still not sure who is going to use sub pieces of Sage 
(the library), I understand that for development purposes, and for being 
able to add more narrow pieces (in the spirit of William's old psage, see 
https://github.com/fredstro/psage for the most recent activity there), it 
is a done deal to modularize.  Kwankyu's last "block" idea seems a good 
compromise, particularly because it could be retrofitted to many other 
optional doctests for truly "optional" packages, which might actually 
streamline documentation for those.

I also think it might be good to have a style guide telling those writing 
such tests to inform users/developers why a particular optional tag is 
needed, in a comment or something.  For a lot of the optional libraries, 
that was fairly obvious (e.g. if you didn't have Mma or Maple installed, 
don't do those tests, or for the various LP solvers), but maybe that isn't 
so clear now.  Just some thoughts.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/5ea2b7cc-aa48-4113-b4d6-e585a191418fn%40googlegroups.com.

[sage-devel] Re: Modularation doctests

[Kwankyu Lee]

Thanks for opening a discussion on this timely topic.

I think that Matthias is well aware of the benefits of the top-of-the-file 
tags over the doctest tags, and is adding doctest tags only when they are 
necessary (this is a lot of work for him). However, I do think that your 
points

(1) hide unnecessary details from all end users
(2) make the public documentation cleaner
(3) reduce the maintenance for modularization
(4) be an easy paradigm for all developers
(5) have a specific location for all relevant information regarding 
dependencies

should be achieved with the doctest tags too. We may also introduce a block 
tag like ":: optional - sage.modules sage.rings.finite_rings" that applies 
to a block of lines.

On Thursday, June 8, 2023 at 10:15:45 AM UTC+9 Travis Scrimshaw wrote:

> Dear everyone,
>I would first like to thank the people who are working to improve the 
> Sage development and build process. However, I am starting to become 
> concerned about what is being done about the modularization of SageMath. 
> Specifically, it is involving the patchbombs (e.g., 
> https://github.com/sagemath/sage/pull/35742) with labeling doctests that 
> nobody will actually maintain beyond possibly Matthias. Furthermore, the 
> large amount of optional labels, especially with no actual optional 
> packages, is starting to scare some users. I tell them they can ignore it, 
> but I feel that is not giving off a good impression. It is even more 
> confounding for people who are starting to develop (e.g., the GSoC 
> students).
>
> In short, the current approach to modularization and doctests does not 
> scale. I also feel the cost-benefit ratio is too high.
>
> I think we need a new approach, something that is both hidden from the 
> end-user (who will essentially never care about anything that is supposed 
> to be in the "standard" distribution of Sage) and allows developers much 
> more ease to actually develop (including Matthias who will largely be the 
> one who will have to fix these).
>
> My proposal is that we *only* have top-of-the-file indicators for 
> doctests unless there is a very compelling reason not to. That is, always do
>
> # sage.doctest: optional - sage.modules sage.rings.finite_rings
>
> at the beginning of files. This will hide what should be unnecessary 
> details from nearly all end users (for example, who really would not 
> install symbolics with Sage on a general install?), make the public 
> documentation cleaner, reduce the maintenance for modularization, be an 
> easy paradigm for all developers, and have a specific location for all 
> relevant information regarding dependencies.
>
> I think there is general consensus that we should make Sage better in 
> terms of being a distribution (albeit with some personal reservations with 
> ending up at the "grass is greener on the other side" feeling). Yet, I 
> really do not see the benefit to having such optional markings localized to 
> individual doctests, which often then have to propagate to subsequent ones.
>
> Best,
> Travis
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/82832aea-dbc2-4109-9e35-9aeebca2b8aen%40googlegroups.com.