Re: Is it time to remove python-numpy from testing?

2020-07-09 Thread Paul Gevers
Hi

On 09-07-2020 21:16, peter green wrote:
> All of the reverse dependencies of python-numpy have already been
> removed from testing. So IMO
> it makes sense to remove python-numpy from testing at this point, do
> other people agree?

I think it makes sense, so I added a removal hint.

Paul



signature.asc
Description: OpenPGP digital signature


Re: cannot allocate memory in static TLS block (Was: Bug#953832: drmaa: autopkgtest failure: RuntimeError: Could not find drmaa library)

2020-03-14 Thread Paul Gevers
Hi Andreas,

On 14-03-2020 14:30, Andreas Tille wrote:
> Hi Paul,

[...]

> Any help would be welcome

I can't help you with this.

Paul



signature.asc
Description: OpenPGP digital signature


Proposal on how to proceed with Python 2 removal from bullseye

2019-12-14 Thread Paul Gevers
Dear all,

TL;DR: as the subject suggests, proposal included, although not fully
aligned with others.

On 08-12-2019 20:55, Sandro Tosi wrote:
> there seems to be disagreement on how to proceed, so for the time
> being i suspended the severity bump part of the py2removal tracking
> script. let me know when everybody agrees on a solution, and what that
> solution is, and i'll code it and re-enable.

[...]

>> On 03-12-2019 13:19, Matthias Klose wrote:
>>> It's unfortunate that issues for some packages only get attention when the
>>> severity of an issue is raised.  Following your proposal means that the 
>>> issue is
>>> probably ignored forever, and you don't propose a better way going forward, 
>>> just
>>> saying we should stop earlier.  I don't think that's the correct choice.  
>>> For
>>> now these seem to be single packages, so please could you name those, and 
>>> we can
>>> look at those with a priority?  That's at least a path that is forward 
>>> looking,
>>> or feel free to propose another approach better than your current proposal 
>>> for
>>> not getting the attention of maintainers.
>>
>> I guess what I'm asking for could be fulfilled by an announcement to
>> d-d-a with a package list (including via which package(s) they are
>> affected) attached including the standard grouping by DD. And then some
>> time to react.

It has been brought to my attention (thanks ginggs) that I have been
focused too much on the release teams side of the story and on the
maintainer of reverse dependencies side of the story. There are more
facets to the Python 2 removal, my aim of this e-mail is to make it more
balanced.

For full clarity, in my mind the discussion about using RC status to
have the autoremoval process enable the clean up in bullseye was about
the question to the release team if they would support making Python 2
removal bugs RC *from the release teams point of view*. In my reasoning
and responses, I fully forgot the description of serious in the BTS that
explicitly (and in my opinion rightfully) mentions the package maintainer:
"""
is a severe violation of Debian policy (roughly, it violates a "must" or
"required" directive), or, in the package maintainer's or release
manager's opinion, makes the package unsuitable for release.
"""
which is reflected in the release team rc_policy for the last couple of
releases [1].

So, to conclude, within Debian it is considered to be in the full power
of a maintainer to raise the severity of their own package to serious.
Hence, in hindsight I believe the question to the release team was
mostly about Python maintainers marking bugs of *other* maintainers as
RC. I believe that for that last category, the careful approach taken by
the people behind the Python 2 removal and requested in my
communications is warranted.

That said, and keeping in mind that autoremovals only work on non-key
packages [2] and that there is always a social side to invasive changes,
I come with the following *proposal* containing a few items. In all, it
may even speed up the Python 2 removal process:

1) all Python 2 removal bugs *can* be raised to RC level by the
maintainers of the packages they belong to, but:

2) maintainers of non-key packages should carefully consider the
backslash for their transitive reverse (normal, build and test)
dependencies before raising the bug severity level and in my opinion
should hold off doing that if there are many that need adaption for the
Python 2 support removal. To be absolutely clear of my intent, if there
is only a *very* limited set that needs adaption but they have a large
set of reverse dependencies that is not what I mean here. The
maintainers of the large set of reverse dependencies have a joint
incentive to fix the issue if the maintainer of the to-be-adapted
package(s) doesn't step up to fix the situation.

3) packages with unfixed reverse (normal, build or test) dependencies,
that want to drop Python 2 support should first make sure their unfixed
reverse dependencies have RC bugs themselves (but please take the
consideration of 2 into account), stating at least something like
"source package $foo is soon going to remove the binary $bar package on
which the $baz package (build) depends, please fix that".

To avoid problems with bug fixes that need to migrate to testing soon,
please don't upload the Python 2 support drop immediately, but wait
until either the reverse dependencies are fixed or the date for the
autoremoval is near.

4) leaf packages (i.e. without normal, build or test reverse
dependencies) that need to be adapted themselves for the Python 2
removal can be marked as RC.

5) Please continue to use the approach used so far as well, but please
also add checks on build dependencies.

Paul

[1] https://release.debian.org/buster/rc_policy.txt
[2] https://udd.debian.org/cgi-bin/key_packages.yaml.cgi



signature.asc
Description: OpenPGP digital signature


Re: Severity bump script

2019-12-05 Thread Paul Gevers
Hi,

On 03-12-2019 13:19, Matthias Klose wrote:
> It's unfortunate that issues for some packages only get attention when the
> severity of an issue is raised.  Following your proposal means that the issue 
> is
> probably ignored forever, and you don't propose a better way going forward, 
> just
> saying we should stop earlier.  I don't think that's the correct choice.  For
> now these seem to be single packages, so please could you name those, and we 
> can
> look at those with a priority?  That's at least a path that is forward 
> looking,
> or feel free to propose another approach better than your current proposal for
> not getting the attention of maintainers.

I guess what I'm asking for could be fulfilled by an announcement to
d-d-a with a package list (including via which package(s) they are
affected) attached including the standard grouping by DD. And then some
time to react.

Paul

PS, my original typed reply below. After having writing it, the idea
above emerged.

My problem with the current approach is that the way that developers
learn that they (potentially transitively) (build) depend on a Python 2
package is by the autoremoval message. I don't think that is acceptable
socially in the project. My proposal is to give the maintainers about
those packages a heads up. Via a bug report may be a bit weird in cases,
but in some cases may be the appropriate thing as the package could work
around the (build) dependency. At least adding "affects" helps a tiny
bit and direct e-mails to the maintainers (e.g. via the
@packages.debian.org address will at least give them a heads
up. Even if the RM bug is coming eventually.

Again, I don't have a problem with Python 2 removal, even at the price
of packages that don't care that their dependency is written in Python.
I do care about proper communication. Using RC severity to trigger
autoremoval for non-leaf packages just yet is not appropriate in the
opinion of the release team. Even though I am well aware of the Python 2
removal effort I can tell from personal experience (cacti) that
receiving an autoremoval e-mail as the first sign of such a dependency
isn't nice, that's the problem I'm having and that needs solving in my
opinion.



signature.asc
Description: OpenPGP digital signature


Re: Severity bump script

2019-12-02 Thread Paul Gevers
Hi,

On 02-12-2019 22:15, Sandro Tosi wrote:
> the blocks are only between py2removal packages, so if a package
> un-related to the py2removal effort
> depend/recomments/b-deps/autotest-triggers a py2removal *application*, that
> is still considered a leaf package

You'll fix that, right? Because why would the tree stop at Python? A
leaf package is a package without Depends/Build-Depends in Debian. (I
appreciate it very much that you consider Recommends and
autopkgtest-triggers as well).

Paul



signature.asc
Description: OpenPGP digital signature


Re: Severity bump script

2019-12-02 Thread Paul Gevers
Hi Ondrej,

On 02-12-2019 20:33, Ondrej Novy wrote:
> Hi,
> 
> po 2. 12. 2019 v 20:28 odesílatel Paul Gevers  <mailto:elb...@debian.org>> napsal:
> 
> I understand the drive to push for Python 2 removal and sympathize with
> it. The issue I had yesterday with the process is that "leaf" was
> wrongly defined, it was looking at Depends, instead of also including
> Build-Depends.
> 
> 
> huh? We are not bumping any "blocked" bugs.
> Depends/Build-Depends/Recommends/autopkgtests usage marks bug as
> blocked. Any example of "wrong definition" please?

#942999 and #936537

Paul



signature.asc
Description: OpenPGP digital signature


Re: Severity bump script

2019-12-02 Thread Paul Gevers
Hi all,

On 01-12-2019 22:45, Sandro Tosi wrote:
> Paul, this is the thread i was talking about.
> 
> you were copied in the original email:
> https://lists.debian.org/debian-python/2019/10/msg00098.html
> 
> if there is something the RT wants to discuss about this effort,
> please do so here, not directly to me (i may be the mail address
> sending those control commands, but the decision was taken here).

I understand the drive to push for Python 2 removal and sympathize with
it. The issue I had yesterday with the process is that "leaf" was
wrongly defined, it was looking at Depends, instead of also including
Build-Depends.

I don't want to stand in the way of Python 2 removal, but as I said
before, pulling packages out from underneath maintainers isn't pretty so
needs to be done with proper notifications and care. An RC bug to ones
own package is acceptable in my opinion as it is a clear discussion
forum, and can be (temporarily) down-graded while the discussion is
ongoing. Being notified about some other package that I not even need
directly but is going to pull "my" package out of testing isn't a nice
experience and should be avoided. Yes, that slows down the process, but
there are people, emotions and all those irrational things involved.

Paul



signature.asc
Description: OpenPGP digital signature


Re: Discussing next steps for the Python2 removal

2019-10-22 Thread Paul Gevers
Dear all,

On 22-10-2019 17:04, Matthias Klose wrote:
> He suggested to make the removal plan more
> concrete and having a timeline.

To be more precise on what I meant, I'm talking here about *every*
package that wants to drop a Python 2 binary package, discuss (and
ideally agree on, but I understand that that can be hard) the time line
with their reverse dependencies.

If this is done globally, fine, but please, give reverse dependencies
time to adapt. I'm already seeing quite some annoyance on the fact that
in several cases the ground is pulled away under one's package.

Yes, there's a bug in britney somewhere that allows the migration, we're
trying to find it.

Paul

For those that are curious, the britney output [1] shows at the end
binary packages in testing that are not build from source anymore but
are left in testing due to dependencies. Those that aren't in
libs/oldlibs shouldn't be there and are make their reverse dependencies
RC buggy. On top of that, Dose [2] shows packages that can't be build
from source in testing. There's quite some python2 packages listing as
causing that. Again, all those packages are RC buggy (often the bug
hasn't been filed yet).

[1] https://release.debian.org/britney/update_output.txt.gz
[2] https://qa.debian.org/dose/debcheck/src_testing_main/ (pick the
latest amd64 log, it should be empty except for cmucl)



signature.asc
Description: OpenPGP digital signature


Re: should Debian add itself to https://python3statement.org ?

2019-09-12 Thread Paul Gevers
Hi,

On 12-09-2019 17:01, Ian Jackson wrote:
> But we need to be clear what's going on and communicate early.

Yes, not on the front page, but there is (first bullet):

https://www.debian.org/releases/buster/amd64/release-notes/ch-information.en.html#deprecated-components

Paul



signature.asc
Description: OpenPGP digital signature


Re: unblock request: python-scipy/1.1.0-4 skimage/0.14.2-2: autopkgtest passes (Re: bug#919929)

2019-03-16 Thread Paul Gevers
Hi Drew,

On 16-03-2019 13:48, Drew Parsons wrote:
>> The numpy.sparse tests pass with this patch, and most of the matrix
>> PendingDeprecationWarnings are gone (the upstream patch missed
>> integrate/tests/test_ivp.py, but the remaining warnings are few enough
>> to not need to worry about).
> 
> Well, turns out the other warnings worried Aurelien enough to file
> Bug#924396.
> 
> Is there enough will to add more scipy patches for the buster release to
> reduce the remaining DeprecationWarnings? (they don't break tests,
> they're just annoying)
> Or should we just let it go at this point and let them get cleared in
> future versions?)

I'd let it be for now.

> That being the case, in the interests of making a stable release that
> passes it own tests, I'd like to request an unblock for
> python-scipy/1.1.0-4 (together with skimage/0.14.2-2)

skimage was already unblocked. I'll unblock python-scipy as well.

Paul



signature.asc
Description: OpenPGP digital signature


Re: python-scipy: autopkgtest fails (Re: bug#919929)

2019-03-07 Thread Paul Gevers
Hi Drew,

On 07-03-2019 14:56, Drew Parsons wrote:
> On 2019-03-07 20:46, Paul Gevers wrote:
>> However, it is probably worth waiting for a resolution of bug
>> 915738 and combine it with that.
> 
> There hasn't been recent movement on 915738. I'll apply Julian's patch
> and see how we go.

Huh. There was a comment from doko that the underlying issue is fixed in
gcc-8, no? I think you only need to switch to the default gfortran. On
the other hand, I don't know how feasible it is at this moment to not
release buster with gcc-7. Maybe other release team members can comment
on that?

Paul



signature.asc
Description: OpenPGP digital signature


Re: python-scipy: autopkgtest fails (Re: bug#919929)

2019-03-07 Thread Paul Gevers
Hi Drew,

On 07-03-2019 13:19, Drew Parsons wrote:
>> Can you elaborate why you think that bug should be RC (as that isn't
>> clear to me from the report itself) and why you haven't marked it as
>> such if you think it should be?
> 
> python-scipy is currently failing all debci tests in both unstable and
> testing.

autopkgtest only, so no FTBFS?

> I haven't marked it RC myself since I'm not 100% certain what the usual
> protocol is for marking the severity of debci test failures.  But as I
> understood it, debci test failures is considered RC under the final
> freeze which we're about to enter (but we're not quite in that deep
> freeze yet).

Some of us want failing autopkgtest to be RC *after* the release of
buster. I am not aware of consensus about that yet. autopkgtest
*regression* in testing is effectively RC since the soft freeze of
12-2-2019. The autopkgtest of python-scipy is already failing in
testing, so it isn't a regression. Hence, it failing is not RC for buster.

> Hi Andreas, I may not have been clear.  What I mean at this point is to
> upload a small patch for scipy 1.1 to ignore the deprecation warnings
> (scipy 1.2 already does that).
> 
> If that doesn't help pass the debci tests then we can consider uploading
> scipy 1.2 instead.  But python-fluids and dipy FTBFS against scipy 1.2
> (a new version of dipy is available which presumeably fixes that)

Please *don't* upload scipy 1.2 to fix only this issue (nor for any
other issue without discussion first), it's for sure not worth it.

>> Slightly depending on the answer above, I'll unblock an upload of
>> python-scipy with only that change.
> 
> Certainly please do unblock if the full freeze is already in place. But
> my intention was to first upload python-scipy 1.1 with a small patch,
> not 1.2 just yet.

If you upload now, your package will not migrate to testing before the
full freeze becomes effective so it would need an unblock. If you want
to fix this issue with the three lines I saw in the bug report, you can
go ahead. However, it is probably worth waiting for a resolution of bug
915738 and combine it with that.

Paul



signature.asc
Description: OpenPGP digital signature


Re: python-scipy: autopkgtest fails (Re: bug#919929)

2019-03-07 Thread Paul Gevers
Hi Drew,

On 07-03-2019 09:03, Andreas Tille wrote:
> On Tue, Mar 05, 2019 at 07:01:54PM +0800, Drew Parsons wrote:
>> python-scipy has recently started failing all debci tests in testing and
>> unstable, exacerbating the bug report in Bug#919929 [1].
>>
>> The failing error is a MemoryError. But understanding the problem is
>> hampered by a flood of deprecation warnings, presumably triggered by numpy
>> 1.16. scipy 1.2 added instructions to pytest.ini to ignore the warnings.
>>
>> Bug#919929 has not yet been marked RC but I guess it's about to happen.

Can you elaborate why you think that bug should be RC (as that isn't
clear to me from the report itself) and why you haven't marked it as
such if you think it should be?

>> I
>> propose in the first instance to apply a patch of the pytest.ini diff
>> between scipy 1.1 and 1.2 and see if that clears things up. I'll commit the
>> patch now. Should I proceed with upload to unstable?
> 
> May be its sensible to coordinate with debian-release (list in CC) and
> file a unblock request *before* uploading.  I confirm that I usually
> upload simple fixes and ask for unblock afterwards but you intend to do
> a version change which does not qualify as simple fix (despite I agree
> with you that it is sensible).

Slightly depending on the answer above, I'll unblock an upload of
python-scipy with only that change.

Paul



signature.asc
Description: OpenPGP digital signature


Re: Numpy migration?

2019-01-07 Thread Paul Gevers
Hi Mattia, all,

On 07-01-2019 17:20, Mattia Rizzolo wrote:
> On Mon, Jan 07, 2019 at 09:03:11AM +0100, Ole Streicher wrote:
>> Mattia Rizzolo  writes:
>>> On Sun, Jan 06, 2019 at 05:07:41PM +0100, Ole Streicher wrote:
 Now it turns out that there is a new migration problem, which is aplpy:
 Current aplpy (2.0~rc2-2) CI test works well
>>>
>>> You probably mean aplpy 1.1.1-4.
>>
>> No, I meant the one above (although the unstable test was not done on
>> ci.d.n when I wrote my last mail).
> 
> Right, that explains why I didn't see it.
> It seems now britney is triggering aplpy test with a big set of
> triggers (I suppose the result of the several Breaks that have been
> added in the meantime), so it's indeed trying to move everything
> together.

Indeed.

>> The problem is that aplpy uses matplotlib, and the old matplotlib uses
>> the deprecated numpy function np.asscalar(), which leads to a
>> DeprecationWarning, which is (on purpose, by upstream) thrown as error.
>>
>> https://ci.debian.net/data/autopkgtest/testing/amd64/a/aplpy/1658948/log.gz
>>
>> This does only happen in the combination "new numpy + old matplotlib",
>> but this is the one that is tested for the CI test.
> 
> mhh… for this, indeed a breaks numpy -> matplotlib could be used, but it
> feels a bit too strong to me, as nothing was really broken since it's
> just deprecations.
> 
> elbrus; this is a case where I think I could use your input (tl;dr: new
> numpy causes deprecations in testing's version of matplotlib (fixed in
> unstable), which in turn causes failures in new aplpy, how to make the
> stack happy?).

The commit, with lots of comments [1], that implemented most of the
related logic is here:
https://salsa.debian.org/release-team/britney2/commit/18633e275c34973379f1426542047bf30c7829c3

With the current code your options are:
- *versioned* depends on one of the binary packages from the source you
want from unstable in any of the binary packages that is going to be
taken from unstable
- *versioned* breaks or conflicts on the binary packages from the source
you want from unstable in any of the binary packages that is going to be
taken from unstable
- *versioned* test dependency in the package that is used for
autopkgtest (only helps if the version that is tested is already taken
from unstable). The version of the test dependency will NOT be added to
the triggers (maybe, I have to think about it, that could be an britney
improvement), but if the version of the autopktest is going to be taken
from unstable already due to other considerations, with the current
settings of ci.d.n and autopkgtest the required version will be installed.

Personally I don't think it is an issue to add the breaks, but I can see
why people find it heavy in cases like this one.

As discussed in the med-fichier bug, britney may want to grow knowledge
from a X-breaks-autopkgtest-of-source header, but that doesn't exist now
and I am not extremely keen about adding it because I fear people may
too easily add it when a normal breaks is in order.

Paul

[1]
+# Here we figure out what is required from the source suite
+# for the test to install successfully.
+#
+# Loop over all binary packages from trigger and
+# recursively look up which *versioned* dependencies are
+# only satisfied in the source suite.
+#
+# For all binaries found, look up which packages they
+# break/conflict with in the target suite, but not in the
+# source suite. The main reason to do this is to cover test
+# dependencies, so we will check Testsuite-Triggers as
+# well.
+#
+# OI: do we need to do the first check in a smart way
+# (i.e. only for the packages that are actully going to be
+# installed) for the breaks/conflicts set as well, i.e. do
+# we need to check if any of the packages that we now
+# enforce being from the source suite, actually have new
+# versioned depends and new breaks/conflicts.
+#
+# For all binaries found, add the set of unique source
+# packages to the list of triggers.



signature.asc
Description: OpenPGP digital signature