Re: Updating python3-xlrd for pandas 1.5 compatibility
Hi Diane, On 23-02-2023 08:12, Diane Trout wrote: the version of python3-xlrd 1.2.0-3 in unstable/testing is too old to be used with pandas 1.5.3. (See Bug #1031701). Do I understand correctly that this isn't an issue from the point of python3-xlrd and that only pandas is effected? While investigating for this reply I noticed src:pandas doesn't even have a dependency in any of its binaries. As it is a really common workflow to use pandas to read excel files, it'd be nice if the version of xlrd in bookworm was compatible. As the maintainer of pandas, do you consider it an RC issue that pandas can't convert it? I guess not because you say "it'd be nice" and you don't even have the required dependency. How severe do you consider this issue for pandas? pandas has a quite extensive autopkgtest, doesn't it cover this use case? Apparently you knew this earlier, why do you bring this up now? Because of the freeze I wanted to check if it was appropriate to upload the new version, I'd hope that the "rules" are clear: https://release.debian.org/testing/freeze_policy.html#soft. You can contact the Release Team if you need further clarification. and what kind of warning I should give to the other developers. It depends. I'm worried about what you write below. THe xlrd changelog says the biggest change in going from 1.2 to 2.0 was they removed the ability to read the newer XML excel files .xslx from xlrd in favor of using openpyxl That sounds like a change that we'd normally consider inappropriate. So we'd need to balance the pain vs the gain and the additional risk of unknown delta's. I updated the source package python-xlrd to 2.0.1 and sent it through experimental, where there were no issues detected by packages that had CI tests. Indeed https://qa.debian.org/excuses.php?experimental=1=python-xlrd is clean. Unfortunately there's packages without tests. Like pandas not testing for xls loading; it wasn't even scheduled as pandas has no binaries depending on python3-xlrd. Here's the list of packages I found that have any relationship to python-xlrd, if it looked like the autopkgtests actually tested using the xlrd library and what the level of declared dependency is. (none means the package lacks autopackage tests) | nemo | none | Recommends| | odoo-14 | none | Depends | | ofxstatement-plugins | none | Depends | | psychopy | unlikely | Depends | | python3-agateexcel | yes | Depends | | python3-canmatrix| no | Recommends| | python3-drslib | no | Recommends| | python3-glue | yes | Depends | | python3-pyspectral | probably | Suggests | | python3-rows | unlikely | Recommends| | python3-tablib | unlikely | Depends | | visidata | none | Build-Depends | | vistrails| none | Build-Depends | | python-xrt | none | Build-Depends | | pyutilib | none | Build-Depends | If I read everything correctly, it seems like you're too late with this change. Paul OpenPGP_signature Description: OpenPGP digital signature
Re: Is it time to remove python-numpy from testing?
Hi On 09-07-2020 21:16, peter green wrote: > All of the reverse dependencies of python-numpy have already been > removed from testing. So IMO > it makes sense to remove python-numpy from testing at this point, do > other people agree? I think it makes sense, so I added a removal hint. Paul signature.asc Description: OpenPGP digital signature
Re: cannot allocate memory in static TLS block (Was: Bug#953832: drmaa: autopkgtest failure: RuntimeError: Could not find drmaa library)
Hi Andreas, On 14-03-2020 14:30, Andreas Tille wrote: > Hi Paul, [...] > Any help would be welcome I can't help you with this. Paul signature.asc Description: OpenPGP digital signature
Proposal on how to proceed with Python 2 removal from bullseye
Dear all, TL;DR: as the subject suggests, proposal included, although not fully aligned with others. On 08-12-2019 20:55, Sandro Tosi wrote: > there seems to be disagreement on how to proceed, so for the time > being i suspended the severity bump part of the py2removal tracking > script. let me know when everybody agrees on a solution, and what that > solution is, and i'll code it and re-enable. [...] >> On 03-12-2019 13:19, Matthias Klose wrote: >>> It's unfortunate that issues for some packages only get attention when the >>> severity of an issue is raised. Following your proposal means that the >>> issue is >>> probably ignored forever, and you don't propose a better way going forward, >>> just >>> saying we should stop earlier. I don't think that's the correct choice. >>> For >>> now these seem to be single packages, so please could you name those, and >>> we can >>> look at those with a priority? That's at least a path that is forward >>> looking, >>> or feel free to propose another approach better than your current proposal >>> for >>> not getting the attention of maintainers. >> >> I guess what I'm asking for could be fulfilled by an announcement to >> d-d-a with a package list (including via which package(s) they are >> affected) attached including the standard grouping by DD. And then some >> time to react. It has been brought to my attention (thanks ginggs) that I have been focused too much on the release teams side of the story and on the maintainer of reverse dependencies side of the story. There are more facets to the Python 2 removal, my aim of this e-mail is to make it more balanced. For full clarity, in my mind the discussion about using RC status to have the autoremoval process enable the clean up in bullseye was about the question to the release team if they would support making Python 2 removal bugs RC *from the release teams point of view*. In my reasoning and responses, I fully forgot the description of serious in the BTS that explicitly (and in my opinion rightfully) mentions the package maintainer: """ is a severe violation of Debian policy (roughly, it violates a "must" or "required" directive), or, in the package maintainer's or release manager's opinion, makes the package unsuitable for release. """ which is reflected in the release team rc_policy for the last couple of releases [1]. So, to conclude, within Debian it is considered to be in the full power of a maintainer to raise the severity of their own package to serious. Hence, in hindsight I believe the question to the release team was mostly about Python maintainers marking bugs of *other* maintainers as RC. I believe that for that last category, the careful approach taken by the people behind the Python 2 removal and requested in my communications is warranted. That said, and keeping in mind that autoremovals only work on non-key packages [2] and that there is always a social side to invasive changes, I come with the following *proposal* containing a few items. In all, it may even speed up the Python 2 removal process: 1) all Python 2 removal bugs *can* be raised to RC level by the maintainers of the packages they belong to, but: 2) maintainers of non-key packages should carefully consider the backslash for their transitive reverse (normal, build and test) dependencies before raising the bug severity level and in my opinion should hold off doing that if there are many that need adaption for the Python 2 support removal. To be absolutely clear of my intent, if there is only a *very* limited set that needs adaption but they have a large set of reverse dependencies that is not what I mean here. The maintainers of the large set of reverse dependencies have a joint incentive to fix the issue if the maintainer of the to-be-adapted package(s) doesn't step up to fix the situation. 3) packages with unfixed reverse (normal, build or test) dependencies, that want to drop Python 2 support should first make sure their unfixed reverse dependencies have RC bugs themselves (but please take the consideration of 2 into account), stating at least something like "source package $foo is soon going to remove the binary $bar package on which the $baz package (build) depends, please fix that". To avoid problems with bug fixes that need to migrate to testing soon, please don't upload the Python 2 support drop immediately, but wait until either the reverse dependencies are fixed or the date for the autoremoval is near. 4) leaf packages (i.e. without normal, build or test reverse dependencies) that need to be adapted themselves for the Python 2 removal can be marked as RC. 5) Please continue to use the approach used so far as well, but please also add checks on build dependencies. Paul [1] https://release.debian.org/buster/rc_policy.txt [2] https://udd.debian.org/cgi-bin/key_packages.yaml.cgi signature.asc Description: OpenPGP digital signature
Re: Severity bump script
Hi, On 03-12-2019 13:19, Matthias Klose wrote: > It's unfortunate that issues for some packages only get attention when the > severity of an issue is raised. Following your proposal means that the issue > is > probably ignored forever, and you don't propose a better way going forward, > just > saying we should stop earlier. I don't think that's the correct choice. For > now these seem to be single packages, so please could you name those, and we > can > look at those with a priority? That's at least a path that is forward > looking, > or feel free to propose another approach better than your current proposal for > not getting the attention of maintainers. I guess what I'm asking for could be fulfilled by an announcement to d-d-a with a package list (including via which package(s) they are affected) attached including the standard grouping by DD. And then some time to react. Paul PS, my original typed reply below. After having writing it, the idea above emerged. My problem with the current approach is that the way that developers learn that they (potentially transitively) (build) depend on a Python 2 package is by the autoremoval message. I don't think that is acceptable socially in the project. My proposal is to give the maintainers about those packages a heads up. Via a bug report may be a bit weird in cases, but in some cases may be the appropriate thing as the package could work around the (build) dependency. At least adding "affects" helps a tiny bit and direct e-mails to the maintainers (e.g. via the @packages.debian.org address will at least give them a heads up. Even if the RM bug is coming eventually. Again, I don't have a problem with Python 2 removal, even at the price of packages that don't care that their dependency is written in Python. I do care about proper communication. Using RC severity to trigger autoremoval for non-leaf packages just yet is not appropriate in the opinion of the release team. Even though I am well aware of the Python 2 removal effort I can tell from personal experience (cacti) that receiving an autoremoval e-mail as the first sign of such a dependency isn't nice, that's the problem I'm having and that needs solving in my opinion. signature.asc Description: OpenPGP digital signature
Re: Severity bump script
Hi, On 02-12-2019 22:15, Sandro Tosi wrote: > the blocks are only between py2removal packages, so if a package > un-related to the py2removal effort > depend/recomments/b-deps/autotest-triggers a py2removal *application*, that > is still considered a leaf package You'll fix that, right? Because why would the tree stop at Python? A leaf package is a package without Depends/Build-Depends in Debian. (I appreciate it very much that you consider Recommends and autopkgtest-triggers as well). Paul signature.asc Description: OpenPGP digital signature
Re: Severity bump script
Hi Ondrej, On 02-12-2019 20:33, Ondrej Novy wrote: > Hi, > > po 2. 12. 2019 v 20:28 odesílatel Paul Gevers <mailto:elb...@debian.org>> napsal: > > I understand the drive to push for Python 2 removal and sympathize with > it. The issue I had yesterday with the process is that "leaf" was > wrongly defined, it was looking at Depends, instead of also including > Build-Depends. > > > huh? We are not bumping any "blocked" bugs. > Depends/Build-Depends/Recommends/autopkgtests usage marks bug as > blocked. Any example of "wrong definition" please? #942999 and #936537 Paul signature.asc Description: OpenPGP digital signature
Re: Severity bump script
Hi all, On 01-12-2019 22:45, Sandro Tosi wrote: > Paul, this is the thread i was talking about. > > you were copied in the original email: > https://lists.debian.org/debian-python/2019/10/msg00098.html > > if there is something the RT wants to discuss about this effort, > please do so here, not directly to me (i may be the mail address > sending those control commands, but the decision was taken here). I understand the drive to push for Python 2 removal and sympathize with it. The issue I had yesterday with the process is that "leaf" was wrongly defined, it was looking at Depends, instead of also including Build-Depends. I don't want to stand in the way of Python 2 removal, but as I said before, pulling packages out from underneath maintainers isn't pretty so needs to be done with proper notifications and care. An RC bug to ones own package is acceptable in my opinion as it is a clear discussion forum, and can be (temporarily) down-graded while the discussion is ongoing. Being notified about some other package that I not even need directly but is going to pull "my" package out of testing isn't a nice experience and should be avoided. Yes, that slows down the process, but there are people, emotions and all those irrational things involved. Paul signature.asc Description: OpenPGP digital signature
Re: Discussing next steps for the Python2 removal
Dear all, On 22-10-2019 17:04, Matthias Klose wrote: > He suggested to make the removal plan more > concrete and having a timeline. To be more precise on what I meant, I'm talking here about *every* package that wants to drop a Python 2 binary package, discuss (and ideally agree on, but I understand that that can be hard) the time line with their reverse dependencies. If this is done globally, fine, but please, give reverse dependencies time to adapt. I'm already seeing quite some annoyance on the fact that in several cases the ground is pulled away under one's package. Yes, there's a bug in britney somewhere that allows the migration, we're trying to find it. Paul For those that are curious, the britney output [1] shows at the end binary packages in testing that are not build from source anymore but are left in testing due to dependencies. Those that aren't in libs/oldlibs shouldn't be there and are make their reverse dependencies RC buggy. On top of that, Dose [2] shows packages that can't be build from source in testing. There's quite some python2 packages listing as causing that. Again, all those packages are RC buggy (often the bug hasn't been filed yet). [1] https://release.debian.org/britney/update_output.txt.gz [2] https://qa.debian.org/dose/debcheck/src_testing_main/ (pick the latest amd64 log, it should be empty except for cmucl) signature.asc Description: OpenPGP digital signature
Re: should Debian add itself to https://python3statement.org ?
Hi, On 12-09-2019 17:01, Ian Jackson wrote: > But we need to be clear what's going on and communicate early. Yes, not on the front page, but there is (first bullet): https://www.debian.org/releases/buster/amd64/release-notes/ch-information.en.html#deprecated-components Paul signature.asc Description: OpenPGP digital signature
Re: unblock request: python-scipy/1.1.0-4 skimage/0.14.2-2: autopkgtest passes (Re: bug#919929)
Hi Drew, On 16-03-2019 13:48, Drew Parsons wrote: >> The numpy.sparse tests pass with this patch, and most of the matrix >> PendingDeprecationWarnings are gone (the upstream patch missed >> integrate/tests/test_ivp.py, but the remaining warnings are few enough >> to not need to worry about). > > Well, turns out the other warnings worried Aurelien enough to file > Bug#924396. > > Is there enough will to add more scipy patches for the buster release to > reduce the remaining DeprecationWarnings? (they don't break tests, > they're just annoying) > Or should we just let it go at this point and let them get cleared in > future versions?) I'd let it be for now. > That being the case, in the interests of making a stable release that > passes it own tests, I'd like to request an unblock for > python-scipy/1.1.0-4 (together with skimage/0.14.2-2) skimage was already unblocked. I'll unblock python-scipy as well. Paul signature.asc Description: OpenPGP digital signature
Re: python-scipy: autopkgtest fails (Re: bug#919929)
Hi Drew, On 07-03-2019 14:56, Drew Parsons wrote: > On 2019-03-07 20:46, Paul Gevers wrote: >> However, it is probably worth waiting for a resolution of bug >> 915738 and combine it with that. > > There hasn't been recent movement on 915738. I'll apply Julian's patch > and see how we go. Huh. There was a comment from doko that the underlying issue is fixed in gcc-8, no? I think you only need to switch to the default gfortran. On the other hand, I don't know how feasible it is at this moment to not release buster with gcc-7. Maybe other release team members can comment on that? Paul signature.asc Description: OpenPGP digital signature
Re: python-scipy: autopkgtest fails (Re: bug#919929)
Hi Drew, On 07-03-2019 13:19, Drew Parsons wrote: >> Can you elaborate why you think that bug should be RC (as that isn't >> clear to me from the report itself) and why you haven't marked it as >> such if you think it should be? > > python-scipy is currently failing all debci tests in both unstable and > testing. autopkgtest only, so no FTBFS? > I haven't marked it RC myself since I'm not 100% certain what the usual > protocol is for marking the severity of debci test failures. But as I > understood it, debci test failures is considered RC under the final > freeze which we're about to enter (but we're not quite in that deep > freeze yet). Some of us want failing autopkgtest to be RC *after* the release of buster. I am not aware of consensus about that yet. autopkgtest *regression* in testing is effectively RC since the soft freeze of 12-2-2019. The autopkgtest of python-scipy is already failing in testing, so it isn't a regression. Hence, it failing is not RC for buster. > Hi Andreas, I may not have been clear. What I mean at this point is to > upload a small patch for scipy 1.1 to ignore the deprecation warnings > (scipy 1.2 already does that). > > If that doesn't help pass the debci tests then we can consider uploading > scipy 1.2 instead. But python-fluids and dipy FTBFS against scipy 1.2 > (a new version of dipy is available which presumeably fixes that) Please *don't* upload scipy 1.2 to fix only this issue (nor for any other issue without discussion first), it's for sure not worth it. >> Slightly depending on the answer above, I'll unblock an upload of >> python-scipy with only that change. > > Certainly please do unblock if the full freeze is already in place. But > my intention was to first upload python-scipy 1.1 with a small patch, > not 1.2 just yet. If you upload now, your package will not migrate to testing before the full freeze becomes effective so it would need an unblock. If you want to fix this issue with the three lines I saw in the bug report, you can go ahead. However, it is probably worth waiting for a resolution of bug 915738 and combine it with that. Paul signature.asc Description: OpenPGP digital signature
Re: python-scipy: autopkgtest fails (Re: bug#919929)
Hi Drew, On 07-03-2019 09:03, Andreas Tille wrote: > On Tue, Mar 05, 2019 at 07:01:54PM +0800, Drew Parsons wrote: >> python-scipy has recently started failing all debci tests in testing and >> unstable, exacerbating the bug report in Bug#919929 [1]. >> >> The failing error is a MemoryError. But understanding the problem is >> hampered by a flood of deprecation warnings, presumably triggered by numpy >> 1.16. scipy 1.2 added instructions to pytest.ini to ignore the warnings. >> >> Bug#919929 has not yet been marked RC but I guess it's about to happen. Can you elaborate why you think that bug should be RC (as that isn't clear to me from the report itself) and why you haven't marked it as such if you think it should be? >> I >> propose in the first instance to apply a patch of the pytest.ini diff >> between scipy 1.1 and 1.2 and see if that clears things up. I'll commit the >> patch now. Should I proceed with upload to unstable? > > May be its sensible to coordinate with debian-release (list in CC) and > file a unblock request *before* uploading. I confirm that I usually > upload simple fixes and ask for unblock afterwards but you intend to do > a version change which does not qualify as simple fix (despite I agree > with you that it is sensible). Slightly depending on the answer above, I'll unblock an upload of python-scipy with only that change. Paul signature.asc Description: OpenPGP digital signature
Re: Numpy migration?
Hi Mattia, all, On 07-01-2019 17:20, Mattia Rizzolo wrote: > On Mon, Jan 07, 2019 at 09:03:11AM +0100, Ole Streicher wrote: >> Mattia Rizzolo writes: >>> On Sun, Jan 06, 2019 at 05:07:41PM +0100, Ole Streicher wrote: Now it turns out that there is a new migration problem, which is aplpy: Current aplpy (2.0~rc2-2) CI test works well >>> >>> You probably mean aplpy 1.1.1-4. >> >> No, I meant the one above (although the unstable test was not done on >> ci.d.n when I wrote my last mail). > > Right, that explains why I didn't see it. > It seems now britney is triggering aplpy test with a big set of > triggers (I suppose the result of the several Breaks that have been > added in the meantime), so it's indeed trying to move everything > together. Indeed. >> The problem is that aplpy uses matplotlib, and the old matplotlib uses >> the deprecated numpy function np.asscalar(), which leads to a >> DeprecationWarning, which is (on purpose, by upstream) thrown as error. >> >> https://ci.debian.net/data/autopkgtest/testing/amd64/a/aplpy/1658948/log.gz >> >> This does only happen in the combination "new numpy + old matplotlib", >> but this is the one that is tested for the CI test. > > mhh… for this, indeed a breaks numpy -> matplotlib could be used, but it > feels a bit too strong to me, as nothing was really broken since it's > just deprecations. > > elbrus; this is a case where I think I could use your input (tl;dr: new > numpy causes deprecations in testing's version of matplotlib (fixed in > unstable), which in turn causes failures in new aplpy, how to make the > stack happy?). The commit, with lots of comments [1], that implemented most of the related logic is here: https://salsa.debian.org/release-team/britney2/commit/18633e275c34973379f1426542047bf30c7829c3 With the current code your options are: - *versioned* depends on one of the binary packages from the source you want from unstable in any of the binary packages that is going to be taken from unstable - *versioned* breaks or conflicts on the binary packages from the source you want from unstable in any of the binary packages that is going to be taken from unstable - *versioned* test dependency in the package that is used for autopkgtest (only helps if the version that is tested is already taken from unstable). The version of the test dependency will NOT be added to the triggers (maybe, I have to think about it, that could be an britney improvement), but if the version of the autopktest is going to be taken from unstable already due to other considerations, with the current settings of ci.d.n and autopkgtest the required version will be installed. Personally I don't think it is an issue to add the breaks, but I can see why people find it heavy in cases like this one. As discussed in the med-fichier bug, britney may want to grow knowledge from a X-breaks-autopkgtest-of-source header, but that doesn't exist now and I am not extremely keen about adding it because I fear people may too easily add it when a normal breaks is in order. Paul [1] +# Here we figure out what is required from the source suite +# for the test to install successfully. +# +# Loop over all binary packages from trigger and +# recursively look up which *versioned* dependencies are +# only satisfied in the source suite. +# +# For all binaries found, look up which packages they +# break/conflict with in the target suite, but not in the +# source suite. The main reason to do this is to cover test +# dependencies, so we will check Testsuite-Triggers as +# well. +# +# OI: do we need to do the first check in a smart way +# (i.e. only for the packages that are actully going to be +# installed) for the breaks/conflicts set as well, i.e. do +# we need to check if any of the packages that we now +# enforce being from the source suite, actually have new +# versioned depends and new breaks/conflicts. +# +# For all binaries found, add the set of unique source +# packages to the list of triggers. signature.asc Description: OpenPGP digital signature
