Re: Bug#1043240: transition: pandas 1.5 -> 2.1 - please upload fixes
I intend to upload pandas 2.x to unstable soon. These packages have a patch in their bug - please upload them (I'm a DM, I can't do that), or if you think this patch won't work or isn't a good idea, tell me why: dials influxdb-python python-altair python-feather-format seaborn tqdm In particular, I'd like the seaborn fix uploaded before pandas, so I can set Breaks for it. (The pandas documentation build-depends on seaborn.)
Re: Bug#1043240: transition: pandas 1.5 -> 2.1
Control: severity 1053943 1053939 1053942 1044053 1044056 serious Control: severity 1044074 1053946 1044078 1044079 1044077 serious Control: severity 1044071 1044067 1044068 1044055 1044060 serious Control: severity 1044072 1044073 1044064 1053945 1044054 serious Control: severity 1044076 1053940 1044057 1053944 1050144 serious As previously discussed in this bug, I'd like to move pandas 2.x into unstable reasonably soon. I'm aiming to get it in before the Ubuntu 24.04 freeze (in about a month), but I am open to disagreement on whether this is a good idea. dask, python-skbio and python-upsetplot have been fixed since the previous discussion, but that still leaves the above 25. 6 of these have a known-to-me fix (dials influxdb-python python-altair python-feather-format seaborn tqdm - see their bugs for details). Hence, doing this transition now would involve breaking some reverse dependencies with no known fix, but given the number of packages involved, trying to wait until they're all fixed is rather likely to instead end in pandas 1.5 being broken by a new Python/numpy/etc.
Bug#1060374: RM: libgpuarray - RoM; abandoned upstream, RC-buggy
Package: ftp.debian.org User: ftp.debian@packages.debian.org Usertags: remove libgpuarray has been de facto abandoned upstream since 2018, and is unused in Debian since theano was removed (#1042574). It currently has what may or may not be a wrong-answer bug (#1031414), and two broken-by-transition bugs. (The latter two _might_ be easy to fix - I haven't tried because I've already mostly given up on it.) Removal was previously suggested in those bugs, without response.
Proposed RM: libgpuarray - abandoned, RC-buggy
libgpuarray has been de facto abandoned upstream since 2018, and is currently unused in Debian. (Its original user was theano, which was removed in #1042574.) It currently has what may or may not be a wrong-answer bug (#1031414), and two broken-by-transition bugs. (The latter two _might_ be easy to fix - I haven't tried because I've already mostly given up on it.) Hence, I intend to request removal.
Bug#1043240: transition: pandas 1.5 -> 2.1
I'd like to move forward with the pandas 1.5 -> 2.1 transition reasonably soon. Given that pandas 2.x is *not* required for Python 3.12 (but is required for Cython 3.0), should we wait for the Python 3.12 transition to be done first? These are broken by pandas 2.x and have a possible (but untested) fix in their bug - please test and apply it: dask(?) dials influxdb-python* python-altair python-feather-format python-upsetplot seaborn tqdm* (* = this package is currently also broken for a non-pandas reason, probably Python 3.12, that I don't have a fix for) These are broken by pandas 2.x and have no known-to-me fix: augur cnvkit dyda emperor esda mirtop pymatgen pyranges python-anndata python-biom-format python-cooler python-nanoget python-skbio python-ulmo q2-quality-control q2-demux q2-taxa q2-types q2templates sklearn-pandas Some generic things to try are pandas.util.testing -> pandas.testing, .iteritems() -> .items(), and if one exists, a more recent upstream version. Is this an acceptable amount of breakage or should we continue to wait? Bear in mind that if we wait too long, we may be forced into it by some transition further up the stack (e.g. a future Python or numpy) that breaks pandas 1.x. Build logs: https://launchpad.net/~rebecca-palmer/+archive/ubuntu/pandas2p1/+builds?build_text=_state=failed https://launchpad.net/~rebecca-palmer/+archive/ubuntu/pandas2p1n/+builds?build_text=_state=failed (The second is more recent, but includes fewer packages.) Autopkgtest logs: https://qa.debian.org/excuses.php?experimental=1=pandas (Because of the Python 3.12 transition, this may currently be wrong about what is a regression and what is not.)
RFS: enabling openpyxl documentation
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Please upload openpyxl 3.1.2+dfsg-3 to unstable (Salsa HEAD = commit ea63a153033c7d6ed426e4931d952d027ce6b855). This enables the documentation, and places it in a new binary package because of its size, which I can't do myself because DMs can't upload NEW packages. -BEGIN PGP SIGNATURE- iQIzBAEBCgAdFiEEZ8sxEAXE7b4yF1MI3uUNDVZ+omYFAmVlCv8ACgkQ3uUNDVZ+ omb4Jw//V/uYocexMQYiGVC8oSMbNC/2ssx2GpNr2XbqKtyY6Z9qhYkeYWKCdFk5 N+g7uhBygWxNj7mle6MGt0lDr//cs79HsVtoXhOzS91xgDJpBvUWWM4NbRSGh7gQ KGRHTU4e3+7N/qQq/BHykUJGPgE6rDuyspCJ8MBkY4J1SbucYyBG0Qtd9zwPiy7X HpBy9b/4gdFJJOWGJnTrfiiOHgHrYVgRfGOoxgTMyYRBcLtsla9n8Wd2cmpWfdJ3 SakVdS9Eeeg3mFRnjQ0mozlLIr0lO2yl5ULAoioLbVG5lDliGB+B/AbMxePfmKiE jnHO5q+10SWOAXwSCraG9wvG/zBe1EXVT3tYP5emrvL0SK7Oa5xR3vU+phfy+8id KPXHDIpKc7FQ6dmJbAOMKuhdMD2BbRQPsGp61lrgxAk/j+CGN63zsZusTjswOtii +aiEnd0KEH81EkXxn5mk8rPOMFqMfHkgHI6hePzNgErfOiigSNaJHok/q/sZdjv3 u3t3TfSLv3CFEqgX43nyEODGzPaombwQln5m2NMwWiaYzJFBn5ynQyQ8RaRrNrDK Nl4sh7of8NsBSDIOiD+Trl/pNfxOeANIiUJKYUsCoIJ3eIFx9syud8Ax1759/aXk XP1Q/zkXOm/WGtE/DLtNObRvW4AxB76geji3ja7kgxH63ZTx7f4= =T4Fn -END PGP SIGNATURE-
Re: Offer to take over openpyxl / request for DM permissions
Thanks. (Please do _not_ immediately upload what I just pushed to Salsa, it still needs more testing.)
Offer to take over openpyxl / request for DM permissions
The version of openpyxl currently in Debian is over a year behind upstream, and this is blocking the pandas 2.x transition. I previously reported this as #1051233, to no response. (The package has been uploaded more recently than that, but keeping the same upstream version; as noted in that bug, it's unclear why.) Hence, if nobody objects I intend to take over maintenance of this package. I will need either DM permissions or sponsorship to upload it.
Bug#1042574: RM: theano, keras, deepnano -- RoM (theano), broken by numpy 1.24, theano mostly abandoned upstream
Package: ftp.debian.org User: ftp.debian@packages.debian.org Usertags: remove Control: affects -1 src:theano src:deepnano src:keras Control: reopen 1026539 Control: reopen 1027215 theano has been mostly abandoned upstream since 2018. (The Aesara fork is not abandoned, but includes interface changes including the import name, so would break reverse dependencies not specifically altered to use it. Neither of the two in Debian are so altered.) Since numpy 1.24, theano has been completely broken (won't even import). Some parts of this are fixable (#1033589), but other parts have no known fix (#1027215). Note that the "OK" status on Salsa CI is *not* an actual fix, but is because theano skips most of its tests in Salsa CI because they take several hours. theano's reverse dependencies (keras and deepnano) are both also broken by this. keras _also_ has apparently unrelated problems, #1026738, and is orphaned, #1027938. This was previously discussed in #1027215 (on, among others, deepnano's team list), where it was noted that removing keras would also block the addition of qmean (ITP #976981), but attempts to fix theano failed.
Re: Bug#1027215: Bug#1026539: How much do we lose if we remove theano (+keras, deepnano, invesalius)?
On 03/03/2023 06:00, Andreas Tille wrote: Ahh, so aesara is not really a "fork" but a "rename"? The original is abandoned (no new development since 2017, and now mostly unmaintained, which is probably why it has this kind of bug). Aesara is a continuation by a new upstream (possibly one that mostly wanted it for their own use), that chose to break compatibility.
Re: Bug#1027215: Bug#1026539: How much do we lose if we remove theano (+keras, deepnano, invesalius)?
On 02/03/2023 10:38, Andreas Tille wrote: I admit I do not see any good reason to stick to the old version if we decided before that keras/deepnano are no real blockers to even drop theano. Thus I was considering it more promising to spent my time on the latest version. 1.1.2 isn't the latest version, just the latest that calls the module theano rather than aesara. (I chose it to package because I didn't want to fully break compatibility, then abandoned it because I didn't like how much it did break.) Do you want to ask release team for permission to do this? If it would have build smoothly on all architectures, yes. If you wouldn't want to do this work if it can't get into bookworm (possibly because you'd prefer to package 2.0 as aesara if you have to wait until trixie anyway), you might want to ask them first. But we have the first trouble on arm64[1] Sorry - I forgot that theano skips most of its tests on Salsa-CI (because there's enough of them to take several hours), which is probably why it "passed" that but failed the full build. Given how many failures there are (everywhere), I don't consider this worth fixing, but you are of course free to disagree. (If you're planning to xfail tests, note that I consider *wrong-answer* bugs (though not crash bugs) in this kind of package to be RC. I haven't checked whether we currently have any.)
Re: Bug#1026539: How much do we lose if we remove theano (+keras, deepnano, invesalius)?
I agree that switching to Aesara is probably the only reasonable option other than removal. (I'd given up on trying to fix 1.0, and was intending to let removal happen.) However, it's a much bigger change than is normally allowed in bookworm at this point. (1.1 includes multiple breaking changes, which is why it's in experimental, but a quick codesearch suggests these parts *may* not be used in keras/deepnano. https://github.com/aesara-devs/aesara/releases?page=8 ) Do you want to ask release team for permission to do this? Or do you want to try the same patches on 1.0? (I suspect that that won't work, but I haven't actually tried it.) (Also, you might not want numpy1p24_compat.patch - the v1p0 branch is currently in whatever state it was in when I gave up on it, and my vague memory is that this was a failed experiment, though I don't know if that meant "actively bad" or just "not a (full) solution".)
scikit-learn Re: Are we still trying to do scipy 1.10, given transition freeze?
Control: tags 1029701 fixed-upstream Full log: https://ci.debian.net/data/autopkgtest/unstable/amd64/s/scikit-learn/30693526/log.gz There appear to be at least 2 separate failures here, both known and probably fixed upstream. So yes, 'new upstream version' is the first thing to try, but we'll need to check what _that_ breaks. https://github.com/scikit-learn/scikit-learn/issues/23626 https://github.com/scikit-learn/scikit-learn/issues/24424
Re: Bug#1022571: transition: pandas 1.3 -> 1.5
The remaining autopkgtest failures blocking pandas are: - dipy/amd64 and snakemake/i386 look like random flakiness: please retry them. (I think DMs can't use the self-service interface.) Or for dipy, see #1029533 for a possible fix. - python-xarray/i386 looks like xarray not pandas: see #1004869. - dask and dask.distributed have to go together, with or before pandas.
Are we still trying to do scipy 1.10, given transition freeze?
Control: tags -1 fixed-upstream patch Control: forwarded -1 https://github.com/scipy/scipy/pull/17033 This "broken by numpy 1.24" bug looks to me like 17033, not 17630. This has what looks like a trivially-backportable patch, though I haven't actually tried that: https://github.com/scipy/scipy/pull/17035 Given that the transition freeze was on 2023-01-12, it's too late to move to scipy 1.10 if that's likely to cause significant breakage. (Do we have a guess of whether it will, or do we need a package that builds in experimental to test that?) The scipy 1.10 FTBFS in salsa ( https://salsa.debian.org/python-team/packages/scipy/-/pipelines/486634 ) is it trying to download test data, which isn't allowed for a build-time test. We could make those tests autopkgtest-only, or if the data is reasonably sized, add it to the package: Pooch can use a local cache. https://sources.debian.org/src/pooch/1.6.0-2/README.rst/
How much do we lose if we remove theano (+keras, deepnano, invesalius)?
theano has been mostly abandoned upstream since 2018. (The Aesara fork is not abandoned, but includes interface changes including the import name, so would break reverse dependencies not specifically altered for it.) Its reverse dependencies are keras, deepnano and invesalius. It is currently broken, probably by numpy 1.24 (#1027215), and the immediately obvious fixes weren't enough (https://salsa.debian.org/science-team/theano/-/pipelines). Is this worth spending more effort on fixing, or should we just remove it?
Re: Bug#1022571: transition: pandas 1.3 -> 1.5
skbio is fixed, ulmo is maybe fixed but untested. dask is unclear (see #1027254 for details): it looks like it's probably fixable, but I don't have an actual working fix yet. There has been no response from its maintainers.
Re: transition: pandas 1.3 -> 1.5
Should this go ahead before the freeze? I think yes if the new dask works, but am open to disagreement. Issues this would fix: python3.11 #1023965: We're currently ignoring these test failures. Mystery autopkgtest failure (fails without any listed individual test failure, started 2022-11-13) https://ci.debian.net/packages/p/pandas/unstable/amd64/ matplotlib 3.6 (not reported yet): test failures (The fixes in Salsa for #1026351 and #1027576 would probably work on either pandas version.) Reverse dependencies that would break: dask #1025393: Probably fixed upstream, but I haven't checked either whether that works or whether it breaks anything else; I'm working on this. ulmo #1017573: Looks easy to fix but I haven't tried yet. skbio #1017574 emperor #1024820: Suggested there that we should ignore this.
Re: pandas 1.3.5+dfsg-2 autopkgtest failures
pandas has now migrated (on both Debian and Ubuntu). I have uploaded versions of statsmodels and snakemake that should (almost always) avoid these test failures, and reported the partd failure as #1005045.
pandas 1.3.5+dfsg-2 autopkgtest failures
In Debian (only change from -1 should be disabling some tests, i.e. real pandas-related failures are unlikely): mdtraj/amd64: upstream tests pass but a warning is printed to stderr, which autopkgtest counts as a fail partd/ppc64el: save/load of a plain string fails snakemake/i386: hang in test_pipes_fail statsmodels/armhf: wrong answer in TestDFMComplex These have all happened before, suggesting they're random, i.e. bugs in these packages not in pandas. (For partd, the other known instance was on a different architecture (s390x), but it looks like partd only uses pandas when it's handling pandas objects, which this test isn't.) In Ubuntu (their first build of 1.3.x, i.e. real pandas-related failures are more plausible): poliastro, python-xarray: These fail with the same error with either new pandas or new matplotlib, and have not had a reference run since then. I suspect they'd now fail every time. python-anndata: Known in Debian as #1001349, fixed there but the fix fails to build in Ubuntu.
Re: Bug#999415: transition: pandas 1.1 -> 1.3 - to unstable now or not?
After build-testing about half of the reverse dependencies, failures that look new-pandas-related are cfgrib #1000726, joypy #1000727, python-skbio #1000752, and maybe hyperspy (not filed yet). python-skbio and hyperspy already FTBFS for unrelated reasons (but fail more tests with new pandas), and joypy looks trivially fixable. Given this and expecting to find a similar number in the other half, against pandas 1.3 working on python3.10 while 1.1 doesn't (#1000422), would you prefer to have pandas 1.3 in unstable now, or not?
Re: Bug#999415: transition: pandas 1.1 -> 1.3
Control: block 996584 by -1 On 21/11/2021 06:30, Graham Inggs wrote: It might be easier to upgrade to 1.2.4 first, then 1,3,x later. Probably not, given that we're now trying to add Python 3.10 support (#996584), and pandas upstream's work on that is recent (maybe still unfinished): https://github.com/pandas-dev/pandas/issues/41940 The 1.3.x currently in Salsa built last time I tried (before Python 3.10), but is (intentionally, to avoid FTBFS) missing some documentation, and I haven't tested the reverse dependencies yet.
Re: transition: pandas 1.1 -> 1.3
I was already planning to try a build+autopkgtest of the reverse dependencies, and probably an upload to experimental (it's also common for pandas to fail a few tests on non-amd64), at some point, but not just yet. (The last time I did this was #969650.) Thanks for the tools suggestions. I am currently working on pandas itself: I suspect the documentation isn't the only issue.
transition: pandas 1.1 -> 1.3
Source: pandas Version: 1.1.5+dfsg-2 Severity: wishlist On 10/11/2021 17:14, Andreas Tille wrote: pandas is lagging behind upstream by several versions. I guess we should try to get in sync with upstream a bit more. Yes, but please don't upload this yet: it's common for a pandas upgrade to break reverse dependencies. The package does not build yet since the documentation needs sphinx_panels. On 10/11/2021 18:32, Timo Röhling wrote> Maybe, as a compromise, we can cut out all the notebooks^H bells and whistles and limit the offline documentation to the API reference itself, which is arguably the most useful part? I don't know how easy it is to trim down the documentation, though. The pandas (and probably statsmodels) documentation already is modified to build with fewer dependencies - see sphinx_no_pandas_theme.patch.
Re: Should the pandas 1.x transition be forced (breaking python-biom-format and q2-demux/q2-types) or keep waiting?
Second attempt uploaded. Of the reverse dependencies with known fixes, python-feather-format has been uploaded but q2templates hasn't.
Re: Should the pandas 1.x transition be forced (breaking python-biom-format and q2-demux/q2-types) or keep waiting?
Package FTBFS everywhere, for multiple reasons but looks like the new pytest_asyncio is at least one of them; will investigate further later.
Re: Should the pandas 1.x transition be forced (breaking python-biom-format and q2-demux/q2-types) or keep waiting?
Not yet, it should at least be 1.0.5 from Salsa. (Also, q2-* have autopkgtests so it will probably get stuck in -proposed.) On 25/08/2020 08:19, Graham Inggs wrote: Hi Rebecca On Mon, 24 Aug 2020 at 01:03, Rebecca N. Palmer wrote: pandas 1.0 has been in experimental for 6+ months, because it breaks some of its reverse dependencies: ... Ubuntu freezes this Thursday, but I suspect I may have left this too late for that. We can sync 1.0.4+dfsg-1 from experimental to Ubuntu right now if you want. Regards Graham
Should the pandas 1.x transition be forced (breaking python-biom-format and q2-demux/q2-types) or keep waiting?
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=950430 pandas 1.0 has been in experimental for 6+ months, because it breaks some of its reverse dependencies: In testing: #950924 python-feather-format (has patch, no maintainer response to it) #950931 q2templates (has upstream fix) #950929 python-biom-format (no known solution) Not in testing: #950932 q2-types #950933 q2-demux This is blocking at least one proposed new package (pymatgen #962268), and making work when a new numpy/etc breaks the old pandas. On the other hand, this year feels like a bad time to break bioinformatics packages. Ubuntu freezes this Thursday, but I suspect I may have left this too late for that. Upstream have now released pandas 1.1, and I might go straight to that but first need to check whether it breaks anything more. (It isn't supposed to have API changes, but the new warnings may break fail-on-warning tests.)
Re: Becoming a package maintainer and best practices for porting RPMs from Fedora
On 08/06/2020 21:04, Qianqian Fang wrote: from browsing some of the existing repos, it appears that the package repo contains all upstream source files (with the addition of the debian/ folder), which is different from Fedora (upstream source package is separate from the .spec and patch* files). Either including or not including the upstream files is allowed, but including them is usually preferred, unless they are very large. If they are included, the packaging repository should have a branch with upstream files only, and a branch with upstream+debian. The upstream branch can be tarball imports (one commit per release) or a clone of the upstream repository (detailed history). (And yes, it is a known issue that having so many choices can be confusing, particularly to beginners.) The git-buildpackage tool can help you with this. if I am willing to be the maintainer of my own packages, do I still need to find a package sponsor (given I am not a maintainer yet)? Yes: the access levels are sponsored maintainer (named in Maintainer: or Uploader: of a debian/control but can't upload) -> Debian Maintainer (can upload their own packages) -> Debian Developer (can upload any package, including as a sponsor) https://wiki.debian.org/DebianMaintainer from the above link, one of the paths is to "Join a packaging team", for example, how do I "join" debian-science or DebianMed? https://salsa.debian.org/groups/science-team/-/group_members/request_access 4. submit an ITP bug Where possible, this should be the first step; otherwise correct. sorry, forgive me for asking these basic questions. It's arguably a bug that this information is so hard to find.
Re: Becoming a package maintainer and best practices for porting RPMs from Fedora
On 08/06/2020 18:35, Qianqian Fang wrote: I picked one of my projects (https://github.com/fangq/zmat) to get started, currently, I managed to create all basic packaging files (3 subpackages, zmat/zmat-dev/octave-zmat) https://github.com/fangq/debpkg/tree/zmat This is a library, so the binary package names should probably be libzmat0/libzmat-dev/octave-zmat. I have not checked the rest of the package. It's generally recommended, and in some teams required, that packaging repositories be hosted on https://salsa.debian.org, not with upstream. I have a couple of questions regarding finishing up the package, including the appropriate octave packaging commands, file permissions etc, but I am not sure if I should post those here or to other lists, debian-mentor for example. debian-mentors is an appropriate place for general or C/C++ packaging questions. Language team lists (e.g. debian-octave) may be an appropriate place for questions related to that programming language. Large teams often have separate "human discussion" (@lists.debian.org) and "package Maintainer: = computer-generated notices" (@lists.alioth.debian.org) lists. You usually *can* post to Debian lists you aren't subscribed to, but it is then up to you to check the web archives (https://lists.debian.org/completeindex.html) if you want a reply. Moving forward from here, I would like to know 1. is there a package review process? how do I initiate it? Yes, and probably by posting to the team's mailing list: https://mentors.debian.net/sponsors (As you have a Git repository, you probably don't need to actually upload to mentors.debian.net - I never did.) 2. do I need to create a bug request on prospective packages (https://www.debian.org/devel/wnpp/) ? Yes, an ITP bug: see https://www.debian.org/devel/wnpp/ for how to do this.
Re: Becoming a package maintainer and best practices for porting RPMs from Fedora
Thank you for offering to contribute. Qianqian Fang wrote: 1. is there an established procedure to convert an official Fedora RPM to a deb package? Not that I'm aware of. (The package 'alien' can install RPM binary packages on Debian, but does not convert source packages.) any best practices guide for creating a package? any links/steps on how to become a maintainer would be fantastic. Any one willing to mentor me would be great! (Non-trust warning: wiki.debian.org is an anyone-can-edit site) General (possibly out of date): https://wiki.debian.org/Packaging/Intro https://www.debian.org/doc/manuals/maint-guide/ https://www.debian.org/devel/ Relevant teams: https://wiki.debian.org/DebianScience https://wiki.debian.org/Teams/DebianOctaveGroup https://neuro.debian.net/ (possibly inactive)
Re: license problem about nvidia's neural network library (cuDNN)
Previous discussions: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=862524 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=887728
Re: using package tests
If upstream provide a test script, you can use "Test-Command" to call it from autopkgtest. Daniel Leidert wrote: However there is a restriction called 'build-needed' which I guess is for cases where you rely on something inside the source. Tests always have the source available, and start in the directory containing it ([0] line 28). "build-needed" builds the source before running the tests, to also allow access to test executables built by the source package but not installed into its binary package(s) ([0] line 211). Depending on the tests run it is possible that tests rely on files inside the source and not the installed package (that's why e.g. the defaukt ruby tests move the lib/ dorectofry in the source away during the tests). It is OK to load the tests themselves from the source tree, but the program/library under test should be loaded from the installed package, not the source ([0] line 30). Some interpreted languages (e.g. Python [1]) search the current directory before the system modules directory, and hence default to loading the module under test from source. One way to avoid this is to change directory to $AUTOPKGTEST_TMP before starting the test [2]. (The source tree is read-only ([0] line 32), so the files Ruby moves [3] probably aren't it.) [0] https://sources.debian.org/src/autopkgtest/5.12.1/doc/README.package-tests.rst [1] https://docs.python.org/3/library/sys.html#sys.path [2] https://sources.debian.org/src/autodep8/0.22/support/python/generate/#L71 [3] https://sources.debian.org/src/gem2deb/1.0.5/lib/gem2deb/test_runner.rb/#L182
State of shiny-server
shiny-server has been mentioned as a potential covid-19 related package [0], though it isn't on the current hackathon list [4]. There is a packaging attempt in science-team Salsa (but no formal ITP) from early 2018. Discussion at the time suggests it builds but possibly doesn't work [1], and was abandoned because neither the science team nor the Javascript team wanted to take responsibility for its Javascript dependencies [2-3]. It appears to have 9 such dependencies that are not already packaged, plus some that are but in the wrong version: see list below. [0] https://salsa.debian.org/blends-team/med/-/blob/master/tasks/covid-19#L80 [1] https://lists.debian.org/debian-science/2018/01/msg00141.html [2] https://lists.debian.org/debian-science/2018/02/msg00095.html [3] https://lists.debian.org/debian-science/2018/09/msg00036.html [4] https://salsa.debian.org/med-team/community/2020-covid19-hackathon/-/wikis/COVID-19-Hackathon-packages-needing-work $ npm2deb depends -r https://github.com/rstudio/shiny-server/raw/master/package.json # plus manual editing Dependencies: NPM Debian shiny-server (1.5.13)None ├─ bash (0.0.1) None ├─ client-sessions (^0.8.0) None (RFP #896975) │ └─ cookies (^0.7.0) node-cookies (0.8.0-2) ├─ compression (^1.7.4) node-compression (1.7.4-2) ├─ express (^4.16.4) node-express (4.17.1-2) ├─ faye-websocket (^0.11.3) too old node-faye-websocket (0.11.1-1) ├─ graceful-fs (^4.1.15) node-graceful-fs (4.2.3-2) ├─ handlebars (^4.5.3) node-handlebars (3:4.7.2-1) ├─ http-proxy (^1.17.0) None (RFP #896978) │ ├─ eventemitter3 (^4.0.0) None │ ├─ follow-redirects (^1.0.0) node-follow-redirects (1.2.4-1) │ └─ requires-port (^1.0.0) node-requires-port (1.0.0-1) ├─ ip-address (^5.9.0) None │ ├─ jsbn (1.1.0) node-jsbn (1.1.0-1) │ ├─ lodash.find (4.6.0)node-lodash (4.17.15+dfsg-2) │ ├─ lodash.max (4.0.1) node-lodash (4.17.15+dfsg-2) │ ├─ lodash.merge (4.6.2) node-lodash (4.17.15+dfsg-2) │ ├─ lodash.padstart (4.6.1)node-lodash (4.17.15+dfsg-2) │ ├─ lodash.repeat (4.1.0) node-lodash (4.17.15+dfsg-2) │ └─ sprintf-js (1.1.2) node-sprintf-js (1.1.2+ds1-1) ├─ log4js (^4.1.1) node-log4js (6.1.0-1) ├─ moment (^2.24.0) node-moment (2.24.0+ds-2) ├─ morgan (^1.9.1) None │ ├─ basic-auth (~2.0.1)None │ │ └─ safe-buffer (5.1.2) node-safe-buffer (5.2.0-1) │ ├─ debug (2.6.9) too new node-debug (4.1.1-2) │ ├─ depd (~2.0.0) node-depd (2.0.0-1) │ ├─ on-finished (~2.3.0) node-on-finished (2.3.0-1) │ └─ on-headers (~1.0.2)node-on-headers (1.0.2-1) ├─ nan (^2.14.0) node-nan (2.14.0-1) ├─ optimist (0.6.1) node-optimist (0.6.1-1) ├─ pause (0.1.0) None ├─ q (^1.5.1)node-q (1.5.1-2) ├─ qs (^6.7.0) node-qs (6.9.1+ds-1) ├─ send (^0.17.0)node-send (0.17.1-2) ├─ shiny-server-client (https://github.com/rstudio/shiny-server-client/archive/fb1aef1.tar.gz)node-shiny-server-client (1.0.0+git20180820.eba5e90+dfsg-2) ├─ sockjs (^0.3.19) too old sockjs-client (0.3.4+dfsg-2) │ ├─ faye-websocket (^0.10.0) node-faye-websocket (0.11.1-1) │ ├─ uuid (^3.4.0) node-uuid (3.3.2-2) │ └─ websocket-driver (0.6.5) too old node-websocket-driver (0.3.5-1) ├─ split (^1.0.1)too old node-split (1.0.0-1) ├─ stable (^0.1.8) None (see below) └─ underscore (^1.9.1) underscore (1.9.1~dfsg-1) Build dependencies: NPM Debian mocha (^6.1.4) too new node-mocha (7.0.1+ds1-2) rewire (^4.0.1) None should (^13.2.3) should.js (13.2.3~dfsg-3) sinon (^7.3.2) too new node-sinon (9.0.1+ds-1) Warnings occurred: [warning] stable: stable is included in node-svgo. Package it separately and remove it from node-svgo if you need it for another module.
Re: Missing dependancies for streamlit
However, streamlit also has a Javascript component (the frontend directory), with ~100 dependencies Debian doesn't have (plus a few that Debian has in too old a version): $ npm2deb depends /home/rnpalmer/Debian/builds/stackbuild/package.json Dependencies: NPM Debian streamlit-browser (0.56.0)None -- @loaders.gl/core (^2.0.2) None -- @loaders.gl/csv (^2.0.2) None -- aws-sdk (^2.524.0) None -- axios (^0.19.0)node-axios (0.19.0+dfsg-2) -- baseui (^8.17.1) None -- bokehjs (^1.2.0) None -- bootstrap (^4.3.1) None -- camelcase (^5.2.0) node-camelcase (5.3.1-1) -- classnames (^2.2.6)None -- clipboard (^2.0.4) node-clipboard (2.0.6+ds-1) -- d3 (^5.12.0) node-d3 (5.12.0-2) -- d3-graphviz (^2.6.1) None -- deck.gl (^8.0.12) None -- hoist-non-react-statics (^3.3.1) None -- immutable (^4.0.0-rc.12) node-immutable (3.8.2+dfsg-3) -- katex (^0.11.1)node-katex (0.8.3+dfsg-1) -- leaflet (^1.3.4) leaflet (1.6.0~dfsg-1) -- lodash (^4.17.15) node-lodash (4.17.15+dfsg-2) -- mapbox-gl (^1.7.0) None -- moment (^2.22.2) node-moment (2.24.0+ds-2) -- moment-duration-format (^2.3.2)None -- node-sass (^4.12.0)node-node-sass (4.13.1-3) -- numbro (^2.1.2)None -- plotly.js (^1.51.2)None -- prismjs (^1.15.0) node-prismjs (1.11.0+dfsg-3) -- protobufjs (^6.8.8)None -- react (^16.8.6)node-react (16.2.0-4) -- react-copy-to-clipboard (^5.0.1) None -- react-dom (^16.8.4)None -- react-feather (^2.0.3) None -- react-google-login (^5.0.4)None -- react-hotkeys (^1.1.4) None -- react-json-view (^1.19.1) None -- react-katex (^2.0.2) None -- react-leaflet (^2.4.0) None -- react-map-gl (^5.2.1) None -- react-markdown (^4.2.2)None -- react-plotly.js (^2.4.0) None -- react-scripts (^3.0.1) None -- react-test-renderer (^16.10.2) None -- react-transition-group (^4.3.0)None -- react-virtualized (^9.21.0)None -- reactstrap (^8.0.1)None -- recharts (^1.6.2) None -- remark-emoji (^2.0.2) None -- remark-math (^2.0.0) None -- sprintf-js (^1.1.2)node-sprintf-js (1.1.2+ds1-1) -- styletron-engine-atomic (^1.4.1) None -- styletron-react (^5.2.1) None -- typed-signals (^1.0.5) None -- vega (^5.7.3) None -- vega-embed (^6.0.0)None -- vega-lite (^4.0.0-beta.10) None -- which-browser (^0.5.1) None -- xxhashjs (^0.2.2) None Build dependencies: NPM Debian @craco/craco (^5.5.0) None @deck.gl/json (^8.0.5)None @types/classnames (^2.2.9)None @types/clipboard (^2.0.1) None @types/d3 (^5.7.2)None @types/d3-graphviz (^2.6.2) None @types/dom-mediacapture-record (^1.0.2) None @types/enzyme (^3.10.3) None @types/fetch-mock (^7.3.1)None @types/hoist-non-react-statics (^3.3.1) None @types/jest (^24.0.11)None @types/katex (^0.10.2)None @types/lodash (^4.14.138) None @types/mocha (^5.2.7) None @types/moment (^2.13.0) None @types/moment-duration-format (^2.2.2)None @types/node (^12.7.4) None @types/numeral (^0.0.26) None @types/plotly.js (^1.44.8)None @types/prismjs (^1.16.0) None @types/react (^16.9.17) None @types/react-copy-to-clipboard (^4.2.6) None @types/react-dom (^16.8.4)
Re: Missing dependancies for streamlit
From searching the source, it appears that only 2 tests (plus some examples) need tensorflow: hashing_test.py:HashTest.test_tensorflow_non_hashable and keras_test.py:KerasTest. Hence, skipping these tests may be a reasonable option after all.
Re: Missing dependancies for streamlit
From #954150, it appears there _is_ a code vs data distinction: tests are allowed to use the network but not to download and run code. https://ftp-master.debian.org/REJECT-FAQ.html Non-Main II I intend to post a patch to autopkgtest to document this. Manually running the tests in question before upload is still an option, but is (a) easy to forget and (b) won't notice breakage due to changes in other packages. (Debian has transition rules for testing API/ABI-breaking changes before they reach unstable, but these in practice apply only to compiled languages, not to Python modules.)
Re: Missing dependancies for streamlit
Internet-using tests are discouraged because they can fail for external reasons, but preferred to leaving online features untested: https://sources.debian.org/src/autopkgtest/5.12.1/doc/README.package-tests.rst/#L431 (Not sure if there are any rules about downloading code vs data) There is a proposal to require such tests to explicitly declare it, but not to ban them altogether: https://lists.debian.org/debian-ci/2020/04/msg00013.html
Re: Missing dependancies for streamlit
A potential workaround for testing: while package builds (including build-time tests) are not allowed to use the network, autopkgtests *are* allowed to => skip the tensorflow-needing tests in build, but run them (with tensorflow from PyPI) in autopkgtest?
Re: unanswered merge requests
Andreas Tille wrote: I was just astonished about the long list of unattended merge requests in Debian Science team. I wonder whether we should somehow prevent people from dumping code there which will simply bitrot since nobody realy cares. Salsa allows users to request notifications of merge requests [0], or project admins to disable merge requests [1]. This was previously discussed on d-d - Sam Hartman wrote (as part of a list of recommended-but-not-required practices) [2]: You should make sure that at least one person sets their notification level to 'watch' rather than global. This way you will receive notifications of merge requests and changes. Hopefully you will choose to monitor merge requests for your repository. If not, turn off merge requests. If you enable merge requests, you should be as responsive to merge requests as you are patches in the BTS. [0] https://salsa.debian.org/help/user/profile/notifications.md [1] https://salsa.debian.org/help/user/project/settings/index.md#sharing-and-permissions [2] https://lists.debian.org/debian-devel/2019/10/msg00107.html
libgcc_s breakage Re: Bug#949187: transition: python3.8
Matthias Klose wrote: On 2/3/20 8:22 PM, Simon McVittie wrote: >> Meanwhile, multiple packages seem to FTBFS on s390x with the new libgcc_s (I've just opened the bug for that, so no bug number known yet), which is going to limit the ability to get things into testing. please retry your package builds. it's fixed in unstable. Do you mean #950525? If so, you might need to wait a few hours because the fix (gcc-9 9.2.1-28) is still being built. When it is ready, please retry pandas and statsmodels in experimental. (DMs can't do self-service give-backs.)
Re: Bug#945389: Tried to upgrade skimage to see whether #945389 is fixed but failed
Stefan van der Walt wrote: One possibility is that NumPy was compiled with an older version of Cython:[...] Do you know how I can check the source files for the NumPy 1.17 build that is used here? Source: https://sources.debian.org/src/numpy/1:1.17.4-3/ Build log: https://buildd.debian.org/status/package.php?p=numpy numpy was built with cython3 0.29.14-0.1. (Not +b1, but I _think_ "Add Python3.8 as supported version" means the ability to run cython3 itself in python3.8, not the ability to compile modules for python3.8.)
Re: Autopkgtest failure in pytables
That looks like a "not built for Python 3.8" error. Does the -lib package contain both Python 3.7 and 3.8 C extensions (both /usr/lib/python3/dist-packages/tables/utilsextension.cpython-38m-x86_64-linux-gnu.so and /usr/lib/python3/dist-packages/tables/utilsextension.cpython-37m-x86_64-linux-gnu.so - note -38m- vs -37m-)? If not, check the build log for attempts to build them, and check whether your build chroot was up to date. The last (failed) build log of the existing 3.5.2-3 suggests it does build them, but it doesn't get far enough to show whether they were installed into the final package.
Re: Adding Python 3.8 as a supported Python3 version
On 18/11/2019 07:07, Matthias Klose wrote: On 18.11.19 07:52, Alastair McKinstry wrote: So I've uploaded a python-xarray 0.14.0-2 which passes on python3.7. Its failing on python 3.8 but apparently due to pandas not ready on 3.8, so it should be ok to pass as the transition completes. I think that error is just pandas *not being built* for 3.8 yet: a Python version transition is ~500 binNMUs and that takes time. xarray really needs dask >= 2.0 but version 1 is in unstable; I've disabled some functionality (dask='parallelize') and marked some tests xfail until this is done. Alastair yes, people are aware of a needed dask update, see #942235. There's now also a dask 2.8 release available, I didn't look at that yet. dask is itself failing autopkgtests, and this is blocking pandas and pytest, so is a problem independently of xarray. (The breakage was originally caused by pytest 4, but pandas 0.25 Build-Depends: on that.)
Re: Remove pyoptical from Debian, [or?] upgrade psychopy to latest upstream to enable Pandas migration
psychopy isn't blocking pandas' testing migration because psychopy already isn't in testing (for reasons unrelated to py2removal). Upstream say it's now Python 3 compatible [0], but I haven't tried to fix its other problems. The ones that are blocking pandas [1] are python-skbio, python-feather-format and dask. [0] https://github.com/psychopy/psychopy/blob/master/setup.cfg#22 [1] https://qa.debian.org/excuses.php?package=pandas (cnvkit since fixed)
Re: Bug#942106: python3.8 / pandas py2removal
I have uploaded pandas and statsmodels. On 10/11/2019 14:18, Matthias Klose wrote: https://people.debian.org/~doko/tmp/ The patsy one has a bug: as debian/tests/nosetests3 was a symlink to nosetests2, it should have deleted this link and renamed nosetests2 to nosetests3, not deleted nosetests2.
Re: Bug#942106: python3.8 / pandas py2removal
mdp isn't in testing either, but if you're using a policy of "no py2removals that break packages in testing", tnseq-transit (Depends: statsmodels) and possibly stimfit (Recommends: pandas) need to be done as well. Those are both thought to need new upstream versions. (patsy isn't a leaf package for py2removal, it's in a cycle with pandas and statsmodels, i.e. for a non-breaking removal they all have to go together.)
Re: theano: remove python2
On 28/10/2019 06:30, Mo Zhou wrote: Hi Rebecca, Theano is a leaf package No it isn't: deepnano Depends on it. (If you used my checker - not finding this is bug https://lists.debian.org/debian-python/2019/10/msg00106.html ) blocking python2 removal. What's the blocker for the pending commits on the master branch? It can be successfully built in a clean chroot. Only the above. (And the same for libgpuarray.)
pandas: Python 2 removal, 0.23 -> 0.25 transition
Python 3.8 is being added (#942106). pandas <0.25 does not support Python 3.8[0] (when Ubuntu tried they got 268 test failures [1]), while pandas>=0.25 does not support Python 2. Hence, our options are: (a) Remove python-pandas and upgrade pandas to 0.25 (b) Split pandas into two source packages (like matplotlib) so we can have python-pandas 0.23 and python3-pandas 0.25 (c) Let pandas be broken on 3.8 for now I currently favour (a), but this is still open for discussion. pandas is currently in a big strongly connected component [2], but it looks possible to cut it out of that tangle without actually breaking anything, by removing these 2 build-dependencies: matplotlib2 (only uses pandas in examples/tests) python-apptools (only uses pandas in H5TableNode which (according to codesearch) nothing uses) This leaves the following needing to drop Python 2 before pandas can: Already supports Python 3: bcolz mdp patsy python-biom-format scikit-learn statsmodels (my package, will deal with it when its dependencies are cleared) Need a new upstream version to support Python 3: metaphlan2 psychopy pymvpa2 (upstream say they don't test this, but the package already fails its tests) stimfit tnseq-transit (needs new dependency pypubsub, in NEW) Also, the move from pandas 0.23 to 0.25 involves API changes [3], and hence may break some reverse dependencies. I haven't tested this yet, but if you maintain a pandas reverse dependency that has a newer upstream version available, this might be a good time to package it. To simplify such testing, I hope to upload pandas 0.25 to experimental today. [0] https://github.com/pandas-dev/pandas/issues/29043 [1] https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-focal/focal/amd64/p/pandas/20191024_181815_7c017@/log.gz [2] https://lists.debian.org/debian-python/2019/10/msg00092.html (not quite that big since we started ignoring Suggests, but still not something to remove just yet) [3] https://pandas.pydata.org/pandas-docs/stable/whatsnew/index.html#release
Re: scientific python stack transitions
On 10/08/2019 04:02, Drew Parsons wrote: Strange, they haven't accepted scipy 1.2.2 yet. But mpi4py 3.0.2 went straight in. Autopkgtest failure: https://autopkgtest.ubuntu.com/packages/python-scipy/eoan/amd64 (Though I'm not sure why they're calling something that's been failing since before their last release a "regression"...)
Re: scientific python stack transitions
statsmodels 0.9 (#931540) turned into a bigger job than I was expecting (in general packaging cleanup, not because it's a transition), but is now almost ready to go - probably tomorrow if there are no unexpected issues. I then intend to look at pandas 0.24 (#931557), but suspect that may be too much to make the Ubuntu freeze.
Re: scientific python stack transitions
matplotlib and scikit-learn have also had upstream releases, and numpy is likely to soon; I do not know their maintainers' intentions. On 07/07/2019 12:59, Drew Parsons wrote: On 2019-07-07 19:29, Rebecca N. Palmer wrote: I'd like to get [pandas+statsmodels] up to latest upstream in time for the next Ubuntu (freeze 22 August[2]), but am not sure if this is realistic for pandas, given the rather large number of rdeps. Speaking of updates, I want to drop scipy 1.2 into unstable in the coming days if not hours. Given that scipy 1.2 is ready to go in experimental (statsmodels isn't - I'd like to at least merge the packaging changes from unstable), and has already been in an Ubuntu release, it probably makes sense for it to go first. scipy 1.3 is now released but we should take it through experimental first, it may have changed some API.
Pandas/Statsmodels
To be clear before I start uploading new upstream versions: was our before-freeze discussion [0-1] a permanent invitation to help (and not just to fix the then-immediate problems)? I'd like to get them up to latest upstream in time for the next Ubuntu (freeze 22 August[2]), but am not sure if this is realistic for pandas, given the rather large number of rdeps. [0] https://lists.debian.org/debian-science/2019/02/msg00070.html [1] https://lists.debian.org/debian-science/2019/03/msg3.html [2] https://wiki.ubuntu.com/EoanErmine/ReleaseSchedule
Re: state of clblas/clfft/clsparse
I haven't found anything officially saying they're abandoned upstream, but none of them have any commits since August 2017, despite open pull requests. clblas and clfft currently appear to be mostly OK in Debian (and most of the bugs that are known have patches; clsparse was removed because of a preference not to have abandoned software, its actual bug was fixable), so this is not an urgent problem. However, they don't run their tests, so this doesn't actually prove that they work. Other projects with similar (but not directly API-compatible) functionality: - rocBLAS/rocFFT/rocSPARSE - https://github.com/ROCmSoftwarePlatform Also by AMD, with no obvious comment on their relationship; I don't currently know whether they are limited to Radeon hardware, or how similar the API is - Apple clfft - supposedly https://developer.apple.com/mac/library/samplecode/OpenCL_FFT/index.html but this now redirects to an archive page; an embedded copy is found at https://github.com/ufo-kit/ufo-filters/tree/master/deps/oclfft - reikna - https://pypi.org/project/reikna/ - FFT and matrix multiplication; in Python - clblast - https://github.com/CNugteren/CLBlast - says it is trying to be similar to the clblas API, but unclear whether this means switching is just a search-and-replace to change the name prefix Reverse dependencies in Debian: - arrayfire (clblas+clfft) - can use clblast instead of clblas (upstream discussion of whether to switch to it by default, doesn't mention clblas abandonment: https://github.com/arrayfire/arrayfire/issues/1956); have their own fork of clfft, but apparently only to get rid of some warnings they don't want - gpyfft (clfft) - Python interface to clfft (not just a 1:1 wrapper, but if clfft goes maybe this should too); no obvious upstream comment - libgpuarray (clblas) - can instead use clblast - ufo-filters (clfft) - can instead use (and upstream embeds) Apple clfft
Re: Pandas
Any comment (and preferably upload) on statsmodels? That needs to be
uploaded by the 2nd (in testing by the 12th) if at all, as it has
changes that wouldn't be allowed under full freeze.
On 27/02/2019 14:46, Yaroslav Halchenko wrote:
On Wed, 27 Feb 2019, Andreas Tille wrote:
I'd prefer if you would move the
0.24 packaging to some separate branch (debian-experimental is covered but may
be debian-0.24 or something like this?) and keep branch debian for what we are
really releasing.
well "releasing" is a loaded term. I guess you are talking about uploading to
unstable so it manages to get into buster. Since "debian" branch already got
its 0.24, what about starting debian-buster branch off debian/0.23.3-1 ?
That's what I was planning, like statsmodels already has. (I was going
to call it 'buster', but I don't care about the name.)
otherwise -- I am ok to hard-reset and force push debian to the debian/0.23.3-1
state -- everyone should just beware of it, and then progress
debian-experimental to current state of debian (v0.24.1-972-g1cfbd07c7)
I'd rather not rebase/force-push a public repository (even if it is in
practice unlikely that anyone other than us has downloaded it). It
should be possible to get to that state without doing so ("merge" debian
into debian-experimental, then commit a "revert to 0.23" to debian).
(In general, I slightly prefer the "'debian' is development head,
whether that's currently unstable or experimental (due to a freeze or a
transition)" layout. However, that may be because the packages I
currently maintain rarely needed any changes in unstable while the
package was also in experimental, and this layout hence avoided
branching most of the time. Something as central as pandas may need two
active branches more often.)
Re: Pandas
On 27/02/2019 07:00, Andreas Tille wrote: Dear Rebecca, I do not think that there is any need for a separate branch. Just stick to the debian branch. It's needed because the debian branch contains the attempt at packaging 0.24 described earlier in this thread, which it's now too late for.
Re: Pandas
I now have fixes for both #918206 (test failure, RC) and #804552 (no documentation), in those bugs. (Would you prefer to have a salsa branch?) Several other open pandas bugs appear to be either already fixed or non-bugs: is it appropriate for me to close them?
Re: statsmodels
On 25/02/2019 10:04, Andreas Tille wrote: Dear Rebecca, thanks for your work on this. I admit it is not clear to me whether you intent / have permission to upload the status of the buster branch It's not entirely clear to me either, given that this is more than (at least I) expected at the time of https://lists.debian.org/debian-science/2019/02/msg00044.html You also mention the necessary pandas upload. How can I help with this? It might be best to wait a bit for that one: I'm currently looking at whether we can fix #804552 (no documentation) as well.
Re: statsmodels
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Fixing statsmodels turned into a much bigger job than I was expecting: * The tests were disabled, possibly by accident. Re-enabling them found that 70 fail, for ~10 different reasons that all look related to newer versions of dependencies. These are all now fixed (except the already-excluded 4, which are still excluded). * The example output in the documentation (produced at build time) contained many error messages: ... A few more issues related to newer versions of dependencies and/or of statsmodels itself. ... Loading data from the cache (which we pre-populate to allow offline building) was broken in Python 3. These have been fixed. There are still several errors from trying to download data we don't have cached (I haven't tried to add it, partly because I am unsure of the legality of doing so, and partly because it's often loaded by methods that don't use our existing cache mechanism), but these are not new. I left #880245 unfixed: upstream's real fix doesn't apply as-is to 0.8 and is big enough that trying to adapt it didn't seem a good idea, and given how long the documentation bugs above probably existed before anyone noticed, I'd rather not do a workaround that would allow even more broken documentation to go unnoticed if we don't have to. This is on salsa as the buster branch (commit 522017cf17578820637ec49614a46b30854f78c3, with unfinalized changelog as per team policy). (Reminder: pandas also needs an upload with the patch in #918206, as this is RC.) -BEGIN PGP SIGNATURE- iQIzBAEBCAAdFiEEZ8sxEAXE7b4yF1MI3uUNDVZ+omYFAlxysN8ACgkQ3uUNDVZ+ ombG5hAAgJX7Is3zEh81T0I9DQEchNbQXD3yPyMjMzrE+kgegyRKcLCkxJ39Y7O9 U2V+NIwIcVenAbyJyseXpOngwbF2EVkCAc3vtewYZknjVwW3f7b64g/L+t6Ou3LI SPs7QWsl8/H2jn/Ws46NYvfUPIS32p/k8r71uwhMbqeOhmEDPnqOYqzc8+NvHe9Y oaP4FGoKxRh7Tmgu6B/xbwhxVhexpp11Bvg6rBqHU3aPMzB54ZWOLMiceDU4xFIa Az9wbR/4KDpZ6vcrh+Xy6/5im8/nd9k7QTvTKsTxWu2ONqeF2vhYdOyemxf/KrK7 QmxXbiWpPBx9xueNTnPEabPi+A2KZiJAWx95bjPDaaVLPucmAW5tqOlzNgtlggDv gaWXQ+kzJgvs4HNDNhD3hV37WdupCt6lOE/5cmtocVwPyekdgDkqJSHsMJNv7Knr 9MAAkLBF5rpO4koronxQkJYRw/lm6pWk7OmG7PvNUjH8KRgRRh3BsXmWRWZgHob4 GbFaSYxqg8djOEqqiN2IJg5Mp/Faf9Jk1CN/4pPjkdfkxQ6L8lIE5WEa5ttpTVyz oFpGrQYH1xuyodWefL29AibWQSN/smsZqdbJE5nZDyNLLb1b5gRHnQ60ukKv5ZnG zZz39bQ5b6XfSUVF2dVViE6hns+XTs3PdyiAbXM4W6yBYZj5d1w= =B8BS -END PGP SIGNATURE-
Re: statsmodels
On 13/02/2019 09:10, Andreas Tille wrote: Any volunteer to backport the relevant changes I pushed to Git right now to 0.8.0? I intend to try tomorrow, if the uploaders don't say anything first. I'm actually wondering why the package did not got any removal from testing warning (or did I missed something)? statsmodels is considered a key package (i.e. can't be autoremoved) because of the statsmodels -> seaborn -> sphinx-gallery -> matplotlib build-dependency chain. The autoremoval list (sorted by maintainer/team) is https://udd.debian.org/cgi-bin/autoremovals.cgi PS: As a general note: We probably need more people dedicated to Debian Science QA work. I guess that's sort of what I do (and if statsmodels and/or pandas were to become available, they are at least things I use).
Re: statsmodels
Is the subject line a leftover, or are you actually proposing to move 0.9.0 from experimental to unstable (which is not generally allowed in soft freeze)? The upstream fix for #880245 (the original reason given for wanting the new version) is supposedly https://github.com/statsmodels/statsmodels/commit/f81cde31421288e669e859d0e798d843ca2ecabe but I haven't tried it by itself.
Bug#921460: dead upstream - keep out of testing
Package: clsparse Severity: serious X-Debbugs-Cc: debian-science@lists.debian.org, ghisv...@gmail.com (plus an identical one for freefem3d) This package is dead upstream, and it has been suggested [0] that because of this, it should not be fixed for buster. I don't know enough about it to have an opinion on this: I am opening this bug as a "don't waste your time fixing it" notice. If this remains the consensus, please consider removing it from unstable as well. [0] https://lists.debian.org/debian-science/2019/02/msg00015.html
freefem3d #897752 and clsparse #897722 RC fixes
Both these packages currently FTBFS (for similar reasons: invalid unused templates, which GCC 8 is stricter at rejecting). I have posted (but can't upload) fixes for both. As neither of these is currently in testing and there is no re-entry after 12 Feb (migration date, not upload date), these urgently need to be uploaded if we want these packages in buster. I don't know if these packages are worth saving (both of them appear to be dead upstream), but if we need more time to think about that, fixing them now gives us the option of deciding whether to remove them later.
Re: Pandas new version
Control: tags -1 fixed-upstream patch The test failure is that np.array @ pd.DataFrame (matrix product) tries to keep both the DataFrame's indices, which fails because the new matrix is a different shape. This appears to be fixed in 0.24.1 from PyPI, but as previously noted, this is a new major version and hence risks breaking rdeps. The relevant change appears to be setting __array_priority__: https://github.com/brute4s99/pandas/commit/a01fa791eafe704ea85e2acc956ad9077e8e7542#diff-03b380f521c43cf003207b0711bac67f but I haven't actually tried applying only that to 0.23.
Re: Pandas new version
Has anyone checked whether this would break pandas' reverse dependencies?
RFS: libgpuarray/0.7.6-1
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Please upload libgpuarray 0.7.6-1 (https://salsa.debian.org/science-team/libgpuarray.git commit c1d24cd9ff81c230b809cc534ff5c59f05b9cf08, not tagged as a previous sponsor preferred not doing this in advance) to experimental. This is binary-NEW (which is why I can't upload it myself), but was ruled not a transition in #919022. Lintian tags: debian-watch-does-not-check-gpg-signature - upstream don't provide one no-symbols-control-file usr/lib/libgpuarray.so.3.0 - would be mostly useless as the only in-archive rdep (theano) uses the Python interface Known issues / testing notes: The BLAS functions in this package use clblas, and can trigger crash bugs on some hardware (#881054/#877316 under beignet-opencl-icd = Intel integrated GPUs, #919824/#920497 under pocl-opencl-icd = CPUs). These are not regressions (0.6.9-3 had the same crashes), and I believe them to be in clblas and/or the ICDs, not libgpuarray itself. 0.6.9-3's autopkgtest skipped the BLAS section entirely; 0.7.6-1's runs it with workarounds for pocl-opencl-icd, on the grounds that some testing is better than none. -BEGIN PGP SIGNATURE- iQIzBAEBCAAdFiEEZ8sxEAXE7b4yF1MI3uUNDVZ+omYFAlxQ2PgACgkQ3uUNDVZ+ omYj2w/7BV0CDC6SGTDpm/6Z7TpIqh4/a7ehJQ1fhxxCx707EpTqPN7GkOcB0Bx3 fWkp/0bu4zFgDOIXFC5SG61BHe37Ut7VBB9ELTJrzxhEsd1jKFUtAvTp4UJkxVvC 6CKH6s9/YdPwJiHSYT1stJbj1F0n9aTCjBVfk06p25eFealVkHSvUv5XRFU/n068 jl4XFCnZPiZvIWV11dCyJRSi8CYWX9e+RFcPBL9KKEmVegBuEJuvM/YZRQKTYiqz UTKzYj2SLePPGT3wVqmeK3vdcXUw9J1YCHZXAhflRj44hbb2XqGnXOw2y+u/n7XK 8S5Y8XZLJWESKtdwGZYMYOuIqCrL3a0ojYj4wibE3mkb+FKFT7TN5wjfkA6LciX8 aVn9XVnCYC8hfL7zcktGMmm/rlwk5cCcWdym66mha0NZ3EiDu7dYOHakRa8hZl+u gUoY9yjGjLz7XvrdDnb0MTwjPWg8Yg4Xyi5FEw6qLSPm200La/39/nC78HKT1Ucp KvQU3Iwiovmyc4rLNv/vwzEkAVWvSegmIq6pSBIfeTOVAguFbldqQyn3ntBqGU/8 MlqqcHoR818lEWHz0rWyaTRCnm6Fgkfs6bFU+peBEklZXTPQxh7E+ESJDqq5JzDZ G3rvwt/HgbimuTxqHhBrlqGdGJN37Vt8C3FOUJ8pAo4iBh+HUJQ= =vC3Q -END PGP SIGNATURE-
Re: Taking over libgpuarray / GPU testing wanted
On 12/01/2019 12:27, Rebecca N. Palmer wrote: 0.6.9-3 [0] is intended for unstable, and is ready for upload; On further investigation, it isn't: while the failing tests on i386 only happen in 0.7.6, the underlying bug (assuming cl_ulong and size_t are the same type - https://github.com/Theano/libgpuarray/issues/581) is there in 0.6.9.
Re: dwz: libsleef3.debug: .debug_line reference above end of section
I suspect this is caused by bumping debhelper compat (while often a good idea, this is intentionally *not* risk-free, which is why it requires explicit action), but I haven't actually tried fixing it by reverting this. codesearch finds that message in dwz: https://sources.debian.org/src/dwz/0.12-3/dwz.c/?hl=1016#L1016 dwz was added to the standard dh sequence in compat 12: https://sources.debian.org/src/debhelper/12/debhelper.pod/#L860 As it seems to relate to debug information, it might be a good idea to test whether that works: # apt-get install libsleef3-dbgsym $ gdb --args (gdb) break (gdb) run [wait for "hit Breakpoint", or if it's a program that spends most of its time in sleef, press Ctrl+C] (gdb) bt full [should show source line numbers and parameter values, i.e. like #909379 not #914021]
Taking over libgpuarray / GPU testing wanted
With its previous Uploader's permission, I have prepared an updated libgpuarray package, available in Salsa. As I am only a DM, I can't currently upload this package. 0.6.9-3 [0] is intended for unstable, and is ready for upload; 0.7.6-1 [1] is intended for experimental (we're in transition freeze from today, but Ubuntu isn't), and is still work in progress (its tests are failing on i386). Testing on other GPU hardware would also be welcome; in particular, the package description says the CUDA interface doesn't work in the packaged version, and I can't see why. The tests are run with DEVICE=opencl0:0 python3 -m nose pygpu (with the python3-pygpu under test installed, *not* from its source directory), Depends: ocl-icd-opencl-dev, python3-nose and your hardware's *-opencl-icd, Recommends: libclblas-dev; depending on your hardware, DEVICE=opencl1:0 (multiple *-opencl-icd) or DEVICE=cuda (nvidia-cuda-dev) may also be valid. [0] https://salsa.debian.org/science-team/libgpuarray/commit/9d6d2668e1e2c3a789b591d8d668662709d4b7c9 [1] https://salsa.debian.org/science-team/libgpuarray/commits/master
Re: Freecad is looking for new maintainers
I see that gdal has moved ahead to 2.3 in Sid, I'm guessing this means opencv needs to be rebuilt (potentially with API changes...). It now has been: https://release.debian.org/transitions/html/gdal-2.3.0.html
Re: [Help] Exclusion did not worked (Was: Bug#877419: Bug#877700: RM: pandas [arm64 armel armhf mips mips64el mipsel s390x] ...)
raise nose.SkipTest("known failure of test_stata on non-little endian")
E NameError: name 'nose' is not defined
You need an 'import nose' first, if the test doesn't already have one.
Re: Python 3 Statsmodels & Pandas
On 30/09/17 16:50, Diane Trout wrote: I am curious if Rebecca is using pandas on a non-intel architecture though (was wondering how she noticed pandas hadn't built) No - I found this email thread and checked https://buildd.debian.org/status/package.php?p=pandas
Re: Python 3 Statsmodels & Pandas
On 29/09/17 23:55, Diane Trout wrote:
I wonder if it's better to filter sphinxdoc out of the dh line, install
sphinx-common, or just always install python3-sphinx?
For local testing of this, use pbuilder --debbuildopts -B
Given earlier messages that we want this working ASAP, always installing
python3-sphinx is the lowest-risk option.
As for nose, python{,3}-nose is listed a build-depends ? Why
is it failing on importing nose? it should either be there or it
shouldn't be running tests?
It's not failing on importing nose - it's failing because the patch
tries to skip a test *without* importing nose.
Re: Python 3 Statsmodels & Pandas
statsmodels passed NEW...and FTBFS with dh: unable to load addon sphinxdoc: Can't locate Debian/Debhelper/Sequence/sphinxdoc.pm in @INC That file is in the sphinx-common package, which suggests that at least some of the sphinx dependencies need to be Build-Depends and not just Build-Depends-Indep. Meanwhile, pandas FTBFS with several failing tests (all involving datetime) on arm*/mips*el and a broken attempt to skip a test (trying to raise nose.SkipTest without importing nose) on mips/s390x.
Is theano worth saving?
I have patches for #848764 and #831540; I haven't yet found the cause of #831541, but as it only affects big-endian systems, partial removal is an option. However, one of these bugs is a point where upstream plan to make an incompatible change (though my fix doesn't), and upstream aren't sure whether having Theano in a long-release-cycle distribution such as Debian is a good idea: https://github.com/Theano/Theano/issues/5494 (Note that their suggested alternative of using pip will involve manually installing libblas-dev, as pip only knows about Python dependencies.) The package is RFAd; I may well end up adopting it, but am not sure I want to commit to that just yet. (I only discovered its existence at a BSP ~10 days ago, but it does look like a reasonable fit for me.)
RFS: blockdiag/1.5.3+dfsg-1.1 [RC] Re: Sympy(+sagemath+nipy+~30 more) removal ahead
Forwarded Message
Subject: intent to NMU #848748: blockdiag FTBFS
Date: Sun, 22 Jan 2017 23:29:24 +
From: Rebecca N. Palmer <rebecca_pal...@zoho.com>
To: 848...@bugs.debian.org
Attached is a proposed NMU fixing this bug (identical to my original
patch other than adding the changelog entry; the olefile issue noted
above was a bug in pillow, which has now been fixed).
As the execnet issue (#840823) now also appears to have a fix,
this will allow blockdiag's ~30 rdeps to stay in stretch.
diff -Nru blockdiag-1.5.3+dfsg/debian/changelog
blockdiag-1.5.3+dfsg/debian/changelog
--- blockdiag-1.5.3+dfsg/debian/changelog 2016-10-11 02:21:04.0
+0100
+++ blockdiag-1.5.3+dfsg/debian/changelog 2017-01-22 22:14:34.0
+
@@ -1,3 +1,10 @@
+blockdiag (1.5.3+dfsg-1.1) unstable; urgency=medium
+
+ * Non-maintainer upload.
+ * Don't fail tests on a harmless wand warning. Closes: #848478
+
+ -- Rebecca N. Palmer <rebecca_pal...@zoho.com> Sun, 22 Jan 2017 22:13:59
+
+
blockdiag (1.5.3+dfsg-1) unstable; urgency=medium
* New upstream release. Closes: #801314
diff -Nru
blockdiag-1.5.3+dfsg/debian/patches/848748-exception-ignored-in-Image-del.patch
blockdiag-1.5.3+dfsg/debian/patches/848748-exception-ignored-in-Image-del.patch
---
blockdiag-1.5.3+dfsg/debian/patches/848748-exception-ignored-in-Image-del.patch
1970-01-01 01:00:00.0 +0100
+++
blockdiag-1.5.3+dfsg/debian/patches/848748-exception-ignored-in-Image-del.patch
2017-01-09 22:32:13.0 +
@@ -0,0 +1,38 @@
+Description: Fix test failure with wand 0.4.1+
+
+wand 0.4.1 added an Image.destroy()
+(https://sources.debian.net/src/wand/0.4.4-1/wand/image.py/#L2760) that
+iterates over self.sequence (the frames of an animation) to free their
+memory; this throws an exception on single images (where
+self.sequence=None), but as this is called from __del__, this
+exception is warned about then ignored
+(https://docs.python.org/3/reference/datamodel.html#object.__del__),
+and hence is not an error in normal use.
+
+However, blockdiag tests that use capture_stderr fail on any output
+containing "Traceback", including this warning message.
+
+This patch ignores this message to allow blockdiag to build.
+
+Author: Rebecca Palmer <rebecca_pal...@zoho.com>
+Bug-Debian: https://bugs.debian.org/848748
+Forwarded: no
+
+--- blockdiag-1.5.3+dfsg.orig/src/blockdiag/tests/utils.py
blockdiag-1.5.3+dfsg/src/blockdiag/tests/utils.py
+@@ -64,7 +64,14 @@ def capture_stderr(func):
+
+ func(*args, **kwargs)
+
+-if re.search('(ERROR|Traceback)', sys.stderr.getvalue()):
++filtered_stderr=re.sub(r"""Exception ignored in: >
++Traceback \(most recent call last\):
++ File "/usr/lib/python3/dist-packages/wand/resource\.py", line [0-9]+, in
__del__
++self\.destroy\(\)
++ File "/usr/lib/python3/dist-packages/wand/image\.py", line [0-9]+, in
destroy
++for i in range\(0, len\(self\.sequence\)\):
++TypeError: object of type 'NoneType' has no
len\(\)""","",sys.stderr.getvalue())#this is the expected result of freeing a
single image (as opposed to an animation) in wand 0.4, not a blockdiag bug -
#848748
++if re.search('(ERROR|Traceback)', filtered_stderr):
+ raise AssertionError('Caught error')
+ finally:
+ if sys.stderr.getvalue():
diff -Nru blockdiag-1.5.3+dfsg/debian/patches/series
blockdiag-1.5.3+dfsg/debian/patches/series
--- blockdiag-1.5.3+dfsg/debian/patches/series 2016-10-11 01:32:49.0
+0100
+++ blockdiag-1.5.3+dfsg/debian/patches/series 2017-01-09 21:50:32.0
+
@@ -1 +1,2 @@
Fixed-remote-image-resouces.patch
+848748-exception-ignored-in-Image-del.patch
Re: quick Q to Matthias Re: Sympy(+sagemath+nipy+~30 more) removal ahead
Why do you think the test failures are due to missing olefile? There are two sets of test failures here: the two in #848748 (test mistakes new-in-0.4.1 wand warning for an error), and the ~70 first reported in this thread (lack of olefile, now probably fixed in pillow). All my testing was in sid, not stretch.
Re: quick Q to Matthias Re: Sympy(+sagemath+nipy+~30 more) removal ahead
I now also see the no-olefile errors in sid:
(This was with a locally built wand - the in-archive one is currently
uninstallable due to #850815 - and is one of many with similar messages.)
I haven't checked whether it works in stretch (which has an older
version of pillow, with an embedded copy of olefile).
==
FAIL: blockdiag.tests.test_generate_diagram.test_generate(at 0x7f65b488db90>, 'svg',
'/tmp/buildd/blockdiag-1.5.3+dfsg/.pybuild/pythonX.Y_2.7/build/src/blockdiag/tests/diagrams/skipped_edge_leftdown.diag',
['-f', '/usr/share/fonts/truetype/vlgothic/VL-Gothic-Regular.ttf'])
--
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in
runTest
self.test(*self.arg)
File
"/tmp/buildd/blockdiag-1.5.3+dfsg/.pybuild/pythonX.Y_2.7/build/src/blockdiag/tests/utils.py",
line 75, in wrap
raise AssertionError('Caught error')
AssertionError: Caught error
>> begin captured stdout << -
---[ stderr ] ---
WARNING: Failed to load blockdiag.imagedraw.svg:
DistributionNotFound(Requirement.parse('olefile'), set(['Pillow']))
WARNING: Failed to load blockdiag.imagedraw.png:
DistributionNotFound(Requirement.parse('olefile'), set(['Pillow']))
WARNING: Failed to load blockdiag.imagedraw.pdf:
DistributionNotFound(Requirement.parse('olefile'), set(['Pillow']))
ERROR: unknown format: SVG
- >> end captured stdout << --
==
Re: Sympy(+sagemath+nipy+~30 more) removal ahead
missing python{,3}-olefile in Build-Depends.
There is no such package in unstable - was this a typo?
(For me the only failed tests were the two in the bug, and just the
patch was enough to fix that.)
