Re: Bug#1043240: transition: pandas 1.5 -> 2.1 - please upload fixes
Hi again, Am Fri, Feb 02, 2024 at 09:56:14PM +0100 schrieb Andreas Tille: > Hi Rebecca, > > Am Tue, Jan 30, 2024 at 08:05:35AM + schrieb Rebecca N. Palmer: > > I intend to upload pandas 2.x to unstable soon. These packages have a patch > > in their bug - please upload them (I'm a DM, I can't do that), or if you > > think this patch won't work or isn't a good idea, tell me why: > > dials > > Was uploaded, all bugs closed. > > > python-altair > > I tried hard to get the latest version which implements what you suggested > independently in the bug report. Unfortunately it needs a new dependency > as I wrote in my comment in the bug report[2] and I was not able to easily > exclude the test that fails due to the missing module. Maybe I'd rather revert to the version currently in Debian. I might check later if nobody will beat me. > > python-feather-format I've followed the hint given by Rebecca. Unfortunately there are new Cython issues as you can see in Salsa CI[1]. Any hint would be welcome. > > seaborn Discussed in other mails > > tqdm > > I try to check later. Kind regards Andreas. [1] https://salsa.debian.org/python-team/packages/python-feather-format/-/jobs/5246082 -- http://fam-tille.de
Re: Bug#1043240: transition: pandas 1.5 -> 2.1 - please upload fixes
Hi Rebecca, Am Tue, Jan 30, 2024 at 08:05:35AM + schrieb Rebecca N. Palmer: > I intend to upload pandas 2.x to unstable soon. These packages have a patch > in their bug - please upload them (I'm a DM, I can't do that), or if you > think this patch won't work or isn't a good idea, tell me why: > dials Was uploaded, all bugs closed. > influxdb-python I've answered to the bug[1] that the hints you gave seem to be incomplete. Unfortunately I have no idea how to fix this. > python-altair I tried hard to get the latest version which implements what you suggested independently in the bug report. Unfortunately it needs a new dependency as I wrote in my comment in the bug report[2] and I was not able to easily exclude the test that fails due to the missing module. > python-feather-format seaborn tqdm I try to check later. > In particular, I'd like the seaborn fix uploaded before pandas, so I can set > Breaks for it. (The pandas documentation build-depends on seaborn.) I will check soon as long as noone will beat me in doing so. Kind regards and thanks for all your work on pandas Andreas. [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1044076#43 [2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1044073#23 -- http://fam-tille.de
Re: Bug#1043240: transition: pandas 1.5 -> 2.1 - please upload fixes
On Tue, Jan 30, 2024 at 08:05:35AM +, Rebecca N. Palmer wrote: > In particular, I'd like the seaborn fix uploaded before pandas, so I can set > Breaks for it. (The pandas documentation build-depends on seaborn.) https://tracker.debian.org/news/1498923/accepted-seaborn-0132-1-source-into-unstable/ Best, Nilesh signature.asc Description: PGP signature
Re: Bug#1043240: transition: pandas 1.5 -> 2.1 - please upload fixes
for dials it seems that the CI works with pandas 2.1 from experimental. https://ci.debian.net/packages/d/dials/unstable/amd64/41962612/#S4 - Le 30 Jan 24, à 9:05, Rebecca N. Palmer rebecca_pal...@zoho.com a écrit : > I intend to upload pandas 2.x to unstable soon. These packages have a > patch in their bug - please upload them (I'm a DM, I can't do that), or > if you think this patch won't work or isn't a good idea, tell me why: > dials influxdb-python python-altair python-feather-format seaborn tqdm > > In particular, I'd like the seaborn fix uploaded before pandas, so I can > set Breaks for it. (The pandas documentation build-depends on seaborn.)
Re: Bug#1043240: transition: pandas 1.5 -> 2.1 - please upload fixes
I intend to upload pandas 2.x to unstable soon. These packages have a patch in their bug - please upload them (I'm a DM, I can't do that), or if you think this patch won't work or isn't a good idea, tell me why: dials influxdb-python python-altair python-feather-format seaborn tqdm In particular, I'd like the seaborn fix uploaded before pandas, so I can set Breaks for it. (The pandas documentation build-depends on seaborn.)
Re: Bug#1043240: transition: pandas 1.5 -> 2.1
On Sun, Jan 21, 2024 at 03:29:21PM +, Rebecca N. Palmer wrote: > Control: severity 1053943 1053939 1053942 1044053 1044056 serious > Control: severity 1044074 1053946 1044078 1044079 1044077 serious > Control: severity 1044071 1044067 1044068 1044055 1044060 serious > Control: severity 1044072 1044073 1044064 1053945 1044054 serious > Control: severity 1044076 1053940 1044057 1053944 1050144 serious > > As previously discussed in this bug, I'd like to move pandas 2.x into > unstable reasonably soon. I'm aiming to get it in before the Ubuntu 24.04 > freeze (in about a month), but I am open to disagreement on whether this is > a good idea. Please could we wait until the "Python 3.12 is a supported version" transition is completed? Adding another 25 or so RC bugs at this point will just slow down that transition. (Unless pandas 1.5 is preventing the transition, that is.) Best wishes, Julian
Re: Bug#1043240: transition: pandas 1.5 -> 2.1
On 2024-01-21 21:22, Andrew M.A. Cater wrote: [Replying off list in case this person is reading this through a digest or some such and is not subscribed to all lists] On Sun, Jan 21, 2024 at 07:31:58PM +, Stelios Moschos wrote: Hi, how to remove myself from these lists? ... The answer is at the bottom of every list mail. You need to send a mail to debian-science-requ...@lists.debian.org with a subject of unsubscribe . To be fair, there was no such notice at the bottom of this current set of emails. Alternatively, you can use the form at https://www.debian.org/MailingLists/ to unsubscribe from several lists at once.
Re: Bug#1043240: transition: pandas 1.5 -> 2.1
[Replying off list in case this person is reading this through a digest or some such and is not subscribed to all lists] On Sun, Jan 21, 2024 at 07:31:58PM +, Stelios Moschos wrote: > Hi, how to remove myself from these lists? > > Thank you > > > The answer is at the bottom of every list mail. You need to send a mail to debian-science-requ...@lists.debian.org with a subject of unsubscribe . (for example) The list mailer will reply with a Confirm mail: a simple reply to that will unsubscribe you. Alternatively, you can use the form at https://www.debian.org/MailingLists/ to unsubscribe from several lists at once. All the very best, Andy (amaca...@debian.org)
Re: Bug#1043240: transition: pandas 1.5 -> 2.1
Hi, how to remove myself from these lists? Thank you On Sun, 21 Jan 2024 at 18:30, Andreas Tille wrote: > Hi Rebecca, > > Am Sun, Jan 21, 2024 at 03:29:21PM + schrieb Rebecca N. Palmer: > > > > Hence, doing this transition now would involve breaking some reverse > > dependencies with no known fix, but given the number of packages > involved, > > trying to wait until they're all fixed is rather likely to instead end in > > pandas 1.5 being broken by a new Python/numpy/etc. > > Just go for it and lets try to fix issues as soon as possible. > > Thanks a lot for all your work on pandas > > Andreas. > > -- > http://fam-tille.de > >
Re: Bug#1043240: transition: pandas 1.5 -> 2.1
Hi Rebecca, Am Sun, Jan 21, 2024 at 03:29:21PM + schrieb Rebecca N. Palmer: > > Hence, doing this transition now would involve breaking some reverse > dependencies with no known fix, but given the number of packages involved, > trying to wait until they're all fixed is rather likely to instead end in > pandas 1.5 being broken by a new Python/numpy/etc. Just go for it and lets try to fix issues as soon as possible. Thanks a lot for all your work on pandas Andreas. -- http://fam-tille.de
Re: Bug#1043240: transition: pandas 1.5 -> 2.1
Control: severity 1053943 1053939 1053942 1044053 1044056 serious Control: severity 1044074 1053946 1044078 1044079 1044077 serious Control: severity 1044071 1044067 1044068 1044055 1044060 serious Control: severity 1044072 1044073 1044064 1053945 1044054 serious Control: severity 1044076 1053940 1044057 1053944 1050144 serious As previously discussed in this bug, I'd like to move pandas 2.x into unstable reasonably soon. I'm aiming to get it in before the Ubuntu 24.04 freeze (in about a month), but I am open to disagreement on whether this is a good idea. dask, python-skbio and python-upsetplot have been fixed since the previous discussion, but that still leaves the above 25. 6 of these have a known-to-me fix (dials influxdb-python python-altair python-feather-format seaborn tqdm - see their bugs for details). Hence, doing this transition now would involve breaking some reverse dependencies with no known fix, but given the number of packages involved, trying to wait until they're all fixed is rather likely to instead end in pandas 1.5 being broken by a new Python/numpy/etc.
Re: Bug#1043240: transition: pandas 1.5 -> 2.1
On 11.12.23 08:12, Matthias Klose wrote: On 10.12.23 14:06, Rebecca N. Palmer wrote: Is this an acceptable amount of breakage or should we continue to wait? Bear in mind that if we wait too long, we may be forced into it by some transition further up the stack (e.g. a future Python or numpy) that breaks pandas 1.x. up to the maintainers. But please wait at least until the current pandas and numpy migrated to testing, e.g. that the autopkg tests of pandas and numpy triggered by python3-defaults pass. I just nmued pyrle and sorted-nearest, having dependencies on cython3-legacy, letting the pyranges autopkg tests fail. Once this succeeds, pandas should be able to migrate.
Re: Bug#1043240: transition: pandas 1.5 -> 2.1
On 12/11/23 08:12, Matthias Klose wrote: On 10.12.23 14:06, Rebecca N. Palmer wrote: I'd like to move forward with the pandas 1.5 -> 2.1 transition reasonably soon. Given that pandas 2.x is *not* required for Python 3.12 (but is required for Cython 3.0), should we wait for the Python 3.12 transition to be done first? These are broken by pandas 2.x and have a possible (but untested) fix in their bug - please test and apply it: dask(?) dials influxdb-python* python-altair python-feather-format python-upsetplot seaborn tqdm* (* = this package is currently also broken for a non-pandas reason, probably Python 3.12, that I don't have a fix for) These are broken by pandas 2.x and have no known-to-me fix: augur cnvkit dyda emperor esda mirtop pymatgen pyranges python-anndata python-biom-format python-cooler python-nanoget python-skbio python-ulmo q2-quality-control q2-demux q2-taxa q2-types q2templates sklearn-pandas Some generic things to try are pandas.util.testing -> pandas.testing, .iteritems() -> .items(), and if one exists, a more recent upstream version. Is this an acceptable amount of breakage or should we continue to wait? Bear in mind that if we wait too long, we may be forced into it by some transition further up the stack (e.g. a future Python or numpy) that breaks pandas 1.x. up to the maintainers. But please wait at least until the current pandas and numpy migrated to testing, e.g. that the autopkg tests of pandas and numpy triggered by python3-defaults pass. Is there a way to see the binNMUs which are still stuck in unstable, and don't migrate? Matthias As a reminder: it's best practice to first upload the new release to Experimental, so we can see what happens with autopkgtest before destroying everything at once... Cheers, Thomas Goirand (zigo)
Re: pandas 1.5 -> 2.1?
Hi Kingsley, On Sun, Dec 10, 2023 at 12:55:43PM -0800, Kingsley G. Morse Jr. wrote: > Hi Rebecca, Julian and all science minded pythonistas of debian, great and > small! > > I like your correspondence about upgrading from > version 1.5 of pandas to 2.1. > > It's open, scientific and explores the ideal of > proceeding wisely in a matter of public interest. > > My humble thoughts are: > > 1.) Rebecca: *Why* did you write that you'd like > to move forward with the pandas 1.5 -> 2.1 > transition? What's your reason? A thought from me on this: pandas 2.1 has many improvements over pandas 1.5. And increasingly, other packages will be requiring these new features. So why would one not want to move forward with it? > 2.) What may be the advantage of migrating to > version 3.0 of Cython? It is compatible with Python 3.12, whereas the current version of Cython in Debian (0.29.x) is not really. (For example, it has an "import imp" in it, and this breaks with Python 3.12, which has removed this deprecated module.) As Cython 0.29.x is no longer maintained upstream, having been superseded by Cython 3.x after many years of development, our options are to either continue to patch Cython 0.29.x within Debian to keep it working with Python 3.12 or to upgrade to Cython 3.x. As there is also software which now depends on Cython 3.x to build, the former option seems unappealing. (At best, we might wish to keep the cython-legacy package around for building packages which can't yet use Cython 3.x, but that should be a short-term thing, not a long-term one.) > 3.) The following one-liner suggests 44 debian > packages might be affected by the breaks > Rebecca said would be caused by pandas 2.x: > > $ for s in augur cnvkit dyda emperor esda mirtop pymatgen pyranges > python-anndata python-biom-format python-cooler python-nanoget python-skbio > python-ulmo q2-quality-control q2-demux q2-taxa q2-types q2templates > sklearn-pandas ; do apt-cache search "$s" ; done | less This does not seem like a particularly helpful one-liner; it picks up packages such as python3-dyda-pipeline-config which are not in the original list. Instead, you perhaps want to count the number of packages depending on these packages. But what Rebecca is looking at (I think) is how many packages would need fixing by the pandas upgrade. (But it is probably worse than this: I'm guessing these are only the packages which fail to build with pandas 2.x or whose autopkgtest fails with pandas 2.x. But there may well be other breakage caused by the upgrade which is not detectable in this way. That is an issue which will have to be handled by individual packages as they are discovered, and the timing of the pandas upgrade is not related to this problem.) > 4.) The break that worries me the most is > sklearn-pandas, because it seems to me that > sklearn is > > popular and > > fundamental. It seems that sklearn-pandas is abandoned; there were just two commits in 2022, and prior to that was May 2021. There has been no activity since. If someone is willing to patch it for Pandas 2.x, great (perhaps you might help the maintainer to do this?), otherwise it might have to drop out of Debian. Best wishes, Julian
Re: Bug#1043240: transition: pandas 1.5 -> 2.1
On 10.12.23 14:06, Rebecca N. Palmer wrote: I'd like to move forward with the pandas 1.5 -> 2.1 transition reasonably soon. Given that pandas 2.x is *not* required for Python 3.12 (but is required for Cython 3.0), should we wait for the Python 3.12 transition to be done first? These are broken by pandas 2.x and have a possible (but untested) fix in their bug - please test and apply it: dask(?) dials influxdb-python* python-altair python-feather-format python-upsetplot seaborn tqdm* (* = this package is currently also broken for a non-pandas reason, probably Python 3.12, that I don't have a fix for) These are broken by pandas 2.x and have no known-to-me fix: augur cnvkit dyda emperor esda mirtop pymatgen pyranges python-anndata python-biom-format python-cooler python-nanoget python-skbio python-ulmo q2-quality-control q2-demux q2-taxa q2-types q2templates sklearn-pandas Some generic things to try are pandas.util.testing -> pandas.testing, .iteritems() -> .items(), and if one exists, a more recent upstream version. Is this an acceptable amount of breakage or should we continue to wait? Bear in mind that if we wait too long, we may be forced into it by some transition further up the stack (e.g. a future Python or numpy) that breaks pandas 1.x. up to the maintainers. But please wait at least until the current pandas and numpy migrated to testing, e.g. that the autopkg tests of pandas and numpy triggered by python3-defaults pass. Is there a way to see the binNMUs which are still stuck in unstable, and don't migrate? Matthias
Re: pandas 1.5 -> 2.1?
On 2023-12-10 21:55, Kingsley G. Morse Jr. wrote: Hi Rebecca, Julian and all science minded pythonistas of debian, great and small! 3.) The following one-liner suggests 44 debian packages might be affected by the breaks Rebecca said would be caused by pandas 2.x: $ for s in augur cnvkit dyda emperor esda mirtop pymatgen pyranges python-anndata python-biom-format python-cooler python-nanoget python-skbio python-ulmo q2-quality-control q2-demux q2-taxa q2-types q2templates sklearn-pandas ; do apt-cache search "$s" ; done | less I'd say don't worry about pymatgen in any case. It's got a new version to upgrade to, and they make regular releases. Drew
pandas 1.5 -> 2.1?
Hi Rebecca, Julian and all science minded pythonistas of debian, great and small! I like your correspondence about upgrading from version 1.5 of pandas to 2.1. It's open, scientific and explores the ideal of proceeding wisely in a matter of public interest. My humble thoughts are: 1.) Rebecca: *Why* did you write that you'd like to move forward with the pandas 1.5 -> 2.1 transition? What's your reason? 2.) What may be the advantage of migrating to version 3.0 of Cython? 3.) The following one-liner suggests 44 debian packages might be affected by the breaks Rebecca said would be caused by pandas 2.x: $ for s in augur cnvkit dyda emperor esda mirtop pymatgen pyranges python-anndata python-biom-format python-cooler python-nanoget python-skbio python-ulmo q2-quality-control q2-demux q2-taxa q2-types q2templates sklearn-pandas ; do apt-cache search "$s" ; done | less 4.) The break that worries me the most is sklearn-pandas, because it seems to me that sklearn is popular and fundamental. Comment welcome, Kingsley On 12/10/2023 20:16, Julian Gilbey wrote: > On Sun, Dec 10, 2023 at 01:06:01PM +, Rebecca N. Palmer wrote: > > I'd like to move forward with the pandas 1.5 -> 2.1 transition reasonably > > soon. > > > > Given that pandas 2.x is *not* required for Python 3.12 (but is required for > > Cython 3.0), should we wait for the Python 3.12 transition to be done first? > > Well, I have seen at least one package that has an RC bug for the > Python 3.12 transition that might be because it's still using an old > version of cython3 :( So it's a bit of chicken-and-egg - having Cython > 3.0 might be very helpful. But then there is this list of 28 packages > broken by pandas 2.x. On the other hand, these will need fixing at > some point soon anyway, so I'd be in favour of doing the pandas > transition now, which will allow Cython 3.0 to move into unstable. > > Just my 2 cents' worth... > > Best wishes, > >Julian > -- Time is the fire in which we all burn.
Re: Bug#1043240: transition: pandas 1.5 -> 2.1
On Sun, Dec 10, 2023 at 01:06:01PM +, Rebecca N. Palmer wrote: > I'd like to move forward with the pandas 1.5 -> 2.1 transition reasonably > soon. > > Given that pandas 2.x is *not* required for Python 3.12 (but is required for > Cython 3.0), should we wait for the Python 3.12 transition to be done first? Well, I have seen at least one package that has an RC bug for the Python 3.12 transition that might be because it's still using an old version of cython3 :( So it's a bit of chicken-and-egg - having Cython 3.0 might be very helpful. But then there is this list of 28 packages broken by pandas 2.x. On the other hand, these will need fixing at some point soon anyway, so I'd be in favour of doing the pandas transition now, which will allow Cython 3.0 to move into unstable. Just my 2 cents' worth... Best wishes, Julian
Bug#1043240: transition: pandas 1.5 -> 2.1
I'd like to move forward with the pandas 1.5 -> 2.1 transition reasonably soon. Given that pandas 2.x is *not* required for Python 3.12 (but is required for Cython 3.0), should we wait for the Python 3.12 transition to be done first? These are broken by pandas 2.x and have a possible (but untested) fix in their bug - please test and apply it: dask(?) dials influxdb-python* python-altair python-feather-format python-upsetplot seaborn tqdm* (* = this package is currently also broken for a non-pandas reason, probably Python 3.12, that I don't have a fix for) These are broken by pandas 2.x and have no known-to-me fix: augur cnvkit dyda emperor esda mirtop pymatgen pyranges python-anndata python-biom-format python-cooler python-nanoget python-skbio python-ulmo q2-quality-control q2-demux q2-taxa q2-types q2templates sklearn-pandas Some generic things to try are pandas.util.testing -> pandas.testing, .iteritems() -> .items(), and if one exists, a more recent upstream version. Is this an acceptable amount of breakage or should we continue to wait? Bear in mind that if we wait too long, we may be forced into it by some transition further up the stack (e.g. a future Python or numpy) that breaks pandas 1.x. Build logs: https://launchpad.net/~rebecca-palmer/+archive/ubuntu/pandas2p1/+builds?build_text=_state=failed https://launchpad.net/~rebecca-palmer/+archive/ubuntu/pandas2p1n/+builds?build_text=_state=failed (The second is more recent, but includes fewer packages.) Autopkgtest logs: https://qa.debian.org/excuses.php?experimental=1=pandas (Because of the Python 3.12 transition, this may currently be wrong about what is a regression and what is not.)
Re: Bug#1022571: transition: pandas 1.3 -> 1.5
Hi Graham, Am Wed, Jan 25, 2023 at 02:28:37PM +0200 schrieb Graham Inggs: > On Wed, 25 Jan 2023 at 11:43, Andreas Tille wrote: > > I was motivated for this patch by Diane's statement on the Debian Med > > matrix channel that dask and dask.distributed are interconnected. If > > the autopkgtest on Salsa CI[4] would have passed I would have uploaded. > > We will temporarily remove dask.distributed from testing, which should > unblock dask and pandas. Thanks a lot! > No uploads please, until dask and pandas have migrated. Yep Andreas. -- http://fam-tille.de
Re: Bug#1022571: transition: pandas 1.3 -> 1.5
Hi Rebecca, Andreas On Wed, 25 Jan 2023 at 11:43, Andreas Tille wrote: > I was motivated for this patch by Diane's statement on the Debian Med > matrix channel that dask and dask.distributed are interconnected. If > the autopkgtest on Salsa CI[4] would have passed I would have uploaded. We will temporarily remove dask.distributed from testing, which should unblock dask and pandas. No uploads please, until dask and pandas have migrated. Regards Graham
Re: Bug#1022571: transition: pandas 1.3 -> 1.5
Hi Graham Am Tue, Jan 24, 2023 at 12:50:41PM +0200 schrieb Graham Inggs: > > On Tue, 24 Jan 2023 at 00:16, Rebecca N. Palmer > wrote: > > The remaining autopkgtest failures blocking pandas are: > > - dipy/amd64 and snakemake/i386 look like random flakiness: please retry > > them. (I think DMs can't use the self-service interface.) Or for dipy, > > see #1029533 for a possible fix. > > These have cleared up. > > > - python-xarray/i386 looks like xarray not pandas: see #1004869. > > Sometimes britney is too greedy and pins too many packages from > unstable. I managed to find the right combination [1] and got the > test to pass without python-xarray. Thanks a lot. > > - dask and dask.distributed have to go together, with or before pandas. > > There's nothing in the currently uploaded packages that specifies > this, although I do see a commit on salsa [2]. I was motivated for this patch by Diane's statement on the Debian Med matrix channel that dask and dask.distributed are interconnected. If the autopkgtest on Salsa CI[4] would have passed I would have uploaded. Unfortunately I have no idea how to deal with those PytestUnraisableExceptionWarnings raised in this test and thus I did not upload. > I tried running dask.distributed's autopkgtest with dask and itself > from unstable [3], but no luck. Diane, could you please comment on this or at least throw some ENOTIME error to let others know. Thanks a lot for you contribution in any case Andreas. > [1] https://ci.debian.net/packages/p/python-xarray/testing/i386/ > [2] > https://salsa.debian.org/python-team/packages/dask.distributed/-/commit/9df3fe1a0ea7538f742ebb162379e3cd50b388e6 > [3] https://ci.debian.net/packages/d/dask.distributed/testing/amd64/ [4] https://salsa.debian.org/python-team/packages/dask.distributed/-/jobs/3834919 -- http://fam-tille.de
Re: Bug#1022571: transition: pandas 1.3 -> 1.5
Hi Rebecca On Tue, 24 Jan 2023 at 00:16, Rebecca N. Palmer wrote: > The remaining autopkgtest failures blocking pandas are: > - dipy/amd64 and snakemake/i386 look like random flakiness: please retry > them. (I think DMs can't use the self-service interface.) Or for dipy, > see #1029533 for a possible fix. These have cleared up. > - python-xarray/i386 looks like xarray not pandas: see #1004869. Sometimes britney is too greedy and pins too many packages from unstable. I managed to find the right combination [1] and got the test to pass without python-xarray. > - dask and dask.distributed have to go together, with or before pandas. There's nothing in the currently uploaded packages that specifies this, although I do see a commit on salsa [2]. I tried running dask.distributed's autopkgtest with dask and itself from unstable [3], but no luck. Regards Graham [1] https://ci.debian.net/packages/p/python-xarray/testing/i386/ [2] https://salsa.debian.org/python-team/packages/dask.distributed/-/commit/9df3fe1a0ea7538f742ebb162379e3cd50b388e6 [3] https://ci.debian.net/packages/d/dask.distributed/testing/amd64/
Re: Bug#1022571: transition: pandas 1.3 -> 1.5
The remaining autopkgtest failures blocking pandas are: - dipy/amd64 and snakemake/i386 look like random flakiness: please retry them. (I think DMs can't use the self-service interface.) Or for dipy, see #1029533 for a possible fix. - python-xarray/i386 looks like xarray not pandas: see #1004869. - dask and dask.distributed have to go together, with or before pandas.
Re: transition: pandas 1.3 -> 1.5
Am Mon, Jan 09, 2023 at 01:44:40PM +0200 schrieb Graham Inggs: > Hi Rebecca > > On Fri, 6 Jan 2023 at 10:30, Rebecca N. Palmer > wrote: > > Should this go ahead before the freeze? I think yes if the new dask > > works, but am open to disagreement. > > Please go ahead. With numpy 1:1.24.1-2 built on all release > architectures, now is a good time. Fine for me. Thanks a lot for working on pandas Andreas. -- http://fam-tille.de
Re: transition: pandas 1.3 -> 1.5
Hi Rebecca On Fri, 6 Jan 2023 at 10:30, Rebecca N. Palmer wrote: > Should this go ahead before the freeze? I think yes if the new dask > works, but am open to disagreement. Please go ahead. With numpy 1:1.24.1-2 built on all release architectures, now is a good time. Regards Graham
Re: Bug#1022571: transition: pandas 1.3 -> 1.5
skbio is fixed, ulmo is maybe fixed but untested. dask is unclear (see #1027254 for details): it looks like it's probably fixable, but I don't have an actual working fix yet. There has been no response from its maintainers.
Re: transition: pandas 1.3 -> 1.5
Should this go ahead before the freeze? I think yes if the new dask works, but am open to disagreement. Issues this would fix: python3.11 #1023965: We're currently ignoring these test failures. Mystery autopkgtest failure (fails without any listed individual test failure, started 2022-11-13) https://ci.debian.net/packages/p/pandas/unstable/amd64/ matplotlib 3.6 (not reported yet): test failures (The fixes in Salsa for #1026351 and #1027576 would probably work on either pandas version.) Reverse dependencies that would break: dask #1025393: Probably fixed upstream, but I haven't checked either whether that works or whether it breaks anything else; I'm working on this. ulmo #1017573: Looks easy to fix but I haven't tried yet. skbio #1017574 emperor #1024820: Suggested there that we should ignore this.
Re: pandas 1.3.5+dfsg-2 autopkgtest failures
pandas has now migrated (on both Debian and Ubuntu). I have uploaded versions of statsmodels and snakemake that should (almost always) avoid these test failures, and reported the partd failure as #1005045.
Re: pandas 1.3.5+dfsg-2 autopkgtest failures
Drew Parsons writes: > Thanks Rebecca. Looks like the mdtraj error is transient, passes > eventually. Makes it hard to debug robustly. FWIW, I've found rr very helpful in such cases. -- Aaron M. Ucko, KB1CJC (amu at alum.mit.edu, ucko at debian.org) http://www.mit.edu/~amu/ | http://stuff.mit.edu/cgi/finger/?a...@monk.mit.edu
Re: pandas 1.3.5+dfsg-2 autopkgtest failures
On 2022-02-03 23:01, Rebecca N. Palmer wrote: In Debian (only change from -1 should be disabling some tests, i.e. real pandas-related failures are unlikely): mdtraj/amd64: upstream tests pass but a warning is printed to stderr, which autopkgtest counts as a fail partd/ppc64el: save/load of a plain string fails snakemake/i386: hang in test_pipes_fail statsmodels/armhf: wrong answer in TestDFMComplex These have all happened before, suggesting they're random, i.e. bugs in these packages not in pandas. (For partd, the other known instance was on a different architecture (s390x), but it looks like partd only uses pandas when it's handling pandas objects, which this test isn't.) Thanks Rebecca. Looks like the mdtraj error is transient, passes eventually. Makes it hard to debug robustly. From the error message "ValueError: Invalid file descriptor: -1" it looks like an actual error, so I don't want to immediately ignore it with allow-stderr. The mdtraj tests themselves pass so I guess it has something to do with shutting down pytest. "Invalid file descriptor" suggests an object is deleted before a reference to it is removed, though strange for that to show up in a python context. Drew
pandas 1.3.5+dfsg-2 autopkgtest failures
In Debian (only change from -1 should be disabling some tests, i.e. real pandas-related failures are unlikely): mdtraj/amd64: upstream tests pass but a warning is printed to stderr, which autopkgtest counts as a fail partd/ppc64el: save/load of a plain string fails snakemake/i386: hang in test_pipes_fail statsmodels/armhf: wrong answer in TestDFMComplex These have all happened before, suggesting they're random, i.e. bugs in these packages not in pandas. (For partd, the other known instance was on a different architecture (s390x), but it looks like partd only uses pandas when it's handling pandas objects, which this test isn't.) In Ubuntu (their first build of 1.3.x, i.e. real pandas-related failures are more plausible): poliastro, python-xarray: These fail with the same error with either new pandas or new matplotlib, and have not had a reference run since then. I suspect they'd now fail every time. python-anndata: Known in Debian as #1001349, fixed there but the fix fails to build in Ubuntu.
Re: Bug#999415: transition: pandas 1.1 -> 1.3 - to unstable now or not?
On Sun, 28 Nov 2021, Rebecca N. Palmer wrote: After build-testing about half of the reverse dependencies, failures that look new-pandas-related are cfgrib #1000726, joypy #1000727, python-skbio #1000752, and maybe hyperspy (not filed yet). python-skbio and hyperspy already FTBFS for unrelated reasons (but fail more tests with new pandas), and joypy looks trivially fixable. Given this and expecting to find a similar number in the other half, against pandas 1.3 working on python3.10 while 1.1 doesn't (#1000422), would you prefer to have pandas 1.3 in unstable now, or not? My vote would be: go for it, but then again, I don't maintain any of pandas BDs. Thanks, Scott
Re: Bug#999415: transition: pandas 1.1 -> 1.3 - to unstable now or not?
After build-testing about half of the reverse dependencies, failures that look new-pandas-related are cfgrib #1000726, joypy #1000727, python-skbio #1000752, and maybe hyperspy (not filed yet). python-skbio and hyperspy already FTBFS for unrelated reasons (but fail more tests with new pandas), and joypy looks trivially fixable. Given this and expecting to find a similar number in the other half, against pandas 1.3 working on python3.10 while 1.1 doesn't (#1000422), would you prefer to have pandas 1.3 in unstable now, or not?
Re: Bug#999415: transition: pandas 1.1 -> 1.3
Control: block 996584 by -1 On 21/11/2021 06:30, Graham Inggs wrote: It might be easier to upgrade to 1.2.4 first, then 1,3,x later. Probably not, given that we're now trying to add Python 3.10 support (#996584), and pandas upstream's work on that is recent (maybe still unfinished): https://github.com/pandas-dev/pandas/issues/41940 The 1.3.x currently in Salsa built last time I tried (before Python 3.10), but is (intentionally, to avoid FTBFS) missing some documentation, and I haven't tested the reverse dependencies yet.
Re: transition: pandas 1.1 -> 1.3
It might be easier to upgrade to 1.2.4 first, then 1,3,x later.
Re: Upgrading pandas
On 2021-11-12 14:06, George N. White III wrote: On Wed, 10 Nov 2021 at 13:14, Andreas Tille wrote: I wonder what you think here whether it makes sense to stop providing the docs for this package. Packaging itself is complex enough and spending extra amounts of work for something users can easily lookup online could be at least discussed. If the answer is: Yes, lets keep the docs we should start packaging sphinx-panels first. There are use cases where having the documentation is important, such as scientific field work, including oceanographic cruises, where "online" is expensive if not impossible, and when available has limited bandwidth needed for data uploads, weather downloads, etc. Agreed. It would be better to have documentation available with pretty formatting switched off (if that is possible, e.g. by commenting out reference to sphinx themes ), than to not have any documentation at all. The package does not build yet since the documentation needs sphinx_panels. Simply patching it out leads to nbsphinx.NotebookError: TypeError in user_guide/style.ipynb: 'coroutine' object is not subscriptable This specific error might be easy to fix. The patch for mdtraj was more or less trivial, https://github.com/mdtraj/mdtraj/pull/1687 , just replace shell.get_msg with kc.get_shell_msg Drew
Re: Upgrading pandas
On Wed, 10 Nov 2021 at 13:14, Andreas Tille wrote: > Hi, > > pandas is lagging behind upstream by several versions. I guess we > should try to get in sync with upstream a bit more. Thus I upgraded Git > and tried to adapt the patches (not sure whether I succeeded completely > - checking would be really welcome). > > The package does not build yet since the documentation needs sphinx_panels. > Simply patching it out leads to > >nbsphinx.NotebookError: TypeError in user_guide/style.ipynb: >'coroutine' object is not subscriptable > > and I stopped here. I wonder what you think here whether it makes sense > to stop providing the docs for this package. Packaging itself is complex > enough and spending extra amounts of work for something users can easily > lookup online could be at least discussed. If the answer is: Yes, lets > keep the docs we should start packaging sphinx-panels first. > > There are use cases where having the documentation is important, such as scientific field work, including oceanographic cruises, where "online" is expensive if not impossible, and when available has limited bandwidth needed for data uploads, weather downloads, etc. -- George N. White III
Re: transition: pandas 1.1 -> 1.3
I was already planning to try a build+autopkgtest of the reverse dependencies, and probably an upload to experimental (it's also common for pandas to fail a few tests on non-amd64), at some point, but not just yet. (The last time I did this was #969650.) Thanks for the tools suggestions. I am currently working on pandas itself: I suspect the documentation isn't the only issue.
Re: transition: pandas 1.1 -> 1.3
Hi Rebecca, Am Thu, Nov 11, 2021 at 09:31:12PM + schrieb Rebecca N. Palmer: > I was already planning to try a build+autopkgtest of the reverse > dependencies, and probably an upload to experimental (it's also common for > pandas to fail a few tests on non-amd64), at some point, but not just yet. > (The last time I did this was #969650.) > > Thanks for the tools suggestions. > > I am currently working on pandas itself: I suspect the documentation isn't > the only issue. Thanks a lot for taking over Andreas. -- http://fam-tille.de
Re: transition: pandas 1.1 -> 1.3
On 11 November 2021 5:44:09 am IST, Drew Parsons wrote: >On 2021-11-10 20:19, Rebecca N. Palmer wrote: >> Source: pandas >> >> On 10/11/2021 17:14, Andreas Tille wrote: >>> pandas is lagging behind upstream by several versions. I guess we >>> should try to get in sync with upstream a bit more. >> >> Yes, but please don't upload this yet: it's common for a pandas >> upgrade to break reverse dependencies. > > >If the new pandas builds successfully, then certainly upload it to >experimental first. We can test dependencies from there. That makes sense, but other than that, I'd really recommend also to build the reverse-"build"-deps with ruby-team/meta or ratt. Uploading to experimental and checking Britney's pseudoexcuses would only warrant for failing autopkgtests in other packages because of new pandas. Regards, Nilesh
Re: transition: pandas 1.1 -> 1.3
Hi, Am Thu, Nov 11, 2021 at 07:17:27PM +0530 schrieb Nilesh Patra: > > >If the new pandas builds successfully, then certainly upload it to > >experimental first. We can test dependencies from there. ^^ > That makes sense, but other than that, I'd really recommend also to build the > reverse-"build"-deps with ruby-team/meta or ratt. > > Uploading to experimental and checking Britney's pseudoexcuses would only > warrant for failing autopkgtests in other packages because of new pandas. That's all fine but the precondition is not fulfilled (see above) and we need people who fix the build first ... than we can follow all those hints. My initial attempt was just a kickstart to get something happen but I do not intend to continue working on this in this month (may be year). Kind regards Andreas. -- http://fam-tille.de
Re: Upgrading pandas
On 11 November 2021 1:37:57 am IST, Gordon Ball wrote: >On Wed, Nov 10, 2021 at 07:32:53PM +0100, Timo Röhling wrote: >> Hi Andreas, >> >> * Andreas Tille [2021-11-10 18:14]: >> > nbsphinx.NotebookError: TypeError in user_guide/style.ipynb: >> > 'coroutine' object is not subscriptable >> This seems to be an issue with nbconvert 5.6.1, which apparently is >> not playing nice with updated Jupyter notebook packages [1]. >> > >The proximate cause was updating jupyter-client to 7, which changes some >existing APIs to be asynchronous. This requires patches or new versions >to some other packages (nbconvert included), and probably some other >packages which import jupyter bits and pieces directly. I'm working on >it. > >The jupyter ecosystem is unfortunately suffering from a bit of runaway >complexity, which is making keeping up with all the bits (particularly >the web-connected bits, like notebook and ipywidgets) difficult. > >> > I wonder what you think here whether it makes sense to stop >> > providing the docs for this package. >> Even though I usually look up documentation online, I'm a bit >> hesitant to drop the documentation completely, because I do need to >> work "in the field" without Internet connection from time to time. >> >> Maybe, as a compromise, we can cut out all the notebooks^H bells and >> whistles and limit the offline documentation to the API reference >> itself, which is arguably the most useful part? I don't know >> how easy it is to trim down the documentation, though. >> > >This I would consider. When working on python packages, my feeling is >that the documentation is a frequent cause of problems (sphinx >extensions, privacy breaches in templates, unpackaged themes, anything >that is meant to embed code), and honestly I don't think compiling it to >HTML adds very much value to users. If the documentation source is in >something which can be directly human-consumed like markdown or >restructuredtext, I'm inclined to think having that in the source >package is sufficient for the majority of use-cases. > >Gordon >
Re: transition: pandas 1.1 -> 1.3
On 2021-11-10 20:19, Rebecca N. Palmer wrote: Source: pandas On 10/11/2021 17:14, Andreas Tille wrote: pandas is lagging behind upstream by several versions. I guess we should try to get in sync with upstream a bit more. Yes, but please don't upload this yet: it's common for a pandas upgrade to break reverse dependencies. If the new pandas builds successfully, then certainly upload it to experimental first. We can test dependencies from there. Drew
Re: Upgrading pandas
Le mercredi 10 novembre 2021 à 19:36 +, Alastair McKinstry a écrit : > > > On 10/11/2021, 16:33, "Timo Röhling" wrote: > > Hi Andreas, > > * Andreas Tille [2021-11-10 18:14]: > > nbsphinx.NotebookError: TypeError in user_guide/style.ipynb: > > 'coroutine' object is not subscriptable > This seems to be an issue with nbconvert 5.6.1, which apparently > is > not playing nice with updated Jupyter notebook packages [1]. > > > Cheers > Timo > > > [1] https://stackoverflow.com/a/67061046/13476707 > > I've the same error in python-xarray, both 0.20.1 I'm packaging and > the current 0.19.0 now FTBFS due to this. > Can we prioritise upgrading nbconvert to 6+ ? > I've wanted to do that since months already, but never found the time to actually get it done. Here are my notes on the matter -- my last serious attempt dates back to july, so things might have moved (hopefully not backwards...) : - nbconvert wants a newer nbsphinx ; - nbsphinx wants a more recent ipywidgets ; - ipywidgets wants jupyterlab ; - I have a very preliminary work on jupyterlab where: * I managed to "compile" its packages coreutils, settingregistry, statedb, nbformat, observables, rendermime-interfaces, mathjax2, metapackage and pdf-extension * for ui-components, typestyle would be needed ; * for vdom, @nteract/transform-vdom would be needed ; - typestyle depends on csstype and free-style ; - csstype officially depends from prettier but that can be worked around. - freestyle I don't know what was lacking. as you'll notice, we quickly go out of Debian Python Team's reach to wander into the savage lands of Debian JavaScript Team -- of which I'm a member too. Cheers, J.Puydt
Re: Upgrading pandas
On Wed, Nov 10, 2021 at 07:32:53PM +0100, Timo Röhling wrote: > Hi Andreas, > > * Andreas Tille [2021-11-10 18:14]: > > nbsphinx.NotebookError: TypeError in user_guide/style.ipynb: > > 'coroutine' object is not subscriptable > This seems to be an issue with nbconvert 5.6.1, which apparently is > not playing nice with updated Jupyter notebook packages [1]. > The proximate cause was updating jupyter-client to 7, which changes some existing APIs to be asynchronous. This requires patches or new versions to some other packages (nbconvert included), and probably some other packages which import jupyter bits and pieces directly. I'm working on it. The jupyter ecosystem is unfortunately suffering from a bit of runaway complexity, which is making keeping up with all the bits (particularly the web-connected bits, like notebook and ipywidgets) difficult. > > I wonder what you think here whether it makes sense to stop > > providing the docs for this package. > Even though I usually look up documentation online, I'm a bit > hesitant to drop the documentation completely, because I do need to > work "in the field" without Internet connection from time to time. > > Maybe, as a compromise, we can cut out all the notebooks^H bells and > whistles and limit the offline documentation to the API reference > itself, which is arguably the most useful part? I don't know > how easy it is to trim down the documentation, though. > This I would consider. When working on python packages, my feeling is that the documentation is a frequent cause of problems (sphinx extensions, privacy breaches in templates, unpackaged themes, anything that is meant to embed code), and honestly I don't think compiling it to HTML adds very much value to users. If the documentation source is in something which can be directly human-consumed like markdown or restructuredtext, I'm inclined to think having that in the source package is sufficient for the majority of use-cases. Gordon
Re: Upgrading pandas
On 10/11/2021, 16:33, "Timo Röhling" wrote: Hi Andreas, * Andreas Tille [2021-11-10 18:14]: > nbsphinx.NotebookError: TypeError in user_guide/style.ipynb: > 'coroutine' object is not subscriptable This seems to be an issue with nbconvert 5.6.1, which apparently is not playing nice with updated Jupyter notebook packages [1]. Cheers Timo [1] https://stackoverflow.com/a/67061046/13476707 I've the same error in python-xarray, both 0.20.1 I'm packaging and the current 0.19.0 now FTBFS due to this. Can we prioritise upgrading nbconvert to 6+ ? Best regards Alastair
transition: pandas 1.1 -> 1.3
Source: pandas Version: 1.1.5+dfsg-2 Severity: wishlist On 10/11/2021 17:14, Andreas Tille wrote: pandas is lagging behind upstream by several versions. I guess we should try to get in sync with upstream a bit more. Yes, but please don't upload this yet: it's common for a pandas upgrade to break reverse dependencies. The package does not build yet since the documentation needs sphinx_panels. On 10/11/2021 18:32, Timo Röhling wrote> Maybe, as a compromise, we can cut out all the notebooks^H bells and whistles and limit the offline documentation to the API reference itself, which is arguably the most useful part? I don't know how easy it is to trim down the documentation, though. The pandas (and probably statsmodels) documentation already is modified to build with fewer dependencies - see sphinx_no_pandas_theme.patch.
Re: Upgrading pandas
Am Wed, Nov 10, 2021 at 07:36:00PM + schrieb Alastair McKinstry: > > I've the same error in python-xarray, both 0.20.1 I'm packaging and the > current 0.19.0 now FTBFS due to this. > Can we prioritise upgrading nbconvert to 6+ ? I guess you should forward this question to Debian Python Team. Kind regards Andreas. -- http://fam-tille.de
Re: Upgrading pandas
Hi Timo, Am Wed, Nov 10, 2021 at 07:32:53PM +0100 schrieb Timo Röhling: > * Andreas Tille [2021-11-10 18:14]: > > nbsphinx.NotebookError: TypeError in user_guide/style.ipynb: > > 'coroutine' object is not subscriptable > This seems to be an issue with nbconvert 5.6.1, which apparently is > not playing nice with updated Jupyter notebook packages [1]. OK, if you think it is not really related to not packaged sphinx-panels I've pushed my workaround for that issue. Any patch that might solve the nbconvert issue is welcome. ;-) > > I wonder what you think here whether it makes sense to stop > > providing the docs for this package. > Even though I usually look up documentation online, I'm a bit > hesitant to drop the documentation completely, because I do need to > work "in the field" without Internet connection from time to time. > > Maybe, as a compromise, we can cut out all the notebooks^H bells and > whistles and limit the offline documentation to the API reference > itself, which is arguably the most useful part? I don't know > how easy it is to trim down the documentation, though. I agree that I do not want to remove the doc if its possible to create it. It would be great if someone would find a solution that enables building it. Kind regards Andreas. > [1] https://stackoverflow.com/a/67061046/13476707 -- http://fam-tille.de
Re: Upgrading pandas
Hi Andreas, * Andreas Tille [2021-11-10 18:14]: nbsphinx.NotebookError: TypeError in user_guide/style.ipynb: 'coroutine' object is not subscriptable This seems to be an issue with nbconvert 5.6.1, which apparently is not playing nice with updated Jupyter notebook packages [1]. I wonder what you think here whether it makes sense to stop providing the docs for this package. Even though I usually look up documentation online, I'm a bit hesitant to drop the documentation completely, because I do need to work "in the field" without Internet connection from time to time. Maybe, as a compromise, we can cut out all the notebooks^H bells and whistles and limit the offline documentation to the API reference itself, which is arguably the most useful part? I don't know how easy it is to trim down the documentation, though. Cheers Timo [1] https://stackoverflow.com/a/67061046/13476707 -- ⢀⣴⠾⠻⢶⣦⠀ ╭╮ ⣾⠁⢠⠒⠀⣿⡁ │ Timo Röhling │ ⢿⡄⠘⠷⠚⠋⠀ │ 9B03 EBB9 8300 DF97 C2B1 23BF CC8C 6BDD 1403 F4CA │ ⠈⠳⣄ ╰╯ signature.asc Description: PGP signature
Upgrading pandas
Hi, pandas is lagging behind upstream by several versions. I guess we should try to get in sync with upstream a bit more. Thus I upgraded Git and tried to adapt the patches (not sure whether I succeeded completely - checking would be really welcome). The package does not build yet since the documentation needs sphinx_panels. Simply patching it out leads to nbsphinx.NotebookError: TypeError in user_guide/style.ipynb: 'coroutine' object is not subscriptable and I stopped here. I wonder what you think here whether it makes sense to stop providing the docs for this package. Packaging itself is complex enough and spending extra amounts of work for something users can easily lookup online could be at least discussed. If the answer is: Yes, lets keep the docs we should start packaging sphinx-panels first. Kind regards Andreas. -- http://fam-tille.de
Re: Should the pandas 1.x transition be forced (breaking python-biom-format and q2-demux/q2-types) or keep waiting?
Second attempt uploaded. Of the reverse dependencies with known fixes, python-feather-format has been uploaded but q2templates hasn't.
Re: Should the pandas 1.x transition be forced (breaking python-biom-format and q2-demux/q2-types) or keep waiting?
Package FTBFS everywhere, for multiple reasons but looks like the new pytest_asyncio is at least one of them; will investigate further later.
Re: Should the pandas 1.x transition be forced (breaking python-biom-format and q2-demux/q2-types) or keep waiting?
Not yet, it should at least be 1.0.5 from Salsa. (Also, q2-* have autopkgtests so it will probably get stuck in -proposed.) On 25/08/2020 08:19, Graham Inggs wrote: Hi Rebecca On Mon, 24 Aug 2020 at 01:03, Rebecca N. Palmer wrote: pandas 1.0 has been in experimental for 6+ months, because it breaks some of its reverse dependencies: ... Ubuntu freezes this Thursday, but I suspect I may have left this too late for that. We can sync 1.0.4+dfsg-1 from experimental to Ubuntu right now if you want. Regards Graham
Re: Should the pandas 1.x transition be forced (breaking python-biom-format and q2-demux/q2-types) or keep waiting?
Hi Rebecca On Mon, 24 Aug 2020 at 01:03, Rebecca N. Palmer wrote: > pandas 1.0 has been in experimental for 6+ months, because it breaks > some of its reverse dependencies: ... > Ubuntu freezes this Thursday, but I suspect I may have left this too > late for that. We can sync 1.0.4+dfsg-1 from experimental to Ubuntu right now if you want. Regards Graham
Re: Should the pandas 1.x transition be forced (breaking python-biom-format and q2-demux/q2-types) or keep waiting?
Hi Rebecca, On Sun, Aug 23, 2020 at 11:30:19PM +0100, Rebecca N. Palmer wrote: > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=950430 > > pandas 1.0 has been in experimental for 6+ months, because it breaks some of > its reverse dependencies: > > In testing: > #950924 python-feather-format (has patch, no maintainer response to it) > #950931 q2templates (has upstream fix) > #950929 python-biom-format (no known solution) > > Not in testing: > #950932 q2-types > #950933 q2-demux > > This is blocking at least one proposed new package (pymatgen #962268), and > making work when a new numpy/etc breaks the old pandas. On the other hand, > this year feels like a bad time to break bioinformatics packages. I personally would voto for a soonish upload of the new pandas version. This gives more time to fix reverse dependencies before the freezw. > Ubuntu freezes this Thursday, but I suspect I may have left this too late > for that. I admit I have no idea how our uploads might influence the Ubuntu release. > Upstream have now released pandas 1.1, and I might go straight to that but > first need to check whether it breaks anything more. (It isn't supposed to > have API changes, but the new warnings may break fail-on-warning tests.) I think following upstream closely and fixing low popcon leaf packages after this is a sensible strategy. Thanks a lot for all your work Andreas. -- http://fam-tille.de
Re: Should the pandas 1.x transition be forced (breaking python-biom-format and q2-demux/q2-types) or keep waiting?
On 2020-08-24 06:30, Rebecca N. Palmer wrote: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=950430 pandas 1.0 has been in experimental for 6+ months, because it breaks some of its reverse dependencies: In testing: #950924 python-feather-format (has patch, no maintainer response to it) python-feather-format is team-maintained, so I can upload its patch. Drew
Should the pandas 1.x transition be forced (breaking python-biom-format and q2-demux/q2-types) or keep waiting?
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=950430 pandas 1.0 has been in experimental for 6+ months, because it breaks some of its reverse dependencies: In testing: #950924 python-feather-format (has patch, no maintainer response to it) #950931 q2templates (has upstream fix) #950929 python-biom-format (no known solution) Not in testing: #950932 q2-types #950933 q2-demux This is blocking at least one proposed new package (pymatgen #962268), and making work when a new numpy/etc breaks the old pandas. On the other hand, this year feels like a bad time to break bioinformatics packages. Ubuntu freezes this Thursday, but I suspect I may have left this too late for that. Upstream have now released pandas 1.1, and I might go straight to that but first need to check whether it breaks anything more. (It isn't supposed to have API changes, but the new warnings may break fail-on-warning tests.)
Help needed to adapt python-skbio to Python3(?) / Pandas
Control: tags -1 help
Control: tags -1 upstream
Control: forwarded -1 https://github.com/biocore/scikit-bio/issues/1678
Hi Rebecca,
thanks a lot for working on the Pandas migration.
On Wed, Nov 13, 2019 at 10:19:56PM +, Rebecca N. Palmer wrote:
> The ones that are blocking pandas [1] are python-skbio,
>
> [1] https://qa.debian.org/excuses.php?package=pandas (cnvkit since fixed)
I've checked the issue with some patch from upstream[1] but failed. I
also tried to build HEAD from upstream which also leaves three remaining
errors which I reported in issue #1678[2]:
==
ERROR: test_munging_invalid_type_to_self_type
(skbio.sequence.tests.test_sequence.TestDistance)
--
Traceback (most recent call last):
File
"/build/python-skbio-0.5.5a/.pybuild/cpython3_3.7_skbio/build/skbio/sequence/tests/test_sequence.py",
line 2369, in test_munging_invalid_type_to_self_type
Sequence("ACGT").distance(42)
File
"/build/python-skbio-0.5.5a/.pybuild/cpython3_3.7_skbio/build/skbio/sequence/_sequence.py",
line 1539, in distance
other = self._munge_to_self_type(other, 'distance')
File
"/build/python-skbio-0.5.5a/.pybuild/cpython3_3.7_skbio/build/skbio/sequence/_sequence.py",
line 2161, in _munge_to_self_type
return self.__class__(other)
File
"/build/python-skbio-0.5.5a/.pybuild/cpython3_3.7_skbio/build/skbio/sequence/_sequence.py",
line 622, in __init__
s = np.frombuffer(sequence, dtype=np.uint8)
TypeError: a bytes-like object is required, not 'int'
==
ERROR: test_init_invalid_sequence
(skbio.sequence.tests.test_sequence.TestSequence)
--
Traceback (most recent call last):
File
"/build/python-skbio-0.5.5a/.pybuild/cpython3_3.7_skbio/build/skbio/sequence/tests/test_sequence.py",
line 465, in test_init_invalid_sequence
Sequence(('a', 'b', 'c'))
File
"/build/python-skbio-0.5.5a/.pybuild/cpython3_3.7_skbio/build/skbio/sequence/_sequence.py",
line 622, in __init__
s = np.frombuffer(sequence, dtype=np.uint8)
TypeError: a bytes-like object is required, not 'tuple'
==
FAIL: test_no_variation_pearson
(skbio.stats.distance.tests.test_mantel.MantelTests)
--
Traceback (most recent call last):
File
"/build/python-skbio-0.5.5a/.pybuild/cpython3_3.7_skbio/build/skbio/stats/distance/tests/test_mantel.py",
line 247, in test_no_variation_pearson
npt.assert_equal(obs, (np.nan, np.nan, 3))
File "/usr/lib/python3/dist-packages/numpy/testing/_private/utils.py", line
355, in assert_equal
assert_equal(actual[k], desired[k], 'item=%r\n%s' % (k, err_msg), verbose)
File "/usr/lib/python3/dist-packages/numpy/testing/_private/utils.py", line
427, in assert_equal
raise AssertionError(msg)
AssertionError:
Items are not equal:
item=0
ACTUAL: 0.0
DESIRED: nan
--
Ran 2352 tests in 44.011s
FAILED (SKIP=30, errors=2, failures=1)
I somehow suspect that the two errors above might be caused by a broken
Python3 conversion - any help to fix this would be welcome.
Kind regards
Andreas.
[1]
https://github.com/biocore/scikit-bio/commit/9c061da7e2746aee403b41621f71b118ce5c52f8
[2] https://github.com/biocore/scikit-bio/issues/1678
--
http://fam-tille.de
Re: Remove pyoptical from Debian, [or?] upgrade psychopy to latest upstream to enable Pandas migration
psychopy isn't blocking pandas' testing migration because psychopy already isn't in testing (for reasons unrelated to py2removal). Upstream say it's now Python 3 compatible [0], but I haven't tried to fix its other problems. The ones that are blocking pandas [1] are python-skbio, python-feather-format and dask. [0] https://github.com/psychopy/psychopy/blob/master/setup.cfg#22 [1] https://qa.debian.org/excuses.php?package=pandas (cnvkit since fixed)
Remove pyoptical from Debian, upgrade psychopy to latest upstream to enable Pandas migration
Hi, I stumbled upon pyoptical since its the only package that uses psychopy which in turn is blocking the python-pandas migration to Python3. Psychopy is extremely lagging behind upstream (which probably - not checked might have switched to Python3). From my point of view it makes sense to 1. Upgrade psychopy to latest upstream 2. Remove pyoptical from Debian (dead upstream, low popcon anyway[1]) Any idea to otherwise smoothen the Pandas migration? Kind regards Andreas. [1] https://qa.debian.org/popcon-graph.php?packages=python-pyoptical_vote=on_legend=on_ticks=on_date=_date=_date=_fmt=%25Y-%25m=1 -- http://fam-tille.de
Re: Bug#942106: python3.8 / pandas py2removal
Hi, On Sun, Nov 10, 2019 at 03:18:31PM +0100, Matthias Klose wrote: > tnseq-transit is a leaf package, raising the severity now, could be removed > and re-enter later. Besides I fully agree to this in general python-pypubsub was accepted yesterday and thus new upstream of tnseq-transit with Python3 support uploaded. I also re-uploaded python-pypubsub as source-only upload so testing migration should be soon. > IMO, we should care about the recommends later ... +1 > > (patsy isn't a leaf package for py2removal, it's in a cycle with pandas > > and statsmodels, i.e. for a non-breaking removal they all have to go > > together.) > > ok I'm undoing the NMU from the delayed queue, you'll find it at > https://people.debian.org/~doko/tmp/ Regarding those packages where I'm listed as Uploader feel free to do non-delayed NMUs. Kind regards Andreas. -- http://fam-tille.de
Re: pandas: Python 2 removal, 0.23 -> 0.25 transition
Hi, On Sun, Oct 27, 2019 at 06:12:38AM -0700, Mo Zhou wrote: > Hi Rebecca, > > Personally I fully support the option (a). Afterall nobody likes > taking extra burden especially when there is no upstream support > anymore. +1 > On 2019-10-27 11:01, Rebecca N. Palmer wrote: > > python-biom-format Solved. > > Need a new upstream version to support Python 3: > > metaphlan2 Solved. > > psychopy > > pymvpa2 (upstream say they don't test this, but the package already > > fails its tests) > > stimfit > > tnseq-transit (needs new dependency pypubsub, in NEW) Solved. > > To simplify such testing, I hope to upload pandas 0.25 to experimental > > today. Thanks a lot. I'm fine with breaking some leaf packages (specifically since after your last mail even less ones are affected). Kind regards and thanks a lot for all who care for pandas Andreas. -- http://fam-tille.de
Re: Bug#942106: python3.8 / pandas py2removal
I have uploaded pandas and statsmodels. On 10/11/2019 14:18, Matthias Klose wrote: https://people.debian.org/~doko/tmp/ The patsy one has a bug: as debian/tests/nosetests3 was a symlink to nosetests2, it should have deleted this link and renamed nosetests2 to nosetests3, not deleted nosetests2.
Re: Bug#942106: python3.8 / pandas py2removal
On 10.11.19 14:46, Rebecca N. Palmer wrote: mdp isn't in testing either, but if you're using a policy of "no py2removals that break packages in testing", tnseq-transit (Depends: statsmodels) and possibly stimfit (Recommends: pandas) need to be done as well. Those are both thought to need new upstream versions. tnseq-transit is a leaf package, raising the severity now, could be removed and re-enter later. IMO, we should care about the recommends later ... (patsy isn't a leaf package for py2removal, it's in a cycle with pandas and statsmodels, i.e. for a non-breaking removal they all have to go together.) ok I'm undoing the NMU from the delayed queue, you'll find it at https://people.debian.org/~doko/tmp/
Re: Bug#942106: python3.8 / pandas py2removal
mdp isn't in testing either, but if you're using a policy of "no py2removals that break packages in testing", tnseq-transit (Depends: statsmodels) and possibly stimfit (Recommends: pandas) need to be done as well. Those are both thought to need new upstream versions. (patsy isn't a leaf package for py2removal, it's in a cycle with pandas and statsmodels, i.e. for a non-breaking removal they all have to go together.)
Re: Bug#942106: python3.8 / pandas py2removal
On 10.11.19 14:22, Matthias Klose wrote: patsy is a leaf package, no problem. scikit-learn, needs mdp, pymvpa2, I think that's manageable, so I'm volunteering to do the NMUs, preferably with a 0 or 1 day delay. PyMVPA has other RC issues, is removed from testing, so ignore it for now. and I'm raising the severity of the py2removal bug.
Re: Bug#942106: python3.8 / pandas py2removal
[CCing debian-science] On 10.11.19 13:18, Rebecca N. Palmer wrote: Matthias Klose wrote: yes, please do [raise pandas 0.25 blocking bugs to "important"] Done, but only 2 of them have been fixed since. This leaves 13: has patch or Ubuntu fix: matplotlib2 patsy python-apptools scikit-learn may need more extensive work: cnvkit python-feather-format python-skbio stimfit tnseq-transit already not in testing: mdp psychopy pymvpa q2-types If you can get that done with [pandas] 0.25, fine, It took longer than I was expecting, but pandas 0.25 / statsmodels 0.10 now appear to be working, including with python3.8. (Though we won't actually know if #943732 is gone until mipsel tries to build it.) \o/ and I wouldn't worry too much about the other four breaking packages at this point. Was this intended as... ... a request to upload pandas 0.25 / statsmodels 0.10 to unstable (and apply the Ubuntu py2removal patches to patsy and scikit-learn?), overriding normal py2removal rules? patsy is a leaf package, no problem. scikit-learn, needs mdp, pymvpa2, I think that's manageable, so I'm volunteering to do the NMUs, preferably with a 0 or 1 day delay. So yes, please go ahead. ... a request to split pandas into a pandas2 0.23 and a pandas 0.25? (since 4 is only the number of non-py2removal breakages, and the wiki page https://wiki.debian.org/Python/Python3.8 says to do this if 2.7 and 3.8 need different upstream versions) Should be technically easy, but means going through NEW. ... a statement that once pandas 0.25 works, this is no longer my problem, i.e. that I don't have to fix 0.23? I don't think that's necessary at this point. matplotlib and pandas don't have Python2 packages in Ubuntu anymore, so I can't tell much what is needed here matplotlib has already been split into Python 2 and Python 3 source packages. (matplotlib2 is in Ubuntu, and unbuildable there due to #943924.) Python2 matplotlib are already removed in Ubuntu, I shouldn't have synced matplotlib2 from experimental. According to its Ubuntu build log: https://launchpad.net/ubuntu/+source/matplotlib/3.0.2-2ubuntu1/+build/17942804 matplotlib has one python3.8-specific test failure, test_axes.py::test_pathological_hexbin. This is currently being ignored (#942766) along with (many) errors that also happen on 3.7.
Re: pandas: Python 2 removal, 0.23 -> 0.25 transition
Hi Rebecca, Personally I fully support the option (a). Afterall nobody likes taking extra burden especially when there is no upstream support anymore. (b) is for good wish but impractical. (c) means we just let pandas package rot and die. Even if some portion of users still need the python2 package, they could still rely on pypi for some time. On 2019-10-27 11:01, Rebecca N. Palmer wrote: > Python 3.8 is being added (#942106). pandas <0.25 does not support > Python 3.8[0] (when Ubuntu tried they got 268 test failures [1]), > while pandas>=0.25 does not support Python 2. > > Hence, our options are: > (a) Remove python-pandas and upgrade pandas to 0.25 > (b) Split pandas into two source packages (like matplotlib) so we can > have python-pandas 0.23 and python3-pandas 0.25 > (c) Let pandas be broken on 3.8 for now > I currently favour (a), but this is still open for discussion. > > pandas is currently in a big strongly connected component [2], but it > looks possible to cut it out of that tangle without actually breaking > anything, by removing these 2 build-dependencies: > matplotlib2 (only uses pandas in examples/tests) > python-apptools (only uses pandas in H5TableNode which (according to > codesearch) nothing uses) > > This leaves the following needing to drop Python 2 before pandas can: > > Already supports Python 3: > bcolz > mdp > patsy > python-biom-format > scikit-learn > statsmodels (my package, will deal with it when its dependencies are cleared) > > Need a new upstream version to support Python 3: > metaphlan2 > psychopy > pymvpa2 (upstream say they don't test this, but the package already > fails its tests) > stimfit > tnseq-transit (needs new dependency pypubsub, in NEW) > > Also, the move from pandas 0.23 to 0.25 involves API changes [3], and > hence may break some reverse dependencies. I haven't tested this yet, > but if you maintain a pandas reverse dependency that has a newer > upstream version available, this might be a good time to package it. > > To simplify such testing, I hope to upload pandas 0.25 to experimental today. > > [0] https://github.com/pandas-dev/pandas/issues/29043 > [1] > https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-focal/focal/amd64/p/pandas/20191024_181815_7c017@/log.gz > [2] https://lists.debian.org/debian-python/2019/10/msg00092.html (not > quite that big since we started ignoring Suggests, but still not > something to remove just yet) > [3] https://pandas.pydata.org/pandas-docs/stable/whatsnew/index.html#release
Re: pandas: Python 2 removal, 0.23 -> 0.25 transition
Hi, ne 27. 10. 2019 v 12:33 odesílatel Rebecca N. Palmer < rebecca_pal...@zoho.com> napsal: > Hence, our options are: > (a) Remove python-pandas and upgrade pandas to 0.25 > (b) Split pandas into two source packages (like matplotlib) so we can > have python-pandas 0.23 and python3-pandas 0.25 > (c) Let pandas be broken on 3.8 for now > I currently favour (a), but this is still open for discussion. > +1 for (a). Imho we are still far away from next stable release a this breakage is feasible. -- Best regards Ondřej Nový
pandas: Python 2 removal, 0.23 -> 0.25 transition
Python 3.8 is being added (#942106). pandas <0.25 does not support Python 3.8[0] (when Ubuntu tried they got 268 test failures [1]), while pandas>=0.25 does not support Python 2. Hence, our options are: (a) Remove python-pandas and upgrade pandas to 0.25 (b) Split pandas into two source packages (like matplotlib) so we can have python-pandas 0.23 and python3-pandas 0.25 (c) Let pandas be broken on 3.8 for now I currently favour (a), but this is still open for discussion. pandas is currently in a big strongly connected component [2], but it looks possible to cut it out of that tangle without actually breaking anything, by removing these 2 build-dependencies: matplotlib2 (only uses pandas in examples/tests) python-apptools (only uses pandas in H5TableNode which (according to codesearch) nothing uses) This leaves the following needing to drop Python 2 before pandas can: Already supports Python 3: bcolz mdp patsy python-biom-format scikit-learn statsmodels (my package, will deal with it when its dependencies are cleared) Need a new upstream version to support Python 3: metaphlan2 psychopy pymvpa2 (upstream say they don't test this, but the package already fails its tests) stimfit tnseq-transit (needs new dependency pypubsub, in NEW) Also, the move from pandas 0.23 to 0.25 involves API changes [3], and hence may break some reverse dependencies. I haven't tested this yet, but if you maintain a pandas reverse dependency that has a newer upstream version available, this might be a good time to package it. To simplify such testing, I hope to upload pandas 0.25 to experimental today. [0] https://github.com/pandas-dev/pandas/issues/29043 [1] https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-focal/focal/amd64/p/pandas/20191024_181815_7c017@/log.gz [2] https://lists.debian.org/debian-python/2019/10/msg00092.html (not quite that big since we started ignoring Suggests, but still not something to remove just yet) [3] https://pandas.pydata.org/pandas-docs/stable/whatsnew/index.html#release
Re: Pandas/Statsmodels
On 2019-07-07 19:29, Rebecca N. Palmer wrote: To be clear before I start uploading new upstream versions: was our before-freeze discussion [0-1] a permanent invitation to help (and not just to fix the then-immediate problems)? Absolutely, join the party! You've done great work with pandas. I haven't used it myself yet, but I admire its capabilities. I'd like to get them up to latest upstream in time for the next Ubuntu (freeze 22 August[2]), but am not sure if this is realistic for pandas, given the rather large number of rdeps. Speaking of updates, I want to drop scipy 1.2 into unstable in the coming days if not hours. scipy 1.3 is now released but we should take it through experimental first, it may have changed some API. Drew
Pandas/Statsmodels
To be clear before I start uploading new upstream versions: was our before-freeze discussion [0-1] a permanent invitation to help (and not just to fix the then-immediate problems)? I'd like to get them up to latest upstream in time for the next Ubuntu (freeze 22 August[2]), but am not sure if this is realistic for pandas, given the rather large number of rdeps. [0] https://lists.debian.org/debian-science/2019/02/msg00070.html [1] https://lists.debian.org/debian-science/2019/03/msg3.html [2] https://wiki.ubuntu.com/EoanErmine/ReleaseSchedule
Re: Pandas
Dear Rebecca, On Wed, Feb 27, 2019 at 10:47:31PM +, Rebecca N. Palmer wrote: > Any comment (and preferably upload) on statsmodels? That needs to be > uploaded by the 2nd (in testing by the 12th) if at all, as it has changes > that wouldn't be allowed under full freeze. Short notice: Thanks a lot for your work on statsmodels and pandas. That's really relaxing for lots or rdepends! Please make sure you merge changes like upstream/metadata and other enhancements into the branches of higher versions in Git. I also think the changelog entries of what was released in the past should be copied to d/changelog to have a complete history of released packages in d/changelog. Would you mind adding your ID to Uploaders of these packages for future versions? Thanks again Andreas. -- http://fam-tille.de
Re: Pandas
never mind -- I was the one too late: Version check failed: Your upload included the source package statsmodels, version 0.8.0-8, however unstable already has version 0.8.0-8. ;-) thanks! -- Yaroslav O. Halchenko Center for Open Neuroscience http://centerforopenneuroscience.org Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik signature.asc Description: PGP signature
Re: Pandas
On Fri, 01 Mar 2019, Yaroslav Halchenko wrote: > On Wed, 27 Feb 2019, Rebecca N. Palmer wrote: > > Any comment (and preferably upload) on statsmodels? That needs to be > > uploaded by the 2nd (in testing by the 12th) if at all, as it has changes > > that wouldn't be allowed under full freeze. > lots of great work there -- thanks again. would be shame to miss it. > looks good - building/testing now Hi Rebecca, I've uploaded the build, and was trying to push the release changelog change + tag, but found some on salsa already... was that version/tag uploaded as well/before me? Please advice on how to proceed -- I would've just force pushed mine as the version actually uploaded (if yours wasn't). -- Yaroslav O. Halchenko Center for Open Neuroscience http://centerforopenneuroscience.org Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik
Re: Pandas
On Wed, 27 Feb 2019, Rebecca N. Palmer wrote: > Any comment (and preferably upload) on statsmodels? That needs to be > uploaded by the 2nd (in testing by the 12th) if at all, as it has changes > that wouldn't be allowed under full freeze. lots of great work there -- thanks again. would be shame to miss it. looks good - building/testing now Cheers! -- Yaroslav O. Halchenko Center for Open Neuroscience http://centerforopenneuroscience.org Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik
Re: Pandas
Hi,
just in case you need my help: I'm offline-ish until 2019-03-04
Kind regards, Andreas.
On Wed, Feb 27, 2019 at 10:47:31PM +, Rebecca N. Palmer wrote:
> Any comment (and preferably upload) on statsmodels? That needs to be
> uploaded by the 2nd (in testing by the 12th) if at all, as it has changes
> that wouldn't be allowed under full freeze.
>
> On 27/02/2019 14:46, Yaroslav Halchenko wrote:
> >
> > On Wed, 27 Feb 2019, Andreas Tille wrote:
> > > I'd prefer if you would move the
> > > 0.24 packaging to some separate branch (debian-experimental is covered
> > > but may
> > > be debian-0.24 or something like this?) and keep branch debian for what
> > > we are
> > > really releasing.
> >
> > well "releasing" is a loaded term. I guess you are talking about uploading
> > to
> > unstable so it manages to get into buster. Since "debian" branch already
> > got
> > its 0.24, what about starting debian-buster branch off debian/0.23.3-1 ?
>
> That's what I was planning, like statsmodels already has. (I was going to
> call it 'buster', but I don't care about the name.)
>
> > otherwise -- I am ok to hard-reset and force push debian to the
> > debian/0.23.3-1
> > state -- everyone should just beware of it, and then progress
> > debian-experimental to current state of debian (v0.24.1-972-g1cfbd07c7)
>
> I'd rather not rebase/force-push a public repository (even if it is in
> practice unlikely that anyone other than us has downloaded it). It should
> be possible to get to that state without doing so ("merge" debian into
> debian-experimental, then commit a "revert to 0.23" to debian).
>
> (In general, I slightly prefer the "'debian' is development head, whether
> that's currently unstable or experimental (due to a freeze or a transition)"
> layout. However, that may be because the packages I currently maintain
> rarely needed any changes in unstable while the package was also in
> experimental, and this layout hence avoided branching most of the time.
> Something as central as pandas may need two active branches more often.)
>
>
--
http://fam-tille.de
Re: Pandas
Any comment (and preferably upload) on statsmodels? That needs to be
uploaded by the 2nd (in testing by the 12th) if at all, as it has
changes that wouldn't be allowed under full freeze.
On 27/02/2019 14:46, Yaroslav Halchenko wrote:
On Wed, 27 Feb 2019, Andreas Tille wrote:
I'd prefer if you would move the
0.24 packaging to some separate branch (debian-experimental is covered but may
be debian-0.24 or something like this?) and keep branch debian for what we are
really releasing.
well "releasing" is a loaded term. I guess you are talking about uploading to
unstable so it manages to get into buster. Since "debian" branch already got
its 0.24, what about starting debian-buster branch off debian/0.23.3-1 ?
That's what I was planning, like statsmodels already has. (I was going
to call it 'buster', but I don't care about the name.)
otherwise -- I am ok to hard-reset and force push debian to the debian/0.23.3-1
state -- everyone should just beware of it, and then progress
debian-experimental to current state of debian (v0.24.1-972-g1cfbd07c7)
I'd rather not rebase/force-push a public repository (even if it is in
practice unlikely that anyone other than us has downloaded it). It
should be possible to get to that state without doing so ("merge" debian
into debian-experimental, then commit a "revert to 0.23" to debian).
(In general, I slightly prefer the "'debian' is development head,
whether that's currently unstable or experimental (due to a freeze or a
transition)" layout. However, that may be because the packages I
currently maintain rarely needed any changes in unstable while the
package was also in experimental, and this layout hence avoided
branching most of the time. Something as central as pandas may need two
active branches more often.)
Re: Pandas
On Wed, 27 Feb 2019, Andreas Tille wrote:
> Dear Rebecca,
> On Wed, Feb 27, 2019 at 07:25:26AM +, Rebecca N. Palmer wrote:
> > On 27/02/2019 07:00, Andreas Tille wrote:
> > > Dear Rebecca,
> > > I do not think that there is any
> > > need for a separate branch. Just stick to the debian branch.
> > It's needed because the debian branch contains the attempt at packaging 0.24
> > described earlier in this thread, which it's now too late for.
> You are right. Considering the branching jungle (Yaroslav, could you possibly
for someone jungle is a sweet sweet home! ;)
$> git br -a | grep salsa | grep -e bf- -e enh- -e cythoned | while
read b; do git push salsa :${b//*salsa\//}; done
To salsa.debian.org:science-team/pandas.git
- [deleted] __tent/debian-cythoned-quilt
To salsa.debian.org:science-team/pandas.git
- [deleted] bf-cython
To salsa.debian.org:science-team/pandas.git
- [deleted] bf-i386
To salsa.debian.org:science-team/pandas.git
- [deleted] bf-native-endianness
To salsa.debian.org:science-team/pandas.git
- [deleted] bf-skips
To salsa.debian.org:science-team/pandas.git
- [deleted] bf-unser
To salsa.debian.org:science-team/pandas.git
- [deleted] bf-versions-etc
To salsa.debian.org:science-team/pandas.git
- [deleted] enh-tests-compat-pytables-2.1
and will try to not forget to not push them again ;)
$> git br -a | grep salsa # -- with my annotations
remotes/salsa/master -- some upstream's point of master -- could
also be removed if you like
remotes/salsa/releases -- linear progression of upstream releases
remotes/salsa/debian -- sits on top of releases and carries
packaging
remotes/salsa/debian-experimental -- the one for uploads to
experimental
better now?
> cleanup branches that are not used any more?) I'd prefer if you would move the
> 0.24 packaging to some separate branch (debian-experimental is covered but may
> be debian-0.24 or something like this?) and keep branch debian for what we are
> really releasing.
well "releasing" is a loaded term. I guess you are talking about uploading to
unstable so it manages to get into buster. Since "debian" branch already got
its 0.24, what about starting debian-buster branch off debian/0.23.3-1 ?
otherwise -- I am ok to hard-reset and force push debian to the debian/0.23.3-1
state -- everyone should just beware of it, and then progress
debian-experimental to current state of debian (v0.24.1-972-g1cfbd07c7)
your call
> Thanks again for your work on this
yeap, Thank you Rebecca!
--
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419
WWW: http://www.linkedin.com/in/yarik
Re: Pandas
Dear Rebecca, On Wed, Feb 27, 2019 at 07:25:26AM +, Rebecca N. Palmer wrote: > On 27/02/2019 07:00, Andreas Tille wrote: > > Dear Rebecca, > > I do not think that there is any > > need for a separate branch. Just stick to the debian branch. > > It's needed because the debian branch contains the attempt at packaging 0.24 > described earlier in this thread, which it's now too late for. You are right. Considering the branching jungle (Yaroslav, could you possibly cleanup branches that are not used any more?) I'd prefer if you would move the 0.24 packaging to some separate branch (debian-experimental is covered but may be debian-0.24 or something like this?) and keep branch debian for what we are really releasing. Thanks again for your work on this Andreas. -- http://fam-tille.de
Re: Pandas
On 27/02/2019 07:00, Andreas Tille wrote: Dear Rebecca, I do not think that there is any need for a separate branch. Just stick to the debian branch. It's needed because the debian branch contains the attempt at packaging 0.24 described earlier in this thread, which it's now too late for.
Re: Pandas
Dear Rebecca, On Tue, Feb 26, 2019 at 10:42:49PM +, Rebecca N. Palmer wrote: > I now have fixes for both #918206 (test failure, RC) and #804552 (no > documentation), in those bugs. (Would you prefer to have a salsa branch?) Thanks a lot for working on this. I do not think that there is any need for a separate branch. Just stick to the debian branch. > Several other open pandas bugs appear to be either already fixed or > non-bugs: is it appropriate for me to close them? Sure. If you realise that a bug is not existent any more just add a comment and close it. Thanks again and ping here for sponsering (or upload permissions - if I remember correctly you are DM, right) Andreas. -- http://fam-tille.de
Re: Pandas
I now have fixes for both #918206 (test failure, RC) and #804552 (no documentation), in those bugs. (Would you prefer to have a salsa branch?) Several other open pandas bugs appear to be either already fixed or non-bugs: is it appropriate for me to close them?
Re: Pandas new version
On Wed, 06 Feb 2019, Yaroslav Halchenko wrote:
...
> = ERRORS
> ==
> _ ERROR collecting
> tests/indexes/datetimes/test_tools.py __
> /usr/lib/python2.7/dist-packages/pluggy/hooks.py:284: in __call__
> return self._hookexec(self, self.get_hookimpls(), kwargs)
> /usr/lib/python2.7/dist-packages/pluggy/manager.py:67: in _hookexec
> return self._inner_hookexec(hook, methods, kwargs)
> /usr/lib/python2.7/dist-packages/pluggy/manager.py:61: in
> firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
> /usr/lib/python2.7/dist-packages/_pytest/python.py:224: in
> pytest_pycollect_makeitem
> res = list(collector._genfunctions(name, obj))
> /usr/lib/python2.7/dist-packages/_pytest/python.py:409: in _genfunctions
> self.ihook.pytest_generate_tests(metafunc=metafunc)
> /usr/lib/python2.7/dist-packages/pluggy/hooks.py:284: in __call__
> return self._hookexec(self, self.get_hookimpls(), kwargs)
> /usr/lib/python2.7/dist-packages/pluggy/manager.py:67: in _hookexec
> return self._inner_hookexec(hook, methods, kwargs)
> /usr/lib/python2.7/dist-packages/pluggy/manager.py:61: in
> firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
> /usr/lib/python2.7/dist-packages/_pytest/python.py:133: in
> pytest_generate_tests
> metafunc.parametrize(*marker.args, **marker.kwargs)
> /usr/lib/python2.7/dist-packages/_pytest/python.py:973: in parametrize
> param_index,
> /usr/lib/python2.7/dist-packages/_pytest/python.py:841: in setmulti2
> self._checkargnotcontained(arg)
> /usr/lib/python2.7/dist-packages/_pytest/python.py:825: in
> _checkargnotcontained
> raise ValueError("duplicate %r" % (arg,))
> E ValueError: duplicate 'box'
Seems one seems to relate to pytest version -- with the most
recent 4.? (installed via pip) it disappears
if I add --continue-on-collection-errors to pytest invocation, I am
ending up with ~50 failing tests :-( and 3 errors.
pushed my changes and now waiting for another "cleanish" run where I
actually store full output since I ran out of terminal buffer to get all
those errors for analysis.
I think part of the problem is upstream chasing bleeding edges for
pytest/numpy/etc in their CI setups and thus loosing idea on how well
pandas behaves on existing deployments. E.g. this pytest 4.0 issue -
versioned dependency was boosted solely to speed up conda's
dependencies resolution... heh heh
--
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419
WWW: http://www.linkedin.com/in/yarik
Re: Pandas new version
any clues on this odd thing? https://github.com/pandas-dev/pandas/issues/25197
(citing here verbatim and entirely)
While updating debian package for pandas 0.24.1 running into this odd error
with pytest during collection. Just wondered if someone might have an
immediate clue so I don't waste too much time trying to catch the ghost (pytest
pkg is 3.10.1-2):
```
collected 47434 items / 2 errors / 778 deselected / 3 skipped
= ERRORS
==
_ ERROR collecting
tests/indexes/datetimes/test_tools.py __
/usr/lib/python2.7/dist-packages/pluggy/hooks.py:284: in __call__
return self._hookexec(self, self.get_hookimpls(), kwargs)
/usr/lib/python2.7/dist-packages/pluggy/manager.py:67: in _hookexec
return self._inner_hookexec(hook, methods, kwargs)
/usr/lib/python2.7/dist-packages/pluggy/manager.py:61: in
firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
/usr/lib/python2.7/dist-packages/_pytest/python.py:224: in
pytest_pycollect_makeitem
res = list(collector._genfunctions(name, obj))
/usr/lib/python2.7/dist-packages/_pytest/python.py:409: in _genfunctions
self.ihook.pytest_generate_tests(metafunc=metafunc)
/usr/lib/python2.7/dist-packages/pluggy/hooks.py:284: in __call__
return self._hookexec(self, self.get_hookimpls(), kwargs)
/usr/lib/python2.7/dist-packages/pluggy/manager.py:67: in _hookexec
return self._inner_hookexec(hook, methods, kwargs)
/usr/lib/python2.7/dist-packages/pluggy/manager.py:61: in
firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
/usr/lib/python2.7/dist-packages/_pytest/python.py:133: in pytest_generate_tests
metafunc.parametrize(*marker.args, **marker.kwargs)
/usr/lib/python2.7/dist-packages/_pytest/python.py:973: in parametrize
param_index,
/usr/lib/python2.7/dist-packages/_pytest/python.py:841: in setmulti2
self._checkargnotcontained(arg)
/usr/lib/python2.7/dist-packages/_pytest/python.py:825: in _checkargnotcontained
raise ValueError("duplicate %r" % (arg,))
E ValueError: duplicate 'box'
__ ERROR collecting tests/resample/test_base.py
___
In test_resample_empty_dataframe_all_ts: function uses no argument
'_index_factory'
! Interrupted: 2 errors during collection
!
=== 3 skipped, 778 deselected, 2 error in 37.76
seconds ===
```
All of it with versions etc (click to expand)
```
echo "backend : Agg" >| /build/pandas-0.24.1/build/matplotlibrc
: # Run unittests here against installed pandas
echo "2.7" | grep -q '^3' && PY=3 || PY=2.7; \
export PYTHONPATH=`/bin/ls -d $PWD/debian/tmp/usr/lib/python$PY/*/`; \
export MPLCONFIGDIR=/build/pandas-0.24.1/build HOME=/build/pandas-0.24.1/build;
\
cd build/; \
python2.7 -c 'import pandas as pd; pd.show_versions()'; \
LOCALE_OVERRIDE=C xvfb-run -a -s "-screen 0 1280x1024x24 -noreset" \
python2.7 -m pytest -s -v -m "not single and not network and not
disabled "$PYTHONPATH/pandas;
INSTALLED VERSIONS
--
commit: None
python: 2.7.15.final.0
python-bits: 64
OS: Linux
OS-release: 4.18.0-0.bpo.1-amd64
machine: x86_64
processor:
byteorder: little
LC_ALL: C
LANG: C
LOCALE: None.None
pandas: 0.24.1
pytest: 3.10.1
pip: None
setuptools: 40.7.1
Cython: 0.29.2
numpy: 1.16.1
scipy: 1.1.0
pyarrow: None
xarray: None
IPython: 5.8.0
sphinx: 1.8.3
patsy: None
dateutil: 2.7.3
pytz: 2018.9
blosc: None
bottleneck: None
tables: 3.4.4
numexpr: 2.6.9
feather: None
matplotlib: 2.2.3
openpyxl: 2.4.9
xlrd: 1.1.0
xlwt: 1.3.0
xlsxwriter: 1.1.2
lxml.etree: 4.3.0
bs4: 4.7.1
html5lib: 1.0.1
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
gcsfs: None
=== test session starts
===
platform linux2 -- Python 2.7.15+, pytest-3.10.1, py-1.7.0, pluggy-0.8.0 --
/usr/bin/python2.7
cachedir: .pytest_cache
hypothesis profile 'default' ->
database=DirectoryBasedExampleDatabase('/build/pandas-0.24.1/build/.hypothesis/examples')
rootdir: /tmp/buildd/pandas-0.24.1, inifile: setup.cfg
plugins: hypothesis-3.71.11
collected 47434 items / 2 errors / 778 deselected / 3 skipped
= ERRORS
==
_ ERROR collecting
tests/indexes/datetimes/test_tools.py __
/usr/lib/python2.7/dist-packages/pluggy/hooks.py:284: in __cal
Re: Pandas new version
On Wed, 06 Feb 2019, Ole Streicher wrote: > Yaroslav Halchenko writes: > >> And it also does not solve the problem that updating can introduce > >> regressions in reverse dependencies, which is not the best thing we can > >> do just before freeze. But finally you are the maintainer, if you think > >> that updating is the way to go, just do it ;-) > > Is there any other commit you would like to push on top of your recent > > 721fcf98d4b79d146a92a8d8def8dee1daecb4c0 > > ? > No; I don't have time to look into this until tomorrow night. If you are > going to fix, I will enjoy a free evening ;-) ok, let's see where I would get, I will keep pushing -- Yaroslav O. Halchenko Center for Open Neuroscience http://centerforopenneuroscience.org Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik
Re: Pandas new version
Yaroslav Halchenko writes: >> And it also does not solve the problem that updating can introduce >> regressions in reverse dependencies, which is not the best thing we can >> do just before freeze. But finally you are the maintainer, if you think >> that updating is the way to go, just do it ;-) > > Is there any other commit you would like to push on top of your recent > 721fcf98d4b79d146a92a8d8def8dee1daecb4c0 > ? No; I don't have time to look into this until tomorrow night. If you are going to fix, I will enjoy a free evening ;-) Cheers Ole
Re: Pandas new version
On Wed, 06 Feb 2019, Ole Streicher wrote: > Yaroslav Halchenko writes: > > On Tue, 05 Feb 2019, Ole Streicher wrote: > >> "Rebecca N. Palmer" writes: > >> > Has anyone checked whether this would break pandas' reverse dependencies? > >> I didn't yet. I just tried to update the packaging to 0.24, which > >> however has a number of test failures, which would need to be discussed > >> with upstream first. > > if not sever - I guess could be patched to be skipped for now and then > > patches picked up to address them? or it was too many/too deep? > If you want: please do it as you like. My own workflow is to first > discuss all failures with upstream, especially when I can't estimate > the impact - sometimes it may be even fault of the package. There were it is the same approach I am taking usually > about ten failures, which makes this already quite an effort. Especially > since I did not work closely with upstream so far (so I don't know them > well, and they don't know me) -- this communication should IMO be done > by the regular maintainer. And, as I wrote, the package rules are a bit > complicated, so I don't want to touch them before they are simplified > (which would add more efforts on top of that). Bringing the other stuff > to gbp standards (pristine-tar etc.) even more. I have been using gbp for it for long time, and there is debian/gbp.conf with all needed settings. pristine-tar is just an option, and I kept burning myself with it too often to rely on it, original author (Joey Hess) even recommended abandoning it at some point in the past. > And it also does not solve the problem that updating can introduce > regressions in reverse dependencies, which is not the best thing we can > do just before freeze. But finally you are the maintainer, if you think > that updating is the way to go, just do it ;-) Is there any other commit you would like to push on top of your recent 721fcf98d4b79d146a92a8d8def8dee1daecb4c0 ? -- Yaroslav O. Halchenko Center for Open Neuroscience http://centerforopenneuroscience.org Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik
Re: Pandas new version
Yaroslav Halchenko writes: > On Tue, 05 Feb 2019, Ole Streicher wrote: >> "Rebecca N. Palmer" writes: >> > Has anyone checked whether this would break pandas' reverse dependencies? > >> I didn't yet. I just tried to update the packaging to 0.24, which >> however has a number of test failures, which would need to be discussed >> with upstream first. > > if not sever - I guess could be patched to be skipped for now and then > patches picked up to address them? or it was too many/too deep? If you want: please do it as you like. My own workflow is to first discuss all failures with upstream, especially when I can't estimate the impact - sometimes it may be even fault of the package. There were about ten failures, which makes this already quite an effort. Especially since I did not work closely with upstream so far (so I don't know them well, and they don't know me) -- this communication should IMO be done by the regular maintainer. And, as I wrote, the package rules are a bit complicated, so I don't want to touch them before they are simplified (which would add more efforts on top of that). Bringing the other stuff to gbp standards (pristine-tar etc.) even more. And it also does not solve the problem that updating can introduce regressions in reverse dependencies, which is not the best thing we can do just before freeze. But finally you are the maintainer, if you think that updating is the way to go, just do it ;-) Best regards Ole
Re: Pandas new version
HI Yaroslav, On Tue, Feb 05, 2019 at 02:07:52PM -0500, Yaroslav Halchenko wrote: > otherwise, I would wholeheartedly welcome simplifications as long as > pandas builds seamlessly also at least on debian stable (or soon > oldstable) and some recent ubuntus. Supporting Stretch (=stable and soon oldstable) is considered sensible. Which one is the oldest Ubuntu version you want to support (and what cython version is supported in this distribution = where can I find this information - sorry for my ignorance). Kind regards Andreas. -- http://fam-tille.de
Re: Pandas new version
On Tue, 05 Feb 2019, Ole Streicher wrote: > "Rebecca N. Palmer" writes: > > Has anyone checked whether this would break pandas' reverse dependencies? > I didn't yet. I just tried to update the packaging to 0.24, which > however has a number of test failures, which would need to be discussed > with upstream first. if not sever - I guess could be patched to be skipped for now and then patches picked up to address them? or it was too many/too deep? -- Yaroslav O. Halchenko Center for Open Neuroscience http://centerforopenneuroscience.org Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik
Re: Pandas new version
On Tue, 05 Feb 2019, Andreas Tille wrote: > On Tue, Feb 05, 2019 at 08:48:09AM +0100, Ole Streicher wrote: > > On a longer term, it would be good to clean this package up (removing > > f.e. the special Cython handling) -- Yaroslav, how much do you still > > these things? Since Panda is one of the universal packages, it would be > > good to have it as standard as possible to make team maintenance simple. > I agree that we should settle with a simple standard and even the last > say 5 Ubuntu versions will be able to cope with this. So may be there > is just historic stuff in the packaging and nobody minded to clean it > up. Yaroslav, if you do not contradict with this we need to assume that > I'm correct with my assumption and whoever might do the upgrade should > make packaging as simple and easily understandable as possible. Is special cython handling too much to handle? if not - I would beg to keep it. I thought it is modular enough to not be a sore point -- if helps, feel welcome to move those rules down in the file. Having it greatly simplifies backporting to older debian/ubuntus which might lack up to date cython and providing cython backport itself would be too destructive (causing other FTBFS etc). May be you see a better way? For the same reason of backports I would ask to not strip all the python2 support in there prematurely. otherwise, I would wholeheartedly welcome simplifications as long as pandas builds seamlessly also at least on debian stable (or soon oldstable) and some recent ubuntus. -- Yaroslav O. Halchenko Center for Open Neuroscience http://centerforopenneuroscience.org Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik
Re: Pandas new version
On Tue, Feb 05, 2019 at 08:48:09AM +0100, Ole Streicher wrote: > > On a longer term, it would be good to clean this package up (removing > f.e. the special Cython handling) -- Yaroslav, how much do you still > these things? Since Panda is one of the universal packages, it would be > good to have it as standard as possible to make team maintenance simple. I agree that we should settle with a simple standard and even the last say 5 Ubuntu versions will be able to cope with this. So may be there is just historic stuff in the packaging and nobody minded to clean it up. Yaroslav, if you do not contradict with this we need to assume that I'm correct with my assumption and whoever might do the upgrade should make packaging as simple and easily understandable as possible. Kind regards Andreas. -- http://fam-tille.de
Re: Pandas new version
"Rebecca N. Palmer" writes: > Has anyone checked whether this would break pandas' reverse dependencies? I didn't yet. I just tried to update the packaging to 0.24, which however has a number of test failures, which would need to be discussed with upstream first. Also, the package is a bit "special", and Yaroslav seems to have the requirement to support some older Ubuntu version with a different handling, keeping this intact requires some attention. Given the upcoming soft freeze, I think the schedule would be too tight to fix them and check the rdeps before Feb 12. And since you already determined the fix, the best would probably be to just apply it. I could to this; however I will have time to do only on Thursday, so you may want to do it before. On a longer term, it would be good to clean this package up (removing f.e. the special Cython handling) -- Yaroslav, how much do you still these things? Since Panda is one of the universal packages, it would be good to have it as standard as possible to make team maintenance simple. Best regards Ole
Re: Pandas new version
Control: tags -1 fixed-upstream patch The test failure is that np.array @ pd.DataFrame (matrix product) tries to keep both the DataFrame's indices, which fails because the new matrix is a different shape. This appears to be fixed in 0.24.1 from PyPI, but as previously noted, this is a new major version and hence risks breaking rdeps. The relevant change appears to be setting __array_priority__: https://github.com/brute4s99/pandas/commit/a01fa791eafe704ea85e2acc956ad9077e8e7542#diff-03b380f521c43cf003207b0711bac67f but I haven't actually tried applying only that to 0.23.
Re: Pandas new version
Has anyone checked whether this would break pandas' reverse dependencies?
Re: Pandas new version
Hi Ole, On Fri, Feb 01, 2019 at 08:54:26AM +0100, Ole Streicher wrote: > the Debian "pandas" package, which is maintained by you, FTBFS since > some time due to an incompatibility with numpy 1.16 (RC bug #918206). > > And there is a new Pandas version 0.24 out since a week, which however > seems to have some API changes (see release notes). > > Since the soft freeze is approaching rapidly: what is your plan here? > IMO it would be great to update Pandas to the latest version, and to fix > the API problems during the freeze in order to not have an outdated > Pandas version in buster. > > What do you think? Could you upload the new release quickly? Or would > you fix the numpy incompatibility manually? If so, it would be nice if > that would happen soon, since during the freeze, once the dependent > packages are removed from testing, they are out of buster. As far as I understood Yaroslav he is fine with team uploads fixing the issues that might occure. I fully trust you to do the right thing here and I agree that we should act as fast as possible. So feel free to upload the new release as quick as possible if you consider this the most promising solution. Kind regards Andreas. -- http://fam-tille.de
Pandas new version
Hi Yaroslav (cc: d-science), the Debian "pandas" package, which is maintained by you, FTBFS since some time due to an incompatibility with numpy 1.16 (RC bug #918206). And there is a new Pandas version 0.24 out since a week, which however seems to have some API changes (see release notes). Since the soft freeze is approaching rapidly: what is your plan here? IMO it would be great to update Pandas to the latest version, and to fix the API problems during the freeze in order to not have an outdated Pandas version in buster. What do you think? Could you upload the new release quickly? Or would you fix the numpy incompatibility manually? If so, it would be nice if that would happen soon, since during the freeze, once the dependent packages are removed from testing, they are out of buster. Best Ole
