Re: Status of https://debian-python.readthedocs.io/en/latest/

[Barry Warsaw]

Hi Sandro,

That RTD project has been failing for years.  It looks like I’m the only 
maintainer for that project.  I’m happy to add others or transfer it to someone 
else.  I don’t think RTD supports maintainer teams.  Just let me know what 
y’all want to do!

Cheers,
-Barry

> On Dec 25, 2020, at 12:52, Sandro Tosi  wrote:
> 
> Hello,
> it looks like Barry (correct me if i'm wrong) set up
> https://debian-python.readthedocs.io/en/latest/ but it has not been
> updated in a while.
> 
> Do we know what's the status of this website, if we want to continue
> to maintain it, or instead we should just consolidate onto
> https://www.debian.org/doc/packaging-manuals/python-policy/ ?
> 
> Thanks,
> --
> Sandro "morph" Tosi
> My website: http://sandrotosi.me/
> Me at Debian: http://wiki.debian.org/SandroTosi
> Twitter: https://twitter.com/sandrotosi



signature.asc
Description: Message signed with OpenPGP

Scaling back for now

[Barry Warsaw]

After the events of April, I've realized that I have a lot less time, and to
be frankly honest, motivation for contributing to Debian right now.  So I'm
scaling back, but not retiring.

I've tried to reflect my current level of participation by updating the
packages for which I was sole or primary Maintainer, putting DPMT or PAPT in
the Maintainer field and moving myself to Uploaders[*].  This reflects my hope
that the team will be better able to maintain these packages than I will right
now, and should remove me as the single point of failure for their continued
improvement in Debian.  As per team conventions, this means you don't have to
ask my permission to make changes to the packages.  If I've missed any, please
do let me know.

I am not giving up on Debian or Ubuntu!  I still feel strongly about their
importance and excellence in the open source and free software world, and
about their strengths as Linux distributions.  I still use them both!  And I'm
still keenly interested in the migration to Python 3 across the whole Python
ecosystem, and Debian in particular.

I'll continue to contribute where I can and as time allows.  But for now, as
my participation in Debian is solely on my own free time, I've found that I
have competing interests for that time, and I plan to spend it in other ways.

The Debian Python community continues to exemplify the best in collaborative
development.  DPMT and PAPT are fantastic teams, filled with great people, and
smart developers, and I have always felt honored to be part of it.  I'm happy
to continue to contribute, even if in a more tangential role for now.  I'm
back on IRC again, so as always, if I can help with anything, please don't
hesitate to reach out.

Cheers,
-Barry

[*] I've made the changes in the git repos, but not necessarily uploaded new
versions.


pgpwPVHzPFQqo.pgp
Description: OpenPGP digital signature

Re: MBF for deprecating Python2 usage

[Barry Warsaw]

On Aug 7, 2017, at 20:52, Ben Finney  wrote:
> 
> So IMO the conservative assumption is that they expect Debian to provide
> the stability it is famous for, and not to break the behaviour of
> commands unnecessarily.

There’s another dimension to that too: it’s people who expect no changes to the 
command called /usr/bin/python *and* want to track the latest and greatest 
Debian releases.  Maybe that changes the equation and maybe not.  Certainly any 
large organization with tons of Python 2 laying around is going to be much more 
conservative about upgrading their entire distro than individual developers who 
just use unstable and get whatever today’s snapshot looks like.

And to reiterate, I’m not arguing that /usr/bin/python change today or even at 
the end of Pycon 2020 when 2.7 is officially EOL’d.  I’m saying that it *will* 
change some day and that we should plan that change, if for no other reason 
than to set the expectations of exactly those folks who are not part of this 
conversation.

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP

Re: MBF for deprecating Python2 usage

[Barry Warsaw]

On Aug 7, 2017, at 18:28, Ben Finney  wrote:

> The issue is: Invoking ‘python’ today gets an interpereter, Python 2,
> that will work with some code and not others. It should not tomorrow
> invoke an incompatible interpreter without *knowing* that the vast
> majority of scripts in the wild no longer expect Python 2 to come from
> that command.

That’s unknowable.  We have no idea what people do with any package of Debian 
outside of the archive.  We can’t even quantify what “vast” means.  So under 
the above definition, you’re right, that day will never come.

> That day might never come, in which case the ‘python’ command will
> forever mean Python 2. That is, I'm saying, better than breaking that
> command in the near or medium future.

I wonder if you’ll say the same thing in 2030 when Python 2.7 is 10 years dead, 
can literally cannot be built on any then-modern Debian, let alone Linux 
distro? That’s not hyperbole; one of the current upstream exceptions to the 2.7 
branch is to adapt to newer build systems and library versions.  That will stop 
in 2020, but of course everything else will continue to evolve.

It might even come sooner.  Once Python 2.7 is EOL’d, it won’t even get source 
releases, and may not even get security releases.  If folks want to keep it 
alive, it’ll take active downstream resources to maintain.  People may pay for 
commercial support from RedHat, Canonical, and others, but how much will 
volunteers want to keep it secure and working?

I can remember when ‘python’ meant Python 1.5.2.  So that name doesn’t have to 
be tied to a specific major version.

> I'm saying it's a bad idea for ‘python’ tomorrow to get an incompatible
> intepreter that won't run the same Python code. That interpreter is
> named ‘python3’ and should never be installed as ‘python’, because
> ‘python’ is a command that many scripts expect to invoke Python 2.

Sure, today.  And tomorrow.  And many tomorrows until at least 2020.  Then I 
think we’ll be having very different conversations about it.  Anybody who will 
still be running Python 2.7 code will have some difficult choices to make.  1) 
port to Python 3; 2) run old versions of distros that still have Python 2.7 
around and live with the unsupported nature of it; 3) roll their own Python 
2.7s; 4) pay some commercial vendors to keep their stuff alive and secure.

IMHO, there will absolutely be a day when it makes no sense to point 
/usr/bin/python to Python 2.

Cheers,
-Barry


signature.asc
Description: Message signed with OpenPGP

Re: MBF for deprecating Python2 usage

[Barry Warsaw]

On Aug 7, 2017, at 14:52, Diane Trout  wrote:
> Perhaps when launching via the command "python" the interpreter could
> hint python2 was available? Something like:
> 
> $ python
> Python 3.5.4rc1 (default, Jul 25 2017, 08:53:34)
> [GCC 6.4.0 20170704] on linux
> Type "python2" for Python 2.7.13
> Type "help", "copyright", "credits" or "license" for more information.

Good idea Diane!
-Barry




signature.asc
Description: Message signed with OpenPGP

Re: a few quick questions on gbp pq workflow

[Barry Warsaw]

On Aug 6, 2017, at 11:11, Ondrej Novy  wrote:

> It's not always possible/simple/nice to use sdist, because it contains 
> prebuild docs. And I don't like to do +dfsg rebuild just for removing docs. 
> Sometimes sdists doesn't contain tests.

I will also generally prefer the PyPI tarballs for the packages I help 
maintain.  There’s good tooling for that and the tarballs are usually fine.  If 
not, and it’s fairly easy to DFSG them (e.g. via d/copyrights or some such), 
then I’ll go that route.  So far I haven’t had to pull things in from upstream 
git although once in a while I’ll just grab a file if needed.

-Barry



signature.asc
Description: Message signed with OpenPGP

Re: MBF for deprecating Python2 usage

[Barry Warsaw]

Scott Kitterman wrote:

> If the primary concern is what happens when a user types "python", then can 
> we 
> address that in command-not-found and leave /usr/bin/python out of it?

I'm definitely willing to for now.  It's clearly not time to remove the
link or point it elsewhere anyway.  I think it'll still be a good idea
some day, but I'll leave it at that.

Can we agree though to start using /usr/bin/python2 in the shebang line
instead of /usr/bin/python, for system installed scripts?

-Barry




signature.asc
Description: OpenPGP digital signature

Re: MBF for deprecating Python2 usage

[Barry Warsaw]

Ole Streicher wrote:

> It is very usual to use "#!/usr/bin/env python" as shebang, exactly for
> the case that python is not installed in /usr/bin.

Sure, but then all bets are off.  The script so shebanged can't assume
anything about $PATH so it gets whatever it gets.  Using /usr/bin/env in
system provided scripts has been known to break stuff, so it's a very
bad idea.

Plus, if that's what you use for your own scripts, then in the future
you'll be in luck because all you'd have to do is change your $PATH
to point to wherever your custom built and installed Python 2 lives
first, and then you'll get exactly the version you want.

> That is -- at least in my environmet -- "ipython", or (for Python 3)
> "ipython3".
>
> IMO it would be better to communicate that the best way for an
> interactive session is "ipython3" (or "python3", if you insist). I would
> wonder when todays tutorials (that cover Python 3) recommend to use
> plain "python".

But ipython doesn't come with upstream Python so that would require that
Something Extra be installed.  That's generally why Getting Started
guides and tutorials usually recommend the built-in interactive prompt.

>> * Port as much as possible to Python 3 (eventually, everything
>> maintained in Debian) and /usr/bin/python3 in their shebang.
> I disagree here, and I don't see an advantage over letting
> /usr/bin/python just die with Python 2.

/usr/bin/python didn't die with Python 2 (who else remembers 1.5.2 as an
effectively LTS release? :) and I don't believe it should with Python 3.
Maybe go on vacation, but eventually return home.

> I would also like to point PEP 394, which has a number of arguments why
> /usr/bin/python should remain as Python 2.
>
> https://www.python.org/dev/peps/pep-0394/

Right, but the ecosystem has evolved since then, so the discussions
going on in linux-sig and the *draft* PR at
https://github.com/python/peps/pull/315 are about updating the
recommendations now that we're somewhere less than three years away from
Python 2 EOL.

Fedora is moving forward with or without us.  I bet that other Linux
distros will too.  We can certainly do our own thing, but I think
that'll make Debian the odd distro out eventually.

Cheers,
-Barry

Re: Python3.6 plans for Buster

[Barry Warsaw]

On Jun 17, 2017, at 04:20 AM, Scott Kitterman wrote:

>Last time I compared my understanding of the Buster schedule with the
>python3.7 schedule, I recall concluding that we'd likely be on python3.6 for
>the Buster release (not sure that's still true).  If so, we only have to do
>this once this cycle and​I think the sooner the better.

Thanks for bringing this up Scott, and thanks for the additional information
about Ubuntu's transition Steve.  For sure, Debian should make 3.6 default as
soon as possible in Buster.

But we may indeed get to 3.7 by the time Buster is released.  According to PEP
537 [*], 3.7.0 final is schedule for almost exactly 1 year from now, on
2018-06-15.  So if Buster is approximately 2 years out from now, then we'll
need to do more transition, and have plenty of time for it.

(Aside: it does mean Ubuntu's next LTS will carry 3.6.)

Cheers,
-Barry

[*] https://www.python.org/dev/peps/pep-0537/

Re: Ad-hoc Debian Python BoF at PyCon US 2017

[Barry Warsaw]

On Jun 06, 2017, at 10:57 AM, Sandro Tosi wrote:

>if we plan (and it looks like we do) to support and distribute 2.7
>with buster, why not support it *properly*? what's the point of
>deprecating python2.7? either we ship it or not, but if we do then
>let's not cripple it by removing python2 modules packages. do yo think
>that just because the module i want to use is not available will make
>realize "oh sh*t, let's migrate this 50k lines of code application to
>py3k so that i can implement this 5-minutes-of-work-funcionality if i
>had the module on py2"?

So what's the plan for when upstream stops supporting Python 2 in 2020?  Given
the pronouncement at Pycon 2017 that maintenance will end at Pycon 2020, we
really need to decide what Debian's official policy will be, and what the
timeline will be to get there.

If Buster is 2 years in development, that means it will be the last Debian
release before Python 2.7 is EOL'd.  Yes, I know it's possible that 2.7 will
get security releases for some time after that, but that's a much reduced
commitment from upstream.

Once upstream stops supporting 2.7, should we also stop supporting it?  That
wouldn't mean that developers on Debian can't use Python 2.7, just that they
will be on their own.  I know it sucks for people who can't port to Python 3,
but if a decade or more isn't enough time to switch, then that's really saying
they'll never switch, and how much responsibility does Debian have at that
point?

Python 2.7 isn't going away today, but 3 years goes by quickly and we need to
decide what our policy will be when the day arrives.

-Barry

Re: PAPT git migration

[Barry Warsaw]

On May 31, 2017, at 08:58 PM, Stefano Rivera wrote:

>It's been a while, but I found some time and energy at PyCon, to work
>our SVN->Git migration.

Thanks for all your great work on this Stefano!

>Here's what the migration currently looks like [0].
>[0]: https://anonscm.debian.org/cgit/users/stefanor/python-apps/

I did a spot check on twine.  I don't see a debian/.git-dpm file on master so
git-dpm won't work.  Is it supposed to?

Let's assume not (which is fine with me; i.e. why adopt git-dpm for PAPT when
we know we just want to get rid of it later?).  Then i tried to import a new
upstream as described here https://wiki.debian.org/Python/GitPackagingPQ

$ gbp pq export
- This doesn't work until you at least do a first pq import, but now I see the
  d/p/changlog-docs patch gets changed in ways that lose information:

modified   debian/patches/changelog-docs
@@ -1,14 +1,17 @@
-Description: link changelog.rst from index.rst
- The generated documentation is otherwise useless so let's just link the
- changelog for now. Upstream is aware of this issue and will fix it with the
- next release.
-Bug: https://github.com/dstufft/twine/issues/46
-Forwarded: yes
-Last-Update: 2014-04-20
+From: Python Applications Packaging Team
+ 
+Date: Wed, 31 May 2017 17:21:14 -0400
+Subject: changelog-docs
 
+---
+ docs/index.rst | 1 +
+ 1 file changed, 1 insertion(+)
+
+diff --git a/docs/index.rst b/docs/index.rst
+index 69d1847..94c5ffb 100644
 --- a/docs/index.rst
 +++ b/docs/index.rst
-@@ -11,6 +11,7 @@
+@@ -11,6 +11,7 @@ Contents:
  .. toctree::
 :maxdepth: 2

Other than that, `gbp import-orig --pristine-tar --uscan` seems to work well,
and the resulting repo produces a good package.

Cheers,
-Barry

Re: python-parse-type

[Barry Warsaw]

On May 16, 2017, at 08:10 PM, Brian May wrote:

>python-enum - robust enumerated type support in Python
>python3-enum34 - backport of Python 3.4's enum package

Oh yeah, that too. :)

-Barry

Re: python-parse-type

[Barry Warsaw]

On May 16, 2017, at 11:51 AM, Piotr Ożarowski wrote:

>packaged as python-enum34 (correct name is python-enum, that's why you
>didn't find it most probably)
>
>Barry: ;-P

Why is that wrong?  Agreed it's perhaps less discoverable in this case, but if
you were looking for the PyPI enum34 package in Debian, you'd find
python-enum34 first, and it would make sense.

-Barry

Re: Request to join DPMT

[Barry Warsaw]

On May 11, 2017, at 10:14 AM, Colin Watson wrote:

>I'd like to join the Debian Python Modules Team.  I maintain a couple of
>module packages already that probably ought to be moved under the DPMT
>umbrella (six, python-tblib), and I'm upstream for some others (e.g.
>lazr.restfulclient, python-launchpadlib) where I could productively help
>with maintaining the packages as well.

+1

I help out already with six and it would be nice to move that (and the others)
to DPMT.

-Barry


pgp76K7A9u8lK.pgp
Description: OpenPGP digital signature

Re: Joining the team

[Barry Warsaw]

On Apr 09, 2017, at 01:13 PM, sab wrote:

>https://mentors.debian.net/package/python-zxcvbn
>
>with DPMT as uploader. What's my next step? Have I to create a Git repository?

Welcome to the team Sabino.

You'll want to take a look at these two wiki pages.  First is the style guide
for DPMT packages.

https://wiki.debian.org/Python/LibraryStyleGuide

The second is our git packaging workflow:

https://wiki.debian.org/Python/GitPackaging

but note that we are still discussing the move away from git-dpm to git-pq for
patch queue management.  When that actually happens we'll use:

https://wiki.debian.org/Python/GitPackagingPQ

as a guideline.

Cheers,
-Barry

Heads up about file name changes

[Barry Warsaw]

As Ubuntu 17.04 development winds down, I've been spending a fair bit of time
looking at FTBFS.  For packages that have been copied from Debian, especially
Python ones, we try really hard not to introduce too many deltas.  Doko
usually does archive-wide test rebuilds at various times during the cycle to
find build failures.  Here are the results of one he did a couple of weeks
ago:

http://qa.ubuntuwire.org/ftbfs/rebuilds/test-rebuild-20170322.1-zesty.html

I've started by looking at the Python package build failures and I've noticed
a bit of a trend, so I thought I'd at least mention it here, because
eventually the same changes I'm uploading as Ubuntu deltas will have to be
ported to Debian.  Yes, I could do that in git or experimental, but right now
I don't have time for that, sorry.  I also haven't investigated exactly what's
changed in the file names, but I'm suspecting something's changed in Sphinx.

So far, I've seen this problem in python-babel and python-pika.  In both
packages, d/rules tries to remove redundant files.  In babel's case it's
license.txt[1] and in pika's case it's version_history.txt (same path
pattern).  In both cases, these d/rules commands fail because the files now
end in .rst.txt, not .txt.

The fixes are easy of course, but this will probably affect many packages in
DPMT so I wanted to give a heads up.

Cheers,
-Barry

[1] rm 
debian/python-babel-doc/usr/share/doc/python-babel-doc/html/_sources/license.txt


pgpszc0EVn_Ml.pgp
Description: OpenPGP digital signature

What should Debian do about Python 2? (was Re: next version of csvkit)

[Barry Warsaw]

On Apr 01, 2017, at 05:12 PM, Ghislain Vaillant wrote:

>Pasting the relevant quotes below:
>
>"The 2.x series of Python is due for deprecation and will not be maintained
>past 2020 so it is recommended that Python 2 modules are not packaged unless
>necessary."

It's true that upstream will almost certainly end bug fix support for Python
2.7 (and thus the 2.x series) in 2020, with the exact date still TBD.  It's
not unlikely that source-only security fixes will continue upstream for some
indeterminate time after that, but it depends on quite a few factors, not the
least of which is core developers and an RM still interested in working on
Python 2.  Keep in mind that 2.7 will be *10 years old* at that point!

Distributions with long term support releases will also be continuing to
support Python 2.7 in some capacity after that.  Ubuntu's 16.04 LTS release
will live until 2021.  Other distros may have longer commitments.  I am hoping
that in Ubuntu, we will be able to demote Python 2.7 for the 18.04 LTS release
so that we won't be committed to supporting it until 2023, but we'll see about
that.

As for Debian, I do think we need to plan on how and when we're going to
downgrade our commitment to Python 2.  Yes, there will probably be Python 2
code bases for effectively forever, but that doesn't necessarily mean Debian's
commitment to it also has to be open ended.  It's not an easy decision or
transition, but I think it's inevitable.  Already we are seeing upstream
library authors beginning to drop Python 2 support, so I suspect at some point
we may have to split source packages if we think we still want to maintain
Python 2 and 3 for some packages.

>"The idea is to basically stop uploading new Python 2 only libraries,
>port things on the critical path, and swap leaf packages to Python 3."

I think we should be very strict about leaf packages.  If they support Python
3 we should not allow them to enter Debian as Python 2 applications, and all
new uploads of leaf applications should use Python 3 if at all possible.

For libraries that already support 2 and 3, I don't think we necessarily need
to actively drop Python 2 yet.  We have great tools, so it's usually just as
easy to continue supporting both.  I'm on the fence for new library packages,
but I suppose if it's also just as easy to support both, we might as well go
through NEW for both, since it's easier to drop binary packages than add them.

I think we should strongly discourage (or even reject?) new Python 2-only
library packages, and I think that's the scenario that the lintian tag is
trying to support.

>csvkit definitely qualifies as such leaf package, since it is a
>collection of command-line tools, not a Python library.

Does it support Python 3?  If so, then why not make them Python 3
applications?

Debian's long term plans for Python 2 would be a great topic to discuss at
Debconf.

-Barry

Re: PyPI source or github source?

[Barry Warsaw]

On Mar 14, 2017, at 08:16 AM, Brian May wrote:

>I just found another one: python-ledger. Unfortunately in this case,
>Python packages are also bundled with the C code (libraries and client).

Not entirely relevant, but pyparted also contained C code, but its mainline
does support Python 3.

-Barry


pgpJS7pdIq9jm.pgp
Description: OpenPGP digital signature

Re: PyPI source or github source?

[Barry Warsaw]

On Mar 13, 2017, at 05:55 PM, Thomas Goirand wrote:

>IMO, upstream are right that the PyPi releases should be minimal. They
>are, from my view point, a binary release, not a source release.

Wheels should be considered a binary release, but tarballs should still be
considered the canonical source release.  OTOH, most non-geeks I know
exclusively top-post so eventually pragmatism will probably win out over
historical conventions.

>It makes a lot of sense to therefore use the git repository, which is
>what I've been doing as much as possible.

So, what do you do about upstreams that don't use git?  (I'm guessing that if
they use git but not GitHub, your workflow doesn't break.)

Cheers,
-Barry

Re: PyPI source or github source?

[Barry Warsaw]

On Mar 12, 2017, at 01:19 PM, Brian May wrote:

>Should we be using PyPI as our source of packages? Or github?

PyPI for two reasons: not every developer uses git, and even those that do may
not use GitHub.

I've so far only encountered one package that releases on GitHub and not PyPI:
pyparted.  And I've filed an upstream bug about that.

>However, often packages in PyPI are not the complete source package you
>would get from github. They may be sufficient for installations, but
>often not for Debian packaging - which really should have the complete
>source package. e.g. they can be missing tests, license files,
>documentation that doesn't get installed, etc.

Yep, and I always file a bug when that happens.  Sometimes I include the file
in debian/ and add rules to move them in place.  Usually the missing file is
just caused by a MANIFEST.in that's out of date or some such (heck, I've done
that myself).  But FWIW, I've yet to encounter an upstream that hasn't been
very helpful in fixing their releases to help downstreams like Debian.

Cheers,
-Barry

Re: GnuPG signatures on PyPI: why so few?

[Barry Warsaw]

On Mar 12, 2017, at 11:46 AM, Ben Finney wrote:

>What prospect is there in the Python community to get signed upstream
>releases become the obvious norm?

I don't know.  Digital security seems to be mostly an afterthought
unfortunately.  I always use `twine upload --sign` so all my projects have
signatures, and for those where I'm also the Debian maintainer or primary
uploader, I try to enable signatures for uscan, but it seems oddly
self-serving. ;)

-Barry

Re: Adopting OpenStack packages

[Barry Warsaw]

On Mar 06, 2017, at 10:32 AM, Scott Kitterman wrote:

>I think it's reasonable to try this out on a branch for some number of
>packages and write documentation (the documentation for using git with
>git-dpm in DPMT is excellent and I don't think we should regress in that
>area).  Once that's done, someone who doesn't know anything about gbp pq
>should try it out and see if the documentation works (I'll be glad to do
>that).

We have this wiki page for describing common workflows for gbp-pq:

https://wiki.debian.org/Python/GitPackagingPQ

It started as a copy of the git-dpm page:

https://wiki.debian.org/Python/GitPackaging

So one thing to do is to debug the instructions there.  I think the pq page
would be a good place to capture whatever foolproof conversion recipe we come
up with.  Even after a mass migration, this might be useful for other
maintainers that want to switch from git-dpm.

Cheers,
-Barry

Re: Adopting OpenStack packages

[Barry Warsaw]

On Mar 05, 2017, at 01:47 AM, Thomas Goirand wrote:

>Why waiting? The freeze is typically a time of very low activity and low
>disturbance. That's a perfect moment for doing the switch.

I think it's generally been the consensus, even outside of this team, that
doing vcs or other disruptive switches are bad ideas during times of release
freezes.  For example, upstream CPython recently switched to git + github, but
while an enormous amount of work went into making that as smooth as possible
beforehand, the actual switch wasn't done until after the 3.6 release.

There are lots of good reasons for that.  I think most importantly is that if
a last minute RC bug were to pop up, no one wants to have to figure out (or
worse, debug) a new maintenance workflow in order to fix that critical
problem.

But that only gates flag day.  Any switch of this nature requires a lot of
work before flag day.  Look at the switch from svn to git,  Stefano did a huge
amount of testing and development to get us to that point, with lots of test
migrations, feedback, etc.  Kudos to him for the tenacity and dedication to
the team on that.

>It took *years* to switch from SVN to Git. It's taking *years* to get
>out of git-dpm. How many centuries until the team realize that others
>are using CI/CD, automated testing, and such, and team member accept
>things changing fast on the right direction? There's always resistance
>for change in this team.

Let's look at the switch from git-dpm as an example.  We *know* there are
challenges in that conversion; I've experimented with it as have others, and
it's not a trivial operation.  So we need one or a few dedicated people to
investigate all the technical details of such a switch.  What are the steps
needed to convert an individual package?  How and when will we convert all
team packages?  What exactly will the new workflow look like?  Does the wiki
page accurate describe all the common tasks that team members will need to
perform?  Is there a test conversion that people can try out?  Where are the
scripts to do the conversion so others can contribute?  What is the process
for providing and addressing feedback as people test it?  What's the timeline,
and when is flag day?

I think one of the problems specifically with getting rid of git-dpm is that,
while the tool is deprecated and there are known problems, it actually kind of
works pretty well for us.  svn clearly was breaking down, but from a global
team point of view, git-dpm is still almost good enough, so the urgency to
switch hasn't been there.  Still, I think there's consensus that we need to be
using supported tools, and git-dpm does occasionally rear up and bite your
head off.  But we need volunteers to say "I am going to do the hard work to
make the conversion happen".  And of course we're all busy, and it's a
thankless job (but thank you Stefano for your previous work!).  So that's why,
IMHO, the git-dpm conversion hasn't happened yet.

If we're just not going to find the round tuits to do the conversion before
then, this would make for a very suitable collaboration for a team Debconf
sprint.

CI/CD, automated testing, etc. cannot just happen by fiat.  They may be great
ideas we can adopt, but *a lot* of hard work and dedicated time goes into
making sure the technology can handle things, but also, and more importantly
IMHO to make sure that everyone on the team knows how it works, understands
and can help debug any problems, knows where the well-written documentation
is, etc.  We need dedicated people to help people on IRC and email when they
get stuck or have a problem.  And we have to consider the needs of those for
whom contribution to DPMT is not a full-time job.

Yes, there is always a natural resistance to change.  But that can be greatly
alleviated by preparation and dedication.  Those things take serious amounts
of time and fortitude.  It's hard to devote that time when you Just Want To
Get Something Done and have to squeeze in an hour here and there to do it.

I look at folks like Brett Cannon doing the CPython GitHub transition as a
model for this.  I'm truly impressed at the work he's done (with lots of help
from others), and at the enormous positive outcome of the effort.  But also
keep in mind that flag day isn't the end of the process.  Things are still
getting tweaked to help smooth the post-migration pain points.  Now, maybe
DPMT won't be as complicated, but it still requires dedication and follow
through.  And faith!  You have to believe (and be convincing and convincingly
friendly in the face of all that inertia) that what you are proposing will
help make everyone's life easier, will make the process of contributions more
efficient for daily maintainers and occasional maintainers alike.

Cheers,
-Barry

Re: Backport of python-lockfile and suggested team maintenance

[Barry Warsaw]

On Mar 03, 2017, at 02:03 PM, Thomas Goirand wrote:

>Please consider that python-lockfile is considered deprecated upstream,
>and only maintained for bugs and security. There's alternative
>available, like python-oslo.concurrency.

ObPlug: Or flufl.lock, albeit with a different API and other (sometimes
interesting) semantics.

Cheers,
-Barry

Adopting OpenStack packages

[Barry Warsaw]

We've talked on various lists about adopting the OpenStack packages into DPMT,
and also adopting the team's standard workflows and helpers.

The way the packages have been maintained in the past isn't aligned with our
team practices, but Allison and I spent a little time today importing alembic
into DPMT git.  After looking at the existing git repo (i.e. what you get with
debcheckout), we both figured it would be better to start fresh, so I used my
import-dscs.py script[1] to import all previous releases into a git-dpm repo
(via debsnap).  Then I pushed that repo to DPMT git.

$ git clone git+ssh://git.debian.org/git/python-modules/packages/alembic.git

That looks like it matches what's in unstable/testing right now.  The package
doesn't use pybuild, and both Allison and I want to convert it to our standard
build system, but before I go hacking around in d/rules, I want to get a
sanity check on the conversion process.  Please poke around and let me know
what you think.  If the results of import-dscs.py look good enough, then the
thinking is that we'll use that to convert the rest of the packages.  If it's
not good enough, then let's work out a better conversion process before we
import more packages.

Things to not worry about, IMHO:

* Anything other than making the package maintainable in git-dpm
* Any commits to the existing git repo that weren't uploaded

I don't think there's a huge urgency here, since I'm not aware of any packages
that need updates before Stretch is released.  (Allison, please confirm.)
E.g. alembic is at 0.9.0 upstream, but it'll stay at 0.8.8 for Stretch.
Still, we'd like to have everything in place once the release happens and
freeze is lifted.

-Barry

[1] git clone https://anonscm.debian.org/git/users/barry/import-dscs.git


pgpUwoOYTa5dc.pgp
Description: OpenPGP digital signature

Re: PAPT and git

[Barry Warsaw]

On Feb 23, 2017, at 10:00 AM, Jordan Justen wrote:

>Looking at http://python-apps.alioth.debian.org/policy.html, it
>appears that PAPT still uses svn.

True.

>I wanted to start working on the 'alot' PAPT package using git. On
>irc, olasd let me know that the current stance is that no PAPT
>packages should be in git until all PAPT packages are transitioned.
>
>So ... can I help to transition the PAPT packages to git?

Yes!  Stefano has a bunch of scripts that he used to convert DPMT svn to git,
so there's a great starting place.  Probably the main thing to change there
would be to forego conversion to git-dpm and work on conversion to gbp-pq.

Or, let's be expedient and use the existing scripts to go to git-dpm for now,
taking up gbp-pq transition along with DPMT after Stretch.

(While you're at it, gosh I'd love the zope packages to be moved to git + DPMT
:).

If it doesn't happen by then, Debconf 2017 would be a good time to work on
this stuff.

Cheers,
-Barry


pgplGLkYUxuFZ.pgp
Description: OpenPGP digital signature

Re: Moving off of git-dpm

[Barry Warsaw]

On Feb 16, 2017, at 07:53 AM, Scott Kitterman wrote:

>Which is completely separate from the question of if we want to use it as a
>team.  Whoever it was that suggested focusing on what (if anything) to
>replace git-dpm with (post-stretch) and leave the dgit discussion for later,
>I completely agree with.

+1
-Barry

Re: Moving off of git-dpm (Re: git-dpm breakage src:faker)

[Barry Warsaw]

On Feb 14, 2017, at 06:30 PM, Arto Jantunen wrote:

>The patch-queue branch is based on the Debian branch, not upstream. Try
>merging the new upstream version to your Debian branch, and then running
>gbp pq rebase.

This confuses me, or I'm doing something wrong.  With git-dpm the way to drop
patches was to rebase interactively against upstream.  That doesn't seem to
work with gbp pq rebase, or with gbp pq import & git rebase -i master (or
upstream).

So how do I drop a patch with gbp-pq?

Cheers,
-Barry

Re: Moving off of git-dpm (Re: git-dpm breakage src:faker)

[Barry Warsaw]

On Feb 14, 2017, at 05:54 PM, Brian May wrote:

>Not sure how to unapply all patches. "quilt pop -a" won't work, quilt
>doesn't realize the patches are applied.

Yep, that does seem to be the problem.

>Maybe something like the following?
>
>git read-tree --reset -u upstream
>git reset -- debian
>git checkout debian
>git rm debian/.git-dpm
>git commit

This gets closer, but there still seems to be problems.

gbp pq import
gbp pq export --drop

That seems to round trip okay, and it just removes a few crufty lines from the
patches.  The problem comes when you try to rebase your patches on top of
upstream.

gbp pq import
git rebase -i upstream
(way way more commits to pick from than expected)

Hmm.
-Barry

Re: Moving off of git-dpm (Re: git-dpm breakage src:faker)

[Barry Warsaw]

On Feb 13, 2017, at 04:56 PM, Brian May wrote:

>There might be errors, as I was going from memory for some of this
>stuff.

Thanks Brian.  I did a quick review (without testing) and it looks pretty
good.

One section I think we should add at some point is instructions on how to
manually convert to gbp-pq, at least until we do a mass conversion.

I don't personally expect to do much package maintenance until after Stretch,
but when I do, I'll pick a package to test this workflow on.

Cheers,
-Barry

Re: Moving off of git-dpm (Re: git-dpm breakage src:faker)

[Barry Warsaw]

On Feb 07, 2017, at 10:47 AM, Ghislain Vaillant wrote:

>I know the discussion is leaning towards replacing usage of git-dpm
>with gbp-pq. I have nothing against it but, since we are talking about
>solutions for a git-centric workflow, has anyone considered the dgit-
>maint-merge workflow [1]?

As I understand it, there are a few design choices that make dgit less
desirable for this team.

The first is that dgit uses single-debian-patch rather than a series of
patches as with quilt.  The individual patches can be viewed in git but that
implies more work for interacting with upstreams and requires the use of the
git repo to examine the individual patches, making it harder for
non-developers or others outside of Debian to see what we've done to their
packages.

The second is that dgit prefers to use the upstream git repo but our work is
heavily orig-tar based since our main input is PyPI and there orig-tars (or
zips) are the predominant distribution format.  This may not be a showstopper
since dgit does say it will work with tarball workflows, but I don't know how
natural that is.

Cheers,
-Barry

Re: Moving off of git-dpm (Re: git-dpm breakage src:faker)

[Barry Warsaw]

On Feb 05, 2017, at 02:07 AM, Scott Kitterman wrote:

>Experimentation with a few packages to prepare for a migration and make sure
>the documentation is good, is fine.  We really ought to switch for real all
>at once like we did for svn -> git.  It's not much of a team repository
>without a consistent approach for VCS use.

+1

-Barry

Re: Moving off of git-dpm (Re: git-dpm breakage src:faker)

[Barry Warsaw]

On Feb 05, 2017, at 04:01 PM, Brian May wrote:

>What should we call the clone page?
>
>PqGitPackaging???

GitPackagingPq ?

Cheers,
-Barry

Moving off of git-dpm (Re: git-dpm breakage src:faker)

[Barry Warsaw]

On Jan 29, 2017, at 09:39 AM, Brian May wrote:

>I would think "gbp pq" is the most popular.

I've used it on some of my non-team packages and while it takes a little
getting used to for the standard git-dpm workflow, it's been mostly fine.

What I'd really like to see is a page like
https://wiki.debian.org/Python/GitPackaging for a non-git-dpm workflow.  (The
page itself could probably use a bit of gardening anyway.)  Specifically, I'd
like to see guidance on any tasks which are different for git-pq (or
non-git-dpm as the case may be).

I'd suggest cloning that page instead of modify that page to cover both
cases.  Edit the clone as if it were the opinionated view of just using gbp
tools and gbp-pq.  The page should also have instructions on opportunistically
switching away from git-dpm.

Then we can start to use those instructions in anger and add any other
recommendations for corner cases.  Once we have enough experience with gpb-pq
throughout the team, we can consider making an official switch.

Cheers,
-Barry

Re: Team maintained packages and git-dpm (was Re: Team upload for python-jedi)

[Barry Warsaw]

On Jan 23, 2017, at 02:41 AM, Scott Kitterman wrote:

>Which would be horrible.  single-debian-patch means that to understand the
>upstream modifications, access to the packaging VCS is required.  I think
>that would be a huge step backwards.

Agreed.

-Barry

Team maintained packages and git-dpm (was Re: Team upload for python-jedi)

[Barry Warsaw]

On Jan 22, 2017, at 03:00 PM, Dmitry Shachnev wrote:

>On Sat, Jan 21, 2017 at 11:54:13AM +, Ghislain Vaillant wrote:
>> "Drop DPMT from Uploaders (due to problems with multiple tarballs in
>> git-dpm)"
>>
>> Then, the package is no longer team-maintained?  
>
>Personally I think we could allow such packages to remain in team, even if
>they are not able to use git-dpm.

In the past, we've discussed the status of git-dpm and team maintained
packages.  I believe I'm accurate in saying:

* git-dpm is no longer actively maintained
* even so, in the majority of cases it Just Works for us

The main thing that git-dpm gives us is patch management with usually good
enough integration with quilt.  FWIW, I use straight-up gbp for most of my
actual package building tasks, but I use git-dpm for pulling in a new
upstream, managing patches, and tagging.

We've talked about eventually dropping git-dpm and just using gbp (with gbp-pq
for patch management).  I think the fact that git-dpm pretty much works fine
in most cases reduces the pressure to drop it.  And it is true that we want
consistency across the team packages so that we can document how you maintain
them in one place (e.g. the wiki[1]), and there's no guesswork when you walk
up to a repository and want to contribute.

But we do have an "out" for team maintained packages where the standard
workflow isn't appropriate.  This can include packages for which git-dpm
doesn't work, for packages which need a different branch naming scheme, etc.
This requires you to document the differences in your debian/README.source
file.

You should be judicious about deviation from our standard team workflow.  Be
kind to your fellow maintainers and really try to work within the standard
team policies and procedures.  But in cases where you must deviate, you can
still be part of the team!  Please discuss your issues on this mailing list,
come to some agreement among the active uploaders, and document your
differences in d/README.source.

Cheers,
-Barry

[1] https://wiki.debian.org/Python/GitPackaging


pgpIs_UFixE1p.pgp
Description: OpenPGP digital signature

Re: pip for stretch

[Barry Warsaw]

On Nov 28, 2016, at 01:56 PM, Barry Warsaw wrote:

>I'm starting to work on pip 9.0.1 but noticed two new dependencies.  appdirs
>1.4.0 we have, but distro 1.0.1 we don't.  It doesn't look like there's an ITP
>for that, so I'll file that bug.

Okay, as soon as python-distro clears new, I'll upload pip 9.0.1.

Cheers,
-Barry


pgpveAufZSqdG.pgp
Description: OpenPGP digital signature

Re: pip for stretch

[Barry Warsaw]

On Nov 21, 2016, at 06:37 PM, Donald Stufft wrote:

>As one might expect, I would prefer it if folks got 9.0.1 as quickly as
>possible. In particular the feature that makes it easier for upstreams to
>drop Python 2 support is one that is really only effective when people can
>consider pip 9 a "minimum" version of pip to support. Getting it into the
>hands of folks as quickly as possible would be a big boon to that.

I'm starting to work on pip 9.0.1 but noticed two new dependencies.  appdirs
1.4.0 we have, but distro 1.0.1 we don't.  It doesn't look like there's an ITP
for that, so I'll file that bug.

Cheers,
-Barry


pgpHuORawNqCM.pgp
Description: OpenPGP digital signature

Re: Binary naming for Django Related Packages

[Barry Warsaw]

On Nov 28, 2016, at 11:11 AM, Scott Kitterman wrote:

>@@ -534,6 +534,13 @@
>  This requirement also applies to extension modules; binaries for all
>  the supported Python versions should be included in a single package.
> 
>+ As a special exception to the `python3-' and `python-' binary naming
>+ policy, Python modules intended for use with Django (`python3-django'/
>+ `python-django') should add django to their binary package names to
>+ make it clear they are intended for use with Django and not general
>+ purpose Python modules, i.e.  `python3-django-' and `python-django-'
>+ respectively.

+1 but I have a question since I'm not a hardcore Django developer.

In many cases we have namespace packages, e.g. zope.*, flufl.*, etc.  Usually
these will be called python-., e.g. python-flufl.i18n.

Is there any risk of having confusing names because of a conflict between a
3rd party Django module and a Django subpackage?  e.g. python3-django-foo
vs. python3-django.foo.

I'm sure it's a non-issue in practice.

Cheers,
-Barry


pgpPkLIxR9pNA.pgp
Description: OpenPGP digital signature

pip for stretch

[Barry Warsaw]

Now that Stretch development is winding down, and I've been doing some recent
maintenance on pip, I wanted to throw this out there and see if anybody has
strong opinions one way or the other.

I'm considering sticking with pip 8.1.2 for Stretch, even though upstream is
at 9.0.1.

Here's the changelog: https://pip.pypa.io/en/stable/news/

Note the backward incompatibility introduced in 9.0.0, though Donald tells me
that it's not a common use case so unlikely to be a problem.  pip 9 will be
the default version in Python 3.6, but that probably doesn't really matter for
us since python3.6 is in experimental.

pip 9 does include a feature that makes it easier for upstreams to drop Python
2, by allowing them to specify a requires-python parameter.  There's some
better automation features and a few other things.  See the changelog linked
above for details.

I have not started to look at what if anything needs to be done to transition
to pip 9, but if you have a strong opinion one way or the other, please weigh
in.

Cheers,
-Barry


pgpQW38U_yLQf.pgp
Description: OpenPGP digital signature

Re: /usr/bin/python2 shebangs

[Barry Warsaw]

On Nov 07, 2016, at 11:44 AM, Thomas Goirand wrote:

>So, I don't agree with you, and believe that gradually using
>#!/usr/bin/python2 is a good approach to the transition. IMO, that's
>what we should start doing as much as possible.
>
>If the dependencies for Python itself aren't calculated well with that
>shebang, then we should address that to make it right regardless of this
>choice.

Clearly we won't do anything for Stretch, but we should consider this for
Buster.

Cheers,
-Barry

Re: DEP 8: Gathering Django usage analytics

[Barry Warsaw]

On Nov 07, 2016, at 01:21 PM, Brian May wrote:

>Should I ask on debian-devel? 

I think you should, and I'll be very interested in that discussion.

Several packages in our team already apply deltas to upstream to disable
certain amounts of information gathering and reporting.  The most common
example I've seen is removing a little bit of JavaScript that reports usage to
Google.

With my upstream hat on, I'm quite sympathetic to the goal of gathering usage
data.  I often get asked "how many sites are using Mailman?" and I have to
answer "I don't know".  Actually, you used to be able to do a Google search to
find references to various clues on the typical Mailman 2 listinfo page, but
since those same searches were being used to spam the FSF (and the Mailman
project), those have long been disabled.  In any case, I'd love to be able to
provide some usage numbers, but oh well.  (Maybe that's why we languish in
Negligible Funding Land. ;)

Donald Stufft mentions in a comment on the Django thread that he's thinking
about doing the same thing with pip.  My initial reaction was that we'd have
to disable that out of privacy concerns.  Or, if it's opt-out, we might have
to change the default to opt-in, which of course reduces participation rates
and makes upstreams unhappy.

I'd love to know if there's a Debian-wide policy on such things.  E.g. if
"opt-out with informed user consent" was an official policy that we could
clearly point to and reference, it would greatly help provide guidance to both
Debian maintainers and upstreams.

Cheers,
-Barry

Re: Packaging new version of ZODB (Zope Object Database)

[Barry Warsaw]

On Nov 03, 2016, at 08:36 PM, Julien Muchembled wrote:

>I'm used to gbp. I don't know git-dpm (or I forgot after seeing I would not
>like?)

git-dpm is usually pretty easy, but it's really only used in a few cases, such
as importing a new upstream, managing the patch stack, and tagging.  We
document the team's use cases pretty well so you don't even really have to
remember much:

https://wiki.debian.org/Python/GitPackaging#New_upstream_release

For a lot of other package management tasks (e.g. building source packages,
cloning, pulling, etc.), gbp works just fine.

That said, we know git-dpm has not been developed in a very long time, and is
for all intents and purposes, abandonware.  It works, so I don't think there's
a huge urgency to get rid of it (obviously, since we haven't ;) but it should
be in our long-term team goals to move away from it.

>Not sure if all python-modules repositories are like persistent, but for me,
>mixing Debian work with imported tarballs in the same branch is
>terrible. When possible, I prefer to fork the upstream repository, otherwise
>no upstream source at all.

You're not alone, but I think that's still a minority opinion in the team.
Our packages are so tightly integrated with PyPI, and source tarballs are such
an ingrained aspect of that service, that a pristine-tar based approach for
team packages still makes sense, IMHO.

Cheers,
-Barry

Re: Packaging new version of ZODB (Zope Object Database)

[Barry Warsaw]

On Nov 02, 2016, at 10:46 AM, Arnaud Fontaine wrote:

>> I write  to debian-python, because  some of the involved  packages are
>> not specific to  Zope. Actually, I even think that  ZODB itself is not
>> specific to Zope, but well,  changing section of existing packages can
>> be another topic.  
>
>This has  already been discussed  but all  the packages in  pkg-zope SVN
>repository will have to be  moved to python-{modules, apps} repositories
>(because there  is almost no activity  on pkg-zope and most  modules are
>used independently  of Zope anyway)  and we should use  debian-python ML
>for the  same reason,  so yes,  please use  debian-python ML  and commit
>everything to python-{modules, apps} repositories.

+1.  I do still touch some of the ztk packages and would dearly love to ditch
svn, but just haven't had the time to think about a proper migration.  Should
we just admit defeat and do on-demand conversions, preserving history if
possible but not worrying about it too much?

And then what about just using gbp and ignoring git-dpm?  The latter still
kind of works but we know it's a dead-end.  Anybody looked at dgit?  Is that
a useful option?

can-of-worms-ly y'rs,
-Barry


pgpgflcDONUfX.pgp
Description: OpenPGP digital signature

Re: Python 4 and ‘python3’

[Barry Warsaw]

Don't panic. :)

On Nov 03, 2016, at 09:28 AM, Ben Finney wrote:

>But the above post implies that pointless confusion will be directly
>courted, merely because of some aesthetic objection to a two-digit
>component in the version string.

Those are Nick's opinions.  Everyone's got one!

AFAIK, there is *no* official declaration (e.g. from Guido or the mythical
Python 4 release manager) about this either way.

>At the current rate of Python releases, it's not very far in the future
>before the Python release managers must decide what the version string
>for “the release that comes after 3.9” will be.

We're up to Python 2.7.12 so the double digit version component ship has
sailed and it wasn't all that Y2K-y, so I doubt there will be a hard and fast
prohibition against 3.10.  Even if there is, we won't see a possible 3.10
until 2022 if I'm doing my math correctly and we stick to the roughly 18 month
release cycle.

I predict that when the time comes, it'll generate a gigathread's worth of
discussion, 3 or 4 competing PEPs, and then Guido will just pronounce.

Cheers,
-Barry

Re: Python 4 and ‘python3’ (was: /usr/bin/python2 shebangs)

[Barry Warsaw]

On Nov 02, 2016, at 01:57 PM, Ben Finney wrote:

>Donald Stufft  writes:
>
>> /usr/bin/python3 being Python 4.x is a bit weird though  
>
>Seriously? Who is proposing that?
>
>> and it’s likely that Python 4.x is not going to be another break the
>> world release.  
>
>Certainly the command ‘python3’ should only ever point to the Python 3
>interpreter.
>
>If upstream ever releases a “Python 4” but expects the interpreter for
>that to also be named ‘python3’, I think we can declare upstream to be
>directly courting user pain, and secede on behalf of our users.

I wouldn't at all be surprised if /usr/bin/python is reclaimed for some future
post-Python2-demise Python 4 interpreter.  It might even be a good thing since
I'm not sure I'd want a /usr/bin/python4.

Not that I'm expecting Python 4 any time soon, but if Larry Hasting's
gilectomy work actually pans out, that'd be a solid contender for it.

Cheers,
-Barry

Re: /usr/bin/python2 shebangs

[Barry Warsaw]

On Nov 01, 2016, at 11:34 AM, Scott Kitterman wrote:

>I don't think /usr/bin/python should ever point to a python3 version.  It
>should be dropped when python2.7 is removed.  I think the existence of
>/usr/bin/python2 is a limited to a work around for a specific distros
>insanity.  There's no need to encourage it's use in Debian as that particular
>insanity hasn't and won't (as long as I'm a python*-defaults co-maintainer).
>
>P.S. But I think you already knew how I feel about it.  ;-)

Oh yes, I knew. :)

Okay, so you don't think we should ever shebang scripts to /usr/bin/python2.
Maybe lintian should warn specifically against that then?

(I still think /usr/bin/python2 is the better long term path, but I don't feel
strongly enough about it to advocate a formal transition.)

Cheers,
-Barry

/usr/bin/python2 shebangs

[Barry Warsaw]

Over in #834193, a user is asking for a /usr/bin/pip2 to mirror /usr/bin/pip
because some uses cases apparently prefer pip2 over pip.  That seems like a
reasonable request on the face of it, and easy to support.

However, I thought, well why not shebang pip2 to /usr/bin/python2 because 1)
it would parallel the script name; 2) I do think it would be in our best long
term interest to start shebanging system scripts explicitly with python3 or
python2 as appropriate.  Yes, I know that we don't intend to make
/usr/bin/python be Python 3 any time soon, but maybe someday we'll change our
minds and PEP 394 will embrace that strategy across Linux distros.

In any case, it produces a mild lintian complaint:

E: python-pip: python-script-but-no-python-dep usr/bin/pip2

Of course the /usr/bin/python shebanged /usr/bin/pip doesn't produce that
warning, and the package's Depends properly has ${python:Depends}.  I think
lintian is wrong, but I'd like to hear any other opinions.

Cheers,
-Barry


pgpa5POdfuIrO.pgp
Description: OpenPGP digital signature

Re: Upstream build system, Sphinx autodoc, Python import path

[Barry Warsaw]

On Sep 29, 2016, at 11:30 PM, Thomas Goirand wrote:

>I just had a quick look to that page, not only it advises to override
>the wrong dh sequence, but also it gives stupid advices for intersphinx:

ObDIY: "It's a wiki". :)

Cheers,
-Barry

Re: can we disable the bounce kicker? Re: confirm

[Barry Warsaw]

On Sep 11, 2016, at 09:41 AM, Sandro Tosi wrote:

>now can we PLEASE stop talking about how the perfect smtp system
>should work and GET BACK to discuss if we are able and want to disable
>the suspend membership upon bounces (that's what the mail you receive
>says, so do not nitpick on this term again)

The downside is that the mail system will no longer track legitimate bounces,
so more traffic will be generated to addresses which won't get delivered, and
more bounce traffic will be received by the incoming mail system and just
thrown away.  The question to answer is whether that's an acceptable trade-off
for reducing the inconvenience of people who need to occasionally re-enable
their subscriptions because of the mail provider they use?

I'm not a Debian system administrator, so I can't answer that question.
However, if the impact on the network is acceptable, then all else being
equal, making users lives better is usually a good choice. :)

(I'd still complain to my mail provider, for whatever good that would do.)

Cheers,
-Barry

Re: can we disable the bounce kicker? Re: confirm

[Barry Warsaw]

On Sep 10, 2016, at 04:32 PM, Santiago Vila wrote:

>AFAIK, Gmail does not bounce spam. It rejects messages with broken
>DKIM signatures.

There are a number of DKIM mitigation features in newer versions of Mailman 2
that should be investigated.  I can't tell what version of Mailman this list
is using, and obviously don't have any access to the list or system
configurations.  If it really is DKIM that's causing your problem, please look
at these.  I haven't seen any reports of changing GMail behavior here on the
mailman-users list, although I haven't been watching carefully lately.

Cheers,
-Barry

Re: can we disable the bounce kicker? Re: confirm

[Barry Warsaw]

On Sep 10, 2016, at 02:46 PM, Sandro Tosi wrote:

>I'm sure i'm not the only member using gmail, which bounces spam
>emails and that what causes this problem.

Are you sure about that?  Bouncing spam has been bad practice for a very long
time.

Cheers,
-Barry

Re: using git-dpm or plain git-buildpackage in PAPT and DPMT

[Barry Warsaw]

On Aug 31, 2016, at 11:31 PM, IOhannes m zmölnig (Debian/GNU) wrote:

>isn't this what `gbp pull` is supposed to do?

Indeed.  Either this is new-ish or I missed it, but it does exactly what I
want.  Thanks!

-Barry


pgpvGDUnhftFZ.pgp
Description: OpenPGP digital signature

Re: on keep providing python 2 packages

[Barry Warsaw]

FWIW,

On Aug 21, 2016, at 12:30 AM, Piotr Ożarowski wrote:

>* all Python applications that support it, should use 3.X only *now*
>  (and do not bother with things like alternatives or "-3" suffixes /
>  "python3-" prefixes - at least for new packages; I'd even slowly start
>  removing alternatives, if it doesn't affect users),

+1

>* libraries in Stretch should support 2.X (i.e. add python-foo binary
>  packages) if that doesn't require too much additional work (py2dsp
>  still creates them). I'm OK with shipping 3.X only packages in NEW
>  packages, though. I'd not encourage people to do so but also not
>  forbid it,

I'd also say that it isn't worth *removing* Python 2.x support right now,
unless of course upstream drops it first.

>* we shouldn't accept 2.X only packages in Buster (Stretch+1, released ~2019)
>  unless they're a dependency of other packages, and start shipping 3.X only
>  packages where it makes sense (and I hope that decision will be mostly made
>  by upstreams by simply dropping 2.X support). We can drop some 2.X packages
>  (problematic to maintain? better alternatives available? low popcon?), but
>  do not do a mass removal yet,

+1

>* for Bullseye (Stretch+2, released ~2021) we should start dropping 2.X
>  packages, and maybe even remove 2.X interpreter,

+1 - remember that this will be 10-11 years after 2.7 was released!

>* Bullseye + 1 (~2023) is the one without 2.X interpreter and
>  python-foo packages for sure (and without /usr/bin/python symlink or
>  at least without Debian packages mentioning it, there should be a rule
>  to not speak about /usr/bin/python symlink! ;)

+1

>Note that Python upstream will stop supporting 2.X in ~2020 so about one
>year (and a half?) after releasing Buster.

As you point out, upstream will stop supporting Python 2.7 for normal bug
fixes in 2020, although I suspect it will be supported in security-fix
source-only mode for some years after that.  There has not be an official
announcement of that IIRC, and PEP 373 doesn't describe that, so it's just my
opinion.

Cheers,
-Barry


pgpyP28DxWK2S.pgp
Description: OpenPGP digital signature

Re: Help for Python mock test suite needed

[Barry Warsaw]

On Aug 17, 2016, at 01:58 PM, Andreas Tille wrote:

>This exactly was my question:  Is there any advise how to inject
>sensible debugging code or any other strategies to find out what might
>be the problem.
>
>> So you can use the full power of your development environment to hunt
>> the problem: an interactive debugging session, etc. Use your Python
>> skills :-)  
>
>I'm afraid at first I need to enhance my Python skills ...

pdb is debugging 101: https://docs.python.org/3/library/pdb.html

Specifically, add this line to the code at the point where you want to start
inspecting the state of the program:

import pdb; pdb.set_trace()

Now run the test suite and when you hit the breakpoint, you can print values,
inspect object, step through calls, etc.  It may or may not help you solve the
problem, but you'll get a much better sense of what's actually going on, and
it should at least improve the quality of an upstream bug report.

Cheers,
-Barry

Re: using git-dpm or plain git-buildpackage in PAPT and DPMT

[Barry Warsaw]

On Aug 15, 2016, at 11:44 PM, Thomas Goirand wrote:

>If we decide to use gbp-pq, in fact, we're deciding to not decide, and anyone
>can use PoQ (my choice, for example). Indeed, the way to store the patches is
>PoQ, and you then "gbp-pq import", modify, then "gbp-pq export" and store the
>packaging branch like this (ie: like a PoQ branch). So, basically, we'll be
>back to what everyone else is doing (ie: 99% of git maintained packages in
>Debian as much as I saw).

That sounds perfect then!

>IMO, that's required if we decide to continue using pristine-tar (which
>I don't think is a good idea, but let's not discuss that, as there seem
>to be a consensus for it).

So at a minimum:

gbp.conf
[import-orig]
pristine-tar = True
gbp.conf

Looking at my ~/.gbp.conf file, I have a bunch of other entries, some of which
are probably not appropriate for team settings.  I'd suggest `sign-tags=True`
though.

*Maybe* also these as conveniences:

[import-orig]
postimport = gbp dch -N%(version)s
import-msg = New upstream release.

What else?

Cheers,
-Barry


pgp_uaUuGilbj.pgp
Description: OpenPGP digital signature

Re: BTS bot in #debian-python IRC channel

[Barry Warsaw]

On Aug 14, 2016, at 01:01 PM, Scott Kitterman wrote:

>On August 14, 2016 12:51:18 PM MDT, "Piotr Ożarowski"  wrote:
>>[Ben Finney, 2016-08-14]  
>>> Would it be a good idea to first have it running in an analogous
>>> channel, ‘#debian-python-changes’?  
>>
>>+1 (I'd move VCS commits messages there too)  
>
>+1 for both.

WFM.  (That's a +1 without just +1'ing :).

-Barry

Re: BTS bot in #debian-python IRC channel

[Barry Warsaw]

On Aug 13, 2016, at 10:48 PM, Ondrej Novy wrote:

>I would like to add BTS bot to IRC channel #debian-python with same
>notifications (uploads, bug reports) as in #debian-devel-changes filtered
>to maintainer/uploaders:

I don't *think* I'd mind getting these notifications in irc, but I guess I
won't know for sure until it happens.  If it's disruptive to real-time
communications (which I think is the most important function of the channel),
then I'd be against it.  But I don't mind an experiment to see if it helps,
hurts, or is neutral.

Cheers,
-Barry

Re: using git-dpm or plain git-buildpackage in PAPT and DPMT

[Barry Warsaw]

On Aug 12, 2016, at 05:50 PM, Sergio Durigan Junior wrote:

>I understand this is a matter of personal taste, but I beg to differ.  I
>have been using git-buildpackage for most of my non-Python packages and,
>despite really small nits here and there, I think it is an awesome tool.

I think there are multiple ways in which git-buildpackage is used.  E.g. I use
`gbp buildpackage` even for git-dpm maintained packages, to build the source
package, etc.  (It even has a nicer `clone` command that makes it easy to
actually get tracking branches; just wish it had something like
`pull-and-update-all-branches` since it's a bit of a PITA to update the
pristine-tar branch.)

It's just the few things that git-dpm adds on top of gbp that we're discussing
getting rid of, like importing a new upstream, managing the quilt patch set,
tagging.

Cheers,
-Barry


pgp7ppU1EAjvh.pgp
Description: OpenPGP digital signature

Re: using git-dpm or plain git-buildpackage in PAPT and DPMT

[Barry Warsaw]

Thanks for the follow up Simon.  One question...

On Aug 11, 2016, at 12:12 AM, Simon McVittie wrote:

>gbp pq works best if all repository users stick to the dialect of DEP-3
>where all Debian-specific pseudo-headers appear at the end of the diff
>(next to the Signed-off-by if any),

Did you mean to say "end of the diff headers"?  I ask because in your "Good
for gbp-pq" example:

>From: Donald Duck 
>Date: Fri, 01 Apr 2016 12:34:00 +0100
>Subject: Reticulate splines correctly
>
>This regressed in 2.0.
>
>In particular, this broke embiggening.
>
>Origin: vendor, Debian
>Forwarded: http://bugs.example.org/123
>---
>[diff goes here]

the DEP-3 headers are *before* the actual diff, separated from the diff
headers by a blank line.

Cheers,
-Barry

Re: using git-dpm or plain git-buildpackage in PAPT and DPMT

[Barry Warsaw]

On Aug 10, 2016, at 11:53 AM, Nikolaus Rath wrote:

>I think the only way to make this less painful is to get rid of the idea
>of managing patches in a VCS and use something like gitpkg. This has the
>drawback source package is now *generated* from the Git repository
>(i.e., you can't do git clone + debuild), but it makes maintaining the
>Git repository much less painful. Judging from all the attempts I've
>seen so far, trying to put patches (i.e., the output of a VCS) under
>version-control is just a doomed endeavor.
>
>I don't believe that switching from git-dpm to git-buildpackage is going
>to make things easier, it'll just be trading one set of problems for
>another.

Two thoughts:

* We probably need to do some actual experimentation on actual packages, so
  we should plan for allowing that on a limited basis, with the caveat that
  once a new workflow is chosen, we'll make all team packages consistent
  again.

* With git-dpm we *had* to enforce the tool choice because git-dpm's artifacts
  had to be preserved.  If we ditch git-dpm, is that still the case?  IOW, if
  you choose to use gbp-pq, am I forced to do so when I modify the same repo?
  Or if you choose to use PoQ (plain 'ol quilt), will that affect how others
  can co-maintain the package in git?

We only need to mandate workflows and conventions where the effects and
artifacts are visible after a package is updated.  Think of it like the choice
of editor - no one cares whether you use vim, emacs, gedit, or whatever to
modify the files, because its effects are only local to you (for the most
part).

Cheers,
-Barry

Re: using git-dpm or plain git-buildpackage in PAPT and DPMT (was: PAPT Git)

[Barry Warsaw]

On Aug 10, 2016, at 08:49 PM, Brian May wrote:

>Most of the time it works pretty well... It looked good compared with
>the alternatives available at the time we made the decision.
>
>However this is irrelevant IMHO if it isn't being mantained.

Yep.  git-dpm was the best of breed at the time we were switching from svn,
and several developers had good experiences with it.  When things are fairly
simple, so is git-dpm, and when it Just Works, it's easy to use.  When it's
*not* -- or when you hit any of the bugs previously mentioned -- then you're
out of luck.

git-dpm is no longer maintained so those bugs won't get fixed.  And gbp-pq has
improved a lot since then.

Moving PAPT to git without git-dpm will gain team experience with that
toolset.  IIRC we figured it was as easy as `rm debian/.git-dpm` to switch,
and we should test that on a few candidate packages.  More important is to
update the documentation:

https://wiki.debian.org/Python/GitPackaging

so we're all on the same page.

Cheers,
-Barry

Re: pypi2deb 1.20160809 and --profile dpmt

[Barry Warsaw]

On Aug 09, 2016, at 11:28 PM, Daniel Stender wrote:

>On this occasion ... let it be me to start the discussion: let's get into Git
>also for Python Apps soon.

You're channeling Stefano on #debian-python:

 barry: I've started poking at PAPT SVN migration  [15:08]

:)

Cheers,
-Barry

Re: pep8 renamed to pycodestyle

[Barry Warsaw]

Hi Ondrej,

On Aug 08, 2016, at 11:46 AM, Ondrej Novy wrote:

>what do you think about this:
>https://anonscm.debian.org/cgit/python-modules/packages/pep8.git/commit/?id=655e6f631fc854c51f48f142e58bdb6b4af8494f
>
>With this change we can make slow transition pep8 -> pycodestyle without
>breaking current pep8 users.

It seems reasonable to me.  We definitely want to keep the pep8 script and
libraries alive for a while so other packages can catch up.  There's also some
discussion of this upstream:

https://github.com/PyCQA/pycodestyle/issues/544

I think we don't need to wait for upstream to decide though, so +1 on your
changes.

>BTW: pycodestyle is still in NEW, so don't upload :)

I'll leave that to you! :)

Thanks,
-Barry


pgpccZFTHACjr.pgp
Description: OpenPGP digital signature

Re: pep8 renamed to pycodestyle

[Barry Warsaw]

On Aug 08, 2016, at 12:52 AM, Ondrej Novy wrote:

>created (https://bugs.debian.org/833683) and uploaded :)

\o/

Thanks,
-Barry


pgpl8ZGPKewXt.pgp
Description: OpenPGP digital signature

Re: voluptuous in DPMT

[Barry Warsaw]

On Aug 02, 2016, at 11:35 PM, Thomas Goirand wrote:

>I need voluptuous for OpenStack. So unless someone needs the version in
>Unstable, I prefer to not risk to break anything, and upload updates to
>Experimental, just in case the new version breaks something in the older
>release of OpenStack. My intention was to upload Voluptuous to Unstable
>when Newton was out. However, if you need it, then it's fine to upload
>to Unstable, and if something break in OpenStack, then it's up to me to
>repair... :)

Okay, I *think* I'm parsing that as: if you need the newer version it's okay
to upload to unstable.

It's kind of hard to tell what's changed.  Upstream doesn't seem to publish a
changelog/news file afaict, and didn't even tag 0.9.2 in their git repo.  But
Ubuntu has the experimental version with a delta and I'd like to get those in
sync, so I think it makes sense to upload the latest release.

>Well, if you look at tracker.debian.org, then you'll see that I uploaded the
>package in early June, when I had still no commit rights to the git of the
>DPMT. Therefore, I couldn't "git push".

Darn.

>Unfortunately, after I got my write access back, I forgot about Voluptuous,
>and therefore didn't git push. Sorry for that, and thanks so much for doing
>the work of pushing the changes.

No worries, and I'm glad you got your write access back.

>The file here:
>https://github.com/openstack/requirements/blob/master/upper-constraints.txt
>
>shows that OpenStack Newton (currently in Experimental) is right now gating
>on 0.9.1 upstream, so 0.9.2 is probably fine as well. Please go ahead, you're
>saving me some work! :) Hopefully, this wont break Mitaka (currently in Sid).

Cool!

This makes me wonder whether we can put something in the d/control or
otherwise to indicate that a package may be used by OpenStack, and where to go
for more information or sanity checks before new versions are uploaded.

It definitely doesn't make sense to have to check for OpenStack compatibility
for every DPMT package.  I only noticed this one after looking at the open
bugs and checking the versions in Debian and Ubuntu.  Then I saw your unpushed
upload and figured I better ask before I break stuff. :)

Maybe we could add a header to d/control that would say something like
"There's an OpenStack constraint on the version numbers, please check before
you upload"?  Then at least for a Maintainer: DPMT package, a quick email like
mine could go out first.

I'll upload 0.9.2 to unstable in a few minutes.

Cheers,
-Barry


pgpZGSYlISNQm.pgp
Description: OpenPGP digital signature

Re: voluptuous in DPMT

[Barry Warsaw]

On Aug 02, 2016, at 01:54 PM, Barry Warsaw wrote:

>You won't mind if I upload 0.9.2?  I'll try to merge your changes in, but
>I've yet to see how easy that will be.

Not too bad actually.  I think I have it all prepped and ready to go in DPMT
git.

Cheers,
-Barry


pgpsyaBqHgRCL.pgp
Description: OpenPGP digital signature

voluptuous in DPMT

[Barry Warsaw]

Thomas, what's going on with the voluptuous package?

I'm looking at starting to use it for a project and I noticed that it was
pretty out of date (the current upstream is 0.9.2).  So I gbp cloned the DPMT
git-dpm repo and started to hack on it.  The last upload appeared to be
0.8.2-1 by Robert S. Edmonds, although Ondřej Nový made some recent
non-uploaded commits.

Then I noticed that you've made two uploads since then, 0.8.8-1 to unstable
and 0.8.11-1 to experimental.  But neither of these changes appears in DPMT
git repo for the package, afaict.

So, questions:

Why was 0.8.11 only uploaded to experimental?  It doesn't seem to have all
that many reverse dependencies, so why not just upload it to unstable?

And, why weren't your changes committed to git and pushed?  Where did you make
your changes?  I don't want to lose them, but since the package *is* DPMT
maintained, its git repo really should reflect all the uploads.

You won't mind if I upload 0.9.2?  I'll try to merge your changes in, but
I've yet to see how easy that will be.

Cheers,
-Barry


pgpsvOHTk59eN.pgp
Description: OpenPGP digital signature

Re: [Python-modules-team] Bug#830186: sphinx: intersphinx mapping extension causes network access during package builds

[Barry Warsaw]

On Jul 12, 2016, at 04:07 PM, Thomas Goirand wrote:

>I find this not the top-noch way. Here's what I do:
>
>override_dh_sphinxdoc:
>ifeq (,$(findstring nodocs, $(DEB_BUILD_OPTIONS)))
>   PYTHONPATH=. sphinx-build -b html doc/source \
>   debian/python-foo-doc/usr/share/doc/python-foo-doc/html
>   dh_sphinxdoc
>endif

I'm going to do this in the next python-webob upload to close #830611.  It had
been building the docs in override_dh_installdocs.  I'll also need to patch
the conf.py file to use the python{,3}-doc files and add those packages as
build dependencies.

Cheers,
-Barry


pgpROt_11jI7v.pgp
Description: OpenPGP digital signature

Re: Removing me from Uploader field of html5lib

[Barry Warsaw]

On Jul 04, 2016, at 04:52 PM, Olivier Berger wrote:

>Unfortunately, I'm no longer able to dedicate time to help maintaining
>the html5lib package.

Thanks so much for your past work on it!

>Thus I'm requesting that anyone uploading the next version as part of
>the team, please remove me from the uploaders field.

Done in vcs.  If it's okay with you, I'll hold off doing a new upload until
there's a new upstream.

>Btw, the docs on the wiki documents how to be added/contribue, but not
>really how to stop ;-)

Despite that, you figured it out. :)

Cheers,
-Barry

Re: Python program as a command-line program (was: Python package providing both modules and an app)

[Barry Warsaw]

On Jun 23, 2016, at 11:17 AM, Ben Finney wrote:

>There isn't, AFAIK, anything portable that I can write in the shebang to
>turn a command invocation of ‘./foo/bar.py’ into ‘python3 -m foo.bar’.

Why not just have a file that contains only?

exec python3 -m foo.bar

Cheers,
-Barry

Re: Best practice for adding python3 version to existing module with helper programs?

[Barry Warsaw]

On Jun 24, 2016, at 03:30 PM, Ben Finney wrote:

>Neil Muller  writes:
>
>> I'm working on packaging the latest version of sqlobject, which adds
>> supprt for python 3, and I'm unsure of how best to handle the helper
>> applications that ship as part of the package. The helper applications
>> will work with both python2 and python3, so it's not clear to me how
>> to apply the advice given in [1], and I'd ideally want the helper
>> programs available regardless of which version of the module is
>> installed.  
>
>What is the purpose of the helper applications? Is there a significant
>use case for “I want specifically the Python 2 version of this
>application”?
>
>If not, I would advise making only the Python 3 applications available,
>in a separate binary package (and put it it “Section: database”) which
>describes the purpose without mentioning Python versions.

I concur with Ben.  If the end user won't care whether Python 2 or 3 is used
and the helper works the same regardless of which Python invokes it, please
just ship the Python 3 version.  It should go in a separate binary package.

Cheers,
-Barry

Re: Python package providing both modules and an app

[Barry Warsaw]

On Jun 22, 2016, at 11:25 AM, Ben Finney wrote:

>This seems to be more common now that command-line invocation is
>becoming even more discouraged. When the upstream documentation
>recommends ‘python3 -m foo.bar’ as the primary means of invoking the
>command line functionality, that really blurs the line between
>command-line application versus library.

Indeed.  It's true that -m invocation isn't as pretty as a /usr/bin script,
but it does have the advantage of unambiguously choosing the Python version
you want to run the script with.  How important that is depends on the
application of course.

>There is a compounding tendency to disparage ‘python3 ./foo/bar.py’,
>which is subject to weird hacks and incomplete workarounds like
>. I wish this trend could
>be effectively reversed, because IMO this is a serious impediment to
>considering Python a good choice for command-line scripting. But this is
>all a digression, my apologies.

Sorry, I don't quite understand the above.  Do you mean that you would rather
use `python3 ./foo/bar.py` more often and `python3 -m foo.bar` less often?

In any case, thanks Hugo for choosing Python 3 as the version to use for the
/usr/bin script. :)

Cheers,
-Barry

Re: pep8 renamed to pycodestyle

[Barry Warsaw]

On Jun 21, 2016, at 10:18 AM, Piotr Ożarowski wrote:

>I'd simply create new source package with new binary packages and remove
>src:pep8 even from Stretch (once all rev. dependencies use new package).

Yep, I'm leaning toward this, or at least creating a new source package for
pycodestyle.  I think there are enough backward compatibility concerns that we
should leave it to reverse-dependencies to make the switch.  Whether or not we
remove pep8 from Stretch is a separate decision, but I'd be okay with that
if/when all the revdeps get ported over.

Cheers,
-Barry

pep8 renamed to pycodestyle

[Barry Warsaw]

You might be aware that the upstream pep8 package has been renamed to
pycodestyle.  Both the command line and imports have changed, although not in
a backward compatible way unfortunately[1].

There's a new upstream flake8 which I'm preparing in git-dpm, but it will be
blocked until we decide what to do about the Debian pep8 package.

I think we eventually want to rename Source:pep8 to Source:pycodestyle, as
well as Package:pep8, Package:python-pep8, and Package:python3-pep8.  I
suppose we could at least keep the source package name unchanged and
transition to the new binary package names.

I'd like to add some backward compatibility hacks, either our own or
upstreams, for the name change.  See the referenced issues.

Looking for any thoughts, comments, or suggestions.  I also don't want to step
on any toes if someone's already starting this transition.  There are some
reverse dependencies, but not too many I think so it should be fairly easy to
just upload new packages (if available; I haven't checked them all).

Cheers,
-Barry

[1] https://github.com/PyCQA/pycodestyle/issues/544
https://github.com/PyCQA/pycodestyle/issues/549


pgpBqAeXpTkEt.pgp
Description: OpenPGP digital signature

Re: Intention to upload 21 packages to jessie-backports

[Barry Warsaw]

On May 31, 2016, at 03:44 PM, Thomas Goirand wrote:

>Changed-By: Barry Warsaw <ba...@debian.org>
>Source: python-webob

I am at Pycon US this week (anybody else?).  Please feel free to do what you
need!

Cheers,
-Barry


pgp4htO8yJedU.pgp
Description: OpenPGP digital signature

PyconUS 2016

[Barry Warsaw]

Pycon 2016 starts in one week in Portland OR, USA.  I'll be there for the
duration (flying in late Friday evening).  Who else is coming?

We should definitely have a Debian Python BOF, and perhaps a keysigning if
there isn't one organized for the larger Python community.  I'm sure we can
find fun things to talk about, like PyPy support, planning for Python 2's EOL,
the inevitable Python 3.6 transition[*], etc.

If you aren't already registered for the conference, sorry but it's too late.
I'm told it sold out months ago.  But there are 4 days of post-conference
sprinting and those are free to all.  You don't even have to be registered for
the conference to attend the sprints.

Hope to see some of you there!

https://us.pycon.org/2016/

Cheers,
-Barry

[*] Yay Doko!

https://packages.debian.org/search?keywords=python3.6=names=all=all


pgpsADXX8kVov.pgp
Description: OpenPGP digital signature

Re: pypy pakages

[Barry Warsaw]

On May 10, 2016, at 09:56 PM, Tristan Seligmann wrote:

>I think it would be great if we could get performance-sensitive applications
>running on PyPy instead of CPython, but of course this requires the whole
>dependency graph to have pypy-* packages built.

That might be a good approach to building out the PyPy stack so we get more
experience with it.  Have you identified a leaf package or two that would
benefit from being run under PyPy in Debian?  That's the first step; then work
backwards in the dep chain.

Cheers,
-Barry

Re: pypy pakages

[Barry Warsaw]

On May 10, 2016, at 07:23 PM, Michael Fladischer wrote:

>is there a specific reason why there are so few pypy-* packages in the
>archive? Is it just a lack of interest or are any practical reasons not
>to have them?

I don't think there are too many practical reasons other than every package
that wants to add PyPy support has to minimally add a binary package section
and some build-deps.  For bonus, build-time and/or autopkgtests should also be
added and of course the whole extra stack needs testing.

I just added PyPy support to pyparsing, but mostly because it's a new
dependency of setuptools and Doko requested it.  I've also submitted a bug and
patch to autodep8 to recognize and generate PyPy test suites (#823883).

>I just tried to add pypy support to some of my packages and it was a
>pretty straight forward thing to do.

\o/

>If it's just a lack of interest, would anyone be willing to work with
>me to add pypy support to packages that are known to be compatible[0]?

JFDI, or do you want reviews of the packaging and/or need sponsorship?

Then the question is, what about Jython?  I guess we don't (yet) have to care
about IronPython.

Cheers,
-Barry


pgphxaAK4A4Z5.pgp
Description: OpenPGP digital signature

Re: [Python-modules-team] [Python-modules-commits] [mpmath] 01/01: d/copyright: Changed licence shortname BSD -> BSD-3-clause

[Barry Warsaw]

On Apr 30, 2016, at 07:21 PM, Ondrej Novy wrote:

>announced before:
>http://lists.alioth.debian.org/pipermail/python-modules-team/2016-March/030256.html

Thanks!  I must have missed it, but you did the right thing.

-Barry

PS: why do we have two mailing lists?! :( :(


pgpnvPRcM4YYn.pgp
Description: OpenPGP digital signature

Re: Test suite in github but missing from pypi tarballs

[Barry Warsaw]

On Apr 21, 2016, at 04:52 PM, Thomas Goirand wrote:

>It's best that the test suite goes within the project. So if project is
>called foo, then best is to get the test folder in foo/tests. This way,
>you don't even need to fix the MANIFEST.in.

+1 - as an upstream I always do this because I like having the tests as close
to the code its testing as possible.  It generally does no harm to include
them in the binary package so why bother, but yes in some cases you might want
to split them out.

Cheers,
-Barry

Re: Test suite in github but missing from pypi tarballs

[Barry Warsaw]

On Apr 21, 2016, at 02:54 PM, Tristan Seligmann wrote:

>With my upstream developer hat on: source packages on PyPI are meant for
>end users to install via pip. They often include generated artifacts, and
>don't include things that aren't intended for installation via pip (tests
>being just one of these).
>
>For distribution packaging purposes, the GitHub tags are generally
>preferrable. GitHub makes archives of tagged releases available as
>tarballs, so this is generally a simple tweak to debian/watch.

With my own upstream hat perched precariously on my head, I disagree.  I
generally prefer to include the test suite in the PyPI tarballs and I also
submit bug reports to upstream that don't include them.  Most have been very
accommodating, especially when you can provide a simple patch to their
MANIFEST.in.

I agree with Fred that PyPI tarballs have multiple intended consumers.

I also don't particularly like relying on GitHub generated tarballs.  Despite
popular believe, not every upstream uses GH or even git, signs their tags or
even tags their releases.  But almost *every* Python package releases tarballs
to PyPI.

There are a few very limited cases where upstreams have balked at supplying
their test suites in their PyPI tarballs.  One that I agree with is pip[*]
because it's downloaded like a jillion times a day so in that case, have a
more svelte tarball makes sense.  I think Donald and I talked about them
spinning a pip-dev tarball for distro consumers, but for now I just override
dh_auto_test in d/rules.

Cheers,
-Barry

[*] https://github.com/pypa/pip/issues/3370

Re: remember to git pull

[Barry Warsaw]

On Apr 17, 2016, at 06:36 PM, Brian May wrote:

>(or even better, use the quick-update script instead)

quick-update script?

Oh, and +1 for the recommendation!  I'd just add to also be sure that you
commit and push your changes. ;)  I've had one or two cases where the repo
doesn't match the archive.  dgit

-Barry

Re: CPython hash randomization makes some Python packages unreproducible

[Barry Warsaw]

On Apr 09, 2016, at 08:11 PM, Julien Cristau wrote:

>FWIW I think that's a bad idea.  A number of packages run their test
>suite at build time, and running the tests with hash randomization
>enabled seems to me like something we shouldn't give up.

Completely agreed.  Although we're in the long tail now, during the bulk of
the Python 3 transition, we found real bugs when hash randomization was
enabled.

>Couldn't packages where the binary packages contents depend on the hash seed
>just set one themselves?

+1

-Barry

Re: Packaging Grip

[Barry Warsaw]

On Apr 06, 2016, at 11:48 PM, Scott Kitterman wrote:

>In my opinion either can be correct depending on the primary purpose of the
>package.

I think that's true; take it on a case-by-case basis.

In general, I like having a separate binary package for the /usr/bin script
because it can be more easily discovered and because it can avoid confusion on
library package naming.  IOW, 'python-foo' is a library, not usually expecting
it to contain a /usr/bin.

As for entry points, I also like to use setuptools eps by default if they
work.  I had to ship a non-ep /usr/bin/pip because of bootstrapping issues.

-Barry

Re: dropping top patch with git-dpm

[Barry Warsaw]

On Apr 04, 2016, at 05:33 PM, Brian May wrote:

>Ok, so as I expected, integrating a new upstream release resolved this
>issue for me. git-dpm import-new-upstream left me in the patched
>directory, and I was able to drop the last patch and then run git-dpm
>update-patches.

When it works, it's magic. ;)

On the rare occasion where I've had to do it manually, I don't use reset.  I
checkout-patched then rebase -i upstream and drop the commit.  I don't know if
that's effectively the same thing as what you've done, and ISTR that it
doesn't always work, but I know that it sometimes does .

-Barry

Re: running tests against installed version of package

[Barry Warsaw]

On Mar 26, 2016, at 01:49 PM, Brian May wrote:

>Barry Warsaw <ba...@debian.org> writes:
>
>> In some cases, I've just taken to adding DEP-8 tests for those.  
>
>Do you have an example I can look at?

Hi Brian.  Take a look at tox for example.

Cheers,
-Barry

Re: running tests against installed version of package

[Barry Warsaw]

On Mar 25, 2016, at 06:17 PM, Brian May wrote:

>However I have a package where the tests require the entry points from
>setup.py to be configured, the tests fail without this.
>
>So what is the appropriate way to modify debian/rules to get the tests
>to run from the installed version with the entry points available?

In some cases, I've just taken to adding DEP-8 tests for those.

Cheers,
-Barry

Re: Packaging dependencies for mailman3-hyperkitty

[Barry Warsaw]

On Mar 25, 2016, at 01:02 PM, Paul Wise wrote:

>Does HyperKitty depend on mailman3 or just enhance it by providing an
>archive web interface?

Although greatly enhanced by it, Mailman 3 (core) doesn't require HyperKitty.
HK isn't currently useful on its own though.

Cheers,
-Barry


pgp3ExyuUZIGA.pgp
Description: OpenPGP digital signature

Re: Use Python3 where possible

[Barry Warsaw]

Probably no surprise, but I agree with everything Matthias said.

On Mar 15, 2016, at 11:20 AM, Matthias Klose wrote:

>I would like to come up with a recommendation that if a python module ships
>scripts, Python3 is used for these scripts, and the Python2 version of these
>scripts should be dropped (and python -m ...) should be used instead.  An
>alternative might be to use separate names for the scripts (e.g. ending with
>2, or like in pillow one set without a suffix (for Python3), and one set with
>a .py suffix (Python2)).  The most conforming name for the scripts should
>always use Python3.

I *really* dislike the -3 suffix on some scripts, e.g. the especially
egregious nosetests2-3.  I wouldn't want to adopt a -2 suffix, so ultimately I
agree that the /usr/bin/foo should be shebanged with python3 and drop back to
`python -m foo` for anyone who needs Python 2.

If there are cases where that won't work, let's try to fix them, or deal with
them on a case-by-case basis.

>Having a lintian warning that a package still uses Python2 instead of Python3
>might help as well, however maybe it should start as an "information" instead
>of a warning.

+1

Cheers,
-Barry

Re: Is pristine-tar failing just for me?

[Barry Warsaw]

On Mar 09, 2016, at 11:21 AM, Nikolaus Rath wrote:

>Whenever I use pristine-tar, I'm getting the following warning:
>
>| warning: pristine-gz cannot reproduce build of [whatever].orig.tar.gz; 
>storing 85% size diff in delta
>| (Please consider filing a bug report so the delta size can be improved.)
>
>I've reported this as a bug, but since pristine-tar is unmaintained I
>don't except any quick fix.
>
>However, I am wondering: am I the only one who sees this, or do other
>people here have the same issue?
>
>The most recent example is the python-llfuse tarball. It is generated by
>uscan after filtering out non-DFSG files.

I think saw it the other day on python-virtualenv, which also filters out
non-DFSG files.  I don't remember seeing it on python-colorama or python-pip
(which doesn't mean it didn't happen ;).  It was just a warning and didn't
seem to affect anything so I ignored it.  Maybe I shouldn't have.

Cheers,
-Barry

Re: Updating python-django-tagging

[Barry Warsaw]

On Mar 07, 2016, at 10:24 AM, Brian May wrote:

>I really like the work flow of git-dpm. Think it is much better then
>gbp-pq, which IIRC stores and distributes the patch files in git.
>
>Shame it isn't better maintained. As a result, seems it is very easy to
>get it in a confused state (e.g. by not noticing the local upstream
>branch needs a fast forward) and doesn't handle merging of branches.

Just to be clear, I agree on all your points.  When git-dpm works, it works
great and you almost don't even have to think about.  I like tools that don't
make me think. :)

Cheers,
-Barry


pgpfNJKylJROt.pgp
Description: OpenPGP digital signature

Re: Updating python-django-tagging

[Barry Warsaw]

On Mar 06, 2016, at 01:12 PM, Christopher Baines wrote:

>However, git-dpm keeps complaining in different ways when I try to do
>this (and I have tried a few different ways). I think the source of the
>issues I am having could be that the debian/.git-dpm file is out of sync
>with the rest of the repository (it seems to still refer to the 0.3.1
>version, whereas the current version is 0.4).
>
>What is the best approach here if I am trying to update the package? I
>can try and fix the debian/.git-dpm file (manually or using some tool?),
>or I could just avoid using git-dpm (but I do need to do some patch
>changes)?

I don't really have any good suggestions.  I've had some weird problems with
git-dpm occasionally and no real solutions.  For example, it kept wanting to
turn a certain git commit in the patched branch into a double quilt patch, and
nothing I did seemed to work (I ended up giving up).

In some of my non-team packages I've been trying to use gbp-pq more.  It seems
not terrible, and at least it's maintained.  Is it time to consider switching?
Perhaps just on a trial basis?  When git-dpm works, it does work well.

Cheers,
-Barry


pgpNZJLDfuqc9.pgp
Description: OpenPGP digital signature

Re: Using 'pyvenv' or 'python3 -m venv' on unstable

[Barry Warsaw]

On Mar 02, 2016, at 11:40 AM, Florent Rougon wrote:

>Sorry for linking to my own bug report, but am I the only one affected
>by bug #815864?

Since the fix for this is in python3.5, I've already provided a patch to
Matthias.  I wanted to get his feedback about one particular point of the
patch, but in the meantime I will attach the patch to that bug and reassign
it.

Cheers,
-Barry

Re: nose2 reverse dependancies

[Barry Warsaw]

On Mar 01, 2016, at 11:02 PM, Mattia Rizzolo wrote:

>FYI, I don't know of a nice way to build a dependency graph, like sometimes I
>see somewhere, with graphiz

That's a tool I'd sometimes love to have too.

Cheers,
-Barry


pgpSD9s_Vutxe.pgp
Description: OpenPGP digital signature

Re: Autopkgtest smoke test for Python libraries

[Barry Warsaw]

On Mar 01, 2016, at 06:10 PM, Martin Pitt wrote:

>You could use it for that purpose indeed, but that's not really what
>it is intended for. The idea is that the generic tests apply to all
>packages of a particular type, so you can change them centrally
>instead of having to modify and upload hundreds of packages.
>
>There is one more thing, though: The test machinery needs to be able
>to discover that a package has an autodep8 test (without having the
>source package already available, as getting and checking that is very
>expensive). So ideally all source packages which want to use that will
>get a "Testsuite: autopkgtest-pkg-python" header in debian/control,
>like for example libnet-oauth-perl. However, until these get uploaded
>we can add some custom code to debci to recognize those packages based
>on the name and dependencies (that's what we did for bootstrapping the
>perl and ruby autodep8 tests).

Ah cool, this is what I didn't understand from reading the README.  I was sort
of looking for the equivalent of README.package-tests.rst, i.e. what do I have
to do to my packages to opt-in to autodep8?  Or on the flip side, what do we
need to do globally to just enable it for all packages (where "all" == 80/20
of course :).

>As I said, running upstream tests works surprisingly well for
>Perl/Ruby, we had about 80% success rate. But they are a bit more
>rigid in structure apparently.

One of the problems is the wide variety of ways for invoking a package's test
suite in Python.  I've been pushing at those edges for a long time.  Ideally I
think the packaging metadata specs should define that, and it's on their
radar, just not any time soon.  My personal favorite is to look for a tox.ini
file and then just invoke `tox` but there are a few upstream features I still
want to make that really work nicely.  (Still, I encourage all upstream Python
authors to look closely at tox, it pretty much rules. :)

pybuild has a bunch of heuristics for trying to figure out how to invoke a
package's test suite, so that could be code/ideas that we could steal or
share.

>But indeed, ensuring that your module still imports already says a
>lot. New dependencies can still break your module in subtle ways, but
>at least things like new/removed Python versions, linker errors, wrong
>paths etc. are spotted.

Yep.  I think that's clearly the first step.

Cheers,
-Barry


pgpa5fEUGQypS.pgp
Description: OpenPGP digital signature

Re: Autopkgtest smoke test for Python libraries

[Barry Warsaw]

On Mar 01, 2016, at 09:05 AM, Martin Pitt wrote:

>Very nice! There's precedent for Perl, Ruby and DKMS packages which
>have a fairly standard way to run the upstream test suite. For Python
>there are some conventions, like "./setup.py test" or running
>nosetests, maybe it's worth experimenting with those after getting the
>initial "import works" smoketest in place.
>
>This situation (testing many related packages in a similar way) is
>what autodep8 is for. When run in a Debian source package tree, it
>checks if it knows something about this type of package, and if so,
>synthesizes a debian/tests/control. Have a look at

I didn't know about autodep8, thanks for the link!

I gather that it's supposed to be used by the maintainer in a source package
($vcs repo) to jumpstart a debian/tests directory?  But after that, would you
use autodep8 again?

I'll try to clear some time to look at adding Python support; shouldn't be too
difficult I think.

I'm not sure how useful things like "setup.py test" are for DEP-8.  In general
I think a package's test suite are best exercised during package build[*].
For the majority of Python packages, I view DEP-8 as ensuring that the user
experience is what they expect, and that means at a minimum that the packages
import and Do Something Simple.  A generic proof of non-broken import can mean
printing the package's version or the module path, and that's something that
autopkg8 could templatize.  Anything more than that requires some more
detailed knowledge about what the package does.

I've tended to taking a pretty dumb approach since I like using Test-Command,
but Ben's smoketests in lockfile are certainly more thorough.  For an autodep8
plugin, I'd probably fall somewhere in between.

>Not sure what you mean with "shortcoming", you can put arbitrary shell
>code into Test-Command:. However, the spirit is that these should be
>very short, usually one line. I suggest that they should mostly just
>call something like "/usr/share/python-something/run-autopkgtest $py $pkgname"
>and you maintain the actual test code centrally in e. g.
>python-defaults.

Oh, that's a good idea.  Then when we come up with better generic tests, we
only need to update one package, not everything.

>That seems like an excellent starting point indeed. Making sure that
>all modules are importable will catch missing builds for particular
>Python versions, regressions in newer Python versions, missing
>dependencies, packaging errors, etc.

+1

>As said above, maybe in the future we want to try and run upstream test
>suites, but this will need a lot more heuristics (starting with additional
>test deps).

I'm skeptical about running the upstream test suite in DEP-8, but baby steps!

Cheers,
-Barry

[*] Although it's true for a small handful of packages, it's not feasible to
run their test suite during package build for various reasons.  In those cases
I've sometimes added DEP-8 tests for running the test suite, but I've also
found those to be rather fragile, so I'm on the fence about gating
e.g. package promotion based on their passing their own test suite.


pgpFtylcbWZXz.pgp
Description: OpenPGP digital signature

Re: Python Policy: Things to consider for Stretch

[Barry Warsaw]

On Feb 16, 2016, at 11:54 AM, Paul Wise wrote:

>I always thought it strange to put site- in /usr/local since
>/usr/local already implies site/system-wide packages. Same for dist-
>since /usr already implies distribution packages.

For as long as I can remember, a from-source 'configure && make && make
install' always put Python in /usr/local by default.  I think it was pretty
much the defacto standard for non-vendor supplied software[*] back in the days
of IRIX, SunOS, and other early *nix OSes that Python was developed on.  So it
was therefore completely natural that you'd end up with a site-packages in
/usr/local.  It was only later that the /usr/local site-packages directory
ended up on a /usr pathed distro-provided Python, which of course led to the
previously discussed dist-packages, the location of which completely mirrors
site-packages.

>I find it weird that site- gets used in ~/ since they are clearly user
>packages not site/system-wide packages.

It's all just a big ball of cruft-on-cruft with backward compatibility
preventing most cullings.  Somewhere out there, the entire world financial
system probably still critically depends on a tiny bit of Python 1.3 that
nobody has anything but the .pyc files for any more. ;)

Cheers,
-Barry

[*] I can't even call it Free Software or Open Source because it predates
those terms.  I mean, I started out sharing split shar files on Usenet with my
UUCP address.  ObGOML.

Re: Python Policy: Things to consider for Stretch

[Barry Warsaw]

On Feb 15, 2016, at 07:42 PM, Barry Warsaw wrote:

>I don't remember exactly why we called it 'site-packages', but I believe it
>was an evolution from the earlier ni.py module, which was where dotted module
>paths first showed up in Python.

And which had a 'site-python' directory, which was kept at least for Python
1.5 also.

Cheers,
-Barry

Re: [Python-modules-team] Git-dpm help for tweepy

[Barry Warsaw]

On Feb 11, 2016, at 09:09 AM, Ross Gammon wrote:

>I was working last night on fixing the RC bug for tweepy, but got myself
>in a muddle with git-dpm.

Yep, git-dpm will do this some times. :/

>Tweepy is currently 3.4.0 & 3.5.0 is a new available release. I cloned,
>fixed the bug (just disabling the unit test), confirmed it was fixed and
>then looked at packaging 3.5.0 as a team upload. But git-dpm seemed to
>be unaware of 3.4.0 (claiming that 3.1.0 was the right tarball), that
>there were non-debian changes in the debian branch and wouldn't let me
>import 3.5.0. So I gave up on 3.5.0, and started trying to work out what
>was wrong with 3.4.0 in git-dpm.

Occasionally I find it necessary to blow away a clone and start over, but even
then, git-dpm sometimes does mysterious things.  (This is one of several
reasons I'm beginning to sour on it, though it works just fine in the majority
of cases.)

One thing I've found is that import-new-upstream will sometimes not record the
pristine-tar, even if you use --ptc.  It seems like this can happen if you
also --rebase and are required to resolve some conflicts.  After finally
git-dpm u-p'ing the pristine-tar will not have been recorded.  Though you have
to remember to do it, it's easy enough to pristine-tar commit the new
orig.tar.gz after the fact.

The main thing is that when you update to a new upstream, be sure to git-dpm
status (and/or prepare) to be sure everything's okay before you start to work
on further changes on top of the new upstream.

Another gotcha is that if you `git clone` the repo, you won't actually have a
pristine-tar or upstream tracking branch, and this will break git-dpm.  Much
better to use `gbp clone` which does the clone and then assures you have the
local tracking branches.  If you do git clone, just be sure to checkout the
pristine-tar branch before you start working on things (and upstream too,
though this matters less in practice).

(Aside: I'm not sure I'd say that disabling a unit test counts as a "fix" ;)

I don't know if any of the above is relevant for your case.  I've seen git-dpm
do weird things like take a single commit in the patches branch and turn it
into a quilt patch where the diff is present *twice*.  I think this was for a
simple fix in pyparsing (trying to change distutils.setup() into
setuptools.setup() in the setup.py), and though I tried a ton of different
things, I was never able to get git-dpm to generate a correct quilt patch
file.

>At this point, I somehow merged pristine-tar into master :-(

You'll be way better off restarting from a fresh clone than trying to fix your
busted one.  That way leads to madness. ;)

Cheers,
-Barry


pgptoBQfNWEsm.pgp
Description: OpenPGP digital signature