Re: setup.py sdist permissions

[Robert Collins]

Yah, packaging permissions are an installation problem, and setup.py
is (no longer) intended for installation.

-Rob

On 5 April 2018 at 10:25, Brian May  wrote:
> Robert Collins  writes:
>
>> Replied on the bug :)
>
> Thanks.
>
> He responded, he is not using pip, but creating a "Void package" from
> source.
>
> I am inclined respond, as he is not using pip, he needs to ensure the
> permissions are correct.
> --
> Brian May 

Re: setup.py sdist permissions

[Robert Collins]

Replied on the bug :)

On 4 April 2018 at 20:04, Brian May  wrote:
> Yaroslav Halchenko  writes:
>
>> just anecdotal support: my umask is 077 as well, have been doing uploads
>> to pypi for a while, never had report from the users about any problem.
>> The reasons could be either it indeed doesn't matter or nobody uses my
>> projects ;-)
>
> Same here.
>
> I just got a bug report :-(
>
> https://github.com/sshuttle/sshuttle/issues/217
>
> Not really sure what circumstances this causes problems, or what I
> should do about it.
> --
> Brian May 
>

Re: Thread on flit...

[Robert Collins]

On 18 February 2018 at 00:03, Paul Wise  wrote:
> On Sat, Feb 17, 2018 at 4:41 PM, Julien Puydt wrote:
>
>> It is intended for trivial packaging... so perhaps dh_python could
>> detect its use and do something smarter.
>
> Ideally, debhelper should DTRT no matter what build system is in use.

Agreed. Its worth noting though that we've finally introduced a build
system abstraction into Python, so dh_python should instead learn to
drive PEP-517 build systems; that will support flit and others over
time.

It does require a python thunk, sadly - see PEP 516 which was the
alternative :) - but its not too deep an issue to write that and slap
a CLI on it for dh_python to use.

-Rob

Re: New, updated pip coming

[Robert Collins]

ch for
> pip, which involves many curse words hurled at git-dpm (and that's a post for
> another day!).  I also have to update my local branch to 8.0.2, but I'm
> assuming there won't be any new surprises there.
>
> Eventually, I'll be using the same techique for virtualenv and pyvenv, but not
> right away.
>
> Huge thanks also go to Donald Stufft, who always is so helpful with various
> technical details, and works really hard to understand our needs and balance
> them with upstream's.
>
> Finally, we need to update Python policy to describe the new wheel policy.
> Fortunately, this is much better because it removes constraints on other
> packages, and reduces the number of -whl binary packages in the distro.  Yes,
> I will update those packages that already ship -whls, or file bugs, but only
> once I'm sure everything is working correctly.
>
> Below is the policy diff, which I'll commit to the python-defaults bzr.
>
> As always, once everything's uploaded, please do let me know via this list or
> BTS, if you notice any problems with pip.
>
> Cheers,
> -Barry
>
> === modified file 'debian/python-policy.sgml'
> --- debian/python-policy.sgml   2016-01-24 06:01:19 +
> +++ debian/python-policy.sgml   2016-01-29 16:49:59 +
> @@ -458,13 +458,12 @@
>  
>http://legacy.python.org/dev/peps/pep-0427/";
> name="PEP 427">
> -  defines a built-package format called "wheels", which is a zip
> -  format archive containing Python code and a "dist-info" metadata
> +  defines a built-package format called "wheels", a zip
> +  archive containing Python code and a "dist-info" metadata
>directory, in a single file named with the .whl suffix.  As zip
>files, wheels containing pure-Python can be put on sys.path and
>modules in the wheel can be imported directly by Python's "import"
> -  statement. (Importing extension modules from wheels is not yet
> -  supported as of Python 3.4.)
> +  statement.
>  
>Except as described below, packages must not build or provide
>wheels.  They are redundant to the established way of providing
> @@ -475,18 +474,20 @@
>  
>A very limited set of wheel packages are available in the archive,
>but these support the narrow purpose of enabling
> -  the pip tool, in a Debian policy compliant way.  The
> -  set of packages providing wheels for this purpose are (by source
> -  package name): chardet, distlib, html5lib, python-colorama,
> -  python-pip, python-setuptools, python-urllib3, requests, and six.
> +  the pip, virtualenv,
> +  and pyvenv tools in a Debian policy compliant way.
> +  These packages build their own dependent wheels through the use of
> +  the dirtbike "rewheeling" tool, which takes installed
> +  Debian packages and turns them back into wheels.  Only universal
> +  wheels (i.e. pure-Python, Python 2 and 3 compatible packages) are
> +  supported.  Since only the programs that require wheels need build
> +  them, only they may provide -whl packages,
> +  e.g. python-pip-whl.
>  
> -  Wheel packages supporting pyvenv and pip
> -  are named with the python- prefix, and the 
> -whl
> -  suffix, e.g. python-chardet-whl.  When these
> -  binary packages are installed, their .whl files must be placed in
> -  the /usr/share/python-wheels directory.  Such wheels must be built
> -  with the --universal flag so as to generate wheels
> -  compatible with both Python 2 and Python 3.
> +  When these binary packages are installed, .whl files must be placed
> +  in the /usr/share/python-wheels directory.  The location inside a
> +  virtual environment will be rooted in the virtual environment,
> +  instead of in /usr.
>  
>
>
>



-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

Re: Bug#802839: django-celery: python 3 tests not invoked and break

[Robert Collins]

I'd probably shut that warning up using the warnings module API rather
than weaking the test more broadly. Also - track down and file a bug
on the leak source.

Re: Python < 3.5 tests

[Robert Collins]

On 9 October 2015 at 10:40, Brian May  wrote:
> On Fri, 9 Oct 2015 at 08:16 Robert Collins 
> wrote:
>>
>> But - ajax_select/__init__.py is going to be imported, and thats whats
>> erroring. I don't think that this is a 3.5 unit testing change - I
>> think its an error importing some bit of django.
>
>
> Except there is no errors under Python2.7. If Python2.7 or Python3.4 loaded
> ajax_select/__init__.py I would expect to see exactly the same Django error.

Right - and if there are any tests they will need to load that :)

The reason it's being discovered is likely due to the pattern bugfix
(also in 3.5) - previously discover couldn't handle directories with
names tht didn't match the file pattern - and that meant nested test
suites didn't load right.

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

Re: Python < 3.5 tests

[Robert Collins]

On 9 October 2015 at 00:43, Dmitry Shachnev  wrote:
> Hi all,
>
> On Thu, 8 Oct 2015 08:12:22 +1100, Brian May wrote:
> [...]
>> So it looks like it can't find the tests under Python 2.7 or Python 3.4,
>> so it use to work, however under Python 3.5 it now does find the tests
>> and they fail.
>>
>> This left me wondering:
>>
>> * Why does it not find the tests under Python 2.7, 3.4? What is
>>   different about Python 3.5 that it does?
>
> There was a change in unittest autoloader in Python 3.5. It now tries to
> import the package even if it has no tests. I do not know if it is an
> intended change, or a side-effect of fixing some bug ([1]?). Maybe Robert
> can enlighten us here.

Imagine I have a directory with __init__.py and test_foo.py, and a
subdir (sub) of that with __init__.py and test_bar.py.
Before that change, __init__ was loaded and test_foo was loaded. and
sub/__init__ was loaded, then if present the load_tests hook in
sub/__init__ is fire, and finally test_bar.py is loaded.

After the change the root __init__ is examined for a load_tests hook
as well. So the only time this specific patch would change behaviour
is if discover's start dir was a package (vs a namespace package aka
directory) and __init__ has a load_tests hook.

As discover is being run with a start of ., thats not the impact here.

But - ajax_select/__init__.py is going to be imported, and thats whats
erroring. I don't think that this is a 3.5 unit testing change - I
think its an error importing some bit of django.

The test suite output should show the actual import error exception,
whats been pasted here so far has just been the loader stage output,
but the loader returns a thunk that will show the exception that
occurred during loading and would help debugging.

> This has caused some my packages to FTBFS too, for example see bugs
> #800671 and #800139. So if it not an intended change, it should be fixed
> upstream.

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=800671 shows this:


"/tmp/buildd/python-changelog-0.3.4/.pybuild/pythonX.Y_3.5/build/changelog/__init__.py",
line 3, in 
  from .changelog import setup
File

"/tmp/buildd/python-changelog-0.3.4/.pybuild/pythonX.Y_3.5/build/changelog/changelog.py",
line 5, in 
  from sphinx.util.compat import Directive
  ImportError: No module named 'sphinx'

And pybuild has cd'd to the build dir there before running discover.
So - changelog's __init__ can't be imported - that should occur in all
versions of discover, when changelog is imported and scanned, because
importing anything from within it will need to initialise the package
first, which is basic Python semantics.

800139 has the same issue.

Any tests of these packages will need to import the package itself to
exercise it, so I don't see a CPython/discover bug here.

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

Re: Python < 3.5 tests

[Robert Collins]

On 8 October 2015 at 18:57, Brian May  wrote:
>
>
> On Thu, 8 Oct 2015 at 08:12 Brian May 
> wrote:
>>
>>I: pybuild base:170: cd /«PKGBUILDDIR»/.pybuild/pythonX.Y_3.5/build;
>>python3.5 -m unittest discover -v
>>ajax_select (unittest.loader._FailedTest) ... ERROR
>>
>
> On second thoughts, wonder what tests it is running. I can't see anything
> that looks like a test in the package... setup.py has nothing either.
>
> Cut and paste doesn't work properly with this stupid Google inbox webclient
> - otherwise I would add a list of the source files here, however this is the
> django-ajax-selects package in Debian.
>
> Only thing I can think of is maybe it is picking up the Django tests, which
> are meaningless here.

Presumably 
https://github.com/crucialfelix/django-ajax-selects/tree/develop/tests
would be the intent?

The tarball on PyPI is missing it - so I think its a broken
MANIFEST.in - it hasn't been updated to include the tests.

So the error thats being reported is that ajax_select can't be
imported - which means that ajax_select/__init__.py can't be imported.

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

Re: Python < 3.5 tests

[Robert Collins]

On 8 October 2015 at 12:05, Ben Finney  wrote:
> Robert Collins  writes:
>
>> On 8 October 2015 at 11:47, Ben Finney  wrote:
>> > If you have a code base that is intended to run unchanged on Python
>> > 2 and Python 3, and that code base imports ‘unittest2’, you need
>> > both the Python 2 and Python 3 version of that package.
>> >
>> > If your code base targets only Python 3, it should not be using
>> > ‘unittest2’ at all.
>>
>> Thats false. unittest2 is a rolling backport.
>
> I'm not sure how that disagrees with what I wrote.
>
> Are you saying ‘unittest2’ is useful on Python 3.3 (for example) because
> it has improvements from later Python 3 standard library ‘unittest’?

Yes, exactly. unittest2 is useful and relevant on Python 3.2/3.3/3.4
today. Once we get some fixes into 3.6, it will be useful and relevant
for 3.5 users.

> Or something else?

-Rob


-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

Re: Python < 3.5 tests

[Robert Collins]

On 8 October 2015 at 11:26, Brian May  wrote:
> On Thu, 8 Oct 2015 at 08:18 Robert Collins 
> wrote:
>>
>> Probably the discover improvements in 3.5 try using python -m
>> unittest2.discover
>
>
> Just trying that now.
>
> I see that there is a python3-unittest2 package - should I be using that one
> or the unittest built in Python 3.5?

let me check I understood the symptoms:
python3.5 - works
python2.x fails
python3.4 fails

So for that - use unittest2 consistently everywhere IMO.

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

Re: Python < 3.5 tests

[Robert Collins]

On 8 October 2015 at 11:47, Ben Finney  wrote:
> Brian May  writes:
>
>> I see that there is a python3-unittest2 package - should I be using that
>> one or the unittest built in Python 3.5?
>
> If you have a code base that is intended to run unchanged on Python 2
> and Python 3, and that code base imports ‘unittest2’, you need both the
> Python 2 and Python 3 version of that package.
>
> If your code base targets only Python 3, it should not be using
> ‘unittest2’ at all.

Thats false. unittest2 is a rolling backport. A true statement would
be 'if your codebase will only ever run on the latest release of
Python then unittest2 offers no value' for it.

-Rob


-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

Re: Python < 3.5 tests

[Robert Collins]

Probably the discover improvements in 3.5 try using python -m
unittest2.discover
On 8 Oct 2015 10:12, "Brian May"  wrote:

> Hello,
>
> When debugging #801208, I noticed the following output:
>
>dh_auto_test -O--buildsystem=pybuild
>I: pybuild base:170: cd /«PKGBUILDDIR»/.pybuild/pythonX.Y_2.7/build;
>python2.7 -m unittest discover -v
>
>--
>Ran 0 tests in 0.000s
>
>OK
>I: pybuild base:170: cd /«PKGBUILDDIR»/.pybuild/pythonX.Y_3.4/build;
>python3.4 -m unittest discover -v
>
>--
>Ran 0 tests in 0.000s
>
>OK
>I: pybuild base:170: cd /«PKGBUILDDIR»/.pybuild/pythonX.Y_3.5/build;
>python3.5 -m unittest discover -v
>ajax_select (unittest.loader._FailedTest) ... ERROR
>
> So it looks like it can't find the tests under Python 2.7 or Python 3.4,
> so it use to work, however under Python 3.5 it now does find the tests
> and they fail.
>
> This left me wondering:
>
> * Why does it not find the tests under Python 2.7, 3.4? What is
>   different about Python 3.5 that it does?
> * Why is Python 3.4 still relevant? Hasn't 3.5 replaced 3.4? Why is it
>   being tested by pybuild?
>
> Regards
>
>

Re: python-networkx_1.10-1_amd64.changes ACCEPTED into experimental

[Robert Collins]

On 6 October 2015 at 01:51, Thomas Goirand  wrote:
> On 10/05/2015 09:37 AM, Michael Fladischer wrote:
>> On 2015-09-30 10:53, Thomas Goirand wrote:
>>> * The maintainer of mock uploaded version 1.3 to Sid, which created RC
>>> bugs (FTBFS) on maybe more than 20 packages currently in Sid, even
>>> though upstream (Robert Collins) is employed by HP and knew OpenStack
>>> Kilo (currently in Sid) would break with his new changes.
>>
>> So much for the finger-pointing. Just to set things straight: Uploading
>> mock-1.3 to unstable was the right thing to do as it uncovered all the
>> broken/misused assertions in the tests that silently passed before but
>> never actually checked anything related to their test-case.
>>
>> I see no use in tests that unconditionally succeed every time they are run.
>
> But this created lots of RC bugs which were not really needed, as the
> affected packages worked otherwise perfectly (I did run Tempest tests to
> functional-validate these packages). Uploading mock to Experimental
> until OpenStack Liberty could be uploaded to Sid was the correct thing
> to do. Never the less, I had (and have) "fixed" many packages that were
> affected (mostly by disabling some tests), but IMO, this didn't improve
> at all their quality.
>
> Site note: at this point, since Liberty will be released in 10 days, I
> wont fix the remaining FTBFS (there's 2 currently in Sid because of mock
> 1.3): I prefer focusing on the next release rather than fixing what's in
> fact already working. Uploading all of Liberty to overwrite Kilo in Sid
> will fix it all anyway.
>
> It is also to be noted that mock is maintained by upstream OpenStack
> people (ie: Robert Collins), and therefore, should be released in Debian
> at the same time as other testing tools and the rest of OpenStack:
> testools, testscenarios, subunit, testrepository, and many more. So in
> the future, I'd advise to follow upstream release schedule. I would
> encourage, if you don't mind, to put mock into the PKG OpenStack team,
> because that's where it belongs. If we don't do that, and without being
> careful, then breakage is to be expected.

Hang on a second here. I, like many others, wear multiple hats.

The things you listed that I help maintain - mock, testtools, etc -
are *not* OpenStack specific. They existed before OpenStack, and
likely will exist after. They have other users, particularly mock
which is very widely used.

Its entirely reasonable to say that known reverse dependencies should
be considered when upgrading packages, but that is in no way OpenStack
specific - and the release schedule of all of the things you listed is
entirely independent of OpenStack.

Its one of the defects of the single-version design of the archive
that we have this uncontrolled use of new releases of software thats
put into it, and - well, thats another discussion. We need to live
with it though.

So I'm +1 on "Check reverse deps aren't significantly broken before
uploading to unstable" as a general principle, not as an OpenStack
specific thing.

-Rob


-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

Re: Application libraries private, Distutils metadata available for console scripts and introspection

[Robert Collins]

On 2 September 2015 at 21:19, Karsten Hilbert  wrote:
> On Wed, Sep 02, 2015 at 09:06:01PM +1200, Robert Collins wrote:
>
>> > Ben Finney  writes:
>>
>> >   According to Robert's earlier message, that means the Distutils
>> >   metadata file needs to be not in the application's private directory,
>> >   but in a directory on the Python module search path. That seems odd to
>> >   me, since this is not amodule import being done.
>>
>> So this is perhaps the disconnect: I did not say the metadata should
>> move: it should be in the private directory - it has to be adjacent to
>> the packages/modules its describing, since if it is present but the
>> package/module is not that is at the very best confusing for tools
>> like pip.
>>
>> That private directory must be on the python search path
>
> Maybe I'm a bit dense but isn't that what OP very much
> DOESN'T want to _happen_ ?

They don't want to have to alter the PYTHONPATH variable as far as I
can tell, but that is only one of a number of mechanisms to get a
directory on the search path.

> Are you, by implication, saying that that's not _possible_ ?

I suspect so.

>> when running the application (or the modules can't be imported).
>
> That reason doesn't hold -- the application can be written to
> (or told to) "know" where to look and manipulate sys.path
> accordingly (regardless of whether that's considered a good
> idea or not).

If sys.path is altered by the application to put the private directory
on the search path, then my assertion (that it needs to be on the
path) would be satisfied, and I'd expect it to work.

-Rob


-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

Re: Application libraries private, Distutils metadata available for console scripts and introspection

[Robert Collins]

On 2 September 2015 at 22:45, Ben Finney  wrote:
> Robert Collins  writes:
>
>> That private directory must be on the python search path when running
>> the application (or the modules can't be imported).
>
> Not at all. The application's private modules are imported only via
> relative imports.

Relative imports still consult the various python module search
mechanisms, which is the top level path, + per package paths (but to
import the highest of those packages you have to import the first
member from the root... which is on the top level path).

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

Re: Application libraries private, Distutils metadata available for console scripts and introspection

[Robert Collins]

On 2 September 2015 at 20:35, Ben Finney  wrote:
> Ben Finney  writes:

>   According to Robert's earlier message, that means the Distutils
>   metadata file needs to be not in the application's private directory,
>   but in a directory on the Python module search path. That seems odd to
>   me, since this is not amodule import being done.

So this is perhaps the disconnect: I did not say the metadata should
move: it should be in the private directory - it has to be adjacent to
the packages/modules its describing, since if it is present but the
package/module is not that is at the very best confusing for tools
like pip.

That private directory must be on the python search path when running
the application (or the modules can't be imported).

-Rob


-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

Re: Application libraries private, Distutils metadata available for console scripts and introspection

[Robert Collins]

On 1 September 2015 at 17:35, Ben Finney  wrote:
> Robert Collins  writes:
>
>> PKG resources should find it anywhere in the python path.
>
> That's exactly the point, though. The packages are private to this
> application, and should not be available for import by other programs.

Right - I did not suggest they should be.

> On the other hand, the Distutils metadata (the ‘foo-1.2.3.egg-info’
> directory) should be available for the program.

How is this the other hand? The Distutils data should be available to
and only to the programs that are allowed to access the packages.

> What I'd like is for the distribution to be installed such that:
>
> * its modules are private (‘--install-lib=/usr/share/FooApp/’), and
>
> * its distribution metadata is discoverable (‘--install-metadata=???’
>   except there doesn't seem to be such an option).

The distribution metadata is located via the python path; whatever
mechanism is making FooApp accessible to the program *should* also
make the metadata accessible, as long as:

>> I'd the path correct within the all processes?
>
> I don't know. I would not expect to need to alter how the program is run
> in order for it to find its own distribution metadata.

That is one of two options for what is happening to make packages in
/usr/share/FooApp/ accessible. Confirming it (or denying it) will let
us debug the issue.

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

Re: Application libraries private, Distutils metadata available for console scripts and introspection

[Robert Collins]

PKG resources should find it anywhere in the python path. I'd the path
correct within the all processes?
On 1 Sep 2015 2:53 pm, "Ben Finney"  wrote:

> Howdy all,
>
> How can I specify to Pybuild that an application should have its modules
> all in a private namespace, but have the Distutils metadata also
> available to `pkg_resources` queries?
>
> I install its libraries to an application-specific space with
> `PYBUILD_INSTALL_ARGS = --install-lib=/usr/share/foo/`::
>
> $ find /usr/share/FooApp/
> /usr/share/FooApp/
> /usr/share/FooApp/lorem
> /usr/share/FooApp/lorem/__init__.py
> /usr/share/FooApp/lorem/dolor.py
> /usr/share/FooApp/lorem/sit.py
> /usr/share/FooApp/lorem/amet.py
> /usr/share/FooApp/FooApp-1.2.3.egg-info
> /usr/share/FooApp/FooApp-1.2.3.egg-info/PKG-INFO
> /usr/share/FooApp/FooApp-1.2.3.egg-info/requires.txt
> /usr/share/FooApp/FooApp-1.2.3.egg-info/entry_points.txt
> /usr/share/FooApp/FooApp-1.2.3.egg-info/not-zip-safe
> /usr/share/FooApp/FooApp-1.2.3.egg-info/dependency_links.txt
> /usr/share/FooApp/FooApp-1.2.3.egg-info/top_level.txt
> /usr/share/FooApp/ipsum
> /usr/share/FooApp/ipsum/__init__.py
> /usr/share/FooApp/ipsum/consecteur.py
> /usr/share/FooApp/ipsum/adipiscing.py
> /usr/share/FooApp/ipsum/elit.py
>
> (The actual code base is a fork of ‘dput’ to modernise its packaging
> https://notabug.org/bignose/dput/> and Python idioms.)
>
> The application's Python package is managed like other operating system
> files, but is not in the general Python namespace for other programs to
> import. Good.
>
> The application has “console scripts” defined in the Distutils
> `entry_points` mapping:
>
> $ cat ./setup.py
> […]
> entry_points={
> 'console_scripts': [
> "foo=FooApp.foo:main",
> ],
> },
> […]
>
> which installs command-line programs at `/usr/bin/foo`, for example.
> Good.
>
> The distribution metadata (the `FooApp-1.2.3.egg-info` directory) is
> also installed to the private directory though, which makes it invisible
> to `pkg_resources`:
>
> $ /usr/bin/foo
> Traceback (most recent call last):
>   File "/usr/bin/foo", line 5, in 
> from pkg_resources import load_entry_point
>   File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py",
> line 3084, in 
> @_call_aside
>   File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py",
> line 3070, in _call_aside
> f(*args, **kwargs)
>   File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py",
> line 3097, in _initialize_master_working_set
> working_set = WorkingSet._build_master()
>   File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py",
> line 651, in _build_master
> ws.require(__requires__)
>   File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py",
> line 952, in require
> needed = self.resolve(parse_requirements(requirements))
>   File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py",
> line 839, in resolve
> raise DistributionNotFound(req, requirers)
> pkg_resources.DistributionNotFound: The FooApp==1.2.3' distribution
> was not found and is required by the application
>
> Other queries to `pkg_resources` for this application's distribution,
> for example to get the distribution version or homepage URL, will also
> fail.
>
> How can I have Pybuild specify to Distutils that the application's
> library modules should be installed away from the public namespace and
> not available for general import, but also that the `pkg_resources`
> functionality should find the distribution metadata where expected?
>
> --
>  \ “When I was a kid I used to pray every night for a new bicycle. |
>   `\Then I realised that the Lord doesn't work that way so I stole |
> _o__)   one and asked Him to forgive me.” —Emo Philips |
> Ben Finney
>
>

Re: Removing some python3-* packages

[Robert Collins]

On 25 August 2015 at 11:49, Thomas Kluyver  wrote:
> On Mon, Aug 24, 2015, at 04:30 PM, Robert Collins wrote:
>> c) write convoluted tricky code to workaround the bugs and differing
>> behaviour on 3.4 vs 3.5.
>
> I use unittest.mock from Python 3.4 on several packages, and it has not
> required convoluted code. I would be very surprised if that code breaks
> when run under Python 3.5.

If it works with mock 1.3 (in stretch) and with unittest.mock from
3.4, then you're fine. Very much depends on the size and variety of
your codebase, whether you will have an issue or not.

But it doesn't take 'works for me' to invalidate the point: it
requires 'works for *everyone*' - and while you haven't experienced
the faults yourself that in no way invalidates the experience of folk
that have experienced it.

-Rob


-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

Re: Removing some python3-* packages

[Robert Collins]

On 25 August 2015 at 11:23, Barry Warsaw  wrote:
> On Aug 25, 2015, at 10:45 AM, Robert Collins wrote:
>
>>Lets take Ironic. While it supports Python 2.7+ and 3.4+ it will
>>depend on 'mock' for unit testing.
>
> That's not unreasonable, and different upstreams will have different policies,
> but if it was *my* upstream package, I'd probably do a conditional import so
> that unittest.mock were used if available, falling back to mock if not.

Except that that will break: mock in 3.4 vs 3.5 are different. Lots of
bugs fixed in 3.5. Upstreams have to choose:
a) use 'mock' everywhere which has all the fixes.
b) use 'mock' everywhere except *3.5 and only 3.5*.
c) write convoluted tricky code to workaround the bugs and differing
behaviour on 3.4 vs 3.5.

> If it's easy to patch upstream to do this, then it seems an acceptable
> approach to take.  I wouldn't say it's *required* (which is why I'm not sure a
> blanket policy is appropriate for us).
>
>> - patch Ironic to use unittest.mock on Python 3.5
>
> That would be my first preference, and I'd also submit the patch to upstream.
>
>> - patch the stdlib to make 'mock' be an alias to unittest.mock
>
> Nope.
>
>> - include 'python3-mock' as a binary package
>
> That's not unreasonable for Debian either.
>
> I'll note another thing that perhaps makes the enum34 case different.  It
> exports the package under the same name in both the standalone version and in
> the stdlib, so there *really* is no difference between the two.  That's not
> the case with unittest.mock/mock.

It certainly makes it a lot harder to tell them apart :). And yeah,
very hard to consume it.

-Rob


-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

Re: Removing some python3-* packages

[Robert Collins]

On 25 August 2015 at 10:37, Barry Warsaw  wrote:
> On Aug 25, 2015, at 10:03 AM, Robert Collins wrote:
>
>>On 25 August 2015 at 09:57, Barry Warsaw  wrote:
>>...
>>>  By all means, if there isn't any
>>> significant difference between a standalone package and what's available in
>>> the current supported Python 3 version, let's not ship unnecessary binary
>>> packages.
>>
>>Even at the cost of having to patch upstream projects?
>
> Sorry, I'm not sure what the question means.

Lets take Ironic. While it supports Python 2.7+ and 3.4+ it will
depend on 'mock' for unit testing.

If Debian has only Python 2.7 and 3.5 and wants to build Ironic for
only Python 3.5 - which is a reasonable thing, then 'mock' is needed
(because Ironic depends on 'mock').

We could:
 - patch Ironic to use unittest.mock on Python 3.5
 - patch the stdlib to make 'mock' be an alias to unittest.mock
 - include 'python3-mock' as a binary package
 - not run the Ironic unit tests.

Perhaps to you 'thing we package X uses library Y' implies 'binary
package X is uncessary' - but the thing is that 'binary package X' may
not be a significant difference vs the Python version in Debian at the
time - even if there are significant differences vs older still
upstream-supported Python versions - which is what the developer
ecosystem will be referencing.

-Rob


-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

Re: Removing some python3-* packages

[Robert Collins]

On 25 August 2015 at 09:57, Barry Warsaw  wrote:
...
>  By all means, if there isn't any
> significant difference between a standalone package and what's available in
> the current supported Python 3 version, let's not ship unnecessary binary
> packages.

Even at the cost of having to patch upstream projects?

-Rob


-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

Re: Removing some python3-* packages

[Robert Collins]

On 3 July 2015 at 08:29, Scott Kitterman  wrote:

> I think dropping these duplicates is the only thing that makes sense. For
> reference, I dropped python3-ipaddr once python3.2 was gone (because 3.3 has
> ipaddress, which does the same thing).

Where its a dupe sure.

unittest2, traceback2, linecache2, mock are not duplicates of the
functionality in 3.4 - they are backports of things in 3.5 (to all
pythons). And they will shortly have more than 3.5 itself has in it,
as they are rolling backports: what lands in 3.6 will go into them. So
I don't think removing them makes sense.

I don't have a view on other packages.

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud


-- 
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/CAJ3HoZ0a9rSSAc-zv+7qhP+0n-wMGQ=hrm__togtuesopuk...@mail.gmail.com

Re: Removing some python3-* packages

[Robert Collins]

On 3 July 2015 at 15:05, Ian Cordasco  wrote:
> On Thu, Jul 2, 2015 at 10:03 PM, Robert Collins
>  wrote:
>> On 3 July 2015 at 11:44, Ian Cordasco  wrote:
>>> On Thu, Jul 2, 2015 at 6:40 PM, Ben Finney  
>>> wrote:
>>>> Barry Warsaw  writes:
>>>>
>>>>> […] there's actually no reason to have a Python 3 version of enum in
>>>>> any version >= Python 3.4. […]
>>>>
>>>> Ian Cordasco  writes:
>>>>
>>>>> Probably a silly question, but are other libraries like unittest2 also
>>>>> being packaged for python3? Another library is mock. That was included
>>>>> in the stdlib in 3.3.
>>>>
>>>> One consideration is: What code is written to be Python 2 and Python 3
>>>> compatible from the same code base, which achieves this by importing a
>>>> module which is backported to Python 2?
>>>>
>>>> In some of my code I'm doing ‘import unit2’ to have features from that
>>>> library available in Python 2 code.
>>>>
>>>> Since those features are all in Python 3's standard library, the case
>>>> could be made that ‘python3-unit2’ is pointless; but against that is the
>>>> fact that a Python 3 ‘unit2’ package means that ‘import unit2’ will work
>>>> the same on both runtime versions.
>>>>
>>>> So I'd argue that ‘python3-mock’ and the like do have a place in Debian:
>>>> they make it easier to follow the recommended strategy of having a code
>>>> base run unchanged on Python2 and Python 3.
>>>
>>> Just to be clear, trying to use mock on 3.4 is thoroughly broken. If
>>> you can install python3-mock right now and use it in 3.4, then y'all
>>> must be carrying patches to make it work. The last 3.x version that
>>> mock works on is 3.3.
>>>
>>> That clearly doesn't have a place on a debian with python 3.5 as the
>>> version of python 3 included, unless you're planning on supporting
>>> packages for python 3.3 as well that will generate a numerous amount
>>> of bugs for you.
>>
>> See my prior mail. I will be backporting all the changes in mock from
>> the stdlib to the mock standalone lib in the near future.
>>
>> I have upload acls from Michael Foord for PyPI and I'm not afraid to
>> use them to fix it up - if you wanted to prep a narrow patch to fix
>> 3.4 then please do so here - https://github.com/testing-cabal/mock
>
> Will do in the next couple weeks. I had chatted with Michael about it
> on Twitter and he expressed no interest in supporting it.

AIUI he's pretty busy stuck in golang land these days, so that doesn't
surprise me. He added me to PyPI etc specifically so I can do /
facilitate this :)

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud


--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/caj3hoz3bpqwfwfda3hec93_2krx1ccvaweyhezelnsugtqx...@mail.gmail.com

Re: Removing some python3-* packages

[Robert Collins]

On 3 July 2015 at 11:44, Ian Cordasco  wrote:
> On Thu, Jul 2, 2015 at 6:40 PM, Ben Finney  wrote:
>> Barry Warsaw  writes:
>>
>>> […] there's actually no reason to have a Python 3 version of enum in
>>> any version >= Python 3.4. […]
>>
>> Ian Cordasco  writes:
>>
>>> Probably a silly question, but are other libraries like unittest2 also
>>> being packaged for python3? Another library is mock. That was included
>>> in the stdlib in 3.3.
>>
>> One consideration is: What code is written to be Python 2 and Python 3
>> compatible from the same code base, which achieves this by importing a
>> module which is backported to Python 2?
>>
>> In some of my code I'm doing ‘import unit2’ to have features from that
>> library available in Python 2 code.
>>
>> Since those features are all in Python 3's standard library, the case
>> could be made that ‘python3-unit2’ is pointless; but against that is the
>> fact that a Python 3 ‘unit2’ package means that ‘import unit2’ will work
>> the same on both runtime versions.
>>
>> So I'd argue that ‘python3-mock’ and the like do have a place in Debian:
>> they make it easier to follow the recommended strategy of having a code
>> base run unchanged on Python2 and Python 3.
>
> Just to be clear, trying to use mock on 3.4 is thoroughly broken. If
> you can install python3-mock right now and use it in 3.4, then y'all
> must be carrying patches to make it work. The last 3.x version that
> mock works on is 3.3.
>
> That clearly doesn't have a place on a debian with python 3.5 as the
> version of python 3 included, unless you're planning on supporting
> packages for python 3.3 as well that will generate a numerous amount
> of bugs for you.

See my prior mail. I will be backporting all the changes in mock from
the stdlib to the mock standalone lib in the near future.

I have upload acls from Michael Foord for PyPI and I'm not afraid to
use them to fix it up - if you wanted to prep a narrow patch to fix
3.4 then please do so here - https://github.com/testing-cabal/mock

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud


--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/caj3hoz1emwkf5st9wburfxv0ffe-ufn+jkugcvvd2ttarnn...@mail.gmail.com

Re: Removing some python3-* packages

[Robert Collins]

On 3 July 2015 at 11:40, Ben Finney  wrote:
> Barry Warsaw  writes:
>
>> […] there's actually no reason to have a Python 3 version of enum in
>> any version >= Python 3.4. […]
>
> Ian Cordasco  writes:
>
>> Probably a silly question, but are other libraries like unittest2 also
>> being packaged for python3? Another library is mock. That was included
>> in the stdlib in 3.3.
>
> One consideration is: What code is written to be Python 2 and Python 3
> compatible from the same code base, which achieves this by importing a
> module which is backported to Python 2?
>
> In some of my code I'm doing ‘import unit2’ to have features from that
> library available in Python 2 code.

I hope you mean 'import unittest2' - and great, thats the intent.

> Since those features are all in Python 3's standard library, the case
> could be made that ‘python3-unit2’ is pointless; but against that is the
> fact that a Python 3 ‘unit2’ package means that ‘import unit2’ will work
> the same on both runtime versions.

unittest2 also has new features in it for python 3.2, 3.3, 3.4, and
will soon have new features for 3.5 as well.

> So I'd argue that ‘python3-mock’ and the like do have a place in Debian:
> they make it easier to follow the recommended strategy of having a code
> base run unchanged on Python2 and Python 3.

+1.

-Rob



-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud


--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/CAJ3HoZ04F=s8af224xl6bceapg8hqeo+tpv+cdfvgkyowji...@mail.gmail.com

Re: Removing some python3-* packages

[Robert Collins]

On 3 July 2015 at 09:53, Ian Cordasco  wrote:
> On Thu, Jul 2, 2015 at 2:55 PM, Barry Warsaw  wrote:
>> As part of the 3.5 test rebuild I noticed an incompatibility with
>> python3-enum, which I reported upstream.  The response was: there's actually
>> no reason to have a Python 3 version of enum in any version >= Python 3.4.
>> Since that's all we have now, maybe it makes more sense to just remove the
>> python3-enum package from Debian.
>>
>> There may be similar packages which are fairly straight backports of Python 3
>> packages.  For those that are no longer necessary, what do you think about
>> removing the python3-* version of the binary package?
>>
>> It seems like a bit of a regression given that we want Python 3 versions of
>> our libraries, but in cases like this it probably makes sense.
>>
>> Thoughts?
>> -Barry
>
> Probably a silly question, but are other libraries like unittest2 also
> being packaged for python3? Another library is mock. That was included
> in the stdlib in 3.3. Is that being packaged for python3 on Debian as
> well? I know it's nice to have the libraries available on both, but
> mock for example stopped working on python3.4 since it's solely
> maintained in the stdlib now. It doesn't make sense to package that
> for both. (Keep in mind, I'm just an observer and I haven't checked if
> those examples are already packaged for Py3)

So, unittest2 is *explicitly* targeted at 3.5, 3.4, 3.3, 3.2, 2.7,
2.6. Its not 'unittest for 2.6', its unittest for not-running-from-hg.

Same for traceback2, linecache2, and shortly, when I get a day to fix
it up, mock.

All those IMO should remain packaged, because the stdlib is moving on
them, and folk like being able to use the new shiny without waiting
years.

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud


-- 
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/caj3hoz1l2x-w0sx4igypngxorh6-wrn+bciducntjrpnvk+...@mail.gmail.com

Re: /usr/bin/python in Python 2 and 3

[Robert Collins]

On 2 May 2015 at 13:39, Geoffrey Thomas  wrote:
> On Sat, 18 Apr 2015, Scott Kitterman wrote:
>
>> I like it and I don't.  If /usr/bin/python is ever going to point to a
>> non-
>> python2 version, then I think the solution is something like this.  OTOH,
>> it
>> adds system complexity and presumably slows interpreter startup.
>>
>> If the implementation were simple enough and well enough tested, then I
>> think
>> the complexity concern is not a major issue.  Interpreter startup time is
>> a
>> big deal in some applications, so that merits some investigation before
>> anyone
>> decides this is "the" solution to the problem.
>
>
> I wrote a proof-of-concept implementation of the algorithm, to time the
> effect. (I mostly say "proof-of-concept" since I feel guilty about
> hand-coded string parsing in C) Anecdotally, testing 1000 runs of
> `python -c True` on my machine takes around 10.5-12 seconds without the
> wrapper, and around 11-12.5 seconds with the wrapper, so I'm not inclined to
> be too concerned. The wrapper is 200 lines of C, which could be reduced with
> a decent string library.
>
> https://github.com/geofft/pythonmux
>
> By the way, note that this could be built into future versions of the Python
> 3 interpreter executable. It's backwards-compatible for Python 3 to live at
> /usr/bin/python, as long as it attempts to exec Python 2 (or prints a useful
> error) whenever basename(argv[0]) is "python" and there's no metadata
> indicating Python 3 support. This avoids the extra exec for Python 3
> applications.
>
> Alternatively, maybe applications that are sensitive about interpreter
> startup time should just call `/usr/bin/python3` (or `python2`) directly. In
> a world with this multiplexer, `python` would be only a compatibility
> interface used by things that need to run on Python 2 in some places and
> Python 3 in other places.
>
>> I wonder if dh_python3 could at some point parse the magic line to assist
>> in
>> dependency generation.
>
>
> That would be useful! I do think there's value in having the metadata of
> Python minor version compatibility, even in a world where major version
> isn't a problem anymore.

This still seems like a profoundly poor idea: 'is X python3
compatible' is a turing completeness problem. There are some scripts
where you can determine compatibility, but many where you cannot until
it blows up (particularly ones with subtle bugs in string type
correctness).

IMO Folks existing scripts should be left to use python, but
encouraged to used python2 if they can't be changed to python3, and
new scripts should just use python3.

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud


-- 
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/CAJ3HoZ2u1i=3goK7uT+3-oE=jc6cgjj_8s2u-e8fz0ljhxs...@mail.gmail.com

Re: PyCon BoF: Stretch goals for cPython, PyPy & CFFI

[Robert Collins]

On 14 April 2015 at 16:38, Scott Kitterman  wrote:
> On Monday, April 13, 2015 10:36:43 PM Barry Warsaw wrote:
>> On Apr 14, 2015, at 01:57 AM, Matthias Klose wrote:
>> >#! /bin/sh
>> >python=$(shuffle /usr/bin/python2 /usr/bin/python3)
>> >exec $python "$@"
>>
>> That was more or less the joke I made at the Pycon Language Summit.  It's
>> too twisted *not* to implement. :)
>>
>> But the idea of a (perhaps less perverse) launcher would solve the PEP 394
>> dilemma of what /usr/bin/python should point to when there's no Python 2 on
>> the system by default.  E.g. maybe it reads a configuration file to
>> determine whether it runs /usr/bin/python2 or /usr/bin/python3.
>
> Even after Python (meaning python2.7) is no longer in the archive (let alone
> part of some type of default install), people might still roll their own.  I
> think the only possible answer to the question of what should /usr/bin/python
> point to when there's no python2 on the system is that it shouldn't exist.
>
> If you want python (which include /usr/bin/python), install it.  If you want
> python3, then the interpreter you're looking for is found at /usr/bin/python3.
>
> There's no dilemma to solve.

I agree.

Maybe in 2029 we can revisit it, but I've seen absolutely no good
reason to push on this other than 'Gentoo did it' - scratch that,
absolutely no good reason to push on it.

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud


-- 
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/CAJ3HoZ0BPr=fbnmqrwc5mz_bcj-bua4+qlu+a64c-vjzn7o...@mail.gmail.com

Re: multiple modules in one source

[Robert Collins]

On 3 February 2015 at 16:42, Brian May  wrote:
> Apologies for resurrecting this old thread.
>
> Is it possible to have a single git tree, containing multiple PyPI
> distributions?

Yes, but each one will ...

> I suspect the answer is "Yes, but each one will have to go into a separate
> subdirectory, each one will have to have a unique setup.py, etc".
>
> Which in turn raises another question of "Why bother?" of which I can't
> think of a good answer.
>
> It might be possible to have just a single debian directory for the entire
> git tree, have a suspicion that might get complicated however...


Whats the actual upstream, and what do you want users to have
available in terms of Debian packages?

-Rob


-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud


-- 
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/CAJ3HoZ2oN87Jxi1i0hwN==y1_ontqyjnuo_bd1ncj9xiaip...@mail.gmail.com

Re: wheel support for Debian?

[Robert Collins]

On 17 May 2014 02:17, Barry Warsaw  wrote:
> On May 15, 2014, at 11:38 PM, Scott Kitterman wrote:
>
>>Shouldn't it be singular?  Each package provides a wheel, not several wheels
>>(and I'd name the location the same for consistency).
>
> I thought about this.  True, in all current cases there's a single wheel file
> in the binary package.  I wanted to leave open the possibility that a binary
> package could contain multiple wheels, but I agree that's a weak argument
> given the scope we're trying to limit here.
>
> Two mild reasons I went with plurals in the package name: I think the
> directory where these get laid down on the file system *should* be plural,
> since it *will* hold multiple wheels, and I wanted the binary package name to
> reflect that.
>
> Also, since we now have python-wheel and python3-wheel packages, which contain
> the PyPI wheel package, I thought it was confusing to name the built-wheel
> packages with the same suffix.  E.g. python-wheel and python-urllib3-wheel
> would by naming appear to have a relationship that they actually don't have.
> It seemed clearer that the latter would be called python-urllib3-wheels.
>
> It's bikeshedding, but anyway that was my rationale for choosing the names I
> did.

Whats the tiebreak logic for a package foo.wheels (e.g. if py.wheels
comes along, will it be python-py-wheels-wheels for the wheels for
it?)

Did you consider using python-wheels-urllib3 ?

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud


-- 
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/caj3hoz1kn0-+hemsjt7_vq3pgsidrhao-eufsn07uk0mfd5...@mail.gmail.com

Re: Python 3.4 and ensurepip (rehashed, long)

[Robert Collins]

On 26 March 2014 17:15, Scott Kitterman  wrote:
> On Tuesday, March 25, 2014 15:29:06 Barry Warsaw wrote:
>> On Mar 25, 2014, at 03:19 PM, Donald Stufft wrote:
>> >I assume once someone has installed pip with apt-get they'd still be able
>> >to run pip install --upgrade pip if they wanted too?
>>
>> I would think they should be able to do that.
>
> If I've install a package and it's upgraded (this is for the system, not for
> any kind of virtualized/isolated environment), I would find it quite 
> surprising
> and unfortunate that it upgraded itself from an external source.

I'd be very surprised if a package manager told to upgrade itself used
a different source for its own code vs things it manages.

Yes, people that use pip to install things globally deserve to keep
both pieces, but either prohibit it entirely, or have it work as
advertised, not some frankenstein.

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud


-- 
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/caj3hoz28tp5udk7inf76rmjxqxc7xy_7ca0ls4p_meqs3m-...@mail.gmail.com

Re: Indeed, python-concurrent.futures is the same

[Robert Collins]

On 25 January 2014 23:01, Sandro Tosi  wrote:
> Sorry, what? and you didn't think to contact me first to almost
> rewrite the package? If there's problems, open bugs. Revert your
> changes or I'll do at the first occasion. and mainly, why don't you go
> away from DPMT once and for all? you're doing more harm than good
> here. you're not welcome here.

Huh? Thomas seemed to be doing the right thing per the DPMT standards
etc; if you don't want the package to be team maintained, perhaps take
it out of team maintenance?

-Rob


-- 
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: 
http://lists.debian.org/caj3hoz1o4gm8kakgz+ddxpxcshzoeqwvpa6kby04_1eeqqt...@mail.gmail.com

Re: Using update-alternatives for /usr/bin provided binaries

[Robert Collins]

I think it's reasonable to get OpenStack to look for python-coverage
to run it's tests when using a system package. Or use python -m
coverage. 'coverage' is indeed super generic and the precedent within
Debian for the package is to call the binary 'python-coverage'.

Is there some reason we can't take that approach?

-Rob


-- 
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: 
http://lists.debian.org/caj3hoz1ll-0ppj0khd2pw4fbn71bu+e9ouyuernui4mu2ss...@mail.gmail.com

Re: PEP 394 and shebang lines for /usr/bin/python2 scripts

[Robert Collins]

On 25 July 2013 11:09, Barry Warsaw  wrote:
> Alright, I obviously haven't convinced anybody, so I'll drop it.  We'll let
> the PEP 394 bug reports speak for themselves .  But the responses I've
> read so far make me think I probably wasn't clear in what I am proposing.
>
> On Jul 25, 2013, at 12:09 AM, Piotr Ożarowski wrote:
>
>>and you want to force an administrator who has a service using Python 2.X
>>with lots of scripts with /usr/bin/python shebang to do additional work?
>
> Of course not.  I'm not proposing that that administrators or really any*body*
> has to do anything different, or even make any changes to their stuff.  I'm
> specifically suggesting that when dh_python2 rewrites shebang lines, it use
> /usr/bin/python2 instead of /usr/bin/python, i.e. when --no-shebang-rewrite or
> --shebang=COMMAND is *not* given.
>
> What does this mean?  Upstreams don't have to change anything.  Developers and
> maintainers don't have to change anything.

Unless they want to install a deb on an older release, as Steve
pointed out. This is actually fairly common. It also introduces a
difference that needs to be documented ('from version X, python2 is
available, but not before') and doesn't really prepare us for some
future move to a different /usr/bin/python, because we haven't even
touched on the implications for:
 - local and 3rd party Python2 packages
 - local unpackaged Python2 scripts

It may be some time *after* Python2.x is gone from the archive that
all that legacy debt is migrated - if ever.

-Rob
-- 
Robert Collins 
Distinguished Technologist
HP Cloud Services


--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: 
http://lists.debian.org/CAJ3HoZ08RAG9mUNCPWbnDC1m4oA7ojtoN5Nyb=KMwWLO5y=7...@mail.gmail.com

Re: How does team maintenace of python module works?

[Robert Collins]

I didn't vote initiall because I read the below as a summary...

On 17 February 2013 01:43, Thomas Kluyver  wrote:
> On 16 February 2013 09:10, Thomas Goirand  wrote:
>>
>> It would be really stupid to only want to "claim" to be working as part
>> of the team, that's not at all what I want to do. I'd like to be able to
>> help when I can, and receive help when I need, which is the point of a
>> team.
>
>
> I agree there are reasonable reasons to want to maintain something in git,
> and it's not ideal to exclude those packages from team maintainership just
> because of the VCS question. Although, if it came to that, the team would be
> happy to offer advice and assistance for Python packages that aren't
> maintained by the team. We all want stuff to work smoothly, whether or not
> it's "our" stuff.
>
> I suggest we take a poll - not as a binding decision, but to get an idea of
> the level of support for different courses of action. You're free to attach
> more weight to the votes of highly involved team members.
>
> The following four positions have all been advocated in this thread:
>
> A - Maintain the status quo, in which DPMT packages may only be maintained
> in SVN.
> B - As A, but encourage the creation of a separate team where Python modules
> can be maintained in git.
> C - Allow DPMT-maintained packages to live in SVN or git, so new packages
> can be committed to git if the packager prefers. Optionally, we could make
> provisions to migrate existing packages.
> D - Migrate all the DPMT-maintained packages to git.

DCA

-Rob


-- 
Robert Collins 
Distinguished Technologist
HP Cloud Services


-- 
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: 
http://lists.debian.org/caj3hoz0r_ssus4ukgywpaqfq8f_p_d4fvvqkozjjdqt-xhc...@mail.gmail.com

Re: [Openstack-devel] New version of python-fixtures for debian experimental

[Robert Collins]

On 18 February 2013 17:03, Thomas Goirand  wrote:
> On 02/15/2013 03:23 PM, Robert Collins wrote:
>> +1 on what you've done and what you propose. If you want to set
>> Maintainer to DPMT and commit it to the DPMT svn repository at the
>> same time, that would be awesome.
>
> Robert,
>
> Both python-fixtures and python-testtools are now stored in the SVN of
> the python module team. Note that python-fixtures doesn't have the
> correct VCS fields yet, but it's not a big deal, since I was planing
> (with your agreement) to switch from CDBS to dh, so this will happen on
> the next upload.
>
> Both the latest versions of python-fixtures and python-testtools are now
> uploaded to Experimental.
>
> By the way, is there a Git repository for these packages? I've found one
> for testtools:
> https://github.com/testing-cabal/testtools
>
> but not for -fixtures.

Awesome, thanks.

-fixtures is currently maintained in bzr; I'm considering its hosting
future at the moment, pending how the testing-cabal feel the git
experiment with testtools pans out.

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Cloud Services


-- 
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: 
http://lists.debian.org/caj3hoz1kl-yqpmeg-pzz9hxg1xdvxerdzaqdrlftsydg1ky...@mail.gmail.com

Re: How does team maintenace of python module works?

[Robert Collins]

On 17 February 2013 08:29, Dmitrijs Ledkovs  wrote:
> On 16 February 2013 14:27, Jakub Wilk  wrote:
>> * Scott Kitterman , 2013-02-16, 09:10:
>>>
>>> On Saturday, February 16, 2013 12:43:02 PM Thomas Kluyver wrote:
>>>>
>>>> The following four positions have all been advocated in this thread:
>>>>
>>>> A - Maintain the status quo, in which DPMT packages may only be
>>>> maintained in SVN.
>>>> B - As A, but encourage the creation of a separate team where Python
>>>> modules can be maintained in git.
>>>> C - Allow DPMT-maintained packages to live in SVN or git, so new packages
>>>> can be committed to git if the packager prefers. Optionally, we could make
>>>> provisions to migrate existing packages.
>>>> D - Migrate all the DPMT-maintained packages to git.
>>>>
>>>> (I suggest we don't consider other VCSs - while we might have our
>>>> favourites, I sampled the list of Debian teams, and found very few using
>>>> anything other than svn or git. So tools & workflows for other VCSs are
>>>> likely to be less well developed.)
>>>>
>>>> So I would vote CDBA, in order of preference.
>>>
>>>
>>> E - Migrated to bzr, but I want someone else to to all the work.
>>>
>>> EA
>>
>>
>> F - Migrate to Mercurial, but I want someone else to do all the work.
>>
>
> A, F.1 - Migrate to Mercurial, if and only if mercurial queues are
> fully functional and are used to maintain the debian/patches
> sub-repository.
>
> realisticly i am opposed to a mix of version control systems.
>
> someone to do the work - means starting a mirror which works in
> read-only / mirror mode only.
>
> When gnome.org was migrating from svn - they had multiple mirrors
> running (bzr, and git, not sure if an hg was there as well)
>
> Regards,
>
> Dmitrijs.

I will *always* have a soft spot in my heart for bzr, having spent
many years working on it.
But consider:

http://qa.debian.org/popcon-graph.php?packages=subversion+git+mercurial+bazaar&show_installed=on&show_vote=on&want_legend=on&want_ticks=on&from_date&to_date&hlght_date&date_fmt=%25Y-%25m&beenhere=1

If we're changing VCS, there is a vastly higher probability that the
folk whose software we are packaging is in git, and that contributors
we get will be familiar with git, than any other system now. That
doesn't mean git is better or worse, but if we're changing systems
because of preference (rather than fitness-for-purpose), then I think
we'd be crazy to consider any other VCS.

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Cloud Services


-- 
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: 
http://lists.debian.org/CAJ3HoZ13=iiu1dBA=YQgsqZqvXHd0=2b=nyk6k8sxw6bmpj...@mail.gmail.com

Re: New upstream release of python-testtools

[Robert Collins]

On 16 February 2013 01:52, Thomas Goirand  wrote:
> Hi Robert & Jelmer,
>
> I would like to upload a new upstream version of python-testtools. I
> have worked out some changes in the packaging, which I have attached to
> this message.
>
> Do you agree with such NMU?

Yes, thank you.

> Can I also upload this package in the python module team SVN repository,
> instead of collab-maint BZR (since we agreed on it for python-fixtures,
> probably you would like this to be done as well for this package)?

Yes please.

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Cloud Services


-- 
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: 
http://lists.debian.org/caj3hoz1arzstdx02unxpyq692g_t8f6zwvehfzjgyzbnrec...@mail.gmail.com

Re: Current state of packaging Python software for Debian

[Robert Collins]

On Sat, Jul 2, 2011 at 2:07 AM, Barry Warsaw  wrote:
>>In some sub-communities, py.test or nosetests are the standard, not
>>setup.py test.
>
> Yes, but if I understand where Michael is taking unittest2, those can just be
> refactored to be plugins instead of separate standards.  Then `python setup.py
> test` can be the preferred and documented standards, while allowing those
> other command lines to also exist.

unittest2 is still a unittest runner at heart; the basic model is
sound to scale up to N processes and so forth (see tox or
testrespository for instance), but compatibility with arbitrary ways
of running is pretty tricky. See for instance the guts of the
following three runners: trial, django and zc.testing.testrunner.

All I'm saying is don't hold your breath: those communities could have
plugged into the original unittest compatibly but didn't - I think it
needs to be massively more clear *how* to do so, and on older Pythons
for those communities - they don't live on the edge ;) - for
one-runner-with-plugins to reach that point.

As a data point, in the java world multiple runners is still the case,
with some common interop on output format.

-Rob


--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: 
http://lists.debian.org/CAJ3HoZ1mOqYGcMfn0Lf514dBmmuBOD5GL78=mvfrewqxvsc...@mail.gmail.com

Re: Switching to git

[Robert Collins]

On Mon, Mar 7, 2011 at 8:09 AM, Scott Kitterman  wrote:
> Robert Collins  wrote:
>
>>On Mon, Mar 7, 2011 at 7:52 AM, Scott Kitterman 
>>wrote:
>>...
>>> reasonably comfortable for both.  It's not as fast a git and it
>>suffers from
>>> not being able to do partial checkouts (like git), so it's very much
>>a middle
>>> ground in both advanatages and disadvantages between svn and git.
>>
>>I believe the partial checkout limitation has been fixed  couple of
>>months back - the next major release will include that facility.
>>
>>-Rob
>
> When is the next release expected?

For clarity, the thing I'm referring to is the ability to commit
directly to a stacked branch - which I think is equivalent to the
partial limitation you're referencing. I've just checked in #bzr, and
that is in 2.3.0, which has been out for ~ a month.

If you are instead referring to getting a subtree like /docs only -
bzr has had that for a couple of years now.

-Rob


--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: 
http://lists.debian.org/AANLkTin_smHGBWParRzZP745Ar_J9NHe2NUJwh=rz...@mail.gmail.com

Re: Switching to git

[Robert Collins]

On Mon, Mar 7, 2011 at 7:52 AM, Scott Kitterman  wrote:
...
> reasonably comfortable for both.  It's not as fast a git and it suffers from
> not being able to do partial checkouts (like git), so it's very much a middle
> ground in both advanatages and disadvantages between svn and git.

I believe the partial checkout limitation has been fixed  couple of
months back - the next major release will include that facility.

-Rob


--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: 
http://lists.debian.org/AANLkTi=lk7xpv24lpk0p+mrdrk4ggmzdjojmohwhx...@mail.gmail.com

Re: coming back to packaging multiple versions of libraries

[Robert Collins]

On Thu, Jan 6, 2011 at 1:14 PM, Barry Warsaw  wrote:
> On Jan 06, 2011, at 12:45 PM, Robert Collins wrote:
>
>>I'm not trying to do this in a hidden way though? Why do you think
>>that that is the case?
>
> Sorry, I meant "automatic".

I'm not proposing anything magical: maintainer intent will always be
present (just as it is for C libraries).

We can look at building out heuristics for setting minimum dependency
deps automatically, but thats orthogonal to basic mechanism which is
what I'm focused on for now.

-Rob


-- 
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: 
http://lists.debian.org/aanlktimnootrarpcxfwf6x=dr94jer3-rlicektmh...@mail.gmail.com

Re: coming back to packaging multiple versions of libraries

[Robert Collins]

On Thu, Jan 6, 2011 at 12:39 PM, Barry Warsaw  wrote:
> These are not necessarily mutually exclusive. ;) #3 and #4 are both worth
> pursuing in any case, but kind of outside the domain of either upstream
> (except were the stdlib is concerned) or debian-python.  Still, as a Python
> programmer, if you find gratuitous API breaks in packages you care about, it's
> worth a bug report.
>
> (I'm still not quite sure why Robert doesn't just get the wadllib developers
> to fix their API breakage?)

wadllib was an example. I'm aware of -very- few libraries that never
change things; what constitutes an API break is vaguely defined; the
functional definition we need here is 'when consuming code is broken
by upgrading the dependency' : that definition is much wider than
'something we advertised is no longer supported' which many upstream
follow.

> I really hate having the version numbers in the module names, and actually in
> package names too, but the latter tend to disappear after transitional
> periods.  Names like urllib2 live forever unfortunately.  I don't have any
> great ideas about how to improve that though.

having the language support N versions of urllib installed at once
seems to work very well for C, C++, C#, Java and other languages.
Perhaps Python should just do that.

> I agree with Piotr, I think trying to do this in a hidden way is fraught with
> problems and lots of unanswered questions.  I will mention that for years
> Mailman shipped with its own version of the email package because it needed an
> exact API that might have been different than the default one, depending on
> the version of Python you were using.  So a specific application that cares
> deeply about its library versions I think has no better choice than to do
> something like that, which is essentially your #2.  There can be more
> principled ways of doing that, e.g. with buildout or virtualenv, but it boils
> down to the same thing.  There's just no good distro-way of doing that
> though.

I'm not trying to do this in a hidden way though? Why do you think
that that is the case?

-Rob


--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: 
http://lists.debian.org/aanlktikbsuaogyztzmtmfnzenccxuxhy3bokdn4wm...@mail.gmail.com

Re: coming back to packaging multiple versions of libraries

[Robert Collins]

2011/1/6 Piotr Ożarowski :
> IMHO installing two versions of the same (public) Python module should
> be simply forbidden in policy. For one simple reason: if module foo uses
> bar in version 1 and spam uses bar in version 2, imagine what will
> happen with egg which uses both foo and spam.

This issue exists with C libraries too, but its not forbidden. Why
should C libraries be expected to permit this, but not Python
libraries?

> Right now I see only these options:
> 1) create separate binary packages for both versions of bar conflicting
>   with each other
> 2) install only one version of bar as public module and the second one as
>   private module

Neither of these meet the use cases I've described comprehensively before.

> 3) convince library author(s) to *not* break API so often (6 months? 1
>   year?)

Once is enough; the ecosystem is huge and unless we handle transitions
gracefully we'll get into nightmare situations routinely - which we do
on complex stacks (like zope) on a regular basis.

> 4) convince library author(s) to rename it (bar, bar2, bar3, etc.) once
>   he/she decides to break API

I'm not sure that this is necessary or worth it. Certainly you'd need
a massive change in the culture of Python upstream (python *itself* to
have this act as a systematic fix).

> 5) convince Python interpreter developers to provide something similar to
>   sonames (bar.py, bar.2/__init__.py, bar.2.3/__init__.py, bar.3.py)
>   and add support for syntax I mentioned last time or use something
>   like 3rd party pkg_resources' require() or... allow
>   foo/lib-packages/bar → ../../bar-1 symlinks (where dist-packages/bar
>   is in version 2 and dist-packages/bar-1 in v1, note that bar-1 is not
>   a valid module name)... with lots of problems Clint already mentioned

Its already developed : pkg_resources.Require()

I'm working on the mechanics to get default programs working; once
thats done things using the soname equivalent should be trivial.

-Rob


--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: 
http://lists.debian.org/aanlkti=j9z+ddf3vdz5lcu3f9__c9zzypff+3qrta...@mail.gmail.com

Re: coming back to packaging multiple versions of libraries

[Robert Collins]

On Mon, Jan 3, 2011 at 9:12 PM, Piotr Ożarowski  wrote:
> what's the point of installing two different versions of the same module
> if only one will be used anyway? If the answer to that question is:
> "in the application where I need different version, I will adjust
> sys.path" - why not simply provide different version of the library as
> private module? Security team will hate you anyway.

For the same reason we support libreadline5 and 6 : to transition
higher level code without a flag day across the entire machine.
Security team should not hate this any more or less than they do for C
libraries, its for precisely the same reasons.

>> I'm very interested in other approaches to make this work;
>
> teach upstream about the importance of stable API (yeah, I know Python
> programmers do not care about it much, it could be worse¹, though)

The best of intentions still sometimes leads to incompatible changes,
but yeah, preach it :).

> You could also provide wadllib/__init__.py file which will modify
> __path__ to point to the newest installed versions (and f.e. check for
> WADLIBVERSION env. variable if someone doesn't want the newest version)
> but it's an ugly hack and could cause more harm than good, IMHO.

I can see that breaking overly-clever packages, or things like the
twisted splitup, bzrlib.plugins etc. I'd like to be minimally
invasive.

@Barry: yes, the C/C++/CLR/ library handling is precisely equivalent.
I would expect only as many concurrent versions in the archive as we
need to support. There are two use cases:
 - in sid, while transitioning an incompatible change
 - users using private packages with their own archive

In the former case, we'd generally(1) drop an old version as soon as
its rdepends(2) is empty. In the latter case, its up to the user to
decide when they don't need a particular version.

(1) Exceptions might occur if a maintainer feels that the package is
extremely widely used outside of the packaging universe and wants to
keep it available.
(2) Figuring out rdepends sanely is another step to be taken, but I'm
hoping to break this down far enough to get consensus on individual
bits.

It really does look like having better upstream facilities would make
this a no-brainer for us; what I'd like to achieve though is something
that /works/ for the existing python platform - for 2.7 which will be
around a good long time, and then we'll have plenty of data to discuss
with upstream.

-Rob


--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: 
http://lists.debian.org/aanlktim7joembx_7nhtpb2mpkaq-9iiet7txbp8cp...@mail.gmail.com

coming back to packaging multiple versions of libraries

[Robert Collins]

So in the thread 'Python packaging, dependencies, upstream facilities'
we had a brief talk that faded out; rather than resurrecting the
entire thread, I'd like to pick one point that seems like a necessary
condition to me: installing the eggs in versioned paths rather than
simple module paths.

Take something like wadllib - a library which has had the odd
effective api break without a namespace change. We currently install
its contents as:

/usr/share/pyshared
/usr/share/pyshared/wadllib
...

/usr/share/pyshared/wadllib-1.1.4.egg-info
/usr/share/pyshared/wadllib-1.1.4.egg-info/PKG-INFO
...

So the question is:
whats the simplest way to install this somewhere else - e.g.
/usr/share/pyshared/wadllib-1.1.4
and have
import wadllib
still work without user intervention.

Two options seem to present themselves to me at the moment:
 - the pyshared symlink logic could select the highest version and
link that into /usr/lib/.../wadllib as well as into
/usr/lib/.../wadllib-1.1.4
   - possibly using a single wadllib->wadllib-1.1.4 symlink, or
possibly using the 'normal' structure of directories + per-file links.
 - we could drop the versioned path into site.py in the same way
python-numeric etc do

I'm very interested in other approaches to make this work; the reason
for doing it were covered in the other thread, but making this one
case work would allow having two packages with slightly different
versions of something to coexist - which is immensely useful for
application (vs library) developers when developing for multiple
versions of dependencies.

It seems to me that the first option is the most efficient performance
wise for now; we should get better upstream support to, of course.

-Rob


-- 
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: 
http://lists.debian.org/aanlkti=6iq2z_d3tdzat0u35kfua_tz_drtutextq...@mail.gmail.com

Re: dh_python and python policy analysis

[Robert Collins]

On Sun, 2006-08-13 at 19:56 -0500, Manoj Srivastava wrote:
>Here there are two cases. Either module foo can't be written at all
>for version 2.6, or it the same functionality can be provided with


This is a little simplistic.

The parser changes fairly routinely in python versions. This means that
a version conditional is not sufficient to provide compatability with
older pythons - the module will not parse.

The usual thing done for cross version support is to write in the older
version of python, or in extreme cases (i.e. where performance really
hurts) have two separate modules _foo_2_5 and _foo_2_6 and conditionally
do
'from _foo_2_5 import *' etc.

-Rob
-- 
GPG key available at: .


signature.asc
Description: This is a digitally signed message part