Re: Removal of Python 2 from the Xfce spin

[Nick Coghlan]

On Sun, 13 Jan 2019 at 19:36, Miro Hrončok  wrote:
> On 13. 01. 19 7:57, Nick Coghlan wrote:
> > Anyway, Zbigniew has resubmitted the change to make the package policy
> > compliant as a new PR:
> > https://src.fedoraproject.org/rpms/catfish/pull-request/2
>
> This is a result of https://pagure.io/fesco/issue/2050

Ah, nice.

Good to see that this particular situation was able to be resolved
relatively amicably, while also providing a precedent that failing to
abide by FPC/FESCo decisions can be approached as a variant of the
"Non-responsive maintainer" case.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org

Re: Removal of Python 2 from the Xfce spin

[Nick Coghlan]

On Thu, 10 Jan 2019 at 03:14, Jason L Tibbitts III  wrote:
>
> >>>>> "MH" == Miro Hrončok  writes:
>
> MH> Aaaand... it's reverted :D
>
> He reverts any of the commits I have made to the packages he maintains
> as well.  Just mass cleanup things like the removal of defattr.
> Reverted with a completely empty commit message.
>
> I really don't want to get into a revert war with the guy.  It's just
> pointless.  I guess at some point either FPC or FESCo will have to talk
> about it.

The reversion from single-binary-package to dual-binary-packages was
similar inaccurate, as it just says "Rebuilt" rather than "Revert to
producing separate Py2 and Py3 packages":
https://src.fedoraproject.org/rpms/catfish/c/7ae2507b600accf729c8a21808d24a8a9db7b8e2?branch=master

It also *deleted* the changelog entry for 1.4.5-2, rather than just
adding the new one.

Given that Miro's change was to bring the package into line with the
distro packaging policy, it seems really strange to me that a package
maintainer would be free to just ignore the policy.

Anyway, Zbigniew has resubmitted the change to make the package policy
compliant as a new PR:
https://src.fedoraproject.org/rpms/catfish/pull-request/2

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org

Fwd: [Distutils]Introducing XAR - SquashFS based mountable executables - Calling OS/Distro Maintainers

[Nick Coghlan]

Hi folks,

Facebook just published an interesting bit of tech that uses squashfs
to create single-file executables that don't require a pre-installed
container engine to execute, but do place some more significant
demands on the FUSE subsystem, and also require a bootstrapping
executable to set up the filesystem mounts.

I recommend taking a look at their blog post, as xar seems like quite
an interesting capability to consider offering: whereas containers and
flatpaks give you both build time and runtime component isolation, xar
files aim to behave more like a statically linked ELF binary with no
supporting data files (no runtime isolation, but no direct
dependencies on the host filesystem either).

Cheers,
Nick.

-- Forwarded message --
From: Cooper Ry Lees 
Date: 14 July 2018 at 06:51
Subject: [Distutils]Introducing XAR - SquashFS based mountable
executables - Calling OS/Distro Maintainers
To: distutils-...@python.org


Hi distutils,

Today Facebook Open Sourced a competitor to PEX and other zip based
distribution methods for Python (and potentially other languages).
Basically it's claim to fame is start up time for large modules being
similar to regular on file system modules due to extracting on read
via SquashFS mounted executables. For more information read our blog
post:
- 
https://code.fb.com/data-infrastructure/xars-a-more-efficient-open-source-system-for-self-contained-executables/

The bdist_xar 'wheel' like plugin is also now in PyPI:
https://pypi.org/project/xar/
(https://github.com/facebookincubator/xar/).

Main reason I wanted to post here was I'd love to reach out to our OS
maintainers on list (e.g. Mr Stinner) and talk about getting the
components of XAR into the OS's package repos. The main components
that make sense are:
- xarexec_fuse: https://github.com/facebookincubator/xar/
- squashfuse: a newer version with squashfuse_ll (and optionally zstd
support) https://github.com/vasi/squashfuse
- squashfs-tools: a newer version with zstd support (made optional I guess)

Once we had this, it would make building XARs as easy as building
wheels, even more so when we can define OS level dependencies for it!

Please feel free to reach out to me directly with any questions etc. -
Also want to note, Sumanah suggested I floated this here, I apologize
if this is a misuse of the list. If so, please ignore. I was torn
wether or not this is appropriate.

Thanks,
Cooper

--
Distutils-SIG mailing list -- distutils-...@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at
https://mail.python.org/mm3/archives/list/distutils-...@python.org/message/TYENF32X7E6U3M2WJWMTH5DTJEXZBVMS/



-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/message/VUE2UMR4EDOHUAO3PONDSEWVI5RXUVPK/

Re: Are 3.6.x release candidates useful to find bugs?

[Nick Coghlan]

On 21 May 2018 at 22:09, Charalampos Stratakis <cstra...@redhat.com> wrote:

> > So, please, if you find any bugs in upcoming Python *3.6.x* release
> > candidates, let me know (or write to Łukasz directly)! If there are no
> > such reports, there will be no 3.7.x RCs.
>


> So I see two potential outcomes here. Either we change our procedures on
> updating the point releases and we test the RC ones as well, or we can just
> say to upstream that from
> Fedora's POV we don't really bother with the rc point releases. I'm more
> inclined towards the second option, as the first one is highly dependent on
> the workload/free cycles
> at that particular time frame, and combined with the short window between
> an RC and the actual release, it could strain a lot of resources for not
> much of a benefit.
>

Note also that part of Łukasz's proposal was that if a significant
regression *was* found in a point release, then the resolution would be to
spin a new point release with a fix ASAP. It's just that in such cases, the
version numbers involved would be X.Y.Z and X.Y.Z+1 rather than X.Y.Zrc1
and X.Y.Z.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/message/MMEWGNMIJLSGBKITBHLDZRYGFNEUEIJZ/

Re: PEP 394 update proposal: Allow changing the `python` command in,some cases

[Nick Coghlan]

On 26 April 2018 at 05:22, Petr Viktorin <pvikt...@redhat.com> wrote:
> If this goes through, in Fedora I would like to:
>
> - Create a "python" package containing *just* the /usr/bin/python symlink.
> This would be a subpackage of python2, and python2 would *suggest* it. In
> other words, it would be installed by default with Python 2, but mock, koji,
> and test tools would omit it (unless something deliberately Requires
> `python` or `/usr/bin/python`).
>
> - Put an unofficial "python 3.6" on COPR (and maybe eventually in a Module),
> which would be the recommended way to make `python` invoke Python 3.

Cool, thanks for moving this forward!

Did you want to update
https://fedora-python.readthedocs.io/en/latest/plans/default-python-module/
accordingly, or should that entire site just be scrubbed as an
experimental idea that didn't work out in practice?

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Building the _uuid accelerator module for CPython 3.7

[Nick Coghlan]

Hi folks,

If anyone has built the development version of CPython on Fedora in
recent weeks you've likely received the following message:

```
The necessary bits to build these optional modules were not found:
_uuid
```

This is a consequence of the new uuid accelerator module introduced in
https://bugs.python.org/issue11063, which means there's a new optional
build dependency in 3.7 that "sudo dnf builddep python3" won't pick up
yet.

The first way that this can fail is simply because the new build
dependency is missing:

$ ./configure | grep uuid
checking for uuid_generate_time_safe... no

The fix for that is to install libuuid-devel:

$ sudo dnf install libuuid-devel
$ ./configure | grep uuid
checking for uuid_generate_time_safe... yes

However, there was also a bug in the build time check for the
existence of `uuid.h` and I've only just submitted the PR to fix that:
https://github.com/python/cpython/pull/4565

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: flit

[Nick Coghlan]

On 18 November 2017 at 06:54, Toshio Kuratomi <a.bad...@gmail.com> wrote:
> On Thu, Nov 16, 2017 at 8:43 PM, Nick Coghlan <ncogh...@gmail.com> wrote:
>> Switching to a wheel based build doesn't change either the starting
>> point (which is still an sdist) or the end point (which is still a
>> policy compliant RPM), it changes the internal interface between the
>> build step and the install step from being a distutils build directory
>> to a wheel archive:
>>
>> "pip wheel" (or "setup.py bdist_wheel") in the %build step
>> "pip install" in the %install step
>>
> So This seems more like "Building packages via pip"  or "Building
> packages that use pyproject.toml".  wheels are actually an
> implementation detail here that doesn't really surface to the either
> the input side or the output side.  we don't care if the build tool
> (pip in this example) takes the sdist with a pyproject.toml file and
> builds a wheel, a _build/ directory, or some other intermediate
> format.  We just care that it builds the appropriate files in this
> step and then installs from that built resource in the second step.
> This is why the name and focus seems wrong to me.  the importance to
> us when building from source is the metadata format and the tools that
> transform the source code with that metadata into a built resource on
> our filesystem.

Ah, OK - this makes sense.

Framing it in terms of the required input file, the two most suitable
names would be:

* "pyproject.toml compatible spec files"
* "setup.py only spec files"

I'd suggest keeping the current wheel-centric macro names, though, and
just adjust the wheel building macro to accept pyproject.toml files in
addition to setup.py files.

>> At a policy management level, I think it makes sense to separate the
>> "these are the policy decisions you *must* abide by" guidelines (which
>> are the domain of FPC) from the "Here are the helper macros that we
>> provide to make it easier to abide by those guidelines". The
>> distinction is that you can still pass a package review without using
>> the helper macros, but you'll still want to use them in most cases
>> simply because they make ongoing package maintenance easier (since the
>> helper macros will adjust automatically to rebases and policy changes,
>> while handcrafted spec files might not).
>>
>
> So, How to write a package that complies with the guidelines is also
> the domain of the FPC (at least, right now).  So you need to have
> something inside of the guidelines that shows how to use these things.
> I leaned towards spelling out the manual steps and then showing that
> there were macros that encapsulated some of those steps but that may
> no longer be the preferred direction.  The guidelines must have at
> least one or the other, though.

In that case, I'd lean towards emphasising the macros in the
guidelines, and then pointing towards the macro definitions in
dist-git if folks really want to dig into the details of what the
macros actually do. However, we'd also keep the various sections that
explain how to do various lower level things correctly (like
precompiling Python files), since those providing the "Why?" of
various decisions, which simply reading the macro source files won't
give you.

Cheers,
NIck.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: flit

[Nick Coghlan]

On 17 November 2017 at 11:55, Toshio Kuratomi <a.bad...@gmail.com> wrote:
>
> On Thu, Nov 16, 2017 at 5:37 PM, Nick Coghlan <ncogh...@gmail.com> wrote:
> > So the two possible approaches are:
> >
> > * traditional sdist: "setup.py build", "setup.py install"
> > * Current wheel macros: "setup.py bdist_wheel", "pip install "
> >
> > If we tweak the %py_build_wheel macro to use `pip wheel` [1] rather than
> > calling `setup.py bdist_wheel` directly, then even the wheel build macro
> > would be usable without a setup.py shim (once pip itself fully supports PEP
> > 517/518)
> >
>
> I'm not sure what you're saying.  I must be further outside of the
> packaging loop than I thought.
>
> The two questions that I need to know the answer to to make sure we're
> even vaguley on the same page are:
>
> * Is the rpm Source: line still going to be an sdist?
> * Are the files that are in the built rpm going to be the same as now?

Yes, those are both unchanged.

The issue is specifically with the example in
https://fedoraproject.org/wiki/Packaging:Python#Example_common_spec_file,
which only shows the %pyX_build and %pyX_install macros, which assume
that the right way to install a Python project inside the buildroot is
to run:

"setup.py build" in the %build step
"setup.py install" in the %install step

That works as long as the project either uses setup.py natively (i.e.
distutils/setuptools projects), or provides a backwards compatibility
shim, but won't work consistently for newer projects that rely on
pyproject.toml and the static build dependency declarations from PEPs
517 & 518.

Switching to a wheel based build doesn't change either the starting
point (which is still an sdist) or the end point (which is still a
policy compliant RPM), it changes the internal interface between the
build step and the install step from being a distutils build directory
to a wheel archive:

"pip wheel" (or "setup.py bdist_wheel") in the %build step
"pip install" in the %install step

At a policy management level, I think it makes sense to separate the
"these are the policy decisions you *must* abide by" guidelines (which
are the domain of FPC) from the "Here are the helper macros that we
provide to make it easier to abide by those guidelines". The
distinction is that you can still pass a package review without using
the helper macros, but you'll still want to use them in most cases
simply because they make ongoing package maintenance easier (since the
helper macros will adjust automatically to rebases and policy changes,
while handcrafted spec files might not).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: flit

[Nick Coghlan]

On 17 November 2017 at 04:50, Jason L Tibbitts III <ti...@math.uh.edu>
wrote:

> Note also that the guidelines are in the wiki for convenience, but
> there have been at least two efforts to move them to a "better" format.
> And I would truly love to have a companion set of documents which aren't
> guidelines but which expand on them and document things like this.
> Whether that ends up being maintained as part of the guidelines or
> separately hasn't even been discussed yet, but at least you'll be able
> to send a PR in pagure to get something updated.
>

OK, it sounds like putting something into
https://fedora-python.readthedocs.io/en/latest/ will actually be the best
way to go, then.

That way, the update to the guidelines page can just be a link to a page
maintained in https://github.com/fedora-python/fedora-python

Cheers,
Nick.

P.S. The latter page is mainly in GitHub because I couldn't figure out how
to get Pagure's RTD integration to work, but I also see some benefits in
lowering barriers to contribution for Python folks that already have GitHub
accounts, but don't have Fedora accounts yet.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: flit

[Nick Coghlan]

On 16 November 2017 at 22:33, Miro Hrončok <mhron...@redhat.com> wrote:

> Adding the link makes sense to me. Adding all the macros definition to the
> wiki does not make sense to me, but form different reasons. I think that
> having %py3_build_egg and %py3_install_egg there is just not necessary.
> Since there are more files at [0] I'd just add that link.
>
> [0] https://src.fedoraproject.org/rpms/python-rpm-macros/tree/master


Even though it's just a new informational link, I'm guessing we still need
to file an FPC ticket for that?


> About flit:
>
>  * is this way of creating Python distributions getting more and
>more popular?
>

It is - Thomas Kluyver (flit's creator) is actually driving a lot of work
to get pip's pluggable build system support working properly.

The key differences between it and setuptools are:

1. It's designed specifically for pure Python projects
2. It assumes it can get the list of files to package from git

Those two assumptions sweep away a *lot* of the complexity in setuptools,
and hence make flit far more beginner friendly.


>  * if so, should we add a new section of the guidelines? something like
>"Packaging setup.py-less projects"?
>

Rather than emphasising the absence of setup.py, I'd emphasise the use of
wheel files:

* "Defining an RPM based on a wheel build process"
* "Defining an RPM based on a setup.py file"

The latter would just be a renamed
https://fedoraproject.org/wiki/Packaging:Python#Example_common_spec_file -
that is the setup.py based build process, but it isn't currently obvious
that `pyX_build` and `pyX_install` assume the use of a setup.py file.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: flit

[Nick Coghlan]

On 16 November 2017 at 16:51, Elliott Sales de Andrade <
quantum.anal...@gmail.com> wrote:

> On 16 November 2017 at 01:31, Nick Coghlan <ncogh...@gmail.com> wrote:
>>
>> However, if flit is now adding its own shim implcitly, then the answer
>> would just be "Yes".
>>
>

> There's no need for a shim; basically use flit to build a wheel and then
> use the wheel installation macros instead.
>

Wait, we have wheel installation macros?

/me goes and checks
https://src.fedoraproject.org/rpms/python-rpm-macros/blob/master/f/macros.python

Huh, so we do. Somehow I managed to miss that :)

How would folks feel about adding a link to the dist-git macro file to the
macros section in https://fedoraproject.org/wiki/Packaging:Python#Macros ?

I'm not currently inclined to add the information directly to the wiki page
as:

- I don't like wikis in general as a docs management tool
- the wiki only shows one version, but the RPM with the default macros is
branched along with the rest of Fedora

We could potentially add something to https://fedora-python.readthedocs.io/
about Fedora's RPM packaging tools for Python projects, but that would be
significantly more work than just linking to dist-git.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: flit

[Nick Coghlan]

On 16 November 2017 at 12:30, Orion Poplawski <or...@nwra.com> wrote:

> So - is it possible to build rpms with flit?
>

I haven't checked recently, but I believe flit still omits `setup.py` from
its sdists by default, and the Python RPM macros are still expecting to be
able to call that directly.

If that's still the case, then the answer would be "Yes, but only if you
make sure the sdist still contains a suitable setup.py shim".

However, if flit is now adding its own shim implcitly, then the answer
would just be "Yes".

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: Modularity, container images, and the default Python stack(s)

[Nick Coghlan]

On 29 August 2017 at 19:33, Petr Viktorin <pvikt...@redhat.com> wrote:
> On 08/25/2017 10:23 AM, Nick Coghlan wrote:
>> Traditionally, that would have meant that we wouldn't get a Fedora
>> based Python 3.7 container out the door until the release of Fedora 29
>> in October, and we'd never ship the F28+3.7 combination at all, even
>> for folks that mainly just want the Python runtime, and then will use
>> pip to install everything else they need.
>
>
> Traditionally, that's very well possible: there'd be a "python37" package
> that provides a /usr/bin/python37. We do the same for old versions, and for
> python36 on f25. It works quite well.
>
> If the intended use for 3.7 in f28 is to make a virtualenv and use pip to
> install things, that virtualenv can very well be made by calling
> /usr/bin/python37. I still see no reason for a stream that makes
> /usr/bin/python3 point to 3.7.

My aim is to be able to define a "Fedora 28 + Python 3.7" container
image that meets the following criteria:

1. Everything that's installed by default is owned by an RPM (no
injecting extra symlinks as part of the container build process)
2. Invocations of "python3" and "/usr/bin/python3" (whether
interactively or via shebang) run Python 3.7
3. I'm able to do this with just the Fedora Modularity tooling - no
extra downstream tools like scl-utils

>> However, I think modularity gives us access to a more flexible
>> approach: we can set up the F28 System Profile to *default* to the 3.6
>> Python App Runtime stream, allowing the sequence of updates to be:
>>
>> 1. In Fedora 28, we split the Python 3 runtime module into two pieces:
>> - the App Runtime, with a 3.6 stream
>> - the Integrated Runtime, with an f28 stream that depends on
>> AppRuntime:3.6
>> 2. After 3.7 hits beta in January 2018, we add a "3.7" stream to the App
>> Runtime
>> 3. After 3.7 is released, we start building a new F28+3.7 app
>> development container
>> - in this configuration, most system packages that need Python 3 can't
>> be installed
>
> How do you achieve this for package that depend on /usr/bin/python3?

If that's all they depend on, I guess they'll still be installable.
However, if they depend on a particular version of the Python ABI,
then the 3.7 package won't provide it.

>>> Also, the names are a mouthful; let's revise naming after we get the
>>> semantics down :)
>>
>>
>> I'm reasonably happy with App Runtime (since I stole that directly
>> from "OpenShift Application Runtimes", which is the downstream use
>> case I'm interested in making easier to maintain), and I think
>> "Integrated Python" or "Integrated Runtime" accurately conveys the
>> significance of the proposed stream mapping between Fedora versions
>> and Python versions ("F28 integration testing uses Python 3.6", "F29
>> integration testing uses Python 3.7", etc).
>>
>> For the Testing Runtimes, we could likely just drop the
>> "interoperability" qualifier, giving:
>>
>> - Platform Python: the private, always installed, potentially
>> non-standards-compliant, Python that dnf uses
>> - Integrated Python: an optional standards-compliant Python used for
>> development and automated scripting *of* Fedora
>> - Application Python: used to run Python apps *on* Fedora (but may not
>> be integrated fully, depending on version)
>> - Testing Python: sufficient for cross-version compatibility testing,
>> but not recommended for app deployments
>
> It seems that the Application and Testing Pythons differ only in "SLA"s and
> availablility of additional packages, is that right?
>
> Also, it seems that the difference between Integrated Python and Application
> Python is that one provides /usr/bin/python3. Is that correct?

Originally that's what I had, but as I wrote it up in more detail (see
https://github.com/fedora-python/fedora-python/commit/e76aa19870627059c9b1539f19060e77ef95317e)
I came to realise that it probably makes more sense to have a module
level dependency from the Integrated Python branches to the
corresponding Application Python branches and then have the "python3
symlink or not?" distinction exist between the Application module
config and the Testing module configs.

That way, the "Integrated Python" module shouldn't actually need to
pull in any new source trees - it just becomes a way of mapping from
Fedora versions to their integrated Python versions in a
machine-readable way such that other Fedora modules can easily declare
either "build for all Fedora Integrated Python streams" (by defining a
mod

Re: Modularity, container images, and the default Python stack(s)

[Nick Coghlan]

On 24 August 2017 at 21:28, Petr Viktorin <pvikt...@redhat.com> wrote:
> On 08/24/2017 12:22 PM, Nick Coghlan wrote:
>>> Stream names match the Platform module. We follow its policy here, even
>>> when
>>> it changes.
>>
>> Oh, interesting, I had that backwards (I thought the planned F27
>> Python modules were the ones named after upstream Python releases).
>>
>> That means the current module definitions would be closer to what I'm
>> calling the "Integrated Python Modules".
>
> My comment was for Platform Python :)

Oops, reading comprehension fail on my part :)

>> * Python Interoperability Testing Modules
>>- one module per Python X.Y release
>>- only one stream per module
>>- conflict with the corresponding Application Runtime stream
>>- provide "/usr/bin/pythonXY" and "/usr/bin/pythonX.Y"
>>- do NOT satisfy "python2-*" and "python3-*" RPM dependencies
>>
>> And looking at their current implementation in Fedora, one notable
>> difference between them and the Application Runtime streams is that
>> these would just use their bundled pip and setuptools, rather than
>> splitting those out the way the regular packages do.
>
> No, I don't want to support *full* parallel-installable stacks. Let's stick
> to what we currently support in Fedora, not more.
>
> "Python Interoperability Testing Modules" get "x.y" streams and provide
> "x.y" binaries;

These modules will probably only have one stream each, so I don't have
a strong personal opinion on what that stream is called (although it's
also possible they could each end up with two streams: one that
actually installs the parallel installable version, and one that just
depends on the corresponding application runtime stream).

> "Python Application Runtime Modules" get one stream for
> python3 and one python2, and provide the python2/python3 binaries.

I think we can do better than that, and the relative timing of the F28
and Python 3.7 releases provides a good illustration of why I think we
should aim to do so:

* F28 code freeze is in March, with the release in May
* 3.7 release candidate is in May, with the release in June

Traditionally, that would have meant that we wouldn't get a Fedora
based Python 3.7 container out the door until the release of Fedora 29
in October, and we'd never ship the F28+3.7 combination at all, even
for folks that mainly just want the Python runtime, and then will use
pip to install everything else they need.

However, I think modularity gives us access to a more flexible
approach: we can set up the F28 System Profile to *default* to the 3.6
Python App Runtime stream, allowing the sequence of updates to be:

1. In Fedora 28, we split the Python 3 runtime module into two pieces:
- the App Runtime, with a 3.6 stream
- the Integrated Runtime, with an f28 stream that depends on AppRuntime:3.6
2. After 3.7 hits beta in January 2018, we add a "3.7" stream to the App Runtime
3. After 3.7 is released, we start building a new F28+3.7 app
development container
- in this configuration, most system packages that need Python 3 can't
be installed
4. In Fedora 29, we add an f29 stream to the integrated runtime that
depends on AppRuntime:3.7
- now it would be the F29+3.6 configuration that prevents use of
system packages that need Py3

> I don't think "Integrated Python Modules" are necessary in Fedora.

Whether they're necessary or not depends on whether we enable the
F28+3.7 configuration: if we allow that, then we need a way to make it
clear that in that configuration, the only Python-3-based system
packages you can install are the ones that rely on
/usr/libexec/platform-python instead of /usr/bin/python3 (since the
others ran their integration testing against the default 3.6 stack).

> Also, the names are a mouthful; let's revise naming after we get the
> semantics down :)

I'm reasonably happy with App Runtime (since I stole that directly
from "OpenShift Application Runtimes", which is the downstream use
case I'm interested in making easier to maintain), and I think
"Integrated Python" or "Integrated Runtime" accurately conveys the
significance of the proposed stream mapping between Fedora versions
and Python versions ("F28 integration testing uses Python 3.6", "F29
integration testing uses Python 3.7", etc).

For the Testing Runtimes, we could likely just drop the
"interoperability" qualifier, giving:

- Platform Python: the private, always installed, potentially
non-standards-compliant, Python that dnf uses
- Integrated Python: an optional standards-compliant Python used for
development and automated scripting *of* Fedora
- Application Python: used to run Python apps *on* F

Re: Modularity, container images, and the default Python stack(s)

[Nick Coghlan]

On 24 August 2017 at 19:02, Petr Viktorin <pvikt...@redhat.com> wrote:
> On 08/24/2017 10:13 AM, Nick Coghlan wrote:
>> My current thinking based on that discussion is that we're actually
>> going to need a module set that looks like this for F28+:
>>
>> * Platform Python (already planned for F27)
>>- part of the Platform module
>>- stream names match Fedora releases (f26, f27, etc)
>
> Stream names match the Platform module. We follow its policy here, even when
> it changes.

Oh, interesting, I had that backwards (I thought the planned F27
Python modules were the ones named after upstream Python releases).

That means the current module definitions would be closer to what I'm
calling the "Integrated Python Modules".

>> * Python Application Runtime Modules (already planned for F27)
>>- one module for Python 2 and one for Python 3
>>- stream names match upstream Python releases (2.7, 3.6, etc)
>>- Application Runtime Modules conflict with the corresponding >
>> Integrated Python Module
>>- provide "/usr/bin/python2" and "/usr/bin/python3" respectively
>>- also satisfy "python2-*" and "python3-*" RPM dependencies
>>- only the Python 2 module satisfies "python-*" RPM dependencies (for
>> now) > * Integrated Python Modules
>>- one module for Python 2 and one for Python 3
>
> These need to be parallel installable, to support tox. So we need separate
> modules for each Python release.

Aye, I agree that will be a good way to tackle the parallel
installable versions that *don't* define "/usr/bin/python3". However,
for containers designed to run Python applications (web or otherwise),
we *will* want mutually conflicting streams that define their symlinks
and RPM provides based only on the Python major version.

So lets make those extra modules their own distinct category:

* Python Interoperability Testing Modules
  - one module per Python X.Y release
  - only one stream per module
  - conflict with the corresponding Application Runtime stream
  - provide "/usr/bin/pythonXY" and "/usr/bin/pythonX.Y"
  - do NOT satisfy "python2-*" and "python3-*" RPM dependencies

And looking at their current implementation in Fedora, one notable
difference between them and the Application Runtime streams is that
these would just use their bundled pip and setuptools, rather than
splitting those out the way the regular packages do.

>>- stream names match Fedora releases (f26, f27, etc)
>>- each depends on an *exact* version of the corresponding Python
>>  Application Runtime module
>
>
> Huh? You just said Python Application Runtime Modules would conflict with
> this.

Sorry about that - when I started writing the email, my plan was to
have them conflict with each other, but as I wrote that initial
version up I went "Hang on, couldn't I use a stream dependency here
and rely on *that* to trigger a conflict if you had an Integrated
Python module installed and then tried to switch to an unsupported
runtime stream, or vice-versa?"

>> * Default Python Module
>>- provides /usr/bin/python (and nothing else)
>>- stream names: no-default, python3-default, python2-default
>
> +1
> I guess this would contain a "python" RPM, containing just a /usr/bin/python
> symlink, which would added to non-modular Fedora as well.

Pretty much, yeah. The "no-default" stream would be a bit different,
as in that case it would install a shell script that printed out a
custom error message.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: Modularity, container images, and the default Python stack(s)

[Nick Coghlan]

On 21 August 2017 at 19:46, Petr Viktorin <pvikt...@redhat.com> wrote:
> On 08/18/2017 01:38 PM, Nick Coghlan wrote:
>> Does that approach sound sufficiently plausible to folks that I can
>> use it to provide feedback to the folks working on the modularity
>> tooling as to the capabilities we think we'll need?
>
> That sounds like it would work. But yes, please talk to the Modularity WG to
> see if modules can be made to work that way.

Aye, this draft proposal was essentially me figuring out what I think
we're going to want/need for Python before I pitched the related
feature ideas to the Modularity WG :)

The proposal then became something I could point to to say "This is
the problem I'm trying to solve" while we discussed various possible
ways of solving it.

I finally had a discussion with Langdon about it today, and he really
isn't a fan of my idea of trying to enhance the modularity tooling to
natively support parallel installation of streams - he'd strongly
prefer that we stick to the idea of "one active stream per module per
system" (at least for now), and then rely on separate SRPMs and/or the
Software Collections parallel installation layout to handle use cases
like tox.

(Note: I'll get the properly formatted proposal updated by tomorrow,
so feel free to wait for that if the email version below is a bit hard
to read)

My current thinking based on that discussion is that we're actually
going to need a module set that looks like this for F28+:

* Platform Python (already planned for F27)
  - part of the Platform module
  - stream names match Fedora releases (f26, f27, etc)
* Python Application Runtime Modules (already planned for F27)
  - one module for Python 2 and one for Python 3
  - stream names match upstream Python releases (2.7, 3.6, etc)
  - Application Runtime Modules conflict with the corresponding
Integrated Python Module
  - provide "/usr/bin/python2" and "/usr/bin/python3" respectively
  - also satisfy "python2-*" and "python3-*" RPM dependencies
  - only the Python 2 module satisfies "python-*" RPM dependencies (for now)
* Integrated Python Modules
  - one module for Python 2 and one for Python 3
  - stream names match Fedora releases (f26, f27, etc)
  - each depends on an *exact* version of the corresponding Python
Application Runtime module
* Default Python Module
  - provides /usr/bin/python (and nothing else)
  - stream names: no-default, python3-default, python2-default

The scope for F27 would specifically be Platform Python and the Python
Application Runtime Modules, with the separate Integrated Python
Modules being defined for F28+

At least for now, the Python XY stacks for Fedora and EPEL, as well as
the Python SCLs, would be treated as something generated downstream
from the app runtime modules, rather than as something that the
modularity tooling would necessarily handle natively.

The general idea is that it would then be up to other modules to
decide whether they specifically needed the Integrated Python Module
with all the related system bindings (in which case they'd either
depend on that module, or depend on another RPM that depended on it),
or were prepared to cope with any installed version of Python 3 (in
which case they'd just use normal RPM level "python3*" dependencies).

Right now, the difference between the Integrated Python Modules and
the Python Application Runtime Modules isn't particularly obvious, but
it hopefully becomes clearer once we consider questions like "Who will
decide when to rebase to Python 3.7?" and "When will a particular
stream stop being updated?".

For the Integrated Python Modules, those are Fedora level design
decisions, as reflected in the stream names being based on Fedora
versions. This means that for a system container, or a traditional
mutable installation, you'd be able to continue to rely on a shared
Python installation with all the operating system level bindings for
things like the RPM database, without Fedora package maintainers
having to provide prebuilt bindings for arbitrary Python versions.
You'd only change streams when you changed base Fedora versions, and
the update cycle for these streams would match the Fedora release
cycle (i.e. each stream would receive updates for 13 months from the
date of the corresponding Fedora release).

By contrast, for the Python App Runtime Modules, when to rebase is an
application developer level decision, as reflected in the stream names
being based directly on Python versions. This means that for an
application container image, you'd be able to select an arbitrary
Python version of your choice, but in doing so, you'd potentially be
giving up ready access to pre-built bindings for various system APIs
(if your choice of stream didn't match the dependencies declared in
the Integrated Python Module). For these streams, the update cycle
would match that 

Re: Setting up a fedora-python pagure.io group

[Nick Coghlan]

On 21 August 2017 at 19:53, Petr Viktorin <pvikt...@redhat.com> wrote:
> On 08/18/2017 12:06 PM, Nick Coghlan wrote:
>> I couldn't get Pagure's webhooks to work properly (see
>> https://pagure.io/pagure/issue/2522), so revising the revised plan:
>> could someone with the appropriate access create me a fedora-python
>> repo under https://github.com/fedora-python and grant me admin access
>> to it? :)
>
> Sure. Added you; you should be getting notifications about this.

Thanks! Repo updated: https://github.com/fedora-python/fedora-python

If you let me know your ReadTheDocs username, I'll add you to the
maintainer list for that as well.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Modularity, container images, and the default Python stack(s)

[Nick Coghlan]

Hi folks,

[Note: this may not make much sense to folks that haven't been closely
following the Modular Server work for F27. If that's you, sorry -
hopefully this will be more comprehensible by the time I officially
propose it for F28, as the relevant terminology becomes more widely
understood. In the meantime, check out
https://docs.pagure.org/modularity/ to learn more]

I'm working on a draft proposal for a "Default Python" module (see
https://fedora-python.readthedocs.io/en/latest/plans/default-python-module/
for details), and hit an interesting challenge when it comes to
defining the containing images that we want to be able to build from
our module definitions.

A quick summary of what I'm expecting we'll have by F28:

- a separate platform-python binary in the Platform module
- python2 modules with a full 2.7 stream and a partial
interpreter-only 2.6 stream
- python3 modules with a full 3.6 stream and a TBD set of other streams
- a default-python module to switch between no-default,
python2-default and python3-default

With those modules defined, a minimal Fedora container image will only
include platform-python, but we'd at least also have Python3-F28, and
Python2-F28 images that included the respective user Python stack in
addition to the platform Python runtime.

The part I'm struggling with is this: Python 3.7.0 is expected to be
released in June 2018, while Fedora 28 is due out in May 2018. It
would be *really nice* to be able to define a Fedora 28 based Python
3.7 container where "/usr/bin/python" and "/usr/bin/python3" were both
references to "/usr/bin/python3.7".

The challenge with actually doing that lies in the "/usr/bin/python3"
symlink: integrated userspace Python applications in F28 are going to
expect that to still refer to the Python 3.6 stream.

One way I could potentially see this working:

* the normal non-conflicting setup is as follows:

  * the python3 3.6 stream includes a "/usr/bin/python3" symlink
  * the other python3 3.x streams do *not* include such a symlink &
hence don't conflict
  * the default-python python3-default stream does *not* include such
a symlink & hence doesn't conflict

We then add some more streams to the default-python module that *do*
include a "/usr/bin/python3" symlink: python3.5-default,
python3.7-default, etc

The trick with these streams is that they would *conflict* with the
regular python3 3.6 stream, due to the disagreement over what
"/usr/bin/python" means. That dependency resolution conflict would
then mean that have a non-default version of Python 3.x configured as
the default when defining your container would *prevent* you from
including any regular userspace Python components - you'd only be able
to include the ones built specifically for that stream.

Does that approach sound sufficiently plausible to folks that I can
use it to provide feedback to the folks working on the modularity
tooling as to the capabilities we think we'll need?

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: Setting up a fedora-python pagure.io group

[Nick Coghlan]

On 15 August 2017 at 19:44, Nick Coghlan <ncogh...@gmail.com> wrote:
> So I decided to set up a build on ReadTheDocs instead, and that looks
> to have just worked, including the logo rendering:
> https://fedora-python.readthedocs.io/en/latest/

I couldn't get Pagure's webhooks to work properly (see
https://pagure.io/pagure/issue/2522), so revising the revised plan:
could someone with the appropriate access create me a fedora-python
repo under https://github.com/fedora-python and grant me admin access
to it? :)

With the Pagure folks being busy with the dist-git integration, that
makes more sense to me than fiddling about trying to get Pagure's
webhooks working.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: Setting up a fedora-python pagure.io group

[Nick Coghlan]

On 11 August 2017 at 17:34, Nick Coghlan <ncogh...@gmail.com> wrote:
> The relocated "default Python module" proposal is at
> https://docs.pagure.org/fedora-python.fedora-python/default-python-module.html
>
> I'm not super fond of that auto-generated URL, nor do I like the
> client-driven update process for pushing changes live, so I'm
> wondering if ReadTheDocs might be a better option for publication,
> rather than using Pagure's native static-site hosting.

After updating the theme to be more Fedora-ish (i.e. I took the Fedora
Modularity theme and removed the bits I didn't need), I ran into a
problem where the Pagure-hosted docs weren't rendering the Fedora
Loves Python logo correctly, even though that worked fine when
browsing the built docs locally, and even though choosing "View Image"
from the Firefox context menu took you to the correct SVG file.

So I decided to set up a build on ReadTheDocs instead, and that looks
to have just worked, including the logo rendering:
https://fedora-python.readthedocs.io/en/latest/

Since RTD's webhook support also solves the autobuild-on-commit
problem, I'm thinking that may be the better way to go (I'll just need
some other folks RTD usernames so I can add you as co-maintainers to
the project)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: Setting up a fedora-python pagure.io group

[Nick Coghlan]

On 2 August 2017 at 21:34, Nick Coghlan <ncogh...@gmail.com> wrote:
> While working on
> https://fedoraproject.org/wiki/User:Ncoghlan/Default_python_module, I
> started getting annoyed at the lack of decent review and commenting
> features in MediaWiki, so prompted by
> https://docs.pagure.org/modularity/ I went ahead and started a
> fedora-python repo at https://pagure.io/fedora-python/fedora-python,
> so we'll be able to use https://docs.pagure.org/fedora-python/ as a
> custom docs site.
>
> I deliberately chose the Pagure group name to match the existing
> GitHub one, but since my main intended use case is to review & discuss
> Fedora design proposals before they make their way into the formal
> Fedora change management processes, it seemed more appropriate to host
> the repo on Fedora managed infrastructure.
>
> Currently myself & Petr Viktorin are the only group members, but
> that's just because I haven't added anyone else yet, not because I
> expect the member list to actually stay that small.

Progress update here: the repo has some useful content!

It's a Sphinx project, currently being published here:
https://docs.pagure.org/fedora-python.fedora-python/

The relocated "default Python module" proposal is at
https://docs.pagure.org/fedora-python.fedora-python/default-python-module.html

I'm not super fond of that auto-generated URL, nor do I like the
client-driven update process for pushing changes live, so I'm
wondering if ReadTheDocs might be a better option for publication,
rather than using Pagure's native static-site hosting.

Cheers,
Nick.

P.S. Now that I have the basic site up, I'd also like to switch to the
more Fedora-ish Sphinx theme used in the Modularity docs:
https://docs.pagure.org/modularity/docs.html

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: Naming policy for application vs library packages in Python

[Nick Coghlan]

On 10 August 2017 at 10:49, Ben Rosser <rosser@gmail.com> wrote:
> As a counter-proposal I would suggest that, as we currently do, we
> require the python3-prefix to be provided by the package, but
> explicitly leave it to the packager+reviewer's discretion whether or
> not the prefix must be part of the real name, too. Some other
> languages do already do this: nodejs [3] and ocaml [4] both explicitly
> have "if this primarily provides a tool or application" clauses in
> their naming guidelines. I think it makes sense to have something
> similar for Python, to help avoid confusion.

This is technically already the de facto policy (if it wasn't, a lot
of Fedora's build tooling would have very different names), but the
Python packaging policy doesn't spell it out explicitly.

So while I think you're right that we should make it officially that
it's OK for application packagers to let it be an implementation
detail whether something is written in Python or not, I also think we
would ideally take more steps than we do today to make sure that such
use of Python actually *is* an implementation detail, rather than
something that can leak out and cause compatibility issues with
upstream Python modules.

For example, some time ago, the "mock" command line tool had to rename
its supporting Python package to "mockbuild" in order to resolve a
file conflict when the python-mock package was first added to the
distro.

In that vein, the potentially more robust approach I've been
considering for managing that kind of situation is the idea of
recommending that these "the use of Python is an implementation
detail" applications be restructured to take advantage of the native
virtual environment support in Python 3 in order to put their
application specific modules in a private virtual environment, rather
than installing them directly into the system Python as generally
importable packages.

Specifically, having venv available by default allows us to do this as
part of an RPM build:

1. Use "python3 -m venv --system-site-packages --without-pip
" to create an empty symlinked virtual environment that
can see system packages
2. From *outside* the venv, run "pip3 install --prefix "
to install the application specific module(s)
3. Copy any generated "bin" scripts out into the FHS binary directory
(their shebang lines will automatically be set up to auto-activate the
private venv, since they'll reference the venv's Python symlink, *not*
the system level path)

Having a private venv like that available would also give application
packagers a relatively clean way to inject "before anything else runs"
behaviour via *.pth files, which will let them do things like
implicitly adjusting sys.path to include additional directories, as
well as tinkering with __main__.__requires__ in order to manipulate
the behaviour of pkg_resources parallel loading support.

The main reason I hadn't gotten around to actually proposing that is
that there are a few significant open questions where I'm not sure
what the right answer would be:

1. Where would these private venvs actually live?
2. Does the answer to (1) differ for pure Python app modules vs binary
extension modules?
3. Would we allow daisy chaining of these private venvs via *.pth
files? (see https://github.com/berdario/pew#add)
4. How would we make this manageable across Fedora/EPEL/SCLo spec files?

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: Finalizing Fedora's Switch to Python 3

[Nick Coghlan]

On 29 July 2017 at 00:25, John Dennis <jden...@redhat.com> wrote:
> I made this comment previously but because I think it's important I'm going
> to repeat it.
>
> Fedora's Python version migration needs to be coordinated with RHEL.
>
> Yes I know Fedora is independent of both Red Hat and RHEL but the real world
> reality is spec files are shared between both. At the moment you cannot
> easily share a spec file between the two, this leads to maintaining two
> independent spec files for something that ought to be nearly identical and
> increases the burden on package maintainers and increases the opportunity
> for errors.

Aye, I agree we should be actively seeking to make single-spec
feasible across Fedora/RHEL/CentOS, at least in combination with EPEL
(without the EPEL dependency, it's hard to consistently make new
Fedora level macro definitions available to older EL releases).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: Finalizing Fedora's Switch to Python 3

[Nick Coghlan]

On 28 July 2017 at 03:15, Zbigniew Jędrzejewski-Szmek <zbys...@in.waw.pl> wrote:
> Do you think it'd be possible to script the python-foo to python2-foo
> renaming? If yes, then maybe it'd make sense to just get some pps to
> do it in rawhide now and get the "easy" part done with? That should
> significantly cut down on the number of misnamed packages and let
> packagers spend their times on the ones where the automatic way is not
> obvious.

I was going to ask whether it might be possible to tweak
http://fedora.portingdb.xyz/ to also report on compliance with the
naming policy, but then I went and saw that it *does* already report
on that: http://fedora.portingdb.xyz/namingpolicy/

While it also turns out the wiki page already links to that page, it
may be good to call it out a second time in a "How can I help?"
section.

Checking an initial sampling of spec files (python-d2to1,
python-BeautfulSoup, python-amqplib) suggests to me that a script
implementing the following rules might offer a reasonably start point,
at least for Python-2-only modules that are remaining Python-2-only:

- immediately before the first BuildRequires or Requires entry, add a
%package section header for "-n python2-" (where "" is the
lowercased package source name with any "python-" prefix stripped)
- add a %python_provides entry after the new package header in
accordance with the current guidelines
- if the original package provided a non-lowercase "python-*"
provides, remote it and add a second %python_provides with the
original capitalisation
- if the source package lacks the "python-" prefix, add a virtual
provides for the unqualified package name
- add a "-n python2-" qualifier to any currently unqualified
description and files sections

A script like that may even do a tolerable job for packages that *do*
offer Python 3 subpackages (since those will already have qualifiers,
and will necessarily appear after any unqualified runtime and build
requirements for the default subpackage).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: Adjusting the way Fedora runs Python's test suite

[Nick Coghlan]

On 20 July 2017 at 16:12, Bohuslav Kabrda <bkab...@redhat.com> wrote:
> On Thu, Jul 20, 2017 at 7:51 AM, Nick Coghlan <ncogh...@gmail.com> wrote:
>> * checking for refleaks means we can't run parallel tests across
>> multiple processes. If we were to move refleak testing out to an
>> integration test in Taskotron, then the tests could be run with the
>> "-j4" option, significantly reducing the time needed to run the %check
>> step
>>
>
> If it's possible to move this testing to taskotron, then it sounds ok to me.

Based on my changes to the sclo-python spec, I can confirm that
passing "-j0" (which autoscales to the number of CPUs +2 to allow for
blocking tests) cuts my local RPM build times from just under 40
minutes to just over 15.

So I'll use that to speed up the sclo-python bootstrapping builds, and
then we can wait for the dist-git move into Pagure and the
introduction of https://fedoraproject.org/wiki/Changes/InvokingTests
before looking at changing the main build.

>> * the `--failfast` option causes the tests to be aborted as soon as
>> one fails. While that doesn't make any difference in the success case,
>> it will mean faster turnaround cycles in the case where one fails. I'm
>> less sure of the value of this one, as the downside is that if you're
>> getting multiple failures, you only find out about them one at a time.
>
> I'd suggest to not use this.

Agreed, especially as it's easy enough to add in to the spec file if
you actually want it in a mock build for some reason.

>> * there's a `--slowest` flag in recent releases that lists the 10
>> slowest tests in a given run. As far as I know, it doesn't add much
>> overhead to the test execution process, and provides an indicator of
>> where to focus efforts if the goal is to make the tests more
>> efficient.
>
> Yeah, I think improving the performance of the slowest tests could be gain
> for everyone, not just for Fedora.

One caveat on this option is that it's a relatively recent addition
that was never backported to the 3.5 branch, so it's only applicable
to builds of 3.6+.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Adjusting the way Fedora runs Python's test suite

[Nick Coghlan]

Hi folks,

While working on bootstrapping a new community sclo-python collection,
I ran into some readability problems with the way the SCL Python spec
file runs the test suite: it uses --verbose mode, which means there's
a lot of noisy output to sort through to find the actual failure when
something goes wrong.

Charalompos pointed out that since the SCL spec file is ultimately
derived from the Fedora one, Fedora would be the best place to file an
issue suggesting some changes, which I've now done:
https://bugzilla.redhat.com/show_bug.cgi?id=1473130

The main suggestion there is one that's a clear win: replace the
current "--verbose" with "-wW" instead, so that the tests are run in
non-verbose mode by default, but if one fails, not only will the test
suite print out any captured output, but it will also immediately
rerun the failed test in verbose mode.

However, I also noted some other possibilities that seem worth discussing:

* checking for refleaks means we can't run parallel tests across
multiple processes. If we were to move refleak testing out to an
integration test in Taskotron, then the tests could be run with the
"-j4" option, significantly reducing the time needed to run the %check
step

* the `--failfast` option causes the tests to be aborted as soon as
one fails. While that doesn't make any difference in the success case,
it will mean faster turnaround cycles in the case where one fails. I'm
less sure of the value of this one, as the downside is that if you're
getting multiple failures, you only find out about them one at a time.

* there's a `--slowest` flag in recent releases that lists the 10
slowest tests in a given run. As far as I know, it doesn't add much
overhead to the test execution process, and provides an indicator of
where to focus efforts if the goal is to make the tests more
efficient.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: Proposed Mass Bug Filing: Renaming "python-" binary packages to "python2-"

[Nick Coghlan]

On 29 June 2017 at 11:39, Adam Williamson <adamw...@fedoraproject.org> wrote:
> On Wed, 2017-06-28 at 16:21 +0200, Iryna Shcherbina wrote:
>> 2) Using `python-` instead of `python2-` in the dependencies for the
>> Python 2 binary RPM [2].
>
> I'm not sure this list is terribly useful, because of the above. There
> are thousands of packages that do this, because the 'python2-' provide
> is not available on some older Fedora release, or on EPEL (and the
> package is maintained for EPEL as well as Fedora). Sprinkling "if (some
> release number condition) then Requires: python2-foo else Requires:
> python-foo" all over your spec is a giant PITA and I for one am not
> very interested in doing it.
>
> IMHO, if there is going to be some kind of requirement that all Python
> requires be explicitly versioned, there needs to be a co-ordinated
> effort to make sure the versioned Provides are available across at
> least EL6, EL7, and all supported Fedora releases *first*.

This was the key concern I raised in response to the initial email,
and our conclusion at the time was:

1. This case does need to be addressed
2. Adding an opaque dependency on buildroot configuration settings
wouldn't be a particularly nice way to handle it, since it forces
every package to switch concurrently, rather than each maintainer
getting to decide when to move from the Python 2 stack to the Python 3
stack for themselves (and that unilateral shift is already going to
happen for unqualified dependency declarations when the virtual
%python_provides macro moves from the Python 2 stack to the Python 3
stack)
3. Ideally, the recommended approach would work for arbitrary RHEL &
CentOS based buildroots, not just those with the EPEL RPM macros
available

The most straightforward solution we came up with was for affected
packages to define their own "%py_prefix" macro that selects the stack
they want to use based on the Python version:

```
# The block below would become the conventional
# "Python stack compatibility" dance for
# EL6, EL7, and Fedora
# Each package can decide for itself which version of
# Fedora had a sufficiently complete Py3 stack for
# them to be able to switch over

# Current EL releases & older Fedora use "python-*"
%if 0%{?el6} || 0%{?el7} || 0%{?fedora} < 25
%define py_prefix python
%if 0%{?el6} || 0%{?el7}
BuildRequires: python-devel
%else
BuildRequires: python2-devel
%endif
%else
# Newer Fedora releases use "pythonX-*"
# A Py2-only project would use "python2" here
%define py_prefix python3
BuildRequires: python3-devel
%endif


# Dependency declarations use stack selected above
BuildRequires: %{py_prefix}-builddep1
BuildRequires: %{py_prefix}-builddep2
Requires: %{py_prefix}-runtimedep1
Requires: %{py_prefix}-runtimedep2
```

For dual stack libraries, the appropriate prefixes to define would be
separate ones for each stack (%py2_prefix and %py3_prefix), and either
leave the latter undefined for systems with no native Py3 stack, or
else set it to rely on EPEL, IUS, or a suitable software collection.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: Should Python 3 macros us UTF-8 locale?

[Nick Coghlan]

On 2 June 2017 at 18:46, Nick Coghlan <ncogh...@redhat.com> wrote:
> On Fri, Jun 2, 2017 at 1:15 PM, Zbigniew Jędrzejewski-Szmek
> <zbys...@in.waw.pl> wrote:
>> It seems that right solution, that would work while the builders are
>> still not F26, would be to configure their locale to C.UTF-8. It'd
>> just be a matter of dropping LANG=C.UTF-8 into /etc/locale.conf.
>> I think a ticket on https://pagure.io/fedora-infrastructure would be
>> the way to request the change.
>
> While this seems like the right next step to me, if it isn't feasible
> for some reason, we may need to reconsider emitting the warning at the
> Python level, and instead just silently change LC_CTYPE to the more
> sensible setting.

A follow-up on this: I just made the change upstream to turn these
warnings off by default, and add a "PYTHONCOERCECLOCALE=warn" runtime
flag to let folks opt in to the warnings as a tool for debugging
potential locale coercion and locale compatibility problems. We were
simply getting far too many false positives where code that was
previously working fine (because it was genuinely only needing to
handle 7-bit ASCII inputs and outputs) was failing solely due to the
new warnings, not because anything was actually broken.

There are also some significant improvements to the test suite to make
sure that it's properly covering the expected behaviour when there
*aren't* any coercion target locales available. While that's not
applicable to Fedora directly, it *will* apply to any future Python
3.6 SCL, as well as to any python36 packages put together for EPEL.

Tracker issue: https://bugs.python.org/issue30565
Squashed commit:
https://github.com/python/cpython/commit/eb81795d7d3a8c898fa89a376d63fc3bbfb9a081

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: Should Python 3 macros us UTF-8 locale?

[Nick Coghlan]

On Fri, Jun 2, 2017 at 1:15 PM, Zbigniew Jędrzejewski-Szmek
<zbys...@in.waw.pl> wrote:
> It seems that right solution, that would work while the builders are
> still not F26, would be to configure their locale to C.UTF-8. It'd
> just be a matter of dropping LANG=C.UTF-8 into /etc/locale.conf.
> I think a ticket on https://pagure.io/fedora-infrastructure would be
> the way to request the change.

While this seems like the right next step to me, if it isn't feasible
for some reason, we may need to reconsider emitting the warning at the
Python level, and instead just silently change LC_CTYPE to the more
sensible setting.

The accepted version of the PEP upstream *only* sets LC_CTYPE, so the
chance of unintended side effects from the coercion is much lower that
it was with the approach that made the F26 Beta cut-off (which also
sets LANG).

Cheers,
Nick.

-- 
Nick Coghlan
Red Hat Platform Engineering, Brisbane
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Speculative idea: incorporating venv into our Python application packaging advice

[Nick Coghlan]

My day job currently involves working on a Python CLI (and potentially
a backing socket-activated service) that needs to run across
Fedora/RHEL/CentOS/SCLs, *without* accidentally exposing a Python
level API that we might inadvertently end up needing to support.

(Note: this CLI is not being, and will likely never be, proposed for
incorporation into Fedora itself - it's a tool to help migrate
applications between different operating system versions without doing
an in-place upgrade, so the update cycles need to be decoupled from
those of the operating system itself)

At the moment, we offer two different ways of installing that:

1. via pipsi, which uses the system Python, but has no access to
system level Python libraries
2. via RPM, which has access to system level Python libraries, exposes
the application's internal libraries for import by other applications
(which we don't really want to do) and also requires that *all*
dependencies be available as system packages

Both approaches have significant downsides:

* the pipsi based approach is *too* decoupled from the host OS,
installing things into the virtual environment even when a perfectly
acceptable version is already installed and maintained as a system
package. It also means we can't benefit from distro level patches to
packages like requests, so the app is decoupled from the system
certificate store
* the RPM based approach isn't decoupled from the OS *enough*, so we
can't readily do things like selectively installing private copies of
newer versions of dependencies on RHEL/CentOS, while using the system
packages on Fedora. It also means the Python packages implementing the
application itself are globally available for import rather than only
being usable from within the application

While we haven't implemented it yet, the approach I'm considering to
tackle this problem [1] involves integrating creation of an
app-specific private virtual environment into the definition of the
application RPM, with the following details:

* unlike pipsi, this virtual environment would be configured to allow
access to the system site packages, giving us the best of both worlds:
we'd use system packages if readily available, otherwise we'd stick
our own pinned dependency in the virtual env and treat it as part of
the application (and hence the app developers' responsibility to keep
up to date)
* we'd come up with some way of turning the Python level dependencies
into additional entries in the RPM's Sources list, and then turn those
into a local sdist index during the %prep phase. That way, we'd
support offline builds automatically, and be well positioned to have
pip autofill any gaps where system level dependencies didn't meet the
needs of the application
* we'd deliberately omit some of the packages injected into the
virtual environment from the resulting RPM (most notably: we'd either
remove pip, wheel, and setuptools, or else avoid installing them in
the first place)

Where I think this idea crosses over into being a suitable topic for
the Fedora Python SIG relates to the current modularity initiatives
and various problems we've faced over the years around separating the
challenges of "provide an application that happens to be written in
Python" and "provide a supported Python API as part of the system
Python installation".

Some examples:

* the helper library for the "mock" CLI tool had to be renamed to
"mockbuild" to fix a conflict with the upstream "mock" testing library
* despite officially having no supported public API, people still
write "import pip" instead of running the pip CLI in a subprocess
* ditto for the yum CLI (and even for DNF, some non-trivial changes
were recently needed to better separate the "supported for third party
use with defined backwards compatibilty guarantees" APIs from the "for
internal use by the DNF CLI and may change at any time" APIs

All of those could have been avoided if the recommended structure for
"applications that happen to be written in Python" included a virtual
environment that isolated the "private to the application" Python
modules (including the application's own source code) from the
"intended for third party consumption" public APIs.

In the near term, my own focus is going to be on figuring out the
details of this structure specifically for LeApp, but I wanted to
raise the notion here early so I didn't go down any paths that would
later prove to be an absolute deal-breaker for updating the distro
level recommendations.

Cheers,
Nick.

[1] https://github.com/leapp-to/prototype/issues/126


-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: What is your opinion on "sudo pip" fix for Fedora 27?

[Nick Coghlan]

On 26 April 2017 at 19:19, Michal Cyprian <mcypr...@redhat.com> wrote:
> The other possibility is to limit the pip install location change
> to distutils and pip [2]. This is the "safer" option, but does
> not cover all corner cases. For example, Python software built
> locally using cmake or similar tools will be installed into
> /usr/lib and can conflict with system tools. Debian chose to
> implement similar solution.

Re-reading https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe
after the discussion on the main devel list, I have a few further
comments on this point:

1. I think the tone of the change proposal should be softened to say
it's about make "sudo pip install" *safer*, rather than about making
it absolutely safe - you can still break your system if you're not
careful, the layout change just makes it easier to recover if you do
(at least in most cases)
2. The question of installation directories other than site-packages
(mostly notably /usr/bin) should be explicitly covered, and either
also moved under /usr/local, or else a rationale given for not moving
them yet
3. The proposal should be more explicitly scoped by specifying some
sequences of operations that you want to make safe, such as:

$ sudo pip3 install contextlib2 && sudo pip3 uninstall contextlib2
$ sudo pipsi install ansible && sudo pipsi uninstall ansible

(Getting the latter example to be safe would require changing the
install directory to /usr/local in a way that "pipsi" detects)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: What is your opinion on "sudo pip" fix for Fedora 27?

[Nick Coghlan]

On 1 May 2017 at 22:47, Nico Kadel-Garcia <nka...@gmail.com> wrote:
> On Sun, Apr 30, 2017 at 10:30 PM, Nick Coghlan <ncogh...@gmail.com> wrote:
>> If the intended benefit of this change remains unclear, it may help to
>> focus on a specific concrete case, which would be that the following
>> operations should be completely indistinguishable at the system level
>> from not having done anything at all (except in the sudo logs):
>>
>> $ sudo pip3 install contextlib2
>> $ sudo pip3 uninstall contextlib2
>
> And this successfully ignores *all* the dependencies which contextlib2
> may add at installation time.

There's a reason I chose contextlib2 as my example rather than
something more complex like ansible - I'm the maintainer of
contextlib2, and I know it doesn't have any runtime dependencies (and
likely never will, since it's a stdlib module backport).

Given the proposal at hand though, writing a
`remove-pip-installed-modules` cleaner utility becomes a lot simpler
that it is today, since it just needs to clear out any Python packages
it finds in /usr/local (based on the Python level installation
records), rather than needing to interact with the RPM database,
figure out which system packages may have potentially been corrupted
and reinstall them.

If the proposal is adjusted to also affect other installation
directories (like the one for scripts), it will also have the
significant benefit that any pip installed binaries will go into
`/usr/local/bin`, so they'll appear on the default path for all
regular users, but *won't* appear on the default path for "root".

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: What is your opinion on "sudo pip" fix for Fedora 27?

[Nick Coghlan]

On 1 May 2017 at 09:59, Nico Kadel-Garcia <nka...@gmail.com> wrote:
> Wouldn't it be more effective, more safe, and more efficient to
> improve the existing support for doing "the right thing", rather than
> trying to outsmart a build and deployment system that is essentially
> outside your control? To encourage the use of "pyvenv" for local such
> modules, in a local userspace, rather than as a default system
> version?

This is already happening, and is entirely independent of the
discussion at hand (e.g. see [1], a presentation I did with Graham
Dumpleton at Red Hat Summit/DevNation last year advocating for Red
Hat's customers to adopt exactly this model). On Fedora's Developer
Portal, virtual environments share the front page of the Python
section with running Python itself:
https://developer.fedoraproject.org/tech/languages/python/python-installation.html

Responding to "We want to make running 'sudo pip install' less harmful
and easier to recover from when people do it by mistake" with "We
should discourage the use of 'sudo pip install'" remains an
unproductive non sequitur - we *know* it's inherently fragile, and
requires deep knowledge of both Python packaging and RPM packaging to
do reliably. However, Red Hat can't (and won't) answer a commercial
support call from a distressed customer with "Ooh, you shouldn't have
done that - sucks to be you" (although these supportability problems
*are* one of the reasons we only ship "pip" commercially through
Software Collections, and not as part of the system Python runtime).

Fortunately, we have the ability to influence *all* of the moving
parts (upstream pip, upstream CPython, downstream distros) here, and
help to incrementally adjust their respective default behaviours so
that the key risks involved in running "sudo pip install" are
comparable to those involved in running "curl $REMOTE_SCRIPT | sudo
sh".

If the intended benefit of this change remains unclear, it may help to
focus on a specific concrete case, which would be that the following
operations should be completely indistinguishable at the system level
from not having done anything at all (except in the sudo logs):

$ sudo pip3 install contextlib2
$ sudo pip3 uninstall contextlib2

At the moment, that's *not* the case if you had previously done "sudo
dnf install python3-contextlib2", as both of the above commands will
mess with the system contextlib2 packages. With the proposed changes,
contextlib2 may behave unexpectedly while the non-system version is
installed, but removing the pip installed version will be sufficient
to restore the affected machine to its previous state. (I chose
contextlib2 as the example here, rather than something more complex
like ansible, as pip installing components with dependencies makes the
cleanup process more complicated. However, the /usr/local/ split helps
there as well, by making the installed-via-pip versions easier to
identify even without checking the PEP 376 installation metadata)

Cheers,
Nick.

[1] 
https://www.slideshare.net/ncoghlan_dev/developing-in-python-on-red-hat-platforms-devnation-2016

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: What is your opinion on "sudo pip" fix for Fedora 27?

[Nick Coghlan]

On 28 April 2017 at 02:01, Charalampos Stratakis <cstra...@redhat.com> wrote:
> Fedora's equivalent to dirtbike is rewheel [0][1] and it would be great if 
> these projects could be unified somehow, however that's a separate 
> conversation for another time.
>
> I'd be happy to take a closer look at dirtbike and maybe initiate some 
> discussions when I get some free time.

My recollection from the last time I looked at this was that where
rewheel assumes that the Python level PEP 376 package metadata will be
available, dirtbike instead queried the operating system level
metadata.

So while they're technically solving the same problem (injecting a
system level Python package or bindings into a virtual environment),
they took radically different paths to get there.

The biggest benefit I see to the dirtbike approach is that it has the
potential to work with Python bindings that aren't built with the
standard Python packaging tools, and hence don't have any PEP 376
metadata.

Conversely, the biggest benefit I see to the rewheel approach is that
it lets system packages more clearly indicate whether or not they want
to support rewheel by including or omitting the PEP 376 metadata.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: What is your opinion on "sudo pip" fix for Fedora 27?

[Nick Coghlan]

On 27 April 2017 at 23:04, Daniel P. Berrange <berra...@redhat.com> wrote:
> On Thu, Apr 27, 2017 at 04:32:09PM +1000, Nick Coghlan wrote:
>> Their approach means that any harm caused by "sudo pip install X" can
>> subsequently be fully reversed by doing "sudo pip uninstall X".
>>
>> At this moment, this is NOT true for Fedora and derivatives. Instead,
>> the remediation step here is "sudo pip uninstall X && sudo dnf
>> reinstall " where you have to:
>>
>> 1. Figure out what "" needs to be
>> 2. Hope that whatever you broke didn't affect your ability to run
>> "sudo dnf reinstall"
>
> Yep, recovering the system python install to a pristine state after
> 'devstack' has done its stuff is very painful right now :-(
>
> devstack was originally written for Ubuntu systems, so I guess they
> don't suffer as much due to the Debian specific change you describe
> above.
>
> Which location takes priority in Debian if the same module is installed
> in both places ? The DPKG location, or the Pip location ? Presumably it
> would have to be the pip version that takes priority if you want it to
> be usable for updating the (likely older) distro provided versions.

Aye, /usr/local/lib has priority:

$ docker run --rm ncoghlan/debian-python python3 -m site
sys.path = [
'/',
'/usr/lib/python3.4',
'/usr/lib/python3.4/plat-x86_64-linux-gnu',
'/usr/lib/python3.4/lib-dynload',
'/usr/local/lib/python3.4/dist-packages',
'/usr/lib/python3/dist-packages',
]
USER_BASE: '/root/.local' (doesn't exist)
USER_SITE: '/root/.local/lib/python3.4/site-packages' (doesn't exist)
ENABLE_USER_SITE: True

User level package installs take priority over system level ones
(regardless of distro), so the current container build use case for
root level pip installations can typically be met equally well by
running as a non-root user inside the container and doing "pip install
--user ...". Unfortunately, as with the venv option, there are a *lot*
of container build scripts out there that don't do that :(

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: What is your opinion on "sudo pip" fix for Fedora 27?

[Nick Coghlan]

On 26 April 2017 at 19:19, Michal Cyprian <mcypr...@redhat.com> wrote:
> The other possibility is to limit the pip install location change
> to distutils and pip [2]. This is the "safer" option, but does
> not cover all corner cases. For example, Python software built
> locally using cmake or similar tools will be installed into
> /usr/lib and can conflict with system tools. Debian chose to
> implement similar solution.
>
> I would like to ask which solution would work for your applications
> better, and what is in your opinion the right way to go.

From my perspective, the main goal of the change is to make it
possible to fully recover from "sudo pip install " by doing
"sudo pip uninstall ", and the approach Debian took is
sufficient to achieve that in almost all cases. So +1 from me for
replicating something similar to Debian's approach in Fedora rather
than blazing new trails by fiddling with `sys.prefix` directly.

For the latter concern with CMake/autotools/Scons/etc, anyone doing
local builds of C/C++ applications already needs to be careful not to
be interfere with system packages, so I'm comfortable leaving that in
the category where running C/C++ build system install commands outside
a package build process without due care and attention is a generally
bad idea (e.g. a "sudo make altinstall" of the Python 3.5 maintenance
branch will clobber the system Python binaries in F25, and a "sudo
make install" from the main Python development branch will mess with
the default symlinks if you didn't set a suitable `/opt/` prefix).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: What is your opinion on "sudo pip" fix for Fedora 27?

[Nick Coghlan]

destructive
when run interactively at a shell prompt, or as part of a
configuration management script.

>> The adjustment of the behavior can be done on different levels.
>> The first option is to set the sys.prefix variable to /usr/local
>> when the interpreter is initialized. This will affect
>> all the install methods, but the solution can cause some
>> problems in applications that rely on the value of sys.prefix.
>> A prototype of this implementation can be seen here [1].
>
> I'm sorry, but ouch. Please stop trying to implement "pip install"
> as a system installation tool.

That is not the goal of the change.

> The effort would be better spent
> upgrading and enhancing "py2pack".

Improving the level of commonality and collaboration between Fedora &
Mageia's pyp2rpm and openSUSE's py2pack would indeed be desirable, but
it's entirely orthogonal to the harm mitigation exercise being
discussed here.

>> The other possibility is to limit the pip install location change
>> to distutils and pip [2]. This is the "safer" option, but does
>> not cover all corner cases. For example, Python software built
>> locally using cmake or similar tools will be installed into
>> /usr/lib and can conflict with system tools. Debian chose to
>> implement similar solution.
>
> It's local environment specific, unstable software sensitive to
> arbitrary and unmanaged upstream dependency updates, and extremely
> sensitive to local system pip modules. It *does not belong* in
> /usr/lib. Because the components are modular and bundled in a non-RPM
> compatible fashion, it behooves developers to use a tool to segregate
> the tools as much as feasible from the Fedora underlying
> infrastructure. i.e., use pyvenv to build them in a contained
> environment segregated from the system files.

Helping users that are already doing the right thing isn't the problem
at hand - this is about making it easier for users to recover from
doing the *wrong* thing.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Status update for PEP 538 (coercing the legacy C locale to C.UTF-8)

[Nick Coghlan]

Hi folks,

The Fedora 26 Alpha currently includes our downstream patch
backporting PEP 538 to Python 3.6 - an update to CPython to implicitly
coerce the legacy C locale to C.UTF-8 as the Python interpreter
process starts up.

I'm still aiming to have that PEP officially accepted for 3.7 upstream
before the F26 beta deadline (currently May 16th), but hit a slight
hitch recently when Barry Warsaw had to withdraw as BDFL-Delegate due
to a lack of sufficient time to devote to the review process.

Fortunately, Naoki Inada has agreed to take over the role, and Guido
signed off on the updated delegation today:
https://mail.python.org/pipermail/python-dev/2017-April/147796.html

So with any luck, we should be able to get this change explicitly
approved by upstream within the next couple of weeks.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: openSUSE Python packaging and Fedora

[Nick Coghlan]

On 14 March 2017 at 02:27, Neal Gompa <ngomp...@gmail.com> wrote:

> However, I think that their work is complementary, and at least we
> could incorporate their macros into our standard set of Python macros.
> I would like to see us work with the openSUSE guys to see if we can
> come together on a unified standard of macros. I believe there's
> certainly a desire on both sides (especially with the OpenStack
> packaging teams, who'd like to not have to duplicate each other's work
> for their respective distributions).
>
> What do you guys think?
>

Improvements to packaging collaboration across the RPM-based distros are
always going to get a big thumbs up from me :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: Python 3.6, Fedora, and the "C" locale

[Nick Coghlan]

On 15 February 2017 at 02:42, Charalampos Stratakis <cstra...@redhat.com>
wrote:

> Bumping this thread.
>
> Draft of the self contained change:
> https://fedoraproject.org/wiki/User:Cstratak/python3.6_c.utf-8_locale
>
> Some of the sections are reworded from PEP 538 [0] as most (if not all) of
> the motivation section for upstream, also applies for Fedora.
>

Sorry for the lack of response! I basically stopped reading email during
PyCon Pune and the trip back to Australia :)

Aside from a typo in the variable name (which I fixed), this looks great to
me, and I see from https://fedoraproject.org/wiki/Category:ChangeAnnounced
that it's already been announced on fedora-devel.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: Warning before sudo pip'ing?

[Nick Coghlan]

On 10 February 2017 at 18:35, Tomas Orsava <tors...@redhat.com> wrote:
> Hi!
> On the last FESCo meeting while discussing the sudo pip Fedora [Change],
> maxamillion proposed that it might be useful to issue a warning when a user
> tries to run pip with root privileges--as in most cases it's not what they
> should be doing (`pip install --user` is usually more appropriate).
>
> What do you think?

I guess when you're building an RPM you're not typically running as
root either, so standard RPM builds would still be fine. So +1 from
me.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: Unable to install packages in a Python 3 virtual environment created with venv module with --system-site-packages option

[Nick Coghlan]

Sorry, missed replying to the second part of your message.

On 8 February 2017 at 13:44, Tadej Janež <tade...@nez.si> wrote:
> As I see it, there are two issues when one has the aforementioned three
> way combination:
>
> 1) Users will probably use 'pip install foo' inside the activated
> virtual environment and be surprised by not being able to install
> anything.
> Many of them might not know the 'python -m pip install foo' variant.

Right, that's why I consider the fact that doesn't work by default a
bug in Fedora's current approach.

> 2) Users that know the 'python -m pip install' variant may want to
> install a package inside the virtual environment that is already
> available in system site-packages . The installation won't work unless
> they pass the '--ignore-installed' option or they install a newer
> version of the same package with '--upgrade' option.

This, on the other hand, is why having the system site-packages
visible is discouraged in general - it really is more complicated than
using an environment that only shares that standard library and the
language runtime.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: Unable to install packages in a Python 3 virtual environment created with venv module with --system-site-packages option

[Nick Coghlan]

On 8 February 2017 at 13:44, Tadej Janež <tade...@nez.si> wrote:
> Nick,
>
> thanks for your thorough answer.
>
> On Mon, 2017-02-06 at 20:07 +0100, Nick Coghlan wrote:
>>
>> It's not specific to Fedora's Python 3 packaging as such, but it *is*
>> specific to:
>>
>> - using --system-site-packages
>> - having pip install in the system site packages
>
> Yes, understood.
>
>> However, the way Fedora handles the native venv support in Python 3
>> means that this case will *always* apply when creating Python 3
>> virtual environments that can see the virtual environments - the pip
>> that gets installed into the virtual environment comes from
>> python-pip, rather than being a bundled copy in the Python RPM.
>
> I'm still a bit puzzled by this part.
>
> If I understand stdlib's venv documentation correctly [1], when one
> creates a virtual environment, it will use ensurepip to bootstrap pip
> into the virtual environment.
>
> However, if pip is installed in system site packages, then ensurepip
> will skip the bootstrapping process, leaving the created virtual
> environment without its bundled copy of pip:
>
> $ python3 -m venv --system-site-packages myvenv3
> $ source myvenv3/bin/activate
> (myvenv3) $ python3 -m ensurepip
> Ignoring indexes: https://pypi.python.org/simple
> Requirement already satisfied (use --upgrade to upgrade): setuptools in
> /usr/lib/python3.5/site-packages
> Requirement already satisfied (use --upgrade to upgrade): pip in
> /usr/lib/python3.5/site-packages
> (myvenv3) [vagrant@tadej-zbook ~]$ ls myvenv3/bin
> activate  activate.csh  activate.fish  python  python3
>
> Do you think this is OK or should stdlib's venv module (or ensurepip)
> be changed to always create a bundled copy of pip in the virtual
> environment?

What's supposed to happen is that ensurepip installs a fresh copy into
the virtual environment.

It *doesn't* happen under Python 3 on Fedora when system site-packages
is visible, because if python3 is installed, then python3-pip will
*also* be installed, and you get the behaviour you're seeing.

So this should be filed as a bug against the Fedora python package for
doing the wrong thing by default.

However, it isn't clear that this qualifies as an upstream bug, as
ensurepip in general is *supposed* to be a no-op when pip is already
installed.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: To use or not to use: packaged flask

[Nick Coghlan]

On 12 January 2017 at 21:26, Alec Leamas <leamas.a...@gmail.com> wrote:
> On 12/01/17 12:02, Alec Leamas wrote:
>> It's "my" code, I'm upstream for an old package for which I'm about to
>> add a python API. Haven't found any pointer how to make pypi package
>> with linux native code... have you?
>
> At a second thought, I don't think I wan't this. My users will  need the
> real package (lirc) anyway; this is compiled code which typically is
> installed using package managers. Adding the python API to the core package
> means that neither building nor headers are required after installing the
> package, and I'm pretty sure pypi doesn't solve that in a better way.

The main cases where I've personally found "system package only" to be
a genuine pain is when doing cross-version compatibility testing with
tox for projects that still support Python 2.6.

As a specific example, you can't readily run the Anitya tests under
Python 2.6 on Fedora, as even though there's a python26 package,
there's no corresponding RPM bindings for it, and hence "import rpm"
fails even with --system-site-packages enabled.

The same problem comes up in reverse on RHEL/CentOS where the newer
runtimes available via EPEL or Software Collections are missing most
system packages and hence rely almost entirely on PyPI.

That said, there's an eminently practical way to avoid those problems
in many cases: run a recent Fedora container image atop a RHEL or
CentOS container host rather than running directly on the host.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: Python 3.6 and blocking-by-default os.urandom()

[Nick Coghlan]

On 5 January 2017 at 03:42, Lukas Slebodnik <lsleb...@fedoraproject.org> wrote:
>> It is not just about python3.6 but I can also see something similar with old
>> kernel and python35 in latest rawhide userspace
>> https://bugzilla.redhat.com/show_bug.cgi?id=1410187
> BTW the explanation is that the latest update of glibc in rawhide provides 
> functions getentropy and getrandom

And Victor Stinner further diagnosed that as a combined bug in
CPython's conditional compilation logic where:

- getentropy was preferred over getrandom when both were available
- only the getrandom code had the ENOSYS handling needed to cope with
newer binaries running on older kernels

http://bugs.python.org/issue29157 has a patch to change the logic so
that getrandom is preferred over getentropy when both are available,
and also to add the ENOSYS handling that getrandom already has to
getentropy.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: Tackling cert bundling with install-time symlink maps?

[Nick Coghlan]

On 24 December 2016 at 02:38, John Dulaney <jdula...@fedoraproject.org>
wrote:

> On Fri, Dec 23, 2016 at 05:37:51PM +1000, Nick Coghlan wrote:
>
> > I filed that idea on the pip issue tracker at
> > https://github.com/pypa/pip/issues/4197 but figured I should raise it
> here
> > as well, as if something like this was added, then Fedora could be
> updated
> > to use a standard symlink map when building RPMs, and the developer
> portal
> > could be updated with suggest `pip.conf` settings to use the system cert
> > bundle by default.
> > Nick.
>
> Seems like a good idea.
>
> Can't we just ship a pip.conf that defaults to using the system cert
> bundle,
> rather than merely suggest it?
>

I'm not sure that makes sense as a general default, as it's relatively rare
for a developer's laptop to 100% match the production deployment
environment, which means not using the system cert store really is a better
default for virtual environments in Fedora Workstation.

The case for using the system certs by default would be stronger for Fedora
Server, Atomic, and the base cloud images, though (as those generally *are*
deployment environments).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Tackling cert bundling with install-time symlink maps?

[Nick Coghlan]

While hacking on Anitya yesterday, an idea occurred to me that may help
address the SSL/TLS cert bundling problem in a way that doesn't require
patching of modules and hence can be made compatible with virtual
environments.

Specifically, I had the idea of adding a new "--symlink
@" option to pip, such that you could do things
like:

"pip install -r requirements.txt --symlink cacerts.txt@
/etc/pki/tls/certs/ca-bundle.crt"

to replace any file called "cacerts.txt" in the packages being installed
with a symlink to "/etc/pki/tls/certs/ca-bundle.crt" instead (cacerts.txt
is the name httplib2 uses for its cert bundle).

Since we patch `python3-certifi` to use the system bundle, and running
`certifi` as a script prints the location of the cert bundle it is using,
that would mean you could do things like:

sudo dnf install python3-certifi
python -m pip install requests --symlink cacert.pem@`/usr/bin/python3
-m certifi`

and end up with an *unpatched* requests in the virtual environment that was
nevertheless still using the system certificate store.

I filed that idea on the pip issue tracker at
https://github.com/pypa/pip/issues/4197 but figured I should raise it here
as well, as if something like this was added, then Fedora could be updated
to use a standard symlink map when building RPMs, and the developer portal
could be updated with suggest `pip.conf` settings to use the system cert
bundle by default.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: Python 3.6, Fedora, and the "C" locale

[Nick Coghlan]

On 15 December 2016 at 23:19, Toshio Kuratomi <a.bad...@gmail.com> wrote:
> The costs here are very steep in both directions... so I don't see any
> good ways to address it yet.  The best I can offer so far is for the
> library to check and warn if an ascii-only locale is used.  That way
> someone who encounters a UnicodeError in code deployed under mod_wsgi
> is shown how to debug this when they run it under /usr/bin/python.  So
> something like this from the library:
>
> libpython detected LC_CTYPE=C.Some encoding errors may occur..
> Use.PYTHONALLOWCLOCALE=1 LC_CTYPE=C /usr/bin/python if debugging this
> under /usr/bin/python.

OK, I've posted a summary at
https://bugzilla.redhat.com/show_bug.cgi?id=1404918#c5 with some
proposed wording for the error messages.

For the embedded runtime warning, I'm suggesting "Some libraries and
operating system interfaces may not work correctly." but could be
pretty easily talked into changing that to "Some text encoding related
errors may occur." (as even if a library bails out the way click does,
it will probably mention ASCII in the error message)

For the PYTHONALLOWCLOCALE case I'm suggesting "Some libraries,
applications and operating system interfaces may not work correctly."
as that's literally the answer to "Why isn't the C locale allowed by
default anymore?".

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: Python 3.6, Fedora, and the "C" locale

[Nick Coghlan]

On 15 December 2016 at 22:39, Toshio Kuratomi <a.bad...@gmail.com> wrote:
> * I'm not 100% certain that LC_CTYPE is the best thing to check.
> People will set LC_CTYPE in conjunction with LC_COLLATE to C to get a
> predictable sort order.(CTYPE is needed because bytes can be
> interpreted as different characters and those differences can affect
> sort order.).  Changing this will mean that their command line sort
> order (ls, sort, etc) could differ from python's sort order.  I
> haven't thought this fully through or a better way to check for our
> actual meaning, though, and perhaps python already uses LC_CTYPE in
> ways that would differ from other unix tools?

Yep, LC_CTYPE is the source of all the pain, as it's what controls the
encoding used for everything that CPython needs to decode before it
gets its own codec machinery bootstrapped. Victor Stinner made a
couple of attempts at overriding it later in the interpreter
bootstrapping process (e.g. based on environment variables) after the
codec system was fully up and running, and the problem you end up with
is that:

1. Within CPython it's easy to lose track of how you decoded system
provided text like sys.argv, sys.warnoptions, sys._xoptions,
sys.executable, os.env, etc, so "fixing" incorrectly decoded values
later is fragile
2. Even if you *do* get all the details right within CPython, you may
be in trouble again as soon as you call out to a third party C/C++
library, especially GUI toolkits like Tcl/Tk, Qt or Gtk that have a
lot of locale dependent behaviour

His conclusion was that letting the locale-as-seen-by-CPython diverge
from that seen by the rest of the process simply doesn't work, and I
don't have any reason to second guess that conclusion.

However, I'll also note that any tooling written in Rust or Go already
makes the "UTF-8 everywhere" assumption at the level of the language
design, so the proposed change would just move tools written in Python
3 into the same category as those written in those languages (unless
you set PYTHONALLOWCLOCALE to request the old behaviour).

> * Thinking about whether this belongs in the library or the
> interpreter some more I'm seeing some hefty cons in both directions.
> already noted that the con for doing it in the interpreter is that we
> get out of sync with other things linking to libpython, therefore
> making debugging harder.

Note that CPython already offers a range of "preconfiguration" APIs
that allow applications embedding the runtime to override otherwise
environment based configuration settings. In particular
https://docs.python.org/3/c-api/init.html#c.Py_SetStandardStreamEncoding
was added specifically so Blender could just tell the Python 3 runtime
"configure the standard streams like *this*", rather than having to
persuade CPython to guess the right answer.

So the fact embedded runtimes can give you different results from what
you get at the command prompt isn't a *new* problem.

[Copying-and-pasting some of your comments from the other subthread to
consolidate the two discussions]

> I'd almost say that internalizing the click behviour could be the
> correct design here.  Have the library check that it has a locale with
> non-ascii capabilities and fail if it doesn't would be helpful.  That
> would quickly point to differences in behaviours running under a
> mod_wsgi vs /usr/bin/python, for instance, prompting the user to fix
> the mod_wsgi deployment in advance.

While I don't like the idea of locale *coercion* inside the library,
I'd be fine with emitting a proper Python level warning inside
Py_Initialize after we get the warnings machinery up and running

>  OTOH, users don't run into the
> problem all the time (it depends on the data being processed and how
> it is handled) so it seems heavy handed to do it this way

I think erroring out would be unduly harsh, but a warning seems
reasonable given the availability of C.UTF-8.

> (I suppose
> by the same argument I'd have to say that click is doing it wrong to
> force users to address ascii-only locales...)

click is younger than Python 3, so Armin did make some initial
attempts to get it working in the C locale on both 2 & 3. However, he
eventually gave the latter up as unsupportable and the error makes it
clear that "I don't need to support ASCII based locales on Python 3"
is a key constraint in deciding whether or not to adopt click.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: Python 3.6, Fedora, and the "C" locale

[Nick Coghlan]

On 15 December 2016 at 21:17, Toshio Kuratomi <a.bad...@gmail.com> wrote:
> On Mon, Dec 12, 2016 at 1:39 AM, Nick Coghlan <ncogh...@gmail.com> wrote:
>> I don't anticipate any major concerns with downstream redistributors
>> adding this behaviour, as the main thing that makes us nervous about
>> globally changing the default upstream is the sheer variety of Linux
>> distros out there, and the fact that folks are inclined to take their
>> Linux integration bugs straight to bugs.python.org rather than first
>> trying the issue tracker for their particular distro.
>>
> My one concern is precisely this variety.  For instance, if I get a
> report that my application is raising a UnicodeError on RHEL7 when run
> under cron (which uses the C locale) I might then try to replicate the
> error on Fedora using the same LC_ALL=C locale.  With this change I
> would fail to reproduce the error.

But with the current patch you *would* get a visible warning on stderr saying:

Python detected LC_CTYPE=C. Setting LC_ALL & LANG to C.UTF-8.

> This is a variation on arguments about why individual sites should not
> change the default encoding via sitecustomize.py.  The changes tend to
> make python applications non-portable.  I don't think it is as severe
> because we're still able to broadly classify things as "Fedora Python"
> vs "Upstream Python" (instead of "Python running at My Business" vs
> "Python running on the rest of the world" but it still is problematic.

Agreed, and my original idea upstream included an environment variable
override to account for that case:
http://bugs.python.org/issue28180#msg282964

I just forgot about that bit while writing the initial patch :(

As documented at
https://docs.python.org/3/using/cmdline.html#environment-variables the
normal convention for Python environment variable toggles is "A
non-empty string setting enables it", so the name I'd suggest here is
PYTHONALLOWCLOCALE.

The error message would then change to:

Python detected LC_CTYPE=C, forcing LC_ALL & LANG to C.UTF-8 (set
PYTHONALLOWCLOCALE to disable this behaviour)

and if the environment variable is already set:

Python detected LC_CTYPE=C, but PYTHONALLOWCLOCALE is set. Some
applications may not work correctly.

Does that approach seem more reasonable than unilateral locale
coercion with no off switch?

> OTOH, if this is a stepping stone and proving ground for getting it
> into upstream Python then we just get this change a little early...
> that's IMHO, a good thing.

Yeah, my goal is to standardise this upstream for 3.7, but I expect
folks to be more willing to make it the default behaviour on *nix
systems if at least some distros are willing to try it out in their
releases of 3.6 first.

> Perhaps what's needed is a locale on Fedora that allows people to
> select an ascii encoding for python which does not coincide with the C
> locale.  This should satisfy the case you mention that *most* of the
> time the C locale is not a conscious desire to select the ascii
> encoding but also, as I'm pointing out, the need to select an
> ascii-only encoding for debugging cross-platform scripts and
> applications.

As in an explicit "LANG=C.ASCII"? While I agree that would work, it's
probably more complexity than is needed vs a dedicated off switch for
the locale coercion.

On the other hand, if *glibc* were to some day start natively
interpreting "no locale set" or an unqualified "C" locale as
"C.UTF-8", then I agree a "C.ASCII" locale to explicitly opt in to the
old behaviour would make sense.

> [..]
>> As far as where we might add that check, I'd suggest the entry point
>> for the `python3` binary itself, rather than in the shared library:
>> https://hg.python.org/cpython/file/3.6/Programs/python.c#l46
>>
> I think the library is the appropriate place.  Otherwise you end up
> with a python application failing when run under mod_wsgi[*]_ which
> you can't debug using the command line interpreter.

There's one pragmatic problem with that, and one that's a question of
appropriate division of responsibilities in terms of understanding the
runtime's context of use.

The pragmatic problem is that the main CPython binary calls
https://docs.python.org/3/c-api/sys.html#c.Py_DecodeLocale to convert
the command line arguments from char* to wchar_t* before it calls
Py_Main, which means we have to override the locale *before* we hand
over control to the dynamically linked library. Otherwise we end up in
exactly the same situation that click complains about: by the time we
find out there's a problem with the locale, some work has already been
done using the wrong setting.

The architectural problem is that when you embed CPython, it really is
one of the embedding appl

Re: Python 3.6, Fedora, and the "C" locale

[Nick Coghlan]

On 12 December 2016 at 20:37, Petr Viktorin <pvikt...@redhat.com> wrote:
> On 12/10/2016 02:56 PM, Nick Coghlan wrote:
>> I also have an upstream motive for suggesting this, though: if we do
>> this in the more constrained environment of "Fedora users" and it
>> doesn't break the world, then that will help build a case for making
>> it the default upstream behaviour in Python 3.7.
>
> +1
> Handling this is on my (and thus python-maint's) TODO list, but python 3.6,
> "sudo pip", and PEP 354 have higher priority for us currently.
> If anyone wants to go ahead, make a patch and put it on Bugzilla, or draft a
> Fedora Change page, please go ahead!

I opened an issue and attached a patch at
https://bugzilla.redhat.com/show_bug.cgi?id=1404918

I also cross-linked it with the most recent upstream issue at
http://bugs.python.org/issue28180

Cheers,
Nick.


-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: Making sudo pip Safe

[Nick Coghlan]

On 13 December 2016 at 00:12, Neal Becker <ndbeck...@gmail.com> wrote:
> Currently, pip3 --user installed stuff can be updated with:
> pip3 install --upgrade --user  $(pip3 list --user -o | cut -f 1 --delim=' ')
>
> But that isn't obvious/discoverable

Note that upstream considers a bulk-upgrade command to be gated on
using a proper dependency solver:

* https://github.com/pypa/pip/issues/59#issuecomment-52424510
* https://github.com/pypa/pip/issues/988

However, using a proper dependency solver is in turn gated on having
programmatic access to the relevant dependency constraints *before*
you start installing anything.

The lack of a proper resolver when doing bulk upgrades is also a much
bigger issue for pip/PyPI than it was for yum as the latter relied
heavily on repo curation to limit the chance of conflicts. PyPI by
contrast is a complete free-for-all where blindly upgrading everything
in --user can easily break previously working setups.

So if we wanted to offer this, it would likely need to be as a
standalone (pip installable?) script that was equivalent to the above
bash snippet.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: Making sudo pip Safe

[Nick Coghlan]

On 12 December 2016 at 19:59, Tomas Orsava <tors...@redhat.com> wrote:
> On 12/12/2016 05:39 AM, Nick Coghlan wrote:
>> On 11 December 2016 at 01:33, Donald Stufft <don...@stufft.io> wrote:
>> The benefit of that approach is that it would not only solve the
>> conflict between dnf and pip at the Fedora level, but also move the
>> packaging user experience forward for the Python ecosystem as a whole
>> (by making the `--global` switch available in anticipation of a future
>> upstream switch to `--user` as the default).
>
> It would be a partial solution I think. People who first try sudo-less `pip3
> install` would be covered, but there are great many people that already go
> with `sudo pip3 install` right away, as well as many install guides on the
> Internet that advise to do so. In that case pip would probably install it
> under /root, which wouldn't be accessible from user-run python.

I think it would make sense to emit a warning when `--user` is used as
UID 0, so you'd end up with the following situation:

* `pip3 install --user ` remains unchanged
* `pip3 install ` usually just works (since we have
`~/.local/bin` on the default PATH)
* `pip3 install --global ` handles RPM and container builds
* `sudo pip3 install ` stops even appearing to work for
user-run scripts (since it installs into /root), no longer conflicts
with DNF, and emits a warning
* `sudo pip3 install --global ` is available if you
really know what you're doing and are prepared to deal with the
consequences

If the main goal is to protect naive users from inadvertently breaking
their system Python, making `sudo pip3 install ` do
something sensible doesn't seem valuable enough to me to maintain an
upstream/downstream behavioural discrepancy over the long term.

However, there's a *different* question, which is whether or not we
want `sudo pip3 install --global ` to be a nominally
supported thing to do, rather than a matter of "You're a DIY system
integrator now, welcome to the club!". If we genuinely want to do
that, then yes, we'd need something more like the Debian
--install-layout patch. However, one of the problems with that is that
it currently targets setuptools/distutils specifically, which may be a
problem as the upstream changes to enable more flexibility in the
build pipeline come into effect.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: Python 3.6, Fedora, and the "C" locale

[Nick Coghlan]

On 12 December 2016 at 16:18, Thomas Spura <toms...@fedoraproject.org> wrote:
> Zbigniew Jędrzejewski-Szmek <zbys...@in.waw.pl> schrieb am Mo., 12. Dez.
> 2016, 02:59:
>> To make things less hand-wavy: there are concrete examples where this
>> would be known to help: containers as mentioned in the original
>> proposal, and mock (in my experience tests most often require setting
>> an utf-8 environment to work). Do we know of examples where the
>> proposed change would make things worse?
>
> I don't and didn't when this change was dropped a few years ago... It would
> certainly help to discuss this also upstream and if there is no objection,
> implement Nick's proposal.

I don't anticipate any major concerns with downstream redistributors
adding this behaviour, as the main thing that makes us nervous about
globally changing the default upstream is the sheer variety of Linux
distros out there, and the fact that folks are inclined to take their
Linux integration bugs straight to bugs.python.org rather than first
trying the issue tracker for their particular distro.

By contrast, for the Fedora system Python, we only need to worry about
doing what's right for *Fedora 26+*, which is a much more constrained
environment than "arbitrary versions of arbitrary Linux distributions
running arbitrary init systems".

Specifically:

- we know that the C.UTF-8 locale should always be available
- we know that system applications don't intentionally use the C locale
- we know that some popular Python libraries will explicitly fail if
you try to use the C locale with Python 3
- we know that anyone trying to use the C locale with a non-UTF-8
default locale already has other problems
- we know that (containers aside) systemd is pretty good about setting
the locale appropriately based on /etc/locale.conf

That lets us pursue a simpler version of the idea that can ignore some
failure modes (like "C.UTF-8" being unavailable).

As far as where we might add that check, I'd suggest the entry point
for the `python3` binary itself, rather than in the shared library:
https://hg.python.org/cpython/file/3.6/Programs/python.c#l46

At various points during start-up CPython uses 'setlocale(LC_ALL, "")'
to force itself back into the default locale for the process based on
its environment, so the locale injection would need to look something
like:

char *loc = setlocale(LC_CTYPE, NULL);
if (loc != NULL && strcmp(loc, "C") == 0) {
setenv("LC_ALL", "C.UTF-8", 1);
setenv("LANG", "C.UTF-8", 1);
   }

(Plus error checking and the warning on stderr that this was being done)

I guess the next step would be for me to try this in a local clone,
and see what happens when running "LANG=C ./python -m regrtest" as
well as when running click.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: [RFC] RPM's Python dependency generator

[Nick Coghlan]

On 3 December 2016 at 05:40, Toshio Kuratomi <a.bad...@gmail.com> wrote:
> On Dec 1, 2016 6:58 AM, "Petr Viktorin" <pvikt...@redhat.com> wrote:
>> Problems with upstreams getting setup.py wrong should be treated as
>> upstream bugs and treated accordingly: reported as pull requests, or, as the
>> last resort, by a Fedora-specific patch to setup.py.
>
> Note that fedora will have to carry more patches to setup.py's (but
> hopefully of limited durations and pretty trivial).  Upstream versioned
> requirements often don't match with distro packages because we backport to
> work around bugs in compatible manners while upstream simply requires newer
> versions that have the bugs fixed.
>
> Not a showstopper by any means; just something to be aware of.

Depending on how deep people want to dive into that aspect, it may
even make sense to start using the "Local version identifier" feature
in PEP 440 to expose RPM release numbers to Python level tooling:
https://www.python.org/dev/peps/pep-0440/#local-version-identifiers

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: upstream dev. asks suggestions about howto make packagers work easier (bundled libraries, etc.)

[Nick Coghlan]

On 21 November 2016 at 21:49, Neal Gompa <ngomp...@gmail.com> wrote:
> On Mon, Nov 21, 2016 at 4:42 AM, Piotr Ozarowski <pi...@debian.org> wrote:
>> In Debian we have https://wiki.debian.org/UpstreamGuide
>> I guess more official, cross-distro document that we all point upstream
>> authors to would be a nice idea as well.
>
> It would be a great idea if there were principles that all
> distributions could agree on and have a document codify that so that
> all distributions can reference it to upstream.

Donald Stufft laid out the Python upstream position in
https://caremad.io/posts/2013/07/setup-vs-requirement/

The gist is that for abstract dependencies (e.g. setup.py), we want
people to specify the cases that they know *don't* work:

- minimum versions
- maximum versions if there's a particular known breaking change, or
if a particular dependency is well-known for API breaks
- exclusions for known incompatibilities with particular versions

This is useful information for both end users and redistributors, as
it indicates the versions of dependencies that upstream will flat out
refuse to support.

By contrast, concrete dependencies (e.g. requirements.txt) are
intended for deployment rather than redistribution, and hence can (and
usually should) pin the exact combination of dependencies that was
tested.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: Python 3.6 beta release in F26 Rawhide?

[Nick Coghlan]

On 4 November 2016 at 02:16, Charalampos Stratakis <cstra...@redhat.com> wrote:
> And FESCo ticket about that[0]
>
> [0] https://pagure.io/fesco/issue/1642

Awesome, thanks for tackling this.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: Python 3.6 beta release in F26 Rawhide?

[Nick Coghlan]

On 18 October 2016 at 00:49, Charalampos Stratakis <cstra...@redhat.com> wrote:
> The current URL should be https://beaker.qa.fedoraproject.org/ if that is the 
> one you have in mind.

Indeed it is, thank you!

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: Python 3.6 beta release in F26 Rawhide?

[Nick Coghlan]

On 17 October 2016 at 19:26, Charalampos Stratakis <cstra...@redhat.com> wrote:
> Hi,
>
> The change is already approved by FESCo [0] although as mentioned at the 
> change page,

Aye, I'd missed that initially, and then saw the ChangeAccepted note on the BZ.

> it is accepted for release candidate, so I guess I should open another ticket 
> for FESCo if they agree for a beta version to be built in rawhide.

Let's see how far we can get with COPR builds before going back to
FESCo - we should at least run
https://beaker-project.org/docs/user-guide/beaker-provided-tasks.html#distribution-rebuild
against the COPR Python and the Fedora packages that depend on Python
before proposing early adoption of the betas.

(Although I'll have to ask what's become of beaker.fedoraproject.org
before doing that - if it's not going to be back any time soon, then I
may need to run that on the Red Hat internal instance instead)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Python 3.6 beta release in F26 Rawhide?

[Nick Coghlan]

Hi folks,

During the discussion of the os.urandom() change in Python 3.6 (the
successor to the accidental change in Python 3.5 that is the subject
of Tomas's emailing about rebasing to 3.5.2 in F24), we came to the
conclusion it would be good to get 3.6 into Rawhide early enough to
impact the upstream beta cycle:
https://lists.fedorahosted.org/archives/list/python-devel@lists.fedoraproject.org/thread/UAB7JJ5VPW2W2QEERZ4HIQZZB3QMB2H5/

However, there's no reference to that discussion in
https://fedoraproject.org/wiki/Changes/Python3.6 - it only refers to
incorporating a release candidate or the actual release, which will be
too late for us to request changes to the upstream default
os.urandom() behaviour if we find unexpected problems with it.

With Python 3.6b2 recently pushed out the door, there's about two
months (including two more beta releases) until the first release
candidate in early December.

Regards,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: python34 for epel6 ?

[Nick Coghlan]

On 3 October 2016 at 23:36, Tomas Orsava <tors...@redhat.com> wrote:
> On 09/27/2016 08:43 AM, Nick Coghlan wrote:
>>>
>>> P.P.S. I realize rh-python34 is available on RHSCL, but it didn't seem to
>>> support "python3" in scripts...
>>
>> Script shebang lines can be supported via SCLs, but they need to run a
>> wrapper script that implicitly enables the SCL, rather than just being
>> a symlink to the SCL's Python executable. I don't recall the exact
>> incantation myself, but hopefully someone else will be able to chime
>> in with that.
>
> Right now you need to create and use a wrapper script that contains:
>
> #!/bin/bash
> /usr/bin/scl enable rh-python34 -- python3 "$@"
>
> However, when this BZ [0] gets into RHEL 7.4 (there's also talk of 7.3
> Z-stream), you'll be able to enable the collection right in the shebang like
> so:
>
> #!/usr/bin/scl enable rh-python34 -- python3

Nice. Would that come with a corresponding update to the pip and
setuptools versions in RHSCL such that their wrapper script generators
did the right thing?

I bring that up, as one of the other challenges with the way SCLs
currently work is that the upstream convention of copying
"sys.executable" into the shebang to generate a script that runs in
the current Python doesn't work as expected.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: Python SIG on-boarding issue

[Nick Coghlan]

On 7 September 2016 at 04:42, Dhanesh Bhalchandra Sabane
<dhanes...@fedoraproject.org> wrote:
> CommOps is currently working on the Python SIG On-Boarding ticket [1] and one 
> of the tasks we have identified is re-writing the Python SIG wiki page to 
> make it more beginner-friendly. I was assigned with this particular task 
> which is now complete [2] [3].  We received some feedback from mhroncok today 
> during the CommOps meeting and following points were discussed:
>
> 1. Splitting Python SIG in two FAS groups:
> 1.1 *python-sig* for newcomers and interested members who hang out and help 
> with Python on Fedora
> 1.2 *python-packagers* for proven members of the community / packagers who 
> will also receive security-related bugs. Promising members from *python-sig* 
> group can be promoted to *python-packagers* given that they have contributed 
> to the package's git.

+1 that sounds like an excellent idea :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org

Re: PEP: Distributing a Subset of the Standard Library

[Nick Coghlan]

On 7 September 2016 at 02:41, Tomas Orsava <tors...@redhat.com> wrote:
> Hi!
>
> I'm currently writing a PEP titled "Distributing a Subset of the Standard
> Library" to standardize and hopefully improve the behavior of Python without
> the its full standard library. This is relevant to Fedora, as we exclude
> several standard library modules into separate optional packages (e.g.
> python3-tkinter).
>
> I have a draft of the first two sections: Motivation and Specification.
> https://fedora-python.github.io/pep-drafts/pep-A.html

Very interesting, although I see a pragmatic problem with trying to
check for explicitly missing packages only after checking for the
standard library ones: the default import system doesn't make a clear
distinction as to which sys.path entries refer to the standard library
and which refer to other directories (like site-packages), so you
can't readily intercept processing after the standard library is
checked but before the rest of sys.path is processed.

However, sys.meta_path *does* let you explicitly block imports before
the default machinery is tried by raising ImportError from find_spec:
https://docs.python.org/3/reference/import.html#the-meta-path

Now, I'm making the assumption that what we need is a model whereby
the base install includes files that tells Python "these stdlib pieces
might be missing", and then the other packages can install files that
mean those "these pieces are missing" markers don't get processed.

One possible way to do that as a pre-import check injected into the
start of sys.meta_path would be to maintain a set of static
"module_name.optional" files in the standard library directory that
included:

- a relative file path to stat to indicate that the optional module is installed
- an import error message to raise if its not found

If the new hook either didn't see an optional module entry, or if it
checked and the file path was present, it would allow the import to
continue as normal. However, if it found the optional declaration file
and the file path missing, it would raise ImportError with the given
message.

CPython by default would provide optional declarations for the modules
that are optional in the upstream build process, then downstreams like
Fedora may add more for things like tkinter, idlelib, and the test
suite.

The downside of this approach is that it does mean the initial import
of optional modules would be a touch slower (since the file path to
stat and the error message would need to be read from file), whereas
the version in the draft PEP has the virtue of having no impact on
import time for modules that are available, even when starting from a
completely cold sys.modules cache.

So if we did want to enable the draft proposal in the PEP, we'd need
to look at proposing a special directory to hold the "missing" markers
that was always placed just after the standard library directory on
sys.path, and then defining a custom path_hook to process it:
https://docs.python.org/3/reference/import.html#path-entry-finders

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org

Re: Renaming python to python2

[Nick Coghlan]

On 1 September 2016 at 22:04, Avram Lubkin <av...@rockhopper.net> wrote:
>
> On Thu, Sep 1, 2016 at 7:44 AM, Nick Coghlan <ncogh...@gmail.com> wrote:
>>
>>
>> The ideal point we'd like to get to is one where all distro provided
>> scripts actually have the appropriate major version in their shebang
>> lines, and the unqualifed "python" is something along the lines of a
>> user-configurable launcher, akin to the "py" launcher for Windows:
>> https://docs.python.org/3/using/windows.html#python-launcher-for-windows
>> (see https://www.python.org/dev/peps/pep-0397/ for more details on
>> that)
>
>
> While I see some benefit to something like this and could see use cases for
> including it in the shebang, it would be overkill for "/usr/bin/python".
> Windows needs something like that because it doesn't support symlinks.

This isn't about Windows, this is about making Fedora a sufficiently
user friendly platform for Python developers that they don't feel
obliged to reach for third party runtime managers like pyenv and conda
(which, make no mistake, a great many of them already do).

We (wearing my upstream CPython hat for this post) would like
developer focused Linux distros to either put individual end users
fully in control of what happens when they type "python" at the shell
(or run a script with "#!/usr/bin/python" as its shebang line), or at
least give system administrators the ability to configure that on a
per-user basis. (There are some other commands like pypy and ipython
that could use a similar treatment, but if we can solve this for
CPython, then it becomes easier to apply a similar solution in other
contexts)

One key advantage this approach offers is that system administrators
can upgrade selected users to "Python 3 by default", while still
keeping Python 2 as the default for their own accounts and for service
accounts (e.g. Ansible's remote access). "You must upgrade entire
systems to Python 3 all at once" simply doesn't give people enough
granularity to do a properly managed transition (cf. the Ansible
bootstrapping problems when Python 2 was first dropped from the Fedora
base install).

Folks that don't want the behaviour of their scripts to be dependent
on a per-user settting should be using the symlinks qualified with the
major version rather than the unqualified path (which is already the
recommendation in PEP 394, and has been for years, since the meaning
of an unqualified Python reference can vary by distro).

> Let's stop trying to reinvent the wheel. There is already a system for
> handling this, alternatives, which is used by other languages and included
> in a minimal Fedora install.

Unfortunately, the alternatives system also requires administrator
access to configure and is also a global setting affecting the entire
system, so it doesn't solve the fundamental usability problem
associated with neither individual users nor system administrators
being able to reliably configure different options for what particular
users get when they type "python" into a shell session or run a script
with "/usr/bin/python" in the shebang line. (Hence the line in
https://lwn.net/Articles/640296/ about "The 'alternatives' mechanism
is another possibility, but that makes more sense for things like
choosing an editor than it does for choosing a version of Python.")

I'll acknowledge that putting `python` under alternatives would be
*better* than the status quo of having it globally locked to a
distro-defined version (since it would at least get us to "admins in
control", even thought it doesn't get us all the way to putting the
user in control of their own developer experience), but it still gets
the user experience wrong, as a particular group of users shouldn't
have to have a shared definition of what 'python' means just because
they all have accounts on the same server, or run on the same Standard
Operating Environment.

? It would only require adding to the %post and
> %postun scripts in the spec file and allows very easy testing to see what
> breaks when changing the default. When we are ready to change the default,
> only one value needs to be changed. And most importantly, it gives the
> end-user an easy way to change their system-wide default either from python
> 2 to 3 or when running a pre-release version of 3.x in parallel.

Switching to putting /usr/bin/python under alternatives would
definitely be a better option than unilaterally switching it to Python
3. However, I'm also confident that it wouldn't be a significant
enough change to slow the rate of adoption of non-distro Python
versions, such as those published for pyenv and conda (note that my
assumption is that we have a shared goal of wanting Python developers
to find value in using the Fedora curated Python runtime bi

Re: Renaming python to python2

[Nick Coghlan]

On 31 August 2016 at 23:54, Charalampos Stratakis <cstra...@redhat.com> wrote:
> Hello Python-SIG,
>
> Currently I am working on some ideas regarding the python package.
>
> Nothing is too concrete yet, however the technical side should be fairly 
> simple.
>
> The first thing is renaming python to python2.

Moving "python" to a virtual provides (currently provided by python2)
makes sense to me as a starting point.

> Currently the python packaging guidelines, suggest that we use the 
> python_provides macro, for both python2 and python3 subpackages of whatever 
> python module someone might be packaging. The macro for python3 subacpakages, 
> currently, does nothing however sometime in the future  it would be great if 
> everything could switch to python 3. When this is done we can flip the switch 
> in the macros and the python3 subpackage will also provide the 
> python- namespace, meaning essentially that when we speak about 
> python we talk about python3, and the python binary will be a symlink to 
> python3.

For /usr/bin/python, we had an upstream discussion about that at the
2015 language summit: https://lwn.net/Articles/640296/

The ideal point we'd like to get to is one where all distro provided
scripts actually have the appropriate major version in their shebang
lines, and the unqualifed "python" is something along the lines of a
user-configurable launcher, akin to the "py" launcher for Windows:
https://docs.python.org/3/using/windows.html#python-launcher-for-windows
(see https://www.python.org/dev/peps/pep-0397/ for more details on
that)

Debian's Geoffrey Thomas actually wrote a proof of concept launcher
that implements something along those lines (described at
https://ldpreload.com/blog/usr-bin-python-23 ), and as far as I know
the main blocker to it going any further has been finding someone that
has the time, energy, and inclination to pursue including it in a
distro as the official implementation of "/usr/bin/python", as well as
proposing it upstream as an amendment to the distro recommendations in
PEP 394.

The nice thing about the design of pythonmux is that, if Python 2 is
installed, it will use it automatically in non-interactive mode for
maximum compatibility with existing scripts, but if only Python 3 is
available, it will implicitly try that, rather than failing outright
the way a missing symlink will.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org

Re: Python 3.4 for Fedora 24+

[Nick Coghlan]

On 16 August 2016 at 20:36, Miro Hrončok <mhron...@redhat.com> wrote:
> On 11.8.2016 11:26, Miro Hrončok wrote:
>>
>> Hi,
>>
>> As a follow up of our Flock discussion, I will build several Python
>> versions in Copr, for development purposes (such as testing your code
>> with tox on multiple Python versions).
>>
>> Those builds will be installable alongside regular python3/python
>> packages and will be normal packages, no software collections etc. One
>> flat package (i.e. no -libs, -devel...) with bundled setuptools and pip.
>>
>> First I've built Python 3.5 for Fedora 23, you can grab it here [1].
>>
>> If you try to use tox with it, you'll have to use Python 3 version
>> (python3-tox is the package and unfortunately also the command), due to
>> a bug in virtualenv [2].
>>
>> I would appreciate any feedback on the build, so I can build python34
>> package for Fedora 24+ in similar manner soon.
>>
>> Also python33 and python26 are planned.
>>
>> If those builds prove themselves useful I'll try to put them in Fedora
>> (with a strict guideline that forbids any other package to depend on
>> them).
>>
>> [1] https://copr.fedorainfracloud.org/coprs/g/python/python35/
>> [2] https://bugzilla.redhat.com/show_bug.cgi?id=1365941
>
>
> You can now also test Python 3.4 for Fedora 24 and 25.
>
> https://copr.fedorainfracloud.org/coprs/g/python/python34/
>
> There is one remaining issue, but it should not block you form using the
> package.
>
> https://github.com/fedora-python/python34/issues/1
>
> Let me know how it works for you.

Nice! Would it make sense for us to have a "tox" section in the
sidebar at 
https://developer.fedoraproject.org/tech/languages/python/python-installation.html
that covers using these COPR builds with tox for cross-version
testing?

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org

Re: Automatic Provides: Discussion summary and plan

[Nick Coghlan]

On 15 August 2016 at 19:42, Igor Gnatenko <ignate...@redhat.com> wrote:
> It can't track/change BR/R's as RPM is Turing complete and impossible to 
> parse.
> Imagine, we have pythonXdist(foo) extracted from PyPI metadata, but in
> Fedora we still need to add some more BR for that, so we add it. When
> new release comes (still without added BR in upstream) rebase-helper
> will not be helpful. It can change only version of spec.

This is a self-inflicted problem arising from our current tooling
design, though, since "generate-and-edit" isn't the only way to
supplement upstream metadata: we can also design spec file generators
to accept a supplemental config file in addition to the upstream
metadata.

If we're using that alternate model, then rebase-helper can have a
much easier time of things, since it isn't trying to edit a previously
generated spec file, it's just generating a new one based on the new
upstream metadata and the old supplemental metadata, and seeing if the
result still passes CI.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org

Re: Python 3 porting: 50% done in Rawhide 

[Nick Coghlan]

On 11 Aug 2016 23:48, "Petr Viktorin"  wrote:
>
> Hello,
> As of now, http://fedora.portingdb.xyz shows that we are 50% done porting
Fedora packages to Python 3. This is a big magic milestone; if you're
looking for a reason to celebrate, this is it! :)

Very cool!

> Now, what's next?
> I can't speak for everyone involved, but at Red Hat's python-maint team,
we'll tone down the focus on getting as many packages ported as possible.
This led to us picking the low-hanging fruit, which is better left to
people that are just getting started. We'll be around to answer questions,
provide hints, and otherwise help others get the badges instead of stealing
them for ourselves :)
>
> Instead, we should shift our focus from porting specfiles to upstream
projects. At this point, if some software is easy to port it was probably
ported already; what we're left with are either tough nuts to crack or
projects with few people relative to the codebase size. Some projects that
come to mind that could use attention are GTK, Mercurial, Samba, wxPython,
PySide, Koji & Fedora infra, Ansible.
> I don't know yet what our priorities should be here, but that's the
general direction.

That makes sense to me. Ansible in particular is going to be a thorny one,
since even after Ansible itself is made Py3 compatible, there's still the
wide array of modules on Ansible Galaxy to deal with.

It may be worth investing some further time in ensuring that porting aids
like your py3c compatibility helper, "pylint --py3k", python-modernize, and
perhaps the "sixer" tool Victor built to help with Python 3 porting are
readily available.

Cheers,
Nick.
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org

Re: [EPEL-devel] Which python3 versions to package for EPEL7?

[Nick Coghlan]

On 6 January 2016 at 13:48, Toshio Kuratomi <a.bad...@gmail.com> wrote:
> Despite the confusion, my feeling is that we want the newer versions.
> People who want to run python3 are willing to live more on the bleeding
> edge.  What I've observed is that they want newer versions of libraries as
> well.  Building packages that nobody wants to use because they are too old
> isn't that helpful to those who want to use the system packages for their
> development.  For us packagers, getting applications to run on python3
> frequently needs newer versions of supporting libraries due to bugfixes for
> python3 going into those libraries.  Attempting to pin the python34 versions
> to the versions that ship with RHEL or in EPEL as the python2 version will
> quickly become a hindrance to us in those efforts as well.

+1 from me for adopting newer package versions as baseline in the
python3X stacks.

As Toshio notes, many of the components currently shipped don't
support Python 3 yet, so they're going to *have* to be rebased before
they can be part of a Python 3 stack: http://fedora.portingdb.xyz/

> If we deem the confusion factor to be too great we could resort to the
> Debian library route and name packages with the library version number as
> well: python34-setuptools19, for example.  But that carries its own set of
> problems: 1) Although we have a way (via setuptools/pkg_resources) to make
> both packaging of alternate modules and usage of the modules work it isn't
> as easy as working with modules packaged in the normal way. 2) If it's
> standard for us to package python34 modules as compat packages, our support
> burden will increase as people create packages for multiple versions of
> the upstream library.  We (EPEL) need to figure out some policies on
> retiring old packages when they're no longer maintained.  3) If we're
> retiring unmaintained older versions of packages during the lifetime of
> a RHEL release then we probably need to figure out if our present method for
> determining the default python module (what version you get if you just do
> "import setuptools") is the best.  The current way is basically, whichever
> version entered EPEL first is the default, all others are forward compat
> packages.

It's also worth keeping in mind that the parallel install model
adopting for the python3X releases in EPEL means there's a chance to
rebase the "default" version of other packages each time "X" is
incremented. While that won't be a big difference for the python34 and
python35 stacks, there will presumably be more significant version
bumps once python36 rolls around.

Regards,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list
python-devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org

Re: New (optional) python egg dependency generator for RPM

[Nick Coghlan]

On 22 November 2015 at 04:18, Neal Gompa <ngomp...@gmail.com> wrote:
> Based on the feedback from you guys, I've made the changes to move to
> pythonX.Ydist() in the dependency generator. That code has been
> submitted as a pull request to the RPM GitHub repository[0]. I also
> added a switch for those who want pythonXdist() Provides, but it is
> opt-in rather than opt-out. The option is only for distributions that
> intend to carry only one Python runtime per major version.

Very cool, thank you!

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list
python-devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org

Re: New (optional) python egg dependency generator for RPM

[Nick Coghlan]

On 17 November 2015 at 23:25, Neal Gompa <ngomp...@gmail.com> wrote:
> As for naming, I'm all ears for a better name, because if the "egg"
> name is going away, I'd rather it not continue to say that.

My suggestions would be either:

python2dist(name)/python3dist(name)

or:

python2(name)/python3(name)

The "dist" suffix comes from:

* "distutils", the standard library's software distribution utilities
* the "sdist" name used for uploading source archives to PyPI
* the "-Dist" suffix used in the never-really-adopted metadata 1.2 spec [1]

While dropping the suffix entirely seems like a potentially attractive
option to me, it may also be ambiguous as to whether it's referring to
import package names (eg. "python2(pkg_resources)") or distribution
package names (e.g. "python2(setuptools)").

Cheers,
Nick.

[1] https://www.python.org/dev/peps/pep-0345/

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/python-devel

Re: Mention dist-info files in the packaging policy?

[Nick Coghlan]

On 23 September 2015 at 02:54, Jason L Tibbitts III <ti...@math.uh.edu> wrote:
>>>>>> "NC" == Nick Coghlan <ncogh...@gmail.com> writes:
>
> NC> I just noticed that the packaging policy doesn't currently mention
> NC> dist-info directories, only the older egg-info:
> NC> https://fedoraproject.org/wiki/Packaging:Python#Files_to_include
>
> dist-info is completely new to me.  I never particularly understood eggs
> so I'm sure I'll understand "dists" or whatever it's called now.

It's a metadata directory installed alongside Python packages to let
Python tools know which packages are installed:
https://www.python.org/dev/peps/pep-0376/#one-dist-info-directory-per-installed-distribution

At various times over the last couple of years we'd discussed the
notion of using pip rather than direct setup.py invocation in the
Python install macros, which would generate that more complete
metadata rather than the partial metadata generated in .egg-info by
setuptools, or the complete lack of installation metadata generated by
plain distutils. (The main discussion I remember is one with Toshio
Kuratomi at Flock 2013, but I think it also came up in a discussion
with Slavek Kabrda last year).

However, it looks like I'd misrembered the situation, and changing the
RPM install macros to generate modern metadata is still in the "this
might be desirable" stage. That's a much bigger change than just
tweaking the docs to mention the updated metadata location.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/python-devel

Mention dist-info files in the packaging policy?

[Nick Coghlan]

I just noticed that the packaging policy doesn't currently mention
dist-info directories, only the older egg-info:
https://fedoraproject.org/wiki/Packaging:Python#Files_to_include

Does that need to go through FPC, or can we just fix it as a bug fix
that brings the downstream docs in line with the upstream metadata
generation?

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/python-devel

Python 3 additions to the Fedora 23 release notes?

[Nick Coghlan]

Something Joe Brockmeier mentioned in his Fedora Marketing talk at
Flock last week was the need for development teams to be active in
providing content for the release announcements.

With the Fedora 23 Alpha coming out last week, I looked at the Release
Notes to see if there was anything about the Python 3 as default
change, but didn't see anything.

Was there something there and I just missed it, or does something need
to be written up and passed to the folks responsible for creating the
release notes?

Regards,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/python-devel

Re: Status update: Python 3 as default on Fedora installs

[Nick Coghlan]

On 14 July 2015 at 21:19, Miro Hrončok mhron...@redhat.com wrote:
 I would like to thank all the contributors, who are helping us with
 this. We are working on a special Fedora Badge for this Python 3 effort [4].

Thanks for the update, and I love the badge idea!

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/python-devel

Re: Python 3.5 as a system-wide change for Fedora 23?

[Nick Coghlan]

On 18 Jun 2015 10:02 pm, Matej Stuchlik mstuc...@redhat.com wrote:
 We feel that that's perhaps a little tight schedule, where things could
 go wrong easily. For that reason we'd like stay with Python 3.4 as system
 python for Fedora 23, while providing Python 3.5 in a Copr. (Perhaps using
 Miro's repo)

 Does that make sense?

It does. I also realised last night that there's a better, less disruptive
path forward that relates to a discussion we had upstream at this year's
language summit regarding the Python 3 transition and /usr/bin/python on
POSIX systems: https://lwn.net/Articles/640296/

The idea I had in relation to that is to wonder whether we could come up
with a self-contained change that allowed the normal python symlink to be
swapped out for a configurable version switcher that was compatible with
the distro independent environment module system now used for SCLs.

That approach would start us down the path of better separating the system
default Python setting from the choice of user's preferred Python,
paving the way for users swapping in alternative implementations like PyPy
and Jython, in addition to choosing between CPython 2.7, 3.4  3.5.

Cheers,
Nick.

P.S. There's at least one current compatibility issue in the upstream
Python test suite related to Fedora's upgrade to more secure default SSL
settings: http://bugs.python.org/issue23965
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/python-devel

Python 3.5 as a system-wide change for Fedora 23?

[Nick Coghlan]

Hi folks,

Toshio pinged me about a problem with dnf using -OO in their shebang
lines earlier today
(https://bugzilla.redhat.com/show_bug.cgi?id=1230820), which got me
thinking about our Python 3.5 adoption timeline (as I note in the
issue there, the reason using -OO in system packages is currently a
bad idea is a limitation in CPython's bytecode caching scheme that
Brett Cannon has fixed for 3.5+).

The two relevant schedule docs are the ones for F23 and Python 3.5:

Fedora 23: https://fedoraproject.org/wiki/Releases/23/Schedule
Python 3.5: https://www.python.org/dev/peps/pep-0478/

The upstream Python 3.5rc1 release is due on August 9, while the final
release is due on September 13. To switch in F23 that would mean:

* getting a system-wide change for a Python 3 upgrade approved by the
F23 deadline on Jun 26
* getting a 3.5 beta release incorporated by the testability deadline
on July 28 (this would likely correspond to 3.5b3 upstream, which is
due for release on July 5)
* F23 Alpha would ship with a Python 3.5 beta release
* F23 Beta would ship with a Python 3.5 release candidate
* F23 final would ship with the Python 3.5.0 final release

Slavek's change proposal for the 3.4 upgrade in F21 is at
https://fedoraproject.org/wiki/Changes/Python_3.4

Progress on the Python 3 as default effort means that the Py3 stack
is significantly more critical now than it was back then. However, we
also have better testing tools available.

In particular, for testing purposes prior to making the change in
Koji, I'd suggest we consider Beaker's /distribution/rebuild task:
https://beaker-project.org/docs/user-guide/beaker-provided-tasks.html#distribution-rebuild

The example given there is for testing GCC changes, but it should work
for this as well (while beaker.fedoraproject.org isn't open for more
general access yet, I still have an account there from when I was
working on the Beaker team, and worst case, we can do the test on Red
Hat's Beaker internal instance instead).

The contingency plan if the Beaker rebuild showed significant problems
that couldn't be resolved by the testability deadline would be to
postpone the system-wide change to Fedora 24 (however, I'd consider
issues of that magnitude to indicate an upstream compatibility
problem, so it hopefully won't come to that)

If folks think this sounds like a plausible approach, I'd volunteer to
work with Matej as Python 3 maintainer to push it forward.

Regards,
Nick.

P.S. Miro's nightly SCLs at
https://copr.fedoraproject.org/coprs/churchyard/python3-nightly/ may
also have a part to play, although I'm not sure what that would be
just yet

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/python-devel

Re: fedora wikipage Packaging:Python

[Nick Coghlan]

On 10 June 2015 at 08:41, Jason L Tibbitts III ti...@math.uh.edu wrote:
 MB == Martin Bukatovič martin.bukato...@gmail.com writes:

 MB The page doesn't discuss much any differences in guidelines for
 MB packages of python modules, python applications and when python
 MB project provides both.

 It shouldn't really need to; the question isn't specific to python at
 all.

 MB Would you consider this to be important/useful enough for new
 MB packager to provide better guidance?

 Perhaps a section in the main guidelines would be warranted.  For some
 reason I already thought it was there, but I don't see it while
 searching.  I'll see if I can draft something.

 MB Moreover would you think that listing few nicely packaged python
 MB projects on the wikipage would make sense?

 That definitely doesn't belong in a packaging guideline.

I agree it doesn't make sense to include that information in the
Python packaging guidelines, but I think it does make sense to provide
such recommendations *somewhere*. We know cargo culting is inevitable,
so it at least makes sense to have a way for folks to find *good*
examples, rather than having them pick at random.

Along those lines, it would potentially be useful to have a Python
specific packaging *tutorial* on the wiki an an alternative to the
generic https://fedoraproject.org/wiki/How_to_create_an_RPM_package
that assumes folks will be doing everything by hand.

For Python, it would make more sense to start with a tool like
Slavek's pyp2rpm to generate the initial SPEC file:
https://pypi.python.org/pypi/pyp2rpm

That will deal with a lot of basic aspects, and let packagers focus on
the delta between what pyp2rpm generates and what they need. As work
on the upstream Python metadata 2.0 spec proceeeds, we'll hopefully be
able to get that delta down to making changes to their *Python* level
metadata, and have the conversion to a policy compliant RPM be fully
automated in the vast majority of cases.

Such a page could also be linked from
https://packaging.python.org/en/latest/deployment.html#os-packaging-installers,
providing a clearer entry point for Pythonistas already familiar with
Python's packaging tools into the RPM ecosystem.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/python-devel

Re: fedora wikipage Packaging:Python

[Nick Coghlan]

On 11 June 2015 at 14:05, Jason L Tibbitts III ti...@math.uh.edu wrote:
 NC == Nick Coghlan ncogh...@gmail.com writes:

 NC I agree it doesn't make sense to include that information in the
 NC Python packaging guidelines, but I think it does make sense to
 NC provide such recommendations *somewhere*.

 I obviously agree, given that I wrote that I would try and draft
 something.  Just haven't had much time yet.

Ah, my apologies - I missed that sentence when I read your reply.

 NC Along those lines, it would potentially be useful to have a Python
 NC specific packaging *tutorial* on the wiki an an alternative to the
 NC generic https://fedoraproject.org/wiki/How_to_create_an_RPM_package
 NC that assumes folks will be doing everything by hand.

 That's certainly not a bad idea.  It's a wiki; feel free to run with it.

My own packaging experience is limited enough that I don't consider
myself qualified to provide advice on how best to go about it - my
involvement in Fedora's Python policy development has been from an
upstream perspective rather than a Fedora perspective.

 NC For Python, it would make more sense to start with a tool like
 NC Slavek's pyp2rpm to generate the initial SPEC file:
 NC https://pypi.python.org/pypi/pyp2rpm

 The problem with doing any of this now is that the python guidelines are
 in significant flux and will hopefully emerge in a much cleaner state
 with various convenience macros to handle the bulk of the heavy lifting.

Yeah, that makes sense. Once it settles down again and pyp2rpm can be
updated accordingly, then a simpler getting started guide will be
easier to write and maintain.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/python-devel

Re: Apps using default Python in Fedora vs. EPEL

[Nick Coghlan]

On 27 February 2015 at 11:02, Toshio Kuratomi a.bad...@gmail.com wrote:

 On Feb 25, 2015 3:14 PM, Nick Coghlan ncogh...@gmail.com wrote:

 For those not following along with the FPC ticket, Toshio and Tomspur
 have a nice write-up of the options at
 https://etherpad.mozilla.org/2Uqk0ikCll

 I came back to this question myself due to a couple of different
 ideas, discussed in https://fedorahosted.org/fpc/ticket/498#comment:19

 * How does the situation in Fedora change if the upstream PEP 494
 recommendation to downstream Linux distros was to change in
 conjunction with the Python 3.5 release currently scheduled for
 September? That release (https://www.python.org/dev/peps/pep-0478/)
 amongst other things, automatically handles EINTR errors for syscalls,
 restores binary interpolation support and adds matrix multiplication
 support and os.scandir().


 I would be against making the switch to /usr/bin/python at this time but
 would do most of that fighting upstream.  If the pep were updated then I'd
 at least want to see that the other major distros are committed to changing
 their /usr/bin/python at the same time.

 Changing the behavior of a well known program like this is a bad idea.  As
 it breaks compatibility: with people's home grown scripts, with their self
 installed programs, and between os's and os releases. The pep is palatable
 because the arguments in favor of someday changing the value revolve around
 someday there not being a /usr/bin/python on most systems. At that point it
 becomes reasonable to reallocate /usr/bin/python and let the systems where
 /usr/bin/python be declared legacy and the behavior of /usr/bin/python on
 those legacy systems is the quirk, not ours.

 We could cut over sooner than this argument actually makes the case for but
 now is definitely not that day.  Fedora, rhel, ubuntu, aren't yet at the
 point where /usr/bin/python isn't present on most of their installed systems
 in their latest version, let alone all of their versions.people are still
 pulling /usr/bin/python onto their systems through dependencies for common
 applications even if their os is advanced enough not to need it by default.
 We have quite a ways to go before /usr/bin/python can be switched.

Yeah, that's fair - a staggered release with the distros switching
first before end user scripts makes sense. That would make the likely
target date for a PEP 394 update some time in early 2017 with Python
3.6.

We could *potentially* switch the recommendation some time in 2016 if
all goes well with the distro migrations and significant libraries
start dropping Python 2.7 support, but switching in conjunction with
Python 3.5 would still be too soon.

 * With the default interpreter change postponed to Fedora 23, would it
 make sense to patch the system Python in Fedora 22 to emit Python 3
 warnings by default when it was run using the unqualified python
 reference rather than being run as the qualified python2? And then
 switch the symlink along with the RPM macros in Fedora 23?


 No to switching the value of /usr/bin/python and stated above.  The test
 makes some sense. If your warning is restricted to warning not to use
 /usr/bin/python (use /usr/bin/python 2 instead) that sounds really good to
 me.  (Your wording sounded like we should turn on warnings like python2 -3
 does which I don't think is such a good idea for fedora 22 but might be good
 in the future... our perhaps, like the kernel does with extra kernel
 debugging, we should turn it on in rawhide and fedora.n+1 but turn it off
 before release.)

I did mean the latter (turning on -3 warnings), but I like your idea
of only doing that in Rawhide and the Alpha releases for F23, and then
switching to a simple use a qualified Python reference warning in
the Betas and the actual release.

 It's also worth noting that the change I am considering to the
 upstream recommendation would place a qualifier on the distro
 providing a C.UTF-8 locale, so the C.UTF-8 in upstream glibc RFE
 (https://sourceware.org/bugzilla/show_bug.cgi?id=17318) would become a
 key enabler for making the symlink switch in Fedora (associated Fedora
 RFE: https://bugzilla.redhat.com/show_bug.cgi?id=902094)

 Like tomspur I'm not sure I see the specific relevance of this to what
 /usr/bin/python invokes although I would welcome the change :-)

Being able to type LANG=C.UTF-8 instead of LANG=C fixes some of
the odd corner cases where the interpreter startup sequence gets
confused. However, I remembered that subtle issues aren't the ones
we're worried about here - it's the big noisy ones like legacy print
and exec statements.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/python-devel

Re: Apps using default Python in Fedora vs. EPEL

[Nick Coghlan]

On 5 February 2015 at 17:48, Nick Coghlan ncogh...@gmail.com wrote:
 On 4 February 2015 at 21:01, Bohuslav Kabrda bkab...@redhat.com wrote:
 - Original Message -
 I don't really feel strongly about this, I agree that your solution has a
 merit (and syspython *is* better than default_python :)). I think I'll open
 an FPC ticket to ask FPC for their opinions on this and then I'll start
 taking concrete steps [1] ;) to get this done.

 Here's the FPC ticket: https://fedorahosted.org/fpc/ticket/498

 In commenting on the ticket, I realised something fundamental: we
 really do need to keep the RPM macro and the binary symlink in sync.
 That way the unversioned shebang lines in any packaged scripts can
 match up with the unversioned macros in the spec file.

 Which seems to bring the viable options down to just two:

 * switch the symlink to Python 3 as well (I don't like this due to the
 impact on end users with custom scripts)
 * hold off on switching the default for the time being and instead
 focus on enabling explicitly opting in to Python 3 in EPEL

For those not following along with the FPC ticket, Toshio and Tomspur
have a nice write-up of the options at
https://etherpad.mozilla.org/2Uqk0ikCll

I came back to this question myself due to a couple of different
ideas, discussed in https://fedorahosted.org/fpc/ticket/498#comment:19

* How does the situation in Fedora change if the upstream PEP 494
recommendation to downstream Linux distros was to change in
conjunction with the Python 3.5 release currently scheduled for
September? That release (https://www.python.org/dev/peps/pep-0478/)
amongst other things, automatically handles EINTR errors for syscalls,
restores binary interpolation support and adds matrix multiplication
support and os.scandir().

* With the default interpreter change postponed to Fedora 23, would it
make sense to patch the system Python in Fedora 22 to emit Python 3
warnings by default when it was run using the unqualified python
reference rather than being run as the qualified python2? And then
switch the symlink along with the RPM macros in Fedora 23?

It's also worth noting that the change I am considering to the
upstream recommendation would place a qualifier on the distro
providing a C.UTF-8 locale, so the C.UTF-8 in upstream glibc RFE
(https://sourceware.org/bugzilla/show_bug.cgi?id=17318) would become a
key enabler for making the symlink switch in Fedora (associated Fedora
RFE: https://bugzilla.redhat.com/show_bug.cgi?id=902094)

Regards,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/python-devel

Re: Apps using default Python in Fedora vs. EPEL

[Nick Coghlan]

On 4 February 2015 at 21:01, Bohuslav Kabrda bkab...@redhat.com wrote:
 - Original Message -
 I don't really feel strongly about this, I agree that your solution has a
 merit (and syspython *is* better than default_python :)). I think I'll open
 an FPC ticket to ask FPC for their opinions on this and then I'll start
 taking concrete steps [1] ;) to get this done.

 Here's the FPC ticket: https://fedorahosted.org/fpc/ticket/498

In commenting on the ticket, I realised something fundamental: we
really do need to keep the RPM macro and the binary symlink in sync.
That way the unversioned shebang lines in any packaged scripts can
match up with the unversioned macros in the spec file.

Which seems to bring the viable options down to just two:

* switch the symlink to Python 3 as well (I don't like this due to the
impact on end users with custom scripts)
* hold off on switching the default for the time being and instead
focus on enabling explicitly opting in to Python 3 in EPEL

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/python-devel

Re: Apps using default Python in Fedora vs. EPEL

[Nick Coghlan]

On 27 January 2015 at 21:26, Bohuslav Kabrda bkab...@redhat.com wrote:
 Hey all,
 I just wanted to ask for some thoughts on a problem (or rather a 
 hardship) that is starting to show with the Python 3 transition in Fedora.
 I've been contacted by two maintainers of applications in Fedora for advice 
 and have been thinking for some time how to solve this:

 Note: by applications I mean packages that provide end-user benefit and 
 don't need to be packaged for both Python interpreters. They just use some 
 Python (preferably the default one) to deliver functionality to user. Let's 
 take copr-cli as an example - this is a thin CLI wrapper around python-copr.

 Current state:
 - Up until F21, maintainers were encouraged to build applications with Python 
 2, but weren't discouraged from building with Python 3.
 - From F22 on, packagers will be encouraged to build with Python 3 rather 
 than Python 2.
 - Lots of packagers want to keep the same specfile for EPEL and Fedora.
 - Fedora guidelines mandate explicit usage of %__python2 and %__python3 (and 
 all the sitelib/sitearch macros).

 The Problem:
 If packagers want to build against Python 3 in Fedora and Python in EPEL 
 *and* keep the same specfile, they have to invent some ugly hacks, since 
 Fedora's guidelines require explicit usage of versioned Python macros. This 
 affects Requires and BuildRequires and %prep, %build, %install, %check 
 sections. People who want to do this either redefine %__python in Fedora to 
 point to Python 3 or something like that - I'm afraid that we could end up 
 with a huge pile of crazy macro redefinitions in tons of packages and I want 
 to avoid that.

 Proposed Solution:
 After thinking a few days about this, here's what I propose:
 - Every specfile will have a minimal header with macro definitions that will 
 look like this:

 %if 0%{?fedora}
 %global default_python python3
 %else
 %global default_python python
 %endif

I'm wary of this proposed solution mostly due to the fact that in the
middle of last year, the Beaker team had to go through and completely
redesign the way the automatic kickstart generation worked, because
the templates were full of checks that assumed RHEL 6 as the default
basis for derived distros. Once RHEL 7 and CentOS 7 were generally
available, that assumption became problematically wrong (e.g. systemd
wasn't a Fedora only feature any more), so the templates were all
rewritten to be based on operating system feature flags instead (which
had the added bonus of also making them more tolerant of Fedora
derivatives).

I see the seeds of a similar problem being planted with the above
proposal: what happens when, at some point in the future, Python 3 as
default is no longer a Fedora-only feature? Do we have to go edit all
the spec files again?

What if, instead, we were able to add a new macro that let folks
*explicitly* opt in to running in the system Python, but then define
the recommended spec file usage such that it falls back to Python 2 if
the system Python macro isn't defined?

That would give people 3 explicit options to choose from:

* always run in Python 2
* always run in Python 3
* run in the same Python as Anaconda and yum

Single source Python 2/3 compatibility could then be made a policy
requirement for packages opting in to running in the system Python
rather than explicitly running in Python 3.

Regards,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/python-devel

Re: Apps using default Python in Fedora vs. EPEL

[Nick Coghlan]

On 28 January 2015 at 03:32, Toshio Kuratomi a.bad...@gmail.com wrote:
 On Tue, Jan 27, 2015 at 10:50:07PM +1000, Nick Coghlan wrote:
 What if, instead, we were able to add a new macro that let folks
 *explicitly* opt in to running in the system Python, but then define
 the recommended spec file usage such that it falls back to Python 2 if
 the system Python macro isn't defined?

 Slavek raises the issue of how we get this into the buildroot.  An idea
 could be to talk to rel-eng and the other packagers about adding these sorts
 of system-feature macros to a package in the buildroot.  We could create
 a new package or add onto an existing one (is epel-release and
 fedora-release in the buildroot?)  the package would contain a small set of
 macros that specified certain features about the OS that packagers need to 
 know
 Then we really could write the conditionals as a feature test instead of
 a distro version test.

 One drawback is that we would have to push the macros out to every release
 that we build for (epel and fedora) otherwise we'd still have to use
 distro+version  conditionals.

That sounds vaguely analogous to the situation we have in Beaker:
adding completely new system features may require a Beaker server
update, while enabling and disabling already known features for a
custom distro is just a configuration setting for that distro in the
database. Actually switching to that model required updating the base
templates for every distro we natively support.

The reason we decided that approach was worth the extra up front
effort was because it meant we just had one place to update in the
future (the code that handles the calculation of the distro - feature
mappings) rather than having to search the templates for all the cases
where we were switching based on the distro.

I personally think do it right (i.e. figuring out how to enable
feature based rather than version based checks) is the direction we
should go for Fedora  EPEL, and then Slavek  I can separately tackle
the challenge of getting key downstreams (i.e. RHEL  CentOS) to go
along with that change. I'm more optimistic than Slavek is about that,
as many of the reasons we made the change for the Beaker kickstart
templates also apply to building for different environments.

Regards,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/python-devel

Updated Python 3 migration tools and HOWTO guide

[Nick Coghlan]

Brett Cannon recently published an updated version of the Python 3
migration guide: https://docs.python.org/3/howto/pyporting.html

One addition I found particularly noteworthy is the pylint --py3k
mode, which is designed to allow a project to keep their own code
Python 3 ready, even if their dependencies aren't available in
Python 3 yet.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/python-devel

Re: [Fedora-packaging] Changes in Guidelines Connected to Python 3 as Default Change

[Nick Coghlan]

On 4 December 2014 at 23:10, Bohuslav Kabrda bkab...@redhat.com wrote:
 - Original Message -
 On Thu, Dec 04, 2014 at 12:51:40AM +1000, Nick Coghlan wrote:
 
  On 4 Dec 2014 00:38, Bohuslav Kabrda bkab...@redhat.com wrote:
   - (This is not really related to the switch, but more of a general
   remark) In
  [4], it says that python 3 version of the executable gains a python3-
  prefix.
  This is IMO bad, since upstream projects tend to name the versioned
  binaries
  foo-3.4, foo-3 or foo3.4, foo3. We should accept one of these - I'm not
  really certain which one of them. I tried to discuss this several times on
  distutils-sig mailing list, but without reaching a consensus. Either way,
  prefixing with python3- doesn't make sense to me, because it's not similar
  to
  any upstream way and you don't find the binaries under their names using
  tab
  completion (e.g. footab doesn't tell you about python3-foo).
 
  Agreed.
 
  CPython  pip use the foo3.4, foo3 convention, so that seems enough of a
  reason to use that convention by default. We may want a unless upstream
  does
  it differently caveat though.
 
 Second caveat here is that currently we use version suffixes to denote
 a command coming from a compat package.  If we make this change we need to
 address that as well.  So, you might have sphinx-build, sphinx-build-2,
 sphinx-build-2.7, sphinx-build-3, and sphinx-build-3.4 for the python
 interpreter.  If you need a forwards or backwards compat package you might
 also have an 0.9 and 1.1 that you need to tack on.  Possible solution here:
 use a v prefix for the compat package's version.  So if the default
 package is 1.1, you would have the python-sphinx0.9 and python3-sphinx0.9
 packages provide:

 * sphinx-build-v0.9
 * sphinx-build-2-v0.9
 * sphinx-build-2.7-v0.9
 * sphinx-build-3-v0.9
 * sphinx-build-3.7-v0.9

 I'd rather see sphinx-build-v0.9-3.4. IMO keeping the Python version at the 
 very end in every case is better. In other words, the binary would normally 
 be sphinx-build-0.9 and we just append -3.4 to it.
 Also, this makes me realize more arguments to append Python version with 
 dash, not without it:
 1) sphinx-build-v0.93.4 would be very confusing (I do understand that this is 
 a downstream problem, but see the following point)
 2) Similarly, if there is an upstream whose entry_point/script ends with a 
 number (pep8 for example), it'd be highly confusing to use the notation 
 without dash, it would yield pep83.4 assuming the upstream community would 
 accept this scheme. This feels just wrong.

I think these are good reasons to default to using the dash if its
Fedora adding it. The guideline could be something like For Python
executables, also provide symlinks with a '-X' and '-X.Y' suffix,
unless upstream already provides appropriately versioned executables
without the dash. For compatibility packages, the Python version is
appended *after* the specific package version.

Cheers,
Nick.


-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/python-devel

Re: Changes in Guidelines Connected to Python 3 as Default Change

[Nick Coghlan]

On 4 Dec 2014 00:59, Donald Stufft don...@stufft.io wrote:


 On Dec 3, 2014, at 9:51 AM, Nick Coghlan ncogh...@gmail.com wrote:

  - (This is not really related to the switch, but more of a general
remark) In [4], it says that python 3 version of the executable gains a
python3- prefix. This is IMO bad, since upstream projects tend to name the
versioned binaries foo-3.4, foo-3 or foo3.4, foo3. We should accept one
of these - I'm not really certain which one of them. I tried to discuss
this several times on distutils-sig mailing list, but without reaching a
consensus. Either way, prefixing with python3- doesn't make sense to me,
because it's not similar to any upstream way and you don't find the
binaries under their names using tab completion (e.g. footab doesn't tell
you about python3-foo).

 Agreed.

 CPython  pip use the foo3.4, foo3 convention, so that seems enough of
a reason to use that convention by default. We may want a unless upstream
does it differently caveat though.

Oops, I just noticed an ambiguity in my caveat. I meant unless the
particular upstream project being packaged does it differently.

 It doesn't really matter right now but long term I think python packaging
should just natively support commands like this. Either just as a matter of
fact, opt in, or by allowing templated command names. Either way I think
the upstream tooling should and likely will follow python's lead for how
these are written.

Yeah. It's somewhere on the list of the umpteen gazillion things that could
still use improving ;)

Cheers,
Nick.


 ___
 python-devel mailing list
 python-devel@lists.fedoraproject.org
 https://admin.fedoraproject.org/mailman/listinfo/python-devel
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/python-devel

Re: [python] Add python2_version_nodots macro

[Nick Coghlan]

On 11/14/2014 12:05 AM, Matej Stuchlik wrote:
 commit 6875a63831616c6c8e722632e24faaa1a09cc831
 Author: Matej Stuchlik mstuc...@redhat.com
 Date:   Thu Nov 13 15:04:28 2014 +0100
 
 Add python2_version_nodots macro
 
  macros.python2 |1 +
  python.spec|5 -
  2 files changed, 5 insertions(+), 1 deletions(-)
 ---
 diff --git a/macros.python2 b/macros.python2
 index 982b51f..d090296 100644
 --- a/macros.python2
 +++ b/macros.python2
 @@ -2,3 +2,4 @@
  %python2_sitelib %(%{__python2} -c from distutils.sysconfig import 
 get_python_lib; print(get_python_lib()))
  %python2_sitearch %(%{__python2} -c from distutils.sysconfig import 
 get_python_lib; print(get_python_lib(1)))
  %python2_version %(%{__python2} -c import sys; 
 sys.stdout.write(sys.version[:3]))
 +%python2_version_nodots %(%{__python2} -c import sys; 
 sys.stdout.write(sys.version[:3].replace('.','')))

I just saw this commit go by on python-owners.

These macros are going to fail when Python 2.7.10 is released next year.
Anything currently depending on the new no dots version in particular
may also fail when it moves from 3 digits to 4 (since 2710 sorts
lexically lower than 279).

The version with dots can be fixed by using
'.'.join(sys.version_info[:3]) instead of ignoring the explicit Do
not extract version information out of it guidance for sys.version.

The version without dots can be addressed by including the leading zero:
{0.major}{0.minor}{0.micro:02d}.format(sys.version_info)

Cheers,
Nick.

-- 
Nick Coghlan
Red Hat Hosted  Shared Services
Software Engineering  Development, Brisbane

HSS Provisioning Architect
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/python-devel

Re: Template for modules that use pbr?

[Nick Coghlan]

On 10/30/2014 06:57 PM, Bohuslav Kabrda wrote:
 - Original Message -
Comparing the two shows a number of additional files in
the latter (AUTHORS, ChangeLog, git_review.egg-info,
PKG-INFO).  Am I correct to assume that this is a problem
upstream, i. e. that the tarballs should not have been
published in this state, and/or is there a way to work
around that in the spec file?
 
 The important thing here is that upstreams do not publish tarballs on 
 Github. Github dynamically generates tarballs of repos for given tag/commit 
 hash when you request it. This generally means that the tarball downloaded 
 from Github is not what upstream meant to release (unlike pypi, which *is* 
 what upstream released).
 The interesting thing about the tarball from pypi is, that all files in the 
 archive are owned by jenkins user. I didn't check, but my guess is, that 
 when upstream wants to do a release, they let Jenkins (with access to 
 upstream git repository, which is what pbr asks you to have) generate the 
 final tarball - and Jenkins generates all the extra files before packing the 
 tarball. If you look at the files missing from the first archive, they do 
 look autogenerated, which seems to confirm my thoughts, but I'm not 100 % 
 sure.

Right, Python sdists include some metadata files that can either be
maintained manually (which is error prone) or automatically generated
from source control (which introduces a build step into the release
archive generation, so the tarballs automatically generated by GitHub
are incomplete).

In this case, if you don't have the pregenerated metadata files from the
sdist, then pbr is going to need access to the original git repo in
order to generate them (see
http://docs.openstack.org/developer/pbr/#what-it-does for the things it
can automatically derive).

Regards,
Nick.

-- 
Nick Coghlan
Red Hat Hosted  Shared Services
Software Engineering  Development, Brisbane

HSS Provisioning Architect
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/python-devel

Re: python-sig in pkgdb2?

[Nick Coghlan]

On 10/07/2014 01:13 PM, Toshio Kuratomi wrote:
 I've stepped back from packaging for the most part but I think this is a
 great idea.  When I was active I'd often find something to cleanup in
 python packaging for each release (pil = pillow; removing
 python-setuptools-devel).  A python-sig group would definitely help with
 future cleanups like those.

I'm not an approved packager myself, but there are some folks on the app
devel teams that I could potentially interest in that (e.g. Dan
Callaghan adopted the TG1 stack for EPEL 7, as we still need it for Beaker).

Cheers,
Nick.

-- 
Nick Coghlan
Red Hat Hosted  Shared Services
Software Engineering  Development, Brisbane

HSS Provisioning Architect
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/python-devel

Re: mock, megadeps and PyPI

[Nick Coghlan]

On 08/22/2014 05:52 AM, Miro Hrončok wrote:
 Dne 21.8.2014 09:40, Bohuslav Kabrda napsal(a):
 I like this idea. Not only it can be beneficial for people creating a whole 
 environment for their Python application, but we would also be able to get a 
 huge feedback on pyp2rpm and would be able to improve it significantly, I'd 
 say.
 /me is wondering whether we could automatically generate SCLs by this, so 
 that the packages wouldn't conflict with system ones. In the best case, one 
 could even be able to do mock-or-a-mock-wrapper 
 --create-scl-from-this-app-and-build-it-in-copr /path/to/my/app scl_name (I 
 admit that naming command line arguments isn't my strongest skill...)
 
 Might also be usable for packagers trying to build something against the
 Python 3.5 SCL [1].
 
 [1] http://copr.fedoraproject.org/coprs/churchyard/python3-nightly/

Oh, I hadn't even thought of that, but yes, rebuilding for an SCL could
be very cool. The How do we rebuild/repackage our dependencies? was
actually one of the problems we ran into when considering using SCLs
rather than putting up with running in an older version of Python.

Cheers,
Nick.

-- 
Nick Coghlan
Red Hat Hosted  Shared Services
Software Engineering  Development, Brisbane

HSS Provisioning Architect
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/python-devel

mock, megadeps and PyPI

[Nick Coghlan]

Jeff Fearn reminded me today of mock --megadeps, a patched version of
mock he created that supports recursively downloading and building
dependencies in a chroot, without incurring the overhead of setting up
and tearing down multiple mock build environments the way the chainbuild
command does.

The mock RFE is at https://bugzilla.redhat.com/show_bug.cgi?id=843745,
while Jeff is now maintaining the patched version at
https://github.com/jfearn/mock

I'm bringing this to folks attention mainly due to one of the neat
features it has embedded in it, which is integration with CPAN: it can
use cpanspec to create an SRPM on the fly. That's likely not going to be
of suitable quality for Fedora itself, but it should be good enough for
COPR and potentially even Playground.

My question is whether it would be practical to do something similar
with pyp2rpm. Bonus points if we can come up with a way to integrate it
nicely with COPR, or even tidy up the implementation to the point where
we can convince Clark to accept the feature as part of mock itself :)

Regards,
Nick.

-- 
Nick Coghlan
Red Hat Hosted  Shared Services
Software Engineering  Development, Brisbane

HSS Provisioning Architect
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/python-devel

Re: Python 2.7 SSL upgrade patch available for testing

[Nick Coghlan]

On 07/30/2014 12:16 AM, Bohuslav Kabrda wrote:
 - Original Message -
 PEP 466 approved bring the core Python 2 network security infrastructure
 up to speed with the modern internet.

 Alex Gaynor has provided a draft patch of the most complex part of that
 PEP, backporting the bulk of the Python 3.4 SSL module to Python 2.7:
 http://bugs.python.org/issue21308#msg223895

 This is also the part of the PEP most likely to break things, so
 figuring out a way to test it in Fedora before it makes it into an
 upstream CPython release would be a good idea...
 
 We could create a copr repo where we would rebuild python (in an SCL?) with 
 these patches and then we'd rebuild some modules that use ssl - to see if the 
 tests pass and if they're actually usable. The disadvantage of this approach 
 is that it just takes lots of time to implement...
 Or, if we're feeling lucky, we can just build Python with these patches in 
 rawhide and see if something breaks :) That's easy and fast (assuming 
 everything works fine).
 
 I'd really love to help here, but I really can't spare enough time to do it 
 properly in Copr as noted above.
 So the question is, are we feeling lucky? :) I'd say yes, since rawhide has 
 just recently become future Fedora 22 and not much is going on in there right 
 now. If we break something, we can just revert it quickly and everything will 
 be fine.
 
 Is someone strictly against this or shall I move on with patching our rawhide 
 Python?

Patching rawhide would be wonderful. The patch is at last passing
Python's own test suite, so it shouldn't have broken anything too
dramatically.

Cheers,
Nick.

-- 
Nick Coghlan
Red Hat Hosted  Shared Services
Software Engineering  Development, Brisbane

HSS Provisioning Architect
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/python-devel

Re: Introducing Python 3.5 nightly builds for Fedora

[Nick Coghlan]

On 07/30/2014 12:04 AM, Bohuslav Kabrda wrote:
 - Original Message -
 On 07/03/2014 11:55 PM, Miro Hrončok wrote:

 Other than the packaging reasons: Fedora might attract Python
 enthusiasts and developers as it is most likely the only Linux distro
 that has development version of Python packaged nightly. Developers and
 powerusers can play with new features committed yesterday just by doing
 `dnf update`.

 I actually mentioned this in my recent SciPy keynote, on the grounds
 that scientists may want to play around with the new matrix
 multiplication operator without having to build Python from source :)
 
 Yeah, I saw that one on YouTube. Good talk :)
 Actually, I'd like to advertise this even more, e.g. on some of Python 
 upstream mailing list - on the other hand, I don't want to look like I'm 
 spamming everyone needlessly... What do you think would be the best place to 
 announce this? python-announce-list?

You can get away with a lot on python-ideas, and you're likely to find
folks potentially interested in playing around with it there.

Cheers,
Nick.

-- 
Nick Coghlan
Red Hat Hosted  Shared Services
Software Engineering  Development, Brisbane

HSS Provisioning Architect
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/python-devel

Python 2.7 SSL upgrade patch available for testing

[Nick Coghlan]

PEP 466 approved bring the core Python 2 network security infrastructure
up to speed with the modern internet.

Alex Gaynor has provided a draft patch of the most complex part of that
PEP, backporting the bulk of the Python 3.4 SSL module to Python 2.7:
http://bugs.python.org/issue21308#msg223895

This is also the part of the PEP most likely to break things, so
figuring out a way to test it in Fedora before it makes it into an
upstream CPython release would be a good idea...

Cheers,
Nick.

-- 
Nick Coghlan
Red Hat Hosted  Shared Services
Software Engineering  Development, Brisbane

HSS Provisioning Architect
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/python-devel

Re: Introducing Python 3.5 nightly builds for Fedora

[Nick Coghlan]

On 07/03/2014 11:55 PM, Miro Hrončok wrote:

 Other than the packaging reasons: Fedora might attract Python
 enthusiasts and developers as it is most likely the only Linux distro
 that has development version of Python packaged nightly. Developers and
 powerusers can play with new features committed yesterday just by doing
 `dnf update`.

I actually mentioned this in my recent SciPy keynote, on the grounds
that scientists may want to play around with the new matrix
multiplication operator without having to build Python from source :)

Cheers,
Nick.

-- 
Nick Coghlan
Red Hat Hosted  Shared Services
Software Engineering  Development, Brisbane

HSS Provisioning Architect
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/python-devel

Python 3.4.0 final has been released

[Nick Coghlan]

From Larry's release announcement:

==
Python 3.4 includes a range of improvements of the 3.x series, including
hundreds of small improvements and bug fixes.  Major new features and
changes in the 3.4 release series include:

* PEP 428, a pathlib module providing object-oriented filesystem paths
* PEP 435, a standardized enum module
* PEP 436, a build enhancement that will help generate introspection
   information for builtins
* PEP 442, improved semantics for object finalization
* PEP 443, adding single-dispatch generic functions to the standard library
* PEP 445, a new C API for implementing custom memory allocators
* PEP 446, changing file descriptors to not be inherited by default
   in subprocesses
* PEP 450, a new statistics module
* PEP 451, standardizing module metadata for Python's module import system
* PEP 453, a bundled installer for the *pip* package manager
* PEP 454, a new tracemalloc module for tracing Python memory allocations
* PEP 456, a new hash algorithm for Python strings and binary data
* PEP 3154, a new and improved protocol for pickled objects
* PEP 3156, a new asyncio module, a new framework for asynchronous I/O


To download Python 3.4.0 visit:

http://www.python.org/download/releases/3.4.0/
==

Direct link to the What's New guide:
http://docs.python.org/3/whatsnew/3.4.html

Cheers,
Nick.

-- 
Nick Coghlan
Red Hat Hosted  Shared Services
Software Engineering  Development, Brisbane

Testing Solutions Team Lead
Beaker Development Lead (http://beaker-project.org/)
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/python-devel

Re: Python 3.4.0 final has been released

[Nick Coghlan]

On 03/17/2014 05:53 PM, Nick Coghlan wrote:
 Direct link to the What's New guide:
 http://docs.python.org/3/whatsnew/3.4.html

Rereading that, I remembered there's more to it for Fedora than just
hey, here's a shiny new version of Python 3 to be packaged, and I
don't mean the stuff Slavek is working on to let ensurepip use the
system pip installation as a base.

Specifically, there may need to be a security-related change to the
Python packaging guidelines, covering the appropriate use of isolated
mode: http://docs.python.org/3/whatsnew/3.4.html#whatsnew-isolated-mode

There's also a simpler workaround for the issues with the standard
streams when running things in the POSIX locale: setting
PYTHONIOENCODING=:surrogateescape will deal with it for user mode scripts.

Cheers,
Nick.

-- 
Nick Coghlan
Red Hat Hosted  Shared Services
Software Engineering  Development, Brisbane

Testing Solutions Team Lead
Beaker Development Lead (http://beaker-project.org/)
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/python-devel

tracemallocqt: GUI to analyze Python tracemalloc snapshots

[Nick Coghlan]

Hey folks,

One of the new features coming in Python 3.4 is Victor Stinner's changes
to allow runtime configurable memory allocators, and the associated
tracemalloc module to record details of memory allocations within the
CPython runtime and extensions modules.

Victor is looking for feedback on the GUI he's creating to explore those
snapshots, so I figured I would pass his request along here.

Cheers,
Nick.

P.S. I also tracked down the BZ for the PySide vs Python 3 issue Victor
mentions below: https://bugzilla.redhat.com/show_bug.cgi?id=1045780

-- Forwarded message --
From: Victor Stinner victor.stin...@gmail.com
Date: 14 March 2014 09:54
Subject: [Python-Dev] tracemallocqt: GUI to analyze tracemalloc snapshots
To: Python Dev python-...@python.org


Hi,

To prepare my conference on tracemalloc for Pycon Montréal next month,
I wrote a GUI to analyze tracemalloc snapshots: tracemallocqt.
https://bitbucket.org/haypo/tracemallocqt

It looks like that:

   http://www.haypocalc.com/tmp/tracemallocqt_python34.png

I'm looking for testers and feedback on the GUI. You may be curious to
see which parts of Python is allocating the most memory, or maybe test
your favorite application to check if it leaks memory or how its
memory footprint can be reduced.

To run tracemallocqt, you have to install PySide and get tracemalloc
snapshots. It looks like most Linux distributions provide PySide
packages.

To take snapshots, see tracemalloc documentation:
http://docs.python.org/dev/library/tracemalloc.html

In short, use Python 3.4 with -X tracemalloc=25 command line option
and then call:
---
import pickle, tracemalloc
snap = tracemalloc.take_snapshot()
with open(dump.pickle, wb) as fp: pickle.dump(snap, fp, 2)
snap = None
---

It's more fun with at least 2 snapshots to compare them ;-)

I force pickle version 2 because tracemallocqt currently only works on
Python 2. (Mostly because I don't see how to get PySide for Python 3
on Fedora 20.)

Not all applications work on Python 3.4 right now, you can use the
pytracemalloc project which is tracemalloc for Python 2.5-3.3. Sorry,
you have to patch Python and recompile it. Here are instructions to
install a patched Python 2.7 with tracemalloc:
http://pytracemalloc.readthedocs.org/#manual-installation

If you don't want to take snapshot, you can compare these two files:

   http://www.haypocalc.com/tmp/python34_start.pickle
   http://www.haypocalc.com/tmp/python34_after_import.pickle

The source browser will not find Python source code, because the
snapshots contain absolute paths.

Victor
___
Python-Dev mailing list
python-...@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com


-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia


___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/python-devel

Re: Python 3.4, ensurepip and wheels

[Nick Coghlan]

On 12/12/2013 01:18 AM, Bohuslav Kabrda wrote:
 Well yeah, my point is that there is no upstream-acceptable way other than 
 checking the file hashes by ensurepip, is there? If I wouldn't want to check 
 file hashes, I'd have to query RPM for release - or is there some other way 
 you're thinking of?

I think doing it initially as a downstream only change where you query
RPM will work for now (perhaps by patching the way pip handles the case
where ENSUREPIP_OPTIONS is set?).

By the time this approach is posted upstream, then PEP 426/440 will
hopefully by Final and we can just use the metadata version field
directly rather than needing to grab the release increment from the RPM
repo. (I think this situation provides a good practical use case for why
it's important to standardise this feature upstream, too).

Cheers,
Nick.

-- 
Nick Coghlan
Red Hat Hosted  Shared Services
Software Engineering  Development, Brisbane

Testing Solutions Team Lead
Beaker Development Lead (http://beaker-project.org/)
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/python-devel