Hello,
After (other parts of) $DAY_JOB delayed this for a while, a slightly
updated draft is at https://hackmd.io/XzJe-sHUQvWK7cSrEH_aKg (and attached).
More importantly, here is the requested summary of changes from the old
guidelines, roughly sorted by "importance":
https://hackmd.io/LPfT6FKQSRyMcnqNU-zw2g (and attached).
There were two major objections to the earlier versions. The discussion
died ou, so I'll repeat them. (Both of these should go in a change
document rather than the guidelines).
- Why are the new macros better than the old ones?
They allow other build tools than just Setuptools, use upstream metadata
for BuildRequires.
And they're being improved, with the vision that metadata should be
taken from upstream rather than duplicated in the spec file. (If
upstream is wrong, metadata should be patched just as you'd patch code.)
- Why do project names need to match PyPI?
This is because everywhere else (except in linux distros or tightly
controlled corporate environments), a package name in a Python
requirements list means a name on PyPI. Even at run time, when Python
code checks if a package is installed, it assumes PyPI names.
Maintaining a separate namespace of project names, which could be mixed
with the PyPI namespace, is *hard*. And building automation on top of
such mixed namespaces is harder still.
Companies hit by the "dependency confusion" attack are to pin versions,
use private proxies and/or block PyPI names.
There are ideas to add namespace support to PyPI, and Fedora should
start using that when/if that happens. But until then, I really think
the best we can do as Fedora packagers is in the draft:
If your package is not or cannot be published on PyPI, you can:
- Ask upstram to publish it
- If you wish: publish it to PyPI yourself and maintain it
- Ask Python SIG to block the name on PyPI for you
- Email PyPI admins to block the name for you, giving the project name and
explaining the situation
Packages that aren't on PyPI are so rare, I'll be happy to handle the
blocking myself (and pass the responsibility on within Red Hat
python-maint when I retire from Fedora).
Also, no other ecosystem in Fedora does this. Python would be the first.
If the other ecosystems also use a flat package namespace, and want to
automate packaging, they'll probably run into the same problem.
Ultimately, syncing Fedora with the wider Python ecosystem is the main
idea behind the draft. I'd be glad to hear how it can be done better,
but to me, the new guidelines wouldn't make sense without this part.
On 4/30/20 3:41 PM, Petr Viktorin wrote:
Hello!
Below is a draft of new Packaging Guidelines! It's full of unfinished
areas (marked with XXX), but it's ready for scrutiny.
A possibly updated version is on
https://hackmd.io/XzJe-sHUQvWK7cSrEH_aKg?view
Generally, for rules marked **SHOULD** we know of cases where they
should be broken; for things marked **MUST** we don't.
We have tried to only take the Good Existing Things™ (macros, practices)
and revise the rest without thinking about the past much. Some used
technology is unfortunately not compatible with current EPELs, but we
have considered Fedora 31+. Using the current Python guidelines will
still be an option for people who target EPEL 6/7/8.
The main controversial idea (of many) is synchronizing Fedora's names
(a.k.a. `python3dist(...)`, a.k.a. `name` in `setup.py`) with the Python
Package Index (PyPI, pypi.org), which has by now become the de-facto
canonical namespace for freely redistributable Python packages.
We believe that this will help both Fedora and the greater Python
ecosystem, but there will definitely be some growing pains.
Most of Fedora Python packages already are on PyPI, but more than 250
are missing. There is software developed within Fedora (e.g. pagure,
fedpkg, rust2rpm); projects that aren't primarily Python packages (e.g.
perf, ceph, xen, setroubleshoot, iotop); and others.
A full list is attached. The names have been temporarily blocked on PyPI
to keep trolls from taking them while this is discussed.
Over at the Python Discourse we are discussing how to properly handle
these; once that discussion is settled they should be unblocked:
https://discuss.python.org/t/pypi-as-a-project-repository-vs-name-registry-a-k-a-pypi-namesquatting-e-g-for-fedora-packages/4045
Another general idea is that package metadata should be kept upstream;
the spec file should duplicate as little of it as possible. Any
adjustments should be done with upstreamable patches.
The draft lives on hackmd.io, which we found easy to collaborate with.
If you have an account there, we can add you. If you'd like to
collaborate some other way, let us know.
Petr and Miro
-----------------------
Current draft for inline comments:
> [IMPORTANT]
> This is a DRAFT; it is not in effect yet.
# Python Packaging Guidelines (draft)
> [IMPORTANT]
> This is a *beta* version of the Python Packaging Guidelines and the
associated RPM macros.
> Packagers that opt in to following this version **MUST** be prepared
to change their packages frequently when the guidelines or macros are
updated.
> These packagers **SHOULD** join [Python SIG mailing
list](https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/)
and monitor/start conversations there.
These Guidelines represent a major rewrite and paradigm shift, and not
all packages are updated to reflect this.
Older guidelines are still being kept up to date, and existing packages
**MAY** use them instead of this document:
* 201x-era Python packaging guidelines (for packages that use e.g.
`%py3_install` or `setup.py install`)
* Python 2 appendix (for e.g. `%py2_install`) (requires FESCo exception)
> [NOTE]
> These guidelines only support Fedora 31+. For older releases (such as
in EPEL 8), consult the older guidelines (XXX link).
The two "Distro-wide guidelines" below apply to all software in Fedora
that uses Python at build- or run-time.
These rest of the Guidelines apply to packages that ship code that *can*
be imported in Python.
Specifically, that is all packages that install files under
`/usr/lib*/python*/`.
Except for the two "Distro-wide guidelines", these Guidelines do not
apply to simple one-file scripts or utilities, especially if these are
included with software not written in Python.
However, if an application (e.g. CLI tool, script or GUI app) needs a
more complex Python library, the library **SHOULD** be packaged as an
importable library under these guidelines.
A major goal for Python packaging in Fedora is to *harmonize with the
wider Python ecosystem*, that is, the [Python Packaging
Authority](https://pypa.io) (PyPA) standards and the [Python Package
Index](https://pypi.org) (PyPI).
Packagers **SHOULD** be prepared to get involved with upstream projects
to establish best practices as outlined here. We wish to improve both
Fedora and the wider Python ecosystem.
> [NOTE]
> Fedora's Python SIG not only develops these guidelines, but it's also
involved in PyPA standards and Python packaging best practices. Check
out [the wiki](https://fedoraproject.org/wiki/SIGs/Python) or [mailing
list](https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/)
if you need help or wish to help out.
## Distro-wide guidelines
### BuildRequire python3-devel
**Every** package that uses Python (at runtime and/or build time),
and/or installs Python modules **MUST** explicitly include
`BuildRequires: python3-devel` in its `.spec` file, even if Python is
not actually invoked during build time.
If the package uses an alternate Python interpreter instead of `python3`
(e.g. `pypy`, `jython`, `python27`), it **MAY** instead require the
coresponding `*-devel` package.
The `*-devel` package brings in relevant RPM macros. It may also enable
automated or manual checks: for example, Python maintainers use this
requirement to list packages that use Python in some way and might be
affected by planned changes.
### Mandatory macros
The following macros **MUST** be used where applicable.
The expansions in parentheses are provided only as reference/examples.
The macros are defined for you in all supported Fedora and EPEL versions.
* `%{python3}` (`/usr/bin/python3`)
The Python interpreter.
For example, this macro should be used for invoking Python from a
`spec` file script, passed to `configure` scripts to select a Python
executable, or used as `%{python3} -m pytest` to run a Python-based tool.
> XXX: Use shebang opts for shebang; document pathfix macro. Document
cases where you don't want this.
* `%{python3_version}` (e.g. `3.9`)
Version of the Python interpreter.
* `%{python3_version_nodots}` (e.g. `39`)
Version of the Python interpreter without the dot.
* `%{python3_sitelib}` (e.g. `/usr/lib/python3.9/site-packages`)
Where pure-Python modules are installed.
* `%{python3_sitearch}` (e.g. `/usr/lib64/python3.9/site-packages`)
Where Python3 extension modules (native code, e.g. compiled from C)
are installed.
The rest of this document uses these macros, along with `%{_bindir}`
(`/usr/bin/`), instead of the raw path names.
## Python implementation support
Fedora primarily targets *CPython*, the reference implementation of the
Python language. We generally use “Python” to mean CPython.
Alternate implementations like `pypy` are available, but currently lack
comprehensive tooling and guidelines. When targetting these, there are
no hard rules (except the general Fedora packaging guidelines). But
please try to abide by the *spirit* of these guidelines. When in doubt,
consider consulting the Python SIG.
## Python version support
Fedora packages **MUST NOT** depend on other versions of the CPython
interpreter than the current `python3`.
In Fedora, Python libraries are packaged for a single version of Python,
called `python3`. For example, in Fedora 32, `python3` is Python 3.8.
In the past, there were multiple Python stacks, e.g. `python3.7` and
`python2.7`, installable together on the same machine. That is also the
case in some projects that build *on top* of Fedora, like RHEL, EPEL and
CentOS. Fedora might re-introduce parallell-installable stacks in the
future (for example if a switch to a new Python version needs a
transition period, or if enough interested maintainers somehow appear).
> XXX dots in package names!
Fedora does include alternate interpreter versions, e.g. `python2.7` or
`python3.5`, but these are meant only for developers that need to test
upstream code. Bug and security fixes for these interpreters only cover
this use case.
Packages such as `pip` or `tox`, which enable setting up isolated
environments and installing third-party packages into them, **MAY**, as
an exception to the rule above, use these interpreters as long as this
is coordinated with the maintainers of the relevant Python interpreter.
## BuildRequire pyproject-rpm-macros
While these guidelines are in Beta, each Python package **MUST** have
`BuildRequires: pyproject-rpm-macros` to access the beta macros.
(When we go out of Beta, listing the dependency won't be necessary:
we'll make `python3-devel` require it.)
## Naming
Python packages have several different names, which should be kept in
sync but will sometimes differ for historical or practical reasons. They
are:
* the Fedora *source package name* (or *component name*, `%{name}`),
* the Fedora *built RPM name*,
* the *project name* used on *PyPI* or by *pip*, and
* the *importable module name* used in Python (a single package may have
multiple importable modules).
Some examples (both good and worse):
| Fedora component | Built RPM | Project name | Importable
module |
| ----------------- | ------------------ | -------------- |
-------------------- |
| `python-requests` | `python3-requests` | `requests` | `requests`
|
| `PyYAML` | `python3-pyyaml` | `pyyaml` | `yaml`
|
| `python-ldap` | `python3-ldap` | `python-ldap` | `ldap`,
`ldif`, etc. |
| `python-pillow` | `python3-pillow` | `pillow` | `PIL` |
Elsewhere in this text, the metavariables `SRPMNAME`, `RPMNAME`,
`PROJECTNAME`, `MODNAME` refer to these names, respectively.
### Canonical project name
Most of these names are case-sensitive machine-friendly identifiers, but
the *project name* has human-friendly semantics: it is case-insensitive
and treats some sets of characters (like `._-`) specially.
For automated use, it needs to be
[normalized](https://www.python.org/dev/peps/pep-0503/#normalized-names)
to a canonical format used by Python tools and services such as
setuptools, pip and PyPI.
The `%{py_dist_name}` macro implements this normalization.
Elsewhere in this text, the metavariable `DISTNAME` refers to the
canonical form of the project name.
> XXX
> ```spec
> # XXX specfile
> %py_set_name Django
> -> %distname django
> -> %pojectname Django
> ```
>
> XXX rewrite: It is customary to define the macro `%{distname}` as the
canonical project name and use it throughout the `spec` file. Some
helper macros use `%{distname}` for this purpose by default.
>
> XXX in some places, the "original project name" must be used -- e.g.
`%pypi_source` and `%autosetup` need `Django`, not `django`.
> XXX The following sections should supersede the [Python section on
the Naming
guidelines](https://docs.fedoraproject.org/en-US/packaging-guidelines/Naming/#_python_modules).
### Library naming
A built (i.e. non-SRPM) package for a *Python library* **MUST** be named
with the `python3-` prefix.
A source package containing primarily a *Python library* **MUST** be
named with the prefix `python-`.
The Fedora package's name **SHOULD** contain the *canonical project
name*. If possible, the project name **SHOULD** be the same as the name
of the main importable module, with underscores (`_`) replaced by dashes
(`-`).
A *Python library* is a package meant to be imported in Python, such as
with `import requests`.
Tools like *Ansible* or *IDLE*, whose code is importable but not
primarily meant to be imported, are not considered libraries in this
sense, so this section does not apply for them.
(See the [general Libraries and Applications
guidelines](https://docs.fedoraproject.org/en-US/packaging-guidelines/#_libraries_and_applications)
for details.)
If the importable module name and the project name do not match, users
frequently end up confused. In this case, packagers *should* ensure that
upstream is aware of the problem and (especially for new packages where
renaming is feasible) strive to get the package renamed. The Python SIG
is available for assistance.
The Fedora package name should be formed by taking the *project name*
and prepending `python-` if it does not already start with `python-`.
This may leads to conflicts (e.g. between
[bugzilla](https://pypi.org/project/bugzilla/) and
[python-bugzilla](https://pypi.org/project/python-bugzilla/)). In that
case, ensure upstream is aware of the potentially confusing naming and
apply best judgment.
### Application naming
Packages that primarily provide executables **SHOULD** be named
according to the general Fedora guidelines (e.g. `ansible`).
Consider adding a virtual provide according to Library naming above
(e.g. `python3-PROJECTNAME`), if it would help users find the package.
## Files to include
### Source files and bytecode cache
Packages **MUST** include the source file (`*.py`) **AND** the bytecode
cache (`*.pyc`) for each pure-Python module.
The cache files are found in a `__pycache__` directory and have an
interpreter-dependent suffix like `.cpython-39.pyc`.
The cache is not necessary to run the software, but if it is not found,
Python will try to create it. If this succeeds, the file is not tracked
by RPM and it will linger on the system after uninstallation. If it does
not succeed, users can get spurious SELinux AVC denials in the logs.
Normally, byte compilation (generating the cache files) is done for you
by the `brp-python-bytecompile` script (XXX link to BRP guidelines),
which runs automatically after the `%install` section of the spec file
has been processed. It byte compiles any `.py` files that it finds in
`%{python3_sitelib}` or `%{python3_sitearch}`.
You must include these files of your package (i.e. in the `%files`
section).
If the code is in a subdirectory (importable package), include the
entire directory:
```
%files
%{python3_sitelib}/foo/
```
Adding the trailing slash is best practice for directories.
However, this cannot be used for top-level scripts (those directly in
e.g. `%{python3_sitelib}`), because both `%{python3_sitelib}` and
`%{python3_sitelib}/__pycache__/` are owned by Python itself.
Here, the `%pycached` macro can help. It expands to the given `*.py`
source file and its corresponding cache file(s). For example:
```
%files
%pycached %{python3_sitelib}/foo.py
```
expands roughly to:
```
%files
%{python3_sitelib}/foo.py
%{python3_sitelib}/__pycache__/foo.cpython-3X{,.opt-?}.pyc
```
#### Manual byte compilation
If you need to bytecompile stuff outside of
`%{python3_sitelib}`/`%{python3_sitearch}`, use the `%py_byte_compile`
macro.
For more details on `%py_byte_compile` and on the internals of bytecode
compilation, please see the
[appendix](https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_Appendix/#manual-bytecompilation).
> XXX: Copy the section from appendix here
### Dist-info metadata
Each Python package **MUST** include *Package Distribution Metadata*
conforming to [PyPA
specifications](https://packaging.python.org/specifications/)
(specifically, [Recording installed
distributons](https://packaging.python.org/specifications/recording-installed-packages/)).
This applies to libraries (e.g. `python-requests`) as well as tools
(e.g. `ansible`).
> XXX what with splitting into subpackages? 1) dist-info always
installed, 2) dist-info installed trough a metapackage?
> * Ideally, do the split upstream
> * Consider package split between library & tool (see poetry, fedpkg)
>
> e.g.
> When software is split into several subpackages, it is OK to only
ship metadata in one built RPM.
The metadata takes the form of a `dist-info` directory installed in
`%{python3_sitelib}` or `%{python3_sitearch}`, and contains information
that tools like
[`importlib.metadata`](https://docs.python.org/3/library/importlib.metadata.html)
use to introspect installed libraries.
> XXX example %files with manual dist-info entry
Note that some older tools instead put metadata in an `egg-info`
directory, or even a single file.
This won't happen if you use the `%pyproject_wheel` macro.
If your package uses a build system that generates an `egg-info`
directory or file, please contact Python SIG.
> XXX We need a better solution before we go out of beta.
As an exception, the Python standard library **MAY** ship without this
metadata.
### Explicit lists
Packagers **SHOULD NOT** simply glob everything under a shared directory.
In particular, the following **SHOULD NOT** be used:
* `%{python3_sitelib}/*`
* `%{python3_sitearch}/*`
* `%{python_sitelib}/*`
* `%{python_sitearch}/*`
* `%{_bindir}/*`
* `%pyproject_save_files *`
* `%pyproject_save_files +bindir`
This rule serves as a check against common mistakes which are otherwise
hard to detect. It does limit some possibilities for automation.
The most common mistake this rule prevents is installing a test suite
system-wide as an importable module named `test`, which would then
conflict with other such packages.
## PyPI parity
Every Python package in Fedora **SHOULD** also be available on [the
Python Package Index](https://pypi.org) (PyPI).
The command `pip install PROJECTNAME` **MUST** install the same package
(possibly in a different version), install nothing, or fail with a
reasonable error message.
If this is not the case, the packager **MUST** contact upstream about
this. The goal is to get the project name registered or blocked on PyPI,
or to otherwise ensure the rule is followed.
> XXX Note that project names that were in Fedora but not on PyPI when
these guidelines were proposed are *blocked* as we discuss how they
should be handled.
> This prevents potential trolls, but also legitimate owners, from
taking them.
This is necessary to protect users, avoid confusion, and enable
automation as Fedora and upstream ecosystems grow more integrated.
As always, specific exceptions can be granted by FPC (XXX link
exceptions rules).
> XXX Write an automated check for this.
## Provides and requirements
### Provides for importable modules
For any module intended to be used in Python 3 with `import MODNAME`,
the package that includes it **SHOULD** provide `python3-MODNAME`, with
underscores (`_`) replaced by dashes (`-`).
This is of course always the case if the package is named
`python3-MODNAME`. If the subpackage has some other name, then add
`%py_provides python3-MODNAME` explicitly. See the following section to
learn about `%py_provides`.
### Automatic unversioned provides
For any `FOO`, a package that provides `python3-FOO` **SHOULD** use
`%py_provides` or an automatic generator to also provide `python-FOO`.
The provide **SHOULD NOT** be added manually: if the generator or macro
is not used, the provide shall not be added at all.
On Fedora 33+, this is done automatically for package names by a
generator. The generator can be disabled by undefining
`%__pythonname_provides`.
The following macro invocation will provide both `python3-FOO` and
`python-FOO`:
%py_provides python3-FOO
> XXX: finalize `%py_provides`
Using the generator or macro is important, because the specific form of
the provide may change in the future.
### Machine-readable provides
Every Python package **MUST** provide `python3dist(DISTNAME)` **and**
`python3.Xdist(DISTNAME)`, where `X` is the minor version of the
interpreter and `DISTNAME` is the *canonical project name* corresponding
to the *dist-info metadata*, for example `python3.9dist(requests)`.
(XXX: add links to previous sections)
This is generated automatically from the dist-info metadata.
The provide **SHOULD NOT** be added manually: if the generator fails to
add it, the metadata **MUST** be fixed.
If necessary, the automatic generator can be disabled by undefining
`%__pythondist_provides`.
These *Provides* are used for automatically generated *Requires*.
### Dependencies
As mentioned above, each Python package **MUST** explicitly BuildRequire
`python3-devel`.
Packages **MUST NOT** have dependencies (either build-time or runtime)
with the unversioned prefix `python-` if the corresponding `python3-`
dependency can be used instead.
Packages **SHOULD NOT** have explicit dependencies (either build-time or
runtime) with a minor-version prefix such as `python3.8-` or
`python3.8dist(`. Such dependencies **SHOULD** instead be automatically
generated or a macro should be used to get the version.
Packages **SHOULD NOT** have an explicit runtime dependency on `python3`.
Instead of depending on `python3`, packges have an automatic dependency
on `python(abi) = 3.X` when they install files to `%{python3_sitelib}`
or `%{python3_sitearch}`, or they have an automatic dependency on
`/usr/bin/python3` if they have executable Python scripts, or they have
an automatic dependency on `libpython3.X.so.1.0()` if they embed Python.
These rules help ensure a smooth upgrade path when `python3` is updated
in new versions of Fedora.
### Automatically generated dependencies
Packages **MUST** use the automatic Python run-time dependency generator.
Packages **SHOULD** use the opt-in build-dependency generator if possible.
Any necessary changes **MUST** be done by patches or modifying the
source (e.g. with `sed`), rather than disabling the generator. The
resulting change **SHOULD** be offered to upstream. As an exception,
[filtering](https://docs.fedoraproject.org/en-US/packaging-guidelines/AutoProvidesAndRequiresFiltering/)
**MAY** be used for temporary workarounds and bootstrapping.
Dependencies covered by the generators **SHOULD NOT** be repeated in the
`.spec` file. (For example, if the generator finds a `requests`
dependency, then `Requires: python3-requests` is redundant.)
The automatically generated requirements are in the form
`python3.Xdist(DISTNAME)`, potentially augmented with version
requirements or combined together with [rich
dependencies](https://rpm.org/user_doc/boolean_dependencies.html).
Note that the generators only cover Python packages. Other dependencies,
often C libraries like `openssl-devel`, must be specified in the `.spec`
file manually.
> XXX When implemented, this goes here: Alternatively, upstream Python
packages may list non-Python dependencies in the
`[tool.fedora.requires]`/`[tool.fedora.buildrequires]` section of
`pyproject.toml`. This is non-standard and only recommended for
Fedora-related packages.
Where the requirements are specified in the source depends on each
project's build system and preferences. Common locations are
`pyproject.toml`, `setup.py`, `setup.cfg`, `config.toml`.
#### Run-time dependency generator
The automatic runtime dependency generator uses package metadata (as
recorded in installed `*.dist-info` directories) to determine what the
package depends on.
In an emergency, you can opt-out from running the requires generator by
adding `%{?python_disable_dependency_generator}` to the package
(usually, just before the main package’s `%description`).
#### Build-time dependency generator
The opt-in (but strongly recommended) build-time dependency generator
gathers information from [`pyproject.toml` build-system
information](https://www.python.org/dev/peps/pep-0517/#source-trees)
(with fallback to `setuptools`) plus a standardized [build-system
hook](https://www.python.org/dev/peps/pep-0517/#get-requires-for-build-wheel)
to gather further requirements. See `%pyproject_buildrequires` (XXX
link) for more details.
### Test dependencies
See the *Tests* section. (XXX link.)
### Extras
> XXX No story here so far.
> XXX Note:
[python-django-storages](https://src.fedoraproject.org/rpms/python-django-storages/blob/master/f/python-django-storages.spec),
`drf-yasg` etc. do `%python_provide python3-%{srcname}+%{distname}` and
`python3dist(%{srcname}/%{extraname})` XXX
`python3dist(%{srcname}[%{extraname}])`
## Interpreter invocation
### Shebangs
Shebangs lines to invoke Python **SHOULD** be `#!%{python3}
-%{py3_shebang_flags}` and it **MAY** include extra flags.
> XXX define py3_shebang_flags
If the default flags from `%{py3_shebang_flags}` are not desirable,
packages **SHOULD** explicitly redefine the macro to remove them.
Using `#!%{python3}` (`#!/usr/bin/python3`) rather than e.g.
`#!/usr/bin/env python` ensures that the system-wide Python interpreter
is used to run the code, even if the user modifies `$PATH` (e.g. by
activating a virtual environment).
By default, `-%{py3_shebang_flags}` expands to `-s`, which means *don't
add user site directory to `sys.path`*. That ensures user-installed
Python packages (e.g. by `pip install --user`) don't interfere with the
RPM installed software. Sometimes, `pip`-installed content is desirable,
such as with plugins. Redefining `%{py3_shebang_flags}` to not include
`s`, rather than not using the macro at all, ensures that existing or
future automation won't add the flag.
The `%pyproject_install` macro automatically changes all Python shebangs
in `%{buildroot}%{_bindir}/*` to use `%{python3}` and add
`%{py3_shebang_flags}` to the existing flags. If you're not using the
macro or you need to change a shebang in a different directory, you can
use the `pathfix.py` command as follows:
```
pathfix.py -n -p -k -i %{python3} -a %{py3_shebang_flags} SCRIPTNAME …
```
> XXX Ouch! Macroize this? Hell yes!
>
> `%py3_shebang_fix <paths>` -- also rename `%{py3_shebang_flags}`
(doesn't include dash)
>
> can be used in `%prep` or `%install`
where:
* `-n` disables ceating backups
* `-p` preserves timestamps
* `-k` keeps existing flags
* `-i` specifies the new interpreter
### Invokable Python modules
Every executable `TOOL` for which the current version of Python matters
**SHOULD** also be invokable by `python3 -m TOOL`.
If the software doesn't provide this functionality, packagers **SHOULD**
ask the upstream to add it.
This applies to tools that modify the current Python environment (like
installing or querying packages), use Python for configuration, or use
Python to run plugins.
It does not apply to tools like GIMP or Bash which support plugins in
multiple languages and/or have other means to specify the interpreter.
For example, `pip` can be invoked as `python3 -m pip`.
This allows users to accurately specify the Python version used to run
the software. This convention works across different environments that
might not always set `$PATH` or install scripts consistently.
## Using Cython
Tightening the general Fedora policy, packages **MUST NOT** use files
pre-generated by Cython. These **MUST** be deleted in `%prep` and
regenerated during the build.
As an exception, these sources **MAY** be used temporarily to prevent
build time circular dependencies by following the
[bootstrapping](https://docs.fedoraproject.org/en-US/packaging-guidelines/#bootstrapping)
guidelines.
Generated files (the ones that must be deleted) have a generic `.c` or
`.cpp` extension.
Cython source files (which should stay) usually have the `.pyx` or
`.pxd` extension.
Cython is a popular tool for writing extension modules for Python. If
compiles a Python-like language to C, which is then fed to the C compiler.
Historically, Cython was hard to use upstream as a build-time
dependency. Many projects include pre-generated C files in source
distribution to avoid users from needing to install the tool.
Cython uses CPython's fast-changing internal API for performance
reasons. For a new release of Python, Cython generally needs to be
updated and the C files regenerated. In Fedora, this is frequently
needed before upstreams release re-generated sources (e.g. for Alpha
versins of Python).
Since we do not have a problem with build-time dependencies, we always
want to run the Cython step.
> XXX example spec snippet
## Tests
### Running tests
If a test suite exists, it **MUST** be run in the `%check` section
and/or in Fedora CI.
You **MAY** exclude specific failing tests.
You **MUST NOT** disable the entire testsuite or ignore the result to
solve a build failure.
As an exception, you **MAY** disable tests with an appropriate `%if`
conditional (e.g. bcond) when
[bootstrapping](https://docs.fedoraproject.org/en-US/packaging-guidelines/#bootstrapping).
A popular testing tool, and one which is well integrated in Fedora, is
`tox`. Upstream, it is commonly used to test against multiple Python
versions. In a Fedora package, BuildRequire test dependencies (see *Test
dependencies* below) and run `tox` with:
```
%tox
```
This sets up the environment (`$PATH`, `$PYTHONPATH`,
`$TOX_TESTENV_PASSENV`) and instructs `tox` to use the current
environment rather than create new ones.
For more options, see *Build macros* (XXX link to section).
When upstream doesn't use `tox`, the tests need to be run directly
depending on upstream choice of a test runner. A popular runner is
`pytest`, which can be run against the package to be installed using:
```
export PATH=%{buildroot}%{_bindir}
export
PYTHONPATH=%{buildroot}%{python3_sitearch}:%{buildroot}%{python3_sitelib}
/usr/bin/pytest # XXX `%{python} -m pytest` doesn't work :( -- sees PWD
```
> XXX Do we want a macro for that? We do, because not all projects have
tox.
>
> Define %__pytest or %_pytest %pytest_command - /usr/bin/pytest
>
> Define %pytest (see above snippet) (not parametric)
Use positional arguments to specify test directory, and `-k` to select
tests (e.g. `-k "not network"` may deselect all network-related tests).
### Test dependencies
One part of the Python packaging ecosystem that is still not
standardized is specifying test dependencies (and development
dependencies in general).
The best practice to specify tests is using an extra (XXX link to
section above, which should be fleshed out) like `[test]` or `[dev]`. In
this case, upstream's instructions to install test dependencies might
look like `pip install -e.[test]`.
Projects using `tox` usually specify test dependencies in a
`tox`-specific format: a
[requires](https://tox.readthedocs.io/en/latest/config.html#conf-requires)
key in the configuration.
Both forms are handled by the `%pyproject_buildrequires` macro, see below.
If upstream does not use either form, list test dependencies as manual
*BuildRequires* in the `spec` file.
### Linters
In `%check`, packages **SHOULD NOT** run “linters”: code style checkers,
test coverage checkers and other tools that check code quality rather
than functionality.
Tools like `black`, `pylint`, `flake8`, or `mypy` are often
“opinionated” and their “opinions” change frequently enough that they
are nuisance in Fedora, where the linter is not pinned to an exact version.
Furthermore, some of these tools take a long time to adapt to new Python
versions, preventing early testing with Aplha and Beta releases of Python.
And they are just not needed: wrongly formatted code is not important
enough for the Fedora packager to bug the upstream about it.
Making such an issue break a package build is entirely unreasonable.
Linters *do* make sense in upstream CI. But not in Fedora.
If a linter is used, disable it and remove the dependency on it. If that
is not easy, talk to upstream about making it easy (for example with a
configuration option or a separate `tox` environment).
For packages that contain such linters, use them at runtime or extend
them, you will usually need to run the linter in `%check`. Run it to
test functionality, not code quality of the packaged software.
## Source files from PyPI
Packages **MAY** use sources from PyPI.
However, packages **SHOULD NOT** use an archive that omits test suites,
licences and/or documentation present in other source archives.
For example, as of this writing `pip` provides a [source tarball
(“sdist”)](https://pypi.org/project/pip/#files) which omits the
relatively large `tests` and `docs` directories present in [the source
on GitHub](https://github.com/pypa/pip). In this case, the tarball from
GitHub should be used (XXX link to source URL guidelines).
When using sources from PyPI, you can use the `%pypi_source` macro to
generate the proper URL. See the *Macro reference* (XXX link directly to
macro) for details.
## Sample spec file
The following is a viable spec file for a hypothetical Python library
called `pello` that follows packaging best practices.
> XXX *Need to get `Pello` uploaded as real example and on GH under
fedora-python.
```
%py_set_name Pello
Name: python-%{distname}
Version: 1.2.3
Release: 1%{?dist}
Summary: Example Python library
License: MIT
URL: https://github.com/fedora-python/Pello
Source0: %{pypi_source}
BuildArch: noarch
BuildRequires: python3-devel
BuildRequires: pyproject-rpm-macros
%global _description %{expand:
A python module which provides a convenient example.
This description provides some details.}
%description %_description
%package -n python3-%{distname}
Summary: %{summary}
%description -n python3-%{distname} %_description
%prep
%autosetup -p1 -n %{projectname}-%{version}
%generate_buildrequires
%pyproject_buildrequires -t
%build
%pyproject_wheel
%install
%pyproject_install
# Here, "pello" is the name of the importable module.
# You may use %%{distname} here if it's the same.
%pyproject_save_files pello
%check
%tox
# Note that there is no %%files section for
# the unversioned python module, python-%%{distname}
%files -n python3-%{distname} -f %{pyproject_files}
%doc README.md
%license LICENSE
%{_bindir}/pello-greeting
```
## Macro Reference
See the *Mandatory macros* section above. (XXX link to section)
<!-- Keep order and examples the same as in Mandatory macros -->
* `%{python3}` (`/usr/bin/python3`)
* `%{python3_version}` (e.g. `3.9`)
* `%{python3_version_nodots}` (e.g. `39`)
* `%{python3_sitelib}` (e.g. `/usr/lib/python3.9/site-packages`)
* `%{python3_sitearch}` (e.g. `/usr/lib64/python3.9/site-packages`)
### Name macros
* `%py_set_name PROJECTNAME` (e.g. `%py_set_name Django`)
Sets `%{projectname}` to the given name, and `%{distname}` to the
canonical version of it. These macros can then be used throughout the
`spec` file.
See *Naming* for details and *Sample spec file* for examples. (XXX
link to sections)
* `%{projectname}` (e.g. `Django`)
Set by `%py_set_name` to the *project name* of the software.
* `%{distname}` (e.g. `django`)
Set by `%py_set_name` to the *canonical project name* of the software.
### Shebang macro
* `%{py3_shebang_flags}` (`s`)
Flags for `%{python3}` to use in shebangs.
Redefine this macro to use a different set of flags. See *Shebangs*
[XXX link section] for details.
### Convenience macros
* `%{pypi_source [PROJECTNAME [VERSION [EXT]]]}` (e.g.
`https://.../Django-3.0.5.tar.gz`)
Evaluates to the appropriate URL for source archive hosted on PyPI.
Accepts up to three optional arguments:
1. The name of the PyPI project. Defaults to `%srcname` if defined,
or to `%pypi_name` if defined, or to `%name` (the package name). [XXX
default to `%projectname`]
2. The version of the PyPI project. Defaults to `%version` (the
package version).
3. The file extension to use. Defaults to `tar.gz`.
In most cases it is not necessary to specify any arguments.
* `%{python3_platform}` (e.g. `linux-x86_64`)
The platform name. Used in some Python build systems.
### Build macros
These macros are most useful for packaging Python projects that use the
`pyproject.toml` file defined in [PEP
518](https://www.python.org/dev/peps/pep-0518/) and [PEP
517](https://www.python.org/dev/peps/pep-0517/), which specifies the
package's build dependencies (including the build system, such as
setuptools, flit or poetry).
If `pyproject.toml` is not found, the macros automatically fall backs to
using `setuptools` with configuration in `setup.cfg`/`setup.py`.
A full tutorial and discussion for the macros is available in the
macros' [README](https://src.fedoraproject.org/rpms/pyproject-rpm-macros/).
* `%pyproject_wheel`
Build the package. Commonly, this is the only macro needed in the
`%build` section.
This macro needs BuildRequires generated by `%pyproject_buildrequires`.
* `%pyproject_install`
Install the package built by `%pyproject_wheel`.
This macro needs BuildRequires generated by `%pyproject_buildrequires`.
* `%pyproject_buildrequires`
Generate BuildRequires for the package. Used in the
`%generate_buildrequires` section of the `spec` file. The macro has
these options:
* `-r`: Include build-time requirements (commonly needed for `%check`).
* `-x EXTRA`: Include dependencies given by the given *extra* (XXX
link). [XXX Multiple comma separated values cannot be given yet.]
* `-t`: Include dependencies for the default *tox* environment.
Implies `-r`.
* `-e ENV`: Include dependencies for the given *tox* environment, and
save the `ENV` name as `%{toxenv}`. Implies `-r`. Multiple comma
separated values can be given, for example:
%pyproject_buildrequires -e %{toxenv}-unit,%{toxenv}-integration
* `%pyproject_save_files MODNAME …`
Generate a list of files corresponding to the given importable
modules, and save it as `%{pyproject_files}`.
Note that README and licence files are not included.
Also, while the macro allows including executable files (using the
`+bindir` flag), this feature **MUST NOT** be used in Fedora.
The `MODNAME` may be a glob pattern, which should be specific to
your package. As mentioned in the *Explicit lists* section, expressions
like `%pyproject_save_files *` are not acceptable.
* `%{pyproject_files}`
Path of the file written by `%pyproject_save_files`, to be used as:
%files -n python3-%{distname} -f %{pyproject_files}
* `%tox`
Run tests using `tox`.
A different environment may be specified with `-e`, for example:
```
%check
%tox
%if %{with integration_test}
%tox -e %{toxenv}-integration
%endif
```
Flags for the `tox` command can be specified after `--`:
%tox -- --parallel 0
Additional arguments for the test runner may be specified after
another `--`:
%tox -- --parallel 0 -- --verbose tests/*
* `%{toxenv}`
The *tox* environment used by the `%tox` macro.
Can be overridden manually or with `%pyproject_buildrequires -t ENV`.
* `%{default_toxenv}` (e.g. `py39`)
The system-wide default value of `%{toxenv}`.
### Manual Generation
The following macros are available for cases where automatic generation
is turned off.
They can also be useful for handling files in non-standard locations
where the generators don't look.
* `%pycached MODNAME.py`
Given a Python file, lists the file and the files with its bytecode
cache. See *Source files and bytecode cache* for more information.
* `%{py_provides python3-MODNAME}`
> XXX
See *The `%python_provide` macro* for more details.
* `%{py_byte_compile INTERPRETER PATH}`
> XXX`%{python3_byte_compile PATH}` XXX?
Byte-compile a Python file into a `__pycache__/*.pyc`.
See [Manual byte
compilation](https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_Appendix/#manual-bytecompilation)
in the Appendix for usage.
* `%{py_dist_name PROJECTNAME}`
Given a *project name* (e.g. `PyYAML`) it will convert it to the
canonical format (e.g. `pyyaml`). See *Canonical project name* for more
information.
* `%{py3_dist PROJECTNAME …}`
Given one or more *project names*, it will convert them to the
canonical format and evaluate to `python3dist(DISTNAME)`, which is
useful when listing dependencies. See *Machine-readable provides* for
more information.
### System Settings
The following macros can be redefined for special use cases. Most of
such cases are unacceptable in Fedora.
* `%{__python}` (`/usr/bin/python`)
Defining this macro changes the meaning of all “unversioned” Python
macros such as `%{python}` or `%{python_sitelib}`.
Don’t use these macros without redefining `%{__python}`.
* `%{__python3}` (`/usr/bin/python3`)
The python 3 interpreter. Redefining this macro changes all the
`%{python3...}` macros, e.g. `%{python3_sitelib}`.
* `%{python3_pkgversion}` (`3`)
Distro-wide Python version, i.e. the `3` in `python3`.
Projects that build on top of Fedora may define it to e.g. `3.9` to
try allowing multiple Python stacks installable in parallel.
### XXX Conditionals?
> XXX How to properly express: "if python_version >= 3.8"?
> The current way is comparing integers from %python3_version_nodots,
but that will break with 3.10/4.0 comparsion.
> Do a Lua macro that splits the versions and compares them?
> This looks more general, is there something in RPM we can use?
### Disabling automation
The following macros can turn off Python-specific automation.
Consider contacting the Python SIG if you need to do this.
* `%{?python_disable_dependency_generator}`
Disables the automatic dependency generator. See *Automatically
generated dependencies* for details.
* `%__pythonname_provides`
> XXX undefine to disable %python-provides generator
* `%__pythondist_requires`
> XXX undefine to disable %python3dist generator
### Deprecated Macros
The following macros are deprecated. See the *201x-era Python Packaging
guidelines* (XXX link) for how some of them were used.
* `%py3_build`
* `%py3_build_wheel`
* `%py3_build_egg`
* `%py3_install`
* `%py3_install_wheel`
* `%py3_install_egg`
* `%py3dir`
* `%py3_other_build`
* `%py3_other_install`
* `%python_provide` (without `s` at the end)
> XXX: `%pyX_build` breaks shebang when upstream already uses some
flags (https://bugzilla.redhat.com/show_bug.cgi?id=1335203) -- should we
document this in the old guidelines?
## Reviewer checklist
> After the guidelines are done, distill the **MUST**s/**SHOULD**s here.
> XXX: Do we need a checklist at all?
> How do we keep it updated (which hasn't happened in the last N years)
_______________________________________________
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org
> [IMPORTANT]
> This is a DRAFT; it is not in effect yet.
>
> Checklist for public Beta:
> * Update the [Python section on the general Naming guidelines](https://docs.fedoraproject.org/en-US/packaging-guidelines/Naming/#_python_modules) – add link to [Name limitations](#Name-limitations), [Library naming](#Library-naming), [Application naming](#Application-naming); and the old guidelines
> * Pello
> * Triage (and possibly fix) [Pello issues](https://github.com/hrnciar/pello/issues)
> * Update the example spec
> * Write a summary of changes from the older guidelines
> * Convert to to AsciiDoc and sembr
> * Update internal links (esp. link directly to individual macros)
>
> Checklist for going out of Beta:
> * Add a note to the [Manual bytecompilation Appendix](https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_Appendix/#manual-bytecompilation) that it's only for the old guidelines. (The [Manual byte compilation](#Manual-byte-compilation) section and macro docs should cover everything needed for Python 3.)
> * Keep "Byte compilation reproducibility" (https://pagure.io/packaging-committee/pull-request/1013) as is; it applies to both old and new guidelines.
> * Make `python3-devel` BuildRequire `pyproject-rpm-macros`; remove text about requiring it manually
> * Write an automated check for [PyPI parity](#PyPI-parity).
> * Move old guidelines to a sub-page
> * Move these guidelines to the main page
> * Update links in both
> * Rewrite the intro
# Python Packaging Guidelines (draft)
> [IMPORTANT]
> This is a *beta* version of the Python Packaging Guidelines and the associated RPM macros.
> Packagers that opt in to following this version **MUST** be prepared to change their packages frequently when the guidelines or macros are updated.
> These packagers **SHOULD** join [Python SIG mailing list](https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/) and monitor/start conversations there.
These Guidelines represent a major rewrite and paradigm shift, and not all packages are updated to reflect this.
Older guidelines are still being kept up to date, and existing packages **MAY** use them instead of this document:
* 201x-era [Python packaging guidelines](https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/) (for packages that use e.g. `%py3_install` or `setup.py install`)
* [Python 2 appendix](https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_Appendix/#_python_2_packages) (for e.g. `%py2_install`) (Python 2 packages require a FESCo exception.)
> [NOTE]
> These guidelines only support Fedora 31+. For older releases (such as in EPEL 8), consult the [older guidelines](https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/).
The two "[Distro-wide guidelines](#Distro-wide-guidelines)" below apply to all software in Fedora that uses Python at build- or run-time.
The rest of the Guidelines apply to packages that ship code that can be imported with Python's `import` statement.
Specifically, that is all packages that install files under `/usr/lib*/python*/`.
Except for the two "Distro-wide guidelines", these Guidelines do not apply to simple one-file scripts or utilities, especially if these are included with software not written in Python.
However, if an application (e.g. CLI tool, script or GUI app) needs a more complex Python library, the library **SHOULD** be packaged as an importable library under these guidelines.
A major goal for Python packaging in Fedora is to *harmonize with the wider Python ecosystem*, that is, the [Python Packaging Authority](https://pypa.io) (PyPA) standards and the [Python Package Index](https://pypi.org) (PyPI).
Packagers **SHOULD** be prepared to get involved with upstream projects to establish best practices as outlined here. We wish to improve both Fedora and the wider Python ecosystem.
> [NOTE]
> Some build tools (like CMake or autotools) may not work with the latest PyPA standards yet.
> (For example, they might generate `.egg-info` directories rather than `.dist-info`.)
> Packages that use these should follow the [older guidelines](https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_Appendix/#_python_2_packages) for the time being.
> We aim to fine-tune these guidelines on Python-native tooling before fixing third-party buildsystems.
> [NOTE]
> Fedora's Python SIG not only develops these guidelines, but it's also involved in PyPA standards and Python packaging best practices. Check out [the wiki](https://fedoraproject.org/wiki/SIGs/Python) or [mailing list](https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/) if you need help or wish to help out.
## Distro-wide guidelines
### BuildRequire python3-devel
**Every** package that uses Python (at runtime and/or build time), and/or installs Python modules **MUST** explicitly include `BuildRequires: python3-devel` in its `.spec` file, even if Python is not actually invoked during build time.
If the package uses an alternate Python interpreter instead of `python3` (e.g. `pypy`, `jython`, `python2.7`), it **MAY** instead require the corresponding `*-devel` package.
The `*-devel` package brings in relevant RPM macros. It may also enable automated or manual checks: for example, Python maintainers use this requirement to list packages that use Python in some way and might be affected by planned changes.
### Mandatory macros
The following macros **MUST** be used where applicable.
The expansions in parentheses are provided only as reference/examples.
The macros are defined for you in all supported Fedora and EPEL versions.
* `%{python3}` (`/usr/bin/python3`)
The Python interpreter.
For example, this macro should be used for invoking Python from a `spec` file script, passed to `configure` scripts to select a Python executable, or used as `%{python3} -m pip` to run a Python-based tool.
If the packaged software invokes Python at *run time* (as opposed to running Python to build/test it), it might be necessary to pass flags to `%{python3}` to isolate it from user-installed packages.
See the [Shebangs section](#Shebangs) for details.
* `%{python3_version}` (e.g. `3.9`, `3.10`)
Version of the Python interpreter.
* `%{python3_version_nodots}` (e.g. `39`, `310` or `3_10`)
Version of the Python interpreter without the dot.
* `%{python3_sitelib}` (e.g. `/usr/lib/python3.9/site-packages`)
Where pure-Python modules are installed.
* `%{python3_sitearch}` (e.g. `/usr/lib64/python3.9/site-packages`)
Where Python extension modules (native code, e.g. compiled from C) are installed.
The rest of this document uses these macros, along with `%{_bindir}` (`/usr/bin/`), instead of the raw path names.
### Python implementation support
Fedora primarily targets *CPython*, the reference implementation of the Python language. We generally use “Python” to mean CPython.
Alternate implementations like `pypy` are available, but currently lack comprehensive tooling and guidelines for packaging. When targetting these, there are no hard rules (except the general Fedora packaging guidelines). But please try to abide by the *spirit* of these guidelines. When in doubt, consider consulting the Python SIG.
### Python version support
Fedora packages **MUST NOT** depend on other versions of the CPython interpreter than the current `python3`.
In Fedora, Python libraries are packaged for a single version of Python, called `python3`. For example, in Fedora 32, `python3` is Python 3.8.
In the past, there were multiple Python stacks, e.g. `python3.7` and `python2.7`, installable together on the same machine. That is also the case in some projects that build *on top* of Fedora, like RHEL, EPEL and CentOS. Fedora might re-introduce parallell-installable stacks in the future (for example if a switch to a new Python version needs a transition period, or if enough interested maintainers somehow appear).
Fedora does include alternate interpreter versions, e.g. `python2.7` or `python3.5`, but these are meant only for developers that need to test upstream code. Bug and security fixes for these interpreters only cover this use case.
Packages such as `pip` or `tox`, which enable setting up isolated environments and installing third-party packages into them, **MAY**, as an exception to the rule above, use these interpreters as long as this is coordinated with the maintainers of the relevant Python interpreter.
## BuildRequire pyproject-rpm-macros
While these guidelines are in Beta, each Python package wishing to follow these guidelines **MUST** have `BuildRequires: pyproject-rpm-macros` to access the macros.
(When the guidelines are out of Beta, requiring the macros explicitly will not be necessary any more.)
## Naming
Python packages have several different names, which should be kept in sync but will sometimes differ for historical or practical reasons. They are:
* the Fedora *source package name* (or *component name*, `%{name}`),
* the Fedora *built RPM name*,
* the *project name* used on [PyPI](https://pypi.org/) or by [pip](https://pip.pypa.io), and
* the *importable module name* used in Python (a single package may have multiple importable modules).
Some examples (both good and worse):
| Fedora component | Built RPM | Project name | Importable module |
| ----------------- | ------------------ | -------------- | -------------------- |
| `python-requests` | `python3-requests` | `requests` | `requests` |
| `python-django` | `python3-django` | `Django` | `django` |
| `PyYAML` | `python3-pyyaml` | `pyyaml` | `yaml` |
| `python-ldap` | `python3-ldap` | `python-ldap` | `ldap`, `ldif`, etc. |
| `python-pillow` | `python3-pillow` | `pillow` | `PIL` |
Elsewhere in this text, the metavariables `SRPMNAME`, `RPMNAME`, `PROJECTNAME`, `MODNAME` refer to these names, respectively.
### Canonical project name
Most of these names are case-sensitive machine-friendly identifiers, but the *project name* has human-friendly semantics: it is case-insensitive and treats some sets of characters (like `._-`) specially.
For automated use, it needs to be normalized to a canonical format used by Python tools and services such as setuptools, pip and PyPI.
For example, the canonical name of the `Django` project is `django` (in lowercase).
This normalization is defined in [PEP 503](https://www.python.org/dev/peps/pep-0503/#normalized-names), and the [`%{py_dist_name}` macro](#Manual-generation) implements it for Fedora packaging.
Elsewhere in this text, the metavariable `DISTNAME` refers to the canonical form of the project name.
Note that in some places, the original, non-normalized project name must be used. For example, the `%pypi_source` and `%autosetup` macros need `Django`, not `django`.
### Name limitations
The character `+` in names of built packages (i.e. non-SRPM)
that include `.dist-info` or `.egg-info` directories
is reserved for [extras](#Extras)
and **MUST NOT** be used for any other purpose.
As an exception, `+` characters **MAY** appear at the *end* of such names.
The `+` character triggers the automatic dependency generator for extras.
Replace any `+` signs in the upstream name with `-`.
Omit `+` signs on the beginning of the name.
Consider adding `Provides` for the original name with `+` characters
to make the package easier to find for users.
### Library naming
A built (i.e. non-SRPM) package for a *Python library* **MUST** be named with the prefix `python3-`.
A source package containing primarily a *Python library* **MUST** be named with the prefix `python-`.
The Fedora package's name **SHOULD** contain the [canonical project name](#Canonical-project-name).
If possible, the project name **SHOULD** be the same
as the name of the main importable module,
with underscores (`_`) replaced by dashes (`-`).
If the importable module name and the project name do not match,
users frequently end up confused.
In this case, packagers **SHOULD** ensure that upstream is aware of the problem
and (especially for new packages where renaming is feasible)
strive to get the package renamed.
The Python SIG is available for assistance.
A *Python library* is a package meant to be imported in Python,
such as with `import requests`.
Tools like *Ansible* or *IDLE*, whose code is importable
but not primarily meant to be imported from other software,
are not considered libraries in this sense.
So, this section does not apply for them.
(See the [general Libraries and Applications guidelines](https://docs.fedoraproject.org/en-US/packaging-guidelines/#_libraries_and_applications) for general guidance.)
The Fedora component (source package) name for a library should be formed
by taking the *canonical project name* and
prepending `python-` if it does not already start with `python-`.
This may leads to conflicts
(e.g. between [bugzilla](https://pypi.org/project/bugzilla/)
and [python-bugzilla](https://pypi.org/project/python-bugzilla/)).
In that case,
ensure upstream is aware of the potentially confusing naming
and apply best judgment.
### Application naming
Packages that primarily provide applications, services or any kind of executables
**SHOULD** be named according to the general [Fedora naming guidelines](https://docs.fedoraproject.org/en-US/packaging-guidelines/Naming/)
(e.g. `ansible`).
Consider adding a virtual provide
according to [Library naming](#Library-naming) above
(e.g. `python3-PROJECTNAME`),
if it would help users find the package.
## Files to include
### Source files and bytecode cache
Packages **MUST** include the source file (`*.py`)
**AND** the bytecode cache (`*.pyc`)
for each pure-Python importable module.
The source files **MUST** be included in the same package as the bytecode cache.
Scripts that are not importable
(typically ones in `%{_bindir}` or `%{_libexecdir}`)
**SHOULD NOT** be byte-compiled.
The cache files are found in a `__pycache__` directory
and have an interpreter-dependent suffix like `.cpython-39.pyc`.
The cache is not necessary to run the software,
but if it is not found,
Python will try to create it when a module is imported.
If this succeeds,
the file is not tracked by RPM
and it will linger on the system after uninstallation.
If it does not succeed,
users can get spurious SELinux AVC denials in the logs.
Normally, byte compilation (generating the cache files)
is done for you by the `brp-python-bytecompile` [BRP script](https://docs.fedoraproject.org/en-US/packaging-guidelines/#_brp_buildroot_policy_scripts),
which runs automatically
after the `%install` section of the spec file has been processed.
It byte-compiles any `.py` files
that it finds in `%{python3_sitelib}` or `%{python3_sitearch}`.
You must include these files of your package (i.e. in the `%files` section).
If the code is in a subdirectory (importable package),
include the entire directory:
```
%files
%{python3_sitelib}/foo/
```
Adding the trailing slash is best practice for directories.
However, this cannot be used for top-level modules
(those directly in e.g. `%{python3_sitelib}`),
because both `%{python3_sitelib}` and `%{python3_sitelib}/__pycache__/`
are owned by Python itself.
Here, the `%pycached` macro can help.
It expands to the given `*.py` source file
and its corresponding cache file(s).
For example:
```
%files
%pycached %{python3_sitelib}/foo.py
```
expands roughly to:
```
%files
%{python3_sitelib}/foo.py
%{python3_sitelib}/__pycache__/foo.cpython-3X{,.opt-?}.pyc
```
#### Manual byte compilation
If you need to bytecompile stuff
outside of `%{python3_sitelib}`/`%{python3_sitearch}`,
use the [`%py_byte_compile` macro](#Manual-Generation).
For example, if your software adds `%{_datadir}/mypackage`
to Python's import path
and imports package `foo` from there,
you will need to compile `foo` with:
```
%py_byte_compile %{python3} %{buildroot}%{_datadir}/mypackage/foo/
```
### Dist-info metadata
Each Python package **MUST** include *Package Distribution Metadata*
conforming to [PyPA specifications](https://packaging.python.org/specifications/)
(specifically, [Recording installed distributons](https://packaging.python.org/specifications/recording-installed-packages/)).
The metadata **SHOULD** be included in the same subpackage
as the main importable module,
if there is one.
This applies to libraries (e.g. `python-requests`)
as well as tools (e.g. `ansible`).
When software is split into several subpackages,
it is OK to only ship metadata in one built RPM.
In this case,
consider working with upstream to also split the upstream project.
The metadata takes the form of a `.dist-info` directory
installed in `%{python3_sitelib}` or `%{python3_sitearch}`,
and contains information that tools like [`importlib.metadata`](https://docs.python.org/3/library/importlib.metadata.html) use
to introspect installed libraries.
For example, a project named `MyLib` with importable package `mylib`
could be packaged with:
```
%files -p python3-mylib
%{python3_sitelib}/mylib/
%{python3_sitelib}/MyLib-%{version}.dist-info/
%doc README.md
%license LICENSE.txt
```
Note that some older tools instead put metadata in an `.egg-info` directory,
or even a single file.
This won't happen if you use the `%pyproject_wheel` macro.
If your package uses a build system that generates
an `.egg-info` directory or file,
please contact Python SIG.
As an exception,
the Python standard library **MAY** ship without this metadata.
### Explicit lists
Packages **MUST NOT** own shared directories owned by Python itself, such as the top-level `__pycache__` directories (`%{python3_sitelib}/__pycache__`, `%{python3_sitearch}/__pycache__`).
Packagers **SHOULD NOT** simply glob everything under a shared directory.
In particular, the following **SHOULD NOT** be used in `%files`:
* `%{python3_sitelib}/*`
* `%{python3_sitearch}/*`
* `%{python_sitelib}/*`
* `%{python_sitearch}/*`
* `%{_bindir}/*`
* `%pyproject_save_files '*'`
* `%pyproject_save_files +auto`
This rule serves as a check against common mistakes
which are otherwise hard to detect.
It does limit some possibilities for automation.
The most common mistakes this rule prevents are:
* installing a test suite system-wide as an importable module named `test`,
which would then conflict with other such packages, and
* upstream adding commands in `%{_bindir}/*` –
you should always check such changes for
[conflicts](https://docs.fedoraproject.org/en-US/packaging-guidelines/Conflicts/#_common_conflicting_files_cases_and_solutions)
(especially with non-Python software),
and keep the list of such files explicit and auditable.
## PyPI parity
Every Python package in Fedora **SHOULD** also be available
on [the Python Package Index](https://pypi.org) (PyPI).
The command `pip install PROJECTNAME` **MUST** install the same package
(possibly in a different version),
install nothing,
or fail with a reasonable error message.
If this is not the case, the packager **SHOULD** contact upstream about this.
The goal is to get the project name registered or blocked on PyPI,
or to otherwise ensure the rule is followed.
If your package is not or cannot be published on PyPI,
you can:
* Ask upstram to publish it
* If you wish: publish it to PyPI yourself and maintain it
* Ask [Python SIG](mailto:python-devel@lists.fedoraproject.org) to *block* the name on PyPI for you
* Email [PyPI admins](mailto:ad...@pypi.org) to block the name for you, giving the project name and explaining the situation
(for example: the package cannot currently be installed via `pip`).
You can ask questions and discuss the process at the [Python Discourse](https://discuss.python.org/t/block-names/4045).
> NOTE: Project names that were in Fedora but not on PyPI
> when these guidelines were proposed
> are *blocked* from being uploaded to PyPI.
> This prevents potential trolls from taking them,
> but it also blocks legitimate owners.
> If your package is affected, contact the Python SIG
> or [file a PyPA issue](https://github.com/pypa/pypi-support/issues/new?labels=PEP+541&template=pep541-request.md&title=PEP+541+Request%3A+PROJECT_NAME)
> and mention `@encukou`.
If your package's project name conflicts with a different package on PyPI, change the project name.
As painful as it is, we need to use a single global namespace across the Python ecosystem.
Software that is not written specifically for Fedora already expects that project names use the PyPI namespace:
for example, if a third-party library identifies a dependency by name, we don't want that dependency satisfied by an unrelated Fedora package.
As always, [specific exceptions can be granted by the Packaging Committee](https://docs.fedoraproject.org/en-US/packaging-guidelines/#_general_exception_policy).
## Provides and requirements
### Provides for importable modules
For any module intended to be used in Python 3 with `import MODNAME`,
the package that includes it **SHOULD** provide `python3-MODNAME`,
with underscores (`_`) replaced by dashes (`-`).
This is of course always the case if the package is named `python3-MODNAME`.
If the subpackage has some other name,
then add `%py_provides python3-MODNAME` explicitly.
See the following section to learn about `%py_provides`.
### Automatic python- and python3.X- provides
For any `FOO`, a package that provides `python3-FOO`
**SHOULD** use `%py_provides` or an automatic generator
to also provide `python-FOO` and `python3.X-FOO`, where `X` is the minor version of the interpreter.
The provide **SHOULD NOT** be added manually:
if a generator or macro is not used,
do not add the `python-FOO` / `python3.X-FOO` provides at all.
On Fedora 33+, this is done automatically for package names by a generator. If absolutely necessary, the generator can be disabled by undefining [`%__pythonname_provides`](#Disabling-automation).
On releases older than Fedora 33, or (for technical reasons) for packages without files, or for provides that aren't package names, the generator will not work.
For these cases, the following invocation will provide `python3-FOO`, `python-FOO` and `python3.X-FOO`:
%py_provides python3-FOO
Using the generator or macro is important, because the specific form of the provide may change in the future.
### Machine-readable provides
Every Python package **MUST** provide `python3dist(DISTNAME)`
**and** `python3.Xdist(DISTNAME)`,
where `X` is the minor version of the interpreter
and `DISTNAME` is the [canonical project name](#Canonical-project-name)
corresponding to the [dist-info metadata](#Dist-info-metadata).
For example, `python3-django` would provide `python3dist(django)`
and `python3.9dist(django)`.
This is generated automatically from the dist-info metadata.
The provide **SHOULD NOT** be added manually:
if the generator fails to add it, the metadata **MUST** be fixed.
These *Provides* are used for automatically generated *Requires*.
If absolutely necessary, the automatic generator can be disabled by using
[`%{?python_disable_dependency_generator}`](#Disabling-automation).
Consider discussing your use case with the Python SIG if you need to do this.
### Dependencies
As mentioned above, each Python package **MUST** explicitly BuildRequire `python3-devel`.
Packages **MUST NOT** have dependencies (either build-time or runtime) with the unversioned prefix `python-` if the corresponding `python3-` dependency can be used instead.
Packages **SHOULD NOT** have explicit dependencies (either build-time or runtime) with a minor-version prefix such as `python3.8-` or `python3.8dist(`. Such dependencies **SHOULD** instead be automatically generated or a macro should be used to get the version.
Packages **SHOULD NOT** have an explicit runtime dependency on `python3`.
Instead of depending on `python3`, packges have an automatic dependency on `python(abi) = 3.X` when they install files to `%{python3_sitelib}` or `%{python3_sitearch}`, or they have an automatic dependency on `/usr/bin/python3` if they have executable Python scripts, or they have an automatic dependency on `libpython3.X.so.1.0()` if they embed Python.
These rules help ensure a smooth upgrade path when `python3` is updated in new versions of Fedora.
### Automatically generated dependencies
Packages **MUST** use the automatic Python run-time dependency generator.
Packages **SHOULD** use the opt-in build-dependency generator if possible.
The packager **MUST** inspect the generated requires for correctness. All dependencies **MUST** be resolvable within the targeted Fedora version.
Any necessary changes **MUST** be done by patches or modifying the source (e.g. with `sed`), rather than disabling the generator. The resulting change **SHOULD** be offered to upstream. As an exception, [filtering](https://docs.fedoraproject.org/en-US/packaging-guidelines/AutoProvidesAndRequiresFiltering/) **MAY** be used for temporary workarounds and bootstrapping.
Dependencies covered by the generators **SHOULD NOT** be repeated in the `.spec` file. (For example, if the generator finds a `requests` dependency, then `Requires: python3-requests` is redundant.)
The automatically generated requirements are in the form `python3.Xdist(DISTNAME)`, potentially augmented with version requirements or combined together with [rich dependencies](https://rpm.org/user_doc/boolean_dependencies.html).
Any `.0` suffixes are removed from version numbers to match the behavior of Python tools. ([PEP 440](https://www.python.org/dev/peps/pep-0440/) specifies that `X.Y` and `X.Y.0` are treated as equal.)
Note that the generators only cover Python packages. Other dependencies, often C libraries like `openssl-devel`, must be specified in the `.spec` file manually.
Where the requirements are specified in the source depends on each project's build system and preferences. Common locations are `pyproject.toml`, `setup.py`, `setup.cfg`, `config.toml`.
#### Run-time dependency generator
The automatic runtime dependency generator uses package metadata (as recorded in installed `*.dist-info` directories) to determine what the package depends on.
In an emergency, you can opt-out from running the requires generator by adding `%{?python_disable_dependency_generator}` to the package (usually, just before the main package’s `%description`).
#### Build-time dependency generator
The opt-in (but strongly recommended) build-time dependency generator gathers information from [`pyproject.toml` build-system information](https://www.python.org/dev/peps/pep-0517/#source-trees) (with fallback to `setuptools`) plus a standardized [build-system hook](https://www.python.org/dev/peps/pep-0517/#get-requires-for-build-wheel) to gather further requirements. See [`%pyproject_buildrequires`](#Build-macros) for more details.
Note that with the `-r` flag, the generator will include run-time requirements in BuildRequires. This is useful for running tests and for checking that the dependencies are available in Fedora.
### Test dependencies
See the [Tests](#Test-dependencies1) section.
### Extras
Python extras are a way for Python projects to declare that extra dependencies are required for additional functionality.
For example, `requests` has several standard dependencies (e.g. `urllib3`).
But it also declares an *extra* named `requests[security]`, which lists
additional dependencies (e.g. `cryptography`).
Unlike RPM subpackages, extras can only specify additional dependencies,
not additional files.
The main package will work if the optional dependency is not installed, but it
might have limited functionality.
Python tools treat extras as virtual packages.
For example, if a user runs `pip install requests[security]`, or installs a
project that depends on `requests[security]`, both `requests` and
`cryptography` will be installed.
In Fedora, extras are usually provided by packages with no files.
Instead of square brackets, Fedora package names conventionally use `+`
character (which is valid in RPM package names, but not in Python canonical
project names nor in extras identifiers).
#### Handling extras
Python packages **SHOULD** have Provides for all extras the upstream project specifies, except:
* those that are not useful for other packages (for example build/development requirements, commonly named `dev`, `doc` or `test`), and
* those that have requirements that are not packaged in Fedora.
A package that provides a Python extra **MUST** provide `python3dist(DISTNAME[EXTRA])` **and** `python3.Xdist(DISTNAME[EXTRA])`, where `X` is the minor version of the interpreter, `DISTNAME` is the [canonical project name](#Canonical-project-name), and `EXTRA` is the name of a single extra. For example, `python3.9dist(requests[security])`.
These requirements **SHOULD** be generated using the automatic dependency
generator.
A package that provides a Python extra **MUST** require the extra's main
package with exact NEVR.
A subpackage that primarily provides one Python extra **SHOULD** be named by appending "+" and the extra name to the main package name. For example, `python3-requests+security`.
The most straightforward way to provide an extra is with a dedicated subpackage
containing no files (a “metapackage”).
This case can be automated with a macro, [`%pyproject_extras_subpkg`](#Extras-macros) or [`%python_extras_subpkg`](#Extras-macros).
This is not the only way: when some extra is always useful in a distro,
it can be provided by the main package; when several extras are related,
they may be provided by a single subpackage.
However, having one dedicated subpackage per extra allows you to use the
automatic dependency generator to ensure that the extras' requirements
will stay in sync with upstream.
If you create a dedicated subpackage and want it to be always/usually
installed, you can *Require*/*Recommend*/*Suggest* it from the main package.
The dependency generator for extras activates if the following holds:
* The package name must end with `+EXTRA` (where `EXTRA` is the extra name).
* The package must contain the `.dist-info` directory, usually as `%ghost`.
##### Example and convenience macros
The extra subpackage for `setuptools_scm[toml]` can be
specified using the `%pyproject_extras_subpkg` convenience macro as follows. The macro takes the main package name and name(s) of the extra(s):
```
%pyproject_extras_subpkg -n python3-setuptools_scm toml
```
If not using `%pyproject_install`, you will instead need to use `%python_extras_subpkg` and pass a path to the `dist-info` directory:
```
%python_extras_subpkg -n python3-setuptools_scm -i %{python3_sitelib}/*.dist-info toml
```
For this case, the extras dependency generator will read upstream metadata from
the `.dist-info` directory.
If it finds that the extra requires on `toml`, it will generate
`Requires: python3.Xdist(toml)`, `Provides: python3dist(setuptools-scm[toml])`
(and the corresponding `python3.Xdist` variant).
If you need additional features that the `*_extras_subpkg` macros do
not cover, you will need to write the subpackage sections manually.
Such features can be, for example:
* Obsoleting/providing other names (e.g. obsoleted extras packages)
* Manual strong or weak dependencies on other (possibly non-Python) packages
* Including files excluded from the main package (if such files only make sense with the extra and the base package does not fail without them)
As an example of what you need to write in these cases,
both of the `*_extras_subpkg` macro invocations above expand to the following:
```
%package -n python3-setuptools_scm+toml
Summary: Metapackage for python3-setuptools_scm: toml extra
Requires: python3-setuptools_scm = %{?epoch:%{epoch}:}%{version}-%{release}
%description -n python3-setuptools_scm+toml
This is a metapackage bringing in toml extra requires for python3-setuptools_scm.
It contains no code, just makes sure the dependencies are installed.
%files -n python3-setuptools_scm+toml
%ghost %{python3_sitelib}/*.dist-info
```
Note that the dependency generator does not add a dependency
on the main package (the `Requires: python3-setuptools_scm = ...` above).
If you are not using the `%python_extras_subpkg` macro, you need to add it manually.
#### Removing extras
If an existing extra is removed from an upstream project, the Fedora maintainer
**SHOULD** try to convince upstream to re-introduce it.
If that fails, the extra **SHOULD** be Obsoleted from either the main package
or another extras subpackage.
Note that removing extras is discouraged in [setuptools documentation](https://setuptools.readthedocs.io/en/latest/setuptools.html#declaring-extras-optional-features-with-their-own-dependencies)
(see the final paragraph of the linked section).
#### Automatic Requires for extras
The automatic [run-time dependency generator](#Run-time-dependency-generator) will generate Requires
on `python3.Xdist(DISTNAME[EXTRA])` from upstream `Requires-Dist` metadata.
If the required package does not yet provide metadata for the extra,
contact the Fedora maintainer to add it.
In an emergency, you can define [`%_python_no_extras_requires`](#Disabling-automation) to avoid
automatically generating *all* extras requirements.
## Interpreter invocation
### Shebangs
Shebang lines to invoke Python **MUST** use `%{python3}` as the interpreter.
Shebang lines to invoke Python **SHOULD** be `#!%{python3} -%{py3_shebang_flags}` and it **MAY** include extra flags.
If the default flags from `%{py3_shebang_flags}` are not desirable, packages **SHOULD** explicitly redefine the macro to remove them.
Using `#!%{python3}` (`#!/usr/bin/python3`) rather than e.g. `#!/usr/bin/env python` ensures that the system-wide Python interpreter is used to run the code, even if the user modifies `$PATH` (e.g. by activating a virtual environment).
By default, `-%{py3_shebang_flags}` expands to `-s`, which means *don't add user site directory to `sys.path`*. That ensures the user's Python packages (e.g. installed by `pip install --user`, or just placed in the current directory) don't interfere with the RPM installed software. Sometimes, such content is desirable, such as with plugins. Undefining `%{py3_shebang_flags}` or redefining it to a set of flags without `s`, rather than not using the macro at all, ensures that existing or future automation won't add the flag.
The `%pyproject_install` macro automatically changes all Python shebangs in `%{buildroot}%{_bindir}/*` to use `%{python3}` and add `%{py3_shebang_flags}` to the existing flags. If you're not using that macro or you need to change a shebang in a different directory, you can use the [`%py3_shebang_fix` macro](#Shebang-macros) as follows:
```
%py3_shebang_fix SCRIPTNAME …
```
### Invokable Python modules
Every executable `TOOL` for which the current version of Python matters **SHOULD** also be invokable by `python3 -m TOOL`.
If the software doesn't provide this functionality, packagers **SHOULD** ask the upstream to add it.
This applies to tools that modify the current Python environment (like installing or querying packages), use Python for configuration, or use Python to run plugins.
It does not apply to tools like GIMP or Bash which support plugins in multiple languages and/or have other means to specify the interpreter.
For example, `pip` can be invoked as `python3 -m pip`.
This allows users to accurately specify the Python version used to run the software. This convention works across different environments that might not always set `$PATH` or install scripts consistently.
## Using Cython
Tightening the [general Fedora policy](https://docs.fedoraproject.org/en-US/packaging-guidelines/what-can-be-packaged/#_pregenerated_code), packages **MUST NOT** use files pre-generated by Cython. These **MUST** be deleted in `%prep` and regenerated during the build.
As an exception, these sources **MAY** be used temporarily to prevent build time circular dependencies by following the [bootstrapping](https://docs.fedoraproject.org/en-US/packaging-guidelines/#bootstrapping) guidelines.
Generated files (the ones that must be deleted) have a generic `.c` or `.cpp` extension.
Cython source files (which should stay) usually have the `.pyx` or `.pxd` extension.
Cython is a popular tool for writing extension modules for Python. If compiles a Python-like language to C, which is then fed to the C compiler.
Historically, Cython was hard to use upstream as a build-time dependency. Many projects include pre-generated C files in source distributions to avoid users from needing to install the tool.
Cython uses CPython's fast-changing internal API for performance reasons. For a new release of Python, Cython generally needs to be updated and the C files regenerated. In Fedora, this is frequently needed before upstreams release re-generated sources (e.g. for Alpha versins of Python).
Since we do not have a problem with build-time dependencies, we always want to run the Cython step.
For example, `PyYAML` removes a generated C file with:
```
rm -rf ext/_yaml.c
```
For another example, in `python-lxml` all C files are generated with Cython, which allows removing them with:
```
# Remove pregenerated Cython C sources
find -type f -name '*.c' -print -delete
```
Some upstreams mix generated and hand-written C files, in such cases a grep like this one from `scipy` helps (but might not be entirely future proof):
```
# Remove pregenerated Cython C sources
rm $(grep -rl '/\* Generated by Cython')
```
## Tests
### Running tests
If a test suite exists upstream, it **MUST** be run in the `%check` section and/or in Fedora CI.
You **MAY** exclude specific failing tests.
You **MUST NOT** disable the entire testsuite or ignore the result to solve a build failure.
As an exception, you **MAY** disable tests with an appropriate `%if` conditional (e.g. bcond) when [bootstrapping](https://docs.fedoraproject.org/en-US/packaging-guidelines/#bootstrapping).
A popular testing tool, and one which is well integrated in Fedora, is `tox`. Upstream, it is commonly used to test against multiple Python versions. In a Fedora package, BuildRequire test dependencies (see *Test dependencies* below) and run `tox` with:
```
%tox
```
This sets up the environment (`$PATH`, `$PYTHONPATH`, `$TOX_TESTENV_PASSENV`) and instructs `tox` to use the current environment rather than create new ones.
For more options, see [Build macros](#Build-macros).
When upstream doesn't use `tox`, the tests need to be run directly depending on upstream choice of a test runner. A popular runner is `pytest`, which can be run against the package to be invoked using `%pytest`.
Use positional arguments to specify the test directory. See `python3 -m pytest --help` for how to select tests. For example, if network-related tests are marked "network", you might use `-m` to deselect them:
```
%pytest -m "not network"
```
The `%pytest` macro sets several environment variables appropriate for `%check`:
* Locations in buildroot are added to `$PATH` and `$PYTHONPATH`.
* `$PYTHONDONTWRITEBYTECODE` is set to avoid writing pytest-specific cache files to buildroot
* If unset, `$CFLAGS` and `$LDFLAGS` are set to match the build flags
### Test dependencies
One part of the Python packaging ecosystem that is still not standardized is specifying test dependencies (and development dependencies in general).
A good, common way for upstreams to specify test dependencies is using an [extra](#Extras) like `[test]`, `[testing]` or `[dev]`. In this case, upstream's instructions to install test dependencies might look like `$ pip install -e.[test]`.
Projects using `tox` usually specify test dependencies in a `tox`-specific format: a [requires](https://tox.readthedocs.io/en/latest/config.html#conf-requires) key in the configuration.
These two forms are handled by the [`%pyproject_buildrequires` macro](#Build-macros).
If upstream does not use either form, list test dependencies as manual *BuildRequires* in the `spec` file, for example:
```
# Test dependencies:
BuildRequires: %{py3_dist pytest}
```
If you need to do this, consider asking upstream to add a `[test]` extra.
### Linters
In `%check`, packages **SHOULD NOT** run “linters”: code style checkers, test coverage checkers and other tools that check code quality rather than functionality.
Tools like `black`, `pylint`, `flake8`, or `mypy` are often “opinionated” and their “opinions” change frequently enough that they are nuisance in Fedora, where the linter is not pinned to an exact version.
Furthermore, some of these tools take a long time to adapt to new Python versions, preventing early testing with Aplha and Beta releases of Python.
And they are just not needed: wrongly formatted code is not important enough for the Fedora packager to bug the upstream about it.
Making such an issue break a package build is entirely unreasonable.
Linters *do* make sense in upstream CI. But not in Fedora.
If a linter is used, disable it and remove the dependency on it. If that is not easy, talk to upstream about making it easy (for example with a configuration option or a separate `tox` environment).
For packages that contain such linters, use them at runtime or extend them, you will usually need to run the linter in `%check`. Run it to test functionality, not code quality of the packaged software.
## Source files from PyPI
Packages **MAY** use sources from PyPI.
However, packages **SHOULD NOT** use an archive that omits test suites, licences and/or documentation present in other source archives.
For example, as of this writing `pip` provides a [source tarball (“sdist”)](https://pypi.org/project/pip/#files) which omits the relatively large `tests` and `docs` directories present in [the source on GitHub](https://github.com/pypa/pip). In this case, the tarball from GitHub should be used. (See the [Git tags](https://docs.fedoraproject.org/en-US/packaging-guidelines/SourceURL/#_git_tags) section of Fedora SourceURL guidelines.)
When using sources from PyPI, you can use the `%pypi_source` macro to generate the proper URL. See the [Macro reference](#Convenience-macros) for details.
## Example spec file
The following is a viable spec file for a Python library called `Pello` that follows packaging best practices.
Note that the project name `Pello` [normalizes](#Canonical-project-name) to the lowercase `pello`.
The example spec shows where each variant is typically used.
```
Name: python-pello
Version: 1.0.1
Release: 1%{?dist}
Summary: Example Python library
License: MIT
URL: https://github.com/fedora-python/Pello
Source0: %{pypi_source Pello}
BuildArch: noarch
BuildRequires: python3-devel
BuildRequires: pyproject-rpm-macros
%global _description %{expand:
A python module which provides a convenient example.
This description provides some details.}
%description %_description
%package -n python3-pello
Summary: %{summary}
%description -n python3-pello %_description
%prep
%autosetup -p1 -n Pello-%{version}
%generate_buildrequires
%pyproject_buildrequires -t
%build
%pyproject_wheel
%install
%pyproject_install
# Here, "pello" is the name of the importable module.
%pyproject_save_files pello
%check
%tox
# Note that there is no %%files section for
# the unversioned python module, python-pello.
# For python3-pello, %%{pyproject_files} handles code files, but
# executables, documentation and licenced must be listed in the spec file:
%files -n python3-pello -f %{pyproject_files}
%doc README.md
%license LICENSE.txt
%{_bindir}/pello_greeting
```
## Macro Reference
This section documents macros that are available to help with Python packaging.
The expansions in parentheses are provided only as reference/examples.
See the [Mandatory macros](#Mandatory-macros) section above for:
<!-- Keep order and examples the same as in Mandatory macros -->
* `%{python3}` (`/usr/bin/python3`)
* `%{python3_version}` (e.g. `3.9`)
* `%{python3_version_nodots}` (e.g. `39`)
* `%{python3_sitelib}` (e.g. `/usr/lib/python3.9/site-packages`)
* `%{python3_sitearch}` (e.g. `/usr/lib64/python3.9/site-packages`)
### Shebang macros
* `%{py3_shebang_flags}` (`s`)
Flags for `%{python3}` to use in shebangs.
Redefine this macro to use a different set of flags. Undefine it to use no flags. See [Shebangs](#Shebangs) for details.
* `%py3_shebang_fix PATHS` (`pathfix.py ... PATHS`)
A macro to fix shebangs in specified `PATHS`.
Only shebnags that already have `python` in them are changed.
If a directory is given, all `.py` files in it are fixed, recursively.
(So, if you need to fix shebangs in files not named `*.py`, you need to list each file separately or use a Shell glob, such as `%{buildroot}%{_libexecdir}/mytool/*`.)
Existing flags are preserved and `%{py3_shebang_flags}` are added (if not there already).
For example, `#! /usr/bin/env python` will be changed to `#! /usr/bin/python3 -s` and `#! /usr/bin/python -u` will be changed to `#! /usr/bin/python3 -su`.
This macro is called automatically by `%pyproject_install` on `%{buildroot}%{_bindir}/*`.
### Convenience macros
* `%{pypi_source PROJECTNAME [VERSION [EXT]]}` (e.g. `https://.../Django-3.0.5.tar.gz`)
Evaluates to the appropriate URL for source archive hosted on PyPI. Accepts the project name and up to two optional arguments:
* The version of the PyPI project. Defaults to `%version` (the package version) with any `~` removed.
* The file extension to use. Defaults to `tar.gz`.
In most cases it is not necessary to specify those two arguments.
For backward compatibility, the first argument is technically optional as well, but omitting it is deprecated. (It defaults to `%srcname` if defined, or to `%pypi_name` if defined, or to `%name`.)
* `%{python3_platform}` (e.g. `linux-x86_64`)
The platform name. Used in some Python build systems.
This corresponds to [`sysconfig.get_platform()`](https://docs.python.org/3/library/sysconfig.html#sysconfig.get_platform).
* `%{python3_ext_suffix}` (e.g. `.cpython-39-x86_64-linux-gnu.so`)
Filename extension for Python extension modules.
This corresponds to the `EXT_SUFFIX` [sysconfig](https://docs.python.org/3/library/sysconfig.html) variable.
* `%{python3_platform_triplet}` (e.g. `x86_64-linux-gnu`)
A string identifying the architecture/platform.
This corresponds to the `MULTIARCH` [sysconfig](https://docs.python.org/3/library/sysconfig.html) variable.
### Build macros
The “pyproject macros” are most useful for packaging Python projects that use the `pyproject.toml` file defined in [PEP 518](https://www.python.org/dev/peps/pep-0518/) and [PEP 517](https://www.python.org/dev/peps/pep-0517/), which specifies the package's build dependencies (including the build system, such as setuptools, flit or poetry).
If `pyproject.toml` is not found, the macros automatically fall backs to using `setuptools` with configuration in `setup.cfg`/`setup.py`.
A full tutorial and discussion for the macros is available in the macros' [README](https://src.fedoraproject.org/rpms/pyproject-rpm-macros/).
* `%pyproject_buildrequires`
Generate BuildRequires for the package. Used in the `%generate_buildrequires` section of the `spec` file. The macro has these options:
* `-r`: Include run-time requirements (commonly needed for `%check`).
* `-x EXTRA`: Include dependencies given by the given [extra](#Extras). Implies `-r`.
* `-t`: Include dependencies for the default *tox* environment. Implies `-r`.
* `-e ENV`: Include dependencies for the given *tox* environment, and save the `ENV` name as `%{toxenv}`. Implies `-r`. Multiple comma separated values can be given, for example:
```
%pyproject_buildrequires -e %{toxenv}-unit,%{toxenv}-integration
```
* `%pyproject_wheel`
Build the package. Commonly, this is the only macro needed in the `%build` section.
This macro needs BuildRequires generated by `%pyproject_buildrequires`.
* `%pyproject_install`
Install the package built by `%pyproject_wheel`.
Calls `%py3_shebang_fix %{_buildroot}%{_bindir}/*`.
This macro needs BuildRequires generated by `%pyproject_buildrequires`.
* `%pyproject_save_files MODNAME …`
Generate a list of files corresponding to the given importable modules globs, and save it as `%{pyproject_files}`.
Note that README and licence files are not included.
Also, while the macro allows including executable and other files (using the `+auto` flag), this feature **MUST NOT** be used in Fedora.
The `MODNAME` may be a glob pattern, which should be specific to your package. To prevent Shell from expanding the globs, put them in `''`, e.g. `%pyproject_save_files '*pytest'`. As mentioned in the *Explicit lists* section, expressions like `%pyproject_save_files '*'` are not acceptable.
* `%{pyproject_files}`
Path of the file written by `%pyproject_save_files`, to be used as:
%files -n python3-DISTNAME -f %{pyproject_files}
### Test macros
* `%tox`
Run tests using `tox`.
Different environments may be specified with `-e`, for example:
```
%check
%tox %{?with_integration_tests:-e %{toxenv},%{toxenv}-integration}
```
Flags for the `tox` command can be specified after `--`:
%tox -- --parallel 0
Additional arguments for the test runner may be specified after another `--`:
%tox -- --parallel 0 -- --verbose tests/*
* `%{toxenv}`
The *tox* environment(s) used by the `%tox` macro. Multiple environments are separated by commas.
Can be overridden manually or with `%pyproject_buildrequires -t ENV1,ENV2`.
* `%{default_toxenv}` (e.g. `py39`)
The system-wide default value of `%{toxenv}`.
* `%pytest`
Run `%__pytest` with environment variables appropriate
for tests in `%check`.
See the [Running tests](#Running-tests) section for details.
* `%__pytest` (`/usr/bin/pytest`)
The command that `%pytest` uses. May be redefined.
### Extras macros
* `%pyproject_extras_subpkg`
Generates a simple subpackage for a Python extra.
See the [Extras](#Extras) section for more information.
This macro needs to be used with `%pyproject_install`
(use `%python_extras_subpkg` in other cases).
Required arguments:
* `-n`: name of the "base" package (e.g. `python3-requests`)
* Positional arguments: the extra name(s). Multiple metapackages are generated when multiple names are provided.
The macro also takes `-i`/`-f`/`-F` arguments for
`%python_extras_subpkg` below, but if they are not given,
a filelist written by `%pyproject_install` is used.
* `%python_extras_subpkg`
Generates a simple subpackage for a Python extra.
See the [Extras](#Extras) section for more information.
Takes these arguments:
* `-n`: name of the "base" package (e.g. `python3-requests`)
* `-i`: the `%files %ghost` path (glob) to the `.dist-info` directory
Positional arguments specify the extra name(s) — multiple metapackages are generated when multiple names are provided.
* `-f`: Relative path to the filelist for this metapackage (which should contain the `%files %ghost` path (glob) to the the metadata directory). Conflicts with `-i` and `-F`.
* `-F`: Skip the %files section entirely (if the packager wants to construct it manually). Conflicts with `-i` and `-f`.
This macro generates all the subpackage definition sections (`%package` including the `Summary` and `Requires` on the base package, `%description` and, by default, `%files`).
Hence, it cannot be extended with custom *Provides*/*Obsoletes*/*Requires*/etc.
This macro is designed to fit only the most common uses.
For more complicated uses, construct the subpackage manually as shown in the [Extras](#Extras) section.
Due to technical limitations, the macro never generates requirement on the arched `BASE_PACKAGE%{?_isa} = %{?epoch:%{epoch}:}%{version}-%{release}`.
It only adds `Requires: BASE_PACKAGE = %{?epoch:%{epoch}:}%{version}-%{release})` because a macro cannot reliably detect if the subpackage is arched or not.
So far, this has not been a problem in practice.
### Manual generation
The following macros are available for cases where automatic generation is turned off.
They can also be useful for handling files in non-standard locations where the generators don't look.
* `%pycached MODNAME.py`
Given a Python file, lists the file and the files with its bytecode cache. See *Source files and bytecode cache* for more information.
* `%{py_provides python3-MODNAME}`
Generates `Provides` for `python3-MODNAME`, `python3.X-MODNAME` and `python-MODNAME`.
See [Automatic unversioned provides](#Automatic-unversioned-provides) for more details.
* `%{py_byte_compile INTERPRETER PATH}`
Byte-compile a Python file into a `__pycache__/*.pyc`.
If the `PATH` argument is a directory, the macro will recursively byte compile all `*.py` files in the directory.
(So, if you need to compile files not named `*.py`, you need to use the macro on each file separately.)
The `INTERPRETER` determines the compiled file name's suffix and the magic number embedded in the file.
These muct match the interpreter that will import the file.
Usually, the `INTERPRETER` should be set to `%{python3}`.
If you are compiling for a non-default interpreter, use that interpreter instead and add a `BuildRequires` line for it.
* `%{py_dist_name PROJECTNAME}`
Given a *project name* (e.g. `PyYAML`) it will convert it to the canonical format (e.g. `pyyaml`). See [Canonical project name](#Canonical-project-name) for more information.
* `%{py3_dist PROJECTNAME …}`
Given one or more *project names*, it will convert them to the canonical format and evaluate to `python3dist(DISTNAME)`, which is useful when listing dependencies. See [Machine-readable provides](#Machine-readable-provides) for more information.
### System Settings
The following macros can be redefined for special use cases.
* `%{__python}` (`/usr/bin/python`)
Defining this macro changes the meaning of all “unversioned” Python macros such as `%{python}` or `%{python_sitelib}`.
Don’t use these macros without redefining `%{__python}`.
* `%{__python3}` (`/usr/bin/python3`)
The python 3 interpreter. Redefining this macro changes all the `%{python3...}` macros, e.g. `%{python3}` or `%{python3_sitelib}`.
* `%{python3_pkgversion}` (`3`)
Distro-wide Python version, i.e. the `3` in `python3`.
Projects that build on top of Fedora might define it to e.g. `3.9` to try allowing multiple Python stacks installable in parallel.
Packages in Fedora **MAY** use it (e.g. in package names: `python%{python3_pkgversion}-requests`), but **MUST NOT** redefine it.
### Comparing Python versions
When comparing Python versions (e.g. to ask: is `%{python3_version}` greater than 3.8?), using naïve `%if %{python3_version} > 3.8` or `%if "%{python3_version}" > "3.8"` is not possible, because the comparison is performed alphabetically on strings. Hence it is true that `"3.10" < "3.8"` (which is not desired).
Starting with RPM 4.16 (Fedora 33), it is possible to explicitly compare version literals by using the `v` prefix, similar to the Python string prefixes:
```
%if v"%{python3_version}" > v"3.8"
...
%endif
```
For compatibility with older releases, a workaround for Python 3.9 and below is to compare integers:
```
%if %{python3_version_nodots} > 38
...
%endif
```
XXX We'll add a recommendation that works with Python 3.10+ after upstream figures out how to represent these versions.
### Disabling automation
The following macros can turn off Python-specific automation.
Consider contacting the Python SIG if you need to do this.
* `%{?python_disable_dependency_generator}`
Disables the automatic dependency generator. See [Automatically generated dependencies](#Automatically-generated-dependencies) for details.
* `%undefine __pythonname_provides`
Disables automatic generation of unversioned provides, e.g. `python-FOO` for `python3-foo`.
See [the corresponding section](#Automatic-unversioned-provides) for more details.
* `%global _python_no_extras_requires 1`
If defined, [automatic Requires for extras](#Automatic-Requires-for-extras) will not be generated.
### Deprecated Macros
The following macros are deprecated. See the [201x-era Python Packaging guidelines](https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/) for how some of them were used.
* `%py3_build`
* `%py3_build_wheel`
* `%py3_build_egg`
* `%py3_install`
* `%py3_install_wheel`
* `%py3_install_egg`
* `%py3dir`
* `%py3_other_build`
* `%py3_other_install`
* `%python_provide` (without `s` at the end)
# Changes in Python packaging guidelines (draft)
These are changes betwween the [current (“201x-era”) Python packaging guidelines](https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/) and the [proposal as of 2020-10-15](https://hackmd.io/@python-maint/H1EWLinvU).
All **MUST** and **SHOULD** items from both versions are covered. (For the old guidelines I added more, where I feel they should have been marked SHOULD.)
If you have comments or reservations, please *read the whole section* of the new guidelines.
Direct your comments to the [Python SIG mailing list](python-devel@lists.fedoraproject.org).
(Comments on HackMD are hard to manage.)
<details>
<summary>Minor changes are hidden in *details* tags like this. To open one, click the summary.</summary>
(Now you can see these details.)
</details>
I tried to list the most important changes first.
## Rationales and explanations
The new guidelines give rationales and explanations where applicable.
In many cases these changed.
This list of changes generally doesn't list these “soft” changes.
## PyPI Parity
The most profound change is synchronization of Python project names with the Python Package Index, allowing Python-level metadata across the ecosystem.
> **NEW:** Every Python package in Fedora **SHOULD** also be available
> on [the Python Package Index](https://pypi.org) (PyPI).
>
> The command `pip install PROJECTNAME` **MUST** install the same package
> (possibly in a different version),
install nothing,
> or fail with a reasonable error message.
>
> If this is not the case, the packager **SHOULD** contact upstream about this.
> The goal is to get the project name registered or blocked on PyPI,
> or to otherwise ensure the rule is followed.
As a reminder: if you want to comment, please read the [relevant section](https://hackmd.io/XzJe-sHUQvWK7cSrEH_aKg?both#PyPI-parity) for details.
## Upstream metadata
The second major paradigm shift is that package metadata is primarily taken from upstream,
since upstream “dist-info” metadata is now standardized and shareable with other distributors (be it Linux distros or others).
Packagers are expected to treat metadata bugs as any other bugs, (ideally, patch them and present the patches upstream).
> NEW: Packagers **SHOULD** be prepared to get involved with upstream projects to establish best practices as outlined here. We wish to improve both Fedora and the wider Python ecosystem.
## Beta period
The new guidelines are “beta” period where they can be used instead of the old ones, but ask for increased cooperation from packagers who opt in to use them.
> NEW: These Guidelines represent a major rewrite and paradigm shift, and not all packages are updated to reflect this.
Older guidelines are still being kept up to date, and existing packages **MAY** use them instead of this document
> NEW: While these guidelines are in Beta, each Python package **MUST** have `BuildRequires: pyproject-rpm-macros` to access the beta macros.
## Python 3
Python 2 has been deprecated for a while now; using it requires a FESCo exception.
Text related to the py2/py3 split is generally removed or simplified.
> NEW: Fedora packages **MUST NOT** depend on other versions of the CPython interpreter than the current `python3`.
> [...]
> Packages such as `pip` or `tox`, which enable setting up isolated environments and installing third-party packages into them, **MAY**, as an exception to the rule above, use these interpreters as long as this is coordinated with the maintainers of the relevant Python interpreter.
> OLD: If a piece of software supports `python3`, it MUST be packaged for `python3`. Software using `python2` MUST NOT be newly packaged into Fedora without FESCo exception.
The following redundant note was put in the guidelines to address pushing "remove of python2 package" changes to stable branches. Since not many python2 packages are left, the note is not necessary any more:
> OLD: Mirroring the policy for regular packages, the Python-version-specific subpackages of your package MUST NOT be removed in a release branch of Fedora.
## Dependency Generator
The dep generator is now mandatory (i.e. upstream metadata must be used/patched):
> NEW: Packages **MUST** use the automatic Python run-time dependency generator.
Packages **SHOULD** use the opt-in build-dependency generator if possible.
he packager **MUST** inspect the generated requires for correctness. All dependencies **MUST** be resolvable within the targeted Fedora version.”
Any necessary changes **MUST** be done by patches or modifying the source (e.g. with `sed`), rather than disabling the generator. The resulting change **SHOULD** be offered to upstream. As an exception, [filtering](https://docs.fedoraproject.org/en-US/packaging-guidelines/AutoProvidesAndRequiresFiltering/) **MAY** be used for temporary workarounds and bootstrapping.
Dependencies covered by the generators **SHOULD NOT** be repeated in the `.spec` file. (For example, if the generator finds a `requests` dependency, then `Requires: python3-requests` is redundant.)
> OLD: This generator is enabled by default in Fedora. If a packager wishes to explicitly opt out of the generator because the upstream metadata are not applicable, a packager SHOULD opt out explicitly by adding: `%{?python_disable_dependency_generator}`
> OLD: [if using dependency generator] The packager MUST inspect the generated requires for correctness. All dependencies MUST be resolvable within the targeted Fedora version.
## Applicability
New clarification:
> NEW: Except for the two "Distro-wide guidelines", these Guidelines do not apply to simple one-file scripts or utilities[...].
> However, if [you need] a more complex Python library, the library **SHOULD** be packaged as an importable library under these guidelines.
## Naming
<details>
<summary>Clarified wording on General naming:</summary>
> NEW: A built (i.e. non-SRPM) package for a *Python library* **MUST** be named with the prefix `python3-`.
>
> A source package containing primarily a *Python library* **MUST** be named with the prefix `python-`.
> OLD: The source package for a Python library MUST be named with the `python-` prefix.
> OLD (in [Naming guidelines](https://docs.fedoraproject.org/en-US/packaging-guidelines/Naming/#_python_modules)): Python2 binary packages MUST be named using a `python2-` prefix.
> Python3 binary packages MUST be named with a prefix of `python3-`.
</details>
<details>
<summary>Clarified wording on `python3-` provides for importable modules:</summary>
> NEW: For any module intended to be used in Python 3 with `import MODNAME`,
the package that includes it **SHOULD** provide `python3-MODNAME`,
with underscores (`_`) replaced by dashes (`-`).
> OLD: For any module foo intended to be used in Python 3 with `import foo`, the package that includes it should provide `python3-foo`
</details>
<details>
<summary>Clarified wording on `python-`/`python3-`/`python3.X-` provides:</summary>
> NEW: For any `FOO`, a package that provides `python3-FOO`
**SHOULD** use `%py_provides` or an automatic generator
to also provide `python-FOO` and `python3.X-FOO`, where `X` is the minor version of the interpreter.
The provide **SHOULD NOT** be added manually:
if a generator or macro is not used,
do not add the `python-FOO` / `python3.X-FOO` provides at all.
> OLD: All packages that provide `python3-...` (for any `...`) SHOULD also provide `python-...` and `python3.X-...`.
> OLD: Any manually added virtual provides of `python3-...` SHOULD be done via the `%py_provides macro`.
</details>
Removed ban on `python-` Requires, since the `python-` prefix now refers to Python 3:
> OLD: Packages MUST NOT have dependencies (either build-time or runtime) on packages named with the unversioned `python-` prefix. Dependencies on Python packages instead MUST use names beginning with `python3-`.
Best practices (SHOULDs):
> NEW: If the importable module name and the project name do not match,
users frequently end up confused.
In this case, packagers **SHOULD** ensure that upstream is aware of the problem
and (especially for new packages where renaming is feasible)
strive to get the package renamed.
The Python SIG is available for assistance.
> NEW: The Fedora package's name **SHOULD** contain the *canonical project name*.
If possible, the project name **SHOULD** be the same
as the name of the main importable module,
with underscores (`_`) replaced by dashes (`-`).
> NEW: Packages that primarily provide applications, services or any kind of executables
**SHOULD** be named according to the general [Fedora naming guidelines](https://docs.fedoraproject.org/en-US/packaging-guidelines/Naming/)
(e.g. `ansible`).
> OLD (in [Naming guidelines](https://docs.fedoraproject.org/en-US/packaging-guidelines/Naming/#_python_modules)): The package name SHOULD reflect the upstream name of the Python module, and SHOULD generally take into account the name of the module used when importing it in Python scripts. This name will be prefixed depending on the type of the package.
Removed rules on removing `%py_provides` (which is documented as being for cases where the generator doesn't work) and `%python_provide` (which is listed in Deprecated macros):
> OLD: Packagers SHOULD try to remove explicit `%py_provides` calls for package names, but MAY preserve them if they aim for compatibility with older releases or packages without files.
> OLD (note): Historically, there was `%python_provide` macro with similar but different semantics. It still works for compatibility reasons but it is deprecated and SHOULD NOT be used and packagers SHOULD replace is with appropriate %py_provides call.
### Naming Extras
Slight addition on including devel extras (diff emphasized):
> NEW: Python packages **SHOULD** have Provides for all extras the upstream project specifies, except: those that are not useful for other packages (for example build/development requirements, commonly named `dev`, `doc` or `test`), **and those that have requirements that are not packaged in Fedora.**
> OLD: Python packages SHOULD have `Provides` for all extras the upstream project specifies, except those that are not useful for other packages (for example build/development requirements, commonly named `dev`, `doc` or `test`).
New best practice re. removing extras:
> NEW: If an existing extra is removed from an upstream project, the Fedora maintainer
**SHOULD** try to convince upstream to re-introduce it.
If that fails, the extra **SHOULD** be Obsoleted from either the main package
or another extras subpackage.
<details>
<summary>Reworded text on <code>+</code>; same meaning</summary>
> NEW: The character `+` in names of built packages (i.e. non-SRPM)
> that include `.dist-info` or `.egg-info` directories
> is reserved for *extras*
> and **MUST NOT** be used for any other purpose.
> As an exception, `+` characters **MAY** appear at the *end* of such names.
> OLD: The character `+` in names of built packages (i.e. non-SRPM) that include `.dist-info` or `.egg-info` directories is reserved for Python Extras and MUST NOT be used for any other purpose.
> As an exception, + characters are permitted at the end of the name.
> NEW: A package that provides a Python extra **MUST** provide `python3dist(DISTNAME[EXTRA])` **and** `python3.Xdist(DISTNAME[EXTRA])`, where `X` is the minor version of the interpreter, `DISTNAME` is the [canonical project name](#Canonical-project-name), and `EXTRA` is the name of a single extra. For example, `python3.9dist(requests[security])`.
> These requirements **SHOULD** be generated using the automatic dependency
generator.
> OLD: A package that provides a Python extra MUST provide `python3dist(…[…])` and `python3.Xdist(…[…])`, for example, `python3.9dist(requests[security])`. These requirements SHOULD be generated using the automatic dependency generator.
</details>
<details>
<summary>Unchanged guidelines</summary>
> NEW: A package that provides a Python extra **MUST** require the extra's main
package with exact NEVR.
> OLD: A package that provides a Python extra MUST require the extra’s main package with exact NEVR.
> NEW: A subpackage that primarily provides one Python extra **SHOULD** be named by appending "+" and the extra name to the main package name. For example, `python3-requests+security`.
> OLD: A subpackage that primarily provides one Python extra SHOULD be named by appending `+` and the extra name to the main package name. For example, `python3-requests+security`.
</details>
## Files to include
More detailed rules on what to include (and where):
> NEW: Packages **MUST** include the source file (`*.py`)
> **AND** the bytecode cache (`*.pyc`)
> for each pure-Python importable module.
> The source files **MUST** be included in the same package as the bytecode cache.
> NEW: Each Python package **MUST** include *Package Distribution Metadata*
conforming to [PyPA specifications](https://packaging.python.org/specifications/)
(specifically, [Recording installed distributons](https://packaging.python.org/specifications/recording-installed-packages/)).
The metadata **SHOULD** be included in the same subpackage
as the main importable module,
if there is one.
[...] As an exception,
the Python standard library **MAY** ship without this metadata.
> OLD: When packaging Python modules, several types of files are included: `*.py source files`[,] `*.pyc byte compiled files`[,] `*.egg-info` or `*.dist-info` files or directories. [technically, there is no MUST here]
>
> The source files MUST be included in the same package as the byte compiled versions.
New best practice: not byte-compiling scripts:
> NEW: Scripts that are not importable
(typically ones in `%{_bindir}` or `%{_libexecdir}`)
**SHOULD NOT** be byte-compiled.
<details>
<summary>Consolidated rules on content in shared directories</summary>
> NEW: Packages **MUST NOT** own shared directories owned by Python itself, such as the top-level `__pycache__` directories (`%{python3_sitelib}/__pycache__`, `%{python3_sitearch}/__pycache__`).
> NEW: Packagers **SHOULD NOT** simply glob everything under a shared directory.
> In particular, the following **SHOULD NOT** be used: `%{python3_sitelib}/*`, `%{python3_sitearch}/*`, `%{python_sitelib}/*`, `%{python_sitearch}/*`, `%{_bindir}/*`, `%pyproject_save_files '*'`, `%pyproject_save_files +auto`.
> OLD: Packagers SHOULD NOT simply glob everything under the sitelib or sitearch directories. The following SHOULD NOT be used: `%{python3_sitelib}/*`, `%{python3_sitearch}/*`, `%{python_sitelib}/*`, `%{python_sitearch}/*`. And packages MUST NOT include the top-level `__pycache__` directory (see below).
>
> OLD: You MUST NOT include the directories `%{python3_sitearch}/__pycache__` or `%{python3_sitelib}/__pycache__` because they are already owned by the `python3-libs` package.
</details>
## Mandatory Requires / Provides
> NEW: **Every** package that uses Python (at runtime and/or build time), and/or installs Python modules **MUST** explicitly include `BuildRequires: python3-devel` in its `.spec` file, even if Python is not actually invoked during build time.
If the package uses an alternate Python interpreter instead of `python3` (e.g. `pypy`, `jython`, `python2.7`), it **MAY** instead require the corresponding `*-devel` package.
> NEW: As mentioned above, each Python package **MUST** explicitly BuildRequire `python3-devel`.
Packages **MUST NOT** have dependencies (either build-time or runtime) with the unversioned prefix `python-` if the corresponding `python3-` dependency can be used instead.
Packages **SHOULD NOT** have explicit dependencies (either build-time or runtime) with a minor-version prefix such as `python3.8-` or `python3.8dist(`. Such dependencies **SHOULD** instead be automatically generated or a macro should be used to get the version.
Packages **SHOULD NOT** have an explicit runtime dependency on `python3`.
> NEW: Every Python package **MUST** provide `python3dist(DISTNAME)`
**and** `python3.Xdist(DISTNAME)`,
where `X` is the minor version of the interpreter
and `DISTNAME` is the *canonical project name*
corresponding to the *dist-info metadata*.
For example, `python3-django` would provide `python3dist(django)`
and `python3.9dist(django)`.
This is generated automatically from the dist-info metadata.
The provide **SHOULD NOT** be added manually:
if the generator fails to add it, the metadata **MUST** be fixed.
## Tests
Running upstream tests is mandatory; linters are discouraged
> NEW: If a test suite exists upstream, it **MUST** be run in the `%check` section and/or in Fedora CI.
You **MAY** exclude specific failing tests.
You **MUST NOT** disable the entire testsuite or ignore the result to solve a build failure.
As an exception, you **MAY** disable tests with an appropriate `%if` conditional (e.g. bcond) when [bootstrapping](https://docs.fedoraproject.org/en-US/packaging-guidelines/#bootstrapping).
> NEW: In `%check`, packages **SHOULD NOT** run “linters”: code style checkers, test coverage checkers and other tools that check code quality rather than functionality.
## Shebangs & Mandatory macros
The rule regarding `/usr/bin/python` has been generalized to mandatory use of some macros, and rules on shebangs were split off and made stricter.
> NEW: The following macros **MUST** be used where applicable. `%{python3}`, `%{python3_version}`, `%{python3_version_nodots}`, `%{python3_sitelib}`, `%{python3_sitearch}`.
> NEW: Shebang lines to invoke Python **MUST** use `%{python3}` as the interpreter.
> NEW: Shebang lines to invoke Python **SHOULD** be `#!%{python3} -%{py3_shebang_flags}` and it **MAY** include extra flags.
If the default flags from `%{py3_shebang_flags}` are not desirable, packages **SHOULD** explicitly redefine the macro to remove them.
> OLD: Packages in Fedora MUST NOT use `/usr/bin/python`. Instead packages for Python 3 MUST use `/usr/bin/python3` (even if upstream supports both Python 2 and 3). As a result of that `/usr/bin/python` (as well as `/usr/bin/env python` and similar) MUST NOT be used in shebang lines or as a dependency of a package. All uses of unversioned python executables in shebang lines will fail the build. These shebangs MUST be fixed (for example by using the `%py3_shebang_fix` macro in the spec file).
## PyPI sources
PyPI archives that omit tests/docs are discouraged:
> NEW: Packages **MAY** use sources from PyPI.
However, packages **SHOULD NOT** use an archive that omits test suites, licences and/or documentation present in other source archives.
## Specific macros
Redefining `%python3_pkgversion` is banned:
> NEW: Packages in Fedora **MAY** use [`%{python3_pkgversion}`] (e.g. in package names: `python%{python3_pkgversion}-requests`), but **MUST NOT** redefine it.
<details>
<summary>Note on `%pyproject_save_files` features unsuitable for Fedora</summary>
> NEW: Note that README and licence files are not included.
Also, while the macro allows including executable and other files (using the `+auto` flag), this feature **MUST NOT** be used in Fedora.
</details>
## Cython
<details>
<summary>Clarification; same meaning</summary>
> NEW: Tightening the [general Fedora policy](https://docs.fedoraproject.org/en-US/packaging-guidelines/what-can-be-packaged/#_pregenerated_code), packages **MUST NOT** use files pre-generated by Cython. These **MUST** be deleted in `%prep` and regenerated during the build.
As an exception, these sources **MAY** be used temporarily to prevent build time circular dependencies by following the [bootstrapping](https://docs.fedoraproject.org/en-US/packaging-guidelines/#bootstrapping) guidelines.
> OLD: Tightening the general Fedora policy, packages MUST NOT use pre-generated Cython sources. They MUST be deleted in %prep and regenerated during the build. Any exception to this rule should be considered a bootstrapping.
</details>
## Best practices
New note on when to prefer `python3 -m tool`:
> NEW: Every executable `TOOL` for which the current version of Python matters **SHOULD** also be invokable by `python3 -m TOOL`.
If the software doesn't provide this functionality, packagers **SHOULD** ask the upstream to add it.
_______________________________________________
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
