Bug#1031716: python3-protobuf: Do reverse dependencies need stricter version constraints?
On Tue, Feb 21, 2023 at 02:23:34PM +0200, Adrian Bunk wrote: > Looking at #1028371, should generated dependencies on python3-protobuf be > python3-protobuf (>= 3.21), python3-protobuf (<< 3.22) > to ensure that the binary package is used with the same version > as the protobuf-compiler used during the build? I'm not the maintainer, but a drive-by contributor. I looked a bit into this, given its RC severity. With my still somewhat limited understanding, a strict version alignment between protobuf-compiler and python3-protobuf would probably resolve this particular symptom, but the issues here seem to run deeper. Specifically: * The protobuf project provides three different versions of Python bindings: pure Python, C++, and libupb-based[1]. These are selectable using the PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION environment variable. * Debian's python3-protobuf, from src:protobuf, ships the pure Python version, as well as the C++ bindings. The default implementation in Debian is "cpp". * The upb implementation is not included in src:protobuf, but in the upb upstream source[2], i.e. what is src:upb in Debian, even though the snapshot we have in Debian does not contain sources to Python bindings. * Upstream has switched the default implementation to "upb", and deprecated the "cpp" implementation. There is, in fact, no way for one to fetch the "cpp" version from PyPI. This is documented extensively in their May 2022 release notes[3]. However, Debian still ships, and defaults to, cpp, a major departure from upstream. * Relatedly, when they made that switch, they also made changes to their versioning scheme, disconnecting the Python library's version from the source version. As a result, the Python API (both upb, as well as pure Python), is now versioned at "4.21", rather than "3.21". The Debian binary package python3-protobuf is versioned as "3.21.12-1", which is not a version that exists, or will ever exist, upstream. That binary package in fact, is shipping an egg named protobuf-4.21.12.egg-info. (This is all also well documented in their release notes[3]). * Finally, in the same release notes document[3], they also state: "Python upb requires generated code that has been generated from protoc 3.19.0 or newer.". Indeed, if one fetches protobuf 4.21 from PyPI, and runs: python3 -c 'import bernhard' or PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=upb python3 -c 'import bernhard' ...a traceback message is emitted, but a much more informative one: > TypeError: Descriptors cannot not be created directly. > > If this call came from a _pb2.py file, your generated code is out of > date and must be regenerated with protoc >= 3.19.0. > > If you cannot immediately regenerate your protos, some other possible workarounds are: > 1. Downgrade the protobuf package to 3.20.x or lower. > 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will > use pure-Python parsing and will be much slower). > > More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates * The release notes specifically mention "upb" requiring protoc (protobuf-compiler) >= 3.19, but not "cpp". However, as established above, "cpp" is deprecated and not used by anyone (but Debian), and therefore they either meant "the non-Pure-Python implementation" there, or did not pay as much attention to forward- and backwards-compatibility, or informative error messages for their deprecated backend. It's likely, but not entirely clear, that the protoc dependency requirement is >= 3.19 here as well. * Finally note that the 3.21.12-1+b2 Python implementation still works with python3-bernhard, Built-Using: protobuf-compiler (= 3.12.4-1+b3): PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python python3 -c 'import bernhard' All in all: it's almost certainly necessary to make the dependency tighter, to something like >= 3.19, if not tight to = 3.21. I still feel uneasy about Debian shipping a version of python3-protobuf that includes, and defaults to, an implementation that is deprecated upstream (and on top of it, is misversioned). I'm not sure what to make of this so late in the release cycle, though. For trixie the path forward is probably something along the lines of updating src:upb to a newer upstream, building the upb-based extension as python3-protobuf-upb, and then changing src:protobuf to not build the cpp extension, make python3-protobuf Arch: all, and then Recommend (or Depend) on python3-protobuf-upb as the native/fast implementation. Faidon 1: https://github.com/protocolbuffers/protobuf/tree/main/python 2: https://github.com/protocolbuffers/upb/tree/main/python 3: https://protobuf.dev/news/2022-05-06/
Bug#1031716: python3-protobuf: Do reverse dependencies need stricter version constraints?
On Tue, Feb 21, 2023 at 08:15:12PM +0100, László Böszörményi (GCS) wrote: > On Tue, Feb 21, 2023 at 1:27 PM Adrian Bunk wrote: > > Looking at #1028371, should generated dependencies on python3-protobuf be > > python3-protobuf (>= 3.21), python3-protobuf (<< 3.22) > You mean on python3-bernhard, right? Yes, sorry. > > to ensure that the binary package is used with the same version > > as the protobuf-compiler used during the build? > Then what? It would become uninstallable when a new protobuf version > (3.22 next time) is uploaded and built. That's the point - an uninstallable package is better than a broken package, especially when being uninstallability blocks upgrading python3-protobuf for users where it is known to cause breakage. And it ensures that the package will be rebuilt (or removed) during protobuf transitions. > > If yes, are other language bindings also affected by this problem? > I still don't get your point. How does a language binding package > version relate to the protobuf-compiler version? I don't follow the > internals of Protobuf, but I would say it's more related to the > library soname and its provided API version. The problem in python3-bernhard is in a Python file generated by protoc: $ python3 -c "import bernhard" Traceback (most recent call last): File "", line 1, in File "/usr/lib/python3/dist-packages/bernhard/__init__.py", line 20, in from . import pb File "/usr/lib/python3/dist-packages/bernhard/pb.py", line 36, in _descriptor.FieldDescriptor( File "/usr/lib/python3/dist-packages/google/protobuf/descriptor.py", line 560, in __new__ _message.Message._CheckCalledFromGeneratedFile() TypeError: Descriptors should not be created directly, but only retrieved from their parent. $ head -2 /usr/lib/python3/dist-packages/bernhard/pb.py # -*- coding: utf-8 -*- # Generated by the protocol buffer compiler. DO NOT EDIT! $ > Regards, > Laszlo/GCS cu Adrian
Bug#1031716: python3-protobuf: Do reverse dependencies need stricter version constraints?
Hello,
On Tue, 21 Feb 2023 20:15:12 +0100
=?UTF-8?B?TMOhc3psw7MgQsO2c3rDtnJtw6lueWkgKEdDUyk=?= wrote:
On Tue, Feb 21, 2023 at 1:27 PM Adrian Bunk wrote:
> Looking at #1028371, should generated dependencies on python3-protobuf be
> python3-protobuf (>= 3.21), python3-protobuf (<< 3.22)
You mean on python3-bernhard, right?
yes
> to ensure that the binary package is used with the same version
> as the protobuf-compiler used during the build?
Then what? It would become uninstallable when a new protobuf version
(3.22 next time) is uploaded and built.
this is the idea, since right now even a basic import of bernhard is just not
working
> If yes, are other language bindings also affected by this problem?
I still don't get your point. How does a language binding package
version relate to the protobuf-compiler version? I don't follow the
internals of Protobuf, but I would say it's more related to the
library soname and its provided API version.
Bernhard uses protobuf to generate a python binding code, and this code is not
working anymore when a new protobuf
version is uploaded.
The idea is to restrict the protobuf version it was built with, to make sure
the import works.
E.g. this is the current pb.py autogenerated code
# -*- coding: utf-8 -*-
# Generated by the protocol buffer compiler. DO NOT EDIT!
# source: riemann.proto
from google.protobuf import descriptor as _descriptor
from google.protobuf import message as _message
from google.protobuf import reflection as _reflection
from google.protobuf import symbol_database as _symbol_database
# @@protoc_insertion_point(imports)
_sym_db = _symbol_database.Default()
DESCRIPTOR = _descriptor.FileDescriptor(
name='riemann.proto',
package='',
syntax='proto2',
serialized_options=b'\n\021com.aphyr.riemannB\005Proto',
create_key=_descriptor._internal_create_key,
serialized_pb=b'\n\rriemann.proto\"\x81\x01\n\x05State\x12\x0c\n\x04time\x18\x01
\x01(\x03\x12\r\n\x05state\x18\x02 \x01(\t\x12\x0f\n\x07service\x18\x03
\x01(\t\x12\x0c\n\x04host\x18\x04 \x01(\t\x12\x13\n\x0b\x64\x65scription\x18\x05
\x01(\t\x12\x0c\n\x04once\x18\x06 \x01(\x08\x12\x0c\n\x04tags\x18\x07 \x03(\t\x12\x0b\n\x03ttl\x18\x08
\x01(\x02\"\xce\x01\n\x05\x45vent\x12\x0c\n\x04time\x18\x01 \x01(\x03\x12\r\n\x05state\x18\x02
\x01(\t\x12\x0f\n\x07service\x18\x03 \x01(\t\x12\x0c\n\x04host\x18\x04
\x01(\t\x12\x13\n\x0b\x64\x65scription\x18\x05 \x01(\t\x12\x0c\n\x04tags\x18\x07
\x03(\t\x12\x0b\n\x03ttl\x18\x08 \x01(\x02\x12\x1e\n\nattributes\x18\t
\x03(\x0b\x32\n.Attribute\x12\x15\n\rmetric_sint64\x18\r \x01(\x12\x12\x10\n\x08metric_d\x18\x0e
\x01(\x01\x12\x10\n\x08metric_f\x18\x0f \x01(\x02\"\x17\n\x05Query\x12\x0e\n\x06string\x18\x01
\x01(\t\"g\n\x03Msg\x12\n\n\x02ok\x18\x02 \x01(\x08\x12\r\n\x05\x65rror\x18\x03
\x01(\t\x12\x16\n\x06states\x18\x04 \x03(\x0b\x32\x06.State\x12\x15\n\x05query\x18\x05
\x01(\x0b\x32\x06.Query\x12\x16\n\x06\x65vents\x18\x06
\x03(\x0b\x32\x06.Event\"\'\n\tAttribute\x12\x0b\n\x03key\x18\x01 \x02(\t\x12\r\n\x05value\x18\x02
\x01(\tB\x1a\n\x11\x63om.aphyr.riemannB\x05Proto'
)
_STATE = _descriptor.Descriptor(
name='State',
full_name='State',
filename=None,
file=DESCRIPTOR,
containing_type=None,
create_key=_descriptor._internal_create_key,
fields=[
_descriptor.FieldDescriptor(
name='time', full_name='State.time', index=0,
number=1, type=3, cpp_type=2, label=1,
has_default_value=False, default_value=0,
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR,
create_key=_descriptor._internal_create_key),
_descriptor.FieldDescriptor(
name='state', full_name='State.state', index=1,
number=2, type=9, cpp_type=9, label=1,
has_default_value=False, default_value=b"".decode('utf-8'),
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR,
create_key=_descriptor._internal_create_key),
_descriptor.FieldDescriptor(
name='service', full_name='State.service', index=2,
number=3, type=9, cpp_type=9, label=1,
has_default_value=False, default_value=b"".decode('utf-8'),
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR,
create_key=_descriptor._internal_create_key),
_descriptor.FieldDescriptor(
name='host', full_name='State.host', index=3,
number=4, type=9, cpp_type=9, label=1,
has_default_value=False, default_value=b"".decode('utf-8'),
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR,
create_key=_descriptor._internal_create_key),
_descriptor.FieldDescriptor(
name='description', full_name='State.description', index=4,
Bug#1031716: python3-protobuf: Do reverse dependencies need stricter version constraints?
On Tue, Feb 21, 2023 at 1:27 PM Adrian Bunk wrote: > Looking at #1028371, should generated dependencies on python3-protobuf be > python3-protobuf (>= 3.21), python3-protobuf (<< 3.22) You mean on python3-bernhard, right? > to ensure that the binary package is used with the same version > as the protobuf-compiler used during the build? Then what? It would become uninstallable when a new protobuf version (3.22 next time) is uploaded and built. > If yes, are other language bindings also affected by this problem? I still don't get your point. How does a language binding package version relate to the protobuf-compiler version? I don't follow the internals of Protobuf, but I would say it's more related to the library soname and its provided API version. Regards, Laszlo/GCS
Bug#1031716: python3-protobuf: Do reverse dependencies need stricter version constraints?
Package: python3-protobuf Version: 3.21.12-1 Severity: serious Control: block 1028371 by -1 Looking at #1028371, should generated dependencies on python3-protobuf be python3-protobuf (>= 3.21), python3-protobuf (<< 3.22) to ensure that the binary package is used with the same version as the protobuf-compiler used during the build? If yes, are other language bindings also affected by this problem?
