Bug#1034668: Please update to new upstream release 3.22.3 in experimental
On Thu, Sep 7, 2023 at 7:24 PM Benjamin Barenblat wrote: > I will do an upload of 20230802.0 to experimental today, but I don’t > think it fixes this issue. scoped_mock_log depends on symbols in > GoogleMock, but there’s no good way to express those dependencies in a > hypothetical libabsl_scoped_mock_log.so since libgmock.so doesn’t exist. Thanks for the upload and the information. I'm going to check it soon. > The way upstream (Google) intends for all this to work is for protobuf > to link its Abseil and GoogleMock dependencies statically. Is that > possible? It is possible, but not recommended. If abseil gets (security) fixes, protobuf will not get it like in the case of loading fresh shared libraries. Also it seems protobuf configure process looks for shared version of abseil. Need to double check. > If not, I have some alternative ideas (like building a > libabsl_scoped_mock_log.so without a DT_NEEDED entry for GoogleMock), > but they all seem like hacks. I’m also open to other ideas if anybody > has them. I'm not sure how to handle this properly for everyone. I think I will solve this on the protobuf side. Use static linking with abseil and record with the build process which version was used. The hard way would be to bundle an abseil source tarball with protobuf and use its shared version of libraries from a private location only for protobuf. The first should get precedence for which abseil should export static linking possibilities of scoped_mock_log in its cmake files. Regards, Laszlo/GCS
Bug#1034668: Please update to new upstream release 3.22.3 in experimental
I will do an upload of 20230802.0 to experimental today, but I don’t think it fixes this issue. scoped_mock_log depends on symbols in GoogleMock, but there’s no good way to express those dependencies in a hypothetical libabsl_scoped_mock_log.so since libgmock.so doesn’t exist. The way upstream (Google) intends for all this to work is for protobuf to link its Abseil and GoogleMock dependencies statically. Is that possible? If not, I have some alternative ideas (like building a libabsl_scoped_mock_log.so without a DT_NEEDED entry for GoogleMock), but they all seem like hacks. I’m also open to other ideas if anybody has them.
Bug#1034668: Please update to new upstream release 3.22.3 in experimental
Hi Nilesh, Benjamin, On Sat, Jul 22, 2023 at 5:06 AM Nilesh Patra wrote: > On Thu, 27 Apr 2023 19:59:38 +0530 Pirate Praveen > wrote: > > Target "tests" links to: > > > >absl::scoped_mock_log > > > > but the target was not found. Possible reasons include: The reason is that it and protobuf 3.24.2 need abseil 20230125.0 or newer. While it is available in experimental, it doesn't export scoped_mock_log targets for cmake. Hence it is not found, I don't know if newer versions like 20230802.0 fix this issue or not. Can you comment on this Benjamin and if you are going to package it soon and/or accept someone as a co-maintainer for abseil? It seems abseil 20230802.0 became available and should be packaged. > Now, the release of abseil in experimental seems to be doing that, but > only for the static library and not the shared library, because it needs > googltest for the same, and googletest does not vendor any shared > library because the SO lib is not maintained properly upstream[2] [...] [...] > This was straightforward was to add "-Dprotobuf_BUILD_TESTS=OFF" to > configure options. Building without tests are always a bad idea. > PS: Sponsor me a party/dinner when? If you are at DebConf, we may talk and eat together. Regards, Laszlo/GCS
Bug#1034668: Please update to new upstream release 3.22.3 in experimental
The last comment that I probbaly should add is that you also need to check if there's api changes between 3.21 and 3.22/3.23 (there probably are). In such a case you need to bump SONAME in d/rules and also change soname in binary packages from 32 to 33/34. This should not be complicated, just need to `mv` some files, that's all (and change soname in rules/cmakelists). Best, Nilesh signature.asc Description: PGP signature
Bug#1034668: Please update to new upstream release 3.22.3 in experimental
On Sat, Jul 22, 2023 at 05:33:03AM +0530, Nilesh Patra wrote:
> **NB1:** The static libs are not present in the protobuf lib at the moment
> because this changed after the build system changed, and needs some work to
> get both shared
> and static libs enabled and it's 5 in the morning and
> I'm too tired and sleepy to work on it. For gitlab's usecase, static libs are
> *probably*
> not needed.
This was actually easy to implement, attached modified patch to get the
static libraries building as well.
The third party utf8's static libs are not installed still
because I don't really know if they are needed.
But if they are, it should be as simple as adding it to an install
file with relevant path, much like how protobuf's static libs are
installed.
That said, my work for the gitlab issue ends here
> **NB2:** bazel buildsystem probably downloads some stuff off the
> internet for this package (I did not check in depth) so the debian way
> of doing it would be to either package the stuff it needs or download
> copies beforehand and use that in build.
This still seems to be an issue, though. During build I could see it
fetch some tarballs from github.
> PS: Sponsor me a party/dinner when?
>
> [0]: https://people.debian.org/~praveen/new/pool/main/p/protobuf/
> [1]: https://github.com/abseil/abseil-cpp/issues/1407
> [2]: https://github.com/google/googletest/issues/2184
> [3]:
> https://salsa.debian.org/debian/abseil/-/commit/136c2572f4d6e6ab8ae02f74d691634874458184
> [4]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1034908#48
Best,
Nilesh
commit 287465705eea4572b76a5d45308c2a5c3377cead
Author: Nilesh Patra
Date: Sat Jul 22 12:53:53 2023 +0530
Vendor static libs as well
diff --git a/debian/changelog b/debian/changelog
index 338446c..03edb4e 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,9 @@
+protobuf (3.23.4-1) UNRELEASED; urgency=medium
+
+ * New upstream version 3.23.4
+
+ -- Nilesh Patra Sat, 22 Jul 2023 00:14:20 +0530
+
protobuf (3.22.3-1) experimental; urgency=medium
* New upstream release 3.22.3.
diff --git a/debian/clean b/debian/clean
index 43ff921..905c503 100644
--- a/debian/clean
+++ b/debian/clean
@@ -1,3 +1,8 @@
protoc.1
ruby/ext/google/protobuf_c/third_party/wyhash/wyhash.h
ruby/tests/multi_level_nesting_test_pb.rb
+ruby/ext/google/protobuf_c/naive.c
+ruby/ext/google/protobuf_c/range2-neon.c
+ruby/ext/google/protobuf_c/range2-sse.c
+ruby/ext/google/protobuf_c/utf8_range.h
+python/google/protobuf/internal/generator_test.py
diff --git a/debian/control b/debian/control
index 6bc74c8..cff61f0 100644
--- a/debian/control
+++ b/debian/control
@@ -10,7 +10,8 @@ Build-Depends:
, cmake
# C/C++
, zlib1g-dev
- , libabsl-dev
+ , bazel-bootstrap
+ , libabsl-dev (>= 20230125.3-1~)
, libgmock-dev
, libgtest-dev
# Python
@@ -19,6 +20,7 @@ Build-Depends:
, libpython3-all-dev
, python3-setuptools
, python3-six
+ , python3-numpy
# Manpage generator
, xmlto
# Tests
diff --git a/debian/libprotobuf-dev.install b/debian/libprotobuf-dev.install
index ad15927..a2c8719 100644
--- a/debian/libprotobuf-dev.install
+++ b/debian/libprotobuf-dev.install
@@ -4,3 +4,4 @@ usr/lib/*/libprotobuf.so
usr/lib/*/libprotobuf-lite.so
usr/lib/*/pkgconfig/*
usr/include
+usr/lib/*/cmake/protobuf/*.cmake
diff --git a/debian/patches/append-lib-dirs.patch b/debian/patches/append-lib-dirs.patch
new file mode 100644
index 000..ab27a3e
--- /dev/null
+++ b/debian/patches/append-lib-dirs.patch
@@ -0,0 +1,21 @@
+--- a/python/setup.py
b/python/setup.py
+@@ -344,6 +344,9 @@
+ else:
+ library_dirs = ['..']
+
++if library_dirs is None:
++library_dirs= []
++library_dirs.append('../shared')
+ TestConformanceCmd.target = ('//python:conformance_test_cpp '
+ '--define=use_fast_cpp_protos=true')
+
+@@ -412,7 +415,7 @@
+ Extension(
+ 'google.protobuf.pyext._message',
+ glob.glob('google/protobuf/pyext/*.cc'),
+-include_dirs=['.', '../src', '../third_party/abseil-cpp'],
++include_dirs=['.', '../src'],
+ libraries=libraries,
+ extra_objects=extra_objects,
+ extra_link_args=message_extra_link_args,
diff --git a/debian/patches/fix_hppa_alignof_assert.patch b/debian/patches/fix_hppa_alignof_assert.patch
index fad5aa3..d4b703b 100644
--- a/debian/patches/fix_hppa_alignof_assert.patch
+++ b/debian/patches/fix_hppa_alignof_assert.patch
@@ -8,7 +8,7 @@ Last-Update: 2023-04-09
--- a/src/google/protobuf/descriptor.cc
+++ b/src/google/protobuf/descriptor.cc
-@@ -381,7 +381,9 @@
+@@ -384,7 +384,9 @@
ABSL_CHECK(!has_allocated());
if (std::is_trivially_destructible::value) {
// Trivial types are aligned to 8 bytes.
@@ -20,7 +20,7 @@ Last-Update: 2023-04-09
// Since we can't use `if constexpr`, just make the expression compile
--- a/src/google/protobuf/map.h
+++ b/src/google/protobuf/map.h
-@@
Bug#1034668: Please update to new upstream release 3.22.3 in experimental
Hi, On Thu, 27 Apr 2023 19:59:38 +0530 Pirate Praveen wrote: > With a newer libabsl-dev from > https://people.debian.org/~praveen/new/pool/main/a/abseil/, the error > changed to > > CMake Error at cmake/tests.cmake:97 (target_link_libraries): > Target "tests" links to: > >absl::scoped_mock_log > > but the target was not found. Possible reasons include: Tl;dr: I targeted the version "3.23.4" of protobuf and could get it building taking[0] changes as reference. Patch attached. This seems to be an issue with abseil itself which is currently unfixed[1]. The workaround suggested is to add in "ABSL_BUILD_TESTING=ON" and "BUILD_TESTING=ON" options to cmake flags. Now, the release of abseil in experimental seems to be doing that, but only for the static library and not the shared library, because it needs googltest for the same, and googletest does not vendor any shared library because the SO lib is not maintained properly upstream[2] and hence trying to enable the above options for abseil for shared object compilation chokes with a dh_shlibdeps error shouting that it also needs a so lib for gtest. The maintainer of abseil has implemented the same for shared lib in this commit[3] and has explained with a solid reasoning, and you were involved (as a reporter) in a bug too, regarding the same. The reason why you get the above issue (as far as i understand) is because protobuf tries to link with abseil in a shared lib form, and I found it non-trivial to change that to fit all usecases. Since this was a test-only thing, the best way for now is to disable the tests for now until this is fixed properly in abseil. This was straightforward was to add "-Dprotobuf_BUILD_TESTS=OFF" to configure options. Now, I had to do multiple changes to get it working. I'm mentioning them incase that helps you/someone else who looks at it: * Refreshed the existing patches * Added in -Dprotobuf_BUILD_TESTS=OFF -DBUILD_SHARED_LIBS=ON to get shared lib building as well. * There's no need of any `-D` in front of dh_auto_build since this translates into a `make` command (which fails with -D), removed that. * Saw some bazel stuff failing, had to add a B-D on bazel-bootstrap and also set HOME to /tmp for the descriptior_proto bash script so it builds fine in sbuild. * The build proceeded forward but choked at python build. Turns out due to buildsystem change, binaries are no longer available at src/ but in obj-* dir. Added a patch to setup.py to link against that dir. * Now the build moved but choked at ruby's build. Turns out it isn't sensible to add entire ruby/ext/google/protobuf_c to d/clean. Adding 4 files is enough. * Now the java part of the build failed. Here, d/p/no_javax.patch has to be updated to remove more instances of javax and @Nullable annotation. * In addition to this change, the java build xml expected protoc binary to be in src/ but it changed to obj-$(DEB_HOST_MULTIARCH) so had to copy it to src/ before build and removed it after build was done (sort of a hack). * Now the python testsuite choked, has to change LD_LIBRARY_PATH to new location. Also needed a B-D on numpy. Had to exclude a generator_test here which looked syntactically incorrect to me. * After this, the install step failed, because no static libs are being built. I've removed them from install files for now, because fixing them would mean a lot of work in either d/rules or patching cmake. * Installed some .cmake files for libprotobuf. There is also a compiled lib for third_party/utf8_range but I suppose we don't need it. So overrided dh_missing to ignore that. After doing all of this, protobuf finally built. All in all, I had to look at multiple languages and buildsystems to get this working. **NB1:** The static libs are not present in the protobuf lib at the moment because this changed after the build system changed, and needs some work to get both shared and static libs enabled and it's 5 in the morning and I'm too tired and sleepy to work on it. For gitlab's usecase, static libs are *probably* not needed. **NB2:** bazel buildsystem probably downloads some stuff off the internet for this package (I did not check in depth) so the debian way of doing it would be to either package the stuff it needs or download copies beforehand and use that in build. PS: Sponsor me a party/dinner when? [0]: https://people.debian.org/~praveen/new/pool/main/p/protobuf/ [1]: https://github.com/abseil/abseil-cpp/issues/1407 [2]: https://github.com/google/googletest/issues/2184 [3]: https://salsa.debian.org/debian/abseil/-/commit/136c2572f4d6e6ab8ae02f74d691634874458184 [4]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1034908#48 Best, Nilesh commit d3abfec3189ca5188fafe09f3003138027774422 Author: Nilesh Patra Date: Sat Jul 22 00:15:06 2023 +0530 New version update diff --git a/debian/changelog b/debian/changelog index 338446c..03edb4e 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,3 +1,9
Bug#1034668: Please update to new upstream release 3.22.3 in experimental
With a newer libabsl-dev from https://people.debian.org/~praveen/new/pool/main/a/abseil/, the error changed to CMake Error at cmake/tests.cmake:97 (target_link_libraries): Target "tests" links to: absl::scoped_mock_log but the target was not found. Possible reasons include: * There is a typo in the target name. * A find_package call is missing for an IMPORTED target. * An ALIAS target is missing. Call Stack (most recent call first): CMakeLists.txt:401 (include)
Bug#1034668: Please update to new upstream release 3.22.3 in experimental
Control: affects -1 gitlab On Wed, 26 Apr 2023 22:51:08 +0530 Pirate Praveen wrote: > On Mon, 24 Apr 2023 12:03:35 +0530 Pirate Praveen > wrote: > > CMake Warning at cmake/abseil-cpp.cmake:28 (message): > > protobuf_ABSL_PROVIDER is "module" but ABSL_ROOT_DIR is wrong > > Call Stack (most recent call first): > > CMakeLists.txt:350 (include) > > > > I could skip building googletest easily, but this is not very > straight > > forward. Any ideas? > > I tried setting -Dprotobuf_ABSL_PROVIDER=package similar to grpc, but > the rules file seems too complex, so I will leave it to the maintainer > for a proper update. In the meanwhile, gitlab can use the version from > rubygems.org (being in contrib). With help from sur5r on irc, diff -Nur protobuf-3.22.3/debian/control protobuf-3.22.3-sur5r/debian/control --- protobuf-3.22.3/debian/control 2023-04-21 12:00:57.0 +0200 +++ protobuf-3.22.3-sur5r/debian/control 2023-04-26 19:23:36.007605725 +0200 @@ -10,6 +10,7 @@ , cmake # C/C++ , zlib1g-dev + , libabsl-dev , libgmock-dev , libgtest-dev # Python diff -Nur protobuf-3.22.3/debian/files protobuf-3.22.3-sur5r/debian/files --- protobuf-3.22.3/debian/files 1970-01-01 01:00:00.0 +0100 +++ protobuf-3.22.3-sur5r/debian/files 2023-04-26 19:23:41.469612410 +0200 @@ -0,0 +1 @@ +protobuf_3.22.3-1_source.buildinfo devel optional diff -Nur protobuf-3.22.3/debian/rules protobuf-3.22.3-sur5r/debian/rules --- protobuf-3.22.3/debian/rules 2023-04-09 07:50:55.0 +0200 +++ protobuf-3.22.3-sur5r/debian/rules 2023-04-26 19:20:30.327913541 +0200 @@ -38,12 +38,15 @@ PYTHON_CROSS_VARS += CXX=$(CXX) endif +override_dh_auto_configure: + dh_auto_configure -- -Dprotobuf_ABSL_PROVIDER=package + override_dh_auto_build-arch: ## Chicken<->Egg problem: protobuf requires self-generated .pb.go files to ## be built. First we build it normally; then "generate_descriptor_proto.sh" ## generates .pb.go files and replaces 'em if they are different as well as ## invokes "make protoc" which re-builds "src/protoc" if required. - dh_auto_build --arch + dh_auto_build --arch -- -Dprotobuf_ABSL_PROVIDER=package bash -x ./generate_descriptor_proto.sh ifneq ($(DEB_BUILD_ARCH),$(DEB_HOST_ARCH)) The error changed to, which he suggested needs a newer absl -- Configuring done CMake Error at cmake/libprotobuf-lite.cmake:26 (target_link_libraries): Target "libprotobuf-lite" links to: absl::absl_check but the target was not found. Possible reasons include: * There is a typo in the target name. * A find_package call is missing for an IMPORTED target. * An ALIAS target is missing.
Bug#1034668: Please update to new upstream release 3.22.3 in experimental
On Mon, 24 Apr 2023 12:03:35 +0530 Pirate Praveen wrote: > CMake Warning at cmake/abseil-cpp.cmake:28 (message): > protobuf_ABSL_PROVIDER is "module" but ABSL_ROOT_DIR is wrong > Call Stack (most recent call first): > CMakeLists.txt:350 (include) > > I could skip building googletest easily, but this is not very straight > forward. Any ideas? I tried setting -Dprotobuf_ABSL_PROVIDER=package similar to grpc, but the rules file seems too complex, so I will leave it to the maintainer for a proper update. In the meanwhile, gitlab can use the version from rubygems.org (being in contrib).
Bug#1034668: Please update to new upstream release 3.22.3 in experimental
On Fri, 21 Apr 2023 14:16:18 +0530 Praveen Arimbrathodiyil wrote: > gitlab 15.10.x needs ruby-google-protobuf 3.22.2 and it'd be great if we > can make a new upload to experimental. If you need help with the update, > I can help. I tried building the updated version, dsc here https://people.debian.org/~praveen/new/pool/main/p/protobuf/ but seems there is some changes in build system (they are using cmake now) CMake Warning at cmake/abseil-cpp.cmake:28 (message): protobuf_ABSL_PROVIDER is "module" but ABSL_ROOT_DIR is wrong Call Stack (most recent call first): CMakeLists.txt:350 (include) I could skip building googletest easily, but this is not very straight forward. Any ideas?
Bug#1034668: Please update to new upstream release 3.22.3 in experimental
Package: protobuf Version: 3.21.12-3 severity: wishlist gitlab 15.10.x needs ruby-google-protobuf 3.22.2 and it'd be great if we can make a new upload to experimental. If you need help with the update, I can help. thanks Praveen OpenPGP_0x8F53E0193B294B75.asc Description: OpenPGP public key OpenPGP_signature Description: OpenPGP digital signature
