Instead of creating a pre-shaded version, we could also just shade+reloacate protobuf into the appropriate Omid client JAR during the build.
In theory, we could also pre-shade protbuf during the omid build in a separate module, but IIRC some Maven limitation made this impossible. This would not require coordinating protobuf changes between components, and would be less of a pain. Unlike HBase, Omid does not need a patched protobuf, we just want to avoid classpath conflicts. On the other hand this would probably require fixing the assembly process, but that is another tech debt we should have tackled a long time ago. I think that fixing the Omid assembly is a better use of our time than creating a new component/artifact. Istvan On Tue, Oct 8, 2024 at 10:38 AM Istvan Toth <st...@cloudera.com> wrote: > correction: > > many of which use unallocated protobuf, > > > I meant unrelocated. > > On Mon, Oct 7, 2024 at 9:37 AM Istvan Toth <st...@cloudera.com> wrote: > >> Unfortunately Protobuf has made the decision of breaking binary >> compatibility between the frequently changing major versions, >> creating a Guava-like problem. >> >> Since Omid (or rather HBase/Phoenix) is frequently used together with >> other Hadoop-adjacent software depending on Protobuf, many of which use >> unallocated protobuf, >> we need to solve this somehow. >> >> To avoid adding another repo, we could add protobuf to the existing >> phoenix-thirdparty repo, since Omid is a Phoenix subproject now. >> >> Istvan >> >> On Wed, Oct 2, 2024 at 1:51 PM Villő Szűcs >> <szucsvi...@cloudera.com.invalid> wrote: >> >>> Hey all, >>> I've been working on OMID-284 and OMID-276, specifically attempting to >>> upgrade Protobuf in OMID, and I've encountered several challenges that I >>> want to bring to the community for discussion. >>> >>> I tried upgrading Protobuf to version 4.28.2: see PR #170 >>> <https://github.com/apache/phoenix-omid/pull/170>. The primary >>> motivation >>> for using this version was to align with hbase-thirdparty, which also >>> uses >>> Protobuf 4. However, during testing, I encountered the following failure: >>> >>> java.lang.NoSuchMethodError:org.apache.hadoop.hdfs.server.namenode.FsImageProto$FileSummary.makeExtensionsImmutable()V >>> This error suggests that Hadoop is still using an older version of >>> Protobuf, and this is causing compatibility issues. >>> I also tried using Protobuf version 3.25.5, and it worked without any >>> test >>> failures. However, this is not aligned with the Protobuf version used in >>> hbase-thirdparty, and if we try to fully switch to hbase-thirdparty's >>> shaded Protobuf, we are likely to run into the same issues that I faced >>> with Protobuf 4. >>> >>> To resolve this, my current suggestion is to create a new Omid-thirdparty >>> repository and shade Protobuf there. This would help us isolate OMID's >>> dependencies and avoid any conflicts with Hadoop's older Protobuf >>> version. >>> >>> Do you think creating an Omid-thirdparty repository for shading Protobuf >>> is >>> a viable solution? >>> Are there any other options that might avoid introducing a new repository >>> while still addressing the compatibility issues? >>> >>> Thanks in advance for your input! >>> Best, >>> Villő >>> >> >> >> -- >> *István Tóth* | Sr. Staff Software Engineer >> *Email*: st...@cloudera.com >> cloudera.com <https://www.cloudera.com> >> [image: Cloudera] <https://www.cloudera.com/> >> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: >> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: >> Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera> >> ------------------------------ >> ------------------------------ >> > > > -- > *István Tóth* | Sr. Staff Software Engineer > *Email*: st...@cloudera.com > cloudera.com <https://www.cloudera.com> > [image: Cloudera] <https://www.cloudera.com/> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: > Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera> > ------------------------------ > ------------------------------ > -- *István Tóth* | Sr. Staff Software Engineer *Email*: st...@cloudera.com cloudera.com <https://www.cloudera.com> [image: Cloudera] <https://www.cloudera.com/> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera> ------------------------------ ------------------------------
