Update: "org.apache.hive:hive-exec" is actually a shaded jar that doesn't do any repackaging for popular libraries (e.g. like "protobuf-java"). My gradle logic was working as intended, but I was still running into issues because multiple classes with the same name were available to the Java class loader (one from my "protobuf-java" dependency, another from the shaded "hive-exec").
We were able to work around this by shading "hive-exec" ourselves and relocating the problematic namespace, similar to how it's done here: https://github.com/GradleUp/shadow/issues/384 Shoutout to @Yi Hu <ya...@google.com> for the help! On Mon, Jul 29, 2024 at 4:48 PM Tomo Suzuki <suzt...@google.com> wrote: > "resolutionStrategy.force" should work. Use "./gradlew dependencies" to > debug the dependencies. > https://docs.gradle.org/current/userguide/viewing_debugging_dependencies.html > > > On Mon, Jul 29, 2024 at 4:23 PM Ahmed Abualsaud via dev < > dev@beam.apache.org> wrote: > >> Hey all, >> >> Does anyone have experience with sticky dependencies in gradle? I'm >> experimenting with the new Managed Iceberg connector and trying to write to >> GCS with a Hive catalog. >> >> I naturally need to pull some hive dependencies, but one particular >> module is very stubborn with its dependencies: >> *org.apache.hive:hive-exec:3.1.3.* >> >> I need to override its dependency on *com.google.protobuf:protobuf-java* >> (version 2.5.0) to use a more up-to-date version, but nothing I do seems >> to work >> >> I tried the following in my gradle file, but running the code always >> defaults to classes in *protobuf-java:2.5.0*: >> >> >> configurations.all { >> resolutionStrategy.force 'com.google.protobuf:protobuf-java:3.25.3' >> } >> >> dependencies { >> testImplementation 'com.google.protobuf:protobuf-java:3.25.3' >> >> testImplementation("org.apache.hive:hive-exec:3.1.3") { >> exclude group: "com.google.protobuf", module: "protobuf-java" >> } >> testImplementation "org.apache.iceberg:iceberg-hive-metastore:1.4.2" >> testImplementation("org.apache.hive.hcatalog:hive-hcatalog-core:3.1.3") { >> exclude group: "org.apache.hive", module: "hive-exec" >> exclude group: "com.google.protobuf", module: "protobuf-java" >> } >> >> constraints { >> testImplementation('com.google.protobuf:protobuf-java') { >> version { >> strictly '3.25.3' >> } >> } >> } >> } >> >> >> I also tried adding the following, which stripped *protobuf-java* from >> everything except *hive-exec:* >> >> >> configurations.all { >> exclude group: 'com.google.protobuf', module: 'protobuf-java' >> } >> >> >> I also had similar pain points when trying to override >> org.apache.parquet:parquet-column. The hive module seems very intent on >> keeping its dependencies. >> >> Can anyone share their experiences with stubborn dependencies such as >> these? Would appreciate any tips! >> >> Best, >> Ahmed >> > > > -- > Regards, > Tomo >
