Hi there, I am currently prepping my PR 2 for Debezium, but I am getting errors from the CI that are related to Java 11. -> Task with path ':sdks:java:container:distroless:java11:docker' not found in project ':runners:google-cloud-dataflow-java' [0]
Is there a way to disable this via a flag or config setting? [0] https://github.com/apache/beam/actions/runs/15115381426/job/42484628013?pr=34763 On Wed, Apr 30, 2025 at 9:29 PM Tobi Kaymak <kay...@google.com> wrote: > This is super helpful, thank you! > I was looking for these kinds of switches in the other build configs, I > agree that a generalization of this would be helpful going forward > (thinking about other libraries that might require higher Java versions in > the future). > Thank you for taking the time to test my PR, I will incorporate your Java > 17/21 check into it and see about the stuck integration test. > > On Wed, Apr 30, 2025 at 6:08 PM Yi Hu <ya...@google.com> wrote: > >> Yeah those configurations are needed to accommodate errorprone. I did >> some test on https://github.com/apache/beam/pull/34788 based on your PR. >> I get Debezium build on CI and unit test passing (except there is an >> integration test stuck). >> >> >> On Wed, Apr 30, 2025 at 11:46 AM Tobi Kaymak <kay...@google.com> wrote: >> >>> Thank you! Yes, that is my understanding too. >>> I tried to force Java 17, in Debezium IOs build.gradle, but I failed so >>> far. >>> I even tried to completely clear the arguments and manually construct >>> them for the gradle command line [0], however then this clashed >>> with ErrorProne. >>> I will give it another shot in the coming days. >>> >>> [0] >>> https://github.com/apache/beam/pull/34763/commits/a779e201941ba10672e8225229bc59e0afd01e21 >>> >>> On Wed, Apr 30, 2025 at 5:18 PM Yi Hu <ya...@google.com> wrote: >>> >>>> The purpose of the fallback [0] is to make other Beam components build >>>> on Java11, while only `:sdks:java:io:debezium` should be configured to >>>> build on Java17, in its build.gradle file. >>>> >>>> On Wed, Apr 30, 2025 at 11:14 AM XQ Hu via dev <dev@beam.apache.org> >>>> wrote: >>>> >>>>> I think we can remove 11. Looks like we always do this: JAVA_HOME: >>>>> /opt/hostedtoolcache/Java_Temurin-Hotspot_jdk/11.0.27-6/x64 >>>>> >>>>> However, we probably need to change that direct github workflow to >>>>> remove :sdks:java:container:java8 and :sdks:java:container:java11 and >>>>> add :sdks:java:container:java17. >>>>> >>>>> On Wed, Apr 30, 2025 at 10:23 AM Tobi Kaymak <kay...@google.com> >>>>> wrote: >>>>> >>>>>> I do have a silly question, I am struggling to find the reason for >>>>>> the fallback need of Java 11 in the >>>>>> beam_PreCommit_Java_Debezium_IO_Direct.yml [0] >>>>>> I was trying to not interfere with the build system outside of the >>>>>> module by doing some tricks within the build.gradle of Debzium IO in the >>>>>> second PR. [1] >>>>>> However, it seems that the CI always picks up Java 11 and therefore >>>>>> fails the build. It builds completely fine with Java 17 on my machine >>>>>> though. >>>>>> Is there a way to force Java 17 for the module by, for example, >>>>>> removing the 11 fallback from the PreCommit Workflow or does this have >>>>>> known side-effects? >>>>>> >>>>>> Thank you! >>>>>> >>>>>> [0] >>>>>> https://github.com/apache/beam/pull/34751/files#diff-de752bcab62398ee28adb75a8fdb1933d8d855f2ae46417ec1d47180174a1800R91 >>>>>> [1] https://github.com/apache/beam/pull/34763 >>>>>> >>>>>> On Mon, Apr 28, 2025 at 5:04 PM Tobi Kaymak <kay...@google.com> >>>>>> wrote: >>>>>> >>>>>>> I did the second PR and removed the enforcement of proto3 in the >>>>>>> build.gradle ( >>>>>>> https://github.com/apache/beam/pull/34763/files#diff-1572bcc322ede7d3a757381b84791a29b691df4d8947f6ad39ce4597653787edL95) >>>>>>> - >>>>>>> All tests are green on my local machine, but they will of course >>>>>>> fail on the CI since the first PR has not been merged yet. >>>>>>> >>>>>>> I would be very happy about a review by Danny McCormick, who did >>>>>>> work on this a few months ago in his PR and added the enforcement ( >>>>>>> https://github.com/apache/beam/pull/33526). >>>>>>> >>>>>>> >>>>>>> On Sun, Apr 27, 2025 at 5:00 PM Tobi Kaymak <kay...@google.com> >>>>>>> wrote: >>>>>>> >>>>>>>> I did create the first PR, after building the IO on my local >>>>>>>> machine with gradle and Java 17. This should not have an effect other >>>>>>>> than >>>>>>>> changing the build system. Feedback is very welcome: >>>>>>>> https://github.com/apache/beam/pull/34751 >>>>>>>> >>>>>>>> On Sat, Apr 26, 2025 at 12:49 AM Tobi Kaymak <kay...@google.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Thank you both! >>>>>>>>> I have checked and couldn't find a matching issue, so I created >>>>>>>>> this one: https://github.com/apache/beam/issues/34747 >>>>>>>>> >>>>>>>>> On Sat, Apr 26, 2025 at 12:32 AM XQ Hu via dev < >>>>>>>>> dev@beam.apache.org> wrote: >>>>>>>>> >>>>>>>>>> Can we create the GitHub issue to track this work if we haven't? >>>>>>>>>> >>>>>>>>>> On Fri, Apr 25, 2025, 4:39 PM Yi Hu via dev <dev@beam.apache.org> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Tobi, >>>>>>>>>>> >>>>>>>>>>> Thanks for your interest and willing to take on the task. First >>>>>>>>>>> of all, could you please create a GitHub Issue to track the effort? >>>>>>>>>>> >>>>>>>>>>> I would like to share some comments if find useful >>>>>>>>>>> >>>>>>>>>>> - We already have infrastructure for the goal of "PR 1". We use >>>>>>>>>>> BeamPluginModulePlugin [1] to initialize java modules and there is a >>>>>>>>>>> setJavaVerOptions method. Here is a module could be an example how >>>>>>>>>>> compiling with different Java version works in Beam: [2]. >>>>>>>>>>> >>>>>>>>>>> - We'll need to handle Debezium with newer Java version as well >>>>>>>>>>> (That's the reason test currently failing on #34733). We'll need to >>>>>>>>>>> modify >>>>>>>>>>> test workflow file [3]. However due to GitHub limitation changes to >>>>>>>>>>> .github/workflows won't be effective on unsubmitted PRs. You'd need >>>>>>>>>>> to test >>>>>>>>>>> your change locally. >>>>>>>>>>> >>>>>>>>>>> - Given that current Debezium (1.x) is already old I'm +1 with >>>>>>>>>>> the change. >>>>>>>>>>> >>>>>>>>>>> [1] >>>>>>>>>>> https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy >>>>>>>>>>> >>>>>>>>>>> [2] >>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/container/agent/build.gradle >>>>>>>>>>> >>>>>>>>>>> [3] >>>>>>>>>>> https://github.com/apache/beam/blob/master/.github/workflows/beam_PreCommit_Java_Debezium_IO_Direct.yml >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Thanks. >>>>>>>>>>> >>>>>>>>>>> Yi >>>>>>>>>>> >>>>>>>>>>> On Fri, Apr 25, 2025 at 3:06 PM Tobi Kaymak via dev < >>>>>>>>>>> dev@beam.apache.org> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Beam Devs, >>>>>>>>>>>> >>>>>>>>>>>> I'm interested in helping to upgrade the Debezium IO connector ( >>>>>>>>>>>> sdks:java:io:debezium) to Debezium version 3.1.1.Final. This >>>>>>>>>>>> version of Debezium requires its dependencies (like >>>>>>>>>>>> debezium-core) to be compiled against Java 17 >>>>>>>>>>>> <https://debezium.io/releases/3.0/release-notes#breaking_changes_14> >>>>>>>>>>>> bytecode. >>>>>>>>>>>> >>>>>>>>>>>> I took the work from Danny McCormick who started doing >>>>>>>>>>>> something similar for Debezium 2.7.4 and created a WIP branch >>>>>>>>>>>> <https://github.com/apache/beam/pull/34733> to get my feet wet >>>>>>>>>>>> with Java and Gradle's build system again. >>>>>>>>>>>> >>>>>>>>>>>> To manage this upgrade gracefully and make the review process >>>>>>>>>>>> smoother, I was thinking to split the work into a two-step PR: >>>>>>>>>>>> >>>>>>>>>>>> 1. >>>>>>>>>>>> >>>>>>>>>>>> *PR 1 (Build Infrastructure):* A smaller, focused PR to >>>>>>>>>>>> enable Java 17 compilation specifically for the >>>>>>>>>>>> sdks:java:io:debezium module. This would likely involve: >>>>>>>>>>>> - Modifying the sdks:java:io:debezium/build.gradle to allow >>>>>>>>>>>> forking its compileJava (and compileTestJava) tasks >>>>>>>>>>>> using a JDK 17 (e.g., via a java17Home Gradle property >>>>>>>>>>>> derived from a CI environment variable like >>>>>>>>>>>> JAVA_HOME_17_X64). >>>>>>>>>>>> - Updating the relevant PreCommit workflow ( >>>>>>>>>>>> beam_PreCommit_Java_Debezium_IO_Direct.yml) >>>>>>>>>>>> >>>>>>>>>>>> <https://github.com/apache/beam/blob/master/.github/workflows/beam_PreCommit_Java_Debezium_IO_Direct.yml#L87> >>>>>>>>>>>> to ensure a JDK 17 is available and its path is passed to >>>>>>>>>>>> Gradle for this >>>>>>>>>>>> module. >>>>>>>>>>>> - The goal of this first PR would be to get the module >>>>>>>>>>>> to compile successfully with Java 17 against the existing >>>>>>>>>>>> (older) Debezium >>>>>>>>>>>> version. >>>>>>>>>>>> 2. >>>>>>>>>>>> >>>>>>>>>>>> *PR 2 (Debezium 3.1.1 Upgrade & API Adaptation):* A >>>>>>>>>>>> subsequent PR that would: >>>>>>>>>>>> - Actually update the Debezium dependencies to 3.1.1.Final. >>>>>>>>>>>> - Incorporate the necessary code changes within >>>>>>>>>>>> sdks:java:io:debezium to adapt to any API changes in >>>>>>>>>>>> Debezium 3.1.1. >>>>>>>>>>>> - Address any new functionality or considerations with >>>>>>>>>>>> Debezium 3.1.1 (which I understand uses now libprotoc-dev >>>>>>>>>>>> 1.5, >>>>>>>>>>>> >>>>>>>>>>>> <https://github.com/debezium/container-images/pull/427/files#diff-bf3b0b8b023cf3cd20ca9d862bb8301523cdb9c934dec1ce96cd9640a613ae7e> >>>>>>>>>>>> - >>>>>>>>>>>> according to its release notes >>>>>>>>>>>> >>>>>>>>>>>> <https://debezium.io/releases/3.1/release-notes#other_changes_5> >>>>>>>>>>>> . >>>>>>>>>>>> >>>>>>>>>>>> My reasoning for this split is to isolate the build system >>>>>>>>>>>> changes from the actual library upgrade and code adaptation, >>>>>>>>>>>> allowing for >>>>>>>>>>>> smaller code reviews. >>>>>>>>>>>> >>>>>>>>>>>> I'd appreciate any feedback or suggestions on the best way to >>>>>>>>>>>> upgrade Debezium IO. >>>>>>>>>>>> >>>>>>>>>>>> Best >>>>>>>>>>>> Tobi >>>>>>>>>>>> >>>>>>>>>>>
