Hello Danny, Thanks for your response. I have couple of questions here.
1. I am currently using beam 2.55.1 with spark 3.3.2. Now I am upgrading to google dataproc 2.2 and planning to continue with 2.55.1 beam version itself. So, I created a PR https://github.com/apache/beam/pull/35998 to run the automated tests of 2.55.1 against 3.5.3 spark runner. But some test cases have been failing. Is this the expected result or should I do something differently ? 1. I was able to run unit test cases of in my local, are these the same test cases that run during the PR CI also or do you have other scenarios that get triggered during CI ? From: Danny McCormick via user <user@beam.apache.org> Date: Thursday, 28 August 2025 at 2:37 AM To: user@beam.apache.org <user@beam.apache.org> Cc: Danny McCormick <dannymccorm...@google.com> Subject: EXT: Re: Apache Beam Spark Runner Support - Spark 3.5.3 EXTERNAL: Report suspicious emails to Email Abuse. Generally we don't do compatibility checks for every patch version within a minor version, but we have tested 3.5.5 and 3.5.0, and both work. I added 3.5.3 in https://github.com/apache/beam/pull/35985<https://urldefense.com/v3/__https:/github.com/apache/beam/pull/35985__;!!IfjTnhH9!Vf4qb_WxIrfMJYI9AGvqorEjAXi-JjiAnxFSUm0u1wubf-TAyp_DKiP7w22_ETKLDhejBiKpqYAm_tnJY4LaSL3-yw$> to quickly test it more directly, and all tests passed. In the future, you're welcome to open PRs to run automated tests against particular minor versions if it is helpful as well, you'd just need to do the same thing I did here. I see 3.5.x support from https://github.com/apache/beam/blob/release-2.53.0/runners/spark/3/build.gradle<https://urldefense.com/v3/__https:/github.com/apache/beam/blob/release-2.53.0/runners/spark/3/build.gradle__;!!IfjTnhH9!Vf4qb_WxIrfMJYI9AGvqorEjAXi-JjiAnxFSUm0u1wubf-TAyp_DKiP7w22_ETKLDhejBiKpqYAm_tnJY4I-ATq3Og$> so I would expect Beam >=2.53.0 to work here. Thanks, Danny On Wed, Aug 27, 2025 at 4:53 AM Santhosh Kumar Srinivasan via user <user@beam.apache.org<mailto:user@beam.apache.org>> wrote: Hello Beam Community, As of now, we are not seeing any version of Apache Beam that is certified for compatibility with Spark 3.5.3 (which comes with Google Dataproc 2.2<https://urldefense.com/v3/__https:/cloud.google.com/dataproc/docs/concepts/versioning/dataproc-release-2.2__;!!IfjTnhH9!Vf4qb_WxIrfMJYI9AGvqorEjAXi-JjiAnxFSUm0u1wubf-TAyp_DKiP7w22_ETKLDhejBiKpqYAm_tnJY4LLku_6xQ$>) https://github.com/apache/beam/blob/release-2.65/runners/spark/3/build.gradle<https://urldefense.com/v3/__https:/github.com/apache/beam/blob/release-2.65/runners/spark/3/build.gradle__;!!IfjTnhH9!Vf4qb_WxIrfMJYI9AGvqorEjAXi-JjiAnxFSUm0u1wubf-TAyp_DKiP7w22_ETKLDhejBiKpqYAm_tnJY4IUCP-DSQ$> https://github.com/apache/beam/blob/release-2.66/runners/spark/3/build.gradle<https://urldefense.com/v3/__https:/github.com/apache/beam/blob/release-2.66/runners/spark/3/build.gradle__;!!IfjTnhH9!Vf4qb_WxIrfMJYI9AGvqorEjAXi-JjiAnxFSUm0u1wubf-TAyp_DKiP7w22_ETKLDhejBiKpqYAm_tnJY4KSbGO64g$> Can you please help with certifying Beam for Spark 3.5.3 also ? Thanks, Santhosh
