> I am currently using beam 2.55.1 with spark 3.3.2. Now I am upgrading to google dataproc 2.2 and planning to continue with 2.55.1 beam version itself. So, I created a PR https://github.com/apache/beam/pull/35998 to run the automated tests of 2.55.1 against 3.5.3 spark runner. But some test cases have been failing. Is this the expected result or should I do something differently ?
Interestingly the test suite that failed was sparkVersion331Test. That is surprising, but probably not something to worry about since that is testing against Spark 3.3.1. The 3.5.3 test suite succeeded so there shouldn't be anything to worry about there. I would assume the failure was just a flake. > I was able to run unit test cases of in my local, are these the same test cases that run during the PR CI also or do you have other scenarios that get triggered during CI ? I'm not sure which tests you ran locally, so I can't say for sure. If you go to https://github.com/apache/beam/actions/runs/17296542424?pr=35998 and download the JUnit Test Results at the bottom, you can see which tests ran on the PR though. Thanks, Danny On Thu, Aug 28, 2025 at 11:41 AM Santhosh Kumar Srinivasan < santhoshkumar.sri...@walmart.com> wrote: > Hello Danny, > > > > Thanks for your response. I have couple of questions here. > > > > 1. I am currently using beam 2.55.1 with spark 3.3.2. Now I am > upgrading to google dataproc 2.2 and planning to continue with 2.55.1 beam > version itself. So, I created a PR > https://github.com/apache/beam/pull/35998 to run the automated tests > of 2.55.1 against 3.5.3 spark runner. But some test cases have been > failing. Is this the expected result or should I do something differently ? > > > > 2. I was able to run unit test cases of in my local, are these the > same test cases that run during the PR CI also or do you have other > scenarios that get triggered during CI ? > > > > *From: *Danny McCormick via user <user@beam.apache.org> > *Date: *Thursday, 28 August 2025 at 2:37 AM > *To: *user@beam.apache.org <user@beam.apache.org> > *Cc: *Danny McCormick <dannymccorm...@google.com> > *Subject: *EXT: Re: Apache Beam Spark Runner Support - Spark 3.5.3 > > *EXTERNAL: *Report suspicious emails to *Email Abuse.* > > Generally we don't do compatibility checks for every patch version within > a minor version, but we have tested 3.5.5 and 3.5.0, and both work. > > > > I added 3.5.3 in https://github.com/apache/beam/pull/35985 > <https://urldefense.com/v3/__https:/github.com/apache/beam/pull/35985__;!!IfjTnhH9!Vf4qb_WxIrfMJYI9AGvqorEjAXi-JjiAnxFSUm0u1wubf-TAyp_DKiP7w22_ETKLDhejBiKpqYAm_tnJY4LaSL3-yw$> > to > quickly test it more directly, and all tests passed. In the future, you're > welcome to open PRs to run automated tests against particular minor > versions if it is helpful as well, you'd just need to do the same thing I > did here. > > > > I see 3.5.x support from > https://github.com/apache/beam/blob/release-2.53.0/runners/spark/3/build.gradle > <https://urldefense.com/v3/__https:/github.com/apache/beam/blob/release-2.53.0/runners/spark/3/build.gradle__;!!IfjTnhH9!Vf4qb_WxIrfMJYI9AGvqorEjAXi-JjiAnxFSUm0u1wubf-TAyp_DKiP7w22_ETKLDhejBiKpqYAm_tnJY4I-ATq3Og$> > so > I would expect Beam >=2.53.0 to work here. > > > > Thanks, > > Danny > > > > On Wed, Aug 27, 2025 at 4:53 AM Santhosh Kumar Srinivasan via user < > user@beam.apache.org> wrote: > > Hello Beam Community, > > > > As of now, we are not seeing any version of Apache Beam that is certified > for compatibility with Spark 3.5.3 (which comes with Google Dataproc 2.2 > <https://urldefense.com/v3/__https:/cloud.google.com/dataproc/docs/concepts/versioning/dataproc-release-2.2__;!!IfjTnhH9!Vf4qb_WxIrfMJYI9AGvqorEjAXi-JjiAnxFSUm0u1wubf-TAyp_DKiP7w22_ETKLDhejBiKpqYAm_tnJY4LLku_6xQ$> > ) > > > > > https://github.com/apache/beam/blob/release-2.65/runners/spark/3/build.gradle > <https://urldefense.com/v3/__https:/github.com/apache/beam/blob/release-2.65/runners/spark/3/build.gradle__;!!IfjTnhH9!Vf4qb_WxIrfMJYI9AGvqorEjAXi-JjiAnxFSUm0u1wubf-TAyp_DKiP7w22_ETKLDhejBiKpqYAm_tnJY4IUCP-DSQ$> > > > > https://github.com/apache/beam/blob/release-2.66/runners/spark/3/build.gradle > <https://urldefense.com/v3/__https:/github.com/apache/beam/blob/release-2.66/runners/spark/3/build.gradle__;!!IfjTnhH9!Vf4qb_WxIrfMJYI9AGvqorEjAXi-JjiAnxFSUm0u1wubf-TAyp_DKiP7w22_ETKLDhejBiKpqYAm_tnJY4KSbGO64g$> > > > > > Can you please help with certifying Beam for Spark 3.5.3 also ? > > > > Thanks, > > Santhosh > > > > > >
