+1 for me to drop Spark 2.x support. Users who want to still use Spark 2.x can be back on a previous Beam release.
Regards JB On Thu, Mar 31, 2022 at 5:51 PM Alexey Romanenko <[email protected]> wrote: > > Hi everyone, > > For the moment, Beam Spark Runner supports two versions of Spark - 2.x and > 3.x. > > Taking into account the several things that: > - almost all cloud providers already mostly moved to Spark 3.x as a main > supported version; > - the latest Spark 2.x release (Spark 2.4.8, maintenance release) was done > almost a year ago; > - Spark 3 is considered as a mainstream Spark version for development and bug > fixing; > - better to avoid the burden of maintenance (there are some incompatibilities > between Spark 2 and 3) of two versions; > > I’d suggest to stop support Spark 2 for the Spark Runner in the one of the > next Beam releases. > > What are your thoughts on this? Are there any principal objections or reasons > for not doing this that I probably missed? > > — > Alexey > >
