For Spark 3.5.5, did you see this which is the best the community offer? https://github.com/apache/spark/pull/49985 [SPARK-51187][SQL][SS][3.5] Implement the graceful deprecation of incorrect config introduced in SPARK-49699
Dongjoon On Mon, Feb 17, 2025 at 14:38 Bjørn Jørgensen <bjornjorgen...@gmail.com> wrote: > > Having breaking changes in a minor seems not that good.. As I'm reading > this, > > "*This could break the query if the rule impacts the query, because the > effectiveness of the fix is flipped.*" > https://github.com/apache/spark/pull/49897#issuecomment-2652567140 > > > What if we have this https://github.com/apache/spark/pull/48149 change in > the branch and remove it only for version 4? That way we dont break > anything. > > > > > man. 17. feb. 2025 kl. 23:03 skrev Dongjoon Hyun <dongjoon.h...@gmail.com > >: > >> Hi, All. >> >> I'd like to highlight this discussion because this is more important and >> tricky in a way. >> >> As already mentioned in the mailing list and PRs, there was an obvious >> mistake >> which missed an improper configuration name, `spark.databricks.*`. >> >> >> https://github.com/apache/spark/blob/a6f220d951742f4074b37772485ee0ec7a774e7d/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L3424 >> >> `spark.databricks.sql.optimizer.pruneFiltersCanPruneStreamingSubplan` >> >> In fact, Apache Spark committers have been preventing this repetitive >> mistake >> pattern during the review stages successfully until we slip the following >> backportings >> at Apache Spark 3.5.4. >> >> https://github.com/apache/spark/pull/45649 >> https://github.com/apache/spark/pull/48149 >> https://github.com/apache/spark/pull/49121 >> >> At this point of writing, `spark.databricks.*` was removed successfully >> from `master` >> and `branch-4.0` and a new ScalaStyle rule was added to protect Apache >> Spark repository >> from future mistakes. >> >> SPARK-51172 Rename to >> spark.sql.optimizer.pruneFiltersCanPruneStreamingSubplan >> SPARK-51173 Add `configName` Scalastyle rule >> >> What I proposed is to release Apache Spark 3.5.5 next week with the >> deprecation >> in order to make Apache Spark 4.0 be free of `spark.databricks.*` >> configuration. >> >> Apache Spark 3.5.5 (2025 February, with deprecation warning with >> alternative) >> Apache Spark 4.0.0 (2025 March, without `spark.databricks.*` config) >> >> In addition, I'd like to volunteer as a release manager of Apache Spark >> 3.5.5 >> for a swift release. WDYT? >> >> FYI, `branch-3.5` has 37 patches currently. >> >> $ git log --oneline v3.5.4..HEAD | wc -l >> 37 >> >> Best Regards, >> Dongjoon. >> > > > -- > Bjørn Jørgensen > Vestre Aspehaug 4, 6010 Ålesund > <https://www.google.com/maps/search/Vestre+Aspehaug+4,+6010+%C3%85lesund++Norge?entry=gmail&source=g> > Norge > <https://www.google.com/maps/search/Vestre+Aspehaug+4,+6010+%C3%85lesund++Norge?entry=gmail&source=g> > > +47 480 94 297 >
