The problem you are mentioning is arguably resolvable if we leave a config
name as it is, and add
withAlternative("spark.databricks.sql.optimizer.pruneFiltersCanPruneStreamingSubplan").
Let's not nitpick just from reverting the PR. We have to revert the PR
"semantically".Btw, from what I understand of dealing with backport PR is, we mostly merge in master to begin with, and down the version line. When we handle my migration PR, we do not follow this practice without any discussion. I submitted PRs for master/4.0/3.5, and 3.5 was merged "first", and when I asked for merging to 4.0, I was pushed back. I don't think this is an ordinary practice of dealing with multiple versions of PRs, especially since the author never agreed with the way of processing. If we were following the practice, we should already have migration logic for master/4.0/3.5, since we should have the same fix in master/4.0. We probably had a discussion about removing config from master/4.0 based on the discussion, and we probably agreed to remove the config since we still have a migration logic. W.r.t migration logic, based on the discussion we are having, we probably can't make an agreement to take it out, then arguably the migration logic is left as it is. This way I never needed to drive such a long and sensitive DISCUSSION and VOTE. But the PR for master and 4.0 weren't merged because of the individual's belief of the rollout plan, which we are seeing does not fit a majority of voices now. Shall we fix the broken process in which we made a huge mistake before moving on? We should have merged the same content in 3.5 to master/4.0 as well, and then have a PR to remove the config. This is totally swapped which does not make sense to me. On Mon, Mar 17, 2025 at 1:57 PM Dongjoon Hyun <dongj...@apache.org> wrote: > I reviewed Wenchen's reverting PR. Although it's a proposal for > discussion, it is another breaking change against Apache Spark 3.5.5, isn't > it? If we consider Apache Spark 3.5.4 users, I believe we need to consider > Apache Spark 3.5.5 users too which uses > `spark.sql.optimizer.pruneFiltersCanPruneStreamingSubplan` already. > > https://github.com/apache/spark/pull/50291 > > - buildConf("spark.sql.optimizer.pruneFiltersCanPruneStreamingSubplan") > + > buildConf("spark.databricks.sql.optimizer.pruneFiltersCanPruneStreamingSubplan") > > Dongjoon. > > On 2025/03/17 02:36:42 Wenchen Fan wrote: > > I've created the revert PR for branch-4.0: > > https://github.com/apache/spark/pull/50291 . We can merge PRs with lazy > > consensus but it's clear that this breaking change PR has failed to > achieve > > consensus. > > > > I hope we now have a clear foundation for discussing solutions. As it > > stands, the misnamed configuration will be released in 4.0.0. I like > > Jungtaek’s proposal to deprecate it, but the decision is up to the > > community. > > --------------------------------------------------------------------- > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >
