One thing I can correct immediately is, downstream does not have any impact at all from this. I believe I clarified that the config will not be modified by anyone, so downstream there is nothing to change. The problem is particular in OSS, downstream does not have any issue with this leak at all. (One thing to clarify, config itself will be removed in Spark 4.0.0. We only propose to retain migration logic.)
I believe there is a huge misunderstanding - we are not proposing this migration logic to ease the vendor's life, no it's not. If I don't care about OSS, there is no incentive for me to propose this. I just wanted to do my best to remove any burden to users who are innocent with this problem. If there is no migration logic, users will be enforced to upgrade to Spark 3.5.5+ before upgrading to Spark 4.0.0+. Isn't it bad enough? Why do we have to let users be bugging while we can avoid it? The problem was introduced in the OSS community (I hope we don't blame ourselves with mistakes. We are human.) so it is up to us to resolve this properly. We don't have the right to break users' queries. On Tue, Mar 11, 2025 at 7:13 AM Andrew Melo <andrew.m...@gmail.com> wrote: > Hello all > > As an outsider, I don't fully understand this discussion. This > particular configuration option "leaked" into the open-source Spark > distribution, and now there is a lot of discussion about how to > mitigate existing workloads. But: presumably the people who are > depending on this configuration flag are already using a downstream > (vendor-specific) fork, and a future update will similarly be > distributed by that downstream provider. > > Which people a) made a workflow using the vendor fork and b) want to > resume it in the OSS version of spark? > > It seems like the people who are affected by this will already be > using someone else's fork, and there's no need to carry this patch in > the mainline Spark code. > > For that reason, I believe the code should be dropped by OSS Spark, > and vendors who need to mitigate it can push the appropriate changes > to their downstreams. > > Thanks > Andrew >
