I was just thinking an info email (perhaps tagged with correctness/dataloss)
to dev rather than an official vote, that way its more visible and if anyone
sees it and disagrees with the targeting it can be discussed on that thread.
It might also just bring more visibility to those important issues and get
people interesting in working on them sooner.
Tom
On Monday, January 27, 2020, 02:31:03 PM CST, Dongjoon Hyun
<dongjoon.h...@gmail.com> wrote:
Yes. That is what I pointed in `Unfortunately, we didn't build a consensus on
what is really blocked by that.` If you are suggesting a vote, do you mean a
majority-win vote or an unanimous decision? Will it be a permanent decision?
> I think the other interesting thing here is how exactly to come to agreement
> on whether it needs to be fixed in a particular release. Like we have been
> discussing on SPARK-29701. This could be a matter of opinion, so should we do
> something like mail the dev list whenever one of these issues is tagged if
> its not going to be back ported to an affected release?
The following seems to happen when the committers initially think like "Seems
behavioral to me and its been consistent so seems ok to skip for 2.4.5"
For example, SPARK-27619 MapType should be prohibited in hash expressions.
> A) I'm not clear on this one as to why affected and target would be different
> initially,
BTW, in this email thread, I'm focusing on the `Target Version` management.That
is the only way to detect the community decision change.
Bests,Dongjoon.
On Mon, Jan 27, 2020 at 11:12 AM Tom Graves <tgraves...@yahoo.com> wrote:
thanks for bringing this up.
A) I'm not clear on this one as to why affected and target would be different
initially, other then the reasons target versions != fixed versions. Is the
intention here just to say, if its already been discussed and came to consensus
not needed in certain release? The only other obvious time is in spark
releases that are no longer maintained.
I think the other interesting thing here is how exactly to come to agreement on
whether it needs to be fixed in a particular release. Like we have been
discussing on SPARK-29701. This could be a matter of opinion, so should we do
something like mail the dev list whenever one of these issues is tagged if its
not going to be back ported to an affected release?
Tom On Sunday, January 26, 2020, 11:22:13 PM CST, Dongjoon Hyun
<dongjoon.h...@gmail.com> wrote:
Hi, All.
After 2.4.5 RC1 vote failure, I asked your opinions about correctness/dataloss
issues (at mailing lists/JIRAs/PRs) in order to collect the current status and
public opinion widely in the community to build a consensus on this at this
time.
Before talking about those issues, please remind that
- Apache Spark 2.4.x is the only live version because 2.3.x is EOL and
3.0.0 is not released. - Apache Spark community has the following rule:
"Correctness and data loss issues should be considered Blockers."
Unfortunately, we didn't build a consensus on what is really blocked by that.
In reality, it was just our resolution for the quality and it works a little
differently.
In this email, I want to talk about correctness/dataloss issues and observed
public opinions. They fall into the following categories roughly.
1. Resolved in both 3.0.0 and 2.4.x - ex) SPARK-30447 Constant propagation
nullability issue - No problem. However, this case sometimes goes to (2)
2. Resolved in both 3.0.0 and 2.4.x. But, reverted in 2.4.x later. - ex)
SPARK-26021 -0.0 and 0.0 not treated consistently, doesn't match Hive - "We
don't want to change the behavior in the maintenence release"
3. Resolved in 3.0.0 and not backported because this is 3.0.0-specific. - ex)
SPARK-29906 Reading of csv file fails with adaptive execution turned on - No
problem.
4. Resolved in 3.0.0 and not backported due to technical difficulty. - ex)
SPARK-26154 Stream-stream joins - left outer join gives inconsistent output -
"This is not backported due to the technical difficulty"
5. Resolved in 3.0.0 and not backported because this is not public API. - ex)
SPARK-29503 MapObjects doesn't copy Unsafe data when nested under Safe data -
"Since `catalyst` is not public, it's less worth backporting this."
6. Resolved in 3.0.0 and not backported because we forget since there was a no
Target Version. - ex) SPARK-28375 Make pullupCorrelatedPredicate idempotent
- "Adding the 'correctness' label so we remember to backport this fix to
2.4.x." - "This is possible, if users add the rule into
postHocOptimizationBatches"
7. Open with Target Version 3.0.0. - ex) SPARK-29701 Correct behaviours of
group analytical queries when empty input given - "We aren't fully SQL
compliant there and I think that has been true since the beginning of spark
sql" - "This is not a regression"
8. Open without Target Version. - I removed this case last week to give more
visibility on them.
Here, I want to focus that Apache Spark is a very healthy community because we
have diverse opinions and reevaluating JIRA issues are the results of the
community decision based on the discusson. I believe that it will go well
eventually. In the above, I added those example JIRA IDs and the collected
reasons just to give some colors to illustrate all cases are the real cases.
There is no case to be blamed in the above.
Although some JIRA issues will jump from one category into another category
time to time, the categories will remain there. I want to propose a small
additional work on `Target Version` to distinguish the above categories easily
to communicate clearly in the community. This should be done by committers
because we have the following policy on `Target Version`.
"Target Version. This is assigned by committers to indicate a PR has been
accepted for possible fix by the target version."
Proposed Idea: A. To reduce the mismatch between `Target Version` vs
`Affected Version`: When a committer set `correctness` or `data-loss`
label, `Target Version` should be set together according to the `Affected
Versions`. In case of the insufficient `Target Version` (e.g. `Target
Version`=`3.0.0` for `Affected Version`=`2.4.4,3.0.0`), he/she need to add a
comment on the JIRA. For example, "This is 3.0.0-specific issue"
B. To reduce the mismatch between `Target Version` vs `Fixed Version`:
When a committer resolve `correctness` or `data-loss` labeled issue, `Target
Version` should be compared with `Fixed Version`. In case of the
insufficient `Fixed Version` (e.g. `Target Version`=`2.4.4,3.0.0` and `Fixed
Version`=`3.0.0`), he/she need to add a comment on the JIRA and adjust `Target
Version` according to his/her decision. For example, "This is not
backported due to the technical difficulty. I'll remove `2.4.4` from `Target
Version`."
With the above rules, the combination of `Affected Version` / `Target Version`
/ `Fixed Version` will serve us with much easier way in searching them,
understanding categories, and discussing how to handle properly.
Bests,Dongjoon.
