Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/17268
I'm mixed if we want this to happen implicitly. Here's how I think about
the tradeoffs for this change: On the pro side, with this change we avoid the
case where the user forgets to include the timestamp and their application
crashes.
On the con, side we are doing something implicit here that might result in
confusing behavior. In the worst case (where the timestamp is not
deterministically correlated with the other deduplication columns), it is
possible that the result actually ends up with duplicates in it.
For that reason, I think its safest to require the user to explicitly
include the timestamp. I'd be curious what others think though.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
