SEPURI-SAI-KRISHNA commented on PR #59583:
URL: https://github.com/apache/spark/pull/59583#issuecomment-6281878966
Thanks for taking a look! They have the same root cause, a float/double
value used as a key without the normalization that grouping applies, but they
fix different code paths and do not overlap:
- #59403 is in the optimizer: a filter pushed below an `Aggregate` reads the
raw grouping key, while the `Aggregate` outputs the normalized one.
- This PR is in the file writer: dynamic partition values are not
normalized, so 0.0 and -0.0 in one task make `DynamicPartitionDataSingleWriter`
reopen `p=0.0` and fail with `FileAlreadyExistsException`.
They touch no common files, and neither fixes the other's repro. The failing
write here has no `Aggregate` (`df.coalesce(1).write.partitionBy("p")`), and
#59403 does not change writes. Both use `NormalizeFloatingNumbers.normalize`,
which #59403 leaves as it is, so they can be merged in either order.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
