Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/6397#issuecomment-106548637
Actually, I spoke slightly too soon: it looks like we _do_ call
`commitAndClose()` in `writePartitionedFile()`, but then it looks like we turn
around and call `revertPartialWritesAndClose()` in `stop()` on a
DiskBlockObjectWriter that's already closed. It looks like this ends up
rolling back the shuffle metrics for the block that we've already committed.
We probably shouldn't be calling `revertPartialWritesAndClose()` here anyways,
but I think that BlockObjectWriter should guard against calling revert on a
writer that's already been closed by making that operation into a no-op if the
writer is already closed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
