LuciferYang commented on pull request #33556: URL: https://github.com/apache/spark/pull/33556#issuecomment-893304021
On the other hand, I found `DiskBlockObjectWriter` use `objOut` to write kv, `objOut` will `close()` after `commitAndGet()` and reassign in next call of `write()`, but `mcs(ManualCloseOutputStream is defined by us)` can still be written before call `closeResources()`. So the potential extra content writing may occur every time `commitAndGet()` is called(by `objOut.close()`)? If so, it seems that potential additional data will be tracked by each `FileSegment`. If the potential extra content writing occur in `closeResources()`, it seems only possible to write some data to `mcs` manually before call `mcs.close()`, if the we write extra content manually, shouldn't we truncate it... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
