Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/6397#issuecomment-106697486
Hmm, looks like this fails nearly all of the HashShuffleSuite tests. I
found the problem: after 8b8fb9e3bba2944ae1967a9a532ab39ff19a7deb, the
`finalPosition` in DiskBlockObjectWriter was no longer being set when calling
`commitAndClose()` on writers that hadn't ever been opened. This caused
`fileSegment()` to return a segment with a negative length, causing problems
during shuffle fetching / serving. I've fixed this problem and added
`require`s to `FileSegment` to guard against negative offsets and lengths.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
