GitHub user omalley opened a pull request:
https://github.com/apache/orc/pull/259
ORC-248. PhysicalFsWriter sometimes passes negative padding down to shims
The PhysicalFsWriter sometimes pass down negative sizes for padding, which
leads to exceptions.
My fix:
* Add a new option to enable writing variable length HDFS blocks
"orc.write.shortened.blocks", which is off by default.
* Modified the shim to shortenBlock so that the PhysicalFsWriter can track
the offsets.
* shortenBlock isn't gated by the padding tolerance, although it is called
when the new stripe would cross a block boundary.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/omalley/orc orc-248
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/orc/pull/259.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #259
----
commit 7a8107b9f6d954ce3d4bddd5245642f58b6bb729
Author: Owen O'Malley <omalley@...>
Date: 2018-04-27T23:58:48Z
ORC-354. Restore the benchmark module.
This reverts commit b86d70aa73289b86e066cc019ea11e0d83c1e40d.
commit 1ee460dd750dace3bee34131ebd2e0eb260bedfc
Author: Owen O'Malley <omalley@...>
Date: 2018-04-30T21:10:07Z
ORC-248. PhysicalFsWriter will sometimes pass a negative amount of
requested padding to the shims.
----
---
