[
https://issues.apache.org/jira/browse/ORC-248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16459173#comment-16459173
]
ASF GitHub Bot commented on ORC-248:
------------------------------------
GitHub user omalley opened a pull request:
https://github.com/apache/orc/pull/259
ORC-248. PhysicalFsWriter sometimes passes negative padding down to shims
The PhysicalFsWriter sometimes pass down negative sizes for padding, which
leads to exceptions.
My fix:
* Add a new option to enable writing variable length HDFS blocks
"orc.write.shortened.blocks", which is off by default.
* Modified the shim to shortenBlock so that the PhysicalFsWriter can track
the offsets.
* shortenBlock isn't gated by the padding tolerance, although it is called
when the new stripe would cross a block boundary.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/omalley/orc orc-248
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/orc/pull/259.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #259
----
commit 7a8107b9f6d954ce3d4bddd5245642f58b6bb729
Author: Owen O'Malley <omalley@...>
Date: 2018-04-27T23:58:48Z
ORC-354. Restore the benchmark module.
This reverts commit b86d70aa73289b86e066cc019ea11e0d83c1e40d.
commit 1ee460dd750dace3bee34131ebd2e0eb260bedfc
Author: Owen O'Malley <omalley@...>
Date: 2018-04-30T21:10:07Z
ORC-248. PhysicalFsWriter will sometimes pass a negative amount of
requested padding to the shims.
----
> The new padStream method can throw a NegativeArraySizeException.
> ----------------------------------------------------------------
>
> Key: ORC-248
> URL: https://issues.apache.org/jira/browse/ORC-248
> Project: ORC
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 1.5.0
>
>
> {code:java}
> Exception in thread "main" java.lang.NegativeArraySizeException
> at
> org.apache.orc.impl.HadoopShimsPre2_3.padStream(HadoopShimsPre2_3.java:51)
> at
> org.apache.orc.impl.HadoopShimsCurrent.padStreamToBlock(HadoopShimsCurrent.java:52)
> at
> org.apache.orc.impl.PhysicalFsWriter.padStripe(PhysicalFsWriter.java:140)
> at
> org.apache.orc.impl.PhysicalFsWriter.finalizeStripe(PhysicalFsWriter.java:341)
> at org.apache.orc.impl.WriterImpl.flushStripe(WriterImpl.java:438)
> at org.apache.orc.impl.WriterImpl.checkMemory(WriterImpl.java:262)
> at
> org.apache.orc.impl.MemoryManagerImpl.notifyWriters(MemoryManagerImpl.java:190)
> at
> org.apache.orc.impl.MemoryManagerImpl.addedRow(MemoryManagerImpl.java:178)
> at org.apache.orc.impl.WriterImpl.addRowBatch(WriterImpl.java:539)
> at
> org.apache.orc.bench.convert.orc.OrcWriter.writeBatch(OrcWriter.java:48)
> at
> org.apache.orc.bench.convert.GenerateVariants.main(GenerateVariants.java:211)
> at org.apache.orc.bench.Driver.main(Driver.java:62)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
