[
https://issues.apache.org/jira/browse/PARQUET-82?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14116047#comment-14116047
]
Ryan Blue commented on PARQUET-82:
----------------------------------
Pull request ready for review:
[#48|https://github.com/apache/incubator-parquet-mr/pull/48]
> ColumnChunkPageWriteStore assumes pages are smaller than Integer.MAX_VALUE
> --------------------------------------------------------------------------
>
> Key: PARQUET-82
> URL: https://issues.apache.org/jira/browse/PARQUET-82
> Project: Parquet
> Issue Type: Bug
> Components: parquet-mr
> Reporter: Ryan Blue
> Assignee: Ryan Blue
>
> The ColumnChunkPageWriteStore casts both the compressed size and uncompressed
> size of a page from a long to an int. If the uncompressed size of a page
> exceeds Integer.MAX_VALUE, the write doesn't fail, although it creates bad
> metadata:
> {code}
> chunk1: BINARY GZIP DO:0 FPO:4 SZ:267184096/-2143335445/-8.02 VC:41
> ENC:BIT_PACKED,PLAIN
> {code}
> At read time, the BytesInput will try to [allocate a byte
> array|https://github.com/apache/incubator-parquet-mr/blob/master/parquet-encoding/src/main/java/parquet/bytes/BytesInput.java#L200]
> for the uncompressed data and fails:
> {code}
> Caused by: parquet.io.ParquetDecodingException: Can not read value at 0 in
> block -1 in file hdfs://nameservice1/OUTPUT/part-m-00000.gz.parquet
> at
> parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:177)
>
> at
> parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:130)
> at
> org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.<init>(ParquetRecordReaderWrapper.java:95)
>
> at
> org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.<init>(ParquetRecordReaderWrapper.java:66)
>
> at
> org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:51)
>
> at
> org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.<init>(CombineHiveRecordReader.java:65)
>
> ... 16 more
> Caused by: java.lang.NegativeArraySizeException
> at parquet.bytes.BytesInput$StreamBytesInput.toByteArray(BytesInput.java:183)
> at parquet.column.impl.ColumnReaderImpl.readPage(ColumnReaderImpl.java:521)
> at parquet.column.impl.ColumnReaderImpl.checkRead(ColumnReaderImpl.java:493)
> at parquet.column.impl.ColumnReaderImpl.consume(ColumnReaderImpl.java:544)
> at parquet.column.impl.ColumnReaderImpl.<init>(ColumnReaderImpl.java:339)
> at
> parquet.column.impl.ColumnReadStoreImpl.newMemColumnReader(ColumnReadStoreImpl.java:63)
>
> at
> parquet.column.impl.ColumnReadStoreImpl.getColumnReader(ColumnReadStoreImpl.java:58)
>
> at
> parquet.io.RecordReaderImplementation.<init>(RecordReaderImplementation.java:265)
>
> at parquet.io.MessageColumnIO.getRecordReader(MessageColumnIO.java:59)
> at parquet.io.MessageColumnIO.getRecordReader(MessageColumnIO.java:73)
> at
> parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:110)
>
> at
> parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:172)
>
> ... 21 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
