[
https://issues.apache.org/jira/browse/HIVE-30048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-30048:
--------------------------------
Description: When MapredParquetInputFormat is listed in
hive.llap.io.encode.formats, SerDeEncodedDataReader builds the source
RecordReader from the caller's vectorized JobConf.
MapredParquetInputFormat.getRecordReader then returns a
VectorizedParquetRecordReader that fills a whole VectorizedRowBatch (up to 1024
rows) per next(), but the encode loop (PassThruOffsetReader →
DeserializerOrcWriter#writeOneRow(Writable)) treats each next() value as a
single row. All but the first row of every batch is silently dropped, so SELECT
queries served from LLAP cache return only ceil(rows / 1024) rows.
> Correctness: LLAP IO SerDe encode path drops rows for MapredParquetInputFormat
> ------------------------------------------------------------------------------
>
> Key: HIVE-30048
> URL: https://issues.apache.org/jira/browse/HIVE-30048
> Project: Hive
> Issue Type: Bug
> Reporter: László Bodor
> Assignee: László Bodor
> Priority: Major
>
> When MapredParquetInputFormat is listed in hive.llap.io.encode.formats,
> SerDeEncodedDataReader builds the source RecordReader from the caller's
> vectorized JobConf. MapredParquetInputFormat.getRecordReader then returns a
> VectorizedParquetRecordReader that fills a whole VectorizedRowBatch (up to
> 1024 rows) per next(), but the encode loop (PassThruOffsetReader →
> DeserializerOrcWriter#writeOneRow(Writable)) treats each next() value as a
> single row. All but the first row of every batch is silently dropped, so
> SELECT queries served from LLAP cache return only ceil(rows / 1024) rows.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
