abstractdog opened a new pull request, #6780:
URL: https://github.com/apache/hive/pull/6780
### What changes were proposed in this pull request?
Pass a `JobConf` clone with vectorization disabled
(`HIVE_VECTORIZATION_ENABLED` and `Utilities.VECTOR_MODE` both `false`) to
`sourceInputFormat.getRecordReader(...)` in `SerDeEncodedDataReader`, so
`MapredParquetInputFormat` picks its row-per-`next()` branch. Added class
Javadocs to `SerDeEncodedDataReader`, `PassThruOffsetReader`, and
`LineRrOffsetReader`.
### Why are the changes needed?
When `MapredParquetInputFormat` is listed in `hive.llap.io.encode.formats`,
the encode path handed it a vectorized `JobConf`. It returned a
`VectorizedParquetRecordReader` that fills a whole `VectorizedRowBatch` (up to
1024 rows) per `next()`, but the encode loop (`PassThruOffsetReader` →
`DeserializerOrcWriter#writeOneRow(Writable)`) treats each `next()` value as
one row. Only the first row of every batch reached the LLAP cache; `SELECT`s
returned `ceil(rows / 1024)` rows.
### Does this PR introduce _any_ user-facing change?
No behavior change in configuration or SQL. Query results served from LLAP
cache for Parquet tables listed under `hive.llap.io.encode.formats` are now
correct — the previous behavior returned far fewer rows than the underlying
file.
### How was this patch tested?
Two new unit tests in `llap-server`:
- **`TestSerDeEncodedDataReader`** — pins the `buildSourceReaderJobConf`
helper contract (clone has vectorization disabled, caller's `JobConf` not
mutated, unrelated entries preserved, `VECTOR_MODE` short-circuit alone flips
`Utilities.getIsVectorized`).
- **`TestSerDeEncodedDataReaderParquetE2E`** — writes a real 2500-row
Parquet file (> 2× `VectorizedRowBatch.DEFAULT_SIZE`) and drives it through
`MapredParquetInputFormat.getRecordReader(...)` wrapped in
`PassThruOffsetReader`. Asserts:
1. all 2500 rows reach the wrapper with the source-reader `JobConf`, and
2. the vectorized branch is batch-shaped (3 iterations for 2500 rows),
documenting why the helper exists.
Verified the E2E test fails without the fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
