[
https://issues.apache.org/jira/browse/IMPALA-9228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17024188#comment-17024188
]
Gabor Kaszab commented on IMPALA-9228:
--------------------------------------
I did another test for a modified lineitem table with 20 int cols (scale factor
is again 25). Querying avg() for all the int cols still shows an increase in
performance with this PoC change. The increase is around 7.5% - 8.5% when
comparing the same metrics as above.
> ORC scanner could be vectorized
> -------------------------------
>
> Key: IMPALA-9228
> URL: https://issues.apache.org/jira/browse/IMPALA-9228
> Project: IMPALA
> Issue Type: Improvement
> Reporter: Zoltán Borók-Nagy
> Assignee: Gabor Kaszab
> Priority: Major
> Labels: orc
> Attachments: 1-4_col_measurement_int_only.png
>
>
> The ORC scanners uses an external library to read ORC files. The library
> reads the file contents into its own memory representation. It is a
> vectorized representation similar to the Arrow format.
> Impala needs to convert the ORC row batch to an Impala row batch. Currently
> the conversion happens row-wise via virtual function calls:
> [https://github.com/apache/impala/blob/85425b81f04c856d7d5ec375242303f78ec7964e/be/src/exec/hdfs-orc-scanner.cc#L671]
> [https://github.com/apache/impala/blob/85425b81f04c856d7d5ec375242303f78ec7964e/be/src/exec/orc-column-readers.cc#L352]
> Instead of this approach it could work similarly to the Parquet scanner that
> fills the columns one-by-one into a scratch batch, then evaluate the
> conjuncts on the scratch batch. For more details see
> HdfsParquetScanner::AssembleRows():
> [https://github.com/apache/impala/blob/85425b81f04c856d7d5ec375242303f78ec7964e/be/src/exec/parquet/hdfs-parquet-scanner.cc#L1077-L1088]
> This way we'll need a lot less virtual function calls, also the memory
> reads/writes will be much more localized and predictable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
