[
https://issues.apache.org/jira/browse/IMPALA-9228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17023032#comment-17023032
]
Gabor Kaszab commented on IMPALA-9228:
--------------------------------------
I've implemented a very basic Poc that covers OrcIntColumnReader only just to
see if this approach brings any improvements to the table
([https://gerrit.cloudera.org/#/c/15104/).]
*Perf. measurement details:*
- Compared the release build of Impala with this change and without.
- MT_DOP=1, NUM_NODES=1
- Ran a select avg(...) from lineitem; query on TPCH scale 25.
- Tests cover 1 column and 4 columns result set as well.
- For comparison I used 3 metrics:
** Runtime of the query
** Time spent on scan node
** MaterializeTupleTime metric
*Results/observations:*
- I ran this first on debug builds and that showed a performance degradation
with this change. Running on a release build shows a different result with the
performance actually increasing with this change.
- When only one column is queried all of the measurement numbers show a 30+%
decrease in runtime.
- When 4 columns are queried the runtime decrease is less than above but it's
still an improvement by 10-12%.
- Worth trying to run this test with 15-20 columns to make sure that
performance is not dropping further when the number of queried columns increase.
- Measurement numbers:
!1-4_col_measurement_int_only.png!
> ORC scanner could be vectorized
> -------------------------------
>
> Key: IMPALA-9228
> URL: https://issues.apache.org/jira/browse/IMPALA-9228
> Project: IMPALA
> Issue Type: Improvement
> Reporter: Zoltán Borók-Nagy
> Assignee: Gabor Kaszab
> Priority: Major
> Labels: orc
> Attachments: 1-4_col_measurement_int_only.png
>
>
> The ORC scanners uses an external library to read ORC files. The library
> reads the file contents into its own memory representation. It is a
> vectorized representation similar to the Arrow format.
> Impala needs to convert the ORC row batch to an Impala row batch. Currently
> the conversion happens row-wise via virtual function calls:
> [https://github.com/apache/impala/blob/85425b81f04c856d7d5ec375242303f78ec7964e/be/src/exec/hdfs-orc-scanner.cc#L671]
> [https://github.com/apache/impala/blob/85425b81f04c856d7d5ec375242303f78ec7964e/be/src/exec/orc-column-readers.cc#L352]
> Instead of this approach it could work similarly to the Parquet scanner that
> fills the columns one-by-one into a scratch batch, then evaluate the
> conjuncts on the scratch batch. For more details see
> HdfsParquetScanner::AssembleRows():
> [https://github.com/apache/impala/blob/85425b81f04c856d7d5ec375242303f78ec7964e/be/src/exec/parquet/hdfs-parquet-scanner.cc#L1077-L1088]
> This way we'll need a lot less virtual function calls, also the memory
> reads/writes will be much more localized and predictable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
