[
https://issues.apache.org/jira/browse/IMPALA-9228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17052155#comment-17052155
]
ASF subversion and git services commented on IMPALA-9228:
---------------------------------------------------------
Commit 0b081aef3fc681b40c9cc45e0387bf7dd84358a9 in impala's branch
refs/heads/master from Gabor Kaszab
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=0b081ae ]
IMPALA-6506: Codegen in ORC scanner for primitives and struct
IMPALA-9228 introduced scratch batch handling for struct and
primitive types in the ORC scanner and the existing scratch batch
logic already supports Codegen for ProcessScratchBatch() function.
This change turns on this Codegen logic for primitives types and
structs in the ORC scanner.
Note, if the query involves collection types then
ProcessScratchBatch() is still codegend but the codegend function
isn't used as the regular row-by-row approach is followed in this
case without using a scratch batch.
Testing:
- Re-run the whole test suite to check for regressions.
- Checked the performance on a scale 25 TPCH workload in ORC format
using single_node_perf_run.py. Comparing the query runtimes it
seems that codegen brings a 1-21% improvement for most of the
queries. There is a slight decrease in 3 queries that are not
scan-heavy where codegen doesn't provide any help for scanning.
However, these are short queries where the size of the
degradation is in subseconds so I'd say the decrease is
negligible.
- Did a manual check for a table that contains both Parquet and ORC
partitions. Verified that in this case ProcessScratchBatch() is
codegend for both formats and the query results are as expected.
Change-Id: I2352d0c8fc75ff722e931bc8c866b3e43d3636f4
Reviewed-on: http://gerrit.cloudera.org:8080/15350
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> ORC scanner could be vectorized
> -------------------------------
>
> Key: IMPALA-9228
> URL: https://issues.apache.org/jira/browse/IMPALA-9228
> Project: IMPALA
> Issue Type: Improvement
> Reporter: Zoltán Borók-Nagy
> Assignee: Gabor Kaszab
> Priority: Major
> Labels: orc
> Fix For: Impala 4.0
>
> Attachments: 1-4_col_measurement_int_only.png
>
>
> The ORC scanners uses an external library to read ORC files. The library
> reads the file contents into its own memory representation. It is a
> vectorized representation similar to the Arrow format.
> Impala needs to convert the ORC row batch to an Impala row batch. Currently
> the conversion happens row-wise via virtual function calls:
> [https://github.com/apache/impala/blob/85425b81f04c856d7d5ec375242303f78ec7964e/be/src/exec/hdfs-orc-scanner.cc#L671]
> [https://github.com/apache/impala/blob/85425b81f04c856d7d5ec375242303f78ec7964e/be/src/exec/orc-column-readers.cc#L352]
> Instead of this approach it could work similarly to the Parquet scanner that
> fills the columns one-by-one into a scratch batch, then evaluate the
> conjuncts on the scratch batch. For more details see
> HdfsParquetScanner::AssembleRows():
> [https://github.com/apache/impala/blob/85425b81f04c856d7d5ec375242303f78ec7964e/be/src/exec/parquet/hdfs-parquet-scanner.cc#L1077-L1088]
> This way we'll need a lot less virtual function calls, also the memory
> reads/writes will be much more localized and predictable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
