Aleksandr Efimov created IMPALA-15303:
-----------------------------------------
Summary: Calcite planner: cover the TPC-DS corpus in the Substrait
export tests
Key: IMPALA-15303
URL: https://issues.apache.org/jira/browse/IMPALA-15303
Project: IMPALA
Issue Type: Sub-task
Components: Frontend
Reporter: Aleksandr Efimov
The exporter is covered by query shapes written by hand, one construct at a
time. Every gap it turned out to have was found by running a real workload
instead, and the numbers those runs produced have been quoted in IMPALA-15297
and after without anything in the tree that reproduces them.
Run the exporter over the TPC-DS queries that carry Calcite goldens, as a test.
The tables come from the schema template rather than from the catalogue, so
this runs wherever the frontend tests run and needs no loaded warehouse. Only
column types matter for an export, so no data or statistics are involved, which
also means the plans are not the ones a loaded warehouse would produce.
The expectation is a named list rather than a count: which queries do not
export and what stops each. A count would only say that the number moved.
Removing the abs mapping, for instance, fails the test with tpcds-q47, q53,
q57, q63 and q89 named against ABS.
Today 102 of 110 export. The eight that do not are outside this exporter: four
stop on SINGLE_VALUE, which Calcite uses to guard a scalar subquery and Impala
turns into a runtime cardinality check that Substrait has no way to express,
and four on GROUPING, which is substrait-java#993. A query leaving that list
matters as much as one joining it, so the test reports both directions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
