Qiegang Long created SPARK-58089:
------------------------------------
Summary: Variant extraction pushdown does not work for Aggregate,
Sort, and Join
Key: SPARK-58089
URL: https://issues.apache.org/jira/browse/SPARK-58089
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.1.2
Reporter: Qiegang Long
While running performance test, noticed that variant extraction pushdown does
does not extend to aggregate and join, losing performance benefits of shreddred
columns.
Extaction pushdown not working for these examples:
Example 1:
{code:sql}
SELECT AVG(variant_get(data, '$.qty', 'double'))
FROM store_sales
GROUP BY variant_get(data, '$.id', 'string')
{code}
Example 2:
{code:sql}
SELECT AVG(variant_get(ss.data, '$.qty', 'double'))
FROM store_sales ss
JOIN date_dim d ON ss.date_sk = d.date_sk
JOIN store st ON ss.store_sk = st.store_sk
GROUP BY variant_get(ss.data, '$.id', 'string')
{code}
Example 3:
{code:sql}
SELECT ss.k
FROM store_sales ss
JOIN date_dim d ON ss.date_sk = variant_get(d.data, '$.sk', 'int')
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
