Asif created SPARK-45926:
----------------------------
Summary: The InMemoryV2FilterBatchScan and InMemoryBatchScan are
not implementing equals and hashCode correctly
Key: SPARK-45926
URL: https://issues.apache.org/jira/browse/SPARK-45926
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.5.1
Reporter: Asif
These InMemoryV2FilterBatchScan and InMemoryBatchScan test classes are not
implementing hashCode and equals correctly as they are not taking into account
the pushed runtime filters. As a result they are unable to expose the TPCDS
test issues which can show whether the reuse of exchange is happening correctly
or not.
If these classes implement equals and hashCode taking into account the pushed
runtime filters, we would see that TPCDS Q14b which should ideally be reusing
the exchange containing Union , is not happening due to multiple bugs which
surface in AQE.
Actual V2 DataSources like iceberg correctly implement equals and hashCode
taking into account pushed runtime filters , which also expose the same issue
of reuse of exchnage not happening
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
