Patrick Woody created SPARK-23819:
-------------------------------------
Summary: InMemoryTableScanExec prunes orderable complex types due
to out of date ColumnStats
Key: SPARK-23819
URL: https://issues.apache.org/jira/browse/SPARK-23819
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.3.0
Reporter: Patrick Woody
The data types that can be compared via BinaryComparison was expanded in
SPARK-21110 now include Arrays/Structs/etc, but ColumnStats would still have
hard coded upper/lower bounds for these types.
InMemoryTableScanExec used to be safe against these comparisons because the
predicate would fail type checking. Now that it passes, the statistics
unintentionally allow pruning of the partition, causing correctness issues.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
