andygrove commented on issue #1839:
URL:
https://github.com/apache/datafusion-comet/issues/1839#issuecomment-2967466668
The "canocilization and exchange reuse" test is expected to fail and should
be ignored until Comet supports DPP. The two exchanges are different. One
contains a `CometScan` and one contains `FileScan` with a
`SubqueryAdaptiveBroadcast` (due to fallback because of no DPP support).
```
Look for CometColumnarExchange hashpartitioning(store_id#3596, 5),
ENSURE_REQUIREMENTS, [plan_id=543]
+- Project [store_id#3596]
+- Filter (isnotnull(units_sold#3595) AND (units_sold#3595 = 70))
+- FileScan parquet
spark_catalog.default.fact_stats[units_sold#3595,store_id#3596] Batched: true,
DataFilters: [isnotnull(units_sold#3595), (units_sold#3595 = 70)], Format:
Parquet, Location: InMemoryFileIndex(25
paths)[file:/home/andy/git/apache/apache-spark-3.5.6/sql/core/spark-warehous...,
PartitionFilters: [isnotnull(store_id#3596),
dynamicpruningexpression(store_id#3596 IN dynamicpruning#3609)], PushedFilters:
[IsNotNull(units_sold), EqualTo(units_sold,70)], ReadSchema:
struct<units_sold:int>
+- SubqueryAdaptiveBroadcast dynamicpruning#3609, 0, true,
Project [store_id#3608], [store_id#3608]
+- AdaptiveSparkPlan isFinalPlan=false
+- CometProject [store_id#3608], [store_id#3608]
+- CometFilter [units_sold#3607, store_id#3608],
(isnotnull(units_sold#3607) AND (units_sold#3607 = 70))
+- CometScan parquet
spark_catalog.default.fact_stats[units_sold#3607,store_id#3608] Batched: true,
DataFilters: [isnotnull(units_sold#3607), (units_sold#3607 = 70)], Format:
CometParquet, Location: InMemoryFileIndex(25
paths)[file:/home/andy/git/apache/apache-spark-3.5.6/sql/core/spark-warehous...,
PartitionFilters: [isnotnull(store_id#3608)], PushedFilters:
[IsNotNull(units_sold), EqualTo(units_sold,70)], ReadSchema:
struct<units_sold:int>
getOrElseUpdate with CometColumnarExchange hashpartitioning(store_id#3596,
5), ENSURE_REQUIREMENTS, [plan_id=568]
+- CometProject [store_id#3596], [store_id#3596]
+- CometFilter [units_sold#3595, store_id#3596],
(isnotnull(units_sold#3595) AND (units_sold#3595 = 70))
+- CometScan parquet
spark_catalog.default.fact_stats[units_sold#3595,store_id#3596] Batched: true,
DataFilters: [isnotnull(units_sold#3595), (units_sold#3595 = 70)], Format:
CometParquet, Location: InMemoryFileIndex(25
paths)[file:/home/andy/git/apache/apache-spark-3.5.6/sql/core/spark-warehous...,
PartitionFilters: [isnotnull(store_id#3596), dynamicpruningexpression(true)],
PushedFilters: [IsNotNull(units_sold), EqualTo(units_sold,70)], ReadSchema:
struct<units_sold:int>
NO REUSE
Look for CometColumnarExchange hashpartitioning(store_id#3608, 5),
ENSURE_REQUIREMENTS, [plan_id=570]
+- CometProject [store_id#3608], [store_id#3608]
+- CometFilter [units_sold#3607, store_id#3608],
(isnotnull(units_sold#3607) AND (units_sold#3607 = 70))
+- CometScan parquet
spark_catalog.default.fact_stats[units_sold#3607,store_id#3608] Batched: true,
DataFilters: [isnotnull(units_sold#3607), (units_sold#3607 = 70)], Format:
CometParquet, Location: InMemoryFileIndex(25
paths)[file:/home/andy/git/apache/apache-spark-3.5.6/sql/core/spark-warehous...,
PartitionFilters: [isnotnull(store_id#3608)], PushedFilters:
[IsNotNull(units_sold), EqualTo(units_sold,70)], ReadSchema:
struct<units_sold:int>
getOrElseUpdate with CometColumnarExchange hashpartitioning(store_id#3608,
5), ENSURE_REQUIREMENTS, [plan_id=580]
+- CometProject [store_id#3608], [store_id#3608]
+- CometFilter [units_sold#3607, store_id#3608],
(isnotnull(units_sold#3607) AND (units_sold#3607 = 70))
+- CometScan parquet
spark_catalog.default.fact_stats[units_sold#3607,store_id#3608] Batched: true,
DataFilters: [isnotnull(units_sold#3607), (units_sold#3607 = 70)], Format:
CometParquet, Location: InMemoryFileIndex(25
paths)[file:/home/andy/git/apache/apache-spark-3.5.6/sql/core/spark-warehous...,
PartitionFilters: [isnotnull(store_id#3608)], PushedFilters:
[IsNotNull(units_sold), EqualTo(units_sold,70)], ReadSchema:
struct<units_sold:int>
NO REUSE
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
