Kimahriman commented on code in PR #793:
URL: https://github.com/apache/datafusion-comet/pull/793#discussion_r1710182306
##########
dev/diffs/3.4.3.diff:
##########
@@ -423,7 +423,17 @@ index f33432ddb6f..9cf7a9dd4e3 100644
Given("dynamic pruning filter on the build side")
withSQLConf(SQLConf.DYNAMIC_PARTITION_PRUNING_REUSE_BROADCAST_ONLY.key ->
"true") {
val df = sql(
-@@ -1311,7 +1320,8 @@ abstract class DynamicPartitionPruningSuiteBase
+@@ -1279,7 +1288,8 @@ abstract class DynamicPartitionPruningSuiteBase
+ }
+ }
+
+- test("SPARK-32659: Fix the data issue when pruning DPP on non-atomic type")
{
Review Comment:
```
[info] - SPARK-32659: Fix the data issue when pruning DPP on non-atomic type
*** FAILED *** (453 milliseconds)
[info] hasReuse was false SubqueryBroadcast dynamicpruning#20445, 0,
[array(store_id#19758)], [id=#26550]
[info] +- BroadcastExchange
HashedRelationBroadcastMode(List(array(input[0, int, true])),false),
[plan_id=26549]
[info] +- *(1) ColumnarToRow
[info] +- CometProject [store_id#19758], [store_id#19758]
[info] +- CometFilter [store_id#19758, country#19760],
(isnotnull(country#19760) AND (country#19760 = DE))
[info] +- CometScan parquet
spark_catalog.default.dim_stats[store_id#19758,country#19760] Batched: true,
DataFilters: [isnotnull(country#19760), (country#19760 = DE)], Format:
CometParquet, Location: InMemoryFileIndex(1
paths)[file:/__w/arrow-datafusion-comet/arrow-datafusion-comet/apache-spark/s...,
PartitionFilters: [], PushedFilters: [IsNotNull(country),
EqualTo(country,DE)], ReadSchema: struct<store_id:int,country:string>
[info]
[info] should have been reused in
[info] *(1) ColumnarToRow
[info] +- CometProject [date_id#19754, product_id#19755, units_sold#19756,
store_id#19757], [date_id#19754, product_id#19755, units_sold#19756,
store_id#19757]
[info] +- CometBroadcastHashJoin [array(store_id#19757)],
[array(store_id#19758)], Inner, BuildRight
[info] :- CometScan parquet
spark_catalog.default.fact_stats[date_id#19754,product_id#19755,units_sold#19756,store_id#19757]
Batched: true, DataFilters: [], Format: CometParquet, Location:
CatalogFileIndex(1
paths)[file:/__w/arrow-datafusion-comet/arrow-datafusion-comet/apache-spark/sq...,
PartitionFilters: [dynamicpruningexpression(array(store_id#19757) IN
dynamicpruning#20445)], PushedFilters: [], ReadSchema:
struct<date_id:int,product_id:int,units_sold:int>
[info] : +- SubqueryBroadcast dynamicpruning#20445, 0,
[array(store_id#19758)], [id=#26550]
[info] : +- BroadcastExchange
HashedRelationBroadcastMode(List(array(input[0, int, true])),false),
[plan_id=26549]
[info] : +- *(1) ColumnarToRow
[info] : +- CometProject [store_id#19758],
[store_id#19758]
[info] : +- CometFilter [store_id#19758,
country#19760], (isnotnull(country#19760) AND (country#19760 = DE))
[info] : +- CometScan parquet
spark_catalog.default.dim_stats[store_id#19758,country#19760] Batched: true,
DataFilters: [isnotnull(country#19760), (country#19760 = DE)], Format:
CometParquet, Location: InMemoryFileIndex(1
paths)[file:/__w/arrow-datafusion-comet/arrow-datafusion-comet/apache-spark/s...,
PartitionFilters: [], PushedFilters: [IsNotNull(country),
EqualTo(country,DE)], ReadSchema: struct<store_id:int,country:string>
[info] +- CometBroadcastExchange [store_id#19758]
[info] +- CometProject [store_id#19758], [store_id#19758]
[info] +- CometFilter [store_id#19758, country#19760],
(isnotnull(country#19760) AND (country#19760 = DE))
[info] +- CometScan parquet
spark_catalog.default.dim_stats[store_id#19758,country#19760] Batched: true,
DataFilters: [isnotnull(country#19760), (country#19760 = DE)], Format:
CometParquet, Location: InMemoryFileIndex(1
paths)[file:/__w/arrow-datafusion-comet/arrow-datafusion-comet/apache-spark/s...,
PartitionFilters: [], PushedFilters: [IsNotNull(country),
EqualTo(country,DE)], ReadSchema: struct<store_id:int,country:string>
(DynamicPartitionPruningSuite.scala:215)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org
