Kimahriman commented on code in PR #793:
URL: https://github.com/apache/datafusion-comet/pull/793#discussion_r1710179762
##########
dev/diffs/3.4.3.diff:
##########
@@ -423,7 +423,17 @@ index f33432ddb6f..9cf7a9dd4e3 100644
Given("dynamic pruning filter on the build side")
withSQLConf(SQLConf.DYNAMIC_PARTITION_PRUNING_REUSE_BROADCAST_ONLY.key ->
"true") {
val df = sql(
-@@ -1311,7 +1320,8 @@ abstract class DynamicPartitionPruningSuiteBase
+@@ -1279,7 +1288,8 @@ abstract class DynamicPartitionPruningSuiteBase
+ }
+ }
+
+- test("SPARK-32659: Fix the data issue when pruning DPP on non-atomic type")
{
Review Comment:
It basically is testing for a reused BroadcastExchange with join on
non-atomic types
```
s"""
|SELECT f.date_id, f.product_id, f.units_sold, f.store_id
FROM fact_stats f
|JOIN dim_stats s
|ON $dataType(f.store_id) = $dataType(s.store_id) WHERE
s.country = 'DE'
""".stripMargin
```
where `dataType` is `Seq("struct", "array").foreach { dataType =>`
Basically not supporting `CreateArray` is what made this fallback to
non-comet, but with `CreateArray` support, this now tries to use Comet but
fails because of the same lack of support for `SubqueryBroadcastExec `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
