samidalouche opened a new issue #1483:
URL: https://github.com/apache/iceberg/issues/1483
Example query plan
```
== Parsed Logical Plan ==
'Project ['leftId.idValue AS leftIdValue#92, 'rightId.idValue AS
rightIdValue#93]
+- Join Inner, ((leftIdSize#29 = length(idValue#11)) && (leftId#24.idValue =
idValue#11))
:- Filter ((rightIdSize#30 = 64) && (rightId#25.idType = sha256_email))
: +- RelationV2 iceberg[eventId#17, companyId#18L, eventTimestamp#19L,
ingestionTimestamp#20L, originalParams#21, version#22, tags#23, leftId#24,
rightId#25, otherIds#26, ipAddress#27, countryCode#28, leftIdSize#29,
rightIdSize#30] (Options:
[path=s3://narrative-tmp/sami/idmapping-lake/left-and-right-ids-v18/,paths=[]])
+- Repartition 20, true
+- Project [idValue#11]
+- Relation[idType#10,idValue#11,ipAddress#12] csv
== Analyzed Logical Plan ==
leftIdValue: string, rightIdValue: string
Project [leftId#24.idValue AS leftIdValue#92, rightId#25.idValue AS
rightIdValue#93]
+- Join Inner, ((leftIdSize#29 = length(idValue#11)) && (leftId#24.idValue =
idValue#11))
:- Filter ((rightIdSize#30 = 64) && (rightId#25.idType = sha256_email))
: +- RelationV2 iceberg[eventId#17, companyId#18L, eventTimestamp#19L,
ingestionTimestamp#20L, originalParams#21, version#22, tags#23, leftId#24,
rightId#25, otherIds#26, ipAddress#27, countryCode#28, leftIdSize#29,
rightIdSize#30] (Options:
[path=s3://narrative-tmp/sami/idmapping-lake/left-and-right-ids-v18/,paths=[]])
+- Repartition 20, true
+- Project [idValue#11]
+- Relation[idType#10,idValue#11,ipAddress#12] csv
== Optimized Logical Plan ==
Project [leftId#24.idValue AS leftIdValue#92, rightId#25.idValue AS
rightIdValue#93]
+- Join Inner, ((leftIdSize#29 = length(idValue#11)) && (leftId#24.idValue =
idValue#11))
:- Project [leftId#24, rightId#25, leftIdSize#29]
: +- Filter (((((isnotnull(rightIdSize#30) && isnotnull(rightId#25)) &&
(rightIdSize#30 = 64)) && (rightId#25.idType = sha256_email)) &&
isnotnull(leftId#24)) && isnotnull(leftIdSize#29))
: +- RelationV2 iceberg[eventId#17, companyId#18L,
eventTimestamp#19L, ingestionTimestamp#20L, originalParams#21, version#22,
tags#23, leftId#24, rightId#25, otherIds#26, ipAddress#27, countryCode#28,
leftIdSize#29, rightIdSize#30] (Options:
[path=s3://narrative-tmp/sami/idmapping-lake/left-and-right-ids-v18/,paths=[]])
+- Repartition 20, true
+- Project [idValue#11]
+- Filter isnotnull(idValue#11)
+- Relation[idType#10,idValue#11,ipAddress#12] csv
== Physical Plan ==
*(5) Project [leftId#24.idValue AS leftIdValue#92, rightId#25.idValue AS
rightIdValue#93]
+- *(5) SortMergeJoin [leftIdSize#29, leftId#24.idValue],
[length(idValue#11), idValue#11], Inner
:- *(2) Sort [leftIdSize#29 ASC NULLS FIRST, leftId#24.idValue ASC NULLS
FIRST], false, 0
: +- Exchange hashpartitioning(leftIdSize#29, leftId#24.idValue, 200)
: +- *(1) Project [leftId#24, rightId#25, leftIdSize#29]
: +- *(1) Filter ((((((rightId#25.idType = sha256_email) &&
isnotnull(rightId#25)) && isnotnull(leftIdSize#29)) &&
isnotnull(rightIdSize#30)) && (rightIdSize#30 = 64)) && isnotnull(leftId#24))
: +- *(1) ScanV2 iceberg[leftId#24, rightId#25, leftIdSize#29,
rightIdSize#30] (Filters: [isnotnull(rightId#25), isnotnull(leftIdSize#29),
isnotnull(rightIdSize#30), (rightIdSize#30 = 64..., Options:
[path=s3://narrative-tmp/sami/idmapping-lake/left-and-right-ids-v18/,paths=[]])
+- *(4) Sort [length(idValue#11) ASC NULLS FIRST, idValue#11 ASC NULLS
FIRST], false, 0
+- Exchange hashpartitioning(length(idValue#11), idValue#11, 200)
+- Exchange RoundRobinPartitioning(20)
+- *(3) Project [idValue#11]
+- *(3) Filter isnotnull(idValue#11)
+- *(3) FileScan csv [idValue#11] Batched: false, Format:
CSV, Location:
InMemoryFileIndex[s3://narrative-tmp/sami/idmapping-lake/id-lists/sample1.csv],
PartitionFilters: [], PushedFilters: [IsNotNull(idValue)], ReadSchema:
struct<idValue:string>
```
More specifically
```
: +- *(1) ScanV2 iceberg[leftId#24, rightId#25, leftIdSize#29,
rightIdSize#30] (Filters: [isnotnull(rightId#25), isnotnull(leftIdSize#29),
isnotnull(rightIdSize#30), (rightIdSize#30 = 64..., Options:
[path=s3://narrative-tmp/sami/idmapping-lake/left-and-right-ids-v18/,paths=[]])
```
The line does not include all the filters that are being pushed to Iceberg.
Is there a way to force iceberg to display all the filters?
(`spark.debug.maxToStringFields` does not have any effect on the output).
Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
