aokolnychyi commented on a change in pull request #1665:
URL: https://github.com/apache/iceberg/pull/1665#discussion_r512220298
##########
File path: core/src/main/java/org/apache/iceberg/TableProperties.java
##########
@@ -84,6 +84,9 @@ private TableProperties() {
public static final String PARQUET_BATCH_SIZE =
"read.parquet.vectorization.batch-size";
public static final int PARQUET_BATCH_SIZE_DEFAULT = 5000;
+ public static final String PARQUET_IN_LIMIT =
"read.parquet.in-predicate-limit";
+ public static final int PARQUET_IN_LIMIT_DEFAULT = 200;
Review comment:
Spark's default value is 10 but it also rewrites IN as OR/EQUALS. We can
do better that this because of the way we evaluate IN predicates. On datasets
we tested this, we saw performance improvements on IN predicates with up to 200
elements (on sorted column). We may increase the default value a bit but I am
very reluctant to make it more than 300 since our jobs were 20% slower with IN
predicates that contained 300 elements. That being said, the optimal value
still depends on a particular query and table.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
