sunchao commented on a change in pull request #2276:
URL: https://github.com/apache/iceberg/pull/2276#discussion_r588798684
##########
File path: core/src/main/java/org/apache/iceberg/TableProperties.java
##########
@@ -78,6 +78,9 @@ private TableProperties() {
public static final String SPLIT_OPEN_FILE_COST =
"read.split.open-file-cost";
public static final long SPLIT_OPEN_FILE_COST_DEFAULT = 4 * 1024 * 1024; //
4MB
+ public static final String SPLIT_BY_PARTITION = "read.split.by-partition";
Review comment:
Yea, I think it's not a good idea to have this as a table property, will
update.
I'll keep it as a Spark read option if ppl think otherwise (cc @rdblue
@prodeezy).
> We may want to default it for bucketed tables in Spark to true unless set
by the user in options.
I assume "bucketed" = "partitioned" :) since the storage-partitioned join on
Spark side could apply to other partition specs than bucketing. If a table is
unpartitioned then I think the behavior is the same no matter the flag is on or
off. Therefore, I think we just need to set the default value to true for Spark?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
