kbendick commented on a change in pull request #2577:
URL: https://github.com/apache/iceberg/pull/2577#discussion_r637331639
##########
File path: spark/src/main/java/org/apache/iceberg/spark/SparkUtil.java
##########
@@ -100,4 +107,11 @@ public static void
validatePartitionTransforms(PartitionSpec spec) {
}
}
}
+
+ public static boolean isLocalityEnabledDefault(Map<String, String>
tableProperties, String fsScheme) {
+ String tableLocalityProp = PropertyUtil.propertyAsString(tableProperties,
TableProperties.LOCALITY_ENABLED,
+ TableProperties.LOCALITY_ENABLED_DEFAULT);
+ return tableLocalityProp == null ?
LOCALITY_WHITELIST_FS.contains(fsScheme) :
+ Boolean.parseBoolean(tableLocalityProp);
Review comment:
I'm a big fan of hints (not that I was asked). But they're a great SQL
first solution to a number of problems.
Assuming you mean like
```
select /* read.locality.enabled=false */ a, b, c from table iceberg_table t
```
We have encountered a few situations internally where updating
`spark.sql.partitions` would make a job unreasonably slow (due to possibly a
large filter prior to the write), so a `COALESCE` hint would be really helpful
in helping with file sizes (which I assume is naturally supported from spark
3.x hints, unless Iceberg operations get in the way, e.g maybe anything to do
with merging or deletes). But I have admittedly not tried it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
