rdblue commented on a change in pull request #2577:
URL: https://github.com/apache/iceberg/pull/2577#discussion_r637332124
##########
File path: spark/src/main/java/org/apache/iceberg/spark/SparkUtil.java
##########
@@ -100,4 +107,11 @@ public static void
validatePartitionTransforms(PartitionSpec spec) {
}
}
}
+
+ public static boolean isLocalityEnabledDefault(Map<String, String>
tableProperties, String fsScheme) {
+ String tableLocalityProp = PropertyUtil.propertyAsString(tableProperties,
TableProperties.LOCALITY_ENABLED,
+ TableProperties.LOCALITY_ENABLED_DEFAULT);
+ return tableLocalityProp == null ?
LOCALITY_WHITELIST_FS.contains(fsScheme) :
+ Boolean.parseBoolean(tableLocalityProp);
Review comment:
Yeah, I'm not a fan either, but the read and write options probably
won't be possible through SQL otherwise. Maybe that's not what we want to do
for things that would ideally have SQL clauses (like `AS OF TIMESTAMP` or `AS
OF VERSION`) but hints like `locality=true` seem like a reasonable path to me.
We may even be able to get that in upstream Spark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
