kbendick commented on a change in pull request #2845:
URL: https://github.com/apache/iceberg/pull/2845#discussion_r687300481
##########
File path: core/src/main/java/org/apache/iceberg/LocationProviders.java
##########
@@ -68,13 +68,15 @@ public static LocationProvider locationsFor(String
location, Map<String, String>
}
}
+ private static String defaultDataLocation(String tableLocation, Map<String,
String> properties) {
+ return properties.getOrDefault(TableProperties.WRITE_NEW_DATA_LOCATION,
String.format("%s/data", tableLocation));
Review comment:
Or would we wind up getting
`s3://bucket/warehouse/<hash>/db/tablename/data`?
My preference would be for the hash to be just after the warehouse for data
files if `write.object-storage.path` is not specified, though I can understand
that we don't want to complicate the resolution to much. However, keeping the
entropy high enough in the path for S3 partitioning seems like a reasonable
goal for allowing it to be somewhat convoluted.
I have not made much usage of `WRITE_NEW_DATA_LOCATION` to have a strong
opinion there. I agree the naming is a bit strange
(`write.folder-storage.path`), but given that this config already exists (with
the given name), I do agree that it would be a bit strange to ignore it
entirely if users have set it.
Is there no convention around `write.folder-storage.path`, and
`write.folder-storage.*` to only be applied to non-object storage layouts?
Given that the property name is `TableProperties.WRITE_NEW_DATA_LOCATION`, I'm
guessing that the name is just an arguably unfortunate side effect, but that we
should respect the intention of this property.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
