mccheah commented on a change in pull request #73: Allow data output streams to
be generated via custom mechanisms when given partitioning and file name
URL: https://github.com/apache/incubator-iceberg/pull/73#discussion_r248511242
##########
File path:
core/src/main/java/com/netflix/iceberg/BaseMetastoreTableOperations.java
##########
@@ -114,30 +103,19 @@ protected void refreshFromMetadataLocation(String
newLocation, int numRetries) {
this.shouldRefresh = false;
}
- private String metadataFileLocation(TableMetadata metadata, String filename)
{
- String metadataLocation = metadata.properties()
- .get(TableProperties.WRITE_METADATA_LOCATION);
-
- if (metadataLocation != null) {
- return String.format("%s/%s", metadataLocation, filename);
- } else {
- return String.format("%s/%s/%s", metadata.location(),
METADATA_FOLDER_NAME, filename);
- }
- }
-
@Override
- public String metadataFileLocation(String filename) {
- return metadataFileLocation(current(), filename);
+ public FileIO io() {
+ return new HadoopFileIO(conf, currentMetadata);
}
@Override
- public FileIO io() {
- return fileIo;
+ public FileIO io(TableMetadata tableMetadata) {
Review comment:
> So unless I'm missing something again, the only thing that adding this
method accomplishes is to be able to update some location and have the change
take effect for a write in the same atomic commit. I'm okay with that
limitation to make this API cleaner, unless you need to be able to change
settings and write in an atomic commit for your use case.
So I accidentally stumbled upon having to do it this way because a lot of
tests just failed if we used the old location to commit when the new metadata
specifies committing elsewhere. Thus I had thought that the current behavior is
part of public API and we don't intend on changing it. I suppose we ought to
document this specific behavior somewhere (if you change the new metadata
location table property on a commit, the change isn't reflected in that current
commit). However it's unclear where the right place to document that is.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
