[
https://issues.apache.org/jira/browse/HADOOP-13448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15415658#comment-15415658
]
Aaron Fabbri commented on HADOOP-13448:
---------------------------------------
To start the interface discussion:
I was looking at the two ConsistentStore/MetadataStore interfaces we posted to
HADOOP-13345, and had a question: Why does your
{{DynamoDBConsistentStore#save()}} implementation walk the path to the root and
save all ancestor paths as well? I understand that this is mirroring what the
S3A client does. I guess I preferred leaving the enumeration of ancestor
directories that need to be created to the client. The case I'm thinking of is
a workload that does this:
{code}
path = "/dirs"
for x in range(0,10) :
path += "/dir" + str(x)
s3aFS.mkdirs(path)
{code}
In this case, we're making N directories that get deeper and deeper with the
same ancestors. This seems like an theta(N^2) cost with your prototype, but
only N cost with ours. The downside to our approach is that the client code
has to explicitly track the missing ancestors that need to be passed to the
MetadataStore. In our patch it looked like this at the end of
{{innerMkdirs()}}:
{code}
} catch (FileNotFoundException fnfe) {
instrumentation.errorIgnored();
if (dirsToCreate != null)
dirsToCreate.add(fPart);
}
fPart = fPart.getParent();
} while (fPart != null);
String key = pathToKey(f);
createFakeDirectory(key);
metadataStoreMkdirs(dirsToCreate, permission);
return true;
}
}
{code}
I might be missing something here, let me know what you think.
> S3Guard: Define MetadataStore interface.
> ----------------------------------------
>
> Key: HADOOP-13448
> URL: https://issues.apache.org/jira/browse/HADOOP-13448
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Chris Nauroth
> Assignee: Chris Nauroth
>
> Define the common interface for metadata store operations. This is the
> interface that any metadata back-end must implement in order to integrate
> with S3Guard.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
