Steve Loughran created HADOOP-16746:
---------------------------------------
Summary: s3a empty dir markers are not created in s3guard as
authoritative
Key: HADOOP-16746
URL: https://issues.apache.org/jira/browse/HADOOP-16746
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.3.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Newly created empty dirs, or markers created after delete operations, are not
marked in S3Guard as auth. This has adverse consequences in that following
changes (i.e. new files) don't get marked as auth either...it needs a listFiles
call to scan the source and mark as auth.
I could stick a quick fix in to HADOOP-16697, but don't want to as I don't like
what that would mean. Essentially, finishedWrite() need to recognise when an
empty directory markers being created (it does this already) and then *always*
declare it as auth.
I'd prefer for the mkdirs operation to pass a flag all the way through to
finishedWrite so that it doesn't need to infer this. The WriteOpContext of
HADOOP-16134 would be the way to do this. Yes it's a big change but it would be
extensible -and I already have some plans there.
Instead it will be a follow-up.
The tests for this problem are part of HADOOP-16697, just disabled for now.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
