[ https://issues.apache.org/jira/browse/HADOOP-16746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17017090#comment-17017090 ]
Steve Loughran commented on HADOOP-16746: ----------------------------------------- Looks suspiciously like when we list a dir and push out the contents, if a subdir was in the listing, that update will overwrite any auth dir entry -that is we can unmark a dir by listing its parent > s3a empty dir markers are not created in s3guard as authoritative > ----------------------------------------------------------------- > > Key: HADOOP-16746 > URL: https://issues.apache.org/jira/browse/HADOOP-16746 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 > Affects Versions: 3.3.0 > Reporter: Steve Loughran > Assignee: Steve Loughran > Priority: Major > > Newly created empty dirs, or markers created after delete operations, are not > marked in S3Guard as auth. This has adverse consequences in that following > changes (i.e. new files) don't get marked as auth either...it needs a > listFiles call to scan the source and mark as auth. > I could stick a quick fix in to HADOOP-16697, but don't want to as I don't > like what that would mean. Essentially, finishedWrite() need to recognise > when an empty directory markers being created (it does this already) and then > *always* declare it as auth. > I'd prefer for the mkdirs operation to pass a flag all the way through to > finishedWrite so that it doesn't need to infer this. The WriteOpContext of > HADOOP-16134 would be the way to do this. Yes it's a big change but it would > be extensible -and I already have some plans there. > Instead it will be a follow-up. > The tests for this problem are part of HADOOP-16697, just disabled for now. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org