[ https://issues.apache.org/jira/browse/HADOOP-15780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gabor Bota resolved HADOOP-15780. --------------------------------- Resolution: Fixed > S3Guard: document how to deal with non-S3Guard processes writing data to > S3Guarded buckets > ------------------------------------------------------------------------------------------ > > Key: HADOOP-15780 > URL: https://issues.apache.org/jira/browse/HADOOP-15780 > Project: Hadoop Common > Issue Type: Sub-task > Affects Versions: 3.2.0 > Reporter: Aaron Fabbri > Assignee: Gabor Bota > Priority: Major > > Our general policy for S3Guard is this: All modifiers of a bucket that is > configured for use with S3Guard, must use S3Guard. Otherwise, the > MetadataStore will not be properly updated as the S3 bucket changes and > problems will arise. > There are limited circumstances in which may be safe to have an external > (non-s3guard) process writing data. There are also scenarios where it > definitely breaks things. > I think we should start by documenting the cases that this works / does not > work for. After we've enumerated that, we can suggest enhancements as needed > to make this sort of configuration easier to use. > To get the ball rolling, some things that do not work: > - Deleting a path *p* with S3Guard, then writing a new file at path *p* > without S3guard (will still have delete marker in S3Guard, making the file > appear to be deleted but still visible in S3 due to false "eventual > consistency") (as [~ste...@apache.org] and I have discussed) > - When fs.s3a.metadatastore.authoritative is true, adding files to > directories without S3Guard, then listing with S3Guard may exclude > externally-written files from listings. > (Note, there are also S3A interop issues with other non-S3A clients even > without S3Guard, due to the unique way S3A interprets empty directory > markers). -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org