It isn't clear whether this S3 consistency change also fixes the negative caching (HEAD when file doesn't exist causes later HEAD to not see the file), but I think that it does not fix it because there was a PR opened to add consistency using LIST before a HEAD operation.
I think it is still a good idea to use the new S3FileIO for S3 tables. On Wed, Dec 2, 2020 at 2:11 AM Jungtaek Lim <kabhwan.opensou...@gmail.com> wrote: > What about S3FileIO implementation? I see some issue filed that even with > Hive catalog working with S3 brings unexpected issues, and S3FileIO > supposed to fix the issue (according to Ryan). Is it safe without S3FileIO > to use Hive catalog + Hadoop API for S3 now? > > 2020년 12월 2일 (수) 오후 6:54, Vivekanand Vellanki <vi...@dremio.com>님이 작성: > >> Iceberg tables backed by HadoopTables and HadoopCatalog require an atomic >> rename. This is not yet supported with S3. >> >> On Wed, Dec 2, 2020 at 3:20 PM Mass Dosage <massdos...@gmail.com> wrote: >> >>> Hello all, >>> >>> Yesterday AWS announced that S3 now has strong read-after-write >>> consistency: >>> >>> >>> https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-read-after-write-consistency >>> >>> https://aws.amazon.com/s3/consistency/ >>> >>> Does this mean that Iceberg tables backed by HadoopTables and >>> HadoopCatalog can now be used on S3 in addition to HDFS? >>> >>> Thanks, >>> >>> Adrian >>> >> -- Ryan Blue Software Engineer Netflix
