Re: AWS Consistent S3 & Apache Hadoop's S3A connector

2020-12-10 Thread Steve Loughran
On Mon, 7 Dec 2020 at 07:36, Chang Chen wrote: > Since S3A now works perfectly with S3Guard turned off, Could Magic > Committor work with S3Guard is off? If Yes, will performance degenerate? Or > if HADOOP-17400 is fixed, then it will have comparable performance? > Yes, works really well. * It

Re: AWS Consistent S3 & Apache Hadoop's S3A connector

2020-12-06 Thread Chang Chen
Since S3A now works perfectly with S3Guard turned off, Could Magic Committor work with S3Guard is off? If Yes, will performance degenerate? Or if HADOOP-17400 is fixed, then it will have comparable performance? Steve Loughran 于2020年12月4日周五 下午10:00写道: > as sent to hadoop-general. > > TL;DR. S3

AWS Consistent S3 & Apache Hadoop's S3A connector

2020-12-04 Thread Steve Loughran
as sent to hadoop-general. TL;DR. S3 is consistent; S3A now works perfectly with S3Guard turned off, if not, file a JIRA. rename still isn't real, so don't rely on that or create(path, overwrite=false) for atomic operations --- If you've missed the announcement, AWS S3 storage is now