Re: Ask for reason for choice of S3 plugins

2020-03-27 Thread David Anderson
If you are using both the Hadoop S3 and Presto S3 filesystems, you should use s3p:// and s3a:// to distinguish between the two. Presto is recommended for checkpointing because the Hadoop implementation has very high latency when creating files, and because it hits request rate limits very

Ask for reason for choice of S3 plugins

2020-03-27 Thread B.Zhou
Hi, In this document https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems/s3.html#hadooppresto-s3-file-systems-plugins, it mentioned that * Presto is the recommended file system for checkpointing to S3. Is there a reason for that? Is there some bottleneck for s3 hadoop