Cheappie commented on issue #2205: URL: https://github.com/apache/arrow-datafusion/issues/2205#issuecomment-1096623689
> [This AWS blog post](https://aws.amazon.com/blogs/aws/the-floodgates-are-open-increased-network-bandwidth-for-ec2-instances/) from 2018 would suggest up to 25Gpbs EC2-S3 is possible and also highlights placement groups as a way to accelerate EC2-EC2. [This support question](https://aws.amazon.com/premiumsupport/knowledge-center/s3-maximum-transfer-speed-ec2/) would suggest the EC2-S3 limit has since been raised to 100Gbps. > > I also found [this benchmark](https://github.com/dvassallo/s3-benchmark#s3-to-ec2-bandwidth) from 2019, which shows speeds in the 1000s of MB/s, including 1,135 MB/s for the r4.2xlarge. I have not been able to find anyone complaining about the network speeds being below what is advertised. > > FWIW if using VPC networking, you need to make sure you have configured a [VPC Gateway](https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-s3.html) and are using a [region-specific endpoint](https://docs.aws.amazon.com/general/latest/gr/rande.html#regional-endpoints) for S3. Otherwise your traffic will transit an Internet Gateway or NAT gateway which will make things a lot slower (and cost a LOT of money). You might be right, I don't recall testing against region specific endpoint. That's really interesting, I will have to check that. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
