singhpk234 commented on code in PR #17457:
URL: https://github.com/apache/iceberg/pull/17457#discussion_r3920274427
##########
aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIO.java:
##########
@@ -100,6 +103,9 @@ public class S3FileIO
private static final String DEFAULT_METRICS_IMPL =
"org.apache.iceberg.hadoop.HadoopMetricsContext";
private static final String ROOT_PREFIX = "s3";
+ // Default HTTP read chunk size for pre-signed URL reads, tuned for S3;
overridable via the
+ // io.http.read.chunk-size-bytes property.
+ private static final int HTTP_READ_CHUNK_SIZE_BYTES_DEFAULT = 8 * 1024 *
1024; // 8 MB
Review Comment:
This is saying s3 prefers 8 MB read ... it might happen ADLS prefers 64 MB
... the generic HTTPInputStream is configurable to what the objectstore
provider FileIO prefers, so s3 here is sending this HTTPInputStream it need to
read it 8 MB chunks ... rather than HTTPInputstream fixing the
http_read_chunk_bytes for everyone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
