danielcweeks commented on code in PR #17457:
URL: https://github.com/apache/iceberg/pull/17457#discussion_r4543288462


##########
aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIO.java:
##########
@@ -88,10 +89,9 @@
  * FileIO implementation backed by S3.
  *
  * <p>Locations used must follow the conventions for S3 URIs (e.g. 
s3://bucket/path...). URIs with
- * schemes s3a, s3n, https are also treated as s3 file paths. Using this 
FileIO with other schemes
- * will result in {@link org.apache.iceberg.exceptions.ValidationException}.
+ * schemes s3a, s3n are also treated as s3 file paths.
  */
-public class S3FileIO
+public class S3FileIO extends BaseFileIO

Review Comment:
   I don't think we can do this silently. Before this change `S3FileIO` treated 
`https://bucket/key` as an S3 path through `S3URI`, using the configured 
credentials. Now reads of those locations become unauthenticated GETs, while 
`newOutputFile`, `deleteFile`, and listing still send the same location to S3. 
A table that read fine before would start failing with 403 after an upgrade, 
and reads and writes of the same location would go through different systems. 
Removing the sentence from the Javadoc doesn't change the code that still 
treats `https` as S3. I'd rather leave `S3FileIO` alone and resolve `http(s)` 
by scheme. If it has to live here, it needs to be explicit and off by default, 
with a test that `https://` without the option still returns an `S3InputFile`.
   
   <!-- lenses: russell-review, blue-review -->



##########
core/src/main/java/org/apache/iceberg/io/HttpInputFile.java:
##########
@@ -0,0 +1,262 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Locale;
+import java.util.Map;
+import org.apache.hc.client5.http.classic.methods.HttpGet;
+import org.apache.hc.client5.http.impl.classic.CloseableHttpClient;
+import org.apache.hc.core5.http.ClassicHttpResponse;
+import org.apache.hc.core5.http.Header;
+import org.apache.hc.core5.http.HttpHeaders;
+import org.apache.iceberg.exceptions.ForbiddenException;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.metrics.MetricsContext;
+import 
org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.PropertyUtil;
+
+/**
+ * An {@link InputFile} backed by an HTTP URL, typically a pre-signed 
object-store URL that encodes
+ * auth in its query parameters.
+ *
+ * <p>A known content length is returned directly; otherwise it is fetched 
lazily via a {@code GET
+ * Range: bytes=0-0} request, which (unlike HEAD) works with pre-signed GET 
URLs.
+ */
+class HttpInputFile extends BaseHttpFile implements InputFile {
+  static final String READ_CHUNK_SIZE_BYTES = "io.http.read.chunk-size-bytes";
+  static final int READ_CHUNK_SIZE_BYTES_DEFAULT = 8 * 1024 * 1024;
+  static final long UNKNOWN_LENGTH = -1L;
+
+  private final int chunkSize;
+
+  private long length;
+
+  static InputFile fromLocation(
+      String location, Map<String, String> properties, MetricsContext metrics) 
{
+    return new HttpInputFile(location, location, UNKNOWN_LENGTH, properties, 
metrics);
+  }
+
+  static InputFile fromLocation(
+      String location, long length, Map<String, String> properties, 
MetricsContext metrics) {
+    return new HttpInputFile(location, location, length, properties, metrics);
+  }
+
+  /** Returns {@code true} if {@code location} is an HTTP(S) URL. */
+  static boolean isHttpUrl(String location) {
+    if (location == null) {
+      return false;
+    }
+
+    try {
+      String scheme = URI.create(location).getScheme();
+      return "https".equalsIgnoreCase(scheme) || 
"http".equalsIgnoreCase(scheme);
+    } catch (IllegalArgumentException e) {
+      return false;
+    }
+  }
+
+  private HttpInputFile(
+      String location,
+      String url,
+      long length,
+      Map<String, String> properties,
+      MetricsContext metrics) {
+    this(null, location, url, length, readChunkSize(properties), properties, 
metrics);
+  }
+
+  HttpInputFile(
+      CloseableHttpClient client,
+      String location,
+      String url,
+      int chunkSize,
+      MetricsContext metrics) {
+    this(client, location, url, UNKNOWN_LENGTH, chunkSize, null, metrics);
+  }
+
+  HttpInputFile(
+      CloseableHttpClient client,
+      String location,
+      String url,
+      long length,
+      int chunkSize,
+      MetricsContext metrics) {
+    this(client, location, url, length, chunkSize, null, metrics);
+  }
+
+  private HttpInputFile(
+      CloseableHttpClient client,
+      String location,
+      String url,
+      long length,
+      int chunkSize,
+      Map<String, String> properties,
+      MetricsContext metrics) {
+    super(client, location, url, properties, metrics);
+    Preconditions.checkArgument(
+        chunkSize > 0, "Invalid %s: %s (must be > 0)", READ_CHUNK_SIZE_BYTES, 
chunkSize);
+    this.chunkSize = chunkSize;
+    this.length = length;
+  }
+
+  @Override
+  public long getLength() {
+    if (length == UNKNOWN_LENGTH) {
+      long fetchedLength = fetchContentLength();
+      if (fetchedLength == UNKNOWN_LENGTH) {
+        throw new RuntimeIOException(
+            "Cannot determine content length for %s", 
BaseHttpFile.redact(location()));
+      }
+
+      this.length = fetchedLength;
+    }
+
+    return length;
+  }
+
+  @Override
+  public SeekableInputStream newStream() {
+    if (hasSharedClient()) {
+      return new HttpInputStream(client(), location(), url(), chunkSize, 
metrics());
+    }
+
+    return new HttpInputStream(newHttpClient(), location(), url(), chunkSize, 
metrics(), true);

Review Comment:
   I'm concerned about the client lifecycle. `BaseFileIO` always goes through 
`fromLocation` without a client, so every `newStream()`, `getLength()`, and 
`exists()` builds and tears down its own `CloseableHttpClient` and pool. On a 
Parquet read that's a new TLS handshake for the footer and for every column 
stream, with no connection reuse, and 
`io.http.max-connections`/`connections-per-route` don't do anything. The 
shared-client path is only used by tests, so the tests don't cover the path 
production uses. The client lifecycle should track with the FileIO: create it 
lazily, pass it in, and close it in `close()` like `S3FileIO` does with its S3 
clients. Then the null-client branches, `hasSharedClient()`, and the per-stream 
`closeClient` can go away.
   
   <!-- lenses: weeks-review, russell-review, blue-review -->



##########
core/src/main/java/org/apache/iceberg/io/HttpInputStream.java:
##########
@@ -0,0 +1,312 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.net.SocketException;
+import java.net.SocketTimeoutException;
+import java.util.Arrays;
+import java.util.Locale;
+import java.util.concurrent.atomic.AtomicReference;
+import javax.net.ssl.SSLException;
+import org.apache.hc.client5.http.classic.methods.HttpGet;
+import org.apache.hc.client5.http.impl.classic.CloseableHttpClient;
+import org.apache.hc.core5.http.HttpHeaders;
+import org.apache.hc.core5.http.io.entity.EntityUtils;
+import org.apache.hc.core5.io.CloseMode;
+import org.apache.iceberg.exceptions.ForbiddenException;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.metrics.Counter;
+import org.apache.iceberg.metrics.MetricsContext;
+import org.apache.iceberg.metrics.MetricsContext.Unit;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.Tasks;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link SeekableInputStream} that reads an HTTP URL via range GETs, for 
pre-signed object-store
+ * URLs that need no object-store credentials on the reader.
+ *
+ * <p>Sequential reads are served from a fixed-size in-memory chunk buffer, 
each chunk fetched with
+ * a single range GET fully consumed within the response handler so 
connections return to the pool.
+ * Positional reads ({@link #readFully}, {@link #readTail}) each issue their 
own range GET.
+ *
+ * <p>Transient socket/TLS errors and retryable HTTP responses are retried 
with exponential backoff
+ * up to {@value #MAX_RETRIES} times, so a throttled or briefly unavailable 
endpoint is not
+ * hammered. A missing location ({@code 404}) surfaces as {@link 
NotFoundException} and a forbidden
+ * response ({@code 403}, e.g. an expired pre-signed URL) as {@link 
ForbiddenException}; both are
+ * terminal, as is any other non-retryable status.
+ */
+class HttpInputStream extends SeekableInputStream implements RangeReadable {
+  private static final Logger LOG = 
LoggerFactory.getLogger(HttpInputStream.class);
+
+  private static final int MAX_RETRIES = 3;
+  private static final int MIN_RETRY_WAIT_MS = 100;
+  private static final int MAX_RETRY_WAIT_MS = 5_000;
+  private static final int MAX_RETRY_DURATION_MS = 30_000;
+  private static final double RETRY_SCALE_FACTOR = 2.0;
+
+  private final StackTraceElement[] createStack;
+  private final CloseableHttpClient client;
+  private final String location;
+  private final String url;
+  private final int chunkSize;
+  private final boolean closeClient;
+
+  private final Counter readBytes;
+  private final Counter readOperations;
+
+  /** Cached chunk buffer. {@code bufferFileStart} is the file offset of 
{@code buffer[0]}. */
+  private byte[] buffer;
+
+  private long bufferFileStart = -1L;
+  private int bufferLimit = 0;
+
+  private long next = 0;
+  private boolean closed = false;
+
+  HttpInputStream(
+      CloseableHttpClient client,
+      String location,
+      String url,
+      int chunkSize,
+      MetricsContext metrics) {
+    this(client, location, url, chunkSize, metrics, false);
+  }
+
+  HttpInputStream(
+      CloseableHttpClient client,
+      String location,
+      String url,
+      int chunkSize,
+      MetricsContext metrics,
+      boolean closeClient) {
+    this.client = client;
+    this.location = location;
+    this.url = url;
+    this.chunkSize = chunkSize;
+    this.closeClient = closeClient;
+    this.readBytes = metrics.counter(FileIOMetricsContext.READ_BYTES, 
Unit.BYTES);
+    this.readOperations = 
metrics.counter(FileIOMetricsContext.READ_OPERATIONS);
+    this.createStack = Thread.currentThread().getStackTrace();
+  }
+
+  @Override
+  public long getPos() {
+    return next;
+  }
+
+  @Override
+  public void seek(long newPos) {
+    Preconditions.checkState(!closed, "Cannot seek: already closed");
+    Preconditions.checkArgument(newPos >= 0, "position is negative: %s", 
newPos);
+    next = newPos;
+  }
+
+  @Override
+  public int read() throws IOException {
+    Preconditions.checkState(!closed, "Cannot read: already closed");
+    ensureBuffered();
+
+    if (buffer == null || !inBuffer(next)) {
+      return -1; // EOF
+    }
+
+    int bufPos = (int) (next - bufferFileStart);
+    next += 1;
+    readBytes.increment();
+    readOperations.increment();
+    return buffer[bufPos] & 0xFF;
+  }
+
+  @Override
+  public int read(byte[] b, int off, int len) throws IOException {
+    Preconditions.checkState(!closed, "Cannot read: already closed");
+    Preconditions.checkPositionIndexes(off, off + len, b.length);
+    if (len == 0) {
+      return 0;
+    }
+
+    ensureBuffered();
+
+    if (buffer == null || !inBuffer(next)) {
+      return -1; // EOF
+    }
+
+    int bufPos = (int) (next - bufferFileStart);
+    int available = bufferLimit - bufPos;
+    int toCopy = Math.min(len, available);
+    System.arraycopy(buffer, bufPos, b, off, toCopy);
+    next += toCopy;
+    readBytes.increment(toCopy);
+    readOperations.increment();
+    return toCopy;
+  }
+
+  @Override
+  public void readFully(long position, byte[] out, int offset, int length) 
throws IOException {
+    Preconditions.checkState(!closed, "Cannot read: already closed");
+    Preconditions.checkPositionIndexes(offset, offset + length, out.length);
+    String range = String.format(Locale.ROOT, "bytes=%s-%s", position, 
position + length - 1);
+    byte[] data = fetchRange(range);
+    if (data.length < length) {
+      throw new EOFException(
+          "Reached end of "
+              + BaseHttpFile.redact(location)
+              + " with "
+              + (length - data.length)
+              + " bytes left to read");
+    }
+
+    System.arraycopy(data, 0, out, offset, length);
+  }
+
+  @Override
+  public int readTail(byte[] out, int offset, int length) throws IOException {
+    Preconditions.checkState(!closed, "Cannot read: already closed");
+    Preconditions.checkPositionIndexes(offset, offset + length, out.length);
+    String range = String.format(Locale.ROOT, "bytes=-%s", length);
+    byte[] data = fetchRange(range);
+    int toCopy = Math.min(data.length, length);
+    System.arraycopy(data, 0, out, offset, toCopy);
+    return toCopy;
+  }
+
+  @Override
+  public void close() throws IOException {
+    try {
+      super.close();
+    } finally {
+      closed = true;
+      buffer = null;
+      if (closeClient) {
+        client.close(CloseMode.GRACEFUL);
+      }
+    }
+  }
+
+  private boolean inBuffer(long filePos) {
+    return filePos >= bufferFileStart && filePos < bufferFileStart + 
bufferLimit;
+  }
+
+  /**
+   * Ensures the buffer covers {@code next}. Issues a new range GET if {@code 
next} is outside the
+   * current buffer window.
+   */
+  private void ensureBuffered() throws IOException {
+    if (buffer != null && inBuffer(next)) {
+      return;
+    }
+
+    // Fetch a new chunk starting at the current position.
+    String range = String.format(Locale.ROOT, "bytes=%s-%s", next, next + 
chunkSize - 1);
+    byte[] data = fetchRange(range);
+    if (data == null || data.length == 0) {
+      buffer = null;
+      return;
+    }
+
+    buffer = data;
+    bufferFileStart = next;
+    bufferLimit = data.length;
+  }
+
+  /**
+   * Fetches a byte range from the URL, retrying transient network and HTTP 
failures with
+   * exponential backoff so a throttled or briefly unavailable endpoint is not 
repeatedly hammered.
+   */
+  private byte[] fetchRange(String range) throws IOException {
+    AtomicReference<byte[]> result = new AtomicReference<>();
+    Tasks.range(1)
+        .retry(MAX_RETRIES)
+        .exponentialBackoff(
+            MIN_RETRY_WAIT_MS, MAX_RETRY_WAIT_MS, MAX_RETRY_DURATION_MS, 
RETRY_SCALE_FACTOR)
+        .onlyRetryOn(
+            TransientHttpException.class,
+            SocketException.class,
+            SocketTimeoutException.class,
+            SSLException.class)
+        .run(ignored -> result.set(doFetchRange(range, url)), 
IOException.class);
+    return result.get();
+  }
+
+  private byte[] doFetchRange(String range, String requestUrl) throws 
IOException {
+    HttpGet request = new HttpGet(requestUrl);
+    request.setHeader(HttpHeaders.RANGE, range);
+
+    return client.execute(
+        request,
+        response -> {
+          int statusCode = response.getCode();
+          return switch (BaseHttpFile.classifyStatus(statusCode)) {
+            case OK, PARTIAL_CONTENT ->
+                response.getEntity() != null
+                    ? EntityUtils.toByteArray(response.getEntity())
+                    : new byte[0];

Review Comment:
   This doesn't look correct. A `200` on a range request means the server 
ignored `Range` and sent the whole object from byte 0, but here we treat it the 
same as `206`. These need to be handled properly as separate results.
   
   <!-- lenses: weeks-review, russell-review, blue-review -->



##########
core/src/main/java/org/apache/iceberg/io/HttpInputFile.java:
##########
@@ -0,0 +1,262 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Locale;
+import java.util.Map;
+import org.apache.hc.client5.http.classic.methods.HttpGet;
+import org.apache.hc.client5.http.impl.classic.CloseableHttpClient;
+import org.apache.hc.core5.http.ClassicHttpResponse;
+import org.apache.hc.core5.http.Header;
+import org.apache.hc.core5.http.HttpHeaders;
+import org.apache.iceberg.exceptions.ForbiddenException;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.metrics.MetricsContext;
+import 
org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.PropertyUtil;
+
+/**
+ * An {@link InputFile} backed by an HTTP URL, typically a pre-signed 
object-store URL that encodes
+ * auth in its query parameters.
+ *
+ * <p>A known content length is returned directly; otherwise it is fetched 
lazily via a {@code GET
+ * Range: bytes=0-0} request, which (unlike HEAD) works with pre-signed GET 
URLs.
+ */
+class HttpInputFile extends BaseHttpFile implements InputFile {
+  static final String READ_CHUNK_SIZE_BYTES = "io.http.read.chunk-size-bytes";
+  static final int READ_CHUNK_SIZE_BYTES_DEFAULT = 8 * 1024 * 1024;
+  static final long UNKNOWN_LENGTH = -1L;
+
+  private final int chunkSize;
+
+  private long length;
+
+  static InputFile fromLocation(
+      String location, Map<String, String> properties, MetricsContext metrics) 
{
+    return new HttpInputFile(location, location, UNKNOWN_LENGTH, properties, 
metrics);
+  }
+
+  static InputFile fromLocation(
+      String location, long length, Map<String, String> properties, 
MetricsContext metrics) {
+    return new HttpInputFile(location, location, length, properties, metrics);
+  }
+
+  /** Returns {@code true} if {@code location} is an HTTP(S) URL. */
+  static boolean isHttpUrl(String location) {
+    if (location == null) {
+      return false;
+    }
+
+    try {
+      String scheme = URI.create(location).getScheme();
+      return "https".equalsIgnoreCase(scheme) || 
"http".equalsIgnoreCase(scheme);
+    } catch (IllegalArgumentException e) {
+      return false;
+    }
+  }
+
+  private HttpInputFile(
+      String location,
+      String url,
+      long length,
+      Map<String, String> properties,
+      MetricsContext metrics) {
+    this(null, location, url, length, readChunkSize(properties), properties, 
metrics);
+  }
+
+  HttpInputFile(
+      CloseableHttpClient client,
+      String location,
+      String url,
+      int chunkSize,
+      MetricsContext metrics) {
+    this(client, location, url, UNKNOWN_LENGTH, chunkSize, null, metrics);
+  }
+
+  HttpInputFile(
+      CloseableHttpClient client,
+      String location,
+      String url,
+      long length,
+      int chunkSize,
+      MetricsContext metrics) {
+    this(client, location, url, length, chunkSize, null, metrics);
+  }
+
+  private HttpInputFile(
+      CloseableHttpClient client,
+      String location,
+      String url,
+      long length,
+      int chunkSize,
+      Map<String, String> properties,
+      MetricsContext metrics) {
+    super(client, location, url, properties, metrics);
+    Preconditions.checkArgument(
+        chunkSize > 0, "Invalid %s: %s (must be > 0)", READ_CHUNK_SIZE_BYTES, 
chunkSize);
+    this.chunkSize = chunkSize;
+    this.length = length;
+  }
+
+  @Override
+  public long getLength() {
+    if (length == UNKNOWN_LENGTH) {
+      long fetchedLength = fetchContentLength();
+      if (fetchedLength == UNKNOWN_LENGTH) {
+        throw new RuntimeIOException(
+            "Cannot determine content length for %s", 
BaseHttpFile.redact(location()));
+      }
+
+      this.length = fetchedLength;
+    }
+
+    return length;
+  }
+
+  @Override
+  public SeekableInputStream newStream() {
+    if (hasSharedClient()) {
+      return new HttpInputStream(client(), location(), url(), chunkSize, 
metrics());
+    }
+
+    return new HttpInputStream(newHttpClient(), location(), url(), chunkSize, 
metrics(), true);
+  }
+
+  @Override
+  public boolean exists() {
+    try (CloseableHttpClient httpClient = hasSharedClient() ? null : 
newHttpClient()) {
+      HttpGet request = new HttpGet(url());
+      request.setHeader(HttpHeaders.RANGE, "bytes=0-0");
+      CloseableHttpClient requestClient = hasSharedClient() ? client() : 
httpClient;
+      return requestClient.execute(
+          request,
+          response ->
+              switch (BaseHttpFile.classifyStatus(response.getCode())) {
+                case OK, PARTIAL_CONTENT -> true;
+                case RANGE_NOT_SATISFIABLE, NOT_FOUND, FORBIDDEN, RETRYABLE, 
FAILURE -> false;
+              });

Review Comment:
   I don't think `exists()` should swallow these. A `503`/`429`, or a `403` 
from an expired pre-signed URL, comes back as "does not exist", while 
`fetchContentLength` fails loudly on the same statuses. Callers make cleanup 
and validation decisions based on `exists()`. Can we return `false` only for 
`NOT_FOUND`, throw `ForbiddenException` for `FORBIDDEN`, and throw 
`RuntimeIOException` for the rest, ideally after the same retry as 
`fetchRange`? `existsReturnsFalseWhenForbidden` would need to change to expect 
the exception.
   
   <!-- lenses: weeks-review, russell-review, blue-review -->



##########
aws/src/test/java/org/apache/iceberg/aws/s3/TestS3FileIOHttpUrlDispatch.java:
##########
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.aws.s3;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Verifies that {@link S3FileIO#newInputFile} dispatches HTTP(S) locations to 
the direct HTTP(S)
+ * read path and everything else to the native S3 client, without performing 
any I/O.
+ */
+class TestS3FileIOHttpUrlDispatch {
+
+  private S3FileIO fileIO;
+
+  @AfterEach
+  void after() {
+    if (fileIO != null) {
+      fileIO.close();
+    }
+  }
+
+  @Test
+  void httpsLocationUsesHttpShortCircuit() {
+    fileIO =
+        new S3FileIO(
+            () -> {
+              throw new AssertionError("Native S3 client should not be used 
for an https:// path");
+            });
+    fileIO.initialize(ImmutableMap.of());
+
+    String url = "https://bucket.s3.amazonaws.com/key?X-Amz-Signature=abc";;
+    InputFile inputFile = fileIO.newInputFile(url);
+
+    assertThat(inputFile.location()).isEqualTo(url);
+    
assertThat(inputFile.getClass().getName()).isEqualTo("org.apache.iceberg.io.HttpInputFile");
+
+    InputFile withLength = fileIO.newInputFile(url, 100L);
+    
assertThat(withLength.getClass().getName()).isEqualTo("org.apache.iceberg.io.HttpInputFile");

Review Comment:
   I don't think the throwing supplier proves anything. The S3 client is built 
lazily, so `s3://` with the same supplier wouldn't throw either.
   
   <!-- lenses: nastra-test-review -->



##########
core/src/test/java/org/apache/iceberg/io/TestHttpInputFile.java:
##########
@@ -0,0 +1,462 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import com.sun.net.httpserver.HttpExchange;
+import com.sun.net.httpserver.HttpServer;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.net.InetAddress;
+import java.net.InetSocketAddress;
+import java.util.Arrays;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Random;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.hc.client5.http.impl.classic.CloseableHttpClient;
+import org.apache.hc.client5.http.impl.classic.HttpClients;
+import org.apache.iceberg.exceptions.ForbiddenException;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.metrics.Counter;
+import org.apache.iceberg.metrics.DefaultMetricsContext;
+import org.apache.iceberg.metrics.MetricsContext;
+import org.apache.iceberg.metrics.MetricsContext.Unit;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Exercises {@link HttpInputFile}/{@link HttpInputStream} against a real 
{@link HttpServer} that
+ * serves ranges from the request {@code Range} header, giving cloud-free 
coverage of the HTTP read
+ * path, including multi-chunk reads that cross the configured chunk-size 
boundary.
+ */
+class TestHttpInputFile {
+
+  private static final String PATH = "/object";
+  private static final int CHUNK_SIZE = 64 * 1024;
+  private static final byte[] DATA = randomBytes(CHUNK_SIZE + 1_000, 42L);
+
+  private static HttpServer server;
+  private static CloseableHttpClient client;
+  private static String url;
+  private static String missingUrl;
+  private static String forbiddenUrl;
+  private static String serverErrorUrl;
+  private static String throttledUrl;
+  private static String notImplementedUrl;
+  private static String redirectUrl;
+  private static String unknownLength200Url;
+  private static String unknownLength206Url;
+  private static final AtomicInteger REQUEST_COUNT = new AtomicInteger();
+
+  @BeforeAll
+  static void startServer() throws IOException {
+    server = HttpServer.create(new 
InetSocketAddress(InetAddress.getLoopbackAddress(), 0), 0);
+    server.createContext(PATH, TestHttpInputFile::handle);
+    server.createContext("/missing", exchange -> respondStatus(exchange, 404));
+    server.createContext("/forbidden", exchange -> respondStatus(exchange, 
403));
+    server.createContext("/server-error", exchange -> respondStatus(exchange, 
500));
+    server.createContext("/throttled", exchange -> respondStatus(exchange, 
429));
+    server.createContext("/not-implemented", exchange -> 
respondStatus(exchange, 501));
+    server.createContext("/redirect", TestHttpInputFile::respondRedirect);
+    server.createContext("/unknown-length-200", 
TestHttpInputFile::respondUnknownLength200);
+    server.createContext("/unknown-length-206", 
TestHttpInputFile::respondUnknownLength206);
+    server.start();
+    // disable the client's built-in retries so tests exercise 
HttpInputStream's own retry logic
+    client = HttpClients.custom().disableAutomaticRetries().build();
+    int port = server.getAddress().getPort();
+    url = String.format(Locale.ROOT, "http://127.0.0.1:%d%s";, port, PATH);
+    missingUrl = String.format(Locale.ROOT, "http://127.0.0.1:%d/missing";, 
port);
+    forbiddenUrl = String.format(Locale.ROOT, "http://127.0.0.1:%d/forbidden";, 
port);
+    serverErrorUrl = String.format(Locale.ROOT, 
"http://127.0.0.1:%d/server-error";, port);
+    throttledUrl = String.format(Locale.ROOT, "http://127.0.0.1:%d/throttled";, 
port);
+    notImplementedUrl = String.format(Locale.ROOT, 
"http://127.0.0.1:%d/not-implemented";, port);
+    redirectUrl = String.format(Locale.ROOT, "http://127.0.0.1:%d/redirect";, 
port);
+    unknownLength200Url =
+        String.format(Locale.ROOT, "http://127.0.0.1:%d/unknown-length-200";, 
port);
+    unknownLength206Url =
+        String.format(Locale.ROOT, "http://127.0.0.1:%d/unknown-length-206";, 
port);
+  }
+
+  @AfterAll
+  static void stopServer() throws IOException {
+    client.close();
+    server.stop(0);
+  }
+
+  @BeforeEach
+  void resetRequestCount() {
+    REQUEST_COUNT.set(0);
+  }
+
+  @Test
+  void getLengthFetchesTotalFromContentRange() {
+    HttpInputFile inputFile = httpInputFile(url, MetricsContext.nullMetrics());
+
+    assertThat(inputFile.getLength()).isEqualTo(DATA.length);
+    assertThat(REQUEST_COUNT.get()).isEqualTo(1);
+  }
+
+  @Test
+  void getLengthUsesKnownLengthWithoutRequest() {
+    HttpInputFile inputFile =
+        new HttpInputFile(
+            client, "s3://bucket/object", url, 123L, CHUNK_SIZE, 
MetricsContext.nullMetrics());
+
+    assertThat(inputFile.getLength()).isEqualTo(123L);
+    assertThat(REQUEST_COUNT.get()).isZero();
+  }
+
+  @Test
+  void readFullyReadsExactRange() throws IOException {
+    HttpInputFile inputFile = httpInputFile(url, MetricsContext.nullMetrics());
+
+    byte[] buffer = new byte[2_048];
+    try (SeekableInputStream stream = inputFile.newStream()) {
+      ((RangeReadable) stream).readFully(1_000, buffer, 0, buffer.length);
+    }
+
+    assertThat(buffer).isEqualTo(Arrays.copyOfRange(DATA, 1_000, 1_000 + 
buffer.length));
+  }
+
+  @Test
+  void readTailReadsSuffix() throws IOException {
+    HttpInputFile inputFile = httpInputFile(url, MetricsContext.nullMetrics());
+
+    byte[] buffer = new byte[512];
+    int read;
+    try (SeekableInputStream stream = inputFile.newStream()) {
+      read = ((RangeReadable) stream).readTail(buffer, 0, buffer.length);
+    }
+
+    assertThat(read).isEqualTo(buffer.length);
+    assertThat(buffer)
+        .isEqualTo(Arrays.copyOfRange(DATA, DATA.length - buffer.length, 
DATA.length));
+  }
+
+  @Test
+  void sequentialReadCrossesChunkBoundary() throws IOException {
+    HttpInputFile inputFile = httpInputFile(url, MetricsContext.nullMetrics());
+
+    byte[] actual = new byte[DATA.length];
+    try (SeekableInputStream stream = inputFile.newStream()) {
+      IOUtil.readFully(stream, actual, 0, actual.length);
+    }
+
+    assertThat(actual).isEqualTo(DATA);
+    // reading past a single chunk requires at least two range fetches
+    assertThat(REQUEST_COUNT.get()).isGreaterThanOrEqualTo(2);
+  }
+
+  @Test
+  void sequentialReadWithinSingleChunkFetchesOnce() throws IOException {
+    // a chunk larger than the object means the whole file is served by one 
range fetch
+    HttpInputFile inputFile =
+        new HttpInputFile(
+            client, "s3://bucket/object", url, DATA.length + 1_000, 
MetricsContext.nullMetrics());
+
+    byte[] actual = new byte[DATA.length];
+    try (SeekableInputStream stream = inputFile.newStream()) {
+      IOUtil.readFully(stream, actual, 0, actual.length);
+    }
+
+    assertThat(actual).isEqualTo(DATA);
+    assertThat(REQUEST_COUNT.get()).isEqualTo(1);
+  }
+
+  @Test
+  void sequentialReadTracksReadMetrics() throws IOException {
+    CachingMetricsContext metrics = new CachingMetricsContext();
+    HttpInputFile inputFile = httpInputFile(url, metrics);
+
+    byte[] actual = new byte[DATA.length];
+    try (SeekableInputStream stream = inputFile.newStream()) {
+      IOUtil.readFully(stream, actual, 0, actual.length);
+    }
+
+    Counter readBytes = metrics.counter(FileIOMetricsContext.READ_BYTES, 
Unit.BYTES);
+    Counter readOperations = 
metrics.counter(FileIOMetricsContext.READ_OPERATIONS);
+    assertThat(readBytes.value()).isEqualTo(DATA.length);
+    assertThat(readOperations.value()).isGreaterThan(0);
+  }
+
+  @Test
+  void getLengthThrowsNotFoundWhenMissing() {
+    HttpInputFile inputFile = httpInputFile(missingUrl, 
MetricsContext.nullMetrics());
+
+    assertThatThrownBy(inputFile::getLength)
+        .isInstanceOf(NotFoundException.class)
+        .hasMessageContaining("Location does not exist");

Review Comment:
   These should check the message too, e.g. `.hasMessage("Location does not 
exist: s3://bucket/object")`. The messages are deterministic, and without that 
the test still passes if the code fails for a different reason. Same for all 
other usages.
   
   <!-- lenses: nastra-test-review -->



##########
core/src/test/java/org/apache/iceberg/io/TestHttpInputFile.java:
##########
@@ -0,0 +1,462 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import com.sun.net.httpserver.HttpExchange;
+import com.sun.net.httpserver.HttpServer;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.net.InetAddress;
+import java.net.InetSocketAddress;
+import java.util.Arrays;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Random;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.hc.client5.http.impl.classic.CloseableHttpClient;
+import org.apache.hc.client5.http.impl.classic.HttpClients;
+import org.apache.iceberg.exceptions.ForbiddenException;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.metrics.Counter;
+import org.apache.iceberg.metrics.DefaultMetricsContext;
+import org.apache.iceberg.metrics.MetricsContext;
+import org.apache.iceberg.metrics.MetricsContext.Unit;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Exercises {@link HttpInputFile}/{@link HttpInputStream} against a real 
{@link HttpServer} that
+ * serves ranges from the request {@code Range} header, giving cloud-free 
coverage of the HTTP read
+ * path, including multi-chunk reads that cross the configured chunk-size 
boundary.
+ */
+class TestHttpInputFile {
+
+  private static final String PATH = "/object";
+  private static final int CHUNK_SIZE = 64 * 1024;
+  private static final byte[] DATA = randomBytes(CHUNK_SIZE + 1_000, 42L);
+
+  private static HttpServer server;
+  private static CloseableHttpClient client;
+  private static String url;
+  private static String missingUrl;
+  private static String forbiddenUrl;
+  private static String serverErrorUrl;
+  private static String throttledUrl;
+  private static String notImplementedUrl;
+  private static String redirectUrl;
+  private static String unknownLength200Url;
+  private static String unknownLength206Url;
+  private static final AtomicInteger REQUEST_COUNT = new AtomicInteger();
+
+  @BeforeAll
+  static void startServer() throws IOException {
+    server = HttpServer.create(new 
InetSocketAddress(InetAddress.getLoopbackAddress(), 0), 0);
+    server.createContext(PATH, TestHttpInputFile::handle);
+    server.createContext("/missing", exchange -> respondStatus(exchange, 404));
+    server.createContext("/forbidden", exchange -> respondStatus(exchange, 
403));
+    server.createContext("/server-error", exchange -> respondStatus(exchange, 
500));
+    server.createContext("/throttled", exchange -> respondStatus(exchange, 
429));
+    server.createContext("/not-implemented", exchange -> 
respondStatus(exchange, 501));
+    server.createContext("/redirect", TestHttpInputFile::respondRedirect);
+    server.createContext("/unknown-length-200", 
TestHttpInputFile::respondUnknownLength200);
+    server.createContext("/unknown-length-206", 
TestHttpInputFile::respondUnknownLength206);
+    server.start();
+    // disable the client's built-in retries so tests exercise 
HttpInputStream's own retry logic
+    client = HttpClients.custom().disableAutomaticRetries().build();
+    int port = server.getAddress().getPort();
+    url = String.format(Locale.ROOT, "http://127.0.0.1:%d%s";, port, PATH);
+    missingUrl = String.format(Locale.ROOT, "http://127.0.0.1:%d/missing";, 
port);
+    forbiddenUrl = String.format(Locale.ROOT, "http://127.0.0.1:%d/forbidden";, 
port);
+    serverErrorUrl = String.format(Locale.ROOT, 
"http://127.0.0.1:%d/server-error";, port);
+    throttledUrl = String.format(Locale.ROOT, "http://127.0.0.1:%d/throttled";, 
port);
+    notImplementedUrl = String.format(Locale.ROOT, 
"http://127.0.0.1:%d/not-implemented";, port);
+    redirectUrl = String.format(Locale.ROOT, "http://127.0.0.1:%d/redirect";, 
port);
+    unknownLength200Url =
+        String.format(Locale.ROOT, "http://127.0.0.1:%d/unknown-length-200";, 
port);
+    unknownLength206Url =
+        String.format(Locale.ROOT, "http://127.0.0.1:%d/unknown-length-206";, 
port);
+  }
+
+  @AfterAll
+  static void stopServer() throws IOException {
+    client.close();
+    server.stop(0);
+  }
+
+  @BeforeEach
+  void resetRequestCount() {
+    REQUEST_COUNT.set(0);
+  }
+
+  @Test
+  void getLengthFetchesTotalFromContentRange() {
+    HttpInputFile inputFile = httpInputFile(url, MetricsContext.nullMetrics());
+
+    assertThat(inputFile.getLength()).isEqualTo(DATA.length);
+    assertThat(REQUEST_COUNT.get()).isEqualTo(1);
+  }
+
+  @Test
+  void getLengthUsesKnownLengthWithoutRequest() {
+    HttpInputFile inputFile =
+        new HttpInputFile(
+            client, "s3://bucket/object", url, 123L, CHUNK_SIZE, 
MetricsContext.nullMetrics());
+
+    assertThat(inputFile.getLength()).isEqualTo(123L);
+    assertThat(REQUEST_COUNT.get()).isZero();
+  }
+
+  @Test
+  void readFullyReadsExactRange() throws IOException {
+    HttpInputFile inputFile = httpInputFile(url, MetricsContext.nullMetrics());
+
+    byte[] buffer = new byte[2_048];
+    try (SeekableInputStream stream = inputFile.newStream()) {
+      ((RangeReadable) stream).readFully(1_000, buffer, 0, buffer.length);
+    }
+
+    assertThat(buffer).isEqualTo(Arrays.copyOfRange(DATA, 1_000, 1_000 + 
buffer.length));
+  }
+
+  @Test
+  void readTailReadsSuffix() throws IOException {
+    HttpInputFile inputFile = httpInputFile(url, MetricsContext.nullMetrics());
+
+    byte[] buffer = new byte[512];
+    int read;
+    try (SeekableInputStream stream = inputFile.newStream()) {
+      read = ((RangeReadable) stream).readTail(buffer, 0, buffer.length);
+    }
+
+    assertThat(read).isEqualTo(buffer.length);
+    assertThat(buffer)
+        .isEqualTo(Arrays.copyOfRange(DATA, DATA.length - buffer.length, 
DATA.length));
+  }
+
+  @Test
+  void sequentialReadCrossesChunkBoundary() throws IOException {
+    HttpInputFile inputFile = httpInputFile(url, MetricsContext.nullMetrics());
+
+    byte[] actual = new byte[DATA.length];
+    try (SeekableInputStream stream = inputFile.newStream()) {
+      IOUtil.readFully(stream, actual, 0, actual.length);
+    }
+
+    assertThat(actual).isEqualTo(DATA);
+    // reading past a single chunk requires at least two range fetches
+    assertThat(REQUEST_COUNT.get()).isGreaterThanOrEqualTo(2);
+  }
+
+  @Test
+  void sequentialReadWithinSingleChunkFetchesOnce() throws IOException {
+    // a chunk larger than the object means the whole file is served by one 
range fetch
+    HttpInputFile inputFile =
+        new HttpInputFile(
+            client, "s3://bucket/object", url, DATA.length + 1_000, 
MetricsContext.nullMetrics());
+
+    byte[] actual = new byte[DATA.length];
+    try (SeekableInputStream stream = inputFile.newStream()) {
+      IOUtil.readFully(stream, actual, 0, actual.length);
+    }
+
+    assertThat(actual).isEqualTo(DATA);
+    assertThat(REQUEST_COUNT.get()).isEqualTo(1);
+  }
+
+  @Test
+  void sequentialReadTracksReadMetrics() throws IOException {
+    CachingMetricsContext metrics = new CachingMetricsContext();
+    HttpInputFile inputFile = httpInputFile(url, metrics);
+
+    byte[] actual = new byte[DATA.length];
+    try (SeekableInputStream stream = inputFile.newStream()) {
+      IOUtil.readFully(stream, actual, 0, actual.length);
+    }
+
+    Counter readBytes = metrics.counter(FileIOMetricsContext.READ_BYTES, 
Unit.BYTES);
+    Counter readOperations = 
metrics.counter(FileIOMetricsContext.READ_OPERATIONS);
+    assertThat(readBytes.value()).isEqualTo(DATA.length);
+    assertThat(readOperations.value()).isGreaterThan(0);
+  }
+
+  @Test
+  void getLengthThrowsNotFoundWhenMissing() {
+    HttpInputFile inputFile = httpInputFile(missingUrl, 
MetricsContext.nullMetrics());
+
+    assertThatThrownBy(inputFile::getLength)
+        .isInstanceOf(NotFoundException.class)
+        .hasMessageContaining("Location does not exist");
+  }
+
+  @Test
+  void getLengthThrowsForbiddenWhenForbidden() {
+    HttpInputFile inputFile = httpInputFile(forbiddenUrl, 
MetricsContext.nullMetrics());
+
+    assertThatThrownBy(inputFile::getLength)
+        .isInstanceOf(ForbiddenException.class)
+        .hasMessageContaining("Access forbidden");
+  }
+
+  @Test
+  void getLengthThrowsWhenLengthCannotBeDetermined() {
+    assertThatThrownBy(
+            () -> httpInputFile(unknownLength200Url, 
MetricsContext.nullMetrics()).getLength())
+        .isInstanceOf(RuntimeIOException.class)
+        .hasMessageContaining("Cannot determine content length");
+
+    assertThatThrownBy(
+            () -> httpInputFile(unknownLength206Url, 
MetricsContext.nullMetrics()).getLength())
+        .isInstanceOf(RuntimeIOException.class)
+        .hasMessageContaining("Cannot determine content length");
+  }
+
+  @Test
+  void readThrowsForbiddenWithoutRetry() {
+    HttpInputFile inputFile = httpInputFile(forbiddenUrl, 
MetricsContext.nullMetrics());
+
+    assertThatThrownBy(
+            () -> {
+              try (SeekableInputStream stream = inputFile.newStream()) {
+                ((RangeReadable) stream).readFully(0, new byte[16], 0, 16);
+              }
+            })
+        .isInstanceOf(ForbiddenException.class)
+        .hasMessageContaining("Access forbidden");
+    assertThat(REQUEST_COUNT.get()).isEqualTo(1);
+  }
+
+  @Test
+  void readRetriesOnServerErrorThenFails() {
+    HttpInputFile inputFile = httpInputFile(serverErrorUrl, 
MetricsContext.nullMetrics());
+
+    assertThatThrownBy(
+            () -> {
+              try (SeekableInputStream stream = inputFile.newStream()) {
+                ((RangeReadable) stream).readFully(0, new byte[16], 0, 16);
+              }
+            })
+        .isInstanceOf(IOException.class)
+        .hasMessageContaining("Transient HTTP");
+    assertThat(REQUEST_COUNT.get()).isGreaterThan(1);

Review Comment:
   `isGreaterThan(1)` passes whether we retried once or ten times. The result 
should be explicit.
   
   <!-- lenses: nastra-test-review -->



##########
core/src/test/java/org/apache/iceberg/io/TestHttpInputFile.java:
##########
@@ -0,0 +1,462 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import com.sun.net.httpserver.HttpExchange;
+import com.sun.net.httpserver.HttpServer;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.net.InetAddress;
+import java.net.InetSocketAddress;
+import java.util.Arrays;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Random;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.hc.client5.http.impl.classic.CloseableHttpClient;
+import org.apache.hc.client5.http.impl.classic.HttpClients;
+import org.apache.iceberg.exceptions.ForbiddenException;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.metrics.Counter;
+import org.apache.iceberg.metrics.DefaultMetricsContext;
+import org.apache.iceberg.metrics.MetricsContext;
+import org.apache.iceberg.metrics.MetricsContext.Unit;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Exercises {@link HttpInputFile}/{@link HttpInputStream} against a real 
{@link HttpServer} that
+ * serves ranges from the request {@code Range} header, giving cloud-free 
coverage of the HTTP read
+ * path, including multi-chunk reads that cross the configured chunk-size 
boundary.
+ */
+class TestHttpInputFile {
+
+  private static final String PATH = "/object";
+  private static final int CHUNK_SIZE = 64 * 1024;
+  private static final byte[] DATA = randomBytes(CHUNK_SIZE + 1_000, 42L);
+
+  private static HttpServer server;
+  private static CloseableHttpClient client;
+  private static String url;
+  private static String missingUrl;
+  private static String forbiddenUrl;
+  private static String serverErrorUrl;
+  private static String throttledUrl;
+  private static String notImplementedUrl;
+  private static String redirectUrl;
+  private static String unknownLength200Url;
+  private static String unknownLength206Url;
+  private static final AtomicInteger REQUEST_COUNT = new AtomicInteger();
+
+  @BeforeAll
+  static void startServer() throws IOException {
+    server = HttpServer.create(new 
InetSocketAddress(InetAddress.getLoopbackAddress(), 0), 0);
+    server.createContext(PATH, TestHttpInputFile::handle);
+    server.createContext("/missing", exchange -> respondStatus(exchange, 404));
+    server.createContext("/forbidden", exchange -> respondStatus(exchange, 
403));
+    server.createContext("/server-error", exchange -> respondStatus(exchange, 
500));
+    server.createContext("/throttled", exchange -> respondStatus(exchange, 
429));
+    server.createContext("/not-implemented", exchange -> 
respondStatus(exchange, 501));
+    server.createContext("/redirect", TestHttpInputFile::respondRedirect);
+    server.createContext("/unknown-length-200", 
TestHttpInputFile::respondUnknownLength200);
+    server.createContext("/unknown-length-206", 
TestHttpInputFile::respondUnknownLength206);
+    server.start();
+    // disable the client's built-in retries so tests exercise 
HttpInputStream's own retry logic
+    client = HttpClients.custom().disableAutomaticRetries().build();
+    int port = server.getAddress().getPort();
+    url = String.format(Locale.ROOT, "http://127.0.0.1:%d%s";, port, PATH);
+    missingUrl = String.format(Locale.ROOT, "http://127.0.0.1:%d/missing";, 
port);
+    forbiddenUrl = String.format(Locale.ROOT, "http://127.0.0.1:%d/forbidden";, 
port);
+    serverErrorUrl = String.format(Locale.ROOT, 
"http://127.0.0.1:%d/server-error";, port);
+    throttledUrl = String.format(Locale.ROOT, "http://127.0.0.1:%d/throttled";, 
port);
+    notImplementedUrl = String.format(Locale.ROOT, 
"http://127.0.0.1:%d/not-implemented";, port);
+    redirectUrl = String.format(Locale.ROOT, "http://127.0.0.1:%d/redirect";, 
port);
+    unknownLength200Url =
+        String.format(Locale.ROOT, "http://127.0.0.1:%d/unknown-length-200";, 
port);
+    unknownLength206Url =
+        String.format(Locale.ROOT, "http://127.0.0.1:%d/unknown-length-206";, 
port);
+  }
+
+  @AfterAll
+  static void stopServer() throws IOException {
+    client.close();
+    server.stop(0);
+  }
+
+  @BeforeEach
+  void resetRequestCount() {
+    REQUEST_COUNT.set(0);
+  }
+
+  @Test
+  void getLengthFetchesTotalFromContentRange() {
+    HttpInputFile inputFile = httpInputFile(url, MetricsContext.nullMetrics());
+
+    assertThat(inputFile.getLength()).isEqualTo(DATA.length);
+    assertThat(REQUEST_COUNT.get()).isEqualTo(1);
+  }
+
+  @Test
+  void getLengthUsesKnownLengthWithoutRequest() {
+    HttpInputFile inputFile =
+        new HttpInputFile(
+            client, "s3://bucket/object", url, 123L, CHUNK_SIZE, 
MetricsContext.nullMetrics());
+
+    assertThat(inputFile.getLength()).isEqualTo(123L);
+    assertThat(REQUEST_COUNT.get()).isZero();
+  }
+
+  @Test
+  void readFullyReadsExactRange() throws IOException {
+    HttpInputFile inputFile = httpInputFile(url, MetricsContext.nullMetrics());
+
+    byte[] buffer = new byte[2_048];
+    try (SeekableInputStream stream = inputFile.newStream()) {
+      ((RangeReadable) stream).readFully(1_000, buffer, 0, buffer.length);
+    }
+
+    assertThat(buffer).isEqualTo(Arrays.copyOfRange(DATA, 1_000, 1_000 + 
buffer.length));
+  }
+
+  @Test
+  void readTailReadsSuffix() throws IOException {
+    HttpInputFile inputFile = httpInputFile(url, MetricsContext.nullMetrics());
+
+    byte[] buffer = new byte[512];
+    int read;
+    try (SeekableInputStream stream = inputFile.newStream()) {
+      read = ((RangeReadable) stream).readTail(buffer, 0, buffer.length);
+    }
+
+    assertThat(read).isEqualTo(buffer.length);
+    assertThat(buffer)
+        .isEqualTo(Arrays.copyOfRange(DATA, DATA.length - buffer.length, 
DATA.length));
+  }
+
+  @Test
+  void sequentialReadCrossesChunkBoundary() throws IOException {
+    HttpInputFile inputFile = httpInputFile(url, MetricsContext.nullMetrics());
+
+    byte[] actual = new byte[DATA.length];
+    try (SeekableInputStream stream = inputFile.newStream()) {
+      IOUtil.readFully(stream, actual, 0, actual.length);
+    }
+
+    assertThat(actual).isEqualTo(DATA);
+    // reading past a single chunk requires at least two range fetches
+    assertThat(REQUEST_COUNT.get()).isGreaterThanOrEqualTo(2);
+  }
+
+  @Test
+  void sequentialReadWithinSingleChunkFetchesOnce() throws IOException {
+    // a chunk larger than the object means the whole file is served by one 
range fetch
+    HttpInputFile inputFile =
+        new HttpInputFile(
+            client, "s3://bucket/object", url, DATA.length + 1_000, 
MetricsContext.nullMetrics());
+
+    byte[] actual = new byte[DATA.length];
+    try (SeekableInputStream stream = inputFile.newStream()) {
+      IOUtil.readFully(stream, actual, 0, actual.length);
+    }
+
+    assertThat(actual).isEqualTo(DATA);
+    assertThat(REQUEST_COUNT.get()).isEqualTo(1);
+  }
+
+  @Test
+  void sequentialReadTracksReadMetrics() throws IOException {
+    CachingMetricsContext metrics = new CachingMetricsContext();
+    HttpInputFile inputFile = httpInputFile(url, metrics);
+
+    byte[] actual = new byte[DATA.length];
+    try (SeekableInputStream stream = inputFile.newStream()) {
+      IOUtil.readFully(stream, actual, 0, actual.length);
+    }
+
+    Counter readBytes = metrics.counter(FileIOMetricsContext.READ_BYTES, 
Unit.BYTES);
+    Counter readOperations = 
metrics.counter(FileIOMetricsContext.READ_OPERATIONS);
+    assertThat(readBytes.value()).isEqualTo(DATA.length);
+    assertThat(readOperations.value()).isGreaterThan(0);
+  }
+
+  @Test
+  void getLengthThrowsNotFoundWhenMissing() {
+    HttpInputFile inputFile = httpInputFile(missingUrl, 
MetricsContext.nullMetrics());
+
+    assertThatThrownBy(inputFile::getLength)
+        .isInstanceOf(NotFoundException.class)
+        .hasMessageContaining("Location does not exist");
+  }
+
+  @Test
+  void getLengthThrowsForbiddenWhenForbidden() {
+    HttpInputFile inputFile = httpInputFile(forbiddenUrl, 
MetricsContext.nullMetrics());
+
+    assertThatThrownBy(inputFile::getLength)
+        .isInstanceOf(ForbiddenException.class)
+        .hasMessageContaining("Access forbidden");
+  }
+
+  @Test
+  void getLengthThrowsWhenLengthCannotBeDetermined() {
+    assertThatThrownBy(
+            () -> httpInputFile(unknownLength200Url, 
MetricsContext.nullMetrics()).getLength())
+        .isInstanceOf(RuntimeIOException.class)
+        .hasMessageContaining("Cannot determine content length");
+
+    assertThatThrownBy(
+            () -> httpInputFile(unknownLength206Url, 
MetricsContext.nullMetrics()).getLength())
+        .isInstanceOf(RuntimeIOException.class)
+        .hasMessageContaining("Cannot determine content length");
+  }

Review Comment:
   This covers two separate failures, a 200 without Content-Length and a 206 
with `bytes 0-0/*`. Can we split it into two tests so a failure tells you which 
one broke?
   
   <!-- lenses: nastra-test-review -->



##########
core/src/main/java/org/apache/iceberg/io/BaseFileIO.java:
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import org.apache.iceberg.metrics.MetricsContext;
+
+/** Base {@link FileIO} implementation with shared handling for HTTP(S) input 
locations. */
+public abstract class BaseFileIO implements FileIO {
+
+  @Override
+  public final InputFile newInputFile(String path) {

Review Comment:
   This should not be public abstract class and the method should not be final.
   
   We don't want to expose this.
   <!-- lenses: russell-review, blue-review -->



##########
core/src/main/java/org/apache/iceberg/io/BaseHttpFile.java:
##########
@@ -0,0 +1,235 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.net.URI;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import org.apache.hc.client5.http.config.ConnectionConfig;
+import org.apache.hc.client5.http.config.RequestConfig;
+import org.apache.hc.client5.http.impl.classic.CloseableHttpClient;
+import org.apache.hc.client5.http.impl.classic.HttpClientBuilder;
+import org.apache.hc.client5.http.impl.classic.HttpClients;
+import 
org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManagerBuilder;
+import org.apache.hc.client5.http.io.HttpClientConnectionManager;
+import org.apache.hc.core5.http.HttpStatus;
+import org.apache.iceberg.metrics.MetricsContext;
+import 
org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.PropertyUtil;
+import org.apache.iceberg.util.SerializableMap;
+
+abstract class BaseHttpFile {
+  static final String CONNECTION_TIMEOUT_MS = "io.http.connection-timeout-ms";
+  static final String SOCKET_TIMEOUT_MS = "io.http.socket-timeout-ms";
+  static final String CONNECTION_ACQUISITION_TIMEOUT_MS =
+      "io.http.connection-acquisition-timeout-ms";
+  static final String MAX_CONNECTIONS = "io.http.max-connections";
+  static final String MAX_CONNECTIONS_PER_ROUTE = 
"io.http.connections-per-route";

Review Comment:
   I don't like that we're exposing so many properties without clear contract 
as to why we need them.  Is this just because we expose them in the other http 
client implementations?
   
   <!-- lenses: weeks-review, blue-review -->



##########
core/src/main/java/org/apache/iceberg/io/HttpInputFile.java:
##########
@@ -0,0 +1,262 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Locale;
+import java.util.Map;
+import org.apache.hc.client5.http.classic.methods.HttpGet;
+import org.apache.hc.client5.http.impl.classic.CloseableHttpClient;
+import org.apache.hc.core5.http.ClassicHttpResponse;
+import org.apache.hc.core5.http.Header;
+import org.apache.hc.core5.http.HttpHeaders;
+import org.apache.iceberg.exceptions.ForbiddenException;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.metrics.MetricsContext;
+import 
org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.PropertyUtil;
+
+/**
+ * An {@link InputFile} backed by an HTTP URL, typically a pre-signed 
object-store URL that encodes
+ * auth in its query parameters.
+ *
+ * <p>A known content length is returned directly; otherwise it is fetched 
lazily via a {@code GET
+ * Range: bytes=0-0} request, which (unlike HEAD) works with pre-signed GET 
URLs.
+ */
+class HttpInputFile extends BaseHttpFile implements InputFile {
+  static final String READ_CHUNK_SIZE_BYTES = "io.http.read.chunk-size-bytes";
+  static final int READ_CHUNK_SIZE_BYTES_DEFAULT = 8 * 1024 * 1024;
+  static final long UNKNOWN_LENGTH = -1L;
+
+  private final int chunkSize;
+
+  private long length;
+
+  static InputFile fromLocation(
+      String location, Map<String, String> properties, MetricsContext metrics) 
{
+    return new HttpInputFile(location, location, UNKNOWN_LENGTH, properties, 
metrics);
+  }
+
+  static InputFile fromLocation(
+      String location, long length, Map<String, String> properties, 
MetricsContext metrics) {
+    return new HttpInputFile(location, location, length, properties, metrics);
+  }
+
+  /** Returns {@code true} if {@code location} is an HTTP(S) URL. */
+  static boolean isHttpUrl(String location) {
+    if (location == null) {
+      return false;
+    }
+
+    try {
+      String scheme = URI.create(location).getScheme();

Review Comment:
   This now runs on every `newInputFile` location, including every `s3://` 
path, so parsing through `java.net.URI`, which is not something we should rely 
on.  In fact, we strictly avoid URI because there are issues with how it parses 
paths.  We should not use URI for parsing.
   
   <!-- lenses: weeks-review -->



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to