wombatu-kun opened a new issue, #18943:
URL: https://github.com/apache/iceberg/issues/18943
### Problem
Since 1.10.0, `s3.preload-client-enabled`
(`S3FileIOProperties.PRELOAD_CLIENT_ENABLED`) no longer creates the S3 client
in `S3FileIO.initialize()`, although its Javadoc still says: "This flag
controls whether the S3 client will be initialized during the S3FileIO
initialization, instead of default lazy initialization upon use."
Up to 1.9.x, `initialize()` called `client()` when the flag was set. #12799
moved client creation into `PrefixedS3Client` and the flag check into its
constructor, but `S3FileIO` creates its `PrefixedS3Client`s lazily in
`clientByPrefix()` on the first file operation. So `initialize()` no longer
creates any client, and the flag only makes the sync client get built together
with the client map. Nothing tests the eager behavior: `TestS3FileIOProperties`
only checks that the flag is parsed.
### Reproduction
This test fails on main:
```java
private static final AtomicInteger S3_CLIENTS = new AtomicInteger();
@Test
void preloadCreatesClientInInitialize() {
try (S3FileIO fileIO = new S3FileIO()) {
fileIO.initialize(
ImmutableMap.of(
S3FileIOProperties.CLIENT_FACTORY,
CountingFactory.class.getName(),
S3FileIOProperties.PRELOAD_CLIENT_ENABLED,
"true"));
assertThat(S3_CLIENTS).hasValue(1);
}
}
public static class CountingFactory implements S3FileIOAwsClientFactory {
@Override
public S3Client s3() {
S3_CLIENTS.incrementAndGet();
return Mockito.mock(S3Client.class);
}
@Override
public S3AsyncClient s3Async() {
return Mockito.mock(S3AsyncClient.class);
}
@Override
public void initialize(Map<String, String> properties) {}
}
```
```
java.lang.AssertionError:
Expecting AtomicInteger(0) to have value:
1
but did not.
```
### Lake Formation
The only in-tree user is `GlueCatalog.newTableOps`, which sets the flag for
every table when `glue.lakeformation-enabled` is true (#5756). The eager
behavior is also what broke #10226 (create table) and #14766 (loading a missing
table from Kafka Connect): `newTableOps` initializes the table's FileIO right
away (today through `FileIOTracker.track`), so `initialize()` reached
`LakeFormationAwsClientFactory.isTableRegisteredWithLakeFormation()` before the
Glue table existed and threw Glue's `EntityNotFoundException`. Since #12799,
`S3FileIO.initialize()` no longer calls the factory's `s3()`, so that path is
gone.
### Question
Does Lake Formation still need the client to be created in `initialize()`?
- If it does: restore the `client()` call in `S3FileIO.initialize()`
(storage credentials are already set by then, because `CatalogUtil.loadFileIO`
calls `setCredentials` first), cover it with the test above, and keep
`GlueCatalog` from requesting preload for a table that does not exist yet, so
#10226 does not come back.
- If it does not: deprecate the flag and stop setting it in `GlueCatalog`.
cc @xiaoxuandev @jackye1995 @singhpk234
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]