wombatu-kun opened a new issue, #18943:
URL: https://github.com/apache/iceberg/issues/18943

   ### Problem
   
   Since 1.10.0, `s3.preload-client-enabled` 
(`S3FileIOProperties.PRELOAD_CLIENT_ENABLED`) no longer creates the S3 client 
in `S3FileIO.initialize()`, although its Javadoc still says: "This flag 
controls whether the S3 client will be initialized during the S3FileIO 
initialization, instead of default lazy initialization upon use."
   
   Up to 1.9.x, `initialize()` called `client()` when the flag was set. #12799 
moved client creation into `PrefixedS3Client` and the flag check into its 
constructor, but `S3FileIO` creates its `PrefixedS3Client`s lazily in 
`clientByPrefix()` on the first file operation. So `initialize()` no longer 
creates any client, and the flag only makes the sync client get built together 
with the client map. Nothing tests the eager behavior: `TestS3FileIOProperties` 
only checks that the flag is parsed.
   
   ### Reproduction
   
   This test fails on main:
   
   ```java
   private static final AtomicInteger S3_CLIENTS = new AtomicInteger();
   
   @Test
   void preloadCreatesClientInInitialize() {
     try (S3FileIO fileIO = new S3FileIO()) {
       fileIO.initialize(
           ImmutableMap.of(
               S3FileIOProperties.CLIENT_FACTORY,
               CountingFactory.class.getName(),
               S3FileIOProperties.PRELOAD_CLIENT_ENABLED,
               "true"));
       assertThat(S3_CLIENTS).hasValue(1);
     }
   }
   
   public static class CountingFactory implements S3FileIOAwsClientFactory {
     @Override
     public S3Client s3() {
       S3_CLIENTS.incrementAndGet();
       return Mockito.mock(S3Client.class);
     }
   
     @Override
     public S3AsyncClient s3Async() {
       return Mockito.mock(S3AsyncClient.class);
     }
   
     @Override
     public void initialize(Map<String, String> properties) {}
   }
   ```
   
   ```
   java.lang.AssertionError:
   Expecting AtomicInteger(0) to have value:
     1
   but did not.
   ```
   
   ### Lake Formation
   
   The only in-tree user is `GlueCatalog.newTableOps`, which sets the flag for 
every table when `glue.lakeformation-enabled` is true (#5756). The eager 
behavior is also what broke #10226 (create table) and #14766 (loading a missing 
table from Kafka Connect): `newTableOps` initializes the table's FileIO right 
away (today through `FileIOTracker.track`), so `initialize()` reached 
`LakeFormationAwsClientFactory.isTableRegisteredWithLakeFormation()` before the 
Glue table existed and threw Glue's `EntityNotFoundException`. Since #12799, 
`S3FileIO.initialize()` no longer calls the factory's `s3()`, so that path is 
gone.
   
   ### Question
   
   Does Lake Formation still need the client to be created in `initialize()`?
   
   - If it does: restore the `client()` call in `S3FileIO.initialize()` 
(storage credentials are already set by then, because `CatalogUtil.loadFileIO` 
calls `setCredentials` first), cover it with the test above, and keep 
`GlueCatalog` from requesting preload for a table that does not exist yet, so 
#10226 does not come back.
   - If it does not: deprecate the flag and stop setting it in `GlueCatalog`.
   
   cc @xiaoxuandev @jackye1995 @singhpk234
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to