sundapeng opened a new pull request, #10991:
URL: https://github.com/apache/paimon/pull/10991

   ### Purpose
   
   `FileIndexerFactoryUtils` and `GlobalIndexerFactoryUtils` discover their 
factories once, in a static initializer, with `ServiceLoader.load(Class)`, i.e. 
with the context classloader of whichever thread touches the class first. If 
that classloader cannot see Paimon's service files, the registry stays empty 
for the life of the classloader and every later lookup fails.
   
   Since #10850 moved schema validation into the first `FileStoreTable#store()` 
call, a Flink job hits this when its `OperatorCoordinator` builds a 
`TableWriteCoordinator` in `start()`. That runs on the JobMaster RPC thread, 
whose context classloader is Flink's own, so the JobMaster fails to start for 
any table that declares a file index:
   
   ```
   Caused by: java.lang.RuntimeException: Can't find file index for type: 
bloom-filter
        at 
org.apache.paimon.fileindex.FileIndexerFactoryUtils.load(FileIndexerFactoryUtils.java:56)
        at 
org.apache.paimon.schema.SchemaValidation.validateFileIndex(SchemaValidation.java:1164)
        at 
org.apache.paimon.schema.SchemaValidation.validateTableSchema(SchemaValidation.java:409)
        at 
org.apache.paimon.table.AbstractFileStoreTable.runtimeStore(AbstractFileStoreTable.java:362)
        at 
org.apache.paimon.table.PrimaryKeyFileStoreTable.store(PrimaryKeyFileStoreTable.java:78)
        at 
org.apache.paimon.flink.sink.coordinator.TableWriteCoordinator.<init>(TableWriteCoordinator.java:71)
        ...
        at 
org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder.start(OperatorCoordinatorHolder.java:204)
   ```
   
   Before #10850 the coordinator path never touched these registries, so the 
same jobs started fine.
   
   This PR discovers the factories with the interfaces' own classloader, as 
`FileIO#discoverLoaders`, `FileFormat#fromIdentifier` and the field aggregator 
factories already do.
   
   ### Tests
   
   - `IndexerFactoryUtilsClassLoaderTest` initializes each utils class in a 
fresh classloader while the context classloader sees nothing, then loads 
`bloom-filter` and `btree`. Both cases fail without the change with the errors 
above, and pass with it.
   - `org.apache.paimon.fileindex.**` and `org.apache.paimon.globalindex.**` 
tests in paimon-common pass (666 tests).
   - Also reproduced against a shaded Flink job jar: deserializing a 
primary-key table with `file-index.bloom-filter.columns` in a new classloader 
and calling `store()` under a context classloader without Paimon fails before 
the change and succeeds after it.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to