Jzjsnow opened a new issue, #3599:
URL: https://github.com/apache/amoro/issues/3599
### What happened?
When I set `flink-conf.classloader.resolve-order=child-first` for the flink
optimizers and the bloom-filter was triggered to to filter eq-delete files, I
encountered the following exception, which blocks the optimization process.
> 2025-05-16 15:40:57,409 ERROR
org.apache.amoro.optimizer.common.OptimizerExecutor [] - Optimizer
executor[0] executed task[OptimizingTaskId(processId:1747381088029, taskId:1)]
failed and cost 876
java.lang.RuntimeException: Run with ugi request failed.
at
org.apache.amoro.table.TableMetaStore.call(TableMetaStore.java:261)
~[optimizer-job.jar:?]
at
org.apache.amoro.table.TableMetaStore.lambda$doAs$0(TableMetaStore.java:231)
~[optimizer-job.jar:?]
at java.security.AccessController.doPrivileged(Native Method)
~[?:1.8.0_352]
at javax.security.auth.Subject.doAs(Subject.java:360) ~[?:1.8.0_352]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1867)
~[hadoop-common-3.3.3.jar:?]
at
org.apache.amoro.table.TableMetaStore.doAs(TableMetaStore.java:231)
~[optimizer-job.jar:?]
at
org.apache.amoro.io.AuthenticatedHadoopFileIO.doAs(AuthenticatedHadoopFileIO.java:201)
~[optimizer-job.jar:?]
at
org.apache.amoro.optimizing.AbstractRewriteFilesExecutor.execute(AbstractRewriteFilesExecutor.java:108)
~[optimizer-job.jar:?]
at
org.apache.amoro.optimizing.AbstractRewriteFilesExecutor.execute(AbstractRewriteFilesExecutor.java:64)
~[optimizer-job.jar:?]
at
org.apache.amoro.optimizer.common.OptimizerExecutor.executeTask(OptimizerExecutor.java:150)
~[optimizer-job.jar:?]
at
org.apache.amoro.optimizer.flink.FlinkOptimizerExecutor.executeTask(FlinkOptimizerExecutor.java:70)
~[optimizer-job.jar:?]
at
org.apache.amoro.optimizer.common.OptimizerExecutor.start(OptimizerExecutor.java:53)
~[optimizer-job.jar:?]
at
org.apache.amoro.optimizer.flink.FlinkExecutor.lambda$open$0(FlinkExecutor.java:64)
~[optimizer-job.jar:?]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_352]
Caused by: java.lang.LinkageError: loader constraint violation: when
resolving method
"com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.setFallbackInstantiatorStrategy(Lorg/objenesis/strategy/InstantiatorStrategy;)V"
the class loader (instance of org/apache/flink/util/ChildFirstClassLoader) of
the current
class, org/apache/amoro/utils/SerializationUtil$KryoInstantiator, and the
class loader (instance of sun/misc/Launcher$AppClassLoader) for the method's
defining class, com/esotericsoftware/kryo/Kryo$DefaultInstantiatorStrategy,
have different Class objects for the type
org/objenesis/strategy/InstantiatorStrategy us
ed in the signature
at
org.apache.amoro.utils.SerializationUtil$KryoInstantiator.newKryo(SerializationUtil.java:128)
~[optimizer-job.jar:?]
at
org.apache.amoro.utils.SerializationUtil$KryoSerializerInstance.<init>(SerializationUtil.java:100)
~[optimizer-job.jar:?]
at
java.lang.ThreadLocal$SuppliedThreadLocal.initialValue(ThreadLocal.java:305)
~[?:1.8.0_352]
at java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:195)
~[?:1.8.0_352]
at java.lang.ThreadLocal.get(ThreadLocal.java:172) ~[?:1.8.0_352]
at
org.apache.amoro.utils.SerializationUtil.kryoSerialize(SerializationUtil.java:73)
~[optimizer-job.jar:?]
at
org.apache.amoro.io.reader.StructLikeFunnel.funnel(StructLikeFunnel.java:35)
~[optimizer-job.jar:?]
at
org.apache.amoro.io.reader.StructLikeFunnel.funnel(StructLikeFunnel.java:27)
~[optimizer-job.jar:?]
at
org.apache.amoro.shade.guava32.com.google.common.hash.AbstractHasher.putObject(AbstractHasher.java:133)
~[optimizer-job.jar:?]
at
org.apache.amoro.shade.guava32.com.google.common.hash.AbstractHashFunction.hashObject(AbstractHashFunction.java:36)
~[optimizer-job.jar:?]
at
org.apache.amoro.shade.guava32.com.google.common.hash.BloomFilterStrategies$2.put(BloomFilterStrategies.java:109)
~[optimizer-job.jar:?]
at
org.apache.amoro.shade.guava32.com.google.common.hash.BloomFilter.put(BloomFilter.java:176)
~[optimizer-job.jar:?]
at
org.apache.amoro.io.reader.CombinedDeleteFilter.lambda$initializeBloomFilter$6(CombinedDeleteFilter.java:299)
~[optimizer-job.jar:?]
at java.util.HashMap.forEach(HashMap.java:1290) ~[?:1.8.0_352]
at
org.apache.amoro.io.reader.CombinedDeleteFilter.initializeBloomFilter(CombinedDeleteFilter.java:285)
~[optimizer-job.jar:?]
at
org.apache.amoro.io.reader.CombinedDeleteFilter.applyEqDeletes(CombinedDeleteFilter.java:247)
~[optimizer-job.jar:?]
at
org.apache.amoro.io.reader.CombinedDeleteFilter.applyEqDeletes(CombinedDeleteFilter.java:379)
~[optimizer-job.jar:?]
at
org.apache.amoro.io.reader.CombinedDeleteFilter.filter(CombinedDeleteFilter.java:216)
~[optimizer-job.jar:?]
at
org.apache.amoro.io.reader.GenericCombinedIcebergDataReader.readData(GenericCombinedIcebergDataReader.java:124)
~[optimizer-job.jar:?]
at
org.apache.amoro.optimizing.AbstractRewriteFilesExecutor.rewriterDataFiles(AbstractRewriteFilesExecutor.java:149)
~[optimizer-job.jar:?]
at
org.apache.amoro.table.TableMetaStore.call(TableMetaStore.java:256)
~[optimizer-job.jar:?]
... 13 more
### Affects Versions
master, 0.7.x
### What table formats are you seeing the problem on?
Iceberg
### What engines are you seeing the problem on?
Optimizer, Flink
### How to reproduce
- flink 1.18/1.16
- set `flink-conf.classloader.resolve-order=child-first` and start a flink
optimizer
- fire the bloom-filter to filter eq-delete files
### Relevant log output
```shell
2025-05-16 15:40:57,409 ERROR
org.apache.amoro.optimizer.common.OptimizerExecutor [] - Optimizer
executor[0] executed task[OptimizingTaskId(processId:1747381088029, taskId:1)]
failed and cost 876
java.lang.RuntimeException: Run with ugi request failed.
at
org.apache.amoro.table.TableMetaStore.call(TableMetaStore.java:261)
~[optimizer-job.jar:?]
at
org.apache.amoro.table.TableMetaStore.lambda$doAs$0(TableMetaStore.java:231)
~[optimizer-job.jar:?]
at java.security.AccessController.doPrivileged(Native Method)
~[?:1.8.0_352]
at javax.security.auth.Subject.doAs(Subject.java:360) ~[?:1.8.0_352]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1867)
~[hadoop-common-3.3.3.jar:?]
at
org.apache.amoro.table.TableMetaStore.doAs(TableMetaStore.java:231)
~[optimizer-job.jar:?]
at
org.apache.amoro.io.AuthenticatedHadoopFileIO.doAs(AuthenticatedHadoopFileIO.java:201)
~[optimizer-job.jar:?]
at
org.apache.amoro.optimizing.AbstractRewriteFilesExecutor.execute(AbstractRewriteFilesExecutor.java:108)
~[optimizer-job.jar:?]
at
org.apache.amoro.optimizing.AbstractRewriteFilesExecutor.execute(AbstractRewriteFilesExecutor.java:64)
~[optimizer-job.jar:?]
at
org.apache.amoro.optimizer.common.OptimizerExecutor.executeTask(OptimizerExecutor.java:150)
~[optimizer-job.jar:?]
at
org.apache.amoro.optimizer.flink.FlinkOptimizerExecutor.executeTask(FlinkOptimizerExecutor.java:70)
~[optimizer-job.jar:?]
at
org.apache.amoro.optimizer.common.OptimizerExecutor.start(OptimizerExecutor.java:53)
~[optimizer-job.jar:?]
at
org.apache.amoro.optimizer.flink.FlinkExecutor.lambda$open$0(FlinkExecutor.java:64)
~[optimizer-job.jar:?]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_352]
Caused by: java.lang.LinkageError: loader constraint violation: when
resolving method
"com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.setFallbackInstantiatorStrategy(Lorg/objenesis/strategy/InstantiatorStrategy;)V"
the class loader (instance of org/apache/flink/util/ChildFirstClassLoader) of
the current
class, org/apache/amoro/utils/SerializationUtil$KryoInstantiator, and the
class loader (instance of sun/misc/Launcher$AppClassLoader) for the method's
defining class, com/esotericsoftware/kryo/Kryo$DefaultInstantiatorStrategy,
have different Class objects for the type
org/objenesis/strategy/InstantiatorStrategy us
ed in the signature
at
org.apache.amoro.utils.SerializationUtil$KryoInstantiator.newKryo(SerializationUtil.java:128)
~[optimizer-job.jar:?]
at
org.apache.amoro.utils.SerializationUtil$KryoSerializerInstance.<init>(SerializationUtil.java:100)
~[optimizer-job.jar:?]
at
java.lang.ThreadLocal$SuppliedThreadLocal.initialValue(ThreadLocal.java:305)
~[?:1.8.0_352]
at java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:195)
~[?:1.8.0_352]
at java.lang.ThreadLocal.get(ThreadLocal.java:172) ~[?:1.8.0_352]
at
org.apache.amoro.utils.SerializationUtil.kryoSerialize(SerializationUtil.java:73)
~[optimizer-job.jar:?]
at
org.apache.amoro.io.reader.StructLikeFunnel.funnel(StructLikeFunnel.java:35)
~[optimizer-job.jar:?]
at
org.apache.amoro.io.reader.StructLikeFunnel.funnel(StructLikeFunnel.java:27)
~[optimizer-job.jar:?]
at
org.apache.amoro.shade.guava32.com.google.common.hash.AbstractHasher.putObject(AbstractHasher.java:133)
~[optimizer-job.jar:?]
at
org.apache.amoro.shade.guava32.com.google.common.hash.AbstractHashFunction.hashObject(AbstractHashFunction.java:36)
~[optimizer-job.jar:?]
at
org.apache.amoro.shade.guava32.com.google.common.hash.BloomFilterStrategies$2.put(BloomFilterStrategies.java:109)
~[optimizer-job.jar:?]
at
org.apache.amoro.shade.guava32.com.google.common.hash.BloomFilter.put(BloomFilter.java:176)
~[optimizer-job.jar:?]
at
org.apache.amoro.io.reader.CombinedDeleteFilter.lambda$initializeBloomFilter$6(CombinedDeleteFilter.java:299)
~[optimizer-job.jar:?]
at java.util.HashMap.forEach(HashMap.java:1290) ~[?:1.8.0_352]
at
org.apache.amoro.io.reader.CombinedDeleteFilter.initializeBloomFilter(CombinedDeleteFilter.java:285)
~[optimizer-job.jar:?]
at
org.apache.amoro.io.reader.CombinedDeleteFilter.applyEqDeletes(CombinedDeleteFilter.java:247)
~[optimizer-job.jar:?]
at
org.apache.amoro.io.reader.CombinedDeleteFilter.applyEqDeletes(CombinedDeleteFilter.java:379)
~[optimizer-job.jar:?]
at
org.apache.amoro.io.reader.CombinedDeleteFilter.filter(CombinedDeleteFilter.java:216)
~[optimizer-job.jar:?]
at
org.apache.amoro.io.reader.GenericCombinedIcebergDataReader.readData(GenericCombinedIcebergDataReader.java:124)
~[optimizer-job.jar:?]
at
org.apache.amoro.optimizing.AbstractRewriteFilesExecutor.rewriterDataFiles(AbstractRewriteFilesExecutor.java:149)
~[optimizer-job.jar:?]
at
org.apache.amoro.table.TableMetaStore.call(TableMetaStore.java:256)
~[optimizer-job.jar:?]
... 13 more
```
### Anything else
_No response_
### Are you willing to submit a PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's Code of Conduct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
