nccuong-vikki commented on issue #8559:
URL: https://github.com/apache/seatunnel/issues/8559#issuecomment-2831897315
@rafi12396 It worked ?
Can you please provide me more information, like what jar files that you
need to manually download and where did you put them ?
I installed the `bundle-2.31.29.jar (AWS SDK V2)` to the **/lib**, and
already did like what you said:
```
sink {
Iceberg {
plugin_input = "sql_transformed"
catalog_name = "hadoop_cat"
iceberg.catalog.config={
warehouse = "My S3"
catalog-impl = "org.apache.iceberg.aws.glue.GlueCatalog"
io-impl = "org.apache.iceberg.aws.s3.S3FileIO"
}
namespace = "analytics_db"
table = "events_iceberg"
iceberg.table.write-props={
write.format.default="parquet"
}
}
}
```
But it then got this error :
```
Exception in thread "main"
org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel
job executed failed
at
org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:228)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at
org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: java.util.concurrent.CompletionException:
org.apache.seatunnel.engine.common.exception.JobException:
org.apache.seatunnel.common.exception.SeaTunnelRuntimeException:
ErrorCode:[API-09], ErrorDescription:[Handle save mode failed]
at
org.apache.seatunnel.engine.server.master.JobMaster.handleSaveMode(JobMaster.java:568)
at
org.apache.seatunnel.engine.server.master.JobMaster.handleSaveMode(JobMaster.java:574)
at
org.apache.seatunnel.engine.server.master.JobMaster.lambda$init$1(JobMaster.java:273)
at
java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at
java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
at
java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
at
java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1845)
at
java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at
java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at
java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at
java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at
java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at
java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
at
org.apache.seatunnel.engine.server.master.JobMaster.init(JobMaster.java:267)
at
org.apache.seatunnel.engine.server.CoordinatorService.lambda$submitJob$6(CoordinatorService.java:649)
at
org.apache.seatunnel.api.tracing.MDCRunnable.run(MDCRunnable.java:43)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.IllegalArgumentException: Cannot initialize Catalog
implementation org.apache.iceberg.aws.glue.GlueCatalog: Cannot find constructor
for interface org.apache.iceberg.catalog.Catalog
Missing org.apache.iceberg.aws.glue.GlueCatalog
[java.lang.ClassNotFoundException: org.apache.iceberg.aws.glue.GlueCatalog]
at org.apache.iceberg.CatalogUtil.loadCatalog(CatalogUtil.java:224)
at
org.apache.iceberg.CatalogUtil.buildIcebergCatalog(CatalogUtil.java:284)
at
org.apache.seatunnel.connectors.seatunnel.iceberg.IcebergCatalogLoader.loadCatalog(IcebergCatalogLoader.java:61)
at
org.apache.seatunnel.connectors.seatunnel.iceberg.catalog.IcebergCatalog.open(IcebergCatalog.java:92)
at
org.apache.seatunnel.api.sink.DefaultSaveModeHandler.open(DefaultSaveModeHandler.java:78)
at
org.apache.seatunnel.engine.server.master.JobMaster.handleSaveMode(JobMaster.java:565)
... 21 more
Caused by: java.lang.NoSuchMethodException: Cannot find constructor for
interface org.apache.iceberg.catalog.Catalog
Missing org.apache.iceberg.aws.glue.GlueCatalog
[java.lang.ClassNotFoundException: org.apache.iceberg.aws.glue.GlueCatalog]
at
org.apache.iceberg.common.DynConstructors.buildCheckedException(DynConstructors.java:250)
at
org.apache.iceberg.common.DynConstructors.access$200(DynConstructors.java:32)
at
org.apache.iceberg.common.DynConstructors$Builder.buildChecked(DynConstructors.java:220)
at org.apache.iceberg.CatalogUtil.loadCatalog(CatalogUtil.java:221)
... 26 more
Suppressed: java.lang.ClassNotFoundException:
org.apache.iceberg.aws.glue.GlueCatalog
at
java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
at
java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:592)
at
org.apache.seatunnel.engine.common.loader.SeaTunnelBaseClassLoader.loadClassWithoutExceptionHandling(SeaTunnelBaseClassLoader.java:56)
at
org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader.loadClassWithoutExceptionHandling(SeaTunnelChildFirstClassLoader.java:97)
at
org.apache.seatunnel.engine.common.loader.SeaTunnelBaseClassLoader.loadClass(SeaTunnelBaseClassLoader.java:47)
at
java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:467)
at
org.apache.iceberg.common.DynConstructors$Builder.impl(DynConstructors.java:149)
... 27 more
at
com.hazelcast.spi.impl.AbstractInvocationFuture.wrapInCompletionException(AbstractInvocationFuture.java:1347)
at
com.hazelcast.spi.impl.AbstractInvocationFuture.cascadeException(AbstractInvocationFuture.java:1340)
at
com.hazelcast.spi.impl.AbstractInvocationFuture.access$200(AbstractInvocationFuture.java:65)
at
com.hazelcast.spi.impl.AbstractInvocationFuture$ApplyNode.execute(AbstractInvocationFuture.java:1478)
at
com.hazelcast.spi.impl.AbstractInvocationFuture.unblockOtherNode(AbstractInvocationFuture.java:797)
at
com.hazelcast.spi.impl.AbstractInvocationFuture.unblockAll(AbstractInvocationFuture.java:759)
at
com.hazelcast.spi.impl.AbstractInvocationFuture.complete0(AbstractInvocationFuture.java:1235)
at
com.hazelcast.spi.impl.AbstractInvocationFuture.completeExceptionallyInternal(AbstractInvocationFuture.java:1223)
at
com.hazelcast.spi.impl.AbstractInvocationFuture.completeExceptionally(AbstractInvocationFuture.java:709)
at
com.hazelcast.client.impl.spi.impl.ClientInvocation.completeExceptionally(ClientInvocation.java:294)
at
com.hazelcast.client.impl.spi.impl.ClientInvocation.notifyExceptionWithOwnedPermission(ClientInvocation.java:321)
at
com.hazelcast.client.impl.spi.impl.ClientInvocation.notifyException(ClientInvocation.java:304)
at
com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier.handleResponse(ClientResponseHandlerSupplier.java:164)
at
com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier.process(ClientResponseHandlerSupplier.java:141)
at
com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier.access$300(ClientResponseHandlerSupplier.java:60)
at
com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier$DynamicResponseHandler.accept(ClientResponseHandlerSupplier.java:251)
at
com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier$DynamicResponseHandler.accept(ClientResponseHandlerSupplier.java:243)
at
com.hazelcast.client.impl.connection.tcp.TcpClientConnection.handleClientMessage(TcpClientConnection.java:245)
at
com.hazelcast.client.impl.protocol.util.ClientMessageDecoder.handleMessage(ClientMessageDecoder.java:135)
at
com.hazelcast.client.impl.protocol.util.ClientMessageDecoder.onRead(ClientMessageDecoder.java:89)
at
com.hazelcast.internal.networking.nio.NioInboundPipeline.process(NioInboundPipeline.java:136)
at
com.hazelcast.internal.networking.nio.NioThread.processSelectionKey(NioThread.java:383)
at
com.hazelcast.internal.networking.nio.NioThread.processSelectionKeys(NioThread.java:368)
at
com.hazelcast.internal.networking.nio.NioThread.selectLoop(NioThread.java:294)
at
com.hazelcast.internal.networking.nio.NioThread.executeRun(NioThread.java:249)
at
com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102)
Caused by: org.apache.seatunnel.engine.common.exception.JobException:
org.apache.seatunnel.common.exception.SeaTunnelRuntimeException:
ErrorCode:[API-09], ErrorDescription:[Handle save mode failed]
at
org.apache.seatunnel.engine.server.master.JobMaster.handleSaveMode(JobMaster.java:568)
at
org.apache.seatunnel.engine.server.master.JobMaster.handleSaveMode(JobMaster.java:574)
at
org.apache.seatunnel.engine.server.master.JobMaster.lambda$init$1(JobMaster.java:273)
at
java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at
java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
at
java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
at
java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1845)
at
java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at
java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at
java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at
java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at
java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at
java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
at
org.apache.seatunnel.engine.server.master.JobMaster.init(JobMaster.java:267)
at
org.apache.seatunnel.engine.server.CoordinatorService.lambda$submitJob$6(CoordinatorService.java:649)
at
org.apache.seatunnel.api.tracing.MDCRunnable.run(MDCRunnable.java:43)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.IllegalArgumentException: Cannot initialize Catalog
implementation org.apache.iceberg.aws.glue.GlueCatalog: Cannot find constructor
for interface org.apache.iceberg.catalog.Catalog
Missing org.apache.iceberg.aws.glue.GlueCatalog
[java.lang.ClassNotFoundException: org.apache.iceberg.aws.glue.GlueCatalog]
at org.apache.iceberg.CatalogUtil.loadCatalog(CatalogUtil.java:224)
at
org.apache.iceberg.CatalogUtil.buildIcebergCatalog(CatalogUtil.java:284)
at
org.apache.seatunnel.connectors.seatunnel.iceberg.IcebergCatalogLoader.loadCatalog(IcebergCatalogLoader.java:61)
at
org.apache.seatunnel.connectors.seatunnel.iceberg.catalog.IcebergCatalog.open(IcebergCatalog.java:92)
at
org.apache.seatunnel.api.sink.DefaultSaveModeHandler.open(DefaultSaveModeHandler.java:78)
at
org.apache.seatunnel.engine.server.master.JobMaster.handleSaveMode(JobMaster.java:565)
... 21 more
Caused by: java.lang.NoSuchMethodException: Cannot find constructor for
interface org.apache.iceberg.catalog.Catalog
Missing org.apache.iceberg.aws.glue.GlueCatalog
[java.lang.ClassNotFoundException: org.apache.iceberg.aws.glue.GlueCatalog]
at
org.apache.iceberg.common.DynConstructors.buildCheckedException(DynConstructors.java:250)
at
org.apache.iceberg.common.DynConstructors.access$200(DynConstructors.java:32)
at
org.apache.iceberg.common.DynConstructors$Builder.buildChecked(DynConstructors.java:220)
at org.apache.iceberg.CatalogUtil.loadCatalog(CatalogUtil.java:221)
... 26 more
Suppressed: java.lang.ClassNotFoundException:
org.apache.iceberg.aws.glue.GlueCatalog
at
java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
at
java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:592)
at
org.apache.seatunnel.engine.common.loader.SeaTunnelBaseClassLoader.loadClassWithoutExceptionHandling(SeaTunnelBaseClassLoader.java:56)
at
org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader.loadClassWithoutExceptionHandling(SeaTunnelChildFirstClassLoader.java:97)
at
org.apache.seatunnel.engine.common.loader.SeaTunnelBaseClassLoader.loadClass(SeaTunnelBaseClassLoader.java:47)
at
java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:467)
at
org.apache.iceberg.common.DynConstructors$Builder.impl(DynConstructors.java:149)
... 27 more
at
org.apache.seatunnel.engine.server.CoordinatorService.lambda$submitJob$6(CoordinatorService.java:656)
at
org.apache.seatunnel.api.tracing.MDCRunnable.run(MDCRunnable.java:43)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.lang.Thread.run(Thread.java:840)
```
I then have a deeper research into the repo, and see that the
GlueCatalog.class only exists in the iceberg-aws, and the pom.xml of the
connector-iceberg-2.3.10 did not have that :
[pom.xml](https://github.com/apache/seatunnel/blob/dev/seatunnel-connectors-v2/connector-iceberg/pom.xml)
So I did try to manually install all of that jar files and desperately put
them into **/lib** or **/connectors** folder, but neither work since they run
only one connector jar file (they have a prefix for each connector plugin at
[plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties))
- I did try to rename to have the same prefix but it will cause this to
trigger SeaTunnelException in selectPluginJar :
[this](https://github.com/apache/seatunnel/blob/dev/seatunnel-plugin-discovery/src/main/java/org/apache/seatunnel/plugin/discovery/AbstractPluginDiscovery.java)
I really need your help, please tell me what did you do @rafi12396
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
