jalousiex commented on issue #4038: URL: https://github.com/apache/incubator-streampark/issues/4038#issuecomment-2342861960
> mysql cdc uses the 3.2.0 package, then you should sink to paimon, and the error shows that paimon catalog does not support s3 for now 实际上0.8.2是支持的，这个任务成功跑起来了，日志显示jobGraph提交失败，restApi成功了 关于s3，flink在不同封装上表现差异比较大，原生支持较好，dinky提交也报类似错误失败，但streampark最终是跑起来了 另外，很奇怪动态参数没有生效，以下为重启streampark之后，运行custom任务的完整日志 ``` 2024-09-11 14:52:37 | INFO | XNIO-1 task-2 | io.undertow.servlet:389] Initializing Spring DispatcherServlet 'dispatcherServlet' 2024-09-11 14:52:37 | INFO | XNIO-1 task-2 | org.springframework.web.servlet.DispatcherServlet:525] Initializing Servlet 'dispatcherServlet' 2024-09-11 14:52:37 | INFO | XNIO-1 task-2 | org.springframework.web.servlet.DispatcherServlet:547] Completed initialization in 3 ms 2024-09-11 14:52:37 | INFO | XNIO-1 task-1 | org.apache.streampark.console.base.handler.GlobalExceptionHandler:61] Permission denied: Unauthorized 2024-09-11 14:52:37 | INFO | XNIO-1 task-2 | org.apache.streampark.console.base.handler.GlobalExceptionHandler:54] Unauthenticated: This subject is anonymous - it does not have any identifying principals and authorization operations require an identity to check against. A Subject instance will acquire these identifying principals automatically after a successful login is performed be executing org.apache.shiro.subject.Subject.login(AuthenticationToken) or when 'Remember Me' functionality is enabled by the SecurityManager. This exception can also occur when a previously logged-in Subject has logged out which makes it anonymous again. Because an identity is currently not known due to any of these conditions, authorization is denied. 2024-09-11 14:52:37 | INFO | XNIO-1 task-4 | org.apache.streampark.console.base.handler.GlobalExceptionHandler:54] Unauthenticated: This subject is anonymous - it does not have any identifying principals and authorization operations require an identity to check against. A Subject instance will acquire these identifying principals automatically after a successful login is performed be executing org.apache.shiro.subject.Subject.login(AuthenticationToken) or when 'Remember Me' functionality is enabled by the SecurityManager. This exception can also occur when a previously logged-in Subject has logged out which makes it anonymous again. Because an identity is currently not known due to any of these conditions, authorization is denied. 14:54:07.624 [streampark-flink-app-bootstrap-0] DEBUG org.apache.streampark.common.util.CommandUtils - [StreamPark] Command execute: java -classpath /opt/flink-1.18.1/lib/flink-dist-1.18.1.jar org.apache.flink.client.cli.CliFrontend --version 14:54:08.065 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.common.conf.FlinkVersion - [StreamPark] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. Version: 1.18.1, Commit ID: a8c8b1c 14:54:08.072 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.flink.proxy.FlinkShimsProxy - [StreamPark] add flink shims urls classloader,flink version: ----------------------------------------- flink version ----------------------------------- flinkHome : /opt/flink-1.18.1 distJarName : flink-dist-1.18.1.jar flinkVersion : 1.18.1 majorVersion : 1.18 scalaVersion : 2.12 shimsVersion : streampark-flink-shims_flink-1.18 ------------------------------------------------------------------------------------------- 14:54:08.082 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.flink.proxy.FlinkShimsProxy - [StreamPark] include streampark lib: streampark-flink-client-core_2.12-2.1.4.jar 14:54:08.083 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.flink.proxy.FlinkShimsProxy - [StreamPark] include flink shims jar lib: streampark-flink-shims_flink-1.18_2.12-2.1.4.jar 14:54:08.083 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.flink.proxy.FlinkShimsProxy - [StreamPark] include streampark lib: streampark-common_2.12-2.1.4.jar 14:54:08.084 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.flink.proxy.FlinkShimsProxy - [StreamPark] include jar lib: streampark-shaded-jackson-1.0.0.jar 14:54:08.084 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.flink.proxy.FlinkShimsProxy - [StreamPark] include streampark lib: streampark-flink-shims-base_2.12-2.1.4.jar 14:54:08.084 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.flink.proxy.FlinkShimsProxy - [StreamPark] include streampark lib: streampark-flink-client-api_2.12-2.1.4.jar 14:54:08.084 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.flink.proxy.FlinkShimsProxy - [StreamPark] include streampark lib: streampark-flink-proxy_2.12-2.1.4.jar 14:54:08.084 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.flink.proxy.FlinkShimsProxy - [StreamPark] include streampark lib: streampark-flink-packer_2.12-2.1.4.jar 14:54:08.085 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.flink.proxy.FlinkShimsProxy - [StreamPark] include streampark lib: streampark-flink-kubernetes_2.12-2.1.4.jar 14:54:08.085 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.flink.proxy.FlinkShimsProxy - [StreamPark] include streampark lib: streampark-flink-core_2.12-2.1.4.jar 14:54:08.926 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.flink.client.impl.RemoteClient - [StreamPark] --------------------------------------- flink job start --------------------------------------- userFlinkHome : /opt/flink-1.18.1 flinkVersion : 1.18.1 appName : mysql-to-paimon-s3 devMode : CUSTOM_CODE execMode : REMOTE k8sNamespace : null flinkExposedType : null clusterId : null applicationType : Apache Flink savePoint : null properties : rest.address -> 172.31.4.220 rest.port -> 8081 execution.checkpointing.interval -> 10s state.checkpoints.num-retained -> 10 execution.checkpointing.num-retained -> 5 classloader.resolve-order -> parent-first args : mysql_sync_database --mysql-conf hostname=172.31.4.149 --mysql-conf port=3306 --mysql-conf username=testuser --mysql-conf password=*** --mysql-conf database-name=testflink --warehouse s3://flink/paimon/ --catalog_conf s3.endpoint=https://ossapi-tst --catalog_conf s3.access-key=ak_flink --catalog_conf s3.secret-key=*** --catalog_conf s3.path.style.access=true --database ods --including_tables 'o.*|product.?|shipments' --table_prefix my_ --table_suffix _001 --table_conf source.checkpoint-align.enabled=true --table_conf changelog-producer=input --table_conf sink.parallelism=1 appConf : json://{"$internal.application.main":"org.apache.paimon.flink.action.FlinkActions"} flinkBuildResult : { workspacePath: /opt/streampark_ws/workspace/100001, shadedJarPath: /opt/streampark_ws/workspace/100001/streampark-flinkjob_mysql-to-paimon-s3.jar, pass: true } ------------------------------------------------------------------------------------------- 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: state.checkpoints.num-retained, 20 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: fs.allowed-fallback-filesystems, s3 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: jobmanager.execution.failover-strategy, region 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: jobmanager.rpc.address, 172.31.4.220 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.num-retained, 10 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: state.savepoints.dir, s3://flink/savepoints/ 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.aligned-checkpoint-timeout, 60s 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: jobmanager.bind-host, 0.0.0.0 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: s3.secret-key, ****** 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: s3.endpoint, https://ossapi-tst.hengrui.com 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.checkpoints-after-tasks-finish.enabled, true 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: parallelism.default, 1 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: taskmanager.numberOfTaskSlots, 4 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: env.java.opts.all, --add-exports=java.base/sun.net.util=ALL-UNNAMED --add-exports=java.rmi/sun.rmi.registry=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED --add-exports=java.security.jgss/sun.security.krb5=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.text=ALL-UNNAMED --add-opens=java.base/jav a.time=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: pekko.ask.timeout, 30s 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: taskmanager.memory.process.size, 3072m 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.mode, EXACTLY_ONCE 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: taskmanager.bind-host, 0.0.0.0 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: classloader.resolve-order, parent-first 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: web.timeout, 50000 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: jobmanager.memory.process.size, 2048m 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: jobmanager.rpc.port, 6123 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: rest.bind-address, 0.0.0.0 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.interval, 1min 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.timeout, 3min 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: s3.access-key, ak_flink 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: taskmanager.memory.managed.size, 0 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.externalized-checkpoint-retention, DELETE_ON_CANCELLATION 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: rest.address, 172.31.4.220 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.max-concurrent-checkpoints, 2 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: state.checkpoints.dir, s3://flink/checkpoints/ 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: s3.path.style.access, true 14:54:09.280 [streampark-flink-app-bootstrap-0] WARN org.apache.streampark.flink.client.impl.RemoteClient - [StreamPark] param:$internal.application.main is error,skip it. 14:54:09.321 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.flink.client.impl.RemoteClient - [StreamPark] submit application dynamicProperties: rest.address :172.31.4.220 14:54:09.322 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.flink.client.impl.RemoteClient - [StreamPark] submit application dynamicProperties: rest.port :8081 14:54:09.322 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.flink.client.impl.RemoteClient - [StreamPark] submit application dynamicProperties: execution.checkpointing.interval :10s 14:54:09.322 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.flink.client.impl.RemoteClient - [StreamPark] submit application dynamicProperties: state.checkpoints.num-retained :10 14:54:09.322 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.flink.client.impl.RemoteClient - [StreamPark] submit application dynamicProperties: execution.checkpointing.num-retained :5 14:54:09.322 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.flink.client.impl.RemoteClient - [StreamPark] submit application dynamicProperties: classloader.resolve-order :parent-first 14:54:09.332 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.flink.client.impl.RemoteClient - cliArgs: -t remote -Drest.address=172.31.4.220 -Drest.port=8081 -Dexecution.checkpointing.interval=10s -Dstate.checkpoints.num-retained=10 -Dexecution.checkpointing.num-retained=5 -Dclassloader.resolve-order=parent-first 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: state.checkpoints.num-retained, 20 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: fs.allowed-fallback-filesystems, s3 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: jobmanager.execution.failover-strategy, region 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: jobmanager.rpc.address, 172.31.4.220 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.num-retained, 10 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: state.savepoints.dir, s3://flink/savepoints/ 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.aligned-checkpoint-timeout, 60s 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: jobmanager.bind-host, 0.0.0.0 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: s3.secret-key, ****** 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: s3.endpoint, https://ossapi-tst.hengrui.com 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.checkpoints-after-tasks-finish.enabled, true 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: parallelism.default, 1 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: taskmanager.numberOfTaskSlots, 4 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: env.java.opts.all, --add-exports=java.base/sun.net.util=ALL-UNNAMED --add-exports=java.rmi/sun.rmi.registry=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED --add-exports=java.security.jgss/sun.security.krb5=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.text=ALL-UNNAMED --add-opens=java.base/jav a.time=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: pekko.ask.timeout, 30s 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: taskmanager.memory.process.size, 3072m 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.mode, EXACTLY_ONCE 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: taskmanager.bind-host, 0.0.0.0 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: classloader.resolve-order, parent-first 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: web.timeout, 50000 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: jobmanager.memory.process.size, 2048m 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: jobmanager.rpc.port, 6123 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: rest.bind-address, 0.0.0.0 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.interval, 1min 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.timeout, 3min 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: s3.access-key, ak_flink 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: taskmanager.memory.managed.size, 0 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.externalized-checkpoint-retention, DELETE_ON_CANCELLATION 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: rest.address, 172.31.4.220 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.max-concurrent-checkpoints, 2 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: state.checkpoints.dir, s3://flink/checkpoints/ 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: s3.path.style.access, true 14:54:09.344 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.flink.client.impl.RemoteClient - [StreamPark] Custom commandline: [org.apache.flink.client.cli.GenericCLI@53ad1d67, org.apache.flink.yarn.cli.FlinkYarnSessionCli@430e4fa6, org.apache.flink.client.cli.DefaultCLI@23c80632] 14:54:09.345 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.flink.client.impl.RemoteClient - [StreamPark] Checking custom commandline org.apache.flink.client.cli.GenericCLI@53ad1d67, isActive: true 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: state.checkpoints.num-retained, 20 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: fs.allowed-fallback-filesystems, s3 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: jobmanager.execution.failover-strategy, region 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: jobmanager.rpc.address, 172.31.4.220 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.num-retained, 10 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: state.savepoints.dir, s3://flink/savepoints/ 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.aligned-checkpoint-timeout, 60s 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: jobmanager.bind-host, 0.0.0.0 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: s3.secret-key, ****** 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: s3.endpoint, https://ossapi-tst.hengrui.com 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.checkpoints-after-tasks-finish.enabled, true 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: parallelism.default, 1 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: taskmanager.numberOfTaskSlots, 4 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: env.java.opts.all, --add-exports=java.base/sun.net.util=ALL-UNNAMED --add-exports=java.rmi/sun.rmi.registry=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED --add-exports=java.security.jgss/sun.security.krb5=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.text=ALL-UNNAMED --add-opens=java.base/jav a.time=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: pekko.ask.timeout, 30s 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: taskmanager.memory.process.size, 3072m 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.mode, EXACTLY_ONCE 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: taskmanager.bind-host, 0.0.0.0 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: classloader.resolve-order, parent-first 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: web.timeout, 50000 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: jobmanager.memory.process.size, 2048m 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: jobmanager.rpc.port, 6123 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: rest.bind-address, 0.0.0.0 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.interval, 1min 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.timeout, 3min 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: s3.access-key, ak_flink 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: taskmanager.memory.managed.size, 0 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.externalized-checkpoint-retention, DELETE_ON_CANCELLATION 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: rest.address, 172.31.4.220 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.max-concurrent-checkpoints, 2 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: state.checkpoints.dir, s3://flink/checkpoints/ 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: s3.path.style.access, true 14:54:09.428 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.flink.client.impl.RemoteClient - [StreamPark] [flink-submit] Submit job with JobGraph Plan. 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: state.checkpoints.num-retained, 20 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: fs.allowed-fallback-filesystems, s3 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: jobmanager.execution.failover-strategy, region 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: jobmanager.rpc.address, 172.31.4.220 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.num-retained, 10 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: state.savepoints.dir, s3://flink/savepoints/ 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.aligned-checkpoint-timeout, 60s 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: jobmanager.bind-host, 0.0.0.0 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: s3.secret-key, ****** 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: s3.endpoint, https://ossapi-tst.hengrui.com 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.checkpoints-after-tasks-finish.enabled, true 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: parallelism.default, 1 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: taskmanager.numberOfTaskSlots, 4 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: env.java.opts.all, --add-exports=java.base/sun.net.util=ALL-UNNAMED --add-exports=java.rmi/sun.rmi.registry=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED --add-exports=java.security.jgss/sun.security.krb5=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.text=ALL-UNNAMED --add-opens=java.base/jav a.time=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: pekko.ask.timeout, 30s 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: taskmanager.memory.process.size, 3072m 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.mode, EXACTLY_ONCE 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: taskmanager.bind-host, 0.0.0.0 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: classloader.resolve-order, parent-first 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: web.timeout, 50000 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: jobmanager.memory.process.size, 2048m 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: jobmanager.rpc.port, 6123 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: rest.bind-address, 0.0.0.0 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.interval, 1min 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.timeout, 3min 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: s3.access-key, ak_flink 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: taskmanager.memory.managed.size, 0 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.externalized-checkpoint-retention, DELETE_ON_CANCELLATION 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: rest.address, 172.31.4.220 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.max-concurrent-checkpoints, 2 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: state.checkpoints.dir, s3://flink/checkpoints/ 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: s3.path.style.access, true 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.paimon.flink.action.ActionFactory:73] mysql_sync_database job args: --mysql-conf hostname=172.31.4.149 --mysql-conf port=3306 --mysql-conf username=testuser --mysql-conf password=testuser --mysql-conf database-name=testflink --warehouse s3://flink/paimon/ --catalog_conf s3.endpoint=https://ossapi-tst --catalog_conf s3.access-key=ak_flink --catalog_conf s3.secret-key=ak_flink --catalog_conf s3.path.style.access=true --database ods --including_tables o.*|product.?|shipments --table_prefix my_ --table_suffix _001 --table_conf source.checkpoint-align.enabled=true --table_conf changelog-producer=input --table_conf sink.parallelism=1 2024-09-11 14:54:09 | WARN | streampark-flink-app-bootstrap-0 | org.apache.paimon.utils.HadoopUtils:125] Could not find Hadoop configuration via any of the supported methods 2024-09-11 14:54:10 | WARN | streampark-flink-app-bootstrap-0 | org.apache.hadoop.util.NativeCodeLoader:60] Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 14:54:10.664 [streampark-flink-app-bootstrap-0] ERROR org.apache.streampark.flink.client.impl.RemoteClient - [StreamPark] REMOTE mode submit by jobGraph fail.org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: org.apache.paimon.fs.UnsupportedSchemeException: Could not find a file io implementation for scheme 's3' in the classpath. FlinkFileIOLoader also cannot access this path. Hadoop FileSystem also cannot access this path 's3://flink/paimon'. at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372) at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) at org.apache.flink.client.program.PackagedProgramUtils.getPipelineFromProgram(PackagedProgramUtils.java:158) at org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:82) at org.apache.streampark.flink.client.trait.FlinkClientTrait.getJobGraph(FlinkClientTrait.scala:252) at org.apache.streampark.flink.client.trait.FlinkClientTrait.getJobGraph$(FlinkClientTrait.scala:231) at org.apache.streampark.flink.client.impl.RemoteClient$.jobGraphSubmit(RemoteClient.scala:139) at org.apache.streampark.flink.client.impl.RemoteClient$.$anonfun$doSubmit$1(RemoteClient.scala:48) at org.apache.streampark.flink.client.trait.FlinkClientTrait.$anonfun$trySubmit$1(FlinkClientTrait.scala:207) at scala.util.Try$.apply(Try.scala:209) at org.apache.streampark.flink.client.trait.FlinkClientTrait.trySubmit(FlinkClientTrait.scala:205) at org.apache.streampark.flink.client.trait.FlinkClientTrait.trySubmit$(FlinkClientTrait.scala:201) at org.apache.streampark.flink.client.impl.RemoteClient$.doSubmit(RemoteClient.scala:49) at org.apache.streampark.flink.client.trait.FlinkClientTrait.submit(FlinkClientTrait.scala:123) at org.apache.streampark.flink.client.trait.FlinkClientTrait.submit$(FlinkClientTrait.scala:60) at org.apache.streampark.flink.client.impl.RemoteClient$.submit(RemoteClient.scala:34) at org.apache.streampark.flink.client.FlinkClientHandler$.submit(FlinkClientHandler.scala:40) at org.apache.streampark.flink.client.FlinkClientHandler.submit(FlinkClientHandler.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.streampark.flink.client.FlinkClient$.$anonfun$proxy$1(FlinkClient.scala:87) at org.apache.streampark.flink.proxy.FlinkShimsProxy$.$anonfun$proxy$1(FlinkShimsProxy.scala:60) at org.apache.streampark.common.util.ClassLoaderUtils$.runAsClassLoader(ClassLoaderUtils.scala:38) at org.apache.streampark.flink.proxy.FlinkShimsProxy$.proxy(FlinkShimsProxy.scala:60) at org.apache.streampark.flink.client.FlinkClient$.proxy(FlinkClient.scala:82) at org.apache.streampark.flink.client.FlinkClient$.submit(FlinkClient.scala:53) at org.apache.streampark.flink.client.FlinkClient.submit(FlinkClient.scala) at org.apache.streampark.console.core.service.impl.ApplicationServiceImpl.lambda$start$8(ApplicationServiceImpl.java:1655) at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.UncheckedIOException: org.apache.paimon.fs.UnsupportedSchemeException: Could not find a file io implementation for scheme 's3' in the classpath. FlinkFileIOLoader also cannot access this path. Hadoop FileSystem also cannot access this path 's3://flink/paimon'. at org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:92) at org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:66) at org.apache.paimon.flink.FlinkCatalogFactory.createPaimonCatalog(FlinkCatalogFactory.java:80) at org.apache.paimon.flink.action.ActionBase.initPaimonCatalog(ActionBase.java:71) at org.apache.paimon.flink.action.ActionBase.<init>(ActionBase.java:58) at org.apache.paimon.flink.action.cdc.SynchronizationActionBase.<init>(SynchronizationActionBase.java:77) at org.apache.paimon.flink.action.cdc.SyncDatabaseActionBase.<init>(SyncDatabaseActionBase.java:61) at org.apache.paimon.flink.action.cdc.mysql.MySqlSyncDatabaseAction.<init>(MySqlSyncDatabaseAction.java:108) at org.apache.paimon.flink.action.cdc.mysql.MySqlSyncDatabaseActionFactory.createAction(MySqlSyncDatabaseActionFactory.java:52) at org.apache.paimon.flink.action.cdc.mysql.MySqlSyncDatabaseActionFactory.createAction(MySqlSyncDatabaseActionFactory.java:31) at org.apache.paimon.flink.action.cdc.SynchronizationActionFactoryBase.create(SynchronizationActionFactoryBase.java:45) at org.apache.paimon.flink.action.cdc.SyncDatabaseActionFactoryBase.create(SyncDatabaseActionFactoryBase.java:44) at org.apache.paimon.flink.action.ActionFactory.createAction(ActionFactory.java:82) at org.apache.paimon.flink.action.FlinkActions.main(FlinkActions.java:38) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) ... 33 more Caused by: org.apache.paimon.fs.UnsupportedSchemeException: Could not find a file io implementation for scheme 's3' in the classpath. FlinkFileIOLoader also cannot access this path. Hadoop FileSystem also cannot access this path 's3://flink/paimon'. at org.apache.paimon.fs.FileIO.get(FileIO.java:420) at org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:89) ... 51 more Suppressed: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 's3'. The scheme is directly supported by Flink through the following plugin(s): flink-s3-fs-hadoop, flink-s3-fs-presto. Please ensure that each plugin resides within its own subfolder within the plugins directory. See https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/plugins/ for more information. If you want to use a Hadoop file system for that scheme, please add the scheme to the configuration fs.allowed-fallback-filesystems. For a full list of supported file systems, please see https://nightlies.apache.org/flink/flink-docs-stable/ops/filesystems/. at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:515) at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:409) at org.apache.flink.core.fs.Path.getFileSystem(Path.java:274) at org.apache.paimon.flink.FlinkFileIO.getFileSystem(FlinkFileIO.java:127) at org.apache.paimon.flink.FlinkFileIO.exists(FlinkFileIO.java:100) at org.apache.paimon.fs.FileIOUtils.checkAccess(FileIOUtils.java:37) at org.apache.paimon.fs.FileIO.get(FileIO.java:388) ... 52 more Suppressed: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "s3" at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3443) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3466) at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365) at org.apache.paimon.fs.hadoop.HadoopFileIO.createFileSystem(HadoopFileIO.java:175) at org.apache.paimon.fs.hadoop.HadoopFileIO.getFileSystem(HadoopFileIO.java:168) at org.apache.paimon.fs.hadoop.HadoopFileIO.getFileSystem(HadoopFileIO.java:145) at org.apache.paimon.fs.hadoop.HadoopFileIO.exists(HadoopFileIO.java:110) at org.apache.paimon.fs.FileIOUtils.checkAccess(FileIOUtils.java:37) at org.apache.paimon.fs.FileIO.get(FileIO.java:397) ... 52 more 14:54:14.550 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.flink.client.impl.RemoteClient - [StreamPark] REMOTE mode submit by restApi, WebInterfaceURL http://172.31.4.220:8081, jobId: 8f0fb2ab846651cf78dd908a5a0bb610 2024-09-11 14:54:14 | INFO | streampark-flink-app-bootstrap-0 | org.apache.streampark.console.core.task.FlinkAppHttpWatcher:635] [StreamPark][FlinkAppHttpWatcher] setOptioning 2024-09-11 14:54:14 | INFO | streampark-flink-app-bootstrap-0 | org.apache.streampark.console.core.task.FlinkAppHttpWatcher:646] [StreamPark][FlinkAppHttpWatcher] add app to tracking,appId:100001 ^C ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@streampark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
