1032851561 opened a new issue, #9339: URL: https://github.com/apache/hudi/issues/9339
**Describe the problem you faced** The taskmanager always exits without error quickly when I restart the job. Before exiting TM, I saw on the UI that the `bucket_assigner` was initializing.  - table : There are 4 tables, every has very little data. ``` table.type=MERGE_ON_READ compaction.tasks=2 compaction.trigger.strategy=num_commits compaction.delta_commits=10 clean.retain_commits=400 write.task.max.size=400 write.tasks=1 ``` - taskmanager Only 1 taskmanager with 6G of memory and 2 slots. **Expected behavior** Then only one TaskManager alway exit without exception. **Environment Description** * Hudi version : 0.13.1 * Table type : MERGE_ON_READ * Flink version : 1.14.5 * Java version : 1.8.0_162 * Storage (HDFS/S3/GCS..) : HDFS (hadoop 2.10.2) * Running on Docker? (yes/no) : Yarn (Application mode) **Stacktrace** - JobManager log ``` 2023-08-01 18:17:56,627 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink@hadoop-02:34821] has failed, address is now gated for [50] ms. Reason: [Disassociated] 2023-08-01 18:17:57,305 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Worker container_e07_1669378398064_0904_01_000002 is terminated. Diagnostics: Container container_e07_1669378398064_0904_01_000002 marked as failed. Exit code:-103. Diagnostics:-M15.jar:/hadoop-2.10.2/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/hadoop-2.10.2/share/hadoop/common/lib/jersey-json-1.9.jar:/hadoop-2.10.2/share/hadoop/common/lib/commons-beanutils-1.9.4.jar:/hadoop-2.10.2/share/hadoop/common/lib/stax2-api-4.2.1.jar:/hadoop-2.10.2/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar:/hadoop-2.10.2/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/hadoop-2.10.2/share/hadoop/hdfs/hadoop-hdfs-client-2.10.2.jar:/hadoop-2.10.2/share/hadoop/hdfs/hadoop-hdfs-native-client-2.10.2.jar:/hadoop-2.10.2/share/hadoop/hdfs/hadoop-hdfs-2.10.2.jar:/hadoop-2.10.2/share/hadoop/hdfs/hadoop-hdfs-rbf-2.10.2.jar:/hadoop-2.10.2/share/hadoop/hdfs/hadoop-hdfs-rbf-2.10.2-tests.jar:/hadoop-2.10.2/share/hadoop/hdfs/hadoop-hdfs-native-client-2.10.2-tests.jar:/hadoop-2.10.2/share/hadoop/hdfs/hadoop-hdfs-client-2.10.2-tests.jar:/hadoop-2.10.2/share/hadoop/hdfs/hadoop-hdfs-nfs-2.10.2.jar:/hadoop-2.10.2/share/hadoop/hdfs/hadoop-hdfs-2.10.2-tests.jar:/hadoop-2 .10.2/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/xercesImpl-2.12.0.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/jsr305-3.0.2.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/jackson-databind-2.9.10.7.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/okio-1.6.0.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/xml-apis-1.4.01.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/reload4j-1.2.18.3.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/hadoop-hdfs-client-2.10.2.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/jackson-annotations-2.9.10.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/commons-io-2.5.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/servlet-api-2 .5.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/asm-3.2.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/htrace-core4-4.1.0-incubating.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/okhttp-2.7.5.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/netty-all-4.1.50.Final.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/guava-11.0.2.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/netty-3.10.6.Final.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/jackson-core-2.9.10.jar:/hadoop-2.10.2/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.10.2.jar:/hadoop-2.10.2/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.10.2.jar:/h adoop-2.10.2/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.10.2.jar:/hadoop-2.10.2/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.10.2.jar:/hadoop-2.10.2/share/hadoop/yarn/hadoop-yarn-server-common-2.10.2.jar:/hadoop-2.10.2/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.10.2.jar:/hadoop-2.10.2/share/hadoop/yarn/hadoop-yarn-api-2.10.2.jar:/hadoop-2.10.2/share/hadoop/yarn/hadoop-yarn-server-tests-2.10.2.jar:/hadoop-2.10.2/share/hadoop/yarn/hadoop-yarn-common-2.10.2.jar:/hadoop-2.10.2/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.10.2.jar:/hadoop-2.10.2/share/hadoop/yarn/hadoop-yarn-server-router-2.10.2.jar:/hadoop-2.10.2/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.10.2.jar:/hadoop-2.10.2/share/hadoop/yarn/hadoop-yarn-registry-2.10.2.jar:/hadoop-2.10.2/share/hadoop/yarn/hadoop-yarn-server-timeline-pluginstorage-2.10.2.jar:/hadoop-2.10.2/share/hadoop/yarn/hadoop-yarn-client-2.10.2.jar:/hadoop-2.10.2/share/hadoop/yarn/li b/jersey-core-1.9.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/jetty-6.1.26.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/curator-client-2.13.0.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/fst-2.50.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/api-util-1.0.0-M20.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/audience-annotations-0.5.0.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/jsr305-3.0.2.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/json-io-2.5.1.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/commons-net-3.1.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/nimbus-jose-jwt-7.9.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/avro-1.7.7.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/jettison-1.1.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/curator-framework-2.13.0.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/hadoop-2.10.2/share/hadoop/yarn/l ib/reload4j-1.2.18.3.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/guice-3.0.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/curator-recipes-2.13.0.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/commons-lang-2.6.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/json-smart-1.3.3.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/commons-codec-1.4.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/commons-io-2.5.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/commons-math3-3.1.1.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/api-asn1-api-1.0.0-M20.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/woodstox-core-5.3.0.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/java-util-1.9.0.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/jcip-annotations-1.0-1.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/httpclient-4.5.13.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/servlet-api-2.5.jar:/ha doop-2.10.2/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/commons-lang3-3.4.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/jsp-api-2.1.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/mssql-jdbc-6.2.1.jre7.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/xmlenc-0.52.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/jersey-server-1.9.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/asm-3.2.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/commons-compress-1.21.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/commons-digester-1.8.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/htrace-core4-4.1.0-incubating.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/snappy-java-1.0.5.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/gson-2.2.4.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/jersey-client-1.9.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/guice-servle t-3.0.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/HikariCP-java7-2.4.12.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/aopalliance-1.0.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/ehcache-3.3.1.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/zookeeper-3.4.14.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/jetty-sslengine-6.1.26.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/metrics-core-3.0.1.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/commons-configuration-1.6.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/spotbugs-annotations-3.1.9.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/guava-11.0.2.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/commons-cli-1.2.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/jsch-0.1.55.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/jets3t-0.9.0.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/activation-1.1.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/ja vax.inject-1.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/netty-3.10.6.Final.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/paranamer-2.3.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/httpcore-4.4.13.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/apacheds-i18n-2.0.0-M15.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/jersey-json-1.9.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/commons-beanutils-1.9.4.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/stax2-api-4.2.1.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/java-xmlbuilder-0.4.jar:/opt/jdk1.8.0_162/jre/../lib/sa-jdi.jar org.apache.hudi.org.openjdk.jol.vm.sa.AttachMain [2023-08-01 18:17:56.141]Container killed on request. Exit code is 143 [2023-08-01 18:17:56.149]Container exited with a non-zero exit code 143. 2023-08-01 18:17:57,305 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Closing TaskExecutor connection container_e07_1669378398064_0904_01_000002 because: Container container_e07_1669378398064_0904_01_000002 marked as failed. Exit code:-103. Diagnostics:-M15.jar:/hadoop-2.10.2/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/hadoop-2.10.2/share/hadoop/common/lib/jersey-json-1.9.jar:/hadoop-2.10.2/share/hadoop/common/lib/commons-beanutils-1.9.4.jar:/hadoop-2.10.2/share/hadoop/common/lib/stax2-api-4.2.1.jar:/hadoop-2.10.2/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar:/hadoop-2.10.2/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/hadoop-2.10.2/share/hadoop/hdfs/hadoop-hdfs-client-2.10.2.jar:/hadoop-2.10.2/share/hadoop/hdfs/hadoop-hdfs-native-client-2.10.2.jar:/hadoop-2.10.2/share/hadoop/hdfs/hadoop-hdfs-2.10.2.jar:/hadoop-2.10.2/share/hadoop/hdfs/hadoop-hdfs-rbf-2.10.2.jar:/hadoop-2.10.2/share/hadoop/hdfs/hadoop-hdfs-rbf-2.10.2-tests.jar:/hadoop-2.10.2/share/hadoop/hdfs/hadoop-hdfs-native-client-2.10.2-tests.jar:/hadoop-2.10.2/share/hadoop/hdfs/hadoop-hdfs-client-2.10.2-tests.jar:/hadoop-2.10.2/share/hadoop/hdfs/hadoop-hdfs-nfs-2.10.2.jar:/hadoop-2.10.2/share/hadoop/hdfs/hadoop-hdfs-2.10.2-tests.jar:/hadoop-2 .10.2/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/xercesImpl-2.12.0.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/jsr305-3.0.2.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/jackson-databind-2.9.10.7.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/okio-1.6.0.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/xml-apis-1.4.01.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/reload4j-1.2.18.3.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/hadoop-hdfs-client-2.10.2.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/jackson-annotations-2.9.10.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/commons-io-2.5.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/servlet-api-2 .5.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/asm-3.2.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/htrace-core4-4.1.0-incubating.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/okhttp-2.7.5.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/netty-all-4.1.50.Final.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/guava-11.0.2.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/netty-3.10.6.Final.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/hadoop-2.10.2/share/hadoop/hdfs/lib/jackson-core-2.9.10.jar:/hadoop-2.10.2/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.10.2.jar:/hadoop-2.10.2/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.10.2.jar:/h adoop-2.10.2/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.10.2.jar:/hadoop-2.10.2/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.10.2.jar:/hadoop-2.10.2/share/hadoop/yarn/hadoop-yarn-server-common-2.10.2.jar:/hadoop-2.10.2/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.10.2.jar:/hadoop-2.10.2/share/hadoop/yarn/hadoop-yarn-api-2.10.2.jar:/hadoop-2.10.2/share/hadoop/yarn/hadoop-yarn-server-tests-2.10.2.jar:/hadoop-2.10.2/share/hadoop/yarn/hadoop-yarn-common-2.10.2.jar:/hadoop-2.10.2/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.10.2.jar:/hadoop-2.10.2/share/hadoop/yarn/hadoop-yarn-server-router-2.10.2.jar:/hadoop-2.10.2/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.10.2.jar:/hadoop-2.10.2/share/hadoop/yarn/hadoop-yarn-registry-2.10.2.jar:/hadoop-2.10.2/share/hadoop/yarn/hadoop-yarn-server-timeline-pluginstorage-2.10.2.jar:/hadoop-2.10.2/share/hadoop/yarn/hadoop-yarn-client-2.10.2.jar:/hadoop-2.10.2/share/hadoop/yarn/li b/jersey-core-1.9.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/jetty-6.1.26.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/curator-client-2.13.0.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/fst-2.50.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/api-util-1.0.0-M20.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/audience-annotations-0.5.0.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/jsr305-3.0.2.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/json-io-2.5.1.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/commons-net-3.1.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/nimbus-jose-jwt-7.9.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/avro-1.7.7.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/jettison-1.1.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/curator-framework-2.13.0.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/hadoop-2.10.2/share/hadoop/yarn/l ib/reload4j-1.2.18.3.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/guice-3.0.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/curator-recipes-2.13.0.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/commons-lang-2.6.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/json-smart-1.3.3.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/commons-codec-1.4.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/commons-io-2.5.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/commons-math3-3.1.1.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/api-asn1-api-1.0.0-M20.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/woodstox-core-5.3.0.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/java-util-1.9.0.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/jcip-annotations-1.0-1.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/httpclient-4.5.13.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/servlet-api-2.5.jar:/ha doop-2.10.2/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/commons-lang3-3.4.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/jsp-api-2.1.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/mssql-jdbc-6.2.1.jre7.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/xmlenc-0.52.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/jersey-server-1.9.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/asm-3.2.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/commons-compress-1.21.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/commons-digester-1.8.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/htrace-core4-4.1.0-incubating.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/snappy-java-1.0.5.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/gson-2.2.4.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/jersey-client-1.9.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/guice-servle t-3.0.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/HikariCP-java7-2.4.12.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/aopalliance-1.0.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/ehcache-3.3.1.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/zookeeper-3.4.14.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/jetty-sslengine-6.1.26.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/metrics-core-3.0.1.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/commons-configuration-1.6.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/spotbugs-annotations-3.1.9.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/guava-11.0.2.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/commons-cli-1.2.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/jsch-0.1.55.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/jets3t-0.9.0.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/activation-1.1.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/ja vax.inject-1.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/netty-3.10.6.Final.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/paranamer-2.3.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/httpcore-4.4.13.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/apacheds-i18n-2.0.0-M15.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/jersey-json-1.9.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/commons-beanutils-1.9.4.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/stax2-api-4.2.1.jar:/hadoop-2.10.2/share/hadoop/yarn/lib/java-xmlbuilder-0.4.jar:/opt/jdk1.8.0_162/jre/../lib/sa-jdi.jar org.apache.hudi.org.openjdk.jol.vm.sa.AttachMain [2023-08-01 18:17:56.141]Container killed on request. Exit code is 143 [2023-08-01 18:17:56.149]Container exited with a non-zero exit code 143. 2023-08-01 18:17:57,306 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Requesting new worker with resource spec WorkerResourceSpec {cpuCores=2.0, taskHeapSize=3.870gb (4155380836 bytes), taskOffHeapSize=0 bytes, networkMemSize=527.360mb (552977046 bytes), managedMemSize=527.360mb (552977046 bytes), numSlots=2}, current pending count: 1. 2023-08-01 18:17:57,307 INFO org.apache.flink.yarn.YarnResourceManagerDriver [] - Requesting new TaskExecutor container with resource TaskExecutorProcessSpec {cpuCores=2.0, frameworkHeapSize=128.000mb (134217728 bytes), frameworkOffHeapSize=128.000mb (134217728 bytes), taskHeapSize=3.870gb (4155380836 bytes), taskOffHeapSize=0 bytes, networkMemSize=527.360mb (552977046 bytes), managedMemorySize=527.360mb (552977046 bytes), jvmMetaspaceSize=256.000mb (268435456 bytes), jvmOverheadSize=614.400mb (644245104 bytes), numSlots=2}, priority 1. 2023-08-01 18:17:57,326 WARN akka.remote.transport.netty.NettyTransport [] - Remote connection to [null] failed with java.net.ConnectException: Connection refused: hadoop-02/191.1.1.2:34821 2023-08-01 18:17:57,328 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink@hadoop-02:34821] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@hadoop-02:34821]] Caused by: [java.net.ConnectException: Connection refused: hadoop-02/191.1.1.2:34821] 2023-08-01 18:17:57,617 WARN akka.remote.transport.netty.NettyTransport [] - Remote connection to [null] failed with java.net.ConnectException: Connection refused: hadoop-02/191.1.1.2:34821 2023-08-01 18:17:57,618 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink@hadoop-02:34821] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@hadoop-02:34821]] Caused by: [java.net.ConnectException: Connection refused: hadoop-02/191.1.1.2:34821] 2023-08-01 18:18:02,807 INFO org.apache.flink.yarn.YarnResourceManagerDriver [] - Received 1 containers. 2023-08-01 18:18:02,807 INFO org.apache.flink.yarn.YarnResourceManagerDriver [] - Received 1 containers with priority 1, 1 pending container requests. 2023-08-01 18:18:02,807 INFO org.apache.flink.yarn.YarnResourceManagerDriver [] - Removing container request Capability[<memory:6144, vCores:2>]Priority[1]AllocationRequestId[0]ExecutionTypeRequest[{Execution Type: GUARANTEED, Enforce Execution Type: false}]. 2023-08-01 18:18:02,807 INFO org.apache.flink.yarn.YarnResourceManagerDriver [] - Accepted 1 requested containers, returned 0 excess containers, 0 pending container requests of resource <memory:6144, vCores:2>. 2023-08-01 18:18:02,807 INFO org.apache.flink.yarn.YarnResourceManagerDriver [] - TaskExecutor container_e07_1669378398064_0904_01_000003(hadoop-03:45245) will be started on hadoop-03 with TaskExecutorProcessSpec {cpuCores=2.0, frameworkHeapSize=128.000mb (134217728 bytes), frameworkOffHeapSize=128.000mb (134217728 bytes), taskHeapSize=3.870gb (4155380836 bytes), taskOffHeapSize=0 bytes, networkMemSize=527.360mb (552977046 bytes), managedMemorySize=527.360mb (552977046 bytes), jvmMetaspaceSize=256.000mb (268435456 bytes), jvmOverheadSize=614.400mb (644245104 bytes), numSlots=2}. 2023-08-01 18:18:02,810 INFO org.apache.flink.yarn.YarnResourceManagerDriver [] - Creating container launch context for TaskManagers 2023-08-01 18:18:02,811 INFO org.apache.flink.yarn.YarnResourceManagerDriver [] - Starting TaskManagers 2023-08-01 18:18:02,812 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Requested worker container_e07_1669378398064_0904_01_000003(hadoop-03:45245) with resource spec WorkerResourceSpec {cpuCores=2.0, taskHeapSize=3.870gb (4155380836 bytes), taskOffHeapSize=0 bytes, networkMemSize=527.360mb (552977046 bytes), managedMemSize=527.360mb (552977046 bytes), numSlots=2}. 2023-08-01 18:18:07,636 WARN akka.remote.transport.netty.NettyTransport [] - Remote connection to [null] failed with java.net.ConnectException: Connection refused: hadoop-02/191.1.1.2:34821 2023-08-01 18:18:07,637 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink@hadoop-02:34821] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@hadoop-02:34821]] Caused by: [java.net.ConnectException: Connection refused: hadoop-02/191.1.1.2:34821] 2023-08-01 18:18:07,638 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - TaskManager with id container_e07_1669378398064_0904_01_000002(hadoop-02:34965) is no longer reachable. 2023-08-01 18:18:07,646 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - compact_task (2/2) (bdf0fcee7bebf275831f45dfa8964416) switched from RUNNING to FAILED on container_e07_1669378398064_0904_01_000002 @ hadoop-02 (dataPort=45779). org.apache.flink.runtime.jobmaster.JobMasterException: TaskManager with id container_e07_1669378398064_0904_01_000002(hadoop-02:34965) is no longer reachable. at org.apache.flink.runtime.jobmaster.JobMaster$TaskManagerHeartbeatListener.notifyTargetUnreachable(JobMaster.java:1359) ~[flink-dist_2.12-1.14.5.jar:1.14.5] at org.apache.flink.runtime.heartbeat.HeartbeatMonitorImpl.reportHeartbeatRpcFailure(HeartbeatMonitorImpl.java:123) ~[flink-dist_2.12-1.14.5.jar:1.14.5] at org.apache.flink.runtime.heartbeat.HeartbeatManagerImpl.runIfHeartbeatMonitorExists(HeartbeatManagerImpl.java:275) ~[flink-dist_2.12-1.14.5.jar:1.14.5] at org.apache.flink.runtime.heartbeat.HeartbeatManagerImpl.reportHeartbeatTargetUnreachable(HeartbeatManagerImpl.java:267) ~[flink-dist_2.12-1.14.5.jar:1.14.5] at org.apache.flink.runtime.heartbeat.HeartbeatManagerImpl.handleHeartbeatRpcFailure(HeartbeatManagerImpl.java:262) ~[flink-dist_2.12-1.14.5.jar:1.14.5] at org.apache.flink.runtime.heartbeat.HeartbeatManagerImpl.lambda$handleHeartbeatRpc$0(HeartbeatManagerImpl.java:248) ~[flink-dist_2.12-1.14.5.jar:1.14.5] at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_162] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_162] at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442) ~[?:1.8.0_162] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.lambda$handleRunAsync$4(AkkaRpcActor.java:455) ~[?:?] at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68) ~[?:?] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:455) ~[?:?] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:213) ~[?:?] at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:78) ~[?:?] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:163) ~[?:?] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24) ~[?:?] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20) ~[?:?] at scala.PartialFunction.applyOrElse(PartialFunction.scala:123) ~[flink-dist_2.12-1.14.5.jar:1.14.5] at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122) ~[flink-dist_2.12-1.14.5.jar:1.14.5] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20) ~[?:?] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.12-1.14.5.jar:1.14.5] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) ~[flink-dist_2.12-1.14.5.jar:1.14.5] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) ~[flink-dist_2.12-1.14.5.jar:1.14.5] at akka.actor.Actor.aroundReceive(Actor.scala:537) ~[?:?] at akka.actor.Actor.aroundReceive$(Actor.scala:535) ~[?:?] at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220) ~[?:?] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580) ~[?:?] at akka.actor.ActorCell.invoke(ActorCell.scala:548) ~[?:?] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270) ~[?:?] at akka.dispatch.Mailbox.run(Mailbox.scala:231) ~[?:?] at akka.dispatch.Mailbox.exec(Mailbox.scala:243) ~[?:?] at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) ~[?:1.8.0_162] at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) ~[?:1.8.0_162] at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) ~[?:1.8.0_162] at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) ~[?:1.8.0_162] ``` - TaskManager log ``` 2023-08-01 19:08:33,244 INFO org.apache.hudi.sink.CleanFunction [] - Executor executes action [wait for cleaning finish] success! 2023-08-01 19:08:33,291 INFO org.apache.hudi.sink.CleanFunction [] - Executor executes action [wait for cleaning finish] success! 2023-08-01 19:08:33,316 INFO org.apache.hudi.sink.CleanFunction [] - Executor executes action [wait for cleaning finish] success! 2023-08-01 19:08:35,811 INFO org.apache.hudi.sink.partitioner.profile.WriteProfile [] - Refresh average bytes per record => 1024 2023-08-01 19:08:35,857 INFO org.apache.flink.runtime.taskmanager.Task [] - bucket_assigner (1/1)#0 (d2440b4a48b9e9206c56ff797dfc5055) switched from INITIALIZING to RUNNING. 2023-08-01 19:08:36,145 INFO org.apache.hudi.sink.partitioner.profile.WriteProfile [] - Refresh average bytes per record => 1024 2023-08-01 19:08:36,195 INFO org.apache.flink.runtime.taskmanager.Task [] - bucket_assigner (1/1)#0 (717f029671a3abf26daaa4fcc54089c2) switched from INITIALIZING to RUNNING. # WARNING: Unable to get Instrumentation. Dynamic Attach failed. You may add this JAR as -javaagent manually, or supply -Djdk.attach.allowAttachSelf # WARNING: Unable to attach Serviceability Agent. You can try again with escalated privileges. Two options: a) use -Djol.tryWithSudo=true to try with sudo; b) echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope 2023-08-01 19:08:37,491 INFO org.apache.flink.yarn.YarnTaskExecutorRunner [] - RECEIVED SIGNAL 15: SIGTERM. Shutting down as requested. 2023-08-01 19:08:37,491 INFO org.apache.flink.runtime.state.TaskExecutorStateChangelogStoragesManager [] - Shutting down TaskExecutorStateChangelogStoragesManager. 2023-08-01 19:08:37,492 INFO org.apache.flink.runtime.state.TaskExecutorLocalStateStoresManager [] - Shutting down TaskExecutorLocalStateStoresManager. 2023-08-01 19:08:37,492 INFO org.apache.flink.runtime.blob.PermanentBlobCache [] - Shutting down BLOB cache 2023-08-01 19:08:37,492 INFO org.apache.flink.runtime.blob.TransientBlobCache [] - Shutting down BLOB cache 2023-08-01 19:08:37,498 ERROR org.apache.hudi.sink.partitioner.profile.WriteProfiles [] - Get write metadata for table xxxxxxx with instant 20230228172445004 and path: hdfs://master/warehouse/xxxxxxxx error 2023-08-01 19:08:37,498 ERROR org.apache.hudi.sink.partitioner.profile.WriteProfiles [] - Get write metadata for table xxxxxxx with instant 20230227172702855 and path: hdfs://master/warehouse/xxxxxxxx error 2023-08-01 19:08:37,498 ERROR org.apache.hudi.sink.partitioner.profile.WriteProfiles [] - Get write metadata for table xxxxxxx with instant 20230224102218187 and path: hdfs://master/warehouse/xxxxxxxx error 2023-08-01 19:08:37,498 ERROR org.apache.hudi.sink.partitioner.profile.WriteProfiles [] - Get write metadata for table xxxxxxx with instant 20230224085023833 and path: hdfs://master/warehouse/xxxxxxxx error 2023-08-01 19:08:37,498 ERROR org.apache.hudi.sink.partitioner.profile.WriteProfiles [] - Get write metadata for table xxxxxxx with instant 20230224085023759 and path: hdfs://master/warehouse/xxxxxxxx error 2023-08-01 19:08:37,498 ERROR org.apache.hudi.sink.partitioner.profile.WriteProfiles [] - Get write metadata for table xxxxxxx with instant 20230222092724224 and path: hdfs://master/warehouse/xxxxxxxx error 2023-08-01 19:08:37,498 ERROR org.apache.hudi.sink.partitioner.profile.WriteProfiles [] - Get write metadata for table xxxxxxx with instant 20230222092523193 and path: hdfs://master/warehouse/xxxxxxxx error 2023-08-01 19:08:37,498 ERROR org.apache.hudi.sink.partitioner.profile.WriteProfiles [] - Get write metadata for table xxxxxxx with instant 20230221111919876 and path: hdfs://master/warehouse/xxxxxxxx error 2023-08-01 19:08:37,498 ERROR org.apache.hudi.sink.partitioner.profile.WriteProfiles [] - Get write metadata for table xxxxxxx with instant 20230221104026736 and path: hdfs://master/warehouse/xxxxxxxx error 2023-08-01 19:08:37,498 ERROR org.apache.hudi.sink.partitioner.profile.WriteProfiles [] - Get write metadata for table xxxxxxx with instant 20230220111820722 and path: hdfs://master/warehouse/xxxxxxxx error 2023-08-01 19:08:37,499 ERROR org.apache.hudi.sink.partitioner.profile.WriteProfiles [] - Get write metadata for table xxxxxxx with instant 20230220111820658 and path: hdfs://master/warehouse/xxxxxxxx error 2023-08-01 19:08:37,501 INFO org.apache.hudi.sink.partitioner.profile.WriteProfile [] - Refresh average bytes per record => 1024 2023-08-01 19:08:37,504 INFO org.apache.flink.runtime.io.disk.FileChannelManagerImpl [] - FileChannelManager removed spill file directory hadoop/tmp/nm-local-dir/usercache/mgr/appcache/application_1669378398064_0907/flink-io-0b1f7e76-f0dc-4f0c-bd69-36a4d5e9ed12 ``` **Additional context** I try to modify `org.apache.hudi.sink.partitioner.profile.WriteProfile#averageBytesPerRecord` as below： ``` private long averageBytesPerRecord() { long avgSize = 1024L; LOG.info("Refresh average bytes per record => " + avgSize); return avgSize; } ``` The job can be successfully started ！ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
