dttlgotv opened a new issue #3500: can not submit to yarn in some centos machine! URL: https://github.com/apache/incubator-heron/issues/3500 issue detail： 1. Same heron version(I compiled using last month codes), same hadoop version: 3.2.1, almost same hadoop config, same heron topology 2. submit to yarn always well on mac sometimes can not submit to yarn cluster on three lab centos machine can not submit to yarn on another company centos machine always. This issue has blocked me for some days, and I have to change to use other cluster..... My suspect: 1. heron protobuf data(version 3.6.1) is not compatible with hadoop protobuf(version 2.5.0), currently I just add 3.6.1 protobuf to external path when submitting to yarn. Please help me to check the error below, other log seems no any hint. The error is below: [2020-03-25 10:36:38 +0800] [信息] org.apache.heron.packing.roundrobin.RoundRobinPacking: Pack internal: container CPU hint: 2.000, RAM hint: ByteAmount{1.0 GB (1073741824 bytes)}, disk hint: ByteAmount{-1 bytes}. [2020-03-25 10:36:38 +0800] [信息] org.apache.heron.packing.roundrobin.RoundRobinPacking: Pack internal finalized: container#1 CPU: 2.000000, RAM: ByteAmount{1.0 GB (1073741824 bytes)}, disk: ByteAmount{13.0 GB (13958643712 bytes)}. [2020-03-25 10:36:38 +0800] [信息] org.apache.heron.packing.roundrobin.RoundRobinPacking: Initalizing RoundRobinPacking. CPU default: 1.000000, RAM default: ByteAmount{1.0 GB (1073741824 bytes)}, DISK default: ByteAmount{1.0 GB (1073741824 bytes)}, RAM padding: ByteAmount{2.0 GB (2147483648 bytes)}. [2020-03-25 10:36:38 +0800] [警告] org.apache.heron.packing.roundrobin.RoundRobinPacking: Container#1 (max RAM: ByteAmount{1.0 GB (1073741824 bytes)}) is now hosting instances that take up to ByteAmount{0 bytes} RAM. The container may not have enough resource to accommodate internal processes which take up to ByteAmount{2.0 GB (2147483648 bytes)} RAM. [2020-03-25 10:36:38 +0800] [信息] org.apache.heron.packing.roundrobin.RoundRobinPacking: Pack internal: container CPU hint: 2.000, RAM hint: ByteAmount{1.0 GB (1073741824 bytes)}, disk hint: ByteAmount{-1 bytes}. [2020-03-25 10:36:38 +0800] [信息] org.apache.heron.packing.roundrobin.RoundRobinPacking: Pack internal finalized: container#1 CPU: 2.000000, RAM: ByteAmount{1.0 GB (1073741824 bytes)}, disk: ByteAmount{13.0 GB (13958643712 bytes)}. [2020-03-25 10:36:38 +0800] [信息] org.apache.heron.scheduler.yarn.YarnLauncher: Initializing topology: Test3Topology, core: /root/.heron/dist/heron-core.tar.gz [2020-03-25 10:36:38 +0800] [信息] org.apache.heron.statemgr.zookeeper.curator.CuratorStateManager: Created node for path: /heron/topologies/Test3Topology [2020-03-25 10:36:38 +0800] [信息] org.apache.heron.statemgr.zookeeper.curator.CuratorStateManager: Created node for path: /heron/packingplans/Test3Topology [2020-03-25 10:36:38 +0800] [信息] org.apache.heron.statemgr.zookeeper.curator.CuratorStateManager: Created node for path: /heron/executionstate/Test3Topology [2020-03-25 10:36:38 +0800] [严重] org.apache.reef.runtime.yarn.YarnClasspathProvider: YarnConfiguration.YARN_APPLICATION_CLASSPATH is empty. This indicates a broken cluster configuration. 2020-03-25 10:36:38,705 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable [2020-03-25 10:36:39 +0800] [信息] org.apache.reef.util.REEFVersion: REEF Version: 0.14.0 [2020-03-25 10:36:39 +0800] [信息] org.apache.heron.scheduler.yarn.ReefClientSideHandlers: Initializing REEF client handlers for Heron, topology: Test3Topology [INFO] RMProxy - Connecting to ResourceManager at guoxinghua1/127.0.0.1:8032 [2020-03-25 10:36:51 +0800] [警告] org.apache.reef.runtime.common.files.JobJarMaker: Failed to delete [/tmp/reef-job-1836122165165029413] 2020-03-25 10:36:54,247 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false 2020-03-25 10:36:54,666 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false 2020-03-25 10:36:54,988 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false 2020-03-25 10:36:55,149 INFO conf.Configuration: resource-types.xml not found 2020-03-25 10:36:55,149 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'. [2020-03-25 10:36:55 +0800] [信息] org.apache.reef.runtime.yarn.client.YarnSubmissionHelper: Submitting REEF Application to YARN. ID: application_1585102108714_0002 2020-03-25 10:36:55,210 INFO impl.YarnClientImpl: Submitted application application_1585102108714_0002 [2020-03-25 10:36:59 +0800] [信息] org.apache.heron.scheduler.yarn.ReefClientSideHandlers: Topology Test3Topology is running, jobId Test3Topology. [2020-03-25 10:36:59 +0800] [信息] org.apache.heron.statemgr.zookeeper.curator.CuratorStateManager: Closing the CuratorClient to: 127.0.0.1:2181 2020-03-25 10:36:59,098 INFO imps.CuratorFrameworkImpl: backgroundOperationsLoop exiting 2020-03-25 10:36:59,104 INFO zookeeper.ZooKeeper: Session: 0x1000030d5e70002 closed [2020-03-25 10:36:59 +0800] [信息] org.apache.heron.statemgr.zookeeper.curator.CuratorStateManager: Closing the tunnel processes 2020-03-25 10:36:59,104 INFO zookeeper.ClientCnxn: EventThread shut down for session: 0x1000030d5e70002 [2020-03-25 10:37:04 +0800] [警告] org.apache.reef.runtime.common.client.RuntimeErrorProtoHandler: socket://127.0.0.1:52988 Runtime Error: com.google.protobuf.Descriptors$Descriptor.getOneofs()Ljava/util/List; [2020-03-25 10:37:04 +0800] [严重] org.apache.heron.scheduler.yarn.ReefClientSideHandlers: Failed to start topology: Test3Topology [2020-03-25 10:37:04 +0800] [警告] org.apache.reef.runtime.common.client.RuntimeErrorProtoHandler: socket://127.0.0.1:52990 Runtime Error: Thread main threw an uncaught exception. [2020-03-25 10:37:04 +0800] [严重] org.apache.heron.scheduler.yarn.ReefClientSideHandlers: Failed to start topology: Test3Topology
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
