jilai liu created KYLIN-3494:
--------------------------------
Summary: build cube with spark ArrayIndexOutOfBoundsException
Key: KYLIN-3494
URL: https://issues.apache.org/jira/browse/KYLIN-3494
Project: Kylin
Issue Type: Bug
Components: Job Engine
Affects Versions: v2.4.0
Reporter: jilai liu
Fix For: v1.6.0
Logged in as: dr.who
Application
About
Jobs
Tools
Log Type: stderr
Log Upload Time: Mon Aug 13 15:50:10 +0800 2018
Log Length: 74544
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/data3/test/data/hadoop/hdfs/data/usercache/hadoop/filecache/17809/__spark_libs__6649521663189541594.zip/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/data1/test/soft/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
18/08/13 15:49:38 INFO util.SignalUtils: Registered signal handler for TERM
18/08/13 15:49:38 INFO util.SignalUtils: Registered signal handler for HUP
18/08/13 15:49:38 INFO util.SignalUtils: Registered signal handler for INT
18/08/13 15:49:38 INFO yarn.ApplicationMaster: Preparing Local resources
18/08/13 15:49:39 INFO yarn.ApplicationMaster: ApplicationAttemptId:
appattempt_1533616206085_5657_000001
18/08/13 15:49:39 INFO spark.SecurityManager: Changing view acls to: hadoop
18/08/13 15:49:39 INFO spark.SecurityManager: Changing modify acls to: hadoop
18/08/13 15:49:39 INFO spark.SecurityManager: Changing view acls groups to:
18/08/13 15:49:39 INFO spark.SecurityManager: Changing modify acls groups to:
18/08/13 15:49:39 INFO spark.SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(hadoop); groups
with view permissions: Set(); users with modify permissions: Set(hadoop);
groups with modify permissions: Set()
18/08/13 15:49:39 INFO yarn.ApplicationMaster: Starting the user application in
a separate Thread
18/08/13 15:49:39 INFO yarn.ApplicationMaster: Waiting for spark context
initialization...
18/08/13 15:49:39 INFO spark.SparkContext: Running Spark version 2.1.2
18/08/13 15:49:39 INFO spark.SecurityManager: Changing view acls to: hadoop
18/08/13 15:49:39 INFO spark.SecurityManager: Changing modify acls to: hadoop
18/08/13 15:49:39 INFO spark.SecurityManager: Changing view acls groups to:
18/08/13 15:49:39 INFO spark.SecurityManager: Changing modify acls groups to:
18/08/13 15:49:39 INFO spark.SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(hadoop); groups
with view permissions: Set(); users with modify permissions: Set(hadoop);
groups with modify permissions: Set()
18/08/13 15:49:40 INFO util.Utils: Successfully started service 'sparkDriver'
on port 40358.
18/08/13 15:49:40 INFO spark.SparkEnv: Registering MapOutputTracker
18/08/13 15:49:40 INFO spark.SparkEnv: Registering BlockManagerMaster
18/08/13 15:49:40 INFO storage.BlockManagerMasterEndpoint: Using
org.apache.spark.storage.DefaultTopologyMapper for getting topology information
18/08/13 15:49:40 INFO storage.BlockManagerMasterEndpoint:
BlockManagerMasterEndpoint up
18/08/13 15:49:40 INFO storage.DiskBlockManager: Created local directory at
/data1/test/data/hadoop/hdfs/data/usercache/hadoop/appcache/application_1533616206085_5657/blockmgr-4cd0fed9-78ae-4e2c-826d-b42a8d6364d2
18/08/13 15:49:40 INFO storage.DiskBlockManager: Created local directory at
/data2/test/data/hadoop/hdfs/data/usercache/hadoop/appcache/application_1533616206085_5657/blockmgr-8cca22e9-ece0-469b-b7fa-3cd9567504d9
18/08/13 15:49:40 INFO storage.DiskBlockManager: Created local directory at
/data3/test/data/hadoop/hdfs/data/usercache/hadoop/appcache/application_1533616206085_5657/blockmgr-15f5073d-5ea2-4766-ab22-b8c68834fb80
18/08/13 15:49:40 INFO memory.MemoryStore: MemoryStore started with capacity
305.3 MB
18/08/13 15:49:40 INFO spark.SparkEnv: Registering OutputCommitCoordinator
18/08/13 15:49:40 INFO util.log: Logging initialized @2958ms
18/08/13 15:49:40 INFO ui.JettyUtils: Adding filter:
org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
18/08/13 15:49:40 INFO server.Server: jetty-9.2.z-SNAPSHOT
18/08/13 15:49:40 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@2506206a\{/jobs,null,AVAILABLE,@Spark}
18/08/13 15:49:40 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@7f1b8616\{/jobs/json,null,AVAILABLE,@Spark}
18/08/13 15:49:40 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@5001120\{/jobs/job,null,AVAILABLE,@Spark}
18/08/13 15:49:40 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@4a662152\{/jobs/job/json,null,AVAILABLE,@Spark}
18/08/13 15:49:40 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@5ef75d04\{/stages,null,AVAILABLE,@Spark}
18/08/13 15:49:40 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@7dff5bfa\{/stages/json,null,AVAILABLE,@Spark}
18/08/13 15:49:40 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@78f3dc74\{/stages/stage,null,AVAILABLE,@Spark}
18/08/13 15:49:40 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@3e40e89\{/stages/stage/json,null,AVAILABLE,@Spark}
18/08/13 15:49:40 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@679f6c8c\{/stages/pool,null,AVAILABLE,@Spark}
18/08/13 15:49:40 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@60b8cb0e\{/stages/pool/json,null,AVAILABLE,@Spark}
18/08/13 15:49:40 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@64eab11\{/storage,null,AVAILABLE,@Spark}
18/08/13 15:49:40 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@25fd6d17\{/storage/json,null,AVAILABLE,@Spark}
18/08/13 15:49:40 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@399a8e28\{/storage/rdd,null,AVAILABLE,@Spark}
18/08/13 15:49:40 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@1fdcd2ae\{/storage/rdd/json,null,AVAILABLE,@Spark}
18/08/13 15:49:40 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@6c39f467\{/environment,null,AVAILABLE,@Spark}
18/08/13 15:49:40 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@22e2b922\{/environment/json,null,AVAILABLE,@Spark}
18/08/13 15:49:40 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@4fe49898\{/executors,null,AVAILABLE,@Spark}
18/08/13 15:49:40 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@36d46a68\{/executors/json,null,AVAILABLE,@Spark}
18/08/13 15:49:40 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@15ed2a19\{/executors/threadDump,null,AVAILABLE,@Spark}
18/08/13 15:49:40 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@22f1aa2f\{/executors/threadDump/json,null,AVAILABLE,@Spark}
18/08/13 15:49:40 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@493ad6b1\{/static,null,AVAILABLE,@Spark}
18/08/13 15:49:40 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@5ca862d4\{/,null,AVAILABLE,@Spark}
18/08/13 15:49:40 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@206926ba\{/api,null,AVAILABLE,@Spark}
18/08/13 15:49:40 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@f66404a\{/jobs/job/kill,null,AVAILABLE,@Spark}
18/08/13 15:49:40 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@10f69049\{/stages/stage/kill,null,AVAILABLE,@Spark}
18/08/13 15:49:40 INFO server.ServerConnector: Started
Spark@6e45bafc\{HTTP/1.1}{0.0.0.0:36502}
18/08/13 15:49:40 INFO server.Server: Started @3103ms
18/08/13 15:49:40 INFO util.Utils: Successfully started service 'SparkUI' on
port 36502.
18/08/13 15:49:40 INFO ui.SparkUI: Bound SparkUI to 0.0.0.0, and started at
http://172.16.19.79:36502
18/08/13 15:49:40 INFO cluster.YarnClusterScheduler: Created
YarnClusterScheduler
18/08/13 15:49:40 INFO cluster.SchedulerExtensionServices: Starting Yarn
extension services with app application_1533616206085_5657 and attemptId
Some(appattempt_1533616206085_5657_000001)
18/08/13 15:49:40 INFO util.Utils: Successfully started service
'org.apache.spark.network.netty.NettyBlockTransferService' on port 46273.
18/08/13 15:49:40 INFO netty.NettyBlockTransferService: Server created on
172.16.19.79:46273
18/08/13 15:49:40 INFO storage.BlockManager: Using
org.apache.spark.storage.RandomBlockReplicationPolicy for block replication
policy
18/08/13 15:49:40 INFO storage.BlockManagerMaster: Registering BlockManager
BlockManagerId(driver, 172.16.19.79, 46273, None)
18/08/13 15:49:40 INFO storage.BlockManagerMasterEndpoint: Registering block
manager 172.16.19.79:46273 with 305.3 MB RAM, BlockManagerId(driver,
172.16.19.79, 46273, None)
18/08/13 15:49:40 INFO storage.BlockManagerMaster: Registered BlockManager
BlockManagerId(driver, 172.16.19.79, 46273, None)
18/08/13 15:49:40 INFO storage.BlockManager: Initialized BlockManager:
BlockManagerId(driver, 172.16.19.79, 46273, None)
18/08/13 15:49:40 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@636d84f8\{/metrics/json,null,AVAILABLE,@Spark}
18/08/13 15:49:41 INFO scheduler.EventLoggingListener: Logging events to
hdfs:///kylin/spark-history/application_1533616206085_5657_1
18/08/13 15:49:41 INFO cluster.YarnSchedulerBackend$YarnSchedulerEndpoint:
ApplicationMaster registered as
NettyRpcEndpointRef(spark://[email protected]:40358)
18/08/13 15:49:41 INFO yarn.ApplicationMaster:
===============================================================================
YARN executor launch context:
env:
CLASSPATH ->
\{{PWD}}<CPS>\{{PWD}}/__spark_conf__<CPS>\{{PWD}}/__spark_libs__/*<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/share/hadoop/common/*<CPS>$HADOOP_COMMON_HOME/share/hadoop/common/lib/*<CPS>$HADOOP_HDFS_HOME/share/hadoop/hdfs/*<CPS>$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*<CPS>$HADOOP_YARN_HOME/share/hadoop/yarn/*<CPS>$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*
SPARK_YARN_STAGING_DIR ->
hdfs://test-online/user/hadoop/.sparkStaging/application_1533616206085_5657
SPARK_USER -> hadoop
SPARK_YARN_MODE -> true
command:
\{{JAVA_HOME}}/bin/java \
-server \
-Xmx8192m \
'-Dhdp.version=current' \
-Djava.io.tmpdir=\{{PWD}}/tmp \
-Dspark.yarn.app.container.log.dir=<LOG_DIR> \
-XX:OnOutOfMemoryError='kill %p' \
org.apache.spark.executor.CoarseGrainedExecutorBackend \
--driver-url \
spark://[email protected]:40358 \
--executor-id \
<executorId> \
--hostname \
<hostname> \
--cores \
2 \
--app-id \
application_1533616206085_5657 \
--user-class-path \
file:$PWD/__app__.jar \
1><LOG_DIR>/stdout \
2><LOG_DIR>/stderr
resources:
__app__.jar -> resource \{ scheme: "hdfs" host: "test-online" port: -1 file:
"/user/hadoop/.sparkStaging/application_1533616206085_5657/kylin-job-2.4.0.jar"
} size: 34310788 timestamp: 1534146575706 type: FILE visibility: PRIVATE
__spark_libs__ -> resource \{ scheme: "hdfs" host: "test-online" port: -1
file:
"/user/hadoop/.sparkStaging/application_1533616206085_5657/__spark_libs__6649521663189541594.zip"
} size: 200815831 timestamp: 1534146575510 type: ARCHIVE visibility: PRIVATE
__spark_conf__ -> resource \{ scheme: "hdfs" host: "test-online" port: -1
file:
"/user/hadoop/.sparkStaging/application_1533616206085_5657/__spark_conf__.zip"
} size: 142316 timestamp: 1534146575796 type: ARCHIVE visibility: PRIVATE
===============================================================================
18/08/13 15:49:41 INFO yarn.YarnRMClient: Registering the ApplicationMaster
18/08/13 15:49:41 INFO client.ConfiguredRMFailoverProxyProvider: Failing over
to rm2
18/08/13 15:49:41 INFO yarn.YarnAllocator: Will request 10 executor
container(s), each with 2 core(s) and 9011 MB memory (including 819 MB of
overhead)
18/08/13 15:49:41 INFO yarn.YarnAllocator: Submitted 10 unlocalized container
requests.
18/08/13 15:49:41 INFO yarn.ApplicationMaster: Started progress reporter thread
with (heartbeat : 3000, initial allocation : 200) intervals
18/08/13 15:49:41 INFO impl.AMRMClientImpl: Received new token for :
hadoop073:46083
18/08/13 15:49:41 INFO impl.AMRMClientImpl: Received new token for :
hadoop057:36259
18/08/13 15:49:41 INFO impl.AMRMClientImpl: Received new token for :
hadoop062:33335
18/08/13 15:49:41 INFO impl.AMRMClientImpl: Received new token for :
hadoop067:44632
18/08/13 15:49:41 INFO impl.AMRMClientImpl: Received new token for :
hadoop075:39257
18/08/13 15:49:41 INFO impl.AMRMClientImpl: Received new token for :
hadoop051:45512
18/08/13 15:49:41 INFO yarn.YarnAllocator: Launching container
container_e79_1533616206085_5657_01_000004 on host hadoop073
18/08/13 15:49:41 INFO yarn.YarnAllocator: Launching container
container_e79_1533616206085_5657_01_000005 on host hadoop057
18/08/13 15:49:41 INFO yarn.YarnAllocator: Launching container
container_e79_1533616206085_5657_01_000007 on host hadoop062
18/08/13 15:49:41 INFO yarn.YarnAllocator: Launching container
container_e79_1533616206085_5657_01_000009 on host hadoop067
18/08/13 15:49:41 INFO yarn.YarnAllocator: Launching container
container_e79_1533616206085_5657_01_000010 on host hadoop075
18/08/13 15:49:41 INFO yarn.YarnAllocator: Launching container
container_e79_1533616206085_5657_01_000011 on host hadoop051
18/08/13 15:49:41 INFO yarn.YarnAllocator: Received 6 containers from YARN,
launching executors on 6 of them.
18/08/13 15:49:41 INFO impl.ContainerManagementProtocolProxy:
yarn.client.max-cached-nodemanagers-proxies : 0
18/08/13 15:49:41 INFO impl.ContainerManagementProtocolProxy:
yarn.client.max-cached-nodemanagers-proxies : 0
18/08/13 15:49:41 INFO impl.ContainerManagementProtocolProxy:
yarn.client.max-cached-nodemanagers-proxies : 0
18/08/13 15:49:41 INFO impl.ContainerManagementProtocolProxy:
yarn.client.max-cached-nodemanagers-proxies : 0
18/08/13 15:49:41 INFO impl.ContainerManagementProtocolProxy:
yarn.client.max-cached-nodemanagers-proxies : 0
18/08/13 15:49:41 INFO impl.ContainerManagementProtocolProxy:
yarn.client.max-cached-nodemanagers-proxies : 0
18/08/13 15:49:41 INFO impl.ContainerManagementProtocolProxy: Opening proxy :
hadoop057:36259
18/08/13 15:49:41 INFO impl.ContainerManagementProtocolProxy: Opening proxy :
hadoop067:44632
18/08/13 15:49:41 INFO impl.ContainerManagementProtocolProxy: Opening proxy :
hadoop073:46083
18/08/13 15:49:41 INFO impl.ContainerManagementProtocolProxy: Opening proxy :
hadoop075:39257
18/08/13 15:49:41 INFO impl.ContainerManagementProtocolProxy: Opening proxy :
hadoop051:45512
18/08/13 15:49:41 INFO impl.ContainerManagementProtocolProxy: Opening proxy :
hadoop062:33335
18/08/13 15:49:42 INFO impl.AMRMClientImpl: Received new token for :
hadoop070:42023
18/08/13 15:49:42 INFO impl.AMRMClientImpl: Received new token for :
hadoop063:34154
18/08/13 15:49:42 INFO impl.AMRMClientImpl: Received new token for :
hadoop053:33601
18/08/13 15:49:42 INFO impl.AMRMClientImpl: Received new token for :
hadoop079:40497
18/08/13 15:49:42 INFO yarn.YarnAllocator: Launching container
container_e79_1533616206085_5657_01_000013 on host hadoop070
18/08/13 15:49:42 INFO yarn.YarnAllocator: Launching container
container_e79_1533616206085_5657_01_000017 on host hadoop063
18/08/13 15:49:42 INFO yarn.YarnAllocator: Launching container
container_e79_1533616206085_5657_01_000018 on host hadoop053
18/08/13 15:49:42 INFO yarn.YarnAllocator: Launching container
container_e79_1533616206085_5657_01_000020 on host hadoop079
18/08/13 15:49:42 INFO yarn.YarnAllocator: Received 4 containers from YARN,
launching executors on 4 of them.
18/08/13 15:49:42 INFO impl.ContainerManagementProtocolProxy:
yarn.client.max-cached-nodemanagers-proxies : 0
18/08/13 15:49:42 INFO impl.ContainerManagementProtocolProxy:
yarn.client.max-cached-nodemanagers-proxies : 0
18/08/13 15:49:42 INFO impl.ContainerManagementProtocolProxy:
yarn.client.max-cached-nodemanagers-proxies : 0
18/08/13 15:49:42 INFO impl.ContainerManagementProtocolProxy:
yarn.client.max-cached-nodemanagers-proxies : 0
18/08/13 15:49:42 INFO impl.ContainerManagementProtocolProxy: Opening proxy :
hadoop079:40497
18/08/13 15:49:42 INFO impl.ContainerManagementProtocolProxy: Opening proxy :
hadoop070:42023
18/08/13 15:49:42 INFO impl.ContainerManagementProtocolProxy: Opening proxy :
hadoop063:34154
18/08/13 15:49:42 INFO impl.ContainerManagementProtocolProxy: Opening proxy :
hadoop053:33601
18/08/13 15:49:44 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint:
Registered executor NettyRpcEndpointRef(null) (172.16.19.79:39942) with ID 10
18/08/13 15:49:44 INFO storage.BlockManagerMasterEndpoint: Registering block
manager hadoop079:44235 with 3.4 GB RAM, BlockManagerId(10, hadoop079, 44235,
None)
18/08/13 15:49:45 INFO impl.AMRMClientImpl: Received new token for :
hadoop060:39607
18/08/13 15:49:45 INFO impl.AMRMClientImpl: Received new token for :
hadoop084:38240
18/08/13 15:49:45 INFO impl.AMRMClientImpl: Received new token for :
hadoop061:40680
18/08/13 15:49:45 INFO impl.AMRMClientImpl: Received new token for :
hadoop074:38414
18/08/13 15:49:45 INFO yarn.YarnAllocator: Received 4 containers from YARN,
launching executors on 0 of them.
18/08/13 15:49:45 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint:
Registered executor NettyRpcEndpointRef(null) (172.16.19.57:35082) with ID 2
18/08/13 15:49:45 INFO storage.BlockManagerMasterEndpoint: Registering block
manager hadoop057:40961 with 3.4 GB RAM, BlockManagerId(2, hadoop057, 40961,
None)
18/08/13 15:49:45 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint:
Registered executor NettyRpcEndpointRef(null) (172.16.19.51:35812) with ID 6
18/08/13 15:49:45 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint:
Registered executor NettyRpcEndpointRef(null) (172.16.19.75:45130) with ID 5
18/08/13 15:49:45 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint:
Registered executor NettyRpcEndpointRef(null) (172.16.19.63:38512) with ID 8
18/08/13 15:49:45 INFO storage.BlockManagerMasterEndpoint: Registering block
manager hadoop075:33510 with 3.4 GB RAM, BlockManagerId(5, hadoop075, 33510,
None)
18/08/13 15:49:45 INFO storage.BlockManagerMasterEndpoint: Registering block
manager hadoop063:46329 with 3.4 GB RAM, BlockManagerId(8, hadoop063, 46329,
None)
18/08/13 15:49:45 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint:
Registered executor NettyRpcEndpointRef(null) (172.16.19.73:33060) with ID 1
18/08/13 15:49:45 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint:
Registered executor NettyRpcEndpointRef(null) (172.16.19.53:38988) with ID 9
18/08/13 15:49:45 INFO storage.BlockManagerMasterEndpoint: Registering block
manager hadoop051:43539 with 3.4 GB RAM, BlockManagerId(6, hadoop051, 43539,
None)
18/08/13 15:49:45 INFO storage.BlockManagerMasterEndpoint: Registering block
manager hadoop053:40239 with 3.4 GB RAM, BlockManagerId(9, hadoop053, 40239,
None)
18/08/13 15:49:45 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint:
Registered executor NettyRpcEndpointRef(null) (172.16.19.67:33364) with ID 4
18/08/13 15:49:45 INFO storage.BlockManagerMasterEndpoint: Registering block
manager hadoop073:39426 with 3.4 GB RAM, BlockManagerId(1, hadoop073, 39426,
None)
18/08/13 15:49:45 INFO storage.BlockManagerMasterEndpoint: Registering block
manager hadoop067:37637 with 3.4 GB RAM, BlockManagerId(4, hadoop067, 37637,
None)
18/08/13 15:49:45 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint:
Registered executor NettyRpcEndpointRef(null) (172.16.19.62:37576) with ID 3
18/08/13 15:49:45 INFO cluster.YarnClusterSchedulerBackend: SchedulerBackend is
ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
18/08/13 15:49:45 INFO cluster.YarnClusterScheduler:
YarnClusterScheduler.postStartHook done
18/08/13 15:49:45 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint:
Registered executor NettyRpcEndpointRef(null) (172.16.19.70:59750) with ID 7
18/08/13 15:49:45 INFO common.AbstractHadoopJob: Ready to load KylinConfig from
uri:
kylin_metadata@hdfs,path=hdfs://test-online/kylin/kylin_metadata/kylin-ed7d5d0d-1007-404d-8512-73e946cfaa73/commonlog_andriod_cube7/metadata
18/08/13 15:49:45 INFO storage.BlockManagerMasterEndpoint: Registering block
manager hadoop062:34791 with 3.4 GB RAM, BlockManagerId(3, hadoop062, 34791,
None)
18/08/13 15:49:45 INFO storage.BlockManagerMasterEndpoint: Registering block
manager hadoop070:34116 with 3.4 GB RAM, BlockManagerId(7, hadoop070, 34116,
None)
18/08/13 15:49:46 INFO common.KylinConfig: Creating new manager instance of
class org.apache.kylin.cube.CubeManager
18/08/13 15:49:46 INFO cube.CubeManager: Initializing CubeManager with config
null
18/08/13 15:49:46 INFO persistence.ResourceStore: Using metadata url
kylin_metadata@hdfs,path=hdfs://test-online/kylin/kylin_metadata/kylin-ed7d5d0d-1007-404d-8512-73e946cfaa73/commonlog_andriod_cube7/metadata
for resource store
18/08/13 15:49:46 INFO persistence.HDFSResourceStore: hdfs meta path :
hdfs://test-online/kylin/kylin_metadata/kylin-ed7d5d0d-1007-404d-8512-73e946cfaa73/commonlog_andriod_cube7/metadata
18/08/13 15:49:46 INFO common.KylinConfig: Creating new manager instance of
class org.apache.kylin.cube.CubeDescManager
18/08/13 15:49:46 INFO cube.CubeDescManager: Initializing CubeDescManager with
config
kylin_metadata@hdfs,path=hdfs://test-online/kylin/kylin_metadata/kylin-ed7d5d0d-1007-404d-8512-73e946cfaa73/commonlog_andriod_cube7/metadata
18/08/13 15:49:46 INFO common.KylinConfig: Creating new manager instance of
class org.apache.kylin.metadata.project.ProjectManager
18/08/13 15:49:46 INFO project.ProjectManager: Initializing ProjectManager with
metadata url
kylin_metadata@hdfs,path=hdfs://test-online/kylin/kylin_metadata/kylin-ed7d5d0d-1007-404d-8512-73e946cfaa73/commonlog_andriod_cube7/metadata
18/08/13 15:49:46 INFO common.KylinConfig: Creating new manager instance of
class org.apache.kylin.metadata.cachesync.Broadcaster
18/08/13 15:49:46 INFO common.KylinConfig: Creating new manager instance of
class org.apache.kylin.metadata.model.DataModelManager
18/08/13 15:49:46 INFO common.KylinConfig: Creating new manager instance of
class org.apache.kylin.metadata.TableMetadataManager
18/08/13 15:49:46 INFO measure.MeasureTypeFactory: Checking custom measure
types from kylin config
18/08/13 15:49:46 INFO measure.MeasureTypeFactory: registering
COUNT_DISTINCT(hllc), class
org.apache.kylin.measure.hllc.HLLCMeasureType$Factory
18/08/13 15:49:46 INFO measure.MeasureTypeFactory: registering
COUNT_DISTINCT(bitmap), class
org.apache.kylin.measure.bitmap.BitmapMeasureType$Factory
18/08/13 15:49:46 INFO measure.MeasureTypeFactory: registering TOP_N(topn),
class org.apache.kylin.measure.topn.TopNMeasureType$Factory
18/08/13 15:49:46 INFO measure.MeasureTypeFactory: registering RAW(raw), class
org.apache.kylin.measure.raw.RawMeasureType$Factory
18/08/13 15:49:46 INFO measure.MeasureTypeFactory: registering
EXTENDED_COLUMN(extendedcolumn), class
org.apache.kylin.measure.extendedcolumn.ExtendedColumnMeasureType$Factory
18/08/13 15:49:46 INFO measure.MeasureTypeFactory: registering
PERCENTILE_APPROX(percentile), class
org.apache.kylin.measure.percentile.PercentileMeasureType$Factory
18/08/13 15:49:46 INFO measure.MeasureTypeFactory: registering
COUNT_DISTINCT(dim_dc), class
org.apache.kylin.measure.dim.DimCountDistinctMeasureType$Factory
18/08/13 15:49:46 INFO model.DataModelManager: Model commonlog_iphone_model is
missing or unloaded yet
18/08/13 15:49:46 INFO model.DataModelManager: Model commonlog_andriod_model is
missing or unloaded yet
18/08/13 15:49:46 INFO spark.SparkCubingByLayer: RDD input path:
hdfs://test-online/kylin/kylin_metadata/kylin-ed7d5d0d-1007-404d-8512-73e946cfaa73/kylin_intermediate_commonlog_andriod_cube7_d1db5d22_cf49_48fc_b9c0_54e8e70cf362
18/08/13 15:49:46 INFO spark.SparkCubingByLayer: RDD Output path:
hdfs://test-online/kylin/kylin_metadata/kylin-ed7d5d0d-1007-404d-8512-73e946cfaa73/commonlog_andriod_cube7/cuboid/
18/08/13 15:49:46 INFO zlib.ZlibFactory: Successfully loaded & initialized
native-zlib library
18/08/13 15:49:46 INFO compress.CodecPool: Got brand-new decompressor [.deflate]
18/08/13 15:49:46 INFO spark.SparkCubingByLayer: All measure are normal (agg on
all cuboids) ? : true
18/08/13 15:49:47 INFO memory.MemoryStore: Block broadcast_0 stored as values
in memory (estimated size 297.1 KB, free 305.0 MB)
18/08/13 15:49:47 INFO memory.MemoryStore: Block broadcast_0_piece0 stored as
bytes in memory (estimated size 25.5 KB, free 304.9 MB)
18/08/13 15:49:47 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in
memory on 172.16.19.79:46273 (size: 25.5 KB, free: 305.2 MB)
18/08/13 15:49:47 INFO spark.SparkContext: Created broadcast 0 from
sequenceFile at SparkCubingByLayer.java:182
18/08/13 15:49:47 INFO common.KylinConfig: Creating new manager instance of
class org.apache.kylin.cube.cuboid.CuboidManager
18/08/13 15:49:47 INFO common.KylinConfig: Creating new manager instance of
class org.apache.kylin.dict.DictionaryManager
18/08/13 15:49:47 INFO dict.DictionaryManager: DictionaryManager(258090034)
loading DictionaryInfo(loadDictObj:true) at
/dict/ODS.COMMONLOG_ANDROID/APP_VERSION/f0950bcb-7ec1-40b2-b7aa-394989976397.dict
18/08/13 15:49:47 INFO dict.DictionaryManager: DictionaryManager(258090034)
loading DictionaryInfo(loadDictObj:true) at
/dict/ODS.COMMONLOG_ANDROID/PID/b91eeb9d-a818-4648-816d-8ec209a97641.dict
18/08/13 15:49:47 INFO dict.DictionaryManager: DictionaryManager(258090034)
loading DictionaryInfo(loadDictObj:true) at
/dict/ODS.COMMONLOG_ANDROID/LANGUAGE/f63e6df9-73b8-47c1-afa4-2d5bd43926c3.dict
18/08/13 15:49:47 INFO dict.DictionaryManager: DictionaryManager(258090034)
loading DictionaryInfo(loadDictObj:true) at
/dict/ODS.COMMONLOG_ANDROID/OS_VERSION/2c3fa118-448e-45b4-b8c7-c8e2c5d3090c.dict
18/08/13 15:49:47 INFO dict.DictionaryManager: DictionaryManager(258090034)
loading DictionaryInfo(loadDictObj:true) at
/dict/ODS.COMMONLOG_ANDROID/VALUE/c1e34ccc-68a2-4acb-a643-8d17c3067e74.dict
18/08/13 15:49:47 INFO dict.DictionaryManager: DictionaryManager(258090034)
loading DictionaryInfo(loadDictObj:true) at
/dict/ODS.COMMONLOG_ANDROID/DU/e125650e-2cb8-4b51-ba99-739476b08d33.dict
18/08/13 15:49:47 INFO dict.DictionaryManager: DictionaryManager(258090034)
loading DictionaryInfo(loadDictObj:true) at
/dict/ODS.COMMONLOG_ANDROID/KEY/b779f6d9-23a6-45e2-8dbf-83f898670ec2.dict
18/08/13 15:49:47 INFO dict.DictionaryManager: DictionaryManager(258090034)
loading DictionaryInfo(loadDictObj:true) at
/dict/ODS.COMMONLOG_ANDROID/NET/dba93801-f8cc-4958-97ba-e2cdeec7abd4.dict
18/08/13 15:49:47 INFO dict.DictionaryManager: DictionaryManager(258090034)
loading DictionaryInfo(loadDictObj:true) at
/dict/ODS.COMMONLOG_ANDROID/CITYID/ebadaeae-c3dd-45d6-9cb8-4c0ecef82cee.dict
18/08/13 15:49:47 INFO common.CubeStatsReader: Estimating size for layer 0, all
cuboids are 511, total size is 449.9567413330078
18/08/13 15:49:47 INFO spark.SparkCubingByLayer: Partition for spark cubing: 44
18/08/13 15:49:47 INFO output.FileOutputCommitter: File Output Committer
Algorithm version is 1
18/08/13 15:49:47 INFO spark.SparkContext: Starting job:
saveAsNewAPIHadoopDataset at SparkCubingByLayer.java:277
18/08/13 15:49:47 INFO mapred.FileInputFormat: Total input paths to process : 44
18/08/13 15:49:48 INFO net.NetworkTopology: Adding a new node:
/dc1/rack6/172.16.19.86:50010
18/08/13 15:49:48 INFO net.NetworkTopology: Adding a new node:
/default/rack/172.16.19.99:50010
18/08/13 15:49:48 INFO net.NetworkTopology: Adding a new node:
/default/rack/172.16.19.106:50010
18/08/13 15:49:48 INFO net.NetworkTopology: Adding a new node:
/dc1/rack4/172.16.19.73:50010
18/08/13 15:49:48 INFO net.NetworkTopology: Adding a new node:
/dc1/rack4/172.16.19.74:50010
18/08/13 15:49:48 INFO net.NetworkTopology: Adding a new node:
/dc1/rack5/172.16.19.84:50010
18/08/13 15:49:48 INFO net.NetworkTopology: Adding a new node:
/default/rack/172.16.19.103:50010
18/08/13 15:49:48 INFO net.NetworkTopology: Adding a new node:
/dc1/rack4/172.16.19.71:50010
18/08/13 15:49:48 INFO net.NetworkTopology: Adding a new node:
/dc1/rack4/172.16.19.72:50010
18/08/13 15:49:48 INFO net.NetworkTopology: Adding a new node:
/dc1/rack2/172.16.19.59:50010
18/08/13 15:49:48 INFO net.NetworkTopology: Adding a new node:
/dc1/rack1/172.16.19.51:50010
18/08/13 15:49:48 INFO net.NetworkTopology: Adding a new node:
/dc1/rack1/172.16.19.50:50010
18/08/13 15:49:48 INFO net.NetworkTopology: Adding a new node:
/dc1/rack5/172.16.19.81:50010
18/08/13 15:49:48 INFO net.NetworkTopology: Adding a new node:
/dc1/rack5/172.16.19.83:50010
18/08/13 15:49:48 INFO scheduler.DAGScheduler: Registering RDD 3 (mapToPair at
SparkCubingByLayer.java:182)
18/08/13 15:49:48 INFO scheduler.DAGScheduler: Got job 0
(saveAsNewAPIHadoopDataset at SparkCubingByLayer.java:277) with 44 output
partitions
18/08/13 15:49:48 INFO scheduler.DAGScheduler: Final stage: ResultStage 1
(saveAsNewAPIHadoopDataset at SparkCubingByLayer.java:277)
18/08/13 15:49:48 INFO scheduler.DAGScheduler: Parents of final stage:
List(ShuffleMapStage 0)
18/08/13 15:49:48 INFO scheduler.DAGScheduler: Missing parents:
List(ShuffleMapStage 0)
18/08/13 15:49:48 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 0
(MapPartitionsRDD[3] at mapToPair at SparkCubingByLayer.java:182), which has no
missing parents
18/08/13 15:49:48 INFO memory.MemoryStore: Block broadcast_1 stored as values
in memory (estimated size 79.7 KB, free 304.9 MB)
18/08/13 15:49:48 INFO memory.MemoryStore: Block broadcast_1_piece0 stored as
bytes in memory (estimated size 30.0 KB, free 304.8 MB)
18/08/13 15:49:48 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in
memory on 172.16.19.79:46273 (size: 30.0 KB, free: 305.2 MB)
18/08/13 15:49:48 INFO spark.SparkContext: Created broadcast 1 from broadcast
at DAGScheduler.scala:996
18/08/13 15:49:48 INFO scheduler.DAGScheduler: Submitting 105 missing tasks
from ShuffleMapStage 0 (MapPartitionsRDD[3] at mapToPair at
SparkCubingByLayer.java:182)
18/08/13 15:49:48 INFO cluster.YarnClusterScheduler: Adding task set 0.0 with
105 tasks
18/08/13 15:49:48 INFO scheduler.TaskSetManager: Starting task 14.0 in stage
0.0 (TID 0, hadoop067, executor 4, partition 14, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:48 INFO scheduler.TaskSetManager: Starting task 4.0 in stage 0.0
(TID 1, hadoop075, executor 5, partition 4, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:48 INFO scheduler.TaskSetManager: Starting task 10.0 in stage
0.0 (TID 2, hadoop057, executor 2, partition 10, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:48 INFO scheduler.TaskSetManager: Starting task 13.0 in stage
0.0 (TID 3, hadoop053, executor 9, partition 13, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:48 INFO scheduler.TaskSetManager: Starting task 11.0 in stage
0.0 (TID 4, hadoop070, executor 7, partition 11, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:48 INFO scheduler.TaskSetManager: Starting task 16.0 in stage
0.0 (TID 5, hadoop062, executor 3, partition 16, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:48 INFO scheduler.TaskSetManager: Starting task 18.0 in stage
0.0 (TID 6, hadoop079, executor 10, partition 18, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:48 INFO scheduler.TaskSetManager: Starting task 31.0 in stage
0.0 (TID 7, hadoop063, executor 8, partition 31, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:48 INFO scheduler.TaskSetManager: Starting task 25.0 in stage
0.0 (TID 8, hadoop073, executor 1, partition 25, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:48 INFO scheduler.TaskSetManager: Starting task 5.0 in stage 0.0
(TID 9, hadoop075, executor 5, partition 5, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:48 INFO scheduler.TaskSetManager: Starting task 45.0 in stage
0.0 (TID 10, hadoop057, executor 2, partition 45, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:48 INFO scheduler.TaskSetManager: Starting task 22.0 in stage
0.0 (TID 11, hadoop053, executor 9, partition 22, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:48 INFO scheduler.TaskSetManager: Starting task 19.0 in stage
0.0 (TID 12, hadoop070, executor 7, partition 19, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:48 INFO scheduler.TaskSetManager: Starting task 42.0 in stage
0.0 (TID 13, hadoop062, executor 3, partition 42, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:48 INFO scheduler.TaskSetManager: Starting task 21.0 in stage
0.0 (TID 14, hadoop079, executor 10, partition 21, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:48 INFO scheduler.TaskSetManager: Starting task 33.0 in stage
0.0 (TID 15, hadoop063, executor 8, partition 33, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:48 INFO scheduler.TaskSetManager: Starting task 26.0 in stage
0.0 (TID 16, hadoop073, executor 1, partition 26, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:48 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in
memory on hadoop079:44235 (size: 30.0 KB, free: 3.4 GB)
18/08/13 15:49:48 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in
memory on hadoop053:40239 (size: 30.0 KB, free: 3.4 GB)
18/08/13 15:49:48 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in
memory on hadoop073:39426 (size: 30.0 KB, free: 3.4 GB)
18/08/13 15:49:48 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in
memory on hadoop067:37637 (size: 30.0 KB, free: 3.4 GB)
18/08/13 15:49:48 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in
memory on hadoop070:34116 (size: 30.0 KB, free: 3.4 GB)
18/08/13 15:49:48 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in
memory on hadoop063:46329 (size: 30.0 KB, free: 3.4 GB)
18/08/13 15:49:48 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in
memory on hadoop075:33510 (size: 30.0 KB, free: 3.4 GB)
18/08/13 15:49:48 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in
memory on hadoop062:34791 (size: 30.0 KB, free: 3.4 GB)
18/08/13 15:49:48 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in
memory on hadoop057:40961 (size: 30.0 KB, free: 3.4 GB)
18/08/13 15:49:49 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in
memory on hadoop062:34791 (size: 25.5 KB, free: 3.4 GB)
18/08/13 15:49:49 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in
memory on hadoop075:33510 (size: 25.5 KB, free: 3.4 GB)
18/08/13 15:49:49 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in
memory on hadoop070:34116 (size: 25.5 KB, free: 3.4 GB)
18/08/13 15:49:49 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in
memory on hadoop067:37637 (size: 25.5 KB, free: 3.4 GB)
18/08/13 15:49:49 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in
memory on hadoop073:39426 (size: 25.5 KB, free: 3.4 GB)
18/08/13 15:49:49 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in
memory on hadoop063:46329 (size: 25.5 KB, free: 3.4 GB)
18/08/13 15:49:49 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in
memory on hadoop057:40961 (size: 25.5 KB, free: 3.4 GB)
18/08/13 15:49:49 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in
memory on hadoop053:40239 (size: 25.5 KB, free: 3.4 GB)
18/08/13 15:49:49 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in
memory on hadoop079:44235 (size: 25.5 KB, free: 3.4 GB)
18/08/13 15:49:51 INFO scheduler.TaskSetManager: Starting task 15.0 in stage
0.0 (TID 17, hadoop067, executor 4, partition 15, RACK_LOCAL, 6146 bytes)
18/08/13 15:49:51 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0
(TID 18, hadoop051, executor 6, partition 1, RACK_LOCAL, 6146 bytes)
18/08/13 15:49:51 INFO scheduler.TaskSetManager: Starting task 12.0 in stage
0.0 (TID 19, hadoop051, executor 6, partition 12, RACK_LOCAL, 6146 bytes)
18/08/13 15:49:51 INFO scheduler.TaskSetManager: Starting task 6.0 in stage 0.0
(TID 20, hadoop075, executor 5, partition 6, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:51 INFO scheduler.TaskSetManager: Starting task 7.0 in stage 0.0
(TID 21, hadoop075, executor 5, partition 7, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:51 WARN scheduler.TaskSetManager: Lost task 4.0 in stage 0.0
(TID 1, hadoop075, executor 5): java.lang.ArrayIndexOutOfBoundsException: 8
at
org.apache.kylin.engine.mr.common.BaseCuboidBuilder.getCell(BaseCuboidBuilder.java:167)
at
org.apache.kylin.engine.mr.common.BaseCuboidBuilder.buildKey(BaseCuboidBuilder.java:116)
at
org.apache.kylin.engine.spark.SparkCubingByLayer$EncodeBaseCuboid.call(SparkCubingByLayer.java:343)
at
org.apache.kylin.engine.spark.SparkCubingByLayer$EncodeBaseCuboid.call(SparkCubingByLayer.java:307)
at
org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1043)
at
org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1043)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at
org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:193)
at
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
18/08/13 15:49:51 INFO scheduler.TaskSetManager: Lost task 5.0 in stage 0.0
(TID 9) on hadoop075, executor 5: java.lang.ArrayIndexOutOfBoundsException (8)
[duplicate 1]
18/08/13 15:49:51 INFO scheduler.TaskSetManager: Starting task 20.0 in stage
0.0 (TID 22, hadoop070, executor 7, partition 20, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:51 INFO scheduler.TaskSetManager: Lost task 19.0 in stage 0.0
(TID 12) on hadoop070, executor 7: java.lang.ArrayIndexOutOfBoundsException (8)
[duplicate 2]
18/08/13 15:49:51 INFO scheduler.TaskSetManager: Starting task 19.1 in stage
0.0 (TID 23, hadoop070, executor 7, partition 19, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:51 INFO scheduler.TaskSetManager: Lost task 11.0 in stage 0.0
(TID 4) on hadoop070, executor 7: java.lang.ArrayIndexOutOfBoundsException (8)
[duplicate 3]
18/08/13 15:49:51 INFO scheduler.TaskSetManager: Lost task 14.0 in stage 0.0
(TID 0) on hadoop067, executor 4: java.lang.ArrayIndexOutOfBoundsException (8)
[duplicate 4]
18/08/13 15:49:51 INFO scheduler.TaskSetManager: Starting task 14.1 in stage
0.0 (TID 24, hadoop067, executor 4, partition 14, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:51 INFO scheduler.TaskSetManager: Starting task 50.0 in stage
0.0 (TID 25, hadoop079, executor 10, partition 50, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:51 INFO scheduler.TaskSetManager: Lost task 21.0 in stage 0.0
(TID 14) on hadoop079, executor 10: java.lang.ArrayIndexOutOfBoundsException
(8) [duplicate 5]
18/08/13 15:49:51 INFO scheduler.TaskSetManager: Starting task 52.0 in stage
0.0 (TID 26, hadoop057, executor 2, partition 52, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:51 INFO scheduler.TaskSetManager: Lost task 45.0 in stage 0.0
(TID 10) on hadoop057, executor 2: java.lang.ArrayIndexOutOfBoundsException (8)
[duplicate 6]
18/08/13 15:49:51 INFO scheduler.TaskSetManager: Starting task 45.1 in stage
0.0 (TID 27, hadoop057, executor 2, partition 45, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:51 INFO scheduler.TaskSetManager: Lost task 10.0 in stage 0.0
(TID 2) on hadoop057, executor 2: java.lang.ArrayIndexOutOfBoundsException (8)
[duplicate 7]
18/08/13 15:49:51 INFO scheduler.TaskSetManager: Lost task 15.0 in stage 0.0
(TID 17) on hadoop067, executor 4: java.lang.ArrayIndexOutOfBoundsException (8)
[duplicate 8]
18/08/13 15:49:51 INFO scheduler.TaskSetManager: Starting task 10.1 in stage
0.0 (TID 28, hadoop062, executor 3, partition 10, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:51 INFO scheduler.TaskSetManager: Lost task 42.0 in stage 0.0
(TID 13) on hadoop062, executor 3: java.lang.ArrayIndexOutOfBoundsException (8)
[duplicate 9]
18/08/13 15:49:51 INFO scheduler.TaskSetManager: Starting task 42.1 in stage
0.0 (TID 29, hadoop062, executor 3, partition 42, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:51 INFO scheduler.TaskSetManager: Lost task 16.0 in stage 0.0
(TID 5) on hadoop062, executor 3: java.lang.ArrayIndexOutOfBoundsException (8)
[duplicate 10]
18/08/13 15:49:52 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in
memory on hadoop051:43539 (size: 30.0 KB, free: 3.4 GB)
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Lost task 14.1 in stage 0.0
(TID 24) on hadoop067, executor 4: java.lang.ArrayIndexOutOfBoundsException (8)
[duplicate 11]
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Starting task 14.2 in stage
0.0 (TID 30, hadoop067, executor 4, partition 14, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Starting task 11.1 in stage
0.0 (TID 31, hadoop075, executor 5, partition 11, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Lost task 7.0 in stage 0.0
(TID 21) on hadoop075, executor 5: java.lang.ArrayIndexOutOfBoundsException (8)
[duplicate 12]
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Starting task 7.1 in stage 0.0
(TID 32, hadoop075, executor 5, partition 7, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Lost task 6.0 in stage 0.0
(TID 20) on hadoop075, executor 5: java.lang.ArrayIndexOutOfBoundsException (8)
[duplicate 13]
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Starting task 27.0 in stage
0.0 (TID 33, hadoop073, executor 1, partition 27, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Lost task 26.0 in stage 0.0
(TID 16) on hadoop073, executor 1: java.lang.ArrayIndexOutOfBoundsException (8)
[duplicate 14]
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Starting task 26.1 in stage
0.0 (TID 34, hadoop073, executor 1, partition 26, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Starting task 90.0 in stage
0.0 (TID 35, hadoop063, executor 8, partition 90, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Lost task 25.0 in stage 0.0
(TID 8) on hadoop073, executor 1: java.lang.ArrayIndexOutOfBoundsException (8)
[duplicate 15]
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Lost task 33.0 in stage 0.0
(TID 15) on hadoop063, executor 8: java.lang.ArrayIndexOutOfBoundsException (8)
[duplicate 16]
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Starting task 21.1 in stage
0.0 (TID 36, hadoop070, executor 7, partition 21, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Lost task 20.0 in stage 0.0
(TID 22) on hadoop070, executor 7: java.lang.ArrayIndexOutOfBoundsException (8)
[duplicate 17]
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Starting task 75.0 in stage
0.0 (TID 37, hadoop057, executor 2, partition 75, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Lost task 52.0 in stage 0.0
(TID 26) on hadoop057, executor 2: java.lang.ArrayIndexOutOfBoundsException (8)
[duplicate 18]
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Starting task 20.1 in stage
0.0 (TID 38, hadoop070, executor 7, partition 20, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Lost task 19.1 in stage 0.0
(TID 23) on hadoop070, executor 7: java.lang.ArrayIndexOutOfBoundsException (8)
[duplicate 19]
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Starting task 52.1 in stage
0.0 (TID 39, hadoop057, executor 2, partition 52, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Lost task 45.1 in stage 0.0
(TID 27) on hadoop057, executor 2: java.lang.ArrayIndexOutOfBoundsException (8)
[duplicate 20]
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Lost task 50.0 in stage 0.0
(TID 25) on hadoop079, executor 10: java.lang.ArrayIndexOutOfBoundsException
(8) [duplicate 21]
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Starting task 50.1 in stage
0.0 (TID 40, hadoop079, executor 10, partition 50, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Starting task 33.1 in stage
0.0 (TID 41, hadoop063, executor 8, partition 33, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Lost task 31.0 in stage 0.0
(TID 7) on hadoop063, executor 8: java.lang.ArrayIndexOutOfBoundsException (8)
[duplicate 22]
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Lost task 18.0 in stage 0.0
(TID 6) on hadoop079, executor 10: java.lang.ArrayIndexOutOfBoundsException (8)
[duplicate 23]
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Starting task 18.1 in stage
0.0 (TID 42, hadoop079, executor 10, partition 18, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Starting task 19.2 in stage
0.0 (TID 43, hadoop070, executor 7, partition 19, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Lost task 21.1 in stage 0.0
(TID 36) on hadoop070, executor 7: java.lang.ArrayIndexOutOfBoundsException (8)
[duplicate 24]
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Lost task 14.2 in stage 0.0
(TID 30) on hadoop067, executor 4: java.lang.ArrayIndexOutOfBoundsException (8)
[duplicate 25]
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Starting task 14.3 in stage
0.0 (TID 44, hadoop067, executor 4, partition 14, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Starting task 31.1 in stage
0.0 (TID 45, hadoop063, executor 8, partition 31, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Lost task 90.0 in stage 0.0
(TID 35) on hadoop063, executor 8: java.lang.ArrayIndexOutOfBoundsException (8)
[duplicate 26]
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Starting task 16.1 in stage
0.0 (TID 46, hadoop062, executor 3, partition 16, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Lost task 42.1 in stage 0.0
(TID 29) on hadoop062, executor 3: java.lang.ArrayIndexOutOfBoundsException (8)
[duplicate 27]
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Starting task 45.2 in stage
0.0 (TID 47, hadoop057, executor 2, partition 45, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Lost task 52.1 in stage 0.0
(TID 39) on hadoop057, executor 2: java.lang.ArrayIndexOutOfBoundsException (8)
[duplicate 28]
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Starting task 42.2 in stage
0.0 (TID 48, hadoop062, executor 3, partition 42, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Lost task 10.1 in stage 0.0
(TID 28) on hadoop062, executor 3: java.lang.ArrayIndexOutOfBoundsException (8)
[duplicate 29]
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Starting task 10.2 in stage
0.0 (TID 49, hadoop075, executor 5, partition 10, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Lost task 7.1 in stage 0.0
(TID 32) on hadoop075, executor 5: java.lang.ArrayIndexOutOfBoundsException (8)
[duplicate 30]
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Starting task 21.2 in stage
0.0 (TID 50, hadoop070, executor 7, partition 21, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Lost task 20.1 in stage 0.0
(TID 38) on hadoop070, executor 7: java.lang.ArrayIndexOutOfBoundsException (8)
[duplicate 31]
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Starting task 7.2 in stage 0.0
(TID 51, hadoop075, executor 5, partition 7, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Lost task 11.1 in stage 0.0
(TID 31) on hadoop075, executor 5: java.lang.ArrayIndexOutOfBoundsException (8)
[duplicate 32]
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Starting task 52.2 in stage
0.0 (TID 52, hadoop057, executor 2, partition 52, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Lost task 75.0 in stage 0.0
(TID 37) on hadoop057, executor 2: java.lang.ArrayIndexOutOfBoundsException (8)
[duplicate 33]
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Starting task 35.0 in stage
0.0 (TID 53, hadoop053, executor 9, partition 35, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Lost task 22.0 in stage 0.0
(TID 11) on hadoop053, executor 9: java.lang.ArrayIndexOutOfBoundsException (8)
[duplicate 34]
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Starting task 22.1 in stage
0.0 (TID 54, hadoop053, executor 9, partition 22, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Lost task 13.0 in stage 0.0
(TID 3) on hadoop053, executor 9: java.lang.ArrayIndexOutOfBoundsException (8)
[duplicate 35]
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Starting task 90.1 in stage
0.0 (TID 55, hadoop073, executor 1, partition 90, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Lost task 26.1 in stage 0.0
(TID 34) on hadoop073, executor 1: java.lang.ArrayIndexOutOfBoundsException (8)
[duplicate 36]
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Starting task 26.2 in stage
0.0 (TID 56, hadoop073, executor 1, partition 26, NODE_LOCAL, 6146 bytes)
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Lost task 27.0 in stage 0.0
(TID 33) on hadoop073, executor 1: java.lang.ArrayIndexOutOfBoundsException (8)
[duplicate 37]
18/08/13 15:49:52 INFO scheduler.TaskSetManager: Lost task 14.3 in stage 0.0
(TID 44) on hadoop067, executor 4: java.lang.ArrayIndexOutOfBoundsException (8)
[duplicate 38]
18/08/13 15:49:52 ERROR scheduler.TaskSetManager: Task 14 in stage 0.0 failed 4
times; aborting job
18/08/13 15:49:52 INFO cluster.YarnClusterScheduler: Cancelling stage 0
18/08/13 15:49:52 INFO cluster.YarnClusterScheduler: Stage 0 was cancelled
18/08/13 15:49:52 INFO scheduler.DAGScheduler: ShuffleMapStage 0 (mapToPair at
SparkCubingByLayer.java:182) failed in 4.198 s due to Job aborted due to stage
failure: Task 14 in stage 0.0 failed 4 times, most recent failure: Lost task
14.3 in stage 0.0 (TID 44, hadoop067, executor 4):
java.lang.ArrayIndexOutOfBoundsException: 8
at
org.apache.kylin.engine.mr.common.BaseCuboidBuilder.getCell(BaseCuboidBuilder.java:167)
at
org.apache.kylin.engine.mr.common.BaseCuboidBuilder.buildKey(BaseCuboidBuilder.java:116)
at
org.apache.kylin.engine.spark.SparkCubingByLayer$EncodeBaseCuboid.call(SparkCubingByLayer.java:343)
at
org.apache.kylin.engine.spark.SparkCubingByLayer$EncodeBaseCuboid.call(SparkCubingByLayer.java:307)
at
org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1043)
at
org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1043)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at
org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:193)
at
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Driver stacktrace:
18/08/13 15:49:52 INFO scheduler.DAGScheduler: Job 0 failed:
saveAsNewAPIHadoopDataset at SparkCubingByLayer.java:277, took 4.768935 s
18/08/13 15:49:52 ERROR yarn.ApplicationMaster: User class threw exception:
java.lang.RuntimeException: error execute
org.apache.kylin.engine.spark.SparkCubingByLayer
java.lang.RuntimeException: error execute
org.apache.kylin.engine.spark.SparkCubingByLayer
at
org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42)
at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:636)
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure:
Task 14 in stage 0.0 failed 4 times, most recent failure: Lost task 14.3 in
stage 0.0 (TID 44, hadoop067, executor 4):
java.lang.ArrayIndexOutOfBoundsException: 8
at
org.apache.kylin.engine.mr.common.BaseCuboidBuilder.getCell(BaseCuboidBuilder.java:167)
at
org.apache.kylin.engine.mr.common.BaseCuboidBuilder.buildKey(BaseCuboidBuilder.java:116)
at
org.apache.kylin.engine.spark.SparkCubingByLayer$EncodeBaseCuboid.call(SparkCubingByLayer.java:343)
at
org.apache.kylin.engine.spark.SparkCubingByLayer$EncodeBaseCuboid.call(SparkCubingByLayer.java:307)
at
org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1043)
at
org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1043)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at
org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:193)
at
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Driver stacktrace:
at
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1435)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1423)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1422)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1422)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802)
at scala.Option.foreach(Option.scala:257)
at
org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:802)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1650)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1605)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1594)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:628)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1928)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1941)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1961)
at
org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply$mcV$sp(PairRDDFunctions.scala:1158)
at
org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1085)
at
org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1085)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
at
org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:1085)
at
org.apache.spark.api.java.JavaPairRDD.saveAsNewAPIHadoopDataset(JavaPairRDD.scala:831)
at
org.apache.kylin.engine.spark.SparkCubingByLayer.saveToHDFS(SparkCubingByLayer.java:277)
at
org.apache.kylin.engine.spark.SparkCubingByLayer.execute(SparkCubingByLayer.java:230)
at
org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
... 6 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: 8
at
org.apache.kylin.engine.mr.common.BaseCuboidBuilder.getCell(BaseCuboidBuilder.java:167)
at
org.apache.kylin.engine.mr.common.BaseCuboidBuilder.buildKey(BaseCuboidBuilder.java:116)
at
org.apache.kylin.engine.spark.SparkCubingByLayer$EncodeBaseCuboid.call(SparkCubingByLayer.java:343)
at
org.apache.kylin.engine.spark.SparkCubingByLayer$EncodeBaseCuboid.call(SparkCubingByLayer.java:307)
at
org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1043)
at
org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1043)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at
org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:193)
at
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
18/08/13 15:49:52 INFO yarn.ApplicationMaster: Final app status: FAILED,
exitCode: 15, (reason: User class threw exception: java.lang.RuntimeException:
error execute org.apache.kylin.engine.spark.SparkCubingByLayer)
18/08/13 15:49:52 INFO spark.SparkContext: Invoking stop() from shutdown hook
18/08/13 15:49:52 INFO server.ServerConnector: Stopped
Spark@6e45bafc\{HTTP/1.1}{0.0.0.0:0}
18/08/13 15:49:52 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@10f69049\{/stages/stage/kill,null,UNAVAILABLE,@Spark}
18/08/13 15:49:52 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@f66404a\{/jobs/job/kill,null,UNAVAILABLE,@Spark}
18/08/13 15:49:52 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@206926ba\{/api,null,UNAVAILABLE,@Spark}
18/08/13 15:49:52 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@5ca862d4\{/,null,UNAVAILABLE,@Spark}
18/08/13 15:49:52 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@493ad6b1\{/static,null,UNAVAILABLE,@Spark}
18/08/13 15:49:52 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@22f1aa2f\{/executors/threadDump/json,null,UNAVAILABLE,@Spark}
18/08/13 15:49:52 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@15ed2a19\{/executors/threadDump,null,UNAVAILABLE,@Spark}
18/08/13 15:49:52 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@36d46a68\{/executors/json,null,UNAVAILABLE,@Spark}
18/08/13 15:49:52 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@4fe49898\{/executors,null,UNAVAILABLE,@Spark}
18/08/13 15:49:52 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@22e2b922\{/environment/json,null,UNAVAILABLE,@Spark}
18/08/13 15:49:52 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@6c39f467\{/environment,null,UNAVAILABLE,@Spark}
18/08/13 15:49:52 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@1fdcd2ae\{/storage/rdd/json,null,UNAVAILABLE,@Spark}
18/08/13 15:49:52 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@399a8e28\{/storage/rdd,null,UNAVAILABLE,@Spark}
18/08/13 15:49:52 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@25fd6d17\{/storage/json,null,UNAVAILABLE,@Spark}
18/08/13 15:49:52 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@64eab11\{/storage,null,UNAVAILABLE,@Spark}
18/08/13 15:49:52 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@60b8cb0e\{/stages/pool/json,null,UNAVAILABLE,@Spark}
18/08/13 15:49:52 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@679f6c8c\{/stages/pool,null,UNAVAILABLE,@Spark}
18/08/13 15:49:52 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@3e40e89\{/stages/stage/json,null,UNAVAILABLE,@Spark}
18/08/13 15:49:52 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@78f3dc74\{/stages/stage,null,UNAVAILABLE,@Spark}
18/08/13 15:49:52 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@7dff5bfa\{/stages/json,null,UNAVAILABLE,@Spark}
18/08/13 15:49:52 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@5ef75d04\{/stages,null,UNAVAILABLE,@Spark}
18/08/13 15:49:52 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@4a662152\{/jobs/job/json,null,UNAVAILABLE,@Spark}
18/08/13 15:49:52 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@5001120\{/jobs/job,null,UNAVAILABLE,@Spark}
18/08/13 15:49:52 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@7f1b8616\{/jobs/json,null,UNAVAILABLE,@Spark}
18/08/13 15:49:52 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@2506206a\{/jobs,null,UNAVAILABLE,@Spark}
18/08/13 15:49:52 INFO ui.SparkUI: Stopped Spark web UI at
http://172.16.19.79:36502
18/08/13 15:49:52 WARN scheduler.TaskSetManager: Lost task 16.1 in stage 0.0
(TID 46, hadoop062, executor 3): TaskKilled (killed intentionally)
18/08/13 15:49:52 ERROR scheduler.LiveListenerBus: SparkListenerBus has already
stopped! Dropping event
SparkListenerTaskEnd(0,0,ShuffleMapTask,TaskKilled,org.apache.spark.scheduler.TaskInfo@5725003b,null)
18/08/13 15:49:52 WARN scheduler.TaskSetManager: Lost task 45.2 in stage 0.0
(TID 47, hadoop057, executor 2): TaskKilled (killed intentionally)
18/08/13 15:49:52 ERROR scheduler.LiveListenerBus: SparkListenerBus has already
stopped! Dropping event
SparkListenerTaskEnd(0,0,ShuffleMapTask,TaskKilled,org.apache.spark.scheduler.TaskInfo@18373e72,null)
18/08/13 15:49:52 WARN scheduler.TaskSetManager: Lost task 22.1 in stage 0.0
(TID 54, hadoop053, executor 9): TaskKilled (killed intentionally)
18/08/13 15:49:52 ERROR scheduler.LiveListenerBus: SparkListenerBus has already
stopped! Dropping event
SparkListenerTaskEnd(0,0,ShuffleMapTask,TaskKilled,org.apache.spark.scheduler.TaskInfo@3b7b08b6,null)
18/08/13 15:49:52 WARN scheduler.TaskSetManager: Lost task 90.1 in stage 0.0
(TID 55, hadoop073, executor 1): TaskKilled (killed intentionally)
18/08/13 15:49:52 ERROR scheduler.LiveListenerBus: SparkListenerBus has already
stopped! Dropping event
SparkListenerTaskEnd(0,0,ShuffleMapTask,TaskKilled,org.apache.spark.scheduler.TaskInfo@1707c5a7,null)
18/08/13 15:49:52 WARN scheduler.TaskSetManager: Lost task 35.0 in stage 0.0
(TID 53, hadoop053, executor 9): TaskKilled (killed intentionally)
18/08/13 15:49:52 ERROR scheduler.LiveListenerBus: SparkListenerBus has already
stopped! Dropping event
SparkListenerTaskEnd(0,0,ShuffleMapTask,TaskKilled,org.apache.spark.scheduler.TaskInfo@68dc3ba4,null)
18/08/13 15:49:52 WARN scheduler.TaskSetManager: Lost task 18.1 in stage 0.0
(TID 42, hadoop079, executor 10): TaskKilled (killed intentionally)
18/08/13 15:49:52 ERROR scheduler.LiveListenerBus: SparkListenerBus has already
stopped! Dropping event
SparkListenerTaskEnd(0,0,ShuffleMapTask,TaskKilled,org.apache.spark.scheduler.TaskInfo@49394c63,null)
18/08/13 15:49:52 WARN scheduler.TaskSetManager: Lost task 7.2 in stage 0.0
(TID 51, hadoop075, executor 5): TaskKilled (killed intentionally)
18/08/13 15:49:52 ERROR scheduler.LiveListenerBus: SparkListenerBus has already
stopped! Dropping event
SparkListenerTaskEnd(0,0,ShuffleMapTask,TaskKilled,org.apache.spark.scheduler.TaskInfo@2a64d34a,null)
18/08/13 15:49:52 WARN scheduler.TaskSetManager: Lost task 26.2 in stage 0.0
(TID 56, hadoop073, executor 1): TaskKilled (killed intentionally)
18/08/13 15:49:52 ERROR scheduler.LiveListenerBus: SparkListenerBus has already
stopped! Dropping event
SparkListenerTaskEnd(0,0,ShuffleMapTask,TaskKilled,org.apache.spark.scheduler.TaskInfo@367115c,null)
18/08/13 15:49:52 WARN scheduler.TaskSetManager: Lost task 21.2 in stage 0.0
(TID 50, hadoop070, executor 7): TaskKilled (killed intentionally)
18/08/13 15:49:52 ERROR scheduler.LiveListenerBus: SparkListenerBus has already
stopped! Dropping event
SparkListenerTaskEnd(0,0,ShuffleMapTask,TaskKilled,org.apache.spark.scheduler.TaskInfo@a6f1b8f,null)
18/08/13 15:49:52 WARN scheduler.TaskSetManager: Lost task 31.1 in stage 0.0
(TID 45, hadoop063, executor 8): TaskKilled (killed intentionally)
18/08/13 15:49:52 ERROR scheduler.LiveListenerBus: SparkListenerBus has already
stopped! Dropping event
SparkListenerTaskEnd(0,0,ShuffleMapTask,TaskKilled,org.apache.spark.scheduler.TaskInfo@93604f5,null)
18/08/13 15:49:52 WARN scheduler.TaskSetManager: Lost task 50.1 in stage 0.0
(TID 40, hadoop079, executor 10): TaskKilled (killed intentionally)
18/08/13 15:49:52 ERROR scheduler.LiveListenerBus: SparkListenerBus has already
stopped! Dropping event
SparkListenerTaskEnd(0,0,ShuffleMapTask,TaskKilled,org.apache.spark.scheduler.TaskInfo@544681a,null)
18/08/13 15:49:52 WARN scheduler.TaskSetManager: Lost task 42.2 in stage 0.0
(TID 48, hadoop062, executor 3): TaskKilled (killed intentionally)
18/08/13 15:49:52 ERROR scheduler.LiveListenerBus: SparkListenerBus has already
stopped! Dropping event
SparkListenerTaskEnd(0,0,ShuffleMapTask,TaskKilled,org.apache.spark.scheduler.TaskInfo@3e12741f,null)
18/08/13 15:49:52 WARN scheduler.TaskSetManager: Lost task 19.2 in stage 0.0
(TID 43, hadoop070, executor 7): TaskKilled (killed intentionally)
18/08/13 15:49:52 ERROR scheduler.LiveListenerBus: SparkListenerBus has already
stopped! Dropping event
SparkListenerTaskEnd(0,0,ShuffleMapTask,TaskKilled,org.apache.spark.scheduler.TaskInfo@4c53b331,null)
18/08/13 15:49:52 WARN scheduler.TaskSetManager: Lost task 52.2 in stage 0.0
(TID 52, hadoop057, executor 2): TaskKilled (killed intentionally)
18/08/13 15:49:52 ERROR scheduler.LiveListenerBus: SparkListenerBus has already
stopped! Dropping event
SparkListenerTaskEnd(0,0,ShuffleMapTask,TaskKilled,org.apache.spark.scheduler.TaskInfo@757b07e2,null)
18/08/13 15:49:52 INFO yarn.YarnAllocator: Driver requested a total number of 0
executor(s).
18/08/13 15:49:52 INFO cluster.YarnClusterSchedulerBackend: Shutting down all
executors
18/08/13 15:49:52 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Asking
each executor to shut down
18/08/13 15:49:52 INFO cluster.SchedulerExtensionServices: Stopping
SchedulerExtensionServices
(serviceOption=None,
services=List(),
started=false)
18/08/13 15:49:52 INFO spark.MapOutputTrackerMasterEndpoint:
MapOutputTrackerMasterEndpoint stopped!
18/08/13 15:49:52 ERROR server.TransportRequestHandler: Error while invoking
RpcHandler#receive() for one-way message.
org.apache.spark.SparkException: Could not find CoarseGrainedScheduler.
at org.apache.spark.rpc.netty.Dispatcher.postMessage(Dispatcher.scala:154)
at
org.apache.spark.rpc.netty.Dispatcher.postOneWayMessage(Dispatcher.scala:134)
at org.apache.spark.rpc.netty.NettyRpcHandler.receive(NettyRpcEnv.scala:570)
at
org.apache.spark.network.server.TransportRequestHandler.processOneWayMessage(TransportRequestHandler.java:180)
at
org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:109)
at
org.apache.spark.network.server.TransportChannelHandler.channelRead(TransportChannelHandler.java:118)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336)
at
io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336)
at
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336)
at
org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:85)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343)
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911)
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:643)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:566)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:480)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:442)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
at
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
at java.lang.Thread.run(Thread.java:748)
18/08/13 15:49:52 ERROR server.TransportRequestHandler: Error while invoking
RpcHandler#receive() for one-way message.
org.apache.spark.SparkException: Could not find CoarseGrainedScheduler.
at org.apache.spark.rpc.netty.Dispatcher.postMessage(Dispatcher.scala:154)
at
org.apache.spark.rpc.netty.Dispatcher.postOneWayMessage(Dispatcher.scala:134)
at org.apache.spark.rpc.netty.NettyRpcHandler.receive(NettyRpcEnv.scala:570)
at
org.apache.spark.network.server.TransportRequestHandler.processOneWayMessage(TransportRequestHandler.java:180)
at
org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:109)
at
org.apache.spark.network.server.TransportChannelHandler.channelRead(TransportChannelHandler.java:118)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336)
at
io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336)
at
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336)
at
org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:85)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343)
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911)
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:643)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:566)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:480)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:442)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
at
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
at java.lang.Thread.run(Thread.java:748)
18/08/13 15:49:52 INFO memory.MemoryStore: MemoryStore cleared
18/08/13 15:49:52 INFO storage.BlockManager: BlockManager stopped
18/08/13 15:49:52 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
18/08/13 15:49:52 INFO
scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:
OutputCommitCoordinator stopped!
18/08/13 15:49:52 INFO spark.SparkContext: Successfully stopped SparkContext
18/08/13 15:49:52 INFO util.ShutdownHookManager: Shutdown hook called
18/08/13 15:49:52 INFO util.ShutdownHookManager: Deleting directory
/data3/test/data/hadoop/hdfs/data/usercache/hadoop/appcache/application_1533616206085_5657/spark-69de25a9-16d2-4eaa-be70-d23ec191776a
18/08/13 15:49:52 INFO util.ShutdownHookManager: Deleting directory
/data2/test/data/hadoop/hdfs/data/usercache/hadoop/appcache/application_1533616206085_5657/spark-a4b4251c-feaf-426a-a363-4159cdc092f6
18/08/13 15:49:52 INFO util.ShutdownHookManager: Deleting directory
/data1/test/data/hadoop/hdfs/data/usercache/hadoop/appcache/application_1533616206085_5657/spark-751dcea0-e9c4-4160-9112-a6fe081ec4fb
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
