[
https://issues.apache.org/jira/browse/KYLIN-4105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16889982#comment-16889982
]
dockj edited comment on KYLIN-4105 at 7/22/19 7:58 AM:
-------------------------------------------------------
use spark engine build cube on step Convert Cuboid Data to HFile ，must invoke
MeasureTypeFactory.init() to get custom MesureType。method call link:
MeasureTypeFactory.init()->KylinConfig.getInstanceFromEnv()->KylinConfig.buildSiteProperties()->KylinConfig.buildSiteOrderedProps()->KylinConfig.getSitePropertiesFile()
when yarn cluster set KYLIN_HOME or KYLIN_CONF environment variable，method
KylinConfig.getSitePropertiesFile() will according to KYLIN_HOME or KYLIN_CONF
to find kylin.properties，but yarn cluster host just set KYLIN_HOME or
KYLIN_CONF,no kylin.properties file.method KylinConfig.buildSiteOrderedProps()
will throws java.lang.RuntimeException: fail to locate kylin.properties。but
method MeasureTypeFactory.init() just catch
KylinConfigCannotInitException,finally cause use spark engine to build cube on
step Convert Cuboid Data to HFile fail,maybe can expand method
MeasureTypeFactory.init() KylinConfigCannotInitException to RuntimeException.
was (Author: dockj):
use spark engine build cube on step Convert Cuboid Data to HFile ，must invoke
MeasureTypeFactory.init() to get custom MesureType。method call link:
+MeasureTypeFactory.init()->KylinConfig.getInstanceFromEnv()->KylinConfig.buildSiteProperties()->KylinConfig.buildSiteOrderedProps()->KylinConfig.getSitePropertiesFile()+
when yarn cluster set KYLIN_HOME or KYLIN_CONF environment variable，method
KylinConfig.getSitePropertiesFile() will according to KYLIN_HOME or KYLIN_CONF
to find kylin.properties，but yarn cluster host just set KYLIN_HOME or
KYLIN_CONF,no kylin.properties file.method KylinConfig.buildSiteOrderedProps()
will throws java.lang.RuntimeException: fail to locate kylin.properties。but
method MeasureTypeFactory.init() just catch
KylinConfigCannotInitException,finally cause use spark engine to build cube on
step Convert Cuboid Data to HFile fail,maybe can expand method
MeasureTypeFactory.init() KylinConfigCannotInitException to RuntimeException.
> use spark engine to build cube on step Convert Cuboid Data to HFile failed
> --------------------------------------------------------------------------
>
> Key: KYLIN-4105
> URL: https://issues.apache.org/jira/browse/KYLIN-4105
> Project: Kylin
> Issue Type: Improvement
> Components: Job Engine
> Affects Versions: v2.4.0, v2.4.1, v2.5.0, v2.6.0, v2.6.1, v2.6.2
> Reporter: dockj
> Priority: Minor
>
> use spark engine to build cube on step Convert Cuboid Data to HFile。visit
> yarn logs has this exception:
> 19/05/05 17:36:42 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 0.0
> (TID 0, spslave4.bigdata.ly, executor 1):
> java.lang.ExceptionInInitializerError
> at org.apache.kylin.metadata.datatype.DataType.<clinit>(DataType.java:134)
> at java.io.ObjectStreamClass.hasStaticInitializer(Native Method)
> at java.io.ObjectStreamClass.computeDefaultSUID(ObjectStreamClass.java:1787)
> at java.io.ObjectStreamClass.access$100(ObjectStreamClass.java:72)
> at java.io.ObjectStreamClass$1.run(ObjectStreamClass.java:253)
> at java.io.ObjectStreamClass$1.run(ObjectStreamClass.java:251)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.io.ObjectStreamClass.getSerialVersionUID(ObjectStreamClass.java:250)
> at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:611)
> at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1630)
> at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1521)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1781)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
> at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1714)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373)
> at java.util.HashSet.readObject(HashSet.java:333)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1909)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373)
> at
> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75)
> at
> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:114)
> at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:85)
> at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
> at org.apache.spark.scheduler.Task.run(Task.scala:99)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: fail to locate kylin.properties
> at
> org.apache.kylin.common.KylinConfig.buildSiteOrderedProps(KylinConfig.java:378)
> at
> org.apache.kylin.common.KylinConfig.buildSiteProperties(KylinConfig.java:354)
> at
> org.apache.kylin.common.KylinConfig.getInstanceFromEnv(KylinConfig.java:134)
> at
> org.apache.kylin.measure.MeasureTypeFactory.init(MeasureTypeFactory.java:120)
> at
> org.apache.kylin.measure.MeasureTypeFactory.<clinit>(MeasureTypeFactory.java:98)
> ... 71 more
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
