While running Spark Cube process, I noticed that this is taking other Cube tables into the consideration , Rather it should take the cube which it isdoing. Not sure why its taking data model of other cubes. Normally its being noticed that Spark is taking almost same time as Maprecuce is taking. We assume that Spark should be faster than MR jobs.
2018-02-05 23:35:07,825 INFO [Job 53f2a470-2973-46ad-9d97-8ddcec1933cc-315] spark.SparkExecutable:38 : 18/02/05 23:35:07 WARN CubeDescManager: Broken cube desc /cube_desc/FRI_CUBE_update.json 2018-02-05 23:35:07,825 INFO [Job 53f2a470-2973-46ad-9d97-8ddcec1933cc-315] spark.SparkExecutable:38 : java.lang.IllegalArgumentException: Table not found by LOAN_POSITION_BAL_009_SS1 2018-02-05 23:35:07,825 INFO [Job 53f2a470-2973-46ad-9d97-8ddcec1933cc-315] spark.SparkExecutable:38 : at org.apache.kylin.metadata.model.DataModelDesc.findTable(DataModelDesc.java:314) 2018-02-05 23:35:07,826 INFO [Job 53f2a470-2973-46ad-9d97-8ddcec1933cc-315] spark.SparkExecutable:38 : at org.apache.kylin.cube.model.DimensionDesc.init(DimensionDesc.java:61) 2018-02-05 23:35:07,826 INFO [Job 53f2a470-2973-46ad-9d97-8ddcec1933cc-315] spark.SparkExecutable:38 : at org.apache.kylin.cube.model.CubeDesc.init(CubeDesc.java:587) 2018-02-05 23:35:07,826 INFO [Job 53f2a470-2973-46ad-9d97-8ddcec1933cc-315] spark.SparkExecutable:38 : at org.apache.kylin.cube.CubeDescManager.loadCubeDesc(CubeDescManager.java:196) 2018-02-05 23:35:07,826 INFO [Job 53f2a470-2973-46ad-9d97-8ddcec1933cc-315] spark.SparkExecutable:38 : at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(CubeDescManager.java:321) 2018-02-05 23:35:07,826 INFO [Job 53f2a470-2973-46ad-9d97-8ddcec1933cc-315] spark.SparkExecutable:38 : at org.apache.kylin.cube.CubeDescManager.<init>(CubeDescManager.java:114) 2018-02-05 23:35:07,827 INFO [Job 53f2a470-2973-46ad-9d97-8ddcec1933cc-315] spark.SparkExecutable:38 : at org.apache.kylin.cube.CubeDescManager.getInstance(CubeDescManager.java:81) 2018-02-05 23:35:07,827 INFO [Job 53f2a470-2973-46ad-9d97-8ddcec1933cc-315] spark.SparkExecutable:38 : at org.apache.kylin.cube.CubeManager.reloadCubeLocalAt(CubeManager.java:811) 2018-02-05 23:35:07,827 INFO [Job 53f2a470-2973-46ad-9d97-8ddcec1933cc-315] spark.SparkExecutable:38 : at org.apache.kylin.cube.CubeManager.loadAllCubeInstance(CubeManager.java:789) 2018-02-05 23:35:07,827 INFO [Job 53f2a470-2973-46ad-9d97-8ddcec1933cc-315] spark.SparkExecutable:38 : at org.apache.kylin.cube.CubeManager.<init>(CubeManager.java:147) 2018-02-05 23:35:07,827 INFO [Job 53f2a470-2973-46ad-9d97-8ddcec1933cc-315] spark.SparkExecutable:38 : at org.apache.kylin.cube.CubeManager.getInstance(CubeManager.java:105) 2018-02-05 23:35:07,827 INFO [Job 53f2a470-2973-46ad-9d97-8ddcec1933cc-315] spark.SparkExecutable:38 : at org.apache.kylin.engine.spark.SparkCubingByLayer.execute(SparkCubingByLayer.java:161) 2018-02-05 23:35:07,828 INFO [Job 53f2a470-2973-46ad-9d97-8ddcec1933cc-315] spark.SparkExecutable:38 : at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37) 2018-02-05 23:35:07,828 INFO [Job 53f2a470-2973-46ad-9d97-8ddcec1933cc-315] spark.SparkExecutable:38 : at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44) 2018-02-05 23:35:07,828 INFO [Job 53f2a470-2973-46ad-9d97-8ddcec1933cc-315] spark.SparkExecutable:38 : at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 2018-02-05 23:35:07,828 INFO [Job 53f2a470-2973-46ad-9d97-8ddcec1933cc-315] spark.SparkExecutable:38 : at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 2018-02-05 23:35:07,828 INFO [Job 53f2a470-2973-46ad-9d97-8ddcec1933cc-315] spark.SparkExecutable:38 : at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 2018-02-05 23:35:07,829 INFO [Job 53f2a470-2973-46ad-9d97-8ddcec1933cc-315] spark.SparkExecutable:38 : at java.lang.reflect.Method.invoke(Method.java:606) 2018-02-05 23:35:07,829 INFO [Job 53f2a470-2973-46ad-9d97-8ddcec1933cc-315] spark.SparkExecutable:38 : at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$r Regards, Manoj This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
