[
https://issues.apache.org/jira/browse/KYLIN-2575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16086788#comment-16086788
]
kangkaisen commented on KYLIN-2575:
-----------------------------------
This commit
https://github.com/apache/kylin/commit/d61f867ff9fa170e155310edeb03db6059433f2f
maybe broken the Spark Cubing. I have registered ComputedColumnDesc to kyro and
changed the engine_type of ci_inner_join_cube to 4.
After I annotated this code:
CalciteParser.ensureNoTableNameExists(expression)， Spark Cubing has run
successfully.
{code:java}
17/07/13 21:33:45 INFO CubeManager: Loading Cube from folder
kylin_metadata(key='/cube')@kylin_metadata@hbase
17/07/13 21:33:45 INFO CubeDescManager: Initializing CubeDescManager with
config kylin_metadata@hbase
17/07/13 21:33:45 INFO CubeDescManager: Reloading Cube Metadata from folder
kylin_metadata(key='/cube_desc')@kylin_metadata@hbase
17/07/13 21:33:45 INFO ProjectManager: Initializing ProjectManager with
metadata url kylin_metadata@hbase
17/07/13 21:33:45 INFO MeasureTypeFactory: Checking custom measure types from
kylin config
17/07/13 21:33:45 INFO MeasureTypeFactory: registering COUNT_DISTINCT(hllc),
class org.apache.kylin.measure.hllc.HLLCMeasureType$Factory
17/07/13 21:33:45 INFO MeasureTypeFactory: registering COUNT_DISTINCT(bitmap),
class org.apache.kylin.measure.bitmap.BitmapMeasureType$Factory
17/07/13 21:33:45 INFO MeasureTypeFactory: registering TOP_N(topn), class
org.apache.kylin.measure.topn.TopNMeasureType$Factory
17/07/13 21:33:45 INFO MeasureTypeFactory: registering RAW(raw), class
org.apache.kylin.measure.raw.RawMeasureType$Factory
17/07/13 21:33:45 INFO MeasureTypeFactory: registering
EXTENDED_COLUMN(extendedcolumn), class
org.apache.kylin.measure.extendedcolumn.ExtendedC
olumnMeasureType$Factory
17/07/13 21:33:45 INFO MeasureTypeFactory: registering PERCENTILE(percentile),
class org.apache.kylin.measure.percentile.PercentileMeasureType$
Factory
17/07/13 21:33:46 INFO MetadataManager: Reloading data model at
/model_desc/ci_inner_join_model.json
Exception in thread "main" java.lang.NoSuchFieldError: names
at
org.apache.kylin.metadata.model.tool.CalciteParser$1.visit(CalciteParser.java:140)
at org.apache.calcite.sql.SqlIdentifier.accept(SqlIdentifier.java:271)
at org.apache.calcite.sql.SqlOperator.acceptCall(SqlOperator.java:670)
at
org.apache.calcite.sql.util.SqlBasicVisitor.visit(SqlBasicVisitor.java:45)
at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:130)
at
org.apache.kylin.metadata.model.tool.CalciteParser.ensureNoTableNameExists(CalciteParser.java:147)
at
org.apache.kylin.metadata.model.ComputedColumnDesc.init(ComputedColumnDesc.java:49)
at
org.apache.kylin.metadata.model.DataModelDesc.initComputedColumns(DataModelDesc.java:490)
at
org.apache.kylin.metadata.model.DataModelDesc.init(DataModelDesc.java:355)
at
org.apache.kylin.metadata.MetadataManager.reloadDataModelDescAt(MetadataManager.java:595)
at
org.apache.kylin.metadata.MetadataManager.reloadAllDataModel(MetadataManager.java:579)
at
org.apache.kylin.metadata.MetadataManager.init(MetadataManager.java:295)
at
org.apache.kylin.metadata.MetadataManager.<init>(MetadataManager.java:122)
at
org.apache.kylin.metadata.MetadataManager.getInstance(MetadataManager.java:87)
at org.apache.kylin.cube.model.CubeDesc.init(CubeDesc.java:589)
at
org.apache.kylin.cube.CubeDescManager.loadCubeDesc(CubeDescManager.java:196)
at
org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(CubeDescManager.java:319)
at
org.apache.kylin.cube.CubeDescManager.<init>(CubeDescManager.java:114)
at
org.apache.kylin.cube.CubeDescManager.getInstance(CubeDescManager.java:81)
at
org.apache.kylin.cube.CubeManager.reloadCubeLocalAt(CubeManager.java:809)
at
org.apache.kylin.cube.CubeManager.loadAllCubeInstance(CubeManager.java:787)
at org.apache.kylin.cube.CubeManager.<init>(CubeManager.java:147)
at org.apache.kylin.cube.CubeManager.getInstance(CubeManager.java:105)
at
org.apache.kylin.engine.spark.SparkCubingByLayer.execute(SparkCubingByLayer.java:160)
at
org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
{code}
> Experimental feature: Computed Column
> -------------------------------------
>
> Key: KYLIN-2575
> URL: https://issues.apache.org/jira/browse/KYLIN-2575
> Project: Kylin
> Issue Type: New Feature
> Reporter: hongbin ma
> Assignee: hongbin ma
> Fix For: v2.1.0
>
>
> Computed column is a virtual column that is calculated from an expression of
> existing columns. For example, TAX is computed from PRICE * TAX_RATE; TX_YEAR
> is from EXTRACT(year from TX_DATE).
> Currently user have to create a view to enrich these computed columns, then
> feed the view to cube. This has two inconvenience:
> Create a view is not easy.
> The query has to be rewritten to use view instead of the original table.
> Let Kylin/KAP directly support computed column will be a big step forward.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
