[
https://issues.apache.org/jira/browse/KYLIN-1077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15243943#comment-15243943
]
Shaofeng SHI commented on KYLIN-1077:
-------------------------------------
xianbin, you are awesome! I went through the patch quickly and have a couple of
comments:
1. "core-cube" and "core-dictionary" are two kylin core modules, which intend
to be independent with the source, storage, and computing framework. That's why
their dependency are very simple; so add "hive-metastore" into these two
modules would break this design. In metadata level, we can just reuse hive
string value to mark the table's type. If must use some hive dependency, add
that in the "source-hive" module.
2. the method "generateHiveViewIntermediateTableName" was declared in a couple
of places; can it be controlled as only 1 across all modules? besides, this
joint name may be long, I remember hive has a restriction on the name length,
we'd better use a shortter name (does segment/cube_name need appear there?) and
cutoff when it exceeds the limit.
3. in HiveMRInput. addStepPhase1_CreateFlatTable(), as the
"LookupHiveViewMaterializationStep" is not a required step, we can check
whether there is such a lookup view first, and then decide whether need add
this step or not. Otherwise there is always such a step even there is no lookup
view, that is confusing;
4. please add unit test for the codes; or make it be covered by the
integration test. Testable is important for such a complexy project as many
factors may break the function.
Thanks for your contribution!
> Support Hive View as Lookup Table
> ---------------------------------
>
> Key: KYLIN-1077
> URL: https://issues.apache.org/jira/browse/KYLIN-1077
> Project: Kylin
> Issue Type: Improvement
> Affects Versions: v1.1
> Reporter: Francesco Palmiotto
> Assignee: Shaofeng SHI
> Labels: newbie
> Attachments: 0001-KYLIN-1077-Support-Hive-View-as-Lookup-Table.patch
>
>
> If we use a view as a lookup table the cube building job fails when executing
> the 3rd step (Build Dimension Dictionary) with this log:
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.kylin.dict.lookup.HiveTable.getSignature(HiveTable.java:72)
> at
> org.apache.kylin.dict.DictionaryManager.buildDictionary(DictionaryManager.java:202)
> at
> org.apache.kylin.cube.CubeManager.buildDictionary(CubeManager.java:166)
> at
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:52)
> at
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:41)
> at
> org.apache.kylin.job.hadoop.dict.CreateDictionaryJob.run(CreateDictionaryJob.java:52)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at
> org.apache.kylin.job.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:62)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> result code:2
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)