[ 
https://issues.apache.org/jira/browse/KYLIN-1077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15243943#comment-15243943
 ] 

Shaofeng SHI commented on KYLIN-1077:
-------------------------------------

xianbin, you are awesome! I went through the patch quickly and have a couple of 
comments:

1. "core-cube" and "core-dictionary" are two kylin core modules, which intend 
to be independent with the source, storage, and computing framework. That's why 
their dependency are very simple; so add "hive-metastore" into these two 
modules would break this design. In metadata level, we can just reuse hive 
string value to mark the table's type. If must use some hive dependency, add 
that in the "source-hive" module.

2. the method "generateHiveViewIntermediateTableName" was declared in a couple 
of places; can it be controlled as only 1 across all modules? besides, this 
joint name may be long, I remember hive has a restriction on the name length, 
we'd better use a shortter name (does segment/cube_name need appear there?) and 
cutoff when it exceeds the limit.

3.  in HiveMRInput. addStepPhase1_CreateFlatTable(), as the 
"LookupHiveViewMaterializationStep" is not a required step, we can check 
whether there is such a lookup view first, and then decide whether need add 
this step or not. Otherwise there is always such a step even there is no lookup 
view, that is confusing;

4. please add unit test for the codes;  or make it be covered by the 
integration test.  Testable is important for such a complexy project as many 
factors may break the function.

Thanks for your contribution!

> Support Hive View as Lookup Table
> ---------------------------------
>
>                 Key: KYLIN-1077
>                 URL: https://issues.apache.org/jira/browse/KYLIN-1077
>             Project: Kylin
>          Issue Type: Improvement
>    Affects Versions: v1.1
>            Reporter: Francesco Palmiotto
>            Assignee: Shaofeng SHI
>              Labels: newbie
>         Attachments: 0001-KYLIN-1077-Support-Hive-View-as-Lookup-Table.patch
>
>
> If we use a view as a lookup table the cube building job fails when executing 
> the 3rd step (Build Dimension Dictionary) with this log:
> java.io.IOException: java.lang.NullPointerException
>       at 
> org.apache.kylin.dict.lookup.HiveTable.getSignature(HiveTable.java:72)
>       at 
> org.apache.kylin.dict.DictionaryManager.buildDictionary(DictionaryManager.java:202)
>       at 
> org.apache.kylin.cube.CubeManager.buildDictionary(CubeManager.java:166)
>       at 
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:52)
>       at 
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:41)
>       at 
> org.apache.kylin.job.hadoop.dict.CreateDictionaryJob.run(CreateDictionaryJob.java:52)
>       at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>       at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>       at 
> org.apache.kylin.job.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:62)
>       at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
>       at 
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)
>       at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
>       at 
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>       at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> result code:2



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to