[ 
https://issues.apache.org/jira/browse/KYLIN-1508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15206035#comment-15206035
 ] 

Shaofeng SHI commented on KYLIN-1508:
-------------------------------------

View as lookup table isn't supported yet, KYLIN-1077 is for that issue, you can 
also comment there.

In the new version (named as v1.5.0), there is no downgrade regarding 
compression as I know. One possible reason is the cube definitions in between 
has some difference, causing more combinations be calculated in the new 
version. If you don't mind, please share the cube JSONs here and then we can 
have a check.

> NPE error ERROR - Build Cube - Step 3 if LOOKUP table is Hive View - : 
> java.lang.NullPointerException at 
> org.apache.kylin.source.hive.HiveTable.getSignature
> ------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: KYLIN-1508
>                 URL: https://issues.apache.org/jira/browse/KYLIN-1508
>             Project: Kylin
>          Issue Type: Bug
>          Components: Job Engine
>    Affects Versions: v1.5.0, v1.3.0, v1.2, all
>         Environment: CentOS 6.7 , Hortonworks Hadoop 2.2.4.2 - sandbox
>            Reporter: Richard Calaba
>            Assignee: Dong Li
>
> Affected releases: The error occurs in 1.2 release (downloaded as binary 
> release from Kylin Home page). The same error occurs also in ??latest?? 
> snapshot 2.1 (compiled today 3/12/16 from Kylin Git using bracnh 
> 2.x-staging). Tests were done on Hortonworks HDP 2.2.4 sandbox. Thus I assume 
> that all current branches are affected.
> Error Summary:   Whereas it is possible to successfully build cube which FACT 
> table referring Hive View – it seems not to be possible to build a cube which 
> has LOOKUP table referring Hive View. It seems to be bug in metadata 
> processing in Step 3 of the Build process. Build process fails with exception:
> (release 1.2) 
> java.io.IOException: java.lang.NullPointerException
>                 at 
> org.apache.kylin.dict.lookup.HiveTable.getSignature(HiveTable.java:72)
>                 at 
> org.apache.kylin.dict.lookup.SnapshotTable.<init>(SnapshotTable.java:62)
>                 at 
> org.apache.kylin.dict.lookup.SnapshotManager.buildSnapshot(SnapshotManager.java:85)
>                 at 
> org.apache.kylin.cube.CubeManager.buildSnapshotTable(CubeManager.java:205)
>                 at 
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:60)
>                 at 
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:41)
>                 at 
> org.apache.kylin.job.hadoop.dict.CreateDictionaryJob.run(CreateDictionaryJob.java:52)
>                 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>                 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>                 at 
> org.apache.kylin.job.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:62)
>                 at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
>                 at 
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)
>                 at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
>                 at 
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)
>                 at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>                 at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>                 at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
>                 at 
> org.apache.kylin.common.util.HadoopUtil.fixWindowsPath(HadoopUtil.java:76)
>                 at 
> org.apache.kylin.common.util.HadoopUtil.makeURI(HadoopUtil.java:68)
>                 at 
> org.apache.kylin.common.util.HadoopUtil.getFileSystem(HadoopUtil.java:63)
>                 at 
> org.apache.kylin.dict.lookup.FileTable.getSizeAndLastModified(FileTable.java:85)
>                 at 
> org.apache.kylin.dict.lookup.HiveTable.getSignature(HiveTable.java:57)
>                 ... 16 more
> Error in latest branch - (snapshoot-2.1) :
> java.io.IOException: java.lang.NullPointerException
>                 at 
> org.apache.kylin.source.hive.HiveTable.getSignature(HiveTable.java:71)
>                 at 
> org.apache.kylin.dict.lookup.SnapshotTable.<init>(SnapshotTable.java:64)
>                 at 
> org.apache.kylin.dict.lookup.SnapshotManager.buildSnapshot(SnapshotManager.java:89)
>                 at 
> org.apache.kylin.cube.CubeManager.buildSnapshotTable(CubeManager.java:208)
>                 at 
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:59)
>                 at 
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:42)
>                 at 
> org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:56)
>                 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>                 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>                 at 
> org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:60)
>                 at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
>                 at 
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50)
>                 at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
>                 at 
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:124)
>                 at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>                 at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>                 at java.lang.Thread.run(Thread.java:745)
> Importance of resolution of this bug: I believe that having the ability to 
> use Hive Views is essential and should work for both FACT tables/views and 
> LOOKUP/DIMENSION tables/views. 1st it helps to do more efficient OLAP staging 
> by not forcing to materialize temporary views used just for OLAP modelling. 
> 2ndly and most importantly having views with left outer and inner joins can 
> further overcome the current Kylin limitation of not beeing to able to 
> provide snowflake-like models.
> Further info - reproduce scenario: I have created 2 Hive Tables (FACT and 
> LOOKUP) using CREATE TABLE AS SELECT … statement in Hive. Then I have created 
> 2 Hive Views using CREATE VIEW AS SELECT … supplying exactly same SQL 
> statements (using left outer and inner joins). All 4 tables/views are 
> correctly producing data in Hive. If I build simple Cube in Kylin using 2 
> tables – everything works fine. If I use Hive view for FACT table and regular 
> Hive Table for LOOKUP – it also works. The error occurs if I try to use Hiev 
> View as LOOKUP table and define dimension from it.
> Just for completeness - below is full Hive QL for creating the table/views. 
> It operates on tables which are created by internal tests while building 
> Kylin 1.2. But I believe the exact view definition should not matter. I did 
> couple of additional tests to create very simple views by doing only "SELECT 
> * FROM one_table" and the error was still the same.
> Hive Query Editor : Create Fact as View 
> CREATE VIEW view_fact_tab_enhanced 
> AS SELECT * 
> FROM DEFAULT.TEST_KYLIN_FACT AS TEST_KYLIN_FACT LEFT OUTER JOIN 
> EDW.TEST_SELLER_TYPE_DIM AS TEST_SELLER_TYPE_DIM 
> ON TEST_KYLIN_FACT.SLR_SEGMENT_CD = TEST_SELLER_TYPE_DIM.SELLER_TYPE_CD
> Hive Query Editor : Create Fact as Table
> CREATE TABLE table_fact_tab_enhanced 
> AS SELECT * 
> FROM DEFAULT.TEST_KYLIN_FACT AS TEST_KYLIN_FACT LEFT OUTER JOIN 
> EDW.TEST_SELLER_TYPE_DIM AS TEST_SELLER_TYPE_DIM 
> ON TEST_KYLIN_FACT.SLR_SEGMENT_CD = TEST_SELLER_TYPE_DIM.SELLER_TYPE_CD
> Hive Query Editor : Create LOOKUP as View
> CREATE VIEW view_dim_tab_self_join AS
> SELECT distinct A.leaf_categ_id, B.leaf_categ_name, A.site_id 
> FROM DEFAULT.TEST_CATEGORY_GROUPINGS AS A LEFT OUTER JOIN 
> DEFAULT.TEST_CATEGORY_GROUPINGS AS B 
> ON A.leaf_categ_id = B.leaf_categ_id AND A.site_id = B.site_id ORDER BY 
> A.leaf_categ_id, B.leaf_categ_name, A.site_id  
> Hive Query Editor : Create LOOKUP as Table
> CREATE TABLE table_dim_tab_self_join AS
> SELECT distinct A.leaf_categ_id, B.leaf_categ_name, A.site_id 
> FROM DEFAULT.TEST_CATEGORY_GROUPINGS AS A LEFT OUTER JOIN 
> DEFAULT.TEST_CATEGORY_GROUPINGS AS B 
> ON A.leaf_categ_id = B.leaf_categ_id AND A.site_id = B.site_id ORDER BY 
> A.leaf_categ_id, B.leaf_categ_name, A.site_id  
> Additional curiosity – seems release 1.2 is handling better the final size of 
> the generated Cube – assuming I didn’t make any mistake in cube definitions 
> and used same in both 1.2-release and 2.1-SNAPSHOT, I am getting smaller cube 
> size in release 1.2 than in 2.1-SNAPSHOT codeline - in both cases the number 
> of rows is exactly same. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to