[ 
https://issues.apache.org/jira/browse/KYLIN-1508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204830#comment-15204830
 ] 

Richard Calaba commented on KYLIN-1508:
---------------------------------------

Little more details in the Issue 1508 - for reproduce scenarios.

> NPE error ERROR - Build Cube - Step 3 if LOOKUP table is Hive View - : 
> java.lang.NullPointerException at 
> org.apache.kylin.source.hive.HiveTable.getSignature
> ------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: KYLIN-1508
>                 URL: https://issues.apache.org/jira/browse/KYLIN-1508
>             Project: Kylin
>          Issue Type: Bug
>          Components: Job Engine
>    Affects Versions: v1.5.0, v1.3.0, v1.2, all
>         Environment: CentOS 6.7 , Hortonworks Hadoop 2.2.4.2 - sandbox
>            Reporter: Richard Calaba
>            Assignee: Dong Li
>
> Affected releases: The error occurs in 1.2 release (downloaded as binary 
> release from Kylin Home page). The same error occurs also in ??latest?? 
> snapshot 2.1 (compiled today 3/12/16 from Kylin Git using bracnh 
> 2.x-staging). Tests were done on Hortonworks HDP 2.2.4 sandbox. Thus I assume 
> that all current branches are affected.
> Error Summary:   Whereas it is possible to successfully build cube which FACT 
> table referring Hive View – it seems not to be possible to build a cube which 
> has LOOKUP table referring Hive View. It seems to be bug in metadata 
> processing in Step 3 of the Build process. Build process fails with exception:
> (release 1.2) 
> java.io.IOException: java.lang.NullPointerException
>                 at 
> org.apache.kylin.dict.lookup.HiveTable.getSignature(HiveTable.java:72)
>                 at 
> org.apache.kylin.dict.lookup.SnapshotTable.<init>(SnapshotTable.java:62)
>                 at 
> org.apache.kylin.dict.lookup.SnapshotManager.buildSnapshot(SnapshotManager.java:85)
>                 at 
> org.apache.kylin.cube.CubeManager.buildSnapshotTable(CubeManager.java:205)
>                 at 
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:60)
>                 at 
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:41)
>                 at 
> org.apache.kylin.job.hadoop.dict.CreateDictionaryJob.run(CreateDictionaryJob.java:52)
>                 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>                 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>                 at 
> org.apache.kylin.job.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:62)
>                 at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
>                 at 
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)
>                 at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
>                 at 
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)
>                 at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>                 at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>                 at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
>                 at 
> org.apache.kylin.common.util.HadoopUtil.fixWindowsPath(HadoopUtil.java:76)
>                 at 
> org.apache.kylin.common.util.HadoopUtil.makeURI(HadoopUtil.java:68)
>                 at 
> org.apache.kylin.common.util.HadoopUtil.getFileSystem(HadoopUtil.java:63)
>                 at 
> org.apache.kylin.dict.lookup.FileTable.getSizeAndLastModified(FileTable.java:85)
>                 at 
> org.apache.kylin.dict.lookup.HiveTable.getSignature(HiveTable.java:57)
>                 ... 16 more
> Error in latest branch - (snapshoot-2.1) :
> java.io.IOException: java.lang.NullPointerException
>                 at 
> org.apache.kylin.source.hive.HiveTable.getSignature(HiveTable.java:71)
>                 at 
> org.apache.kylin.dict.lookup.SnapshotTable.<init>(SnapshotTable.java:64)
>                 at 
> org.apache.kylin.dict.lookup.SnapshotManager.buildSnapshot(SnapshotManager.java:89)
>                 at 
> org.apache.kylin.cube.CubeManager.buildSnapshotTable(CubeManager.java:208)
>                 at 
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:59)
>                 at 
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:42)
>                 at 
> org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:56)
>                 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>                 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>                 at 
> org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:60)
>                 at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
>                 at 
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50)
>                 at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
>                 at 
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:124)
>                 at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>                 at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>                 at java.lang.Thread.run(Thread.java:745)
> Importance of resolution of this bug: I believe that having the ability to 
> use Hive Views is essential and should work for both FACT tables/views and 
> LOOKUP/DIMENSION tables/views. 1st it helps to do more efficient OLAP staging 
> by not forcing to materialize temporary views used just for OLAP modelling. 
> 2ndly and most importantly having views with left outer and inner joins can 
> further overcome the current Kylin limitation of not beeing to able to 
> provide snowflake-like models.
> Further info - reproduce scenario: I have created 2 Hive Tables (FACT and 
> LOOKUP) using CREATE TABLE AS SELECT … statement in Hive. Then I have created 
> 2 Hive Views using CREATE VIEW AS SELECT … supplying exactly same SQL 
> statements (using left outer and inner joins). All 4 tables/views are 
> correctly producing data in Hive. If I build simple Cube in Kylin using 2 
> tables – everything works fine. If I use Hive view for FACT table and regular 
> Hive Table for LOOKUP – it also works. The error occurs if I try to use Hiev 
> View as LOOKUP table and define dimension from it.
> Just for completeness - below is full Hive QL for creating the table/views. 
> It operates on tables which are created by internal tests while building 
> Kylin 1.2. But I believe the exact view definition should not matter. I did 
> couple of additional tests to create very simple views by doing only "SELECT 
> * FROM one_table" and the error was still the same.
> Hive Query Editor : Create Fact as View 
> CREATE VIEW view_fact_tab_enhanced 
> AS SELECT * 
> FROM DEFAULT.TEST_KYLIN_FACT AS TEST_KYLIN_FACT LEFT OUTER JOIN 
> EDW.TEST_SELLER_TYPE_DIM AS TEST_SELLER_TYPE_DIM 
> ON TEST_KYLIN_FACT.SLR_SEGMENT_CD = TEST_SELLER_TYPE_DIM.SELLER_TYPE_CD
> Hive Query Editor : Create Fact as Table
> CREATE TABLE table_fact_tab_enhanced 
> AS SELECT * 
> FROM DEFAULT.TEST_KYLIN_FACT AS TEST_KYLIN_FACT LEFT OUTER JOIN 
> EDW.TEST_SELLER_TYPE_DIM AS TEST_SELLER_TYPE_DIM 
> ON TEST_KYLIN_FACT.SLR_SEGMENT_CD = TEST_SELLER_TYPE_DIM.SELLER_TYPE_CD
> Hive Query Editor : Create LOOKUP as View
> CREATE VIEW view_dim_tab_self_join AS
> SELECT distinct A.leaf_categ_id, B.leaf_categ_name, A.site_id 
> FROM DEFAULT.TEST_CATEGORY_GROUPINGS AS A LEFT OUTER JOIN 
> DEFAULT.TEST_CATEGORY_GROUPINGS AS B 
> ON A.leaf_categ_id = B.leaf_categ_id AND A.site_id = B.site_id ORDER BY 
> A.leaf_categ_id, B.leaf_categ_name, A.site_id  
> Hive Query Editor : Create LOOKUP as Table
> CREATE TABLE table_dim_tab_self_join AS
> SELECT distinct A.leaf_categ_id, B.leaf_categ_name, A.site_id 
> FROM DEFAULT.TEST_CATEGORY_GROUPINGS AS A LEFT OUTER JOIN 
> DEFAULT.TEST_CATEGORY_GROUPINGS AS B 
> ON A.leaf_categ_id = B.leaf_categ_id AND A.site_id = B.site_id ORDER BY 
> A.leaf_categ_id, B.leaf_categ_name, A.site_id  
> Additional curiosity – seems release 1.2 is handling better the final size of 
> the generated Cube – assuming I didn’t make any mistake in cube definitions 
> and used same in both 1.2-release and 2.1-SNAPSHOT, I am getting smaller cube 
> size in release 1.2 than in 2.1-SNAPSHOT codeline - in both cases the number 
> of rows is exactly same. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to