[
https://issues.apache.org/jira/browse/KYLIN-1508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15206452#comment-15206452
]
Richard Calaba commented on KYLIN-1508:
---------------------------------------
I am sorry I do not have the Cube JSONs anymore - those were very simple Cubes
- one dimension from fact table, one derrived from lookup table and one
additional keyfigure/measure with MAX aggregation.
Anyway the ask here is the support of the vHive Views for lookup tables so as
the https://issues.apache.org/jira/browse/KYLIN-1077 is already asking for it -
we might close this issue as duplicate.
> NPE error ERROR - Build Cube - Step 3 if LOOKUP table is Hive View - :
> java.lang.NullPointerException at
> org.apache.kylin.source.hive.HiveTable.getSignature
> ------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: KYLIN-1508
> URL: https://issues.apache.org/jira/browse/KYLIN-1508
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
> Affects Versions: v1.5.0, v1.3.0, v1.2, all
> Environment: CentOS 6.7 , Hortonworks Hadoop 2.2.4.2 - sandbox
> Reporter: Richard Calaba
> Assignee: Dong Li
>
> Affected releases: The error occurs in 1.2 release (downloaded as binary
> release from Kylin Home page). The same error occurs also in ??latest??
> snapshot 2.1 (compiled today 3/12/16 from Kylin Git using bracnh
> 2.x-staging). Tests were done on Hortonworks HDP 2.2.4 sandbox. Thus I assume
> that all current branches are affected.
> Error Summary: Whereas it is possible to successfully build cube which FACT
> table referring Hive View – it seems not to be possible to build a cube which
> has LOOKUP table referring Hive View. It seems to be bug in metadata
> processing in Step 3 of the Build process. Build process fails with exception:
> (release 1.2)
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.kylin.dict.lookup.HiveTable.getSignature(HiveTable.java:72)
> at
> org.apache.kylin.dict.lookup.SnapshotTable.<init>(SnapshotTable.java:62)
> at
> org.apache.kylin.dict.lookup.SnapshotManager.buildSnapshot(SnapshotManager.java:85)
> at
> org.apache.kylin.cube.CubeManager.buildSnapshotTable(CubeManager.java:205)
> at
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:60)
> at
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:41)
> at
> org.apache.kylin.job.hadoop.dict.CreateDictionaryJob.run(CreateDictionaryJob.java:52)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at
> org.apache.kylin.job.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:62)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at
> org.apache.kylin.common.util.HadoopUtil.fixWindowsPath(HadoopUtil.java:76)
> at
> org.apache.kylin.common.util.HadoopUtil.makeURI(HadoopUtil.java:68)
> at
> org.apache.kylin.common.util.HadoopUtil.getFileSystem(HadoopUtil.java:63)
> at
> org.apache.kylin.dict.lookup.FileTable.getSizeAndLastModified(FileTable.java:85)
> at
> org.apache.kylin.dict.lookup.HiveTable.getSignature(HiveTable.java:57)
> ... 16 more
> Error in latest branch - (snapshoot-2.1) :
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.kylin.source.hive.HiveTable.getSignature(HiveTable.java:71)
> at
> org.apache.kylin.dict.lookup.SnapshotTable.<init>(SnapshotTable.java:64)
> at
> org.apache.kylin.dict.lookup.SnapshotManager.buildSnapshot(SnapshotManager.java:89)
> at
> org.apache.kylin.cube.CubeManager.buildSnapshotTable(CubeManager.java:208)
> at
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:59)
> at
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:42)
> at
> org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:56)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at
> org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:60)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:124)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Importance of resolution of this bug: I believe that having the ability to
> use Hive Views is essential and should work for both FACT tables/views and
> LOOKUP/DIMENSION tables/views. 1st it helps to do more efficient OLAP staging
> by not forcing to materialize temporary views used just for OLAP modelling.
> 2ndly and most importantly having views with left outer and inner joins can
> further overcome the current Kylin limitation of not beeing to able to
> provide snowflake-like models.
> Further info - reproduce scenario: I have created 2 Hive Tables (FACT and
> LOOKUP) using CREATE TABLE AS SELECT … statement in Hive. Then I have created
> 2 Hive Views using CREATE VIEW AS SELECT … supplying exactly same SQL
> statements (using left outer and inner joins). All 4 tables/views are
> correctly producing data in Hive. If I build simple Cube in Kylin using 2
> tables – everything works fine. If I use Hive view for FACT table and regular
> Hive Table for LOOKUP – it also works. The error occurs if I try to use Hiev
> View as LOOKUP table and define dimension from it.
> Just for completeness - below is full Hive QL for creating the table/views.
> It operates on tables which are created by internal tests while building
> Kylin 1.2. But I believe the exact view definition should not matter. I did
> couple of additional tests to create very simple views by doing only "SELECT
> * FROM one_table" and the error was still the same.
> Hive Query Editor : Create Fact as View
> CREATE VIEW view_fact_tab_enhanced
> AS SELECT *
> FROM DEFAULT.TEST_KYLIN_FACT AS TEST_KYLIN_FACT LEFT OUTER JOIN
> EDW.TEST_SELLER_TYPE_DIM AS TEST_SELLER_TYPE_DIM
> ON TEST_KYLIN_FACT.SLR_SEGMENT_CD = TEST_SELLER_TYPE_DIM.SELLER_TYPE_CD
> Hive Query Editor : Create Fact as Table
> CREATE TABLE table_fact_tab_enhanced
> AS SELECT *
> FROM DEFAULT.TEST_KYLIN_FACT AS TEST_KYLIN_FACT LEFT OUTER JOIN
> EDW.TEST_SELLER_TYPE_DIM AS TEST_SELLER_TYPE_DIM
> ON TEST_KYLIN_FACT.SLR_SEGMENT_CD = TEST_SELLER_TYPE_DIM.SELLER_TYPE_CD
> Hive Query Editor : Create LOOKUP as View
> CREATE VIEW view_dim_tab_self_join AS
> SELECT distinct A.leaf_categ_id, B.leaf_categ_name, A.site_id
> FROM DEFAULT.TEST_CATEGORY_GROUPINGS AS A LEFT OUTER JOIN
> DEFAULT.TEST_CATEGORY_GROUPINGS AS B
> ON A.leaf_categ_id = B.leaf_categ_id AND A.site_id = B.site_id ORDER BY
> A.leaf_categ_id, B.leaf_categ_name, A.site_id
> Hive Query Editor : Create LOOKUP as Table
> CREATE TABLE table_dim_tab_self_join AS
> SELECT distinct A.leaf_categ_id, B.leaf_categ_name, A.site_id
> FROM DEFAULT.TEST_CATEGORY_GROUPINGS AS A LEFT OUTER JOIN
> DEFAULT.TEST_CATEGORY_GROUPINGS AS B
> ON A.leaf_categ_id = B.leaf_categ_id AND A.site_id = B.site_id ORDER BY
> A.leaf_categ_id, B.leaf_categ_name, A.site_id
> Additional curiosity – seems release 1.2 is handling better the final size of
> the generated Cube – assuming I didn’t make any mistake in cube definitions
> and used same in both 1.2-release and 2.1-SNAPSHOT, I am getting smaller cube
> size in release 1.2 than in 2.1-SNAPSHOT codeline - in both cases the number
> of rows is exactly same.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)