Hi,
I am trying to create a cube from a star schema created using Hive External
tables (below an example) stored as TEXT FILE (CSV).
CREATE EXTERNAL TABLE IF NOT EXISTS USERS_TABLE (
uid INT,
name STRING
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\073' LINES TERMINATED BY '\012'
STORED AS TEXTFILE
LOCATION '/data/users';
To CSV files are obtained from Spark RDDs, so they are saved as part-xxxx.
Below the HDFS listing
hdfs dfs -ls /data/users
Found 12 items
-rw-r--r-- 3 hdfs hdfs 0 2015-07-07 12:05 /data/users/_SUCCESS
-rw-r--r-- 3 hdfs hdfs 3699360 2015-07-07 12:05 /data/users/part-00000
-rw-r--r-- 3 hdfs hdfs 3694740 2015-07-07 12:05 /data/users/part-00001
-rw-r--r-- 3 hdfs hdfs 3685374 2015-07-07 12:05 /data/users/part-00002
-rw-r--r-- 3 hdfs hdfs 3719646 2015-07-07 12:05 /data/users/part-00003
-rw-r--r-- 3 hdfs hdfs 3682476 2015-07-07 12:05 /data/users/part-00004
-rw-r--r-- 3 hdfs hdfs 3679956 2015-07-07 12:05 /data/users/part-00005
-rw-r--r-- 3 hdfs hdfs 3700242 2015-07-07 12:05 /data/users/part-00006
-rw-r--r-- 3 hdfs hdfs 3672186 2015-07-07 12:05 /data/users/part-00007
-rw-r--r-- 3 hdfs hdfs 3682350 2015-07-07 12:05 /data/users/part-00008
-rw-r--r-- 3 hdfs hdfs 3680292 2015-07-07 12:05 /data/users/part-00009
-rw-r--r-- 3 hdfs hdfs 3697722 2015-07-07 12:05 /data/users/part-00010
The CUBE build JOB fails when try to build the Dimension Dictionary with the
following exception (it seems that the Hive Table data directory MUST contain
only one file)
java.lang.IllegalStateException: Expect 1 and only 1 non-zero file under
hdfs://gas.gfmintegration.it:8020/data/cdr/bb/dimensions/users, but find 11
at
org.apache.kylin.dict.lookup.HiveTable.findOnlyFile(HiveTable.java:123)
at
org.apache.kylin.dict.lookup.HiveTable.computeHDFSLocation(HiveTable.java:107)
at
org.apache.kylin.dict.lookup.HiveTable.getHDFSLocation(HiveTable.java:83)
at
org.apache.kylin.dict.lookup.HiveTable.getFileTable(HiveTable.java:76)
at
org.apache.kylin.dict.lookup.HiveTable.getSignature(HiveTable.java:71)
at
org.apache.kylin.dict.DictionaryManager.buildDictionary(DictionaryManager.java:164)
at
org.apache.kylin.cube.CubeManager.buildDictionary(CubeManager.java:154)
at
org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:53)
at
org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:42)
at
org.apache.kylin.job.hadoop.dict.CreateDictionaryJob.run(CreateDictionaryJob.java:53)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at
org.apache.kylin.job.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
result code:2
Do you have any indications on how to create a proper Hive star schema for
Kylin?
I would like to use external tables (stored as CSV, parquet files or HBase)
because I need to process the same data also from Spark.
Thanks in advance.
BR,
-- gas
