[
https://issues.apache.org/jira/browse/KYLIN-1515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15248188#comment-15248188
]
Rachel Asher Silver edited comment on KYLIN-1515 at 4/19/16 5:44 PM:
---------------------------------------------------------------------
Experiencing the same with Kylin 1.5.1 (Hbase 1.3 build) on MapR 5.1 when
running through the tutorial cube build. Work around didn't work but produced
the second log:
Pre-workaround
===================================================
Number of reduce tasks is set to 0 since there's no reduce operator
java.io.IOException: Cannot initialize Cluster. Please check your configuration
for mapreduce.framework.name and the correspond server addresses.
at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:121)
at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:83)
at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:76)
at org.apache.hadoop.mapred.JobClient.init(JobClient.java:470)
at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:449)
at
org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:401)
at
org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:137)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1670)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1429)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1212)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1076)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1066)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:311)
at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:708)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Job Submission failed with exception 'java.io.IOException(Cannot initialize
Cluster. Please check your configuration for mapreduce.framework.name and the
correspond server addresses.)'
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.mr.MapRedTask
Post-workaround
===================================================
Kylin
tutorial
Insight
Model
Monitor
System
Help
Welcome, ADMIN
Cube Name:
Filter ...
JobsJobs in: NEW PENDING RUNNING FINISHED ERROR DISCARDED
Job Name Cube Progress Last Modified Time Duration
Actions
tutorial_cube - 20111230000000_20130531000000 - BUILD - PDT 2016-04-19 10:40:13
tutorial_cube
ERROR
2016-04-19 09:40:47 PST 0.23 mins Action
tutorial_cube - 20111230000000_20130531000000 - BUILD - PDT 2016-04-19 09:46:36
tutorial_cube
0.00%
2016-04-19 09:39:47 PST 0.17 mins N/A
Total: 2
Detail Information
Job Name tutorial_cube - 20111230000000_20130531000000 - BUILD - PDT
2016-04-19 10:40:13
Job ID 7d311b43-a377-4e48-95d0-16a1c5a73197
Status ERROR
Duration 0.23 mins
MapReduce Waiting 0.00 mins
Start 2016-04-19 09:40:32 PST
2016-04-19 09:40:32 PST
#1 Step Name: Create Intermediate Flat Hive Table
Duration: 0.25 mins
#2 Step Name: Extract Fact Table Distinct Columns
Duration: 0 seconds
#3 Step Name: Build Dimension Dictionary
Duration: 0 seconds
#4 Step Name: Save Cuboid Statistics
Duration: 0 seconds
#5 Step Name: Create HTable
Duration: 0 seconds
#6 Step Name: Build Base Cuboid Data
Duration: 0 seconds
#7 Step Name: Build N-Dimension Cuboid Data : 0-Dimension
Duration: 0 seconds
#8 Step Name: Build Cube
Duration: 0 seconds
#9 Step Name: Convert Cuboid Data to HFile
Duration: 0 seconds
#10 Step Name: Load HFile to HBase Table
Duration: 0 seconds
#11 Step Name: Update Cube Info
Duration: 0 seconds
#12 Step Name: Garbage Collection
Duration: 0 seconds
End
Apache Kylin | Apache Kylin Community
Output
OS command error exit with 2 -- hive -e "USE default;
DROP TABLE IF EXISTS
kylin_intermediate_tutorial_cube_20111230000000_20130531000000;
CREATE EXTERNAL TABLE IF NOT EXISTS
kylin_intermediate_tutorial_cube_20111230000000_20130531000000
(
DEFAULT_KYLIN_SALES_PART_DT date
,DEFAULT_KYLIN_SALES_PRICE decimal(19,4)
,DEFAULT_KYLIN_SALES_SELLER_ID bigint
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\177'
STORED AS SEQUENCEFILE
LOCATION
'/kylin/kylin_metadata/kylin-7d311b43-a377-4e48-95d0-16a1c5a73197/kylin_intermediate_tutorial_cube_20111230000000_20130531000000';
SET dfs.replication=2;
SET hive.exec.compress.output=true;
SET hive.auto.convert.join.noconditionaltask=true;
SET hive.auto.convert.join.noconditionaltask.size=300000000;
SET
mapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.SnappyCodec;
SET
mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.SnappyCodec;
SET hive.merge.mapfiles=true;
SET hive.merge.mapredfiles=true;
SET mapred.output.compression.type=BLOCK;
SET hive.merge.size.per.task=256000000;
SET hive.support.concurrency=false;
SET mapreduce.job.split.metainfo.maxsize=-1;
INSERT OVERWRITE TABLE
kylin_intermediate_tutorial_cube_20111230000000_20130531000000 SELECT
KYLIN_SALES.PART_DT
,KYLIN_SALES.PRICE
,KYLIN_SALES.SELLER_ID
FROM DEFAULT.KYLIN_SALES as KYLIN_SALES
INNER JOIN DEFAULT.KYLIN_CAL_DT as KYLIN_CAL_DT
ON KYLIN_SALES.PART_DT = KYLIN_CAL_DT.CAL_DT
WHERE (price>0) AND (KYLIN_SALES.PART_DT >= '2011-12-30' AND
KYLIN_SALES.PART_DT < '2013-05-31')
;
"
Logging initialized using configuration in
jar:file:/opt/mapr/hive/hive-1.2/lib/hive-common-1.2.0-mapr-1603.jar!/hive-log4j.properties
OK
Time taken: 1.046 seconds
OK
Time taken: 0.165 seconds
OK
Time taken: 0.264 seconds
Query ID = root_20160419174040_1d28cd91-5ced-42f1-b448-8cce1df3fc38
Total jobs = 3
2016-04-19 17:40:41 Starting to launch local task to process map join;
maximum memory = 477102080
2016-04-19 17:40:42 Dump the side-table for tag: 0 with group count: 515
into file:
file:/tmp/root/63131c16-a7ce-4e7c-aace-094a68643cd9/hive_2016-04-19_17-40-40_050_2641925910235619022-1/-local-10003/HashTable-Stage-9/MapJoin-mapfile00--.hashtable
2016-04-19 17:40:42 Uploaded 1 File to:
file:/tmp/root/63131c16-a7ce-4e7c-aace-094a68643cd9/hive_2016-04-19_17-40-40_050_2641925910235619022-1/-local-10003/HashTable-Stage-9/MapJoin-mapfile00--.hashtable
(106315 bytes)
2016-04-19 17:40:42 End of local task; Time Taken: 1.166 sec.
Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1461006272498_0006, Tracking URL = <>/
Kill Command = /opt/mapr/hadoop/hadoop-2.7.0/bin/hadoop job -kill
job_1461006272498_0006
Hadoop job information for Stage-9: number of mappers: 0; number of reducers: 0
2016-04-19 17:40:47,126 Stage-9 map = 0%, reduce = 0%
Ended Job = job_1461006272498_0006 with errors
Error during job, obtaining debugging information...
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Stage-Stage-9: MAPRFS Read: 0 MAPRFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec
was (Author: raichlea):
Experiencing the same with Kylin 1.5.1 (Hbase 1.3 build) on MapR 5.1 when
running through the tutorial cube build:
Number of reduce tasks is set to 0 since there's no reduce operator
java.io.IOException: Cannot initialize Cluster. Please check your configuration
for mapreduce.framework.name and the correspond server addresses.
at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:121)
at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:83)
at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:76)
at org.apache.hadoop.mapred.JobClient.init(JobClient.java:470)
at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:449)
at
org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:401)
at
org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:137)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1670)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1429)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1212)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1076)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1066)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:311)
at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:708)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Job Submission failed with exception 'java.io.IOException(Cannot initialize
Cluster. Please check your configuration for mapreduce.framework.name and the
correspond server addresses.)'
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> Cube Build - java.io.IOException: Cannot initialize Cluster. Please check
> your configuration for mapreduce.framework.name and the correspond server
> addresses
> -------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: KYLIN-1515
> URL: https://issues.apache.org/jira/browse/KYLIN-1515
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
> Affects Versions: v1.5.0
> Environment: MapR - Hadoop 2.5.1
> Reporter: Richard Calaba
> Assignee: Dong Li
>
> Knowing that MapR is not officially supported we were able to use Kylin 1.2
> in our MapR distro successfully.
> After upgrade to Kylin 1.5.0 we are facing issue with the Cube Build process
> - the one which worked on 1.2 without issues. The Cube is created from
> scratch (no Kylin metadata migration) on clean install of Kylinn1.5.0 (HDFS
> directory /kytlin and HBase tables KYLIN* and kylin* deleted prior upgrade
> from 1.2 to 1.5.0).
> The build process is Failing in Step 1 complaining about property value
> "mapreduce.framework.name". According to this post
> https://stackoverflow.com/questions/19642862/cannot-initialize-cluster-exception-while-running-job-on-hadoop-2
> - the solution should be to ensure the respective property is correctly set
> in the file mapred-site.xml.
> Originally in our MapR distro the property was commented (and having value
> yarn-tez) - even after adding the "yarn" value -> the Build process still
> fails with same exception - I am not sure what is wrong with our cluster
> configuration. Anyone has an idea ???
> Below is our mapred-site.xml content:
> ==============================
> cat /opt/mapr/hadoop/hadoop-2.5.1/etc/hadoop/mapred-site.xml
> <!-- Put site-specific property overrides in this file. -->
> <configuration>
> <property>
> <name>mapreduce.jobhistory.address</name>
> <value>node1:10020</value>
> </property>
> <property>
> <name>mapreduce.jobhistory.webapp.address</name>
> <value>node1:19888</value>
> </property>
> <!--
> <property>
> <name>mapreduce.framework.name</name>
> <value>yarn-tez</value>
> </property>
> -->
> <property>
> <name>mapreduce.framework.name</name>
> <value>yarn</value>
> </property>
> </configuration>
> Known workaround:
> ================
> Know workaround to make this error to disappear is to delete from
> conf/kylin_hive_conf.xml this property section:
> <property>
> <name>dfs.block.size</name>
> <value>32000000</value>
> <description>Want more mappers for in-mem cubing, thus smaller the DFS block
> size</description>
> </property>
> The full log output of Cube Build Step 1 - attached below:
> ==============================================
> OS command error exit with 1 -- hive -e "USE default;
> DROP TABLE IF EXISTS
> kylin_intermediate_TestCube_clone2_19700101000000_2922789940817071255;
> CREATE EXTERNAL TABLE IF NOT EXISTS
> kylin_intermediate_TestCube_clone2_19700101000000_2922789940817071255
> (
> DEFAULT_BATTING_PLAYER_ID string
> ,DEFAULT_BATTING_YEAR int
> ,DEFAULT_BATTING_RUNS int
> )
> ROW FORMAT DELIMITED FIELDS TERMINATED BY '\177'
> STORED AS SEQUENCEFILE
> LOCATION
> '/kylin/kylin_metadata/kylin-3eb4b652-a2a4-4659-8b6a-dc822e1341fb/kylin_intermediate_TestCube_clone2_19700101000000_2922789940817071255';
> SET dfs.replication=2;
> SET dfs.block.size=32000000;
> SET hive.exec.compress.output=true;
> SET hive.auto.convert.join.noconditionaltask=true;
> SET hive.auto.convert.join.noconditionaltask.size=300000000;
> SET
> mapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.SnappyCodec;
> SET
> mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.SnappyCodec;
> SET hive.merge.mapfiles=true;
> SET hive.merge.mapredfiles=true;
> SET mapred.output.compression.type=BLOCK;
> SET hive.merge.size.per.task=256000000;
> SET hive.support.concurrency=false;
> SET mapreduce.job.split.metainfo.maxsize=-1;
> INSERT OVERWRITE TABLE
> kylin_intermediate_TestCube_clone2_19700101000000_2922789940817071255 SELECT
> BATTING.PLAYER_ID
> ,BATTING.YEAR
> ,BATTING.RUNS
> FROM DEFAULT.BATTING as BATTING
> LEFT JOIN DEFAULT.TEMP_BATTING as TEMP_BATTING
> ON BATTING.PLAYER_ID = TEMP_BATTING.COL_VALUE
> ;
> "
> Logging initialized using configuration in
> jar:file:/opt/mapr/hive/hive-1.0/lib/hive-common-1.0.0-mapr-1510.jar!/hive-log4j.properties
> OK
> Time taken: 0.611 seconds
> OK
> Time taken: 0.83 seconds
> OK
> Time taken: 0.474 seconds
> Query ID = mapr_20160321201212_610078b4-5805-43eb-8fd1-87304530a84e
> Total jobs = 3
> 2016-03-21 08:12:32 Starting to launch local task to process map join;
> maximum memory = 477102080
> 2016-03-21 08:12:32 Dump the side-table for tag: 1 with group count: 95196
> into file:
> file:/tmp/mapr/b35c5ac2-3231-4ef1-9e6b-216c0a1bd9ef/hive_2016-03-21_20-12-31_085_8296009472449837835-1/-local-10003/HashTable-Stage-9/MapJoin-mapfile01--.hashtable
> 2016-03-21 08:12:32 Uploaded 1 File to:
> file:/tmp/mapr/b35c5ac2-3231-4ef1-9e6b-216c0a1bd9ef/hive_2016-03-21_20-12-31_085_8296009472449837835-1/-local-10003/HashTable-Stage-9/MapJoin-mapfile01--.hashtable
> (7961069 bytes)
> 2016-03-21 08:12:32 End of local task; Time Taken: 0.853 sec.
> Launching Job 1 out of 3
> Number of reduce tasks is set to 0 since there's no reduce operator
> java.io.IOException: Cannot initialize Cluster. Please check your
> configuration for mapreduce.framework.name and the correspond server
> addresses.
> at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:121)
> at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:83)
> at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:76)
> at org.apache.hadoop.mapred.JobClient.init(JobClient.java:470)
> at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:449)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:399)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:137)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1619)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1379)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1192)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1019)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1009)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:201)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:153)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:364)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:299)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:662)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:631)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:570)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> Job Submission failed with exception 'java.io.IOException(Cannot initialize
> Cluster. Please check your configuration for mapreduce.framework.name and the
> correspond server addresses.)'
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
