Nick Muerdter created KYLIN-1745:
------------------------------------
Summary: Java heap space errors for all builds in Kylin 1.5.2
Key: KYLIN-1745
URL: https://issues.apache.org/jira/browse/KYLIN-1745
Project: Kylin
Issue Type: Bug
Affects Versions: v1.5.2
Reporter: Nick Muerdter
When trying to build any cubes in Kylin 1.5.2, I get Java heap space errors on
step 2 (Extract Fact Table Distinct Columns). I experience the same heap space
error on my own cube, as well as using the sample cube data (from
{{sample.sh}}).
I'm testing on a fairly stock HDP-2.2.9.0-3393 installation. I've tried
tweaking the memory settings, but even with the map memory tuned to allow 8GB,
I still get the same heap space error even for the small sample data (where 8GB
seems like it should be more than plenty).
If I try the same build against the sample data on a fresh 1.5.1 installation,
the cube builds as expected, but with a fresh 1.5.2 installation, I get these
heap space errors on any build attempts.
Here's the backtrace from the failed map job:
{code}
2016-05-29 01:51:57,910 WARN [main]
org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration:
tried hadoop-metrics2-maptask.properties,hadoop-metrics2.properties
2016-05-29 01:51:57,977 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at
10 second(s).
2016-05-29 01:51:57,977 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system
started
2016-05-29 01:51:57,984 INFO [main] org.apache.hadoop.mapred.YarnChild:
Executing with tokens:
2016-05-29 01:51:58,017 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind:
mapreduce.job, Service: job_1464481246919_0079, Ident:
(org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@548e6d58)
2016-05-29 01:51:58,045 INFO [main] org.apache.hadoop.mapred.YarnChild:
Sleeping for 0ms before retrying again. Got null now.
2016-05-29 01:51:58,249 INFO [main] org.apache.hadoop.mapred.YarnChild:
mapreduce.cluster.local.dir for child:
/hadoop/yarn/local/usercache/api-umbrella/appcache/application_1464481246919_0079
2016-05-29 01:51:58,443 INFO [main]
org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated.
Instead, use dfs.metrics.session-id
2016-05-29 01:51:58,792 INFO [main]
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output
Committer Algorithm version is 1
2016-05-29 01:51:58,801 INFO [main] org.apache.hadoop.mapred.Task: Using
ResourceCalculatorProcessTree : [ ]
2016-05-29 01:51:59,078 INFO [main] org.apache.hadoop.mapred.MapTask:
Processing split: org.apache.hive.hcatalog.mapreduce.HCatSplit@12dae582
2016-05-29 01:51:59,218 ERROR [main] org.apache.hadoop.mapred.YarnChild: Error
running child : java.lang.OutOfMemoryError: Java heap space
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:983)
at
org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:401)
at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:81)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:695)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:767)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
{code}
Any ideas? Let me know if I can supply any other details.
Thanks!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
