Jihoon Son created TAJO-2012:
--------------------------------
Summary: OOM while writing history
Key: TAJO-2012
URL: https://issues.apache.org/jira/browse/TAJO-2012
Project: Tajo
Issue Type: Bug
Reporter: Jihoon Son
Fix For: 0.12.0
I met OOM when querying on 10TB data set.
Here is the log.
{noformat}
2015-12-08 03:39:13,591 INFO org.apache.tajo.util.history.HistoryWriter: Saving
query unit:
hdfs://appliance007:9010/tmp/tajo-hadoop/staging/history/20151208/query-detail/q_1449490267507_0003/eb_1449490267507_0003_000001.hist
2015-12-08 03:39:22,520 ERROR org.apache.tajo.util.history.HistoryWriter: Error
while saving query history: q_1449490267507_0003:Requested array size exceeds
VM limit
java.lang.OutOfMemoryError: Requested array size exceeds VM limit
at java.util.Arrays.copyOf(Arrays.java:3332)
at
java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
at
java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
at
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:421)
at java.lang.StringBuffer.append(StringBuffer.java:272)
at java.io.StringWriter.write(StringWriter.java:112)
at com.google.gson.stream.JsonWriter.string(JsonWriter.java:538)
at com.google.gson.stream.JsonWriter.value(JsonWriter.java:404)
at
com.google.gson.internal.bind.TypeAdapters$13.write(TypeAdapters.java:353)
at
com.google.gson.internal.bind.TypeAdapters$13.write(TypeAdapters.java:337)
at
com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68)
at
com.google.gson.internal.bind.ArrayTypeAdapter.write(ArrayTypeAdapter.java:93)
at
com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68)
at
com.google.gson.internal.bind.ArrayTypeAdapter.write(ArrayTypeAdapter.java:93)
at
com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68)
at
com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.write(ReflectiveTypeAdapterFactory.java:89)
at
com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.write(ReflectiveTypeAdapterFactory.java:195)
at
com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68)
at
com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.write(CollectionTypeAdapterFactory.java:96)
at
com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.write(CollectionTypeAdapterFactory.java:60)
at com.google.gson.Gson.toJson(Gson.java:586)
at com.google.gson.Gson.toJson(Gson.java:565)
at com.google.gson.Gson.toJson(Gson.java:520)
at
org.apache.tajo.util.history.StageHistory.toTasksJson(StageHistory.java:233)
at
org.apache.tajo.util.history.HistoryWriter$WriterThread.writeQueryHistory(HistoryWriter.java:378)
at
org.apache.tajo.util.history.HistoryWriter$WriterThread.writeHistory(HistoryWriter.java:318)
at
org.apache.tajo.util.history.HistoryWriter$WriterThread.run(HistoryWriter.java:244)
2015-12-08 09:11:08,538 INFO org.apache.tajo.util.history.HistoryCleaner:
Cleaning task history dir: file:/tmp/tajo-hadoop/history/20151201
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
