[
https://issues.apache.org/jira/browse/TAJO-548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880147#comment-13880147
]
Min Zhou commented on TAJO-548:
-------------------------------
For reducing frequent young gc, we have 2 choices.
One is to tune the young generation heap size and other related options , the
other is to figure out which method create so many temporary objects.
After some investigation, I found 2 suspicions
One is
{noformat}
at java.lang.String.<init>(String.java:570)
at
org.apache.tajo.storage.TextSerializerDeserializer.deserialize(TextSerializerDeserializer.java:139)
at org.apache.tajo.storage.LazyTuple.get(LazyTuple.java:123)
at org.apache.tajo.engine.eval.FieldEval.eval(FieldEval.java:51)
at org.apache.tajo.engine.eval.BinaryEval.eval(BinaryEval.java:140)
at org.apache.tajo.engine.eval.BinaryEval.eval(BinaryEval.java:140)
at org.apache.tajo.engine.planner.Projector.eval(Projector.java:48)
at
org.apache.tajo.engine.planner.physical.SeqScanExec.next(SeqScanExec.java:186)
at
org.apache.tajo.engine.planner.physical.HashAggregateExec.compute(HashAggregateExec.java:57)
at
org.apache.tajo.engine.planner.physical.HashAggregateExec.next(HashAggregateExec.java:83)
at
org.apache.tajo.engine.planner.physical.HashShuffleFileWriteExec.next(HashShuffleFileWriteExec.java:117)
at org.apache.tajo.worker.Task.run(Task.java:370)
at org.apache.tajo.worker.TaskRunner$1.run(TaskRunner.java:392)
at java.lang.Thread.run(Thread.java:662)
{noformat}
The other is
{noformat}
at java.util.ArrayList.<init>(ArrayList.java:119)
at org.apache.tajo.util.Bytes.splitWorker(Bytes.java:1490)
at org.apache.tajo.util.Bytes.splitPreserveAllTokens(Bytes.java:1451)
at org.apache.tajo.storage.CSVFile$CSVScanner.next(CSVFile.java:429)
at
org.apache.tajo.engine.planner.physical.SeqScanExec.next(SeqScanExec.java:183)
at
org.apache.tajo.engine.planner.physical.HashAggregateExec.compute(HashAggregateExec.java:57)
at
org.apache.tajo.engine.planner.physical.HashAggregateExec.next(HashAggregateExec.java:83)
at
org.apache.tajo.engine.planner.physical.HashShuffleFileWriteExec.next(HashShuffleFileWriteExec.java:117)
at org.apache.tajo.worker.Task.run(Task.java:370)
at org.apache.tajo.worker.TaskRunner$1.run(TaskRunner.java:392)
at java.lang.Thread.run(Thread.java:662)
{noformat}
The first one is easy to improve. That line parse an byte array, convert it
into a double value. The original way is firstly convert the byte array to a
String, which involved decoding from byte array to char array and creating char
array and string instance. After that, using Double.valueOf(String) to achieve
the conversion purpose. Actually, we can directly parse a byte array into
double , without intermediary conversion, and without allocating memory for
intermediary char array and string objects.
> Investigate frequent young gc
> ------------------------------
>
> Key: TAJO-548
> URL: https://issues.apache.org/jira/browse/TAJO-548
> Project: Tajo
> Issue Type: Improvement
> Affects Versions: 0.8-incubating
> Reporter: Min Zhou
>
> Two days ago, I ran a 300X TPCH Q1 on a 7 slaves , 1 master tajo clsuter.
> Here is my configuration for each worker jvm
> {noformat}
> export TAJO_WORKER_OPTS="-Xmx20g -Xms20g -XX:MaxPermSize=512m -verbose:gc
> -Xloggc:$TAJO_LOG_DIR/worker-gc.log -XX:+UseConcMarkSweepGC
> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCompressedOops "
> {noformat}
> I observed very frequent young gc
> {noformat}
> $ jstat -gcutil `pgrep -f TajoWorker` 1000
> S0 S1 E O P YGC YGCT FGC FGCT GCT
> 0.00 2.13 57.40 3.79 99.65 5185 75.042 0 0.000 75.042
> 0.00 2.13 57.40 3.79 99.65 5185 75.042 0 0.000 75.042
> 0.00 2.13 57.41 3.79 99.65 5185 75.042 0 0.000 75.042
> 0.00 2.13 57.42 3.79 99.65 5185 75.042 0 0.000 75.042
> 0.00 2.13 57.42 3.79 99.65 5185 75.042 0 0.000 75.042
> 0.00 2.13 57.43 3.79 99.65 5185 75.042 0 0.000 75.042
> 0.00 2.13 57.43 3.79 99.65 5185 75.042 0 0.000 75.042
> 0.00 2.13 57.43 3.79 99.65 5185 75.042 0 0.000 75.042
> 0.00 2.13 57.43 3.79 99.65 5185 75.042 0 0.000 75.042
> 0.00 2.13 57.43 3.79 99.65 5185 75.042 0 0.000 75.042
> 0.00 2.13 57.44 3.79 99.65 5185 75.042 0 0.000 75.042
> 0.00 2.13 57.53 3.79 99.65 5185 75.042 0 0.000 75.042
> 0.00 2.13 57.57 3.79 99.65 5185 75.042 0 0.000 75.042
> 0.00 2.13 57.58 3.79 99.65 5185 75.042 0 0.000 75.042
> 0.00 2.13 57.58 3.79 99.65 5185 75.042 0 0.000 75.042
> 0.00 2.13 57.58 3.79 99.65 5185 75.042 0 0.000 75.042
> 0.00 2.13 57.59 3.79 99.65 5185 75.042 0 0.000 75.042
> 0.00 2.13 57.59 3.79 99.65 5185 75.042 0 0.000 75.042
> 0.00 2.13 57.60 3.79 99.65 5185 75.042 0 0.000 75.042
> 0.00 2.13 57.60 3.79 99.65 5185 75.042 0 0.000 75.042
> 0.00 2.13 66.73 3.79 99.66 5185 75.042 0 0.000 75.042
> 0.00 66.46 17.89 3.79 99.66 5190 75.082 0 0.000 75.082
> 8.79 0.00 37.36 3.88 99.66 5202 75.533 0 0.000 75.533
> 0.00 9.10 21.61 3.88 99.66 5216 75.926 0 0.000 75.926
> ...
> 105.523
> {noformat}
> After finish the Q1 query, GCT is at 105.523, which means young gc cost 30
> seconds. As well known, ParNew YGC is a kind of stop the world gc. The
> whole query need about 98 secs. So in this case, young gc waste nearly 1
> third of the query time.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
