The follwing are the parameters: -XX:+UseG1GC -XX:+UnlockDiagnostivVMOptions -XX:G1SummarizeConcMark -XX:InitiatingHeapOccupancyPercent=35 spark.executor.memory=4G
________________________________ 发件人: Ted Yu <yuzhih...@gmail.com> 发送时间: 2016年5月30日 9:47:05 收件人: condor join 抄送: user@spark.apache.org 主题: Re: G1 GC takes too much time bq. It happens during the Reduce majority. Did the above refer to reduce operation ? Can you share your G1GC parameters (and heap size for workers) ? Thanks On Sun, May 29, 2016 at 6:15 PM, condor join <spark_ker...@outlook.com<mailto:spark_ker...@outlook.com>> wrote: Hi, my spark application failed due to take too much time during GC. Looking at the logs I found these things: 1.there are Young GC takes too much time,and not found Full GC happen this; 2.The time takes too much during the object copy; 3.It happened more easily when there were not enough resources; 4.It happens during the Reduce majority. have anyone met the same question? thanks --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org<mailto:user-unsubscr...@spark.apache.org> For additional commands, e-mail: user-h...@spark.apache.org<mailto:user-h...@spark.apache.org>
