Thanks Jeff! I did as you suggested and the job went through in a reasonable time. However the performance is not perfect. With mr engine the performance was better.
I didn't set hive.tez.container.size before. The default seems to be -1. I wonder what that means in practice. Any way seems like quite many containers were spawned, so it must be using something rather small value. SET hive.tez.container.size; hive.tez.container.size=-1 Also I was running with this default value: hive> SET hive.auto.convert.join.noconditionaltask.size; hive.auto.convert.join.noconditionaltask.size=10000000 So I changed them to*: SET hive.tez.container.size=4096; SET hive.auto.convert.join.noconditionaltask.size=1252698795; This seems to mean that tez container size needs to be configured depending on yarn container configuration (so basically need to provide different values depending on the cluster size). If the relationship is really so strict, couldn't Tez pick a suitable container size automatically to avoid running out of memory? Any way, thanks a lot for solving my problem! Juho *) P.S. I tried to follow this guide basically: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_performance_tuning/content/hive_perf_best_pract_config_tez.html Do you know if it's with some special hive release that you can specify 4096MB (and ..noconditionaltask.size=1370MB), because for me Hive didn't accept MB suffix. Also this seems to be more or less the only document I've found about tuning Tez for Hive. I wonder if there are some other options and if I could find extensive documentation somewhere. On Fri, Oct 23, 2015 at 11:25 AM, Jianfeng (Jeff) Zhang < [email protected]> wrote: > > Maybe your tez container size is less than container of mr. You can try > increasing the container size of tez (You can check the container size of > your job in RM WebUI) > e.g. Set hive.tez.container.size=4096 > > BTW, not sure your cluster environment. I meet the similar issue under the > cluster managed by ambari. By default in ambari the hive container size is > less than the container size of reducer in MR. > This may cause performance issue in the reduce stage when input data size > is much large. > > > > Best Regard, > Jeff Zhang > > > From: Juho Autio <[email protected]> > Reply-To: "[email protected]" <[email protected]> > Date: Friday, October 23, 2015 at 4:08 PM > To: "[email protected]" <[email protected]> > Subject: Constant Full GC making Tez Hive job take almost forever > > Hi, > > I'm running a Hive script with tez-0.7.0. The progress is real slow and in > the container logs I'm seeing constant Full GC lines, so that there doesn't > seem to be no time for the JVM to actually execute anything between the GC > pauses. > > When running the same Hive script with mr execution engine, the job goes > through normally. > > So there's something specific to Tez's memory usage that causes the Full > GC issue. > > Also with similar clusters & configuration other Hive jobs have gone > through with Tez just fine. This issue happens when I just add a little > more data to be processed by the script. With a smaller workload it goes > through also with Tez engine with the expected execution time. > > For example an extract from one of the container logs: > > > application_1445328511212_0001/container_1445328511212_0001_01_000292/stdout.gz > > 791.208: [Full GC > [PSYoungGen: 58368K->56830K(116736K)] > [ParOldGen: 348914K->348909K(349184K)] > 407282K->405740K(465920K) > [PSPermGen: 43413K->43413K(43520K)], 1.4063790 secs] [Times: user=5.22 > sys=0.04, real=1.40 secs] > Heap > PSYoungGen total 116736K, used 58000K [0x00000000f5500000, > 0x0000000100000000, 0x0000000100000000) > eden space 58368K, 99% used > [0x00000000f5500000,0x00000000f8da41a0,0x00000000f8e00000) > from space 58368K, 0% used > [0x00000000f8e00000,0x00000000f8e00000,0x00000000fc700000) > to space 58368K, 0% used > [0x00000000fc700000,0x00000000fc700000,0x0000000100000000) > ParOldGen total 349184K, used 348909K [0x00000000e0000000, > 0x00000000f5500000, 0x00000000f5500000) > object space 349184K, 99% used > [0x00000000e0000000,0x00000000f54bb4b0,0x00000000f5500000) > PSPermGen total 43520K, used 43413K [0x00000000d5a00000, > 0x00000000d8480000, 0x00000000e0000000) > object space 43520K, 99% used > [0x00000000d5a00000,0x00000000d84657a8,0x00000000d8480000) > > If I understand the GC log correctly, it seems like ParOldGen is full and > Full GC doesn't manage to free space from there. So maybe Tez has created > too many objects that can't be released. It could be a memory leak. Or > maybe this is just not big enough minimum heap for Tez in general? I could > probably fix the problem by changing configuration somehow to simply have > less containers and thus bigger heap size per container? Still, changing to > bigger nodes doesn't seem like a solution that would eventually scale, so I > would prefer to resolve this properly. > > Please, could you help me with how to troubleshoot & fix this issue? > > Cheers, > Juho > -- *Juho Autio* Analytics Developer Hatch Rovio Entertainment Ltd Mobile: + 358 (0)45 313 0122 [email protected] www.rovio.com *This message and its attachments may contain confidential information and is intended solely for the attention and use of the named addressee(s). If you are not the intended recipient and / or you have received this message in error, please contact the sender immediately and delete all material you have received in this message. You are hereby notified that any use of the information, which you have received in error in whatsoever form, is strictly prohibited. Thank you for your co-operation.*
