Need hive-site and tez-site configuration samples

[Jyotsna G]

All,
Can anyone share their Hive-site.xml and tez-site.xml config settings wrt
hive on TEZ?

Specifically looking for hive server 2 config settings for TEZ.

Thanks,
Jyotsna

Re: Best way/tool to debug memory leaks in HiveServer2

[Gopal Vijayaraghavan]

> It also shows that the process is consuming more than 30GB. However, it is 
> not clear what is causing the process to consume more than 30GB.

The Xmx only applies to the heap size, there's another factor that is usually 
ignored which are the network buffers and compression buffers used by Java.

Most of these are only collected on a full GC pause (or OS memory pressure, I 
think).

Try running a jmap -histo:live and see what happens to the extra buffers on 
that list.

Cheers,
Gopal

Re: Best way/tool to debug memory leaks in HiveServer2

[Prasanth Jayachandran]

One option to check for is if THP is enabled on the node running HS2. If enable 
try disabling THP and see if it helps

# Disable THP echo never | sudo tee -a 
/sys/kernel/mm/transparent_hugepage/enabled echo never | sudo tee -a 
/sys/kernel/mm/transparent_hugepage/defrag Thanks Prasanth

On Mar 13, 2018, at 5:32 PM, rajan sthapit 
> wrote:

Hi,

I am currently trying to debug a memory leak in a HiveServer2 java process. The 
heap size for the jvm is set to 16GB. When I look at the heap usage it is well 
under 16 GB, However, if I look at the total memory consumed by the process(top 
command)  it's more than 30 GB.

Currently, I am trying to use pmap to see the memory consumption by the 
process. Here is the result of the pmap

https://gist.github.com/anonymous/977b2ba8e7484d02dcec00a10439e516

It also shows that the process is consuming more than 30GB. However, it is not 
clear what is causing the process to consume more than 30GB.

Any suggestion, on how I can debug this issue. The JVM heap memory usage seems 
to be fine. It is well under 16GB. However, the total process memory consumed 
by the java process is more than 30GB.

Could you please recommend any tools/methods that I could use to debug this 
issue?

Thanks

Best way/tool to debug memory leaks in HiveServer2

[rajan sthapit]

Hi,

I am currently trying to debug a memory leak in a HiveServer2 java process.
The heap size for the jvm is set to 16GB. When I look at the heap usage it
is well under 16 GB, However, if I look at the total memory consumed by the
process(top command)  it's more than 30 GB.

Currently, I am trying to use pmap to see the memory consumption by the
process. Here is the result of the pmap

https://gist.github.com/anonymous/977b2ba8e7484d02dcec00a10439e516


It also shows that the process is consuming more than 30GB. However, it is
not clear what is causing the process to consume more than 30GB.


Any suggestion, on how I can debug this issue. The JVM heap memory usage
seems to be fine. It is well under 16GB. However, the total process memory
consumed by the java process is more than 30GB.


Could you please recommend any tools/methods that I could use to debug this
issue?


Thanks