Catalog Server Out of memory when upgrading from CDH 5.5 -> 5.7

[廖松博]

Hi guys,



 We upgrade out CDH cluster from 5.5 to 5.7 at Sep 20, and from the 
following figure:

 [cid:image001.jpg@01D21574.46A788E0]

 You can see the catalog server in impala 2.5 use much more memory than 
what in impala 2.3 and causes the out of memory error.

 We set the Load Catalog In Background = false and restart the service 
but not help.

 We set Java Heap Size of Catalog Server in Bytes from 32G to 64G ,but 
the memory usage also increase to 64G.



 Is there any bugs in catalog server for impala 2.5? Any suggestion for 
us?



 Thanks.



Songbo

答复: 答复: 答复: Catalog Server Out of memory when upgrading from CDH 5.5 -> 5.7

[廖松博]

Hi Dimitris,

 Thanks for fast response.

1.   I double checked,  load_catalog_in_background is false , so even if I 
restart catalog separately, it should not ask for full updates, right ?

2.   Start loading those big tables, thanks, we will try this.

3.   The catalog actually already takes 3-4 times memory than what in 
impala 2.3 , and I doubt if we set Java Heap Size of Catalog Server in Bytes a 
bigger value, it will eat more memory.

Thanks.

Songbo

发件人: Dimitris Tsirogiannis [mailto:dtsirogian...@cloudera.com]
发送时间: 2016年9月23日 13:03
收件人: user@impala.incubator.apache.org
主题: Re: 答复: 答复: Catalog Server Out of memory when upgrading from CDH 5.5 -> 5.7

You may want to consider the following steps:
1. Restart the entire cluster and not just the catalog. If an impalad detects a 
change in the catalog (caused by a restart) it asks for full updates.
2. Verify that the load_catalog_in_background is set to false. If that is the 
case and there is no workload, the catalog shouldn't be active and the memory 
utilization should stay flat.
3. Start loading those big tables, ideally one at a time by issuing a REFRESH 
statement. See if the metadata of some tables are too big causing the 2GB limit 
to be exceeded. If so, you may want to consider dropping incremental stats (if 
used) or merging small files (if any).
4. Monitor memory consumption. There might be some increase in memory 
consumption in 5.7 but not in the order of 3-4X.
5. Resume normal workloads.

Dimitris

On Thu, Sep 22, 2016 at 9:45 PM, 廖松博 
> wrote:
Hi  Dimitris,

 statestore_frequency_update_ms not change during the update, is 2000ms 
by default. And not more frequent REFRESH statement.
 Actually we have 2 problem here:

1.   the size of the serialized metadata of some tables exceed the maximum 
allowed size of a byte array (2GB). The potential reason are :

a) Before the upgrading, the number of partitions and metadata of per 
table are added day by day , but once we upgrade and restart the catalog, the 
entire metadata of per table is transferred  to catalog, so this transferring 
exceed the maximum allowed size of a byte array, does it make sense?

2.   The memory usage of catalog server  increase continuously , and before 
the upgrade, catalog server only use 20G memory but now it reach our limit 
whether the limit is 32G or 96G. So I doubt there is a memory leak or 
inefficient GC

And I see the issue https://issues.cloudera.org/browse/IMPALA-3910, seems that 
it is related to my problem, but why the memory usage of catalog server 
increase continuously after upgrading.
Thanks


Songbo

发件人: Dimitris Tsirogiannis 
[mailto:dtsirogian...@cloudera.com]
发送时间: 2016年9月23日 11:33
收件人: user@impala.incubator.apache.org
抄送: Matthew Jacobs
主题: Re: 答复: Catalog Server Out of memory when upgrading from CDH 5.5 -> 5.7

That error message indicates that the size of the serialized metadata of some 
tables exceed the maximum allowed size of a byte array (2GB) and doesn't have 
anything to do with the size of the heap. The logs indicate a large number of 
refresh tables that could trigger this kind of errors, especially if these 
tables have large metadata and/or incremental stats. You may want to see if 
there are any changes in the workload (e.g. more frequent REFRESH statements 
than before) and if the value of the statestore_frequency_update_ms was changed 
after the upgrade.

On Thu, Sep 22, 2016 at 7:24 PM, 廖松博 
> wrote:
Hi,

 The event is the upgrading from CDH 5.5 -> 5.7.  And the Operation is 
the regular operations with Impala 2.3 which we work well with for a long time 
, but after the upgrading, it consume so much memory.
 It seems a memory leak, because as follow chart:
[cid:image001.jpg@01D21582.157D2320]
when we use impala 2.3,  it use 20G memory more or less, but when we upgrade to 
2.5 and set Java Heap Size of Catalog Server in Bytes = 32 G, but the catalog 
server consume the 32G rapidly, and we reset Java Heap Size of Catalog Server 
in Bytes to 64G , it also increase to 64G in a not long time.
So I double if it is a memory leak and I see the issue 
https://issues.cloudera.org/browse/IMPALA-3568 , is it the root cause ?
And the attached is the catalog server log and it contains the : 
java.lang.OutOfMemoryError: Requested array size exceeds VM limit.

1.   Is the root cause of the memory leak?

2.   Any way to find the table which cause the exception?

Thanks.

Songbo

发件人: Dimitris Tsirogiannis 
[mailto:dtsirogian...@cloudera.com]
发送时间: 2016年9月23日 8:49
收件人: user@impala.incubator.apache.org
抄送: Matthew Jacobs
主题: Re: Catalog Server Out of memory when upgrading from CDH 5.5 -> 5.7

There seem to be some