That error message indicates that the size of the serialized metadata of some tables exceed the maximum allowed size of a byte array (2GB) and doesn't have anything to do with the size of the heap. The logs indicate a large number of refresh tables that could trigger this kind of errors, especially if these tables have large metadata and/or incremental stats. You may want to see if there are any changes in the workload (e.g. more frequent REFRESH statements than before) and if the value of the statestore_frequency_update_ms was changed after the upgrade.
On Thu, Sep 22, 2016 at 7:24 PM, 廖松博 <liaoson...@gridsum.com> wrote: > Hi, > > > > The event is the upgrading from CDH 5.5 -> 5.7. And the > Operation is the regular operations with Impala 2.3 which we work well with > for a long time , but after the upgrading, it consume so much memory. > > It seems a memory leak, because as follow chart: > > [image: cid:image001.jpg@01D21582.157D2320] > > when we use impala 2.3, it use 20G memory more or less, but when we > upgrade to 2.5 and set Java Heap Size of Catalog Server in Bytes = 32 G, > but the catalog server consume the 32G rapidly, and we reset Java Heap Size > of Catalog Server in Bytes to 64G , it also increase to 64G in a not long > time. > > So I double if it is a memory leak and I see the issue > https://issues.cloudera.org/browse/IMPALA-3568 , is it the root cause ? > > And the attached is the catalog server log and it contains the : > java.lang.OutOfMemoryError: > Requested array size exceeds VM limit. > > 1. Is the root cause of the memory leak? > > 2. Any way to find the table which cause the exception? > > > > Thanks. > > > > Songbo > > > > *发件人:* Dimitris Tsirogiannis [mailto:dtsirogian...@cloudera.com] > *发送时间:* 2016年9月23日 8:49 > *收件人:* user@impala.incubator.apache.org > *抄送:* Matthew Jacobs > *主题:* Re: Catalog Server Out of memory when upgrading from CDH 5.5 -> 5.7 > > > > There seem to be some events after sept 21 that trigger a spike in memory > usage. Can you identify from CM what these events might be? For > example, refreshing tables with lots of metadata. Looking at the catalog > logs might also give you some hints on what the system is doing during > those memory spikes. > > > > Dimitris > > On Thursday, September 22, 2016, 廖松博 <liaoson...@gridsum.com> wrote: > > Hi guys, > > > > We upgrade out CDH cluster from 5.5 to 5.7 at Sep 20, and from > the following figure: > > [image: cid:image001.jpg@01D21582.157D2320] > > You can see the catalog server in impala 2.5 use much more memory > than what in impala 2.3 and causes the out of memory error. > > We set the Load Catalog In Background = false and restart the > service but not help. > > We set Java Heap Size of Catalog Server in Bytes from 32G to 64G > ,but the memory usage also increase to 64G. > > > > Is there any bugs in catalog server for impala 2.5? Any > suggestion for us? > > > > Thanks. > > > > Songbo > >
