The query returned partial results warning in CM
Hi guys, We are using CDH 5.10.0, the impala page of CM shows: [cid:image001.png@01D2C8B0.C07C0C00] We set the limit to 10: [cid:image002.png@01D2C8B0.C07C0C00] But not help. I found some similar issues: http://community.cloudera.com/t5/Cloudera-Manager-Installation/Error-occurred-during-obtaining-time-series-data-using-the-cm/td-p/47256 Any idea to solve the problem ? Thanks Best Regards, Songbo
答复: Use Centos 6.7 servers with Centos 6.5 servers in one CDH cluster
Thanks a lot, I will do it. 发件人: Bharath Vissapragada [mailto:bhara...@cloudera.com] 发送时间: 2016年10月20日 0:29 收件人: user@impala.incubator.apache.org 主题: Re: Use Centos 6.7 servers with Centos 6.5 servers in one CDH cluster That page talks about "major" OS releases and both your OS versions are in 6.x series. So I think that configuration is fine. On Wed, Oct 19, 2016 at 9:04 AM, Henry Robinson <he...@apache.org<mailto:he...@apache.org>> wrote: This isn't a recommendation so much as a requirement for users who wish to purchase commercial support from Cloudera. From our - the Apache project's - perspective, I don't think there is much reason to be concerned about mixing 6.5 and 6.7 on the same cluster. On 19 October 2016 at 02:14, 廖松博 <liaoson...@gridsum.com<mailto:liaoson...@gridsum.com>> wrote: Hi all, I notice from http://www.cloudera.com/documentation/enterprise/5-7-x/topics/cm_ig_cm_requirements.html#cmig_topic_4_1 [cid:image001.png@01D22AA6.F6803220] Most of server in our CDH cluster is Centos 6.5 , we’d like add some servers of Centos 6.7 to our clusters, per this docs above it is not recommended , right ? Does it really matters? Thanks 廖松博 | 技术研发部 Add：上海市静安区南京西路1468号中欣大厦3604室 邮编：200040 [说明: 说明: 说明: 说明: 说明: cid:image002.png@01CC5AC4.C3D5CB80] www.gridsum.com<http://www.gridsum.com/>
答复: 答复: 答复: Catalog Server Out of memory when upgrading from CDH 5.5 -> 5.7
Hi Dimitris, Thanks for fast response. 1. I double checked, load_catalog_in_background is false , so even if I restart catalog separately, it should not ask for full updates, right ? 2. Start loading those big tables, thanks, we will try this. 3. The catalog actually already takes 3-4 times memory than what in impala 2.3 , and I doubt if we set Java Heap Size of Catalog Server in Bytes a bigger value, it will eat more memory. Thanks. Songbo 发件人: Dimitris Tsirogiannis [mailto:dtsirogian...@cloudera.com] 发送时间: 2016年9月23日 13:03 收件人: user@impala.incubator.apache.org 主题: Re: 答复: 答复: Catalog Server Out of memory when upgrading from CDH 5.5 -> 5.7 You may want to consider the following steps: 1. Restart the entire cluster and not just the catalog. If an impalad detects a change in the catalog (caused by a restart) it asks for full updates. 2. Verify that the load_catalog_in_background is set to false. If that is the case and there is no workload, the catalog shouldn't be active and the memory utilization should stay flat. 3. Start loading those big tables, ideally one at a time by issuing a REFRESH statement. See if the metadata of some tables are too big causing the 2GB limit to be exceeded. If so, you may want to consider dropping incremental stats (if used) or merging small files (if any). 4. Monitor memory consumption. There might be some increase in memory consumption in 5.7 but not in the order of 3-4X. 5. Resume normal workloads. Dimitris On Thu, Sep 22, 2016 at 9:45 PM, 廖松博 <liaoson...@gridsum.com<mailto:liaoson...@gridsum.com>> wrote: Hi Dimitris, statestore_frequency_update_ms not change during the update, is 2000ms by default. And not more frequent REFRESH statement. Actually we have 2 problem here: 1. the size of the serialized metadata of some tables exceed the maximum allowed size of a byte array (2GB). The potential reason are : a) Before the upgrading, the number of partitions and metadata of per table are added day by day , but once we upgrade and restart the catalog, the entire metadata of per table is transferred to catalog, so this transferring exceed the maximum allowed size of a byte array, does it make sense? 2. The memory usage of catalog server increase continuously , and before the upgrade, catalog server only use 20G memory but now it reach our limit whether the limit is 32G or 96G. So I doubt there is a memory leak or inefficient GC And I see the issue https://issues.cloudera.org/browse/IMPALA-3910, seems that it is related to my problem, but why the memory usage of catalog server increase continuously after upgrading. Thanks Songbo 发件人: Dimitris Tsirogiannis [mailto:dtsirogian...@cloudera.com<mailto:dtsirogian...@cloudera.com>] 发送时间: 2016年9月23日 11:33 收件人: user@impala.incubator.apache.org<mailto:user@impala.incubator.apache.org> 抄送: Matthew Jacobs 主题: Re: 答复: Catalog Server Out of memory when upgrading from CDH 5.5 -> 5.7 That error message indicates that the size of the serialized metadata of some tables exceed the maximum allowed size of a byte array (2GB) and doesn't have anything to do with the size of the heap. The logs indicate a large number of refresh tables that could trigger this kind of errors, especially if these tables have large metadata and/or incremental stats. You may want to see if there are any changes in the workload (e.g. more frequent REFRESH statements than before) and if the value of the statestore_frequency_update_ms was changed after the upgrade. On Thu, Sep 22, 2016 at 7:24 PM, 廖松博 <liaoson...@gridsum.com<mailto:liaoson...@gridsum.com>> wrote: Hi, The event is the upgrading from CDH 5.5 -> 5.7. And the Operation is the regular operations with Impala 2.3 which we work well with for a long time , but after the upgrading, it consume so much memory. It seems a memory leak, because as follow chart: [cid:image001.jpg@01D21582.157D2320] when we use impala 2.3, it use 20G memory more or less, but when we upgrade to 2.5 and set Java Heap Size of Catalog Server in Bytes = 32 G, but the catalog server consume the 32G rapidly, and we reset Java Heap Size of Catalog Server in Bytes to 64G , it also increase to 64G in a not long time. So I double if it is a memory leak and I see the issue https://issues.cloudera.org/browse/IMPALA-3568 , is it the root cause ? And the attached is the catalog server log and it contains the : java.lang.OutOfMemoryError: Requested array size exceeds VM limit. 1. Is the root cause of the memory leak? 2. Any way to find the table which cause the exception? Thanks. Songbo 发件人: Dimitris Tsirogiannis [mailto:dtsirogian...@cloudera.com<mailto:dtsirogian...@cloudera.com>] 发送时间: 2016年9月23日 8:49 收件人: user@impala.incubator.apache.org<mailto:user@impala.incubator.apache.org> 抄送: Matthew Jacobs 主题: Re: Catalog Server Out of memory
Catalog Server Out of memory when upgrading from CDH 5.5 -> 5.7
Hi guys, We upgrade out CDH cluster from 5.5 to 5.7 at Sep 20, and from the following figure: [cid:image001.jpg@01D21574.46A788E0] You can see the catalog server in impala 2.5 use much more memory than what in impala 2.3 and causes the out of memory error. We set the Load Catalog In Background = false and restart the service but not help. We set Java Heap Size of Catalog Server in Bytes from 32G to 64G ,but the memory usage also increase to 64G. Is there any bugs in catalog server for impala 2.5? Any suggestion for us? Thanks. Songbo
RE: Impala user resource isolation best practice
Thanks a lot for your patient. Songbo -邮件原件- 发件人: Matthew Jacobs [mailto:m...@cloudera.com] 发送时间: 2016年7月26日 13:40 收件人: 廖松博 抄送: user@impala.incubator.apache.org 主题: Re: Impala user resource isolation best practice If memory cannot be spilled to disk then the query will be killed. Best, Matt On Mon, Jul 25, 2016 at 7:26 PM, 廖松博 <liaoson...@gridsum.com> wrote: > Hi Matt, > > You mentioned that " If the query tries to use more than the mem > limit it will spill if possible" , if not possible what happened, the query > will be killed or any unexpected result ? > Thanks. > > Songbo > > -邮件原件- > 发件人: Matthew Jacobs [mailto:m...@cloudera.com] > 发送时间: 2016年7月26日 8:33 > 收件人: 廖松博 > 抄送: user@impala.incubator.apache.org > 主题: Re: Impala user resource isolation best practice > > On Mon, Jul 25, 2016 at 4:36 PM, 廖松博 <liaoson...@gridsum.com> wrote: >> Hi Matt, >> Thanks for reply, but there are still something confuse me: >> 1. Per >> http://www.cloudera.com/documentation/archive/impala/2-x/2-0-x/topics/impala_mem_limit.html >> mem_limit is the maximum amount of memory a query can allocate on each >> node, not across the entire cluster, right ? If we set mem limit, the >> estimation does not work ? > > Basically yes. You don't want to rely on estimates because they're going to > be wrong. > >> 2. So if we set query mem limit = 5G, the coordinator will check if >> all the workers has at least 5G memory , if not it will reject the query, >> right? > > Yes, partially. Right now it's not smart enough to actually reserve memory, > but if it thinks it the resources are available it will let the query start. > If the coordinator thinks the resources are already being used, the query > will queue. If there isn't enough memory for Impala on the node, then yes, > the query will be rejected. > >> 3. If the coordinator accept the query and dispatch to workers, what >> will happen if the query use 10G (more than mem_limit we set ) on that node >> during execution？Be canceled or it is possible out of memory to crash the >> impalad process ? Does it use estimation to do anything during execution ? >> Or estimation is not useful when we set query mem limit ? > > If the query tries to use more than the mem limit it will spill if possible. > If it can't spill, the query will be cancelled. It should not crash -- that > would be a bug. Estimates are not used at runtime. > >> 4. If we does not set query mem limit, coordinator will use its >> estimation instead of query limit to check if workers have enough memory to >> execute, right? > > You probably won't get what you expect. The estimates will be very far off > and there are no mem limits enforced, so all queries will just start > consuming whatever they want. > >> >> Thanks a lot for your patient >> >> Songbo >> >> -邮件原件- >> 发件人: Matthew Jacobs [mailto:m...@cloudera.com] >> 发送时间: 2016年7月26日 0:24 >> 收件人: 廖松博 <liaoson...@gridsum.com> >> 抄送: user@impala.incubator.apache.org >> 主题: Re: Impala user resource isolation best practice >> >> Hi, >> >> If you set the query mem_limit the estimates are not used in this way, and >> they're enforced at runtime. We recommend that you always set the query >> mem_limits. In Impala 2.5 you can configure the admission control pools to >> have a default query option mem_limit, e.g. every query in poolA gets a >> default query mem_limit=5G, and perhaps poolB gets a default query >> mem_limit=10g. You can still override those query options if you'd like with >> "set mem_limit=X;". There is also an impalad command line argument to set >> the default query options (fallback in all cases): >> --default_query_options=‘mem_limit=X’ >> >> Best, >> Matt >> >> On Sun, Jul 24, 2016 at 8:25 PM, 廖松博 <liaoson...@gridsum.com> wrote: >>> Hi Matt, >>> >>> Thanks for suggestion. Even though we can solve "soft limit" >>> problem using single coordinator, the coordinator doesn't know how much >>> memory will be used during execution. The limit is actually based on the >>> memory size estimated by coordinator, right. So it is possible that the >>> coordinator accept a query which using estimating 5G memory which under >>> limit and dispatch to workers, but the query actually use 15G memory during >>> execution so limit doesn’t make sense. >>> And as I know the estimation will be more accurate if the table h
RE: Impala user resource isolation best practice
Hi Matt, You mentioned that " If the query tries to use more than the mem limit it will spill if possible" , if not possible what happened, the query will be killed or any unexpected result ? Thanks. Songbo -邮件原件- 发件人: Matthew Jacobs [mailto:m...@cloudera.com] 发送时间: 2016年7月26日 8:33 收件人: 廖松博 抄送: user@impala.incubator.apache.org 主题: Re: Impala user resource isolation best practice On Mon, Jul 25, 2016 at 4:36 PM, 廖松博 <liaoson...@gridsum.com> wrote: > Hi Matt, > Thanks for reply, but there are still something confuse me: > 1. Per > http://www.cloudera.com/documentation/archive/impala/2-x/2-0-x/topics/impala_mem_limit.html > mem_limit is the maximum amount of memory a query can allocate on each node, > not across the entire cluster, right ? If we set mem limit, the estimation > does not work ? Basically yes. You don't want to rely on estimates because they're going to be wrong. > 2. So if we set query mem limit = 5G, the coordinator will check if > all the workers has at least 5G memory , if not it will reject the query, > right? Yes, partially. Right now it's not smart enough to actually reserve memory, but if it thinks it the resources are available it will let the query start. If the coordinator thinks the resources are already being used, the query will queue. If there isn't enough memory for Impala on the node, then yes, the query will be rejected. > 3. If the coordinator accept the query and dispatch to workers, what > will happen if the query use 10G (more than mem_limit we set ) on that node > during execution？Be canceled or it is possible out of memory to crash the > impalad process ? Does it use estimation to do anything during execution ? Or > estimation is not useful when we set query mem limit ? If the query tries to use more than the mem limit it will spill if possible. If it can't spill, the query will be cancelled. It should not crash -- that would be a bug. Estimates are not used at runtime. > 4. If we does not set query mem limit, coordinator will use its > estimation instead of query limit to check if workers have enough memory to > execute, right? You probably won't get what you expect. The estimates will be very far off and there are no mem limits enforced, so all queries will just start consuming whatever they want. > > Thanks a lot for your patient > > Songbo > > -邮件原件- > 发件人: Matthew Jacobs [mailto:m...@cloudera.com] > 发送时间: 2016年7月26日 0:24 > 收件人: 廖松博 <liaoson...@gridsum.com> > 抄送: user@impala.incubator.apache.org > 主题: Re: Impala user resource isolation best practice > > Hi, > > If you set the query mem_limit the estimates are not used in this way, and > they're enforced at runtime. We recommend that you always set the query > mem_limits. In Impala 2.5 you can configure the admission control pools to > have a default query option mem_limit, e.g. every query in poolA gets a > default query mem_limit=5G, and perhaps poolB gets a default query > mem_limit=10g. You can still override those query options if you'd like with > "set mem_limit=X;". There is also an impalad command line argument to set the > default query options (fallback in all cases): > --default_query_options=‘mem_limit=X’ > > Best, > Matt > > On Sun, Jul 24, 2016 at 8:25 PM, 廖松博 <liaoson...@gridsum.com> wrote: >> Hi Matt, >> >> Thanks for suggestion. Even though we can solve "soft limit" problem >> using single coordinator, the coordinator doesn't know how much memory will >> be used during execution. The limit is actually based on the memory size >> estimated by coordinator, right. So it is possible that the coordinator >> accept a query which using estimating 5G memory which under limit and >> dispatch to workers, but the query actually use 15G memory during execution >> so limit doesn’t make sense. >> And as I know the estimation will be more accurate if the table has >> stats, right ? But it does not promise to be accurate even if stats exists. >> Does the wrongly estimation matters ? For example will it cause the >> out of memory problem ? If yes, is there any best practice to avoid that ? >> Thank you very much. >> >> Songbo >> >> -邮件原件- >> 发件人: Matthew Jacobs [mailto:m...@cloudera.com] >> 发送时间: 2016年7月19日 11:35 >> 收件人: 廖松博 >> 抄送: user@impala.incubator.apache.org >> 主题: Re: Impala user resource isolation best practice >> >> Hi Songbo, >> >> Yes, setting the MEM_LIMITs is a good idea. In Impala 2.5 and after, you can >> set a per-pool default query mem limit. Cloudera Manager makes this e
RE: Impala user resource isolation best practice
Hi Matt, Thanks for suggestion. Even though we can solve "soft limit" problem using single coordinator, the coordinator doesn't know how much memory will be used during execution. The limit is actually based on the memory size estimated by coordinator, right. So it is possible that the coordinator accept a query which using estimating 5G memory which under limit and dispatch to workers, but the query actually use 15G memory during execution so limit doesn’t make sense. And as I know the estimation will be more accurate if the table has stats, right ? But it does not promise to be accurate even if stats exists. Does the wrongly estimation matters ? For example will it cause the out of memory problem ? If yes, is there any best practice to avoid that ? Thank you very much. Songbo -邮件原件- 发件人: Matthew Jacobs [mailto:m...@cloudera.com] 发送时间: 2016年7月19日 11:35 收件人: 廖松博 抄送: user@impala.incubator.apache.org 主题: Re: Impala user resource isolation best practice Hi Songbo, Yes, setting the MEM_LIMITs is a good idea. In Impala 2.5 and after, you can set a per-pool default query mem limit. Cloudera Manager makes this easy in the Dynamic Resource Pools page. You are right about the possibility of overwhelming a single coordinator. You can consider setting up one larger node to be a coordinator (e.g. a lot of RAM & CPUs), and setting it so that it *does not* have a datanode on it to minimize the number of query fragments scheduled on it. In order to do that you'll also need to set the impalad cmd line argument "-abort_on_config_error=false" because it normally expects a DN to be colocated with the Impalad. It is still possible to overwhelm a single coordinator like this, but we've found it to work reasonably well in our tests. Of course it is very dependent on your hardware and workload. Best, Matt On Mon, Jul 18, 2016 at 6:31 PM, 廖松博 <liaoson...@gridsum.com> wrote: > Hi Matt, > > If we use a single coordinator, if a lot of queries come in, the > coordinator may be out of memory, right ? Because the coordinator take charge > of merge results from workers and send back to client. > Thanks. > > Songbo > > -邮件原件- > 发件人: Matthew Jacobs [mailto:m...@cloudera.com] > 发送时间: 2016年7月19日 7:50 > 收件人: 廖松博 > 抄送: user@impala.incubator.apache.org > 主题: Re: Impala user resource isolation best practice > > The 'soft limits' refer to the behavior that, when multiple coordinators are > used and queries are submitted concurrently, the admission control cannot > guarantee the limits because the decisions are made based on some stale > information that needs to be updated by the statestore. If you use a single > coordinator you avoid that problem. That said, a separate problem is that > there is no way to guarantee that other resources, e.g. network, IO, etc. are > not abused by a single user. > > On Mon, Jul 18, 2016 at 4:29 PM, 廖松博 <liaoson...@gridsum.com> wrote: >> Hi Matthew， >> Thanks for your reply. The point is , per admission control >> documents, most of impala limits are "soft limit", are the 2 settings you >> mentioned also "soft limit" ? The soft limit means the pool will exceed the >> memory/concurrency limit at some moment when impala is not aware of. But it >> is affecting other pool at that moment. >> Thanks. >> >> Songbo >> >> -邮件原件- >> 发件人: Matthew Jacobs [mailto:m...@cloudera.com] >> 发送时间: 2016年7月19日 0:44 >> 收件人: user@impala.incubator.apache.org >> 主题: Re: Impala user resource isolation best practice >> >> By the way, some of the controls I mentioned were added in Impala 2.5, so >> you should consider upgrading if you're not already using a newer version of >> Impala. >> >> Thanks, >> Matt >> >> On Mon, Jul 18, 2016 at 9:20 AM, Matthew Jacobs <m...@cloudera.com> wrote: >>> Hi Songbo, >>> >>> Right now the best you can do is with admission control with: >>> (a) a single coordinator to avoid the possibility of over-admitting >>> by different coordinators >>> (b) setting default query mem limits so that individual queries are >>> limited >>> >>> For your scenario, I'd recommend setting up 2 pools, one for user A >>> and a second for user B. Set the max number of running queries for >>> user A to something reasonable for the concurrency for that workload. >>> Set the max memory for the user B pool to the portion of cluster >>> memory you're willing to give to those queries. (Notice the pool >>> with the small queries has the max number of running queries set and >>> the pool with
