Balazs Jeszenszky has posted comments on this change. ( http://gerrit.cloudera.org:8080/10457 )
Change subject: IMPALA-7050: [DOCS] Document the max serialized incremental stat size setting ...................................................................... Patch Set 3: (7 comments) initial pass http://gerrit.cloudera.org:8080/#/c/10457/3/docs/topics/impala_perf_stats.xml File docs/topics/impala_perf_stats.xml: http://gerrit.cloudera.org:8080/#/c/10457/3/docs/topics/impala_perf_stats.xml@709 PS3, Line 709: When you need to run <codeph>COMPUTE INCREMENTAL STATS</codeph> on : very large tables, you can use the configuration setting : <codeph>inc_stats_size_limit_bytes</codeph> to reduce the load on : the catalog server. turn into passive? http://gerrit.cloudera.org:8080/#/c/10457/3/docs/topics/impala_perf_stats.xml@714 PS3, Line 714: This limit is set as a safety check, to prevent the JVM from hitting : a maximum array limit of 1 GB or runs out of memory. the limit is 2GB with suggested JVM versions. Readers would need to understand that this is only a part of the entire table's metadata (all of which together must be below 2GB when serialized as Thrift). So while we're trying to avoid a 2GB limit, increasing the 200MB limit can easily lead to crashes (other parts of the table's metadata can add up quickly). http://gerrit.cloudera.org:8080/#/c/10457/3/docs/topics/impala_perf_stats.xml@724 PS3, Line 724: To change the <codeph>inc_stats_size_limit_bytes</codeph> value This is a startup option for the impala daemons and the catalogd, not a client (impala-shell) side config. http://gerrit.cloudera.org:8080/#/c/10457/3/docs/topics/impala_perf_stats.xml@733 PS3, Line 733: 1 GB I'd put 500MB http://gerrit.cloudera.org:8080/#/c/10457/3/docs/topics/impala_perf_stats.xml@733 PS3, Line 733: spike in heap usage as well as crashes. http://gerrit.cloudera.org:8080/#/c/10457/3/docs/topics/impala_perf_stats.xml@736 PS3, Line 736: <p> : Setting <codeph>inc_stats_size_limit_bytes</codeph> to a big value, : such as 1 GB or more, can result in a spike in heap usage. : </p> duplicate of above? http://gerrit.cloudera.org:8080/#/c/10457/3/docs/topics/impala_shell_options.xml File docs/topics/impala_shell_options.xml: http://gerrit.cloudera.org:8080/#/c/10457/3/docs/topics/impala_shell_options.xml@570 PS3, Line 570: --inc_stats_size_limit_bytes this is an impalad-side option -- To view, visit http://gerrit.cloudera.org:8080/10457 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifa80325f0008d42a9cc8178e7c144fc2b49d7d4e Gerrit-Change-Number: 10457 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni <arod...@cloudera.com> Gerrit-Reviewer: Balazs Jeszenszky <jes...@gmail.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Comment-Date: Tue, 05 Jun 2018 20:31:20 +0000 Gerrit-HasComments: Yes
