[jira] [Updated] (HIVE-15882) HS2 generating high memory pressure with many partitions and concurrent queries
[ https://issues.apache.org/jira/browse/HIVE-15882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15882: -- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed to master. Thanks Misha for the contribution! > HS2 generating high memory pressure with many partitions and concurrent > queries > --- > > Key: HIVE-15882 > URL: https://issues.apache.org/jira/browse/HIVE-15882 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Misha Dmitriev >Assignee: Misha Dmitriev > Fix For: 2.2.0 > > Attachments: HIVE-15882.01.patch, HIVE-15882.02.patch, > HIVE-15882.03.patch, HIVE-15882.04.patch, hs2-crash-2000p-500m-50q.txt > > > I've created a Hive table with 2000 partitions, each backed by two files, > with one row in each file. When I execute some number of concurrent queries > against this table, e.g. as follows > {code} > for i in `seq 1 50`; do beeline -u jdbc:hive2://localhost:1 -n admin -p > admin -e "select count(i_f_1) from misha_table;" & done > {code} > it results in a big memory spike. With 20 queries I caused an OOM in a HS2 > server with -Xmx200m and with 50 queries - in the one with -Xmx500m. > I am attaching the results of jxray (www.jxray.com) analysis of a heap dump > that was generated in the 50queries/500m heap scenario. It suggests that > there are several opportunities to reduce memory pressure with not very > invasive changes to the code: > 1. 24.5% of memory is wasted by duplicate strings (see section 6). With > String.intern() calls added in the ~10 relevant places in the code, this > overhead can be highly reduced. > 2. Almost 20% of memory is wasted due to various suboptimally used > collections (see section 8). There are many maps and lists that are either > empty or have just 1 element. By modifying the code that creates and > populates these collections, we may likely save 5-10% of memory. > 3. Almost 20% of memory is used by instances of java.util.Properties. It > looks like these objects are highly duplicate, since for each Partition each > concurrently running query creates its own copy of Partion, PartitionDesc and > Properties. Thus we have nearly 100,000 (50 queries * 2,000 partitions) > Properties in memory. By interning/deduplicating these objects we may be able > to save perhaps 15% of memory. > So overall, I think there is a good chance to reduce HS2 memory consumption > in this scenario by ~40%. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15882) HS2 generating high memory pressure with many partitions and concurrent queries
[ https://issues.apache.org/jira/browse/HIVE-15882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-15882: -- Attachment: HIVE-15882.04.patch Addressed the most recent Rui's comments. > HS2 generating high memory pressure with many partitions and concurrent > queries > --- > > Key: HIVE-15882 > URL: https://issues.apache.org/jira/browse/HIVE-15882 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Misha Dmitriev >Assignee: Misha Dmitriev > Attachments: HIVE-15882.01.patch, HIVE-15882.02.patch, > HIVE-15882.03.patch, HIVE-15882.04.patch, hs2-crash-2000p-500m-50q.txt > > > I've created a Hive table with 2000 partitions, each backed by two files, > with one row in each file. When I execute some number of concurrent queries > against this table, e.g. as follows > {code} > for i in `seq 1 50`; do beeline -u jdbc:hive2://localhost:1 -n admin -p > admin -e "select count(i_f_1) from misha_table;" & done > {code} > it results in a big memory spike. With 20 queries I caused an OOM in a HS2 > server with -Xmx200m and with 50 queries - in the one with -Xmx500m. > I am attaching the results of jxray (www.jxray.com) analysis of a heap dump > that was generated in the 50queries/500m heap scenario. It suggests that > there are several opportunities to reduce memory pressure with not very > invasive changes to the code: > 1. 24.5% of memory is wasted by duplicate strings (see section 6). With > String.intern() calls added in the ~10 relevant places in the code, this > overhead can be highly reduced. > 2. Almost 20% of memory is wasted due to various suboptimally used > collections (see section 8). There are many maps and lists that are either > empty or have just 1 element. By modifying the code that creates and > populates these collections, we may likely save 5-10% of memory. > 3. Almost 20% of memory is used by instances of java.util.Properties. It > looks like these objects are highly duplicate, since for each Partition each > concurrently running query creates its own copy of Partion, PartitionDesc and > Properties. Thus we have nearly 100,000 (50 queries * 2,000 partitions) > Properties in memory. By interning/deduplicating these objects we may be able > to save perhaps 15% of memory. > So overall, I think there is a good chance to reduce HS2 memory consumption > in this scenario by ~40%. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15882) HS2 generating high memory pressure with many partitions and concurrent queries
[ https://issues.apache.org/jira/browse/HIVE-15882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-15882: -- Attachment: HIVE-15882.03.patch Attaching the 3rd version of the patch, addressing the recent minor review comments. > HS2 generating high memory pressure with many partitions and concurrent > queries > --- > > Key: HIVE-15882 > URL: https://issues.apache.org/jira/browse/HIVE-15882 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Misha Dmitriev >Assignee: Misha Dmitriev > Attachments: HIVE-15882.01.patch, HIVE-15882.02.patch, > HIVE-15882.03.patch, hs2-crash-2000p-500m-50q.txt > > > I've created a Hive table with 2000 partitions, each backed by two files, > with one row in each file. When I execute some number of concurrent queries > against this table, e.g. as follows > {code} > for i in `seq 1 50`; do beeline -u jdbc:hive2://localhost:1 -n admin -p > admin -e "select count(i_f_1) from misha_table;" & done > {code} > it results in a big memory spike. With 20 queries I caused an OOM in a HS2 > server with -Xmx200m and with 50 queries - in the one with -Xmx500m. > I am attaching the results of jxray (www.jxray.com) analysis of a heap dump > that was generated in the 50queries/500m heap scenario. It suggests that > there are several opportunities to reduce memory pressure with not very > invasive changes to the code: > 1. 24.5% of memory is wasted by duplicate strings (see section 6). With > String.intern() calls added in the ~10 relevant places in the code, this > overhead can be highly reduced. > 2. Almost 20% of memory is wasted due to various suboptimally used > collections (see section 8). There are many maps and lists that are either > empty or have just 1 element. By modifying the code that creates and > populates these collections, we may likely save 5-10% of memory. > 3. Almost 20% of memory is used by instances of java.util.Properties. It > looks like these objects are highly duplicate, since for each Partition each > concurrently running query creates its own copy of Partion, PartitionDesc and > Properties. Thus we have nearly 100,000 (50 queries * 2,000 partitions) > Properties in memory. By interning/deduplicating these objects we may be able > to save perhaps 15% of memory. > So overall, I think there is a good chance to reduce HS2 memory consumption > in this scenario by ~40%. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15882) HS2 generating high memory pressure with many partitions and concurrent queries
[ https://issues.apache.org/jira/browse/HIVE-15882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-15882: -- Attachment: HIVE-15882.02.patch Uploaded the second version of the patch, with comments made in reviewboard addressed. > HS2 generating high memory pressure with many partitions and concurrent > queries > --- > > Key: HIVE-15882 > URL: https://issues.apache.org/jira/browse/HIVE-15882 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Misha Dmitriev >Assignee: Misha Dmitriev > Attachments: HIVE-15882.01.patch, HIVE-15882.02.patch, > hs2-crash-2000p-500m-50q.txt > > > I've created a Hive table with 2000 partitions, each backed by two files, > with one row in each file. When I execute some number of concurrent queries > against this table, e.g. as follows > {code} > for i in `seq 1 50`; do beeline -u jdbc:hive2://localhost:1 -n admin -p > admin -e "select count(i_f_1) from misha_table;" & done > {code} > it results in a big memory spike. With 20 queries I caused an OOM in a HS2 > server with -Xmx200m and with 50 queries - in the one with -Xmx500m. > I am attaching the results of jxray (www.jxray.com) analysis of a heap dump > that was generated in the 50queries/500m heap scenario. It suggests that > there are several opportunities to reduce memory pressure with not very > invasive changes to the code: > 1. 24.5% of memory is wasted by duplicate strings (see section 6). With > String.intern() calls added in the ~10 relevant places in the code, this > overhead can be highly reduced. > 2. Almost 20% of memory is wasted due to various suboptimally used > collections (see section 8). There are many maps and lists that are either > empty or have just 1 element. By modifying the code that creates and > populates these collections, we may likely save 5-10% of memory. > 3. Almost 20% of memory is used by instances of java.util.Properties. It > looks like these objects are highly duplicate, since for each Partition each > concurrently running query creates its own copy of Partion, PartitionDesc and > Properties. Thus we have nearly 100,000 (50 queries * 2,000 partitions) > Properties in memory. By interning/deduplicating these objects we may be able > to save perhaps 15% of memory. > So overall, I think there is a good chance to reduce HS2 memory consumption > in this scenario by ~40%. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15882) HS2 generating high memory pressure with many partitions and concurrent queries
[ https://issues.apache.org/jira/browse/HIVE-15882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-15882: -- Status: Patch Available (was: In Progress) The supplied patch de-dupes most of the duplicate strings. > HS2 generating high memory pressure with many partitions and concurrent > queries > --- > > Key: HIVE-15882 > URL: https://issues.apache.org/jira/browse/HIVE-15882 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Misha Dmitriev >Assignee: Misha Dmitriev > Attachments: HIVE-15882.01.patch, hs2-crash-2000p-500m-50q.txt > > > I've created a Hive table with 2000 partitions, each backed by two files, > with one row in each file. When I execute some number of concurrent queries > against this table, e.g. as follows > {code} > for i in `seq 1 50`; do beeline -u jdbc:hive2://localhost:1 -n admin -p > admin -e "select count(i_f_1) from misha_table;" & done > {code} > it results in a big memory spike. With 20 queries I caused an OOM in a HS2 > server with -Xmx200m and with 50 queries - in the one with -Xmx500m. > I am attaching the results of jxray (www.jxray.com) analysis of a heap dump > that was generated in the 50queries/500m heap scenario. It suggests that > there are several opportunities to reduce memory pressure with not very > invasive changes to the code: > 1. 24.5% of memory is wasted by duplicate strings (see section 6). With > String.intern() calls added in the ~10 relevant places in the code, this > overhead can be highly reduced. > 2. Almost 20% of memory is wasted due to various suboptimally used > collections (see section 8). There are many maps and lists that are either > empty or have just 1 element. By modifying the code that creates and > populates these collections, we may likely save 5-10% of memory. > 3. Almost 20% of memory is used by instances of java.util.Properties. It > looks like these objects are highly duplicate, since for each Partition each > concurrently running query creates its own copy of Partion, PartitionDesc and > Properties. Thus we have nearly 100,000 (50 queries * 2,000 partitions) > Properties in memory. By interning/deduplicating these objects we may be able > to save perhaps 15% of memory. > So overall, I think there is a good chance to reduce HS2 memory consumption > in this scenario by ~40%. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15882) HS2 generating high memory pressure with many partitions and concurrent queries
[ https://issues.apache.org/jira/browse/HIVE-15882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-15882: -- Attachment: HIVE-15882.01.patch > HS2 generating high memory pressure with many partitions and concurrent > queries > --- > > Key: HIVE-15882 > URL: https://issues.apache.org/jira/browse/HIVE-15882 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Misha Dmitriev >Assignee: Misha Dmitriev > Attachments: HIVE-15882.01.patch, hs2-crash-2000p-500m-50q.txt > > > I've created a Hive table with 2000 partitions, each backed by two files, > with one row in each file. When I execute some number of concurrent queries > against this table, e.g. as follows > {code} > for i in `seq 1 50`; do beeline -u jdbc:hive2://localhost:1 -n admin -p > admin -e "select count(i_f_1) from misha_table;" & done > {code} > it results in a big memory spike. With 20 queries I caused an OOM in a HS2 > server with -Xmx200m and with 50 queries - in the one with -Xmx500m. > I am attaching the results of jxray (www.jxray.com) analysis of a heap dump > that was generated in the 50queries/500m heap scenario. It suggests that > there are several opportunities to reduce memory pressure with not very > invasive changes to the code: > 1. 24.5% of memory is wasted by duplicate strings (see section 6). With > String.intern() calls added in the ~10 relevant places in the code, this > overhead can be highly reduced. > 2. Almost 20% of memory is wasted due to various suboptimally used > collections (see section 8). There are many maps and lists that are either > empty or have just 1 element. By modifying the code that creates and > populates these collections, we may likely save 5-10% of memory. > 3. Almost 20% of memory is used by instances of java.util.Properties. It > looks like these objects are highly duplicate, since for each Partition each > concurrently running query creates its own copy of Partion, PartitionDesc and > Properties. Thus we have nearly 100,000 (50 queries * 2,000 partitions) > Properties in memory. By interning/deduplicating these objects we may be able > to save perhaps 15% of memory. > So overall, I think there is a good chance to reduce HS2 memory consumption > in this scenario by ~40%. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15882) HS2 generating high memory pressure with many partitions and concurrent queries
[ https://issues.apache.org/jira/browse/HIVE-15882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-15882: -- Attachment: hs2-crash-2000p-500m-50q.txt > HS2 generating high memory pressure with many partitions and concurrent > queries > --- > > Key: HIVE-15882 > URL: https://issues.apache.org/jira/browse/HIVE-15882 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Misha Dmitriev >Assignee: Misha Dmitriev > Attachments: hs2-crash-2000p-500m-50q.txt > > > I've created a Hive table with 2000 partitions, each backed by two files, > with one row in each file. When I execute some number of concurrent queries > against this table, e.g. as follows > {code} > for i in `seq 1 50`; do beeline -u jdbc:hive2://localhost:1 -n admin -p > admin -e "select count(i_f_1) from misha_table;" & done > {code} > it results in a big memory spike. With 20 queries I caused an OOM in a HS2 > server with -Xmx200m and with 50 queries - in the one with -Xmx500m. > I am attaching the results of jxray (www.jxray.com) analysis of a heap dump > that was generated in the 50queries/500m heap scenario. It suggests that > there are several opportunities to reduce memory pressure with not very > invasive changes to the code: > 1. 24.5% of memory is wasted by duplicate strings (see section 6). With > String.intern() calls added in the ~10 relevant places in the code, this > overhead can be highly reduced. > 2. Almost 20% of memory is wasted due to various suboptimally used > collections (see section 8). There are many maps and lists that are either > empty or have just 1 element. By modifying the code that creates and > populates these collections, we may likely save 5-10% of memory. > 3. Almost 20% of memory is used by instances of java.util.Properties. It > looks like these objects are highly duplicate, since for each Partition each > concurrently running query creates its own copy of Partion, PartitionDesc and > Properties. Thus we have nearly 100,000 (50 queries * 2,000 partitions) > Properties in memory. By interning/deduplicating these objects we may be able > to save perhaps 15% of memory. > So overall, I think there is a good chance to reduce HS2 memory consumption > in this scenario by ~40%. -- This message was sent by Atlassian JIRA (v6.3.15#6346)