[jira] [Commented] (HIVE-15882) HS2 generating high memory pressure with many partitions and concurrent queries

2017-02-23 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881788#comment-15881788 ] Misha Dmitriev commented on HIVE-15882: --- [~lirui] sure - the RB for the first change (string

[jira] [Commented] (HIVE-15882) HS2 generating high memory pressure with many partitions and concurrent queries

2017-02-23 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881399#comment-15881399 ] Misha Dmitriev commented on HIVE-15882: --- I've just measured the CPU performance impact of my changes

[jira] [Commented] (HIVE-15882) HS2 generating high memory pressure with many partitions and concurrent queries

2017-02-23 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881244#comment-15881244 ] Misha Dmitriev commented on HIVE-15882: --- I've measured how much memory is saved with my change. It

[jira] [Commented] (HIVE-15882) HS2 generating high memory pressure with many partitions and concurrent queries

2017-02-22 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879349#comment-15879349 ] Misha Dmitriev commented on HIVE-15882: --- Yes, I did take a heap dump and rerun the tool after

[jira] [Updated] (HIVE-15882) HS2 generating high memory pressure with many partitions and concurrent queries

2017-02-24 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-15882: -- Attachment: HIVE-15882.02.patch Uploaded the second version of the patch, with comments made in

[jira] [Commented] (HIVE-15882) HS2 generating high memory pressure with many partitions and concurrent queries

2017-02-22 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879949#comment-15879949 ] Misha Dmitriev commented on HIVE-15882: --- Hi [~lirui], this is a legitimate concern. Regarding the

[jira] [Assigned] (HIVE-15882) HS2 generating high memory pressure with many partitions and concurrent queries

2017-02-10 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev reassigned HIVE-15882: - > HS2 generating high memory pressure with many partitions and concurrent > queries >

[jira] [Updated] (HIVE-15882) HS2 generating high memory pressure with many partitions and concurrent queries

2017-02-10 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-15882: -- Attachment: hs2-crash-2000p-500m-50q.txt > HS2 generating high memory pressure with many

[jira] [Work started] (HIVE-15882) HS2 generating high memory pressure with many partitions and concurrent queries

2017-02-14 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-15882 started by Misha Dmitriev. - > HS2 generating high memory pressure with many partitions and concurrent >

[jira] [Updated] (HIVE-15882) HS2 generating high memory pressure with many partitions and concurrent queries

2017-02-14 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-15882: -- Status: Patch Available (was: In Progress) The supplied patch de-dupes most of the duplicate

[jira] [Updated] (HIVE-15882) HS2 generating high memory pressure with many partitions and concurrent queries

2017-02-14 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-15882: -- Attachment: HIVE-15882.01.patch > HS2 generating high memory pressure with many partitions and

[jira] [Commented] (HIVE-15882) HS2 generating high memory pressure with many partitions and concurrent queries

2017-02-14 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866879#comment-15866879 ] Misha Dmitriev commented on HIVE-15882: --- For convenience, I've created a code review here:

[jira] [Commented] (HIVE-15882) HS2 generating high memory pressure with many partitions and concurrent queries

2017-02-15 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868357#comment-15868357 ] Misha Dmitriev commented on HIVE-15882: --- I've checked the failed tests. They either pass for me

[jira] [Commented] (HIVE-15882) HS2 generating high memory pressure with many partitions and concurrent queries

2017-02-28 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1541#comment-1541 ] Misha Dmitriev commented on HIVE-15882: --- [~lirui] thank you for your constructive feedback, I hope

[jira] [Updated] (HIVE-15882) HS2 generating high memory pressure with many partitions and concurrent queries

2017-02-28 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-15882: -- Attachment: HIVE-15882.04.patch Addressed the most recent Rui's comments. > HS2 generating

[jira] [Commented] (HIVE-16079) HS2: high memory pressure due to duplicate Properties objects

2017-03-02 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15891821#comment-15891821 ] Misha Dmitriev commented on HIVE-16079: --- Thank you for paying attention, [~prasanth_j] Actually, the

[jira] [Updated] (HIVE-15882) HS2 generating high memory pressure with many partitions and concurrent queries

2017-02-27 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-15882: -- Attachment: HIVE-15882.03.patch Attaching the 3rd version of the patch, addressing the recent

[jira] [Commented] (HIVE-15882) HS2 generating high memory pressure with many partitions and concurrent queries

2017-02-27 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886651#comment-15886651 ] Misha Dmitriev commented on HIVE-15882: --- The same tests fail in other builds, so they are not due to

[jira] [Updated] (HIVE-16079) HS2: high memory pressure due to duplicate Properties objects

2017-03-24 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-16079: -- Status: Open (was: Patch Available) > HS2: high memory pressure due to duplicate Properties

[jira] [Updated] (HIVE-16079) HS2: high memory pressure due to duplicate Properties objects

2017-03-24 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-16079: -- Status: Patch Available (was: Open) > HS2: high memory pressure due to duplicate Properties

[jira] [Commented] (HIVE-16166) HS2 may still waste up to 15% of memory on duplicate strings

2017-03-17 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931072#comment-15931072 ] Misha Dmitriev commented on HIVE-16166: --- [~spena] thank you very much for the logs. I found that the

[jira] [Commented] (HIVE-16166) HS2 may still waste up to 15% of memory on duplicate strings

2017-03-20 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15933583#comment-15933583 ] Misha Dmitriev commented on HIVE-16166: --- I ran 'mvn test -Dtest=TestMiniLlapLocalCliDriver

[jira] [Updated] (HIVE-16079) HS2: high memory pressure due to duplicate Properties objects

2017-03-15 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-16079: -- Status: Patch Available (was: Open) > HS2: high memory pressure due to duplicate Properties

[jira] [Updated] (HIVE-16166) HS2 may still waste up to 15% of memory on duplicate strings

2017-03-15 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-16166: -- Status: Patch Available (was: Open) > HS2 may still waste up to 15% of memory on duplicate

[jira] [Commented] (HIVE-16166) HS2 may still waste up to 15% of memory on duplicate strings

2017-03-15 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15926592#comment-15926592 ] Misha Dmitriev commented on HIVE-16166: --- Done. > HS2 may still waste up to 15% of memory on

[jira] [Updated] (HIVE-16166) HS2 may still waste up to 15% of memory on duplicate strings

2017-03-20 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-16166: -- Attachment: HIVE-16166.02.patch Fixed the problem with a List not providing the proper

[jira] [Commented] (HIVE-16166) HS2 may still waste up to 15% of memory on duplicate strings

2017-03-16 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928635#comment-15928635 ] Misha Dmitriev commented on HIVE-16166: --- @spena I ran all these tests locally, and they passed for

[jira] [Comment Edited] (HIVE-16166) HS2 may still waste up to 15% of memory on duplicate strings

2017-03-16 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928635#comment-15928635 ] Misha Dmitriev edited comment on HIVE-16166 at 3/16/17 6:53 PM: [~spena] I

[jira] [Commented] (HIVE-16166) HS2 may still waste up to 15% of memory on duplicate strings

2017-03-17 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929493#comment-15929493 ] Misha Dmitriev commented on HIVE-16166: --- Hm, the same tests failed again. Now it starts to look

[jira] [Commented] (HIVE-16079) HS2: high memory pressure due to duplicate Properties objects

2017-04-20 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977805#comment-15977805 ] Misha Dmitriev commented on HIVE-16079: --- It looks like testCliDriver[vector_if_expr] test is broken

[jira] [Updated] (HIVE-16489) HMS wastes 26.4% of memory due to dup strings in metastore.api.Partition.parameters

2017-04-20 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-16489: -- Component/s: (was: HiveServer2) Hive > HMS wastes 26.4% of memory due to

[jira] [Assigned] (HIVE-16489) HMS wastes 26.4% of memory due to dup strings in metastore.api.Partition.parameters

2017-04-20 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev reassigned HIVE-16489: - > HMS wastes 26.4% of memory due to dup strings in > metastore.api.Partition.parameters >

[jira] [Updated] (HIVE-16489) HMS wastes 26.4% of memory due to dup strings in metastore.api.Partition.parameters

2017-04-20 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-16489: -- Description: I've just analyzed an HMS heap dump. It turns out that it contains a lot of

[jira] [Updated] (HIVE-16079) HS2: high memory pressure due to duplicate Properties objects

2017-04-19 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-16079: -- Attachment: HIVE-16079.02.patch Attached the second version of the patch, with all JDK8 methods

[jira] [Updated] (HIVE-16079) HS2: high memory pressure due to duplicate Properties objects

2017-04-19 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-16079: -- Status: Patch Available (was: Open) > HS2: high memory pressure due to duplicate Properties

[jira] [Updated] (HIVE-16079) HS2: high memory pressure due to duplicate Properties objects

2017-04-19 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-16079: -- Status: Open (was: Patch Available) > HS2: high memory pressure due to duplicate Properties

[jira] [Updated] (HIVE-16079) HS2: high memory pressure due to duplicate Properties objects

2017-04-21 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-16079: -- Status: Open (was: Patch Available) > HS2: high memory pressure due to duplicate Properties

[jira] [Updated] (HIVE-16079) HS2: high memory pressure due to duplicate Properties objects

2017-04-21 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-16079: -- Attachment: HIVE-16079.03.patch > HS2: high memory pressure due to duplicate Properties objects

[jira] [Updated] (HIVE-16079) HS2: high memory pressure due to duplicate Properties objects

2017-04-21 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-16079: -- Status: Patch Available (was: Open) > HS2: high memory pressure due to duplicate Properties

[jira] [Commented] (HIVE-16079) HS2: high memory pressure due to duplicate Properties objects

2017-04-21 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15979322#comment-15979322 ] Misha Dmitriev commented on HIVE-16079: --- Uploaded the 3rd patch, that addresses Sergio's comments.

[jira] [Assigned] (HIVE-16166) HS2 may still waste up to 15% of memory on duplicate strings

2017-03-09 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev reassigned HIVE-16166: - > HS2 may still waste up to 15% of memory on duplicate strings >

[jira] [Updated] (HIVE-16166) HS2 may still waste up to 15% of memory on duplicate strings

2017-03-09 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-16166: -- Attachment: ch_2_excerpt.txt Results of jxray analysis for duplicate strings, baseline code. >

[jira] [Updated] (HIVE-16166) HS2 may still waste up to 15% of memory on duplicate strings

2017-03-09 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-16166: -- Attachment: HIVE-16166.01.patch > HS2 may still waste up to 15% of memory on duplicate strings

[jira] [Commented] (HIVE-16079) HS2: high memory pressure due to duplicate Properties objects

2017-03-06 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898533#comment-15898533 ] Misha Dmitriev commented on HIVE-16079: --- I've just created an RB:

[jira] [Updated] (HIVE-16079) HS2: high memory pressure due to duplicate Properties objects

2017-03-01 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-16079: -- Attachment: (was: HIVE-15882.04.patch) > HS2: high memory pressure due to duplicate

[jira] [Updated] (HIVE-16079) HS2: high memory pressure due to duplicate Properties objects

2017-03-01 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-16079: -- Attachment: (was: HIVE-15882.01.patch) > HS2: high memory pressure due to duplicate

[jira] [Assigned] (HIVE-16079) HS2: high memory pressure due to duplicate Properties objects

2017-03-01 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev reassigned HIVE-16079: - > HS2: high memory pressure due to duplicate Properties objects >

[jira] [Updated] (HIVE-16079) HS2: high memory pressure due to duplicate Properties objects

2017-03-01 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-16079: -- Description: I've created a Hive table with 2000 partitions, each backed by two files, with

[jira] [Updated] (HIVE-16079) HS2: high memory pressure due to duplicate Properties objects

2017-03-01 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-16079: -- Attachment: (was: HIVE-15882.02.patch) > HS2: high memory pressure due to duplicate

[jira] [Updated] (HIVE-16079) HS2: high memory pressure due to duplicate Properties objects

2017-03-01 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-16079: -- Attachment: (was: HIVE-15882.03.patch) > HS2: high memory pressure due to duplicate

[jira] [Updated] (HIVE-16079) HS2: high memory pressure due to duplicate Properties objects

2017-03-01 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-16079: -- Attachment: HIVE-16079.01.patch > HS2: high memory pressure due to duplicate Properties objects

[jira] [Commented] (HIVE-16079) HS2: high memory pressure due to duplicate Properties objects

2017-04-18 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15973751#comment-15973751 ] Misha Dmitriev commented on HIVE-16079: --- Hi [~ashutoshc], sorry for a delay with response - I've

[jira] [Commented] (HIVE-16079) HS2: high memory pressure due to duplicate Properties objects

2017-04-24 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981588#comment-15981588 ] Misha Dmitriev commented on HIVE-16079: --- vector_if_expr test fails in pretty much every Hive build.

[jira] [Assigned] (HIVE-17237) HMS wastes 26.4% of memory due to dup strings in metastore.api.Partition.parameters

2017-08-02 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-17237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev reassigned HIVE-17237: - > HMS wastes 26.4% of memory due to dup strings in > metastore.api.Partition.parameters >

[jira] [Updated] (HIVE-17237) HMS wastes 26.4% of memory due to dup strings in metastore.api.Partition.parameters

2017-08-02 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-17237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-17237: -- Attachment: HIVE-17237.01.patch > HMS wastes 26.4% of memory due to dup strings in >

[jira] [Updated] (HIVE-17237) HMS wastes 26.4% of memory due to dup strings in metastore.api.Partition.parameters

2017-08-02 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-17237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-17237: -- Status: Patch Available (was: Open) > HMS wastes 26.4% of memory due to dup strings in >

[jira] [Commented] (HIVE-16079) HS2: high memory pressure due to duplicate Properties objects

2017-04-27 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15987241#comment-15987241 ] Misha Dmitriev commented on HIVE-16079: --- Thank you [~spena]. Yes, this patch contains changes that

[jira] [Commented] (HIVE-17237) HMS wastes 26.4% of memory due to dup strings in metastore.api.Partition.parameters

2017-08-07 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-17237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16116963#comment-16116963 ] Misha Dmitriev commented on HIVE-17237: --- This is to save memory and improve performance.

[jira] [Updated] (HIVE-17237) HMS wastes 26.4% of memory due to dup strings in metastore.api.Partition.parameters

2017-08-21 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-17237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-17237: -- Status: In Progress (was: Patch Available) > HMS wastes 26.4% of memory due to dup strings in

[jira] [Updated] (HIVE-17237) HMS wastes 26.4% of memory due to dup strings in metastore.api.Partition.parameters

2017-08-21 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-17237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-17237: -- Attachment: HIVE-17237.02.patch > HMS wastes 26.4% of memory due to dup strings in >

[jira] [Updated] (HIVE-17237) HMS wastes 26.4% of memory due to dup strings in metastore.api.Partition.parameters

2017-08-21 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-17237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-17237: -- Status: Patch Available (was: In Progress) Just rebased the change and submitted the new

[jira] [Commented] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler

2017-11-15 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16254068#comment-16254068 ] Misha Dmitriev commented on HIVE-17684: --- The problem with {{MapJoinMemoryExhaustionHandler}} is a

[jira] [Work started] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler

2017-12-18 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-17684 started by Misha Dmitriev. - > HoS memory issues with MapJoinMemoryExhaustionHandler >

[jira] [Assigned] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler

2017-12-18 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev reassigned HIVE-17684: - Assignee: Misha Dmitriev (was: Sahil Takiar) > HoS memory issues with

[jira] [Updated] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler

2017-12-18 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-17684: -- Attachment: HIVE-17684.01.patch > HoS memory issues with MapJoinMemoryExhaustionHandler >

[jira] [Updated] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler

2017-12-18 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-17684: -- Status: Patch Available (was: In Progress) > HoS memory issues with

[jira] [Updated] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler

2017-12-19 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-17684: -- Status: Patch Available (was: In Progress) > HoS memory issues with

[jira] [Updated] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler

2017-12-19 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-17684: -- Attachment: HIVE-17684.02.patch > HoS memory issues with MapJoinMemoryExhaustionHandler >

[jira] [Updated] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler

2017-12-19 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-17684: -- Status: In Progress (was: Patch Available) > HoS memory issues with

[jira] [Commented] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler

2017-12-19 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297728#comment-16297728 ] Misha Dmitriev commented on HIVE-17684: --- I've fixed some checkstyle warnings (several others, e.g.

[jira] [Commented] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler

2017-12-18 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296040#comment-16296040 ] Misha Dmitriev commented on HIVE-17684: --- Thank you for taking a look, [~stakiar]. Yes, naturally

[jira] [Commented] (HIVE-19041) Thrift deserialization of Partition objects should intern fields

2018-05-09 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469733#comment-16469733 ] Misha Dmitriev commented on HIVE-19041: --- Agree - in our internal heap dump analysis, the aboveĀ 

[jira] [Commented] (HIVE-19041) Thrift deserialization of Partition objects should intern fields

2018-05-10 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16470954#comment-16470954 ] Misha Dmitriev commented on HIVE-19041: --- Thank you for looking into the details, [~vihangk1] I've

[jira] [Assigned] (HIVE-19668) 11.8% of the heap wasted due to duplicate org.antlr.runtime.CommonToken's

2018-05-22 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev reassigned HIVE-19668: - > 11.8% of the heap wasted due to duplicate org.antlr.runtime.CommonToken's >

[jira] [Updated] (HIVE-19668) Over 30% of the heap wasted by duplicate org.antlr.runtime.CommonToken's and duplicate strings

2018-06-02 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-19668: -- Attachment: HIVE-19668.01.patch > Over 30% of the heap wasted by duplicate

[jira] [Updated] (HIVE-19668) Over 30% of the heap wasted by duplicate org.antlr.runtime.CommonToken's and duplicate strings

2018-06-02 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-19668: -- Status: Patch Available (was: In Progress) > Over 30% of the heap wasted by duplicate

[jira] [Work started] (HIVE-19668) Over 30% of the heap wasted by duplicate org.antlr.runtime.CommonToken's and duplicate strings

2018-06-02 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-19668 started by Misha Dmitriev. - > Over 30% of the heap wasted by duplicate org.antlr.runtime.CommonToken's

[jira] [Updated] (HIVE-19668) 11.8% of the heap wasted due to duplicate org.antlr.runtime.CommonToken's

2018-06-01 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-19668: -- Description: I've recently analyzed a HS2 heap dump, obtained when there was a huge memory

[jira] [Updated] (HIVE-19668) Over 30% of the heap wasted by duplicate org.antlr.runtime.CommonToken's and duplicate strings

2018-06-01 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-19668: -- Summary: Over 30% of the heap wasted by duplicate org.antlr.runtime.CommonToken's and

[jira] [Updated] (HIVE-19668) Over 30% of the heap wasted by duplicate org.antlr.runtime.CommonToken's and duplicate strings

2018-06-27 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-19668: -- Status: In Progress (was: Patch Available) > Over 30% of the heap wasted by duplicate

[jira] [Updated] (HIVE-19668) Over 30% of the heap wasted by duplicate org.antlr.runtime.CommonToken's and duplicate strings

2018-06-27 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-19668: -- Status: Patch Available (was: In Progress) > Over 30% of the heap wasted by duplicate

[jira] [Updated] (HIVE-19668) Over 30% of the heap wasted by duplicate org.antlr.runtime.CommonToken's and duplicate strings

2018-06-27 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-19668: -- Attachment: HIVE-19668.02.patch > Over 30% of the heap wasted by duplicate

[jira] [Commented] (HIVE-19937) Intern JobConf objects in Spark tasks

2018-06-30 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16528876#comment-16528876 ] Misha Dmitriev commented on HIVE-19937: --- [~stakiar] regarding the behavior of

[jira] [Commented] (HIVE-19668) Over 30% of the heap wasted by duplicate org.antlr.runtime.CommonToken's and duplicate strings

2018-06-30 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16528883#comment-16528883 ] Misha Dmitriev commented on HIVE-19668: --- [~aihuaxu] I've checked the logs of failed tests, but

[jira] [Commented] (HIVE-19041) Thrift deserialization of Partition objects should intern fields

2018-05-03 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462886#comment-16462886 ] Misha Dmitriev commented on HIVE-19041: --- [~vihangk1] does the jxray report show that comments (or

[jira] [Commented] (HIVE-19041) Thrift deserialization of Partition objects should intern fields

2018-05-03 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463236#comment-16463236 ] Misha Dmitriev commented on HIVE-19041: --- [~gopalv] yes, since JDK 1.7 built-in string interning is

[jira] [Commented] (HIVE-19041) Thrift deserialization of Partition objects should intern fields

2018-05-03 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463235#comment-16463235 ] Misha Dmitriev commented on HIVE-19041: --- Yes, all interned strings are kept in the JVM internal

[jira] [Commented] (HIVE-19937) Intern JobConf objects in Spark tasks

2018-07-03 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16532187#comment-16532187 ] Misha Dmitriev commented on HIVE-19937: --- Thank you for sharing the jxray report, [~stakiar]. If it

[jira] [Resolved] (HIVE-16489) HMS wastes 26.4% of memory due to dup strings in metastore.api.Partition.parameters

2018-01-05 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev resolved HIVE-16489. --- Resolution: Duplicate > HMS wastes 26.4% of memory due to dup strings in >

[jira] [Commented] (HIVE-16489) HMS wastes 26.4% of memory due to dup strings in metastore.api.Partition.parameters

2018-01-05 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313767#comment-16313767 ] Misha Dmitriev commented on HIVE-16489: --- Hi [~szita] - apparently yes, just closed it. > HMS wastes

[jira] [Commented] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler

2018-01-09 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319247#comment-16319247 ] Misha Dmitriev commented on HIVE-17684: --- Hi [~stakiar], just a reminder that I cannot make further

[jira] [Commented] (HIVE-6430) MapJoin hash table has large memory overhead

2018-02-13 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-6430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363134#comment-16363134 ] Misha Dmitriev commented on HIVE-6430: -- Thank you [~akolb]! This is nice work of the kind I wish I can

[jira] [Commented] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler

2017-12-20 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299373#comment-16299373 ] Misha Dmitriev commented on HIVE-17684: --- [~stakiar] How do I run these {{TestSparkCliDriver}} tests?

[jira] [Commented] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler

2018-07-30 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16562445#comment-16562445 ] Misha Dmitriev commented on HIVE-17684: --- [~stakiar] a large number of CLI tests failed because of

[jira] [Commented] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler

2018-07-30 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16562481#comment-16562481 ] Misha Dmitriev commented on HIVE-17684: --- Actually, looks like some of these CLI tests fail for me

[jira] [Commented] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler

2018-07-31 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564366#comment-16564366 ] Misha Dmitriev commented on HIVE-17684: --- [~stakiar] tried to debug it, but not sure what's going

[jira] [Updated] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler

2018-07-27 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-17684: -- Status: Patch Available (was: In Progress) > HoS memory issues with

[jira] [Updated] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler

2018-07-27 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-17684: -- Attachment: HIVE-17684.04.patch > HoS memory issues with MapJoinMemoryExhaustionHandler >

[jira] [Updated] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler

2018-07-27 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated HIVE-17684: -- Status: In Progress (was: Patch Available) > HoS memory issues with

[jira] [Commented] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler

2018-08-14 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16580366#comment-16580366 ] Misha Dmitriev commented on HIVE-17684: --- I've resumed working on this, and finally found the reason

  1   2   >