[jira] [Updated] (HIVE-13410) PerfLog metrics scopes not closed if there are exceptions on HS2
[ https://issues.apache.org/jira/browse/HIVE-13410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-13410: - Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Committed to master, thanks Aihua for review! > PerfLog metrics scopes not closed if there are exceptions on HS2 > > > Key: HIVE-13410 > URL: https://issues.apache.org/jira/browse/HIVE-13410 > Project: Hive > Issue Type: Bug > Components: Diagnosability >Affects Versions: 2.0.0 >Reporter: Szehon Ho >Assignee: Szehon Ho > Fix For: 2.1.0 > > Attachments: HIVE-13410.2.patch, HIVE-13410.3.patch, > HIVE-13410.4.patch, HIVE-13410.4.patch, HIVE-13410.5.patch, HIVE-13410.patch > > > If there are errors, the HS2 PerfLog api scopes are not closed. Then there > are sometimes messages like 'java.io.IOException: Scope named api_parse is > not closed, cannot be opened.' > I had simply forgetting to close the dangling scopes if there is an > exception. Doing so now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13410) PerfLog metrics scopes not closed if there are exceptions on HS2
[ https://issues.apache.org/jira/browse/HIVE-13410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-13410: - Attachment: HIVE-13410.5.patch Rebase the patch > PerfLog metrics scopes not closed if there are exceptions on HS2 > > > Key: HIVE-13410 > URL: https://issues.apache.org/jira/browse/HIVE-13410 > Project: Hive > Issue Type: Bug > Components: Diagnosability >Affects Versions: 2.0.0 >Reporter: Szehon Ho >Assignee: Szehon Ho > Attachments: HIVE-13410.2.patch, HIVE-13410.3.patch, > HIVE-13410.4.patch, HIVE-13410.4.patch, HIVE-13410.5.patch, HIVE-13410.patch > > > If there are errors, the HS2 PerfLog api scopes are not closed. Then there > are sometimes messages like 'java.io.IOException: Scope named api_parse is > not closed, cannot be opened.' > I had simply forgetting to close the dangling scopes if there is an > exception. Doing so now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13410) PerfLog metrics scopes not closed if there are exceptions on HS2
[ https://issues.apache.org/jira/browse/HIVE-13410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-13410: - Attachment: HIVE-13410.4.patch Not sure why precommit did not pick this up. > PerfLog metrics scopes not closed if there are exceptions on HS2 > > > Key: HIVE-13410 > URL: https://issues.apache.org/jira/browse/HIVE-13410 > Project: Hive > Issue Type: Bug > Components: Diagnosability >Affects Versions: 2.0.0 >Reporter: Szehon Ho >Assignee: Szehon Ho > Attachments: HIVE-13410.2.patch, HIVE-13410.3.patch, > HIVE-13410.4.patch, HIVE-13410.4.patch, HIVE-13410.patch > > > If there are errors, the HS2 PerfLog api scopes are not closed. Then there > are sometimes messages like 'java.io.IOException: Scope named api_parse is > not closed, cannot be opened.' > I had simply forgetting to close the dangling scopes if there is an > exception. Doing so now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13410) PerfLog metrics scopes not closed if there are exceptions on HS2
[ https://issues.apache.org/jira/browse/HIVE-13410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-13410: - Attachment: (was: HIVE-13410.4.patch) > PerfLog metrics scopes not closed if there are exceptions on HS2 > > > Key: HIVE-13410 > URL: https://issues.apache.org/jira/browse/HIVE-13410 > Project: Hive > Issue Type: Bug > Components: Diagnosability >Affects Versions: 2.0.0 >Reporter: Szehon Ho >Assignee: Szehon Ho > Attachments: HIVE-13410.2.patch, HIVE-13410.3.patch, > HIVE-13410.4.patch, HIVE-13410.patch > > > If there are errors, the HS2 PerfLog api scopes are not closed. Then there > are sometimes messages like 'java.io.IOException: Scope named api_parse is > not closed, cannot be opened.' > I had simply forgetting to close the dangling scopes if there is an > exception. Doing so now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13410) PerfLog metrics scopes not closed if there are exceptions on HS2
[ https://issues.apache.org/jira/browse/HIVE-13410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-13410: - Attachment: HIVE-13410.4.patch > PerfLog metrics scopes not closed if there are exceptions on HS2 > > > Key: HIVE-13410 > URL: https://issues.apache.org/jira/browse/HIVE-13410 > Project: Hive > Issue Type: Bug > Components: Diagnosability >Affects Versions: 2.0.0 >Reporter: Szehon Ho >Assignee: Szehon Ho > Attachments: HIVE-13410.2.patch, HIVE-13410.3.patch, > HIVE-13410.4.patch, HIVE-13410.patch > > > If there are errors, the HS2 PerfLog api scopes are not closed. Then there > are sometimes messages like 'java.io.IOException: Scope named api_parse is > not closed, cannot be opened.' > I had simply forgetting to close the dangling scopes if there is an > exception. Doing so now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13410) PerfLog metrics scopes not closed if there are exceptions on HS2
[ https://issues.apache.org/jira/browse/HIVE-13410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-13410: - Attachment: HIVE-13410.4.patch A lot of test failures may not be related. But one of them (TestMetaStoreMetrics) is perhaps happening because its running in the same JVM as the other Metrics test, and thus the Metrics singleton is not in a clean state. Explicitly reset it in the beginning of that test, and adjust the counts in the test accordingly (if run independently, it was initialized in another place before, which is during startMetastore). > PerfLog metrics scopes not closed if there are exceptions on HS2 > > > Key: HIVE-13410 > URL: https://issues.apache.org/jira/browse/HIVE-13410 > Project: Hive > Issue Type: Bug > Components: Diagnosability >Affects Versions: 2.0.0 >Reporter: Szehon Ho >Assignee: Szehon Ho > Attachments: HIVE-13410.2.patch, HIVE-13410.3.patch, > HIVE-13410.4.patch, HIVE-13410.patch > > > If there are errors, the HS2 PerfLog api scopes are not closed. Then there > are sometimes messages like 'java.io.IOException: Scope named api_parse is > not closed, cannot be opened.' > I had simply forgetting to close the dangling scopes if there is an > exception. Doing so now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13410) PerfLog metrics scopes not closed if there are exceptions on HS2
[ https://issues.apache.org/jira/browse/HIVE-13410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-13410: - Attachment: HIVE-13410.patch Good point! Today I could not find any perfLogging happening during parallel task execution phase but it is very possible in future. The perf-logger is thread-local, made a fix so that each perf-logger keeps track of its own scope and its lifecycle. > PerfLog metrics scopes not closed if there are exceptions on HS2 > > > Key: HIVE-13410 > URL: https://issues.apache.org/jira/browse/HIVE-13410 > Project: Hive > Issue Type: Bug > Components: Diagnosability >Affects Versions: 2.0.0 >Reporter: Szehon Ho >Assignee: Szehon Ho > Attachments: HIVE-13410.2.patch, HIVE-13410.3.patch, HIVE-13410.patch > > > If there are errors, the HS2 PerfLog api scopes are not closed. Then there > are sometimes messages like 'java.io.IOException: Scope named api_parse is > not closed, cannot be opened.' > I had simply forgetting to close the dangling scopes if there is an > exception. Doing so now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13410) PerfLog metrics scopes not closed if there are exceptions on HS2
[ https://issues.apache.org/jira/browse/HIVE-13410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-13410: - Attachment: HIVE-13410.2.patch Thanks a lot for taking a look! Addressed review comments. For the first questions, I do check if the scope is closed via 'active_calls' which is incremented when scope is open and decremented when scope is closed. I added comments in the test to try to make that clearer. For second question, the perf logger is reset at the beginning of every query (Driver.runInternal) so I think its lifecycle is more tied to a query rather than session, hence I reset the metrics on it as part of the return of that query. > PerfLog metrics scopes not closed if there are exceptions on HS2 > > > Key: HIVE-13410 > URL: https://issues.apache.org/jira/browse/HIVE-13410 > Project: Hive > Issue Type: Bug > Components: Diagnosability >Affects Versions: 2.0.0 >Reporter: Szehon Ho >Assignee: Szehon Ho > Attachments: HIVE-13410.2.patch, HIVE-13410.patch > > > If there are errors, the HS2 PerfLog api scopes are not closed. Then there > are sometimes messages like 'java.io.IOException: Scope named api_parse is > not closed, cannot be opened.' > I had simply forgetting to close the dangling scopes if there is an > exception. Doing so now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13410) PerfLog metrics scopes not closed if there are exceptions on HS2
[ https://issues.apache.org/jira/browse/HIVE-13410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-13410: - Assignee: Szehon Ho Status: Patch Available (was: Open) [~aihuaxu] would you mind taking a look at this simple patch? Thanks > PerfLog metrics scopes not closed if there are exceptions on HS2 > > > Key: HIVE-13410 > URL: https://issues.apache.org/jira/browse/HIVE-13410 > Project: Hive > Issue Type: Bug > Components: Diagnosability >Affects Versions: 2.0.0 >Reporter: Szehon Ho >Assignee: Szehon Ho > Attachments: HIVE-13410.patch > > > If there are errors, the HS2 PerfLog api scopes are not closed. Then there > are sometimes messages like 'java.io.IOException: Scope named api_parse is > not closed, cannot be opened.' > I had simply forgetting to close the dangling scopes if there is an > exception. Doing so now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13410) PerfLog metrics scopes not closed if there are exceptions on HS2
[ https://issues.apache.org/jira/browse/HIVE-13410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-13410: - Attachment: HIVE-13410.patch Fixing this issue, and moving the metrics back out from LOG.debug section. Seems it had been moved inside in some other change. > PerfLog metrics scopes not closed if there are exceptions on HS2 > > > Key: HIVE-13410 > URL: https://issues.apache.org/jira/browse/HIVE-13410 > Project: Hive > Issue Type: Bug > Components: Diagnosability >Affects Versions: 2.0.0 >Reporter: Szehon Ho > Attachments: HIVE-13410.patch > > > If there are errors, the HS2 PerfLog api scopes are not closed. Then there > are sometimes messages like 'java.io.IOException: Scope named api_parse is > not closed, cannot be opened.' > I had simply forgetting to close the dangling scopes if there is an > exception. Doing so now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
