[jira] [Work logged] (HIVE-25916) Optimise updateCompactionMetricsData
[ https://issues.apache.org/jira/browse/HIVE-25916?focusedWorklogId=720815=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-720815 ] ASF GitHub Bot logged work on HIVE-25916: - Author: ASF GitHub Bot Created on: 04/Feb/22 11:32 Start Date: 04/Feb/22 11:32 Worklog Time Spent: 10m Work Description: lcspinter merged pull request #2989: URL: https://github.com/apache/hive/pull/2989 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 720815) Time Spent: 40m (was: 0.5h) > Optimise updateCompactionMetricsData > > > Key: HIVE-25916 > URL: https://issues.apache.org/jira/browse/HIVE-25916 > Project: Hive > Issue Type: Improvement >Reporter: László Pintér >Assignee: László Pintér >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > TxnStore#updateCompactionMetricsData can be optimised to use the same DB > connection during create/update/delete if the delta metric threshold > evaluation is moved inside this method. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25916) Optimise updateCompactionMetricsData
[
https://issues.apache.org/jira/browse/HIVE-25916?focusedWorklogId=719291=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-719291
]
ASF GitHub Bot logged work on HIVE-25916:
-
Author: ASF GitHub Bot
Created on: 02/Feb/22 10:59
Start Date: 02/Feb/22 10:59
Worklog Time Spent: 10m
Work Description: lcspinter commented on a change in pull request #2989:
URL: https://github.com/apache/hive/pull/2989#discussion_r797489255
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
##
@@ -1584,38 +1584,17 @@ public boolean
updateCompactionMetricsData(CompactionMetricsData data) throws Me
try {
dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
boolean updateRes;
-CompactionMetricsData prevMetricsData = getCompactionMetricsData(data,
dbConn);
-if (prevMetricsData != null) {
- String query = UPDATE_COMPACTION_METRICS_CACHE_QUERY;
- if (data.getPartitionName() != null) {
-query += " AND \"CMC_PARTITION\" = ?";
+if (data.getMetricValue() >= data.getThreshold()) {
+ CompactionMetricsData prevMetricsData =
getCompactionMetricsData(data, dbConn);
+ if (prevMetricsData != null) {
+updateRes = updateCompactionMetricsData(dbConn, data,
prevMetricsData);
} else {
-query += " AND \"CMC_PARTITION\" IS NULL";
- }
- try (PreparedStatement pstmt = dbConn.prepareStatement(query)) {
-pstmt.setInt(1, data.getMetricValue());
-pstmt.setInt(2, prevMetricsData.getVersion() + 1);
-pstmt.setString(3, data.getDbName());
-pstmt.setString(4, data.getTblName());
-pstmt.setString(5, data.getMetricType().toString());
-pstmt.setInt(6, prevMetricsData.getVersion());
-if (data.getPartitionName() != null) {
- pstmt.setString(7, data.getPartitionName());
-}
-updateRes = pstmt.executeUpdate() > 0;
+updateRes = createCompactionMetricsData(dbConn, data);
}
} else {
- try (PreparedStatement pstmt =
dbConn.prepareStatement(INSERT_COMPACTION_METRICS_CACHE_QUERY)) {
-pstmt.setString(1, data.getDbName());
-pstmt.setString(2, data.getTblName());
-pstmt.setString(3, data.getPartitionName());
-pstmt.setString(4, data.getMetricType().toString());
-pstmt.setInt(5, data.getMetricValue());
-pstmt.setInt(6, 1);
-updateRes = pstmt.executeUpdate() > 0;
- }
+ updateRes = removeCompactionMetricsData(dbConn, data.getDbName(),
data.getTblName(), data.getPartitionName(),
Review comment:
Good point. I changed the logic to run remove in case the new metric is
below threshold and it is already present in the cache.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 719291)
Time Spent: 0.5h (was: 20m)
> Optimise updateCompactionMetricsData
>
>
> Key: HIVE-25916
> URL: https://issues.apache.org/jira/browse/HIVE-25916
> Project: Hive
> Issue Type: Improvement
>Reporter: László Pintér
>Assignee: László Pintér
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> TxnStore#updateCompactionMetricsData can be optimised to use the same DB
> connection during create/update/delete if the delta metric threshold
> evaluation is moved inside this method.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25916) Optimise updateCompactionMetricsData
[
https://issues.apache.org/jira/browse/HIVE-25916?focusedWorklogId=718073=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-718073
]
ASF GitHub Bot logged work on HIVE-25916:
-
Author: ASF GitHub Bot
Created on: 31/Jan/22 16:49
Start Date: 31/Jan/22 16:49
Worklog Time Spent: 10m
Work Description: klcopp commented on a change in pull request #2989:
URL: https://github.com/apache/hive/pull/2989#discussion_r795854302
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
##
@@ -1584,38 +1584,17 @@ public boolean
updateCompactionMetricsData(CompactionMetricsData data) throws Me
try {
dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
boolean updateRes;
-CompactionMetricsData prevMetricsData = getCompactionMetricsData(data,
dbConn);
-if (prevMetricsData != null) {
- String query = UPDATE_COMPACTION_METRICS_CACHE_QUERY;
- if (data.getPartitionName() != null) {
-query += " AND \"CMC_PARTITION\" = ?";
+if (data.getMetricValue() >= data.getThreshold()) {
+ CompactionMetricsData prevMetricsData =
getCompactionMetricsData(data, dbConn);
+ if (prevMetricsData != null) {
+updateRes = updateCompactionMetricsData(dbConn, data,
prevMetricsData);
} else {
-query += " AND \"CMC_PARTITION\" IS NULL";
- }
- try (PreparedStatement pstmt = dbConn.prepareStatement(query)) {
-pstmt.setInt(1, data.getMetricValue());
-pstmt.setInt(2, prevMetricsData.getVersion() + 1);
-pstmt.setString(3, data.getDbName());
-pstmt.setString(4, data.getTblName());
-pstmt.setString(5, data.getMetricType().toString());
-pstmt.setInt(6, prevMetricsData.getVersion());
-if (data.getPartitionName() != null) {
- pstmt.setString(7, data.getPartitionName());
-}
-updateRes = pstmt.executeUpdate() > 0;
+updateRes = createCompactionMetricsData(dbConn, data);
}
} else {
- try (PreparedStatement pstmt =
dbConn.prepareStatement(INSERT_COMPACTION_METRICS_CACHE_QUERY)) {
-pstmt.setString(1, data.getDbName());
-pstmt.setString(2, data.getTblName());
-pstmt.setString(3, data.getPartitionName());
-pstmt.setString(4, data.getMetricType().toString());
-pstmt.setInt(5, data.getMetricValue());
-pstmt.setInt(6, 1);
-updateRes = pstmt.executeUpdate() > 0;
- }
+ updateRes = removeCompactionMetricsData(dbConn, data.getDbName(),
data.getTblName(), data.getPartitionName(),
Review comment:
It's possible that:
- this table/partition is not in the "cache", because it was under the
threshold
- it remains under the threshold, so nothing is deleted from the "cache".
Then updateRes==false, which is not really what we want, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 718073)
Time Spent: 20m (was: 10m)
> Optimise updateCompactionMetricsData
>
>
> Key: HIVE-25916
> URL: https://issues.apache.org/jira/browse/HIVE-25916
> Project: Hive
> Issue Type: Improvement
>Reporter: László Pintér
>Assignee: László Pintér
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> TxnStore#updateCompactionMetricsData can be optimised to use the same DB
> connection during create/update/delete if the delta metric threshold
> evaluation is moved inside this method.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25916) Optimise updateCompactionMetricsData
[ https://issues.apache.org/jira/browse/HIVE-25916?focusedWorklogId=717865=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-717865 ] ASF GitHub Bot logged work on HIVE-25916: - Author: ASF GitHub Bot Created on: 31/Jan/22 12:36 Start Date: 31/Jan/22 12:36 Worklog Time Spent: 10m Work Description: lcspinter opened a new pull request #2989: URL: https://github.com/apache/hive/pull/2989 ### What changes were proposed in this pull request? TxnStore#updateCompactionMetricsData can be optimised to use the same DB connection during create/update/delete if the delta metric threshold evaluation is moved inside this method. ### Why are the changes needed? Performance optimisation ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Manual test, unit test -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 717865) Remaining Estimate: 0h Time Spent: 10m > Optimise updateCompactionMetricsData > > > Key: HIVE-25916 > URL: https://issues.apache.org/jira/browse/HIVE-25916 > Project: Hive > Issue Type: Improvement >Reporter: László Pintér >Assignee: László Pintér >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > TxnStore#updateCompactionMetricsData can be optimised to use the same DB > connection during create/update/delete if the delta metric threshold > evaluation is moved inside this method. -- This message was sent by Atlassian Jira (v8.20.1#820001)
