[jira] [Work logged] (HIVE-24450) DbNotificationListener Request Notification IDs in Batches
[ https://issues.apache.org/jira/browse/HIVE-24450?focusedWorklogId=519033=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-519033 ] ASF GitHub Bot logged work on HIVE-24450: - Author: ASF GitHub Bot Created on: 02/Dec/20 14:20 Start Date: 02/Dec/20 14:20 Worklog Time Spent: 10m Work Description: belugabehr commented on pull request #1718: URL: https://github.com/apache/hive/pull/1718#issuecomment-737258804 Thank you all for the feedback. Please review my notes: https://issues.apache.org/jira/browse/HIVE-24450 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 519033) Time Spent: 50m (was: 40m) > DbNotificationListener Request Notification IDs in Batches > -- > > Key: HIVE-24450 > URL: https://issues.apache.org/jira/browse/HIVE-24450 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > Every time a new notification event is logged into the database, the sequence > number for the ID of the even is incremented by one. It is very standard in > database design to instead request a block of IDs for each fetch from the > database. The sequence numbers are then handed out locally until the block > of IDs is exhausted. This allows for fewer database round-trips and > transactions, at the expense of perhaps burning a few IDs. > Burning of IDs happens when the server is restarted in the middle of a block > of sequence IDs. That is, if the HMS requests a block of 10 ids, and only > three have been assigned, after the restart, the HMS will request another > block of 10, burning (wasting) 7 IDs. As long as the blocks are not too > small, and restarts are infrequent, then few IDs are lost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24450) DbNotificationListener Request Notification IDs in Batches
[ https://issues.apache.org/jira/browse/HIVE-24450?focusedWorklogId=519032=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-519032 ] ASF GitHub Bot logged work on HIVE-24450: - Author: ASF GitHub Bot Created on: 02/Dec/20 14:20 Start Date: 02/Dec/20 14:20 Worklog Time Spent: 10m Work Description: belugabehr closed pull request #1718: URL: https://github.com/apache/hive/pull/1718 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 519032) Time Spent: 40m (was: 0.5h) > DbNotificationListener Request Notification IDs in Batches > -- > > Key: HIVE-24450 > URL: https://issues.apache.org/jira/browse/HIVE-24450 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Every time a new notification event is logged into the database, the sequence > number for the ID of the even is incremented by one. It is very standard in > database design to instead request a block of IDs for each fetch from the > database. The sequence numbers are then handed out locally until the block > of IDs is exhausted. This allows for fewer database round-trips and > transactions, at the expense of perhaps burning a few IDs. > Burning of IDs happens when the server is restarted in the middle of a block > of sequence IDs. That is, if the HMS requests a block of 10 ids, and only > three have been assigned, after the restart, the HMS will request another > block of 10, burning (wasting) 7 IDs. As long as the blocks are not too > small, and restarts are infrequent, then few IDs are lost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24450) DbNotificationListener Request Notification IDs in Batches
[ https://issues.apache.org/jira/browse/HIVE-24450?focusedWorklogId=518891=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-518891 ] ASF GitHub Bot logged work on HIVE-24450: - Author: ASF GitHub Bot Created on: 02/Dec/20 09:19 Start Date: 02/Dec/20 09:19 Worklog Time Spent: 10m Work Description: aasha commented on pull request #1718: URL: https://github.com/apache/hive/pull/1718#issuecomment-737101280 In HA case how will the ordering of events be maintained? Acid Replication relies on the event sequence. So the ordering needs to be maintained. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 518891) Time Spent: 0.5h (was: 20m) > DbNotificationListener Request Notification IDs in Batches > -- > > Key: HIVE-24450 > URL: https://issues.apache.org/jira/browse/HIVE-24450 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Every time a new notification event is logged into the database, the sequence > number for the ID of the even is incremented by one. It is very standard in > database design to instead request a block of IDs for each fetch from the > database. The sequence numbers are then handed out locally until the block > of IDs is exhausted. This allows for fewer database round-trips and > transactions, at the expense of perhaps burning a few IDs. > Burning of IDs happens when the server is restarted in the middle of a block > of sequence IDs. That is, if the HMS requests a block of 10 ids, and only > three have been assigned, after the restart, the HMS will request another > block of 10, burning (wasting) 7 IDs. As long as the blocks are not too > small, and restarts are infrequent, then few IDs are lost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24450) DbNotificationListener Request Notification IDs in Batches
[ https://issues.apache.org/jira/browse/HIVE-24450?focusedWorklogId=518138=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-518138 ] ASF GitHub Bot logged work on HIVE-24450: - Author: ASF GitHub Bot Created on: 30/Nov/20 21:53 Start Date: 30/Nov/20 21:53 Worklog Time Spent: 10m Work Description: pvargacl commented on pull request #1718: URL: https://github.com/apache/hive/pull/1718#issuecomment-736078852 @belugabehr Although I am not very familiar with this area, but what happens if multiple HMS are running in HA? Wouldn't this solution mean, that potentially the order of the notification events will change? Two HMS are running HMS 1 gets id range 1-10, HMS 2 gets 11-20. Then openTxn notification goes to HMS 2 and allocateWriteId notification goes to HMS1. The sequence of the ids will not represent the sequence of the events. Wouldn't this mess up acid table replication? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 518138) Time Spent: 20m (was: 10m) > DbNotificationListener Request Notification IDs in Batches > -- > > Key: HIVE-24450 > URL: https://issues.apache.org/jira/browse/HIVE-24450 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Every time a new notification event is logged into the database, the sequence > number for the ID of the even is incremented by one. It is very standard in > database design to instead request a block of IDs for each fetch from the > database. The sequence numbers are then handed out locally until the block > of IDs is exhausted. This allows for fewer database round-trips and > transactions, at the expense of perhaps burning a few IDs. > Burning of IDs happens when the server is restarted in the middle of a block > of sequence IDs. That is, if the HMS requests a block of 10 ids, and only > three have been assigned, after the restart, the HMS will request another > block of 10, burning (wasting) 7 IDs. As long as the blocks are not too > small, and restarts are infrequent, then few IDs are lost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24450) DbNotificationListener Request Notification IDs in Batches
[ https://issues.apache.org/jira/browse/HIVE-24450?focusedWorklogId=518075=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-518075 ] ASF GitHub Bot logged work on HIVE-24450: - Author: ASF GitHub Bot Created on: 30/Nov/20 18:36 Start Date: 30/Nov/20 18:36 Worklog Time Spent: 10m Work Description: belugabehr opened a new pull request #1718: URL: https://github.com/apache/hive/pull/1718 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 518075) Remaining Estimate: 0h Time Spent: 10m > DbNotificationListener Request Notification IDs in Batches > -- > > Key: HIVE-24450 > URL: https://issues.apache.org/jira/browse/HIVE-24450 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Every time a new notification event is logged into the database, the sequence > number for the ID of the even is incremented by one. It is very standard in > database design to instead request a block of IDs for each fetch from the > database. The sequence numbers are then handed out locally until the block > of IDs is exhausted. This allows for fewer database round-trips and > transactions, at the expense of perhaps burning a few IDs. > Burning of IDs happens when the server is restarted in the middle of a block > of sequence IDs. That is, if the HMS requests a block of 10 ids, and only > three have been assigned, after the restart, the HMS will request another > block of 10, burning (wasting) 7 IDs. As long as the blocks are not too > small, and restarts are infrequent, then few IDs are lost. -- This message was sent by Atlassian Jira (v8.3.4#803005)