[jira] [Comment Edited] (HBASE-14070) Hybrid Logical Clocks for HBase
[ https://issues.apache.org/jira/browse/HBASE-14070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16142478#comment-16142478 ] Amit Patel edited comment on HBASE-14070 at 8/26/17 12:15 AM: -- Just as a status update, I'll be finishing my internship so I will be much less active in the future (I still hope to maintain some degree of involvement with the work). I am extremely grateful for the help and mentorship that I have received from the community. At the moment there are still a few outstanding issues that should be addressed before merging the work into master: * Performance of HLC (Clock#now, Clock#update are expensive, not as big of an issue if HLC is only used for meta but certainly is for user tables) * Huge number of tests timing out in in the pre-commit/[public branch|https://builds.apache.org/job/HBASE-14070.HLC/] builds (but haven't been able to replicate the test time-out and they pass just fine if ran locally) In terms of enabling HLC on user tables, remaining issues would include: * Some tests that are still explicitly setting the timestamp and need to be converted to instead manipulate the current time via mocking the clocks * Bulk load does not update the HLC Some additional brainstorming/discussions includes: * [Deprecate setting of timestamp in client for HLC|https://issues.apache.org/jira/browse/HBASE-18642] * [Add 'Transaction ID' to Result for HLC|https://issues.apache.org/jira/browse/HBASE-18643] was (Author: amit.patel): Just as a status update, I'll be finishing my internship so I will be much less active in the future. I am extremely grateful for the help and support that I have received from the community. At the moment there are still a few outstanding issues that should be addressed before merging the work into master: * Performance of HLC (Clock#now, Clock#update are expensive, not as big of an issue if HLC is only used for meta but certainly is for user tables) * Huge number of tests timing out in in the pre-commit/[public branch|https://builds.apache.org/job/HBASE-14070.HLC/] builds (but haven't been able to replicate the test time-out and they pass just fine if ran locally) In terms of enabling HLC on user tables, remaining issues would include: * Some tests that are still explicitly setting the timestamp and need to be converted to instead manipulate the current time via mocking the clocks * Bulk load does not update the HLC Some additional brainstorming/discussions includes: * [Deprecate setting of timestamp in client for HLC|https://issues.apache.org/jira/browse/HBASE-18642] * [Add 'Transaction ID' to Result for HLC|https://issues.apache.org/jira/browse/HBASE-18643] > Hybrid Logical Clocks for HBase > --- > > Key: HBASE-14070 > URL: https://issues.apache.org/jira/browse/HBASE-14070 > Project: HBase > Issue Type: New Feature >Reporter: Enis Soztutar >Assignee: Amit Patel > Attachments: HBASE-14070.master.001.patch, > HybridLogicalClocksforHBaseandPhoenix.docx, > HybridLogicalClocksforHBaseandPhoenix.pdf > > > HBase and Phoenix uses systems physical clock (PT) to give timestamps to > events (read and writes). This works mostly when the system clock is strictly > monotonically increasing and there is no cross-dependency between servers > clocks. However we know that leap seconds, general clock skew and clock drift > are in fact real. > This jira proposes using Hybrid Logical Clocks (HLC) as an implementation of > hybrid physical clock + a logical clock. HLC is best of both worlds where it > keeps causality relationship similar to logical clocks, but still is > compatible with NTP based physical system clock. HLC can be represented in > 64bits. > A design document is attached and also can be found here: > https://docs.google.com/document/d/1LL2GAodiYi0waBz5ODGL4LDT4e_bXy8P9h6kWC05Bhw/edit# -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HBASE-14070) Hybrid Logical Clocks for HBase
[
https://issues.apache.org/jira/browse/HBASE-14070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16097089#comment-16097089
]
Appy edited comment on HBASE-14070 at 7/22/17 3:19 AM:
---
I think a simple solution here is, keep track of skew in clock. And instead of
keeping track of physical time, always compute it by ST + skew.
On update(), recalculate {{skew}} and validate if it's greater than max_skew.
On toTimestamp(), calculate PT = ST+skew.
The biggest advantage is, PT will keep moving forward. It'll fix most problems
mentioned earlier which arise because PT gets stuck.
I'll make a patch to show what i mean.
Here's the jira which has the patch:
https://issues.apache.org/jira/browse/HBASE-18432
was (Author: appy):
I think a simple solution here is, keep track of skew in clock. And instead of
keeping track of physical time, always compute it by ST + skew.
On update(), recalculate {{skew}} and validate if it's greater than max_skew.
On toTimestamp(), calculate PT = ST+skew.
The biggest advantage is, PT will keep moving forward. It'll fix most problems
mentioned earlier which arise because PT gets stuck.
I'll make a patch to show what i mean.
> Hybrid Logical Clocks for HBase
> ---
>
> Key: HBASE-14070
> URL: https://issues.apache.org/jira/browse/HBASE-14070
> Project: HBase
> Issue Type: New Feature
>Reporter: Enis Soztutar
>Assignee: Amit Patel
> Attachments: HBASE-14070.master.001.patch,
> HybridLogicalClocksforHBaseandPhoenix.docx,
> HybridLogicalClocksforHBaseandPhoenix.pdf
>
>
> HBase and Phoenix uses systems physical clock (PT) to give timestamps to
> events (read and writes). This works mostly when the system clock is strictly
> monotonically increasing and there is no cross-dependency between servers
> clocks. However we know that leap seconds, general clock skew and clock drift
> are in fact real.
> This jira proposes using Hybrid Logical Clocks (HLC) as an implementation of
> hybrid physical clock + a logical clock. HLC is best of both worlds where it
> keeps causality relationship similar to logical clocks, but still is
> compatible with NTP based physical system clock. HLC can be represented in
> 64bits.
> A design document is attached and also can be found here:
> https://docs.google.com/document/d/1LL2GAodiYi0waBz5ODGL4LDT4e_bXy8P9h6kWC05Bhw/edit#
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HBASE-14070) Hybrid Logical Clocks for HBase
[ https://issues.apache.org/jira/browse/HBASE-14070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16094013#comment-16094013 ] Appy edited comment on HBASE-14070 at 7/20/17 12:44 AM: While reviewing one of the patches, a question popped in my mind: We are passing master clock to RS in open/close region request and RS clock to master in the responses. And they both update their own time on receiving these request/response. (x= don't care) Say RS hlc time is (10,x) and master time is (20,x). Master sends request to RS which then updates its own time to (20,x). While RS's physical java clock (which is backing up physical component of hlc clock) will still take 10 sec to catch up, we'll keep incrementing logical component. That means, in worst case, our logical clock window should be big enough to support all the events that can happen in max skew time. The problem is, that doesn't seem to be the case. Our logical window is 1M events (20bits) and max skew time is 30 sec, that results in 33k max write qps, which is quite low. We can easily see 150k update qps per beefy server with 1k values. Even 22 bits won't be enough. We'll need minimum of 23 bits and 20 sec max skew time to support ~420k max events per second in worst case clock skew. was (Author: appy): While reviewing one of the patches, a question popped in my mind: We are passing master clock to RS in open/close region request and RS clock to master in the responses. And they both update their own time on receiving these request/response. (x= don't care) Say RS hlc time is (10,x) and master time is (20,x). Master sends request to RS which then updates its own time to (20,x). While RS's physical java clock (which is backing up physical component of hlc clock) will still take 10 sec to catch up, we'll keep incrementing logical component. That means, in worst case, our logical clock window should be big enough to support all the events that can happen in max skew time. The problem is, that doesn't seem to be the case. Our logical window is 1M events (20bits) and max skew time is 30 sec, that results in 33k max write qps, which is quite low. We can easily see 150k update qps per beefy server with 1k values. Even 22 bits won't be enough. We'll need minimum of 23 bits and 20 sec max skew time to get to ~420k events per second (worst case). > Hybrid Logical Clocks for HBase > --- > > Key: HBASE-14070 > URL: https://issues.apache.org/jira/browse/HBASE-14070 > Project: HBase > Issue Type: New Feature >Reporter: Enis Soztutar >Assignee: Amit Patel > Attachments: HBASE-14070.master.001.patch, > HybridLogicalClocksforHBaseandPhoenix.docx, > HybridLogicalClocksforHBaseandPhoenix.pdf > > > HBase and Phoenix uses systems physical clock (PT) to give timestamps to > events (read and writes). This works mostly when the system clock is strictly > monotonically increasing and there is no cross-dependency between servers > clocks. However we know that leap seconds, general clock skew and clock drift > are in fact real. > This jira proposes using Hybrid Logical Clocks (HLC) as an implementation of > hybrid physical clock + a logical clock. HLC is best of both worlds where it > keeps causality relationship similar to logical clocks, but still is > compatible with NTP based physical system clock. HLC can be represented in > 64bits. > A design document is attached and also can be found here: > https://docs.google.com/document/d/1LL2GAodiYi0waBz5ODGL4LDT4e_bXy8P9h6kWC05Bhw/edit# -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HBASE-14070) Hybrid Logical Clocks for HBase
[ https://issues.apache.org/jira/browse/HBASE-14070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15478593#comment-15478593 ] stack edited comment on HBASE-14070 at 9/9/16 11:56 PM: The patch - HBase-14070.master.001.patch contains the current unfinished changes for the HLC project. A document describing current state of the work and work yet to be done is attached in the HBASE-14070. It is present in the links section of this JIRA, it is named Current Status of HLC. h2. Work Done Timestamp Enum Class with all the methods with the tests Clock Class supporting three clock types with the tests HLC clock updates during recovery and replication Clock per Region Server notion TTL and Time to Purge work well across all Clock Types Most of the time related tests are parameterized to run against all the three clock types. Test cases setting timestamps in Put were changed so that timestamp is not set with the help of manual environment edge The tests were run with HLC as default clock type of the tables and most of the bugs have been removed. There is still some work to be done to get all the tests passed. h2. Work Yet to done Time range should be dealt with as per section 3.3 A check, for HLC and System Monotonic tables, disallowing clients to set the timestamps in the server side and not on the client side (Keep Client dumb) Currently, still some test cases are flaky, some are failing. Need to clear them. HLC clock should be updates for more events such open, close regions etc. as per various use cases. Deciding which events we should track is important. Bulk Loads need to be thought upon. Mechanism to ensure that the bulk loaded files are having correct timestamp types. If we can have the highest timestamp of all the cells, we can update the local clock with it. This is something to think about. Test cases written currently are very local, need to write some integration tests which tests the HLC clock properties w.r.t recovery, replication and more. Currently some of the tests were rewritten using manual environment edge to get around the problem of not setting timestamps for HLC tables. A suggestion was made by Stack and Enis to not use environment edge in the clocks. Instead have a pluggable manual clock in place of Environment Edge. was (Author: saitejar): The patch - HBase-14070.master.001.patch contains the current unfinished changes for the HLC project. A document describing current state of the work and work yet to be done is attached in the HBASE-14070. It is present in the links section of this JIRA, it is named Current Status of HLC. Work Done: Timestamp Enum Class with all the methods with the tests Clock Class supporting three clock types with the tests HLC clock updates during recovery and replication Clock per Region Server notion TTL and Time to Purge work well across all Clock Types Most of the time related tests are parameterized to run against all the three clock types. Test cases setting timestamps in Put were changed so that timestamp is not set with the help of manual environment edge The tests were run with HLC as default clock type of the tables and most of the bugs have been removed. There is still some work to be done to get all the tests passed. Work Yet to done: Time range should be dealt with as per section 3.3 A check, for HLC and System Monotonic tables, disallowing clients to set the timestamps in the server side and not on the client side (Keep Client dumb) Currently, still some test cases are flaky, some are failing. Need to clear them. HLC clock should be updates for more events such open, close regions etc. as per various use cases. Deciding which events we should track is important. Bulk Loads need to be thought upon. Mechanism to ensure that the bulk loaded files are having correct timestamp types. If we can have the highest timestamp of all the cells, we can update the local clock with it. This is something to think about. Test cases written currently are very local, need to write some integration tests which tests the HLC clock properties w.r.t recovery, replication and more. Currently some of the tests were rewritten using manual environment edge to get around the problem of not setting timestamps for HLC tables. A suggestion was made by Stack and Enis to not use environment edge in the clocks. Instead have a pluggable manual clock in place of Environment Edge. > Hybrid Logical Clocks for HBase > --- > > Key: HBASE-14070 > URL: https://issues.apache.org/jira/browse/HBASE-14070 > Project: HBase > Issue Type: New Feature >Reporter: Enis Soztutar >Assignee: Sai Teja Ranuva > Attachments: HBASE-14070.master.001.patch, > HybridLogicalClocksforHBaseandPhoenix.docx, > HybridLogicalClocksforHBaseandPhoenix.pdf > > > HBase and
[jira] [Comment Edited] (HBASE-14070) Hybrid Logical Clocks for HBase
[ https://issues.apache.org/jira/browse/HBASE-14070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15478593#comment-15478593 ] Sai Teja Ranuva edited comment on HBASE-14070 at 9/9/16 11:53 PM: -- The patch - HBase-14070.master.001.patch contains the current unfinished changes for the HLC project. A document describing current state of the work and work yet to be done is attached in the HBASE-14070. It is present in the links section of this JIRA, it is named Current Status of HLC. Work Done: Timestamp Enum Class with all the methods with the tests Clock Class supporting three clock types with the tests HLC clock updates during recovery and replication Clock per Region Server notion TTL and Time to Purge work well across all Clock Types Most of the time related tests are parameterized to run against all the three clock types. Test cases setting timestamps in Put were changed so that timestamp is not set with the help of manual environment edge The tests were run with HLC as default clock type of the tables and most of the bugs have been removed. There is still some work to be done to get all the tests passed. Work Yet to done: Time range should be dealt with as per section 3.3 A check, for HLC and System Monotonic tables, disallowing clients to set the timestamps in the server side and not on the client side (Keep Client dumb) Currently, still some test cases are flaky, some are failing. Need to clear them. HLC clock should be updates for more events such open, close regions etc. as per various use cases. Deciding which events we should track is important. Bulk Loads need to be thought upon. Mechanism to ensure that the bulk loaded files are having correct timestamp types. If we can have the highest timestamp of all the cells, we can update the local clock with it. This is something to think about. Test cases written currently are very local, need to write some integration tests which tests the HLC clock properties w.r.t recovery, replication and more. Currently some of the tests were rewritten using manual environment edge to get around the problem of not setting timestamps for HLC tables. A suggestion was made by Stack and Enis to not use environment edge in the clocks. Instead have a pluggable manual clock in place of Environment Edge. was (Author: saitejar): The patch - HBase-14070.master.001.patch contains the current unfinished changes for the HLC project. A document describing current state of the work and work yet to be done is attached in the HBASE-14070. To summarise it: Work Done: Timestamp Enum Class with all the methods with the tests Clock Class supporting three clock types with the tests HLC clock updates during recovery and replication Clock per Region Server notion TTL and Time to Purge work well across all Clock Types Most of the time related tests are parameterized to run against all the three clock types. Test cases setting timestamps in Put were changed so that timestamp is not set with the help of manual environment edge The tests were run with HLC as default clock type of the tables and most of the bugs have been removed. There is still some work to be done to get all the tests passed. Work Yet to done: Time range should be dealt with as per section 3.3 A check, for HLC and System Monotonic tables, disallowing clients to set the timestamps in the server side and not on the client side (Keep Client dumb) Currently, still some test cases are flaky, some are failing. Need to clear them. HLC clock should be updates for more events such open, close regions etc. as per various use cases. Deciding which events we should track is important. Bulk Loads need to be thought upon. Mechanism to ensure that the bulk loaded files are having correct timestamp types. If we can have the highest timestamp of all the cells, we can update the local clock with it. This is something to think about. Test cases written currently are very local, need to write some integration tests which tests the HLC clock properties w.r.t recovery, replication and more. Currently some of the tests were rewritten using manual environment edge to get around the problem of not setting timestamps for HLC tables. A suggestion was made by Stack and Enis to not use environment edge in the clocks. Instead have a pluggable manual clock in place of Environment Edge. > Hybrid Logical Clocks for HBase > --- > > Key: HBASE-14070 > URL: https://issues.apache.org/jira/browse/HBASE-14070 > Project: HBase > Issue Type: New Feature >Reporter: Enis Soztutar >Assignee: Sai Teja Ranuva > Attachments: HBASE-14070.master.001.patch, > HybridLogicalClocksforHBaseandPhoenix.docx, > HybridLogicalClocksforHBaseandPhoenix.pdf > > > HBase and Phoenix uses systems physical clock (PT) to give timestamps
[jira] [Comment Edited] (HBASE-14070) Hybrid Logical Clocks for HBase
[ https://issues.apache.org/jira/browse/HBASE-14070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15478493#comment-15478493 ] Sai Teja Ranuva edited comment on HBASE-14070 at 9/9/16 11:13 PM: -- The document(Current Status of HLC - google doc) contains the current status of the project with some details of the decisions made in the process. It also contains the work that is yet to be done. Open to comments. was (Author: saitejar): This document contains the current status of the project with some details of the decisions made in the process. It also contains the work that is yet to be done. > Hybrid Logical Clocks for HBase > --- > > Key: HBASE-14070 > URL: https://issues.apache.org/jira/browse/HBASE-14070 > Project: HBase > Issue Type: New Feature >Reporter: Enis Soztutar >Assignee: Sai Teja Ranuva > Attachments: HBASE-14070.master.001.patch, > HybridLogicalClocksforHBaseandPhoenix.docx, > HybridLogicalClocksforHBaseandPhoenix.pdf > > > HBase and Phoenix uses systems physical clock (PT) to give timestamps to > events (read and writes). This works mostly when the system clock is strictly > monotonically increasing and there is no cross-dependency between servers > clocks. However we know that leap seconds, general clock skew and clock drift > are in fact real. > This jira proposes using Hybrid Logical Clocks (HLC) as an implementation of > hybrid physical clock + a logical clock. HLC is best of both worlds where it > keeps causality relationship similar to logical clocks, but still is > compatible with NTP based physical system clock. HLC can be represented in > 64bits. > A design document is attached and also can be found here: > https://docs.google.com/document/d/1LL2GAodiYi0waBz5ODGL4LDT4e_bXy8P9h6kWC05Bhw/edit# -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-14070) Hybrid Logical Clocks for HBase
[ https://issues.apache.org/jira/browse/HBASE-14070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15384889#comment-15384889 ] Sai Teja Ranuva edited comment on HBASE-14070 at 7/19/16 9:42 PM: -- [~enis] [~stack] Is there a reason why we are introducing System Monotonic clock apart from HLC clock ? Do we need it anywhere specifically ? Can we not use HLC in places where we might use System Monotonic ? I feel it might be better to have clock implementations that will be used. was (Author: saitejar): [~enis] [~stack] Is there a reason why we are introducing System Monotonic clock apart from HLC clock ? Do we need it anywhere specifically ? Can we not use HLC in places where we might System Monotonic ? I feel it might be better to have clock implementations that will be used. > Hybrid Logical Clocks for HBase > --- > > Key: HBASE-14070 > URL: https://issues.apache.org/jira/browse/HBASE-14070 > Project: HBase > Issue Type: New Feature >Reporter: Enis Soztutar >Assignee: Sai Teja Ranuva > Attachments: HybridLogicalClocksforHBaseandPhoenix.docx, > HybridLogicalClocksforHBaseandPhoenix.pdf > > > HBase and Phoenix uses systems physical clock (PT) to give timestamps to > events (read and writes). This works mostly when the system clock is strictly > monotonically increasing and there is no cross-dependency between servers > clocks. However we know that leap seconds, general clock skew and clock drift > are in fact real. > This jira proposes using Hybrid Logical Clocks (HLC) as an implementation of > hybrid physical clock + a logical clock. HLC is best of both worlds where it > keeps causality relationship similar to logical clocks, but still is > compatible with NTP based physical system clock. HLC can be represented in > 64bits. > A design document is attached and also can be found here: > https://docs.google.com/document/d/1LL2GAodiYi0waBz5ODGL4LDT4e_bXy8P9h6kWC05Bhw/edit# -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-14070) Hybrid Logical Clocks for HBase
[ https://issues.apache.org/jira/browse/HBASE-14070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15384889#comment-15384889 ] Sai Teja Ranuva edited comment on HBASE-14070 at 7/19/16 9:42 PM: -- [~enis] [~stack] Is there a reason why we are introducing System Monotonic clock apart from HLC clock ? Do we need it anywhere specifically ? Can we not use HLC in places where we might System Monotonic ? I feel it might be better to have clock implementations that will be used. was (Author: saitejar): [~enis] [~stack] Is there a reason why we are introducing System Monotonic clock apart from HLC clock ? Do we need it anywhere specifically ? Can we not use HLC in places where we might System Monotonic ? > Hybrid Logical Clocks for HBase > --- > > Key: HBASE-14070 > URL: https://issues.apache.org/jira/browse/HBASE-14070 > Project: HBase > Issue Type: New Feature >Reporter: Enis Soztutar >Assignee: Sai Teja Ranuva > Attachments: HybridLogicalClocksforHBaseandPhoenix.docx, > HybridLogicalClocksforHBaseandPhoenix.pdf > > > HBase and Phoenix uses systems physical clock (PT) to give timestamps to > events (read and writes). This works mostly when the system clock is strictly > monotonically increasing and there is no cross-dependency between servers > clocks. However we know that leap seconds, general clock skew and clock drift > are in fact real. > This jira proposes using Hybrid Logical Clocks (HLC) as an implementation of > hybrid physical clock + a logical clock. HLC is best of both worlds where it > keeps causality relationship similar to logical clocks, but still is > compatible with NTP based physical system clock. HLC can be represented in > 64bits. > A design document is attached and also can be found here: > https://docs.google.com/document/d/1LL2GAodiYi0waBz5ODGL4LDT4e_bXy8P9h6kWC05Bhw/edit# -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-14070) Hybrid Logical Clocks for HBase
[ https://issues.apache.org/jira/browse/HBASE-14070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15378417#comment-15378417 ] Sai Teja Ranuva edited comment on HBASE-14070 at 7/14/16 9:26 PM: -- [~enis] Referring to description in RB link - "TTL works with HLC timestamps and SYSTEM timestamps". I feel TTL might not work well with HLC, as there could be a message with much greater PT than the current system time, say a second (but less than max delta we set), which can take the HLC physical time forward. If you read the time the HLC time before the message was received and after the message was received, the difference will be inflated by one second. Can you clarify this aspect ? was (Author: saitejar): [~enis] Referring to description in RB link - "TTL works with HLC timestamps and SYSTEM timestamps". I feel TTL might not work well with HLC, as there could be a message with much greater PT than the current system time, say a second (but less than max delta we set), which can take the HLC physical time forward. If you read the time the HLC time before the message was received and after the message was received, the difference will be inflated by one second. Can you clarity this aspect ? > Hybrid Logical Clocks for HBase > --- > > Key: HBASE-14070 > URL: https://issues.apache.org/jira/browse/HBASE-14070 > Project: HBase > Issue Type: New Feature >Reporter: Enis Soztutar >Assignee: Sai Teja Ranuva > Attachments: HybridLogicalClocksforHBaseandPhoenix.docx, > HybridLogicalClocksforHBaseandPhoenix.pdf > > > HBase and Phoenix uses systems physical clock (PT) to give timestamps to > events (read and writes). This works mostly when the system clock is strictly > monotonically increasing and there is no cross-dependency between servers > clocks. However we know that leap seconds, general clock skew and clock drift > are in fact real. > This jira proposes using Hybrid Logical Clocks (HLC) as an implementation of > hybrid physical clock + a logical clock. HLC is best of both worlds where it > keeps causality relationship similar to logical clocks, but still is > compatible with NTP based physical system clock. HLC can be represented in > 64bits. > A design document is attached and also can be found here: > https://docs.google.com/document/d/1LL2GAodiYi0waBz5ODGL4LDT4e_bXy8P9h6kWC05Bhw/edit# -- This message was sent by Atlassian JIRA (v6.3.4#6332)
