Nice work Amit. Idea of merging sounds good to me. Our metadata system will run w/ a new rigor as HLC allows us leave behind a whole class of issues.
I ran the HLC branch on small cluster for a while w/ chaos. The ITBLL job passed so basically works I'd say. +1 from me on merge. St.Ack On Mon, Aug 21, 2017 at 1:29 PM, Amit Patel <[email protected]> wrote: > Hi everyone, > > I’d like to begin the discussion of merging the current HLC work with > master. Note that the current work is not ready to merge yet since there > are a few remaining issues (see below). For reference, HBASE-14070 > <https://issues.apache.org/jira/browse/HBASE-14070> proposes using hybrid > logical clocks, which combine both logical and physical clocks to capture > not only causality relationship between events, but also physical time at > which events occur. HLC can be enabled on a per-table basis and at the > moment just the meta table has HLC enabled by default (the other tables > default to use the system clock). An initial design document created by > Enis Soztutar can be found here > <https://docs.google.com/document/d/1LL2GAodiYi0waBz5ODGL4LDT4e_ > bXy8P9h6kWC05Bhw/edit#>. > The more recent status document created by Sai Teja Ranuva can be found > here > <https://docs.google.com/document/d/1n32DUKoL3LSoKQ1_ > NuF8TbQkZiZCwFhbHZ7JXRjg3oA/edit#heading=h.881hwl2sdbvx>. > > > Changes > > The new classes that would be introduced include: a Clock interface with > three clock implementations mentioned above and enums for the clock and > timestamp type. The three clock implementations are hybrid logical, system > monotonic, and system clocks and each generate 64-bit timestamps. > Timestamps from the system monotonic and system clock represent physical > time and are directly backwards compatible. > > Users can set the clock type of a table during table creation. A table’s > clock type cannot be changed after the table has been created. For tables > whose clock type is HLC, users would not be permitted to set their own > timestamps for mutations. Certain events such as region assignment, > unassignment, and recovery update the clocks upon receiving timestamps > externally. > > Benefits > > > - > > Hybrid logical clocks (and even system monotonic clocks) can solve a > variety of long standing issues related to time highlighted here > <https://docs.google.com/document/d/1LL2GAodiYi0waBz5ODGL4LDT4e_ > bXy8P9h6kWC05Bhw/edit#heading=h.msabldfs8oly>. > > - > > Future work with hybrid logical clocks would include enabling them with > user tables. Ideally we would want HLC to be used everywhere. Further > work > such as global point-in-time snapshots can leverage HLC > > > Remaining issues > > There are still a few remaining issues to close out before actually > merging, but I wanted to at least start the discussion. Currently I am > working on fixing remaining tests that are either failing or timing out in > the public branch <https://builds.apache.org/job/HBASE-14070.HLC/> as well > as doing local performance tests, and cluster tests. Some of the remaining > issues we are tracking are: > > > - > > HBASE-18542 (Perf numbers) > <https://issues.apache.org/jira/browse/HBASE-18542> > - > > HBASE-18508 (Fixing unit tests) > <https://issues.apache.org/jira/browse/HBASE-18508> > - > > HBASE-18432 (Clock skew) > <https://issues.apache.org/jira/browse/HBASE-18432> > > Some brainstorming/discussions we also are tracking: > > - > > (HBASE-18643) Adding transaction id to Result > <https://issues.apache.org/jira/browse/HBASE-18643> > - > > (HBASE-18642) Deprecate setting of timestamp in client for HLC > <https://issues.apache.org/jira/browse/HBASE-18642> >
