Thanks Amit for continuing on the work. I am biased, but of course would like to see this merged back to master (and possibly make it to 2.0) once the issues are ironed out. We can eliminate a very large class of problems with a mathematically sound strategy.
Enis On Tue, Aug 22, 2017 at 1:28 PM, Stack <[email protected]> wrote: > Look at the pretty HLC timestamps in meta after a run: > > hbase:meta > column=table:state, timestamp=1576463246879096833, value=\x08\x00 > hbase:namespace > column=table:state, timestamp=1576460369910562817, > value=\x08\x00 > hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5. > column=info:regioninfo, timestamp=1576462593304821761, > value={ENCODED => 6d365281d5ca8ee0bc0fa12cdf33ecb5, NAME => > 'hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5.', > STARTKEY => '', ENDKEY => ''} > hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5. > column=info:seqnumDuringOpen, timestamp=1576462593304821761, > value=\x00\x00\x00\x00\x00\x00\x00\x15 > hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5. > column=info:server, timestamp=1576462593304821761, value= > ve0536.halxg.cloudera.com:16020 > hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5. > column=info:serverstartcode, timestamp=1576462593304821761, > value=1503431013602 > hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5. > column=info:sn, timestamp=1576462593050017795, value= > ve0536.halxg.cloudera.com,16020,1503431013602 > hbase:namespace,,1503429763610.6d365281d5ca8ee0bc0fa12cdf33ecb5. > column=info:state, timestamp=1576462593304821761, value=OPEN > > St.Ack > > > > On Tue, Aug 22, 2017 at 1:26 PM, Stack <[email protected]> wrote: > > > Nice work Amit. Idea of merging sounds good to me. Our metadata system > > will run w/ a new rigor as HLC allows us leave behind a whole class of > > issues. > > > > I ran the HLC branch on small cluster for a while w/ chaos. The ITBLL job > > passed so basically works I'd say. +1 from me on merge. > > > > St.Ack > > > > On Mon, Aug 21, 2017 at 1:29 PM, Amit Patel <[email protected]> > > wrote: > > > >> Hi everyone, > >> > >> I’d like to begin the discussion of merging the current HLC work with > >> master. Note that the current work is not ready to merge yet since there > >> are a few remaining issues (see below). For reference, HBASE-14070 > >> <https://issues.apache.org/jira/browse/HBASE-14070> proposes using > hybrid > >> logical clocks, which combine both logical and physical clocks to > capture > >> not only causality relationship between events, but also physical time > at > >> which events occur. HLC can be enabled on a per-table basis and at the > >> moment just the meta table has HLC enabled by default (the other tables > >> default to use the system clock). An initial design document created by > >> Enis Soztutar can be found here > >> <https://docs.google.com/document/d/1LL2GAodiYi0waBz5ODGL4LD > >> T4e_bXy8P9h6kWC05Bhw/edit#>. > >> The more recent status document created by Sai Teja Ranuva can be found > >> here > >> <https://docs.google.com/document/d/1n32DUKoL3LSoKQ1_NuF8TbQ > >> kZiZCwFhbHZ7JXRjg3oA/edit#heading=h.881hwl2sdbvx>. > >> > >> > >> Changes > >> > >> The new classes that would be introduced include: a Clock interface with > >> three clock implementations mentioned above and enums for the clock and > >> timestamp type. The three clock implementations are hybrid logical, > >> system > >> monotonic, and system clocks and each generate 64-bit timestamps. > >> Timestamps from the system monotonic and system clock represent physical > >> time and are directly backwards compatible. > >> > >> Users can set the clock type of a table during table creation. A table’s > >> clock type cannot be changed after the table has been created. For > tables > >> whose clock type is HLC, users would not be permitted to set their own > >> timestamps for mutations. Certain events such as region assignment, > >> unassignment, and recovery update the clocks upon receiving timestamps > >> externally. > >> > >> Benefits > >> > >> > >> - > >> > >> Hybrid logical clocks (and even system monotonic clocks) can solve a > >> variety of long standing issues related to time highlighted here > >> <https://docs.google.com/document/d/1LL2GAodiYi0waBz5ODGL4L > >> DT4e_bXy8P9h6kWC05Bhw/edit#heading=h.msabldfs8oly>. > >> > >> - > >> > >> Future work with hybrid logical clocks would include enabling them > with > >> user tables. Ideally we would want HLC to be used everywhere. Further > >> work > >> such as global point-in-time snapshots can leverage HLC > >> > >> > >> Remaining issues > >> > >> There are still a few remaining issues to close out before actually > >> merging, but I wanted to at least start the discussion. Currently I am > >> working on fixing remaining tests that are either failing or timing out > in > >> the public branch <https://builds.apache.org/job/HBASE-14070.HLC/> as > >> well > >> as doing local performance tests, and cluster tests. Some of the > remaining > >> issues we are tracking are: > >> > >> > >> - > >> > >> HBASE-18542 (Perf numbers) > >> <https://issues.apache.org/jira/browse/HBASE-18542> > >> - > >> > >> HBASE-18508 (Fixing unit tests) > >> <https://issues.apache.org/jira/browse/HBASE-18508> > >> - > >> > >> HBASE-18432 (Clock skew) > >> <https://issues.apache.org/jira/browse/HBASE-18432> > >> > >> Some brainstorming/discussions we also are tracking: > >> > >> - > >> > >> (HBASE-18643) Adding transaction id to Result > >> <https://issues.apache.org/jira/browse/HBASE-18643> > >> - > >> > >> (HBASE-18642) Deprecate setting of timestamp in client for HLC > >> <https://issues.apache.org/jira/browse/HBASE-18642> > >> > > > > >
