[jira] [Commented] (HBASE-20065) Revisit the timestamp usage in MetaTableAccessor
[ https://issues.apache.org/jira/browse/HBASE-20065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16376561#comment-16376561 ] Hudson commented on HBASE-20065: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4651 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4651/]) HBASE-20065 Addendum remove wrong comment (zhangduo: rev a8471bd98736c7ee387e268415bfd3ff96d8655d) * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java > Revisit the timestamp usage in MetaTableAccessor > > > Key: HBASE-20065 > URL: https://issues.apache.org/jira/browse/HBASE-20065 > Project: HBase > Issue Type: Improvement >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-20065-v1.patch, HBASE-20065.patch > > > It is totally a mess and makes me confusing when reimplementing the serial > replication feature. Let me do a clean up first. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20065) Revisit the timestamp usage in MetaTableAccessor
[ https://issues.apache.org/jira/browse/HBASE-20065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16376326#comment-16376326 ] Duo Zhang commented on HBASE-20065: --- Pushed an addendum to master and branch-2 which removes the wrong comment. > Revisit the timestamp usage in MetaTableAccessor > > > Key: HBASE-20065 > URL: https://issues.apache.org/jira/browse/HBASE-20065 > Project: HBase > Issue Type: Improvement >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-20065-v1.patch, HBASE-20065.patch > > > It is totally a mess and makes me confusing when reimplementing the serial > replication feature. Let me do a clean up first. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20065) Revisit the timestamp usage in MetaTableAccessor
[ https://issues.apache.org/jira/browse/HBASE-20065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16376007#comment-16376007 ] Duo Zhang commented on HBASE-20065: --- I always wonder why we need to set timestamp explicitly when updating meta region. But if we have a bad clock then only something like HLC can save us... For example, if you use the local time for master as timestamp, then a new master with a smaller time(no sure how to describe this in English...) will kill you. But if you use HConstants.LATEST_TIMESTAMP, which meas you want to use the local time of the RS which holds meta, then if meta is assigned to another RS with a smaller time will kill you... {quote} HBCK doesn't work against hbase2 and setting time on client-side seems problematic to me. {quote} [~openinx] said above that it is a little confusing that we use current time everywhere but HConstants.LATEST_TIMESTAMP only in hbck. For me I'm OK with both approach... Thanks. > Revisit the timestamp usage in MetaTableAccessor > > > Key: HBASE-20065 > URL: https://issues.apache.org/jira/browse/HBASE-20065 > Project: HBase > Issue Type: Improvement >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-20065-v1.patch, HBASE-20065.patch > > > It is totally a mess and makes me confusing when reimplementing the serial > replication feature. Let me do a clean up first. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20065) Revisit the timestamp usage in MetaTableAccessor
[ https://issues.apache.org/jira/browse/HBASE-20065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16376004#comment-16376004 ] stack commented on HBASE-20065: --- Nice cleanup. Why not let the server set the time rather than do it in client: 1449 Put put = makePutFromRegionInfo(regionInfo, EnvironmentEdgeManager.currentTime()); ... previous we did not pass a time. Yeah, it is done again later in the class when we remove this and pass currentTime instead... addRegionsToMeta(connection, regionInfos, regionReplication, HConstants.LATEST_TIMESTAMP); This could go badly wrong if a new Master has a clock that is behind that of the server hosting hbase:meta. nit: Comment is wrong here now... 1531 // use the maximum of what master passed us vs local time. 1595 long time = Math.max(EnvironmentEdgeManager.currentTime(), masterSystemTime); 1532 long time = EnvironmentEdgeManager.currentTime(); Not sure what was going on before... we had a master time? Removing this is nice cleanup. 225 // Start the RegionStateStore 226 regionStateStore.start(); HBCK doesn't work against hbase2 and setting time on client-side seems problematic to me. Otherwise, really nice cleanup. I like the removal of unused overrides. > Revisit the timestamp usage in MetaTableAccessor > > > Key: HBASE-20065 > URL: https://issues.apache.org/jira/browse/HBASE-20065 > Project: HBase > Issue Type: Improvement >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-20065-v1.patch, HBASE-20065.patch > > > It is totally a mess and makes me confusing when reimplementing the serial > replication feature. Let me do a clean up first. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20065) Revisit the timestamp usage in MetaTableAccessor
[ https://issues.apache.org/jira/browse/HBASE-20065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16375989#comment-16375989 ] Hudson commented on HBASE-20065: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4645 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4645/]) HBASE-20065 Revisit the timestamp usage in MetaTableAccessor (zhangduo: rev ba5fb53d147ef63415027f776928c478af37f515) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestGetClosestAtOrBefore.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStateStore.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/favored/FavoredNodeAssignmentHelper.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/TestMetaTableAccessor.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/assignment/MockMasterServices.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsckRepair.java > Revisit the timestamp usage in MetaTableAccessor > > > Key: HBASE-20065 > URL: https://issues.apache.org/jira/browse/HBASE-20065 > Project: HBase > Issue Type: Improvement >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-20065-v1.patch, HBASE-20065.patch > > > It is totally a mess and makes me confusing when reimplementing the serial > replication feature. Let me do a clean up first. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20065) Revisit the timestamp usage in MetaTableAccessor
[ https://issues.apache.org/jira/browse/HBASE-20065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16375557#comment-16375557 ] Hadoop QA commented on HBASE-20065: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 3 new or modified test files. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 20s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 39s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m 21s{color} | {color:green} branch has no errors when building our shaded downstream artifacts. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 6s{color} | {color:red} hbase-server in master has 24 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 31s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 3s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 3s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m 24s{color} | {color:red} hbase-client: The patch generated 1 new + 66 unchanged - 43 fixed = 67 total (was 109) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 26s{color} | {color:red} hbase-server: The patch generated 1 new + 380 unchanged - 53 fixed = 381 total (was 433) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m 8s{color} | {color:green} patch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 24m 43s{color} | {color:green} Patch does not cause any errors with Hadoop 2.6.5 2.7.4 or 3.0.0. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 45s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 58s{color} | {color:green} hbase-client in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green}116m 55s{color} | {color:green} hbase-server in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 35s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}175m 17s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 | | JIRA Issue | HBASE-20065 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12911895/HBASE-20065-v1.patch | | Optional Tests | asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 387f18008728 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19 15:49:21 UTC 2017 x86_64 GNU/Linux | | Build tool | maven | | Personality |
[jira] [Commented] (HBASE-20065) Revisit the timestamp usage in MetaTableAccessor
[ https://issues.apache.org/jira/browse/HBASE-20065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16375469#comment-16375469 ] Duo Zhang commented on HBASE-20065: --- Address the review comments, and also the failed UTs. > Revisit the timestamp usage in MetaTableAccessor > > > Key: HBASE-20065 > URL: https://issues.apache.org/jira/browse/HBASE-20065 > Project: HBase > Issue Type: Improvement >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-20065-v1.patch, HBASE-20065.patch > > > It is totally a mess and makes me confusing when reimplementing the serial > replication feature. Let me do a clean up first. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20065) Revisit the timestamp usage in MetaTableAccessor
[ https://issues.apache.org/jira/browse/HBASE-20065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16375462#comment-16375462 ] Duo Zhang commented on HBASE-20065: --- Filed HBASE-20067 for the findbugs warnings. > Revisit the timestamp usage in MetaTableAccessor > > > Key: HBASE-20065 > URL: https://issues.apache.org/jira/browse/HBASE-20065 > Project: HBase > Issue Type: Improvement >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-20065.patch > > > It is totally a mess and makes me confusing when reimplementing the serial > replication feature. Let me do a clean up first. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20065) Revisit the timestamp usage in MetaTableAccessor
[ https://issues.apache.org/jira/browse/HBASE-20065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16375452#comment-16375452 ] Hadoop QA commented on HBASE-20065: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 3 new or modified test files. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 20s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 14s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 49s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 7m 17s{color} | {color:green} branch has no errors when building our shaded downstream artifacts. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 30s{color} | {color:red} hbase-server in master has 24 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 59s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 20s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 20s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 37s{color} | {color:red} hbase-client: The patch generated 1 new + 91 unchanged - 18 fixed = 92 total (was 109) {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 19s{color} | {color:green} hbase-server: The patch generated 0 new + 170 unchanged - 53 fixed = 170 total (was 223) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m 37s{color} | {color:green} patch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 22m 10s{color} | {color:green} Patch does not cause any errors with Hadoop 2.6.5 2.7.4 or 3.0.0. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 50s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 26s{color} | {color:red} hbase-client generated 1 new + 2 unchanged - 0 fixed = 3 total (was 2) {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 25s{color} | {color:green} hbase-client in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red}115m 41s{color} | {color:red} hbase-server in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 41s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}174m 57s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hbase.regionserver.TestHdfsSnapshotHRegion | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 | | JIRA Issue | HBASE-20065 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12911879/HBASE-20065.patch | | Optional Tests | asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname
[jira] [Commented] (HBASE-20065) Revisit the timestamp usage in MetaTableAccessor
[ https://issues.apache.org/jira/browse/HBASE-20065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16375423#comment-16375423 ] Zheng Hu commented on HBASE-20065: -- No other concerns except the above two. > Revisit the timestamp usage in MetaTableAccessor > > > Key: HBASE-20065 > URL: https://issues.apache.org/jira/browse/HBASE-20065 > Project: HBase > Issue Type: Improvement >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-20065.patch > > > It is totally a mess and makes me confusing when reimplementing the serial > replication feature. Let me do a clean up first. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20065) Revisit the timestamp usage in MetaTableAccessor
[ https://issues.apache.org/jira/browse/HBASE-20065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16375418#comment-16375418 ] Duo Zhang commented on HBASE-20065: --- Use HConstants.LATEST_TIMESTAMP is almost the same with current time, it will be replaced at RS side. It is OK to change it to current time. Any other problems? Thanks. > Revisit the timestamp usage in MetaTableAccessor > > > Key: HBASE-20065 > URL: https://issues.apache.org/jira/browse/HBASE-20065 > Project: HBase > Issue Type: Improvement >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-20065.patch > > > It is totally a mess and makes me confusing when reimplementing the serial > replication feature. Let me do a clean up first. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20065) Revisit the timestamp usage in MetaTableAccessor
[ https://issues.apache.org/jira/browse/HBASE-20065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16375411#comment-16375411 ] Zheng Hu commented on HBASE-20065: -- #1 {code} /** * @param t Table to use (will be closed when done). * @param p put to make - * @throws IOException */ - private static void put(final Table t, final Put p) throws IOException { -try { - debugLogMutation(p); - t.put(p); -} finally { - t.close(); -} + private static void put(Table t, Put p) throws IOException { +debugLogMutation(p); +t.put(p); } {code} The javadoc mismatched the implementation ? #2 Why we use Long.MAX_VALUE as the ts in HBaseFsck.java ? IMHO, should use current timestamp (or a future timestamp slight large than the current ts to make sure the region info from HBCK to be the latest) ? > Revisit the timestamp usage in MetaTableAccessor > > > Key: HBASE-20065 > URL: https://issues.apache.org/jira/browse/HBASE-20065 > Project: HBase > Issue Type: Improvement >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-20065.patch > > > It is totally a mess and makes me confusing when reimplementing the serial > replication feature. Let me do a clean up first. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20065) Revisit the timestamp usage in MetaTableAccessor
[ https://issues.apache.org/jira/browse/HBASE-20065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16375367#comment-16375367 ] Duo Zhang commented on HBASE-20065: --- Must provide a timestamp when constructing a Put or Delete, and except one of the updateRegionLocation, do not pass timestamp for methods which are used to update meta. Make methods private if possible. > Revisit the timestamp usage in MetaTableAccessor > > > Key: HBASE-20065 > URL: https://issues.apache.org/jira/browse/HBASE-20065 > Project: HBase > Issue Type: Improvement >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-20065.patch > > > It is totally a mess and makes me confusing when reimplementing the serial > replication feature. Let me do a clean up first. -- This message was sent by Atlassian JIRA (v7.6.3#76005)