[jira] [Commented] (HBASE-20065) Revisit the timestamp usage in MetaTableAccessor
[ https://issues.apache.org/jira/browse/HBASE-20065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16376561#comment-16376561 ] Hudson commented on HBASE-20065: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4651 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4651/]) HBASE-20065 Addendum remove wrong comment (zhangduo: rev a8471bd98736c7ee387e268415bfd3ff96d8655d) * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java > Revisit the timestamp usage in MetaTableAccessor > > > Key: HBASE-20065 > URL: https://issues.apache.org/jira/browse/HBASE-20065 > Project: HBase > Issue Type: Improvement >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-20065-v1.patch, HBASE-20065.patch > > > It is totally a mess and makes me confusing when reimplementing the serial > replication feature. Let me do a clean up first. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20065) Revisit the timestamp usage in MetaTableAccessor
[ https://issues.apache.org/jira/browse/HBASE-20065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16376326#comment-16376326 ] Duo Zhang commented on HBASE-20065: --- Pushed an addendum to master and branch-2 which removes the wrong comment. > Revisit the timestamp usage in MetaTableAccessor > > > Key: HBASE-20065 > URL: https://issues.apache.org/jira/browse/HBASE-20065 > Project: HBase > Issue Type: Improvement >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-20065-v1.patch, HBASE-20065.patch > > > It is totally a mess and makes me confusing when reimplementing the serial > replication feature. Let me do a clean up first. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20065) Revisit the timestamp usage in MetaTableAccessor
[
https://issues.apache.org/jira/browse/HBASE-20065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16376007#comment-16376007
]
Duo Zhang commented on HBASE-20065:
---
I always wonder why we need to set timestamp explicitly when updating meta
region.
But if we have a bad clock then only something like HLC can save us... For
example, if you use the local time for master as timestamp, then a new master
with a smaller time(no sure how to describe this in English...) will kill you.
But if you use HConstants.LATEST_TIMESTAMP, which meas you want to use the
local time of the RS which holds meta, then if meta is assigned to another RS
with a smaller time will kill you...
{quote}
HBCK doesn't work against hbase2 and setting time on client-side seems
problematic to me.
{quote}
[~openinx] said above that it is a little confusing that we use current time
everywhere but HConstants.LATEST_TIMESTAMP only in hbck. For me I'm OK with
both approach...
Thanks.
> Revisit the timestamp usage in MetaTableAccessor
>
>
> Key: HBASE-20065
> URL: https://issues.apache.org/jira/browse/HBASE-20065
> Project: HBase
> Issue Type: Improvement
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-20065-v1.patch, HBASE-20065.patch
>
>
> It is totally a mess and makes me confusing when reimplementing the serial
> replication feature. Let me do a clean up first.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20065) Revisit the timestamp usage in MetaTableAccessor
[ https://issues.apache.org/jira/browse/HBASE-20065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16376004#comment-16376004 ] stack commented on HBASE-20065: --- Nice cleanup. Why not let the server set the time rather than do it in client: 1449 Put put = makePutFromRegionInfo(regionInfo, EnvironmentEdgeManager.currentTime()); ... previous we did not pass a time. Yeah, it is done again later in the class when we remove this and pass currentTime instead... addRegionsToMeta(connection, regionInfos, regionReplication, HConstants.LATEST_TIMESTAMP); This could go badly wrong if a new Master has a clock that is behind that of the server hosting hbase:meta. nit: Comment is wrong here now... 1531 // use the maximum of what master passed us vs local time. 1595 long time = Math.max(EnvironmentEdgeManager.currentTime(), masterSystemTime); 1532 long time = EnvironmentEdgeManager.currentTime(); Not sure what was going on before... we had a master time? Removing this is nice cleanup. 225 // Start the RegionStateStore 226 regionStateStore.start(); HBCK doesn't work against hbase2 and setting time on client-side seems problematic to me. Otherwise, really nice cleanup. I like the removal of unused overrides. > Revisit the timestamp usage in MetaTableAccessor > > > Key: HBASE-20065 > URL: https://issues.apache.org/jira/browse/HBASE-20065 > Project: HBase > Issue Type: Improvement >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-20065-v1.patch, HBASE-20065.patch > > > It is totally a mess and makes me confusing when reimplementing the serial > replication feature. Let me do a clean up first. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20065) Revisit the timestamp usage in MetaTableAccessor
[ https://issues.apache.org/jira/browse/HBASE-20065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16375989#comment-16375989 ] Hudson commented on HBASE-20065: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4645 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4645/]) HBASE-20065 Revisit the timestamp usage in MetaTableAccessor (zhangduo: rev ba5fb53d147ef63415027f776928c478af37f515) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestGetClosestAtOrBefore.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStateStore.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/favored/FavoredNodeAssignmentHelper.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/TestMetaTableAccessor.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/assignment/MockMasterServices.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsckRepair.java > Revisit the timestamp usage in MetaTableAccessor > > > Key: HBASE-20065 > URL: https://issues.apache.org/jira/browse/HBASE-20065 > Project: HBase > Issue Type: Improvement >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-20065-v1.patch, HBASE-20065.patch > > > It is totally a mess and makes me confusing when reimplementing the serial > replication feature. Let me do a clean up first. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20065) Revisit the timestamp usage in MetaTableAccessor
[
https://issues.apache.org/jira/browse/HBASE-20065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16375557#comment-16375557
]
Hadoop QA commented on HBASE-20065:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m
21s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
6s{color} | {color:red} hbase-server in master has 24 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m
24s{color} | {color:red} hbase-client: The patch generated 1 new + 66 unchanged
- 43 fixed = 67 total (was 109) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
26s{color} | {color:red} hbase-server: The patch generated 1 new + 380
unchanged - 53 fixed = 381 total (was 433) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m
8s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
24m 43s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
58s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}116m
55s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}175m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-20065 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12911895/HBASE-20065-v1.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 387f18008728 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19
15:49:21 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HBASE-20065) Revisit the timestamp usage in MetaTableAccessor
[ https://issues.apache.org/jira/browse/HBASE-20065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16375469#comment-16375469 ] Duo Zhang commented on HBASE-20065: --- Address the review comments, and also the failed UTs. > Revisit the timestamp usage in MetaTableAccessor > > > Key: HBASE-20065 > URL: https://issues.apache.org/jira/browse/HBASE-20065 > Project: HBase > Issue Type: Improvement >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-20065-v1.patch, HBASE-20065.patch > > > It is totally a mess and makes me confusing when reimplementing the serial > replication feature. Let me do a clean up first. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20065) Revisit the timestamp usage in MetaTableAccessor
[ https://issues.apache.org/jira/browse/HBASE-20065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16375462#comment-16375462 ] Duo Zhang commented on HBASE-20065: --- Filed HBASE-20067 for the findbugs warnings. > Revisit the timestamp usage in MetaTableAccessor > > > Key: HBASE-20065 > URL: https://issues.apache.org/jira/browse/HBASE-20065 > Project: HBase > Issue Type: Improvement >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-20065.patch > > > It is totally a mess and makes me confusing when reimplementing the serial > replication feature. Let me do a clean up first. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20065) Revisit the timestamp usage in MetaTableAccessor
[
https://issues.apache.org/jira/browse/HBASE-20065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16375452#comment-16375452
]
Hadoop QA commented on HBASE-20065:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 7m
17s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
30s{color} | {color:red} hbase-server in master has 24 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
37s{color} | {color:red} hbase-client: The patch generated 1 new + 91 unchanged
- 18 fixed = 92 total (was 109) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
19s{color} | {color:green} hbase-server: The patch generated 0 new + 170
unchanged - 53 fixed = 170 total (was 223) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
37s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
22m 10s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
50s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
26s{color} | {color:red} hbase-client generated 1 new + 2 unchanged - 0 fixed =
3 total (was 2) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
25s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}115m 41s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}174m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestHdfsSnapshotHRegion |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-20065 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12911879/HBASE-20065.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname
[jira] [Commented] (HBASE-20065) Revisit the timestamp usage in MetaTableAccessor
[ https://issues.apache.org/jira/browse/HBASE-20065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16375423#comment-16375423 ] Zheng Hu commented on HBASE-20065: -- No other concerns except the above two. > Revisit the timestamp usage in MetaTableAccessor > > > Key: HBASE-20065 > URL: https://issues.apache.org/jira/browse/HBASE-20065 > Project: HBase > Issue Type: Improvement >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-20065.patch > > > It is totally a mess and makes me confusing when reimplementing the serial > replication feature. Let me do a clean up first. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20065) Revisit the timestamp usage in MetaTableAccessor
[ https://issues.apache.org/jira/browse/HBASE-20065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16375418#comment-16375418 ] Duo Zhang commented on HBASE-20065: --- Use HConstants.LATEST_TIMESTAMP is almost the same with current time, it will be replaced at RS side. It is OK to change it to current time. Any other problems? Thanks. > Revisit the timestamp usage in MetaTableAccessor > > > Key: HBASE-20065 > URL: https://issues.apache.org/jira/browse/HBASE-20065 > Project: HBase > Issue Type: Improvement >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-20065.patch > > > It is totally a mess and makes me confusing when reimplementing the serial > replication feature. Let me do a clean up first. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20065) Revisit the timestamp usage in MetaTableAccessor
[
https://issues.apache.org/jira/browse/HBASE-20065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16375411#comment-16375411
]
Zheng Hu commented on HBASE-20065:
--
#1
{code}
/**
* @param t Table to use (will be closed when done).
* @param p put to make
- * @throws IOException
*/
- private static void put(final Table t, final Put p) throws IOException {
-try {
- debugLogMutation(p);
- t.put(p);
-} finally {
- t.close();
-}
+ private static void put(Table t, Put p) throws IOException {
+debugLogMutation(p);
+t.put(p);
}
{code}
The javadoc mismatched the implementation ?
#2
Why we use Long.MAX_VALUE as the ts in HBaseFsck.java ? IMHO, should use
current timestamp (or a future timestamp slight large than the current ts to
make sure the region info from HBCK to be the latest) ?
> Revisit the timestamp usage in MetaTableAccessor
>
>
> Key: HBASE-20065
> URL: https://issues.apache.org/jira/browse/HBASE-20065
> Project: HBase
> Issue Type: Improvement
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Attachments: HBASE-20065.patch
>
>
> It is totally a mess and makes me confusing when reimplementing the serial
> replication feature. Let me do a clean up first.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20065) Revisit the timestamp usage in MetaTableAccessor
[ https://issues.apache.org/jira/browse/HBASE-20065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16375367#comment-16375367 ] Duo Zhang commented on HBASE-20065: --- Must provide a timestamp when constructing a Put or Delete, and except one of the updateRegionLocation, do not pass timestamp for methods which are used to update meta. Make methods private if possible. > Revisit the timestamp usage in MetaTableAccessor > > > Key: HBASE-20065 > URL: https://issues.apache.org/jira/browse/HBASE-20065 > Project: HBase > Issue Type: Improvement >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-20065.patch > > > It is totally a mess and makes me confusing when reimplementing the serial > replication feature. Let me do a clean up first. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
