[jira] [Commented] (HBASE-21440) Assign procedure on the crashed server is not properly interrupted
[
https://issues.apache.org/jira/browse/HBASE-21440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16686172#comment-16686172
]
Hadoop QA commented on HBASE-21440:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.0 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
47s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
27s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
43s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
45s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
31s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} branch-2.0 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
27s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 36s{color} | {color:green} Patch does not cause any errors with Hadoop 2.6.5
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
22s{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}123m
55s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
43s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}172m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:6f01af0 |
| JIRA Issue | HBASE-21440 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12948071/HBASE-21440.branch-2.0.005.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 7d04b089f69f 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | branch-2.0 / 2fcf961e73 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe;
[jira] [Commented] (HBASE-21468) separate workers for meta table is not working
[ https://issues.apache.org/jira/browse/HBASE-21468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16686149#comment-16686149 ] Allan Yang commented on HBASE-21468: RN added, thanks for reminding,[~anoop.hbase]. > separate workers for meta table is not working > -- > > Key: HBASE-21468 > URL: https://issues.apache.org/jira/browse/HBASE-21468 > Project: HBase > Issue Type: Bug >Affects Versions: 2.1.1, 2.0.2 >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > Fix For: 2.0.3, 2.1.2 > > Attachments: HBASE-21468.branch-2.0.001.patch > > > This is an addendum for HBASE-21423, since HBASE-21423 is already closed, the > QA won't be triggered. > It is my mistake that the separate workers for meta table is not working, > since when polling from queue, the onlyUrgent flag is not passed in. > And for some UT that only require one worker thread, urgent workers should > set to 0 to ensure there is one worker at time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21423) Procedures for meta table/region should be able to execute in separate workers
[ https://issues.apache.org/jira/browse/HBASE-21423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allan Yang updated HBASE-21423: --- Release Note: The procedure for meta table will be executed in a separate worker thread named 'Urgent Worker' to avoid stuck. A new config named 'hbase.master.urgent.procedure.threads' is added, the default value for it is 1. To disable the separate worker, set it to 0. > Procedures for meta table/region should be able to execute in separate > workers > --- > > Key: HBASE-21423 > URL: https://issues.apache.org/jira/browse/HBASE-21423 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.1.1, 2.0.2 >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > Fix For: 2.0.3, 2.1.2 > > Attachments: HBASE-21423.branch-2.0.001.patch, > HBASE-21423.branch-2.0.002.patch, HBASE-21423.branch-2.0.003.patch, > HBASE-21423.branch-2.0.addendum.patch > > > We have higher priority for meta table procedures, but only in queue level. > There is a case that the meta table is closed and a AssignProcedure(or RTSP > in branch-2+) is waiting there to be executed, but at the same time, all the > Work threads are executing procedures need to write to meta table, then all > the worker will be stuck and retry for writing meta, no worker will take the > AP for meta. > Though we have a mechanism that will detect stuck and adding more > ''KeepAlive'' workers to the pool to resolve the stuck. It is already stuck a > long time. > This is a real case I encountered in ITBLL. > So, I add one 'Urgent work' to the ProceudureExecutor, which only take meta > procedures(other workers can take meta procedures too), which can resolve > this kind of stuck. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21452) Illegal character in hbase counters group name
[
https://issues.apache.org/jira/browse/HBASE-21452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-21452:
--
Resolution: Fixed
Hadoop Flags: Incompatible change,Reviewed (was: Incompatible change)
Status: Resolved (was: Patch Available)
Pushed to branch-2+. Thanks for review [~busbey]
> Illegal character in hbase counters group name
> --
>
> Key: HBASE-21452
> URL: https://issues.apache.org/jira/browse/HBASE-21452
> Project: HBase
> Issue Type: Bug
> Components: spark
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21452.branch-2.001.patch
>
>
> Messing w/ spark counting RDD rows, spark dumps out following complaint:
> {code}
> 2018-11-07 20:03:29,132 ERROR [Executor task launch worker for task 0]
> repl.ExecutorClassLoader: Failed to check existence of class HBase
> Counters_en_US on REPL class server at spark://192.168.1.139:61037/classes
> java.net.URISyntaxException: Illegal character in path at index 41:
> spark://192.168.1.139:61037/classes/HBase Counters_en_US.class
> at java.net.URI$Parser.fail(URI.java:2848)
> at java.net.URI$Parser.checkChars(URI.java:3021)
> at java.net.URI$Parser.parseHierarchical(URI.java:3105)
> at java.net.URI$Parser.parse(URI.java:3053)
> at java.net.URI.(URI.java:588)
> at
> org.apache.spark.rpc.netty.NettyRpcEnv.openChannel(NettyRpcEnv.scala:328)
> at
> org.apache.spark.repl.ExecutorClassLoader.org$apache$spark$repl$ExecutorClassLoader$$getClassFileInputStreamFromSparkRPC(ExecutorClassLoader.scala:95)
> at
> org.apache.spark.repl.ExecutorClassLoader$$anonfun$1.apply(ExecutorClassLoader.scala:62)
> at
> org.apache.spark.repl.ExecutorClassLoader$$anonfun$1.apply(ExecutorClassLoader.scala:62)
> at
> org.apache.spark.repl.ExecutorClassLoader.findClassLocally(ExecutorClassLoader.scala:167)
> at
> org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:85)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.util.ResourceBundle$Control.newBundle(ResourceBundle.java:2649)
> at java.util.ResourceBundle.loadBundle(ResourceBundle.java:1510)
> at java.util.ResourceBundle.findBundle(ResourceBundle.java:1474)
> at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1370)
> at java.util.ResourceBundle.getBundle(ResourceBundle.java:1091)
> at
> org.apache.hadoop.mapreduce.util.ResourceBundles.getBundle(ResourceBundles.java:37)
> at
> org.apache.hadoop.mapreduce.util.ResourceBundles.getValue(ResourceBundles.java:56)
> at
> org.apache.hadoop.mapreduce.util.ResourceBundles.getCounterGroupName(ResourceBundles.java:77)
> at
> org.apache.hadoop.mapreduce.counters.CounterGroupFactory.newGroup(CounterGroupFactory.java:94)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounters.getGroup(AbstractCounters.java:226)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounters.findCounter(AbstractCounters.java:153)
> at
> org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl$DummyReporter.getCounter(TaskAttemptContextImpl.java:110)
> at
> org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl.getCounter(TaskAttemptContextImpl.java:76)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.updateCounters(TableRecordReaderImpl.java:298)
> at
> org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.updateCounters(TableRecordReaderImpl.java:286)
> at
> org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.nextKeyValue(TableRecordReaderImpl.java:257)
> at
> org.apache.hadoop.hbase.mapreduce.TableRecordReader.nextKeyValue(TableRecordReader.java:133)
> at
> org.apache.hadoop.hbase.mapreduce.TableInputFormatBase$1.nextKeyValue(TableInputFormatBase.java:220)
> at
> org.apache.spark.rdd.NewHadoopRDD$$anon$1.hasNext(NewHadoopRDD.scala:214)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
> at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1837)
> at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1168)
> at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1168)
> at
> org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)
> at
>
[jira] [Commented] (HBASE-21410) A helper page that help find all problematic regions and procedures
[ https://issues.apache.org/jira/browse/HBASE-21410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16686121#comment-16686121 ] stack commented on HBASE-21410: --- bq. I misunderstood your reply 2 days ago as a +1. I will pay more attention next time. [~tianjingyun] No harm done. You learning the ropes. Nice feature. Thanks. > A helper page that help find all problematic regions and procedures > --- > > Key: HBASE-21410 > URL: https://issues.apache.org/jira/browse/HBASE-21410 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0, 2.2.0, 2.1.1 >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Major > Fix For: 3.0.0, 2.1.0, 2.2.0 > > Attachments: HBASE-21410.branch-2.1.001.patch, > HBASE-21410.branch-2.1.002.patch, HBASE-21410.master.001.patch, > HBASE-21410.master.002.patch, HBASE-21410.master.003.patch, > HBASE-21410.master.004.patch, Screenshot from 2018-10-30 19-06-21.png, > Screenshot from 2018-10-30 19-06-42.png, Screenshot from 2018-10-31 > 10-11-38.png, Screenshot from 2018-10-31 10-11-56.png, Screenshot from > 2018-11-01 17-56-02.png, Screenshot from 2018-11-01 17-56-15.png > > > *This page is mainly focus on finding the regions stuck in some state that > cannot be assigned. My proposal of the page is as follows:* > !Screenshot from 2018-10-30 19-06-21.png! > *From this page we can see all regions in RIT queue and their related > procedures. If we can determine that these regions' state are abnormal, we > can click the link 'Procedures as TXT' to get a full list of procedure IDs to > bypass them. Then click 'Regions as TXT' to get a full list of encoded region > names to assign.* > !Screenshot from 2018-10-30 19-06-42.png! > *Some region names are covered by the navigator bar, I'll fix it later.* -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21460) correct Document Configurable Bucket Sizes in bucketCache
[
https://issues.apache.org/jira/browse/HBASE-21460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16686117#comment-16686117
]
Yechao Chen commented on HBASE-21460:
-
[~te...@apache.org] mind have a look ,thanks.
> correct Document Configurable Bucket Sizes in bucketCache
> -
>
> Key: HBASE-21460
> URL: https://issues.apache.org/jira/browse/HBASE-21460
> Project: HBase
> Issue Type: Improvement
> Components: documentation
>Reporter: Yechao Chen
>Assignee: Yechao Chen
>Priority: Major
> Attachments: HBASE-21460.v1.patch
>
>
> we use the bucket cache(offheap),found the doc was error,
> the property bucket sizes shoul be "hbase.bucketcache.bucket.sizes" instead
> of "hfile.block.cache.sizes"
> CacheConfig.java
> /**
> * A comma-delimited array of values for use as bucket sizes.
> */
> public static final String BUCKET_CACHE_BUCKETS_KEY =
> "hbase.bucketcache.bucket.sizes";
> the doc was:
>
>
> HBASE-10641 introduced the ability to configure multiple sizes for the
> buckets of the BucketCache, in HBase 0.98 and newer. To configurable multiple
> bucket sizes, configure the new property
> {color:#ff}{{hfile.block.cache.sizes}}{color} (instead of{color:#ff}
> {{hfile.block.cache.size}}{color}) to a comma-separated list of block sizes,
> ordered from smallest to largest, with no spaces. The goal is to optimize the
> bucket sizes based on your data access patterns. The following example
> configures buckets of size 4096 and 8192.
> {color:#ff}hfile.block.cache.sizes{color}
> 4096,8192
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21465) Retry on reportRegionStateTransition can lead to unexpected errors
[ https://issues.apache.org/jira/browse/HBASE-21465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-21465: -- Status: Patch Available (was: Reopened) > Retry on reportRegionStateTransition can lead to unexpected errors > -- > > Key: HBASE-21465 > URL: https://issues.apache.org/jira/browse/HBASE-21465 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-21465-UT.patch, HBASE-21465-v1.patch, > HBASE-21465-v2.patch, HBASE-21465-v3.patch, HBASE-21465-v4.patch, > HBASE-21465.patch > > > It is possible that the reportRegionStateTransition method is succeeded at > master side, but before returning the the region server, the rpc connection > is broken, or the master restarts. So next when the region server try > again,we will find that the state for the region and the TRSP is not correct, > and can lead to a RS abort or something even worse. > We should be able to determine whether a reportRegionStateTransition call is > just a retry and has already been succeeded, and just ignore it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21034) Add new throttle type: read/write capacity unit
[
https://issues.apache.org/jira/browse/HBASE-21034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16686115#comment-16686115
]
Hadoop QA commented on HBASE-21034:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
59s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 3m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} The patch passed checkstyle in hbase-protocol-shaded
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} The patch passed checkstyle in hbase-client {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s{color} | {color:green} hbase-server: The patch generated 0 new + 19
unchanged - 12 fixed = 19 total (was 31) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} The patch passed checkstyle in hbase-shell {color} |
| {color:red}-1{color} | {color:red} rubocop {color} | {color:red} 0m
7s{color} | {color:red} The patch generated 5 new + 90 unchanged - 5 fixed = 95
total (was 95) {color} |
| {color:green}+1{color} | {color:green} ruby-lint {color} | {color:green} 0m
3s{color} | {color:green} There were no new ruby-lint issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
56s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 4s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green}
1m 50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
32s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
[jira] [Updated] (HBASE-21440) Assign procedure on the crashed server is not properly interrupted
[ https://issues.apache.org/jira/browse/HBASE-21440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Singhal updated HBASE-21440: -- Attachment: HBASE-21440.branch-2.0.005.patch > Assign procedure on the crashed server is not properly interrupted > -- > > Key: HBASE-21440 > URL: https://issues.apache.org/jira/browse/HBASE-21440 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.2 >Reporter: Ankit Singhal >Assignee: Ankit Singhal >Priority: Major > Attachments: HBASE-21440.branch-2.0.001.patch, > HBASE-21440.branch-2.0.002.patch, HBASE-21440.branch-2.0.003.patch, > HBASE-21440.branch-2.0.004.patch, HBASE-21440.branch-2.0.005.patch > > > When the server crashes, it's SCP checks if there is already a procedure > assigning the region on this crashed server. If we found one, SCP will just > interrupt the already running AssignProcedure by calling remoteCallFailed > which internally just changes the region node state to OFFLINE and send the > procedure back with transition queue state for assignment with a new plan. > But, due to the race condition between the calling of the remoteCallFailed > and current state of the already running assign > procedure(REGION_TRANSITION_FINISH: where the region is already opened), it > is possible that assign procedure goes ahead in updating the regionStateNode > to OPEN on a crashed server. > As SCP had already skipped this region for assignment as it was relying on > existing assign procedure to do the right thing, this whole confusion leads > region to a not accessible state. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21468) separate workers for meta table is not working
[ https://issues.apache.org/jira/browse/HBASE-21468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16686093#comment-16686093 ] Anoop Sam John commented on HBASE-21468: Pls update the RN in main issue with this new config info. Tks > separate workers for meta table is not working > -- > > Key: HBASE-21468 > URL: https://issues.apache.org/jira/browse/HBASE-21468 > Project: HBase > Issue Type: Bug >Affects Versions: 2.1.1, 2.0.2 >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > Fix For: 2.0.3, 2.1.2 > > Attachments: HBASE-21468.branch-2.0.001.patch > > > This is an addendum for HBASE-21423, since HBASE-21423 is already closed, the > QA won't be triggered. > It is my mistake that the separate workers for meta table is not working, > since when polling from queue, the onlyUrgent flag is not passed in. > And for some UT that only require one worker thread, urgent workers should > set to 0 to ensure there is one worker at time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21440) Assign procedure on the crashed server is not properly interrupted
[ https://issues.apache.org/jira/browse/HBASE-21440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16686091#comment-16686091 ] Ankit Singhal commented on HBASE-21440: --- Thanks [~allan163] for the confirmation, Uploaded 005.patch with checkstyle fix and to re-run the test. > Assign procedure on the crashed server is not properly interrupted > -- > > Key: HBASE-21440 > URL: https://issues.apache.org/jira/browse/HBASE-21440 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.2 >Reporter: Ankit Singhal >Assignee: Ankit Singhal >Priority: Major > Attachments: HBASE-21440.branch-2.0.001.patch, > HBASE-21440.branch-2.0.002.patch, HBASE-21440.branch-2.0.003.patch, > HBASE-21440.branch-2.0.004.patch, HBASE-21440.branch-2.0.005.patch > > > When the server crashes, it's SCP checks if there is already a procedure > assigning the region on this crashed server. If we found one, SCP will just > interrupt the already running AssignProcedure by calling remoteCallFailed > which internally just changes the region node state to OFFLINE and send the > procedure back with transition queue state for assignment with a new plan. > But, due to the race condition between the calling of the remoteCallFailed > and current state of the already running assign > procedure(REGION_TRANSITION_FINISH: where the region is already opened), it > is possible that assign procedure goes ahead in updating the regionStateNode > to OPEN on a crashed server. > As SCP had already skipped this region for assignment as it was relying on > existing assign procedure to do the right thing, this whole confusion leads > region to a not accessible state. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21440) Assign procedure on the crashed server is not properly interrupted
[ https://issues.apache.org/jira/browse/HBASE-21440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16686060#comment-16686060 ] Allan Yang edited comment on HBASE-21440 at 11/14/18 4:28 AM: -- +1 on v4 patch(fix the checkstyle please). Maybe it is HBASE-21468 making these test flaky, I have committed HBASE-21468 to branch-2.0 and branch-2.1. [~an...@apache.org], you can re-trigger a QA run here. Sorry for that. was (Author: allan163): +1 on v4 patch. Maybe it is HBASE-21468 making these test flaky, I have committed HBASE-21468 to branch-2.0 and branch-2.1. [~an...@apache.org], you can re-trigger a QA run here. Sorry for that. > Assign procedure on the crashed server is not properly interrupted > -- > > Key: HBASE-21440 > URL: https://issues.apache.org/jira/browse/HBASE-21440 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.2 >Reporter: Ankit Singhal >Assignee: Ankit Singhal >Priority: Major > Attachments: HBASE-21440.branch-2.0.001.patch, > HBASE-21440.branch-2.0.002.patch, HBASE-21440.branch-2.0.003.patch, > HBASE-21440.branch-2.0.004.patch > > > When the server crashes, it's SCP checks if there is already a procedure > assigning the region on this crashed server. If we found one, SCP will just > interrupt the already running AssignProcedure by calling remoteCallFailed > which internally just changes the region node state to OFFLINE and send the > procedure back with transition queue state for assignment with a new plan. > But, due to the race condition between the calling of the remoteCallFailed > and current state of the already running assign > procedure(REGION_TRANSITION_FINISH: where the region is already opened), it > is possible that assign procedure goes ahead in updating the regionStateNode > to OPEN on a crashed server. > As SCP had already skipped this region for assignment as it was relying on > existing assign procedure to do the right thing, this whole confusion leads > region to a not accessible state. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21476) Support for nanosecond timestamps
[ https://issues.apache.org/jira/browse/HBASE-21476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16686071#comment-16686071 ] Sean Busbey commented on HBASE-21476: - Please write a scope document. Consider emailing dev@hbase to gather feedback once you have one. Important bits to cover: * What use cases this enables * How will this impact upgrades * How/where we'll document this * How we'll test this feature Some example scope documents: * [HBase Spark Integration|https://issues.apache.org/jira/secure/attachment/12878023/Apache%20HBase%20-%20Apache%20Spark%20Integration%20Scope%20-%20update%201.pdf] (HBASE-18405) * [Read Replica Clusters|https://issues.apache.org/jira/secure/attachment/12888376/HBase%20Read-Replica%20Clusters%20Scope%20doc_v2.pdf] (HBASE-18477) > Support for nanosecond timestamps > - > > Key: HBASE-21476 > URL: https://issues.apache.org/jira/browse/HBASE-21476 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.1.1 >Reporter: Andrey Elenskiy >Assignee: Andrey Elenskiy >Priority: Major > Labels: features, patch > Attachments: nanosecond_timestamps_v1.patch > > > Introducing a new table attribute "NANOSECOND_TIMESTAMPS" to tell HBase to > handle timestamps with nanosecond precision. This is useful for applications > that timestamp updates at the source with nanoseconds and still want features > like column family TTL and "hbase.hstore.time.to.purge.deletes" to work. > The attribute should be specified either on new tables or on existing tables > which have timestamps only with nanosecond precision. There's no migration > from milliseconds to nanoseconds for already existing tables. We could add > this migration as part of compaction if you think that would be useful, but > that would obviously make the change more complex. > I've added a new EnvironmentEdge method "currentTimeNano()" that uses > [java.time.Instant|https://docs.oracle.com/javase/8/docs/api/java/time/Instant.html] > to get time in nanoseconds which means it will only work with Java 8. The > idea is to gradually replace all places where "EnvironmentEdge.currentTime()" > is used to have HBase working purely with nanoseconds (which is a > prerequisite for HBASE-14070). Also, I've refactored ScanInfo and > PartitionedMobCompactor to expect TableDescriptor as an argument which makes > code a little cleaner and easier to extend. > Couple more points: > - column family TTL (specified in seconds) and > "hbase.hstore.time.to.purge.deletes" (specified in milliseconds) options > don't need to be changed, those are adjusted automatically. > - Per cell TTL needs to be scaled by clients accordingly after > "NANOSECOND_TIMESTAMPS" table attribute is specified. > Looking for everyone's feedback to know if that's a worthwhile direction. > Will add more comprehensive tests in a later patch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20952) Re-visit the WAL API
[
https://issues.apache.org/jira/browse/HBASE-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16686063#comment-16686063
]
Hudson commented on HBASE-20952:
Results for branch HBASE-20952
[build #49 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/49/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/49//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/49//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/49//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Re-visit the WAL API
>
>
> Key: HBASE-20952
> URL: https://issues.apache.org/jira/browse/HBASE-20952
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Josh Elser
>Priority: Major
> Attachments: 20952.v1.txt
>
>
> Take a step back from the current WAL implementations and think about what an
> HBase WAL API should look like. What are the primitive calls that we require
> to guarantee durability of writes with a high degree of performance?
> The API needs to take the current implementations into consideration. We
> should also have a mind for what is happening in the Ratis LogService (but
> the LogService should not dictate what HBase's WAL API looks like RATIS-272).
> Other "systems" inside of HBase that use WALs are replication and
> backup Replication has the use-case for "tail"'ing the WAL which we
> should provide via our new API. B doesn't do anything fancy (IIRC). We
> should make sure all consumers are generally going to be OK with the API we
> create.
> The API may be "OK" (or OK in a part). We need to also consider other methods
> which were "bolted" on such as {{AbstractFSWAL}} and
> {{WALFileLengthProvider}}. Other corners of "WAL use" (like the
> {{WALSplitter}} should also be looked at to use WAL-APIs only).
> We also need to make sure that adequate interface audience and stability
> annotations are chosen.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21468) separate workers for meta table is not working
[ https://issues.apache.org/jira/browse/HBASE-21468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allan Yang updated HBASE-21468: --- Resolution: Fixed Fix Version/s: 2.1.2 2.0.3 Status: Resolved (was: Patch Available) > separate workers for meta table is not working > -- > > Key: HBASE-21468 > URL: https://issues.apache.org/jira/browse/HBASE-21468 > Project: HBase > Issue Type: Bug >Affects Versions: 2.1.1, 2.0.2 >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > Fix For: 2.0.3, 2.1.2 > > Attachments: HBASE-21468.branch-2.0.001.patch > > > This is an addendum for HBASE-21423, since HBASE-21423 is already closed, the > QA won't be triggered. > It is my mistake that the separate workers for meta table is not working, > since when polling from queue, the onlyUrgent flag is not passed in. > And for some UT that only require one worker thread, urgent workers should > set to 0 to ensure there is one worker at time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21440) Assign procedure on the crashed server is not properly interrupted
[ https://issues.apache.org/jira/browse/HBASE-21440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16686060#comment-16686060 ] Allan Yang commented on HBASE-21440: +1 on v4 patch. Maybe it is HBASE-21468 making these test flaky, I have committed HBASE-21468 to branch-2.0 and branch-2.1. [~an...@apache.org], you can re-trigger a QA run here. Sorry for that. > Assign procedure on the crashed server is not properly interrupted > -- > > Key: HBASE-21440 > URL: https://issues.apache.org/jira/browse/HBASE-21440 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.2 >Reporter: Ankit Singhal >Assignee: Ankit Singhal >Priority: Major > Attachments: HBASE-21440.branch-2.0.001.patch, > HBASE-21440.branch-2.0.002.patch, HBASE-21440.branch-2.0.003.patch, > HBASE-21440.branch-2.0.004.patch > > > When the server crashes, it's SCP checks if there is already a procedure > assigning the region on this crashed server. If we found one, SCP will just > interrupt the already running AssignProcedure by calling remoteCallFailed > which internally just changes the region node state to OFFLINE and send the > procedure back with transition queue state for assignment with a new plan. > But, due to the race condition between the calling of the remoteCallFailed > and current state of the already running assign > procedure(REGION_TRANSITION_FINISH: where the region is already opened), it > is possible that assign procedure goes ahead in updating the regionStateNode > to OPEN on a crashed server. > As SCP had already skipped this region for assignment as it was relying on > existing assign procedure to do the right thing, this whole confusion leads > region to a not accessible state. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21468) separate workers for meta table is not working
[
https://issues.apache.org/jira/browse/HBASE-21468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16686050#comment-16686050
]
Allan Yang commented on HBASE-21468:
{quote}
{quote}
[~anoop.hbase], yes, there is one called
''hbase.master.urgent.procedure.threads'', and the default value is 1, we only
need one urgent worker to handle meta table.
I will commit this one to branch-2.0 and branch-2.1 soon
> separate workers for meta table is not working
> --
>
> Key: HBASE-21468
> URL: https://issues.apache.org/jira/browse/HBASE-21468
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.1.1, 2.0.2
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Major
> Attachments: HBASE-21468.branch-2.0.001.patch
>
>
> This is an addendum for HBASE-21423, since HBASE-21423 is already closed, the
> QA won't be triggered.
> It is my mistake that the separate workers for meta table is not working,
> since when polling from queue, the onlyUrgent flag is not passed in.
> And for some UT that only require one worker thread, urgent workers should
> set to 0 to ensure there is one worker at time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21255) [acl] Refactor TablePermission into three classes (Global, Namespace, Table)
[
https://issues.apache.org/jira/browse/HBASE-21255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16686047#comment-16686047
]
Reid Chan edited comment on HBASE-21255 at 11/14/18 3:33 AM:
-
bq. address code duplication
As suggestion, v9 is much cleaner.
And all related tests passed locally, let's wait QA.
Updated RB as well.
was (Author: reidchan):
bq. address code duplication
As suggestion, it is much cleaner.
And all related tests passed locally, let's wait QA.
Updated RB as well.
> [acl] Refactor TablePermission into three classes (Global, Namespace, Table)
>
>
> Key: HBASE-21255
> URL: https://issues.apache.org/jira/browse/HBASE-21255
> Project: HBase
> Issue Type: Improvement
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21225.master.001.patch,
> HBASE-21225.master.002.patch, HBASE-21225.master.007.patch,
> HBASE-21225.master.008.patch, HBASE-21225.master.009.patch,
> HBASE-21255.master.003.patch, HBASE-21255.master.004.patch,
> HBASE-21255.master.005.patch, HBASE-21255.master.006.patch,
> HBASE-21255.master.006.patch
>
>
> A TODO in {{TablePermission.java}}
> {code:java}
> //TODO refactor this class
> //we need to refacting this into three classes (Global, Table, Namespace)
> {code}
> Change Notes:
> * Divide origin TablePermission into three classes GlobalPermission,
> NamespacePermission, TablePermission
> * New UserPermission consists of a user name(string, not byte[], for
> convenience) and a permission in one of [Global, Namespace, Table]Permission.
> * Rename TableAuthManager to AuthManager(it is IA.P), and rename some
> methods for readability.
> * Make PermissionCache thread safe, and the ListMultiMap is changed to Set.
> * User cache and group cache in AuthManager is combined together.
> * Wire proto is kept, BC should be under guarantee.
> * Fix HBASE-21390.
> * Resolve a small {{TODO}} global entry should be handled differently in
> AccessControlLists
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21255) [acl] Refactor TablePermission into three classes (Global, Namespace, Table)
[
https://issues.apache.org/jira/browse/HBASE-21255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16686047#comment-16686047
]
Reid Chan commented on HBASE-21255:
---
bq. address code duplication
As suggestion, it is much cleaner.
And all related tests passed locally, let's wait QA.
Updated RB as well.
> [acl] Refactor TablePermission into three classes (Global, Namespace, Table)
>
>
> Key: HBASE-21255
> URL: https://issues.apache.org/jira/browse/HBASE-21255
> Project: HBase
> Issue Type: Improvement
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21225.master.001.patch,
> HBASE-21225.master.002.patch, HBASE-21225.master.007.patch,
> HBASE-21225.master.008.patch, HBASE-21225.master.009.patch,
> HBASE-21255.master.003.patch, HBASE-21255.master.004.patch,
> HBASE-21255.master.005.patch, HBASE-21255.master.006.patch,
> HBASE-21255.master.006.patch
>
>
> A TODO in {{TablePermission.java}}
> {code:java}
> //TODO refactor this class
> //we need to refacting this into three classes (Global, Table, Namespace)
> {code}
> Change Notes:
> * Divide origin TablePermission into three classes GlobalPermission,
> NamespacePermission, TablePermission
> * New UserPermission consists of a user name(string, not byte[], for
> convenience) and a permission in one of [Global, Namespace, Table]Permission.
> * Rename TableAuthManager to AuthManager(it is IA.P), and rename some
> methods for readability.
> * Make PermissionCache thread safe, and the ListMultiMap is changed to Set.
> * User cache and group cache in AuthManager is combined together.
> * Wire proto is kept, BC should be under guarantee.
> * Fix HBASE-21390.
> * Resolve a small {{TODO}} global entry should be handled differently in
> AccessControlLists
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21469) Re-visit post* hooks in DDL operations
[ https://issues.apache.org/jira/browse/HBASE-21469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16686045#comment-16686045 ] Allan Yang commented on HBASE-21469: [~anoop.hbase], yes, I want to solve the RPC time out issue here mainly. > Re-visit post* hooks in DDL operations > -- > > Key: HBASE-21469 > URL: https://issues.apache.org/jira/browse/HBASE-21469 > Project: HBase > Issue Type: Bug >Affects Versions: 2.1.1, 2.0.2 >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > > I have some discuss in HBASE-19953 from > [here|https://issues.apache.org/jira/browse/HBASE-19953?focusedCommentId=16673126=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16673126] > In HBASE-19953,[~elserj] want to make sure that the post* hooks are called > only when the procedures finish.But it accidentally turns modifytable and > truncate table request into a sync call, which make clients RPC timeout > easily on big tables. > We should re-visit those postxxx hooks in DDL operations, because they are > now not consistent now: > For DDLs other than modifytable and truncate table, although the call will > wait on the latch, the latch is actually released just after prepare state, > so we still call postxxx hooks before the operation finish. > For DDLs of modifytable and truncate, the latch is only released after the > whole procedure finish. So the effort works(but will cause RPC timeout) > I think these latches are designed for compatibility with 1.x clients. Take > ModifyTable for example, in 1.x, we use admin.getAlterStauts() to check the > alter status, but in 2.x, this method is deprecated and returning inaccurate > result, we have to make 1.x client in a sync wait. > And for the semantics of postxxx hooks in 1.x, we will call them after the > corresponding DDL request return, but actually, the DDL request may not > finished also since we don't want for region assignment. > So, here, we need to discuss the semantics of postxxx hooks in DDL > operations, we need to make it consistent in every DDL operations, do we > really need to make sure this hooks being called only after the operation > finish? What's more, we have postCompletedxxx hooks for that need. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21255) [acl] Refactor TablePermission into three classes (Global, Namespace, Table)
[
https://issues.apache.org/jira/browse/HBASE-21255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reid Chan updated HBASE-21255:
--
Attachment: HBASE-21225.master.009.patch
> [acl] Refactor TablePermission into three classes (Global, Namespace, Table)
>
>
> Key: HBASE-21255
> URL: https://issues.apache.org/jira/browse/HBASE-21255
> Project: HBase
> Issue Type: Improvement
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21225.master.001.patch,
> HBASE-21225.master.002.patch, HBASE-21225.master.007.patch,
> HBASE-21225.master.008.patch, HBASE-21225.master.009.patch,
> HBASE-21255.master.003.patch, HBASE-21255.master.004.patch,
> HBASE-21255.master.005.patch, HBASE-21255.master.006.patch,
> HBASE-21255.master.006.patch
>
>
> A TODO in {{TablePermission.java}}
> {code:java}
> //TODO refactor this class
> //we need to refacting this into three classes (Global, Table, Namespace)
> {code}
> Change Notes:
> * Divide origin TablePermission into three classes GlobalPermission,
> NamespacePermission, TablePermission
> * New UserPermission consists of a user name(string, not byte[], for
> convenience) and a permission in one of [Global, Namespace, Table]Permission.
> * Rename TableAuthManager to AuthManager(it is IA.P), and rename some
> methods for readability.
> * Make PermissionCache thread safe, and the ListMultiMap is changed to Set.
> * User cache and group cache in AuthManager is combined together.
> * Wire proto is kept, BC should be under guarantee.
> * Fix HBASE-21390.
> * Resolve a small {{TODO}} global entry should be handled differently in
> AccessControlLists
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21410) A helper page that help find all problematic regions and procedures
[ https://issues.apache.org/jira/browse/HBASE-21410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16686033#comment-16686033 ] Jingyun Tian edited comment on HBASE-21410 at 11/14/18 2:55 AM: [~stack] Thank you. And Sorry for my careless. I misunderstood your reply 2 days ago as a +1. I will pay more attention next time. For branch-2 and master, Guanghao Zhang already pushed the patch a few days ago. was (Author: tianjingyun): [~stack] Thank you. And Sorry for my careless. I misunderstood your reply 2 days ago as a +1. I will pay more attention next time. > A helper page that help find all problematic regions and procedures > --- > > Key: HBASE-21410 > URL: https://issues.apache.org/jira/browse/HBASE-21410 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0, 2.2.0, 2.1.1 >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Major > Fix For: 3.0.0, 2.1.0, 2.2.0 > > Attachments: HBASE-21410.branch-2.1.001.patch, > HBASE-21410.branch-2.1.002.patch, HBASE-21410.master.001.patch, > HBASE-21410.master.002.patch, HBASE-21410.master.003.patch, > HBASE-21410.master.004.patch, Screenshot from 2018-10-30 19-06-21.png, > Screenshot from 2018-10-30 19-06-42.png, Screenshot from 2018-10-31 > 10-11-38.png, Screenshot from 2018-10-31 10-11-56.png, Screenshot from > 2018-11-01 17-56-02.png, Screenshot from 2018-11-01 17-56-15.png > > > *This page is mainly focus on finding the regions stuck in some state that > cannot be assigned. My proposal of the page is as follows:* > !Screenshot from 2018-10-30 19-06-21.png! > *From this page we can see all regions in RIT queue and their related > procedures. If we can determine that these regions' state are abnormal, we > can click the link 'Procedures as TXT' to get a full list of procedure IDs to > bypass them. Then click 'Regions as TXT' to get a full list of encoded region > names to assign.* > !Screenshot from 2018-10-30 19-06-42.png! > *Some region names are covered by the navigator bar, I'll fix it later.* -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21410) A helper page that help find all problematic regions and procedures
[ https://issues.apache.org/jira/browse/HBASE-21410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16686033#comment-16686033 ] Jingyun Tian commented on HBASE-21410: -- [~stack] Thank you. And Sorry for my careless. I misunderstood your reply 2 days ago as a +1. I will pay more attention next time. > A helper page that help find all problematic regions and procedures > --- > > Key: HBASE-21410 > URL: https://issues.apache.org/jira/browse/HBASE-21410 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0, 2.2.0, 2.1.1 >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Major > Fix For: 3.0.0, 2.1.0, 2.2.0 > > Attachments: HBASE-21410.branch-2.1.001.patch, > HBASE-21410.branch-2.1.002.patch, HBASE-21410.master.001.patch, > HBASE-21410.master.002.patch, HBASE-21410.master.003.patch, > HBASE-21410.master.004.patch, Screenshot from 2018-10-30 19-06-21.png, > Screenshot from 2018-10-30 19-06-42.png, Screenshot from 2018-10-31 > 10-11-38.png, Screenshot from 2018-10-31 10-11-56.png, Screenshot from > 2018-11-01 17-56-02.png, Screenshot from 2018-11-01 17-56-15.png > > > *This page is mainly focus on finding the regions stuck in some state that > cannot be assigned. My proposal of the page is as follows:* > !Screenshot from 2018-10-30 19-06-21.png! > *From this page we can see all regions in RIT queue and their related > procedures. If we can determine that these regions' state are abnormal, we > can click the link 'Procedures as TXT' to get a full list of procedure IDs to > bypass them. Then click 'Regions as TXT' to get a full list of encoded region > names to assign.* > !Screenshot from 2018-10-30 19-06-42.png! > *Some region names are covered by the navigator bar, I'll fix it later.* -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21464) Splitting blocked with meta NSRE during split transaction
[
https://issues.apache.org/jira/browse/HBASE-21464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16686027#comment-16686027
]
Andrew Purtell commented on HBASE-21464:
Hmm, we leak Table references in MetaTableAccessor. Might not be related but
let me open a local branch and start making changes...
> Splitting blocked with meta NSRE during split transaction
> -
>
> Key: HBASE-21464
> URL: https://issues.apache.org/jira/browse/HBASE-21464
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.5.0, 1.4.3, 1.4.4, 1.4.5, 1.4.6, 1.4.8, 1.4.7
>Reporter: Andrew Purtell
>Priority: Blocker
> Fix For: 1.5.0, 1.4.9
>
>
> Splitting is blocked during split transaction. The split worker is trying to
> update meta but isn't able to relocate it after NSRE:
> {noformat}
> 2018-11-09 17:50:45,277 INFO
> [regionserver/ip-172-31-5-92.us-west-2.compute.internal/172.31.5.92:8120-splits-1541785709434]
> client.RpcRetryingCaller: Call exception, tries=13, retries=350,
> started=88590 ms ago, cancelled=false,
> msg=org.apache.hadoop.hbase.NotServingRegionException: Region hbase:meta,,1
> is not online on ip-172-31-13-83.us-west-2.compute.internal,8120,1541785618832
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3088)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1271)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:2198)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:36617)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2396)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:297)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:277)row
> 'test,,1541785709452.5ba6596f0050c2dab969d152829227c6.44' on table
> 'hbase:meta' at region=hbase:meta,1.1588230740,
> hostname=ip-172-31-15-225.us-west-2.compute.internal,8120,1541785640586,
> seqNum=0{noformat}
> Clients, in this case YCSB, are hung with part of the keyspace missing:
> {noformat}
> 2018-11-09 17:51:06,033 DEBUG [hconnection-0x5739e567-shared--pool1-t165]
> client.ConnectionManager$HConnectionImplementation: locateRegionInMeta
> parentTable=hbase:meta, metaLocation=, attempt=14 of 35 failed; retrying
> after sleep of 20158 because: No server address listed in hbase:meta for
> region
> test,user307326104267982763,1541785754600.ef90030b05cb02305b75e9bfbc3ee081.
> containing row user3301635648728421323{noformat}
> Balancing cannot run indefinitely because the split transaction is stuck
> {noformat}
> 2018-11-09 17:49:55,478 DEBUG
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=8100] master.HMaster:
> Not running balancer because 3 region(s) in transition:
> [{ef90030b05cb02305b75e9bfbc3ee081 state=SPLITTING_NEW, ts=1541785754606,
> server=ip-172-31-5-92.us-west-2.compute.internal,8120,1541785626417},
> {5ba6596f0050c2dab969d152829227c6 state=SPLITTING, ts=1541785754606,
> server=ip-172-31-5-92.us-west-2.compute{noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21034) Add new throttle type: read/write capacity unit
[ https://issues.apache.org/jira/browse/HBASE-21034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Mei updated HBASE-21034: --- Attachment: HBASE-21034.master.004.patch > Add new throttle type: read/write capacity unit > --- > > Key: HBASE-21034 > URL: https://issues.apache.org/jira/browse/HBASE-21034 > Project: HBase > Issue Type: Sub-task >Affects Versions: 3.0.0, 2.2.0 >Reporter: Yi Mei >Assignee: Yi Mei >Priority: Major > Attachments: HBASE-21034.master.001.patch, > HBASE-21034.master.002.patch, HBASE-21034.master.003.patch, > HBASE-21034.master.004.patch > > > Add new throttle type: read/write capacity unit like DynamoDB. > One read capacity unit represents that read up to 1K data per time unit. If > data size is more than 1K, then consume additional read capacity units. > One write capacity unit represents that one write for an item up to 1 KB in > size per time unit. If data size is more than 1K, then consume additional > write capacity units. > For example, 100 read capacity units per second means that, HBase user can > read 100 times for 1K data in every second, or 50 times for 2K data in every > second and so on. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21464) Splitting blocked with meta NSRE during split transaction
[
https://issues.apache.org/jira/browse/HBASE-21464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16686000#comment-16686000
]
Andrew Purtell commented on HBASE-21464:
On the regionserver doing the splitting, we get to the point where its' time to
update meta at 18:07:35
{noformat}
2018-11-09 18:07:35,704 DEBUG
[regionserver/ip-172-31-13-83.us-west-2.compute.internal/172.31.13.83:8120-splits-1541786530557]
regionserver.SplitTransaction: Split storefiles for \
region
test,user4112339446054425864,1541786730764.2802f0bfbe9e7d88d530c16539f95cfd.
Daughter A: 6 storefiles, Daughter B: 6 storefiles.
...
2018-11-09 18:07:35,757 DEBUG
[regionserver/ip-172-31-13-83.us-west-2.compute.internal/172.31.13.83:8120-splits-1541786530557]
ipc.BlockingRpcConnection: Connecting to
ip-172-31-5-92.us-west-2.compute.internal/172.31.5.92:8120
...
2018-11-09 18:08:14,168 INFO
[regionserver/ip-172-31-13-83.us-west-2.compute.internal/172.31.13.83:8120-splits-1541786530557]
client.RpcRetryingCaller: Call exception, tries=10, retries=350, started=38412
ms ago, cancelled=false, msg=org.apache.hadoop.hbase.NotServingRegionException:
Region hbase:meta,,1 is not online on
ip-172-31-5-92.us-west-2.compute.internal,8120,1541786481463{noformat}
However META recently moved. It's not on ip-172-31-5-92 any longer. It moved to
ip-172-31-15-225 a minute prior, at 18:06:24.
>From master
{noformat}
2018-11-09 18:06:24,690 DEBUG [AM.ZK.Worker-pool5-t64]
master.AssignmentManager: Znode hbase:meta,,1.1588230740 deleted, state:
{1588230740 state=OPEN, ts=1541786784688,
server=ip-172-31-15-225.us-west-2.compute.internal,8120,1541786485409}
{noformat}
>From regionserver ip-172-31-15-225:
{noformat}
2018-11-09 18:06:24,686 DEBUG [PostOpenDeployTasks:1588230740]
regionserver.HRegionServer: Finished post open deploy task for
hbase:meta,,1.1588230740
...
2018-11-09 18:06:24,688 DEBUG [RS_OPEN_META-ip-172-31-15-225:8120-0]
handler.OpenRegionHandler: Opened hbase:meta,,1.1588230740 on
ip-172-31-15-225.us-west-2.compute.internal,8120,1541786485409
{noformat}
The stuck split happens a minute later after META is redeployed and is live on
ip-172-31-15-225.
The relevant code attempting the update is in SplitTransactionImpl.
{code:java}
if (!testing && useZKForAssignment) {
if (metaEntries == null || metaEntries.isEmpty()) {
MetaTableAccessor.splitRegion(server.getConnection(),
parent.getRegionInfo(), daughterRegions.getFirst().getRegionInfo(),
daughterRegions.getSecond().getRegionInfo(), server.getServerName(),
parent.getTableDesc().getRegionReplication());
} else {
offlineParentInMetaAndputMetaEntries(server.getConnection(),
parent.getRegionInfo(), daughterRegions.getFirst().getRegionInfo(),
daughterRegions
.getSecond().getRegionInfo(), server.getServerName(), metaEntries,
parent.getTableDesc().getRegionReplication());
}
{code}
(and not relevant, this bit tells the master directly if using zk-less
assignment)
{code:java}
} else if (services != null && !useZKForAssignment) {
if (!services.reportRegionStateTransition(TransitionCode.SPLIT_PONR,
parent.getRegionInfo(), hri_a, hri_b)) {
// Passed PONR, let SSH clean it up
throw new IOException("Failed to notify master that split passed PONR: "
+ parent.getRegionInfo().getRegionNameWithoutKeyAsString());
}
}
{code}
So we either call MetaTableAccessor.splitRegion here or
MetaTableAccessor.mutateMetaTable via offlineParentInMetaAndputMetaEntries.
The question is why the connection (acquired by server.getConnection()) used by
MetaTableAccessor is not relocating META's location.
There is one change to MetaTableAccessor between 1.4.2, which does not
reproduce, and 1.4.3, which does reproduce the problem, but looking at it I
can't see how it it would be related.
{noformat}
commit 0d8fee2158e08bc6d0907d4abbe1215eaded6ce3
Author: Pankaj Kumar
Date: Thu Dec 7 22:51:01 2017 +0530
HBASE-19364, Truncate_preserve fails with table when replica region > 1
{noformat}
Looking at ConnectionManager now.
> Splitting blocked with meta NSRE during split transaction
> -
>
> Key: HBASE-21464
> URL: https://issues.apache.org/jira/browse/HBASE-21464
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.5.0, 1.4.3, 1.4.4, 1.4.5, 1.4.6, 1.4.8, 1.4.7
>Reporter: Andrew Purtell
>Priority: Blocker
> Fix For: 1.5.0, 1.4.9
>
>
> Splitting is blocked during split transaction. The split worker is trying to
> update meta but isn't able to relocate it after NSRE:
> {noformat}
> 2018-11-09 17:50:45,277 INFO
> [regionserver/ip-172-31-5-92.us-west-2.compute.internal/172.31.5.92:8120-splits-1541785709434]
>
[jira] [Commented] (HBASE-21405) [DOC] Add Details about Output of "status 'replication'"
[
https://issues.apache.org/jira/browse/HBASE-21405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685964#comment-16685964
]
Roland Teague commented on HBASE-21405:
---
[~wchevreuil] I have the following definitions for the metrics.
{noformat}
Source:
ageOfLastShippedOp - Age of the last operation that was shipped by the source
sizeOfLogQueue - Current size of the queue of logs to replicate, excluding
the one being processed at the moment
Sink:
ageOfLastAppliedOp - Age of the last operation that was applied by the sink
Other related metrics:
appliedBatchesRate - Rate of batches (of operations) applied by the sink
appliedOpsRate - Rate of operations applied by the sink
logEditsFilteredRate - Rate of log entries filtered by the source
logEditsReadRate - Rate of log entries (can be multiple Puts) read from the
logs
shippedBatchesRate - Rate of shipped batches by the source
shippedOpsRate - Rate of shipped operations by the source
{noformat}
> [DOC] Add Details about Output of "status 'replication'"
>
>
> Key: HBASE-21405
> URL: https://issues.apache.org/jira/browse/HBASE-21405
> Project: HBase
> Issue Type: Improvement
> Components: documentation
>Affects Versions: 1.2.0, 1.4.0, 2.0.0
>Reporter: Daisuke Kobayashi
>Assignee: Wellington Chevreuil
>Priority: Minor
>
> Add more information about the meaning of each metric on to
> [http://hbase.apache.org/book.html#_monitoring_replication_status].
> {noformat}
> SOURCE:
> PeerID
> AgeOfLastShippedOp
> SizeOfLogQueue
> TimeStampsOfLastShippedOp
> Replication Lag
> SINK
> AgeOfLastAppliedOp
> TimeStampsOfLastAppliedOp
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21465) Retry on reportRegionStateTransition can lead to unexpected errors
[ https://issues.apache.org/jira/browse/HBASE-21465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685971#comment-16685971 ] Duo Zhang commented on HBASE-21465: --- Or maybe the v4 is better, where we do not change the logic of writing sequence id file. FWIW, we always need the logic in v4 patch even if we go v3, as we need to consider rolling upgrading, where the region servers in old version still run the old code, where we write sequence id file before writing open marker. > Retry on reportRegionStateTransition can lead to unexpected errors > -- > > Key: HBASE-21465 > URL: https://issues.apache.org/jira/browse/HBASE-21465 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-21465-UT.patch, HBASE-21465-v1.patch, > HBASE-21465-v2.patch, HBASE-21465-v3.patch, HBASE-21465-v4.patch, > HBASE-21465.patch > > > It is possible that the reportRegionStateTransition method is succeeded at > master side, but before returning the the region server, the rpc connection > is broken, or the master restarts. So next when the region server try > again,we will find that the state for the region and the TRSP is not correct, > and can lead to a RS abort or something even worse. > We should be able to determine whether a reportRegionStateTransition call is > just a retry and has already been succeeded, and just ignore it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21465) Retry on reportRegionStateTransition can lead to unexpected errors
[ https://issues.apache.org/jira/browse/HBASE-21465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-21465: -- Attachment: HBASE-21465-v4.patch > Retry on reportRegionStateTransition can lead to unexpected errors > -- > > Key: HBASE-21465 > URL: https://issues.apache.org/jira/browse/HBASE-21465 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-21465-UT.patch, HBASE-21465-v1.patch, > HBASE-21465-v2.patch, HBASE-21465-v3.patch, HBASE-21465-v4.patch, > HBASE-21465.patch > > > It is possible that the reportRegionStateTransition method is succeeded at > master side, but before returning the the region server, the rpc connection > is broken, or the master restarts. So next when the region server try > again,we will find that the state for the region and the TRSP is not correct, > and can lead to a RS abort or something even worse. > We should be able to determine whether a reportRegionStateTransition call is > just a retry and has already been succeeded, and just ignore it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21465) Retry on reportRegionStateTransition can lead to unexpected errors
[ https://issues.apache.org/jira/browse/HBASE-21465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685935#comment-16685935 ] Duo Zhang commented on HBASE-21465: --- Let's see if the patch could solve the problem. I moved the writing sequence id file after writing the open marker. Not sure if this could break any UTs. > Retry on reportRegionStateTransition can lead to unexpected errors > -- > > Key: HBASE-21465 > URL: https://issues.apache.org/jira/browse/HBASE-21465 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-21465-UT.patch, HBASE-21465-v1.patch, > HBASE-21465-v2.patch, HBASE-21465-v3.patch, HBASE-21465.patch > > > It is possible that the reportRegionStateTransition method is succeeded at > master side, but before returning the the region server, the rpc connection > is broken, or the master restarts. So next when the region server try > again,we will find that the state for the region and the TRSP is not correct, > and can lead to a RS abort or something even worse. > We should be able to determine whether a reportRegionStateTransition call is > just a retry and has already been succeeded, and just ignore it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21465) Retry on reportRegionStateTransition can lead to unexpected errors
[ https://issues.apache.org/jira/browse/HBASE-21465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-21465: -- Attachment: HBASE-21465-v3.patch > Retry on reportRegionStateTransition can lead to unexpected errors > -- > > Key: HBASE-21465 > URL: https://issues.apache.org/jira/browse/HBASE-21465 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-21465-UT.patch, HBASE-21465-v1.patch, > HBASE-21465-v2.patch, HBASE-21465-v3.patch, HBASE-21465.patch > > > It is possible that the reportRegionStateTransition method is succeeded at > master side, but before returning the the region server, the rpc connection > is broken, or the master restarts. So next when the region server try > again,we will find that the state for the region and the TRSP is not correct, > and can lead to a RS abort or something even worse. > We should be able to determine whether a reportRegionStateTransition call is > just a retry and has already been succeeded, and just ignore it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21477) [documentation] doc against an hbase2 client connecting to an hbase1 cluster
stack created HBASE-21477: - Summary: [documentation] doc against an hbase2 client connecting to an hbase1 cluster Key: HBASE-21477 URL: https://issues.apache.org/jira/browse/HBASE-21477 Project: HBase Issue Type: Bug Components: documentation Reporter: stack Currently nothing in the refguide that doing this is a bad idea (https://stackoverflow.com/questions/46468430/hbase-client-2-0-x-error). Should at least say don't do it if only to quote when folks try. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21476) Support for nanosecond timestamps
Andrey Elenskiy created HBASE-21476: --- Summary: Support for nanosecond timestamps Key: HBASE-21476 URL: https://issues.apache.org/jira/browse/HBASE-21476 Project: HBase Issue Type: New Feature Affects Versions: 2.1.1 Reporter: Andrey Elenskiy Assignee: Andrey Elenskiy Attachments: nanosecond_timestamps_v1.patch Introducing a new table attribute "NANOSECOND_TIMESTAMPS" to tell HBase to handle timestamps with nanosecond precision. This is useful for applications that timestamp updates at the source with nanoseconds and still want features like column family TTL and "hbase.hstore.time.to.purge.deletes" to work. The attribute should be specified either on new tables or on existing tables which have timestamps only with nanosecond precision. There's no migration from milliseconds to nanoseconds for already existing tables. We could add this migration as part of compaction if you think that would be useful, but that would obviously make the change more complex. I've added a new EnvironmentEdge method "currentTimeNano()" that uses [java.time.Instant|https://docs.oracle.com/javase/8/docs/api/java/time/Instant.html] to get time in nanoseconds which means it will only work with Java 8. The idea is to gradually replace all places where "EnvironmentEdge.currentTime()" is used to have HBase working purely with nanoseconds (which is a prerequisite for HBASE-14070). Also, I've refactored ScanInfo and PartitionedMobCompactor to expect TableDescriptor as an argument which makes code a little cleaner and easier to extend. Couple more points: - column family TTL (specified in seconds) and "hbase.hstore.time.to.purge.deletes" (specified in milliseconds) options don't need to be changed, those are adjusted automatically. - Per cell TTL needs to be scaled by clients accordingly after "NANOSECOND_TIMESTAMPS" table attribute is specified. Looking for everyone's feedback to know if that's a worthwhile direction. Will add more comprehensive tests in a later patch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21465) Retry on reportRegionStateTransition can lead to unexpected errors
[
https://issues.apache.org/jira/browse/HBASE-21465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685891#comment-16685891
]
Hudson commented on HBASE-21465:
Results for branch branch-2
[build #1501 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1501/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1501//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1501//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1501//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Retry on reportRegionStateTransition can lead to unexpected errors
> --
>
> Key: HBASE-21465
> URL: https://issues.apache.org/jira/browse/HBASE-21465
> Project: HBase
> Issue Type: Sub-task
> Components: amv2
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21465-UT.patch, HBASE-21465-v1.patch,
> HBASE-21465-v2.patch, HBASE-21465.patch
>
>
> It is possible that the reportRegionStateTransition method is succeeded at
> master side, but before returning the the region server, the rpc connection
> is broken, or the master restarts. So next when the region server try
> again,we will find that the state for the region and the TRSP is not correct,
> and can lead to a RS abort or something even worse.
> We should be able to determine whether a reportRegionStateTransition call is
> just a retry and has already been succeeded, and just ignore it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21410) A helper page that help find all problematic regions and procedures
[ https://issues.apache.org/jira/browse/HBASE-21410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685846#comment-16685846 ] stack commented on HBASE-21410: --- I tried this new UI. Its really great. Much better than I expected. Thanks Jingyun Tian. Make sure it is in branch-2.1, branch-2, and master. Thanks. > A helper page that help find all problematic regions and procedures > --- > > Key: HBASE-21410 > URL: https://issues.apache.org/jira/browse/HBASE-21410 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0, 2.2.0, 2.1.1 >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Major > Fix For: 3.0.0, 2.1.0, 2.2.0 > > Attachments: HBASE-21410.branch-2.1.001.patch, > HBASE-21410.branch-2.1.002.patch, HBASE-21410.master.001.patch, > HBASE-21410.master.002.patch, HBASE-21410.master.003.patch, > HBASE-21410.master.004.patch, Screenshot from 2018-10-30 19-06-21.png, > Screenshot from 2018-10-30 19-06-42.png, Screenshot from 2018-10-31 > 10-11-38.png, Screenshot from 2018-10-31 10-11-56.png, Screenshot from > 2018-11-01 17-56-02.png, Screenshot from 2018-11-01 17-56-15.png > > > *This page is mainly focus on finding the regions stuck in some state that > cannot be assigned. My proposal of the page is as follows:* > !Screenshot from 2018-10-30 19-06-21.png! > *From this page we can see all regions in RIT queue and their related > procedures. If we can determine that these regions' state are abnormal, we > can click the link 'Procedures as TXT' to get a full list of procedure IDs to > bypass them. Then click 'Regions as TXT' to get a full list of encoded region > names to assign.* > !Screenshot from 2018-10-30 19-06-42.png! > *Some region names are covered by the navigator bar, I'll fix it later.* -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21475) Put mutation (having TTL set) added via co-processor is retrieved even after TTL expires
[
https://issues.apache.org/jira/browse/HBASE-21475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685739#comment-16685739
]
Hadoop QA commented on HBASE-21475:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
8s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
17s{color} | {color:red} hbase-server: The patch generated 1 new + 212
unchanged - 0 fixed = 213 total (was 212) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
7s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 14s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}134m
22s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}174m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21475 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12948006/HBASE-21475.master.001.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux c5a9a4696768 4.4.0-134-generic #160~14.04.1-Ubuntu SMP Fri Aug
17 11:07:07 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 00acda3c58 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/15037/artifact/patchprocess/diff-checkstyle-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/15037/testReport/ |
| Max. process+thread count | 4703 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
[jira] [Commented] (HBASE-21255) [acl] Refactor TablePermission into three classes (Global, Namespace, Table)
[
https://issues.apache.org/jira/browse/HBASE-21255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685710#comment-16685710
]
Hadoop QA commented on HBASE-21255:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 8 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
5s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
28s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} The patch passed checkstyle in hbase-common {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
38s{color} | {color:red} hbase-client: The patch generated 6 new + 79 unchanged
- 30 fixed = 85 total (was 109) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
26s{color} | {color:green} hbase-server: The patch generated 0 new + 101
unchanged - 53 fixed = 101 total (was 154) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} The patch passed checkstyle in hbase-rsgroup {color}
|
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
43s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 24s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
48s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
29s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}151m 47s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
54s{color} | {color:green} hbase-rsgroup in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color}
[jira] [Commented] (HBASE-18273) hbase_rotate_log in hbase-daemon.sh script not working for some JDK
[
https://issues.apache.org/jira/browse/HBASE-18273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685700#comment-16685700
]
Hadoop QA commented on HBASE-18273:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
0s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
0s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 0m 42s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-18273 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12874799/HBASE-18273.2.patch |
| Optional Tests | dupname asflicense shellcheck shelldocs |
| uname | Linux 03569304ef41 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 00acda3c58 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| shellcheck | v0.4.4 |
| Max. process+thread count | 48 (vs. ulimit of 1) |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/15038/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> hbase_rotate_log in hbase-daemon.sh script not working for some JDK
> ---
>
> Key: HBASE-18273
> URL: https://issues.apache.org/jira/browse/HBASE-18273
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.6, 1.1.11, 2.0.0-alpha-1
>Reporter: Fangyuan Deng
>Assignee: Fangyuan Deng
>Priority: Major
> Attachments: HBASE-18273.0.patch, HBASE-18273.1.patch,
> HBASE-18273.2.patch
>
>
> When restarting a hbase process, hbase_rotate_log $HBASE_LOGGC will rotate
> GC logs.
> the code looks like this,
> if [ -f "$log" ]; then # rotate logs
> while [ $num -gt 1 ]; do
> prev=`expr $num - 1`
> [ -f "$log.$prev" ] && mv -f "$log.$prev" "$log.$num"
> num=$prev
> done
> But, some version JDK will add a suffix (.0) to the gc file, like
> hbase-xxx.gc.0, rather than hbase-xxx.gc.
> So I add a check before rotate,
> if [ ! -f "$log" ]; then #for some jdk, gc log has a postfix 0
> if [ -f "$log.0" ]; then
> mv -f "$log.0" "$log";
> fi
> fi
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-18273) hbase_rotate_log in hbase-daemon.sh script not working for some JDK
[ https://issues.apache.org/jira/browse/HBASE-18273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685698#comment-16685698 ] Peter Somogyi commented on HBASE-18273: --- [~whisper_deng], could you share what JDK has this behavior? > hbase_rotate_log in hbase-daemon.sh script not working for some JDK > --- > > Key: HBASE-18273 > URL: https://issues.apache.org/jira/browse/HBASE-18273 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.6, 1.1.11, 2.0.0-alpha-1 >Reporter: Fangyuan Deng >Assignee: Fangyuan Deng >Priority: Major > Attachments: HBASE-18273.0.patch, HBASE-18273.1.patch, > HBASE-18273.2.patch > > > When restarting a hbase process, hbase_rotate_log $HBASE_LOGGC will rotate > GC logs. > the code looks like this, > if [ -f "$log" ]; then # rotate logs > while [ $num -gt 1 ]; do > prev=`expr $num - 1` > [ -f "$log.$prev" ] && mv -f "$log.$prev" "$log.$num" > num=$prev > done > But, some version JDK will add a suffix (.0) to the gc file, like > hbase-xxx.gc.0, rather than hbase-xxx.gc. > So I add a check before rotate, > if [ ! -f "$log" ]; then #for some jdk, gc log has a postfix 0 > if [ -f "$log.0" ]; then > mv -f "$log.0" "$log"; > fi > fi -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21404) Master/RS navbar active state does not work
[
https://issues.apache.org/jira/browse/HBASE-21404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685689#comment-16685689
]
Hadoop QA commented on HBASE-21404:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
59s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}129m
37s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}139m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21404 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12948004/HBASE-21404.master.001.patch
|
| Optional Tests | dupname asflicense javac javadoc unit |
| uname | Linux 01f463a4b654 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 00acda3c58 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| whitespace |
https://builds.apache.org/job/PreCommit-HBASE-Build/15036/artifact/patchprocess/whitespace-eol.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/15036/testReport/ |
| Max. process+thread count | 4939 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/15036/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Master/RS navbar active state does not work
> ---
>
> Key: HBASE-21404
> URL: https://issues.apache.org/jira/browse/HBASE-21404
> Project: HBase
> Issue Type: Bug
> Components: UI
>Affects Versions: 3.0.0, 2.0.0
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HBASE-21404.master.001.patch,
> HBASE-21404.master.001.patch, master_after_patch.png, master_before.png,
> rs_after_patch.png, rs_before.png
>
>
> In master/rs web UI, the current active tab is not updated when user switches
> to any tab other than "Home" tab.
> For example: even though say if we are on "tabledetailed.jsp", the navbar
> does not update the active state of that tab. See master_before.png
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21473) RowIndexSeekerV1 may return cell with extra two \x00\x00 bytes which has no tags
[
https://issues.apache.org/jira/browse/HBASE-21473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685634#comment-16685634
]
Hadoop QA commented on HBASE-21473:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
2s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
58s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
8m 45s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
29s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}147m 13s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
50s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}191m 52s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestBlockEvictionFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21473 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12947993/HBASE-21473.v1.patch |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux cb87b260362d 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 00acda3c58 |
| maven | version: Apache Maven 3.5.4
[jira] [Commented] (HBASE-21440) Assign procedure on the crashed server is not properly interrupted
[
https://issues.apache.org/jira/browse/HBASE-21440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685628#comment-16685628
]
Ankit Singhal commented on HBASE-21440:
---
bq. Legit failures then in your opinion sir? Related?
Actually, test failures seem to be not related. (and also the code path in the
patch is not even accessed during these tests).
TestMasterFailoverWithProcedures is flaky, I can see it is getting failed
multiple times in night builds.
{code}
https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1065/
https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1064/
https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1068/
{code}
So asked [~te...@apache.org] help to verify TestMasterFailoverWithProcedures
with my patch, and he also finds it passing locally. Thanks, Ted.
TestMergeTableRegionsProcedure fails everytime in branch-2.0/2.1 at least.
{code}
https://builds.apache.org/job/HBase-Flaky-Tests/job/branch-2.0/1904/#showFailuresLink
{code}
[~stack] , WDYT, can this be committed now?
> Assign procedure on the crashed server is not properly interrupted
> --
>
> Key: HBASE-21440
> URL: https://issues.apache.org/jira/browse/HBASE-21440
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.2
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HBASE-21440.branch-2.0.001.patch,
> HBASE-21440.branch-2.0.002.patch, HBASE-21440.branch-2.0.003.patch,
> HBASE-21440.branch-2.0.004.patch
>
>
> When the server crashes, it's SCP checks if there is already a procedure
> assigning the region on this crashed server. If we found one, SCP will just
> interrupt the already running AssignProcedure by calling remoteCallFailed
> which internally just changes the region node state to OFFLINE and send the
> procedure back with transition queue state for assignment with a new plan.
> But, due to the race condition between the calling of the remoteCallFailed
> and current state of the already running assign
> procedure(REGION_TRANSITION_FINISH: where the region is already opened), it
> is possible that assign procedure goes ahead in updating the regionStateNode
> to OPEN on a crashed server.
> As SCP had already skipped this region for assignment as it was relying on
> existing assign procedure to do the right thing, this whole confusion leads
> region to a not accessible state.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20662) Increasing space quota on a violated table does not remove SpaceViolationPolicy.DISABLE enforcement
[
https://issues.apache.org/jira/browse/HBASE-20662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nihal Jain updated HBASE-20662:
---
Affects Version/s: 3.0.0
2.0.0
> Increasing space quota on a violated table does not remove
> SpaceViolationPolicy.DISABLE enforcement
> ---
>
> Key: HBASE-20662
> URL: https://issues.apache.org/jira/browse/HBASE-20662
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.0.0
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20662.master.001.patch,
> HBASE-20662.master.002.patch, HBASE-20662.master.003.patch,
> HBASE-20662.master.004.patch
>
>
> *Steps to reproduce*
> * Create a table and set quota with {{SpaceViolationPolicy.DISABLE}} having
> limit say 2MB
> * Now put rows until space quota is violated and table gets disabled
> * Next, increase space quota with limit say 4MB on the table
> * Now try putting a row into the table
> {code:java}
> private void testSetQuotaThenViolateAndFinallyIncreaseQuota() throws
> Exception {
> SpaceViolationPolicy policy = SpaceViolationPolicy.DISABLE;
> Put put = new Put(Bytes.toBytes("to_reject"));
> put.addColumn(Bytes.toBytes(SpaceQuotaHelperForTests.F1),
> Bytes.toBytes("to"),
> Bytes.toBytes("reject"));
> // Do puts until we violate space policy
> final TableName tn = writeUntilViolationAndVerifyViolation(policy, put);
> // Now, increase limit
> setQuotaLimit(tn, policy, 4L);
> // Put some row now: should not violate as quota limit increased
> verifyNoViolation(policy, tn, put);
> }
> {code}
> *Expected*
> We should be able to put data as long as newly set quota limit is not reached
> *Actual*
> We fail to put any new row even after increasing limit
> *Root cause*
> Increasing quota on a violated table triggers the table to be enabled, but
> since the table is already in violation, the system does not allow it to be
> enabled (may be thinking that a user is trying to enable it)
> *Relevant exception trace*
> {noformat}
> 2018-05-31 00:34:27,563 INFO [regionserver/root1-ThinkPad-T440p:0.Chore.1]
> client.HBaseAdmin$14(844): Started enable of
> testSetQuotaAndThenIncreaseQuotaWithDisable0
> 2018-05-31 00:34:27,571 DEBUG
> [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=42525]
> ipc.CallRunner(142): callId: 11 service: MasterService methodName:
> EnableTable size: 104 connection: 127.0.0.1:38030 deadline: 1527707127568,
> exception=org.apache.hadoop.hbase.security.AccessDeniedException: Enabling
> the table 'testSetQuotaAndThenIncreaseQuotaWithDisable0' is disallowed due to
> a violated space quota.
> 2018-05-31 00:34:27,571 ERROR [regionserver/root1-ThinkPad-T440p:0.Chore.1]
> quotas.RegionServerSpaceQuotaManager(210): Failed to disable space violation
> policy for testSetQuotaAndThenIncreaseQuotaWithDisable0. This table will
> remain in violation.
> org.apache.hadoop.hbase.security.AccessDeniedException:
> org.apache.hadoop.hbase.security.AccessDeniedException: Enabling the table
> 'testSetQuotaAndThenIncreaseQuotaWithDisable0' is disallowed due to a
> violated space quota.
> at org.apache.hadoop.hbase.master.HMaster$6.run(HMaster.java:2275)
> at
> org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:131)
> at org.apache.hadoop.hbase.master.HMaster.enableTable(HMaster.java:2258)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.enableTable(MasterRpcServices.java:725)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:100)
> at
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:90)
> at
>
[jira] [Commented] (HBASE-21243) Correct java-doc for the method RpcServer.getRemoteAddress()
[
https://issues.apache.org/jira/browse/HBASE-21243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685575#comment-16685575
]
Nihal Jain commented on HBASE-21243:
[~elserj] Can you please review [^HBASE-21243.master.002.patch]?
> Correct java-doc for the method RpcServer.getRemoteAddress()
>
>
> Key: HBASE-21243
> URL: https://issues.apache.org/jira/browse/HBASE-21243
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.0.0
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Trivial
> Labels: beginner, beginners, documentaion
> Fix For: 3.0.0
>
> Attachments: HBASE-21243.master.001.patch,
> HBASE-21243.master.002.patch
>
>
> Correct the java-doc for the method {{RpcServer.getRemoteAddress()}}.
> Currently it look like as below:
> {code:java}
> /**
>* @return Address of remote client if a request is ongoing, else null
>*/
> public static Optional getRemoteAddress() {
> return getCurrentCall().map(RpcCall::getRemoteAddress);
> }
> {code}
> Contrary to the doc the method will never return null. Rather it may return
> an empty Optional.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21297) ModifyTableProcedure can throw TNDE instead of IOE in case of REGION_REPLICATION change
[
https://issues.apache.org/jira/browse/HBASE-21297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nihal Jain updated HBASE-21297:
---
Affects Version/s: 3.0.0
2.0.0
> ModifyTableProcedure can throw TNDE instead of IOE in case of
> REGION_REPLICATION change
> ---
>
> Key: HBASE-21297
> URL: https://issues.apache.org/jira/browse/HBASE-21297
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.0.0
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HBASE-21297.master.001.patch
>
>
> Currently {{ModifyTableProcedure}} throws an {{IOException}} (See
> [ModifyTableProcedure.java#L252|https://github.com/apache/hbase/blob/924d183ba0e67b975e998f6006c993f457e03c20/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ModifyTableProcedure.java#L252])
> when a user tries to modify REGION_REPLICATION for an enabled table.
> Instead, it can throw a more specific {{TableNotDisabledException}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21475) Put mutation (having TTL set) added via co-processor is retrieved even after TTL expires
[
https://issues.apache.org/jira/browse/HBASE-21475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nihal Jain updated HBASE-21475:
---
Description:
*Steps to reproduce*
# Create a region co-processor and override {{preBatchMutate}} such that it
creates a put corresponding to a user put having same timestamp and TTL as the
user put.
# Create a table and add a row having TTL set to 3000 ms.
# Wait for > 3000 ms.
# Scan the table.
*Expected Result*
No rows should be retrieved in step 4
*Actual Result*
User row is not retreived, while put created via co-processor is still
retrieved.
*Analysis/Issue*
Unlike user mutations, the mutations added by coprocessor do not have tags
corresponding to TTL, hence they are retrieved in scan even after TTL expires.
was:
*Steps to reproduce*
# Create a region co-processor and override {{preBatchMutate}} such that it
creates a put corresponding to a user put having same timestamp and TTL as the
user put.
# Create a table and add a row having TTL set to 3000 ms.
# Wait for > 3000 ms.
# Scan the table.
*Expected Result*
No rows should be retrieved in step 4
*Actual Result*
User row is not retreived, while put created via co-processor is still
retrieved.
*Analysis/Issue*
Unlike user mutations, the mutations added by coprocessor do not have tags
corresponding to TTL, hence they retrieved even after TTL expires.
> Put mutation (having TTL set) added via co-processor is retrieved even after
> TTL expires
>
>
> Key: HBASE-21475
> URL: https://issues.apache.org/jira/browse/HBASE-21475
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors
>Affects Versions: 3.0.0, 2.0.0, 2.1.1
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-21475.master.001.patch
>
>
> *Steps to reproduce*
> # Create a region co-processor and override {{preBatchMutate}} such that it
> creates a put corresponding to a user put having same timestamp and TTL as
> the user put.
> # Create a table and add a row having TTL set to 3000 ms.
> # Wait for > 3000 ms.
> # Scan the table.
> *Expected Result*
> No rows should be retrieved in step 4
> *Actual Result*
> User row is not retreived, while put created via co-processor is still
> retrieved.
> *Analysis/Issue*
> Unlike user mutations, the mutations added by coprocessor do not have tags
> corresponding to TTL, hence they are retrieved in scan even after TTL expires.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21475) Put mutation (having TTL set) added via co-processor is retrieved even after TTL expires
[
https://issues.apache.org/jira/browse/HBASE-21475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nihal Jain updated HBASE-21475:
---
Description:
*Steps to reproduce*
# Create a region co-processor and override {{preBatchMutate}} such that it
creates a put corresponding to a user put having same timestamp and TTL as the
user put.
# Create a table and add a row having TTL set to 3000 ms.
# Wait for > 3000 ms.
# Scan the table.
*Expected Result*
No rows should be retrieved in step 4
*Actual Result*
User row is not retreived, while put created via co-processor is still
retrieved.
*Analysis/Issue*
Unlike user mutations, the mutations added by coprocessor do not have tags
corresponding to TTL, hence they retrieved even after TTL expires.
was:
*Steps to reproduce*
1) Create a region coprocessor an override {{preBatchMutate}} such that it
create a put corresponding to a user put having same timestamp and TTL as the
user put.
2) Create a table and add a row having TTL set to 3000 ms
3) Wait for > 3000 ms
4) Scan the table.
*Result*
User row is not retreived while put created via coprocessor is still retreived.
*Analysis/Issue*
Unlike user mutations, the mutations added by coprocessor do not have tags
corresponding to TTL, hence they retreived even after TTL expires.
> Put mutation (having TTL set) added via co-processor is retrieved even after
> TTL expires
>
>
> Key: HBASE-21475
> URL: https://issues.apache.org/jira/browse/HBASE-21475
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors
>Affects Versions: 3.0.0, 2.0.0, 2.1.1
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-21475.master.001.patch
>
>
> *Steps to reproduce*
> # Create a region co-processor and override {{preBatchMutate}} such that it
> creates a put corresponding to a user put having same timestamp and TTL as
> the user put.
> # Create a table and add a row having TTL set to 3000 ms.
> # Wait for > 3000 ms.
> # Scan the table.
> *Expected Result*
> No rows should be retrieved in step 4
> *Actual Result*
> User row is not retreived, while put created via co-processor is still
> retrieved.
> *Analysis/Issue*
> Unlike user mutations, the mutations added by coprocessor do not have tags
> corresponding to TTL, hence they retrieved even after TTL expires.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21475) Put mutation (having TTL set) added via co-processor is retrieved even after TTL expires
[
https://issues.apache.org/jira/browse/HBASE-21475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685552#comment-16685552
]
Nihal Jain commented on HBASE-21475:
Submitted a patch which reproduces the scenario along with the proposed
solution.
The test contained in the patch adds a put (via a coprocessor) corresponding to
the user put having same timestamp and TTL as user stamp.
{noformat}
2018-11-13 23:09:00,277 INFO
[RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=45641]
coprocessor.TestRegionObserverForAddingMutationsFromCoprocessors$TestPutWithTTLCoprocessor(231):
Putting:[{"totalColumns":1,"row":"cpPut","families":{"test":[{"qualifier":"dummy","vlen":7,"tag":[],"timestamp":1542130740276}]},"ttl":3000,"ts":9223372036854775807}]
2018-11-13 23:09:00,288 INFO [Time-limited test]
coprocessor.TestRegionObserverForAddingMutationsFromCoprocessors(142):
keyvalues={cpPut/test:dummy/1542130740276/Put/vlen=7/seqid=0}
2018-11-13 23:09:00,288 INFO [Time-limited test]
coprocessor.TestRegionObserverForAddingMutationsFromCoprocessors(142):
keyvalues={r1/test:dummy/1542130740276/Put/vlen=5/seqid=0}{noformat}
After TTL expires we can still retrieve the row added by coprocessor, while the
user put gets deleted succesfully.
{noformat}
2018-11-13 23:09:05,292 INFO [Time-limited test]
coprocessor.TestRegionObserverForAddingMutationsFromCoprocessors(142):
keyvalues={cpPut/test:dummy/1542130740276/Put/vlen=7/seqid=0}
{noformat}
> Put mutation (having TTL set) added via co-processor is retrieved even after
> TTL expires
>
>
> Key: HBASE-21475
> URL: https://issues.apache.org/jira/browse/HBASE-21475
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors
>Affects Versions: 3.0.0, 2.0.0, 2.1.1
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-21475.master.001.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21475) Put mutation (having TTL set) added via co-processor is retrieved even after TTL expires
[
https://issues.apache.org/jira/browse/HBASE-21475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685566#comment-16685566
]
Nihal Jain edited comment on HBASE-21475 at 11/13/18 6:12 PM:
--
[~apurtell], [~elserj] Can you please review this?
was (Author: nihaljain.cs):
[~apurtell] Can you please review this?
> Put mutation (having TTL set) added via co-processor is retrieved even after
> TTL expires
>
>
> Key: HBASE-21475
> URL: https://issues.apache.org/jira/browse/HBASE-21475
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors
>Affects Versions: 3.0.0, 2.0.0, 2.1.1
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-21475.master.001.patch
>
>
> *Steps to reproduce*
> 1) Create a region coprocessor an override {{preBatchMutate}} such that it
> create a put corresponding to a user put having same timestamp and TTL as the
> user put.
> 2) Create a table and add a row having TTL set to 3000 ms
> 3) Wait for > 3000 ms
> 4) Scan the table.
> *Result*
> User row is not retreived while put created via coprocessor is still
> retreived.
> *Analysis/Issue*
> Unlike user mutations, the mutations added by coprocessor do not have tags
> corresponding to TTL, hence they retreived even after TTL expires.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21475) Put mutation (having TTL set) added via co-processor is retrieved even after TTL expires
[
https://issues.apache.org/jira/browse/HBASE-21475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685566#comment-16685566
]
Nihal Jain commented on HBASE-21475:
[~apurtell] Can you please review this?
> Put mutation (having TTL set) added via co-processor is retrieved even after
> TTL expires
>
>
> Key: HBASE-21475
> URL: https://issues.apache.org/jira/browse/HBASE-21475
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors
>Affects Versions: 3.0.0, 2.0.0, 2.1.1
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-21475.master.001.patch
>
>
> *Steps to reproduce*
> 1) Create a region coprocessor an override {{preBatchMutate}} such that it
> create a put corresponding to a user put having same timestamp and TTL as the
> user put.
> 2) Create a table and add a row having TTL set to 3000 ms
> 3) Wait for > 3000 ms
> 4) Scan the table.
> *Result*
> User row is not retreived while put created via coprocessor is still
> retreived.
> *Analysis/Issue*
> Unlike user mutations, the mutations added by coprocessor do not have tags
> corresponding to TTL, hence they retreived even after TTL expires.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21475) Put mutation (having TTL set) added via co-processor is retrieved even after TTL expires
[
https://issues.apache.org/jira/browse/HBASE-21475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nihal Jain updated HBASE-21475:
---
Description:
*Steps to reproduce*
1) Create a region coprocessor an override {{preBatchMutate}} such that it
create a put corresponding to a user put having same timestamp and TTL as the
user put.
2) Create a table and add a row having TTL set to 3000 ms
3) Wait for > 3000 ms
4) Scan the table.
*Result*
User row is not retreived while put created via coprocessor is still retreived.
*Analysis/Issue*
Unlike user mutations, the mutations added by coprocessor do not have tags
corresponding to TTL, hence they retreived even after TTL expires.
> Put mutation (having TTL set) added via co-processor is retrieved even after
> TTL expires
>
>
> Key: HBASE-21475
> URL: https://issues.apache.org/jira/browse/HBASE-21475
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors
>Affects Versions: 3.0.0, 2.0.0, 2.1.1
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-21475.master.001.patch
>
>
> *Steps to reproduce*
> 1) Create a region coprocessor an override {{preBatchMutate}} such that it
> create a put corresponding to a user put having same timestamp and TTL as the
> user put.
> 2) Create a table and add a row having TTL set to 3000 ms
> 3) Wait for > 3000 ms
> 4) Scan the table.
> *Result*
> User row is not retreived while put created via coprocessor is still
> retreived.
> *Analysis/Issue*
> Unlike user mutations, the mutations added by coprocessor do not have tags
> corresponding to TTL, hence they retreived even after TTL expires.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21475) Put mutation (having TTL set) added via co-processor is retrieved even after TTL expires
[ https://issues.apache.org/jira/browse/HBASE-21475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nihal Jain updated HBASE-21475: --- Fix Version/s: 3.0.0 Affects Version/s: 2.0.0 Status: Patch Available (was: Open) > Put mutation (having TTL set) added via co-processor is retrieved even after > TTL expires > > > Key: HBASE-21475 > URL: https://issues.apache.org/jira/browse/HBASE-21475 > Project: HBase > Issue Type: Bug > Components: Coprocessors >Affects Versions: 2.1.1, 2.0.0, 3.0.0 >Reporter: Nihal Jain >Assignee: Nihal Jain >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-21475.master.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21475) Put mutation (having TTL set) added via co-processor is retrieved even after TTL expires
[ https://issues.apache.org/jira/browse/HBASE-21475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nihal Jain updated HBASE-21475: --- Attachment: HBASE-21475.master.001.patch > Put mutation (having TTL set) added via co-processor is retrieved even after > TTL expires > > > Key: HBASE-21475 > URL: https://issues.apache.org/jira/browse/HBASE-21475 > Project: HBase > Issue Type: Bug > Components: Coprocessors >Affects Versions: 3.0.0, 2.1.1 >Reporter: Nihal Jain >Assignee: Nihal Jain >Priority: Major > Attachments: HBASE-21475.master.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21475) Put mutation (having TTL set) added via co-processor is retrieved even after TTL expires
Nihal Jain created HBASE-21475: -- Summary: Put mutation (having TTL set) added via co-processor is retrieved even after TTL expires Key: HBASE-21475 URL: https://issues.apache.org/jira/browse/HBASE-21475 Project: HBase Issue Type: Bug Components: Coprocessors Affects Versions: 2.1.1, 3.0.0 Reporter: Nihal Jain Assignee: Nihal Jain -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21404) Master/RS navbar active state does not work
[ https://issues.apache.org/jira/browse/HBASE-21404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685533#comment-16685533 ] Nihal Jain commented on HBASE-21404: Resubmitted patch. > Master/RS navbar active state does not work > --- > > Key: HBASE-21404 > URL: https://issues.apache.org/jira/browse/HBASE-21404 > Project: HBase > Issue Type: Bug > Components: UI >Affects Versions: 3.0.0, 2.0.0 >Reporter: Nihal Jain >Assignee: Nihal Jain >Priority: Minor > Fix For: 3.0.0 > > Attachments: HBASE-21404.master.001.patch, > HBASE-21404.master.001.patch, master_after_patch.png, master_before.png, > rs_after_patch.png, rs_before.png > > > In master/rs web UI, the current active tab is not updated when user switches > to any tab other than "Home" tab. > For example: even though say if we are on "tabledetailed.jsp", the navbar > does not update the active state of that tab. See master_before.png -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21404) Master/RS navbar active state does not work
[ https://issues.apache.org/jira/browse/HBASE-21404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nihal Jain updated HBASE-21404: --- Attachment: HBASE-21404.master.001.patch > Master/RS navbar active state does not work > --- > > Key: HBASE-21404 > URL: https://issues.apache.org/jira/browse/HBASE-21404 > Project: HBase > Issue Type: Bug > Components: UI >Affects Versions: 3.0.0, 2.0.0 >Reporter: Nihal Jain >Assignee: Nihal Jain >Priority: Minor > Fix For: 3.0.0 > > Attachments: HBASE-21404.master.001.patch, > HBASE-21404.master.001.patch, master_after_patch.png, master_before.png, > rs_after_patch.png, rs_before.png > > > In master/rs web UI, the current active tab is not updated when user switches > to any tab other than "Home" tab. > For example: even though say if we are on "tabledetailed.jsp", the navbar > does not update the active state of that tab. See master_before.png -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21246) Introduce WALIdentity interface
[ https://issues.apache.org/jira/browse/HBASE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685515#comment-16685515 ] Ted Yu commented on HBASE-21246: Patch v37 fixes infinite call in RegionGroupingProvider#createWALIdentity > Introduce WALIdentity interface > --- > > Key: HBASE-21246 > URL: https://issues.apache.org/jira/browse/HBASE-21246 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Fix For: HBASE-20952 > > Attachments: 21246.003.patch, 21246.20.txt, 21246.21.txt, > 21246.23.txt, 21246.24.txt, 21246.25.txt, 21246.26.txt, 21246.34.txt, > 21246.37.txt, 21246.HBASE-20952.001.patch, 21246.HBASE-20952.002.patch, > 21246.HBASE-20952.004.patch, 21246.HBASE-20952.005.patch, > 21246.HBASE-20952.007.patch, 21246.HBASE-20952.008.patch, > replication-src-creates-wal-reader.jpg, wal-factory-providers.png, > wal-providers.png, wal-splitter-reader.jpg, wal-splitter-writer.jpg > > > We are introducing WALIdentity interface so that the WAL representation can > be decoupled from distributed filesystem. > The interface provides getName method whose return value can represent > filename in distributed filesystem environment or, the name of the stream > when the WAL is backed by log stream. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21246) Introduce WALIdentity interface
[ https://issues.apache.org/jira/browse/HBASE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21246: --- Attachment: 21246.37.txt > Introduce WALIdentity interface > --- > > Key: HBASE-21246 > URL: https://issues.apache.org/jira/browse/HBASE-21246 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Fix For: HBASE-20952 > > Attachments: 21246.003.patch, 21246.20.txt, 21246.21.txt, > 21246.23.txt, 21246.24.txt, 21246.25.txt, 21246.26.txt, 21246.34.txt, > 21246.37.txt, 21246.HBASE-20952.001.patch, 21246.HBASE-20952.002.patch, > 21246.HBASE-20952.004.patch, 21246.HBASE-20952.005.patch, > 21246.HBASE-20952.007.patch, 21246.HBASE-20952.008.patch, > replication-src-creates-wal-reader.jpg, wal-factory-providers.png, > wal-providers.png, wal-splitter-reader.jpg, wal-splitter-writer.jpg > > > We are introducing WALIdentity interface so that the WAL representation can > be decoupled from distributed filesystem. > The interface provides getName method whose return value can represent > filename in distributed filesystem environment or, the name of the stream > when the WAL is backed by log stream. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20586) SyncTable tool: Add support for cross-realm remote clusters
[
https://issues.apache.org/jira/browse/HBASE-20586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685464#comment-16685464
]
Sean Busbey commented on HBASE-20586:
-
A doc jira blocked by this one sounds like a good idea.
> SyncTable tool: Add support for cross-realm remote clusters
> ---
>
> Key: HBASE-20586
> URL: https://issues.apache.org/jira/browse/HBASE-20586
> Project: HBase
> Issue Type: Improvement
> Components: mapreduce, Operability, Replication
>Affects Versions: 1.2.0, 2.0.0
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Major
> Fix For: 1.5.0, 2.2.0
>
> Attachments: HBASE-20586.master.001.patch
>
>
> One possible scenario for HashTable/SyncTable is for synchronize different
> clusters, for instance, when replication has been enabled but data existed
> already, or due replication issues that may had caused long lags in the
> replication.
> For secured clusters under different kerberos realms (with cross-realm
> properly set), though, current SyncTable version would fail to authenticate
> with the remote cluster when trying to read HashTable outputs (when
> *sourcehashdir* is remote) and also when trying to read table data on the
> remote cluster (when *sourcezkcluster* is remote).
> The hdfs error would look like this:
> {noformat}
> INFO mapreduce.Job: Task Id : attempt_1524358175778_105392_m_00_0, Status
> : FAILED
> Error: java.io.IOException: Failed on local exception: java.io.IOException:
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate
> via:[TOKEN, KERBEROS]; Host Details : local host is: "local-host/1.1.1.1";
> destination host is: "remote-nn":8020;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
> at org.apache.hadoop.ipc.Client.call(Client.java:1506)
> at org.apache.hadoop.ipc.Client.call(Client.java:1439)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
> at com.sun.proxy.$Proxy13.getBlockLocations(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:256)
> ...
> at
> org.apache.hadoop.hbase.mapreduce.HashTable$TableHash.readPropertiesFile(HashTable.java:144)
> at
> org.apache.hadoop.hbase.mapreduce.HashTable$TableHash.read(HashTable.java:105)
> at
> org.apache.hadoop.hbase.mapreduce.SyncTable$SyncMapper.setup(SyncTable.java:188)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
> ...
> Caused by: java.io.IOException:
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate
> via:[TOKEN, KERBEROS]{noformat}
> The above can be sorted if the SyncTable job acquires a DT for the remote NN.
> Once hdfs related authentication is done, it's also necessary to authenticate
> against remote HBase, as the below error would arise:
> {noformat}
> INFO mapreduce.Job: Task Id : attempt_1524358175778_172414_m_00_0, Status
> : FAILED
> Error: org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get
> the location
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:326)
> ...
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:867)
> at
> org.apache.hadoop.hbase.mapreduce.SyncTable$SyncMapper.syncRange(SyncTable.java:331)
> ...
> Caused by: java.io.IOException: Could not set up IO Streams to
> remote-rs-host/1.1.1.2:60020
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:786)
> ...
> Caused by: java.lang.RuntimeException: SASL authentication failed. The most
> likely cause is missing or invalid credentials. Consider 'kinit'.
> ...
> Caused by: GSSException: No valid credentials provided (Mechanism level:
> Failed to find any Kerberos tgt)
> ...{noformat}
> The above would need additional authentication logic against the remote hbase
> cluster.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21255) [acl] Refactor TablePermission into three classes (Global, Namespace, Table)
[
https://issues.apache.org/jira/browse/HBASE-21255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685461#comment-16685461
]
stack commented on HBASE-21255:
---
You did not address code duplication in you last patch [~reidchan]? Otherwise,
patch is good by me. Let me retry.
> [acl] Refactor TablePermission into three classes (Global, Namespace, Table)
>
>
> Key: HBASE-21255
> URL: https://issues.apache.org/jira/browse/HBASE-21255
> Project: HBase
> Issue Type: Improvement
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21225.master.001.patch,
> HBASE-21225.master.002.patch, HBASE-21225.master.007.patch,
> HBASE-21225.master.008.patch, HBASE-21255.master.003.patch,
> HBASE-21255.master.004.patch, HBASE-21255.master.005.patch,
> HBASE-21255.master.006.patch, HBASE-21255.master.006.patch
>
>
> A TODO in {{TablePermission.java}}
> {code:java}
> //TODO refactor this class
> //we need to refacting this into three classes (Global, Table, Namespace)
> {code}
> Change Notes:
> * Divide origin TablePermission into three classes GlobalPermission,
> NamespacePermission, TablePermission
> * New UserPermission consists of a user name(string, not byte[], for
> convenience) and a permission in one of [Global, Namespace, Table]Permission.
> * Rename TableAuthManager to AuthManager(it is IA.P), and rename some
> methods for readability.
> * Make PermissionCache thread safe, and the ListMultiMap is changed to Set.
> * User cache and group cache in AuthManager is combined together.
> * Wire proto is kept, BC should be under guarantee.
> * Fix HBASE-21390.
> * Resolve a small {{TODO}} global entry should be handled differently in
> AccessControlLists
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21255) [acl] Refactor TablePermission into three classes (Global, Namespace, Table)
[
https://issues.apache.org/jira/browse/HBASE-21255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-21255:
--
Attachment: HBASE-21255.master.006.patch
> [acl] Refactor TablePermission into three classes (Global, Namespace, Table)
>
>
> Key: HBASE-21255
> URL: https://issues.apache.org/jira/browse/HBASE-21255
> Project: HBase
> Issue Type: Improvement
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21225.master.001.patch,
> HBASE-21225.master.002.patch, HBASE-21225.master.007.patch,
> HBASE-21225.master.008.patch, HBASE-21255.master.003.patch,
> HBASE-21255.master.004.patch, HBASE-21255.master.005.patch,
> HBASE-21255.master.006.patch, HBASE-21255.master.006.patch
>
>
> A TODO in {{TablePermission.java}}
> {code:java}
> //TODO refactor this class
> //we need to refacting this into three classes (Global, Table, Namespace)
> {code}
> Change Notes:
> * Divide origin TablePermission into three classes GlobalPermission,
> NamespacePermission, TablePermission
> * New UserPermission consists of a user name(string, not byte[], for
> convenience) and a permission in one of [Global, Namespace, Table]Permission.
> * Rename TableAuthManager to AuthManager(it is IA.P), and rename some
> methods for readability.
> * Make PermissionCache thread safe, and the ListMultiMap is changed to Set.
> * User cache and group cache in AuthManager is combined together.
> * Wire proto is kept, BC should be under guarantee.
> * Fix HBASE-21390.
> * Resolve a small {{TODO}} global entry should be handled differently in
> AccessControlLists
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21467) Fix flaky test TestCoprocessorClassLoader.testCleanupOldJars
[
https://issues.apache.org/jira/browse/HBASE-21467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685455#comment-16685455
]
stack commented on HBASE-21467:
---
Added below on PR: "What versions of hbase do you see this failure in? I wonder
why your PR didn't show up on the JIRA? Normally if the commit has the jira as
first thing, it attaches to the JIRA. You don't seem to have an account on
apache JIRA? Maybe that is it? Mind creating one so we can assign you the
issue? Thanks."
> Fix flaky test TestCoprocessorClassLoader.testCleanupOldJars
>
>
> Key: HBASE-21467
> URL: https://issues.apache.org/jira/browse/HBASE-21467
> Project: HBase
> Issue Type: Bug
> Environment: Ubuntu 18.04 LTS
> java version "1.8.0_181"
> Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
> Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)
> Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe;
> 2018-06-17T13:33:14-05:00)
> Maven home: /home/OrdTesters/apache-maven-3.5.4
> Java version: 1.8.0_172, vendor: Oracle Corporation, runtime:
> /usr/lib/jvm/jdk1.8.0_172/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux", version: "4.15.0-36-generic", arch: "amd64", family: "unix"
>Reporter: OrDTesters
>Priority: Minor
>
> TestCoprocessorClassLoader.testCleanupOldJars fails when run by itself.
> This can be fixed by calling CoprocessorClassLoader.getClassLoader, which
> initializes the jar file needed by testCleanupOldJars.
> Pull request: [https://github.com/apache/hbase/pull/100]
> Steps to reproduce:
> {code:java}
> git clone https://github.com/apache/hbase
> cd hbase/hbase-common
> mvn install -DskipTests -am
> mvn test -Dtest="TestCoprocessorClassLoader#testCleanupOldJars"{code}
>
> Stack trace of the failure:
>
> {code:java}
> java.io.FileNotFoundException:
> /home/OrdTesters/Java/hbase/hbase-common/target/test-data/64b7e1ae-1c34-48f7-d731-9af59256dc0a/hbase-local-dir/jars/tmp/TestCleanupOldJars.test.jar
> (Aucun fichier ou dossier de ce type)
> at java.io.FileOutputStream.open0(Native Method)
> at java.io.FileOutputStream.open(FileOutputStream.java:270)
> at java.io.FileOutputStream.(FileOutputStream.java:213)
> at java.io.FileOutputStream.(FileOutputStream.java:162)
> at
> org.apache.hadoop.hbase.util.TestCoprocessorClassLoader.testCleanupOldJars(TestCoprocessorClassLoader.java:65)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21452) Illegal character in hbase counters group name
[
https://issues.apache.org/jira/browse/HBASE-21452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685454#comment-16685454
]
Sean Busbey commented on HBASE-21452:
-
Okay so release note marking it as an incompat that calls out folks looking at
Hadoop counters (either in MapReduce jobs or Spark) will see a change in the
names? If that sounds correct, then +1 from me for master and branch-2.
> Illegal character in hbase counters group name
> --
>
> Key: HBASE-21452
> URL: https://issues.apache.org/jira/browse/HBASE-21452
> Project: HBase
> Issue Type: Bug
> Components: spark
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21452.branch-2.001.patch
>
>
> Messing w/ spark counting RDD rows, spark dumps out following complaint:
> {code}
> 2018-11-07 20:03:29,132 ERROR [Executor task launch worker for task 0]
> repl.ExecutorClassLoader: Failed to check existence of class HBase
> Counters_en_US on REPL class server at spark://192.168.1.139:61037/classes
> java.net.URISyntaxException: Illegal character in path at index 41:
> spark://192.168.1.139:61037/classes/HBase Counters_en_US.class
> at java.net.URI$Parser.fail(URI.java:2848)
> at java.net.URI$Parser.checkChars(URI.java:3021)
> at java.net.URI$Parser.parseHierarchical(URI.java:3105)
> at java.net.URI$Parser.parse(URI.java:3053)
> at java.net.URI.(URI.java:588)
> at
> org.apache.spark.rpc.netty.NettyRpcEnv.openChannel(NettyRpcEnv.scala:328)
> at
> org.apache.spark.repl.ExecutorClassLoader.org$apache$spark$repl$ExecutorClassLoader$$getClassFileInputStreamFromSparkRPC(ExecutorClassLoader.scala:95)
> at
> org.apache.spark.repl.ExecutorClassLoader$$anonfun$1.apply(ExecutorClassLoader.scala:62)
> at
> org.apache.spark.repl.ExecutorClassLoader$$anonfun$1.apply(ExecutorClassLoader.scala:62)
> at
> org.apache.spark.repl.ExecutorClassLoader.findClassLocally(ExecutorClassLoader.scala:167)
> at
> org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:85)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.util.ResourceBundle$Control.newBundle(ResourceBundle.java:2649)
> at java.util.ResourceBundle.loadBundle(ResourceBundle.java:1510)
> at java.util.ResourceBundle.findBundle(ResourceBundle.java:1474)
> at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1370)
> at java.util.ResourceBundle.getBundle(ResourceBundle.java:1091)
> at
> org.apache.hadoop.mapreduce.util.ResourceBundles.getBundle(ResourceBundles.java:37)
> at
> org.apache.hadoop.mapreduce.util.ResourceBundles.getValue(ResourceBundles.java:56)
> at
> org.apache.hadoop.mapreduce.util.ResourceBundles.getCounterGroupName(ResourceBundles.java:77)
> at
> org.apache.hadoop.mapreduce.counters.CounterGroupFactory.newGroup(CounterGroupFactory.java:94)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounters.getGroup(AbstractCounters.java:226)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounters.findCounter(AbstractCounters.java:153)
> at
> org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl$DummyReporter.getCounter(TaskAttemptContextImpl.java:110)
> at
> org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl.getCounter(TaskAttemptContextImpl.java:76)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.updateCounters(TableRecordReaderImpl.java:298)
> at
> org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.updateCounters(TableRecordReaderImpl.java:286)
> at
> org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.nextKeyValue(TableRecordReaderImpl.java:257)
> at
> org.apache.hadoop.hbase.mapreduce.TableRecordReader.nextKeyValue(TableRecordReader.java:133)
> at
> org.apache.hadoop.hbase.mapreduce.TableInputFormatBase$1.nextKeyValue(TableInputFormatBase.java:220)
> at
> org.apache.spark.rdd.NewHadoopRDD$$anon$1.hasNext(NewHadoopRDD.scala:214)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
> at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1837)
> at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1168)
> at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1168)
> at
>
[jira] [Commented] (HBASE-21034) Add new throttle type: read/write capacity unit
[
https://issues.apache.org/jira/browse/HBASE-21034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685446#comment-16685446
]
Hadoop QA commented on HBASE-21034:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
45s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
26s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 4m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch passed checkstyle in hbase-protocol-shaded
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} The patch passed checkstyle in hbase-client {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
20s{color} | {color:green} hbase-server: The patch generated 0 new + 19
unchanged - 12 fixed = 19 total (was 31) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch passed checkstyle in hbase-shell {color} |
| {color:red}-1{color} | {color:red} rubocop {color} | {color:red} 0m
8s{color} | {color:red} The patch generated 5 new + 90 unchanged - 5 fixed = 95
total (was 95) {color} |
| {color:green}+1{color} | {color:green} ruby-lint {color} | {color:green} 0m
2s{color} | {color:green} There were no new ruby-lint issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
39s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 31s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green}
2m 13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
40s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
[jira] [Commented] (HBASE-21410) A helper page that help find all problematic regions and procedures
[ https://issues.apache.org/jira/browse/HBASE-21410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685445#comment-16685445 ] stack commented on HBASE-21410: --- [~tianjingyun] FYI, when folks reviewing, you should wait till they +1 before pushing. Let me take a look at this... Also, why push to branch-2.1 only? Should be on branch-2 and master also? (Congrats on becoming a committer!) > A helper page that help find all problematic regions and procedures > --- > > Key: HBASE-21410 > URL: https://issues.apache.org/jira/browse/HBASE-21410 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0, 2.2.0, 2.1.1 >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Major > Fix For: 3.0.0, 2.1.0, 2.2.0 > > Attachments: HBASE-21410.branch-2.1.001.patch, > HBASE-21410.branch-2.1.002.patch, HBASE-21410.master.001.patch, > HBASE-21410.master.002.patch, HBASE-21410.master.003.patch, > HBASE-21410.master.004.patch, Screenshot from 2018-10-30 19-06-21.png, > Screenshot from 2018-10-30 19-06-42.png, Screenshot from 2018-10-31 > 10-11-38.png, Screenshot from 2018-10-31 10-11-56.png, Screenshot from > 2018-11-01 17-56-02.png, Screenshot from 2018-11-01 17-56-15.png > > > *This page is mainly focus on finding the regions stuck in some state that > cannot be assigned. My proposal of the page is as follows:* > !Screenshot from 2018-10-30 19-06-21.png! > *From this page we can see all regions in RIT queue and their related > procedures. If we can determine that these regions' state are abnormal, we > can click the link 'Procedures as TXT' to get a full list of procedure IDs to > bypass them. Then click 'Regions as TXT' to get a full list of encoded region > names to assign.* > !Screenshot from 2018-10-30 19-06-42.png! > *Some region names are covered by the navigator bar, I'll fix it later.* -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21440) Assign procedure on the crashed server is not properly interrupted
[ https://issues.apache.org/jira/browse/HBASE-21440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685439#comment-16685439 ] stack commented on HBASE-21440: --- [~an...@apache.org] Legit failures then in your opinion sir? Related? > Assign procedure on the crashed server is not properly interrupted > -- > > Key: HBASE-21440 > URL: https://issues.apache.org/jira/browse/HBASE-21440 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.2 >Reporter: Ankit Singhal >Assignee: Ankit Singhal >Priority: Major > Attachments: HBASE-21440.branch-2.0.001.patch, > HBASE-21440.branch-2.0.002.patch, HBASE-21440.branch-2.0.003.patch, > HBASE-21440.branch-2.0.004.patch > > > When the server crashes, it's SCP checks if there is already a procedure > assigning the region on this crashed server. If we found one, SCP will just > interrupt the already running AssignProcedure by calling remoteCallFailed > which internally just changes the region node state to OFFLINE and send the > procedure back with transition queue state for assignment with a new plan. > But, due to the race condition between the calling of the remoteCallFailed > and current state of the already running assign > procedure(REGION_TRANSITION_FINISH: where the region is already opened), it > is possible that assign procedure goes ahead in updating the regionStateNode > to OPEN on a crashed server. > As SCP had already skipped this region for assignment as it was relying on > existing assign procedure to do the right thing, this whole confusion leads > region to a not accessible state. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21470) [hbase-connectors] Build shaded versions of the connectors libs
[ https://issues.apache.org/jira/browse/HBASE-21470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685440#comment-16685440 ] Sean Busbey commented on HBASE-21470: - the spark module should already work with the shaded hbase module(s). IIRC it requires the mapreduce specific one. I would expect that Kafka will end up being the same. If we can avoid relying on the shaded plugin again within hbase-connectors we should do so. > [hbase-connectors] Build shaded versions of the connectors libs > --- > > Key: HBASE-21470 > URL: https://issues.apache.org/jira/browse/HBASE-21470 > Project: HBase > Issue Type: Task > Components: build, hbase-connectors >Affects Versions: connector-1.0.0 >Reporter: Adrian Muraru >Priority: Major > > For downstream users it would be helpful to generate shaded versions of the > connectors libs, e.g hbase-shaded-spark and hbase-shaded-kafka. > These would ease integrating this libs in Spark/Hadoop projects where > transitive dependencies of the connectors libs conflict with the runtime ones -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20604) ProtobufLogReader#readNext can incorrectly loop to the same position in the stream until the the WAL is rolled
[
https://issues.apache.org/jira/browse/HBASE-20604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685419#comment-16685419
]
Sean Busbey commented on HBASE-20604:
-
My understanding is that a good portion of our issues in this code are caused
by Hadoop not really defining what to expect when a client has a concurrent
read open on a file that's still open for write. This is usually where the
problems in our WAL reading code comes up; our replication system is relying on
assumptions that aren't really documented anywhere.
AFAIK there's no UT because we have never been able to isolate the problem(s)
that poke up in production around this, and trying to mock out the various
levels of abstraction went poorly when last I tried.
> ProtobufLogReader#readNext can incorrectly loop to the same position in the
> stream until the the WAL is rolled
> --
>
> Key: HBASE-20604
> URL: https://issues.apache.org/jira/browse/HBASE-20604
> Project: HBase
> Issue Type: Bug
> Components: Replication, wal
>Affects Versions: 3.0.0
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
>Priority: Critical
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.0.3, 1.4.9, 2.1.2, 1.2.9
>
> Attachments: HBASE-20604.002.patch, HBASE-20604.003.patch,
> HBASE-20604.004.patch, HBASE-20604.005.patch, HBASE-20604.patch
>
>
> Every time we call {{ProtobufLogReader#readNext}} we consume the input stream
> associated to the {{FSDataInputStream}} from the WAL that we are reading.
> Under certain conditions, e.g. when using the encryption at rest
> ({{CryptoInputStream}}) the stream can return partial data which can cause a
> premature EOF that cause {{inputStream.getPos()}} to return to the same
> origina position causing {{ProtobufLogReader#readNext}} to re-try over the
> reads until the WAL is rolled.
> The side effect of this issue is that {{ReplicationSource}} can get stuck
> until the WAL is rolled and causing replication delays up to an hour in some
> cases.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20604) ProtobufLogReader#readNext can incorrectly loop to the same position in the stream until the the WAL is rolled
[
https://issues.apache.org/jira/browse/HBASE-20604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685426#comment-16685426
]
Sean Busbey commented on HBASE-20604:
-
Also also I would like to start the RC process for 1.2.9 this week, so it'd be
very helpful if this critical issue didn't reopen. ;)
> ProtobufLogReader#readNext can incorrectly loop to the same position in the
> stream until the the WAL is rolled
> --
>
> Key: HBASE-20604
> URL: https://issues.apache.org/jira/browse/HBASE-20604
> Project: HBase
> Issue Type: Bug
> Components: Replication, wal
>Affects Versions: 3.0.0
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
>Priority: Critical
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.0.3, 1.4.9, 2.1.2, 1.2.9
>
> Attachments: HBASE-20604.002.patch, HBASE-20604.003.patch,
> HBASE-20604.004.patch, HBASE-20604.005.patch, HBASE-20604.patch
>
>
> Every time we call {{ProtobufLogReader#readNext}} we consume the input stream
> associated to the {{FSDataInputStream}} from the WAL that we are reading.
> Under certain conditions, e.g. when using the encryption at rest
> ({{CryptoInputStream}}) the stream can return partial data which can cause a
> premature EOF that cause {{inputStream.getPos()}} to return to the same
> origina position causing {{ProtobufLogReader#readNext}} to re-try over the
> reads until the WAL is rolled.
> The side effect of this issue is that {{ReplicationSource}} can get stuck
> until the WAL is rolled and causing replication delays up to an hour in some
> cases.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20604) ProtobufLogReader#readNext can incorrectly loop to the same position in the stream until the the WAL is rolled
[
https://issues.apache.org/jira/browse/HBASE-20604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685425#comment-16685425
]
Sean Busbey commented on HBASE-20604:
-
Also note that while we can try to isolate problems into things that Hadoop is
doing wrong (either in CryptoInputStream or the other implementation classes),
that project defines "what HDFS did in ~Hadoop 1" as canonical. This can
include things that some folks might consider incorrect implementation details,
if the project believes downstream has come to rely on it.
So to some degree we'll be stuck with defensive workarounds in our client usage
regardless.
> ProtobufLogReader#readNext can incorrectly loop to the same position in the
> stream until the the WAL is rolled
> --
>
> Key: HBASE-20604
> URL: https://issues.apache.org/jira/browse/HBASE-20604
> Project: HBase
> Issue Type: Bug
> Components: Replication, wal
>Affects Versions: 3.0.0
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
>Priority: Critical
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.0.3, 1.4.9, 2.1.2, 1.2.9
>
> Attachments: HBASE-20604.002.patch, HBASE-20604.003.patch,
> HBASE-20604.004.patch, HBASE-20604.005.patch, HBASE-20604.patch
>
>
> Every time we call {{ProtobufLogReader#readNext}} we consume the input stream
> associated to the {{FSDataInputStream}} from the WAL that we are reading.
> Under certain conditions, e.g. when using the encryption at rest
> ({{CryptoInputStream}}) the stream can return partial data which can cause a
> premature EOF that cause {{inputStream.getPos()}} to return to the same
> origina position causing {{ProtobufLogReader#readNext}} to re-try over the
> reads until the WAL is rolled.
> The side effect of this issue is that {{ReplicationSource}} can get stuck
> until the WAL is rolled and causing replication delays up to an hour in some
> cases.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21473) RowIndexSeekerV1 may return cell with extra two \x00\x00 bytes which has no tags
[ https://issues.apache.org/jira/browse/HBASE-21473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685420#comment-16685420 ] stack commented on HBASE-21473: --- Retry > RowIndexSeekerV1 may return cell with extra two \x00\x00 bytes which has no > tags > > > Key: HBASE-21473 > URL: https://issues.apache.org/jira/browse/HBASE-21473 > Project: HBase > Issue Type: Sub-task >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Major > Attachments: HBASE-21473.v1.patch, HBASE-21473.v1.patch, > HBASE-21473.v1.patch > > > When testing HBASE-21401, we found that two UT related to RowIndexSeekerV1 > always failed [1]. because RowIndexSeekerV1 may return cell with extra two > \x00\x00 bytes which has no tags.. > I plan to fix this bug in HBASE-21401, but seems the checkKeyValueBytes > need more performance evaluation, as [~Apache9] & [~stack] comment. I opened > issue HBASE-21459 to address it. > So, let me fix this bug firstly (Also will provide a seperate UT for this). > 1. > https://issues.apache.org/jira/browse/HBASE-21401?focusedCommentId=1671=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-1671 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21473) RowIndexSeekerV1 may return cell with extra two \x00\x00 bytes which has no tags
[ https://issues.apache.org/jira/browse/HBASE-21473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-21473: -- Attachment: HBASE-21473.v1.patch > RowIndexSeekerV1 may return cell with extra two \x00\x00 bytes which has no > tags > > > Key: HBASE-21473 > URL: https://issues.apache.org/jira/browse/HBASE-21473 > Project: HBase > Issue Type: Sub-task >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Major > Attachments: HBASE-21473.v1.patch, HBASE-21473.v1.patch, > HBASE-21473.v1.patch > > > When testing HBASE-21401, we found that two UT related to RowIndexSeekerV1 > always failed [1]. because RowIndexSeekerV1 may return cell with extra two > \x00\x00 bytes which has no tags.. > I plan to fix this bug in HBASE-21401, but seems the checkKeyValueBytes > need more performance evaluation, as [~Apache9] & [~stack] comment. I opened > issue HBASE-21459 to address it. > So, let me fix this bug firstly (Also will provide a seperate UT for this). > 1. > https://issues.apache.org/jira/browse/HBASE-21401?focusedCommentId=1671=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-1671 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21473) RowIndexSeekerV1 may return cell with extra two \x00\x00 bytes which has no tags
[ https://issues.apache.org/jira/browse/HBASE-21473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685418#comment-16685418 ] stack commented on HBASE-21473: --- Why the test failures? Because they depend on old format? If tag length == 0, we don't add the tag length short? Thanks for working on this. > RowIndexSeekerV1 may return cell with extra two \x00\x00 bytes which has no > tags > > > Key: HBASE-21473 > URL: https://issues.apache.org/jira/browse/HBASE-21473 > Project: HBase > Issue Type: Sub-task >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Major > Attachments: HBASE-21473.v1.patch, HBASE-21473.v1.patch > > > When testing HBASE-21401, we found that two UT related to RowIndexSeekerV1 > always failed [1]. because RowIndexSeekerV1 may return cell with extra two > \x00\x00 bytes which has no tags.. > I plan to fix this bug in HBASE-21401, but seems the checkKeyValueBytes > need more performance evaluation, as [~Apache9] & [~stack] comment. I opened > issue HBASE-21459 to address it. > So, let me fix this bug firstly (Also will provide a seperate UT for this). > 1. > https://issues.apache.org/jira/browse/HBASE-21401?focusedCommentId=1671=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-1671 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21457) BackupUtils#getWALFilesOlderThan refers to wrong FileSystem
[ https://issues.apache.org/jira/browse/HBASE-21457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685384#comment-16685384 ] Ted Yu edited comment on HBASE-21457 at 11/13/18 3:50 PM: -- Thanks for the review, Stephen and Vlad. was (Author: yuzhih...@gmail.com): Thanks for the review, Stephen. > BackupUtils#getWALFilesOlderThan refers to wrong FileSystem > --- > > Key: HBASE-21457 > URL: https://issues.apache.org/jira/browse/HBASE-21457 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Janos Gub >Assignee: Ted Yu >Priority: Major > Fix For: 3.0.0 > > Attachments: 21457.v1.txt, 21457.v2.txt, 21457.v3.txt, 21457.v3.txt, > 21457.v4.txt > > > Janos reported seeing backup test failure when testing a local HDFS for WALs > while using WASB/ADLS only for store files. > Janos spotted the code in BackupUtils#getWALFilesOlderThan which uses HBase > root dir for retrieving WAL files. > We should use the helper methods from CommonFSUtils. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20952) Re-visit the WAL API
[
https://issues.apache.org/jira/browse/HBASE-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685390#comment-16685390
]
ramkrishna.s.vasudevan commented on HBASE-20952:
OOO for personal reasons. No access to official emails during this period.
> Re-visit the WAL API
>
>
> Key: HBASE-20952
> URL: https://issues.apache.org/jira/browse/HBASE-20952
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Josh Elser
>Priority: Major
> Attachments: 20952.v1.txt
>
>
> Take a step back from the current WAL implementations and think about what an
> HBase WAL API should look like. What are the primitive calls that we require
> to guarantee durability of writes with a high degree of performance?
> The API needs to take the current implementations into consideration. We
> should also have a mind for what is happening in the Ratis LogService (but
> the LogService should not dictate what HBase's WAL API looks like RATIS-272).
> Other "systems" inside of HBase that use WALs are replication and
> backup Replication has the use-case for "tail"'ing the WAL which we
> should provide via our new API. B doesn't do anything fancy (IIRC). We
> should make sure all consumers are generally going to be OK with the API we
> create.
> The API may be "OK" (or OK in a part). We need to also consider other methods
> which were "bolted" on such as {{AbstractFSWAL}} and
> {{WALFileLengthProvider}}. Other corners of "WAL use" (like the
> {{WALSplitter}} should also be looked at to use WAL-APIs only).
> We also need to make sure that adequate interface audience and stability
> annotations are chosen.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21457) BackupUtils#getWALFilesOlderThan refers to wrong FileSystem
[ https://issues.apache.org/jira/browse/HBASE-21457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21457: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Thanks for the review, Stephen. > BackupUtils#getWALFilesOlderThan refers to wrong FileSystem > --- > > Key: HBASE-21457 > URL: https://issues.apache.org/jira/browse/HBASE-21457 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Janos Gub >Assignee: Ted Yu >Priority: Major > Fix For: 3.0.0 > > Attachments: 21457.v1.txt, 21457.v2.txt, 21457.v3.txt, 21457.v3.txt, > 21457.v4.txt > > > Janos reported seeing backup test failure when testing a local HDFS for WALs > while using WASB/ADLS only for store files. > Janos spotted the code in BackupUtils#getWALFilesOlderThan which uses HBase > root dir for retrieving WAL files. > We should use the helper methods from CommonFSUtils. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20952) Re-visit the WAL API
[
https://issues.apache.org/jira/browse/HBASE-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685385#comment-16685385
]
Sean Busbey commented on HBASE-20952:
-
How about if I move the job to only run weekly? Since my last comment this job
has run 5 times, which means for about 20 hours of testing that has gotten us
no new information.
> Re-visit the WAL API
>
>
> Key: HBASE-20952
> URL: https://issues.apache.org/jira/browse/HBASE-20952
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Josh Elser
>Priority: Major
> Attachments: 20952.v1.txt
>
>
> Take a step back from the current WAL implementations and think about what an
> HBase WAL API should look like. What are the primitive calls that we require
> to guarantee durability of writes with a high degree of performance?
> The API needs to take the current implementations into consideration. We
> should also have a mind for what is happening in the Ratis LogService (but
> the LogService should not dictate what HBase's WAL API looks like RATIS-272).
> Other "systems" inside of HBase that use WALs are replication and
> backup Replication has the use-case for "tail"'ing the WAL which we
> should provide via our new API. B doesn't do anything fancy (IIRC). We
> should make sure all consumers are generally going to be OK with the API we
> create.
> The API may be "OK" (or OK in a part). We need to also consider other methods
> which were "bolted" on such as {{AbstractFSWAL}} and
> {{WALFileLengthProvider}}. Other corners of "WAL use" (like the
> {{WALSplitter}} should also be looked at to use WAL-APIs only).
> We also need to make sure that adequate interface audience and stability
> annotations are chosen.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21473) RowIndexSeekerV1 may return cell with extra two \x00\x00 bytes which has no tags
[
https://issues.apache.org/jira/browse/HBASE-21473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685331#comment-16685331
]
Hadoop QA commented on HBASE-21473:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
0s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
58s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
8m 50s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
26s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}214m 29s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
54s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}259m 14s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.master.assignment.TestCloseRegionWhileRSCrash |
| | hadoop.hbase.client.TestRestoreSnapshotFromClientSchemaChange |
| | hadoop.hbase.client.TestSnapshotTemporaryDirectoryWithRegionReplicas |
| | hadoop.hbase.client.TestRestoreSnapshotFromClientClone |
| | hadoop.hbase.regionserver.wal.TestAsyncWALReplay |
| |
hadoop.hbase.security.visibility.TestVisibilityLabelsWithCustomVisLabService |
| | hadoop.hbase.security.access.TestRpcAccessChecks |
| | hadoop.hbase.master.procedure.TestProcedurePriority |
| | hadoop.hbase.regionserver.TestRegionReplicasAreDistributed |
| | hadoop.hbase.regionserver.wal.TestWALReplayBoundedLogWriterCreation |
| | hadoop.hbase.replication.TestReplicationDroppedTables |
| |
[jira] [Commented] (HBASE-21465) Retry on reportRegionStateTransition can lead to unexpected errors
[ https://issues.apache.org/jira/browse/HBASE-21465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685327#comment-16685327 ] Duo Zhang commented on HBASE-21465: --- OK, I think the problem is that, if we fail to close the region normally, the openSeqNum may not be increased after a reopen, if we haven't written anything into it. Let me think how to deal with this. > Retry on reportRegionStateTransition can lead to unexpected errors > -- > > Key: HBASE-21465 > URL: https://issues.apache.org/jira/browse/HBASE-21465 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-21465-UT.patch, HBASE-21465-v1.patch, > HBASE-21465-v2.patch, HBASE-21465.patch > > > It is possible that the reportRegionStateTransition method is succeeded at > master side, but before returning the the region server, the rpc connection > is broken, or the master restarts. So next when the region server try > again,we will find that the state for the region and the TRSP is not correct, > and can lead to a RS abort or something even worse. > We should be able to determine whether a reportRegionStateTransition call is > just a retry and has already been succeeded, and just ignore it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21428) Performance issue due to userRegionLock in the ConnectionManager.
[
https://issues.apache.org/jira/browse/HBASE-21428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685285#comment-16685285
]
Sean Busbey commented on HBASE-21428:
-
This should be impacting all branches, correct?
> Performance issue due to userRegionLock in the ConnectionManager.
> -
>
> Key: HBASE-21428
> URL: https://issues.apache.org/jira/browse/HBASE-21428
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.7
>Reporter: koo
>Priority: Major
>
> My service is that execute a lot of puts using HTableMultiplexer.
> After the version change, most of the requests are rejected.
> It works fine in 1.2.6.1, but there is a problem in 1.2.7.
> This issue is related with the HBASE-19260.
> Most of my threads are using a lot of time as below.
>
> |"Worker-972" #2479 daemon prio=5 os_prio=0 tid=0x7f8cea86b000 nid=0x4c8c
> waiting on condition [0x7f8b78104000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x0005dd703b78> (a
> java.util.concurrent.locks.ReentrantLock$NonfairSync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
> at
> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
> at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1274)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1186)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1170)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1127)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:962)
> at
> org.apache.hadoop.hbase.client.HTableMultiplexer.put(HTableMultiplexer.java:206)
> at
> org.apache.hadoop.hbase.client.HTableMultiplexer.put(HTableMultiplexer.java:150)|
>
> When I looked at the issue(HBASE-19260), I recognized the dangerous of to
> allow accessessing multiple threads.
> However, Already create many threads with the limitations
> I think it is very inefficient to allow only one thread access.
>
> | this.metaLookupPool = getThreadPool(
> conf.getInt("hbase.hconnection.meta.lookup.threads.max", 128),
> conf.getInt("hbase.hconnection.meta.lookup.threads.core", 10),
> "-metaLookup-shared-", new LinkedBlockingQueue());|
>
> I want to suggest changing it that allow to have multiple locks.(but not the
> entire thread)
> The following is pseudocode.
>
> |int lockSize = conf.getInt("hbase.hconnection.meta.lookup.threads.max", 128)
> / 2;
> BlockingQueue userRegionLockQueue = new
> LinkedBlockingQueue();
> for (int i=0; i userRegionLockQueue.put(new ReentrantLock());
> }|
>
> thanks.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21466) WALProcedureStore uses wrong FileSystem if wal.dir is not under rootdir
[
https://issues.apache.org/jira/browse/HBASE-21466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685279#comment-16685279
]
Hudson commented on HBASE-21466:
Results for branch master
[build #602 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/602/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/602//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/602//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/602//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> WALProcedureStore uses wrong FileSystem if wal.dir is not under rootdir
> ---
>
> Key: HBASE-21466
> URL: https://issues.apache.org/jira/browse/HBASE-21466
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: 21466.v2.txt, 21466.v3.txt, 21466.v3.txt
>
>
> In WALProcedureStore ctor , the fs field is initialized this way:
> {code}
> this.fs = walDir.getFileSystem(conf);
> {code}
> However, when wal.dir is not under rootdir, the above would return wrong
> FileSystem.
> In the modified TestMasterProcedureEvents, without fix, the master wouldn't
> initialize.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21465) Retry on reportRegionStateTransition can lead to unexpected errors
[
https://issues.apache.org/jira/browse/HBASE-21465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685281#comment-16685281
]
Hudson commented on HBASE-21465:
Results for branch master
[build #602 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/602/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/602//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/602//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/602//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Retry on reportRegionStateTransition can lead to unexpected errors
> --
>
> Key: HBASE-21465
> URL: https://issues.apache.org/jira/browse/HBASE-21465
> Project: HBase
> Issue Type: Sub-task
> Components: amv2
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21465-UT.patch, HBASE-21465-v1.patch,
> HBASE-21465-v2.patch, HBASE-21465.patch
>
>
> It is possible that the reportRegionStateTransition method is succeeded at
> master side, but before returning the the region server, the rpc connection
> is broken, or the master restarts. So next when the region server try
> again,we will find that the state for the region and the TRSP is not correct,
> and can lead to a RS abort or something even worse.
> We should be able to determine whether a reportRegionStateTransition call is
> just a retry and has already been succeeded, and just ignore it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21463) The checkOnlineRegionsReport can accidentally complete a TRSP
[
https://issues.apache.org/jira/browse/HBASE-21463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685280#comment-16685280
]
Hudson commented on HBASE-21463:
Results for branch master
[build #602 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/602/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/602//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/602//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/602//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> The checkOnlineRegionsReport can accidentally complete a TRSP
> -
>
> Key: HBASE-21463
> URL: https://issues.apache.org/jira/browse/HBASE-21463
> Project: HBase
> Issue Type: Sub-task
> Components: amv2
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21463-UT.patch, HBASE-21463-v1.patch,
> HBASE-21463-v2.patch, HBASE-21463.patch
>
>
> On our testing cluster, we observe a race condition:
> 1. A regionServerReport request is built
> 2. A TRSP is scheduled to reopen the region
> 3. The region is closed at RS side
> 4. The OpenRegionProcedure is created
> 5. The regionServerReport generated at step 1 is executed, and we find that
> the region is opened on the RS, so we update the region state to OPEN.
> 6. The OpenRegionProcedure notices that the region has already been in the
> OPEN state so gives up and finishes.
> 7. The TRSP finishes.
> 8. The region is recorded as OPEN on the RS but actually not, and can not
> recover unless we use HBCK2.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (HBASE-21465) Retry on reportRegionStateTransition can lead to unexpected errors
[ https://issues.apache.org/jira/browse/HBASE-21465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reopened HBASE-21465: --- A bunch UTs are failed. Let me revert the patch first. > Retry on reportRegionStateTransition can lead to unexpected errors > -- > > Key: HBASE-21465 > URL: https://issues.apache.org/jira/browse/HBASE-21465 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-21465-UT.patch, HBASE-21465-v1.patch, > HBASE-21465-v2.patch, HBASE-21465.patch > > > It is possible that the reportRegionStateTransition method is succeeded at > master side, but before returning the the region server, the rpc connection > is broken, or the master restarts. So next when the region server try > again,we will find that the state for the region and the TRSP is not correct, > and can lead to a RS abort or something even worse. > We should be able to determine whether a reportRegionStateTransition call is > just a retry and has already been succeeded, and just ignore it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21472) Should not persist the dispatched field for RegionRemoteProcedureBase
[ https://issues.apache.org/jira/browse/HBASE-21472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685271#comment-16685271 ] Duo Zhang commented on HBASE-21472: --- Seems introduced by HBASE-21465. > Should not persist the dispatched field for RegionRemoteProcedureBase > - > > Key: HBASE-21472 > URL: https://issues.apache.org/jira/browse/HBASE-21472 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-21472-UT.patch, HBASE-21472.patch > > > It is OK to send the same open/close region request to a RS multiple times > but if we persist the dispatched field, after master restarts the procedure > will finish immediately, even if the reportRegionStateTransition has not been > called yet. This may lead to assign a region to multiple region servers... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21472) Should not persist the dispatched field for RegionRemoteProcedureBase
[
https://issues.apache.org/jira/browse/HBASE-21472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685216#comment-16685216
]
Hadoop QA commented on HBASE-21472:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
50s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
44s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
7s{color} | {color:red} hbase-server: The patch generated 1 new + 0 unchanged -
0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
46s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
8m 21s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green}
1m 9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
32s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}197m 51s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
51s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}244m 9s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestRegionReplicaFailover |
| | hadoop.hbase.client.TestRestoreSnapshotFromClientAfterTruncate |
| | hadoop.hbase.client.TestCloneSnapshotFromClientAfterSplittingRegion |
| | hadoop.hbase.master.assignment.TestRegionMoveAndAbandon |
| |
hadoop.hbase.security.visibility.TestVisibilityLabelsWithCustomVisLabService |
| | hadoop.hbase.procedure.TestFailedProcCleanup |
| |
[jira] [Commented] (HBASE-21468) separate workers for meta table is not working
[ https://issues.apache.org/jira/browse/HBASE-21468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685095#comment-16685095 ] Anoop Sam John commented on HBASE-21468: Is there any new configs around the original issue patch? Did not check the patches. > separate workers for meta table is not working > -- > > Key: HBASE-21468 > URL: https://issues.apache.org/jira/browse/HBASE-21468 > Project: HBase > Issue Type: Bug >Affects Versions: 2.1.1, 2.0.2 >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > Attachments: HBASE-21468.branch-2.0.001.patch > > > This is an addendum for HBASE-21423, since HBASE-21423 is already closed, the > QA won't be triggered. > It is my mistake that the separate workers for meta table is not working, > since when polling from queue, the onlyUrgent flag is not passed in. > And for some UT that only require one worker thread, urgent workers should > set to 0 to ensure there is one worker at time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21469) Re-visit post* hooks in DDL operations
[ https://issues.apache.org/jira/browse/HBASE-21469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685085#comment-16685085 ] Anoop Sam John commented on HBASE-21469: DDL ops other than modifytable and truncate table way make sense IMO. bq.but will cause RPC timeout) So jira to address this issue? > Re-visit post* hooks in DDL operations > -- > > Key: HBASE-21469 > URL: https://issues.apache.org/jira/browse/HBASE-21469 > Project: HBase > Issue Type: Bug >Affects Versions: 2.1.1, 2.0.2 >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > > I have some discuss in HBASE-19953 from > [here|https://issues.apache.org/jira/browse/HBASE-19953?focusedCommentId=16673126=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16673126] > In HBASE-19953,[~elserj] want to make sure that the post* hooks are called > only when the procedures finish.But it accidentally turns modifytable and > truncate table request into a sync call, which make clients RPC timeout > easily on big tables. > We should re-visit those postxxx hooks in DDL operations, because they are > now not consistent now: > For DDLs other than modifytable and truncate table, although the call will > wait on the latch, the latch is actually released just after prepare state, > so we still call postxxx hooks before the operation finish. > For DDLs of modifytable and truncate, the latch is only released after the > whole procedure finish. So the effort works(but will cause RPC timeout) > I think these latches are designed for compatibility with 1.x clients. Take > ModifyTable for example, in 1.x, we use admin.getAlterStauts() to check the > alter status, but in 2.x, this method is deprecated and returning inaccurate > result, we have to make 1.x client in a sync wait. > And for the semantics of postxxx hooks in 1.x, we will call them after the > corresponding DDL request return, but actually, the DDL request may not > finished also since we don't want for region assignment. > So, here, we need to discuss the semantics of postxxx hooks in DDL > operations, we need to make it consistent in every DDL operations, do we > really need to make sure this hooks being called only after the operation > finish? What's more, we have postCompletedxxx hooks for that need. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21473) RowIndexSeekerV1 may return cell with extra two \x00\x00 bytes which has no tags
[ https://issues.apache.org/jira/browse/HBASE-21473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685075#comment-16685075 ] Anoop Sam John commented on HBASE-21473: +1 > RowIndexSeekerV1 may return cell with extra two \x00\x00 bytes which has no > tags > > > Key: HBASE-21473 > URL: https://issues.apache.org/jira/browse/HBASE-21473 > Project: HBase > Issue Type: Sub-task >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Major > Attachments: HBASE-21473.v1.patch, HBASE-21473.v1.patch > > > When testing HBASE-21401, we found that two UT related to RowIndexSeekerV1 > always failed [1]. because RowIndexSeekerV1 may return cell with extra two > \x00\x00 bytes which has no tags.. > I plan to fix this bug in HBASE-21401, but seems the checkKeyValueBytes > need more performance evaluation, as [~Apache9] & [~stack] comment. I opened > issue HBASE-21459 to address it. > So, let me fix this bug firstly (Also will provide a seperate UT for this). > 1. > https://issues.apache.org/jira/browse/HBASE-21401?focusedCommentId=1671=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-1671 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21473) RowIndexSeekerV1 may return cell with extra two \x00\x00 bytes which has no tags
[ https://issues.apache.org/jira/browse/HBASE-21473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Hu updated HBASE-21473: - Attachment: HBASE-21473.v1.patch > RowIndexSeekerV1 may return cell with extra two \x00\x00 bytes which has no > tags > > > Key: HBASE-21473 > URL: https://issues.apache.org/jira/browse/HBASE-21473 > Project: HBase > Issue Type: Sub-task >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Major > Attachments: HBASE-21473.v1.patch, HBASE-21473.v1.patch > > > When testing HBASE-21401, we found that two UT related to RowIndexSeekerV1 > always failed [1]. because RowIndexSeekerV1 may return cell with extra two > \x00\x00 bytes which has no tags.. > I plan to fix this bug in HBASE-21401, but seems the checkKeyValueBytes > need more performance evaluation, as [~Apache9] & [~stack] comment. I opened > issue HBASE-21459 to address it. > So, let me fix this bug firstly (Also will provide a seperate UT for this). > 1. > https://issues.apache.org/jira/browse/HBASE-21401?focusedCommentId=1671=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-1671 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21473) RowIndexSeekerV1 may return cell with extra two \x00\x00 bytes which has no tags
[ https://issues.apache.org/jira/browse/HBASE-21473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685001#comment-16685001 ] Zheng Hu commented on HBASE-21473: -- Retry... > RowIndexSeekerV1 may return cell with extra two \x00\x00 bytes which has no > tags > > > Key: HBASE-21473 > URL: https://issues.apache.org/jira/browse/HBASE-21473 > Project: HBase > Issue Type: Sub-task >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Major > Attachments: HBASE-21473.v1.patch, HBASE-21473.v1.patch > > > When testing HBASE-21401, we found that two UT related to RowIndexSeekerV1 > always failed [1]. because RowIndexSeekerV1 may return cell with extra two > \x00\x00 bytes which has no tags.. > I plan to fix this bug in HBASE-21401, but seems the checkKeyValueBytes > need more performance evaluation, as [~Apache9] & [~stack] comment. I opened > issue HBASE-21459 to address it. > So, let me fix this bug firstly (Also will provide a seperate UT for this). > 1. > https://issues.apache.org/jira/browse/HBASE-21401?focusedCommentId=1671=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-1671 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21473) RowIndexSeekerV1 may return cell with extra two \x00\x00 bytes which has no tags
[
https://issues.apache.org/jira/browse/HBASE-21473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684973#comment-16684973
]
Hadoop QA commented on HBASE-21473:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
4s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
10s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 2s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
25s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}181m 24s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
50s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}227m 33s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.master.assignment.TestCloseRegionWhileRSCrash |
| | hadoop.hbase.client.TestRestoreSnapshotFromClientSchemaChange |
| | hadoop.hbase.client.TestSnapshotTemporaryDirectoryWithRegionReplicas |
| | hadoop.hbase.client.TestRestoreSnapshotFromClientClone |
| | hadoop.hbase.regionserver.wal.TestAsyncWALReplay |
| |
hadoop.hbase.security.visibility.TestVisibilityLabelsWithCustomVisLabService |
| | hadoop.hbase.master.procedure.TestProcedurePriority |
| |
hadoop.hbase.replication.TestSyncReplicationMoreLogsInLocalGiveUpSplitting |
| | hadoop.hbase.regionserver.TestRegionReplicasAreDistributed |
| | hadoop.hbase.regionserver.wal.TestWALReplayBoundedLogWriterCreation |
| | hadoop.hbase.regionserver.TestRegionReplicaFailover |
| |
[jira] [Commented] (HBASE-21472) Should not persist the dispatched field for RegionRemoteProcedureBase
[ https://issues.apache.org/jira/browse/HBASE-21472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684960#comment-16684960 ] Duo Zhang commented on HBASE-21472: --- Review board link: https://reviews.apache.org/r/69326/ > Should not persist the dispatched field for RegionRemoteProcedureBase > - > > Key: HBASE-21472 > URL: https://issues.apache.org/jira/browse/HBASE-21472 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-21472-UT.patch, HBASE-21472.patch > > > It is OK to send the same open/close region request to a RS multiple times > but if we persist the dispatched field, after master restarts the procedure > will finish immediately, even if the reportRegionStateTransition has not been > called yet. This may lead to assign a region to multiple region servers... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21034) Add new throttle type: read/write capacity unit
[ https://issues.apache.org/jira/browse/HBASE-21034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Mei updated HBASE-21034: --- Attachment: HBASE-21034.master.003.patch > Add new throttle type: read/write capacity unit > --- > > Key: HBASE-21034 > URL: https://issues.apache.org/jira/browse/HBASE-21034 > Project: HBase > Issue Type: Sub-task >Affects Versions: 3.0.0, 2.2.0 >Reporter: Yi Mei >Assignee: Yi Mei >Priority: Major > Attachments: HBASE-21034.master.001.patch, > HBASE-21034.master.002.patch, HBASE-21034.master.003.patch > > > Add new throttle type: read/write capacity unit like DynamoDB. > One read capacity unit represents that read up to 1K data per time unit. If > data size is more than 1K, then consume additional read capacity units. > One write capacity unit represents that one write for an item up to 1 KB in > size per time unit. If data size is more than 1K, then consume additional > write capacity units. > For example, 100 read capacity units per second means that, HBase user can > read 100 times for 1K data in every second, or 50 times for 2K data in every > second and so on. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21472) Should not persist the dispatched field for RegionRemoteProcedureBase
[ https://issues.apache.org/jira/browse/HBASE-21472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-21472: -- Attachment: HBASE-21472.patch > Should not persist the dispatched field for RegionRemoteProcedureBase > - > > Key: HBASE-21472 > URL: https://issues.apache.org/jira/browse/HBASE-21472 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Reporter: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-21472-UT.patch, HBASE-21472.patch > > > It is OK to send the same open/close region request to a RS multiple times > but if we persist the dispatched field, after master restarts the procedure > will finish immediately, even if the reportRegionStateTransition has not been > called yet. This may lead to assign a region to multiple region servers... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21472) Should not persist the dispatched field for RegionRemoteProcedureBase
[ https://issues.apache.org/jira/browse/HBASE-21472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-21472: -- Assignee: Duo Zhang Status: Patch Available (was: Open) > Should not persist the dispatched field for RegionRemoteProcedureBase > - > > Key: HBASE-21472 > URL: https://issues.apache.org/jira/browse/HBASE-21472 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-21472-UT.patch, HBASE-21472.patch > > > It is OK to send the same open/close region request to a RS multiple times > but if we persist the dispatched field, after master restarts the procedure > will finish immediately, even if the reportRegionStateTransition has not been > called yet. This may lead to assign a region to multiple region servers... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21472) Should not persist the dispatched field for RegionRemoteProcedureBase
[ https://issues.apache.org/jira/browse/HBASE-21472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-21472: -- Attachment: HBASE-21472-UT.patch > Should not persist the dispatched field for RegionRemoteProcedureBase > - > > Key: HBASE-21472 > URL: https://issues.apache.org/jira/browse/HBASE-21472 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Reporter: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-21472-UT.patch > > > It is OK to send the same open/close region request to a RS multiple times > but if we persist the dispatched field, after master restarts the procedure > will finish immediately, even if the reportRegionStateTransition has not been > called yet. This may lead to assign a region to multiple region servers... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
