[jira] [Updated] (HBASE-20866) HBase 1.x scan performance degradation compared to 0.98 version

2018-07-11 Thread Vikas Vishwakarma (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-20866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vikas Vishwakarma updated HBASE-20866:
--
Attachment: HBASE-20866.branch-1.3.004.patch

> HBase 1.x scan performance degradation compared to 0.98 version
> ---
>
> Key: HBASE-20866
> URL: https://issues.apache.org/jira/browse/HBASE-20866
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.2
>Reporter: Vikas Vishwakarma
>Assignee: Vikas Vishwakarma
>Priority: Critical
> Fix For: 1.5.0, 1.2.7, 1.3.3, 1.4.6
>
> Attachments: HBASE-20866.branch-1.3.001.patch, 
> HBASE-20866.branch-1.3.002.patch, HBASE-20866.branch-1.3.003.patch
>
>
> Internally while testing 1.3 as part of migration from 0.98 to 1.3 we 
> observed perf degradation in scan performance for phoenix queries varying 
> from few 10's to upto 200% depending on the query being executed. We tried 
> simple native HBase scan and there also we saw upto 40% degradation in 
> performance when the number of column qualifiers are high (40-50+)
> To identify the root cause of performance diff between 0.98 and 1.3 we 
> carried out lot of experiments with profiling and git bisect iterations, 
> however we were not able to identify any particular source of scan 
> performance degradation and it looked like this is an accumulated degradation 
> of 5-10% over various enhancements and refactoring.
> We identified few major enhancements like partialResult handling, 
> ScannerContext with heartbeat processing, time/size limiting, RPC 
> refactoring, etc that could have contributed to small degradation in 
> performance which put together could be leading to large overall degradation.
> One of the changes is 
> [HBASE-11544|https://jira.apache.org/jira/browse/HBASE-11544] which 
> implements partialResult handling. In ClientScanner.java the results received 
> from server are cached on the client side by converting the result array into 
> an ArrayList. This function gets called in a loop depending on the number of 
> rows in the scan result. Example for ten’s of millions of rows scanned, this 
> can be called in the order of millions of times.
> In almost all the cases 99% of the time (except for handling partial results, 
> etc). We are just taking the resultsFromServer converting it into a ArrayList 
> resultsToAddToCache in addResultsToList(..) and then iterating over the list 
> again and adding it to cache in loadCache(..) as given in the code path below
> In ClientScanner → loadCache(..) → getResultsToAddToCache(..) → 
> addResultsToList(..) →
> {code:java}
> loadCache() {
> ...
>  List resultsToAddToCache =
>  getResultsToAddToCache(values, callable.isHeartbeatMessage());
> ...
> …
>    for (Result rs : resultsToAddToCache) {
>  rs = filterLoadedCell(rs);
>  cache.add(rs);
> ...
>    }
> }
> getResultsToAddToCache(..) {
> ..
>    final boolean isBatchSet = scan != null && scan.getBatch() > 0;
>    final boolean allowPartials = scan != null && 
> scan.getAllowPartialResults();
> ..
>    if (allowPartials || isBatchSet) {
>  addResultsToList(resultsToAddToCache, resultsFromServer, 0,
>    (null == resultsFromServer ? 0 : resultsFromServer.length));
>  return resultsToAddToCache;
>    }
> ...
> }
> private void addResultsToList(List outputList, Result[] inputArray, 
> int start, int end) {
>    if (inputArray == null || start < 0 || end > inputArray.length) return;
>    for (int i = start; i < end; i++) {
>  outputList.add(inputArray[i]);
>    }
>  }{code}
>  
> It looks like we can avoid the result array to arraylist conversion 
> (resultsFromServer --> resultsToAddToCache ) for the first case which is also 
> the most frequent case and instead directly take the values arraay returned 
> by callable and add it to the cache without converting it into ArrayList.
> I have taken both these flags allowPartials and isBatchSet out in loadcahe() 
> and I am directly adding values to scanner cache if the above condition is 
> pass instead of coverting it into arrayList by calling 
> getResultsToAddToCache(). For example:
> {code:java}
> protected void loadCache() throws IOException {
> Result[] values = null;
> ..
> final boolean isBatchSet = scan != null && scan.getBatch() > 0;
> final boolean allowPartials = scan != null && scan.getAllowPartialResults();
> ..
> for (;;) {
> try {
> values = call(callable, caller, scannerTimeout);
> ..
> } catch (DoNotRetryIOException | NeedUnmanagedConnectionException e) {
> ..
> }
> if (allowPartials || isBatchSet) {  // DIRECTLY COPY values TO CACHE
> if (values != null) {
> for (int v=0; v Result rs = values[v];
> 
> cache.add(rs);
> ...
> } else { // DO ALL THE REGULAR PARTIAL RESULT HANDLING ..
> List resultsToAddToCache =
> 

[jira] [Updated] (HBASE-20866) HBase 1.x scan performance degradation compared to 0.98 version

2018-07-11 Thread Vikas Vishwakarma (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-20866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vikas Vishwakarma updated HBASE-20866:
--
Attachment: (was: HBASE-20866.branch-1.3.004.patch)

> HBase 1.x scan performance degradation compared to 0.98 version
> ---
>
> Key: HBASE-20866
> URL: https://issues.apache.org/jira/browse/HBASE-20866
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.2
>Reporter: Vikas Vishwakarma
>Assignee: Vikas Vishwakarma
>Priority: Critical
> Fix For: 1.5.0, 1.2.7, 1.3.3, 1.4.6
>
> Attachments: HBASE-20866.branch-1.3.001.patch, 
> HBASE-20866.branch-1.3.002.patch, HBASE-20866.branch-1.3.003.patch
>
>
> Internally while testing 1.3 as part of migration from 0.98 to 1.3 we 
> observed perf degradation in scan performance for phoenix queries varying 
> from few 10's to upto 200% depending on the query being executed. We tried 
> simple native HBase scan and there also we saw upto 40% degradation in 
> performance when the number of column qualifiers are high (40-50+)
> To identify the root cause of performance diff between 0.98 and 1.3 we 
> carried out lot of experiments with profiling and git bisect iterations, 
> however we were not able to identify any particular source of scan 
> performance degradation and it looked like this is an accumulated degradation 
> of 5-10% over various enhancements and refactoring.
> We identified few major enhancements like partialResult handling, 
> ScannerContext with heartbeat processing, time/size limiting, RPC 
> refactoring, etc that could have contributed to small degradation in 
> performance which put together could be leading to large overall degradation.
> One of the changes is 
> [HBASE-11544|https://jira.apache.org/jira/browse/HBASE-11544] which 
> implements partialResult handling. In ClientScanner.java the results received 
> from server are cached on the client side by converting the result array into 
> an ArrayList. This function gets called in a loop depending on the number of 
> rows in the scan result. Example for ten’s of millions of rows scanned, this 
> can be called in the order of millions of times.
> In almost all the cases 99% of the time (except for handling partial results, 
> etc). We are just taking the resultsFromServer converting it into a ArrayList 
> resultsToAddToCache in addResultsToList(..) and then iterating over the list 
> again and adding it to cache in loadCache(..) as given in the code path below
> In ClientScanner → loadCache(..) → getResultsToAddToCache(..) → 
> addResultsToList(..) →
> {code:java}
> loadCache() {
> ...
>  List resultsToAddToCache =
>  getResultsToAddToCache(values, callable.isHeartbeatMessage());
> ...
> …
>    for (Result rs : resultsToAddToCache) {
>  rs = filterLoadedCell(rs);
>  cache.add(rs);
> ...
>    }
> }
> getResultsToAddToCache(..) {
> ..
>    final boolean isBatchSet = scan != null && scan.getBatch() > 0;
>    final boolean allowPartials = scan != null && 
> scan.getAllowPartialResults();
> ..
>    if (allowPartials || isBatchSet) {
>  addResultsToList(resultsToAddToCache, resultsFromServer, 0,
>    (null == resultsFromServer ? 0 : resultsFromServer.length));
>  return resultsToAddToCache;
>    }
> ...
> }
> private void addResultsToList(List outputList, Result[] inputArray, 
> int start, int end) {
>    if (inputArray == null || start < 0 || end > inputArray.length) return;
>    for (int i = start; i < end; i++) {
>  outputList.add(inputArray[i]);
>    }
>  }{code}
>  
> It looks like we can avoid the result array to arraylist conversion 
> (resultsFromServer --> resultsToAddToCache ) for the first case which is also 
> the most frequent case and instead directly take the values arraay returned 
> by callable and add it to the cache without converting it into ArrayList.
> I have taken both these flags allowPartials and isBatchSet out in loadcahe() 
> and I am directly adding values to scanner cache if the above condition is 
> pass instead of coverting it into arrayList by calling 
> getResultsToAddToCache(). For example:
> {code:java}
> protected void loadCache() throws IOException {
> Result[] values = null;
> ..
> final boolean isBatchSet = scan != null && scan.getBatch() > 0;
> final boolean allowPartials = scan != null && scan.getAllowPartialResults();
> ..
> for (;;) {
> try {
> values = call(callable, caller, scannerTimeout);
> ..
> } catch (DoNotRetryIOException | NeedUnmanagedConnectionException e) {
> ..
> }
> if (allowPartials || isBatchSet) {  // DIRECTLY COPY values TO CACHE
> if (values != null) {
> for (int v=0; v Result rs = values[v];
> 
> cache.add(rs);
> ...
> } else { // DO ALL THE REGULAR PARTIAL RESULT HANDLING ..
> List resultsToAddToCache 

[jira] [Commented] (HBASE-20866) HBase 1.x scan performance degradation compared to 0.98 version

2018-07-11 Thread Vikas Vishwakarma (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541158#comment-16541158
 ] 

Vikas Vishwakarma commented on HBASE-20866:
---

thanks for the review and suggestions.

I have just committed a change with a trivial change in HRegion to get a full 
QA run as suggested by [~yuzhih...@gmail.com]

I will also put up the patch on RB and take a look at the master branch

> HBase 1.x scan performance degradation compared to 0.98 version
> ---
>
> Key: HBASE-20866
> URL: https://issues.apache.org/jira/browse/HBASE-20866
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.2
>Reporter: Vikas Vishwakarma
>Assignee: Vikas Vishwakarma
>Priority: Critical
> Fix For: 1.5.0, 1.2.7, 1.3.3, 1.4.6
>
> Attachments: HBASE-20866.branch-1.3.001.patch, 
> HBASE-20866.branch-1.3.002.patch, HBASE-20866.branch-1.3.003.patch
>
>
> Internally while testing 1.3 as part of migration from 0.98 to 1.3 we 
> observed perf degradation in scan performance for phoenix queries varying 
> from few 10's to upto 200% depending on the query being executed. We tried 
> simple native HBase scan and there also we saw upto 40% degradation in 
> performance when the number of column qualifiers are high (40-50+)
> To identify the root cause of performance diff between 0.98 and 1.3 we 
> carried out lot of experiments with profiling and git bisect iterations, 
> however we were not able to identify any particular source of scan 
> performance degradation and it looked like this is an accumulated degradation 
> of 5-10% over various enhancements and refactoring.
> We identified few major enhancements like partialResult handling, 
> ScannerContext with heartbeat processing, time/size limiting, RPC 
> refactoring, etc that could have contributed to small degradation in 
> performance which put together could be leading to large overall degradation.
> One of the changes is 
> [HBASE-11544|https://jira.apache.org/jira/browse/HBASE-11544] which 
> implements partialResult handling. In ClientScanner.java the results received 
> from server are cached on the client side by converting the result array into 
> an ArrayList. This function gets called in a loop depending on the number of 
> rows in the scan result. Example for ten’s of millions of rows scanned, this 
> can be called in the order of millions of times.
> In almost all the cases 99% of the time (except for handling partial results, 
> etc). We are just taking the resultsFromServer converting it into a ArrayList 
> resultsToAddToCache in addResultsToList(..) and then iterating over the list 
> again and adding it to cache in loadCache(..) as given in the code path below
> In ClientScanner → loadCache(..) → getResultsToAddToCache(..) → 
> addResultsToList(..) →
> {code:java}
> loadCache() {
> ...
>  List resultsToAddToCache =
>  getResultsToAddToCache(values, callable.isHeartbeatMessage());
> ...
> …
>    for (Result rs : resultsToAddToCache) {
>  rs = filterLoadedCell(rs);
>  cache.add(rs);
> ...
>    }
> }
> getResultsToAddToCache(..) {
> ..
>    final boolean isBatchSet = scan != null && scan.getBatch() > 0;
>    final boolean allowPartials = scan != null && 
> scan.getAllowPartialResults();
> ..
>    if (allowPartials || isBatchSet) {
>  addResultsToList(resultsToAddToCache, resultsFromServer, 0,
>    (null == resultsFromServer ? 0 : resultsFromServer.length));
>  return resultsToAddToCache;
>    }
> ...
> }
> private void addResultsToList(List outputList, Result[] inputArray, 
> int start, int end) {
>    if (inputArray == null || start < 0 || end > inputArray.length) return;
>    for (int i = start; i < end; i++) {
>  outputList.add(inputArray[i]);
>    }
>  }{code}
>  
> It looks like we can avoid the result array to arraylist conversion 
> (resultsFromServer --> resultsToAddToCache ) for the first case which is also 
> the most frequent case and instead directly take the values arraay returned 
> by callable and add it to the cache without converting it into ArrayList.
> I have taken both these flags allowPartials and isBatchSet out in loadcahe() 
> and I am directly adding values to scanner cache if the above condition is 
> pass instead of coverting it into arrayList by calling 
> getResultsToAddToCache(). For example:
> {code:java}
> protected void loadCache() throws IOException {
> Result[] values = null;
> ..
> final boolean isBatchSet = scan != null && scan.getBatch() > 0;
> final boolean allowPartials = scan != null && scan.getAllowPartialResults();
> ..
> for (;;) {
> try {
> values = call(callable, caller, scannerTimeout);
> ..
> } catch (DoNotRetryIOException | NeedUnmanagedConnectionException e) {
> ..
> }
> if (allowPartials || isBatchSet) 

[jira] [Updated] (HBASE-20866) HBase 1.x scan performance degradation compared to 0.98 version

2018-07-11 Thread Vikas Vishwakarma (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-20866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vikas Vishwakarma updated HBASE-20866:
--
Attachment: HBASE-20866.branch-1.3.003.patch

> HBase 1.x scan performance degradation compared to 0.98 version
> ---
>
> Key: HBASE-20866
> URL: https://issues.apache.org/jira/browse/HBASE-20866
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.2
>Reporter: Vikas Vishwakarma
>Assignee: Vikas Vishwakarma
>Priority: Critical
> Fix For: 1.5.0, 1.2.7, 1.3.3, 1.4.6
>
> Attachments: HBASE-20866.branch-1.3.001.patch, 
> HBASE-20866.branch-1.3.002.patch, HBASE-20866.branch-1.3.003.patch
>
>
> Internally while testing 1.3 as part of migration from 0.98 to 1.3 we 
> observed perf degradation in scan performance for phoenix queries varying 
> from few 10's to upto 200% depending on the query being executed. We tried 
> simple native HBase scan and there also we saw upto 40% degradation in 
> performance when the number of column qualifiers are high (40-50+)
> To identify the root cause of performance diff between 0.98 and 1.3 we 
> carried out lot of experiments with profiling and git bisect iterations, 
> however we were not able to identify any particular source of scan 
> performance degradation and it looked like this is an accumulated degradation 
> of 5-10% over various enhancements and refactoring.
> We identified few major enhancements like partialResult handling, 
> ScannerContext with heartbeat processing, time/size limiting, RPC 
> refactoring, etc that could have contributed to small degradation in 
> performance which put together could be leading to large overall degradation.
> One of the changes is 
> [HBASE-11544|https://jira.apache.org/jira/browse/HBASE-11544] which 
> implements partialResult handling. In ClientScanner.java the results received 
> from server are cached on the client side by converting the result array into 
> an ArrayList. This function gets called in a loop depending on the number of 
> rows in the scan result. Example for ten’s of millions of rows scanned, this 
> can be called in the order of millions of times.
> In almost all the cases 99% of the time (except for handling partial results, 
> etc). We are just taking the resultsFromServer converting it into a ArrayList 
> resultsToAddToCache in addResultsToList(..) and then iterating over the list 
> again and adding it to cache in loadCache(..) as given in the code path below
> In ClientScanner → loadCache(..) → getResultsToAddToCache(..) → 
> addResultsToList(..) →
> {code:java}
> loadCache() {
> ...
>  List resultsToAddToCache =
>  getResultsToAddToCache(values, callable.isHeartbeatMessage());
> ...
> …
>    for (Result rs : resultsToAddToCache) {
>  rs = filterLoadedCell(rs);
>  cache.add(rs);
> ...
>    }
> }
> getResultsToAddToCache(..) {
> ..
>    final boolean isBatchSet = scan != null && scan.getBatch() > 0;
>    final boolean allowPartials = scan != null && 
> scan.getAllowPartialResults();
> ..
>    if (allowPartials || isBatchSet) {
>  addResultsToList(resultsToAddToCache, resultsFromServer, 0,
>    (null == resultsFromServer ? 0 : resultsFromServer.length));
>  return resultsToAddToCache;
>    }
> ...
> }
> private void addResultsToList(List outputList, Result[] inputArray, 
> int start, int end) {
>    if (inputArray == null || start < 0 || end > inputArray.length) return;
>    for (int i = start; i < end; i++) {
>  outputList.add(inputArray[i]);
>    }
>  }{code}
>  
> It looks like we can avoid the result array to arraylist conversion 
> (resultsFromServer --> resultsToAddToCache ) for the first case which is also 
> the most frequent case and instead directly take the values arraay returned 
> by callable and add it to the cache without converting it into ArrayList.
> I have taken both these flags allowPartials and isBatchSet out in loadcahe() 
> and I am directly adding values to scanner cache if the above condition is 
> pass instead of coverting it into arrayList by calling 
> getResultsToAddToCache(). For example:
> {code:java}
> protected void loadCache() throws IOException {
> Result[] values = null;
> ..
> final boolean isBatchSet = scan != null && scan.getBatch() > 0;
> final boolean allowPartials = scan != null && scan.getAllowPartialResults();
> ..
> for (;;) {
> try {
> values = call(callable, caller, scannerTimeout);
> ..
> } catch (DoNotRetryIOException | NeedUnmanagedConnectionException e) {
> ..
> }
> if (allowPartials || isBatchSet) {  // DIRECTLY COPY values TO CACHE
> if (values != null) {
> for (int v=0; v Result rs = values[v];
> 
> cache.add(rs);
> ...
> } else { // DO ALL THE REGULAR PARTIAL RESULT HANDLING ..
> List resultsToAddToCache =
> 

[jira] [Commented] (HBASE-20847) The parent procedure of RegionTransitionProcedure may not have the table lock

2018-07-11 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541154#comment-16541154
 ] 

Hudson commented on HBASE-20847:


Results for branch branch-2
[build #972 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/972/]: 
(x) *{color:red}-1 overall{color}*

details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/972//General_Nightly_Build_Report/]




(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/972//JDK8_Nightly_Build_Report_(Hadoop2)/]


(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/972//JDK8_Nightly_Build_Report_(Hadoop3)/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


(/) {color:green}+1 client integration test{color}


> The parent procedure of RegionTransitionProcedure may not have the table lock
> -
>
> Key: HBASE-20847
> URL: https://issues.apache.org/jira/browse/HBASE-20847
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2, Region Assignment
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1
>
> Attachments: HBASE-20847-addendum.patch, 
> HBASE-20847-branch-2.0-v1.patch, HBASE-20847-branch-2.0.patch, 
> HBASE-20847-v1.patch, HBASE-20847-v2.patch, HBASE-20847-v3.patch, 
> HBASE-20847.patch
>
>
> For example, SCP can also schedule AssignProcedure and obviously it will not 
> hold the table lock.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-19572) RegionMover should use the configured default port number and not the one from HConstants

2018-07-11 Thread Reid Chan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-19572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541150#comment-16541150
 ] 

Reid Chan commented on HBASE-19572:
---

Oh ye, it's {{RegionMoverBuilder}}.
{code}
+private static Configuration createConf() {
+  Configuration conf = HBaseConfiguration.create();
+  conf.setInt("hbase.client.prefetch.limit", 1);
+  conf.setInt("hbase.client.pause", 500);
+  conf.setInt("hbase.client.retries.number", 100);
+  return conf;
+}
{code}
I think there's a conf in super class {{AbstractHBaseTool.conf}}, should we 
create a new one unless is null? Following should be fine, (origin one also has 
problem).
{code}
+private static Configuration createConf() {
+  if (conf == null) {
+conf = HBaseConfiguration.create();
+  }
+  conf.setInt("hbase.client.prefetch.limit", 1);
+  conf.setInt("hbase.client.pause", 500);
+  conf.setInt("hbase.client.retries.number", 100);
+  return conf;
+}
{code}

bq. What did you mean?
{code}
+  } else {
+this.port = conf.getInt(HConstants.REGIONSERVER_PORT, 
HConstants.DEFAULT_REGIONSERVER_PORT);
   }
   setDefaultfilename(this.hostname);
+  this.conf = conf;
{code}
I only see it used for initializing port and does nothing afterwards, so i 
doubt it to be a member of  {{RegionMoverBuilder}}.

> RegionMover should use the configured default port number and not the one 
> from HConstants
> -
>
> Key: HBASE-19572
> URL: https://issues.apache.org/jira/browse/HBASE-19572
> Project: HBase
>  Issue Type: Bug
>Reporter: Esteban Gutierrez
>Assignee: Toshihiro Suzuki
>Priority: Major
> Attachments: HBASE-19572.master.001.patch, 
> HBASE-19572.master.001.patch, HBASE-19572.master.003.patch, 
> HBASE-19572.master.004.patch, HBASE-19572.master.004.patch, 
> HBASE-19572.patch, HBASE-19572.patch
>
>
> The issue I ran into HBASE-19499 was due RegionMover not using the port used 
> by {{hbase-site.xml}}. The tool should use the value used in the 
> configuration before falling back to the hardcoded value 
> {{HConstants.DEFAULT_REGIONSERVER_PORT}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20704) Sometimes some compacted storefiles are not archived on region close

2018-07-11 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541148#comment-16541148
 ] 

Hadoop QA commented on HBASE-20704:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
 5s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
28s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
46s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  1m  
6s{color} | {color:red} hbase-server: The patch generated 7 new + 43 unchanged 
- 0 fixed = 50 total (was 43) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
16s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}  
9m 23s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4 
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}171m 
51s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
19s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}210m 59s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20704 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12931253/HBASE-20704.003.patch 
|
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  shadedjars  
hadoopcheck  hbaseanti  checkstyle  compile  |
| uname | Linux 33abdcb37624 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14 
08:53:28 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 4a08135dd8 |
| maven | version: Apache Maven 3.5.4 
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC3 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HBASE-Build/13601/artifact/patchprocess/diff-checkstyle-hbase-server.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/13601/testReport/ |
| Max. process+thread count | 5204 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output | 

[jira] [Commented] (HBASE-20697) Can't cache All region locations of the specify table by calling table.getRegionLocator().getAllRegionLocations()

2018-07-11 Thread Andrew Purtell (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541147#comment-16541147
 ] 

Andrew Purtell commented on HBASE-20697:


+1 on addendum, thanks

> Can't cache All region locations of the specify table by calling 
> table.getRegionLocator().getAllRegionLocations()
> -
>
> Key: HBASE-20697
> URL: https://issues.apache.org/jira/browse/HBASE-20697
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.1, 1.2.6, 2.0.1
>Reporter: zhaoyuan
>Assignee: zhaoyuan
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.4.6, 2.0.2, 2.2.0, 2.1.1
>
> Attachments: HBASE-20697.branch-1.2.001.patch, 
> HBASE-20697.branch-1.2.002.patch, HBASE-20697.branch-1.2.003.patch, 
> HBASE-20697.branch-1.2.004.patch, HBASE-20697.branch-1.addendum.patch, 
> HBASE-20697.master.001.patch, HBASE-20697.master.002.patch, 
> HBASE-20697.master.002.patch, HBASE-20697.master.003.patch
>
>
> When we upgrade and restart  a new version application which will read and 
> write to HBase, we will get some operation timeout. The time out is expected 
> because when the application restarts,It will not hold any region locations 
> cache and do communication with zk and meta regionserver to get region 
> locations.
> We want to avoid these timeouts so we do warmup work and as far as I am 
> concerned,the method table.getRegionLocator().getAllRegionLocations() will 
> fetch all region locations and cache them. However, it didn't work good. 
> There are still a lot of time outs,so it confused me. 
> I dig into the source code and find something below
> {code:java}
> // code placeholder
> public List getAllRegionLocations() throws IOException {
>   TableName tableName = getName();
>   NavigableMap locations =
>   MetaScanner.allTableRegions(this.connection, tableName);
>   ArrayList regions = new ArrayList<>(locations.size());
>   for (Entry entry : locations.entrySet()) {
> regions.add(new HRegionLocation(entry.getKey(), entry.getValue()));
>   }
>   if (regions.size() > 0) {
> connection.cacheLocation(tableName, new RegionLocations(regions));
>   }
>   return regions;
> }
> In MetaCache
> public void cacheLocation(final TableName tableName, final RegionLocations 
> locations) {
>   byte [] startKey = 
> locations.getRegionLocation().getRegionInfo().getStartKey();
>   ConcurrentMap tableLocations = 
> getTableLocations(tableName);
>   RegionLocations oldLocation = tableLocations.putIfAbsent(startKey, 
> locations);
>   boolean isNewCacheEntry = (oldLocation == null);
>   if (isNewCacheEntry) {
> if (LOG.isTraceEnabled()) {
>   LOG.trace("Cached location: " + locations);
> }
> addToCachedServers(locations);
> return;
>   }
> {code}
> It will collect all regions into one RegionLocations object and only cache 
> the first not null region location and then when we put or get to hbase, we 
> do getCacheLocation() 
> {code:java}
> // code placeholder
> public RegionLocations getCachedLocation(final TableName tableName, final 
> byte [] row) {
>   ConcurrentNavigableMap tableLocations =
> getTableLocations(tableName);
>   Entry e = tableLocations.floorEntry(row);
>   if (e == null) {
> if (metrics!= null) metrics.incrMetaCacheMiss();
> return null;
>   }
>   RegionLocations possibleRegion = e.getValue();
>   // make sure that the end key is greater than the row we're looking
>   // for, otherwise the row actually belongs in the next region, not
>   // this one. the exception case is when the endkey is
>   // HConstants.EMPTY_END_ROW, signifying that the region we're
>   // checking is actually the last region in the table.
>   byte[] endKey = 
> possibleRegion.getRegionLocation().getRegionInfo().getEndKey();
>   if (Bytes.equals(endKey, HConstants.EMPTY_END_ROW) ||
>   getRowComparator(tableName).compareRows(
>   endKey, 0, endKey.length, row, 0, row.length) > 0) {
> if (metrics != null) metrics.incrMetaCacheHit();
> return possibleRegion;
>   }
>   // Passed all the way through, so we got nothing - complete cache miss
>   if (metrics != null) metrics.incrMetaCacheMiss();
>   return null;
> }
> {code}
> It will choose the first location to be possibleRegion and possibly it will 
> miss match.
> So did I forget something or may be wrong somewhere? If this is indeed a bug 
> I think it can be fixed not very hard.
> Hope commiters and PMC review this !
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HBASE-17877) Improve HBase's byte[] comparator

2018-07-11 Thread Vikas Vishwakarma (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vikas Vishwakarma updated HBASE-17877:
--
Attachment: HBASE-17877.branch-1.3.004.patch

> Improve HBase's byte[] comparator
> -
>
> Key: HBASE-17877
> URL: https://issues.apache.org/jira/browse/HBASE-17877
> Project: HBase
>  Issue Type: Improvement
>  Components: util
>Reporter: Lars Hofhansl
>Assignee: Vikas Vishwakarma
>Priority: Major
> Fix For: 1.4.0, 1.3.2, 2.0.0
>
> Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch, 
> 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf, 
> HBASE-17877.branch-1.3.001.patch, HBASE-17877.branch-1.3.002.patch, 
> HBASE-17877.branch-1.3.003.patch, HBASE-17877.branch-1.3.004.patch, 
> HBASE-17877.master.001.patch, HBASE-17877.master.002.patch, 
> HBASE-17877.master.003.patch
>
>
> [~vik.karma] did some extensive tests and found that Hadoop's version is 
> faster - dramatically faster in some cases.
> Patch forthcoming.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20847) The parent procedure of RegionTransitionProcedure may not have the table lock

2018-07-11 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541134#comment-16541134
 ] 

Hudson commented on HBASE-20847:


Results for branch branch-2.1
[build #50 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/50/]: 
(x) *{color:red}-1 overall{color}*

details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/50//General_Nightly_Build_Report/]




(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/50//JDK8_Nightly_Build_Report_(Hadoop2)/]


(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/50//JDK8_Nightly_Build_Report_(Hadoop3)/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


(/) {color:green}+1 client integration test{color}


> The parent procedure of RegionTransitionProcedure may not have the table lock
> -
>
> Key: HBASE-20847
> URL: https://issues.apache.org/jira/browse/HBASE-20847
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2, Region Assignment
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1
>
> Attachments: HBASE-20847-addendum.patch, 
> HBASE-20847-branch-2.0-v1.patch, HBASE-20847-branch-2.0.patch, 
> HBASE-20847-v1.patch, HBASE-20847-v2.patch, HBASE-20847-v3.patch, 
> HBASE-20847.patch
>
>
> For example, SCP can also schedule AssignProcedure and obviously it will not 
> hold the table lock.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-19572) RegionMover should use the configured default port number and not the one from HConstants

2018-07-11 Thread Toshihiro Suzuki (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-19572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541128#comment-16541128
 ] 

Toshihiro Suzuki commented on HBASE-19572:
--

Thank you for reviewing [~reidchan].

Regarding 1, If I remember correctly, I thought we needed to make "final int 
port" package private for test, so I made all members package private for 
consistency. But I don't think it's strong reason, I'll revert it.

Regarding 2, I think "final Configuration conf" is not a member of RegionMover, 
but RegionMoverBuilder. What did you mean?

Thanks.

> RegionMover should use the configured default port number and not the one 
> from HConstants
> -
>
> Key: HBASE-19572
> URL: https://issues.apache.org/jira/browse/HBASE-19572
> Project: HBase
>  Issue Type: Bug
>Reporter: Esteban Gutierrez
>Assignee: Toshihiro Suzuki
>Priority: Major
> Attachments: HBASE-19572.master.001.patch, 
> HBASE-19572.master.001.patch, HBASE-19572.master.003.patch, 
> HBASE-19572.master.004.patch, HBASE-19572.master.004.patch, 
> HBASE-19572.patch, HBASE-19572.patch
>
>
> The issue I ran into HBASE-19499 was due RegionMover not using the port used 
> by {{hbase-site.xml}}. The tool should use the value used in the 
> configuration before falling back to the hardcoded value 
> {{HConstants.DEFAULT_REGIONSERVER_PORT}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20697) Can't cache All region locations of the specify table by calling table.getRegionLocator().getAllRegionLocations()

2018-07-11 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541127#comment-16541127
 ] 

Hadoop QA commented on HBASE-20697:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
40s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
15s{color} | {color:green} branch-1 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
19s{color} | {color:green} branch-1 passed with JDK v1.7.0_181 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
28s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  2m 
39s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
16s{color} | {color:green} branch-1 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
20s{color} | {color:green} branch-1 passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
28s{color} | {color:green} hbase-client: The patch generated 0 new + 1 
unchanged - 4 fixed = 1 total (was 5) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  2m 
36s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}  
1m 35s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
20s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
22s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
 9s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 37s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:1f3957d |
| JIRA Issue | HBASE-20697 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12931267/HBASE-20697.branch-1.addendum.patch
 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  shadedjars  
hadoopcheck  hbaseanti  

[jira] [Updated] (HBASE-20697) Can't cache All region locations of the specify table by calling table.getRegionLocator().getAllRegionLocations()

2018-07-11 Thread Guanghao Zhang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-20697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guanghao Zhang updated HBASE-20697:
---
Status: Patch Available  (was: Reopened)

> Can't cache All region locations of the specify table by calling 
> table.getRegionLocator().getAllRegionLocations()
> -
>
> Key: HBASE-20697
> URL: https://issues.apache.org/jira/browse/HBASE-20697
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.1, 1.2.6, 1.3.1
>Reporter: zhaoyuan
>Assignee: zhaoyuan
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.4.6, 2.0.2, 2.2.0, 2.1.1
>
> Attachments: HBASE-20697.branch-1.2.001.patch, 
> HBASE-20697.branch-1.2.002.patch, HBASE-20697.branch-1.2.003.patch, 
> HBASE-20697.branch-1.2.004.patch, HBASE-20697.branch-1.addendum.patch, 
> HBASE-20697.master.001.patch, HBASE-20697.master.002.patch, 
> HBASE-20697.master.002.patch, HBASE-20697.master.003.patch
>
>
> When we upgrade and restart  a new version application which will read and 
> write to HBase, we will get some operation timeout. The time out is expected 
> because when the application restarts,It will not hold any region locations 
> cache and do communication with zk and meta regionserver to get region 
> locations.
> We want to avoid these timeouts so we do warmup work and as far as I am 
> concerned,the method table.getRegionLocator().getAllRegionLocations() will 
> fetch all region locations and cache them. However, it didn't work good. 
> There are still a lot of time outs,so it confused me. 
> I dig into the source code and find something below
> {code:java}
> // code placeholder
> public List getAllRegionLocations() throws IOException {
>   TableName tableName = getName();
>   NavigableMap locations =
>   MetaScanner.allTableRegions(this.connection, tableName);
>   ArrayList regions = new ArrayList<>(locations.size());
>   for (Entry entry : locations.entrySet()) {
> regions.add(new HRegionLocation(entry.getKey(), entry.getValue()));
>   }
>   if (regions.size() > 0) {
> connection.cacheLocation(tableName, new RegionLocations(regions));
>   }
>   return regions;
> }
> In MetaCache
> public void cacheLocation(final TableName tableName, final RegionLocations 
> locations) {
>   byte [] startKey = 
> locations.getRegionLocation().getRegionInfo().getStartKey();
>   ConcurrentMap tableLocations = 
> getTableLocations(tableName);
>   RegionLocations oldLocation = tableLocations.putIfAbsent(startKey, 
> locations);
>   boolean isNewCacheEntry = (oldLocation == null);
>   if (isNewCacheEntry) {
> if (LOG.isTraceEnabled()) {
>   LOG.trace("Cached location: " + locations);
> }
> addToCachedServers(locations);
> return;
>   }
> {code}
> It will collect all regions into one RegionLocations object and only cache 
> the first not null region location and then when we put or get to hbase, we 
> do getCacheLocation() 
> {code:java}
> // code placeholder
> public RegionLocations getCachedLocation(final TableName tableName, final 
> byte [] row) {
>   ConcurrentNavigableMap tableLocations =
> getTableLocations(tableName);
>   Entry e = tableLocations.floorEntry(row);
>   if (e == null) {
> if (metrics!= null) metrics.incrMetaCacheMiss();
> return null;
>   }
>   RegionLocations possibleRegion = e.getValue();
>   // make sure that the end key is greater than the row we're looking
>   // for, otherwise the row actually belongs in the next region, not
>   // this one. the exception case is when the endkey is
>   // HConstants.EMPTY_END_ROW, signifying that the region we're
>   // checking is actually the last region in the table.
>   byte[] endKey = 
> possibleRegion.getRegionLocation().getRegionInfo().getEndKey();
>   if (Bytes.equals(endKey, HConstants.EMPTY_END_ROW) ||
>   getRowComparator(tableName).compareRows(
>   endKey, 0, endKey.length, row, 0, row.length) > 0) {
> if (metrics != null) metrics.incrMetaCacheHit();
> return possibleRegion;
>   }
>   // Passed all the way through, so we got nothing - complete cache miss
>   if (metrics != null) metrics.incrMetaCacheMiss();
>   return null;
> }
> {code}
> It will choose the first location to be possibleRegion and possibly it will 
> miss match.
> So did I forget something or may be wrong somewhere? If this is indeed a bug 
> I think it can be fixed not very hard.
> Hope commiters and PMC review this !
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HBASE-20697) Can't cache All region locations of the specify table by calling table.getRegionLocator().getAllRegionLocations()

2018-07-11 Thread Guanghao Zhang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-20697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guanghao Zhang updated HBASE-20697:
---
Attachment: HBASE-20697.branch-1.addendum.patch

> Can't cache All region locations of the specify table by calling 
> table.getRegionLocator().getAllRegionLocations()
> -
>
> Key: HBASE-20697
> URL: https://issues.apache.org/jira/browse/HBASE-20697
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.1, 1.2.6, 2.0.1
>Reporter: zhaoyuan
>Assignee: zhaoyuan
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.4.6, 2.0.2, 2.2.0, 2.1.1
>
> Attachments: HBASE-20697.branch-1.2.001.patch, 
> HBASE-20697.branch-1.2.002.patch, HBASE-20697.branch-1.2.003.patch, 
> HBASE-20697.branch-1.2.004.patch, HBASE-20697.branch-1.addendum.patch, 
> HBASE-20697.master.001.patch, HBASE-20697.master.002.patch, 
> HBASE-20697.master.002.patch, HBASE-20697.master.003.patch
>
>
> When we upgrade and restart  a new version application which will read and 
> write to HBase, we will get some operation timeout. The time out is expected 
> because when the application restarts,It will not hold any region locations 
> cache and do communication with zk and meta regionserver to get region 
> locations.
> We want to avoid these timeouts so we do warmup work and as far as I am 
> concerned,the method table.getRegionLocator().getAllRegionLocations() will 
> fetch all region locations and cache them. However, it didn't work good. 
> There are still a lot of time outs,so it confused me. 
> I dig into the source code and find something below
> {code:java}
> // code placeholder
> public List getAllRegionLocations() throws IOException {
>   TableName tableName = getName();
>   NavigableMap locations =
>   MetaScanner.allTableRegions(this.connection, tableName);
>   ArrayList regions = new ArrayList<>(locations.size());
>   for (Entry entry : locations.entrySet()) {
> regions.add(new HRegionLocation(entry.getKey(), entry.getValue()));
>   }
>   if (regions.size() > 0) {
> connection.cacheLocation(tableName, new RegionLocations(regions));
>   }
>   return regions;
> }
> In MetaCache
> public void cacheLocation(final TableName tableName, final RegionLocations 
> locations) {
>   byte [] startKey = 
> locations.getRegionLocation().getRegionInfo().getStartKey();
>   ConcurrentMap tableLocations = 
> getTableLocations(tableName);
>   RegionLocations oldLocation = tableLocations.putIfAbsent(startKey, 
> locations);
>   boolean isNewCacheEntry = (oldLocation == null);
>   if (isNewCacheEntry) {
> if (LOG.isTraceEnabled()) {
>   LOG.trace("Cached location: " + locations);
> }
> addToCachedServers(locations);
> return;
>   }
> {code}
> It will collect all regions into one RegionLocations object and only cache 
> the first not null region location and then when we put or get to hbase, we 
> do getCacheLocation() 
> {code:java}
> // code placeholder
> public RegionLocations getCachedLocation(final TableName tableName, final 
> byte [] row) {
>   ConcurrentNavigableMap tableLocations =
> getTableLocations(tableName);
>   Entry e = tableLocations.floorEntry(row);
>   if (e == null) {
> if (metrics!= null) metrics.incrMetaCacheMiss();
> return null;
>   }
>   RegionLocations possibleRegion = e.getValue();
>   // make sure that the end key is greater than the row we're looking
>   // for, otherwise the row actually belongs in the next region, not
>   // this one. the exception case is when the endkey is
>   // HConstants.EMPTY_END_ROW, signifying that the region we're
>   // checking is actually the last region in the table.
>   byte[] endKey = 
> possibleRegion.getRegionLocation().getRegionInfo().getEndKey();
>   if (Bytes.equals(endKey, HConstants.EMPTY_END_ROW) ||
>   getRowComparator(tableName).compareRows(
>   endKey, 0, endKey.length, row, 0, row.length) > 0) {
> if (metrics != null) metrics.incrMetaCacheHit();
> return possibleRegion;
>   }
>   // Passed all the way through, so we got nothing - complete cache miss
>   if (metrics != null) metrics.incrMetaCacheMiss();
>   return null;
> }
> {code}
> It will choose the first location to be possibleRegion and possibly it will 
> miss match.
> So did I forget something or may be wrong somewhere? If this is indeed a bug 
> I think it can be fixed not very hard.
> Hope commiters and PMC review this !
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20679) Add the ability to compile JSP dynamically in Jetty

2018-07-11 Thread Allan Yang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541098#comment-16541098
 ] 

Allan Yang commented on HBASE-20679:


[~stack], are we still need this? Today, I use this trick fix the RIT regions 
again in our test env. The Procedure wal is Corrupted with no reason.  
{quote}
2018-07-12 10:11:01,082 ERROR [master/test-sy-vpc-219-002:16000] 
procedure2.ProcedureExecutor(327): Corrupt pid=362, ppid=333, 
state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=h, regi
on=b3f59c4fb17d0b5a4836479e931d5640, 
server=test-sy-vpc-219-004.hbase.9b78df04-b.rds.aliyuncs.com,16020,1529657866452
{quote}

> Add the ability to compile JSP dynamically in Jetty
> ---
>
> Key: HBASE-20679
> URL: https://issues.apache.org/jira/browse/HBASE-20679
> Project: HBase
>  Issue Type: New Feature
>Affects Versions: 2.0.0
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20679.002.patch, HBASE-20679.003.patch, 
> HBASE-20679.004.patch, HBASE-20679.005.patch, HBASE-20679.patch
>
>
> As discussed in HBASE-20617, adding the ability to dynamically compile jsp 
> enable us to do some hot fix. 
>  For example, several days ago, in our testing HBase-2.0 cluster, 
> procedureWals were corrupted due to some unknown reasons. After restarting 
> the cluster, since some procedures(AssignProcedure for example) were 
> corrupted and couldn't be replayed. Some regions were stuck in RIT forever. 
> We couldn't use HBCK since it haven't support AssignmentV2 yet. As a matter 
> of fact, the namespace region was not online, so the master was not inited, 
> we even couldn't use shell command like assign/move. But, we wrote a jsp and 
> fix this issue easily. The jsp file is like this:
> {code:java}
> <%
>   String action = request.getParameter("action");
>   HMaster master = (HMaster)getServletContext().getAttribute(HMaster.MASTER);
>   List offlineRegionsToAssign = new ArrayList<>();
>   List regionRITs = 
> master.getAssignmentManager()
>   .getRegionStates().getRegionsInTransition();
>   for (RegionStates.RegionStateNode regionStateNode :  regionRITs) {
> // if regionStateNode don't have a procedure attached, but meta state 
> shows
> // this region is in RIT, that means the previous procedure may be 
> corrupted
> // we need to create a new assignProcedure to assign them
> if (!regionStateNode.isInTransition()) {
>   offlineRegionsToAssign.add(regionStateNode.getRegionInfo());
>   out.println("RIT region:" + regionStateNode);
> }
>   }
>   // Assign offline regions. Uses round-robin.
>   if ("fix".equals(action) && offlineRegionsToAssign.size() > 0) {
> 
> master.getMasterProcedureExecutor().submitProcedures(master.getAssignmentManager().
> createRoundRobinAssignProcedures(offlineRegionsToAssign));
>   } else {
> out.println("use ?action=fix to fix RIT regions");
>   }
> %>
> {code}
> Above it is only one example we can do if we have the ability to compile jsp 
> dynamically. We think it is very useful.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Reopened] (HBASE-20697) Can't cache All region locations of the specify table by calling table.getRegionLocator().getAllRegionLocations()

2018-07-11 Thread Guanghao Zhang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-20697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guanghao Zhang reopened HBASE-20697:


Reopen for fix checkstyle for branch-1.

> Can't cache All region locations of the specify table by calling 
> table.getRegionLocator().getAllRegionLocations()
> -
>
> Key: HBASE-20697
> URL: https://issues.apache.org/jira/browse/HBASE-20697
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.1, 1.2.6, 2.0.1
>Reporter: zhaoyuan
>Assignee: zhaoyuan
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.4.6, 2.0.2, 2.2.0, 2.1.1
>
> Attachments: HBASE-20697.branch-1.2.001.patch, 
> HBASE-20697.branch-1.2.002.patch, HBASE-20697.branch-1.2.003.patch, 
> HBASE-20697.branch-1.2.004.patch, HBASE-20697.master.001.patch, 
> HBASE-20697.master.002.patch, HBASE-20697.master.002.patch, 
> HBASE-20697.master.003.patch
>
>
> When we upgrade and restart  a new version application which will read and 
> write to HBase, we will get some operation timeout. The time out is expected 
> because when the application restarts,It will not hold any region locations 
> cache and do communication with zk and meta regionserver to get region 
> locations.
> We want to avoid these timeouts so we do warmup work and as far as I am 
> concerned,the method table.getRegionLocator().getAllRegionLocations() will 
> fetch all region locations and cache them. However, it didn't work good. 
> There are still a lot of time outs,so it confused me. 
> I dig into the source code and find something below
> {code:java}
> // code placeholder
> public List getAllRegionLocations() throws IOException {
>   TableName tableName = getName();
>   NavigableMap locations =
>   MetaScanner.allTableRegions(this.connection, tableName);
>   ArrayList regions = new ArrayList<>(locations.size());
>   for (Entry entry : locations.entrySet()) {
> regions.add(new HRegionLocation(entry.getKey(), entry.getValue()));
>   }
>   if (regions.size() > 0) {
> connection.cacheLocation(tableName, new RegionLocations(regions));
>   }
>   return regions;
> }
> In MetaCache
> public void cacheLocation(final TableName tableName, final RegionLocations 
> locations) {
>   byte [] startKey = 
> locations.getRegionLocation().getRegionInfo().getStartKey();
>   ConcurrentMap tableLocations = 
> getTableLocations(tableName);
>   RegionLocations oldLocation = tableLocations.putIfAbsent(startKey, 
> locations);
>   boolean isNewCacheEntry = (oldLocation == null);
>   if (isNewCacheEntry) {
> if (LOG.isTraceEnabled()) {
>   LOG.trace("Cached location: " + locations);
> }
> addToCachedServers(locations);
> return;
>   }
> {code}
> It will collect all regions into one RegionLocations object and only cache 
> the first not null region location and then when we put or get to hbase, we 
> do getCacheLocation() 
> {code:java}
> // code placeholder
> public RegionLocations getCachedLocation(final TableName tableName, final 
> byte [] row) {
>   ConcurrentNavigableMap tableLocations =
> getTableLocations(tableName);
>   Entry e = tableLocations.floorEntry(row);
>   if (e == null) {
> if (metrics!= null) metrics.incrMetaCacheMiss();
> return null;
>   }
>   RegionLocations possibleRegion = e.getValue();
>   // make sure that the end key is greater than the row we're looking
>   // for, otherwise the row actually belongs in the next region, not
>   // this one. the exception case is when the endkey is
>   // HConstants.EMPTY_END_ROW, signifying that the region we're
>   // checking is actually the last region in the table.
>   byte[] endKey = 
> possibleRegion.getRegionLocation().getRegionInfo().getEndKey();
>   if (Bytes.equals(endKey, HConstants.EMPTY_END_ROW) ||
>   getRowComparator(tableName).compareRows(
>   endKey, 0, endKey.length, row, 0, row.length) > 0) {
> if (metrics != null) metrics.incrMetaCacheHit();
> return possibleRegion;
>   }
>   // Passed all the way through, so we got nothing - complete cache miss
>   if (metrics != null) metrics.incrMetaCacheMiss();
>   return null;
> }
> {code}
> It will choose the first location to be possibleRegion and possibly it will 
> miss match.
> So did I forget something or may be wrong somewhere? If this is indeed a bug 
> I think it can be fixed not very hard.
> Hope commiters and PMC review this !
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-19572) RegionMover should use the configured default port number and not the one from HConstants

2018-07-11 Thread Reid Chan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-19572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541081#comment-16541081
 ] 

Reid Chan commented on HBASE-19572:
---

LGTM overall.

Two questions,
1. Any particular reason why changes the visibility to package visible 
{code}
boolean ack = true;
{code}
2. {{final Configuration conf}} do we have to make it a member of 
{{RegionMover}}. Remove it otherwise

> RegionMover should use the configured default port number and not the one 
> from HConstants
> -
>
> Key: HBASE-19572
> URL: https://issues.apache.org/jira/browse/HBASE-19572
> Project: HBase
>  Issue Type: Bug
>Reporter: Esteban Gutierrez
>Assignee: Toshihiro Suzuki
>Priority: Major
> Attachments: HBASE-19572.master.001.patch, 
> HBASE-19572.master.001.patch, HBASE-19572.master.003.patch, 
> HBASE-19572.master.004.patch, HBASE-19572.master.004.patch, 
> HBASE-19572.patch, HBASE-19572.patch
>
>
> The issue I ran into HBASE-19499 was due RegionMover not using the port used 
> by {{hbase-site.xml}}. The tool should use the value used in the 
> configuration before falling back to the hardcoded value 
> {{HConstants.DEFAULT_REGIONSERVER_PORT}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20697) Can't cache All region locations of the specify table by calling table.getRegionLocator().getAllRegionLocations()

2018-07-11 Thread Guanghao Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541082#comment-16541082
 ] 

Guanghao Zhang commented on HBASE-20697:


Pushed to branch-2.0, too.

> Can't cache All region locations of the specify table by calling 
> table.getRegionLocator().getAllRegionLocations()
> -
>
> Key: HBASE-20697
> URL: https://issues.apache.org/jira/browse/HBASE-20697
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.1, 1.2.6, 2.0.1
>Reporter: zhaoyuan
>Assignee: zhaoyuan
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.4.6, 2.0.2, 2.2.0, 2.1.1
>
> Attachments: HBASE-20697.branch-1.2.001.patch, 
> HBASE-20697.branch-1.2.002.patch, HBASE-20697.branch-1.2.003.patch, 
> HBASE-20697.branch-1.2.004.patch, HBASE-20697.master.001.patch, 
> HBASE-20697.master.002.patch, HBASE-20697.master.002.patch, 
> HBASE-20697.master.003.patch
>
>
> When we upgrade and restart  a new version application which will read and 
> write to HBase, we will get some operation timeout. The time out is expected 
> because when the application restarts,It will not hold any region locations 
> cache and do communication with zk and meta regionserver to get region 
> locations.
> We want to avoid these timeouts so we do warmup work and as far as I am 
> concerned,the method table.getRegionLocator().getAllRegionLocations() will 
> fetch all region locations and cache them. However, it didn't work good. 
> There are still a lot of time outs,so it confused me. 
> I dig into the source code and find something below
> {code:java}
> // code placeholder
> public List getAllRegionLocations() throws IOException {
>   TableName tableName = getName();
>   NavigableMap locations =
>   MetaScanner.allTableRegions(this.connection, tableName);
>   ArrayList regions = new ArrayList<>(locations.size());
>   for (Entry entry : locations.entrySet()) {
> regions.add(new HRegionLocation(entry.getKey(), entry.getValue()));
>   }
>   if (regions.size() > 0) {
> connection.cacheLocation(tableName, new RegionLocations(regions));
>   }
>   return regions;
> }
> In MetaCache
> public void cacheLocation(final TableName tableName, final RegionLocations 
> locations) {
>   byte [] startKey = 
> locations.getRegionLocation().getRegionInfo().getStartKey();
>   ConcurrentMap tableLocations = 
> getTableLocations(tableName);
>   RegionLocations oldLocation = tableLocations.putIfAbsent(startKey, 
> locations);
>   boolean isNewCacheEntry = (oldLocation == null);
>   if (isNewCacheEntry) {
> if (LOG.isTraceEnabled()) {
>   LOG.trace("Cached location: " + locations);
> }
> addToCachedServers(locations);
> return;
>   }
> {code}
> It will collect all regions into one RegionLocations object and only cache 
> the first not null region location and then when we put or get to hbase, we 
> do getCacheLocation() 
> {code:java}
> // code placeholder
> public RegionLocations getCachedLocation(final TableName tableName, final 
> byte [] row) {
>   ConcurrentNavigableMap tableLocations =
> getTableLocations(tableName);
>   Entry e = tableLocations.floorEntry(row);
>   if (e == null) {
> if (metrics!= null) metrics.incrMetaCacheMiss();
> return null;
>   }
>   RegionLocations possibleRegion = e.getValue();
>   // make sure that the end key is greater than the row we're looking
>   // for, otherwise the row actually belongs in the next region, not
>   // this one. the exception case is when the endkey is
>   // HConstants.EMPTY_END_ROW, signifying that the region we're
>   // checking is actually the last region in the table.
>   byte[] endKey = 
> possibleRegion.getRegionLocation().getRegionInfo().getEndKey();
>   if (Bytes.equals(endKey, HConstants.EMPTY_END_ROW) ||
>   getRowComparator(tableName).compareRows(
>   endKey, 0, endKey.length, row, 0, row.length) > 0) {
> if (metrics != null) metrics.incrMetaCacheHit();
> return possibleRegion;
>   }
>   // Passed all the way through, so we got nothing - complete cache miss
>   if (metrics != null) metrics.incrMetaCacheMiss();
>   return null;
> }
> {code}
> It will choose the first location to be possibleRegion and possibly it will 
> miss match.
> So did I forget something or may be wrong somewhere? If this is indeed a bug 
> I think it can be fixed not very hard.
> Hope commiters and PMC review this !
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20875) MemStoreLABImp::copyIntoCell uses 7% CPU when writing

2018-07-11 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541074#comment-16541074
 ] 

Hadoop QA commented on HBASE-20875:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
33s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red}  0m  3s{color} 
| {color:red} HBASE-20875 does not apply to master. Rebase required? Wrong 
Branch? See https://yetus.apache.org/documentation/0.7.0/precommit-patchnames 
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20875 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12931264/0001-HBASE-20875-MemStoreLABImp-copyIntoCell-uses-7-CPU-w.patch
 |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/13602/console |
| Powered by | Apache Yetus 0.7.0   http://yetus.apache.org |


This message was automatically generated.



> MemStoreLABImp::copyIntoCell uses 7% CPU when writing
> -
>
> Key: HBASE-20875
> URL: https://issues.apache.org/jira/browse/HBASE-20875
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Affects Versions: 2.0.1
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.0.2
>
> Attachments: 
> 0001-HBASE-20875-MemStoreLABImp-copyIntoCell-uses-7-CPU-w.patch, 
> 2.0707.baseline.91935.cpu.svg, 2.0711.patched.145414.cpu.svg, Screen Shot 
> 2018-07-11 at 9.52.46 PM.png
>
>
> Looks like this with a PE random write loading:
> {code}
>  ./hbase/bin/hbase  --config ~/conf_hbase 
> org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --presplit=40  
> --size=30 --columns=10 --valueSize=100 randomWrite 200
> {code}
> ... against a single server.
> {code}
> 12.47%  perf-91935.map
> [.] Lorg/apache/hadoop/hbase/BBKVComparator;::compare
>  10.42%  libjvm.so
>  [.] 
> ParNewGeneration::copy_to_survivor_space_avoiding_promotion_undo(ParScanThreadState*,
>  oopDesc*, unsigned long, markOopDesc*)
>   6.78%  perf-91935.map   
>  [.] 
> Lorg/apache/hadoop/hbase/regionserver/MemStoreLABImpl;::copyCellInto
> 
> {code}
> These are top CPU consumers using perf-map-agent ./bin/perf-java-top... 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20875) MemStoreLABImp::copyIntoCell uses 7% CPU when writing

2018-07-11 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541072#comment-16541072
 ] 

stack commented on HBASE-20875:
---

Makes for a small improvement in throughput. Nothing dramatic so no graph.

#compare still the long pole.

> MemStoreLABImp::copyIntoCell uses 7% CPU when writing
> -
>
> Key: HBASE-20875
> URL: https://issues.apache.org/jira/browse/HBASE-20875
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Affects Versions: 2.0.1
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.0.2
>
> Attachments: 
> 0001-HBASE-20875-MemStoreLABImp-copyIntoCell-uses-7-CPU-w.patch, 
> 2.0707.baseline.91935.cpu.svg, 2.0711.patched.145414.cpu.svg, Screen Shot 
> 2018-07-11 at 9.52.46 PM.png
>
>
> Looks like this with a PE random write loading:
> {code}
>  ./hbase/bin/hbase  --config ~/conf_hbase 
> org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --presplit=40  
> --size=30 --columns=10 --valueSize=100 randomWrite 200
> {code}
> ... against a single server.
> {code}
> 12.47%  perf-91935.map
> [.] Lorg/apache/hadoop/hbase/BBKVComparator;::compare
>  10.42%  libjvm.so
>  [.] 
> ParNewGeneration::copy_to_survivor_space_avoiding_promotion_undo(ParScanThreadState*,
>  oopDesc*, unsigned long, markOopDesc*)
>   6.78%  perf-91935.map   
>  [.] 
> Lorg/apache/hadoop/hbase/regionserver/MemStoreLABImpl;::copyCellInto
> 
> {code}
> These are top CPU consumers using perf-map-agent ./bin/perf-java-top... 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HBASE-20875) MemStoreLABImp::copyIntoCell uses 7% CPU when writing

2018-07-11 Thread stack (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-20875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-20875:
--
 Assignee: stack
Fix Version/s: 2.0.2
Affects Version/s: 2.0.1
   Status: Patch Available  (was: Open)

Small patch. Adds a smaller copyCellInto called copyBBECellInto picked up by 
the hotpath.

> MemStoreLABImp::copyIntoCell uses 7% CPU when writing
> -
>
> Key: HBASE-20875
> URL: https://issues.apache.org/jira/browse/HBASE-20875
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Affects Versions: 2.0.1
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.0.2
>
> Attachments: 
> 0001-HBASE-20875-MemStoreLABImp-copyIntoCell-uses-7-CPU-w.patch, 
> 2.0707.baseline.91935.cpu.svg, 2.0711.patched.145414.cpu.svg, Screen Shot 
> 2018-07-11 at 9.52.46 PM.png
>
>
> Looks like this with a PE random write loading:
> {code}
>  ./hbase/bin/hbase  --config ~/conf_hbase 
> org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --presplit=40  
> --size=30 --columns=10 --valueSize=100 randomWrite 200
> {code}
> ... against a single server.
> {code}
> 12.47%  perf-91935.map
> [.] Lorg/apache/hadoop/hbase/BBKVComparator;::compare
>  10.42%  libjvm.so
>  [.] 
> ParNewGeneration::copy_to_survivor_space_avoiding_promotion_undo(ParScanThreadState*,
>  oopDesc*, unsigned long, markOopDesc*)
>   6.78%  perf-91935.map   
>  [.] 
> Lorg/apache/hadoop/hbase/regionserver/MemStoreLABImpl;::copyCellInto
> 
> {code}
> These are top CPU consumers using perf-map-agent ./bin/perf-java-top... 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HBASE-20875) MemStoreLABImp::copyIntoCell uses 7% CPU when writing

2018-07-11 Thread stack (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-20875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-20875:
--
Component/s: Performance

> MemStoreLABImp::copyIntoCell uses 7% CPU when writing
> -
>
> Key: HBASE-20875
> URL: https://issues.apache.org/jira/browse/HBASE-20875
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Affects Versions: 2.0.1
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.0.2
>
> Attachments: 
> 0001-HBASE-20875-MemStoreLABImp-copyIntoCell-uses-7-CPU-w.patch, 
> 2.0707.baseline.91935.cpu.svg, 2.0711.patched.145414.cpu.svg, Screen Shot 
> 2018-07-11 at 9.52.46 PM.png
>
>
> Looks like this with a PE random write loading:
> {code}
>  ./hbase/bin/hbase  --config ~/conf_hbase 
> org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --presplit=40  
> --size=30 --columns=10 --valueSize=100 randomWrite 200
> {code}
> ... against a single server.
> {code}
> 12.47%  perf-91935.map
> [.] Lorg/apache/hadoop/hbase/BBKVComparator;::compare
>  10.42%  libjvm.so
>  [.] 
> ParNewGeneration::copy_to_survivor_space_avoiding_promotion_undo(ParScanThreadState*,
>  oopDesc*, unsigned long, markOopDesc*)
>   6.78%  perf-91935.map   
>  [.] 
> Lorg/apache/hadoop/hbase/regionserver/MemStoreLABImpl;::copyCellInto
> 
> {code}
> These are top CPU consumers using perf-map-agent ./bin/perf-java-top... 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HBASE-20875) MemStoreLABImp::copyIntoCell uses 7% CPU when writing

2018-07-11 Thread stack (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-20875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-20875:
--
Attachment: 0001-HBASE-20875-MemStoreLABImp-copyIntoCell-uses-7-CPU-w.patch

> MemStoreLABImp::copyIntoCell uses 7% CPU when writing
> -
>
> Key: HBASE-20875
> URL: https://issues.apache.org/jira/browse/HBASE-20875
> Project: HBase
>  Issue Type: Sub-task
>Reporter: stack
>Priority: Major
> Attachments: 
> 0001-HBASE-20875-MemStoreLABImp-copyIntoCell-uses-7-CPU-w.patch, 
> 2.0707.baseline.91935.cpu.svg, 2.0711.patched.145414.cpu.svg, Screen Shot 
> 2018-07-11 at 9.52.46 PM.png
>
>
> Looks like this with a PE random write loading:
> {code}
>  ./hbase/bin/hbase  --config ~/conf_hbase 
> org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --presplit=40  
> --size=30 --columns=10 --valueSize=100 randomWrite 200
> {code}
> ... against a single server.
> {code}
> 12.47%  perf-91935.map
> [.] Lorg/apache/hadoop/hbase/BBKVComparator;::compare
>  10.42%  libjvm.so
>  [.] 
> ParNewGeneration::copy_to_survivor_space_avoiding_promotion_undo(ParScanThreadState*,
>  oopDesc*, unsigned long, markOopDesc*)
>   6.78%  perf-91935.map   
>  [.] 
> Lorg/apache/hadoop/hbase/regionserver/MemStoreLABImpl;::copyCellInto
> 
> {code}
> These are top CPU consumers using perf-map-agent ./bin/perf-java-top... 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20875) MemStoreLABImp::copyIntoCell uses 7% CPU when writing

2018-07-11 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541070#comment-16541070
 ] 

stack commented on HBASE-20875:
---

 !Screen Shot 2018-07-11 at 9.52.46 PM.png! is from jitwatch. See how 
copyCellInto is just over inlin'ing threashold of 3k. This is before patch. 
Look at  [^2.0707.baseline.91935.cpu.svg] to the right of the biggest hump 
where we are doing maybeCloneWithAllocator. See how it is > 10%. In patched 
view, it is < 5%... copyBBECellInto is the diff.

> MemStoreLABImp::copyIntoCell uses 7% CPU when writing
> -
>
> Key: HBASE-20875
> URL: https://issues.apache.org/jira/browse/HBASE-20875
> Project: HBase
>  Issue Type: Sub-task
>Reporter: stack
>Priority: Major
> Attachments: 2.0707.baseline.91935.cpu.svg, 
> 2.0711.patched.145414.cpu.svg, Screen Shot 2018-07-11 at 9.52.46 PM.png
>
>
> Looks like this with a PE random write loading:
> {code}
>  ./hbase/bin/hbase  --config ~/conf_hbase 
> org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --presplit=40  
> --size=30 --columns=10 --valueSize=100 randomWrite 200
> {code}
> ... against a single server.
> {code}
> 12.47%  perf-91935.map
> [.] Lorg/apache/hadoop/hbase/BBKVComparator;::compare
>  10.42%  libjvm.so
>  [.] 
> ParNewGeneration::copy_to_survivor_space_avoiding_promotion_undo(ParScanThreadState*,
>  oopDesc*, unsigned long, markOopDesc*)
>   6.78%  perf-91935.map   
>  [.] 
> Lorg/apache/hadoop/hbase/regionserver/MemStoreLABImpl;::copyCellInto
> 
> {code}
> These are top CPU consumers using perf-map-agent ./bin/perf-java-top... 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20866) HBase 1.x scan performance degradation compared to 0.98 version

2018-07-11 Thread Reid Chan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541069#comment-16541069
 ] 

Reid Chan commented on HBASE-20866:
---

The core is to avoid conversion (resultsFromServer --> resultsToAddToCache)
LGTM. 

Same concern as Mike, and i skimmed through branch-1 and master branch. The 
conversion becomes [] to sub[], then add to cache. and there're much more work.


> HBase 1.x scan performance degradation compared to 0.98 version
> ---
>
> Key: HBASE-20866
> URL: https://issues.apache.org/jira/browse/HBASE-20866
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.2
>Reporter: Vikas Vishwakarma
>Assignee: Vikas Vishwakarma
>Priority: Critical
> Fix For: 1.5.0, 1.2.7, 1.3.3, 1.4.6
>
> Attachments: HBASE-20866.branch-1.3.001.patch, 
> HBASE-20866.branch-1.3.002.patch
>
>
> Internally while testing 1.3 as part of migration from 0.98 to 1.3 we 
> observed perf degradation in scan performance for phoenix queries varying 
> from few 10's to upto 200% depending on the query being executed. We tried 
> simple native HBase scan and there also we saw upto 40% degradation in 
> performance when the number of column qualifiers are high (40-50+)
> To identify the root cause of performance diff between 0.98 and 1.3 we 
> carried out lot of experiments with profiling and git bisect iterations, 
> however we were not able to identify any particular source of scan 
> performance degradation and it looked like this is an accumulated degradation 
> of 5-10% over various enhancements and refactoring.
> We identified few major enhancements like partialResult handling, 
> ScannerContext with heartbeat processing, time/size limiting, RPC 
> refactoring, etc that could have contributed to small degradation in 
> performance which put together could be leading to large overall degradation.
> One of the changes is 
> [HBASE-11544|https://jira.apache.org/jira/browse/HBASE-11544] which 
> implements partialResult handling. In ClientScanner.java the results received 
> from server are cached on the client side by converting the result array into 
> an ArrayList. This function gets called in a loop depending on the number of 
> rows in the scan result. Example for ten’s of millions of rows scanned, this 
> can be called in the order of millions of times.
> In almost all the cases 99% of the time (except for handling partial results, 
> etc). We are just taking the resultsFromServer converting it into a ArrayList 
> resultsToAddToCache in addResultsToList(..) and then iterating over the list 
> again and adding it to cache in loadCache(..) as given in the code path below
> In ClientScanner → loadCache(..) → getResultsToAddToCache(..) → 
> addResultsToList(..) →
> {code:java}
> loadCache() {
> ...
>  List resultsToAddToCache =
>  getResultsToAddToCache(values, callable.isHeartbeatMessage());
> ...
> …
>    for (Result rs : resultsToAddToCache) {
>  rs = filterLoadedCell(rs);
>  cache.add(rs);
> ...
>    }
> }
> getResultsToAddToCache(..) {
> ..
>    final boolean isBatchSet = scan != null && scan.getBatch() > 0;
>    final boolean allowPartials = scan != null && 
> scan.getAllowPartialResults();
> ..
>    if (allowPartials || isBatchSet) {
>  addResultsToList(resultsToAddToCache, resultsFromServer, 0,
>    (null == resultsFromServer ? 0 : resultsFromServer.length));
>  return resultsToAddToCache;
>    }
> ...
> }
> private void addResultsToList(List outputList, Result[] inputArray, 
> int start, int end) {
>    if (inputArray == null || start < 0 || end > inputArray.length) return;
>    for (int i = start; i < end; i++) {
>  outputList.add(inputArray[i]);
>    }
>  }{code}
>  
> It looks like we can avoid the result array to arraylist conversion 
> (resultsFromServer --> resultsToAddToCache ) for the first case which is also 
> the most frequent case and instead directly take the values arraay returned 
> by callable and add it to the cache without converting it into ArrayList.
> I have taken both these flags allowPartials and isBatchSet out in loadcahe() 
> and I am directly adding values to scanner cache if the above condition is 
> pass instead of coverting it into arrayList by calling 
> getResultsToAddToCache(). For example:
> {code:java}
> protected void loadCache() throws IOException {
> Result[] values = null;
> ..
> final boolean isBatchSet = scan != null && scan.getBatch() > 0;
> final boolean allowPartials = scan != null && scan.getAllowPartialResults();
> ..
> for (;;) {
> try {
> values = call(callable, caller, scannerTimeout);
> ..
> } catch (DoNotRetryIOException | NeedUnmanagedConnectionException e) {
> ..
> }
> if (allowPartials || isBatchSet) {  // DIRECTLY COPY values TO CACHE
> if 

[jira] [Updated] (HBASE-20875) MemStoreLABImp::copyIntoCell uses 7% CPU when writing

2018-07-11 Thread stack (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-20875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-20875:
--
Attachment: 2.0711.patched.145414.cpu.svg

> MemStoreLABImp::copyIntoCell uses 7% CPU when writing
> -
>
> Key: HBASE-20875
> URL: https://issues.apache.org/jira/browse/HBASE-20875
> Project: HBase
>  Issue Type: Sub-task
>Reporter: stack
>Priority: Major
> Attachments: 2.0707.baseline.91935.cpu.svg, 
> 2.0711.patched.145414.cpu.svg, Screen Shot 2018-07-11 at 9.52.46 PM.png
>
>
> Looks like this with a PE random write loading:
> {code}
>  ./hbase/bin/hbase  --config ~/conf_hbase 
> org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --presplit=40  
> --size=30 --columns=10 --valueSize=100 randomWrite 200
> {code}
> ... against a single server.
> {code}
> 12.47%  perf-91935.map
> [.] Lorg/apache/hadoop/hbase/BBKVComparator;::compare
>  10.42%  libjvm.so
>  [.] 
> ParNewGeneration::copy_to_survivor_space_avoiding_promotion_undo(ParScanThreadState*,
>  oopDesc*, unsigned long, markOopDesc*)
>   6.78%  perf-91935.map   
>  [.] 
> Lorg/apache/hadoop/hbase/regionserver/MemStoreLABImpl;::copyCellInto
> 
> {code}
> These are top CPU consumers using perf-map-agent ./bin/perf-java-top... 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HBASE-20875) MemStoreLABImp::copyIntoCell uses 7% CPU when writing

2018-07-11 Thread stack (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-20875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-20875:
--
Attachment: 2.0707.baseline.91935.cpu.svg

> MemStoreLABImp::copyIntoCell uses 7% CPU when writing
> -
>
> Key: HBASE-20875
> URL: https://issues.apache.org/jira/browse/HBASE-20875
> Project: HBase
>  Issue Type: Sub-task
>Reporter: stack
>Priority: Major
> Attachments: 2.0707.baseline.91935.cpu.svg, 
> 2.0711.patched.145414.cpu.svg, Screen Shot 2018-07-11 at 9.52.46 PM.png
>
>
> Looks like this with a PE random write loading:
> {code}
>  ./hbase/bin/hbase  --config ~/conf_hbase 
> org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --presplit=40  
> --size=30 --columns=10 --valueSize=100 randomWrite 200
> {code}
> ... against a single server.
> {code}
> 12.47%  perf-91935.map
> [.] Lorg/apache/hadoop/hbase/BBKVComparator;::compare
>  10.42%  libjvm.so
>  [.] 
> ParNewGeneration::copy_to_survivor_space_avoiding_promotion_undo(ParScanThreadState*,
>  oopDesc*, unsigned long, markOopDesc*)
>   6.78%  perf-91935.map   
>  [.] 
> Lorg/apache/hadoop/hbase/regionserver/MemStoreLABImpl;::copyCellInto
> 
> {code}
> These are top CPU consumers using perf-map-agent ./bin/perf-java-top... 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20865) CreateTableProcedure is stuck in retry loop in CREATE_TABLE_WRITE_FS_LAYOUT state

2018-07-11 Thread Ted Yu (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541067#comment-16541067
 ] 

Ted Yu commented on HBASE-20865:


lgtm

[~Apache9]:
Can you take a look as well ?

> CreateTableProcedure is stuck in retry loop in CREATE_TABLE_WRITE_FS_LAYOUT 
> state
> -
>
> Key: HBASE-20865
> URL: https://issues.apache.org/jira/browse/HBASE-20865
> Project: HBase
>  Issue Type: Bug
>  Components: amv2
>Reporter: Toshihiro Suzuki
>Assignee: Toshihiro Suzuki
>Priority: Major
> Attachments: HBASE-20865.master.001.patch
>
>
> Similar to HBASE-20616, CreateTableProcedure gets stuck in retry loop in 
> CREATE_TABLE_WRITE_FS_LAYOUT state when writing HDFS fails.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20875) MemStoreLABImp::copyIntoCell uses 7% CPU when writing

2018-07-11 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541066#comment-16541066
 ] 

stack commented on HBASE-20875:
---

Let me upload a patch. Has the method now using 1.6% of CPU. Here is the top 
methods:

{code}
 16.31%  libjvm.so  
   [.] 
ParNewGeneration::copy_to_survivor_space_avoiding_promotion_undo(ParScanThreadState*,
 oopDesc*, unsigned long, markOopDesc*)
 11.97%  perf-145414.map
   [.] Lorg/apache/hadoop/hbase/BBKVComparator;::compare
  6.16%  perf-145414.map
   [.] Lorg/apache/hadoop/hbase/BBKVComparator;::compare
  4.41%  perf-145414.map
   [.] Ljava/util/concurrent/ConcurrentSkipListMap;::doPut
  3.93%  perf-145414.map
   [.] 
Lorg/apache/hadoop/hbase/regionserver/AbstractMemStore;::add
  2.86%  perf-145414.map
   [.] Lorg/apache/hadoop/hbase/client/Put;::add
  2.39%  perf-145414.map
   [.] 
Lorg/apache/hadoop/hbase/regionserver/HRegion;::updateCellTimestamps
  1.96%  perf-145414.map
   [.] 
Lorg/apache/hadoop/hbase/regionserver/RSRpcServices;::checkCellSizeLimit
  1.71%  perf-145414.map
   [.] 
Lorg/apache/hadoop/hbase/regionserver/wal/AsyncProtobufLogWriter;::append
  1.64%  perf-145414.map
   [.] 
Lorg/apache/hadoop/hbase/regionserver/wal/AbstractFSWAL;::stampSequenceIdAndPublishToRingBuffer
  1.61%  perf-145414.map
   [.] 
Lorg/apache/hadoop/hbase/regionserver/MemStoreLABImpl;::copyBBECellInto

{code}

The new, purposed method name is copyBBECellInto, the equiv of the old 
copyCellInto.


> MemStoreLABImp::copyIntoCell uses 7% CPU when writing
> -
>
> Key: HBASE-20875
> URL: https://issues.apache.org/jira/browse/HBASE-20875
> Project: HBase
>  Issue Type: Sub-task
>Reporter: stack
>Priority: Major
> Attachments: Screen Shot 2018-07-11 at 9.52.46 PM.png
>
>
> Looks like this with a PE random write loading:
> {code}
>  ./hbase/bin/hbase  --config ~/conf_hbase 
> org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --presplit=40  
> --size=30 --columns=10 --valueSize=100 randomWrite 200
> {code}
> ... against a single server.
> {code}
> 12.47%  perf-91935.map
> [.] Lorg/apache/hadoop/hbase/BBKVComparator;::compare
>  10.42%  libjvm.so
>  [.] 
> ParNewGeneration::copy_to_survivor_space_avoiding_promotion_undo(ParScanThreadState*,
>  oopDesc*, unsigned long, markOopDesc*)
>   6.78%  perf-91935.map   
>  [.] 
> Lorg/apache/hadoop/hbase/regionserver/MemStoreLABImpl;::copyCellInto
> 
> {code}
> These are top CPU consumers using perf-map-agent ./bin/perf-java-top... 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-18840) Add functionality to refresh meta table at master startup

2018-07-11 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541064#comment-16541064
 ] 

Hadoop QA commented on HBASE-18840:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} HBASE-18477 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
18s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
21s{color} | {color:green} HBASE-18477 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  3m 
13s{color} | {color:green} HBASE-18477 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
 1s{color} | {color:green} HBASE-18477 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  5m 
30s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
47s{color} | {color:green} HBASE-18477 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
9s{color} | {color:green} HBASE-18477 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  5m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  3m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  3m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
40s{color} | {color:green} hbase-client: The patch generated 0 new + 124 
unchanged - 1 fixed = 124 total (was 125) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  1m 
22s{color} | {color:red} hbase-server: The patch generated 2 new + 242 
unchanged - 0 fixed = 244 total (was 242) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  5m 
31s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
13m 29s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m 
44s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}206m 40s{color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
43s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}273m 39s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestMetaWithReplicas |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-18840 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12931227/HBASE-18840.HBASE-18477.007.patch
 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  shadedjars  
hadoopcheck  hbaseanti  checkstyle  compile  |
| uname | Linux a5afd9e4e399 4.4.0-130-generic 

[jira] [Updated] (HBASE-20875) MemStoreLABImp::copyIntoCell uses 7% CPU when writing

2018-07-11 Thread stack (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-20875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-20875:
--
Attachment: Screen Shot 2018-07-11 at 9.52.46 PM.png

> MemStoreLABImp::copyIntoCell uses 7% CPU when writing
> -
>
> Key: HBASE-20875
> URL: https://issues.apache.org/jira/browse/HBASE-20875
> Project: HBase
>  Issue Type: Sub-task
>Reporter: stack
>Priority: Major
> Attachments: Screen Shot 2018-07-11 at 9.52.46 PM.png
>
>
> Looks like this with a PE random write loading:
> {code}
>  ./hbase/bin/hbase  --config ~/conf_hbase 
> org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --presplit=40  
> --size=30 --columns=10 --valueSize=100 randomWrite 200
> {code}
> ... against a single server.
> {code}
> 12.47%  perf-91935.map
> [.] Lorg/apache/hadoop/hbase/BBKVComparator;::compare
>  10.42%  libjvm.so
>  [.] 
> ParNewGeneration::copy_to_survivor_space_avoiding_promotion_undo(ParScanThreadState*,
>  oopDesc*, unsigned long, markOopDesc*)
>   6.78%  perf-91935.map   
>  [.] 
> Lorg/apache/hadoop/hbase/regionserver/MemStoreLABImpl;::copyCellInto
> 
> {code}
> These are top CPU consumers using perf-map-agent ./bin/perf-java-top... 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20865) CreateTableProcedure is stuck in retry loop in CREATE_TABLE_WRITE_FS_LAYOUT state

2018-07-11 Thread Toshihiro Suzuki (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541054#comment-16541054
 ] 

Toshihiro Suzuki commented on HBASE-20865:
--

Could you please review the patch? [~yuzhih...@gmail.com]
Thanks.

> CreateTableProcedure is stuck in retry loop in CREATE_TABLE_WRITE_FS_LAYOUT 
> state
> -
>
> Key: HBASE-20865
> URL: https://issues.apache.org/jira/browse/HBASE-20865
> Project: HBase
>  Issue Type: Bug
>  Components: amv2
>Reporter: Toshihiro Suzuki
>Assignee: Toshihiro Suzuki
>Priority: Major
> Attachments: HBASE-20865.master.001.patch
>
>
> Similar to HBASE-20616, CreateTableProcedure gets stuck in retry loop in 
> CREATE_TABLE_WRITE_FS_LAYOUT state when writing HDFS fails.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (HBASE-20875) MemStoreLABImp::copyIntoCell uses 7% CPU when writing

2018-07-11 Thread stack (JIRA)
stack created HBASE-20875:
-

 Summary: MemStoreLABImp::copyIntoCell uses 7% CPU when writing
 Key: HBASE-20875
 URL: https://issues.apache.org/jira/browse/HBASE-20875
 Project: HBase
  Issue Type: Sub-task
Reporter: stack


Looks like this with a PE random write loading:

{code}
 ./hbase/bin/hbase  --config ~/conf_hbase 
org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --presplit=40  
--size=30 --columns=10 --valueSize=100 randomWrite 200
{code}

... against a single server.

{code}
12.47%  perf-91935.map  
  [.] Lorg/apache/hadoop/hbase/BBKVComparator;::compare
 10.42%  libjvm.so  
   [.] 
ParNewGeneration::copy_to_survivor_space_avoiding_promotion_undo(ParScanThreadState*,
 oopDesc*, unsigned long, markOopDesc*)
  6.78%  perf-91935.map 
   [.] 
Lorg/apache/hadoop/hbase/regionserver/MemStoreLABImpl;::copyCellInto

{code}

These are top CPU consumers using perf-map-agent ./bin/perf-java-top... 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20846) Table's shared lock is not held by sub-procedures after master restart

2018-07-11 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541048#comment-16541048
 ] 

Duo Zhang commented on HBASE-20846:
---

Anyway let me provide a patch first. The optimization can be done later.

> Table's shared lock is not held by sub-procedures after master restart
> --
>
> Key: HBASE-20846
> URL: https://issues.apache.org/jira/browse/HBASE-20846
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.1.0
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Major
> Fix For: 3.0.0, 2.0.2, 2.1.1
>
> Attachments: HBASE-20846.branch-2.0.002.patch, 
> HBASE-20846.branch-2.0.patch
>
>
> Found this one when investigating ModifyTableProcedure got stuck while there 
> was a MoveRegionProcedure going on after master restart.
> Though this issue can be solved by HBASE-20752. But I discovered something 
> else.
> Before a MoveRegionProcedure can execute, it will hold the table's shared 
> lock. so,, when a UnassignProcedure was spwaned, it will not check the 
> table's shared lock since it is sure that its parent(MoveRegionProcedure) has 
> aquired the table's lock.
> {code:java}
> // If there is parent procedure, it would have already taken xlock, so no 
> need to take
>   // shared lock here. Otherwise, take shared lock.
>   if (!procedure.hasParent()
>   && waitTableQueueSharedLock(procedure, table) == null) {
>   return true;
>   }
> {code}
> But, it is not the case when Master was restarted. The child 
> procedure(UnassignProcedure) will be executed first after restart. Though it 
> has a parent(MoveRegionProcedure), but apprently the parent didn't hold the 
> table's lock.
> So, since it began to execute without hold the table's shared lock. A 
> ModifyTableProcedure can aquire the table's exclusive lock and execute at the 
> same time. Which is not possible if the master was not restarted.
> This will cause a stuck before HBASE-20752. But since HBASE-20752 has fixed, 
> I wrote a simple UT to repo this case.
> I think we don't have to check the parent for table's shared lock. It is a 
> shared lock, right? I think we can acquire it every time we need it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20838) Include hbase-server in precommit test if CommonFSUtils is changed

2018-07-11 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541046#comment-16541046
 ] 

Duo Zhang commented on HBASE-20838:
---

+1 on committing to branch-2.1.

> Include hbase-server in precommit test if CommonFSUtils is changed
> --
>
> Key: HBASE-20838
> URL: https://issues.apache.org/jira/browse/HBASE-20838
> Project: HBase
>  Issue Type: Test
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-20838.patch, HBASE-20838.patch, 
> HBASE-20838.v2.patch
>
>
> -As per 
> [discussed|https://issues.apache.org/jira/browse/HBASE-20691?focusedCommentId=16517662=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16517662]
>  in HBASE-20691, since the setStoragePolicy code is in CommonFSUtils, the 
> test should be in TestCommonFSUtils-
> We don't want to introduce dependency on hadoop-hdfs to hbase-common so 
> decided to leave the setStoragePolicy related tests in TestCommonFSUtils. 
> Instead, we will change the personality script to include hbase-server in 
> unit test if any change made against {{CommonFSUtils}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HBASE-20838) Include hbase-server in precommit test if CommonFSUtils is changed

2018-07-11 Thread Yu Li (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-20838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yu Li updated HBASE-20838:
--
   Resolution: Fixed
Fix Version/s: 2.2.0
   3.0.0
   Status: Resolved  (was: Patch Available)

Pushed into master and branch-2. [~stack] [~Apache9] please let me know if 
you'd like this one in branch-2.0/2.1, thanks.

Closing issue for now.

> Include hbase-server in precommit test if CommonFSUtils is changed
> --
>
> Key: HBASE-20838
> URL: https://issues.apache.org/jira/browse/HBASE-20838
> Project: HBase
>  Issue Type: Test
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-20838.patch, HBASE-20838.patch, 
> HBASE-20838.v2.patch
>
>
> -As per 
> [discussed|https://issues.apache.org/jira/browse/HBASE-20691?focusedCommentId=16517662=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16517662]
>  in HBASE-20691, since the setStoragePolicy code is in CommonFSUtils, the 
> test should be in TestCommonFSUtils-
> We don't want to introduce dependency on hadoop-hdfs to hbase-common so 
> decided to leave the setStoragePolicy related tests in TestCommonFSUtils. 
> Instead, we will change the personality script to include hbase-server in 
> unit test if any change made against {{CommonFSUtils}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HBASE-20838) Include hbase-server in precommit test if CommonFSUtils is changed

2018-07-11 Thread Yu Li (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-20838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yu Li updated HBASE-20838:
--
Hadoop Flags: Reviewed
 Description: 
-As per 
[discussed|https://issues.apache.org/jira/browse/HBASE-20691?focusedCommentId=16517662=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16517662]
 in HBASE-20691, since the setStoragePolicy code is in CommonFSUtils, the test 
should be in TestCommonFSUtils-
We don't want to introduce dependency on hadoop-hdfs to hbase-common so decided 
to leave the setStoragePolicy related tests in TestCommonFSUtils. Instead, we 
will change the personality script to include hbase-server in unit test if any 
change made against {{CommonFSUtils}}

  was:As per 
[discussed|https://issues.apache.org/jira/browse/HBASE-20691?focusedCommentId=16517662=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16517662]
 in HBASE-20691, since the setStoragePolicy code is in CommonFSUtils, the test 
should be in TestCommonFSUtils


Updated title and description.

Thanks [~busbey] for review, will commit this soon.

> Include hbase-server in precommit test if CommonFSUtils is changed
> --
>
> Key: HBASE-20838
> URL: https://issues.apache.org/jira/browse/HBASE-20838
> Project: HBase
>  Issue Type: Test
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Major
> Attachments: HBASE-20838.patch, HBASE-20838.patch, 
> HBASE-20838.v2.patch
>
>
> -As per 
> [discussed|https://issues.apache.org/jira/browse/HBASE-20691?focusedCommentId=16517662=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16517662]
>  in HBASE-20691, since the setStoragePolicy code is in CommonFSUtils, the 
> test should be in TestCommonFSUtils-
> We don't want to introduce dependency on hadoop-hdfs to hbase-common so 
> decided to leave the setStoragePolicy related tests in TestCommonFSUtils. 
> Instead, we will change the personality script to include hbase-server in 
> unit test if any change made against {{CommonFSUtils}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HBASE-20838) Include hbase-server in precommit test if CommonFSUtils is changed

2018-07-11 Thread Yu Li (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-20838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yu Li updated HBASE-20838:
--
Summary: Include hbase-server in precommit test if CommonFSUtils is changed 
 (was: Move all setStorage related UT cases from TestFSUtils to 
TestCommonFSUtils)

> Include hbase-server in precommit test if CommonFSUtils is changed
> --
>
> Key: HBASE-20838
> URL: https://issues.apache.org/jira/browse/HBASE-20838
> Project: HBase
>  Issue Type: Test
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Major
> Attachments: HBASE-20838.patch, HBASE-20838.patch, 
> HBASE-20838.v2.patch
>
>
> As per 
> [discussed|https://issues.apache.org/jira/browse/HBASE-20691?focusedCommentId=16517662=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16517662]
>  in HBASE-20691, since the setStoragePolicy code is in CommonFSUtils, the 
> test should be in TestCommonFSUtils



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20870) Wrong HBase root dir in ITBLL's Search Tool

2018-07-11 Thread Mike Drob (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541020#comment-16541020
 ] 

Mike Drob commented on HBASE-20870:
---

I'm a little confused on what we're setting and unsetting. IIRC I had tested 
these on distributed clusters already.

{code}
-conf.set("fs.defaultFS", conf.get("original.defaultFS"));
+Configuration localConf = HBaseConfiguration.create();
+conf.set("fs.defaultFS", localConf.get("fs.defaultFS"));
+conf.set(HConstants.HBASE_DIR, localConf.get(HConstants.HBASE_DIR));
{code}

Is the defaultFS in localConf different from the old value that we saved? I 
wouldn't expect it to be.

Can we give HBASE_DIR the same treatment? I think I was saving it in 
HBaseTestingUtility in some version of my patch but then it appeared that 
wasn't necessary. See HBASE-19947

> Wrong HBase root dir in ITBLL's Search Tool
> ---
>
> Key: HBASE-20870
> URL: https://issues.apache.org/jira/browse/HBASE-20870
> Project: HBase
>  Issue Type: Bug
>  Components: integration tests
>Affects Versions: 3.0.0, 2.1.0, 2.0.1
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Minor
> Attachments: HBASE-20870.branch-2.0.001.patch
>
>
> When using IntegrationTestBigLinkedList's Search tools, it always fails since 
> it tries to read WALs in a wrong HBase root dir. Turned out that when 
> initializing IntegrationTestingUtility in IntegrationTestBigLinkedList, its 
> super class HBaseTestingUtility will change hbase.rootdir to a local random 
> dir. It is not wrong since HBaseTestingUtility is mostly used by Minicluster. 
> But for IntegrationTest runs on distributed clusters, we should change it 
> back.
>  Here is the error info.
> {code:java}
> 2018-07-11 16:35:49,679 DEBUG [main] hbase.HBaseCommonTestingUtility: Setting 
> hbase.rootdir to 
> /home/hadoop/target/test-data/deb67611-2737-4696-abe9-32a7783df7bb
> 2018-07-11 16:35:50,736 ERROR [main] util.AbstractHBaseTool: Error running 
> command-line tool java.io.FileNotFoundException: File 
> file:/home/hadoop/target/test-data/deb67611-2737-4696-abe9-32a7783df7bb/WALs 
> does not exist
> at 
> org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:431)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1517)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20859) Backup and incremental load could fail in secure clusters

2018-07-11 Thread Reid Chan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541018#comment-16541018
 ] 

Reid Chan commented on HBASE-20859:
---

{code}
  /** Don't use it! This'll get you the wrong path in a secure cluster.
   * Use FileSystem.getHomeDirectory() or
   * "/user/" + UserGroupInformation.getCurrentUser().getShortUserName()  */
{code}
This docs style is not good??
Or at least we can file a jira for {{beginner}}.

> Backup and incremental load could fail in secure clusters
> -
>
> Key: HBASE-20859
> URL: https://issues.apache.org/jira/browse/HBASE-20859
> Project: HBase
>  Issue Type: Bug
>  Components: backuprestore
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20859.master.001.patch, 
> HBASE-20859.master.002.patch, HBASE-20859.master.003.patch
>
>
> HBase Backup and incremental load uses 
> HConstants.DEFAULT_TEMPORARY_HDFS_DIRECTORY for temporary path.
> HConstants.DEFAULT_TEMPORARY_HDFS_DIRECTORY uses the Java runtime user name 
> to generate a temporary path on HDFS. This can be a wrong assumption in a 
> secure cluster where Kerberos principal name can be different from the system 
> user name.
> {code:java}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
>   + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This constant variable is used in BackupUtils.java and HFileOutputFormat2.java
> In such cases, you will not be able to write files to the temporary location 
> on HDFS due to permission error, and therefore operations such as backup will 
> fail.
> This bug is similar in nature to HDFS-12485.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20846) Table's shared lock is not held by sub-procedures after master restart

2018-07-11 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540992#comment-16540992
 ] 

stack commented on HBASE-20846:
---

bq. We have already done one step, and we expect that no one can interrupt us, 
so obviously this behavior is broken.

Good objection.

So, persist wasExecuted before we do first step?

bq. For now, this is done by a very tricky way - we always put the already 
being executed table procedure to the front of the queue, even in the yield 
method.

Yes, these should take exclusive lock for life of the procedure.




> Table's shared lock is not held by sub-procedures after master restart
> --
>
> Key: HBASE-20846
> URL: https://issues.apache.org/jira/browse/HBASE-20846
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.1.0
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Major
> Fix For: 3.0.0, 2.0.2, 2.1.1
>
> Attachments: HBASE-20846.branch-2.0.002.patch, 
> HBASE-20846.branch-2.0.patch
>
>
> Found this one when investigating ModifyTableProcedure got stuck while there 
> was a MoveRegionProcedure going on after master restart.
> Though this issue can be solved by HBASE-20752. But I discovered something 
> else.
> Before a MoveRegionProcedure can execute, it will hold the table's shared 
> lock. so,, when a UnassignProcedure was spwaned, it will not check the 
> table's shared lock since it is sure that its parent(MoveRegionProcedure) has 
> aquired the table's lock.
> {code:java}
> // If there is parent procedure, it would have already taken xlock, so no 
> need to take
>   // shared lock here. Otherwise, take shared lock.
>   if (!procedure.hasParent()
>   && waitTableQueueSharedLock(procedure, table) == null) {
>   return true;
>   }
> {code}
> But, it is not the case when Master was restarted. The child 
> procedure(UnassignProcedure) will be executed first after restart. Though it 
> has a parent(MoveRegionProcedure), but apprently the parent didn't hold the 
> table's lock.
> So, since it began to execute without hold the table's shared lock. A 
> ModifyTableProcedure can aquire the table's exclusive lock and execute at the 
> same time. Which is not possible if the master was not restarted.
> This will cause a stuck before HBASE-20752. But since HBASE-20752 has fixed, 
> I wrote a simple UT to repo this case.
> I think we don't have to check the parent for table's shared lock. It is a 
> shared lock, right? I think we can acquire it every time we need it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20870) Wrong HBase root dir in ITBLL's Search Tool

2018-07-11 Thread Allan Yang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540975#comment-16540975
 ] 

Allan Yang commented on HBASE-20870:


{quote}
lgtm, assuming you have tested on cluster.
{quote}
Yes, I have it tested on a real cluster, thanks for reviewing, 
[~yuzhih...@gmail.com].

> Wrong HBase root dir in ITBLL's Search Tool
> ---
>
> Key: HBASE-20870
> URL: https://issues.apache.org/jira/browse/HBASE-20870
> Project: HBase
>  Issue Type: Bug
>  Components: integration tests
>Affects Versions: 3.0.0, 2.1.0, 2.0.1
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Minor
> Attachments: HBASE-20870.branch-2.0.001.patch
>
>
> When using IntegrationTestBigLinkedList's Search tools, it always fails since 
> it tries to read WALs in a wrong HBase root dir. Turned out that when 
> initializing IntegrationTestingUtility in IntegrationTestBigLinkedList, its 
> super class HBaseTestingUtility will change hbase.rootdir to a local random 
> dir. It is not wrong since HBaseTestingUtility is mostly used by Minicluster. 
> But for IntegrationTest runs on distributed clusters, we should change it 
> back.
>  Here is the error info.
> {code:java}
> 2018-07-11 16:35:49,679 DEBUG [main] hbase.HBaseCommonTestingUtility: Setting 
> hbase.rootdir to 
> /home/hadoop/target/test-data/deb67611-2737-4696-abe9-32a7783df7bb
> 2018-07-11 16:35:50,736 ERROR [main] util.AbstractHBaseTool: Error running 
> command-line tool java.io.FileNotFoundException: File 
> file:/home/hadoop/target/test-data/deb67611-2737-4696-abe9-32a7783df7bb/WALs 
> does not exist
> at 
> org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:431)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1517)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HBASE-20704) Sometimes some compacted storefiles are not archived on region close

2018-07-11 Thread Francis Liu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Francis Liu updated HBASE-20704:

Attachment: HBASE-20704.003.patch

> Sometimes some compacted storefiles are not archived on region close
> 
>
> Key: HBASE-20704
> URL: https://issues.apache.org/jira/browse/HBASE-20704
> Project: HBase
>  Issue Type: Bug
>  Components: Compaction
>Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0
>Reporter: Francis Liu
>Assignee: Francis Liu
>Priority: Critical
> Attachments: HBASE-20704.001.patch, HBASE-20704.002.patch, 
> HBASE-20704.003.patch
>
>
> During region close compacted files which have not yet been archived by the 
> discharger are archived as part of the region closing process. It is 
> important that these files are wholly archived to insure data consistency. ie 
> a storefile containing delete tombstones can be archived while older 
> storefiles containing cells that were supposed to be deleted are left 
> unarchived thereby undeleting those cells. 
> On region close a compacted storefile is skipped from archiving if it has 
> read references (ie open scanners). This behavior is correct for when the 
> discharger chore runs but on region close consistency is of course more 
> important so we should add a special case to ignore any references on the 
> storefile and go ahead and archive it. 
> Attached patch contains a unit test that reproduces the problem and the 
> proposed fix.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20704) Sometimes some compacted storefiles are not archived on region close

2018-07-11 Thread Francis Liu (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540972#comment-16540972
 ] 

Francis Liu commented on HBASE-20704:
-

OK I did some testing NPE only actually occurs in the 1.x. For the master 
branch it's one of two scenarios depending on wether a pread was used or not as 
stream scanners open their own inputstream, reader, etc. 
 # pread - you'll get a "stream closed" IOException. In 1.x it's an NPE because 
the stream references are set to null after the streams are closed.
 # not pread - Since the HStore has no knowledge of the created stream it does 
not close them. What happens is either the current running scan request is 
processed or it will get an IOException (replica not found), since the region 
close operation may have archived and the cleaner chore deleted the file. 

It sounds to me that #1 is good and #2 is acceptable? Let me know. If #2 is not 
acceptable I'd have to add a map to keep track of the new readers created when 
a stream scanner is created and close those on region close so exceptions will 
be like #1.

I've attached another patch which add another test method to surface the 
exceptions.

> Sometimes some compacted storefiles are not archived on region close
> 
>
> Key: HBASE-20704
> URL: https://issues.apache.org/jira/browse/HBASE-20704
> Project: HBase
>  Issue Type: Bug
>  Components: Compaction
>Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0
>Reporter: Francis Liu
>Assignee: Francis Liu
>Priority: Critical
> Attachments: HBASE-20704.001.patch, HBASE-20704.002.patch
>
>
> During region close compacted files which have not yet been archived by the 
> discharger are archived as part of the region closing process. It is 
> important that these files are wholly archived to insure data consistency. ie 
> a storefile containing delete tombstones can be archived while older 
> storefiles containing cells that were supposed to be deleted are left 
> unarchived thereby undeleting those cells. 
> On region close a compacted storefile is skipped from archiving if it has 
> read references (ie open scanners). This behavior is correct for when the 
> discharger chore runs but on region close consistency is of course more 
> important so we should add a special case to ignore any references on the 
> storefile and go ahead and archive it. 
> Attached patch contains a unit test that reproduces the problem and the 
> proposed fix.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20846) Table's shared lock is not held by sub-procedures after master restart

2018-07-11 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540955#comment-16540955
 ] 

Duo Zhang commented on HBASE-20846:
---

{quote}
Do we need a new boolean? What cases do we need to handle on load post-crash? 
To my mind, the procedure that holds a lock for the life of the procedure is 
the only case that needs re-institution?
{quote}

Yes, this is the truth for now, holdLock is true, and wasExecuted is true, then 
we give it the lock when loading procedures. But there is still a problem, as 
we will only persist the wasExecuted state into the procedure store after we 
finish at least one step of the procedure. So it is possible that, a procedure 
has acquired the lock, and done some modifications, and then we crashed. On 
restart, we will not give the lock to it, and another procedure may jump it. We 
have already done one step, and we expect that no one can interrupt us, so 
obviously this behavior is broken.

And when reviewing the table related procedures, I think we should make 
holdLock depend on the state of the procedure. For table related procedure, 
ideally we should acquire the exclusive lock all the time, until we schedule 
the reopen regions procedure. If not there will be problem, think of two 
ModifyTableProcedures run at the same time and both want to modify the table 
descriptor and layout... For now, this is done by a very tricky way - we always 
put the already being executed table procedure to the front of the queue, even 
in the yield method. This is not a good design as it will be easily broken in 
the future development.

> Table's shared lock is not held by sub-procedures after master restart
> --
>
> Key: HBASE-20846
> URL: https://issues.apache.org/jira/browse/HBASE-20846
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.1.0
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Major
> Fix For: 3.0.0, 2.0.2, 2.1.1
>
> Attachments: HBASE-20846.branch-2.0.002.patch, 
> HBASE-20846.branch-2.0.patch
>
>
> Found this one when investigating ModifyTableProcedure got stuck while there 
> was a MoveRegionProcedure going on after master restart.
> Though this issue can be solved by HBASE-20752. But I discovered something 
> else.
> Before a MoveRegionProcedure can execute, it will hold the table's shared 
> lock. so,, when a UnassignProcedure was spwaned, it will not check the 
> table's shared lock since it is sure that its parent(MoveRegionProcedure) has 
> aquired the table's lock.
> {code:java}
> // If there is parent procedure, it would have already taken xlock, so no 
> need to take
>   // shared lock here. Otherwise, take shared lock.
>   if (!procedure.hasParent()
>   && waitTableQueueSharedLock(procedure, table) == null) {
>   return true;
>   }
> {code}
> But, it is not the case when Master was restarted. The child 
> procedure(UnassignProcedure) will be executed first after restart. Though it 
> has a parent(MoveRegionProcedure), but apprently the parent didn't hold the 
> table's lock.
> So, since it began to execute without hold the table's shared lock. A 
> ModifyTableProcedure can aquire the table's exclusive lock and execute at the 
> same time. Which is not possible if the master was not restarted.
> This will cause a stuck before HBASE-20752. But since HBASE-20752 has fixed, 
> I wrote a simple UT to repo this case.
> I think we don't have to check the parent for table's shared lock. It is a 
> shared lock, right? I think we can acquire it every time we need it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HBASE-20847) The parent procedure of RegionTransitionProcedure may not have the table lock

2018-07-11 Thread Duo Zhang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-20847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang updated HBASE-20847:
--
   Resolution: Fixed
Fix Version/s: 2.1.1
   2.2.0
   3.0.0
   Status: Resolved  (was: Patch Available)

Pushed the addendum to branch-2,1+.

Thanks all for reviewing.

> The parent procedure of RegionTransitionProcedure may not have the table lock
> -
>
> Key: HBASE-20847
> URL: https://issues.apache.org/jira/browse/HBASE-20847
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2, Region Assignment
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1
>
> Attachments: HBASE-20847-addendum.patch, 
> HBASE-20847-branch-2.0-v1.patch, HBASE-20847-branch-2.0.patch, 
> HBASE-20847-v1.patch, HBASE-20847-v2.patch, HBASE-20847-v3.patch, 
> HBASE-20847.patch
>
>
> For example, SCP can also schedule AssignProcedure and obviously it will not 
> hold the table lock.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20869) Endpoint-based Export use incorrect user to write to destination

2018-07-11 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540928#comment-16540928
 ] 

Hadoop QA commented on HBASE-20869:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
34s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
32s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
13s{color} | {color:red} hbase-endpoint: The patch generated 2 new + 0 
unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
35s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
10m 14s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
49s{color} | {color:green} hbase-endpoint in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
10s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 35m 58s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20869 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12931247/HBASE-20869.master.001.patch
 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  shadedjars  
hadoopcheck  hbaseanti  checkstyle  compile  |
| uname | Linux 03a4deae3b14 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27 
10:45:36 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 0d33caa39a |
| maven | version: Apache Maven 3.5.4 
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC3 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HBASE-Build/13600/artifact/patchprocess/diff-checkstyle-hbase-endpoint.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/13600/testReport/ |
| Max. process+thread count | 3086 (vs. ulimit of 1) |
| modules | C: hbase-endpoint U: hbase-endpoint |
| Console output | 

[jira] [Commented] (HBASE-20800) Master-orchestrated compactions

2018-07-11 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540929#comment-16540929
 ] 

Duo Zhang commented on HBASE-20800:
---

Can do this on a feature branch?

> Master-orchestrated compactions
> ---
>
> Key: HBASE-20800
> URL: https://issues.apache.org/jira/browse/HBASE-20800
> Project: HBase
>  Issue Type: Umbrella
>  Components: Compaction
>Reporter: stack
>Assignee: Mohit Goel
>Priority: Major
>
> An umbrella issue for having compactions go via the Master so we can have a 
> centralized arbitrator of cluster i/o. If we put Master in the way, we can do 
> stuff like:
>  * Ask the Master for current cluster compaction state; what is running, what 
> is blocked
>  * Master can manage cluster-wide compaction policy and/or throttling/or 
> blocking of compaction i/os.
>  * Master can schedule when and where compactions run so we can guard against 
> the pathological where all RegionServers decide now is the time to major 
> compact bringing on a compaction storm.
> Other side-benefits might include being able to farm out the compaction work 
> to another process -- e.g. the splice machine model of having spark run the 
> compactions -- or just to a separate compactor that we might i/o nice.
>  * We'll need to figure how to externalize the CompactionRequest so it can be 
> passed over RPC.
>  * We'll need to have something like a CompactionManager in the Master 
> process that keeps up current cluster state.
> MOB needs a compaction fabric it can use. Its compactions are currently 
> Master-based only and so don't scale. It could make use of this mechanism to 
> ask the Master to farm out its compaction requests.
> This is an umbrella issue. I thought I'd filed one already on this topic but 
> can't find it. Will shut it down if I trip over it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (HBASE-20874) Sending compaction descriptions from all regionservers to master.

2018-07-11 Thread Mohit Goel (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-20874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mohit Goel reassigned HBASE-20874:
--

Assignee: Mohit Goel

> Sending compaction descriptions from all regionservers to master.
> -
>
> Key: HBASE-20874
> URL: https://issues.apache.org/jira/browse/HBASE-20874
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Mohit Goel
>Assignee: Mohit Goel
>Priority: Minor
>
> Need to send the compaction description from region servers to Master , to 
> let master know of the entire compaction state of the cluster. Further need 
> to change the implementation of client Side API than like getCompactionState, 
> which will consult master for the result instead of sending individual 
> request to regionservers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (HBASE-20874) Sending compaction descriptions from all regionservers to master.

2018-07-11 Thread Mohit Goel (JIRA)
Mohit Goel created HBASE-20874:
--

 Summary: Sending compaction descriptions from all regionservers to 
master.
 Key: HBASE-20874
 URL: https://issues.apache.org/jira/browse/HBASE-20874
 Project: HBase
  Issue Type: Sub-task
Reporter: Mohit Goel


Need to send the compaction description from region servers to Master , to let 
master know of the entire compaction state of the cluster. Further need to 
change the implementation of client Side API than like getCompactionState, 
which will consult master for the result instead of sending individual request 
to regionservers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20859) Backup and incremental load could fail in secure clusters

2018-07-11 Thread Wei-Chiu Chuang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540910#comment-16540910
 ] 

Wei-Chiu Chuang commented on HBASE-20859:
-

Thanks [~yuzhih...@gmail.com]!

> Backup and incremental load could fail in secure clusters
> -
>
> Key: HBASE-20859
> URL: https://issues.apache.org/jira/browse/HBASE-20859
> Project: HBase
>  Issue Type: Bug
>  Components: backuprestore
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20859.master.001.patch, 
> HBASE-20859.master.002.patch, HBASE-20859.master.003.patch
>
>
> HBase Backup and incremental load uses 
> HConstants.DEFAULT_TEMPORARY_HDFS_DIRECTORY for temporary path.
> HConstants.DEFAULT_TEMPORARY_HDFS_DIRECTORY uses the Java runtime user name 
> to generate a temporary path on HDFS. This can be a wrong assumption in a 
> secure cluster where Kerberos principal name can be different from the system 
> user name.
> {code:java}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
>   + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This constant variable is used in BackupUtils.java and HFileOutputFormat2.java
> In such cases, you will not be able to write files to the temporary location 
> on HDFS due to permission error, and therefore operations such as backup will 
> fail.
> This bug is similar in nature to HDFS-12485.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HBASE-20859) Backup and incremental load could fail in secure clusters

2018-07-11 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-20859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-20859:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Thanks for the patch, Wei-Chiu

Thanks for the reviews.

> Backup and incremental load could fail in secure clusters
> -
>
> Key: HBASE-20859
> URL: https://issues.apache.org/jira/browse/HBASE-20859
> Project: HBase
>  Issue Type: Bug
>  Components: backuprestore
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20859.master.001.patch, 
> HBASE-20859.master.002.patch, HBASE-20859.master.003.patch
>
>
> HBase Backup and incremental load uses 
> HConstants.DEFAULT_TEMPORARY_HDFS_DIRECTORY for temporary path.
> HConstants.DEFAULT_TEMPORARY_HDFS_DIRECTORY uses the Java runtime user name 
> to generate a temporary path on HDFS. This can be a wrong assumption in a 
> secure cluster where Kerberos principal name can be different from the system 
> user name.
> {code:java}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
>   + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This constant variable is used in BackupUtils.java and HFileOutputFormat2.java
> In such cases, you will not be able to write files to the temporary location 
> on HDFS due to permission error, and therefore operations such as backup will 
> fail.
> This bug is similar in nature to HDFS-12485.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20870) Wrong HBase root dir in ITBLL's Search Tool

2018-07-11 Thread Ted Yu (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540905#comment-16540905
 ] 

Ted Yu commented on HBASE-20870:


lgtm, assuming you have tested on cluster.

> Wrong HBase root dir in ITBLL's Search Tool
> ---
>
> Key: HBASE-20870
> URL: https://issues.apache.org/jira/browse/HBASE-20870
> Project: HBase
>  Issue Type: Bug
>  Components: integration tests
>Affects Versions: 3.0.0, 2.1.0, 2.0.1
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Minor
> Attachments: HBASE-20870.branch-2.0.001.patch
>
>
> When using IntegrationTestBigLinkedList's Search tools, it always fails since 
> it tries to read WALs in a wrong HBase root dir. Turned out that when 
> initializing IntegrationTestingUtility in IntegrationTestBigLinkedList, its 
> super class HBaseTestingUtility will change hbase.rootdir to a local random 
> dir. It is not wrong since HBaseTestingUtility is mostly used by Minicluster. 
> But for IntegrationTest runs on distributed clusters, we should change it 
> back.
>  Here is the error info.
> {code:java}
> 2018-07-11 16:35:49,679 DEBUG [main] hbase.HBaseCommonTestingUtility: Setting 
> hbase.rootdir to 
> /home/hadoop/target/test-data/deb67611-2737-4696-abe9-32a7783df7bb
> 2018-07-11 16:35:50,736 ERROR [main] util.AbstractHBaseTool: Error running 
> command-line tool java.io.FileNotFoundException: File 
> file:/home/hadoop/target/test-data/deb67611-2737-4696-abe9-32a7783df7bb/WALs 
> does not exist
> at 
> org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:431)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1517)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (HBASE-20873) Update doc for Endpoint-based Export

2018-07-11 Thread Wei-Chiu Chuang (JIRA)
Wei-Chiu Chuang created HBASE-20873:
---

 Summary: Update doc for Endpoint-based Export
 Key: HBASE-20873
 URL: https://issues.apache.org/jira/browse/HBASE-20873
 Project: HBase
  Issue Type: Improvement
  Components: documentation
Affects Versions: 2.0.0
Reporter: Wei-Chiu Chuang
Assignee: Wei-Chiu Chuang


The current documentation on the usage is a little vague. I'd like to take a 
stab at expanding it, based on my experience.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HBASE-20869) Endpoint-based Export use incorrect user to write to destination

2018-07-11 Thread Wei-Chiu Chuang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-20869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HBASE-20869:

Status: Patch Available  (was: Open)

> Endpoint-based Export use incorrect user to write to destination
> 
>
> Key: HBASE-20869
> URL: https://issues.apache.org/jira/browse/HBASE-20869
> Project: HBase
>  Issue Type: Bug
>  Components: Coprocessors
>Affects Versions: 2.0.0
> Environment: Hadoop 3.0.0 + HBase 2.0.0, Kerberos.
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HBASE-20869.master.001.patch
>
>
> HBASE-15806 implemented an endpoint based export. It gets caller's HDFS 
> delegation token, and RegionServer is supposed to write out exported files as 
> the caller.
> Everything works fine if you use run export as hbase user. However, once you 
> use a different user to export, it fails.
> To reproduce,
> Add to configuration key hbase.coprocessor.region.classes the coprocessor 
> class org.apache.hadoop.hbase.coprocessor.Export.
> create a table t1, assign permission to a user foo:
>  
> {noformat}
> hbase(main):004:0> user_permission 't1'
> User Namespace,Table,Family,Qualifier:Permission
> hbase default,t1,,: [Permission: actions=READ,WRITE,EXEC,CREATE,ADMIN]
> foo default,t1,,: [Permission: actions=READ,WRITE,EXEC,CREATE,ADMIN]{noformat}
>  
> As user foo, execute the following command:
>  
> {noformat}
> $ hdfs dfs -mkdir /tmp/export_hbase2
> $ hbase org.apache.hadoop.hbase.coprocessor.Export t1 /tmp/export_hbase2/t2/
> 
> 18/07/10 14:03:59 INFO client.RpcRetryingCallerImpl: Call exception, tries=6, 
> retries=6, started=4457 ms ago, cancelled=false, 
> msg=org.apache.hadoop.security.AccessControlException: Permission denied: 
> user=hbase, access=WRITE, 
> inode="/tmp/export_hbase2/t2":foo:supergroup:drwxr-xr-x
> at 
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:400)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:256)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:194)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1846)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1830)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1789)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.resolvePathForStartFile(FSDirWriteFileOp.java:316)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2411)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2343)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:764)
> at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:451)
> at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
> at sun.reflect.GeneratedConstructorAccessor25.newInstance(Unknown Source)
> at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at 
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121)
> at 
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88)
> at 
> org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:278)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1195)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1174)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1112)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:462)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:459)
> at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at 

[jira] [Updated] (HBASE-20869) Endpoint-based Export use incorrect user to write to destination

2018-07-11 Thread Wei-Chiu Chuang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-20869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HBASE-20869:

Attachment: HBASE-20869.master.001.patch

> Endpoint-based Export use incorrect user to write to destination
> 
>
> Key: HBASE-20869
> URL: https://issues.apache.org/jira/browse/HBASE-20869
> Project: HBase
>  Issue Type: Bug
>  Components: Coprocessors
>Affects Versions: 2.0.0
> Environment: Hadoop 3.0.0 + HBase 2.0.0, Kerberos.
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HBASE-20869.master.001.patch
>
>
> HBASE-15806 implemented an endpoint based export. It gets caller's HDFS 
> delegation token, and RegionServer is supposed to write out exported files as 
> the caller.
> Everything works fine if you use run export as hbase user. However, once you 
> use a different user to export, it fails.
> To reproduce,
> Add to configuration key hbase.coprocessor.region.classes the coprocessor 
> class org.apache.hadoop.hbase.coprocessor.Export.
> create a table t1, assign permission to a user foo:
>  
> {noformat}
> hbase(main):004:0> user_permission 't1'
> User Namespace,Table,Family,Qualifier:Permission
> hbase default,t1,,: [Permission: actions=READ,WRITE,EXEC,CREATE,ADMIN]
> foo default,t1,,: [Permission: actions=READ,WRITE,EXEC,CREATE,ADMIN]{noformat}
>  
> As user foo, execute the following command:
>  
> {noformat}
> $ hdfs dfs -mkdir /tmp/export_hbase2
> $ hbase org.apache.hadoop.hbase.coprocessor.Export t1 /tmp/export_hbase2/t2/
> 
> 18/07/10 14:03:59 INFO client.RpcRetryingCallerImpl: Call exception, tries=6, 
> retries=6, started=4457 ms ago, cancelled=false, 
> msg=org.apache.hadoop.security.AccessControlException: Permission denied: 
> user=hbase, access=WRITE, 
> inode="/tmp/export_hbase2/t2":foo:supergroup:drwxr-xr-x
> at 
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:400)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:256)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:194)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1846)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1830)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1789)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.resolvePathForStartFile(FSDirWriteFileOp.java:316)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2411)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2343)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:764)
> at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:451)
> at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
> at sun.reflect.GeneratedConstructorAccessor25.newInstance(Unknown Source)
> at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at 
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121)
> at 
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88)
> at 
> org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:278)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1195)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1174)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1112)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:462)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:459)
> at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)

[jira] [Commented] (HBASE-20175) hbase-spark needs scala dependency convergance

2018-07-11 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540878#comment-16540878
 ] 

Hadoop QA commented on HBASE-20175:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
31s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  5m 
 8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
34s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  5m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  8m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  5m 
 6s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
11m 54s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  3m 
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}202m 
37s{color} | {color:green} root in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
52s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}259m 31s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20175 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12931207/HBASE-20175.v04.patch 
|
| Optional Tests |  asflicense  javac  javadoc  unit  shadedjars  hadoopcheck  
xml  compile  |
| uname | Linux 806715fa4ced 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27 
10:45:36 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 0d33caa39a |
| maven | version: Apache Maven 3.5.4 
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_171 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/13597/testReport/ |
| Max. process+thread count | 5009 (vs. ulimit of 1) |
| modules | C: hbase-spark . U: . |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/13597/console |
| Powered by | Apache Yetus 0.7.0   http://yetus.apache.org |


This message was automatically generated.



> hbase-spark needs scala dependency convergance
> --
>
> Key: HBASE-20175
> URL: https://issues.apache.org/jira/browse/HBASE-20175
> Project: 

[jira] [Updated] (HBASE-20578) Support region server group in target cluster

2018-07-11 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-20578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-20578:
---
Component/s: rsgroup

> Support region server group in target cluster
> -
>
> Key: HBASE-20578
> URL: https://issues.apache.org/jira/browse/HBASE-20578
> Project: HBase
>  Issue Type: Improvement
>  Components: Replication, rsgroup
>Reporter: Ted Yu
>Assignee: Albert Lee
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20578-001.patch
>
>
> When source tables belong to non-default region server group(s) and there are 
> region server group counterpart in the target cluster, we should support 
> replicating to target cluster using the region server group mapping.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20734) Colocate recovered edits directory with hbase.wal.dir

2018-07-11 Thread Zach York (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540796#comment-16540796
 ] 

Zach York commented on HBASE-20734:
---

Yep, I'll work on getting a patch for master branch. It was just easier for me 
to test on cluster with branch-1.

> Colocate recovered edits directory with hbase.wal.dir
> -
>
> Key: HBASE-20734
> URL: https://issues.apache.org/jira/browse/HBASE-20734
> Project: HBase
>  Issue Type: Improvement
>  Components: MTTR, Recovery, wal
>Reporter: Ted Yu
>Assignee: Zach York
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20734.branch-1.001.patch
>
>
> During investigation of HBASE-20723, I realized that we wouldn't get the best 
> performance when hbase.wal.dir is configured to be on different (fast) media 
> than hbase rootdir w.r.t. recovered edits since recovered edits directory is 
> currently under rootdir.
> Such setup may not result in fast recovery when there is region server 
> failover.
> This issue is to find proper (hopefully backward compatible) way in 
> colocating recovered edits directory with hbase.wal.dir .



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20838) Move all setStorage related UT cases from TestFSUtils to TestCommonFSUtils

2018-07-11 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540764#comment-16540764
 ] 

Sean Busbey commented on HBASE-20838:
-

+1

> Move all setStorage related UT cases from TestFSUtils to TestCommonFSUtils
> --
>
> Key: HBASE-20838
> URL: https://issues.apache.org/jira/browse/HBASE-20838
> Project: HBase
>  Issue Type: Test
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Major
> Attachments: HBASE-20838.patch, HBASE-20838.patch, 
> HBASE-20838.v2.patch
>
>
> As per 
> [discussed|https://issues.apache.org/jira/browse/HBASE-20691?focusedCommentId=16517662=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16517662]
>  in HBASE-20691, since the setStoragePolicy code is in CommonFSUtils, the 
> test should be in TestCommonFSUtils



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20847) The parent procedure of RegionTransitionProcedure may not have the table lock

2018-07-11 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540766#comment-16540766
 ] 

Hudson commented on HBASE-20847:


Results for branch branch-2.1
[build #49 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/49/]: 
(x) *{color:red}-1 overall{color}*

details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/49//General_Nightly_Build_Report/]




(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/49//JDK8_Nightly_Build_Report_(Hadoop2)/]


(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/49//JDK8_Nightly_Build_Report_(Hadoop3)/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


(/) {color:green}+1 client integration test{color}


> The parent procedure of RegionTransitionProcedure may not have the table lock
> -
>
> Key: HBASE-20847
> URL: https://issues.apache.org/jira/browse/HBASE-20847
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2, Region Assignment
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Attachments: HBASE-20847-addendum.patch, 
> HBASE-20847-branch-2.0-v1.patch, HBASE-20847-branch-2.0.patch, 
> HBASE-20847-v1.patch, HBASE-20847-v2.patch, HBASE-20847-v3.patch, 
> HBASE-20847.patch
>
>
> For example, SCP can also schedule AssignProcedure and obviously it will not 
> hold the table lock.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20847) The parent procedure of RegionTransitionProcedure may not have the table lock

2018-07-11 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540756#comment-16540756
 ] 

Hudson commented on HBASE-20847:


Results for branch branch-2
[build #971 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/971/]: 
(/) *{color:green}+1 overall{color}*

details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/971//General_Nightly_Build_Report/]




(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/971//JDK8_Nightly_Build_Report_(Hadoop2)/]


(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/971//JDK8_Nightly_Build_Report_(Hadoop3)/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


(/) {color:green}+1 client integration test{color}


> The parent procedure of RegionTransitionProcedure may not have the table lock
> -
>
> Key: HBASE-20847
> URL: https://issues.apache.org/jira/browse/HBASE-20847
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2, Region Assignment
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Attachments: HBASE-20847-addendum.patch, 
> HBASE-20847-branch-2.0-v1.patch, HBASE-20847-branch-2.0.patch, 
> HBASE-20847-v1.patch, HBASE-20847-v2.patch, HBASE-20847-v3.patch, 
> HBASE-20847.patch
>
>
> For example, SCP can also schedule AssignProcedure and obviously it will not 
> hold the table lock.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20617) Upgrade/remove jetty-jsp

2018-07-11 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540757#comment-16540757
 ] 

Hudson commented on HBASE-20617:


Results for branch branch-2
[build #971 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/971/]: 
(/) *{color:green}+1 overall{color}*

details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/971//General_Nightly_Build_Report/]




(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/971//JDK8_Nightly_Build_Report_(Hadoop2)/]


(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/971//JDK8_Nightly_Build_Report_(Hadoop3)/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


(/) {color:green}+1 client integration test{color}


> Upgrade/remove jetty-jsp
> 
>
> Key: HBASE-20617
> URL: https://issues.apache.org/jira/browse/HBASE-20617
> Project: HBase
>  Issue Type: Improvement
>Reporter: Sakthi
>Assignee: Sakthi
>Priority: Minor
> Fix For: 3.0.0, 2.2.0
>
> Attachments: hbase-20617.master.001.patch
>
>
> jetty-jsp removed after jetty-9.2.x version. We use the 9.2 version. Research 
> so far brings out that apache-jsp might be of interest to us in jetty-9.4.x 
> version(as JettyJspServlet.class is in apache-jsp). Yet to figure out about 
> jetty-9.3.x.
> Filing to track this along.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20542) Better heap utilization for IMC with MSLABs

2018-07-11 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540755#comment-16540755
 ] 

Hudson commented on HBASE-20542:


Results for branch branch-2
[build #971 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/971/]: 
(/) *{color:green}+1 overall{color}*

details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/971//General_Nightly_Build_Report/]




(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/971//JDK8_Nightly_Build_Report_(Hadoop2)/]


(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/971//JDK8_Nightly_Build_Report_(Hadoop3)/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


(/) {color:green}+1 client integration test{color}


> Better heap utilization for IMC with MSLABs
> ---
>
> Key: HBASE-20542
> URL: https://issues.apache.org/jira/browse/HBASE-20542
> Project: HBase
>  Issue Type: Sub-task
>  Components: in-memory-compaction
>Reporter: Eshcar Hillel
>Assignee: Eshcar Hillel
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-20542-addendum.master.005.patch, 
> HBASE-20542.branch-2.001.patch, HBASE-20542.branch-2.003.patch, 
> HBASE-20542.branch-2.004.patch, HBASE-20542.branch-2.005.patch, 
> HBASE-20542.master.003.patch, HBASE-20542.master.005-addendum.patch, run.sh, 
> workloada, workloadc, workloadx, workloady
>
>
> Following HBASE-20188 we realized in-memory compaction combined with MSLABs 
> may suffer from heap under-utilization due to internal fragmentation. This 
> jira presents a solution to circumvent this problem. The main idea is to have 
> each update operation check if it will cause overflow in the active segment 
> *before* it is writing the new value (instead of checking the size after the 
> write is completed), and if it is then the active segment is atomically 
> swapped with a new empty segment, and is pushed (full-yet-not-overflowed) to 
> the compaction pipeline. Later on the IMC deamon will run its compaction 
> operation (flatten index/merge indices/data compaction) in the background. 
> Some subtle concurrency issues should be handled with care. We next elaborate 
> on them.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20734) Colocate recovered edits directory with hbase.wal.dir

2018-07-11 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540750#comment-16540750
 ] 

Hadoop QA commented on HBASE-20734:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 4 new or modified test 
files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
55s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
39s{color} | {color:green} branch-1 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
38s{color} | {color:green} branch-1 passed with JDK v1.7.0_181 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
27s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  2m 
38s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
32s{color} | {color:green} branch-1 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
34s{color} | {color:green} branch-1 passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
33s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
34s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
34s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  1m 
25s{color} | {color:red} hbase-server: The patch generated 12 new + 659 
unchanged - 12 fixed = 671 total (was 671) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  2m 
29s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}  
1m 32s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
26s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
36s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 28m 56s{color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
17s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 44s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.io.TestHeapSize |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:1f3957d |
| JIRA Issue | HBASE-20734 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12931222/HBASE-20734.branch-1.001.patch
 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  shadedjars  
hadoopcheck  hbaseanti  checkstyle  compile  |
| uname | Linux 9fa7a9a9a09d 

[jira] [Commented] (HBASE-20866) HBase 1.x scan performance degradation compared to 0.98 version

2018-07-11 Thread Ted Yu (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540742#comment-16540742
 ] 

Ted Yu commented on HBASE-20866:


Vikas:
Can you include trivial change to hbase-server module so that the tests in that 
module get run by QA ?

Thanks

> HBase 1.x scan performance degradation compared to 0.98 version
> ---
>
> Key: HBASE-20866
> URL: https://issues.apache.org/jira/browse/HBASE-20866
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.2
>Reporter: Vikas Vishwakarma
>Assignee: Vikas Vishwakarma
>Priority: Critical
> Fix For: 1.5.0, 1.2.7, 1.3.3, 1.4.6
>
> Attachments: HBASE-20866.branch-1.3.001.patch, 
> HBASE-20866.branch-1.3.002.patch
>
>
> Internally while testing 1.3 as part of migration from 0.98 to 1.3 we 
> observed perf degradation in scan performance for phoenix queries varying 
> from few 10's to upto 200% depending on the query being executed. We tried 
> simple native HBase scan and there also we saw upto 40% degradation in 
> performance when the number of column qualifiers are high (40-50+)
> To identify the root cause of performance diff between 0.98 and 1.3 we 
> carried out lot of experiments with profiling and git bisect iterations, 
> however we were not able to identify any particular source of scan 
> performance degradation and it looked like this is an accumulated degradation 
> of 5-10% over various enhancements and refactoring.
> We identified few major enhancements like partialResult handling, 
> ScannerContext with heartbeat processing, time/size limiting, RPC 
> refactoring, etc that could have contributed to small degradation in 
> performance which put together could be leading to large overall degradation.
> One of the changes is 
> [HBASE-11544|https://jira.apache.org/jira/browse/HBASE-11544] which 
> implements partialResult handling. In ClientScanner.java the results received 
> from server are cached on the client side by converting the result array into 
> an ArrayList. This function gets called in a loop depending on the number of 
> rows in the scan result. Example for ten’s of millions of rows scanned, this 
> can be called in the order of millions of times.
> In almost all the cases 99% of the time (except for handling partial results, 
> etc). We are just taking the resultsFromServer converting it into a ArrayList 
> resultsToAddToCache in addResultsToList(..) and then iterating over the list 
> again and adding it to cache in loadCache(..) as given in the code path below
> In ClientScanner → loadCache(..) → getResultsToAddToCache(..) → 
> addResultsToList(..) →
> {code:java}
> loadCache() {
> ...
>  List resultsToAddToCache =
>  getResultsToAddToCache(values, callable.isHeartbeatMessage());
> ...
> …
>    for (Result rs : resultsToAddToCache) {
>  rs = filterLoadedCell(rs);
>  cache.add(rs);
> ...
>    }
> }
> getResultsToAddToCache(..) {
> ..
>    final boolean isBatchSet = scan != null && scan.getBatch() > 0;
>    final boolean allowPartials = scan != null && 
> scan.getAllowPartialResults();
> ..
>    if (allowPartials || isBatchSet) {
>  addResultsToList(resultsToAddToCache, resultsFromServer, 0,
>    (null == resultsFromServer ? 0 : resultsFromServer.length));
>  return resultsToAddToCache;
>    }
> ...
> }
> private void addResultsToList(List outputList, Result[] inputArray, 
> int start, int end) {
>    if (inputArray == null || start < 0 || end > inputArray.length) return;
>    for (int i = start; i < end; i++) {
>  outputList.add(inputArray[i]);
>    }
>  }{code}
>  
> It looks like we can avoid the result array to arraylist conversion 
> (resultsFromServer --> resultsToAddToCache ) for the first case which is also 
> the most frequent case and instead directly take the values arraay returned 
> by callable and add it to the cache without converting it into ArrayList.
> I have taken both these flags allowPartials and isBatchSet out in loadcahe() 
> and I am directly adding values to scanner cache if the above condition is 
> pass instead of coverting it into arrayList by calling 
> getResultsToAddToCache(). For example:
> {code:java}
> protected void loadCache() throws IOException {
> Result[] values = null;
> ..
> final boolean isBatchSet = scan != null && scan.getBatch() > 0;
> final boolean allowPartials = scan != null && scan.getAllowPartialResults();
> ..
> for (;;) {
> try {
> values = call(callable, caller, scannerTimeout);
> ..
> } catch (DoNotRetryIOException | NeedUnmanagedConnectionException e) {
> ..
> }
> if (allowPartials || isBatchSet) {  // DIRECTLY COPY values TO CACHE
> if (values != null) {
> for (int v=0; v Result rs = values[v];
> 
> cache.add(rs);
> ...
> } else { // DO ALL THE REGULAR PARTIAL 

[jira] [Updated] (HBASE-18840) Add functionality to refresh meta table at master startup

2018-07-11 Thread Zach York (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-18840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zach York updated HBASE-18840:
--
Attachment: HBASE-18840.HBASE-18477.007.patch

> Add functionality to refresh meta table at master startup
> -
>
> Key: HBASE-18840
> URL: https://issues.apache.org/jira/browse/HBASE-18840
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: HBASE-18477
>Reporter: Zach York
>Assignee: Zach York
>Priority: Major
> Attachments: HBASE-18840.HBASE-18477.001.patch, 
> HBASE-18840.HBASE-18477.002.patch, HBASE-18840.HBASE-18477.003 (2) (1).patch, 
> HBASE-18840.HBASE-18477.003 (2).patch, HBASE-18840.HBASE-18477.003.patch, 
> HBASE-18840.HBASE-18477.004.patch, HBASE-18840.HBASE-18477.005.patch, 
> HBASE-18840.HBASE-18477.006.patch, HBASE-18840.HBASE-18477.007.patch
>
>
> If a HBase cluster’s hbase:meta table is deleted or a cluster is started with 
> a new meta table, HBase needs the functionality to synchronize it’s metadata 
> from Storage.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HBASE-20868) Fix TestCheckTestClasses on HBASE-18477

2018-07-11 Thread Zach York (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-20868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zach York updated HBASE-20868:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

pushed to HBASE-18477

> Fix TestCheckTestClasses on HBASE-18477
> ---
>
> Key: HBASE-20868
> URL: https://issues.apache.org/jira/browse/HBASE-20868
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: HBASE-18477
>Reporter: Zach York
>Assignee: Zach York
>Priority: Minor
> Fix For: HBASE-18477
>
> Attachments: HBASE-20868.HBASE-18477.001.patch, 
> HBASE-20868.HBASE-18477.002.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20734) Colocate recovered edits directory with hbase.wal.dir

2018-07-11 Thread Ted Yu (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540733#comment-16540733
 ] 

Ted Yu commented on HBASE-20734:


If it is not too much trouble, can you base patch on master branch ?

Please also publish it on reviewboard.

> Colocate recovered edits directory with hbase.wal.dir
> -
>
> Key: HBASE-20734
> URL: https://issues.apache.org/jira/browse/HBASE-20734
> Project: HBase
>  Issue Type: Improvement
>  Components: MTTR, Recovery, wal
>Reporter: Ted Yu
>Assignee: Zach York
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20734.branch-1.001.patch
>
>
> During investigation of HBASE-20723, I realized that we wouldn't get the best 
> performance when hbase.wal.dir is configured to be on different (fast) media 
> than hbase rootdir w.r.t. recovered edits since recovered edits directory is 
> currently under rootdir.
> Such setup may not result in fast recovery when there is region server 
> failover.
> This issue is to find proper (hopefully backward compatible) way in 
> colocating recovered edits directory with hbase.wal.dir .



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HBASE-20651) Master, prevents hbck or shell command to reassign the split parent region

2018-07-11 Thread Andrew Purtell (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-20651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-20651:
---
Fix Version/s: 1.4.6
   1.3.3
   1.2.7

> Master, prevents hbck or shell command to reassign the split parent region
> --
>
> Key: HBASE-20651
> URL: https://issues.apache.org/jira/browse/HBASE-20651
> Project: HBase
>  Issue Type: Improvement
>  Components: master
>Affects Versions: 1.2.6
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Fix For: 1.5.0, 1.2.7, 1.3.3, 1.4.6
>
> Attachments: HBASE-20651-branch-1-v001.patch, 
> HBASE-20651-branch-1-v002.patch, HBASE-20651-branch-1-v003.patch
>
>
> We are seeing that hbck brings back split parent region and this causes 
> region inconsistency. More details will be filled as reproduce is still 
> ongoing. Might need to do something at hbck or master to prevent this from 
> happening.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HBASE-20734) Colocate recovered edits directory with hbase.wal.dir

2018-07-11 Thread Zach York (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-20734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zach York updated HBASE-20734:
--
Status: Patch Available  (was: Open)

> Colocate recovered edits directory with hbase.wal.dir
> -
>
> Key: HBASE-20734
> URL: https://issues.apache.org/jira/browse/HBASE-20734
> Project: HBase
>  Issue Type: Improvement
>  Components: MTTR, Recovery, wal
>Reporter: Ted Yu
>Assignee: Zach York
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20734.branch-1.001.patch
>
>
> During investigation of HBASE-20723, I realized that we wouldn't get the best 
> performance when hbase.wal.dir is configured to be on different (fast) media 
> than hbase rootdir w.r.t. recovered edits since recovered edits directory is 
> currently under rootdir.
> Such setup may not result in fast recovery when there is region server 
> failover.
> This issue is to find proper (hopefully backward compatible) way in 
> colocating recovered edits directory with hbase.wal.dir .



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HBASE-20734) Colocate recovered edits directory with hbase.wal.dir

2018-07-11 Thread Zach York (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-20734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zach York updated HBASE-20734:
--
Attachment: HBASE-20734.branch-1.001.patch

> Colocate recovered edits directory with hbase.wal.dir
> -
>
> Key: HBASE-20734
> URL: https://issues.apache.org/jira/browse/HBASE-20734
> Project: HBase
>  Issue Type: Improvement
>  Components: MTTR, Recovery, wal
>Reporter: Ted Yu
>Assignee: Zach York
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20734.branch-1.001.patch
>
>
> During investigation of HBASE-20723, I realized that we wouldn't get the best 
> performance when hbase.wal.dir is configured to be on different (fast) media 
> than hbase rootdir w.r.t. recovered edits since recovered edits directory is 
> currently under rootdir.
> Such setup may not result in fast recovery when there is region server 
> failover.
> This issue is to find proper (hopefully backward compatible) way in 
> colocating recovered edits directory with hbase.wal.dir .



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-18477) Umbrella JIRA for HBase Read Replica clusters

2018-07-11 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540615#comment-16540615
 ] 

Sean Busbey commented on HBASE-18477:
-

FYI, haven't forgotten about this. just a bit underwater. should be back with 
bandwidth next week.

> Umbrella JIRA for HBase Read Replica clusters
> -
>
> Key: HBASE-18477
> URL: https://issues.apache.org/jira/browse/HBASE-18477
> Project: HBase
>  Issue Type: New Feature
>Reporter: Zach York
>Assignee: Zach York
>Priority: Major
> Attachments: HBase Read-Replica Clusters Scope doc.docx, HBase 
> Read-Replica Clusters Scope doc.pdf, HBase Read-Replica Clusters Scope 
> doc_v2.docx, HBase Read-Replica Clusters Scope doc_v2.pdf
>
>
> Recently, changes (such as HBASE-17437) have unblocked HBase to run with a 
> root directory external to the cluster (such as in Amazon S3). This means 
> that the data is stored outside of the cluster and can be accessible after 
> the cluster has been terminated. One use case that is often asked about is 
> pointing multiple clusters to one root directory (sharing the data) to have 
> read resiliency in the case of a cluster failure.
>  
> This JIRA is an umbrella JIRA to contain all the tasks necessary to create a 
> read-replica HBase cluster that is pointed at the same root directory.
>  
> This requires making the Read-Replica cluster Read-Only (no metadata 
> operation or data operations).
> Separating the hbase:meta table for each cluster (Otherwise HBase gets 
> confused with multiple clusters trying to update the meta table with their ip 
> addresses)
> Adding refresh functionality for the meta table to ensure new metadata is 
> picked up on the read replica cluster.
> Adding refresh functionality for HFiles for a given table to ensure new data 
> is picked up on the read replica cluster.
>  
> This can be used with any existing cluster that is backed by an external 
> filesystem.
>  
> Please note that this feature is still quite manual (with the potential for 
> automation later).
>  
> More information on this particular feature can be found here: 
> https://aws.amazon.com/blogs/big-data/setting-up-read-replica-clusters-with-hbase-on-amazon-s3/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20697) Can't cache All region locations of the specify table by calling table.getRegionLocator().getAllRegionLocations()

2018-07-11 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540608#comment-16540608
 ] 

Hudson commented on HBASE-20697:


Results for branch branch-1.4
[build #382 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/382/]: 
(x) *{color:red}-1 overall{color}*

details (if available):

(x) {color:red}-1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/382//General_Nightly_Build_Report/]


(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/382//JDK7_Nightly_Build_Report/]


(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/382//JDK8_Nightly_Build_Report_(Hadoop2)/]




(x) {color:red}-1 source release artifact{color}
-- See build output for details.


> Can't cache All region locations of the specify table by calling 
> table.getRegionLocator().getAllRegionLocations()
> -
>
> Key: HBASE-20697
> URL: https://issues.apache.org/jira/browse/HBASE-20697
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.1, 1.2.6, 2.0.1
>Reporter: zhaoyuan
>Assignee: zhaoyuan
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.4.6, 2.0.2, 2.2.0, 2.1.1
>
> Attachments: HBASE-20697.branch-1.2.001.patch, 
> HBASE-20697.branch-1.2.002.patch, HBASE-20697.branch-1.2.003.patch, 
> HBASE-20697.branch-1.2.004.patch, HBASE-20697.master.001.patch, 
> HBASE-20697.master.002.patch, HBASE-20697.master.002.patch, 
> HBASE-20697.master.003.patch
>
>
> When we upgrade and restart  a new version application which will read and 
> write to HBase, we will get some operation timeout. The time out is expected 
> because when the application restarts,It will not hold any region locations 
> cache and do communication with zk and meta regionserver to get region 
> locations.
> We want to avoid these timeouts so we do warmup work and as far as I am 
> concerned,the method table.getRegionLocator().getAllRegionLocations() will 
> fetch all region locations and cache them. However, it didn't work good. 
> There are still a lot of time outs,so it confused me. 
> I dig into the source code and find something below
> {code:java}
> // code placeholder
> public List getAllRegionLocations() throws IOException {
>   TableName tableName = getName();
>   NavigableMap locations =
>   MetaScanner.allTableRegions(this.connection, tableName);
>   ArrayList regions = new ArrayList<>(locations.size());
>   for (Entry entry : locations.entrySet()) {
> regions.add(new HRegionLocation(entry.getKey(), entry.getValue()));
>   }
>   if (regions.size() > 0) {
> connection.cacheLocation(tableName, new RegionLocations(regions));
>   }
>   return regions;
> }
> In MetaCache
> public void cacheLocation(final TableName tableName, final RegionLocations 
> locations) {
>   byte [] startKey = 
> locations.getRegionLocation().getRegionInfo().getStartKey();
>   ConcurrentMap tableLocations = 
> getTableLocations(tableName);
>   RegionLocations oldLocation = tableLocations.putIfAbsent(startKey, 
> locations);
>   boolean isNewCacheEntry = (oldLocation == null);
>   if (isNewCacheEntry) {
> if (LOG.isTraceEnabled()) {
>   LOG.trace("Cached location: " + locations);
> }
> addToCachedServers(locations);
> return;
>   }
> {code}
> It will collect all regions into one RegionLocations object and only cache 
> the first not null region location and then when we put or get to hbase, we 
> do getCacheLocation() 
> {code:java}
> // code placeholder
> public RegionLocations getCachedLocation(final TableName tableName, final 
> byte [] row) {
>   ConcurrentNavigableMap tableLocations =
> getTableLocations(tableName);
>   Entry e = tableLocations.floorEntry(row);
>   if (e == null) {
> if (metrics!= null) metrics.incrMetaCacheMiss();
> return null;
>   }
>   RegionLocations possibleRegion = e.getValue();
>   // make sure that the end key is greater than the row we're looking
>   // for, otherwise the row actually belongs in the next region, not
>   // this one. the exception case is when the endkey is
>   // HConstants.EMPTY_END_ROW, signifying that the region we're
>   // checking is actually the last region in the table.
>   byte[] endKey = 
> possibleRegion.getRegionLocation().getRegionInfo().getEndKey();
>   if (Bytes.equals(endKey, HConstants.EMPTY_END_ROW) ||
>   getRowComparator(tableName).compareRows(
>   endKey, 0, endKey.length, row, 0, row.length) > 0) {
> if (metrics != null) metrics.incrMetaCacheHit();

[jira] [Commented] (HBASE-20257) hbase-spark should not depend on com.google.code.findbugs.jsr305

2018-07-11 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540606#comment-16540606
 ] 

Sean Busbey commented on HBASE-20257:
-

sweet. checking again.

> hbase-spark should not depend on com.google.code.findbugs.jsr305
> 
>
> Key: HBASE-20257
> URL: https://issues.apache.org/jira/browse/HBASE-20257
> Project: HBase
>  Issue Type: Task
>  Components: build, spark
>Affects Versions: 3.0.0
>Reporter: Ted Yu
>Assignee: Artem Ervits
>Priority: Minor
>  Labels: beginner
> Attachments: HBASE-20257.v01.patch, HBASE-20257.v02.patch, 
> HBASE-20257.v03.patch, HBASE-20257.v04.patch, HBASE-20257.v05.patch
>
>
> The following can be observed in the build output of master branch:
> {code}
> [WARNING] Rule 0: org.apache.maven.plugins.enforcer.BannedDependencies failed 
> with message:
> We don't allow the JSR305 jar from the Findbugs project, see HBASE-16321.
> Found Banned Dependency: com.google.code.findbugs:jsr305:jar:1.3.9
> Use 'mvn dependency:tree' to locate the source of the banned dependencies.
> {code}
> Here is related snippet from hbase-spark/pom.xml:
> {code}
> 
>   com.google.code.findbugs
>   jsr305
> {code}
> Dependency on jsr305 should be dropped.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20651) Master, prevents hbck or shell command to reassign the split parent region

2018-07-11 Thread huaxiang sun (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540598#comment-16540598
 ] 

huaxiang sun commented on HBASE-20651:
--

Yes, this applies to 1.2, 1.3 and 1.4. Thanks [~apurtell] for taking care of it.

> Master, prevents hbck or shell command to reassign the split parent region
> --
>
> Key: HBASE-20651
> URL: https://issues.apache.org/jira/browse/HBASE-20651
> Project: HBase
>  Issue Type: Improvement
>  Components: master
>Affects Versions: 1.2.6
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Fix For: 1.5.0
>
> Attachments: HBASE-20651-branch-1-v001.patch, 
> HBASE-20651-branch-1-v002.patch, HBASE-20651-branch-1-v003.patch
>
>
> We are seeing that hbck brings back split parent region and this causes 
> region inconsistency. More details will be filled as reproduce is still 
> ongoing. Might need to do something at hbck or master to prevent this from 
> happening.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-18477) Umbrella JIRA for HBase Read Replica clusters

2018-07-11 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540590#comment-16540590
 ] 

Hudson commented on HBASE-18477:


Results for branch HBASE-18477
[build #261 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18477/261/]: 
(x) *{color:red}-1 overall{color}*

details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18477/261//General_Nightly_Build_Report/]




(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18477/261//JDK8_Nightly_Build_Report_(Hadoop2)/]


(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18477/261//JDK8_Nightly_Build_Report_(Hadoop3)/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for 
details|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18477/261//artifact/output-integration/hadoop-2.log].
 (note that this means we didn't run on Hadoop 3)


> Umbrella JIRA for HBase Read Replica clusters
> -
>
> Key: HBASE-18477
> URL: https://issues.apache.org/jira/browse/HBASE-18477
> Project: HBase
>  Issue Type: New Feature
>Reporter: Zach York
>Assignee: Zach York
>Priority: Major
> Attachments: HBase Read-Replica Clusters Scope doc.docx, HBase 
> Read-Replica Clusters Scope doc.pdf, HBase Read-Replica Clusters Scope 
> doc_v2.docx, HBase Read-Replica Clusters Scope doc_v2.pdf
>
>
> Recently, changes (such as HBASE-17437) have unblocked HBase to run with a 
> root directory external to the cluster (such as in Amazon S3). This means 
> that the data is stored outside of the cluster and can be accessible after 
> the cluster has been terminated. One use case that is often asked about is 
> pointing multiple clusters to one root directory (sharing the data) to have 
> read resiliency in the case of a cluster failure.
>  
> This JIRA is an umbrella JIRA to contain all the tasks necessary to create a 
> read-replica HBase cluster that is pointed at the same root directory.
>  
> This requires making the Read-Replica cluster Read-Only (no metadata 
> operation or data operations).
> Separating the hbase:meta table for each cluster (Otherwise HBase gets 
> confused with multiple clusters trying to update the meta table with their ip 
> addresses)
> Adding refresh functionality for the meta table to ensure new metadata is 
> picked up on the read replica cluster.
> Adding refresh functionality for HFiles for a given table to ensure new data 
> is picked up on the read replica cluster.
>  
> This can be used with any existing cluster that is backed by an external 
> filesystem.
>  
> Please note that this feature is still quite manual (with the potential for 
> automation later).
>  
> More information on this particular feature can be found here: 
> https://aws.amazon.com/blogs/big-data/setting-up-read-replica-clusters-with-hbase-on-amazon-s3/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20860) Merged region's RIT state may not be cleaned after master restart

2018-07-11 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540546#comment-16540546
 ] 

stack commented on HBASE-20860:
---

+1 Looks great. +1 for branch-2.0 too please (Or just say and I'll backport). 
Thanks [~allan163]

> Merged region's RIT state may not be cleaned after master restart
> -
>
> Key: HBASE-20860
> URL: https://issues.apache.org/jira/browse/HBASE-20860
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 3.0.0, 2.1.0, 2.0.1
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Major
> Fix For: 3.0.0, 2.0.2, 2.1.1
>
> Attachments: HBASE-20860.branch-2.0.002.patch, 
> HBASE-20860.branch-2.0.003.patch, HBASE-20860.branch-2.0.004.patch, 
> HBASE-20860.branch-2.0.005.patch, HBASE-20860.branch-2.0.patch
>
>
> In MergeTableRegionsProcedure, we issue UnassignProcedures to offline regions 
> to merge. But if we restart master just after MergeTableRegionsProcedure 
> finished these two UnassignProcedure and before it can delete their meta 
> entries. The new master will found these two region is CLOSED but no 
> procedures are attached to them. They will be regard as RIT regions and 
> nobody will clean the RIT state for them later.
> A quick way to resolve this stuck situation in the production env is 
> restarting master again, since the meta entries are deleted in 
> MergeTableRegionsProcedure. Here, I offer a fix for this problem.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20257) hbase-spark should not depend on com.google.code.findbugs.jsr305

2018-07-11 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540547#comment-16540547
 ] 

Hadoop QA commented on HBASE-20257:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
 5s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
 7s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
10m  1s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  5m 
42s{color} | {color:green} hbase-spark in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
 9s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 36m 25s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20257 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12931202/HBASE-20257.v05.patch 
|
| Optional Tests |  asflicense  javac  javadoc  unit  shadedjars  hadoopcheck  
xml  compile  |
| uname | Linux 53b859710d0c 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14 
08:53:28 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 0d33caa39a |
| maven | version: Apache Maven 3.5.4 
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_171 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/13596/testReport/ |
| Max. process+thread count | 841 (vs. ulimit of 1) |
| modules | C: hbase-spark U: hbase-spark |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/13596/console |
| Powered by | Apache Yetus 0.7.0   http://yetus.apache.org |


This message was automatically generated.



> hbase-spark should not depend on com.google.code.findbugs.jsr305
> 
>
> Key: HBASE-20257
> URL: https://issues.apache.org/jira/browse/HBASE-20257
> Project: HBase
>  Issue Type: Task
>  Components: build, spark
>Affects Versions: 3.0.0
>Reporter: Ted Yu
>Assignee: Artem Ervits
>Priority: Minor
>  Labels: beginner
> 

[jira] [Commented] (HBASE-20846) Table's shared lock is not held by sub-procedures after master restart

2018-07-11 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540528#comment-16540528
 ] 

stack commented on HBASE-20846:
---

Updating procedure store is expensive...takes time.

Do we need a new boolean?  What cases do we need to handle on load post-crash? 
To my mind, the procedure that holds a lock for the life of the procedure is 
the only case that needs re-institution?

When we read the store WALs on startup, we read in order. Can we not just look 
for Procedures with holdLock=true and if procedure is not yet finished -- it 
has sub-procedures or not  -- just re-create their lock before starting the 
procedure workers?

> Table's shared lock is not held by sub-procedures after master restart
> --
>
> Key: HBASE-20846
> URL: https://issues.apache.org/jira/browse/HBASE-20846
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.1.0
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Major
> Fix For: 3.0.0, 2.0.2, 2.1.1
>
> Attachments: HBASE-20846.branch-2.0.002.patch, 
> HBASE-20846.branch-2.0.patch
>
>
> Found this one when investigating ModifyTableProcedure got stuck while there 
> was a MoveRegionProcedure going on after master restart.
> Though this issue can be solved by HBASE-20752. But I discovered something 
> else.
> Before a MoveRegionProcedure can execute, it will hold the table's shared 
> lock. so,, when a UnassignProcedure was spwaned, it will not check the 
> table's shared lock since it is sure that its parent(MoveRegionProcedure) has 
> aquired the table's lock.
> {code:java}
> // If there is parent procedure, it would have already taken xlock, so no 
> need to take
>   // shared lock here. Otherwise, take shared lock.
>   if (!procedure.hasParent()
>   && waitTableQueueSharedLock(procedure, table) == null) {
>   return true;
>   }
> {code}
> But, it is not the case when Master was restarted. The child 
> procedure(UnassignProcedure) will be executed first after restart. Though it 
> has a parent(MoveRegionProcedure), but apprently the parent didn't hold the 
> table's lock.
> So, since it began to execute without hold the table's shared lock. A 
> ModifyTableProcedure can aquire the table's exclusive lock and execute at the 
> same time. Which is not possible if the master was not restarted.
> This will cause a stuck before HBASE-20752. But since HBASE-20752 has fixed, 
> I wrote a simple UT to repo this case.
> I think we don't have to check the parent for table's shared lock. It is a 
> shared lock, right? I think we can acquire it every time we need it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20175) hbase-spark needs scala dependency convergance

2018-07-11 Thread Artem Ervits (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540529#comment-16540529
 ] 

Artem Ervits commented on HBASE-20175:
--

ping [~mdrob] [~yuzhih...@gmail.com] updated patch with latest 
scala.maven.version

> hbase-spark needs scala dependency convergance
> --
>
> Key: HBASE-20175
> URL: https://issues.apache.org/jira/browse/HBASE-20175
> Project: HBase
>  Issue Type: Bug
>  Components: dependencies, spark
>Reporter: Mike Drob
>Assignee: Artem Ervits
>Priority: Major
> Attachments: HBASE-20175.v01.patch, HBASE-20175.v04.patch
>
>
> This is a follow-on to HBASE-16179 - I think we might need to specify an 
> exclude in the dependency management.
> {noformat}
> [INFO] --- scala-maven-plugin:3.2.0:compile (scala-compile-first) @ 
> hbase-spark ---
> [WARNING]  Expected all dependencies to require Scala version: 2.11.8
> [WARNING]  org.apache.hbase:hbase-spark:3.0.0-SNAPSHOT requires scala 
> version: 2.11.8
> [WARNING]  org.apache.spark:spark-streaming_2.11:2.1.1 requires scala 
> version: 2.11.8
> [WARNING]  org.apache.spark:spark-streaming_2.11:2.1.1 requires scala 
> version: 2.11.8
> [WARNING]  org.scalatest:scalatest_2.11:2.2.4 requires scala version: 2.11.2
> {noformat}
> [~tedyu] - since you're already fiddling in this area, do you want to take a 
> look?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HBASE-20175) hbase-spark needs scala dependency convergance

2018-07-11 Thread Artem Ervits (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-20175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Artem Ervits updated HBASE-20175:
-
Attachment: HBASE-20175.v04.patch

> hbase-spark needs scala dependency convergance
> --
>
> Key: HBASE-20175
> URL: https://issues.apache.org/jira/browse/HBASE-20175
> Project: HBase
>  Issue Type: Bug
>  Components: dependencies, spark
>Reporter: Mike Drob
>Assignee: Artem Ervits
>Priority: Major
> Attachments: HBASE-20175.v01.patch, HBASE-20175.v04.patch
>
>
> This is a follow-on to HBASE-16179 - I think we might need to specify an 
> exclude in the dependency management.
> {noformat}
> [INFO] --- scala-maven-plugin:3.2.0:compile (scala-compile-first) @ 
> hbase-spark ---
> [WARNING]  Expected all dependencies to require Scala version: 2.11.8
> [WARNING]  org.apache.hbase:hbase-spark:3.0.0-SNAPSHOT requires scala 
> version: 2.11.8
> [WARNING]  org.apache.spark:spark-streaming_2.11:2.1.1 requires scala 
> version: 2.11.8
> [WARNING]  org.apache.spark:spark-streaming_2.11:2.1.1 requires scala 
> version: 2.11.8
> [WARNING]  org.scalatest:scalatest_2.11:2.2.4 requires scala version: 2.11.2
> {noformat}
> [~tedyu] - since you're already fiddling in this area, do you want to take a 
> look?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HBASE-20175) hbase-spark needs scala dependency convergance

2018-07-11 Thread Artem Ervits (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-20175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Artem Ervits updated HBASE-20175:
-
Attachment: (was: HBASE-20175.v03.patch)

> hbase-spark needs scala dependency convergance
> --
>
> Key: HBASE-20175
> URL: https://issues.apache.org/jira/browse/HBASE-20175
> Project: HBase
>  Issue Type: Bug
>  Components: dependencies, spark
>Reporter: Mike Drob
>Assignee: Artem Ervits
>Priority: Major
> Attachments: HBASE-20175.v01.patch
>
>
> This is a follow-on to HBASE-16179 - I think we might need to specify an 
> exclude in the dependency management.
> {noformat}
> [INFO] --- scala-maven-plugin:3.2.0:compile (scala-compile-first) @ 
> hbase-spark ---
> [WARNING]  Expected all dependencies to require Scala version: 2.11.8
> [WARNING]  org.apache.hbase:hbase-spark:3.0.0-SNAPSHOT requires scala 
> version: 2.11.8
> [WARNING]  org.apache.spark:spark-streaming_2.11:2.1.1 requires scala 
> version: 2.11.8
> [WARNING]  org.apache.spark:spark-streaming_2.11:2.1.1 requires scala 
> version: 2.11.8
> [WARNING]  org.scalatest:scalatest_2.11:2.2.4 requires scala version: 2.11.2
> {noformat}
> [~tedyu] - since you're already fiddling in this area, do you want to take a 
> look?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HBASE-20175) hbase-spark needs scala dependency convergance

2018-07-11 Thread Artem Ervits (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-20175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Artem Ervits updated HBASE-20175:
-
Attachment: HBASE-20175.v03.patch

> hbase-spark needs scala dependency convergance
> --
>
> Key: HBASE-20175
> URL: https://issues.apache.org/jira/browse/HBASE-20175
> Project: HBase
>  Issue Type: Bug
>  Components: dependencies, spark
>Reporter: Mike Drob
>Assignee: Artem Ervits
>Priority: Major
> Attachments: HBASE-20175.v01.patch, HBASE-20175.v03.patch
>
>
> This is a follow-on to HBASE-16179 - I think we might need to specify an 
> exclude in the dependency management.
> {noformat}
> [INFO] --- scala-maven-plugin:3.2.0:compile (scala-compile-first) @ 
> hbase-spark ---
> [WARNING]  Expected all dependencies to require Scala version: 2.11.8
> [WARNING]  org.apache.hbase:hbase-spark:3.0.0-SNAPSHOT requires scala 
> version: 2.11.8
> [WARNING]  org.apache.spark:spark-streaming_2.11:2.1.1 requires scala 
> version: 2.11.8
> [WARNING]  org.apache.spark:spark-streaming_2.11:2.1.1 requires scala 
> version: 2.11.8
> [WARNING]  org.scalatest:scalatest_2.11:2.2.4 requires scala version: 2.11.2
> {noformat}
> [~tedyu] - since you're already fiddling in this area, do you want to take a 
> look?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20651) Master, prevents hbck or shell command to reassign the split parent region

2018-07-11 Thread Andrew Purtell (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540505#comment-16540505
 ] 

Andrew Purtell commented on HBASE-20651:


Doesn't this impact all of the 1.x branches? The change picks cleanly to 
branch-1.2, branch-1.3, and branch-1.4. Going to pick it after some local 
checks. In the future, we should be committing relevant fixes to all of the 
live branches, yes?

> Master, prevents hbck or shell command to reassign the split parent region
> --
>
> Key: HBASE-20651
> URL: https://issues.apache.org/jira/browse/HBASE-20651
> Project: HBase
>  Issue Type: Improvement
>  Components: master
>Affects Versions: 1.2.6
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Fix For: 1.5.0
>
> Attachments: HBASE-20651-branch-1-v001.patch, 
> HBASE-20651-branch-1-v002.patch, HBASE-20651-branch-1-v003.patch
>
>
> We are seeing that hbck brings back split parent region and this causes 
> region inconsistency. More details will be filled as reproduce is still 
> ongoing. Might need to do something at hbck or master to prevent this from 
> happening.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20617) Upgrade/remove jetty-jsp

2018-07-11 Thread Sakthi (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540502#comment-16540502
 ] 

Sakthi commented on HBASE-20617:


Thanks [~mdrob].

> Upgrade/remove jetty-jsp
> 
>
> Key: HBASE-20617
> URL: https://issues.apache.org/jira/browse/HBASE-20617
> Project: HBase
>  Issue Type: Improvement
>Reporter: Sakthi
>Assignee: Sakthi
>Priority: Minor
> Fix For: 3.0.0, 2.2.0
>
> Attachments: hbase-20617.master.001.patch
>
>
> jetty-jsp removed after jetty-9.2.x version. We use the 9.2 version. Research 
> so far brings out that apache-jsp might be of interest to us in jetty-9.4.x 
> version(as JettyJspServlet.class is in apache-jsp). Yet to figure out about 
> jetty-9.3.x.
> Filing to track this along.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20257) hbase-spark should not depend on com.google.code.findbugs.jsr305

2018-07-11 Thread Artem Ervits (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540487#comment-16540487
 ] 

Artem Ervits commented on HBASE-20257:
--

[~busbey] [~yuzhih...@gmail.com] please see v5 patch, added additional 
exclusion to hadoop-common for both hadoop2 and hadoop3 profiles. Cannot run 
tests on hbase-spark module due to 
https://issues.apache.org/jira/browse/HBASE-20872 but patch does compile 
against both hadoop 2 and 3 now.

> hbase-spark should not depend on com.google.code.findbugs.jsr305
> 
>
> Key: HBASE-20257
> URL: https://issues.apache.org/jira/browse/HBASE-20257
> Project: HBase
>  Issue Type: Task
>  Components: build, spark
>Affects Versions: 3.0.0
>Reporter: Ted Yu
>Assignee: Artem Ervits
>Priority: Minor
>  Labels: beginner
> Attachments: HBASE-20257.v01.patch, HBASE-20257.v02.patch, 
> HBASE-20257.v03.patch, HBASE-20257.v04.patch, HBASE-20257.v05.patch
>
>
> The following can be observed in the build output of master branch:
> {code}
> [WARNING] Rule 0: org.apache.maven.plugins.enforcer.BannedDependencies failed 
> with message:
> We don't allow the JSR305 jar from the Findbugs project, see HBASE-16321.
> Found Banned Dependency: com.google.code.findbugs:jsr305:jar:1.3.9
> Use 'mvn dependency:tree' to locate the source of the banned dependencies.
> {code}
> Here is related snippet from hbase-spark/pom.xml:
> {code}
> 
>   com.google.code.findbugs
>   jsr305
> {code}
> Dependency on jsr305 should be dropped.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HBASE-20257) hbase-spark should not depend on com.google.code.findbugs.jsr305

2018-07-11 Thread Artem Ervits (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-20257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Artem Ervits updated HBASE-20257:
-
Attachment: HBASE-20257.v05.patch

> hbase-spark should not depend on com.google.code.findbugs.jsr305
> 
>
> Key: HBASE-20257
> URL: https://issues.apache.org/jira/browse/HBASE-20257
> Project: HBase
>  Issue Type: Task
>  Components: build, spark
>Affects Versions: 3.0.0
>Reporter: Ted Yu
>Assignee: Artem Ervits
>Priority: Minor
>  Labels: beginner
> Attachments: HBASE-20257.v01.patch, HBASE-20257.v02.patch, 
> HBASE-20257.v03.patch, HBASE-20257.v04.patch, HBASE-20257.v05.patch
>
>
> The following can be observed in the build output of master branch:
> {code}
> [WARNING] Rule 0: org.apache.maven.plugins.enforcer.BannedDependencies failed 
> with message:
> We don't allow the JSR305 jar from the Findbugs project, see HBASE-16321.
> Found Banned Dependency: com.google.code.findbugs:jsr305:jar:1.3.9
> Use 'mvn dependency:tree' to locate the source of the banned dependencies.
> {code}
> Here is related snippet from hbase-spark/pom.xml:
> {code}
> 
>   com.google.code.findbugs
>   jsr305
> {code}
> Dependency on jsr305 should be dropped.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (HBASE-20872) Cause: java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMasterUncompilable source code - package org.apache.hbase.thirdpa

2018-07-11 Thread Artem Ervits (JIRA)
Artem Ervits created HBASE-20872:


 Summary: Cause: java.lang.RuntimeException: Failed construction of 
Master: class org.apache.hadoop.hbase.master.HMasterUncompilable source code - 
package org.apache.hbase.thirdparty.io.netty.channel does not exist
 Key: HBASE-20872
 URL: https://issues.apache.org/jira/browse/HBASE-20872
 Project: HBase
  Issue Type: Bug
Reporter: Artem Ervits


running
{code:java}
mvn clean test{code}
on hbase-spark fails with
{code:java}
Cause: java.lang.RuntimeException: Failed construction of Master: class 
org.apache.hadoop.hbase.master.HMasterUncompilable source code - package 
org.apache.hbase.thirdparty.io.netty.channel does not exist
at 
org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:136)
at 
org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:212)
at org.apache.hadoop.hbase.LocalHBaseCluster.(LocalHBaseCluster.java:159)
at org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:250)
at org.apache.hadoop.hbase.MiniHBaseCluster.(MiniHBaseCluster.java:121)
at 
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:1042)
at 
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:988)
at 
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:859)
at 
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:853)
at 
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:782)
...
Cause: java.lang.ExceptionInInitializerError:
at 
org.apache.hadoop.hbase.regionserver.HRegionServer.setupNetty(HRegionServer.java:688)
at 
org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:547)
at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:486)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at 
org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:131)
at 
org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:212)
at org.apache.hadoop.hbase.LocalHBaseCluster.(LocalHBaseCluster.java:159)
...
Cause: java.lang.RuntimeException: Uncompilable source code - package 
org.apache.hbase.thirdparty.io.netty.channel does not exist
at 
org.apache.hadoop.hbase.util.NettyEventLoopGroupConfig.(NettyEventLoopGroupConfig.java:20)
at 
org.apache.hadoop.hbase.regionserver.HRegionServer.setupNetty(HRegionServer.java:688)
at 
org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:547)
at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:486)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at 
org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:131)
at 
org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:212){code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (HBASE-20871) Backport HBASE-20847 to branch-2.0: "The parent procedure of RegionTransitionProcedure may not have the table lock"

2018-07-11 Thread stack (JIRA)
stack created HBASE-20871:
-

 Summary: Backport HBASE-20847 to branch-2.0: "The parent procedure 
of RegionTransitionProcedure may not have the table lock"
 Key: HBASE-20871
 URL: https://issues.apache.org/jira/browse/HBASE-20871
 Project: HBase
  Issue Type: Bug
  Components: amv2
Reporter: stack
Assignee: stack
 Fix For: 2.0.2


Evaluate HBASE-20847 for backport before we cut 2.0.2.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Comment Edited] (HBASE-20847) The parent procedure of RegionTransitionProcedure may not have the table lock

2018-07-11 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540460#comment-16540460
 ] 

stack edited comment on HBASE-20847 at 7/11/18 5:54 PM:


+1 on v3. I like explanation in releaseExclusiveLock.

On backport to branch-2.0, I'm worried there'll be side-effects and that we 
need more than this fix. Let me make a separate issue for backport to 
branch-2.0. Will evaluate before pushing a 2.0.2. I can do it. HBASE-20871




was (Author: stack):
+1 on v3. I like explanation in releaseExclusiveLock.

On backport to branch-2.0, I'm worried there'll be side-effects and that we 
need more than this fix. Let me make a separate issue for backport to 
branch-2.0. Will evaluate before pushing a 2.0.2. I can do it.



> The parent procedure of RegionTransitionProcedure may not have the table lock
> -
>
> Key: HBASE-20847
> URL: https://issues.apache.org/jira/browse/HBASE-20847
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2, Region Assignment
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Attachments: HBASE-20847-addendum.patch, 
> HBASE-20847-branch-2.0-v1.patch, HBASE-20847-branch-2.0.patch, 
> HBASE-20847-v1.patch, HBASE-20847-v2.patch, HBASE-20847-v3.patch, 
> HBASE-20847.patch
>
>
> For example, SCP can also schedule AssignProcedure and obviously it will not 
> hold the table lock.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20847) The parent procedure of RegionTransitionProcedure may not have the table lock

2018-07-11 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540460#comment-16540460
 ] 

stack commented on HBASE-20847:
---

+1 on v3. I like explanation in releaseExclusiveLock.

On backport to branch-2.0, I'm worried there'll be side-effects and that we 
need more than this fix. Let me make a separate issue for backport to 
branch-2.0. Will evaluate before pushing a 2.0.2. I can do it.



> The parent procedure of RegionTransitionProcedure may not have the table lock
> -
>
> Key: HBASE-20847
> URL: https://issues.apache.org/jira/browse/HBASE-20847
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2, Region Assignment
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Attachments: HBASE-20847-addendum.patch, 
> HBASE-20847-branch-2.0-v1.patch, HBASE-20847-branch-2.0.patch, 
> HBASE-20847-v1.patch, HBASE-20847-v2.patch, HBASE-20847-v3.patch, 
> HBASE-20847.patch
>
>
> For example, SCP can also schedule AssignProcedure and obviously it will not 
> hold the table lock.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20866) HBase 1.x scan performance degradation compared to 0.98 version

2018-07-11 Thread Mike Drob (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540438#comment-16540438
 ] 

Mike Drob commented on HBASE-20866:
---

I have a little concern about how different this code is between 1.x and 2.x -- 
would love to see if the same logic can be applied to the newer branches as 
well, but I'm having a hard time following.

> HBase 1.x scan performance degradation compared to 0.98 version
> ---
>
> Key: HBASE-20866
> URL: https://issues.apache.org/jira/browse/HBASE-20866
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.2
>Reporter: Vikas Vishwakarma
>Assignee: Vikas Vishwakarma
>Priority: Critical
> Fix For: 1.5.0, 1.2.7, 1.3.3, 1.4.6
>
> Attachments: HBASE-20866.branch-1.3.001.patch, 
> HBASE-20866.branch-1.3.002.patch
>
>
> Internally while testing 1.3 as part of migration from 0.98 to 1.3 we 
> observed perf degradation in scan performance for phoenix queries varying 
> from few 10's to upto 200% depending on the query being executed. We tried 
> simple native HBase scan and there also we saw upto 40% degradation in 
> performance when the number of column qualifiers are high (40-50+)
> To identify the root cause of performance diff between 0.98 and 1.3 we 
> carried out lot of experiments with profiling and git bisect iterations, 
> however we were not able to identify any particular source of scan 
> performance degradation and it looked like this is an accumulated degradation 
> of 5-10% over various enhancements and refactoring.
> We identified few major enhancements like partialResult handling, 
> ScannerContext with heartbeat processing, time/size limiting, RPC 
> refactoring, etc that could have contributed to small degradation in 
> performance which put together could be leading to large overall degradation.
> One of the changes is 
> [HBASE-11544|https://jira.apache.org/jira/browse/HBASE-11544] which 
> implements partialResult handling. In ClientScanner.java the results received 
> from server are cached on the client side by converting the result array into 
> an ArrayList. This function gets called in a loop depending on the number of 
> rows in the scan result. Example for ten’s of millions of rows scanned, this 
> can be called in the order of millions of times.
> In almost all the cases 99% of the time (except for handling partial results, 
> etc). We are just taking the resultsFromServer converting it into a ArrayList 
> resultsToAddToCache in addResultsToList(..) and then iterating over the list 
> again and adding it to cache in loadCache(..) as given in the code path below
> In ClientScanner → loadCache(..) → getResultsToAddToCache(..) → 
> addResultsToList(..) →
> {code:java}
> loadCache() {
> ...
>  List resultsToAddToCache =
>  getResultsToAddToCache(values, callable.isHeartbeatMessage());
> ...
> …
>    for (Result rs : resultsToAddToCache) {
>  rs = filterLoadedCell(rs);
>  cache.add(rs);
> ...
>    }
> }
> getResultsToAddToCache(..) {
> ..
>    final boolean isBatchSet = scan != null && scan.getBatch() > 0;
>    final boolean allowPartials = scan != null && 
> scan.getAllowPartialResults();
> ..
>    if (allowPartials || isBatchSet) {
>  addResultsToList(resultsToAddToCache, resultsFromServer, 0,
>    (null == resultsFromServer ? 0 : resultsFromServer.length));
>  return resultsToAddToCache;
>    }
> ...
> }
> private void addResultsToList(List outputList, Result[] inputArray, 
> int start, int end) {
>    if (inputArray == null || start < 0 || end > inputArray.length) return;
>    for (int i = start; i < end; i++) {
>  outputList.add(inputArray[i]);
>    }
>  }{code}
>  
> It looks like we can avoid the result array to arraylist conversion 
> (resultsFromServer --> resultsToAddToCache ) for the first case which is also 
> the most frequent case and instead directly take the values arraay returned 
> by callable and add it to the cache without converting it into ArrayList.
> I have taken both these flags allowPartials and isBatchSet out in loadcahe() 
> and I am directly adding values to scanner cache if the above condition is 
> pass instead of coverting it into arrayList by calling 
> getResultsToAddToCache(). For example:
> {code:java}
> protected void loadCache() throws IOException {
> Result[] values = null;
> ..
> final boolean isBatchSet = scan != null && scan.getBatch() > 0;
> final boolean allowPartials = scan != null && scan.getAllowPartialResults();
> ..
> for (;;) {
> try {
> values = call(callable, caller, scannerTimeout);
> ..
> } catch (DoNotRetryIOException | NeedUnmanagedConnectionException e) {
> ..
> }
> if (allowPartials || isBatchSet) {  // DIRECTLY COPY values TO CACHE
> if (values != null) {
> for (int v=0; v Result 

[jira] [Updated] (HBASE-20697) Can't cache All region locations of the specify table by calling table.getRegionLocator().getAllRegionLocations()

2018-07-11 Thread Andrew Purtell (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-20697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-20697:
---
  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

> Can't cache All region locations of the specify table by calling 
> table.getRegionLocator().getAllRegionLocations()
> -
>
> Key: HBASE-20697
> URL: https://issues.apache.org/jira/browse/HBASE-20697
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.1, 1.2.6, 2.0.1
>Reporter: zhaoyuan
>Assignee: zhaoyuan
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.4.6, 2.0.2, 2.2.0, 2.1.1
>
> Attachments: HBASE-20697.branch-1.2.001.patch, 
> HBASE-20697.branch-1.2.002.patch, HBASE-20697.branch-1.2.003.patch, 
> HBASE-20697.branch-1.2.004.patch, HBASE-20697.master.001.patch, 
> HBASE-20697.master.002.patch, HBASE-20697.master.002.patch, 
> HBASE-20697.master.003.patch
>
>
> When we upgrade and restart  a new version application which will read and 
> write to HBase, we will get some operation timeout. The time out is expected 
> because when the application restarts,It will not hold any region locations 
> cache and do communication with zk and meta regionserver to get region 
> locations.
> We want to avoid these timeouts so we do warmup work and as far as I am 
> concerned,the method table.getRegionLocator().getAllRegionLocations() will 
> fetch all region locations and cache them. However, it didn't work good. 
> There are still a lot of time outs,so it confused me. 
> I dig into the source code and find something below
> {code:java}
> // code placeholder
> public List getAllRegionLocations() throws IOException {
>   TableName tableName = getName();
>   NavigableMap locations =
>   MetaScanner.allTableRegions(this.connection, tableName);
>   ArrayList regions = new ArrayList<>(locations.size());
>   for (Entry entry : locations.entrySet()) {
> regions.add(new HRegionLocation(entry.getKey(), entry.getValue()));
>   }
>   if (regions.size() > 0) {
> connection.cacheLocation(tableName, new RegionLocations(regions));
>   }
>   return regions;
> }
> In MetaCache
> public void cacheLocation(final TableName tableName, final RegionLocations 
> locations) {
>   byte [] startKey = 
> locations.getRegionLocation().getRegionInfo().getStartKey();
>   ConcurrentMap tableLocations = 
> getTableLocations(tableName);
>   RegionLocations oldLocation = tableLocations.putIfAbsent(startKey, 
> locations);
>   boolean isNewCacheEntry = (oldLocation == null);
>   if (isNewCacheEntry) {
> if (LOG.isTraceEnabled()) {
>   LOG.trace("Cached location: " + locations);
> }
> addToCachedServers(locations);
> return;
>   }
> {code}
> It will collect all regions into one RegionLocations object and only cache 
> the first not null region location and then when we put or get to hbase, we 
> do getCacheLocation() 
> {code:java}
> // code placeholder
> public RegionLocations getCachedLocation(final TableName tableName, final 
> byte [] row) {
>   ConcurrentNavigableMap tableLocations =
> getTableLocations(tableName);
>   Entry e = tableLocations.floorEntry(row);
>   if (e == null) {
> if (metrics!= null) metrics.incrMetaCacheMiss();
> return null;
>   }
>   RegionLocations possibleRegion = e.getValue();
>   // make sure that the end key is greater than the row we're looking
>   // for, otherwise the row actually belongs in the next region, not
>   // this one. the exception case is when the endkey is
>   // HConstants.EMPTY_END_ROW, signifying that the region we're
>   // checking is actually the last region in the table.
>   byte[] endKey = 
> possibleRegion.getRegionLocation().getRegionInfo().getEndKey();
>   if (Bytes.equals(endKey, HConstants.EMPTY_END_ROW) ||
>   getRowComparator(tableName).compareRows(
>   endKey, 0, endKey.length, row, 0, row.length) > 0) {
> if (metrics != null) metrics.incrMetaCacheHit();
> return possibleRegion;
>   }
>   // Passed all the way through, so we got nothing - complete cache miss
>   if (metrics != null) metrics.incrMetaCacheMiss();
>   return null;
> }
> {code}
> It will choose the first location to be possibleRegion and possibly it will 
> miss match.
> So did I forget something or may be wrong somewhere? If this is indeed a bug 
> I think it can be fixed not very hard.
> Hope commiters and PMC review this !
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20866) HBase 1.x scan performance degradation compared to 0.98 version

2018-07-11 Thread Andrew Purtell (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540407#comment-16540407
 ] 

Andrew Purtell commented on HBASE-20866:


The patch looks sane to be, but I am not super familiar with this code. It 
would help to view it again in context. Suggest putting the patch up on RB. 
Might help others too.

> HBase 1.x scan performance degradation compared to 0.98 version
> ---
>
> Key: HBASE-20866
> URL: https://issues.apache.org/jira/browse/HBASE-20866
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.2
>Reporter: Vikas Vishwakarma
>Assignee: Vikas Vishwakarma
>Priority: Critical
> Fix For: 1.5.0, 1.2.7, 1.3.3, 1.4.6
>
> Attachments: HBASE-20866.branch-1.3.001.patch, 
> HBASE-20866.branch-1.3.002.patch
>
>
> Internally while testing 1.3 as part of migration from 0.98 to 1.3 we 
> observed perf degradation in scan performance for phoenix queries varying 
> from few 10's to upto 200% depending on the query being executed. We tried 
> simple native HBase scan and there also we saw upto 40% degradation in 
> performance when the number of column qualifiers are high (40-50+)
> To identify the root cause of performance diff between 0.98 and 1.3 we 
> carried out lot of experiments with profiling and git bisect iterations, 
> however we were not able to identify any particular source of scan 
> performance degradation and it looked like this is an accumulated degradation 
> of 5-10% over various enhancements and refactoring.
> We identified few major enhancements like partialResult handling, 
> ScannerContext with heartbeat processing, time/size limiting, RPC 
> refactoring, etc that could have contributed to small degradation in 
> performance which put together could be leading to large overall degradation.
> One of the changes is 
> [HBASE-11544|https://jira.apache.org/jira/browse/HBASE-11544] which 
> implements partialResult handling. In ClientScanner.java the results received 
> from server are cached on the client side by converting the result array into 
> an ArrayList. This function gets called in a loop depending on the number of 
> rows in the scan result. Example for ten’s of millions of rows scanned, this 
> can be called in the order of millions of times.
> In almost all the cases 99% of the time (except for handling partial results, 
> etc). We are just taking the resultsFromServer converting it into a ArrayList 
> resultsToAddToCache in addResultsToList(..) and then iterating over the list 
> again and adding it to cache in loadCache(..) as given in the code path below
> In ClientScanner → loadCache(..) → getResultsToAddToCache(..) → 
> addResultsToList(..) →
> {code:java}
> loadCache() {
> ...
>  List resultsToAddToCache =
>  getResultsToAddToCache(values, callable.isHeartbeatMessage());
> ...
> …
>    for (Result rs : resultsToAddToCache) {
>  rs = filterLoadedCell(rs);
>  cache.add(rs);
> ...
>    }
> }
> getResultsToAddToCache(..) {
> ..
>    final boolean isBatchSet = scan != null && scan.getBatch() > 0;
>    final boolean allowPartials = scan != null && 
> scan.getAllowPartialResults();
> ..
>    if (allowPartials || isBatchSet) {
>  addResultsToList(resultsToAddToCache, resultsFromServer, 0,
>    (null == resultsFromServer ? 0 : resultsFromServer.length));
>  return resultsToAddToCache;
>    }
> ...
> }
> private void addResultsToList(List outputList, Result[] inputArray, 
> int start, int end) {
>    if (inputArray == null || start < 0 || end > inputArray.length) return;
>    for (int i = start; i < end; i++) {
>  outputList.add(inputArray[i]);
>    }
>  }{code}
>  
> It looks like we can avoid the result array to arraylist conversion 
> (resultsFromServer --> resultsToAddToCache ) for the first case which is also 
> the most frequent case and instead directly take the values arraay returned 
> by callable and add it to the cache without converting it into ArrayList.
> I have taken both these flags allowPartials and isBatchSet out in loadcahe() 
> and I am directly adding values to scanner cache if the above condition is 
> pass instead of coverting it into arrayList by calling 
> getResultsToAddToCache(). For example:
> {code:java}
> protected void loadCache() throws IOException {
> Result[] values = null;
> ..
> final boolean isBatchSet = scan != null && scan.getBatch() > 0;
> final boolean allowPartials = scan != null && scan.getAllowPartialResults();
> ..
> for (;;) {
> try {
> values = call(callable, caller, scannerTimeout);
> ..
> } catch (DoNotRetryIOException | NeedUnmanagedConnectionException e) {
> ..
> }
> if (allowPartials || isBatchSet) {  // DIRECTLY COPY values TO CACHE
> if (values != null) {
> for (int v=0; v Result rs = values[v];

[jira] [Commented] (HBASE-20617) Upgrade/remove jetty-jsp

2018-07-11 Thread Mike Drob (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540364#comment-16540364
 ] 

Mike Drob commented on HBASE-20617:
---

Great, thanks for the testing, [~jatsakthi]! Pushed this patch to branch-2 and 
master.

> Upgrade/remove jetty-jsp
> 
>
> Key: HBASE-20617
> URL: https://issues.apache.org/jira/browse/HBASE-20617
> Project: HBase
>  Issue Type: Improvement
>Reporter: Sakthi
>Assignee: Sakthi
>Priority: Minor
> Fix For: 3.0.0, 2.2.0
>
> Attachments: hbase-20617.master.001.patch
>
>
> jetty-jsp removed after jetty-9.2.x version. We use the 9.2 version. Research 
> so far brings out that apache-jsp might be of interest to us in jetty-9.4.x 
> version(as JettyJspServlet.class is in apache-jsp). Yet to figure out about 
> jetty-9.3.x.
> Filing to track this along.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HBASE-20617) Upgrade/remove jetty-jsp

2018-07-11 Thread Mike Drob (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-20617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mike Drob updated HBASE-20617:
--
   Resolution: Fixed
Fix Version/s: 2.2.0
   3.0.0
   Status: Resolved  (was: Patch Available)

> Upgrade/remove jetty-jsp
> 
>
> Key: HBASE-20617
> URL: https://issues.apache.org/jira/browse/HBASE-20617
> Project: HBase
>  Issue Type: Improvement
>Reporter: Sakthi
>Assignee: Sakthi
>Priority: Minor
> Fix For: 3.0.0, 2.2.0
>
> Attachments: hbase-20617.master.001.patch
>
>
> jetty-jsp removed after jetty-9.2.x version. We use the 9.2 version. Research 
> so far brings out that apache-jsp might be of interest to us in jetty-9.4.x 
> version(as JettyJspServlet.class is in apache-jsp). Yet to figure out about 
> jetty-9.3.x.
> Filing to track this along.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20847) The parent procedure of RegionTransitionProcedure may not have the table lock

2018-07-11 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540336#comment-16540336
 ] 

Hadoop QA commented on HBASE-20847:
---

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  5m 
 6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
43s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
30s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  5m 
 2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
31s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
10m  0s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}118m 
51s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
21s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}160m 44s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20847 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12931167/HBASE-20847-addendum.patch
 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  shadedjars  
hadoopcheck  hbaseanti  checkstyle  compile  |
| uname | Linux 05cf9c955e1e 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27 
10:45:36 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / a838f7631f |
| maven | version: Apache Maven 3.5.4 
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC3 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/13594/testReport/ |
| Max. process+thread count | 4845 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/13594/console |
| Powered by | Apache Yetus 0.7.0   http://yetus.apache.org |


This message was automatically generated.



> The parent procedure of 

[jira] [Commented] (HBASE-20835) Document how to get replication reporting

2018-07-11 Thread Mike Drob (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540319#comment-16540319
 ] 

Mike Drob commented on HBASE-20835:
---

I think part of my confusion is the state of replication internals, maybe. The 
layout in ZK seems like it is private and subject to change at any version 
boundary because the constants and config properties are defined in a bunch of 
IA.Private classes like ZKReplicationPeerStorage. Is there a stable, public 
facing method for getting the same metrics that I'm looking for?

Currently we are looking directly in ZK because that is the only place to get 
this information, and it happens to be very brittle. I would be very happy to 
move to a more well defined API.

> Document how to get replication reporting
> -
>
> Key: HBASE-20835
> URL: https://issues.apache.org/jira/browse/HBASE-20835
> Project: HBase
>  Issue Type: Task
>  Components: Replication
>Affects Versions: 2.1.0
>Reporter: Mike Drob
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 3.0.0
>
>
> Based on my questions at the tail end of HBASE-19543
> bq. We have some tooling that checks on replication queues and reads the 
> znode as the source of truth. When replication is disabled, it's expected 
> that the node was still there, but just empty. Is there a better way to get 
> this same information?
> I understand that with table based replication it doesn't make sense to check 
> ZK for status. However, losing the ability to inspect the data and get 
> information is a tough hit for operators. Do we have APIs that expose the 
> same sort of metrics?
> bq. how many peers/queues, queue size, position in the queue, and age of last 
> op
> Assigning to you for now, Duo, since you were both primary implementor and RM 
> for 2.1.0 and I'm not sure who else would know the answers. If the docs 
> already exist, then nothing to do but we should include them in the RN. Maybe 
> this will need additional code, but I hope it's already there and is 
> something we can write a workaround for.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20853) Polish "Add defaults to Table Interface so Implementors don't have to"

2018-07-11 Thread Balazs Meszaros (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540251#comment-16540251
 ] 

Balazs Meszaros commented on HBASE-20853:
-

Ok thanks for the clarification [~chia7712]. I will add an update soon.

> Polish "Add defaults to Table Interface so Implementors don't have to"
> --
>
> Key: HBASE-20853
> URL: https://issues.apache.org/jira/browse/HBASE-20853
> Project: HBase
>  Issue Type: Sub-task
>  Components: API
>Reporter: stack
>Assignee: Balazs Meszaros
>Priority: Major
>  Labels: beginner, beginners
> Fix For: 3.0.0, 2.0.2, 2.1.1
>
> Attachments: HBASE-20853.master.001.patch
>
>
> This issue is to address feedback that came in after commit on the parent 
> (FYI [~chia7712]). See tail of parent issue and amendment attached to parent 
> adding better defaults to the Table Interface.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


  1   2   >