[jira] [Commented] (HBASE-18479) should apply HBASE-18255 to HBASE_MASTER_OPTS too
[ https://issues.apache.org/jira/browse/HBASE-18479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122893#comment-16122893 ] Hudson commented on HBASE-18479: FAILURE: Integrated in Jenkins build HBase-1.4 #846 (See [https://builds.apache.org/job/HBase-1.4/846/]) HBASE-18479 should apply HBASE-18255 to HBASE_MASTER_OPTS too (tedyu: rev 97c6ee5124f2dc34bc9d8678594d9a86dbb4b25b) * (edit) conf/hbase-env.sh * (edit) conf/hbase-env.cmd > should apply HBASE-18255 to HBASE_MASTER_OPTS too > - > > Key: HBASE-18479 > URL: https://issues.apache.org/jira/browse/HBASE-18479 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.1, 1.2.6, 1.1.11 >Reporter: Yechao Chen >Assignee: Yechao Chen >Priority: Critical > Fix For: 1.4.0, 1.3.2, 1.5.0, 1.2.7 > > Attachments: HBASE-18479-branch-1.v1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18255) Time-Delayed HBase Performance Degradation with Java 7
[ https://issues.apache.org/jira/browse/HBASE-18255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122894#comment-16122894 ] Hudson commented on HBASE-18255: FAILURE: Integrated in Jenkins build HBase-1.4 #846 (See [https://builds.apache.org/job/HBase-1.4/846/]) HBASE-18479 should apply HBASE-18255 to HBASE_MASTER_OPTS too (tedyu: rev 97c6ee5124f2dc34bc9d8678594d9a86dbb4b25b) * (edit) conf/hbase-env.sh * (edit) conf/hbase-env.cmd > Time-Delayed HBase Performance Degradation with Java 7 > -- > > Key: HBASE-18255 > URL: https://issues.apache.org/jira/browse/HBASE-18255 > Project: HBase > Issue Type: Bug > Components: Performance >Affects Versions: 1.3.1, 1.2.6, 1.1.11 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Critical > Labels: jdk7 > Fix For: 1.4.0, 1.3.2, 1.2.7, 1.1.12 > > Attachments: HBASE-18255-branch-1.x.v1.patch > > > The good summary of the issue and provided resolution can be found in this > article: > https://community.hortonworks.com/articles/105802/time-delayed-hbase-performance-degradation-with-ja.html > In a few words, due to internal JVM 7 bug (which has been addressed only in > Java 8), HotSpot code cache can become full and after that ALL JIT > compilations get suspended indefinitely. The default value for code cache > size in JVM 7 is quite low: 48MB. It is recommended to increase this value at > least to 256MB (default in JVM 8). > This BUG affects only 1.x -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18387) [Thrift] Make principal configurable in DemoClient.java
[
https://issues.apache.org/jira/browse/HBASE-18387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122892#comment-16122892
]
Hudson commented on HBASE-18387:
FAILURE: Integrated in Jenkins build HBase-2.0 #308 (See
[https://builds.apache.org/job/HBase-2.0/308/])
HBASE-18387: [Thrift] Make principal configurable in DemoClient.java (elserj:
rev ee15c2c296d49e943c61b99e851449ee94394621)
* (edit)
hbase-examples/src/main/java/org/apache/hadoop/hbase/thrift/DemoClient.java
* (edit) hbase-examples/README.txt
> [Thrift] Make principal configurable in DemoClient.java
> ---
>
> Key: HBASE-18387
> URL: https://issues.apache.org/jira/browse/HBASE-18387
> Project: HBase
> Issue Type: Improvement
>Reporter: Lars George
>Assignee: Tamas Penzes
>Priority: Minor
> Labels: beginner
> Fix For: 2.0.0, 1.4.0, 1.3.2, 1.2.7, 1.1.12
>
> Attachments: HBASE-18387.master.001.patch,
> HBASE-18387.master.002.patch, HBASE-18387.master.003.patch
>
>
> In the Thrift1 demo client we have this code:
> {code}
> transport = new TSaslClientTransport("GSSAPI", null,
> "hbase", // Thrift server user name, should be an authorized proxy user.
> host, // Thrift server domain
> saslProperties, null, transport);
> {code}
> This will only work when the Thrift server is started with the {{hbase}}
> principal. Often this may deviate, for example I am using {{hbase-thrift}} to
> separate the names from those of backend servers.
> What we need is either an additional command line option to specify the name,
> or a property that can be set with -D and can be passed at runtime. I prefer
> the former, as the latter is making this a little convoluted.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-17125) Inconsistent result when use filter to read data
[
https://issues.apache.org/jira/browse/HBASE-17125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122891#comment-16122891
]
Hudson commented on HBASE-17125:
FAILURE: Integrated in Jenkins build HBase-2.0 #308 (See
[https://builds.apache.org/job/HBase-2.0/308/])
HBASE-17125 Inconsistent result when use filter to read data (zghao: rev
8197a31bbc4c49b4edfc2a0f01b3ef29b40e268d)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMinVersions.java
* (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/Query.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/ScanQueryMatcher.java
* (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/Get.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/UserScanQueryMatcher.java
* (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/Scan.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/ScanWildcardColumnTracker.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
> Inconsistent result when use filter to read data
>
>
> Key: HBASE-17125
> URL: https://issues.apache.org/jira/browse/HBASE-17125
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 2.0.0, 3.0.0
>
> Attachments: 17125-slack-13.txt, example.diff,
> HBASE-17125.master.001.patch, HBASE-17125.master.002.patch,
> HBASE-17125.master.002.patch, HBASE-17125.master.003.patch,
> HBASE-17125.master.004.patch, HBASE-17125.master.005.patch,
> HBASE-17125.master.006.patch, HBASE-17125.master.007.patch,
> HBASE-17125.master.008.patch, HBASE-17125.master.009.patch,
> HBASE-17125.master.009.patch, HBASE-17125.master.010.patch,
> HBASE-17125.master.011.patch, HBASE-17125.master.011.patch,
> HBASE-17125.master.012.patch, HBASE-17125.master.013.patch,
> HBASE-17125.master.014.patch, HBASE-17125.master.015.patch,
> HBASE-17125.master.016.patch, HBASE-17125.master.017.patch,
> HBASE-17125.master.018.patch, HBASE-17125.master.019.patch,
> HBASE-17125.master.020.patch, HBASE-17125.master.020.patch,
> HBASE-17125.master.021.patch, HBASE-17125.master.022.patch,
> HBASE-17125.master.checkReturnedVersions.patch,
> HBASE-17125.master.no-specified-filter.patch
>
>
> Assume a cloumn's max versions is 3, then we write 4 versions of this column.
> The oldest version doesn't remove immediately. But from the user view, the
> oldest version has gone. When user use a filter to query, if the filter skip
> a new version, then the oldest version will be seen again. But after compact
> the region, then the oldest version will never been seen. So it is weird for
> user. The query will get inconsistent result before and after region
> compaction.
> The reason is matchColumn method of UserScanQueryMatcher. It first check the
> cell by filter, then check the number of versions needed. So if the filter
> skip the new version, then the oldest version will be seen again when it is
> not removed.
> Have a discussion offline with [~Apache9] and [~fenghh], now we have two
> solution for this problem. The first idea is check the number of versions
> first, then check the cell by filter. As the comment of setFilter, the filter
> is called after all tests for ttl, column match, deletes and max versions
> have been run.
> {code}
> /**
>* Apply the specified server-side filter when performing the Query.
>* Only {@link Filter#filterKeyValue(Cell)} is called AFTER all tests
>* for ttl, column match, deletes and max versions have been run.
>* @param filter filter to run on the server
>* @return this for invocation chaining
>*/
> public Query setFilter(Filter filter) {
> this.filter = filter;
> return this;
> }
> {code}
> But this idea has another problem, if a column's max version is 5 and the
> user query only need 3 versions. It first check the version's number, then
> check the cell by filter. So the cells number of the result may less than 3.
> But there are 2 versions which don't read anymore.
> So the second idea has three steps.
> 1. check by the max versions of this column
> 2. check the kv by filter
> 3. check the versions which user need.
> But this will lead the ScanQueryMatcher more complicated. And this will break
> the javadoc of Query.setFilter.
> Now we don't have a final solution for this problem. Suggestions are welcomed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18563) Fix RAT License complaint about website jenkins scripts
[
https://issues.apache.org/jira/browse/HBASE-18563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122887#comment-16122887
]
Hudson commented on HBASE-18563:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3510 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/3510/])
HBASE-18563 Fix RAT License complaint about website jenkins scripts (esteban:
rev c37432fefbf4d1ff5bf80f5227c986f3bde281a1)
* (edit) dev-support/jenkins-scripts/generate-hbase-website.sh
* (edit) dev-support/jenkins-scripts/check-website-links.sh
> Fix RAT License complaint about website jenkins scripts
> ---
>
> Key: HBASE-18563
> URL: https://issues.apache.org/jira/browse/HBASE-18563
> Project: HBase
> Issue Type: Bug
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
>Priority: Trivial
> Fix For: 3.0.0
>
> Attachments: HBASE-18563.0001.patch
>
>
> {{2 Unknown Licenses
> *
> Files with unapproved licenses:
> dev-support/jenkins-scripts/check-website-links.sh
> dev-support/jenkins-scripts/generate-hbase-website.sh
> *
> }}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18024) HRegion#initializeRegionInternals should not re-create .hregioninfo file when the region directory no longer exists
[
https://issues.apache.org/jira/browse/HBASE-18024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122886#comment-16122886
]
Hudson commented on HBASE-18024:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3510 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/3510/])
HBASE-18024 HRegion#initializeRegionInternals should not re-create (esteban:
rev efd211debd8a37f215b1a6fdb982aa3ca890bc40)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionOpen.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestWALMonotonicallyIncreasingSeqId.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileRefresherChore.java
> HRegion#initializeRegionInternals should not re-create .hregioninfo file when
> the region directory no longer exists
> ---
>
> Key: HBASE-18024
> URL: https://issues.apache.org/jira/browse/HBASE-18024
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment, regionserver
>Affects Versions: 2.0.0, 1.4.0, 1.3.1, 1.2.5
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
> Fix For: 2.0.0, 1.4.0, 1.3.2, 1.5.0, 1.2.7
>
> Attachments: HBASE-18024.001.patch, HBASE-18024.002.patch,
> HBASE-18024.003.patch, HBASE-18024.004.patch
>
>
> When a RegionSever attempts to open a region, during initialization the RS
> tries to open the {{/data///.hregioninfo}}
> file, however if the {{.hregioninfofile}} doesn't exist, the RegionServer
> will create a new one on {{HRegionFileSystem#checkRegionInfoOnFilesystem}}. A
> side effect of that tools like hbck will incorrectly assume an inconsistency
> due the presence of this new {{.hregioninfofile}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18479) should apply HBASE-18255 to HBASE_MASTER_OPTS too
[ https://issues.apache.org/jira/browse/HBASE-18479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122876#comment-16122876 ] Hudson commented on HBASE-18479: SUCCESS: Integrated in Jenkins build HBase-1.2-JDK8 #185 (See [https://builds.apache.org/job/HBase-1.2-JDK8/185/]) HBASE-18479 should apply HBASE-18255 to HBASE_MASTER_OPTS too (tedyu: rev 9f809748d72b39489f202a56aba107cf7386f30c) * (edit) conf/hbase-env.cmd * (edit) conf/hbase-env.sh > should apply HBASE-18255 to HBASE_MASTER_OPTS too > - > > Key: HBASE-18479 > URL: https://issues.apache.org/jira/browse/HBASE-18479 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.1, 1.2.6, 1.1.11 >Reporter: Yechao Chen >Assignee: Yechao Chen >Priority: Critical > Fix For: 1.4.0, 1.3.2, 1.5.0, 1.2.7 > > Attachments: HBASE-18479-branch-1.v1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18255) Time-Delayed HBase Performance Degradation with Java 7
[ https://issues.apache.org/jira/browse/HBASE-18255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122877#comment-16122877 ] Hudson commented on HBASE-18255: SUCCESS: Integrated in Jenkins build HBase-1.2-JDK8 #185 (See [https://builds.apache.org/job/HBase-1.2-JDK8/185/]) HBASE-18479 should apply HBASE-18255 to HBASE_MASTER_OPTS too (tedyu: rev 9f809748d72b39489f202a56aba107cf7386f30c) * (edit) conf/hbase-env.cmd * (edit) conf/hbase-env.sh > Time-Delayed HBase Performance Degradation with Java 7 > -- > > Key: HBASE-18255 > URL: https://issues.apache.org/jira/browse/HBASE-18255 > Project: HBase > Issue Type: Bug > Components: Performance >Affects Versions: 1.3.1, 1.2.6, 1.1.11 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Critical > Labels: jdk7 > Fix For: 1.4.0, 1.3.2, 1.2.7, 1.1.12 > > Attachments: HBASE-18255-branch-1.x.v1.patch > > > The good summary of the issue and provided resolution can be found in this > article: > https://community.hortonworks.com/articles/105802/time-delayed-hbase-performance-degradation-with-ja.html > In a few words, due to internal JVM 7 bug (which has been addressed only in > Java 8), HotSpot code cache can become full and after that ALL JIT > compilations get suspended indefinitely. The default value for code cache > size in JVM 7 is quite low: 48MB. It is recommended to increase this value at > least to 256MB (default in JVM 8). > This BUG affects only 1.x -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18387) [Thrift] Make principal configurable in DemoClient.java
[
https://issues.apache.org/jira/browse/HBASE-18387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122874#comment-16122874
]
Hudson commented on HBASE-18387:
FAILURE: Integrated in Jenkins build HBase-1.1-JDK7 #1899 (See
[https://builds.apache.org/job/HBase-1.1-JDK7/1899/])
HBASE-18387: [Thrift] Make principal configurable in DemoClient.java (elserj:
rev ebc1d2bf2c52bbaf5e2bc81afab6298558978239)
* (edit) hbase-examples/README.txt
* (edit)
hbase-examples/src/main/java/org/apache/hadoop/hbase/thrift/DemoClient.java
> [Thrift] Make principal configurable in DemoClient.java
> ---
>
> Key: HBASE-18387
> URL: https://issues.apache.org/jira/browse/HBASE-18387
> Project: HBase
> Issue Type: Improvement
>Reporter: Lars George
>Assignee: Tamas Penzes
>Priority: Minor
> Labels: beginner
> Fix For: 2.0.0, 1.4.0, 1.3.2, 1.2.7, 1.1.12
>
> Attachments: HBASE-18387.master.001.patch,
> HBASE-18387.master.002.patch, HBASE-18387.master.003.patch
>
>
> In the Thrift1 demo client we have this code:
> {code}
> transport = new TSaslClientTransport("GSSAPI", null,
> "hbase", // Thrift server user name, should be an authorized proxy user.
> host, // Thrift server domain
> saslProperties, null, transport);
> {code}
> This will only work when the Thrift server is started with the {{hbase}}
> principal. Often this may deviate, for example I am using {{hbase-thrift}} to
> separate the names from those of backend servers.
> What we need is either an additional command line option to specify the name,
> or a property that can be set with -D and can be passed at runtime. I prefer
> the former, as the latter is making this a little convoluted.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-15511) ClusterStatus should be able to return responses by scope
[ https://issues.apache.org/jira/browse/HBASE-15511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan updated HBASE-15511: -- Status: Patch Available (was: Open) > ClusterStatus should be able to return responses by scope > - > > Key: HBASE-15511 > URL: https://issues.apache.org/jira/browse/HBASE-15511 > Project: HBase > Issue Type: Improvement >Reporter: Esteban Gutierrez >Assignee: Reid Chan > Fix For: 2.0.0 > > Attachments: HBASE-15511.master.001.patch, > HBASE-15511.master.002.patch, HBASE-15511.master.003.patch, > HBASE-15511.master.004.patch, HBASE-15511.master.005.patch, > HBASE-15511.master.006.patch, HBASE-15511.master.007.patch, > HBASE-15511.master.008.patch, HBASE-15511.master.009.patch, > HBASE-15511.master.010.patch, HBASE-15511.master.011.patch, > HBASE-15511.master.012.patch, HBASE-15511.master.013.patch > > > The current ClusterStatus response returns too much information about the > load per region and replication cluster wide. Sometimes that response can be > quite large (10s or 100s of MBs) and methods like getServerSize() or > getRegionsCount() don't really need the full response. One possibility is to > provide a scope (or filter) for the ClusterStatus requests to limit the > response back to the client. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-15511) ClusterStatus should be able to return responses by scope
[ https://issues.apache.org/jira/browse/HBASE-15511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan updated HBASE-15511: -- Attachment: HBASE-15511.master.013.patch set optional fields in Options default value "true" > ClusterStatus should be able to return responses by scope > - > > Key: HBASE-15511 > URL: https://issues.apache.org/jira/browse/HBASE-15511 > Project: HBase > Issue Type: Improvement >Reporter: Esteban Gutierrez >Assignee: Reid Chan > Fix For: 2.0.0 > > Attachments: HBASE-15511.master.001.patch, > HBASE-15511.master.002.patch, HBASE-15511.master.003.patch, > HBASE-15511.master.004.patch, HBASE-15511.master.005.patch, > HBASE-15511.master.006.patch, HBASE-15511.master.007.patch, > HBASE-15511.master.008.patch, HBASE-15511.master.009.patch, > HBASE-15511.master.010.patch, HBASE-15511.master.011.patch, > HBASE-15511.master.012.patch, HBASE-15511.master.013.patch > > > The current ClusterStatus response returns too much information about the > load per region and replication cluster wide. Sometimes that response can be > quite large (10s or 100s of MBs) and methods like getServerSize() or > getRegionsCount() don't really need the full response. One possibility is to > provide a scope (or filter) for the ClusterStatus requests to limit the > response back to the client. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-15511) ClusterStatus should be able to return responses by scope
[ https://issues.apache.org/jira/browse/HBASE-15511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan updated HBASE-15511: -- Status: Open (was: Patch Available) > ClusterStatus should be able to return responses by scope > - > > Key: HBASE-15511 > URL: https://issues.apache.org/jira/browse/HBASE-15511 > Project: HBase > Issue Type: Improvement >Reporter: Esteban Gutierrez >Assignee: Reid Chan > Fix For: 2.0.0 > > Attachments: HBASE-15511.master.001.patch, > HBASE-15511.master.002.patch, HBASE-15511.master.003.patch, > HBASE-15511.master.004.patch, HBASE-15511.master.005.patch, > HBASE-15511.master.006.patch, HBASE-15511.master.007.patch, > HBASE-15511.master.008.patch, HBASE-15511.master.009.patch, > HBASE-15511.master.010.patch, HBASE-15511.master.011.patch, > HBASE-15511.master.012.patch > > > The current ClusterStatus response returns too much information about the > load per region and replication cluster wide. Sometimes that response can be > quite large (10s or 100s of MBs) and methods like getServerSize() or > getRegionsCount() don't really need the full response. One possibility is to > provide a scope (or filter) for the ClusterStatus requests to limit the > response back to the client. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (HBASE-17797) Add a filter to implement the function which return the special number of versions of each column
[ https://issues.apache.org/jira/browse/HBASE-17797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang resolved HBASE-17797. Resolution: Duplicate As we solved this in HBASE-17125, so resolve this as duplicate. > Add a filter to implement the function which return the special number of > versions of each column > - > > Key: HBASE-17797 > URL: https://issues.apache.org/jira/browse/HBASE-17797 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > > After HBASE-17125, ScanQueryMatch will first check column then check by > filter. The scan/get will get consistent result when use filter to read data. > But scan/get setMaxVersions() can not return the special number of versions > of each column. So this issue will introduce a new filter to implement this > function which return the special number of versions of each column. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17125) Inconsistent result when use filter to read data
[
https://issues.apache.org/jira/browse/HBASE-17125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122841#comment-16122841
]
Guanghao Zhang commented on HBASE-17125:
Pushed to master and branch-2.
> Inconsistent result when use filter to read data
>
>
> Key: HBASE-17125
> URL: https://issues.apache.org/jira/browse/HBASE-17125
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 2.0.0, 3.0.0
>
> Attachments: 17125-slack-13.txt, example.diff,
> HBASE-17125.master.001.patch, HBASE-17125.master.002.patch,
> HBASE-17125.master.002.patch, HBASE-17125.master.003.patch,
> HBASE-17125.master.004.patch, HBASE-17125.master.005.patch,
> HBASE-17125.master.006.patch, HBASE-17125.master.007.patch,
> HBASE-17125.master.008.patch, HBASE-17125.master.009.patch,
> HBASE-17125.master.009.patch, HBASE-17125.master.010.patch,
> HBASE-17125.master.011.patch, HBASE-17125.master.011.patch,
> HBASE-17125.master.012.patch, HBASE-17125.master.013.patch,
> HBASE-17125.master.014.patch, HBASE-17125.master.015.patch,
> HBASE-17125.master.016.patch, HBASE-17125.master.017.patch,
> HBASE-17125.master.018.patch, HBASE-17125.master.019.patch,
> HBASE-17125.master.020.patch, HBASE-17125.master.020.patch,
> HBASE-17125.master.021.patch, HBASE-17125.master.022.patch,
> HBASE-17125.master.checkReturnedVersions.patch,
> HBASE-17125.master.no-specified-filter.patch
>
>
> Assume a cloumn's max versions is 3, then we write 4 versions of this column.
> The oldest version doesn't remove immediately. But from the user view, the
> oldest version has gone. When user use a filter to query, if the filter skip
> a new version, then the oldest version will be seen again. But after compact
> the region, then the oldest version will never been seen. So it is weird for
> user. The query will get inconsistent result before and after region
> compaction.
> The reason is matchColumn method of UserScanQueryMatcher. It first check the
> cell by filter, then check the number of versions needed. So if the filter
> skip the new version, then the oldest version will be seen again when it is
> not removed.
> Have a discussion offline with [~Apache9] and [~fenghh], now we have two
> solution for this problem. The first idea is check the number of versions
> first, then check the cell by filter. As the comment of setFilter, the filter
> is called after all tests for ttl, column match, deletes and max versions
> have been run.
> {code}
> /**
>* Apply the specified server-side filter when performing the Query.
>* Only {@link Filter#filterKeyValue(Cell)} is called AFTER all tests
>* for ttl, column match, deletes and max versions have been run.
>* @param filter filter to run on the server
>* @return this for invocation chaining
>*/
> public Query setFilter(Filter filter) {
> this.filter = filter;
> return this;
> }
> {code}
> But this idea has another problem, if a column's max version is 5 and the
> user query only need 3 versions. It first check the version's number, then
> check the cell by filter. So the cells number of the result may less than 3.
> But there are 2 versions which don't read anymore.
> So the second idea has three steps.
> 1. check by the max versions of this column
> 2. check the kv by filter
> 3. check the versions which user need.
> But this will lead the ScanQueryMatcher more complicated. And this will break
> the javadoc of Query.setFilter.
> Now we don't have a final solution for this problem. Suggestions are welcomed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18368) FilterList with multiple FamilyFilters concatenated by OR does not work.
[
https://issues.apache.org/jira/browse/HBASE-18368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122839#comment-16122839
]
Guanghao Zhang commented on HBASE-18368:
Even the filter return a NEXT_ROW, it still may get a cell from the same row's
different family. You can't guarantee NEXT_ROW so why you provide it to user? I
thought we can tell user, if your filter need NEXT_ROW state, one method is use
the reset method to know the row changed. Another method is use
CellUtil.matchingRows to know the row changed.
> FilterList with multiple FamilyFilters concatenated by OR does not work.
>
>
> Key: HBASE-18368
> URL: https://issues.apache.org/jira/browse/HBASE-18368
> Project: HBase
> Issue Type: Sub-task
> Components: Filters
>Affects Versions: 3.0.0, 2.0.0-alpha-1
>Reporter: Peter Somogyi
>Assignee: Zheng Hu
>Priority: Critical
> Attachments: HBASE-18368.branch-1.patch,
> HBASE-18368.branch-1.v2.patch, HBASE-18368.branch-1.v3.patch,
> HBASE-18368.patch, HBASE-18368.v2.patch
>
>
> Scan gives back incomplete list if multiple filters are combined with OR /
> MUST_PASS_ONE.
> Using 2 FamilyFilters in a FilterList using MUST_PASS_ONE operator will give
> back results for only the first Filter.
> {code:java|title=Test code}
> @Test
> public void testFiltersWithOr() throws Exception {
> TableName tn = TableName.valueOf("MyTest");
> Table table = utility.createTable(tn, new String[] {"cf1", "cf2"});
> byte[] CF1 = Bytes.toBytes("cf1");
> byte[] CF2 = Bytes.toBytes("cf2");
> Put put1 = new Put(Bytes.toBytes("0"));
> put1.addColumn(CF1, Bytes.toBytes("col_a"), Bytes.toBytes(0));
> table.put(put1);
> Put put2 = new Put(Bytes.toBytes("0"));
> put2.addColumn(CF2, Bytes.toBytes("col_b"), Bytes.toBytes(0));
> table.put(put2);
> FamilyFilter filterCF1 = new FamilyFilter(CompareFilter.CompareOp.EQUAL,
> new BinaryComparator(CF1));
> FamilyFilter filterCF2 = new FamilyFilter(CompareFilter.CompareOp.EQUAL,
> new BinaryComparator(CF2));
> FilterList filterList = new FilterList(FilterList.Operator.MUST_PASS_ONE);
> filterList.addFilter(filterCF1);
> filterList.addFilter(filterCF2);
> Scan scan = new Scan();
> scan.setFilter(filterList);
> ResultScanner scanner = table.getScanner(scan);
> System.out.println(filterList);
> for (Result rr = scanner.next(); rr != null; rr = scanner.next()) {
> System.out.println(rr);
> }
> }
> {code}
> {noformat:title=Output}
> FilterList OR (2/2): [FamilyFilter (EQUAL, cf1), FamilyFilter (EQUAL, cf2)]
> keyvalues={0/cf1:col_a/1499852754957/Put/vlen=4/seqid=0}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (HBASE-18567) VerifyReplication does not show any mistmatch if source table has no data
Vishal Khandelwal created HBASE-18567: - Summary: VerifyReplication does not show any mistmatch if source table has no data Key: HBASE-18567 URL: https://issues.apache.org/jira/browse/HBASE-18567 Project: HBase Issue Type: Bug Reporter: Vishal Khandelwal Assuming situation when source table does not have any data where as DR has data. In such case if we run Verify Replication from pr it says no mismatch whereas if you run from DR, it as X ROWS only in Primary where as if you atleast 1 row in primary it would show correct behvaiour like 1 GOOD and X ONLY IN PEER. This give in consistent results and also veirifyreplication is no more idempotent. This behavior is true for search with filter, timerange etc. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18262) name of parameter quote need update in hbase-default.xml
[ https://issues.apache.org/jira/browse/HBASE-18262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122821#comment-16122821 ] Hudson commented on HBASE-18262: SUCCESS: Integrated in Jenkins build HBase-2.0 #307 (See [https://builds.apache.org/job/HBase-2.0/307/]) HBASE-18262 name of parameter quote need update (stack: rev 8da096b231b236e80e60412d5bef08ded0ab2a1e) * (edit) hbase-common/src/main/resources/hbase-default.xml > name of parameter quote need update in hbase-default.xml > > > Key: HBASE-18262 > URL: https://issues.apache.org/jira/browse/HBASE-18262 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Dongtao Zhang >Assignee: Dongtao Zhang >Priority: Minor > Fix For: 2.0.0-alpha-2 > > Attachments: HBASE-18262-v001.patch > > > In description of parameter "hbase.zookeeper.quorum", old name > "hbase.zookeeper.clientPort " should be replaced to > "hbase.zookeeper.property.clientPort". -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18024) HRegion#initializeRegionInternals should not re-create .hregioninfo file when the region directory no longer exists
[
https://issues.apache.org/jira/browse/HBASE-18024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122823#comment-16122823
]
Hudson commented on HBASE-18024:
SUCCESS: Integrated in Jenkins build HBase-2.0 #307 (See
[https://builds.apache.org/job/HBase-2.0/307/])
HBASE-18024 HRegion#initializeRegionInternals should not re-create (esteban:
rev b3e7e31dee77afa0053c798bb45c105323930774)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileRefresherChore.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionOpen.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestWALMonotonicallyIncreasingSeqId.java
> HRegion#initializeRegionInternals should not re-create .hregioninfo file when
> the region directory no longer exists
> ---
>
> Key: HBASE-18024
> URL: https://issues.apache.org/jira/browse/HBASE-18024
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment, regionserver
>Affects Versions: 2.0.0, 1.4.0, 1.3.1, 1.2.5
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
> Fix For: 2.0.0, 1.4.0, 1.3.2, 1.5.0, 1.2.7
>
> Attachments: HBASE-18024.001.patch, HBASE-18024.002.patch,
> HBASE-18024.003.patch, HBASE-18024.004.patch
>
>
> When a RegionSever attempts to open a region, during initialization the RS
> tries to open the {{/data///.hregioninfo}}
> file, however if the {{.hregioninfofile}} doesn't exist, the RegionServer
> will create a new one on {{HRegionFileSystem#checkRegionInfoOnFilesystem}}. A
> side effect of that tools like hbck will incorrectly assume an inconsistency
> due the presence of this new {{.hregioninfofile}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18560) Test master.assignment.TestAssignmentManager hangs on master and its in flaky list
[ https://issues.apache.org/jira/browse/HBASE-18560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122822#comment-16122822 ] Hudson commented on HBASE-18560: SUCCESS: Integrated in Jenkins build HBase-2.0 #307 (See [https://builds.apache.org/job/HBase-2.0/307/]) HBASE-18560 Fixed master.assignment.TestAssignmentManager hangs on (stack: rev ad266a4b669d2276f64e638ebc21ca2c70e69781) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockNoopMasterServices.java > Test master.assignment.TestAssignmentManager hangs on master and its in flaky > list > -- > > Key: HBASE-18560 > URL: https://issues.apache.org/jira/browse/HBASE-18560 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Umesh Agashe >Assignee: Umesh Agashe > Fix For: 2.0.0 > > Attachments: hbase-18560.master.001.patch > > > Test master.assignment.TestAssignmentManager hangs on master and showing up > in flaky list -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18551) [AMv2] UnassignProcedure and crashed regionservers
[
https://issues.apache.org/jira/browse/HBASE-18551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122815#comment-16122815
]
Hadoop QA commented on HBASE-18551:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 18m
12s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
33s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
34m 14s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
9s{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}154m 54s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}231m 43s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestSplitTransactionOnCluster |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.13.1 Server=1.13.1 Image:yetus/hbase:bdc94b1 |
| JIRA Issue | HBASE-18551 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881361/HBASE-18551.master.003.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 785f0f6d07a1 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / efd211d |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC3 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/8026/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/8026/testReport/ |
| modules | C: hbase-procedure hbase-server U: . |
| Console
[jira] [Commented] (HBASE-18387) [Thrift] Make principal configurable in DemoClient.java
[
https://issues.apache.org/jira/browse/HBASE-18387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122816#comment-16122816
]
Hudson commented on HBASE-18387:
SUCCESS: Integrated in Jenkins build HBase-1.2-IT #926 (See
[https://builds.apache.org/job/HBase-1.2-IT/926/])
HBASE-18387: [Thrift] Make principal configurable in DemoClient.java (elserj:
rev a732b67ce8c7c08f867807ce7f1b0b47352a7d5e)
* (edit)
hbase-examples/src/main/java/org/apache/hadoop/hbase/thrift/DemoClient.java
* (edit) hbase-examples/README.txt
> [Thrift] Make principal configurable in DemoClient.java
> ---
>
> Key: HBASE-18387
> URL: https://issues.apache.org/jira/browse/HBASE-18387
> Project: HBase
> Issue Type: Improvement
>Reporter: Lars George
>Assignee: Tamas Penzes
>Priority: Minor
> Labels: beginner
> Fix For: 2.0.0, 1.4.0, 1.3.2, 1.2.7, 1.1.12
>
> Attachments: HBASE-18387.master.001.patch,
> HBASE-18387.master.002.patch, HBASE-18387.master.003.patch
>
>
> In the Thrift1 demo client we have this code:
> {code}
> transport = new TSaslClientTransport("GSSAPI", null,
> "hbase", // Thrift server user name, should be an authorized proxy user.
> host, // Thrift server domain
> saslProperties, null, transport);
> {code}
> This will only work when the Thrift server is started with the {{hbase}}
> principal. Often this may deviate, for example I am using {{hbase-thrift}} to
> separate the names from those of backend servers.
> What we need is either an additional command line option to specify the name,
> or a property that can be set with -D and can be passed at runtime. I prefer
> the former, as the latter is making this a little convoluted.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18548) Move sources of important Jenkins jobs into source control
[ https://issues.apache.org/jira/browse/HBASE-18548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-18548: Issue Type: Improvement (was: Bug) > Move sources of important Jenkins jobs into source control > -- > > Key: HBASE-18548 > URL: https://issues.apache.org/jira/browse/HBASE-18548 > Project: HBase > Issue Type: Improvement > Components: documentation, scripts >Affects Versions: 2.0.0-alpha-1 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones >Priority: Critical > Fix For: 3.0.0 > > Attachments: HBASE-18548.patch > > > Move sources of the following Jenkins jobs into source control: > https://builds.apache.org/job/hbase_generate_website/ > https://builds.apache.org/view/All/job/HBase%20Website%20Link%20Checker/ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18387) [Thrift] Make principal configurable in DemoClient.java
[
https://issues.apache.org/jira/browse/HBASE-18387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122807#comment-16122807
]
Hudson commented on HBASE-18387:
SUCCESS: Integrated in Jenkins build HBase-1.3-IT #165 (See
[https://builds.apache.org/job/HBase-1.3-IT/165/])
HBASE-18387: [Thrift] Make principal configurable in DemoClient.java (elserj:
rev a2a28a7806549b3dd337163432ad12d944add634)
* (edit) hbase-examples/README.txt
* (edit)
hbase-examples/src/main/java/org/apache/hadoop/hbase/thrift/DemoClient.java
> [Thrift] Make principal configurable in DemoClient.java
> ---
>
> Key: HBASE-18387
> URL: https://issues.apache.org/jira/browse/HBASE-18387
> Project: HBase
> Issue Type: Improvement
>Reporter: Lars George
>Assignee: Tamas Penzes
>Priority: Minor
> Labels: beginner
> Fix For: 2.0.0, 1.4.0, 1.3.2, 1.2.7, 1.1.12
>
> Attachments: HBASE-18387.master.001.patch,
> HBASE-18387.master.002.patch, HBASE-18387.master.003.patch
>
>
> In the Thrift1 demo client we have this code:
> {code}
> transport = new TSaslClientTransport("GSSAPI", null,
> "hbase", // Thrift server user name, should be an authorized proxy user.
> host, // Thrift server domain
> saslProperties, null, transport);
> {code}
> This will only work when the Thrift server is started with the {{hbase}}
> principal. Often this may deviate, for example I am using {{hbase-thrift}} to
> separate the names from those of backend servers.
> What we need is either an additional command line option to specify the name,
> or a property that can be set with -D and can be passed at runtime. I prefer
> the former, as the latter is making this a little convoluted.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18548) Move sources of important Jenkins jobs into source control
[ https://issues.apache.org/jira/browse/HBASE-18548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-18548: Fix Version/s: (was: 2.0.0) 3.0.0 > Move sources of important Jenkins jobs into source control > -- > > Key: HBASE-18548 > URL: https://issues.apache.org/jira/browse/HBASE-18548 > Project: HBase > Issue Type: Bug > Components: documentation, scripts >Affects Versions: 2.0.0-alpha-1 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones >Priority: Critical > Fix For: 3.0.0 > > Attachments: HBASE-18548.patch > > > Move sources of the following Jenkins jobs into source control: > https://builds.apache.org/job/hbase_generate_website/ > https://builds.apache.org/view/All/job/HBase%20Website%20Link%20Checker/ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18548) Move sources of important Jenkins jobs into source control
[ https://issues.apache.org/jira/browse/HBASE-18548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122804#comment-16122804 ] Sean Busbey commented on HBASE-18548: - Ah, I see someone already filed HBASE-18563 for the license thing, nevermind. > Move sources of important Jenkins jobs into source control > -- > > Key: HBASE-18548 > URL: https://issues.apache.org/jira/browse/HBASE-18548 > Project: HBase > Issue Type: Bug > Components: documentation, scripts >Affects Versions: 2.0.0-alpha-1 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-18548.patch > > > Move sources of the following Jenkins jobs into source control: > https://builds.apache.org/job/hbase_generate_website/ > https://builds.apache.org/view/All/job/HBase%20Website%20Link%20Checker/ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18548) Move sources of important Jenkins jobs into source control
[ https://issues.apache.org/jira/browse/HBASE-18548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122802#comment-16122802 ] Sean Busbey commented on HBASE-18548: - the asf license header and shellcheck things flagged by precommit need to get cleaned up still. [~misty] you prefer an addendum or a new JIRA? > Move sources of important Jenkins jobs into source control > -- > > Key: HBASE-18548 > URL: https://issues.apache.org/jira/browse/HBASE-18548 > Project: HBase > Issue Type: Bug > Components: documentation, scripts >Affects Versions: 2.0.0-alpha-1 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-18548.patch > > > Move sources of the following Jenkins jobs into source control: > https://builds.apache.org/job/hbase_generate_website/ > https://builds.apache.org/view/All/job/HBase%20Website%20Link%20Checker/ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18387) [Thrift] Make principal configurable in DemoClient.java
[
https://issues.apache.org/jira/browse/HBASE-18387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122801#comment-16122801
]
Hudson commented on HBASE-18387:
FAILURE: Integrated in Jenkins build HBase-1.3-JDK7 #229 (See
[https://builds.apache.org/job/HBase-1.3-JDK7/229/])
HBASE-18387: [Thrift] Make principal configurable in DemoClient.java (elserj:
rev a2a28a7806549b3dd337163432ad12d944add634)
* (edit)
hbase-examples/src/main/java/org/apache/hadoop/hbase/thrift/DemoClient.java
* (edit) hbase-examples/README.txt
> [Thrift] Make principal configurable in DemoClient.java
> ---
>
> Key: HBASE-18387
> URL: https://issues.apache.org/jira/browse/HBASE-18387
> Project: HBase
> Issue Type: Improvement
>Reporter: Lars George
>Assignee: Tamas Penzes
>Priority: Minor
> Labels: beginner
> Fix For: 2.0.0, 1.4.0, 1.3.2, 1.2.7, 1.1.12
>
> Attachments: HBASE-18387.master.001.patch,
> HBASE-18387.master.002.patch, HBASE-18387.master.003.patch
>
>
> In the Thrift1 demo client we have this code:
> {code}
> transport = new TSaslClientTransport("GSSAPI", null,
> "hbase", // Thrift server user name, should be an authorized proxy user.
> host, // Thrift server domain
> saslProperties, null, transport);
> {code}
> This will only work when the Thrift server is started with the {{hbase}}
> principal. Often this may deviate, for example I am using {{hbase-thrift}} to
> separate the names from those of backend servers.
> What we need is either an additional command line option to specify the name,
> or a property that can be set with -D and can be passed at runtime. I prefer
> the former, as the latter is making this a little convoluted.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18387) [Thrift] Make principal configurable in DemoClient.java
[
https://issues.apache.org/jira/browse/HBASE-18387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122800#comment-16122800
]
Hudson commented on HBASE-18387:
FAILURE: Integrated in Jenkins build HBase-1.2-JDK7 #191 (See
[https://builds.apache.org/job/HBase-1.2-JDK7/191/])
HBASE-18387: [Thrift] Make principal configurable in DemoClient.java (elserj:
rev a732b67ce8c7c08f867807ce7f1b0b47352a7d5e)
* (edit)
hbase-examples/src/main/java/org/apache/hadoop/hbase/thrift/DemoClient.java
* (edit) hbase-examples/README.txt
> [Thrift] Make principal configurable in DemoClient.java
> ---
>
> Key: HBASE-18387
> URL: https://issues.apache.org/jira/browse/HBASE-18387
> Project: HBase
> Issue Type: Improvement
>Reporter: Lars George
>Assignee: Tamas Penzes
>Priority: Minor
> Labels: beginner
> Fix For: 2.0.0, 1.4.0, 1.3.2, 1.2.7, 1.1.12
>
> Attachments: HBASE-18387.master.001.patch,
> HBASE-18387.master.002.patch, HBASE-18387.master.003.patch
>
>
> In the Thrift1 demo client we have this code:
> {code}
> transport = new TSaslClientTransport("GSSAPI", null,
> "hbase", // Thrift server user name, should be an authorized proxy user.
> host, // Thrift server domain
> saslProperties, null, transport);
> {code}
> This will only work when the Thrift server is started with the {{hbase}}
> principal. Often this may deviate, for example I am using {{hbase-thrift}} to
> separate the names from those of backend servers.
> What we need is either an additional command line option to specify the name,
> or a property that can be set with -D and can be passed at runtime. I prefer
> the former, as the latter is making this a little convoluted.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18368) FilterList with multiple FamilyFilters concatenated by OR does not work.
[
https://issues.apache.org/jira/browse/HBASE-18368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122797#comment-16122797
]
Zheng Hu commented on HBASE-18368:
--
bq. How about we rename NEXT_ROW to NEXT_FAMILY directly?
I would like to separate them into NEXT_ROW & NEXT_FAMILY.for
ScanQueryMatcher , they are the same logic, but for filter, their meanings are
different, the former means skip to the next row, and the later means skip to
the next family.besides, for FilterList ,their logic are different
now.
> FilterList with multiple FamilyFilters concatenated by OR does not work.
>
>
> Key: HBASE-18368
> URL: https://issues.apache.org/jira/browse/HBASE-18368
> Project: HBase
> Issue Type: Sub-task
> Components: Filters
>Affects Versions: 3.0.0, 2.0.0-alpha-1
>Reporter: Peter Somogyi
>Assignee: Zheng Hu
>Priority: Critical
> Attachments: HBASE-18368.branch-1.patch,
> HBASE-18368.branch-1.v2.patch, HBASE-18368.branch-1.v3.patch,
> HBASE-18368.patch, HBASE-18368.v2.patch
>
>
> Scan gives back incomplete list if multiple filters are combined with OR /
> MUST_PASS_ONE.
> Using 2 FamilyFilters in a FilterList using MUST_PASS_ONE operator will give
> back results for only the first Filter.
> {code:java|title=Test code}
> @Test
> public void testFiltersWithOr() throws Exception {
> TableName tn = TableName.valueOf("MyTest");
> Table table = utility.createTable(tn, new String[] {"cf1", "cf2"});
> byte[] CF1 = Bytes.toBytes("cf1");
> byte[] CF2 = Bytes.toBytes("cf2");
> Put put1 = new Put(Bytes.toBytes("0"));
> put1.addColumn(CF1, Bytes.toBytes("col_a"), Bytes.toBytes(0));
> table.put(put1);
> Put put2 = new Put(Bytes.toBytes("0"));
> put2.addColumn(CF2, Bytes.toBytes("col_b"), Bytes.toBytes(0));
> table.put(put2);
> FamilyFilter filterCF1 = new FamilyFilter(CompareFilter.CompareOp.EQUAL,
> new BinaryComparator(CF1));
> FamilyFilter filterCF2 = new FamilyFilter(CompareFilter.CompareOp.EQUAL,
> new BinaryComparator(CF2));
> FilterList filterList = new FilterList(FilterList.Operator.MUST_PASS_ONE);
> filterList.addFilter(filterCF1);
> filterList.addFilter(filterCF2);
> Scan scan = new Scan();
> scan.setFilter(filterList);
> ResultScanner scanner = table.getScanner(scan);
> System.out.println(filterList);
> for (Result rr = scanner.next(); rr != null; rr = scanner.next()) {
> System.out.println(rr);
> }
> }
> {code}
> {noformat:title=Output}
> FilterList OR (2/2): [FamilyFilter (EQUAL, cf1), FamilyFilter (EQUAL, cf2)]
> keyvalues={0/cf1:col_a/1499852754957/Put/vlen=4/seqid=0}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-18500:
---
Attachment: HBASE-18500-v6.patch
Add a TODO(Copied the comments from [~anoop.hbase]) in v6.
> Performance issue: Don't use BufferedMutator for HTable's put method
>
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
> Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch,
> HBASE-18500-v3.patch, HBASE-18500-v4.patch, HBASE-18500-v5.patch,
> HBASE-18500-v5.patch, HBASE-18500-v5.patch, HBASE-18500-v6.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the
> almost same latency with the blocking write. But for master branch, the
> result shows the async write has better latency than the blocking client.
> Take a look about the code, I thought the difference is the BufferedMutator.
> For master branch, HTable don't have a write buffer and all write request
> will be flushed directly. And user can use BufferedMutator when user want to
> perform client-side buffering of writes. For the performance issue
> (autoFlush=True), I thought we can use rpc caller directly in HTable's put
> method. Thanks.
> Review: https://reviews.apache.org/r/61454/
> Copy the comments from [~chia7712]. Remove the BufferdMutator brings four
> benefits.
> 1. correct the metrics (see HBASE-18476)
> 2. make HTable thread-safe (see HBASE-17368)
> 3. reduce the latency
> 4. get rid of some deprecated methods in Table
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18387) [Thrift] Make principal configurable in DemoClient.java
[
https://issues.apache.org/jira/browse/HBASE-18387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser updated HBASE-18387:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Release Note: This change allows the demonstration Thrift client to
customize the server principal used by the Thrift server for instances secured
with Kerberos.
Status: Resolved (was: Patch Available)
Thanks for the patch, [~tamaas]!
> [Thrift] Make principal configurable in DemoClient.java
> ---
>
> Key: HBASE-18387
> URL: https://issues.apache.org/jira/browse/HBASE-18387
> Project: HBase
> Issue Type: Improvement
>Reporter: Lars George
>Assignee: Tamas Penzes
>Priority: Minor
> Labels: beginner
> Fix For: 2.0.0, 1.4.0, 1.3.2, 1.2.7, 1.1.12
>
> Attachments: HBASE-18387.master.001.patch,
> HBASE-18387.master.002.patch, HBASE-18387.master.003.patch
>
>
> In the Thrift1 demo client we have this code:
> {code}
> transport = new TSaslClientTransport("GSSAPI", null,
> "hbase", // Thrift server user name, should be an authorized proxy user.
> host, // Thrift server domain
> saslProperties, null, transport);
> {code}
> This will only work when the Thrift server is started with the {{hbase}}
> principal. Often this may deviate, for example I am using {{hbase-thrift}} to
> separate the names from those of backend servers.
> What we need is either an additional command line option to specify the name,
> or a property that can be set with -D and can be passed at runtime. I prefer
> the former, as the latter is making this a little convoluted.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (HBASE-18390) Sleep too long when finding region location failed
[
https://issues.apache.org/jira/browse/HBASE-18390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang resolved HBASE-18390.
---
Resolution: Fixed
I tried reverted to this commit
{quote}
commit 4bd5f03d22be5f8ce09c1c67cfe5d1dcc603446a
Author: Chia-Ping Tsai
Date: Thu Jul 13 19:37:15 2017 +0800
HBASE-18365 Eliminate the findbugs warnings for hbase-common
{quote}
The tests are still failing for me. So I do no think it is introduced by this
issue.
And the actual problem is ToolProvider returns a null JavaCompiler. It seems
that tools.jar is not on our classpath. If I manually add tools.jar then the
tests work fine.
[~tedyu] You can open a new issue to address this.
> Sleep too long when finding region location failed
> --
>
> Key: HBASE-18390
> URL: https://issues.apache.org/jira/browse/HBASE-18390
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.1, 1.2.6, 1.1.11, 2.0.0-alpha-1
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 3.0.0, 1.4.0, 1.3.2, 1.5.0, 1.2.7, 2.0.0-alpha-2, 1.1.12
>
> Attachments: HBASE-18390.v01.patch, HBASE-18390.v02.patch,
> HBASE-18390.v03.patch
>
>
> If RegionServerCallable#prepare failed when getRegionLocation, the location
> in this callable object is null. And before we retry we will sleep. However,
> when location is null we will sleep at least 10 seconds. And the request will
> be failed directly if operation timeout is less than 10 seconds. I think it
> is no need to keep MIN_WAIT_DEAD_SERVER logic. Use backoff sleeping logic is
> ok for most cases.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-18500:
---
Description:
Copied the test result from HBASE-17994.
Run start-hbase.sh in my local computer and use the default config to test with
PE tool.
{code}
./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10
--nomapred --autoFlush=True randomWrite 1
./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10
--nomapred --autoFlush=True asyncRandomWrite 1
{code}
Mean latency test result.
|| || Test1 || Test2 || Test3 || Test4 || Test5 ||
| randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
| asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
50th latency test result.
|| || Test1 || Test2 || Test3 || Test4 || Test5 ||
| randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
| asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
99th latency test result.
|| || Test1 || Test2 || Test3 || Test4 || Test5 ||
| randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
| asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
In our internal 0.98 branch, the PE test result shows the async write has the
almost same latency with the blocking write. But for master branch, the result
shows the async write has better latency than the blocking client. Take a look
about the code, I thought the difference is the BufferedMutator. For master
branch, HTable don't have a write buffer and all write request will be flushed
directly. And user can use BufferedMutator when user want to perform
client-side buffering of writes. For the performance issue (autoFlush=True), I
thought we can use rpc caller directly in HTable's put method. Thanks.
Review: https://reviews.apache.org/r/61454/
Copy the comments from [~chia7712]. Remove the BufferdMutator brings four
benefits.
1. correct the metrics (see HBASE-18476)
2. make HTable thread-safe (see HBASE-17368)
3. reduce the latency
4. get rid of some deprecated methods in Table
was:
Copied the test result from HBASE-17994.
Run start-hbase.sh in my local computer and use the default config to test with
PE tool.
{code}
./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10
--nomapred --autoFlush=True randomWrite 1
./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10
--nomapred --autoFlush=True asyncRandomWrite 1
{code}
Mean latency test result.
|| || Test1 || Test2 || Test3 || Test4 || Test5 ||
| randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
| asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
50th latency test result.
|| || Test1 || Test2 || Test3 || Test4 || Test5 ||
| randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
| asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
99th latency test result.
|| || Test1 || Test2 || Test3 || Test4 || Test5 ||
| randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
| asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
In our internal 0.98 branch, the PE test result shows the async write has the
almost same latency with the blocking write. But for master branch, the result
shows the async write has better latency than the blocking client. Take a look
about the code, I thought the difference is the BufferedMutator. For master
branch, HTable don't have a write buffer and all write request will be flushed
directly. And user can use BufferedMutator when user want to perform
client-side buffering of writes. For the performance issue (autoFlush=True), I
thought we can use rpc caller directly in HTable's put method. Thanks.
Review: https://reviews.apache.org/r/61454/
Copy the comments from [~chia7712]. Remove the BufferdMutator brings some
1. correct the metrics (see HBASE-18476)
2. make HTable thread-safe (see HBASE-17368)
3. reduce the latency
4. get rid of some deprecated methods in Table
> Performance issue: Don't use BufferedMutator for HTable's put method
>
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
> Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch,
> HBASE-18500-v3.patch, HBASE-18500-v4.patch, HBASE-18500-v5.patch,
> HBASE-18500-v5.patch, HBASE-18500-v5.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean
[jira] [Updated] (HBASE-18566) [RSGROUP]Log the client IP/port of the rsgroup admin
[ https://issues.apache.org/jira/browse/HBASE-18566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guangxu Cheng updated HBASE-18566: -- Attachment: (was: HBASE-18566.patch) > [RSGROUP]Log the client IP/port of the rsgroup admin > > > Key: HBASE-18566 > URL: https://issues.apache.org/jira/browse/HBASE-18566 > Project: HBase > Issue Type: Improvement > Components: rsgroup >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng > > It is necessary to log the client IP/port of the rsgroup admin command as > discussion in HBASE-8754,so that we know who moved the servers from one group > to another etc. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-18500:
---
Description:
Copied the test result from HBASE-17994.
Run start-hbase.sh in my local computer and use the default config to test with
PE tool.
{code}
./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10
--nomapred --autoFlush=True randomWrite 1
./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10
--nomapred --autoFlush=True asyncRandomWrite 1
{code}
Mean latency test result.
|| || Test1 || Test2 || Test3 || Test4 || Test5 ||
| randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
| asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
50th latency test result.
|| || Test1 || Test2 || Test3 || Test4 || Test5 ||
| randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
| asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
99th latency test result.
|| || Test1 || Test2 || Test3 || Test4 || Test5 ||
| randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
| asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
In our internal 0.98 branch, the PE test result shows the async write has the
almost same latency with the blocking write. But for master branch, the result
shows the async write has better latency than the blocking client. Take a look
about the code, I thought the difference is the BufferedMutator. For master
branch, HTable don't have a write buffer and all write request will be flushed
directly. And user can use BufferedMutator when user want to perform
client-side buffering of writes. For the performance issue (autoFlush=True), I
thought we can use rpc caller directly in HTable's put method. Thanks.
Review: https://reviews.apache.org/r/61454/
Copy the comments from [~chia7712]. Remove the brings
1. correct the metrics (see HBASE-18476)
2. make HTable thread-safe (see HBASE-17368)
3. reduce the latency
4. get rid of some deprecated methods in Table
was:
Copied the test result from HBASE-17994.
Run start-hbase.sh in my local computer and use the default config to test with
PE tool.
{code}
./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10
--nomapred --autoFlush=True randomWrite 1
./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10
--nomapred --autoFlush=True asyncRandomWrite 1
{code}
Mean latency test result.
|| || Test1 || Test2 || Test3 || Test4 || Test5 ||
| randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
| asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
50th latency test result.
|| || Test1 || Test2 || Test3 || Test4 || Test5 ||
| randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
| asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
99th latency test result.
|| || Test1 || Test2 || Test3 || Test4 || Test5 ||
| randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
| asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
In our internal 0.98 branch, the PE test result shows the async write has the
almost same latency with the blocking write. But for master branch, the result
shows the async write has better latency than the blocking client. Take a look
about the code, I thought the difference is the BufferedMutator. For master
branch, HTable don't have a write buffer and all write request will be flushed
directly. And user can use BufferedMutator when user want to perform
client-side buffering of writes. For the performance issue (autoFlush=True), I
thought we can use rpc caller directly in HTable's put method. Thanks.
Review: https://reviews.apache.org/r/61454/
> Performance issue: Don't use BufferedMutator for HTable's put method
>
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
> Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch,
> HBASE-18500-v3.patch, HBASE-18500-v4.patch, HBASE-18500-v5.patch,
> HBASE-18500-v5.patch, HBASE-18500-v5.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 ||
[jira] [Updated] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-18500:
---
Description:
Copied the test result from HBASE-17994.
Run start-hbase.sh in my local computer and use the default config to test with
PE tool.
{code}
./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10
--nomapred --autoFlush=True randomWrite 1
./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10
--nomapred --autoFlush=True asyncRandomWrite 1
{code}
Mean latency test result.
|| || Test1 || Test2 || Test3 || Test4 || Test5 ||
| randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
| asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
50th latency test result.
|| || Test1 || Test2 || Test3 || Test4 || Test5 ||
| randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
| asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
99th latency test result.
|| || Test1 || Test2 || Test3 || Test4 || Test5 ||
| randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
| asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
In our internal 0.98 branch, the PE test result shows the async write has the
almost same latency with the blocking write. But for master branch, the result
shows the async write has better latency than the blocking client. Take a look
about the code, I thought the difference is the BufferedMutator. For master
branch, HTable don't have a write buffer and all write request will be flushed
directly. And user can use BufferedMutator when user want to perform
client-side buffering of writes. For the performance issue (autoFlush=True), I
thought we can use rpc caller directly in HTable's put method. Thanks.
Review: https://reviews.apache.org/r/61454/
Copy the comments from [~chia7712]. Remove the BufferdMutator brings some
1. correct the metrics (see HBASE-18476)
2. make HTable thread-safe (see HBASE-17368)
3. reduce the latency
4. get rid of some deprecated methods in Table
was:
Copied the test result from HBASE-17994.
Run start-hbase.sh in my local computer and use the default config to test with
PE tool.
{code}
./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10
--nomapred --autoFlush=True randomWrite 1
./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10
--nomapred --autoFlush=True asyncRandomWrite 1
{code}
Mean latency test result.
|| || Test1 || Test2 || Test3 || Test4 || Test5 ||
| randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
| asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
50th latency test result.
|| || Test1 || Test2 || Test3 || Test4 || Test5 ||
| randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
| asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
99th latency test result.
|| || Test1 || Test2 || Test3 || Test4 || Test5 ||
| randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
| asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
In our internal 0.98 branch, the PE test result shows the async write has the
almost same latency with the blocking write. But for master branch, the result
shows the async write has better latency than the blocking client. Take a look
about the code, I thought the difference is the BufferedMutator. For master
branch, HTable don't have a write buffer and all write request will be flushed
directly. And user can use BufferedMutator when user want to perform
client-side buffering of writes. For the performance issue (autoFlush=True), I
thought we can use rpc caller directly in HTable's put method. Thanks.
Review: https://reviews.apache.org/r/61454/
Copy the comments from [~chia7712]. Remove the brings
1. correct the metrics (see HBASE-18476)
2. make HTable thread-safe (see HBASE-17368)
3. reduce the latency
4. get rid of some deprecated methods in Table
> Performance issue: Don't use BufferedMutator for HTable's put method
>
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
> Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch,
> HBASE-18500-v3.patch, HBASE-18500-v4.patch, HBASE-18500-v5.patch,
> HBASE-18500-v5.patch, HBASE-18500-v5.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1
[jira] [Commented] (HBASE-14135) HBase Backup/Restore Phase 3: Merge backup images
[
https://issues.apache.org/jira/browse/HBASE-14135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122785#comment-16122785
]
Josh Elser commented on HBASE-14135:
+1 once we get a proper QA back.
bq. What is a blocker is moving backup out of hbase-server module, HBASE-17614
[~vrodionov], can you find the cycles to also take on this work in the
near-term, please?
> HBase Backup/Restore Phase 3: Merge backup images
> -
>
> Key: HBASE-14135
> URL: https://issues.apache.org/jira/browse/HBASE-14135
> Project: HBase
> Issue Type: New Feature
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Critical
> Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-14135-v10.patch, HBASE-14135-v3.patch,
> HBASE-14135-v5.patch, HBASE-14135-v6.patch, HBASE-14135-v7.patch,
> HBASE-14135-v8.patch, HBASE-14135-v9.patch
>
>
> User can merge incremental backup images into single incremental backup image.
> # Merge supports only incremental images
> # Merge supports only images for the same backup destinations
> Command:
> {code}
> hbase backup merge image1,image2,..imageK
> {code}
> Example:
> {code}
> hbase backup merge backup_143126764557,backup_143126764456
> {code}
> When operation is complete, only the most recent backup image will be kept
> (in above example - backup_143126764557) as a merged backup image, all other
> images will be deleted from both: file system and backup system tables,
> corresponding backup manifest for the merged backup image will be updated to
> remove dependencies from deleted images. Merged backup image will contains
> all the data from original image and from deleted images.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HBASE-14135) HBase Backup/Restore Phase 3: Merge backup images
[
https://issues.apache.org/jira/browse/HBASE-14135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122777#comment-16122777
]
Josh Elser edited comment on HBASE-14135 at 8/11/17 3:33 AM:
-
Re-attaching v10 since qa -since- seems to have not been accurate.
was (Author: elserj):
Re-attaching v10 since qa since to have not been accurate.
> HBase Backup/Restore Phase 3: Merge backup images
> -
>
> Key: HBASE-14135
> URL: https://issues.apache.org/jira/browse/HBASE-14135
> Project: HBase
> Issue Type: New Feature
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Critical
> Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-14135-v10.patch, HBASE-14135-v3.patch,
> HBASE-14135-v5.patch, HBASE-14135-v6.patch, HBASE-14135-v7.patch,
> HBASE-14135-v8.patch, HBASE-14135-v9.patch
>
>
> User can merge incremental backup images into single incremental backup image.
> # Merge supports only incremental images
> # Merge supports only images for the same backup destinations
> Command:
> {code}
> hbase backup merge image1,image2,..imageK
> {code}
> Example:
> {code}
> hbase backup merge backup_143126764557,backup_143126764456
> {code}
> When operation is complete, only the most recent backup image will be kept
> (in above example - backup_143126764557) as a merged backup image, all other
> images will be deleted from both: file system and backup system tables,
> corresponding backup manifest for the merged backup image will be updated to
> remove dependencies from deleted images. Merged backup image will contains
> all the data from original image and from deleted images.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18566) [RSGROUP]Log the client IP/port of the rsgroup admin
[ https://issues.apache.org/jira/browse/HBASE-18566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guangxu Cheng updated HBASE-18566: -- Attachment: HBASE-18566.patch > [RSGROUP]Log the client IP/port of the rsgroup admin > > > Key: HBASE-18566 > URL: https://issues.apache.org/jira/browse/HBASE-18566 > Project: HBase > Issue Type: Improvement > Components: rsgroup >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng > Attachments: HBASE-18566.patch > > > It is necessary to log the client IP/port of the rsgroup admin command as > discussion in HBASE-8754,so that we know who moved the servers from one group > to another etc. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18566) [RSGROUP]Log the client IP/port of the rsgroup admin
[ https://issues.apache.org/jira/browse/HBASE-18566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122778#comment-16122778 ] Guangxu Cheng commented on HBASE-18566: --- Upload the first patch for review.Thanks > [RSGROUP]Log the client IP/port of the rsgroup admin > > > Key: HBASE-18566 > URL: https://issues.apache.org/jira/browse/HBASE-18566 > Project: HBase > Issue Type: Improvement > Components: rsgroup >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng > Attachments: HBASE-18566.patch > > > It is necessary to log the client IP/port of the rsgroup admin command as > discussion in HBASE-8754,so that we know who moved the servers from one group > to another etc. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-14135) HBase Backup/Restore Phase 3: Merge backup images
[
https://issues.apache.org/jira/browse/HBASE-14135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser updated HBASE-14135:
---
Attachment: HBASE-14135-v10.patch
Re-attaching v10 since qa since to have not been accurate.
> HBase Backup/Restore Phase 3: Merge backup images
> -
>
> Key: HBASE-14135
> URL: https://issues.apache.org/jira/browse/HBASE-14135
> Project: HBase
> Issue Type: New Feature
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Critical
> Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-14135-v10.patch, HBASE-14135-v3.patch,
> HBASE-14135-v5.patch, HBASE-14135-v6.patch, HBASE-14135-v7.patch,
> HBASE-14135-v8.patch, HBASE-14135-v9.patch
>
>
> User can merge incremental backup images into single incremental backup image.
> # Merge supports only incremental images
> # Merge supports only images for the same backup destinations
> Command:
> {code}
> hbase backup merge image1,image2,..imageK
> {code}
> Example:
> {code}
> hbase backup merge backup_143126764557,backup_143126764456
> {code}
> When operation is complete, only the most recent backup image will be kept
> (in above example - backup_143126764557) as a merged backup image, all other
> images will be deleted from both: file system and backup system tables,
> corresponding backup manifest for the merged backup image will be updated to
> remove dependencies from deleted images. Merged backup image will contains
> all the data from original image and from deleted images.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-14135) HBase Backup/Restore Phase 3: Merge backup images
[
https://issues.apache.org/jira/browse/HBASE-14135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser updated HBASE-14135:
---
Attachment: (was: HBASE-14135-v10.patch)
> HBase Backup/Restore Phase 3: Merge backup images
> -
>
> Key: HBASE-14135
> URL: https://issues.apache.org/jira/browse/HBASE-14135
> Project: HBase
> Issue Type: New Feature
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Critical
> Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-14135-v3.patch, HBASE-14135-v5.patch,
> HBASE-14135-v6.patch, HBASE-14135-v7.patch, HBASE-14135-v8.patch,
> HBASE-14135-v9.patch
>
>
> User can merge incremental backup images into single incremental backup image.
> # Merge supports only incremental images
> # Merge supports only images for the same backup destinations
> Command:
> {code}
> hbase backup merge image1,image2,..imageK
> {code}
> Example:
> {code}
> hbase backup merge backup_143126764557,backup_143126764456
> {code}
> When operation is complete, only the most recent backup image will be kept
> (in above example - backup_143126764557) as a merged backup image, all other
> images will be deleted from both: file system and backup system tables,
> corresponding backup manifest for the merged backup image will be updated to
> remove dependencies from deleted images. Merged backup image will contains
> all the data from original image and from deleted images.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18024) HRegion#initializeRegionInternals should not re-create .hregioninfo file when the region directory no longer exists
[
https://issues.apache.org/jira/browse/HBASE-18024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122769#comment-16122769
]
Hudson commented on HBASE-18024:
FAILURE: Integrated in Jenkins build HBase-1.4 #845 (See
[https://builds.apache.org/job/HBase-1.4/845/])
HBASE-18024 HRegion#initializeRegionInternals should not re-create (esteban:
rev 15bad3c0367717e3e79fe5b2ac42ab713fbeddb6)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileRefresherChore.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionOpen.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> HRegion#initializeRegionInternals should not re-create .hregioninfo file when
> the region directory no longer exists
> ---
>
> Key: HBASE-18024
> URL: https://issues.apache.org/jira/browse/HBASE-18024
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment, regionserver
>Affects Versions: 2.0.0, 1.4.0, 1.3.1, 1.2.5
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
> Fix For: 2.0.0, 1.4.0, 1.3.2, 1.5.0, 1.2.7
>
> Attachments: HBASE-18024.001.patch, HBASE-18024.002.patch,
> HBASE-18024.003.patch, HBASE-18024.004.patch
>
>
> When a RegionSever attempts to open a region, during initialization the RS
> tries to open the {{/data///.hregioninfo}}
> file, however if the {{.hregioninfofile}} doesn't exist, the RegionServer
> will create a new one on {{HRegionFileSystem#checkRegionInfoOnFilesystem}}. A
> side effect of that tools like hbck will incorrectly assume an inconsistency
> due the presence of this new {{.hregioninfofile}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Reopened] (HBASE-18390) Sleep too long when finding region location failed
[ https://issues.apache.org/jira/browse/HBASE-18390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reopened HBASE-18390: --- > Sleep too long when finding region location failed > -- > > Key: HBASE-18390 > URL: https://issues.apache.org/jira/browse/HBASE-18390 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.1, 1.2.6, 1.1.11, 2.0.0-alpha-1 >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 3.0.0, 1.4.0, 1.3.2, 1.5.0, 1.2.7, 2.0.0-alpha-2, 1.1.12 > > Attachments: HBASE-18390.v01.patch, HBASE-18390.v02.patch, > HBASE-18390.v03.patch > > > If RegionServerCallable#prepare failed when getRegionLocation, the location > in this callable object is null. And before we retry we will sleep. However, > when location is null we will sleep at least 10 seconds. And the request will > be failed directly if operation timeout is less than 10 seconds. I think it > is no need to keep MIN_WAIT_DEAD_SERVER logic. Use backoff sleeping logic is > ok for most cases. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17125) Inconsistent result when use filter to read data
[
https://issues.apache.org/jira/browse/HBASE-17125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-17125:
---
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
> Inconsistent result when use filter to read data
>
>
> Key: HBASE-17125
> URL: https://issues.apache.org/jira/browse/HBASE-17125
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 2.0.0, 3.0.0
>
> Attachments: 17125-slack-13.txt, example.diff,
> HBASE-17125.master.001.patch, HBASE-17125.master.002.patch,
> HBASE-17125.master.002.patch, HBASE-17125.master.003.patch,
> HBASE-17125.master.004.patch, HBASE-17125.master.005.patch,
> HBASE-17125.master.006.patch, HBASE-17125.master.007.patch,
> HBASE-17125.master.008.patch, HBASE-17125.master.009.patch,
> HBASE-17125.master.009.patch, HBASE-17125.master.010.patch,
> HBASE-17125.master.011.patch, HBASE-17125.master.011.patch,
> HBASE-17125.master.012.patch, HBASE-17125.master.013.patch,
> HBASE-17125.master.014.patch, HBASE-17125.master.015.patch,
> HBASE-17125.master.016.patch, HBASE-17125.master.017.patch,
> HBASE-17125.master.018.patch, HBASE-17125.master.019.patch,
> HBASE-17125.master.020.patch, HBASE-17125.master.020.patch,
> HBASE-17125.master.021.patch, HBASE-17125.master.022.patch,
> HBASE-17125.master.checkReturnedVersions.patch,
> HBASE-17125.master.no-specified-filter.patch
>
>
> Assume a cloumn's max versions is 3, then we write 4 versions of this column.
> The oldest version doesn't remove immediately. But from the user view, the
> oldest version has gone. When user use a filter to query, if the filter skip
> a new version, then the oldest version will be seen again. But after compact
> the region, then the oldest version will never been seen. So it is weird for
> user. The query will get inconsistent result before and after region
> compaction.
> The reason is matchColumn method of UserScanQueryMatcher. It first check the
> cell by filter, then check the number of versions needed. So if the filter
> skip the new version, then the oldest version will be seen again when it is
> not removed.
> Have a discussion offline with [~Apache9] and [~fenghh], now we have two
> solution for this problem. The first idea is check the number of versions
> first, then check the cell by filter. As the comment of setFilter, the filter
> is called after all tests for ttl, column match, deletes and max versions
> have been run.
> {code}
> /**
>* Apply the specified server-side filter when performing the Query.
>* Only {@link Filter#filterKeyValue(Cell)} is called AFTER all tests
>* for ttl, column match, deletes and max versions have been run.
>* @param filter filter to run on the server
>* @return this for invocation chaining
>*/
> public Query setFilter(Filter filter) {
> this.filter = filter;
> return this;
> }
> {code}
> But this idea has another problem, if a column's max version is 5 and the
> user query only need 3 versions. It first check the version's number, then
> check the cell by filter. So the cells number of the result may less than 3.
> But there are 2 versions which don't read anymore.
> So the second idea has three steps.
> 1. check by the max versions of this column
> 2. check the kv by filter
> 3. check the versions which user need.
> But this will lead the ScanQueryMatcher more complicated. And this will break
> the javadoc of Query.setFilter.
> Now we don't have a final solution for this problem. Suggestions are welcomed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Issue Comment Deleted] (HBASE-18566) [RSGROUP]Log the client IP/port of the rsgroup admin
[ https://issues.apache.org/jira/browse/HBASE-18566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guangxu Cheng updated HBASE-18566: -- Comment: was deleted (was: Upload the first patch for review.) > [RSGROUP]Log the client IP/port of the rsgroup admin > > > Key: HBASE-18566 > URL: https://issues.apache.org/jira/browse/HBASE-18566 > Project: HBase > Issue Type: Improvement > Components: rsgroup >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng > > It is necessary to log the client IP/port of the rsgroup admin command as > discussion in HBASE-8754,so that we know who moved the servers from one group > to another etc. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122763#comment-16122763
]
Guanghao Zhang commented on HBASE-18500:
bq. We should address it via some other jira. As of here, just do as what u did
in patch with a TODO to fix the issue.
Will add a TODO when commit it.
Thanks all for reviewing.
> Performance issue: Don't use BufferedMutator for HTable's put method
>
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
> Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch,
> HBASE-18500-v3.patch, HBASE-18500-v4.patch, HBASE-18500-v5.patch,
> HBASE-18500-v5.patch, HBASE-18500-v5.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the
> almost same latency with the blocking write. But for master branch, the
> result shows the async write has better latency than the blocking client.
> Take a look about the code, I thought the difference is the BufferedMutator.
> For master branch, HTable don't have a write buffer and all write request
> will be flushed directly. And user can use BufferedMutator when user want to
> perform client-side buffering of writes. For the performance issue
> (autoFlush=True), I thought we can use rpc caller directly in HTable's put
> method. Thanks.
> Review: https://reviews.apache.org/r/61454/
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-17125) Inconsistent result when use filter to read data
[
https://issues.apache.org/jira/browse/HBASE-17125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-17125:
---
Hadoop Flags: Incompatible change
Release Note: Marked Scan and Get's setMaxVersions() and
setMaxVersions(int) as deprecated. They are easy to misunderstand with column
family's max versions, so use readAllVersions() and readVersions(int) instead.
> Inconsistent result when use filter to read data
>
>
> Key: HBASE-17125
> URL: https://issues.apache.org/jira/browse/HBASE-17125
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 17125-slack-13.txt, example.diff,
> HBASE-17125.master.001.patch, HBASE-17125.master.002.patch,
> HBASE-17125.master.002.patch, HBASE-17125.master.003.patch,
> HBASE-17125.master.004.patch, HBASE-17125.master.005.patch,
> HBASE-17125.master.006.patch, HBASE-17125.master.007.patch,
> HBASE-17125.master.008.patch, HBASE-17125.master.009.patch,
> HBASE-17125.master.009.patch, HBASE-17125.master.010.patch,
> HBASE-17125.master.011.patch, HBASE-17125.master.011.patch,
> HBASE-17125.master.012.patch, HBASE-17125.master.013.patch,
> HBASE-17125.master.014.patch, HBASE-17125.master.015.patch,
> HBASE-17125.master.016.patch, HBASE-17125.master.017.patch,
> HBASE-17125.master.018.patch, HBASE-17125.master.019.patch,
> HBASE-17125.master.020.patch, HBASE-17125.master.020.patch,
> HBASE-17125.master.021.patch, HBASE-17125.master.022.patch,
> HBASE-17125.master.checkReturnedVersions.patch,
> HBASE-17125.master.no-specified-filter.patch
>
>
> Assume a cloumn's max versions is 3, then we write 4 versions of this column.
> The oldest version doesn't remove immediately. But from the user view, the
> oldest version has gone. When user use a filter to query, if the filter skip
> a new version, then the oldest version will be seen again. But after compact
> the region, then the oldest version will never been seen. So it is weird for
> user. The query will get inconsistent result before and after region
> compaction.
> The reason is matchColumn method of UserScanQueryMatcher. It first check the
> cell by filter, then check the number of versions needed. So if the filter
> skip the new version, then the oldest version will be seen again when it is
> not removed.
> Have a discussion offline with [~Apache9] and [~fenghh], now we have two
> solution for this problem. The first idea is check the number of versions
> first, then check the cell by filter. As the comment of setFilter, the filter
> is called after all tests for ttl, column match, deletes and max versions
> have been run.
> {code}
> /**
>* Apply the specified server-side filter when performing the Query.
>* Only {@link Filter#filterKeyValue(Cell)} is called AFTER all tests
>* for ttl, column match, deletes and max versions have been run.
>* @param filter filter to run on the server
>* @return this for invocation chaining
>*/
> public Query setFilter(Filter filter) {
> this.filter = filter;
> return this;
> }
> {code}
> But this idea has another problem, if a column's max version is 5 and the
> user query only need 3 versions. It first check the version's number, then
> check the cell by filter. So the cells number of the result may less than 3.
> But there are 2 versions which don't read anymore.
> So the second idea has three steps.
> 1. check by the max versions of this column
> 2. check the kv by filter
> 3. check the versions which user need.
> But this will lead the ScanQueryMatcher more complicated. And this will break
> the javadoc of Query.setFilter.
> Now we don't have a final solution for this problem. Suggestions are welcomed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18390) Sleep too long when finding region location failed
[ https://issues.apache.org/jira/browse/HBASE-18390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122754#comment-16122754 ] Ted Yu commented on HBASE-18390: 15 unit tests failed since this went in: https://builds.apache.org/job/HBase-1.2-JDK7/165/ TestClassFinder and TestCoprocessorClassLoader > Sleep too long when finding region location failed > -- > > Key: HBASE-18390 > URL: https://issues.apache.org/jira/browse/HBASE-18390 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.1, 1.2.6, 1.1.11, 2.0.0-alpha-1 >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 3.0.0, 1.4.0, 1.3.2, 1.5.0, 1.2.7, 2.0.0-alpha-2, 1.1.12 > > Attachments: HBASE-18390.v01.patch, HBASE-18390.v02.patch, > HBASE-18390.v03.patch > > > If RegionServerCallable#prepare failed when getRegionLocation, the location > in this callable object is null. And before we retry we will sleep. However, > when location is null we will sleep at least 10 seconds. And the request will > be failed directly if operation timeout is less than 10 seconds. I think it > is no need to keep MIN_WAIT_DEAD_SERVER logic. Use backoff sleeping logic is > ok for most cases. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18566) [RSGROUP]Log the client IP/port of the rsgroup admin
[ https://issues.apache.org/jira/browse/HBASE-18566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122753#comment-16122753 ] Guangxu Cheng commented on HBASE-18566: --- Upload the first patch for review. > [RSGROUP]Log the client IP/port of the rsgroup admin > > > Key: HBASE-18566 > URL: https://issues.apache.org/jira/browse/HBASE-18566 > Project: HBase > Issue Type: Improvement > Components: rsgroup >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng > Attachments: HBASE-18566.patch > > > It is necessary to log the client IP/port of the rsgroup admin command as > discussion in HBASE-8754,so that we know who moved the servers from one group > to another etc. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18566) [RSGROUP]Log the client IP/port of the rsgroup admin
[ https://issues.apache.org/jira/browse/HBASE-18566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guangxu Cheng updated HBASE-18566: -- Attachment: HBASE-18566.patch > [RSGROUP]Log the client IP/port of the rsgroup admin > > > Key: HBASE-18566 > URL: https://issues.apache.org/jira/browse/HBASE-18566 > Project: HBase > Issue Type: Improvement > Components: rsgroup >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng > Attachments: HBASE-18566.patch > > > It is necessary to log the client IP/port of the rsgroup admin command as > discussion in HBASE-8754,so that we know who moved the servers from one group > to another etc. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18024) HRegion#initializeRegionInternals should not re-create .hregioninfo file when the region directory no longer exists
[
https://issues.apache.org/jira/browse/HBASE-18024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122751#comment-16122751
]
Hudson commented on HBASE-18024:
FAILURE: Integrated in Jenkins build HBase-1.3-JDK8 #240 (See
[https://builds.apache.org/job/HBase-1.3-JDK8/240/])
HBASE-18024 HRegion#initializeRegionInternals should not re-create (esteban:
rev 77847da928f9ee79674f54f8dc517fdbde35287f)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileRefresherChore.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionOpen.java
> HRegion#initializeRegionInternals should not re-create .hregioninfo file when
> the region directory no longer exists
> ---
>
> Key: HBASE-18024
> URL: https://issues.apache.org/jira/browse/HBASE-18024
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment, regionserver
>Affects Versions: 2.0.0, 1.4.0, 1.3.1, 1.2.5
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
> Fix For: 2.0.0, 1.4.0, 1.3.2, 1.5.0, 1.2.7
>
> Attachments: HBASE-18024.001.patch, HBASE-18024.002.patch,
> HBASE-18024.003.patch, HBASE-18024.004.patch
>
>
> When a RegionSever attempts to open a region, during initialization the RS
> tries to open the {{/data///.hregioninfo}}
> file, however if the {{.hregioninfofile}} doesn't exist, the RegionServer
> will create a new one on {{HRegionFileSystem#checkRegionInfoOnFilesystem}}. A
> side effect of that tools like hbck will incorrectly assume an inconsistency
> due the presence of this new {{.hregioninfofile}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18387) [Thrift] Make principal configurable in DemoClient.java
[
https://issues.apache.org/jira/browse/HBASE-18387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser updated HBASE-18387:
---
Fix Version/s: 1.1.12
1.2.7
1.3.2
1.4.0
2.0.0
> [Thrift] Make principal configurable in DemoClient.java
> ---
>
> Key: HBASE-18387
> URL: https://issues.apache.org/jira/browse/HBASE-18387
> Project: HBase
> Issue Type: Improvement
>Reporter: Lars George
>Assignee: Tamas Penzes
>Priority: Minor
> Labels: beginner
> Fix For: 2.0.0, 1.4.0, 1.3.2, 1.2.7, 1.1.12
>
> Attachments: HBASE-18387.master.001.patch,
> HBASE-18387.master.002.patch, HBASE-18387.master.003.patch
>
>
> In the Thrift1 demo client we have this code:
> {code}
> transport = new TSaslClientTransport("GSSAPI", null,
> "hbase", // Thrift server user name, should be an authorized proxy user.
> host, // Thrift server domain
> saslProperties, null, transport);
> {code}
> This will only work when the Thrift server is started with the {{hbase}}
> principal. Often this may deviate, for example I am using {{hbase-thrift}} to
> separate the names from those of backend servers.
> What we need is either an additional command line option to specify the name,
> or a property that can be set with -D and can be passed at runtime. I prefer
> the former, as the latter is making this a little convoluted.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (HBASE-18566) [RSGROUP]Log the client IP/port of the rsgroup admin
Guangxu Cheng created HBASE-18566: - Summary: [RSGROUP]Log the client IP/port of the rsgroup admin Key: HBASE-18566 URL: https://issues.apache.org/jira/browse/HBASE-18566 Project: HBase Issue Type: Improvement Components: rsgroup Reporter: Guangxu Cheng Assignee: Guangxu Cheng It is necessary to log the client IP/port of the rsgroup admin command as discussion in HBASE-8754,so that we know who moved the servers from one group to another etc. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18024) HRegion#initializeRegionInternals should not re-create .hregioninfo file when the region directory no longer exists
[
https://issues.apache.org/jira/browse/HBASE-18024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122744#comment-16122744
]
Hudson commented on HBASE-18024:
SUCCESS: Integrated in Jenkins build HBase-1.2-JDK8 #184 (See
[https://builds.apache.org/job/HBase-1.2-JDK8/184/])
HBASE-18024 HRegion#initializeRegionInternals should not re-create (esteban:
rev 11503fc6e65a6e350d4a17cd6031d6ae4d204bdc)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileRefresherChore.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
> HRegion#initializeRegionInternals should not re-create .hregioninfo file when
> the region directory no longer exists
> ---
>
> Key: HBASE-18024
> URL: https://issues.apache.org/jira/browse/HBASE-18024
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment, regionserver
>Affects Versions: 2.0.0, 1.4.0, 1.3.1, 1.2.5
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
> Fix For: 2.0.0, 1.4.0, 1.3.2, 1.5.0, 1.2.7
>
> Attachments: HBASE-18024.001.patch, HBASE-18024.002.patch,
> HBASE-18024.003.patch, HBASE-18024.004.patch
>
>
> When a RegionSever attempts to open a region, during initialization the RS
> tries to open the {{/data///.hregioninfo}}
> file, however if the {{.hregioninfofile}} doesn't exist, the RegionServer
> will create a new one on {{HRegionFileSystem#checkRegionInfoOnFilesystem}}. A
> side effect of that tools like hbck will incorrectly assume an inconsistency
> due the presence of this new {{.hregioninfofile}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18368) FilterList with multiple FamilyFilters concatenated by OR does not work.
[
https://issues.apache.org/jira/browse/HBASE-18368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122741#comment-16122741
]
Guanghao Zhang commented on HBASE-18368:
How about we rename NEXT_ROW to NEXT_FAMILY directly?
> FilterList with multiple FamilyFilters concatenated by OR does not work.
>
>
> Key: HBASE-18368
> URL: https://issues.apache.org/jira/browse/HBASE-18368
> Project: HBase
> Issue Type: Sub-task
> Components: Filters
>Affects Versions: 3.0.0, 2.0.0-alpha-1
>Reporter: Peter Somogyi
>Assignee: Zheng Hu
>Priority: Critical
> Attachments: HBASE-18368.branch-1.patch,
> HBASE-18368.branch-1.v2.patch, HBASE-18368.branch-1.v3.patch,
> HBASE-18368.patch, HBASE-18368.v2.patch
>
>
> Scan gives back incomplete list if multiple filters are combined with OR /
> MUST_PASS_ONE.
> Using 2 FamilyFilters in a FilterList using MUST_PASS_ONE operator will give
> back results for only the first Filter.
> {code:java|title=Test code}
> @Test
> public void testFiltersWithOr() throws Exception {
> TableName tn = TableName.valueOf("MyTest");
> Table table = utility.createTable(tn, new String[] {"cf1", "cf2"});
> byte[] CF1 = Bytes.toBytes("cf1");
> byte[] CF2 = Bytes.toBytes("cf2");
> Put put1 = new Put(Bytes.toBytes("0"));
> put1.addColumn(CF1, Bytes.toBytes("col_a"), Bytes.toBytes(0));
> table.put(put1);
> Put put2 = new Put(Bytes.toBytes("0"));
> put2.addColumn(CF2, Bytes.toBytes("col_b"), Bytes.toBytes(0));
> table.put(put2);
> FamilyFilter filterCF1 = new FamilyFilter(CompareFilter.CompareOp.EQUAL,
> new BinaryComparator(CF1));
> FamilyFilter filterCF2 = new FamilyFilter(CompareFilter.CompareOp.EQUAL,
> new BinaryComparator(CF2));
> FilterList filterList = new FilterList(FilterList.Operator.MUST_PASS_ONE);
> filterList.addFilter(filterCF1);
> filterList.addFilter(filterCF2);
> Scan scan = new Scan();
> scan.setFilter(filterList);
> ResultScanner scanner = table.getScanner(scan);
> System.out.println(filterList);
> for (Result rr = scanner.next(); rr != null; rr = scanner.next()) {
> System.out.println(rr);
> }
> }
> {code}
> {noformat:title=Output}
> FilterList OR (2/2): [FamilyFilter (EQUAL, cf1), FamilyFilter (EQUAL, cf2)]
> keyvalues={0/cf1:col_a/1499852754957/Put/vlen=4/seqid=0}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122740#comment-16122740
]
Guanghao Zhang commented on HBASE-18500:
Thanks for your explanation.
bq. The table on which put was called is created via the RegionCP env and that
uses a special Connection.
Checked the code, I thought this was introduced by HBASE-9534.
bq. For #1 we may have to use InheritableThreadLocal instead of ThreadLocal.
Agreed. But I don't know if it has any other side-effects.
bq. When doing a runAsUser, how/whether we should reset the Rpc context to have
the new user.
Do we have other scenario which need reset the Rpc context? If not, I thought
we can only fixed this for this problem. We can introduce a new config or
method to disable the "Short-Circuit Coprocessor HTable access" feature. The
code may be:
{code}
User.runAsLoginUser(new PrivilegedExceptionAction() {
@Override
public Void run() throws Exception {
// disable the "Short-Circuit Coprocessor HTable access" feature
first
// Then start a new rpc call
AccessControlLists.addUserPermission(regionEnv.getConfiguration(),
perm,
regionEnv.getTable(AccessControlLists.ACL_TABLE_NAME),
request.getMergeExistingPermissions());
return null;
}
});
{code}
> Performance issue: Don't use BufferedMutator for HTable's put method
>
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
> Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch,
> HBASE-18500-v3.patch, HBASE-18500-v4.patch, HBASE-18500-v5.patch,
> HBASE-18500-v5.patch, HBASE-18500-v5.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the
> almost same latency with the blocking write. But for master branch, the
> result shows the async write has better latency than the blocking client.
> Take a look about the code, I thought the difference is the BufferedMutator.
> For master branch, HTable don't have a write buffer and all write request
> will be flushed directly. And user can use BufferedMutator when user want to
> perform client-side buffering of writes. For the performance issue
> (autoFlush=True), I thought we can use rpc caller directly in HTable's put
> method. Thanks.
> Review: https://reviews.apache.org/r/61454/
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18255) Time-Delayed HBase Performance Degradation with Java 7
[ https://issues.apache.org/jira/browse/HBASE-18255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122735#comment-16122735 ] Hudson commented on HBASE-18255: SUCCESS: Integrated in Jenkins build HBase-1.2-IT #925 (See [https://builds.apache.org/job/HBase-1.2-IT/925/]) HBASE-18479 should apply HBASE-18255 to HBASE_MASTER_OPTS too (tedyu: rev 9f809748d72b39489f202a56aba107cf7386f30c) * (edit) conf/hbase-env.sh * (edit) conf/hbase-env.cmd > Time-Delayed HBase Performance Degradation with Java 7 > -- > > Key: HBASE-18255 > URL: https://issues.apache.org/jira/browse/HBASE-18255 > Project: HBase > Issue Type: Bug > Components: Performance >Affects Versions: 1.3.1, 1.2.6, 1.1.11 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Critical > Labels: jdk7 > Fix For: 1.4.0, 1.3.2, 1.2.7, 1.1.12 > > Attachments: HBASE-18255-branch-1.x.v1.patch > > > The good summary of the issue and provided resolution can be found in this > article: > https://community.hortonworks.com/articles/105802/time-delayed-hbase-performance-degradation-with-ja.html > In a few words, due to internal JVM 7 bug (which has been addressed only in > Java 8), HotSpot code cache can become full and after that ALL JIT > compilations get suspended indefinitely. The default value for code cache > size in JVM 7 is quite low: 48MB. It is recommended to increase this value at > least to 256MB (default in JVM 8). > This BUG affects only 1.x -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18479) should apply HBASE-18255 to HBASE_MASTER_OPTS too
[ https://issues.apache.org/jira/browse/HBASE-18479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122734#comment-16122734 ] Hudson commented on HBASE-18479: SUCCESS: Integrated in Jenkins build HBase-1.2-IT #925 (See [https://builds.apache.org/job/HBase-1.2-IT/925/]) HBASE-18479 should apply HBASE-18255 to HBASE_MASTER_OPTS too (tedyu: rev 9f809748d72b39489f202a56aba107cf7386f30c) * (edit) conf/hbase-env.sh * (edit) conf/hbase-env.cmd > should apply HBASE-18255 to HBASE_MASTER_OPTS too > - > > Key: HBASE-18479 > URL: https://issues.apache.org/jira/browse/HBASE-18479 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.1, 1.2.6, 1.1.11 >Reporter: Yechao Chen >Assignee: Yechao Chen >Priority: Critical > Fix For: 1.4.0, 1.3.2, 1.5.0, 1.2.7 > > Attachments: HBASE-18479-branch-1.v1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18255) Time-Delayed HBase Performance Degradation with Java 7
[ https://issues.apache.org/jira/browse/HBASE-18255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122730#comment-16122730 ] Hudson commented on HBASE-18255: FAILURE: Integrated in Jenkins build HBase-1.3-JDK7 #227 (See [https://builds.apache.org/job/HBase-1.3-JDK7/227/]) HBASE-18479 should apply HBASE-18255 to HBASE_MASTER_OPTS too (tedyu: rev d87b63e7319303a01d2a71fc022e69c434163cce) * (edit) conf/hbase-env.sh * (edit) conf/hbase-env.cmd > Time-Delayed HBase Performance Degradation with Java 7 > -- > > Key: HBASE-18255 > URL: https://issues.apache.org/jira/browse/HBASE-18255 > Project: HBase > Issue Type: Bug > Components: Performance >Affects Versions: 1.3.1, 1.2.6, 1.1.11 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Critical > Labels: jdk7 > Fix For: 1.4.0, 1.3.2, 1.2.7, 1.1.12 > > Attachments: HBASE-18255-branch-1.x.v1.patch > > > The good summary of the issue and provided resolution can be found in this > article: > https://community.hortonworks.com/articles/105802/time-delayed-hbase-performance-degradation-with-ja.html > In a few words, due to internal JVM 7 bug (which has been addressed only in > Java 8), HotSpot code cache can become full and after that ALL JIT > compilations get suspended indefinitely. The default value for code cache > size in JVM 7 is quite low: 48MB. It is recommended to increase this value at > least to 256MB (default in JVM 8). > This BUG affects only 1.x -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18255) Time-Delayed HBase Performance Degradation with Java 7
[ https://issues.apache.org/jira/browse/HBASE-18255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122726#comment-16122726 ] Hudson commented on HBASE-18255: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #164 (See [https://builds.apache.org/job/HBase-1.3-IT/164/]) HBASE-18479 should apply HBASE-18255 to HBASE_MASTER_OPTS too (tedyu: rev d87b63e7319303a01d2a71fc022e69c434163cce) * (edit) conf/hbase-env.cmd * (edit) conf/hbase-env.sh > Time-Delayed HBase Performance Degradation with Java 7 > -- > > Key: HBASE-18255 > URL: https://issues.apache.org/jira/browse/HBASE-18255 > Project: HBase > Issue Type: Bug > Components: Performance >Affects Versions: 1.3.1, 1.2.6, 1.1.11 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Critical > Labels: jdk7 > Fix For: 1.4.0, 1.3.2, 1.2.7, 1.1.12 > > Attachments: HBASE-18255-branch-1.x.v1.patch > > > The good summary of the issue and provided resolution can be found in this > article: > https://community.hortonworks.com/articles/105802/time-delayed-hbase-performance-degradation-with-ja.html > In a few words, due to internal JVM 7 bug (which has been addressed only in > Java 8), HotSpot code cache can become full and after that ALL JIT > compilations get suspended indefinitely. The default value for code cache > size in JVM 7 is quite low: 48MB. It is recommended to increase this value at > least to 256MB (default in JVM 8). > This BUG affects only 1.x -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18479) should apply HBASE-18255 to HBASE_MASTER_OPTS too
[ https://issues.apache.org/jira/browse/HBASE-18479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122729#comment-16122729 ] Hudson commented on HBASE-18479: FAILURE: Integrated in Jenkins build HBase-1.3-JDK7 #227 (See [https://builds.apache.org/job/HBase-1.3-JDK7/227/]) HBASE-18479 should apply HBASE-18255 to HBASE_MASTER_OPTS too (tedyu: rev d87b63e7319303a01d2a71fc022e69c434163cce) * (edit) conf/hbase-env.sh * (edit) conf/hbase-env.cmd > should apply HBASE-18255 to HBASE_MASTER_OPTS too > - > > Key: HBASE-18479 > URL: https://issues.apache.org/jira/browse/HBASE-18479 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.1, 1.2.6, 1.1.11 >Reporter: Yechao Chen >Assignee: Yechao Chen >Priority: Critical > Fix For: 1.4.0, 1.3.2, 1.5.0, 1.2.7 > > Attachments: HBASE-18479-branch-1.v1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18479) should apply HBASE-18255 to HBASE_MASTER_OPTS too
[ https://issues.apache.org/jira/browse/HBASE-18479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122725#comment-16122725 ] Hudson commented on HBASE-18479: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #164 (See [https://builds.apache.org/job/HBase-1.3-IT/164/]) HBASE-18479 should apply HBASE-18255 to HBASE_MASTER_OPTS too (tedyu: rev d87b63e7319303a01d2a71fc022e69c434163cce) * (edit) conf/hbase-env.cmd * (edit) conf/hbase-env.sh > should apply HBASE-18255 to HBASE_MASTER_OPTS too > - > > Key: HBASE-18479 > URL: https://issues.apache.org/jira/browse/HBASE-18479 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.1, 1.2.6, 1.1.11 >Reporter: Yechao Chen >Assignee: Yechao Chen >Priority: Critical > Fix For: 1.4.0, 1.3.2, 1.5.0, 1.2.7 > > Attachments: HBASE-18479-branch-1.v1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18479) should apply HBASE-18255 to HBASE_MASTER_OPTS too
[ https://issues.apache.org/jira/browse/HBASE-18479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122723#comment-16122723 ] Hudson commented on HBASE-18479: FAILURE: Integrated in Jenkins build HBase-1.2-JDK7 #190 (See [https://builds.apache.org/job/HBase-1.2-JDK7/190/]) HBASE-18479 should apply HBASE-18255 to HBASE_MASTER_OPTS too (tedyu: rev 9f809748d72b39489f202a56aba107cf7386f30c) * (edit) conf/hbase-env.cmd * (edit) conf/hbase-env.sh > should apply HBASE-18255 to HBASE_MASTER_OPTS too > - > > Key: HBASE-18479 > URL: https://issues.apache.org/jira/browse/HBASE-18479 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.1, 1.2.6, 1.1.11 >Reporter: Yechao Chen >Assignee: Yechao Chen >Priority: Critical > Fix For: 1.4.0, 1.3.2, 1.5.0, 1.2.7 > > Attachments: HBASE-18479-branch-1.v1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18255) Time-Delayed HBase Performance Degradation with Java 7
[ https://issues.apache.org/jira/browse/HBASE-18255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122724#comment-16122724 ] Hudson commented on HBASE-18255: FAILURE: Integrated in Jenkins build HBase-1.2-JDK7 #190 (See [https://builds.apache.org/job/HBase-1.2-JDK7/190/]) HBASE-18479 should apply HBASE-18255 to HBASE_MASTER_OPTS too (tedyu: rev 9f809748d72b39489f202a56aba107cf7386f30c) * (edit) conf/hbase-env.sh * (edit) conf/hbase-env.cmd > Time-Delayed HBase Performance Degradation with Java 7 > -- > > Key: HBASE-18255 > URL: https://issues.apache.org/jira/browse/HBASE-18255 > Project: HBase > Issue Type: Bug > Components: Performance >Affects Versions: 1.3.1, 1.2.6, 1.1.11 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Critical > Labels: jdk7 > Fix For: 1.4.0, 1.3.2, 1.2.7, 1.1.12 > > Attachments: HBASE-18255-branch-1.x.v1.patch > > > The good summary of the issue and provided resolution can be found in this > article: > https://community.hortonworks.com/articles/105802/time-delayed-hbase-performance-degradation-with-ja.html > In a few words, due to internal JVM 7 bug (which has been addressed only in > Java 8), HotSpot code cache can become full and after that ALL JIT > compilations get suspended indefinitely. The default value for code cache > size in JVM 7 is quite low: 48MB. It is recommended to increase this value at > least to 256MB (default in JVM 8). > This BUG affects only 1.x -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18479) should apply HBASE-18255 to HBASE_MASTER_OPTS too
[ https://issues.apache.org/jira/browse/HBASE-18479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-18479: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: (was: 1.1.12) 1.5.0 Status: Resolved (was: Patch Available) > should apply HBASE-18255 to HBASE_MASTER_OPTS too > - > > Key: HBASE-18479 > URL: https://issues.apache.org/jira/browse/HBASE-18479 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.1, 1.2.6, 1.1.11 >Reporter: Yechao Chen >Assignee: Yechao Chen >Priority: Critical > Fix For: 1.4.0, 1.3.2, 1.5.0, 1.2.7 > > Attachments: HBASE-18479-branch-1.v1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18537) [C++] Improvements to load-client
[
https://issues.apache.org/jira/browse/HBASE-18537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122705#comment-16122705
]
Ted Yu commented on HBASE-18537:
{code}
+ if (end != max_row && end != max_row + 1) {
{code}
Why is the above check needed ?
For DoGet(),
{code}
-for (int m = 1; m <= cols; m++) {
- if (!Verify(result, family, m)) return false;
{code}
Where does the above go ?
> [C++] Improvements to load-client
> -
>
> Key: HBASE-18537
> URL: https://issues.apache.org/jira/browse/HBASE-18537
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: HBASE-14850
>
> Attachments: hbase-18537.patch, hbase-18537_v2.patch
>
>
> A couple of improvements to the load-client after spending some time with
> testing:
> - better log messages
> - support for progress
> - minor bug fixes
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18565) [C++] Fix deadlock in AsyncScanRetryingCaller and other RPCs
[
https://issues.apache.org/jira/browse/HBASE-18565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-18565:
--
Attachment: hbase-18565_v1.patch
v1 patch. The problem is that in AsyncScanRetryingCaller, the chain generated
from Future.then() invocations results in a case where we are executing an RPC
(thus trying to write to the pipeline) while still being inside the
{{hbase::ClientHandler::read}} call chain (look at the above stack traces). The
solution is moving the work for the new Scan.Next() call from IO thread to CPU
executor thread. The patch also contains another bug fix.
> [C++] Fix deadlock in AsyncScanRetryingCaller and other RPCs
>
>
> Key: HBASE-18565
> URL: https://issues.apache.org/jira/browse/HBASE-18565
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: HBASE-14850
>
> Attachments: hbase-18565_v1.patch
>
>
> When running the load-client test, sometimes we get a deadlock.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18479) should apply HBASE-18255 to HBASE_MASTER_OPTS too
[ https://issues.apache.org/jira/browse/HBASE-18479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yechao Chen updated HBASE-18479: Status: Patch Available (was: Reopened) > should apply HBASE-18255 to HBASE_MASTER_OPTS too > - > > Key: HBASE-18479 > URL: https://issues.apache.org/jira/browse/HBASE-18479 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.11, 1.2.6, 1.3.1 >Reporter: Yechao Chen >Assignee: Yechao Chen >Priority: Critical > Fix For: 1.4.0, 1.3.2, 1.2.7, 1.1.12 > > Attachments: HBASE-18479-branch-1.v1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Reopened] (HBASE-18479) should apply HBASE-18255 to HBASE_MASTER_OPTS too
[ https://issues.apache.org/jira/browse/HBASE-18479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yechao Chen reopened HBASE-18479: - not commited > should apply HBASE-18255 to HBASE_MASTER_OPTS too > - > > Key: HBASE-18479 > URL: https://issues.apache.org/jira/browse/HBASE-18479 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.1, 1.2.6, 1.1.11 >Reporter: Yechao Chen >Assignee: Yechao Chen >Priority: Critical > Fix For: 1.4.0, 1.3.2, 1.2.7, 1.1.12 > > Attachments: HBASE-18479-branch-1.v1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18565) [C++] Fix deadlock in AsyncScanRetryingCaller and other RPCs
[
https://issues.apache.org/jira/browse/HBASE-18565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122693#comment-16122693
]
Enis Soztutar commented on HBASE-18565:
---
Attaching to the process using gdb reveals these stack traces:
A lot of threads are in trying to acquire the mutex for the rpc-connection like
this:
{code}
Thread 10 (Thread 0x7febf1ffb700 (LWP 6210)):
#0 0x7fec0eea81bd in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x7fec0eea3d1d in _L_lock_840 () from /lib64/libpthread.so.0
#2 0x7fec0eea3c3a in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x7fec135fe083 in __gthread_mutex_lock (__mutex=0x7febec001a58) at
/opt/rh/devtoolset-6/root/usr/include/c++/6.2.1/x86_64-redhat-linux/bits/gthr-default.h:748
#4 0x7fec135fe0d3 in __gthread_recursive_mutex_lock
(__mutex=0x7febec001a58) at
/opt/rh/devtoolset-6/root/usr/include/c++/6.2.1/x86_64-redhat-linux/bits/gthr-default.h:810
#5 0x7fec135feb56 in std::recursive_mutex::lock (this=0x7febec001a58) at
/opt/rh/devtoolset-6/root/usr/include/c++/6.2.1/mutex:105
#6 0x7fec136005c6 in std::lock_guard::lock_guard
(this=0x7febf1ff9800, __m=...) at
/opt/rh/devtoolset-6/root/usr/include/c++/6.2.1/bits/std_mutex.h:162
#7 0x7fec135ff9e0 in hbase::RpcConnection::SendRequest
(this=0x7febec001a40, req=std::unique_ptr containing
0x7feb9c01e300) at ./connection/rpc-connection.h:43
#8 0x7fec135bfca0 in hbase::RpcClient::SendRequest (this=0xa6f890,
remote_id=std::shared_ptr (count 3, weak 0) 0x7feb9c025f40,
req=std::unique_ptr containing 0x0)
at connection/rpc-client.cc:85
#9 0x7fec135bf85c in hbase::RpcClient::AsyncCall (this=0xa6f890,
host="localhost", port=42985, req=std::unique_ptr containing
0x0,
ticket=std::shared_ptr (count 3, weak 0) 0x7feb9c01e1c0,
service_name="ClientService") at connection/rpc-client.cc:71
Python Exception There is no member or method named
_M_bbegin.:
#10 0x7fec1370e1b1 in hbase::AsyncBatchRpcRetryingCaller::GetMultiResponse
(this=0x7feb9c0036c0, actions_by_server=std::unordered_map with 1 elements) at
core/async-batch-rpc-retrying-caller.cc:313
Python Exception There is no member or method named
_M_bbegin.:
#11 0x7fec1370f048 in hbase::AsyncBatchRpcRetryingCaller::Send
(this=0x7feb9c0036c0, actions_by_server=std::unordered_map with 1 elements,
tries=1) at core/async-batch-rpc-retrying-caller.cc:343
#12 0x7fec1370d819 in
hbase::AsyncBatchRpcRetryingCaller::, std::allocator >
>&)>::operator()(std::vector, std::allocator > > &)
const (__closure=0x7feb9c001790,
loc=std::vector of length 100, capacity 100 = {...}) at
core/async-batch-rpc-retrying-caller.cc:283
#13 0x7fec137112d0 in folly::detail::CoreCallbackState&,
int32_t)::, std::allocator > >&)>
>::invoke,
std::allocator >
>&>(std::vector,
std::allocator > > &)
(this=0x7feb9c001790, args#0=std::vector of length 100, capacity 100 = {...})
at /usr/local/include/folly/futures/Future-inl.h:85
#14 0x7fec13711309 in
folly::Future,
std::allocator > >
>::, std::allocator > >
>&&)>::::operator()(void) const (__closure=0x7febf1ffa1a0)
at /usr/local/include/folly/futures/Future-inl.h:243
#15 0x7fec13712520 in
folly::makeTryWith)::
mutable [with F = hbase::AsyncBatchRpcRetryingCaller::GroupAndSend(const
std::vector&,
int32_t):: >&)>; R =
folly::detail::callableResult >, hbase::AsyncBatchRpcRetryingCaller::GroupAndSend(const
std::vector&,
int32_t):: >&)> >; bool isTry = false; Args =
{std::vector,
std::allocator > >&}; T =
std::vector >]::
>() (
f=) at /usr/local/include/folly/Try-inl.h:165
#16 0x7fec1371137b in
folly::Future,
std::allocator > >
>::, std::allocator > >
>&&)>::operator()() (__closure=0x7feb9c001790, t=) at
[jira] [Commented] (HBASE-18479) should apply HBASE-18255 to HBASE_MASTER_OPTS too
[ https://issues.apache.org/jira/browse/HBASE-18479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122692#comment-16122692 ] Yechao Chen commented on HBASE-18479: - [~tedyu] sorry the Resolved option says "has been commited",so ... but ,I see the git log ,not commited > should apply HBASE-18255 to HBASE_MASTER_OPTS too > - > > Key: HBASE-18479 > URL: https://issues.apache.org/jira/browse/HBASE-18479 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.1, 1.2.6, 1.1.11 >Reporter: Yechao Chen >Assignee: Yechao Chen >Priority: Critical > Fix For: 1.4.0, 1.3.2, 1.2.7, 1.1.12 > > Attachments: HBASE-18479-branch-1.v1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18565) [C++] Fix deadlock in AsyncScanRetryingCaller and other RPCs
Enis Soztutar created HBASE-18565: - Summary: [C++] Fix deadlock in AsyncScanRetryingCaller and other RPCs Key: HBASE-18565 URL: https://issues.apache.org/jira/browse/HBASE-18565 Project: HBase Issue Type: Sub-task Reporter: Enis Soztutar Assignee: Enis Soztutar Fix For: HBASE-14850 When running the load-client test, sometimes we get a deadlock. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18479) should apply HBASE-18255 to HBASE_MASTER_OPTS too
[ https://issues.apache.org/jira/browse/HBASE-18479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122689#comment-16122689 ] Ted Yu commented on HBASE-18479: Is the patch committed ? > should apply HBASE-18255 to HBASE_MASTER_OPTS too > - > > Key: HBASE-18479 > URL: https://issues.apache.org/jira/browse/HBASE-18479 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.1, 1.2.6, 1.1.11 >Reporter: Yechao Chen >Assignee: Yechao Chen >Priority: Critical > Fix For: 1.4.0, 1.3.2, 1.2.7, 1.1.12 > > Attachments: HBASE-18479-branch-1.v1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18564) [C++] Problems compiling with GCC
Enis Soztutar created HBASE-18564:
-
Summary: [C++] Problems compiling with GCC
Key: HBASE-18564
URL: https://issues.apache.org/jira/browse/HBASE-18564
Project: HBase
Issue Type: Sub-task
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: HBASE-14850
I am compiling with gcc-6, and getting some errors, mostly due to the lack of
{{operator bool}} in {{std::experimental::optional}} (and some other small
cases).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18551) [AMv2] UnassignProcedure and crashed regionservers
[ https://issues.apache.org/jira/browse/HBASE-18551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122672#comment-16122672 ] Hudson commented on HBASE-18551: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #3509 (See [https://builds.apache.org/job/HBase-Trunk_matrix/3509/]) HBASE-18551 [AMv2] UnassignProcedure and crashed regionservers (stack: rev 2dd75d10f8818ed31fcc36bd89024e9ad728ae41) * (edit) hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignProcedure.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionTransitionProcedure.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableNamespaceManager.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DisableTableProcedure.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/UnassignProcedure.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashException.java Revert "HBASE-18551 [AMv2] UnassignProcedure and crashed regionservers" (stack: rev e4ba404a5a65d522421e26045b8d37fbfda8) * (edit) hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionTransitionProcedure.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/UnassignProcedure.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DisableTableProcedure.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableNamespaceManager.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignProcedure.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashException.java > [AMv2] UnassignProcedure and crashed regionservers > -- > > Key: HBASE-18551 > URL: https://issues.apache.org/jira/browse/HBASE-18551 > Project: HBase > Issue Type: Bug > Components: amv2 >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: HBASE-18551.master.001.patch, > HBASE-18551.master.002.patch, HBASE-18551.master.003.patch > > > This has been [~uagashe] and my obsession over the last few days, what should > an UnassignProcedure do when it dispatches a CLOSE but the CLOSE fails > because of ConnectException or SocketTimeout. > + We used to let UnassignProcedure continue presuming the Region would be > closed since the server is dead. BUT, if the unassign was part of a > MoveProcedure, the unassign would proceed and the Move would then run WITHOUT > first splitting logs. Bad. > + So, we made it so UnassignProcedure failed; let the upper layers take care > of the failure. See HBASE-18491 that enabled this behavior. BUT, we are since > figuring that even if the UP completes as a failure, since it gives up the > Region lock on completion, another procedure -- say an AssignProcedure -- > could cut in before the ServerCrashProcedure had finished and again there > could be dataloss. > + Now we are thinking the UP should hold on to the Region lock until we are > signalled by a ServerCrashProcedure; only then let go of the region. The UP > has context that is hard to pass another. Waiting on a SCP has the UP living > on for what could be a good amount of time. It might be ok if we can suspend > the procedure. > There is a good sample scenario that came up doing the no-regions-on-master > issue, HBASE-18511. When meta is not on master, TestSplitTransactionOnCluster > is failing. It fails because though the test completes, the tests commonly > kill a
[jira] [Commented] (HBASE-18548) Move sources of important Jenkins jobs into source control
[ https://issues.apache.org/jira/browse/HBASE-18548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122671#comment-16122671 ] Hudson commented on HBASE-18548: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #3509 (See [https://builds.apache.org/job/HBase-Trunk_matrix/3509/]) HBASE-18548 Move sources of website gen and check jobs into source (misty: rev 6114824b53c1299d2d79572800fb1e1cbc96cb66) * (edit) pom.xml * (add) dev-support/jenkins-scripts/generate-hbase-website.sh * (edit) README.txt * (edit) LICENSE.txt * (edit) CHANGES.txt * (edit) src/main/asciidoc/_chapters/appendix_contributing_to_documentation.adoc * (add) dev-support/jenkins-scripts/check-website-links.sh * (edit) NOTICE.txt > Move sources of important Jenkins jobs into source control > -- > > Key: HBASE-18548 > URL: https://issues.apache.org/jira/browse/HBASE-18548 > Project: HBase > Issue Type: Bug > Components: documentation, scripts >Affects Versions: 2.0.0-alpha-1 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-18548.patch > > > Move sources of the following Jenkins jobs into source control: > https://builds.apache.org/job/hbase_generate_website/ > https://builds.apache.org/view/All/job/HBase%20Website%20Link%20Checker/ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18543) Re-enable test master.TestMasterFailover on master
[ https://issues.apache.org/jira/browse/HBASE-18543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122669#comment-16122669 ] Hudson commented on HBASE-18543: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #3509 (See [https://builds.apache.org/job/HBase-Trunk_matrix/3509/]) HBASE-18543 Disabled test TestMasterFailover (stack: rev d5f34adcdb2b8e1ab4c4d811b801f016ca81fbcd) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java > Re-enable test master.TestMasterFailover on master > -- > > Key: HBASE-18543 > URL: https://issues.apache.org/jira/browse/HBASE-18543 > Project: HBase > Issue Type: Bug > Components: amv2 >Affects Versions: 2.0.0 >Reporter: Umesh Agashe >Assignee: Umesh Agashe > Fix For: 2.0.0 > > Attachments: hbase-18543.master.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18551) [AMv2] UnassignProcedure and crashed regionservers
[ https://issues.apache.org/jira/browse/HBASE-18551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122679#comment-16122679 ] Umesh Agashe commented on HBASE-18551: -- +1 > [AMv2] UnassignProcedure and crashed regionservers > -- > > Key: HBASE-18551 > URL: https://issues.apache.org/jira/browse/HBASE-18551 > Project: HBase > Issue Type: Bug > Components: amv2 >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: HBASE-18551.master.001.patch, > HBASE-18551.master.002.patch, HBASE-18551.master.003.patch > > > This has been [~uagashe] and my obsession over the last few days, what should > an UnassignProcedure do when it dispatches a CLOSE but the CLOSE fails > because of ConnectException or SocketTimeout. > + We used to let UnassignProcedure continue presuming the Region would be > closed since the server is dead. BUT, if the unassign was part of a > MoveProcedure, the unassign would proceed and the Move would then run WITHOUT > first splitting logs. Bad. > + So, we made it so UnassignProcedure failed; let the upper layers take care > of the failure. See HBASE-18491 that enabled this behavior. BUT, we are since > figuring that even if the UP completes as a failure, since it gives up the > Region lock on completion, another procedure -- say an AssignProcedure -- > could cut in before the ServerCrashProcedure had finished and again there > could be dataloss. > + Now we are thinking the UP should hold on to the Region lock until we are > signalled by a ServerCrashProcedure; only then let go of the region. The UP > has context that is hard to pass another. Waiting on a SCP has the UP living > on for what could be a good amount of time. It might be ok if we can suspend > the procedure. > There is a good sample scenario that came up doing the no-regions-on-master > issue, HBASE-18511. When meta is not on master, TestSplitTransactionOnCluster > is failing. It fails because though the test completes, the tests commonly > kill a RegionServer. The teardown for the test runs before we've noticed the > aborted RS. So, the disable of the table in the teardown prepartory to our > deleting the test table as part of clean up, goes to unassign regions but the > unassign fails against the aborted server. > Good stuff. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18537) [C++] Improvements to load-client
[ https://issues.apache.org/jira/browse/HBASE-18537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-18537: -- Attachment: hbase-18537_v2.patch v2 patch. > [C++] Improvements to load-client > - > > Key: HBASE-18537 > URL: https://issues.apache.org/jira/browse/HBASE-18537 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: HBASE-14850 > > Attachments: hbase-18537.patch, hbase-18537_v2.patch > > > A couple of improvements to the load-client after spending some time with > testing: > - better log messages > - support for progress > - minor bug fixes -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18560) Test master.assignment.TestAssignmentManager hangs on master and its in flaky list
[ https://issues.apache.org/jira/browse/HBASE-18560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122673#comment-16122673 ] Hudson commented on HBASE-18560: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #3509 (See [https://builds.apache.org/job/HBase-Trunk_matrix/3509/]) HBASE-18560 Fixed master.assignment.TestAssignmentManager hangs on (stack: rev e98b38bf6c112c00f573896053504e8605b95122) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockNoopMasterServices.java > Test master.assignment.TestAssignmentManager hangs on master and its in flaky > list > -- > > Key: HBASE-18560 > URL: https://issues.apache.org/jira/browse/HBASE-18560 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Umesh Agashe >Assignee: Umesh Agashe > Fix For: 2.0.0 > > Attachments: hbase-18560.master.001.patch > > > Test master.assignment.TestAssignmentManager hangs on master and showing up > in flaky list -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18262) name of parameter quote need update in hbase-default.xml
[ https://issues.apache.org/jira/browse/HBASE-18262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122674#comment-16122674 ] Hudson commented on HBASE-18262: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #3509 (See [https://builds.apache.org/job/HBase-Trunk_matrix/3509/]) HBASE-18262 name of parameter quote need update (stack: rev 5507150a163f08c966f4cd55607feff8e2570c17) * (edit) hbase-common/src/main/resources/hbase-default.xml > name of parameter quote need update in hbase-default.xml > > > Key: HBASE-18262 > URL: https://issues.apache.org/jira/browse/HBASE-18262 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Dongtao Zhang >Assignee: Dongtao Zhang >Priority: Minor > Fix For: 2.0.0-alpha-2 > > Attachments: HBASE-18262-v001.patch > > > In description of parameter "hbase.zookeeper.quorum", old name > "hbase.zookeeper.clientPort " should be replaced to > "hbase.zookeeper.property.clientPort". -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18398) Snapshot operation fails with FileNotFoundException
[ https://issues.apache.org/jira/browse/HBASE-18398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122670#comment-16122670 ] Hudson commented on HBASE-18398: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #3509 (See [https://builds.apache.org/job/HBase-Trunk_matrix/3509/]) HBASE-18398: Snapshot operation fails with FileNotFoundException (ashu: rev ded0842cafbccff0ad64137abf4690c781d86e65) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/snapshot/FlushSnapshotSubprocedure.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Region.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotManifest.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestRegionSnapshotTask.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java > Snapshot operation fails with FileNotFoundException > --- > > Key: HBASE-18398 > URL: https://issues.apache.org/jira/browse/HBASE-18398 > Project: HBase > Issue Type: Sub-task > Components: snapshots >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri > Fix For: 3.0.0, 1.4.0, 1.3.2, 1.5.0, 2.0.0-alpha-2 > > Attachments: HBASE-18398.branch-1.3.001.patch, > HBASE-18398.branch-1.3.002.patch, HBASE-18398.branch-1.3.003.patch, > HBASE-18398.master.001.patch, HBASE-18398.master.002.patch, > HBASE-18398.master.003.patch, HBASE-18398.master.004.patch > > > Failing to take snapshot due to FileNotFoundException > * FlushSnapshotSubprocedure.RegionSnapshotTask takes a region level read > lock > * Call to HRegion#addRegionToSnapshot. > * Call to SnapshotManifest#addRegion. This gets the current list of store > files. > * RACE → File is marked as compacted away and HFileArchiver moves the > file to archive under store level lock. > * SnapshotManifest#addRegion visits the stale list of store files one by > one. It does a file.getStatus() call to get length of each file. Since the > file object still points to the original file, file.getStatus() fails with > FileNotFoundException. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18479) should apply HBASE-18255 to HBASE_MASTER_OPTS too
[ https://issues.apache.org/jira/browse/HBASE-18479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yechao Chen updated HBASE-18479: Resolution: Fixed Status: Resolved (was: Patch Available) > should apply HBASE-18255 to HBASE_MASTER_OPTS too > - > > Key: HBASE-18479 > URL: https://issues.apache.org/jira/browse/HBASE-18479 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.1, 1.2.6, 1.1.11 >Reporter: Yechao Chen >Assignee: Yechao Chen >Priority: Critical > Fix For: 1.4.0, 1.3.2, 1.2.7, 1.1.12 > > Attachments: HBASE-18479-branch-1.v1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17125) Inconsistent result when use filter to read data
[
https://issues.apache.org/jira/browse/HBASE-17125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122653#comment-16122653
]
Guanghao Zhang commented on HBASE-17125:
Thanks all for reviewing. Will commit it later if no other objections.
> Inconsistent result when use filter to read data
>
>
> Key: HBASE-17125
> URL: https://issues.apache.org/jira/browse/HBASE-17125
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 17125-slack-13.txt, example.diff,
> HBASE-17125.master.001.patch, HBASE-17125.master.002.patch,
> HBASE-17125.master.002.patch, HBASE-17125.master.003.patch,
> HBASE-17125.master.004.patch, HBASE-17125.master.005.patch,
> HBASE-17125.master.006.patch, HBASE-17125.master.007.patch,
> HBASE-17125.master.008.patch, HBASE-17125.master.009.patch,
> HBASE-17125.master.009.patch, HBASE-17125.master.010.patch,
> HBASE-17125.master.011.patch, HBASE-17125.master.011.patch,
> HBASE-17125.master.012.patch, HBASE-17125.master.013.patch,
> HBASE-17125.master.014.patch, HBASE-17125.master.015.patch,
> HBASE-17125.master.016.patch, HBASE-17125.master.017.patch,
> HBASE-17125.master.018.patch, HBASE-17125.master.019.patch,
> HBASE-17125.master.020.patch, HBASE-17125.master.020.patch,
> HBASE-17125.master.021.patch, HBASE-17125.master.022.patch,
> HBASE-17125.master.checkReturnedVersions.patch,
> HBASE-17125.master.no-specified-filter.patch
>
>
> Assume a cloumn's max versions is 3, then we write 4 versions of this column.
> The oldest version doesn't remove immediately. But from the user view, the
> oldest version has gone. When user use a filter to query, if the filter skip
> a new version, then the oldest version will be seen again. But after compact
> the region, then the oldest version will never been seen. So it is weird for
> user. The query will get inconsistent result before and after region
> compaction.
> The reason is matchColumn method of UserScanQueryMatcher. It first check the
> cell by filter, then check the number of versions needed. So if the filter
> skip the new version, then the oldest version will be seen again when it is
> not removed.
> Have a discussion offline with [~Apache9] and [~fenghh], now we have two
> solution for this problem. The first idea is check the number of versions
> first, then check the cell by filter. As the comment of setFilter, the filter
> is called after all tests for ttl, column match, deletes and max versions
> have been run.
> {code}
> /**
>* Apply the specified server-side filter when performing the Query.
>* Only {@link Filter#filterKeyValue(Cell)} is called AFTER all tests
>* for ttl, column match, deletes and max versions have been run.
>* @param filter filter to run on the server
>* @return this for invocation chaining
>*/
> public Query setFilter(Filter filter) {
> this.filter = filter;
> return this;
> }
> {code}
> But this idea has another problem, if a column's max version is 5 and the
> user query only need 3 versions. It first check the version's number, then
> check the cell by filter. So the cells number of the result may less than 3.
> But there are 2 versions which don't read anymore.
> So the second idea has three steps.
> 1. check by the max versions of this column
> 2. check the kv by filter
> 3. check the versions which user need.
> But this will lead the ScanQueryMatcher more complicated. And this will break
> the javadoc of Query.setFilter.
> Now we don't have a final solution for this problem. Suggestions are welcomed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18563) Fix RAT License complaint about website jenkins scripts
[
https://issues.apache.org/jira/browse/HBASE-18563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Esteban Gutierrez updated HBASE-18563:
--
Status: Patch Available (was: Open)
> Fix RAT License complaint about website jenkins scripts
> ---
>
> Key: HBASE-18563
> URL: https://issues.apache.org/jira/browse/HBASE-18563
> Project: HBase
> Issue Type: Bug
>Reporter: Esteban Gutierrez
>Priority: Trivial
> Attachments: HBASE-18563.0001.patch
>
>
> {{2 Unknown Licenses
> *
> Files with unapproved licenses:
> dev-support/jenkins-scripts/check-website-links.sh
> dev-support/jenkins-scripts/generate-hbase-website.sh
> *
> }}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18563) Fix RAT License complaint about website jenkins scripts
[
https://issues.apache.org/jira/browse/HBASE-18563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Esteban Gutierrez updated HBASE-18563:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
trivial fix, committed to master.
> Fix RAT License complaint about website jenkins scripts
> ---
>
> Key: HBASE-18563
> URL: https://issues.apache.org/jira/browse/HBASE-18563
> Project: HBase
> Issue Type: Bug
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
>Priority: Trivial
> Attachments: HBASE-18563.0001.patch
>
>
> {{2 Unknown Licenses
> *
> Files with unapproved licenses:
> dev-support/jenkins-scripts/check-website-links.sh
> dev-support/jenkins-scripts/generate-hbase-website.sh
> *
> }}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18563) Fix RAT License complaint about website jenkins scripts
[
https://issues.apache.org/jira/browse/HBASE-18563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Esteban Gutierrez updated HBASE-18563:
--
Attachment: HBASE-18563.0001.patch
> Fix RAT License complaint about website jenkins scripts
> ---
>
> Key: HBASE-18563
> URL: https://issues.apache.org/jira/browse/HBASE-18563
> Project: HBase
> Issue Type: Bug
>Reporter: Esteban Gutierrez
>Priority: Trivial
> Fix For: 3.0.0
>
> Attachments: HBASE-18563.0001.patch
>
>
> {{2 Unknown Licenses
> *
> Files with unapproved licenses:
> dev-support/jenkins-scripts/check-website-links.sh
> dev-support/jenkins-scripts/generate-hbase-website.sh
> *
> }}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18563) Fix RAT License complaint about website jenkins scripts
[
https://issues.apache.org/jira/browse/HBASE-18563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Esteban Gutierrez updated HBASE-18563:
--
Fix Version/s: 3.0.0
> Fix RAT License complaint about website jenkins scripts
> ---
>
> Key: HBASE-18563
> URL: https://issues.apache.org/jira/browse/HBASE-18563
> Project: HBase
> Issue Type: Bug
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
>Priority: Trivial
> Fix For: 3.0.0
>
> Attachments: HBASE-18563.0001.patch
>
>
> {{2 Unknown Licenses
> *
> Files with unapproved licenses:
> dev-support/jenkins-scripts/check-website-links.sh
> dev-support/jenkins-scripts/generate-hbase-website.sh
> *
> }}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HBASE-18563) Fix RAT License complaint about website jenkins scripts
[
https://issues.apache.org/jira/browse/HBASE-18563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Esteban Gutierrez reassigned HBASE-18563:
-
Assignee: Esteban Gutierrez
> Fix RAT License complaint about website jenkins scripts
> ---
>
> Key: HBASE-18563
> URL: https://issues.apache.org/jira/browse/HBASE-18563
> Project: HBase
> Issue Type: Bug
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
>Priority: Trivial
> Fix For: 3.0.0
>
> Attachments: HBASE-18563.0001.patch
>
>
> {{2 Unknown Licenses
> *
> Files with unapproved licenses:
> dev-support/jenkins-scripts/check-website-links.sh
> dev-support/jenkins-scripts/generate-hbase-website.sh
> *
> }}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18398) Snapshot operation fails with FileNotFoundException
[ https://issues.apache.org/jira/browse/HBASE-18398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122636#comment-16122636 ] Hudson commented on HBASE-18398: SUCCESS: Integrated in Jenkins build HBase-1.3-JDK8 #239 (See [https://builds.apache.org/job/HBase-1.3-JDK8/239/]) HBASE-18398: Snapshot operation fails with FileNotFoundException (ashu: rev 52c2dcbaa321705df143e1dc9dca28c849f8f9bb) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestRegionSnapshotTask.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Region.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotManifest.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/snapshot/FlushSnapshotSubprocedure.java > Snapshot operation fails with FileNotFoundException > --- > > Key: HBASE-18398 > URL: https://issues.apache.org/jira/browse/HBASE-18398 > Project: HBase > Issue Type: Sub-task > Components: snapshots >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri > Fix For: 3.0.0, 1.4.0, 1.3.2, 1.5.0, 2.0.0-alpha-2 > > Attachments: HBASE-18398.branch-1.3.001.patch, > HBASE-18398.branch-1.3.002.patch, HBASE-18398.branch-1.3.003.patch, > HBASE-18398.master.001.patch, HBASE-18398.master.002.patch, > HBASE-18398.master.003.patch, HBASE-18398.master.004.patch > > > Failing to take snapshot due to FileNotFoundException > * FlushSnapshotSubprocedure.RegionSnapshotTask takes a region level read > lock > * Call to HRegion#addRegionToSnapshot. > * Call to SnapshotManifest#addRegion. This gets the current list of store > files. > * RACE → File is marked as compacted away and HFileArchiver moves the > file to archive under store level lock. > * SnapshotManifest#addRegion visits the stale list of store files one by > one. It does a file.getStatus() call to get length of each file. Since the > file object still points to the original file, file.getStatus() fails with > FileNotFoundException. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18557) change splitable to mergeable in MergeTableRegionsProcedure
[
https://issues.apache.org/jira/browse/HBASE-18557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122625#comment-16122625
]
Hadoop QA commented on HBASE-18557:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 17m
7s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 2s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}117m
2s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}177m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:bdc94b1 |
| JIRA Issue | HBASE-18557 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881335/HBASE-18557-master-v1.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 1482b4b4fcbc 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 5507150 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/8022/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/8022/console |
| Powered by | Apache Yetus 0.4.0 http://yetus.apache.org |
This message was automatically generated.
> change splitable to mergeable in MergeTableRegionsProcedure
> ---
>
> Key: HBASE-18557
> URL: https://issues.apache.org/jira/browse/HBASE-18557
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Yi
[jira] [Created] (HBASE-18563) Fix RAT License complaint about website jenkins scripts
Esteban Gutierrez created HBASE-18563:
-
Summary: Fix RAT License complaint about website jenkins scripts
Key: HBASE-18563
URL: https://issues.apache.org/jira/browse/HBASE-18563
Project: HBase
Issue Type: Bug
Reporter: Esteban Gutierrez
Priority: Trivial
{{2 Unknown Licenses
*
Files with unapproved licenses:
dev-support/jenkins-scripts/check-website-links.sh
dev-support/jenkins-scripts/generate-hbase-website.sh
*
}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18024) HRegion#initializeRegionInternals should not re-create .hregioninfo file when the region directory no longer exists
[
https://issues.apache.org/jira/browse/HBASE-18024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122613#comment-16122613
]
Hudson commented on HBASE-18024:
SUCCESS: Integrated in Jenkins build HBase-1.2-IT #924 (See
[https://builds.apache.org/job/HBase-1.2-IT/924/])
HBASE-18024 HRegion#initializeRegionInternals should not re-create (esteban:
rev 11503fc6e65a6e350d4a17cd6031d6ae4d204bdc)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileRefresherChore.java
> HRegion#initializeRegionInternals should not re-create .hregioninfo file when
> the region directory no longer exists
> ---
>
> Key: HBASE-18024
> URL: https://issues.apache.org/jira/browse/HBASE-18024
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment, regionserver
>Affects Versions: 2.0.0, 1.4.0, 1.3.1, 1.2.5
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
> Fix For: 2.0.0, 1.4.0, 1.3.2, 1.5.0, 1.2.7
>
> Attachments: HBASE-18024.001.patch, HBASE-18024.002.patch,
> HBASE-18024.003.patch, HBASE-18024.004.patch
>
>
> When a RegionSever attempts to open a region, during initialization the RS
> tries to open the {{/data///.hregioninfo}}
> file, however if the {{.hregioninfofile}} doesn't exist, the RegionServer
> will create a new one on {{HRegionFileSystem#checkRegionInfoOnFilesystem}}. A
> side effect of that tools like hbck will incorrectly assume an inconsistency
> due the presence of this new {{.hregioninfofile}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18398) Snapshot operation fails with FileNotFoundException
[ https://issues.apache.org/jira/browse/HBASE-18398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122609#comment-16122609 ] Hudson commented on HBASE-18398: FAILURE: Integrated in Jenkins build HBase-1.4 #844 (See [https://builds.apache.org/job/HBase-1.4/844/]) HBASE-18398: Snapshot operation fails with FileNotFoundException (ashu: rev cab492d34f6441d11b5f7e154ee5cfe9b44bdf8e) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Region.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotManifest.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/snapshot/FlushSnapshotSubprocedure.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestRegionSnapshotTask.java > Snapshot operation fails with FileNotFoundException > --- > > Key: HBASE-18398 > URL: https://issues.apache.org/jira/browse/HBASE-18398 > Project: HBase > Issue Type: Sub-task > Components: snapshots >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri > Fix For: 3.0.0, 1.4.0, 1.3.2, 1.5.0, 2.0.0-alpha-2 > > Attachments: HBASE-18398.branch-1.3.001.patch, > HBASE-18398.branch-1.3.002.patch, HBASE-18398.branch-1.3.003.patch, > HBASE-18398.master.001.patch, HBASE-18398.master.002.patch, > HBASE-18398.master.003.patch, HBASE-18398.master.004.patch > > > Failing to take snapshot due to FileNotFoundException > * FlushSnapshotSubprocedure.RegionSnapshotTask takes a region level read > lock > * Call to HRegion#addRegionToSnapshot. > * Call to SnapshotManifest#addRegion. This gets the current list of store > files. > * RACE → File is marked as compacted away and HFileArchiver moves the > file to archive under store level lock. > * SnapshotManifest#addRegion visits the stale list of store files one by > one. It does a file.getStatus() call to get length of each file. Since the > file object still points to the original file, file.getStatus() fails with > FileNotFoundException. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18024) HRegion#initializeRegionInternals should not re-create .hregioninfo file when the region directory no longer exists
[
https://issues.apache.org/jira/browse/HBASE-18024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122605#comment-16122605
]
Hudson commented on HBASE-18024:
FAILURE: Integrated in Jenkins build HBase-1.2-JDK7 #189 (See
[https://builds.apache.org/job/HBase-1.2-JDK7/189/])
HBASE-18024 HRegion#initializeRegionInternals should not re-create (esteban:
rev 11503fc6e65a6e350d4a17cd6031d6ae4d204bdc)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileRefresherChore.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
> HRegion#initializeRegionInternals should not re-create .hregioninfo file when
> the region directory no longer exists
> ---
>
> Key: HBASE-18024
> URL: https://issues.apache.org/jira/browse/HBASE-18024
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment, regionserver
>Affects Versions: 2.0.0, 1.4.0, 1.3.1, 1.2.5
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
> Fix For: 2.0.0, 1.4.0, 1.3.2, 1.5.0, 1.2.7
>
> Attachments: HBASE-18024.001.patch, HBASE-18024.002.patch,
> HBASE-18024.003.patch, HBASE-18024.004.patch
>
>
> When a RegionSever attempts to open a region, during initialization the RS
> tries to open the {{/data///.hregioninfo}}
> file, however if the {{.hregioninfofile}} doesn't exist, the RegionServer
> will create a new one on {{HRegionFileSystem#checkRegionInfoOnFilesystem}}. A
> side effect of that tools like hbck will incorrectly assume an inconsistency
> due the presence of this new {{.hregioninfofile}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18024) HRegion#initializeRegionInternals should not re-create .hregioninfo file when the region directory no longer exists
[
https://issues.apache.org/jira/browse/HBASE-18024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122592#comment-16122592
]
Hudson commented on HBASE-18024:
SUCCESS: Integrated in Jenkins build HBase-1.3-IT #163 (See
[https://builds.apache.org/job/HBase-1.3-IT/163/])
HBASE-18024 HRegion#initializeRegionInternals should not re-create (esteban:
rev 77847da928f9ee79674f54f8dc517fdbde35287f)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionOpen.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileRefresherChore.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> HRegion#initializeRegionInternals should not re-create .hregioninfo file when
> the region directory no longer exists
> ---
>
> Key: HBASE-18024
> URL: https://issues.apache.org/jira/browse/HBASE-18024
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment, regionserver
>Affects Versions: 2.0.0, 1.4.0, 1.3.1, 1.2.5
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
> Fix For: 2.0.0, 1.4.0, 1.3.2, 1.5.0, 1.2.7
>
> Attachments: HBASE-18024.001.patch, HBASE-18024.002.patch,
> HBASE-18024.003.patch, HBASE-18024.004.patch
>
>
> When a RegionSever attempts to open a region, during initialization the RS
> tries to open the {{/data///.hregioninfo}}
> file, however if the {{.hregioninfofile}} doesn't exist, the RegionServer
> will create a new one on {{HRegionFileSystem#checkRegionInfoOnFilesystem}}. A
> side effect of that tools like hbck will incorrectly assume an inconsistency
> due the presence of this new {{.hregioninfofile}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18024) HRegion#initializeRegionInternals should not re-create .hregioninfo file when the region directory no longer exists
[
https://issues.apache.org/jira/browse/HBASE-18024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Esteban Gutierrez updated HBASE-18024:
--
Resolution: Fixed
Fix Version/s: 1.2.7
1.5.0
1.3.2
1.4.0
2.0.0
Status: Resolved (was: Patch Available)
> HRegion#initializeRegionInternals should not re-create .hregioninfo file when
> the region directory no longer exists
> ---
>
> Key: HBASE-18024
> URL: https://issues.apache.org/jira/browse/HBASE-18024
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment, regionserver
>Affects Versions: 2.0.0, 1.4.0, 1.3.1, 1.2.5
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
> Fix For: 2.0.0, 1.4.0, 1.3.2, 1.5.0, 1.2.7
>
> Attachments: HBASE-18024.001.patch, HBASE-18024.002.patch,
> HBASE-18024.003.patch, HBASE-18024.004.patch
>
>
> When a RegionSever attempts to open a region, during initialization the RS
> tries to open the {{/data///.hregioninfo}}
> file, however if the {{.hregioninfofile}} doesn't exist, the RegionServer
> will create a new one on {{HRegionFileSystem#checkRegionInfoOnFilesystem}}. A
> side effect of that tools like hbck will incorrectly assume an inconsistency
> due the presence of this new {{.hregioninfofile}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18024) HRegion#initializeRegionInternals should not re-create .hregioninfo file when the region directory no longer exists
[
https://issues.apache.org/jira/browse/HBASE-18024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122565#comment-16122565
]
Hudson commented on HBASE-18024:
FAILURE: Integrated in Jenkins build HBase-1.3-JDK7 #226 (See
[https://builds.apache.org/job/HBase-1.3-JDK7/226/])
HBASE-18024 HRegion#initializeRegionInternals should not re-create (esteban:
rev 77847da928f9ee79674f54f8dc517fdbde35287f)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileRefresherChore.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionOpen.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
> HRegion#initializeRegionInternals should not re-create .hregioninfo file when
> the region directory no longer exists
> ---
>
> Key: HBASE-18024
> URL: https://issues.apache.org/jira/browse/HBASE-18024
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment, regionserver
>Affects Versions: 2.0.0, 1.4.0, 1.3.1, 1.2.5
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
> Fix For: 2.0.0, 1.4.0, 1.3.2, 1.5.0, 1.2.7
>
> Attachments: HBASE-18024.001.patch, HBASE-18024.002.patch,
> HBASE-18024.003.patch, HBASE-18024.004.patch
>
>
> When a RegionSever attempts to open a region, during initialization the RS
> tries to open the {{/data///.hregioninfo}}
> file, however if the {{.hregioninfofile}} doesn't exist, the RegionServer
> will create a new one on {{HRegionFileSystem#checkRegionInfoOnFilesystem}}. A
> side effect of that tools like hbck will incorrectly assume an inconsistency
> due the presence of this new {{.hregioninfofile}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18551) [AMv2] UnassignProcedure and crashed regionservers
[ https://issues.apache.org/jira/browse/HBASE-18551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122560#comment-16122560 ] stack commented on HBASE-18551: --- .003 addresses [~uagashe] reviews up on rb. > [AMv2] UnassignProcedure and crashed regionservers > -- > > Key: HBASE-18551 > URL: https://issues.apache.org/jira/browse/HBASE-18551 > Project: HBase > Issue Type: Bug > Components: amv2 >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: HBASE-18551.master.001.patch, > HBASE-18551.master.002.patch, HBASE-18551.master.003.patch > > > This has been [~uagashe] and my obsession over the last few days, what should > an UnassignProcedure do when it dispatches a CLOSE but the CLOSE fails > because of ConnectException or SocketTimeout. > + We used to let UnassignProcedure continue presuming the Region would be > closed since the server is dead. BUT, if the unassign was part of a > MoveProcedure, the unassign would proceed and the Move would then run WITHOUT > first splitting logs. Bad. > + So, we made it so UnassignProcedure failed; let the upper layers take care > of the failure. See HBASE-18491 that enabled this behavior. BUT, we are since > figuring that even if the UP completes as a failure, since it gives up the > Region lock on completion, another procedure -- say an AssignProcedure -- > could cut in before the ServerCrashProcedure had finished and again there > could be dataloss. > + Now we are thinking the UP should hold on to the Region lock until we are > signalled by a ServerCrashProcedure; only then let go of the region. The UP > has context that is hard to pass another. Waiting on a SCP has the UP living > on for what could be a good amount of time. It might be ok if we can suspend > the procedure. > There is a good sample scenario that came up doing the no-regions-on-master > issue, HBASE-18511. When meta is not on master, TestSplitTransactionOnCluster > is failing. It fails because though the test completes, the tests commonly > kill a RegionServer. The teardown for the test runs before we've noticed the > aborted RS. So, the disable of the table in the teardown prepartory to our > deleting the test table as part of clean up, goes to unassign regions but the > unassign fails against the aborted server. > Good stuff. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
