[jira] [Updated] (HBASE-18113) Handle old client without include_stop_row flag when startRow equals endRow
[ https://issues.apache.org/jira/browse/HBASE-18113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-18113: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to master and branch-1. Thanks [~Apache9] for review. > Handle old client without include_stop_row flag when startRow equals endRow > --- > > Key: HBASE-18113 > URL: https://issues.apache.org/jira/browse/HBASE-18113 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.4.0 >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-18113.v02.patch, HBASE-18113.v02.patch, > HBASE-18113.v03.patch > > > Now we have include_start_row/include_stop_row flag in new version. Before > it, startRow is include and stopRow is exclude, but when startRow=endRow > there is a special logic that we consider it as a get and we should return > the value of this row to client. > In the new client, if user set start=end we will change include_stop_row to > true for behavior compatibility. > We should also do a special logic if old client access new server for > compatibility at server. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18118) Default storage policy if not configured cannot be "NONE"
[ https://issues.apache.org/jira/browse/HBASE-18118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025838#comment-16025838 ] Yu Li commented on HBASE-18118: --- +1 > Default storage policy if not configured cannot be "NONE" > - > > Key: HBASE-18118 > URL: https://issues.apache.org/jira/browse/HBASE-18118 > Project: HBase > Issue Type: Bug > Components: wal >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-18118.patch > > > HBase can't use 'NONE' as default storage policy if not configured because > HDFS supports no such policy. This policy name was probably available in a > precommit or early version of the HDFS side support for heterogeneous > storage. Now the best default is 'HOT'. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16011) TableSnapshotScanner and TableSnapshotInputFormat can produce duplicate rows
[ https://issues.apache.org/jira/browse/HBASE-16011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025834#comment-16025834 ] Hudson commented on HBASE-16011: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #3077 (See [https://builds.apache.org/job/HBase-Trunk_matrix/3077/]) HBASE-16011 TableSnapshotScanner and TableSnapshotInputFormat can (tedyu: rev f441ca0458b40556a64d564cf2d4aa15a0c0be1e) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/client/TableSnapshotScanner.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/SnapshotTestingUtils.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableSnapshotInputFormat.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestTableSnapshotScanner.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormatImpl.java > TableSnapshotScanner and TableSnapshotInputFormat can produce duplicate rows > > > Key: HBASE-16011 > URL: https://issues.apache.org/jira/browse/HBASE-16011 > Project: HBase > Issue Type: Bug > Components: snapshots >Affects Versions: 2.0.0, 1.2.2 >Reporter: Youngjoon Kim >Assignee: Zheng Hu > Attachments: HBASE-16011.v1.patch, HBASE-16011.v2.patch, > HBASE-16011.v2.patch, snapshot_bug_test.patch > > > A snapshot of (non-pre) split table can include both a parent region and > daughter regions. If run TableSnapshotScanner or TableSnapshotInputFormat on > the such snapshot, duplicate rows are produced. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18078) [C++] Harden RPC by handling various communication abnormalities
[
https://issues.apache.org/jira/browse/HBASE-18078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025824#comment-16025824
]
Xiaobing Zhou commented on HBASE-18078:
---
Coming next are:
# apply the APIs (e.g. ConnectionFactory and ConnectionPool) changes to all
related places.
# In order to simulate various abnormal behaviors, it necessitates building a
TestRpcServer, which can be started from samples in
[BootstrapTest|https://github.com/facebook/wangle/blob/master/wangle/bootstrap/BootstrapTest.cpp]
# handle other abnormal cases such as defined in folly::AsyncSocketException
{code}
class AsyncSocketException : public std::runtime_error {
public:
enum AsyncSocketExceptionType {
UNKNOWN = 0,
NOT_OPEN = 1,
ALREADY_OPEN = 2,
TIMED_OUT = 3,
END_OF_FILE = 4,
INTERRUPTED = 5,
BAD_ARGS = 6,
CORRUPTED_DATA = 7,
INTERNAL_ERROR = 8,
NOT_SUPPORTED = 9,
INVALID_STATE = 10,
SSL_ERROR = 12,
COULD_NOT_BIND = 13,
SASL_HANDSHAKE_TIMEOUT = 14,
NETWORK_ERROR = 15
};
{code}
# add unit tests to cover them.
> [C++] Harden RPC by handling various communication abnormalities
>
>
> Key: HBASE-18078
> URL: https://issues.apache.org/jira/browse/HBASE-18078
> Project: HBase
> Issue Type: Sub-task
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HBASE-18078.000.patch
>
>
> RPC layer should handle various communication abnormalities (e.g. connection
> timeout, server aborted connection, and so on). Ideally, the corresponding
> exceptions should be raised and propagated through handlers of pipeline in
> client.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18078) [C++] Harden RPC by handling various communication abnormalities
[
https://issues.apache.org/jira/browse/HBASE-18078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025812#comment-16025812
]
Xiaobing Zhou commented on HBASE-18078:
---
Posted v0 patch to handle connection related issues, such as timeout.
The idea is to catch and propagate ConnectionException up. Therefore, from
bottom of stack, whatever issues from establishing connection (i.e. Wangle
pipeline) will be folly::AsyncSocketException wrapped within
ConnectionException.
Note that there's no way to set socket options in Wangle ClientBootstrap, but
fortunately, ClientBootstrap::connect supports setting our own timeout which is
enough for now.
See also ConnectionFactory::Connect
{code}
try {
/* any connection error (e.g. timeout) will be
folly::AsyncSocketException */
auto pipeline = client->connect(
SocketAddress(hostname, port, true),
std::chrono::duration_cast(connect_timeout_)).get();
auto dispatcher = std::make_shared();
dispatcher->setPipeline(pipeline);
promise.setValue(dispatcher);
} catch(const folly::AsyncSocketException ) {
promise.setException(
folly::make_exception_wrapper(
folly::make_exception_wrapper(e)));
}
{code}
> [C++] Harden RPC by handling various communication abnormalities
>
>
> Key: HBASE-18078
> URL: https://issues.apache.org/jira/browse/HBASE-18078
> Project: HBase
> Issue Type: Sub-task
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HBASE-18078.000.patch
>
>
> RPC layer should handle various communication abnormalities (e.g. connection
> timeout, server aborted connection, and so on). Ideally, the corresponding
> exceptions should be raised and propagated through handlers of pipeline in
> client.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-18078) [C++] Harden RPC by handling various communication abnormalities
[ https://issues.apache.org/jira/browse/HBASE-18078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HBASE-18078: -- Attachment: (was: HBASE-18078.000.patch) > [C++] Harden RPC by handling various communication abnormalities > > > Key: HBASE-18078 > URL: https://issues.apache.org/jira/browse/HBASE-18078 > Project: HBase > Issue Type: Sub-task >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HBASE-18078.000.patch > > > RPC layer should handle various communication abnormalities (e.g. connection > timeout, server aborted connection, and so on). Ideally, the corresponding > exceptions should be raised and propagated through handlers of pipeline in > client. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18078) [C++] Harden RPC by handling various communication abnormalities
[ https://issues.apache.org/jira/browse/HBASE-18078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HBASE-18078: -- Attachment: HBASE-18078.000.patch > [C++] Harden RPC by handling various communication abnormalities > > > Key: HBASE-18078 > URL: https://issues.apache.org/jira/browse/HBASE-18078 > Project: HBase > Issue Type: Sub-task >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HBASE-18078.000.patch > > > RPC layer should handle various communication abnormalities (e.g. connection > timeout, server aborted connection, and so on). Ideally, the corresponding > exceptions should be raised and propagated through handlers of pipeline in > client. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18078) [C++] Harden RPC by handling various communication abnormalities
[ https://issues.apache.org/jira/browse/HBASE-18078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HBASE-18078: -- Attachment: (was: HBASE-18078.000.patch) > [C++] Harden RPC by handling various communication abnormalities > > > Key: HBASE-18078 > URL: https://issues.apache.org/jira/browse/HBASE-18078 > Project: HBase > Issue Type: Sub-task >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HBASE-18078.000.patch > > > RPC layer should handle various communication abnormalities (e.g. connection > timeout, server aborted connection, and so on). Ideally, the corresponding > exceptions should be raised and propagated through handlers of pipeline in > client. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18078) [C++] Harden RPC by handling various communication abnormalities
[ https://issues.apache.org/jira/browse/HBASE-18078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HBASE-18078: -- Attachment: HBASE-18078.000.patch > [C++] Harden RPC by handling various communication abnormalities > > > Key: HBASE-18078 > URL: https://issues.apache.org/jira/browse/HBASE-18078 > Project: HBase > Issue Type: Sub-task >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HBASE-18078.000.patch > > > RPC layer should handle various communication abnormalities (e.g. connection > timeout, server aborted connection, and so on). Ideally, the corresponding > exceptions should be raised and propagated through handlers of pipeline in > client. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18078) [C++] Harden RPC by handling various communication abnormalities
[ https://issues.apache.org/jira/browse/HBASE-18078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HBASE-18078: -- Description: RPC layer should handle various communication abnormalities (e.g. connection timeout, server aborted connection, and so on). Ideally, the corresponding exceptions should be raised and propagated through handlers of pipeline in client. was: RPC layer should handle various communication abnormality (e.g. connection timeout, server aborted connection, and so on). Ideally, the corresponding exceptions should be raised and propagated through handlers of pipeline in client. > [C++] Harden RPC by handling various communication abnormalities > > > Key: HBASE-18078 > URL: https://issues.apache.org/jira/browse/HBASE-18078 > Project: HBase > Issue Type: Sub-task >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HBASE-18078.000.patch > > > RPC layer should handle various communication abnormalities (e.g. connection > timeout, server aborted connection, and so on). Ideally, the corresponding > exceptions should be raised and propagated through handlers of pipeline in > client. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18078) [C++] Harden RPC by handling various communication abnormalities
[ https://issues.apache.org/jira/browse/HBASE-18078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HBASE-18078: -- Summary: [C++] Harden RPC by handling various communication abnormalities (was: [C++] Implement RPC timeout) > [C++] Harden RPC by handling various communication abnormalities > > > Key: HBASE-18078 > URL: https://issues.apache.org/jira/browse/HBASE-18078 > Project: HBase > Issue Type: Sub-task >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HBASE-18078.000.patch > > > RPC layer should handle various communication abnormality (e.g. connection > timeout, server aborted connection, and so on). Ideally, the corresponding > exceptions should be raised and propagated through handlers of pipeline in > client. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18078) [C++] Implement RPC timeout
[ https://issues.apache.org/jira/browse/HBASE-18078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HBASE-18078: -- Description: RPC layer should handle various communication abnormality (e.g. connection timeout, server aborted connection, and so on). Ideally, the corresponding exceptions should be raised and propagated through handlers of pipeline in client. was: RPC layer should handle various communication abnormality (e.g. connection timeout, server aborted connection, and so on). Ideally, exceptions should be raised and propagated through handlers of pipeline in client. > [C++] Implement RPC timeout > --- > > Key: HBASE-18078 > URL: https://issues.apache.org/jira/browse/HBASE-18078 > Project: HBase > Issue Type: Sub-task >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HBASE-18078.000.patch > > > RPC layer should handle various communication abnormality (e.g. connection > timeout, server aborted connection, and so on). Ideally, the corresponding > exceptions should be raised and propagated through handlers of pipeline in > client. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18078) [C++] Implement RPC timeout
[ https://issues.apache.org/jira/browse/HBASE-18078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HBASE-18078: -- Description: RPC layer should handle various communication abnormality (e.g. connection timeout, server aborted connection, and so on). Ideally, exceptions should be raised and propagated through handlers of pipeline in client. was: RPC layer should handle various connection abnormality (e.g. server aborted connection) through timeout. Ideally, exceptions should be raised and propagated through handlers of pipeline in client. > [C++] Implement RPC timeout > --- > > Key: HBASE-18078 > URL: https://issues.apache.org/jira/browse/HBASE-18078 > Project: HBase > Issue Type: Sub-task >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HBASE-18078.000.patch > > > RPC layer should handle various communication abnormality (e.g. connection > timeout, server aborted connection, and so on). Ideally, exceptions should be > raised and propagated through handlers of pipeline in client. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18113) Handle old client without include_stop_row flag when startRow equals endRow
[
https://issues.apache.org/jira/browse/HBASE-18113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025799#comment-16025799
]
Hadoop QA commented on HBASE-18113:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
20s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 6s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
36m 4s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 37s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 49m 25s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:757bf37 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12869991/HBASE-18113.v03.patch
|
| JIRA Issue | HBASE-18113 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 3322641fdaa9 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / f441ca0 |
| Default Java | 1.8.0_131 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6964/testReport/ |
| modules | C: hbase-client U: hbase-client |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6964/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Handle old client without include_stop_row flag when startRow equals endRow
> ---
>
> Key: HBASE-18113
> URL: https://issues.apache.org/jira/browse/HBASE-18113
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.4.0
>
[jira] [Commented] (HBASE-18113) Handle old client without include_stop_row flag when startRow equals endRow
[ https://issues.apache.org/jira/browse/HBASE-18113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025772#comment-16025772 ] Duo Zhang commented on HBASE-18113: --- +1. > Handle old client without include_stop_row flag when startRow equals endRow > --- > > Key: HBASE-18113 > URL: https://issues.apache.org/jira/browse/HBASE-18113 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.4.0 >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-18113.v02.patch, HBASE-18113.v02.patch, > HBASE-18113.v03.patch > > > Now we have include_start_row/include_stop_row flag in new version. Before > it, startRow is include and stopRow is exclude, but when startRow=endRow > there is a special logic that we consider it as a get and we should return > the value of this row to client. > In the new client, if user set start=end we will change include_stop_row to > true for behavior compatibility. > We should also do a special logic if old client access new server for > compatibility at server. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18042) Client Compatibility breaks between versions 1.2 and 1.3
[
https://issues.apache.org/jira/browse/HBASE-18042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025769#comment-16025769
]
Duo Zhang commented on HBASE-18042:
---
OK, now branch-1.3 is also failed...
We need to find a solution...
[~busbey]
> Client Compatibility breaks between versions 1.2 and 1.3
>
>
> Key: HBASE-18042
> URL: https://issues.apache.org/jira/browse/HBASE-18042
> Project: HBase
> Issue Type: Bug
> Components: regionserver, scan
>Affects Versions: 2.0.0, 1.4.0, 1.3.1
>Reporter: Karan Mehta
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0, 1.4.0, 1.3.2
>
> Attachments: HBASE-18042-branch-1.3.patch,
> HBASE-18042-branch-1.3-v1.patch, HBASE-18042-branch-1.patch,
> HBASE-18042-branch-1.patch, HBASE-18042-branch-1-v1.patch,
> HBASE-18042-branch-1-v1.patch, HBASE-18042.patch, HBASE-18042-v1.patch,
> HBASE-18042-v2.patch
>
>
> OpenTSDB uses AsyncHBase as its client, rather than using the traditional
> HBase Client. From version 1.2 to 1.3, the {{ClientProtos}} have been
> changed. Newer fields are added to {{ScanResponse}} proto.
> For a typical Scan request in 1.2, would require caller to make an
> OpenScanner Request, GetNextRows Request and a CloseScanner Request, based on
> {{more_rows}} boolean field in the {{ScanResponse}} proto.
> However, from 1.3, new parameter {{more_results_in_region}} was added, which
> limits the results per region. Therefore the client has to now manage sending
> all the requests for each region. Further more, if the results are exhausted
> from a particular region, the {{ScanResponse}} will set
> {{more_results_in_region}} to false, but {{more_results}} can still be true.
> Whenever the former is set to false, the {{RegionScanner}} will also be
> closed.
> OpenTSDB makes an OpenScanner Request and receives all its results in the
> first {{ScanResponse}} itself, thus creating a condition as described in
> above paragraph. Since {{more_rows}} is true, it will proceed to send next
> request at which point the {{RSRpcServices}} will throw
> {{UnknownScannerException}}. The protobuf client compatibility is maintained
> but expected behavior is modified.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-18113) Handle old client without include_stop_row flag when startRow equals endRow
[ https://issues.apache.org/jira/browse/HBASE-18113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-18113: -- Attachment: HBASE-18113.v03.patch Add a new ClientUtil > Handle old client without include_stop_row flag when startRow equals endRow > --- > > Key: HBASE-18113 > URL: https://issues.apache.org/jira/browse/HBASE-18113 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.4.0 >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-18113.v02.patch, HBASE-18113.v02.patch, > HBASE-18113.v03.patch > > > Now we have include_start_row/include_stop_row flag in new version. Before > it, startRow is include and stopRow is exclude, but when startRow=endRow > there is a special logic that we consider it as a get and we should return > the value of this row to client. > In the new client, if user set start=end we will change include_stop_row to > true for behavior compatibility. > We should also do a special logic if old client access new server for > compatibility at server. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18042) Client Compatibility breaks between versions 1.2 and 1.3
[
https://issues.apache.org/jira/browse/HBASE-18042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025765#comment-16025765
]
Hadoop QA commented on HBASE-18042:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 0m 24s
{color} | {color:red} Docker failed to build yetus/hbase:e1e11ad. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12869990/HBASE-18042-branch-1.3-v1.patch
|
| JIRA Issue | HBASE-18042 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6963/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Client Compatibility breaks between versions 1.2 and 1.3
>
>
> Key: HBASE-18042
> URL: https://issues.apache.org/jira/browse/HBASE-18042
> Project: HBase
> Issue Type: Bug
> Components: regionserver, scan
>Affects Versions: 2.0.0, 1.4.0, 1.3.1
>Reporter: Karan Mehta
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0, 1.4.0, 1.3.2
>
> Attachments: HBASE-18042-branch-1.3.patch,
> HBASE-18042-branch-1.3-v1.patch, HBASE-18042-branch-1.patch,
> HBASE-18042-branch-1.patch, HBASE-18042-branch-1-v1.patch,
> HBASE-18042-branch-1-v1.patch, HBASE-18042.patch, HBASE-18042-v1.patch,
> HBASE-18042-v2.patch
>
>
> OpenTSDB uses AsyncHBase as its client, rather than using the traditional
> HBase Client. From version 1.2 to 1.3, the {{ClientProtos}} have been
> changed. Newer fields are added to {{ScanResponse}} proto.
> For a typical Scan request in 1.2, would require caller to make an
> OpenScanner Request, GetNextRows Request and a CloseScanner Request, based on
> {{more_rows}} boolean field in the {{ScanResponse}} proto.
> However, from 1.3, new parameter {{more_results_in_region}} was added, which
> limits the results per region. Therefore the client has to now manage sending
> all the requests for each region. Further more, if the results are exhausted
> from a particular region, the {{ScanResponse}} will set
> {{more_results_in_region}} to false, but {{more_results}} can still be true.
> Whenever the former is set to false, the {{RegionScanner}} will also be
> closed.
> OpenTSDB makes an OpenScanner Request and receives all its results in the
> first {{ScanResponse}} itself, thus creating a condition as described in
> above paragraph. Since {{more_rows}} is true, it will proceed to send next
> request at which point the {{RSRpcServices}} will throw
> {{UnknownScannerException}}. The protobuf client compatibility is maintained
> but expected behavior is modified.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-18042) Client Compatibility breaks between versions 1.2 and 1.3
[
https://issues.apache.org/jira/browse/HBASE-18042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-18042:
--
Attachment: HBASE-18042-branch-1.3-v1.patch
> Client Compatibility breaks between versions 1.2 and 1.3
>
>
> Key: HBASE-18042
> URL: https://issues.apache.org/jira/browse/HBASE-18042
> Project: HBase
> Issue Type: Bug
> Components: regionserver, scan
>Affects Versions: 2.0.0, 1.4.0, 1.3.1
>Reporter: Karan Mehta
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0, 1.4.0, 1.3.2
>
> Attachments: HBASE-18042-branch-1.3.patch,
> HBASE-18042-branch-1.3-v1.patch, HBASE-18042-branch-1.patch,
> HBASE-18042-branch-1.patch, HBASE-18042-branch-1-v1.patch,
> HBASE-18042-branch-1-v1.patch, HBASE-18042.patch, HBASE-18042-v1.patch,
> HBASE-18042-v2.patch
>
>
> OpenTSDB uses AsyncHBase as its client, rather than using the traditional
> HBase Client. From version 1.2 to 1.3, the {{ClientProtos}} have been
> changed. Newer fields are added to {{ScanResponse}} proto.
> For a typical Scan request in 1.2, would require caller to make an
> OpenScanner Request, GetNextRows Request and a CloseScanner Request, based on
> {{more_rows}} boolean field in the {{ScanResponse}} proto.
> However, from 1.3, new parameter {{more_results_in_region}} was added, which
> limits the results per region. Therefore the client has to now manage sending
> all the requests for each region. Further more, if the results are exhausted
> from a particular region, the {{ScanResponse}} will set
> {{more_results_in_region}} to false, but {{more_results}} can still be true.
> Whenever the former is set to false, the {{RegionScanner}} will also be
> closed.
> OpenTSDB makes an OpenScanner Request and receives all its results in the
> first {{ScanResponse}} itself, thus creating a condition as described in
> above paragraph. Since {{more_rows}} is true, it will proceed to send next
> request at which point the {{RSRpcServices}} will throw
> {{UnknownScannerException}}. The protobuf client compatibility is maintained
> but expected behavior is modified.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-15160) Put back HFile's HDFS op latency sampling code and add metrics for monitoring
[ https://issues.apache.org/jira/browse/HBASE-15160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025754#comment-16025754 ] Yu Li commented on HBASE-15160: --- Thanks for reviving this [~enis], will check the patch and try the YCSB tests ASAP bq. BTW, these metrics would have saved us days worth of debugging in a recent case, so let's get this patch in one way or the other. Understood, the same in our case and it's the exact reason we already put this (a customized version) online for one and a half years (smile). > Put back HFile's HDFS op latency sampling code and add metrics for monitoring > - > > Key: HBASE-15160 > URL: https://issues.apache.org/jira/browse/HBASE-15160 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0, 1.1.2 >Reporter: Yu Li >Assignee: Yu Li >Priority: Critical > Attachments: HBASE-15160.patch, HBASE-15160_v2.patch, > HBASE-15160_v3.patch, hbase-15160_v4.patch, hbase-15160_v5.patch, > hbase-15160_v6.patch > > > In HBASE-11586 all HDFS op latency sampling code, including fsReadLatency, > fsPreadLatency and fsWriteLatency, have been removed. There was some > discussion about putting them back in a new JIRA but never happened. > According to our experience, these metrics are useful to judge whether issue > lies on HDFS when slow request occurs, so we propose to put them back in this > JIRA, and add the metrics for monitoring as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16261) MultiHFileOutputFormat Enhancement
[
https://issues.apache.org/jira/browse/HBASE-16261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025744#comment-16025744
]
Hadoop QA commented on HBASE-16261:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 26s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 20s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 9s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
58m 9s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 215m 15s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
54s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 299m 39s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestAsyncProcedureAdminApi |
| | hadoop.hbase.client.TestAsyncRegionAdminApi |
| | hadoop.hbase.util.TestHBaseFsckTwoRS |
| | hadoop.hbase.client.TestAsyncBalancerAdminApi |
| | hadoop.hbase.client.TestAsyncTableAdminApi |
| | hadoop.hbase.client.TestBlockEvictionFromClient |
| Timed out junit tests | org.apache.hadoop.hbase.master.TestMasterFailover |
| | org.apache.hadoop.hbase.client.TestMultiRespectsLimits |
| | org.apache.hadoop.hbase.master.TestDistributedLogSplitting |
| | org.apache.hadoop.hbase.snapshot.TestMobSecureExportSnapshot |
| | org.apache.hadoop.hbase.master.TestMasterOperationsForRegionReplicas |
| | org.apache.hadoop.hbase.util.TestHBaseFsckTwoRS |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.03.0-ce Server=17.03.0-ce Image:yetus/hbase:757bf37 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12869942/HBase-16261-V9.patch |
| JIRA Issue | HBASE-16261 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 59a5b0b6e526 4.8.3-std-1 #1 SMP Fri Oct 21 11:15:43 UTC 2016
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / f441ca0 |
| Default Java | 1.8.0_131 |
| findbugs | v3.0.0 |
| unit |
[jira] [Updated] (HBASE-18119) Improve HFile readability and modify ChecksumUtil log level
[ https://issues.apache.org/jira/browse/HBASE-18119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qilin Cao updated HBASE-18119: -- Affects Version/s: 2.0.0 Status: Patch Available (was: Open) > Improve HFile readability and modify ChecksumUtil log level > --- > > Key: HBASE-18119 > URL: https://issues.apache.org/jira/browse/HBASE-18119 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Qilin Cao >Assignee: Qilin Cao >Priority: Minor > Attachments: HBASE-18119-v1.patch > > > It is confused when I read the HFile.checkHFileVersion method, so I change > the source code. Simultaneously, I change ChecksumUtil the info log level to > trace. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18119) Improve HFile readability and modify ChecksumUtil log level
[ https://issues.apache.org/jira/browse/HBASE-18119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qilin Cao updated HBASE-18119: -- Attachment: HBASE-18119-v1.patch Submit a patch! > Improve HFile readability and modify ChecksumUtil log level > --- > > Key: HBASE-18119 > URL: https://issues.apache.org/jira/browse/HBASE-18119 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Qilin Cao >Assignee: Qilin Cao >Priority: Minor > Attachments: HBASE-18119-v1.patch > > > It is confused when I read the HFile.checkHFileVersion method, so I change > the source code. Simultaneously, I change ChecksumUtil the info log level to > trace. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HBASE-18119) Improve HFile readability and modify ChecksumUtil log level
[ https://issues.apache.org/jira/browse/HBASE-18119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qilin Cao reassigned HBASE-18119: - Assignee: Qilin Cao > Improve HFile readability and modify ChecksumUtil log level > --- > > Key: HBASE-18119 > URL: https://issues.apache.org/jira/browse/HBASE-18119 > Project: HBase > Issue Type: Improvement >Reporter: Qilin Cao >Assignee: Qilin Cao >Priority: Minor > > It is confused when I read the HFile.checkHFileVersion method, so I change > the source code. Simultaneously, I change ChecksumUtil the info log level to > trace. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18066) Get with closest_row_before on "hbase:meta" can return empty Cell during region merge/split
[ https://issues.apache.org/jira/browse/HBASE-18066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025730#comment-16025730 ] Zheng Hu commented on HBASE-18066: -- [~busbey], FYI. > Get with closest_row_before on "hbase:meta" can return empty Cell during > region merge/split > --- > > Key: HBASE-18066 > URL: https://issues.apache.org/jira/browse/HBASE-18066 > Project: HBase > Issue Type: Bug > Components: hbase, regionserver >Affects Versions: 1.3.1 > Environment: Linux (16.04.2), MacOS 10.11.6. > Standalone and distributed HBase setup. >Reporter: Andrey Elenskiy >Assignee: Zheng Hu > Attachments: HBASE-18066.branch-1.1.v1.patch, > HBASE-18066.branch-1.1.v1.patch, HBASE-18066.branch-1.1.v1.patch, > HBASE-18066.branch-1.3.v1.patch, HBASE-18066.branch-1.3.v1.patch, > HBASE-18066.branch-1.v1.patch, HBASE-18066.branch-1.v2.patch, > HBASE-18066.branch-1.v3.patch, TestGetWithClosestRowBeforeWhenSplit.java > > > During region split/merge there's a brief period of time where doing a "Get" > with "closest_row_before=true" on "hbase:meta" may return empty > "GetResponse.result.cell" field even though parent, splitA and splitB regions > are all in "hbase:meta". Both gohbase (https://github.com/tsuna/gohbase) and > AsyncHBase (https://github.com/OpenTSDB/asynchbase) interprets this as > "TableDoesNotExist", which is returned to the client. > Here's a gist that reproduces this problem: > https://gist.github.com/Timoha/c7a236b768be9220e85e53e1ca53bf96. Note that > you have to use older HTable client (I used 1.2.4) as current versions ignore > `Get.setClosestRowBefore(bool)` option. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-18119) Improve HFile readability and modify ChecksumUtil log level
Qilin Cao created HBASE-18119: - Summary: Improve HFile readability and modify ChecksumUtil log level Key: HBASE-18119 URL: https://issues.apache.org/jira/browse/HBASE-18119 Project: HBase Issue Type: Improvement Reporter: Qilin Cao Priority: Minor It is confused when I read the HFile.checkHFileVersion method, so I change the source code. Simultaneously, I change ChecksumUtil the info log level to trace. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18066) Get with closest_row_before on "hbase:meta" can return empty Cell during region merge/split
[
https://issues.apache.org/jira/browse/HBASE-18066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025726#comment-16025726
]
Zheng Hu commented on HBASE-18066:
--
I've tried following shell commands to install oracle-java7-installer and
failed.
{code}
Step 8 : RUN apt-get -q update && apt-get -q install --no-install-recommends -y
software-properties-common
---> Using cache
---> e8bb99c68d6d
Step 9 : RUN add-apt-repository -y ppa:webupd8team/java
---> Using cache
---> e5856402bd14
Step 10 : RUN echo oracle-java7-installer shared/accepted-oracle-license-v1-1
select true | sudo /usr/bin/debconf-set-selections
---> Running in 0d3c79d66aa7
---> 1172173f42b9
Removing intermediate container 0d3c79d66aa7
Step 11 : RUN echo oracle-java8-installer shared/accepted-oracle-license-v1-1
select true | sudo /usr/bin/debconf-set-selections
---> Running in d2e322271e4e
---> d911c02484a3
Removing intermediate container d2e322271e4e
Step 12 : RUN apt-get -q update && apt-get -q install -y oracle-java7-installer
{code}
> Get with closest_row_before on "hbase:meta" can return empty Cell during
> region merge/split
> ---
>
> Key: HBASE-18066
> URL: https://issues.apache.org/jira/browse/HBASE-18066
> Project: HBase
> Issue Type: Bug
> Components: hbase, regionserver
>Affects Versions: 1.3.1
> Environment: Linux (16.04.2), MacOS 10.11.6.
> Standalone and distributed HBase setup.
>Reporter: Andrey Elenskiy
>Assignee: Zheng Hu
> Attachments: HBASE-18066.branch-1.1.v1.patch,
> HBASE-18066.branch-1.1.v1.patch, HBASE-18066.branch-1.1.v1.patch,
> HBASE-18066.branch-1.3.v1.patch, HBASE-18066.branch-1.3.v1.patch,
> HBASE-18066.branch-1.v1.patch, HBASE-18066.branch-1.v2.patch,
> HBASE-18066.branch-1.v3.patch, TestGetWithClosestRowBeforeWhenSplit.java
>
>
> During region split/merge there's a brief period of time where doing a "Get"
> with "closest_row_before=true" on "hbase:meta" may return empty
> "GetResponse.result.cell" field even though parent, splitA and splitB regions
> are all in "hbase:meta". Both gohbase (https://github.com/tsuna/gohbase) and
> AsyncHBase (https://github.com/OpenTSDB/asynchbase) interprets this as
> "TableDoesNotExist", which is returned to the client.
> Here's a gist that reproduces this problem:
> https://gist.github.com/Timoha/c7a236b768be9220e85e53e1ca53bf96. Note that
> you have to use older HTable client (I used 1.2.4) as current versions ignore
> `Get.setClosestRowBefore(bool)` option.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (HBASE-18066) Get with closest_row_before on "hbase:meta" can return empty Cell during region merge/split
[
https://issues.apache.org/jira/browse/HBASE-18066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025726#comment-16025726
]
Zheng Hu edited comment on HBASE-18066 at 5/26/17 2:47 AM:
---
I've tried following shell commands to install oracle-java7-installer under my
ubuntu and failed.
{code}
Step 8 : RUN apt-get -q update && apt-get -q install --no-install-recommends -y
software-properties-common
---> Using cache
---> e8bb99c68d6d
Step 9 : RUN add-apt-repository -y ppa:webupd8team/java
---> Using cache
---> e5856402bd14
Step 10 : RUN echo oracle-java7-installer shared/accepted-oracle-license-v1-1
select true | sudo /usr/bin/debconf-set-selections
---> Running in 0d3c79d66aa7
---> 1172173f42b9
Removing intermediate container 0d3c79d66aa7
Step 11 : RUN echo oracle-java8-installer shared/accepted-oracle-license-v1-1
select true | sudo /usr/bin/debconf-set-selections
---> Running in d2e322271e4e
---> d911c02484a3
Removing intermediate container d2e322271e4e
Step 12 : RUN apt-get -q update && apt-get -q install -y oracle-java7-installer
{code}
was (Author: openinx):
I've tried following shell commands to install oracle-java7-installer and
failed.
{code}
Step 8 : RUN apt-get -q update && apt-get -q install --no-install-recommends -y
software-properties-common
---> Using cache
---> e8bb99c68d6d
Step 9 : RUN add-apt-repository -y ppa:webupd8team/java
---> Using cache
---> e5856402bd14
Step 10 : RUN echo oracle-java7-installer shared/accepted-oracle-license-v1-1
select true | sudo /usr/bin/debconf-set-selections
---> Running in 0d3c79d66aa7
---> 1172173f42b9
Removing intermediate container 0d3c79d66aa7
Step 11 : RUN echo oracle-java8-installer shared/accepted-oracle-license-v1-1
select true | sudo /usr/bin/debconf-set-selections
---> Running in d2e322271e4e
---> d911c02484a3
Removing intermediate container d2e322271e4e
Step 12 : RUN apt-get -q update && apt-get -q install -y oracle-java7-installer
{code}
> Get with closest_row_before on "hbase:meta" can return empty Cell during
> region merge/split
> ---
>
> Key: HBASE-18066
> URL: https://issues.apache.org/jira/browse/HBASE-18066
> Project: HBase
> Issue Type: Bug
> Components: hbase, regionserver
>Affects Versions: 1.3.1
> Environment: Linux (16.04.2), MacOS 10.11.6.
> Standalone and distributed HBase setup.
>Reporter: Andrey Elenskiy
>Assignee: Zheng Hu
> Attachments: HBASE-18066.branch-1.1.v1.patch,
> HBASE-18066.branch-1.1.v1.patch, HBASE-18066.branch-1.1.v1.patch,
> HBASE-18066.branch-1.3.v1.patch, HBASE-18066.branch-1.3.v1.patch,
> HBASE-18066.branch-1.v1.patch, HBASE-18066.branch-1.v2.patch,
> HBASE-18066.branch-1.v3.patch, TestGetWithClosestRowBeforeWhenSplit.java
>
>
> During region split/merge there's a brief period of time where doing a "Get"
> with "closest_row_before=true" on "hbase:meta" may return empty
> "GetResponse.result.cell" field even though parent, splitA and splitB regions
> are all in "hbase:meta". Both gohbase (https://github.com/tsuna/gohbase) and
> AsyncHBase (https://github.com/OpenTSDB/asynchbase) interprets this as
> "TableDoesNotExist", which is returned to the client.
> Here's a gist that reproduces this problem:
> https://gist.github.com/Timoha/c7a236b768be9220e85e53e1ca53bf96. Note that
> you have to use older HTable client (I used 1.2.4) as current versions ignore
> `Get.setClosestRowBefore(bool)` option.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18066) Get with closest_row_before on "hbase:meta" can return empty Cell during region merge/split
[ https://issues.apache.org/jira/browse/HBASE-18066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025718#comment-16025718 ] Zheng Hu commented on HBASE-18066: -- Seems like we often fail to install jdk7 for Hadoop QA. ( RUN apt-get -q update && apt-get -q install -y oracle-java7-installer failed) 1. https://builds.apache.org/job/PreCommit-HBASE-Build/6921/console 2. https://builds.apache.org/job/PreCommit-HBASE-Build/6960/console > Get with closest_row_before on "hbase:meta" can return empty Cell during > region merge/split > --- > > Key: HBASE-18066 > URL: https://issues.apache.org/jira/browse/HBASE-18066 > Project: HBase > Issue Type: Bug > Components: hbase, regionserver >Affects Versions: 1.3.1 > Environment: Linux (16.04.2), MacOS 10.11.6. > Standalone and distributed HBase setup. >Reporter: Andrey Elenskiy >Assignee: Zheng Hu > Attachments: HBASE-18066.branch-1.1.v1.patch, > HBASE-18066.branch-1.1.v1.patch, HBASE-18066.branch-1.1.v1.patch, > HBASE-18066.branch-1.3.v1.patch, HBASE-18066.branch-1.3.v1.patch, > HBASE-18066.branch-1.v1.patch, HBASE-18066.branch-1.v2.patch, > HBASE-18066.branch-1.v3.patch, TestGetWithClosestRowBeforeWhenSplit.java > > > During region split/merge there's a brief period of time where doing a "Get" > with "closest_row_before=true" on "hbase:meta" may return empty > "GetResponse.result.cell" field even though parent, splitA and splitB regions > are all in "hbase:meta". Both gohbase (https://github.com/tsuna/gohbase) and > AsyncHBase (https://github.com/OpenTSDB/asynchbase) interprets this as > "TableDoesNotExist", which is returned to the client. > Here's a gist that reproduces this problem: > https://gist.github.com/Timoha/c7a236b768be9220e85e53e1ca53bf96. Note that > you have to use older HTable client (I used 1.2.4) as current versions ignore > `Get.setClosestRowBefore(bool)` option. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18118) Default storage policy if not configured cannot be "NONE"
[
https://issues.apache.org/jira/browse/HBASE-18118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025711#comment-16025711
]
Hadoop QA commented on HBASE-18118:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 23s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
38s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
32m 40s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 57s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 5s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:757bf37 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12869973/HBASE-18118.patch |
| JIRA Issue | HBASE-18118 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux a3135f265126 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / f441ca0 |
| Default Java | 1.8.0_131 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6959/testReport/ |
| modules | C: hbase-common U: hbase-common |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6959/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Default storage policy if not configured cannot be "NONE"
> -
>
> Key: HBASE-18118
> URL: https://issues.apache.org/jira/browse/HBASE-18118
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.0
>Reporter: Andrew
[jira] [Commented] (HBASE-18066) Get with closest_row_before on "hbase:meta" can return empty Cell during region merge/split
[
https://issues.apache.org/jira/browse/HBASE-18066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025701#comment-16025701
]
Hadoop QA commented on HBASE-18066:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 0m 37s
{color} | {color:red} Docker failed to build yetus/hbase:de9b245. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12869976/HBASE-18066.branch-1.1.v1.patch
|
| JIRA Issue | HBASE-18066 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6960/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Get with closest_row_before on "hbase:meta" can return empty Cell during
> region merge/split
> ---
>
> Key: HBASE-18066
> URL: https://issues.apache.org/jira/browse/HBASE-18066
> Project: HBase
> Issue Type: Bug
> Components: hbase, regionserver
>Affects Versions: 1.3.1
> Environment: Linux (16.04.2), MacOS 10.11.6.
> Standalone and distributed HBase setup.
>Reporter: Andrey Elenskiy
>Assignee: Zheng Hu
> Attachments: HBASE-18066.branch-1.1.v1.patch,
> HBASE-18066.branch-1.1.v1.patch, HBASE-18066.branch-1.1.v1.patch,
> HBASE-18066.branch-1.3.v1.patch, HBASE-18066.branch-1.3.v1.patch,
> HBASE-18066.branch-1.v1.patch, HBASE-18066.branch-1.v2.patch,
> HBASE-18066.branch-1.v3.patch, TestGetWithClosestRowBeforeWhenSplit.java
>
>
> During region split/merge there's a brief period of time where doing a "Get"
> with "closest_row_before=true" on "hbase:meta" may return empty
> "GetResponse.result.cell" field even though parent, splitA and splitB regions
> are all in "hbase:meta". Both gohbase (https://github.com/tsuna/gohbase) and
> AsyncHBase (https://github.com/OpenTSDB/asynchbase) interprets this as
> "TableDoesNotExist", which is returned to the client.
> Here's a gist that reproduces this problem:
> https://gist.github.com/Timoha/c7a236b768be9220e85e53e1ca53bf96. Note that
> you have to use older HTable client (I used 1.2.4) as current versions ignore
> `Get.setClosestRowBefore(bool)` option.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-18114) Update the config of TestAsync*AdminApi to make test stable
[
https://issues.apache.org/jira/browse/HBASE-18114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-18114:
---
Attachment: HBASE-18114-v2.patch
> Update the config of TestAsync*AdminApi to make test stable
> ---
>
> Key: HBASE-18114
> URL: https://issues.apache.org/jira/browse/HBASE-18114
> Project: HBase
> Issue Type: Sub-task
> Components: Client
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-18114-v1.patch, HBASE-18114-v1.patch,
> HBASE-18114-v1.patch, HBASE-18114-v2.patch, HBASE-18114-v2.patch
>
>
> {code}
> 2017-05-25 17:56:34,967 INFO
> [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=50801]
> master.HMaster$11(2297): Client=hao//127.0.0.1 disable testModifyColumnFamily
> 2017-05-25 17:56:37,974 INFO [RpcClient-timer-pool1-t1]
> client.AsyncHBaseAdmin$TableProcedureBiConsumer(2219): Operation: DISABLE,
> Table Name: default:testModifyColumnFamily failed with Failed after
> attempts=3, exceptions:
> Thu May 25 17:56:35 CST 2017, , java.io.IOException: Call to
> localhost/127.0.0.1:50801 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=294, waitTime=1008,
> rpcTimeout=1000
> Thu May 25 17:56:37 CST 2017, , java.io.IOException: Call to
> localhost/127.0.0.1:50801 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=295, waitTime=1299,
> rpcTimeout=1000
> Thu May 25 17:56:37 CST 2017, , java.io.IOException: Call to
> localhost/127.0.0.1:50801 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=296, waitTime=668,
> rpcTimeout=660
> 017-05-25 17:56:38,936 DEBUG
> [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=50801]
> procedure2.ProcedureExecutor(788): Stored procId=15, owner=hao,
> state=RUNNABLE:DISABLE_TABLE_PREPARE, DisableTableProcedure
> table=testModifyColumnFamily
> {code}
> For this disable table procedure, master return the procedure id when it
> submit the procedure to ProcedureExecutor. And the above procedure take 4
> seconds to submit. So the disable table call failed because the rpc timeout
> is 1 seconds and the retry number is 3.
> For admin operation, I thought we don't need change the default timeout
> config in unit test. And the retry is not need, too. (Or we can set a retry >
> 1 to test nonce thing). Meanwhile, the default timeout is 60 seconds. So the
> test type may need change to LargeTests.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-18066) Get with closest_row_before on "hbase:meta" can return empty Cell during region merge/split
[ https://issues.apache.org/jira/browse/HBASE-18066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Hu updated HBASE-18066: - Attachment: HBASE-18066.branch-1.1.v1.patch Trigger Hadoop QA for branch-1.1 again. > Get with closest_row_before on "hbase:meta" can return empty Cell during > region merge/split > --- > > Key: HBASE-18066 > URL: https://issues.apache.org/jira/browse/HBASE-18066 > Project: HBase > Issue Type: Bug > Components: hbase, regionserver >Affects Versions: 1.3.1 > Environment: Linux (16.04.2), MacOS 10.11.6. > Standalone and distributed HBase setup. >Reporter: Andrey Elenskiy >Assignee: Zheng Hu > Attachments: HBASE-18066.branch-1.1.v1.patch, > HBASE-18066.branch-1.1.v1.patch, HBASE-18066.branch-1.1.v1.patch, > HBASE-18066.branch-1.3.v1.patch, HBASE-18066.branch-1.3.v1.patch, > HBASE-18066.branch-1.v1.patch, HBASE-18066.branch-1.v2.patch, > HBASE-18066.branch-1.v3.patch, TestGetWithClosestRowBeforeWhenSplit.java > > > During region split/merge there's a brief period of time where doing a "Get" > with "closest_row_before=true" on "hbase:meta" may return empty > "GetResponse.result.cell" field even though parent, splitA and splitB regions > are all in "hbase:meta". Both gohbase (https://github.com/tsuna/gohbase) and > AsyncHBase (https://github.com/OpenTSDB/asynchbase) interprets this as > "TableDoesNotExist", which is returned to the client. > Here's a gist that reproduces this problem: > https://gist.github.com/Timoha/c7a236b768be9220e85e53e1ca53bf96. Note that > you have to use older HTable client (I used 1.2.4) as current versions ignore > `Get.setClosestRowBefore(bool)` option. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18118) Default storage policy if not configured cannot be "NONE"
[ https://issues.apache.org/jira/browse/HBASE-18118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025684#comment-16025684 ] Ted Yu commented on HBASE-18118: lgtm > Default storage policy if not configured cannot be "NONE" > - > > Key: HBASE-18118 > URL: https://issues.apache.org/jira/browse/HBASE-18118 > Project: HBase > Issue Type: Bug > Components: wal >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-18118.patch > > > HBase can't use 'NONE' as default storage policy if not configured because > HDFS supports no such policy. This policy name was probably available in a > precommit or early version of the HDFS side support for heterogeneous > storage. Now the best default is 'HOT'. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18118) Default storage policy if not configured cannot be "NONE"
[ https://issues.apache.org/jira/browse/HBASE-18118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-18118: --- Attachment: HBASE-18118.patch Simple patch. Eliminates log noise that can come about when WAL storage policy isn't set in the site file. > Default storage policy if not configured cannot be "NONE" > - > > Key: HBASE-18118 > URL: https://issues.apache.org/jira/browse/HBASE-18118 > Project: HBase > Issue Type: Bug > Components: wal >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-18118.patch > > > HBase can't use 'NONE' as default storage policy if not configured because > HDFS supports no such policy. This policy name was probably available in a > precommit or early version of the HDFS side support for heterogeneous > storage. Now the best default is 'HOT'. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18118) Default storage policy if not configured cannot be "NONE"
[ https://issues.apache.org/jira/browse/HBASE-18118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-18118: --- Status: Patch Available (was: Open) > Default storage policy if not configured cannot be "NONE" > - > > Key: HBASE-18118 > URL: https://issues.apache.org/jira/browse/HBASE-18118 > Project: HBase > Issue Type: Bug > Components: wal >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-18118.patch > > > HBase can't use 'NONE' as default storage policy if not configured because > HDFS supports no such policy. This policy name was probably available in a > precommit or early version of the HDFS side support for heterogeneous > storage. Now the best default is 'HOT'. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-18118) Default storage policy if not configured cannot be "NONE"
Andrew Purtell created HBASE-18118: -- Summary: Default storage policy if not configured cannot be "NONE" Key: HBASE-18118 URL: https://issues.apache.org/jira/browse/HBASE-18118 Project: HBase Issue Type: Bug Components: wal Affects Versions: 2.0.0 Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor Fix For: 2.0.0 HBase can't use 'NONE' as default storage policy if not configured because HDFS supports no such policy. This policy name was probably available in a precommit or early version of the HDFS side support for heterogeneous storage. Now the best default is 'HOT'. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18042) Client Compatibility breaks between versions 1.2 and 1.3
[
https://issues.apache.org/jira/browse/HBASE-18042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025654#comment-16025654
]
Duo Zhang commented on HBASE-18042:
---
Seems the download url is changed by oracle.
https://stackoverflow.com/questions/44142450/apt-get-installing-oracle-java-7-stopped-working.
> Client Compatibility breaks between versions 1.2 and 1.3
>
>
> Key: HBASE-18042
> URL: https://issues.apache.org/jira/browse/HBASE-18042
> Project: HBase
> Issue Type: Bug
> Components: regionserver, scan
>Affects Versions: 2.0.0, 1.4.0, 1.3.1
>Reporter: Karan Mehta
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0, 1.4.0, 1.3.2
>
> Attachments: HBASE-18042-branch-1.3.patch,
> HBASE-18042-branch-1.patch, HBASE-18042-branch-1.patch,
> HBASE-18042-branch-1-v1.patch, HBASE-18042-branch-1-v1.patch,
> HBASE-18042.patch, HBASE-18042-v1.patch, HBASE-18042-v2.patch
>
>
> OpenTSDB uses AsyncHBase as its client, rather than using the traditional
> HBase Client. From version 1.2 to 1.3, the {{ClientProtos}} have been
> changed. Newer fields are added to {{ScanResponse}} proto.
> For a typical Scan request in 1.2, would require caller to make an
> OpenScanner Request, GetNextRows Request and a CloseScanner Request, based on
> {{more_rows}} boolean field in the {{ScanResponse}} proto.
> However, from 1.3, new parameter {{more_results_in_region}} was added, which
> limits the results per region. Therefore the client has to now manage sending
> all the requests for each region. Further more, if the results are exhausted
> from a particular region, the {{ScanResponse}} will set
> {{more_results_in_region}} to false, but {{more_results}} can still be true.
> Whenever the former is set to false, the {{RegionScanner}} will also be
> closed.
> OpenTSDB makes an OpenScanner Request and receives all its results in the
> first {{ScanResponse}} itself, thus creating a condition as described in
> above paragraph. Since {{more_rows}} is true, it will proceed to send next
> request at which point the {{RSRpcServices}} will throw
> {{UnknownScannerException}}. The protobuf client compatibility is maintained
> but expected behavior is modified.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-18114) Update the config of TestAsync*AdminApi to make test stable
[
https://issues.apache.org/jira/browse/HBASE-18114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-18114:
---
Attachment: HBASE-18114-v2.patch
> Update the config of TestAsync*AdminApi to make test stable
> ---
>
> Key: HBASE-18114
> URL: https://issues.apache.org/jira/browse/HBASE-18114
> Project: HBase
> Issue Type: Sub-task
> Components: Client
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-18114-v1.patch, HBASE-18114-v1.patch,
> HBASE-18114-v1.patch, HBASE-18114-v2.patch
>
>
> {code}
> 2017-05-25 17:56:34,967 INFO
> [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=50801]
> master.HMaster$11(2297): Client=hao//127.0.0.1 disable testModifyColumnFamily
> 2017-05-25 17:56:37,974 INFO [RpcClient-timer-pool1-t1]
> client.AsyncHBaseAdmin$TableProcedureBiConsumer(2219): Operation: DISABLE,
> Table Name: default:testModifyColumnFamily failed with Failed after
> attempts=3, exceptions:
> Thu May 25 17:56:35 CST 2017, , java.io.IOException: Call to
> localhost/127.0.0.1:50801 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=294, waitTime=1008,
> rpcTimeout=1000
> Thu May 25 17:56:37 CST 2017, , java.io.IOException: Call to
> localhost/127.0.0.1:50801 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=295, waitTime=1299,
> rpcTimeout=1000
> Thu May 25 17:56:37 CST 2017, , java.io.IOException: Call to
> localhost/127.0.0.1:50801 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=296, waitTime=668,
> rpcTimeout=660
> 017-05-25 17:56:38,936 DEBUG
> [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=50801]
> procedure2.ProcedureExecutor(788): Stored procId=15, owner=hao,
> state=RUNNABLE:DISABLE_TABLE_PREPARE, DisableTableProcedure
> table=testModifyColumnFamily
> {code}
> For this disable table procedure, master return the procedure id when it
> submit the procedure to ProcedureExecutor. And the above procedure take 4
> seconds to submit. So the disable table call failed because the rpc timeout
> is 1 seconds and the retry number is 3.
> For admin operation, I thought we don't need change the default timeout
> config in unit test. And the retry is not need, too. (Or we can set a retry >
> 1 to test nonce thing). Meanwhile, the default timeout is 60 seconds. So the
> test type may need change to LargeTests.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18117) Increase resiliency by allowing more parameters for online config change
[
https://issues.apache.org/jira/browse/HBASE-18117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025640#comment-16025640
]
Karan Mehta commented on HBASE-18117:
-
The current framework only allows changes on the config parameters that are
accessed only on the server side. If {{ConfigurationObserver}} is implemented
by any of the classes from {{hbase-client}}, this will introduce a cyclic
dependency between hbase-client and hbase-server projects and thus the build
would fail.
> Increase resiliency by allowing more parameters for online config change
>
>
> Key: HBASE-18117
> URL: https://issues.apache.org/jira/browse/HBASE-18117
> Project: HBase
> Issue Type: Improvement
>Reporter: Karan Mehta
>Assignee: Karan Mehta
>
> HBASE-8544 adds the feature to change config online without having a server
> restart. This JIRA is to work on new parameters for the utilizing that
> feature.
> As [~apurtell] suggested, following are the useful and frequently changing
> parameters in production.
> - RPC limits, timeouts, and other performance relevant settings
> - Replication limits and batch sizes
> - Region carrying limit
> - WAL retention and cleaning parameters
> I will try to make the RPC timeout parameter online as a part of this JIRA.
> If it seems suitable then we can extend it to other params.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-18027) Replication should respect RPC size limits when batching edits
[ https://issues.apache.org/jira/browse/HBASE-18027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-18027: --- Attachment: HBASE-18027-branch-1.patch The branch-1 patch is simpler. I keep the logging changes and only the clamping of replication queue capacity limit. I did look into changing logic in ReplicationSource to check if we go overlimit _before_ adding an entry to the replication worklist, but the question then is what to do with that last read entry. Must be robust to failure. In-memory only 'put back' queue will lose data upon failure. Seeking the reader back to before the most recent read did not test out to be robust with active chaos. Instead we try to avoid creating an overlarge RPC by setting the replication queue capacity limit to the lesser of replication.source.size.capacity or 95% of the RPC size limit. This is trying a lot harder than we did previously to avoid the problem. > Replication should respect RPC size limits when batching edits > -- > > Key: HBASE-18027 > URL: https://issues.apache.org/jira/browse/HBASE-18027 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.4.0, 1.3.1 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.4.0, 1.3.2 > > Attachments: HBASE-18027-branch-1.patch, HBASE-18027.patch, > HBASE-18027.patch, HBASE-18027.patch, HBASE-18027.patch, HBASE-18027.patch > > > In HBaseInterClusterReplicationEndpoint#replicate we try to replicate in > batches. We create N lists. N is the minimum of configured replicator > threads, number of 100-waledit batches, or number of current sinks. Every > pending entry in the replication context is then placed in order by hash of > encoded region name into one of these N lists. Each of the N lists is then > sent all at once in one replication RPC. We do not test if the sum of data in > each N list will exceed RPC size limits. This code presumes each individual > edit is reasonably small. Not checking for aggregate size while assembling > the lists into RPCs is an oversight and can lead to replication failure when > that assumption is violated. > We can fix this by generating as many replication RPC calls as we need to > drain a list, keeping each RPC under limit, instead of assuming the whole > list will fit in one. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17997) In dev environment, add jruby-complete jar to classpath only when jruby is needed
[
https://issues.apache.org/jira/browse/HBASE-17997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025604#comment-16025604
]
Xiang Li commented on HBASE-17997:
--
Ted, thanks for that!
> In dev environment, add jruby-complete jar to classpath only when jruby is
> needed
> -
>
> Key: HBASE-17997
> URL: https://issues.apache.org/jira/browse/HBASE-17997
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Xiang Li
> Fix For: 2.0.0
>
> Attachments: HBASE-17997.master.000.patch,
> HBASE-17997.master.001.patch, HBASE-17997.master.002.patch,
> HBASE-17997.master.003.debug.diff, HBASE-17997.master.003.patch
>
>
> HBASE-15199 moves jruby-complete-1.6.8.jar to lib/ruby directory.
> However, jruby-complete-1.6.8.jar still appears in cached_classpath.txt
> This means that user would see exception similar to the following when
> starting hbase in standalone mode with s3a as rootdir :
> {code}
> 2017-05-04 16:41:32,854 WARN
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=38659]
> internal.S3MetadataResponseHandler: Unable to parse last modified date: Thu,
> 04 May 2017 16:27:09 GMT
> java.lang.IllegalStateException: Joda-time 2.2 or later version is required,
> but found version: null
> at com.amazonaws.util.DateUtils.handleException(DateUtils.java:149)
> at com.amazonaws.util.DateUtils.parseRFC822Date(DateUtils.java:195)
> at
> com.amazonaws.services.s3.internal.ServiceUtils.parseRfc822Date(ServiceUtils.java:78)
> at
> com.amazonaws.services.s3.internal.AbstractS3ResponseHandler.populateObjectMetadata(AbstractS3ResponseHandler.java:115)
> at
> com.amazonaws.services.s3.internal.S3ObjectResponseHandler.handle(S3ObjectResponseHandler.java:52)
> at
> com.amazonaws.services.s3.internal.S3ObjectResponseHandler.handle(S3ObjectResponseHandler.java:30)
> at
> com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:1072)
> at
> com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:746)
> at
> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
> at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
> at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785)
> at
> com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1191)
> at org.apache.hadoop.fs.s3a.S3AInputStream.reopen(S3AInputStream.java:148)
> at org.apache.hadoop.fs.s3a.S3AInputStream.lazySeek(S3AInputStream.java:281)
> at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:364)
> at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:75)
> at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:92)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.positionalReadWithExtra(HFileBlock.java:722)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1420)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1677)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1504)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:439)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:904)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:267)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:169)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:363)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:217)
> at
> org.apache.hadoop.hbase.regionserver.HStore.createScanner(HStore.java:2132)
> at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2122)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:5687)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2679)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2665)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2647)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6906)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6885)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2007)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-15160) Put back HFile's HDFS op latency sampling code and add metrics for monitoring
[ https://issues.apache.org/jira/browse/HBASE-15160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-15160: -- Attachment: hbase-15160_v6.patch [~carp84] how about this patch. I've removed the extra counters and made it so that we are passing a boolean down from the getMetaBlock() function so that metrics are not updated for the meta blocks. The reason that we cannot move the timing and updating of metrics up the stack is that, the callers of readBlock() do not know whether the returned block is read from disk, or comes from cache. Is it easy enough for you to replicate the YCSB tests? I've done some basic testing, and did not find meaningful perf regression. BTW, these metrics would have saved us days worth of debugging in a recent case, so let's get this patch in one way or the other. > Put back HFile's HDFS op latency sampling code and add metrics for monitoring > - > > Key: HBASE-15160 > URL: https://issues.apache.org/jira/browse/HBASE-15160 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0, 1.1.2 >Reporter: Yu Li >Assignee: Yu Li > Attachments: HBASE-15160.patch, HBASE-15160_v2.patch, > HBASE-15160_v3.patch, hbase-15160_v4.patch, hbase-15160_v5.patch, > hbase-15160_v6.patch > > > In HBASE-11586 all HDFS op latency sampling code, including fsReadLatency, > fsPreadLatency and fsWriteLatency, have been removed. There was some > discussion about putting them back in a new JIRA but never happened. > According to our experience, these metrics are useful to judge whether issue > lies on HDFS when slow request occurs, so we propose to put them back in this > JIRA, and add the metrics for monitoring as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-15160) Put back HFile's HDFS op latency sampling code and add metrics for monitoring
[ https://issues.apache.org/jira/browse/HBASE-15160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-15160: -- Priority: Critical (was: Major) > Put back HFile's HDFS op latency sampling code and add metrics for monitoring > - > > Key: HBASE-15160 > URL: https://issues.apache.org/jira/browse/HBASE-15160 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0, 1.1.2 >Reporter: Yu Li >Assignee: Yu Li >Priority: Critical > Attachments: HBASE-15160.patch, HBASE-15160_v2.patch, > HBASE-15160_v3.patch, hbase-15160_v4.patch, hbase-15160_v5.patch, > hbase-15160_v6.patch > > > In HBASE-11586 all HDFS op latency sampling code, including fsReadLatency, > fsPreadLatency and fsWriteLatency, have been removed. There was some > discussion about putting them back in a new JIRA but never happened. > According to our experience, these metrics are useful to judge whether issue > lies on HDFS when slow request occurs, so we propose to put them back in this > JIRA, and add the metrics for monitoring as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16392) Backup delete fault tolerance
[
https://issues.apache.org/jira/browse/HBASE-16392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025575#comment-16025575
]
Vladimir Rodionov commented on HBASE-16392:
---
{quote}
If deleteSnapshot() fails in the middle,
BackupSystemTable.restoreFromSnapshot(conn) call in catch block would fail as
well, right ?
{quote}
Good catch. Will need to fix this.
> Backup delete fault tolerance
> -
>
> Key: HBASE-16392
> URL: https://issues.apache.org/jira/browse/HBASE-16392
> Project: HBase
> Issue Type: Sub-task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-16392-v1.patch
>
>
> Backup delete modified file system and backup system table. We have to make
> sure that operation is atomic, durable and isolated.
> Delete operation:
> # Start backup session (this guarantees) that system will be blocked for all
> backup commands during delete operation
> # Save list of tables being deleted to system table
> # Before delete operation we take backup system table snapshot
> # During delete operation we detect any failures and restore backup system
> table from snapshot, then finish backup session
> # To guarantee consistency of the data, delete operation MUST be repeated
> # We guarantee that all file delete operations are idempotent, can be
> repeated multiple times
> # Any backup operations will be blocked until consistency is restored
> # To restore consistency, repair command must be executed.
> # Repair command checks if there is failed delete op in a backup system
> table, and repeats delete operation
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-18078) [C++] Implement RPC timeout
[ https://issues.apache.org/jira/browse/HBASE-18078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HBASE-18078: -- Attachment: HBASE-18078.000.patch > [C++] Implement RPC timeout > --- > > Key: HBASE-18078 > URL: https://issues.apache.org/jira/browse/HBASE-18078 > Project: HBase > Issue Type: Sub-task >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HBASE-18078.000.patch > > > RPC layer should handle various connection abnormality (e.g. server aborted > connection) through timeout. Ideally, exceptions should be raised and > propagated through handlers of pipeline in client. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18090) Improve TableSnapshotInputFormat to allow more multiple mappers per region
[ https://issues.apache.org/jira/browse/HBASE-18090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025520#comment-16025520 ] Mikhail Antonov commented on HBASE-18090: - I actually just want to decouple two things: 1) decision making guiding table split layout on the cluster running HBase and serving traffic to applications and 2) amount of parallelism available in the batch processing. So it's not necessarily about knowing the size of regions, its about knowing that if number of regions in snapshot is X and number of mapreduce slots is, for example, 5x, then I can run 5 tasks per region. > Improve TableSnapshotInputFormat to allow more multiple mappers per region > -- > > Key: HBASE-18090 > URL: https://issues.apache.org/jira/browse/HBASE-18090 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Affects Versions: 1.4.0 >Reporter: Mikhail Antonov > Attachments: HBASE-18090-branch-1.3-v1.patch > > > TableSnapshotInputFormat runs one map task per region in the table snapshot. > This places unnecessary restriction that the region layout of the original > table needs to take the processing resources available to MR job into > consideration. Allowing to run multiple mappers per region (assuming > reasonably even key distribution) would be useful. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18090) Improve TableSnapshotInputFormat to allow more multiple mappers per region
[ https://issues.apache.org/jira/browse/HBASE-18090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025515#comment-16025515 ] Yi Liang commented on HBASE-18090: -- ok, so if user know the size of region, they can considering split it into a reasonable number. Cool, i will review the code today and tomorrow. > Improve TableSnapshotInputFormat to allow more multiple mappers per region > -- > > Key: HBASE-18090 > URL: https://issues.apache.org/jira/browse/HBASE-18090 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Affects Versions: 1.4.0 >Reporter: Mikhail Antonov > Attachments: HBASE-18090-branch-1.3-v1.patch > > > TableSnapshotInputFormat runs one map task per region in the table snapshot. > This places unnecessary restriction that the region layout of the original > table needs to take the processing resources available to MR job into > consideration. Allowing to run multiple mappers per region (assuming > reasonably even key distribution) would be useful. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18090) Improve TableSnapshotInputFormat to allow more multiple mappers per region
[ https://issues.apache.org/jira/browse/HBASE-18090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025511#comment-16025511 ] Mikhail Antonov commented on HBASE-18090: - `int numSplitsPerRegion` param, passed in :) > Improve TableSnapshotInputFormat to allow more multiple mappers per region > -- > > Key: HBASE-18090 > URL: https://issues.apache.org/jira/browse/HBASE-18090 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Affects Versions: 1.4.0 >Reporter: Mikhail Antonov > Attachments: HBASE-18090-branch-1.3-v1.patch > > > TableSnapshotInputFormat runs one map task per region in the table snapshot. > This places unnecessary restriction that the region layout of the original > table needs to take the processing resources available to MR job into > consideration. Allowing to run multiple mappers per region (assuming > reasonably even key distribution) would be useful. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18090) Improve TableSnapshotInputFormat to allow more multiple mappers per region
[ https://issues.apache.org/jira/browse/HBASE-18090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025497#comment-16025497 ] Yi Liang commented on HBASE-18090: -- Hi Mikhail, Before dig into your patch, just have a questions, how do you decide how many mappers per region. > Improve TableSnapshotInputFormat to allow more multiple mappers per region > -- > > Key: HBASE-18090 > URL: https://issues.apache.org/jira/browse/HBASE-18090 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Affects Versions: 1.4.0 >Reporter: Mikhail Antonov > Attachments: HBASE-18090-branch-1.3-v1.patch > > > TableSnapshotInputFormat runs one map task per region in the table snapshot. > This places unnecessary restriction that the region layout of the original > table needs to take the processing resources available to MR job into > consideration. Allowing to run multiple mappers per region (assuming > reasonably even key distribution) would be useful. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HBASE-16894) Create more than 1 split per region, generalize HBASE-12590
[
https://issues.apache.org/jira/browse/HBASE-16894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025492#comment-16025492
]
Yi Liang edited comment on HBASE-16894 at 5/25/17 10:36 PM:
Hi Mikhail,
I will take a look at the HBASE-18090. I have a draft patch there in the
review board, but it does not matter, let's what we can do to improve it.
was (Author: easyliangjob):
Hi Mikhail,
I will take a look at the HBASE-18090 to see what we can do to improve it.
> Create more than 1 split per region, generalize HBASE-12590
> ---
>
> Key: HBASE-16894
> URL: https://issues.apache.org/jira/browse/HBASE-16894
> Project: HBase
> Issue Type: Improvement
>Reporter: Enis Soztutar
>Assignee: Yi Liang
> Attachments: ImplementaionAndSomeQuestion.docx
>
>
> A common request from users is to be able to better control how many map
> tasks are created per region. Right now, it is always 1 region = 1 input
> split = 1 map task. Same goes for Spark since it uses the TIF. With region
> sizes as large as 50 GBs, it is desirable to be able to create more than 1
> split per region.
> HBASE-12590 adds a config property for MR jobs to be able to handle skew in
> region sizes. The algorithm is roughly:
> {code}
> If (region size >= average size*ratio) : cut the region into two MR input
> splits
> If (average size <= region size < average size*ratio) : one region as one MR
> input split
> If (sum of several continuous regions size < average size * ratio): combine
> these regions into one MR input split.
> {code}
> Although we can set data skew ratio to be 0.5 or something to abuse
> HBASE-12590 into creating more than 1 split task per region, it is not ideal.
> But there is no way to create more with the patch as it is. For example we
> cannot create more than 2 tasks per region.
> If we want to fix this properly, we should extend the approach in
> HBASE-12590, and make it so that the client can specify the desired num of
> mappers, or desired split size, and the TIF generates the splits based on the
> current region sizes very similar to the algorithm in HBASE-12590, but a more
> generic way. This also would eliminate the hand tuning of data skew ratio.
> We also can think about the guidepost approach that Phoenix has in the stats
> table which is used for exactly this purpose. Right now, the region can be
> split into powers of two assuming uniform distribution within the region.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-16894) Create more than 1 split per region, generalize HBASE-12590
[
https://issues.apache.org/jira/browse/HBASE-16894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025492#comment-16025492
]
Yi Liang commented on HBASE-16894:
--
Hi Mikhail,
I will take a look at the HBASE-18090 to see what we can do to improve it.
> Create more than 1 split per region, generalize HBASE-12590
> ---
>
> Key: HBASE-16894
> URL: https://issues.apache.org/jira/browse/HBASE-16894
> Project: HBase
> Issue Type: Improvement
>Reporter: Enis Soztutar
>Assignee: Yi Liang
> Attachments: ImplementaionAndSomeQuestion.docx
>
>
> A common request from users is to be able to better control how many map
> tasks are created per region. Right now, it is always 1 region = 1 input
> split = 1 map task. Same goes for Spark since it uses the TIF. With region
> sizes as large as 50 GBs, it is desirable to be able to create more than 1
> split per region.
> HBASE-12590 adds a config property for MR jobs to be able to handle skew in
> region sizes. The algorithm is roughly:
> {code}
> If (region size >= average size*ratio) : cut the region into two MR input
> splits
> If (average size <= region size < average size*ratio) : one region as one MR
> input split
> If (sum of several continuous regions size < average size * ratio): combine
> these regions into one MR input split.
> {code}
> Although we can set data skew ratio to be 0.5 or something to abuse
> HBASE-12590 into creating more than 1 split task per region, it is not ideal.
> But there is no way to create more with the patch as it is. For example we
> cannot create more than 2 tasks per region.
> If we want to fix this properly, we should extend the approach in
> HBASE-12590, and make it so that the client can specify the desired num of
> mappers, or desired split size, and the TIF generates the splits based on the
> current region sizes very similar to the algorithm in HBASE-12590, but a more
> generic way. This also would eliminate the hand tuning of data skew ratio.
> We also can think about the guidepost approach that Phoenix has in the stats
> table which is used for exactly this purpose. Right now, the region can be
> split into powers of two assuming uniform distribution within the region.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-16261) MultiHFileOutputFormat Enhancement
[ https://issues.apache.org/jira/browse/HBASE-16261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liang updated HBASE-16261: - Attachment: HBase-16261-V9.patch > MultiHFileOutputFormat Enhancement > > > Key: HBASE-16261 > URL: https://issues.apache.org/jira/browse/HBASE-16261 > Project: HBase > Issue Type: Sub-task > Components: hbase, mapreduce >Affects Versions: 2.0.0 >Reporter: Yi Liang >Assignee: Yi Liang >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16261-V1.patch, HBASE-16261-V2.patch, > HBASE-16261-V3.patch, HBASE-16261-V4.patch, HBASE-16261-V5.patch, > HBase-16261-V6.patch, HBase-16261-V7.patch, HBase-16261-V8.patch, > HBase-16261-V9.patch > > > MultiHFileOutputFormat follow HFileOutputFormat2 > (1) HFileOutputFormat2 can read one table's region split keys. and then > output multiple hfiles for one family, and each hfile map to one region. We > can add partitioner in MultiHFileOutputFormat to make it support this feature. > (2) HFileOutputFormat2 support Customized Compression algorithm for column > family and BloomFilter, also support customized DataBlockEncoding for the > output hfiles. We can also make MultiHFileOutputFormat to support these > features. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HBASE-18117) Increase resiliency by allowing more parameters for online config change
[ https://issues.apache.org/jira/browse/HBASE-18117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karan Mehta reassigned HBASE-18117: --- Assignee: Karan Mehta > Increase resiliency by allowing more parameters for online config change > > > Key: HBASE-18117 > URL: https://issues.apache.org/jira/browse/HBASE-18117 > Project: HBase > Issue Type: Improvement >Reporter: Karan Mehta >Assignee: Karan Mehta > > HBASE-8544 adds the feature to change config online without having a server > restart. This JIRA is to work on new parameters for the utilizing that > feature. > As [~apurtell] suggested, following are the useful and frequently changing > parameters in production. > - RPC limits, timeouts, and other performance relevant settings > - Replication limits and batch sizes > - Region carrying limit > - WAL retention and cleaning parameters > I will try to make the RPC timeout parameter online as a part of this JIRA. > If it seems suitable then we can extend it to other params. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-18117) Increase resiliency by allowing more parameters for online config change
Karan Mehta created HBASE-18117: --- Summary: Increase resiliency by allowing more parameters for online config change Key: HBASE-18117 URL: https://issues.apache.org/jira/browse/HBASE-18117 Project: HBase Issue Type: Improvement Reporter: Karan Mehta HBASE-8544 adds the feature to change config online without having a server restart. This JIRA is to work on new parameters for the utilizing that feature. As [~apurtell] suggested, following are the useful and frequently changing parameters in production. - RPC limits, timeouts, and other performance relevant settings - Replication limits and batch sizes - Region carrying limit - WAL retention and cleaning parameters I will try to make the RPC timeout parameter online as a part of this JIRA. If it seems suitable then we can extend it to other params. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-14614) Procedure v2: Core Assignment Manager
[
https://issues.apache.org/jira/browse/HBASE-14614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025397#comment-16025397
]
Hadoop QA commented on HBASE-14614:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 30s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 98 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 39s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
11s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 5m 4s
{color} | {color:red} hbase-protocol-shaded in master has 24 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 18s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 20s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
2s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 4m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
12s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 654 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
62m 12s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 3m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 19m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 38s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 23s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 5s
{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 8s
{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 31s
{color} | {color:green} hbase-hadoop-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 43s
{color} | {color:green} hbase-hadoop2-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 6s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 251m 0s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 48s
{color} | {color:green} hbase-rsgroup in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 2m
57s {color} | {color:green} The patch does not
[jira] [Updated] (HBASE-16392) Backup delete fault tolerance
[ https://issues.apache.org/jira/browse/HBASE-16392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16392: -- Description: Backup delete modified file system and backup system table. We have to make sure that operation is atomic, durable and isolated. Delete operation: # Start backup session (this guarantees) that system will be blocked for all backup commands during delete operation # Save list of tables being deleted to system table # Before delete operation we take backup system table snapshot # During delete operation we detect any failures and restore backup system table from snapshot, then finish backup session # To guarantee consistency of the data, delete operation MUST be repeated # We guarantee that all file delete operations are idempotent, can be repeated multiple times # Any backup operations will be blocked until consistency is restored # To restore consistency, repair command must be executed. # Repair command checks if there is failed delete op in a backup system table, and repeats delete operation was:Backup delete modified file system and backup system table. We have to make sure that operation is atomic, durable and isolated. > Backup delete fault tolerance > - > > Key: HBASE-16392 > URL: https://issues.apache.org/jira/browse/HBASE-16392 > Project: HBase > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Fix For: 2.0.0 > > Attachments: HBASE-16392-v1.patch > > > Backup delete modified file system and backup system table. We have to make > sure that operation is atomic, durable and isolated. > Delete operation: > # Start backup session (this guarantees) that system will be blocked for all > backup commands during delete operation > # Save list of tables being deleted to system table > # Before delete operation we take backup system table snapshot > # During delete operation we detect any failures and restore backup system > table from snapshot, then finish backup session > # To guarantee consistency of the data, delete operation MUST be repeated > # We guarantee that all file delete operations are idempotent, can be > repeated multiple times > # Any backup operations will be blocked until consistency is restored > # To restore consistency, repair command must be executed. > # Repair command checks if there is failed delete op in a backup system > table, and repeats delete operation -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17997) In dev environment, add jruby-complete jar to classpath only when jruby is needed
[
https://issues.apache.org/jira/browse/HBASE-17997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025389#comment-16025389
]
Hudson commented on HBASE-17997:
SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #3076 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/3076/])
HBASE-17997: In dev environment, add jruby-complete jar to classpath (tedyu:
rev 4ab94744e2e17a9ec4614ebb5ad80598e5dfbcd5)
* (edit) bin/hbase
* (edit) bin/hbase.cmd
* (edit) hbase-assembly/pom.xml
> In dev environment, add jruby-complete jar to classpath only when jruby is
> needed
> -
>
> Key: HBASE-17997
> URL: https://issues.apache.org/jira/browse/HBASE-17997
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Xiang Li
> Fix For: 2.0.0
>
> Attachments: HBASE-17997.master.000.patch,
> HBASE-17997.master.001.patch, HBASE-17997.master.002.patch,
> HBASE-17997.master.003.debug.diff, HBASE-17997.master.003.patch
>
>
> HBASE-15199 moves jruby-complete-1.6.8.jar to lib/ruby directory.
> However, jruby-complete-1.6.8.jar still appears in cached_classpath.txt
> This means that user would see exception similar to the following when
> starting hbase in standalone mode with s3a as rootdir :
> {code}
> 2017-05-04 16:41:32,854 WARN
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=38659]
> internal.S3MetadataResponseHandler: Unable to parse last modified date: Thu,
> 04 May 2017 16:27:09 GMT
> java.lang.IllegalStateException: Joda-time 2.2 or later version is required,
> but found version: null
> at com.amazonaws.util.DateUtils.handleException(DateUtils.java:149)
> at com.amazonaws.util.DateUtils.parseRFC822Date(DateUtils.java:195)
> at
> com.amazonaws.services.s3.internal.ServiceUtils.parseRfc822Date(ServiceUtils.java:78)
> at
> com.amazonaws.services.s3.internal.AbstractS3ResponseHandler.populateObjectMetadata(AbstractS3ResponseHandler.java:115)
> at
> com.amazonaws.services.s3.internal.S3ObjectResponseHandler.handle(S3ObjectResponseHandler.java:52)
> at
> com.amazonaws.services.s3.internal.S3ObjectResponseHandler.handle(S3ObjectResponseHandler.java:30)
> at
> com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:1072)
> at
> com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:746)
> at
> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
> at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
> at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785)
> at
> com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1191)
> at org.apache.hadoop.fs.s3a.S3AInputStream.reopen(S3AInputStream.java:148)
> at org.apache.hadoop.fs.s3a.S3AInputStream.lazySeek(S3AInputStream.java:281)
> at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:364)
> at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:75)
> at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:92)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.positionalReadWithExtra(HFileBlock.java:722)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1420)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1677)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1504)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:439)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:904)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:267)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:169)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:363)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:217)
> at
> org.apache.hadoop.hbase.regionserver.HStore.createScanner(HStore.java:2132)
> at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2122)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:5687)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2679)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2665)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2647)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6906)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6885)
> at
>
[jira] [Commented] (HBASE-16894) Create more than 1 split per region, generalize HBASE-12590
[
https://issues.apache.org/jira/browse/HBASE-16894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025366#comment-16025366
]
Mikhail Antonov commented on HBASE-16894:
-
I was unaware of this jira so drafted something similar on HBASE-18090, see
the patch attached.
> Create more than 1 split per region, generalize HBASE-12590
> ---
>
> Key: HBASE-16894
> URL: https://issues.apache.org/jira/browse/HBASE-16894
> Project: HBase
> Issue Type: Improvement
>Reporter: Enis Soztutar
>Assignee: Yi Liang
> Attachments: ImplementaionAndSomeQuestion.docx
>
>
> A common request from users is to be able to better control how many map
> tasks are created per region. Right now, it is always 1 region = 1 input
> split = 1 map task. Same goes for Spark since it uses the TIF. With region
> sizes as large as 50 GBs, it is desirable to be able to create more than 1
> split per region.
> HBASE-12590 adds a config property for MR jobs to be able to handle skew in
> region sizes. The algorithm is roughly:
> {code}
> If (region size >= average size*ratio) : cut the region into two MR input
> splits
> If (average size <= region size < average size*ratio) : one region as one MR
> input split
> If (sum of several continuous regions size < average size * ratio): combine
> these regions into one MR input split.
> {code}
> Although we can set data skew ratio to be 0.5 or something to abuse
> HBASE-12590 into creating more than 1 split task per region, it is not ideal.
> But there is no way to create more with the patch as it is. For example we
> cannot create more than 2 tasks per region.
> If we want to fix this properly, we should extend the approach in
> HBASE-12590, and make it so that the client can specify the desired num of
> mappers, or desired split size, and the TIF generates the splits based on the
> current region sizes very similar to the algorithm in HBASE-12590, but a more
> generic way. This also would eliminate the hand tuning of data skew ratio.
> We also can think about the guidepost approach that Phoenix has in the stats
> table which is used for exactly this purpose. Right now, the region can be
> split into powers of two assuming uniform distribution within the region.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18090) Improve TableSnapshotInputFormat to allow more multiple mappers per region
[ https://issues.apache.org/jira/browse/HBASE-18090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025363#comment-16025363 ] Mikhail Antonov commented on HBASE-18090: - Oh, thanks for reference! I didn't see this one. I don't see patches there, so might be this one would do some good. any feedback on the patch? I think assuming reasonably even key distribution across regions, giving just number of splits per region and split algo should suffice. Simpler and cheaper then trying to compute actual distribution based on data in HFiles. Still need to address feedback from [~tedyu], as well as some rough edges around how we create recovered.edits files during openRegion sequence. cc [~enis] [~esteban] ? > Improve TableSnapshotInputFormat to allow more multiple mappers per region > -- > > Key: HBASE-18090 > URL: https://issues.apache.org/jira/browse/HBASE-18090 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Affects Versions: 1.4.0 >Reporter: Mikhail Antonov > Attachments: HBASE-18090-branch-1.3-v1.patch > > > TableSnapshotInputFormat runs one map task per region in the table snapshot. > This places unnecessary restriction that the region layout of the original > table needs to take the processing resources available to MR job into > consideration. Allowing to run multiple mappers per region (assuming > reasonably even key distribution) would be useful. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16415) Replication in different namespace
[
https://issues.apache.org/jira/browse/HBASE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025323#comment-16025323
]
Jean-Marc Spaggiari commented on HBASE-16415:
-
Indeed, it will be very nice! Anyone want to take this?
> Replication in different namespace
> --
>
> Key: HBASE-16415
> URL: https://issues.apache.org/jira/browse/HBASE-16415
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Reporter: Christian Guegi
>
> It would be nice to replicate tables from one namespace to another namespace.
> Example:
> Master cluster, namespace=default, table=bar
> Slave cluster, namespace=dr, table=bar
> Replication happens in class ReplicationSink:
> public void replicateEntries(List entries, final CellScanner
> cells, ...){
> ...
> TableName table =
> TableName.valueOf(entry.getKey().getTableName().toByteArray());
> ...
> addToHashMultiMap(rowMap, table, clusterIds, m);
> ...
> for (Entry> entry :
> rowMap.entrySet()) {
> batch(entry.getKey(), entry.getValue().values());
> }
>}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-16011) TableSnapshotScanner and TableSnapshotInputFormat can produce duplicate rows
[ https://issues.apache.org/jira/browse/HBASE-16011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025281#comment-16025281 ] Ted Yu commented on HBASE-16011: Please attach branch-1 patch. > TableSnapshotScanner and TableSnapshotInputFormat can produce duplicate rows > > > Key: HBASE-16011 > URL: https://issues.apache.org/jira/browse/HBASE-16011 > Project: HBase > Issue Type: Bug > Components: snapshots >Affects Versions: 2.0.0, 1.2.2 >Reporter: Youngjoon Kim >Assignee: Zheng Hu > Attachments: HBASE-16011.v1.patch, HBASE-16011.v2.patch, > HBASE-16011.v2.patch, snapshot_bug_test.patch > > > A snapshot of (non-pre) split table can include both a parent region and > daughter regions. If run TableSnapshotScanner or TableSnapshotInputFormat on > the such snapshot, duplicate rows are produced. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16011) TableSnapshotScanner and TableSnapshotInputFormat can produce duplicate rows
[ https://issues.apache.org/jira/browse/HBASE-16011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025276#comment-16025276 ] Ted Yu commented on HBASE-16011: Second attempt ended up with same error. I ran failed tests with patch v2 which passed. > TableSnapshotScanner and TableSnapshotInputFormat can produce duplicate rows > > > Key: HBASE-16011 > URL: https://issues.apache.org/jira/browse/HBASE-16011 > Project: HBase > Issue Type: Bug > Components: snapshots >Affects Versions: 2.0.0, 1.2.2 >Reporter: Youngjoon Kim >Assignee: Zheng Hu > Attachments: HBASE-16011.v1.patch, HBASE-16011.v2.patch, > HBASE-16011.v2.patch, snapshot_bug_test.patch > > > A snapshot of (non-pre) split table can include both a parent region and > daughter regions. If run TableSnapshotScanner or TableSnapshotInputFormat on > the such snapshot, duplicate rows are produced. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (HBASE-17966) Inconsistent logic in Admin.createTable vs Admin.tableExists
[ https://issues.apache.org/jira/browse/HBASE-17966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinmay Kulkarni resolved HBASE-17966. -- Resolution: Not A Bug Assignee: Chinmay Kulkarni This issue does not exist for master branch. Closing this JIRA. > Inconsistent logic in Admin.createTable vs Admin.tableExists > > > Key: HBASE-17966 > URL: https://issues.apache.org/jira/browse/HBASE-17966 > Project: HBase > Issue Type: Bug > Components: Admin >Affects Versions: 1.2.2 >Reporter: Jeff Saremi >Assignee: Chinmay Kulkarni > > tableExists can return false for cases where createTable fails. Obviously the > logic implemented in these two methods are different. > This is to make this implementation consistent -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18066) Get with closest_row_before on "hbase:meta" can return empty Cell during region merge/split
[
https://issues.apache.org/jira/browse/HBASE-18066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025242#comment-16025242
]
Hadoop QA commented on HBASE-18066:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 31s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 1s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
55s {color} | {color:green} branch-1.3 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 9s
{color} | {color:green} branch-1.3 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
37s {color} | {color:green} branch-1.3 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s {color} | {color:green} branch-1.3 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 6s
{color} | {color:red} hbase-server in branch-1.3 has 1 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s
{color} | {color:green} branch-1.3 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 13s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 13s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
35m 44s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 175m 28s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
44s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 235m 44s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestFromClientSideWithCoprocessor |
| | hadoop.hbase.regionserver.TestHRegion |
| | hadoop.hbase.client.TestAdmin2 |
| | hadoop.hbase.client.TestFromClientSide |
| Timed out junit tests | org.apache.hadoop.hbase.util.TestHBaseFsck |
| | org.apache.hadoop.hbase.snapshot.TestSecureExportSnapshot |
| | org.apache.hadoop.hbase.snapshot.TestExportSnapshot |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.03.0-ce Server=17.03.0-ce Image:yetus/hbase:e1e11ad |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12869884/HBASE-18066.branch-1.3.v1.patch
|
| JIRA Issue | HBASE-18066 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 07e134febb31 4.8.3-std-1 #1 SMP Fri Oct 21 11:15:43 UTC 2016
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/hbase.sh |
| git revision | branch-1.3 / 17e87ae |
| Default Java | 1.8.0_131 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HBASE-Build/6953/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/6953/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/6953/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results
[jira] [Updated] (HBASE-18116) Replication source in-memory accounting should not include bulk transfer hfiles
[ https://issues.apache.org/jira/browse/HBASE-18116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-18116: --- Summary: Replication source in-memory accounting should not include bulk transfer hfiles (was: Replication buffer quota accounting should not include bulk transfer hfiles) > Replication source in-memory accounting should not include bulk transfer > hfiles > --- > > Key: HBASE-18116 > URL: https://issues.apache.org/jira/browse/HBASE-18116 > Project: HBase > Issue Type: Bug > Components: Replication >Reporter: Andrew Purtell > > In ReplicationSourceWALReaderThread we maintain a global quota on enqueued > replication work for preventing OOM by queuing up too many edits into queues > on heap. When calculating the size of a given replication queue entry, if it > has associated hfiles (is a bulk load to be replicated as a batch of hfiles), > we get the file sizes and include the sum. We then apply that result to the > quota. This isn't quite right. Those hfiles will be pulled by the sink as a > file copy, not pushed by the source. The cells in those files are not queued > in memory at the source and therefore shouldn't be counted against the quota. > Related, the sum of the hfile sizes are also included when checking if queued > work exceeds the configured replication queue capacity, which is by default > 64 MB. HFiles are commonly much larger than this. > So what happens is when we encounter a bulk load replication entry typically > both the quota and capacity limits are exceeded, we break out of loops, and > send right away. What is transferred on the wire via HBase RPC though has > only a partial relationship to the calculation. > Depending how you look at it, it makes sense to factor hfile file sizes > against replication queue capacity limits. The sink will be occupied > transferring those files at the HDFS level. Anyway, this is how we have been > doing it and it is too late to change now. I do not however think it is > correct to apply hfile file sizes against a quota for in memory state on the > source. The source doesn't queue or even transfer those bytes. > Something I noticed while working on HBASE-18027. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HBASE-18116) Replication buffer quota accounting should not include bulk transfer hfiles
[ https://issues.apache.org/jira/browse/HBASE-18116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025196#comment-16025196 ] Andrew Purtell edited comment on HBASE-18116 at 5/25/17 7:00 PM: - In addition, when estimating the size of a replication queue entry we only consider the WALEdit objects, not the also associated WALKey objects. was (Author: apurtell): Also, when calculating the heap size of a replication queue entry we only track the WALEdit objects, not the associated WALKey objects. > Replication buffer quota accounting should not include bulk transfer hfiles > --- > > Key: HBASE-18116 > URL: https://issues.apache.org/jira/browse/HBASE-18116 > Project: HBase > Issue Type: Bug > Components: Replication >Reporter: Andrew Purtell > > In ReplicationSourceWALReaderThread we maintain a global quota on enqueued > replication work for preventing OOM by queuing up too many edits into queues > on heap. When calculating the size of a given replication queue entry, if it > has associated hfiles (is a bulk load to be replicated as a batch of hfiles), > we get the file sizes and include the sum. We then apply that result to the > quota. This isn't quite right. Those hfiles will be pulled by the sink as a > file copy, not pushed by the source. The cells in those files are not queued > in memory at the source and therefore shouldn't be counted against the quota. > Related, the sum of the hfile sizes are also included when checking if queued > work exceeds the configured replication queue capacity, which is by default > 64 MB. HFiles are commonly much larger than this. > So what happens is when we encounter a bulk load replication entry typically > both the quota and capacity limits are exceeded, we break out of loops, and > send right away. What is transferred on the wire via HBase RPC though has > only a partial relationship to the calculation. > Depending how you look at it, it makes sense to factor hfile file sizes > against replication queue capacity limits. The sink will be occupied > transferring those files at the HDFS level. Anyway, this is how we have been > doing it and it is too late to change now. I do not however think it is > correct to apply hfile file sizes against a quota for in memory state on the > source. The source doesn't queue or even transfer those bytes. > Something I noticed while working on HBASE-18027. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18027) Replication should respect RPC size limits when batching edits
[ https://issues.apache.org/jira/browse/HBASE-18027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025200#comment-16025200 ] Andrew Purtell commented on HBASE-18027: Nothing I can do about HBASE-18116 if separating it out. So will commit this. It doesn't make the situation any worse. > Replication should respect RPC size limits when batching edits > -- > > Key: HBASE-18027 > URL: https://issues.apache.org/jira/browse/HBASE-18027 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.4.0, 1.3.1 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.4.0, 1.3.2 > > Attachments: HBASE-18027.patch, HBASE-18027.patch, HBASE-18027.patch, > HBASE-18027.patch, HBASE-18027.patch > > > In HBaseInterClusterReplicationEndpoint#replicate we try to replicate in > batches. We create N lists. N is the minimum of configured replicator > threads, number of 100-waledit batches, or number of current sinks. Every > pending entry in the replication context is then placed in order by hash of > encoded region name into one of these N lists. Each of the N lists is then > sent all at once in one replication RPC. We do not test if the sum of data in > each N list will exceed RPC size limits. This code presumes each individual > edit is reasonably small. Not checking for aggregate size while assembling > the lists into RPCs is an oversight and can lead to replication failure when > that assumption is violated. > We can fix this by generating as many replication RPC calls as we need to > drain a list, keeping each RPC under limit, instead of assuming the whole > list will fit in one. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HBASE-18027) Replication should respect RPC size limits when batching edits
[ https://issues.apache.org/jira/browse/HBASE-18027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025200#comment-16025200 ] Andrew Purtell edited comment on HBASE-18027 at 5/25/17 6:47 PM: - Nothing I can do about HBASE-18116 here if separating it out. So will commit this. It doesn't make the situation any worse. was (Author: apurtell): Nothing I can do about HBASE-18116 if separating it out. So will commit this. It doesn't make the situation any worse. > Replication should respect RPC size limits when batching edits > -- > > Key: HBASE-18027 > URL: https://issues.apache.org/jira/browse/HBASE-18027 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.4.0, 1.3.1 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.4.0, 1.3.2 > > Attachments: HBASE-18027.patch, HBASE-18027.patch, HBASE-18027.patch, > HBASE-18027.patch, HBASE-18027.patch > > > In HBaseInterClusterReplicationEndpoint#replicate we try to replicate in > batches. We create N lists. N is the minimum of configured replicator > threads, number of 100-waledit batches, or number of current sinks. Every > pending entry in the replication context is then placed in order by hash of > encoded region name into one of these N lists. Each of the N lists is then > sent all at once in one replication RPC. We do not test if the sum of data in > each N list will exceed RPC size limits. This code presumes each individual > edit is reasonably small. Not checking for aggregate size while assembling > the lists into RPCs is an oversight and can lead to replication failure when > that assumption is violated. > We can fix this by generating as many replication RPC calls as we need to > drain a list, keeping each RPC under limit, instead of assuming the whole > list will fit in one. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18116) Replication buffer quota accounting should not include bulk transfer hfiles
[ https://issues.apache.org/jira/browse/HBASE-18116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025196#comment-16025196 ] Andrew Purtell commented on HBASE-18116: Also, when calculating the heap size of a replication queue entry we only track the WALEdit objects, not the associated WALKey objects. > Replication buffer quota accounting should not include bulk transfer hfiles > --- > > Key: HBASE-18116 > URL: https://issues.apache.org/jira/browse/HBASE-18116 > Project: HBase > Issue Type: Bug > Components: Replication >Reporter: Andrew Purtell > > In ReplicationSourceWALReaderThread we maintain a global quota on enqueued > replication work for preventing OOM by queuing up too many edits into queues > on heap. When calculating the size of a given replication queue entry, if it > has associated hfiles (is a bulk load to be replicated as a batch of hfiles), > we get the file sizes and include the sum. We then apply that result to the > quota. This isn't quite right. Those hfiles will be pulled by the sink as a > file copy, not pushed by the source. The cells in those files are not queued > in memory at the source and therefore shouldn't be counted against the quota. > Related, the sum of the hfile sizes are also included when checking if queued > work exceeds the configured replication queue capacity, which is by default > 64 MB. HFiles are commonly much larger than this. > So what happens is when we encounter a bulk load replication entry typically > both the quota and capacity limits are exceeded, we break out of loops, and > send right away. What is transferred on the wire via HBase RPC though has > only a partial relationship to the calculation. > Depending how you look at it, it makes sense to factor hfile file sizes > against replication queue capacity limits. The sink will be occupied > transferring those files at the HDFS level. Anyway, this is how we have been > doing it and it is too late to change now. I do not however think it is > correct to apply hfile file sizes against a quota for in memory state on the > source. The source doesn't queue or even transfer those bytes. > Something I noticed while working on HBASE-18027. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18116) Replication buffer quota accounting should not include bulk transfer hfiles
[ https://issues.apache.org/jira/browse/HBASE-18116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-18116: --- Description: In ReplicationSourceWALReaderThread we maintain a global quota on enqueued replication work for preventing OOM by queuing up too many edits into queues on heap. When calculating the size of a given replication queue entry, if it has associated hfiles (is a bulk load to be replicated as a batch of hfiles), we get the file sizes and include the sum. We then apply that result to the quota. This isn't quite right. Those hfiles will be pulled by the sink as a file copy, not pushed by the source. The cells in those files are not queued in memory at the source and therefore shouldn't be counted against the quota. Related, the sum of the hfile sizes are also included when checking if queued work exceeds the configured replication queue capacity, which is by default 64 MB. HFiles are commonly much larger than this. So what happens is when we encounter a bulk load replication entry typically both the quota and capacity limits are exceeded, we break out of loops, and send right away. What is transferred on the wire via HBase RPC though has only a partial relationship to the calculation. Depending how you look at it, it makes sense to factor hfile file sizes against replication queue capacity limits. The sink will be occupied transferring those files at the HDFS level. Anyway, this is how we have been doing it and it is too late to change now. I do not however think it is correct to apply hfile file sizes against a quota for in memory state on the source. The source doesn't queue or even transfer those bytes. Something I noticed while working on HBASE-18027. was: In ReplicationSourceWALReaderThread we maintain a global quota on enqueued replication work for preventing OOM by queuing up too many edits into queues on heap. When calculating the size of a given replication queue entry, if it has associated hfiles (is a bulk load to be replicated as a batch of hfiles), we get the file sizes and include the sum. We then apply that result to the quota. This isn't quite right. Those hfiles will be transferred by the sink as a file copy and then the files will be bulk loaded on the sink. The cells in those files are not queued in memory at the source and therefore shouldn't be counted against the quota. Related, the sum of the hfile sizes are also included when checking if queued work exceeds the configured replication queue capacity, which is by default 64 MB. HFiles are commonly much larger than this. So what happens is when we encounter a bulk load replication entry typically both the quota and capacity limits are exceeded, we break out of loops, and send right away. What is transferred on the wire via HBase RPC though has only a partial relationship to the calculation. Depending how you look at it, it makes sense to factor hfile file sizes against replication queue capacity limits. The sink will be occupied transferring those files at the HDFS level. Anyway, this is how we have been doing it and it is too late to change now. I do not however think it is correct to apply hfile file sizes against a quota for in memory state on the source. The source doesn't queue or even transfer those bytes. Something I noticed while working on HBASE-18027. > Replication buffer quota accounting should not include bulk transfer hfiles > --- > > Key: HBASE-18116 > URL: https://issues.apache.org/jira/browse/HBASE-18116 > Project: HBase > Issue Type: Bug > Components: Replication >Reporter: Andrew Purtell > > In ReplicationSourceWALReaderThread we maintain a global quota on enqueued > replication work for preventing OOM by queuing up too many edits into queues > on heap. When calculating the size of a given replication queue entry, if it > has associated hfiles (is a bulk load to be replicated as a batch of hfiles), > we get the file sizes and include the sum. We then apply that result to the > quota. This isn't quite right. Those hfiles will be pulled by the sink as a > file copy, not pushed by the source. The cells in those files are not queued > in memory at the source and therefore shouldn't be counted against the quota. > Related, the sum of the hfile sizes are also included when checking if queued > work exceeds the configured replication queue capacity, which is by default > 64 MB. HFiles are commonly much larger than this. > So what happens is when we encounter a bulk load replication entry typically > both the quota and capacity limits are exceeded, we break out of loops, and > send right away. What is transferred on the wire via HBase RPC though has > only a partial relationship to the calculation. >
[jira] [Created] (HBASE-18116) Replication buffer quota accounting should not include bulk transfer hfiles
Andrew Purtell created HBASE-18116: -- Summary: Replication buffer quota accounting should not include bulk transfer hfiles Key: HBASE-18116 URL: https://issues.apache.org/jira/browse/HBASE-18116 Project: HBase Issue Type: Bug Components: Replication Reporter: Andrew Purtell In ReplicationSourceWALReaderThread we maintain a global quota on enqueued replication work for preventing OOM by queuing up too many edits into queues on heap. When calculating the size of a given replication queue entry, if it has associated hfiles (is a bulk load to be replicated as a batch of hfiles), we get the file sizes and include the sum. We then apply that result to the quota. This isn't quite right. Those hfiles will be transferred by the sink as a file copy and then the files will be bulk loaded on the sink. The cells in those files are not queued in memory at the source and therefore shouldn't be counted against the quota. Related, the sum of the hfile sizes are also included when checking if queued work exceeds the configured replication queue capacity, which is by default 64 MB. HFiles are commonly much larger than this. So what happens is when we encounter a bulk load replication entry typically both the quota and capacity limits are exceeded, we break out of loops, and send right away. What is transferred on the wire via HBase RPC though has only a partial relationship to the calculation. Depending how you look at it, it makes sense to factor hfile file sizes against replication queue capacity limits. The sink will be occupied transferring those files at the HDFS level. Anyway, this is how we have been doing it and it is too late to change now. I do not however think it is correct to apply hfile file sizes against a quota for in memory state on the source. The source doesn't queue or even transfer those bytes. Something I noticed while working on HBASE-18027. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18027) Replication should respect RPC size limits when batching edits
[ https://issues.apache.org/jira/browse/HBASE-18027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025161#comment-16025161 ] Andrew Purtell commented on HBASE-18027: I filed HBASE-18116 for some accounting improvements outside the scope of this work. > Replication should respect RPC size limits when batching edits > -- > > Key: HBASE-18027 > URL: https://issues.apache.org/jira/browse/HBASE-18027 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.4.0, 1.3.1 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.4.0, 1.3.2 > > Attachments: HBASE-18027.patch, HBASE-18027.patch, HBASE-18027.patch, > HBASE-18027.patch, HBASE-18027.patch > > > In HBaseInterClusterReplicationEndpoint#replicate we try to replicate in > batches. We create N lists. N is the minimum of configured replicator > threads, number of 100-waledit batches, or number of current sinks. Every > pending entry in the replication context is then placed in order by hash of > encoded region name into one of these N lists. Each of the N lists is then > sent all at once in one replication RPC. We do not test if the sum of data in > each N list will exceed RPC size limits. This code presumes each individual > edit is reasonably small. Not checking for aggregate size while assembling > the lists into RPCs is an oversight and can lead to replication failure when > that assumption is violated. > We can fix this by generating as many replication RPC calls as we need to > drain a list, keeping each RPC under limit, instead of assuming the whole > list will fit in one. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18090) Improve TableSnapshotInputFormat to allow more multiple mappers per region
[ https://issues.apache.org/jira/browse/HBASE-18090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025154#comment-16025154 ] Jerry He commented on HBASE-18090: -- HBASE-16894 tries to accomplish something similar. > Improve TableSnapshotInputFormat to allow more multiple mappers per region > -- > > Key: HBASE-18090 > URL: https://issues.apache.org/jira/browse/HBASE-18090 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Affects Versions: 1.4.0 >Reporter: Mikhail Antonov > Attachments: HBASE-18090-branch-1.3-v1.patch > > > TableSnapshotInputFormat runs one map task per region in the table snapshot. > This places unnecessary restriction that the region layout of the original > table needs to take the processing resources available to MR job into > consideration. Allowing to run multiple mappers per region (assuming > reasonably even key distribution) would be useful. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18114) Update the config of TestAsync*AdminApi to make test stable
[
https://issues.apache.org/jira/browse/HBASE-18114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025151#comment-16025151
]
Hadoop QA commented on HBASE-18114:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 35s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
59s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 0s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
31m 43s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 100m 40s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
56s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 148m 4s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.TestZooKeeper |
| | org.apache.hadoop.hbase.mob.compactions.TestPartitionedMobCompactor |
| | org.apache.hadoop.hbase.mob.compactions.TestMobCompactor |
| | org.apache.hadoop.hbase.snapshot.TestMobExportSnapshot |
| | org.apache.hadoop.hbase.TestIOFencing |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.10.1 Server=1.10.1 Image:yetus/hbase:757bf37 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12869890/HBASE-18114-v1.patch |
| JIRA Issue | HBASE-18114 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux cbf7c4ba8b8c 3.19.0-25-generic #26~14.04.1-Ubuntu SMP Fri Jul
24 21:16:20 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 4ab9474 |
| Default Java | 1.8.0_131 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/6955/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/6955/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6955/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6955/console |
| Powered
[jira] [Commented] (HBASE-16392) Backup delete fault tolerance
[
https://issues.apache.org/jira/browse/HBASE-16392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025146#comment-16025146
]
Ted Yu commented on HBASE-16392:
For restoreFromSnapshot() :
{code}
+ if (snapshotExists(admin, snapshotName)) {
+admin.disableTable(BackupSystemTable.getTableName(conf));
+admin.restoreSnapshot(snapshotName);
+admin.enableTable(BackupSystemTable.getTableName(conf));
{code}
What if one of the 3 actions above fails ?
{code}
+// In this case we log WARN and proceed
+LOG.error("Could not restore backup system table. Snapshot " +
snapshotName+
{code}
Comment and actual log level are different.
{code}
+ protected static boolean snapshotExists(Admin admin, String snapshotName)
throws IOException {
{code}
TableBackupClient.java already has snapshotExists() method.
{code}
+ private Delete createDeleteForDeleteOperation() {
{code}
createDeleteForDeleteOperation -> createDeleteForBackupDeleteOperation
> Backup delete fault tolerance
> -
>
> Key: HBASE-16392
> URL: https://issues.apache.org/jira/browse/HBASE-16392
> Project: HBase
> Issue Type: Sub-task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-16392-v1.patch
>
>
> Backup delete modified file system and backup system table. We have to make
> sure that operation is atomic, durable and isolated.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18114) Update the config of TestAsync*AdminApi to make test stable
[
https://issues.apache.org/jira/browse/HBASE-18114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025139#comment-16025139
]
Hadoop QA commented on HBASE-18114:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
7s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 59s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 110m 20s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 149m 37s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:757bf37 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12869888/HBASE-18114-v1.patch |
| JIRA Issue | HBASE-18114 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux d57641140c34 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 4ab9474 |
| Default Java | 1.8.0_131 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6954/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6954/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Update the config of TestAsync*AdminApi to make test stable
> ---
>
> Key: HBASE-18114
> URL: https://issues.apache.org/jira/browse/HBASE-18114
> Project: HBase
> Issue Type: Sub-task
> Components: Client
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-18114-v1.patch,
[jira] [Commented] (HBASE-18099) FlushSnapshotSubprocedure should wait for concurrent Region#flush() to finish
[
https://issues.apache.org/jira/browse/HBASE-18099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025110#comment-16025110
]
Hudson commented on HBASE-18099:
SUCCESS: Integrated in Jenkins build HBase-1.4 #748 (See
[https://builds.apache.org/job/HBase-1.4/748/])
HBASE-18099 FlushSnapshotSubprocedure should wait for concurrent (tedyu: rev
109e0d5485af3abf6914c8c782c3700477f47646)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Region.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/snapshot/FlushSnapshotSubprocedure.java
> FlushSnapshotSubprocedure should wait for concurrent Region#flush() to finish
> -
>
> Key: HBASE-18099
> URL: https://issues.apache.org/jira/browse/HBASE-18099
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.4.0
>
> Attachments: 18099.v1.txt, 18099.v2.txt, 18099.v3.txt, 18099.v4.txt
>
>
> In the following thread:
> http://search-hadoop.com/m/HBase/YGbbMXkeHlI9zo
> Jacob described the scenario where data from certain region were missing in
> the snapshot.
> Here was related region server log:
> https://pastebin.com/1ECXjhRp
> He pointed out that concurrent flush from MemStoreFlusher.1 thread was not
> initiated from the thread pool for snapshot.
> In RegionSnapshotTask#call() method there is this:
> {code}
> region.flush(true);
> {code}
> The return value is not checked.
> In HRegion#flushcache(), Result.CANNOT_FLUSH may be returned due to:
> {code}
> String msg = "Not flushing since "
> + (writestate.flushing ? "already flushing"
> : "writes not enabled");
> {code}
> This implies that FlushSnapshotSubprocedure may incorrectly skip waiting for
> the concurrent flush to complete.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18027) Replication should respect RPC size limits when batching edits
[ https://issues.apache.org/jira/browse/HBASE-18027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025101#comment-16025101 ] Andrew Purtell commented on HBASE-18027: Although this works, I'd like to improve it before commit by making an estimate of RPC size independent of accounting against the replication queue limit, mainly to exclude in the bulk load case the bytes in the storefiles, which will not be transferred via RPC. Current patch lets this case through but it is not quite right. Back with another rev shortly. > Replication should respect RPC size limits when batching edits > -- > > Key: HBASE-18027 > URL: https://issues.apache.org/jira/browse/HBASE-18027 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.4.0, 1.3.1 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.4.0, 1.3.2 > > Attachments: HBASE-18027.patch, HBASE-18027.patch, HBASE-18027.patch, > HBASE-18027.patch, HBASE-18027.patch > > > In HBaseInterClusterReplicationEndpoint#replicate we try to replicate in > batches. We create N lists. N is the minimum of configured replicator > threads, number of 100-waledit batches, or number of current sinks. Every > pending entry in the replication context is then placed in order by hash of > encoded region name into one of these N lists. Each of the N lists is then > sent all at once in one replication RPC. We do not test if the sum of data in > each N list will exceed RPC size limits. This code presumes each individual > edit is reasonably small. Not checking for aggregate size while assembling > the lists into RPCs is an oversight and can lead to replication failure when > that assumption is violated. > We can fix this by generating as many replication RPC calls as we need to > drain a list, keeping each RPC under limit, instead of assuming the whole > list will fit in one. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18027) Replication should respect RPC size limits when batching edits
[ https://issues.apache.org/jira/browse/HBASE-18027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-18027: --- Status: Open (was: Patch Available) > Replication should respect RPC size limits when batching edits > -- > > Key: HBASE-18027 > URL: https://issues.apache.org/jira/browse/HBASE-18027 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.3.1, 2.0.0, 1.4.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.4.0, 1.3.2 > > Attachments: HBASE-18027.patch, HBASE-18027.patch, HBASE-18027.patch, > HBASE-18027.patch, HBASE-18027.patch > > > In HBaseInterClusterReplicationEndpoint#replicate we try to replicate in > batches. We create N lists. N is the minimum of configured replicator > threads, number of 100-waledit batches, or number of current sinks. Every > pending entry in the replication context is then placed in order by hash of > encoded region name into one of these N lists. Each of the N lists is then > sent all at once in one replication RPC. We do not test if the sum of data in > each N list will exceed RPC size limits. This code presumes each individual > edit is reasonably small. Not checking for aggregate size while assembling > the lists into RPCs is an oversight and can lead to replication failure when > that assumption is violated. > We can fix this by generating as many replication RPC calls as we need to > drain a list, keeping each RPC under limit, instead of assuming the whole > list will fit in one. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16392) Backup delete fault tolerance
[
https://issues.apache.org/jira/browse/HBASE-16392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025055#comment-16025055
]
Ted Yu commented on HBASE-16392:
{code}
++ "If there is no active backup session is running, run backup
repair utility to restore \n"
{code}
First part of the sentence has double 'is'
{code}
+finalizeDelete(allTablesMap, sysTable);
+// Finish
+sysTable.finishDeleteOperation();
+// delete snapshot
+BackupSystemTable.deleteSnapshot(conn);
{code}
Do we need to distinguish potential failure in each of the above calls ?
If deleteSnapshot() fails in the middle,
BackupSystemTable.restoreFromSnapshot(conn) call in catch block would fail as
well, right ?
{code}
+ String[] backupIds = sysTable.getDeleteOperationListOfBackupIds();
{code}
getDeleteOperationListOfBackupIds -> getListOfBackupIdsFromDeleteOperation
{code}
+ System.out.println("Delete operation finished OK");
{code}
Add information about the backup Ids involved.
Please put the next patch on review board.
> Backup delete fault tolerance
> -
>
> Key: HBASE-16392
> URL: https://issues.apache.org/jira/browse/HBASE-16392
> Project: HBase
> Issue Type: Sub-task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-16392-v1.patch
>
>
> Backup delete modified file system and backup system table. We have to make
> sure that operation is atomic, durable and isolated.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18042) Client Compatibility breaks between versions 1.2 and 1.3
[
https://issues.apache.org/jira/browse/HBASE-18042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025047#comment-16025047
]
Sean Busbey commented on HBASE-18042:
-
looks like the jdk the branch-1 dockerfile wants isn't available from oracle.
> Client Compatibility breaks between versions 1.2 and 1.3
>
>
> Key: HBASE-18042
> URL: https://issues.apache.org/jira/browse/HBASE-18042
> Project: HBase
> Issue Type: Bug
> Components: regionserver, scan
>Affects Versions: 2.0.0, 1.4.0, 1.3.1
>Reporter: Karan Mehta
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0, 1.4.0, 1.3.2
>
> Attachments: HBASE-18042-branch-1.3.patch,
> HBASE-18042-branch-1.patch, HBASE-18042-branch-1.patch,
> HBASE-18042-branch-1-v1.patch, HBASE-18042-branch-1-v1.patch,
> HBASE-18042.patch, HBASE-18042-v1.patch, HBASE-18042-v2.patch
>
>
> OpenTSDB uses AsyncHBase as its client, rather than using the traditional
> HBase Client. From version 1.2 to 1.3, the {{ClientProtos}} have been
> changed. Newer fields are added to {{ScanResponse}} proto.
> For a typical Scan request in 1.2, would require caller to make an
> OpenScanner Request, GetNextRows Request and a CloseScanner Request, based on
> {{more_rows}} boolean field in the {{ScanResponse}} proto.
> However, from 1.3, new parameter {{more_results_in_region}} was added, which
> limits the results per region. Therefore the client has to now manage sending
> all the requests for each region. Further more, if the results are exhausted
> from a particular region, the {{ScanResponse}} will set
> {{more_results_in_region}} to false, but {{more_results}} can still be true.
> Whenever the former is set to false, the {{RegionScanner}} will also be
> closed.
> OpenTSDB makes an OpenScanner Request and receives all its results in the
> first {{ScanResponse}} itself, thus creating a condition as described in
> above paragraph. Since {{more_rows}} is true, it will proceed to send next
> request at which point the {{RSRpcServices}} will throw
> {{UnknownScannerException}}. The protobuf client compatibility is maintained
> but expected behavior is modified.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18095) Provide an option for clients to find the server hosting META that does not involve the ZooKeeper client
[ https://issues.apache.org/jira/browse/HBASE-18095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025003#comment-16025003 ] Andrew Purtell commented on HBASE-18095: bq. As far as I know, in branch-1, replication information, like replication peers are stored in ZK, if user want to list or modify replication relationship using ReplicationAdmin, they need zookeeper client Yes, but this is admin action. The scope of the JIRA is for normal client access. Just to be clear. :-) > Provide an option for clients to find the server hosting META that does not > involve the ZooKeeper client > > > Key: HBASE-18095 > URL: https://issues.apache.org/jira/browse/HBASE-18095 > Project: HBase > Issue Type: New Feature > Components: Client >Reporter: Andrew Purtell > > Clients are required to connect to ZooKeeper to find the location of the > regionserver hosting the meta table region. Site configuration provides the > client a list of ZK quorum peers and the client uses an embedded ZK client to > query meta location. Timeouts and retry behavior of this embedded ZK client > are managed orthogonally to HBase layer settings and in some cases the ZK > cannot manage what in theory the HBase client can, i.e. fail fast upon outage > or network partition. > We should consider new configuration settings that provide a list of > well-known master and backup master locations, and with this information the > client can contact any of the master processes directly. Any master in either > active or passive state will track meta location and respond to requests for > it with its cached last known location. If this location is stale, the client > can ask again with a flag set that requests the master refresh its location > cache and return the up-to-date location. Every client interaction with the > cluster thus uses only HBase RPC as transport, with appropriate settings > applied to the connection. The configuration toggle that enables this > alternative meta location lookup should be false by default. > This removes the requirement that HBase clients embed the ZK client and > contact the ZK service directly at the beginning of the connection lifecycle. > This has several benefits. ZK service need not be exposed to clients, and > their potential abuse, yet no benefit ZK provides the HBase server cluster is > compromised. Normalizing HBase client and ZK client timeout settings and > retry behavior - in some cases, impossible, i.e. for fail-fast - is no longer > necessary. > And, from [~ghelmling]: There is an additional complication here for > token-based authentication. When a delegation token is used for SASL > authentication, the client uses the cluster ID obtained from Zookeeper to > select the token identifier to use. So there would also need to be some > Zookeeper-less, unauthenticated way to obtain the cluster ID as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-15903) Delete Object
[ https://issues.apache.org/jira/browse/HBASE-15903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024986#comment-16024986 ] Ted Yu commented on HBASE-15903: In patch v2, I enhanced ClientTest.PutGet with deleting the row just written and verifying that subsequent Get returns nothing. > Delete Object > - > > Key: HBASE-15903 > URL: https://issues.apache.org/jira/browse/HBASE-15903 > Project: HBase > Issue Type: Sub-task >Reporter: Sudeep Sunthankar >Assignee: Ted Yu > Attachments: 15903.v2.txt, HBASE-15903.HBASE-14850.v1.patch > > > Patch for creating Delete objects. These Delete objects are used by the Table > implementation to delete rowkey from a table. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-15903) Delete Object
[ https://issues.apache.org/jira/browse/HBASE-15903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15903: --- Attachment: (was: 15903.v2.txt) > Delete Object > - > > Key: HBASE-15903 > URL: https://issues.apache.org/jira/browse/HBASE-15903 > Project: HBase > Issue Type: Sub-task >Reporter: Sudeep Sunthankar >Assignee: Ted Yu > Attachments: 15903.v2.txt, HBASE-15903.HBASE-14850.v1.patch > > > Patch for creating Delete objects. These Delete objects are used by the Table > implementation to delete rowkey from a table. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-15903) Delete Object
[ https://issues.apache.org/jira/browse/HBASE-15903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15903: --- Attachment: 15903.v2.txt > Delete Object > - > > Key: HBASE-15903 > URL: https://issues.apache.org/jira/browse/HBASE-15903 > Project: HBase > Issue Type: Sub-task >Reporter: Sudeep Sunthankar >Assignee: Ted Yu > Attachments: 15903.v2.txt, HBASE-15903.HBASE-14850.v1.patch > > > Patch for creating Delete objects. These Delete objects are used by the Table > implementation to delete rowkey from a table. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18099) FlushSnapshotSubprocedure should wait for concurrent Region#flush() to finish
[
https://issues.apache.org/jira/browse/HBASE-18099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024929#comment-16024929
]
Hudson commented on HBASE-18099:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3075 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/3075/])
HBASE-18099 FlushSnapshotSubprocedure should wait for concurrent (tedyu: rev
3e426b2f851f40dbc4e8d53c607cddebdb8a73e0)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Region.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/snapshot/FlushSnapshotSubprocedure.java
> FlushSnapshotSubprocedure should wait for concurrent Region#flush() to finish
> -
>
> Key: HBASE-18099
> URL: https://issues.apache.org/jira/browse/HBASE-18099
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.4.0
>
> Attachments: 18099.v1.txt, 18099.v2.txt, 18099.v3.txt, 18099.v4.txt
>
>
> In the following thread:
> http://search-hadoop.com/m/HBase/YGbbMXkeHlI9zo
> Jacob described the scenario where data from certain region were missing in
> the snapshot.
> Here was related region server log:
> https://pastebin.com/1ECXjhRp
> He pointed out that concurrent flush from MemStoreFlusher.1 thread was not
> initiated from the thread pool for snapshot.
> In RegionSnapshotTask#call() method there is this:
> {code}
> region.flush(true);
> {code}
> The return value is not checked.
> In HRegion#flushcache(), Result.CANNOT_FLUSH may be returned due to:
> {code}
> String msg = "Not flushing since "
> + (writestate.flushing ? "already flushing"
> : "writes not enabled");
> {code}
> This implies that FlushSnapshotSubprocedure may incorrectly skip waiting for
> the concurrent flush to complete.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-18114) Update the config of TestAsync*AdminApi to make test stable
[
https://issues.apache.org/jira/browse/HBASE-18114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-18114:
---
Attachment: HBASE-18114-v1.patch
> Update the config of TestAsync*AdminApi to make test stable
> ---
>
> Key: HBASE-18114
> URL: https://issues.apache.org/jira/browse/HBASE-18114
> Project: HBase
> Issue Type: Sub-task
> Components: Client
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-18114-v1.patch, HBASE-18114-v1.patch,
> HBASE-18114-v1.patch
>
>
> {code}
> 2017-05-25 17:56:34,967 INFO
> [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=50801]
> master.HMaster$11(2297): Client=hao//127.0.0.1 disable testModifyColumnFamily
> 2017-05-25 17:56:37,974 INFO [RpcClient-timer-pool1-t1]
> client.AsyncHBaseAdmin$TableProcedureBiConsumer(2219): Operation: DISABLE,
> Table Name: default:testModifyColumnFamily failed with Failed after
> attempts=3, exceptions:
> Thu May 25 17:56:35 CST 2017, , java.io.IOException: Call to
> localhost/127.0.0.1:50801 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=294, waitTime=1008,
> rpcTimeout=1000
> Thu May 25 17:56:37 CST 2017, , java.io.IOException: Call to
> localhost/127.0.0.1:50801 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=295, waitTime=1299,
> rpcTimeout=1000
> Thu May 25 17:56:37 CST 2017, , java.io.IOException: Call to
> localhost/127.0.0.1:50801 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=296, waitTime=668,
> rpcTimeout=660
> 017-05-25 17:56:38,936 DEBUG
> [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=50801]
> procedure2.ProcedureExecutor(788): Stored procId=15, owner=hao,
> state=RUNNABLE:DISABLE_TABLE_PREPARE, DisableTableProcedure
> table=testModifyColumnFamily
> {code}
> For this disable table procedure, master return the procedure id when it
> submit the procedure to ProcedureExecutor. And the above procedure take 4
> seconds to submit. So the disable table call failed because the rpc timeout
> is 1 seconds and the retry number is 3.
> For admin operation, I thought we don't need change the default timeout
> config in unit test. And the retry is not need, too. (Or we can set a retry >
> 1 to test nonce thing). Meanwhile, the default timeout is 60 seconds. So the
> test type may need change to LargeTests.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-15930) Make IntegrationTestReplication's waitForReplication() smarter
[
https://issues.apache.org/jira/browse/HBASE-15930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mike Drob updated HBASE-15930:
--
Status: Open (was: Patch Available)
Ha, good catch, [~jmhsieh]! I started the test in the pre-setup case and
stopped it after I saw that it didn't error at the beginning. I probably would
have seen the failure at the end if I had waited that long. Will do an
additional round of testing and then post an updated patch.
> Make IntegrationTestReplication's waitForReplication() smarter
> --
>
> Key: HBASE-15930
> URL: https://issues.apache.org/jira/browse/HBASE-15930
> Project: HBase
> Issue Type: Improvement
> Components: integration tests
>Reporter: Dima Spivak
>Assignee: Mike Drob
> Fix For: 2.0.0
>
> Attachments: HBASE-15930.patch
>
>
> {{IntegrationTestReplication}} is a great test, but can improved by changing
> how we handle waiting between generation of the linked list on the source
> cluster and verifying the linked list on the destination cluster. [Even the
> code suggests this should be
> done|https://github.com/apache/hbase/blob/master/hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestReplication.java#L251-252],
> so I'd like to take it on. [~mbertozzi] and [~busbey] have both suggested a
> simple solution wherein we write a row into each region on the source cluster
> after the linked list generation and then assume replication has gone through
> once these rows are detected on the destination cluster.
> Since you lads at Facebook are some of the heaviest users, [~eclark], would
> you prefer I maintain the API and add a new command line option (say {{\-c |
> \-\-check-replication}}) that would run before any {{--generateVerifyGap}}
> sleep is carried out as it is now?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-18114) Update the config of TestAsync*AdminApi to make test stable
[
https://issues.apache.org/jira/browse/HBASE-18114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-18114:
---
Attachment: HBASE-18114-v1.patch
> Update the config of TestAsync*AdminApi to make test stable
> ---
>
> Key: HBASE-18114
> URL: https://issues.apache.org/jira/browse/HBASE-18114
> Project: HBase
> Issue Type: Sub-task
> Components: Client
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-18114-v1.patch, HBASE-18114-v1.patch
>
>
> {code}
> 2017-05-25 17:56:34,967 INFO
> [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=50801]
> master.HMaster$11(2297): Client=hao//127.0.0.1 disable testModifyColumnFamily
> 2017-05-25 17:56:37,974 INFO [RpcClient-timer-pool1-t1]
> client.AsyncHBaseAdmin$TableProcedureBiConsumer(2219): Operation: DISABLE,
> Table Name: default:testModifyColumnFamily failed with Failed after
> attempts=3, exceptions:
> Thu May 25 17:56:35 CST 2017, , java.io.IOException: Call to
> localhost/127.0.0.1:50801 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=294, waitTime=1008,
> rpcTimeout=1000
> Thu May 25 17:56:37 CST 2017, , java.io.IOException: Call to
> localhost/127.0.0.1:50801 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=295, waitTime=1299,
> rpcTimeout=1000
> Thu May 25 17:56:37 CST 2017, , java.io.IOException: Call to
> localhost/127.0.0.1:50801 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=296, waitTime=668,
> rpcTimeout=660
> 017-05-25 17:56:38,936 DEBUG
> [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=50801]
> procedure2.ProcedureExecutor(788): Stored procId=15, owner=hao,
> state=RUNNABLE:DISABLE_TABLE_PREPARE, DisableTableProcedure
> table=testModifyColumnFamily
> {code}
> For this disable table procedure, master return the procedure id when it
> submit the procedure to ProcedureExecutor. And the above procedure take 4
> seconds to submit. So the disable table call failed because the rpc timeout
> is 1 seconds and the retry number is 3.
> For admin operation, I thought we don't need change the default timeout
> config in unit test. And the retry is not need, too. (Or we can set a retry >
> 1 to test nonce thing). Meanwhile, the default timeout is 60 seconds. So the
> test type may need change to LargeTests.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-18066) Get with closest_row_before on "hbase:meta" can return empty Cell during region merge/split
[ https://issues.apache.org/jira/browse/HBASE-18066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Hu updated HBASE-18066: - Attachment: HBASE-18066.branch-1.3.v1.patch HBASE-18066.branch-1.1.v1.patch Trigger Hadoop QA again. > Get with closest_row_before on "hbase:meta" can return empty Cell during > region merge/split > --- > > Key: HBASE-18066 > URL: https://issues.apache.org/jira/browse/HBASE-18066 > Project: HBase > Issue Type: Bug > Components: hbase, regionserver >Affects Versions: 1.3.1 > Environment: Linux (16.04.2), MacOS 10.11.6. > Standalone and distributed HBase setup. >Reporter: Andrey Elenskiy >Assignee: Zheng Hu > Attachments: HBASE-18066.branch-1.1.v1.patch, > HBASE-18066.branch-1.1.v1.patch, HBASE-18066.branch-1.3.v1.patch, > HBASE-18066.branch-1.3.v1.patch, HBASE-18066.branch-1.v1.patch, > HBASE-18066.branch-1.v2.patch, HBASE-18066.branch-1.v3.patch, > TestGetWithClosestRowBeforeWhenSplit.java > > > During region split/merge there's a brief period of time where doing a "Get" > with "closest_row_before=true" on "hbase:meta" may return empty > "GetResponse.result.cell" field even though parent, splitA and splitB regions > are all in "hbase:meta". Both gohbase (https://github.com/tsuna/gohbase) and > AsyncHBase (https://github.com/OpenTSDB/asynchbase) interprets this as > "TableDoesNotExist", which is returned to the client. > Here's a gist that reproduces this problem: > https://gist.github.com/Timoha/c7a236b768be9220e85e53e1ca53bf96. Note that > you have to use older HTable client (I used 1.2.4) as current versions ignore > `Get.setClosestRowBefore(bool)` option. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17997) In dev environment, add jruby-complete jar to classpath only when jruby is needed
[
https://issues.apache.org/jira/browse/HBASE-17997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-17997:
---
Summary: In dev environment, add jruby-complete jar to classpath only when
jruby is needed (was: jruby-complete-1.6.8.jar is in cached_classpath.txt)
> In dev environment, add jruby-complete jar to classpath only when jruby is
> needed
> -
>
> Key: HBASE-17997
> URL: https://issues.apache.org/jira/browse/HBASE-17997
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Xiang Li
> Fix For: 2.0.0
>
> Attachments: HBASE-17997.master.000.patch,
> HBASE-17997.master.001.patch, HBASE-17997.master.002.patch,
> HBASE-17997.master.003.debug.diff, HBASE-17997.master.003.patch
>
>
> HBASE-15199 moves jruby-complete-1.6.8.jar to lib/ruby directory.
> However, jruby-complete-1.6.8.jar still appears in cached_classpath.txt
> This means that user would see exception similar to the following when
> starting hbase in standalone mode with s3a as rootdir :
> {code}
> 2017-05-04 16:41:32,854 WARN
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=38659]
> internal.S3MetadataResponseHandler: Unable to parse last modified date: Thu,
> 04 May 2017 16:27:09 GMT
> java.lang.IllegalStateException: Joda-time 2.2 or later version is required,
> but found version: null
> at com.amazonaws.util.DateUtils.handleException(DateUtils.java:149)
> at com.amazonaws.util.DateUtils.parseRFC822Date(DateUtils.java:195)
> at
> com.amazonaws.services.s3.internal.ServiceUtils.parseRfc822Date(ServiceUtils.java:78)
> at
> com.amazonaws.services.s3.internal.AbstractS3ResponseHandler.populateObjectMetadata(AbstractS3ResponseHandler.java:115)
> at
> com.amazonaws.services.s3.internal.S3ObjectResponseHandler.handle(S3ObjectResponseHandler.java:52)
> at
> com.amazonaws.services.s3.internal.S3ObjectResponseHandler.handle(S3ObjectResponseHandler.java:30)
> at
> com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:1072)
> at
> com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:746)
> at
> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
> at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
> at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785)
> at
> com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1191)
> at org.apache.hadoop.fs.s3a.S3AInputStream.reopen(S3AInputStream.java:148)
> at org.apache.hadoop.fs.s3a.S3AInputStream.lazySeek(S3AInputStream.java:281)
> at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:364)
> at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:75)
> at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:92)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.positionalReadWithExtra(HFileBlock.java:722)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1420)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1677)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1504)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:439)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:904)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:267)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:169)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:363)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:217)
> at
> org.apache.hadoop.hbase.regionserver.HStore.createScanner(HStore.java:2132)
> at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2122)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:5687)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2679)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2665)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2647)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6906)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6885)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2007)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17997) In dev environment, add jruby-complete jar to classpath only when jruby is needed
[
https://issues.apache.org/jira/browse/HBASE-17997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-17997:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
Thanks for the patch, Xiang.
> In dev environment, add jruby-complete jar to classpath only when jruby is
> needed
> -
>
> Key: HBASE-17997
> URL: https://issues.apache.org/jira/browse/HBASE-17997
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Xiang Li
> Fix For: 2.0.0
>
> Attachments: HBASE-17997.master.000.patch,
> HBASE-17997.master.001.patch, HBASE-17997.master.002.patch,
> HBASE-17997.master.003.debug.diff, HBASE-17997.master.003.patch
>
>
> HBASE-15199 moves jruby-complete-1.6.8.jar to lib/ruby directory.
> However, jruby-complete-1.6.8.jar still appears in cached_classpath.txt
> This means that user would see exception similar to the following when
> starting hbase in standalone mode with s3a as rootdir :
> {code}
> 2017-05-04 16:41:32,854 WARN
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=38659]
> internal.S3MetadataResponseHandler: Unable to parse last modified date: Thu,
> 04 May 2017 16:27:09 GMT
> java.lang.IllegalStateException: Joda-time 2.2 or later version is required,
> but found version: null
> at com.amazonaws.util.DateUtils.handleException(DateUtils.java:149)
> at com.amazonaws.util.DateUtils.parseRFC822Date(DateUtils.java:195)
> at
> com.amazonaws.services.s3.internal.ServiceUtils.parseRfc822Date(ServiceUtils.java:78)
> at
> com.amazonaws.services.s3.internal.AbstractS3ResponseHandler.populateObjectMetadata(AbstractS3ResponseHandler.java:115)
> at
> com.amazonaws.services.s3.internal.S3ObjectResponseHandler.handle(S3ObjectResponseHandler.java:52)
> at
> com.amazonaws.services.s3.internal.S3ObjectResponseHandler.handle(S3ObjectResponseHandler.java:30)
> at
> com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:1072)
> at
> com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:746)
> at
> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
> at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
> at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785)
> at
> com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1191)
> at org.apache.hadoop.fs.s3a.S3AInputStream.reopen(S3AInputStream.java:148)
> at org.apache.hadoop.fs.s3a.S3AInputStream.lazySeek(S3AInputStream.java:281)
> at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:364)
> at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:75)
> at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:92)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.positionalReadWithExtra(HFileBlock.java:722)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1420)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1677)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1504)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:439)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:904)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:267)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:169)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:363)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:217)
> at
> org.apache.hadoop.hbase.regionserver.HStore.createScanner(HStore.java:2132)
> at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2122)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:5687)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2679)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2665)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2647)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6906)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6885)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2007)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17997) jruby-complete-1.6.8.jar is in cached_classpath.txt
[
https://issues.apache.org/jira/browse/HBASE-17997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-17997:
---
Release Note:
When JRUBY_HOME is specified, if the command is "hbase shell" or "hbase
org.jruby.Main", CLASSPATH and HBASE_OPTS will be updated according to
JRUBY_HOME specified
* Jar under JRUBY_HOME is added to CLASSPATH
* The following will be added into HBASE_OPTS
-Djruby.home=$JRUBY_HOME -Djruby.lib=$JRUBY_HOME/lib
That is, as long as JRUBY_HOME is specified, JRUBY_HOME specified will take
precedence.
* In dev env, the jar recorded in cached_classpath_jruby.txt will be ignored
* In non dev env, jruby-complete jar packaged with HBase will be ignored
was:
When JRUBY_HOME is specified, if the command is "hbase shell" or "hbase
org.jruby.Main", CLASSPATH and HBASE_OPTS will be updated according to
JRUBY_HOME specified
* Jar under JRUBY_HOME is added to CLASSPATH
* The following will be added into HBASE_OPTS
{code}
-Djruby.home=$JRUBY_HOME -Djruby.lib=$JRUBY_HOME/lib
{code}
That is, as long as JRUBY_HOME is specified, JRUBY_HOME specified will take
precedence.
* In dev env, the jar recorded in cached_classpath_jruby.txt will be ignored
* In non dev env, jruby-complete jar packaged with HBase will be ignored
> jruby-complete-1.6.8.jar is in cached_classpath.txt
> ---
>
> Key: HBASE-17997
> URL: https://issues.apache.org/jira/browse/HBASE-17997
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Xiang Li
> Fix For: 2.0.0
>
> Attachments: HBASE-17997.master.000.patch,
> HBASE-17997.master.001.patch, HBASE-17997.master.002.patch,
> HBASE-17997.master.003.debug.diff, HBASE-17997.master.003.patch
>
>
> HBASE-15199 moves jruby-complete-1.6.8.jar to lib/ruby directory.
> However, jruby-complete-1.6.8.jar still appears in cached_classpath.txt
> This means that user would see exception similar to the following when
> starting hbase in standalone mode with s3a as rootdir :
> {code}
> 2017-05-04 16:41:32,854 WARN
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=38659]
> internal.S3MetadataResponseHandler: Unable to parse last modified date: Thu,
> 04 May 2017 16:27:09 GMT
> java.lang.IllegalStateException: Joda-time 2.2 or later version is required,
> but found version: null
> at com.amazonaws.util.DateUtils.handleException(DateUtils.java:149)
> at com.amazonaws.util.DateUtils.parseRFC822Date(DateUtils.java:195)
> at
> com.amazonaws.services.s3.internal.ServiceUtils.parseRfc822Date(ServiceUtils.java:78)
> at
> com.amazonaws.services.s3.internal.AbstractS3ResponseHandler.populateObjectMetadata(AbstractS3ResponseHandler.java:115)
> at
> com.amazonaws.services.s3.internal.S3ObjectResponseHandler.handle(S3ObjectResponseHandler.java:52)
> at
> com.amazonaws.services.s3.internal.S3ObjectResponseHandler.handle(S3ObjectResponseHandler.java:30)
> at
> com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:1072)
> at
> com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:746)
> at
> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
> at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
> at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785)
> at
> com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1191)
> at org.apache.hadoop.fs.s3a.S3AInputStream.reopen(S3AInputStream.java:148)
> at org.apache.hadoop.fs.s3a.S3AInputStream.lazySeek(S3AInputStream.java:281)
> at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:364)
> at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:75)
> at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:92)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.positionalReadWithExtra(HFileBlock.java:722)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1420)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1677)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1504)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:439)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:904)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:267)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:169)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:363)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:217)
> at
>
[jira] [Updated] (HBASE-17997) jruby-complete-1.6.8.jar is in cached_classpath.txt
[
https://issues.apache.org/jira/browse/HBASE-17997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-17997:
---
Hadoop Flags: Reviewed
Fix Version/s: 2.0.0
Release Note:
When JRUBY_HOME is specified, if the command is "hbase shell" or "hbase
org.jruby.Main", CLASSPATH and HBASE_OPTS will be updated according to
JRUBY_HOME specified
* Jar under JRUBY_HOME is added to CLASSPATH
* The following will be added into HBASE_OPTS
{code}
-Djruby.home=$JRUBY_HOME -Djruby.lib=$JRUBY_HOME/lib
{code}
That is, as long as JRUBY_HOME is specified, JRUBY_HOME specified will take
precedence.
* In dev env, the jar recorded in cached_classpath_jruby.txt will be ignored
* In non dev env, jruby-complete jar packaged with HBase will be ignored
> jruby-complete-1.6.8.jar is in cached_classpath.txt
> ---
>
> Key: HBASE-17997
> URL: https://issues.apache.org/jira/browse/HBASE-17997
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Xiang Li
> Fix For: 2.0.0
>
> Attachments: HBASE-17997.master.000.patch,
> HBASE-17997.master.001.patch, HBASE-17997.master.002.patch,
> HBASE-17997.master.003.debug.diff, HBASE-17997.master.003.patch
>
>
> HBASE-15199 moves jruby-complete-1.6.8.jar to lib/ruby directory.
> However, jruby-complete-1.6.8.jar still appears in cached_classpath.txt
> This means that user would see exception similar to the following when
> starting hbase in standalone mode with s3a as rootdir :
> {code}
> 2017-05-04 16:41:32,854 WARN
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=38659]
> internal.S3MetadataResponseHandler: Unable to parse last modified date: Thu,
> 04 May 2017 16:27:09 GMT
> java.lang.IllegalStateException: Joda-time 2.2 or later version is required,
> but found version: null
> at com.amazonaws.util.DateUtils.handleException(DateUtils.java:149)
> at com.amazonaws.util.DateUtils.parseRFC822Date(DateUtils.java:195)
> at
> com.amazonaws.services.s3.internal.ServiceUtils.parseRfc822Date(ServiceUtils.java:78)
> at
> com.amazonaws.services.s3.internal.AbstractS3ResponseHandler.populateObjectMetadata(AbstractS3ResponseHandler.java:115)
> at
> com.amazonaws.services.s3.internal.S3ObjectResponseHandler.handle(S3ObjectResponseHandler.java:52)
> at
> com.amazonaws.services.s3.internal.S3ObjectResponseHandler.handle(S3ObjectResponseHandler.java:30)
> at
> com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:1072)
> at
> com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:746)
> at
> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
> at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
> at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785)
> at
> com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1191)
> at org.apache.hadoop.fs.s3a.S3AInputStream.reopen(S3AInputStream.java:148)
> at org.apache.hadoop.fs.s3a.S3AInputStream.lazySeek(S3AInputStream.java:281)
> at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:364)
> at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:75)
> at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:92)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.positionalReadWithExtra(HFileBlock.java:722)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1420)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1677)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1504)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:439)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:904)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:267)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:169)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:363)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:217)
> at
> org.apache.hadoop.hbase.regionserver.HStore.createScanner(HStore.java:2132)
> at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2122)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:5687)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2679)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2665)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2647)
> at
[jira] [Commented] (HBASE-16011) TableSnapshotScanner and TableSnapshotInputFormat can produce duplicate rows
[
https://issues.apache.org/jira/browse/HBASE-16011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024838#comment-16024838
]
Ted Yu commented on HBASE-16011:
>From https://builds.apache.org/job/PreCommit-HBASE-Build/6947/console :
{code}
HEAD is now at a3c5a74 Revert "HBASE-14614 Procedure v2 - Core Assignment
Manager (Matteo Bertozzi)"
error: pathspec 'master' did not match any file(s) known to git.
ERROR: git checkout --force master is failing
{code}
I re-triggered QA run.
> TableSnapshotScanner and TableSnapshotInputFormat can produce duplicate rows
>
>
> Key: HBASE-16011
> URL: https://issues.apache.org/jira/browse/HBASE-16011
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 2.0.0, 1.2.2
>Reporter: Youngjoon Kim
>Assignee: Zheng Hu
> Attachments: HBASE-16011.v1.patch, HBASE-16011.v2.patch,
> HBASE-16011.v2.patch, snapshot_bug_test.patch
>
>
> A snapshot of (non-pre) split table can include both a parent region and
> daughter regions. If run TableSnapshotScanner or TableSnapshotInputFormat on
> the such snapshot, duplicate rows are produced.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-18099) FlushSnapshotSubprocedure should wait for concurrent Region#flush() to finish
[
https://issues.apache.org/jira/browse/HBASE-18099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-18099:
---
Resolution: Fixed
Assignee: Ted Yu
Hadoop Flags: Reviewed
Fix Version/s: 1.4.0
2.0.0
Status: Resolved (was: Patch Available)
Thanks for the review, Jerry
> FlushSnapshotSubprocedure should wait for concurrent Region#flush() to finish
> -
>
> Key: HBASE-18099
> URL: https://issues.apache.org/jira/browse/HBASE-18099
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.4.0
>
> Attachments: 18099.v1.txt, 18099.v2.txt, 18099.v3.txt, 18099.v4.txt
>
>
> In the following thread:
> http://search-hadoop.com/m/HBase/YGbbMXkeHlI9zo
> Jacob described the scenario where data from certain region were missing in
> the snapshot.
> Here was related region server log:
> https://pastebin.com/1ECXjhRp
> He pointed out that concurrent flush from MemStoreFlusher.1 thread was not
> initiated from the thread pool for snapshot.
> In RegionSnapshotTask#call() method there is this:
> {code}
> region.flush(true);
> {code}
> The return value is not checked.
> In HRegion#flushcache(), Result.CANNOT_FLUSH may be returned due to:
> {code}
> String msg = "Not flushing since "
> + (writestate.flushing ? "already flushing"
> : "writes not enabled");
> {code}
> This implies that FlushSnapshotSubprocedure may incorrectly skip waiting for
> the concurrent flush to complete.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18114) Update the config of TestAsync*AdminApi to make test stable
[
https://issues.apache.org/jira/browse/HBASE-18114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024828#comment-16024828
]
Hadoop QA commented on HBASE-18114:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 1s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
12s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 6s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
30m 2s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 116m 7s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 173m 56s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:757bf37 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12869849/HBASE-18114-v1.patch |
| JIRA Issue | HBASE-18114 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux e8c68d1cafa4 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 3e426b2 |
| Default Java | 1.8.0_131 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6948/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6948/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Update the config of TestAsync*AdminApi to make test stable
> ---
>
> Key: HBASE-18114
> URL: https://issues.apache.org/jira/browse/HBASE-18114
> Project: HBase
> Issue Type: Sub-task
> Components: Client
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-18114-v1.patch
>
>
> {code}
>
[jira] [Commented] (HBASE-18042) Client Compatibility breaks between versions 1.2 and 1.3
[
https://issues.apache.org/jira/browse/HBASE-18042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024809#comment-16024809
]
Duo Zhang commented on HBASE-18042:
---
[~busbey] Seems yetus always failed to build the docker image for branch-1?
> Client Compatibility breaks between versions 1.2 and 1.3
>
>
> Key: HBASE-18042
> URL: https://issues.apache.org/jira/browse/HBASE-18042
> Project: HBase
> Issue Type: Bug
> Components: regionserver, scan
>Affects Versions: 2.0.0, 1.4.0, 1.3.1
>Reporter: Karan Mehta
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0, 1.4.0, 1.3.2
>
> Attachments: HBASE-18042-branch-1.3.patch,
> HBASE-18042-branch-1.patch, HBASE-18042-branch-1.patch,
> HBASE-18042-branch-1-v1.patch, HBASE-18042-branch-1-v1.patch,
> HBASE-18042.patch, HBASE-18042-v1.patch, HBASE-18042-v2.patch
>
>
> OpenTSDB uses AsyncHBase as its client, rather than using the traditional
> HBase Client. From version 1.2 to 1.3, the {{ClientProtos}} have been
> changed. Newer fields are added to {{ScanResponse}} proto.
> For a typical Scan request in 1.2, would require caller to make an
> OpenScanner Request, GetNextRows Request and a CloseScanner Request, based on
> {{more_rows}} boolean field in the {{ScanResponse}} proto.
> However, from 1.3, new parameter {{more_results_in_region}} was added, which
> limits the results per region. Therefore the client has to now manage sending
> all the requests for each region. Further more, if the results are exhausted
> from a particular region, the {{ScanResponse}} will set
> {{more_results_in_region}} to false, but {{more_results}} can still be true.
> Whenever the former is set to false, the {{RegionScanner}} will also be
> closed.
> OpenTSDB makes an OpenScanner Request and receives all its results in the
> first {{ScanResponse}} itself, thus creating a condition as described in
> above paragraph. Since {{more_rows}} is true, it will proceed to send next
> request at which point the {{RSRpcServices}} will throw
> {{UnknownScannerException}}. The protobuf client compatibility is maintained
> but expected behavior is modified.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-14614) Procedure v2: Core Assignment Manager
[ https://issues.apache.org/jira/browse/HBASE-14614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14614: -- Attachment: HBASE-14614.master.044.patch > Procedure v2: Core Assignment Manager > - > > Key: HBASE-14614 > URL: https://issues.apache.org/jira/browse/HBASE-14614 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Stephen Yuan Jiang >Assignee: Matteo Bertozzi > Fix For: 2.0.0 > > Attachments: 14614.040.patch, HBASE-14614.master.003.patch, > HBASE-14614.master.004.patch, HBASE-14614.master.005.patch, > HBASE-14614.master.006.patch, HBASE-14614.master.007.patch, > HBASE-14614.master.008.patch, HBASE-14614.master.009.patch, > HBASE-14614.master.010.patch, HBASE-14614.master.012.patch, > HBASE-14614.master.012.patch, HBASE-14614.master.013.patch, > HBASE-14614.master.014.patch, HBASE-14614.master.015.patch, > HBASE-14614.master.017.patch, HBASE-14614.master.018.patch, > HBASE-14614.master.019.patch, HBASE-14614.master.020.patch, > HBASE-14614.master.022.patch, HBASE-14614.master.023.patch, > HBASE-14614.master.024.patch, HBASE-14614.master.025.patch, > HBASE-14614.master.026.patch, HBASE-14614.master.027.patch, > HBASE-14614.master.028.patch, HBASE-14614.master.029.patch, > HBASE-14614.master.030.patch, HBASE-14614.master.033.patch, > HBASE-14614.master.038.patch, HBASE-14614.master.039.patch, > HBASE-14614.master.040.patch, HBASE-14614.master.041.patch, > HBASE-14614.master.042.patch, HBASE-14614.master.043.patch, > HBASE-14614.master.044.patch > > > New AssignmentManager implemented using proc-v2. > - AssignProcedure handle assignment operation > - UnassignProcedure handle unassign operation > - MoveRegionProcedure handle move/balance operation > Concurrent Assign operations are batched together and sent to the balancer > Concurrent Assign and Unassign operation ready to be sent to the RS are > batched together > This patch is an intermediate state where we add the new AM as > AssignmentManager2() to the master, to be reached by tests. but the new AM > will not be integrated with the rest of the system. Only new am unit-tests > will exercise the new assigment manager. The integration with the master code > is part of HBASE-14616 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
