[jira] [Commented] (HBASE-20893) Data loss if splitting region while ServerCrashProcedure executing
[ https://issues.apache.org/jira/browse/HBASE-20893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550260#comment-16550260 ] stack commented on HBASE-20893: --- +1 Thanks. Test repros the condition? > Data loss if splitting region while ServerCrashProcedure executing > -- > > Key: HBASE-20893 > URL: https://issues.apache.org/jira/browse/HBASE-20893 > Project: HBase > Issue Type: Sub-task >Affects Versions: 3.0.0, 2.1.0, 2.0.1 >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > Attachments: HBASE-20893.branch-2.0.001.patch, > HBASE-20893.branch-2.0.002.patch, HBASE-20893.branch-2.0.003.patch > > > Similar case as HBASE-20878. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-17885) Backport HBASE-15871 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-17885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Toshihiro Suzuki updated HBASE-17885: - Attachment: HBASE-17885.branch-1.001.patch > Backport HBASE-15871 to branch-1 > > > Key: HBASE-17885 > URL: https://issues.apache.org/jira/browse/HBASE-17885 > Project: HBase > Issue Type: Bug > Components: Scanners >Affects Versions: 1.3.1, 1.2.5, 1.1.8 >Reporter: ramkrishna.s.vasudevan >Priority: Major > Fix For: 1.5.0 > > Attachments: HBASE-17885.branch-1.001.patch > > > Will try to rebase the branch-1 patch at the earliest. Hope the fix versions > are correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-17885) Backport HBASE-15871 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-17885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Toshihiro Suzuki updated HBASE-17885: - Assignee: Toshihiro Suzuki Status: Patch Available (was: Open) > Backport HBASE-15871 to branch-1 > > > Key: HBASE-17885 > URL: https://issues.apache.org/jira/browse/HBASE-17885 > Project: HBase > Issue Type: Bug > Components: Scanners >Affects Versions: 1.1.8, 1.2.5, 1.3.1 >Reporter: ramkrishna.s.vasudevan >Assignee: Toshihiro Suzuki >Priority: Major > Fix For: 1.5.0 > > Attachments: HBASE-17885.branch-1.001.patch > > > Will try to rebase the branch-1 patch at the earliest. Hope the fix versions > are correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20901) Reducing region replica has no effect
[
https://issues.apache.org/jira/browse/HBASE-20901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550253#comment-16550253
]
Hudson commented on HBASE-20901:
Results for branch branch-2
[build #1003 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1003/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1003//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1003//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1003//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Reducing region replica has no effect
> -
>
> Key: HBASE-20901
> URL: https://issues.apache.org/jira/browse/HBASE-20901
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Labels: replica
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-20901.patch, HBASE-20901_v1.patch,
> HBASE-20901_v2.patch
>
>
> While reducing the region replica, server name(sn) and state column of the
> replica are not getting deleted, resulting in assignment manager to think
> that these regions are CLOSED and assign them again.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20672) New metrics ReadRequestRate and WriteRequestRate
[
https://issues.apache.org/jira/browse/HBASE-20672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550254#comment-16550254
]
Hudson commented on HBASE-20672:
Results for branch branch-2
[build #1003 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1003/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1003//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1003//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1003//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> New metrics ReadRequestRate and WriteRequestRate
>
>
> Key: HBASE-20672
> URL: https://issues.apache.org/jira/browse/HBASE-20672
> Project: HBase
> Issue Type: Improvement
> Components: metrics
>Reporter: Ankit Jain
>Assignee: Ankit Jain
>Priority: Minor
> Fix For: 3.0.0, 1.5.0, 2.2.0
>
> Attachments: HBASE-20672.branch-1.001.patch,
> HBASE-20672.branch-1.002.patch, HBASE-20672.branch-2.001.patch,
> HBASE-20672.master.001.patch, HBASE-20672.master.002.patch,
> HBASE-20672.master.003.patch, hits1vs2.4.40.400.png
>
>
> Hbase currently provides counter read/write requests (ReadRequestCount,
> WriteRequestCount). That said it is not easy to use counter that reset only
> after a restart of the service, we would like to expose 2 new metrics in
> HBase to provide ReadRequestRate and WriteRequestRate at region server level.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20893) Data loss if splitting region while ServerCrashProcedure executing
[ https://issues.apache.org/jira/browse/HBASE-20893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allan Yang updated HBASE-20893: --- Attachment: HBASE-20893.branch-2.0.003.patch > Data loss if splitting region while ServerCrashProcedure executing > -- > > Key: HBASE-20893 > URL: https://issues.apache.org/jira/browse/HBASE-20893 > Project: HBase > Issue Type: Sub-task >Affects Versions: 3.0.0, 2.1.0, 2.0.1 >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > Attachments: HBASE-20893.branch-2.0.001.patch, > HBASE-20893.branch-2.0.002.patch, HBASE-20893.branch-2.0.003.patch > > > Similar case as HBASE-20878. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[
https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550244#comment-16550244
]
Ted Yu commented on HBASE-20565:
I ran the new test with ColumnPaginationFilter placed first in branch-1.3 where
there is no HBASE-18410 :
{code}
TestColumnRangeFilterWithColumnPaginationFilter(org.apache.hadoop.hbase.filter.TestColumnRangeFilter)
Time elapsed: 1.679 sec <<< FAILURE!
java.lang.AssertionError: expected:<5> but was:<4>
at
org.apache.hadoop.hbase.filter.TestColumnRangeFilter.TestColumnRangeFilterWithColumnPaginationFilter(TestColumnRangeFilter.java:278)
{code}
So the patch here is good by me.
> ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect
> result since 1.4
> -
>
> Key: HBASE-20565
> URL: https://issues.apache.org/jira/browse/HBASE-20565
> Project: HBase
> Issue Type: Bug
> Components: Filters
>Affects Versions: 2.1.0, 1.4.4, 2.0.1
>Reporter: Jerry He
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 2.1.0, 1.5.0, 1.4.6, 2.0.2
>
> Attachments: HBASE-20565.v1.patch, HBASE-20565.v2.patch, debug.diff,
> debug.log, test-branch-1.4.patch
>
>
> When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see
> incorrect result.
> Here is a simple example.
> One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for
> range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset
> 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter,
> ColumnPaginationFilter).
> We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns
> are returned.
> In 1.2.x, the correct 5 columns are returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20401) Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable
[
https://issues.apache.org/jira/browse/HBASE-20401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550223#comment-16550223
]
Tak Lon (Stephen) Wu commented on HBASE-20401:
--
[~reidchan] thanks for catching the {{Dynamic Configuration}} in the book
chapter and it's been updated. RN has been updated based on your suggestion,
and sorry that It's my first time writing the RN that may be too nerdy (will
try to improve it over time :))
> Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable
> --
>
> Key: HBASE-20401
> URL: https://issues.apache.org/jira/browse/HBASE-20401
> Project: HBase
> Issue Type: Improvement
> Components: master
>Affects Versions: 3.0.0, 1.5.0, 2.0.0-beta-1, 1.4.4, 2.0.0
>Reporter: Tak Lon (Stephen) Wu
>Assignee: Tak Lon (Stephen) Wu
>Priority: Minor
> Labels: beginner
> Attachments: HBASE-20401.branch-1.001.patch,
> HBASE-20401.master.001.patch, HBASE-20401.master.002.patch,
> HBASE-20401.master.003.patch, HBASE-20401.master.004.patch,
> HBASE-20401.master.005.patch, HBASE-20401.master.006.patch
>
>
> When backporting HBASE-18309 in HBASE-20352, the deleteFiles calls
> CleanerContext.java#getResult with a waitIfNotFinished timeout to wait for
> notification (notify) from the fs.delete file thread. there might be two
> situation need to tune the MAX_WAIT in CleanerContext or waitIfNotFinished
> when LogClearner call getResult.
> # fs.delete never complete (strange but possible), then we need to wait for
> a max of 60 seconds. here, 60 seconds might be too long
> # getResult is waiting in the period of 500 milliseconds, but the fs.delete
> has completed and setFromClear is set but yet notify(). one might want to
> tune this 500 milliseconds to 200 or less .
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20870) Wrong HBase root dir in ITBLL's Search Tool
[
https://issues.apache.org/jira/browse/HBASE-20870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allan Yang updated HBASE-20870:
---
Resolution: Fixed
Fix Version/s: 2.1.1
2.2.0
2.0.2
3.0.0
Status: Resolved (was: Patch Available)
> Wrong HBase root dir in ITBLL's Search Tool
> ---
>
> Key: HBASE-20870
> URL: https://issues.apache.org/jira/browse/HBASE-20870
> Project: HBase
> Issue Type: Bug
> Components: integration tests
>Affects Versions: 3.0.0, 2.1.0, 2.0.1
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Minor
> Fix For: 3.0.0, 2.0.2, 2.2.0, 2.1.1
>
> Attachments: HBASE-20870.branch-2.0.001.patch,
> HBASE-20870.branch-2.0.002.patch, HBASE-20870.branch-2.0.003.patch
>
>
> When using IntegrationTestBigLinkedList's Search tools, it always fails since
> it tries to read WALs in a wrong HBase root dir. Turned out that when
> initializing IntegrationTestingUtility in IntegrationTestBigLinkedList, its
> super class HBaseTestingUtility will change hbase.rootdir to a local random
> dir. It is not wrong since HBaseTestingUtility is mostly used by Minicluster.
> But for IntegrationTest runs on distributed clusters, we should change it
> back.
> Here is the error info.
> {code:java}
> 2018-07-11 16:35:49,679 DEBUG [main] hbase.HBaseCommonTestingUtility: Setting
> hbase.rootdir to
> /home/hadoop/target/test-data/deb67611-2737-4696-abe9-32a7783df7bb
> 2018-07-11 16:35:50,736 ERROR [main] util.AbstractHBaseTool: Error running
> command-line tool java.io.FileNotFoundException: File
> file:/home/hadoop/target/test-data/deb67611-2737-4696-abe9-32a7783df7bb/WALs
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:431)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1517)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20401) Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable
[ https://issues.apache.org/jira/browse/HBASE-20401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak Lon (Stephen) Wu updated HBASE-20401: - Release Note: When oldwals (and hfile) cleaner cleans stale wals (and hfiles), it will periodically check and wait the clean results from filesystem, the total wait time will be no more than a max time. The periodically wait and check configurations are hbase.oldwals.cleaner.thread.check.interval.msec (default is 500 ms) and hbase.regionserver.hfilecleaner.thread.check.interval.msec (default is 1000 ms). Meanwhile, The max time configurations are hbase.oldwals.cleaner.thread.timeout.msec and hbase.regionserver.hfilecleaner.thread.timeout.msec, they are set to 60 seconds by default. All support dynamic configuration. e.g. in the oldwals cleaning scenario, one may consider tuning hbase.oldwals.cleaner.thread.timeout.msec and hbase.oldwals.cleaner.thread.check.interval.msec 1. While deleting a oldwal never complete (strange but possible), then delete file task needs to wait for a max of 60 seconds. Here, 60 seconds might be too long, or the opposite way is to increase more than 60 seconds in the use cases of slow file delete. 2. The check and wait of a file delete is set to default in the period of 500 milliseconds, one might want to tune this checking period to a short interval to check more frequently or to a longer interval to avoid checking too often to manage their delete file task checking period (the longer interval may be use to avoid checking too fast while using a high latency storage). was: Add thread-level timeout and check interval configurations in LogCleaner and HFileCleaner, such that one can tune the file cleaners to match their environment. 1. hbase.oldwals.cleaner.thread.timeout.msec 2. hbase.oldwals.cleaner.thread.check.interval.msec 3. hbase.regionserver.hfilecleaner.thread.timeout.msec 4. hbase.regionserver.hfilecleaner.thread.check.interval.msec For example in the LogCleaner scenario, one might consider tuning hbase.oldwals.cleaner.thread.timeout.msec and hbase.oldwals.cleaner.thread.check.interval.msec 1. fs.delete never complete (strange but possible), then we need to wait for a max of 60 seconds. here, 60 seconds might be too long. or the opposite way is to increase more than 60 seconds in the use cases of slow oldwals delete. 2. getResult is waiting in the period of 500 milliseconds, but the fs.delete has completed and setFromCleaner is set but yet notify(). one might want to tune this 500 milliseconds to 200 or less . > Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable > -- > > Key: HBASE-20401 > URL: https://issues.apache.org/jira/browse/HBASE-20401 > Project: HBase > Issue Type: Improvement > Components: master >Affects Versions: 3.0.0, 1.5.0, 2.0.0-beta-1, 1.4.4, 2.0.0 >Reporter: Tak Lon (Stephen) Wu >Assignee: Tak Lon (Stephen) Wu >Priority: Minor > Labels: beginner > Attachments: HBASE-20401.branch-1.001.patch, > HBASE-20401.master.001.patch, HBASE-20401.master.002.patch, > HBASE-20401.master.003.patch, HBASE-20401.master.004.patch, > HBASE-20401.master.005.patch, HBASE-20401.master.006.patch > > > When backporting HBASE-18309 in HBASE-20352, the deleteFiles calls > CleanerContext.java#getResult with a waitIfNotFinished timeout to wait for > notification (notify) from the fs.delete file thread. there might be two > situation need to tune the MAX_WAIT in CleanerContext or waitIfNotFinished > when LogClearner call getResult. > # fs.delete never complete (strange but possible), then we need to wait for > a max of 60 seconds. here, 60 seconds might be too long > # getResult is waiting in the period of 500 milliseconds, but the fs.delete > has completed and setFromClear is set but yet notify(). one might want to > tune this 500 milliseconds to 200 or less . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550193#comment-16550193 ] Zheng Hu commented on HBASE-20565: -- bq. I am yet to see the relationship between SQL and FilterList. SQL is just an example to expression the semantics of ColumnPaginationFilter. I mean ColumnPaginationFilter is order dependence filter. so it's not worth us to eliminate order dependence in the upper layer (FilterList). [~anoop.hbase], [~jinghe], [~apurtell], If you have some time, please take a look. > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 2.1.0, 1.4.4, 2.0.1 >Reporter: Jerry He >Assignee: Zheng Hu >Priority: Major > Fix For: 2.1.0, 1.5.0, 1.4.6, 2.0.2 > > Attachments: HBASE-20565.v1.patch, HBASE-20565.v2.patch, debug.diff, > debug.log, test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20401) Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable
[ https://issues.apache.org/jira/browse/HBASE-20401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak Lon (Stephen) Wu updated HBASE-20401: - Attachment: HBASE-20401.master.006.patch > Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable > -- > > Key: HBASE-20401 > URL: https://issues.apache.org/jira/browse/HBASE-20401 > Project: HBase > Issue Type: Improvement > Components: master >Affects Versions: 3.0.0, 1.5.0, 2.0.0-beta-1, 1.4.4, 2.0.0 >Reporter: Tak Lon (Stephen) Wu >Assignee: Tak Lon (Stephen) Wu >Priority: Minor > Labels: beginner > Attachments: HBASE-20401.branch-1.001.patch, > HBASE-20401.master.001.patch, HBASE-20401.master.002.patch, > HBASE-20401.master.003.patch, HBASE-20401.master.004.patch, > HBASE-20401.master.005.patch, HBASE-20401.master.006.patch > > > When backporting HBASE-18309 in HBASE-20352, the deleteFiles calls > CleanerContext.java#getResult with a waitIfNotFinished timeout to wait for > notification (notify) from the fs.delete file thread. there might be two > situation need to tune the MAX_WAIT in CleanerContext or waitIfNotFinished > when LogClearner call getResult. > # fs.delete never complete (strange but possible), then we need to wait for > a max of 60 seconds. here, 60 seconds might be too long > # getResult is waiting in the period of 500 milliseconds, but the fs.delete > has completed and setFromClear is set but yet notify(). one might want to > tune this 500 milliseconds to 200 or less . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550187#comment-16550187 ] Ted Yu commented on HBASE-20565: I am yet to see the relationship between SQL and FilterList. If you're confident in the current solution, I don't want to block it. Please get +1 from Anoop or Jerry. > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 2.1.0, 1.4.4, 2.0.1 >Reporter: Jerry He >Assignee: Zheng Hu >Priority: Major > Fix For: 2.1.0, 1.5.0, 1.4.6, 2.0.2 > > Attachments: HBASE-20565.v1.patch, HBASE-20565.v2.patch, debug.diff, > debug.log, test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20401) Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable
[
https://issues.apache.org/jira/browse/HBASE-20401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550155#comment-16550155
]
Reid Chan commented on HBASE-20401:
---
Can the RN be more user friendly?
Those who doesn't read source code may feel hard to understand what those
configurations are for and tune them for what..
Here is mine (as reference, it will be good if you have better),
{code}
When oldwals(hfile) cleaner cleans stale wals(hfiles), it will periodically
check and wait the clean results from filesystem, the total wait time will be
no more than a max time.
The periodically wait and check configurations are ...
The max time configurations are ...
All support dynamic configuration.
...(Then here it is your tuning advice, please get rid of any code reference)...
{code}
> Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable
> --
>
> Key: HBASE-20401
> URL: https://issues.apache.org/jira/browse/HBASE-20401
> Project: HBase
> Issue Type: Improvement
> Components: master
>Affects Versions: 3.0.0, 1.5.0, 2.0.0-beta-1, 1.4.4, 2.0.0
>Reporter: Tak Lon (Stephen) Wu
>Assignee: Tak Lon (Stephen) Wu
>Priority: Minor
> Labels: beginner
> Attachments: HBASE-20401.branch-1.001.patch,
> HBASE-20401.master.001.patch, HBASE-20401.master.002.patch,
> HBASE-20401.master.003.patch, HBASE-20401.master.004.patch,
> HBASE-20401.master.005.patch
>
>
> When backporting HBASE-18309 in HBASE-20352, the deleteFiles calls
> CleanerContext.java#getResult with a waitIfNotFinished timeout to wait for
> notification (notify) from the fs.delete file thread. there might be two
> situation need to tune the MAX_WAIT in CleanerContext or waitIfNotFinished
> when LogClearner call getResult.
> # fs.delete never complete (strange but possible), then we need to wait for
> a max of 60 seconds. here, 60 seconds might be too long
> # getResult is waiting in the period of 500 milliseconds, but the fs.delete
> has completed and setFromClear is set but yet notify(). one might want to
> tune this 500 milliseconds to 200 or less .
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Hu updated HBASE-20565: - Affects Version/s: (was: 2.0.0) 2.0.1 > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 2.1.0, 1.4.4, 2.0.1 >Reporter: Jerry He >Assignee: Zheng Hu >Priority: Major > Fix For: 2.1.0, 1.5.0, 1.4.6, 2.0.2 > > Attachments: HBASE-20565.v1.patch, HBASE-20565.v2.patch, debug.diff, > debug.log, test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20151) Bug with SingleColumnValueFilter and FamilyFilter
[
https://issues.apache.org/jira/browse/HBASE-20151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zheng Hu updated HBASE-20151:
-
Affects Version/s: 2.1.0
2.0.1
1.4.5
Fix Version/s: (was: 2.2.0)
2.1.1
2.0.2
1.4.6
> Bug with SingleColumnValueFilter and FamilyFilter
> -
>
> Key: HBASE-20151
> URL: https://issues.apache.org/jira/browse/HBASE-20151
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.1.0, 2.0.1, 1.4.5
> Environment: MacOS 10.13.3
> HBase 1.3.1
>Reporter: Steven Sadowski
>Assignee: Reid Chan
>Priority: Major
> Fix For: 3.0.0, 1.4.6, 2.0.2, 2.1.1
>
> Attachments: HBASE-20151.master.001.patch,
> HBASE-20151.master.002.patch, HBASE-20151.master.003.patch,
> HBASE-20151.master.004.patch, HBASE-20151.master.004.patch,
> HBASE-20151.master.005.patch
>
>
> When running the following queries, the result is sometimes return correctly
> and other times incorrectly based on the qualifier queried.
> Setup:
> {code:java}
> create 'test', 'a', 'b'
> test = get_table 'test'
> test.put '1', 'a:1', nil
> test.put '1', 'a:10', nil
> test.put '1', 'b:2', nil
> {code}
>
> This query works fine when the SCVF's qualifier has length 1 (i.e. '1') :
> {code:java}
> test.scan({ FILTER => "(
> SingleColumnValueFilter('a','1',=,'binary:',true,true) AND
> FamilyFilter(=,'binary:b') )"})
> ROW COLUMN+CELL
> 1column=b:2,
> timestamp=1520455888059, value=
> 1 row(s) in 0.0060 seconds
> {code}
>
> The query should return the same result when passed a qualifier of length 2
> (i.e. '10') :
> {code:java}
> test.scan({ FILTER => "(
> SingleColumnValueFilter('a','10',=,'binary:',true,true) AND
> FamilyFilter(=,'binary:b') )"})
> ROW COLUMN+CELL
> 0 row(s) in 0.0110 seconds
> {code}
> However, in this case, it does not return any row (expected result would be
> to return the same result as the first query).
>
> Removing the family filter while the qualifier is '10' yields expected
> results:
> {code:java}
> test.scan({ FILTER => "(
> SingleColumnValueFilter('a','10',=,'binary:',true,true) )"})
> ROW COLUMN+CELL
> 1column=a:1,
> timestamp=1520455887954, value=
> 1column=a:10,
> timestamp=1520455888024, value=
> 1column=b:2,
> timestamp=1520455888059, value=
> 1 row(s) in 0.0140 seconds
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20908) Infinite loop on regionserver if region replica are reduced
[
https://issues.apache.org/jira/browse/HBASE-20908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550134#comment-16550134
]
Hadoop QA commented on HBASE-20908:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m
39s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
12s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
22s{color} | {color:red} hbase-server: The patch generated 1 new + 3 unchanged
- 0 fixed = 4 total (was 3) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m
49s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
19m 15s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}310m 55s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}382m 28s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestFromClientSide |
| | hadoop.hbase.master.procedure.TestProcedureAdmin |
| | hadoop.hbase.replication.TestSyncReplicationMoreLogsInLocalCopyToRemote |
| |
hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaLargeCluster
|
| | hadoop.hbase.filter.TestFuzzyRowFilterEndToEnd |
| |
hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaSameHosts |
| | hadoop.hbase.quotas.TestSpaceQuotas |
| | hadoop.hbase.regionserver.TestCompactingToCellFlatMapMemStore |
| | hadoop.hbase.client.TestBlockEvictionFromClient |
| | hadoop.hbase.replication.TestMasterReplication |
| |
hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaHighReplication
|
| | hadoop.hbase.replication.regionserver.TestWALEntryStream |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20908 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12932299/HBASE-20908_v1.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550132#comment-16550132 ] Zheng Hu commented on HBASE-20565: -- Only for those filters who maintain some row-global state such as offset/count, it will be order dependence...otherwise, it's still order independence. Just as I said before, user will write SQL like : select * from test where x = 1 and y < 10 limit 0, 100 (For HBase, limit 0, 100 is also a Filter, ColumnPaginationFilter ex), If select * from test where limit 0, 100 and x = 1 and y < 10 has the same meaning as the former one, don't you think it's more confusing? > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 2.1.0, 1.4.4, 2.0.0 >Reporter: Jerry He >Assignee: Zheng Hu >Priority: Major > Fix For: 2.1.0, 1.5.0, 1.4.6, 2.0.2 > > Attachments: HBASE-20565.v1.patch, HBASE-20565.v2.patch, debug.diff, > debug.log, test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20855) PeerConfigTracker only supporting one listener will cause problem when there is a recovered replication queue
[
https://issues.apache.org/jira/browse/HBASE-20855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550115#comment-16550115
]
Jingyun Tian edited comment on HBASE-20855 at 7/20/18 2:18 AM:
---
[~yuzhih...@gmail.com] Sorry, branch-1.3 doesn't have
ReplicationPeerConfigListener, thus it doesn't get affected. My bad..
was (Author: tianjingyun):
[~yuzhih...@gmail.com] Sure. I'll update a patch asap.
> PeerConfigTracker only supporting one listener will cause problem when there
> is a recovered replication queue
> -
>
> Key: HBASE-20855
> URL: https://issues.apache.org/jira/browse/HBASE-20855
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0, 1.5.0
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
>Priority: Major
> Fix For: 1.5.0, 1.4.6
>
> Attachments: HBASE-20855.branch-1.001.patch,
> HBASE-20855.branch-1.002.patch, HBASE-20855.branch-1.003.patch,
> HBASE-20855.branch-1.004.patch, HBASE-20855.branch-1.005.patch,
> HBASE-20855.branch-1.006.patch, HBASE-20855.branch-1.007.patch
>
>
> {code}
> public void init(Context context) throws IOException {
> this.ctx = context;
> if (this.ctx != null){
> ReplicationPeer peer = this.ctx.getReplicationPeer();
> if (peer != null){
> peer.trackPeerConfigChanges(this);
> } else {
> LOG.warn("Not tracking replication peer config changes for Peer Id " +
> this.ctx.getPeerId() +
> " because there's no such peer");
> }
> }
> }
> {code}
> As we know, replication source will set itself to the PeerConfigTracker in
> ReplicationPeer. When there is one or more recovered queue, each queue will
> generate a new replication source, But they share the same ReplicationPeer.
> Then when it calls setListener, the new generated one will cover the older
> one. Thus there will only has one ReplicationPeer that receive the peer
> config change notify.
> {code}
> public synchronized void setListener(ReplicationPeerConfigListener listener){
> this.listener = listener;
> }
> {code}
>
> To solve this, PeerConfigTracker need to support multiple listener and
> listener should be removed when the replication endpoint terminated.
> I will upload a patch later with fix and UT.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20855) PeerConfigTracker only supporting one listener will cause problem when there is a recovered replication queue
[
https://issues.apache.org/jira/browse/HBASE-20855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingyun Tian updated HBASE-20855:
-
Affects Version/s: (was: 1.3.0)
> PeerConfigTracker only supporting one listener will cause problem when there
> is a recovered replication queue
> -
>
> Key: HBASE-20855
> URL: https://issues.apache.org/jira/browse/HBASE-20855
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0, 1.5.0
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
>Priority: Major
> Fix For: 1.5.0, 1.4.6
>
> Attachments: HBASE-20855.branch-1.001.patch,
> HBASE-20855.branch-1.002.patch, HBASE-20855.branch-1.003.patch,
> HBASE-20855.branch-1.004.patch, HBASE-20855.branch-1.005.patch,
> HBASE-20855.branch-1.006.patch, HBASE-20855.branch-1.007.patch
>
>
> {code}
> public void init(Context context) throws IOException {
> this.ctx = context;
> if (this.ctx != null){
> ReplicationPeer peer = this.ctx.getReplicationPeer();
> if (peer != null){
> peer.trackPeerConfigChanges(this);
> } else {
> LOG.warn("Not tracking replication peer config changes for Peer Id " +
> this.ctx.getPeerId() +
> " because there's no such peer");
> }
> }
> }
> {code}
> As we know, replication source will set itself to the PeerConfigTracker in
> ReplicationPeer. When there is one or more recovered queue, each queue will
> generate a new replication source, But they share the same ReplicationPeer.
> Then when it calls setListener, the new generated one will cover the older
> one. Thus there will only has one ReplicationPeer that receive the peer
> config change notify.
> {code}
> public synchronized void setListener(ReplicationPeerConfigListener listener){
> this.listener = listener;
> }
> {code}
>
> To solve this, PeerConfigTracker need to support multiple listener and
> listener should be removed when the replication endpoint terminated.
> I will upload a patch later with fix and UT.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20401) Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable
[
https://issues.apache.org/jira/browse/HBASE-20401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550126#comment-16550126
]
Reid Chan edited comment on HBASE-20401 at 7/20/18 2:15 AM:
Modified and new added configurations support {{Dynamic Configuration}}, please
update the hbase book as well.
was (Author: reidchan):
Modified and new added configurations support {{Dynamic Configuration}, please
update the hbase book as well.
> Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable
> --
>
> Key: HBASE-20401
> URL: https://issues.apache.org/jira/browse/HBASE-20401
> Project: HBase
> Issue Type: Improvement
> Components: master
>Affects Versions: 3.0.0, 1.5.0, 2.0.0-beta-1, 1.4.4, 2.0.0
>Reporter: Tak Lon (Stephen) Wu
>Assignee: Tak Lon (Stephen) Wu
>Priority: Minor
> Labels: beginner
> Attachments: HBASE-20401.branch-1.001.patch,
> HBASE-20401.master.001.patch, HBASE-20401.master.002.patch,
> HBASE-20401.master.003.patch, HBASE-20401.master.004.patch,
> HBASE-20401.master.005.patch
>
>
> When backporting HBASE-18309 in HBASE-20352, the deleteFiles calls
> CleanerContext.java#getResult with a waitIfNotFinished timeout to wait for
> notification (notify) from the fs.delete file thread. there might be two
> situation need to tune the MAX_WAIT in CleanerContext or waitIfNotFinished
> when LogClearner call getResult.
> # fs.delete never complete (strange but possible), then we need to wait for
> a max of 60 seconds. here, 60 seconds might be too long
> # getResult is waiting in the period of 500 milliseconds, but the fs.delete
> has completed and setFromClear is set but yet notify(). one might want to
> tune this 500 milliseconds to 200 or less .
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20401) Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable
[
https://issues.apache.org/jira/browse/HBASE-20401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550126#comment-16550126
]
Reid Chan commented on HBASE-20401:
---
Modified and new added configurations support {{Dynamic Configuration}, please
update the hbase book as well.
> Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable
> --
>
> Key: HBASE-20401
> URL: https://issues.apache.org/jira/browse/HBASE-20401
> Project: HBase
> Issue Type: Improvement
> Components: master
>Affects Versions: 3.0.0, 1.5.0, 2.0.0-beta-1, 1.4.4, 2.0.0
>Reporter: Tak Lon (Stephen) Wu
>Assignee: Tak Lon (Stephen) Wu
>Priority: Minor
> Labels: beginner
> Attachments: HBASE-20401.branch-1.001.patch,
> HBASE-20401.master.001.patch, HBASE-20401.master.002.patch,
> HBASE-20401.master.003.patch, HBASE-20401.master.004.patch,
> HBASE-20401.master.005.patch
>
>
> When backporting HBASE-18309 in HBASE-20352, the deleteFiles calls
> CleanerContext.java#getResult with a waitIfNotFinished timeout to wait for
> notification (notify) from the fs.delete file thread. there might be two
> situation need to tune the MAX_WAIT in CleanerContext or waitIfNotFinished
> when LogClearner call getResult.
> # fs.delete never complete (strange but possible), then we need to wait for
> a max of 60 seconds. here, 60 seconds might be too long
> # getResult is waiting in the period of 500 milliseconds, but the fs.delete
> has completed and setFromClear is set but yet notify(). one might want to
> tune this 500 milliseconds to 200 or less .
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20855) PeerConfigTracker only supporting one listener will cause problem when there is a recovered replication queue
[
https://issues.apache.org/jira/browse/HBASE-20855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550115#comment-16550115
]
Jingyun Tian commented on HBASE-20855:
--
[~yuzhih...@gmail.com] Sure. I'll update a patch asap.
> PeerConfigTracker only supporting one listener will cause problem when there
> is a recovered replication queue
> -
>
> Key: HBASE-20855
> URL: https://issues.apache.org/jira/browse/HBASE-20855
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.0, 1.4.0, 1.5.0
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
>Priority: Major
> Fix For: 1.5.0, 1.4.6
>
> Attachments: HBASE-20855.branch-1.001.patch,
> HBASE-20855.branch-1.002.patch, HBASE-20855.branch-1.003.patch,
> HBASE-20855.branch-1.004.patch, HBASE-20855.branch-1.005.patch,
> HBASE-20855.branch-1.006.patch, HBASE-20855.branch-1.007.patch
>
>
> {code}
> public void init(Context context) throws IOException {
> this.ctx = context;
> if (this.ctx != null){
> ReplicationPeer peer = this.ctx.getReplicationPeer();
> if (peer != null){
> peer.trackPeerConfigChanges(this);
> } else {
> LOG.warn("Not tracking replication peer config changes for Peer Id " +
> this.ctx.getPeerId() +
> " because there's no such peer");
> }
> }
> }
> {code}
> As we know, replication source will set itself to the PeerConfigTracker in
> ReplicationPeer. When there is one or more recovered queue, each queue will
> generate a new replication source, But they share the same ReplicationPeer.
> Then when it calls setListener, the new generated one will cover the older
> one. Thus there will only has one ReplicationPeer that receive the peer
> config change notify.
> {code}
> public synchronized void setListener(ReplicationPeerConfigListener listener){
> this.listener = listener;
> }
> {code}
>
> To solve this, PeerConfigTracker need to support multiple listener and
> listener should be removed when the replication endpoint terminated.
> I will upload a patch later with fix and UT.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HBASE-20864) RS was killed due to master thought the region should be on a already dead server
[
https://issues.apache.org/jira/browse/HBASE-20864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allan Yang resolved HBASE-20864.
Resolution: Resolved
Fix Version/s: (was: 2.0.2)
HBASE-20792 solved this issue
> RS was killed due to master thought the region should be on a already dead
> server
> -
>
> Key: HBASE-20864
> URL: https://issues.apache.org/jira/browse/HBASE-20864
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Major
> Attachments: log.zip
>
>
> When I was running ITBLL with our internal 2.0.0 version(with 2.0.1
> backported and with other two issues: HBASE-20706, HBASE-20752). I found two
> of my RS killed by master since master has a different region state with
> those RS. It is very strange that master thought these region should be on a
> already dead server. There might be a serious bug, but I haven't found it
> yet. Here is the process:
> 1. e010125048153.bja,60020,1531137365840 is crashed, and clearly
> 4423e4182457c5b573729be4682cc3a3 was assigned to
> e010125049164.bja,60020,1531136465378 during ServerCrashProcedure
> {code:java}
> 2018-07-09 20:03:32,443 INFO [PEWorker-10] procedure.ServerCrashProcedure:
> Start pid=2303, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure
> server=e010125048153.bja,60020,1531137365840, splitWal=true, meta=false
> 2018-07-09 20:03:39,220 DEBUG
> [RpcServer.default.FPBQ.Fifo.handler=294,queue=24,port=6]
> assignment.RegionTransitionProcedure: Received report OPENED seqId=16021,
> pid=2305, ppid=2303, state=RUNNABLE:REGION_TRANSITION_DISPATCH;
> AssignProcedure table=IntegrationTestBigLinkedList,
> region=4423e4182457c5b573729be4682cc3a3; rit=OPENING,
> location=e010125049164.bja,60020,1531136465378
> 2018-07-09 20:03:39,220 INFO [PEWorker-13] assignment.RegionStateStore:
> pid=2305 updating hbase:meta row=4423e4182457c5b573729be4682cc3a3,
> regionState=OPEN, openSeqNum=16021,
> regionLocation=e010125049164.bja,60020,1531136465378
> 2018-07-09 20:03:43,190 INFO [PEWorker-12] procedure2.ProcedureExecutor:
> Finished pid=2303, state=SUCCESS; ServerCrashProcedure
> server=e010125048153.bja,60020,1531137365840, splitWal=true, meta=false in
> 10.7490sec
> {code}
> 2. A modify table happened later, the 4423e4182457c5b573729be4682cc3a3 was
> reopend on e010125049164.bja,60020,1531136465378
> {code:java}
> 2018-07-09 20:04:39,929 DEBUG
> [RpcServer.default.FPBQ.Fifo.handler=295,queue=25,port=6]
> assignment.RegionTransitionProcedure: Received report OPENED seqId=16024,
> pid=2351, ppid=2314, state=RUNNABLE:REGION_TRANSITION_DISPATCH;
> AssignProcedure table=IntegrationTestBigLinkedList,
> region=4423e4182457c5b573729be4682cc3a3,
> target=e010125049164.bja,60020,1531136465378; rit=OPENING,
> location=e010125049164.bja,60020,1531136465378
> 2018-07-09 20:04:40,554 INFO [PEWorker-6] assignment.RegionStateStore:
> pid=2351 updating hbase:meta row=4423e4182457c5b573729be4682cc3a3,
> regionState=OPEN, openSeqNum=16024,
> regionLocation=e010125049164.bja,60020,1531136465378
> {code}
> 3. Active master was killed, the backup master took over, but when loading
> meta entry, it clearly showed 4423e4182457c5b573729be4682cc3a3 is on the
> privous dead server e010125048153.bja,60020,1531137365840. That is very very
> strange!!!
> {code:java}
> 2018-07-09 20:06:17,985 INFO [master/e010125048016:6]
> assignment.RegionStateStore: Load hbase:meta entry
> region=4423e4182457c5b573729be4682cc3a3, regionState=OPEN,
> lastHost=e010125049164.bja,60020,1531136465378,
> regionLocation=e010125048153.bja,60020,1531137365840, openSeqNum=16024
> {code}
> 4. the rs was killed
> {code:java}
> 2018-07-09 20:06:20,265 WARN
> [RpcServer.default.FPBQ.Fifo.handler=297,queue=27,port=6]
> assignment.AssignmentManager: Killing e010125049164.bja,60020,1531136465378:
> rit=OPEN, location=e010125048153.bja,60020,1531137365840,
> table=IntegrationTestBigLinkedList,
> region=4423e4182457c5b573729be4682cc3a3reported OPEN on
> server=e010125049164.bja,60020,1531136465378 but state has otherwise.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20914) Trim Master memory usage
[
https://issues.apache.org/jira/browse/HBASE-20914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550106#comment-16550106
]
Ted Yu commented on HBASE-20914:
+1
{code}
- rackLocalities = new float[numRegions][numServers];
+ rackLocalities = new float[numRegions][numRacks];
{code}
Good catch.
> Trim Master memory usage
>
>
> Key: HBASE-20914
> URL: https://issues.apache.org/jira/browse/HBASE-20914
> Project: HBase
> Issue Type: Sub-task
> Components: master
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 3.0.0, 2.0.2, 2.2.0, 2.1.1
>
> Attachments: HBASE-20914.branch-2.0.001.patch, Screen Shot 2018-07-19
> at 1.20.23 PM.png, Screen Shot 2018-07-19 at 1.20.23 PM.png, Screen Shot
> 2018-07-19 at 1.38.56 PM.png, Screen Shot 2018-07-19 at 1.38.56 PM.png,
> Screen Shot 2018-07-19 at 2.22.42 PM.png, Screen Shot 2018-07-19 at 2.22.42
> PM.png, Screen Shot 2018-07-19 at 2.43.42 PM.png, Screen Shot 2018-07-19 at
> 2.43.42 PM.png
>
>
> While working on the parent issue, looking at a heap from a Master tha was
> running ~650 servers and > 300k regions, I tripped over some silly items in
> the heap:
> 1. Balancer has a regions x server matrix which takes up 18% of the Master
> heap according to jxray and 40% according to eclipse. Looks like the matrix
> should be regions x racks which would be much smaller (Issue came in with
> HBASE-18164 Fast locality computation in balancer -Added new
> LocalityCostFunction and LocalityCandidateGenerator ..). See !Screen Shot
> 2018-07-19 at 1.20.23 PM.png! !Screen Shot 2018-07-19 at 1.38.56 PM.png!
> 2. Duplicate Strings make up ~5% of the Master heap. Of these, ServerName
> seems to be the font. Interesting is report that there 54k instances of
> ServerName in this heap though there are only 650 Servers. See !Screen Shot
> 2018-07-19 at 2.22.42 PM.png!
> 3. ArrayDequeue initializes its internal elements array with 16 elements. We
> use this in a few places. In Procedures, of which there are many in this
> heap, we near never make use of this array. See !Screen Shot 2018-07-19 at
> 2.43.42 PM.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20895) NPE in RpcServer#readAndProcess
[
https://issues.apache.org/jira/browse/HBASE-20895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550092#comment-16550092
]
Hadoop QA commented on HBASE-20895:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
41s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} branch-1 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} branch-1 passed with JDK v1.7.0_181 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
21s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
35s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} branch-1 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} branch-1 passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
36s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
1m 35s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 97m 31s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}115m 49s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:61288f8 |
| JIRA Issue | HBASE-20895 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12932156/HBASE-20895-branch-1.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 7010020a45a8 3.13.0-153-generic #203-Ubuntu
[jira] [Commented] (HBASE-20914) Trim Master memory usage
[
https://issues.apache.org/jira/browse/HBASE-20914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550090#comment-16550090
]
Hadoop QA commented on HBASE-20914:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.0 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
56s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
36s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
49s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
13s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
21s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} branch-2.0 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
43s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
23s{color} | {color:red} hbase-common: The patch generated 1 new + 24 unchanged
- 0 fixed = 25 total (was 24) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
17s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
11m 41s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
38s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
54s{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}120m
11s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
4s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}174m 51s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:6f01af0 |
| JIRA Issue | HBASE-20914 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12932334/HBASE-20914.branch-2.0.001.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 3e0036a5b640 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HBASE-20908) Infinite loop on regionserver if region replica are reduced
[
https://issues.apache.org/jira/browse/HBASE-20908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550084#comment-16550084
]
Ted Yu commented on HBASE-20908:
Failed test was not related to patch.
> Infinite loop on regionserver if region replica are reduced
>
>
> Key: HBASE-20908
> URL: https://issues.apache.org/jira/browse/HBASE-20908
> Project: HBase
> Issue Type: Bug
> Components: read replicas
>Affects Versions: 1.2.0, 2.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HBASE-20908.patch, HBASE-20908_v1.patch,
> HBASE-20908_v3.patch
>
>
> Steps to reproduce
> {code}
> hbase(main):003:0> create 'myTable','cf',{REGION_REPLICATION=>3}
> hbase(main):003:0> put 'myTable','r1','cf:col1','1'
> 0 row(s) in 0.1230 seconds
> hbase(main):004:0> disable 'myTable'
> alter '0 row(s) in 2.3040 seconds
> hbase(main):005:0> alter 'myTable',{REGION_REPLICATION=>1}
> Updating all regions with the new schema...
> 1/1 regions updated.
> Done.
> 0 row(s) in 11.9550 seconds
> hbase(main):006:0> enable 'myTable'
> 0 row(s) in 1.2620 seconds
> hbase(main):007:0> put 'myTable1','r2','cf:col1','1'
> 0 row(s) in 0.0060 seconds
> {code}
> This is the replica region request which will not be present now in Meta but
> was there in cache. Server will say that he is not serving this region.
> {code}
> com.google.protobuf.ServiceException:
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.NotServingRegionException):
> org.apache.hadoop.hbase.NotServingRegionException: Region
> d997d9b47a106216b9b117617ec09015 is not online on
> 10.22.9.76,16020,1531341039091
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3124)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3106)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.replay(RSRpcServices.java:1714)
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22773)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2150)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167)
> {code}
> Eventually, when we will update our cache after looking into meta , we will
> get into an infinite loop as this event will not be replicated because the
> location of the replica will not appear again.
> {code}
> java.net.SocketTimeoutException: callTimeout=120, callDuration=2181316:
> Can't get the location null
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:170)
> at
> org.apache.hadoop.hbase.replication.regionserver.RegionReplicaReplicationEndpoint$RetryingRpcCallable.call(RegionReplicaReplicationEndpoint.java:606)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't
> get the location
> at
> org.apache.hadoop.hbase.client.RegionAdminServiceCallable.getRegionLocations(RegionAdminServiceCallable.java:178)
> at
> org.apache.hadoop.hbase.client.RegionAdminServiceCallable.getLocation(RegionAdminServiceCallable.java:105)
> at
> org.apache.hadoop.hbase.client.RegionAdminServiceCallable.prepare(RegionAdminServiceCallable.java:89)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:134)
> ... 5 more
> Caused by: java.io.IOException: HRegionInfo was null in myTable,
> row=keyvalues={myTable,,1531262022075.f2b68622cfd5851023be29d5599db6c9./info:regioninfo/1531262022425/Put/vlen=41/seqid=0,
>
> myTable,,1531262022075.f2b68622cfd5851023be29d5599db6c9./info:seqnumDuringOpen/1531341209944/Put/vlen=8/seqid=0,
>
> myTable,,1531262022075.f2b68622cfd5851023be29d5599db6c9./info:server/1531341209944/Put/vlen=16/seqid=0,
>
> myTable,,1531262022075.f2b68622cfd5851023be29d5599db6c9./info:serverstartcode/1531341209944/Put/vlen=8/seqid=0}
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1289)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1179)
> at
>
[jira] [Commented] (HBASE-20908) Infinite loop on regionserver if region replica are reduced
[
https://issues.apache.org/jira/browse/HBASE-20908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550080#comment-16550080
]
Hadoop QA commented on HBASE-20908:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
34s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
43s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 50s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}128m 36s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}171m 6s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.regionserver.TestCompactingToCellFlatMapMemStore |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20908 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12932323/HBASE-20908_v3.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 231db05fe108 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / b4eacdabd6 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC3 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/13693/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/13693/testReport/ |
| Max. process+thread count | 4773 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console
[jira] [Commented] (HBASE-20026) Add 1.4 release line to the JDK and Hadoop expectation tables
[ https://issues.apache.org/jira/browse/HBASE-20026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550038#comment-16550038 ] Andrew Purtell commented on HBASE-20026: That was my thought... If I think of it while putting together the RC I'll do it > Add 1.4 release line to the JDK and Hadoop expectation tables > - > > Key: HBASE-20026 > URL: https://issues.apache.org/jira/browse/HBASE-20026 > Project: HBase > Issue Type: Task > Components: documentation >Affects Versions: 1.4.0 >Reporter: Sean Busbey >Priority: Critical > Fix For: 1.4.7 > > > the ref guide currently doesn't have any expectations listed for branch-1.4 > releases around JDK and Hadoop versions. > either add it, or maybe update the existing entries so we have "1.2, 1.3, > 1.4" in a single entry. unless we're ready to include something different > among them. (Maybe note the default Hadoop we ship with? Or Hadoop 2.8.2+ > moving to S maybe? if we've actually done any of the legwork.) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20026) Add 1.4 release line to the JDK and Hadoop expectation tables
[ https://issues.apache.org/jira/browse/HBASE-20026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550037#comment-16550037 ] Mike Drob commented on HBASE-20026: --- [~apurtell] - this one would be nice to get done before the next release, OTOH it hasn't ever been done up until now so what's one more release.. > Add 1.4 release line to the JDK and Hadoop expectation tables > - > > Key: HBASE-20026 > URL: https://issues.apache.org/jira/browse/HBASE-20026 > Project: HBase > Issue Type: Task > Components: documentation >Affects Versions: 1.4.0 >Reporter: Sean Busbey >Priority: Critical > Fix For: 1.4.7 > > > the ref guide currently doesn't have any expectations listed for branch-1.4 > releases around JDK and Hadoop versions. > either add it, or maybe update the existing entries so we have "1.2, 1.3, > 1.4" in a single entry. unless we're ready to include something different > among them. (Maybe note the default Hadoop we ship with? Or Hadoop 2.8.2+ > moving to S maybe? if we've actually done any of the legwork.) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20401) Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable
[
https://issues.apache.org/jira/browse/HBASE-20401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550033#comment-16550033
]
Hadoop QA commented on HBASE-20401:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 4m
50s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m
1s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
43s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
13m 42s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}177m
26s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}237m 27s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20401 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12932296/HBASE-20401.master.005.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux bbcfae429eae 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14
08:53:28 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / c92cda8420 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/13690/testReport/ |
| Max. process+thread count | 4779 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/13690/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Make `MAX_WAIT` and
[jira] [Commented] (HBASE-20901) Reducing region replica has no effect
[
https://issues.apache.org/jira/browse/HBASE-20901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550019#comment-16550019
]
Hadoop QA commented on HBASE-20901:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
15s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
56s{color} | {color:green} root: The patch generated 0 new + 83 unchanged - 63
fixed = 83 total (was 146) {color} |
| {color:green}+1{color} | {color:green} pylint {color} | {color:green} 0m
5s{color} | {color:green} There were no new pylint issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
22s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 6s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}181m 31s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}242m 49s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.replication.TestSyncReplicationStandbyKillMaster |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20901 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12932289/HBASE-20901_v2.patch |
| Optional Tests | asflicense pylint javac javadoc unit findbugs
shadedjars hadoopcheck
[jira] [Updated] (HBASE-16071) The VisibilityLabelFilter and AccessControlFilter should not count the "delete cell"
[
https://issues.apache.org/jira/browse/HBASE-16071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-16071:
---
Fix Version/s: (was: 1.4.6)
(was: 1.3.3)
2.2.0
> The VisibilityLabelFilter and AccessControlFilter should not count the
> "delete cell"
>
>
> Key: HBASE-16071
> URL: https://issues.apache.org/jira/browse/HBASE-16071
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Minor
> Fix For: 3.0.0, 1.5.0, 2.2.0
>
> Attachments: HBASE-16071-v1.patch, HBASE-16071-v2.patch,
> HBASE-16071-v3.patch
>
>
> The VisibilityLabelFilter will see and count the "delete cell" if the
> scan.isRaw() returns true, so the (put) cell will be skipped if it has lower
> version than "delete cell"
> The critical code is shown below:
> {code:title=VisibilityLabelFilter.java|borderStyle=solid}
> public ReturnCode filterKeyValue(Cell cell) throws IOException {
> if (curFamily.getBytes() == null
> || !(CellUtil.matchingFamily(cell, curFamily.getBytes(),
> curFamily.getOffset(),
> curFamily.getLength( {
> curFamily.set(cell.getFamilyArray(), cell.getFamilyOffset(),
> cell.getFamilyLength());
> // For this family, all the columns can have max of
> curFamilyMaxVersions versions. No need to
> // consider the older versions for visibility label check.
> // Ideally this should have been done at a lower layer by HBase (?)
> curFamilyMaxVersions = cfVsMaxVersions.get(curFamily);
> // Family is changed. Just unset curQualifier.
> curQualifier.unset();
> }
> if (curQualifier.getBytes() == null
> || !(CellUtil.matchingQualifier(cell, curQualifier.getBytes(),
> curQualifier.getOffset(),
> curQualifier.getLength( {
> curQualifier.set(cell.getQualifierArray(), cell.getQualifierOffset(),
> cell.getQualifierLength());
> curQualMetVersions = 0;
> }
> curQualMetVersions++;
> if (curQualMetVersions > curFamilyMaxVersions) {
> return ReturnCode.SKIP;
> }
> return this.expEvaluator.evaluate(cell) ? ReturnCode.INCLUDE :
> ReturnCode.SKIP;
> }
> {code}
> [VisibilityLabelFilter.java|https://github.com/apache/hbase/blob/d7a4499dfc8b3936a0eca867589fc2b23b597866/hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityLabelFilter.java]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-14223) Meta WALs are not cleared if meta region was closed and RS aborts
[
https://issues.apache.org/jira/browse/HBASE-14223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-14223:
---
Fix Version/s: (was: 1.4.6)
1.4.7
2.2.0
> Meta WALs are not cleared if meta region was closed and RS aborts
> -
>
> Key: HBASE-14223
> URL: https://issues.apache.org/jira/browse/HBASE-14223
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.3.3, 1.2.8, 2.2.0, 1.4.7
>
> Attachments: HBASE-14223logs, hbase-14223_v0.patch,
> hbase-14223_v1-branch-1.patch, hbase-14223_v2-branch-1.patch,
> hbase-14223_v3-branch-1.patch, hbase-14223_v3-branch-1.patch,
> hbase-14223_v3-master.patch
>
>
> When an RS opens meta, and later closes it, the WAL(FSHlog) is not closed.
> The last WAL file just sits there in the RS WAL directory. If RS stops
> gracefully, the WAL file for meta is deleted. Otherwise if RS aborts, WAL for
> meta is not cleaned. It is also not split (which is correct) since master
> determines that the RS no longer hosts meta at the time of RS abort.
> From a cluster after running ITBLL with CM, I see a lot of {{-splitting}}
> directories left uncleaned:
> {code}
> [root@os-enis-dal-test-jun-4-7 cluster-os]# sudo -u hdfs hadoop fs -ls
> /apps/hbase/data/WALs
> Found 31 items
> drwxr-xr-x - hbase hadoop 0 2015-06-05 01:14
> /apps/hbase/data/WALs/hregion-58203265
> drwxr-xr-x - hbase hadoop 0 2015-06-05 07:54
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-1.openstacklocal,16020,1433489308745-splitting
> drwxr-xr-x - hbase hadoop 0 2015-06-05 09:28
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-1.openstacklocal,16020,1433494382959-splitting
> drwxr-xr-x - hbase hadoop 0 2015-06-05 10:01
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-1.openstacklocal,16020,1433498252205-splitting
> ...
> {code}
> The directories contain WALs from meta:
> {code}
> [root@os-enis-dal-test-jun-4-7 cluster-os]# sudo -u hdfs hadoop fs -ls
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting
> Found 2 items
> -rw-r--r-- 3 hbase hadoop 201608 2015-06-05 03:15
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433470511501.meta
> -rw-r--r-- 3 hbase hadoop 44420 2015-06-05 04:36
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433474111645.meta
> {code}
> The RS hosted the meta region for some time:
> {code}
> 2015-06-05 03:14:28,692 INFO [PostOpenDeployTasks:1588230740]
> zookeeper.MetaTableLocator: Setting hbase:meta region location in ZooKeeper
> as os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285
> ...
> 2015-06-05 03:15:17,302 INFO
> [RS_CLOSE_META-os-enis-dal-test-jun-4-5:16020-0] regionserver.HRegion: Closed
> hbase:meta,,1.1588230740
> {code}
> In between, a WAL is created:
> {code}
> 2015-06-05 03:15:11,707 INFO
> [RS_OPEN_META-os-enis-dal-test-jun-4-5:16020-0-MetaLogRoller] wal.FSHLog:
> Rolled WAL
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433470511501.meta
> with entries=385, filesize=196.88 KB; new WAL
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433474111645.meta
> {code}
> When CM killed the region server later master did not see these WAL files:
> {code}
> ./hbase-hbase-master-os-enis-dal-test-jun-4-3.log:2015-06-05 03:36:46,075
> INFO [MASTER_SERVER_OPERATIONS-os-enis-dal-test-jun-4-3:16000-0]
> master.SplitLogManager: started splitting 2 logs in
> [hdfs://os-enis-dal-test-jun-4-1.openstacklocal:8020/apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting]
> for [os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285]
> ./hbase-hbase-master-os-enis-dal-test-jun-4-3.log:2015-06-05 03:36:47,300
> INFO [main-EventThread] wal.WALSplitter: Archived processed log
> hdfs://os-enis-dal-test-jun-4-1.openstacklocal:8020/apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285.default.1433475074436
> to
> hdfs://os-enis-dal-test-jun-4-1.openstacklocal:8020/apps/hbase/data/oldWALs/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285.default.1433475074436
>
[jira] [Updated] (HBASE-12943) Set sun.net.inetaddr.ttl in HBase
[ https://issues.apache.org/jira/browse/HBASE-12943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-12943: --- Fix Version/s: (was: 1.4.6) (was: 1.3.3) > Set sun.net.inetaddr.ttl in HBase > - > > Key: HBASE-12943 > URL: https://issues.apache.org/jira/browse/HBASE-12943 > Project: HBase > Issue Type: Bug >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Major > Fix For: 3.0.0, 1.5.0, 2.2.0 > > Attachments: 12943-1-master.txt > > > The default value of config: sun.net.inetaddr.ttl is -1 and the java > processes will cache the mapping of hostname to ip address forever, See: > http://docs.oracle.com/javase/7/docs/technotes/guides/net/properties.html > But things go wrong when a regionserver with same hostname and different ip > address rejoins the hbase cluster. The HMaster will get wrong ip address of > the regionserver from this cache and every region assignment to this > regionserver will be blocked for a time because the HMaster can't communicate > with the regionserver. > A tradeoff is to set the sun.net.inetaddr.ttl to 10m or 1h and make the wrong > cache expired. > Suggestions are welcomed. Thanks~ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-13160) SplitLogWorker does not pick up the task immediately
[
https://issues.apache.org/jira/browse/HBASE-13160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-13160:
---
Fix Version/s: (was: 1.4.6)
(was: 1.3.3)
> SplitLogWorker does not pick up the task immediately
>
>
> Key: HBASE-13160
> URL: https://issues.apache.org/jira/browse/HBASE-13160
> Project: HBase
> Issue Type: Improvement
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Major
> Fix For: 1.5.0
>
> Attachments: hbase-13160_v1.patch
>
>
> We were reading some code with Jeffrey, and we realized that the
> SplitLogWorker's internal task loop is weird. It does {{ls}} every second and
> sleeps, but have another mechanism to learn about new tasks, but does not
> make affective use of the zk notification.
> I have a simple patch which might improve this area.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-17885) Backport HBASE-15871 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-17885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-17885: --- Fix Version/s: (was: 1.4.6) (was: 1.2.8) (was: 1.3.3) > Backport HBASE-15871 to branch-1 > > > Key: HBASE-17885 > URL: https://issues.apache.org/jira/browse/HBASE-17885 > Project: HBase > Issue Type: Bug > Components: Scanners >Affects Versions: 1.3.1, 1.2.5, 1.1.8 >Reporter: ramkrishna.s.vasudevan >Priority: Major > Fix For: 1.5.0 > > > Will try to rebase the branch-1 patch at the earliest. Hope the fix versions > are correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-18228) HBCK improvements
[ https://issues.apache.org/jira/browse/HBASE-18228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-18228: --- Fix Version/s: (was: 1.4.6) > HBCK improvements > - > > Key: HBASE-18228 > URL: https://issues.apache.org/jira/browse/HBASE-18228 > Project: HBase > Issue Type: Improvement > Components: hbck >Reporter: Lars Hofhansl >Assignee: Karan Mehta >Priority: Critical > Fix For: 3.0.0, 1.5.0, 2.2.0 > > Attachments: HBASE-18228.branch-1.3.patch > > > We just had a prod issue and running HBCK the way we did actually causes more > problems. > In part HBCK did stuff we did not expect, in part we had little visibility > into what HBCK was doing, and in part the logging was confusing. > I'm proposing 2 improvements: > 1. A dry-run mode. Run, and just list what would have been done. > 2. An interactive mode. Run, and for each action request Y/N user input. So > that a user can opt-out of stuff. > [~jmhsieh], FYI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-18639) nightly jobs that need to do jdk7 + jdk8 need more than 6 hours
[ https://issues.apache.org/jira/browse/HBASE-18639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-18639: --- Fix Version/s: (was: 1.4.6) 1.4.7 > nightly jobs that need to do jdk7 + jdk8 need more than 6 hours > --- > > Key: HBASE-18639 > URL: https://issues.apache.org/jira/browse/HBASE-18639 > Project: HBase > Issue Type: Task > Components: test >Reporter: Sean Busbey >Priority: Major > Labels: beginner > Fix For: 1.5.0, 1.3.3, 1.2.8, 1.4.7 > > > looks like the branches that need to do 2 jdks will need more than 6 hours, > at least until we can parallelize things. > e.g. > * > [branch-1.2|https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Nightly/job/branch-1.2/buildTimeTrend] > * > [branch-1.3|https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Nightly/job/branch-1.3/buildTimeTrend] > branch-1.4 and branch-1 are failing in jdk7 checks, so they don't go that > long yet, but once we get those tests fixed presumably they'll need the time > as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-18549) Unclaimed replication queues can go undetected
[ https://issues.apache.org/jira/browse/HBASE-18549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-18549: --- Fix Version/s: (was: 1.4.6) 1.4.7 > Unclaimed replication queues can go undetected > -- > > Key: HBASE-18549 > URL: https://issues.apache.org/jira/browse/HBASE-18549 > Project: HBase > Issue Type: Bug > Components: Replication >Reporter: Ashu Pachauri >Priority: Critical > Fix For: 1.5.0, 1.3.3, 1.4.7 > > > We have come across this situation multiple times where a zookeeper issues > can cause NodeFailoverWorker to fail picking up replication queue for a dead > region server silently. One example is when the znode size for a particular > queue exceed jute.maxBuffer value. > There can be other situations that may lead to this and just go undetected. > We need to have a metric for number of unclaimed replication queues. This > will help in mitigating the problem through alerting on the metric and > identifying underlying issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-18415) The local timeout may cause Admin to submit duplicate request
[ https://issues.apache.org/jira/browse/HBASE-18415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-18415: --- Fix Version/s: (was: 1.4.6) 1.4.7 > The local timeout may cause Admin to submit duplicate request > - > > Key: HBASE-18415 > URL: https://issues.apache.org/jira/browse/HBASE-18415 > Project: HBase > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Major > Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 1.4.7 > > Attachments: HBASE-18415.branch-1.ut.patch, > HBASE-18415.branch-1.v0.patch, HBASE-18415.branch-1.v1.patch, > HBASE-18415.branch-1.v2.patch, HBASE-18415.branch-1.v3.patch, > HBASE-18415.branch-1.v3.patch, HBASE-18415.branch-1.v3.patch, > HBASE-18415.branch-1.v4.patch, HBASE-18415.branch-1.v4.patch, > HBASE-18415.branch-1.v4.patch > > > After a timeout occurs on first request, client will retry the request with > distinct group/nonce. The second request may bring the TableXXXException back > if the first request have changed the table state. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19444) RSGroups test units cannot be concurrently executed
[ https://issues.apache.org/jira/browse/HBASE-19444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-19444: --- Fix Version/s: (was: 1.4.6) 1.4.7 2.2.0 > RSGroups test units cannot be concurrently executed > --- > > Key: HBASE-19444 > URL: https://issues.apache.org/jira/browse/HBASE-19444 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.0 >Reporter: Andrew Purtell >Priority: Minor > Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.7 > > > TestRSGroups and friends cannot be concurrently executed or they are very > likely to flake, failing with constraint exceptions. If executed serially all > units pass. Fix for concurrent execution. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20546) Improve perf of RegionLocationFinder.mapHostNameToServerName
[ https://issues.apache.org/jira/browse/HBASE-20546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-20546: --- Fix Version/s: (was: 2.0.2) (was: 1.4.6) Status: Open (was: Patch Available) > Improve perf of RegionLocationFinder.mapHostNameToServerName > > > Key: HBASE-20546 > URL: https://issues.apache.org/jira/browse/HBASE-20546 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0, 1.4.4 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan >Priority: Major > Fix For: 1.5.0 > > Attachments: HBASE-20546.branch-1.4.001.patch > > > RegionLocationFinder.getTopBlockLocations() is called multiple times during > balancer. While profiling on a large table balance, mapHostNameToServerName() > seem to take a lot of time. One of the maps is repeatedly created for each > iteration, while we can just initialize it once. > Goes into both branch-1 and branch-2, although patches differ slightly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20026) Add 1.4 release line to the JDK and Hadoop expectation tables
[ https://issues.apache.org/jira/browse/HBASE-20026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-20026: --- Fix Version/s: (was: 1.4.6) 1.4.7 > Add 1.4 release line to the JDK and Hadoop expectation tables > - > > Key: HBASE-20026 > URL: https://issues.apache.org/jira/browse/HBASE-20026 > Project: HBase > Issue Type: Task > Components: documentation >Affects Versions: 1.4.0 >Reporter: Sean Busbey >Priority: Critical > Fix For: 1.4.7 > > > the ref guide currently doesn't have any expectations listed for branch-1.4 > releases around JDK and Hadoop versions. > either add it, or maybe update the existing entries so we have "1.2, 1.3, > 1.4" in a single entry. unless we're ready to include something different > among them. (Maybe note the default Hadoop we ship with? Or Hadoop 2.8.2+ > moving to S maybe? if we've actually done any of the legwork.) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20869) Endpoint-based Export use incorrect user to write to destination
[
https://issues.apache.org/jira/browse/HBASE-20869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550011#comment-16550011
]
Hudson commented on HBASE-20869:
Results for branch branch-2.1
[build #80 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/80/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/80//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/80//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/80//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Endpoint-based Export use incorrect user to write to destination
>
>
> Key: HBASE-20869
> URL: https://issues.apache.org/jira/browse/HBASE-20869
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors
>Affects Versions: 2.0.0
> Environment: Hadoop 3.0.0 + HBase 2.0.0, Kerberos.
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Fix For: 3.0.0, 2.0.2, 2.2.0, 2.1.1
>
> Attachments: HBASE-20869.master.001.patch,
> HBASE-20869.master.002.patch
>
>
> HBASE-15806 implemented an endpoint based export. It gets caller's HDFS
> delegation token, and RegionServer is supposed to write out exported files as
> the caller.
> Everything works fine if you use run export as hbase user. However, once you
> use a different user to export, it fails.
> To reproduce,
> Add to configuration key hbase.coprocessor.region.classes the coprocessor
> class org.apache.hadoop.hbase.coprocessor.Export.
> create a table t1, assign permission to a user foo:
>
> {noformat}
> hbase(main):004:0> user_permission 't1'
> User Namespace,Table,Family,Qualifier:Permission
> hbase default,t1,,: [Permission: actions=READ,WRITE,EXEC,CREATE,ADMIN]
> foo default,t1,,: [Permission: actions=READ,WRITE,EXEC,CREATE,ADMIN]{noformat}
>
> As user foo, execute the following command:
>
> {noformat}
> $ hdfs dfs -mkdir /tmp/export_hbase2
> $ hbase org.apache.hadoop.hbase.coprocessor.Export t1 /tmp/export_hbase2/t2/
>
> 18/07/10 14:03:59 INFO client.RpcRetryingCallerImpl: Call exception, tries=6,
> retries=6, started=4457 ms ago, cancelled=false,
> msg=org.apache.hadoop.security.AccessControlException: Permission denied:
> user=hbase, access=WRITE,
> inode="/tmp/export_hbase2/t2":foo:supergroup:drwxr-xr-x
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:400)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:256)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:194)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1846)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1830)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1789)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.resolvePathForStartFile(FSDirWriteFileOp.java:316)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2411)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2343)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:764)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:451)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
> at
[jira] [Commented] (HBASE-20853) Polish "Add defaults to Table Interface so Implementors don't have to"
[
https://issues.apache.org/jira/browse/HBASE-20853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550010#comment-16550010
]
Hudson commented on HBASE-20853:
Results for branch branch-2.1
[build #80 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/80/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/80//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/80//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/80//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Polish "Add defaults to Table Interface so Implementors don't have to"
> --
>
> Key: HBASE-20853
> URL: https://issues.apache.org/jira/browse/HBASE-20853
> Project: HBase
> Issue Type: Sub-task
> Components: API
>Reporter: stack
>Assignee: Balazs Meszaros
>Priority: Major
> Labels: beginner, beginners
> Fix For: 3.0.0, 2.0.2, 2.2.0, 2.1.1
>
> Attachments: HBASE-20853.master.001.patch,
> HBASE-20853.master.002.patch, HBASE-20853.master.003.patch,
> HBASE-20853.master.004.patch, HBASE-20853.master.005.patch
>
>
> This issue is to address feedback that came in after commit on the parent
> (FYI [~chia7712]). See tail of parent issue and amendment attached to parent
> adding better defaults to the Table Interface.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20618) Skip large rows instead of throwing an exception to client
[
https://issues.apache.org/jira/browse/HBASE-20618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-20618:
---
Fix Version/s: (was: 2.0.2)
(was: 1.4.6)
2.2.0
1.5.0
> Skip large rows instead of throwing an exception to client
> --
>
> Key: HBASE-20618
> URL: https://issues.apache.org/jira/browse/HBASE-20618
> Project: HBase
> Issue Type: New Feature
>Reporter: Swapna
>Priority: Minor
> Fix For: 3.0.0, 1.5.0, 2.2.0
>
> Attachments: HBASE-20618.hbasemaster.v01.patch,
> HBASE-20618.hbasemaster.v02.patch, HBASE-20618.v1.branch-1.patch,
> HBASE-20618.v1.branch-1.patch
>
>
> Currently HBase supports throwing RowTooBigException incase there is a row
> with one of the column family data exceeds the configured maximum
> https://issues.apache.org/jira/browse/HBASE-10925?attachmentOrder=desc
> We have some bad rows growing very large. We need a way to skip these rows
> for most of our jobs.
> Some of the options we considered:
> Option 1:
> Hbase client handle the exception and restart the scanner past bad row by
> capturing the row key where it failed. Can be by adding the rowkey to the
> exception stack trace, which seems brittle. Client would ignore the setting
> if its upgraded before server.
> Option 2:
> Skip through big rows on Server.Go with server level config similar to
> "hbase.table.max.rowsize" or request based by changing the scan request api.
> If allowed to do per request, based on the scan request config, Client will
> have to ignore the setting if its upgraded before server.
> {code}
> try {
> populateResult(results, this.storeHeap, scannerContext, current);
> } catch(RowTooBigException e) {
> LOG.info("Row exceeded the limit in storeheap. Skipping row with
> key:"+Bytes.toString(current.getRowArray()));
> this.storeHeap.reseek(PrivateCellUtil.createLastOnRow(current));
> results.clear();
> scannerContext.clearProgress();
> continue;
> }
> {code}
> Prefer the option 2 with server level config. Please share your inputs
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20895) NPE in RpcServer#readAndProcess
[
https://issues.apache.org/jira/browse/HBASE-20895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-20895:
---
Status: Patch Available (was: Open)
> NPE in RpcServer#readAndProcess
> ---
>
> Key: HBASE-20895
> URL: https://issues.apache.org/jira/browse/HBASE-20895
> Project: HBase
> Issue Type: Bug
> Components: rpc
>Affects Versions: 1.3.2
>Reporter: Andrew Purtell
>Assignee: Monani Mihir
>Priority: Major
> Fix For: 1.5.0, 1.3.3, 1.4.6
>
> Attachments: HBASE-20895-branch-1.patch
>
>
> {noformat}
> 2018-07-10 16:25:55,005 DEBUG [.sfdc.net,port=60020] ipc.RpcServer -
> RpcServer.listener,port=60020: Caught exception while reading:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1761)
> at
> org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:949)
> at
> org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:730)
> at
> org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:706)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> This looks like it could be a use after close problem if there is concurrent
> access to a Connection.
> In process() we might store a null back to the 'data' field.
> Meanwhile in readAndProcess() we have a case where we might be blocked on a
> channel read and then after coming back from the read we go to use 'data'
> after a null has been written back, leading to a NPE.
> {quote}count = channelRead(channel, data);
> 1761 ---> if (count >= 0 && *data.remaining()* == 0)
> \{ process(); }{quote}
> Whether a NPE happens or not is going to depend on the timing of the store
> back to 'data' in another thread and use of 'data' in this thread and whether
> or not the JVM has optimized away a reload of 'data' (it's not declared
> volatile)
> We should do a null check here just to be defensive. We should also look at
> whether concurrent access to the Connection is happening and intended.The
> above is just a theory. We should also look at other execution sequences that
> could lead to 'data' being null in this location. At a glance I didn't find
> one but the store to 'data' happens behind conditionals so it is possible.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20401) Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable
[
https://issues.apache.org/jira/browse/HBASE-20401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550005#comment-16550005
]
Hadoop QA commented on HBASE-20401:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
31s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
12s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} branch-1 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} branch-1 passed with JDK v1.7.0_181 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
29s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
43s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} branch-1 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} branch-1 passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
14s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
1m 55s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}135m 45s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}158m 43s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.mapreduce.TestLoadIncrementalHFiles |
| | hadoop.hbase.mapreduce.TestLoadIncrementalHFilesUseSecurityEndPoint |
| | hadoop.hbase.mapreduce.TestSecureLoadIncrementalHFiles |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:61288f8 |
| JIRA Issue | HBASE-20401 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12932310/HBASE-20401.branch-1.001.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs
[jira] [Updated] (HBASE-20643) Getting HDFSBlockDist in Master by querying RegionServers
[ https://issues.apache.org/jira/browse/HBASE-20643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-20643: --- Fix Version/s: (was: 1.4.6) 1.4.7 > Getting HDFSBlockDist in Master by querying RegionServers > - > > Key: HBASE-20643 > URL: https://issues.apache.org/jira/browse/HBASE-20643 > Project: HBase > Issue Type: Improvement > Components: Balancer >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan >Priority: Major > Fix For: 1.5.0, 2.2.0, 1.4.7 > > > Region locality information is needed by the balancer to generate region > plans. Computing HDFSBlockDistribution is expensive on larger clusters and > adds load to the NameNode. This also needs to be recomputed on a master > restart. The proposal is to get the HDFSBlockDistribution from the > RegionServers instead of computing it in Master. RS already has this > information and we could just reuse it by querying it. RS already passes > dataLocality info via RegionLoad today. > Proposed Implementation: This is a high-level overview. > # A RegionServer API has to be added which will return HDFSBlockDistribution > for all the regions it hosts. RS already has this info. Since ClusterStatus > has already become bulky and we don’t need updated locality so fast, it’s > better to have another API rather than add this to RegionLoad and pass it > along with RSReport. > # Master will have a Chore to query all RegionServers and will cache the > HDFSBlockDistribution for those regions. This is easy and quick. Admins can > tune the frequency based on size of the cluster. On a ~90 nodes cluster with > 500k regions and a prototype implementation and no load, it took about 5 > seconds to get all HDFSBlockDistribution from RS. > # The cache will be an extension of RegionLocationFinder (subclass), if > needed to keep the implementation simple. Probably will get clear with > implementation. > # Balancer will use the new cache to get all HDFSBlockDistribution. If there > is a new region and Chore didn’t get the block distribution from RS during > its previous run, then it will be computed by RegionLocationFinder the same > way it has been done now. If the Chore runs more frequently like every hour, > then this recomputation will be drastically reduced. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20855) PeerConfigTracker only supporting one listener will cause problem when there is a recovered replication queue
[
https://issues.apache.org/jira/browse/HBASE-20855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-20855:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
> PeerConfigTracker only supporting one listener will cause problem when there
> is a recovered replication queue
> -
>
> Key: HBASE-20855
> URL: https://issues.apache.org/jira/browse/HBASE-20855
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.0, 1.4.0, 1.5.0
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
>Priority: Major
> Fix For: 1.5.0, 1.4.6
>
> Attachments: HBASE-20855.branch-1.001.patch,
> HBASE-20855.branch-1.002.patch, HBASE-20855.branch-1.003.patch,
> HBASE-20855.branch-1.004.patch, HBASE-20855.branch-1.005.patch,
> HBASE-20855.branch-1.006.patch, HBASE-20855.branch-1.007.patch
>
>
> {code}
> public void init(Context context) throws IOException {
> this.ctx = context;
> if (this.ctx != null){
> ReplicationPeer peer = this.ctx.getReplicationPeer();
> if (peer != null){
> peer.trackPeerConfigChanges(this);
> } else {
> LOG.warn("Not tracking replication peer config changes for Peer Id " +
> this.ctx.getPeerId() +
> " because there's no such peer");
> }
> }
> }
> {code}
> As we know, replication source will set itself to the PeerConfigTracker in
> ReplicationPeer. When there is one or more recovered queue, each queue will
> generate a new replication source, But they share the same ReplicationPeer.
> Then when it calls setListener, the new generated one will cover the older
> one. Thus there will only has one ReplicationPeer that receive the peer
> config change notify.
> {code}
> public synchronized void setListener(ReplicationPeerConfigListener listener){
> this.listener = listener;
> }
> {code}
>
> To solve this, PeerConfigTracker need to support multiple listener and
> listener should be removed when the replication endpoint terminated.
> I will upload a patch later with fix and UT.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20694) Consolidate warning on SecureBulkLoad directory permissions
[ https://issues.apache.org/jira/browse/HBASE-20694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-20694: --- Fix Version/s: (was: 1.4.6) 1.4.7 > Consolidate warning on SecureBulkLoad directory permissions > --- > > Key: HBASE-20694 > URL: https://issues.apache.org/jira/browse/HBASE-20694 > Project: HBase > Issue Type: Improvement > Components: documentation >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Minor > Fix For: 3.0.0, 1.5.0, 1.2.7, 1.3.3, 2.2.0, 1.4.7 > > > Follow-on from HBASE-20605: > HBase 1.x has a check which ignores a directory permission check if you're > using a specific filesystem which we think doesnt' do security properly. > HBase 2.x dropped this check. > Since the security of bulk-loaded data is dependent upon this directory > permission (and thus the capabilities of the FileSystem), it would be better > to have a consistent warning across branches. > [~busbey] suggested that we make a WARN message which points admins to our > Book (and write such a section if we don't have something sufficient already) > and our supported filesystems, coupled with an option to disable that warning > message. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20896) Port HBASE-20866 to branch-1 and branch-1.4
[ https://issues.apache.org/jira/browse/HBASE-20896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-20896: --- Fix Version/s: (was: 1.4.6) 1.4.7 > Port HBASE-20866 to branch-1 and branch-1.4 > > > Key: HBASE-20896 > URL: https://issues.apache.org/jira/browse/HBASE-20896 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Vikas Vishwakarma >Priority: Major > Fix For: 1.5.0, 1.4.7 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20901) Reducing region replica has no effect
[
https://issues.apache.org/jira/browse/HBASE-20901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550001#comment-16550001
]
Hadoop QA commented on HBASE-20901:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
25s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
41s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
49s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} The patch hbase-client passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
20s{color} | {color:green} hbase-server: The patch generated 0 new + 17
unchanged - 63 fixed = 17 total (was 80) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
49s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
12m 54s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
19s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}149m
48s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
46s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}213m 0s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20901 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12932293/HBASE-20901_v2.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 0af377ca787d 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HBASE-20908) Infinite loop on regionserver if region replica are reduced
[
https://issues.apache.org/jira/browse/HBASE-20908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549993#comment-16549993
]
Ted Yu commented on HBASE-20908:
Thanks for the quick turn around.
lgtm, pending QA
> Infinite loop on regionserver if region replica are reduced
>
>
> Key: HBASE-20908
> URL: https://issues.apache.org/jira/browse/HBASE-20908
> Project: HBase
> Issue Type: Bug
> Components: read replicas
>Affects Versions: 1.2.0, 2.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HBASE-20908.patch, HBASE-20908_v1.patch,
> HBASE-20908_v3.patch
>
>
> Steps to reproduce
> {code}
> hbase(main):003:0> create 'myTable','cf',{REGION_REPLICATION=>3}
> hbase(main):003:0> put 'myTable','r1','cf:col1','1'
> 0 row(s) in 0.1230 seconds
> hbase(main):004:0> disable 'myTable'
> alter '0 row(s) in 2.3040 seconds
> hbase(main):005:0> alter 'myTable',{REGION_REPLICATION=>1}
> Updating all regions with the new schema...
> 1/1 regions updated.
> Done.
> 0 row(s) in 11.9550 seconds
> hbase(main):006:0> enable 'myTable'
> 0 row(s) in 1.2620 seconds
> hbase(main):007:0> put 'myTable1','r2','cf:col1','1'
> 0 row(s) in 0.0060 seconds
> {code}
> This is the replica region request which will not be present now in Meta but
> was there in cache. Server will say that he is not serving this region.
> {code}
> com.google.protobuf.ServiceException:
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.NotServingRegionException):
> org.apache.hadoop.hbase.NotServingRegionException: Region
> d997d9b47a106216b9b117617ec09015 is not online on
> 10.22.9.76,16020,1531341039091
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3124)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3106)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.replay(RSRpcServices.java:1714)
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22773)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2150)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167)
> {code}
> Eventually, when we will update our cache after looking into meta , we will
> get into an infinite loop as this event will not be replicated because the
> location of the replica will not appear again.
> {code}
> java.net.SocketTimeoutException: callTimeout=120, callDuration=2181316:
> Can't get the location null
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:170)
> at
> org.apache.hadoop.hbase.replication.regionserver.RegionReplicaReplicationEndpoint$RetryingRpcCallable.call(RegionReplicaReplicationEndpoint.java:606)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't
> get the location
> at
> org.apache.hadoop.hbase.client.RegionAdminServiceCallable.getRegionLocations(RegionAdminServiceCallable.java:178)
> at
> org.apache.hadoop.hbase.client.RegionAdminServiceCallable.getLocation(RegionAdminServiceCallable.java:105)
> at
> org.apache.hadoop.hbase.client.RegionAdminServiceCallable.prepare(RegionAdminServiceCallable.java:89)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:134)
> ... 5 more
> Caused by: java.io.IOException: HRegionInfo was null in myTable,
> row=keyvalues={myTable,,1531262022075.f2b68622cfd5851023be29d5599db6c9./info:regioninfo/1531262022425/Put/vlen=41/seqid=0,
>
> myTable,,1531262022075.f2b68622cfd5851023be29d5599db6c9./info:seqnumDuringOpen/1531341209944/Put/vlen=8/seqid=0,
>
> myTable,,1531262022075.f2b68622cfd5851023be29d5599db6c9./info:server/1531341209944/Put/vlen=16/seqid=0,
>
> myTable,,1531262022075.f2b68622cfd5851023be29d5599db6c9./info:serverstartcode/1531341209944/Put/vlen=8/seqid=0}
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1289)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1179)
> at
>
[jira] [Updated] (HBASE-20901) Reducing region replica has no effect
[ https://issues.apache.org/jira/browse/HBASE-20901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-20901: --- Fix Version/s: 3.0.0 > Reducing region replica has no effect > - > > Key: HBASE-20901 > URL: https://issues.apache.org/jira/browse/HBASE-20901 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Ankit Singhal >Assignee: Ankit Singhal >Priority: Major > Labels: replica > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-20901.patch, HBASE-20901_v1.patch, > HBASE-20901_v2.patch > > > While reducing the region replica, server name(sn) and state column of the > replica are not getting deleted, resulting in assignment manager to think > that these regions are CLOSED and assign them again. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20672) New metrics ReadRequestRate and WriteRequestRate
[ https://issues.apache.org/jira/browse/HBASE-20672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-20672: --- Resolution: Fixed Fix Version/s: 2.2.0 1.5.0 Status: Resolved (was: Patch Available) > New metrics ReadRequestRate and WriteRequestRate > > > Key: HBASE-20672 > URL: https://issues.apache.org/jira/browse/HBASE-20672 > Project: HBase > Issue Type: Improvement > Components: metrics >Reporter: Ankit Jain >Assignee: Ankit Jain >Priority: Minor > Fix For: 3.0.0, 1.5.0, 2.2.0 > > Attachments: HBASE-20672.branch-1.001.patch, > HBASE-20672.branch-1.002.patch, HBASE-20672.branch-2.001.patch, > HBASE-20672.master.001.patch, HBASE-20672.master.002.patch, > HBASE-20672.master.003.patch, hits1vs2.4.40.400.png > > > Hbase currently provides counter read/write requests (ReadRequestCount, > WriteRequestCount). That said it is not easy to use counter that reset only > after a restart of the service, we would like to expose 2 new metrics in > HBase to provide ReadRequestRate and WriteRequestRate at region server level. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-6028) Implement a cancel for in-progress compactions
[
https://issues.apache.org/jira/browse/HBASE-6028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549962#comment-16549962
]
Hudson commented on HBASE-6028:
---
Results for branch branch-2
[build #1002 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1002/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1002//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1002//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1002//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Implement a cancel for in-progress compactions
> --
>
> Key: HBASE-6028
> URL: https://issues.apache.org/jira/browse/HBASE-6028
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Reporter: Derek Wollenstein
>Assignee: Mohit Goel
>Priority: Minor
> Labels: beginner
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-6028.master.007.patch,
> HBASE-6028.master.008.patch, HBASE-6028.master.008.patch,
> HBASE-6028.master.009.patch
>
>
> Depending on current server load, it can be extremely expensive to run
> periodic minor / major compactions. It would be helpful to have a feature
> where a user could use the shell or a client tool to explicitly cancel an
> in-progress compactions. This would allow a system to recover when too many
> regions became eligible for compactions at once
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20869) Endpoint-based Export use incorrect user to write to destination
[
https://issues.apache.org/jira/browse/HBASE-20869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549961#comment-16549961
]
Hudson commented on HBASE-20869:
Results for branch branch-2
[build #1002 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1002/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1002//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1002//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1002//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Endpoint-based Export use incorrect user to write to destination
>
>
> Key: HBASE-20869
> URL: https://issues.apache.org/jira/browse/HBASE-20869
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors
>Affects Versions: 2.0.0
> Environment: Hadoop 3.0.0 + HBase 2.0.0, Kerberos.
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Fix For: 3.0.0, 2.0.2, 2.2.0, 2.1.1
>
> Attachments: HBASE-20869.master.001.patch,
> HBASE-20869.master.002.patch
>
>
> HBASE-15806 implemented an endpoint based export. It gets caller's HDFS
> delegation token, and RegionServer is supposed to write out exported files as
> the caller.
> Everything works fine if you use run export as hbase user. However, once you
> use a different user to export, it fails.
> To reproduce,
> Add to configuration key hbase.coprocessor.region.classes the coprocessor
> class org.apache.hadoop.hbase.coprocessor.Export.
> create a table t1, assign permission to a user foo:
>
> {noformat}
> hbase(main):004:0> user_permission 't1'
> User Namespace,Table,Family,Qualifier:Permission
> hbase default,t1,,: [Permission: actions=READ,WRITE,EXEC,CREATE,ADMIN]
> foo default,t1,,: [Permission: actions=READ,WRITE,EXEC,CREATE,ADMIN]{noformat}
>
> As user foo, execute the following command:
>
> {noformat}
> $ hdfs dfs -mkdir /tmp/export_hbase2
> $ hbase org.apache.hadoop.hbase.coprocessor.Export t1 /tmp/export_hbase2/t2/
>
> 18/07/10 14:03:59 INFO client.RpcRetryingCallerImpl: Call exception, tries=6,
> retries=6, started=4457 ms ago, cancelled=false,
> msg=org.apache.hadoop.security.AccessControlException: Permission denied:
> user=hbase, access=WRITE,
> inode="/tmp/export_hbase2/t2":foo:supergroup:drwxr-xr-x
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:400)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:256)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:194)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1846)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1830)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1789)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.resolvePathForStartFile(FSDirWriteFileOp.java:316)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2411)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2343)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:764)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:451)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
> at
[jira] [Updated] (HBASE-20672) New metrics ReadRequestRate and WriteRequestRate
[ https://issues.apache.org/jira/browse/HBASE-20672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-20672: --- Summary: New metrics ReadRequestRate and WriteRequestRate (was: Create new HBase metrics ReadRequestRate and WriteRequestRate that reset at every monitoring interval) > New metrics ReadRequestRate and WriteRequestRate > > > Key: HBASE-20672 > URL: https://issues.apache.org/jira/browse/HBASE-20672 > Project: HBase > Issue Type: Improvement > Components: metrics >Reporter: Ankit Jain >Assignee: Ankit Jain >Priority: Minor > Fix For: 3.0.0 > > Attachments: HBASE-20672.branch-1.001.patch, > HBASE-20672.branch-1.002.patch, HBASE-20672.branch-2.001.patch, > HBASE-20672.master.001.patch, HBASE-20672.master.002.patch, > HBASE-20672.master.003.patch, hits1vs2.4.40.400.png > > > Hbase currently provides counter read/write requests (ReadRequestCount, > WriteRequestCount). That said it is not easy to use counter that reset only > after a restart of the service, we would like to expose 2 new metrics in > HBase to provide ReadRequestRate and WriteRequestRate at region server level. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20914) Trim Master memory usage
[ https://issues.apache.org/jira/browse/HBASE-20914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20914: -- Fix Version/s: 2.1.1 2.2.0 3.0.0 Status: Patch Available (was: Open) > Trim Master memory usage > > > Key: HBASE-20914 > URL: https://issues.apache.org/jira/browse/HBASE-20914 > Project: HBase > Issue Type: Sub-task > Components: master >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 3.0.0, 2.0.2, 2.2.0, 2.1.1 > > Attachments: HBASE-20914.branch-2.0.001.patch, Screen Shot 2018-07-19 > at 1.20.23 PM.png, Screen Shot 2018-07-19 at 1.20.23 PM.png, Screen Shot > 2018-07-19 at 1.38.56 PM.png, Screen Shot 2018-07-19 at 1.38.56 PM.png, > Screen Shot 2018-07-19 at 2.22.42 PM.png, Screen Shot 2018-07-19 at 2.22.42 > PM.png, Screen Shot 2018-07-19 at 2.43.42 PM.png, Screen Shot 2018-07-19 at > 2.43.42 PM.png > > > While working on the parent issue, looking at a heap from a Master tha was > running ~650 servers and > 300k regions, I tripped over some silly items in > the heap: > 1. Balancer has a regions x server matrix which takes up 18% of the Master > heap according to jxray and 40% according to eclipse. Looks like the matrix > should be regions x racks which would be much smaller (Issue came in with > HBASE-18164 Fast locality computation in balancer -Added new > LocalityCostFunction and LocalityCandidateGenerator ..). See !Screen Shot > 2018-07-19 at 1.20.23 PM.png! !Screen Shot 2018-07-19 at 1.38.56 PM.png! > 2. Duplicate Strings make up ~5% of the Master heap. Of these, ServerName > seems to be the font. Interesting is report that there 54k instances of > ServerName in this heap though there are only 650 Servers. See !Screen Shot > 2018-07-19 at 2.22.42 PM.png! > 3. ArrayDequeue initializes its internal elements array with 16 elements. We > use this in a few places. In Procedures, of which there are many in this > heap, we near never make use of this array. See !Screen Shot 2018-07-19 at > 2.43.42 PM.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20913) List memstore direct memory/heap memory usage
[ https://issues.apache.org/jira/browse/HBASE-20913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549947#comment-16549947 ] Ted Yu commented on HBASE-20913: Nice enhancement. > List memstore direct memory/heap memory usage > - > > Key: HBASE-20913 > URL: https://issues.apache.org/jira/browse/HBASE-20913 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Attachments: Screen Shot 2018-07-13 at 4.44.07 PM.png > > > With offheap write path support, mslab can be allocated at offheap memory. > Currently, RS Server Metrics, only show Memstore Size and it does not list > the DM usage for memstore. It will be good that memstore's offheap memory > usage along with the heap memory usage be shown at the webpage so we will > know how much DM is used for memstore. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20914) Trim Master memory usage
[ https://issues.apache.org/jira/browse/HBASE-20914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20914: -- Attachment: HBASE-20914.branch-2.0.001.patch > Trim Master memory usage > > > Key: HBASE-20914 > URL: https://issues.apache.org/jira/browse/HBASE-20914 > Project: HBase > Issue Type: Sub-task > Components: master >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 3.0.0, 2.0.2, 2.2.0, 2.1.1 > > Attachments: HBASE-20914.branch-2.0.001.patch, Screen Shot 2018-07-19 > at 1.20.23 PM.png, Screen Shot 2018-07-19 at 1.20.23 PM.png, Screen Shot > 2018-07-19 at 1.38.56 PM.png, Screen Shot 2018-07-19 at 1.38.56 PM.png, > Screen Shot 2018-07-19 at 2.22.42 PM.png, Screen Shot 2018-07-19 at 2.22.42 > PM.png, Screen Shot 2018-07-19 at 2.43.42 PM.png, Screen Shot 2018-07-19 at > 2.43.42 PM.png > > > While working on the parent issue, looking at a heap from a Master tha was > running ~650 servers and > 300k regions, I tripped over some silly items in > the heap: > 1. Balancer has a regions x server matrix which takes up 18% of the Master > heap according to jxray and 40% according to eclipse. Looks like the matrix > should be regions x racks which would be much smaller (Issue came in with > HBASE-18164 Fast locality computation in balancer -Added new > LocalityCostFunction and LocalityCandidateGenerator ..). See !Screen Shot > 2018-07-19 at 1.20.23 PM.png! !Screen Shot 2018-07-19 at 1.38.56 PM.png! > 2. Duplicate Strings make up ~5% of the Master heap. Of these, ServerName > seems to be the font. Interesting is report that there 54k instances of > ServerName in this heap though there are only 650 Servers. See !Screen Shot > 2018-07-19 at 2.22.42 PM.png! > 3. ArrayDequeue initializes its internal elements array with 16 elements. We > use this in a few places. In Procedures, of which there are many in this > heap, we near never make use of this array. See !Screen Shot 2018-07-19 at > 2.43.42 PM.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20914) Trim Master memory usage
[ https://issues.apache.org/jira/browse/HBASE-20914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20914: -- Attachment: Screen Shot 2018-07-19 at 2.43.42 PM.png > Trim Master memory usage > > > Key: HBASE-20914 > URL: https://issues.apache.org/jira/browse/HBASE-20914 > Project: HBase > Issue Type: Sub-task > Components: master >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.2 > > Attachments: Screen Shot 2018-07-19 at 1.20.23 PM.png, Screen Shot > 2018-07-19 at 1.20.23 PM.png, Screen Shot 2018-07-19 at 1.38.56 PM.png, > Screen Shot 2018-07-19 at 1.38.56 PM.png, Screen Shot 2018-07-19 at 2.22.42 > PM.png, Screen Shot 2018-07-19 at 2.22.42 PM.png, Screen Shot 2018-07-19 at > 2.43.42 PM.png, Screen Shot 2018-07-19 at 2.43.42 PM.png > > > While working on the parent issue, looking at a heap from a Master tha was > running ~650 servers and > 300k regions, I tripped over some silly items in > the heap: > 1. Balancer has a regions x server matrix which takes up 18% of the Master > heap according to jxray and 40% according to eclipse. Looks like the matrix > should be regions x racks which would be much smaller (Issue came in with > HBASE-18164 Fast locality computation in balancer -Added new > LocalityCostFunction and LocalityCandidateGenerator ..). See !Screen Shot > 2018-07-19 at 1.20.23 PM.png! !Screen Shot 2018-07-19 at 1.38.56 PM.png! > 2. Duplicate Strings make up ~5% of the Master heap. Of these, ServerName > seems to be the font. Interesting is report that there 54k instances of > ServerName in this heap though there are only 650 Servers. See !Screen Shot > 2018-07-19 at 1.20.23 PM.png! > 3. ArrayDequeue initializes its internal elements array with 16 elements. We > use this in a few places. In Procedures, of which there are many in this > heap, we near never make use of this array. See !Screen Shot 2018-07-19 at > 1.20.23 PM.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20914) Trim Master memory usage
[ https://issues.apache.org/jira/browse/HBASE-20914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20914: -- Description: While working on the parent issue, looking at a heap from a Master tha was running ~650 servers and > 300k regions, I tripped over some silly items in the heap: 1. Balancer has a regions x server matrix which takes up 18% of the Master heap according to jxray and 40% according to eclipse. Looks like the matrix should be regions x racks which would be much smaller (Issue came in with HBASE-18164 Fast locality computation in balancer -Added new LocalityCostFunction and LocalityCandidateGenerator ..). See !Screen Shot 2018-07-19 at 1.20.23 PM.png! !Screen Shot 2018-07-19 at 1.38.56 PM.png! 2. Duplicate Strings make up ~5% of the Master heap. Of these, ServerName seems to be the font. Interesting is report that there 54k instances of ServerName in this heap though there are only 650 Servers. See !Screen Shot 2018-07-19 at 2.22.42 PM.png! 3. ArrayDequeue initializes its internal elements array with 16 elements. We use this in a few places. In Procedures, of which there are many in this heap, we near never make use of this array. See !Screen Shot 2018-07-19 at 2.43.42 PM.png! was: While working on the parent issue, looking at a heap from a Master tha was running ~650 servers and > 300k regions, I tripped over some silly items in the heap: 1. Balancer has a regions x server matrix which takes up 18% of the Master heap according to jxray and 40% according to eclipse. Looks like the matrix should be regions x racks which would be much smaller (Issue came in with HBASE-18164 Fast locality computation in balancer -Added new LocalityCostFunction and LocalityCandidateGenerator ..). See !Screen Shot 2018-07-19 at 1.20.23 PM.png! !Screen Shot 2018-07-19 at 1.38.56 PM.png! 2. Duplicate Strings make up ~5% of the Master heap. Of these, ServerName seems to be the font. Interesting is report that there 54k instances of ServerName in this heap though there are only 650 Servers. See !Screen Shot 2018-07-19 at 1.20.23 PM.png! 3. ArrayDequeue initializes its internal elements array with 16 elements. We use this in a few places. In Procedures, of which there are many in this heap, we near never make use of this array. See !Screen Shot 2018-07-19 at 1.20.23 PM.png! > Trim Master memory usage > > > Key: HBASE-20914 > URL: https://issues.apache.org/jira/browse/HBASE-20914 > Project: HBase > Issue Type: Sub-task > Components: master >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.2 > > Attachments: Screen Shot 2018-07-19 at 1.20.23 PM.png, Screen Shot > 2018-07-19 at 1.20.23 PM.png, Screen Shot 2018-07-19 at 1.38.56 PM.png, > Screen Shot 2018-07-19 at 1.38.56 PM.png, Screen Shot 2018-07-19 at 2.22.42 > PM.png, Screen Shot 2018-07-19 at 2.22.42 PM.png, Screen Shot 2018-07-19 at > 2.43.42 PM.png, Screen Shot 2018-07-19 at 2.43.42 PM.png > > > While working on the parent issue, looking at a heap from a Master tha was > running ~650 servers and > 300k regions, I tripped over some silly items in > the heap: > 1. Balancer has a regions x server matrix which takes up 18% of the Master > heap according to jxray and 40% according to eclipse. Looks like the matrix > should be regions x racks which would be much smaller (Issue came in with > HBASE-18164 Fast locality computation in balancer -Added new > LocalityCostFunction and LocalityCandidateGenerator ..). See !Screen Shot > 2018-07-19 at 1.20.23 PM.png! !Screen Shot 2018-07-19 at 1.38.56 PM.png! > 2. Duplicate Strings make up ~5% of the Master heap. Of these, ServerName > seems to be the font. Interesting is report that there 54k instances of > ServerName in this heap though there are only 650 Servers. See !Screen Shot > 2018-07-19 at 2.22.42 PM.png! > 3. ArrayDequeue initializes its internal elements array with 16 elements. We > use this in a few places. In Procedures, of which there are many in this > heap, we near never make use of this array. See !Screen Shot 2018-07-19 at > 2.43.42 PM.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20914) Trim Master memory usage
[ https://issues.apache.org/jira/browse/HBASE-20914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20914: -- Description: While working on the parent issue, looking at a heap from a Master tha was running ~650 servers and > 300k regions, I tripped over some silly items in the heap: 1. Balancer has a regions x server matrix which takes up 18% of the Master heap according to jxray and 40% according to eclipse. Looks like the matrix should be regions x racks which would be much smaller (Issue came in with HBASE-18164 Fast locality computation in balancer -Added new LocalityCostFunction and LocalityCandidateGenerator ..). See !Screen Shot 2018-07-19 at 1.20.23 PM.png! !Screen Shot 2018-07-19 at 1.38.56 PM.png! 2. Duplicate Strings make up ~5% of the Master heap. Of these, ServerName seems to be the font. Interesting is report that there 54k instances of ServerName in this heap though there are only 650 Servers. See !Screen Shot 2018-07-19 at 1.20.23 PM.png! 3. ArrayDequeue initializes its internal elements array with 16 elements. We use this in a few places. In Procedures, of which there are many in this heap, we near never make use of this array. See !Screen Shot 2018-07-19 at 1.20.23 PM.png! was: While working on the parent issue, looking at a heap from a Master tha was running ~650 servers and > 300k regions, I tripped over some silly items in the heap: 1. Balancer has a regions x server matrix which takes up 18% of the Master heap according to jxray and 40% according to eclipse. Looks like the matrix should be regions x racks which would be much smaller (Issue came in with HBASE-18164 Fast locality computation in balancer -Added new LocalityCostFunction and LocalityCandidateGenerator ..). See !Screen Shot 2018-07-19 at 2.22.42 PM.png! and !Screen Shot 2018-07-19 at 2.43.42 PM.png! 2. Duplicate Strings make up ~5% of the Master heap. Of these, ServerName seems to be the font. Interesting is report that there 54k instances of ServerName in this heap though there are only 650 Servers. See !Screen Shot 2018-07-19 at 1.38.56 PM.png! 3. ArrayDequeue initializes its internal elements array with 16 elements. We use this in a few places. In Procedures, of which there are many in this heap, we near never make use of this array. See !Screen Shot 2018-07-19 at 1.20.23 PM.png! > Trim Master memory usage > > > Key: HBASE-20914 > URL: https://issues.apache.org/jira/browse/HBASE-20914 > Project: HBase > Issue Type: Sub-task > Components: master >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.2 > > Attachments: Screen Shot 2018-07-19 at 1.20.23 PM.png, Screen Shot > 2018-07-19 at 1.20.23 PM.png, Screen Shot 2018-07-19 at 1.38.56 PM.png, > Screen Shot 2018-07-19 at 1.38.56 PM.png, Screen Shot 2018-07-19 at 2.22.42 > PM.png, Screen Shot 2018-07-19 at 2.22.42 PM.png, Screen Shot 2018-07-19 at > 2.43.42 PM.png > > > While working on the parent issue, looking at a heap from a Master tha was > running ~650 servers and > 300k regions, I tripped over some silly items in > the heap: > 1. Balancer has a regions x server matrix which takes up 18% of the Master > heap according to jxray and 40% according to eclipse. Looks like the matrix > should be regions x racks which would be much smaller (Issue came in with > HBASE-18164 Fast locality computation in balancer -Added new > LocalityCostFunction and LocalityCandidateGenerator ..). See !Screen Shot > 2018-07-19 at 1.20.23 PM.png! !Screen Shot 2018-07-19 at 1.38.56 PM.png! > 2. Duplicate Strings make up ~5% of the Master heap. Of these, ServerName > seems to be the font. Interesting is report that there 54k instances of > ServerName in this heap though there are only 650 Servers. See !Screen Shot > 2018-07-19 at 1.20.23 PM.png! > 3. ArrayDequeue initializes its internal elements array with 16 elements. We > use this in a few places. In Procedures, of which there are many in this > heap, we near never make use of this array. See !Screen Shot 2018-07-19 at > 1.20.23 PM.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20914) Trim Master memory usage
[ https://issues.apache.org/jira/browse/HBASE-20914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20914: -- Attachment: Screen Shot 2018-07-19 at 2.22.42 PM.png > Trim Master memory usage > > > Key: HBASE-20914 > URL: https://issues.apache.org/jira/browse/HBASE-20914 > Project: HBase > Issue Type: Sub-task > Components: master >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.2 > > Attachments: Screen Shot 2018-07-19 at 1.20.23 PM.png, Screen Shot > 2018-07-19 at 1.20.23 PM.png, Screen Shot 2018-07-19 at 1.38.56 PM.png, > Screen Shot 2018-07-19 at 1.38.56 PM.png, Screen Shot 2018-07-19 at 2.22.42 > PM.png, Screen Shot 2018-07-19 at 2.22.42 PM.png, Screen Shot 2018-07-19 at > 2.43.42 PM.png > > > While working on the parent issue, looking at a heap from a Master tha was > running ~650 servers and > 300k regions, I tripped over some silly items in > the heap: > 1. Balancer has a regions x server matrix which takes up 18% of the Master > heap according to jxray and 40% according to eclipse. Looks like the matrix > should be regions x racks which would be much smaller (Issue came in with > HBASE-18164 Fast locality computation in balancer -Added new > LocalityCostFunction and LocalityCandidateGenerator ..). See !Screen Shot > 2018-07-19 at 1.20.23 PM.png! !Screen Shot 2018-07-19 at 1.38.56 PM.png! > 2. Duplicate Strings make up ~5% of the Master heap. Of these, ServerName > seems to be the font. Interesting is report that there 54k instances of > ServerName in this heap though there are only 650 Servers. See !Screen Shot > 2018-07-19 at 1.20.23 PM.png! > 3. ArrayDequeue initializes its internal elements array with 16 elements. We > use this in a few places. In Procedures, of which there are many in this > heap, we near never make use of this array. See !Screen Shot 2018-07-19 at > 1.20.23 PM.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20914) Trim Master memory usage
[ https://issues.apache.org/jira/browse/HBASE-20914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20914: -- Attachment: Screen Shot 2018-07-19 at 1.20.23 PM.png > Trim Master memory usage > > > Key: HBASE-20914 > URL: https://issues.apache.org/jira/browse/HBASE-20914 > Project: HBase > Issue Type: Sub-task > Components: master >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.2 > > Attachments: Screen Shot 2018-07-19 at 1.20.23 PM.png, Screen Shot > 2018-07-19 at 1.20.23 PM.png, Screen Shot 2018-07-19 at 1.38.56 PM.png, > Screen Shot 2018-07-19 at 1.38.56 PM.png, Screen Shot 2018-07-19 at 2.22.42 > PM.png, Screen Shot 2018-07-19 at 2.43.42 PM.png > > > While working on the parent issue, looking at a heap from a Master tha was > running ~650 servers and > 300k regions, I tripped over some silly items in > the heap: > 1. Balancer has a regions x server matrix which takes up 18% of the Master > heap according to jxray and 40% according to eclipse. Looks like the matrix > should be regions x racks which would be much smaller (Issue came in with > HBASE-18164 Fast locality computation in balancer -Added new > LocalityCostFunction and LocalityCandidateGenerator ..). See !Screen Shot > 2018-07-19 at 2.22.42 PM.png! and !Screen Shot 2018-07-19 at 2.43.42 > PM.png! > 2. Duplicate Strings make up ~5% of the Master heap. Of these, ServerName > seems to be the font. Interesting is report that there 54k instances of > ServerName in this heap though there are only 650 Servers. See !Screen Shot > 2018-07-19 at 1.38.56 PM.png! > 3. ArrayDequeue initializes its internal elements array with 16 elements. We > use this in a few places. In Procedures, of which there are many in this > heap, we near never make use of this array. See !Screen Shot 2018-07-19 at > 1.20.23 PM.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20914) Trim Master memory usage
[ https://issues.apache.org/jira/browse/HBASE-20914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20914: -- Attachment: Screen Shot 2018-07-19 at 1.38.56 PM.png > Trim Master memory usage > > > Key: HBASE-20914 > URL: https://issues.apache.org/jira/browse/HBASE-20914 > Project: HBase > Issue Type: Sub-task > Components: master >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.2 > > Attachments: Screen Shot 2018-07-19 at 1.20.23 PM.png, Screen Shot > 2018-07-19 at 1.20.23 PM.png, Screen Shot 2018-07-19 at 1.38.56 PM.png, > Screen Shot 2018-07-19 at 1.38.56 PM.png, Screen Shot 2018-07-19 at 2.22.42 > PM.png, Screen Shot 2018-07-19 at 2.43.42 PM.png > > > While working on the parent issue, looking at a heap from a Master tha was > running ~650 servers and > 300k regions, I tripped over some silly items in > the heap: > 1. Balancer has a regions x server matrix which takes up 18% of the Master > heap according to jxray and 40% according to eclipse. Looks like the matrix > should be regions x racks which would be much smaller (Issue came in with > HBASE-18164 Fast locality computation in balancer -Added new > LocalityCostFunction and LocalityCandidateGenerator ..). See !Screen Shot > 2018-07-19 at 2.22.42 PM.png! and !Screen Shot 2018-07-19 at 2.43.42 > PM.png! > 2. Duplicate Strings make up ~5% of the Master heap. Of these, ServerName > seems to be the font. Interesting is report that there 54k instances of > ServerName in this heap though there are only 650 Servers. See !Screen Shot > 2018-07-19 at 1.38.56 PM.png! > 3. ArrayDequeue initializes its internal elements array with 16 elements. We > use this in a few places. In Procedures, of which there are many in this > heap, we near never make use of this array. See !Screen Shot 2018-07-19 at > 1.20.23 PM.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20869) Endpoint-based Export use incorrect user to write to destination
[
https://issues.apache.org/jira/browse/HBASE-20869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549938#comment-16549938
]
Hudson commented on HBASE-20869:
Results for branch branch-2.0
[build #568 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/568/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/568//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/568//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/568//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Endpoint-based Export use incorrect user to write to destination
>
>
> Key: HBASE-20869
> URL: https://issues.apache.org/jira/browse/HBASE-20869
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors
>Affects Versions: 2.0.0
> Environment: Hadoop 3.0.0 + HBase 2.0.0, Kerberos.
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Fix For: 3.0.0, 2.0.2, 2.2.0, 2.1.1
>
> Attachments: HBASE-20869.master.001.patch,
> HBASE-20869.master.002.patch
>
>
> HBASE-15806 implemented an endpoint based export. It gets caller's HDFS
> delegation token, and RegionServer is supposed to write out exported files as
> the caller.
> Everything works fine if you use run export as hbase user. However, once you
> use a different user to export, it fails.
> To reproduce,
> Add to configuration key hbase.coprocessor.region.classes the coprocessor
> class org.apache.hadoop.hbase.coprocessor.Export.
> create a table t1, assign permission to a user foo:
>
> {noformat}
> hbase(main):004:0> user_permission 't1'
> User Namespace,Table,Family,Qualifier:Permission
> hbase default,t1,,: [Permission: actions=READ,WRITE,EXEC,CREATE,ADMIN]
> foo default,t1,,: [Permission: actions=READ,WRITE,EXEC,CREATE,ADMIN]{noformat}
>
> As user foo, execute the following command:
>
> {noformat}
> $ hdfs dfs -mkdir /tmp/export_hbase2
> $ hbase org.apache.hadoop.hbase.coprocessor.Export t1 /tmp/export_hbase2/t2/
>
> 18/07/10 14:03:59 INFO client.RpcRetryingCallerImpl: Call exception, tries=6,
> retries=6, started=4457 ms ago, cancelled=false,
> msg=org.apache.hadoop.security.AccessControlException: Permission denied:
> user=hbase, access=WRITE,
> inode="/tmp/export_hbase2/t2":foo:supergroup:drwxr-xr-x
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:400)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:256)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:194)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1846)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1830)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1789)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.resolvePathForStartFile(FSDirWriteFileOp.java:316)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2411)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2343)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:764)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:451)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
> at sun.reflect.GeneratedConstructorAccessor25.newInstance(Unknown
[jira] [Commented] (HBASE-20903) backport HBASE-20792 "info:servername and info:sn inconsistent for OPEN region" to branch-2.0
[
https://issues.apache.org/jira/browse/HBASE-20903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549939#comment-16549939
]
Hudson commented on HBASE-20903:
Results for branch branch-2.0
[build #568 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/568/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/568//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/568//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/568//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> backport HBASE-20792 "info:servername and info:sn inconsistent for OPEN
> region" to branch-2.0
> -
>
> Key: HBASE-20903
> URL: https://issues.apache.org/jira/browse/HBASE-20903
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.1
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Major
> Fix For: 2.0.2
>
> Attachments: HBASE-20903.branch-2.0.001.patch
>
>
> As discussed in HBASE-20864. This is a very serious bug which can cause RS
> being killed or data loss. Should be backported to branch-2.0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20914) Trim Master memory usage
[ https://issues.apache.org/jira/browse/HBASE-20914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20914: -- Description: While working on the parent issue, looking at a heap from a Master tha was running ~650 servers and > 300k regions, I tripped over some silly items in the heap: 1. Balancer has a regions x server matrix which takes up 18% of the Master heap according to jxray and 40% according to eclipse. Looks like the matrix should be regions x racks which would be much smaller (Issue came in with HBASE-18164 Fast locality computation in balancer -Added new LocalityCostFunction and LocalityCandidateGenerator ..). See !Screen Shot 2018-07-19 at 2.22.42 PM.png! and !Screen Shot 2018-07-19 at 2.43.42 PM.png! 2. Duplicate Strings make up ~5% of the Master heap. Of these, ServerName seems to be the font. Interesting is report that there 54k instances of ServerName in this heap though there are only 650 Servers. See !Screen Shot 2018-07-19 at 1.38.56 PM.png! 3. ArrayDequeue initializes its internal elements array with 16 elements. We use this in a few places. In Procedures, of which there are many in this heap, we near never make use of this array. See !Screen Shot 2018-07-19 at 1.20.23 PM.png! was: While working on the parent issue, looking at a heap from a Master tha was running ~650 servers and > 300k regions, I tripped over some silly items in the heap: 1. Balancer has a regions x server matrix which takes up 18% of the Master heap according to jxray and 40% according to eclipse. Looks like the matrix should be regions x racks which would be much smaller (Issue came in with HBASE-18164 Fast locality computation in balancer -Added new LocalityCostFunction and LocalityCandidateGenerator ..). 2. Duplicate Strings make up ~5% of the Master heap. Of these, ServerName seems to be the font. Interesting is report that there 54k instances of ServerName in this heap though there are only 650 Servers. 3. ArrayDequeue initializes its internal elements array with 16 elements. We use this in a few places. In Procedures, of which there are many in this heap, we near never make use of this array. > Trim Master memory usage > > > Key: HBASE-20914 > URL: https://issues.apache.org/jira/browse/HBASE-20914 > Project: HBase > Issue Type: Sub-task > Components: master >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.2 > > Attachments: Screen Shot 2018-07-19 at 1.20.23 PM.png, Screen Shot > 2018-07-19 at 1.38.56 PM.png, Screen Shot 2018-07-19 at 2.22.42 PM.png, > Screen Shot 2018-07-19 at 2.43.42 PM.png > > > While working on the parent issue, looking at a heap from a Master tha was > running ~650 servers and > 300k regions, I tripped over some silly items in > the heap: > 1. Balancer has a regions x server matrix which takes up 18% of the Master > heap according to jxray and 40% according to eclipse. Looks like the matrix > should be regions x racks which would be much smaller (Issue came in with > HBASE-18164 Fast locality computation in balancer -Added new > LocalityCostFunction and LocalityCandidateGenerator ..). See !Screen Shot > 2018-07-19 at 2.22.42 PM.png! and !Screen Shot 2018-07-19 at 2.43.42 > PM.png! > 2. Duplicate Strings make up ~5% of the Master heap. Of these, ServerName > seems to be the font. Interesting is report that there 54k instances of > ServerName in this heap though there are only 650 Servers. See !Screen Shot > 2018-07-19 at 1.38.56 PM.png! > 3. ArrayDequeue initializes its internal elements array with 16 elements. We > use this in a few places. In Procedures, of which there are many in this > heap, we near never make use of this array. See !Screen Shot 2018-07-19 at > 1.20.23 PM.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20792) info:servername and info:sn inconsistent for OPEN region
[
https://issues.apache.org/jira/browse/HBASE-20792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549940#comment-16549940
]
Hudson commented on HBASE-20792:
Results for branch branch-2.0
[build #568 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/568/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/568//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/568//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/568//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> info:servername and info:sn inconsistent for OPEN region
>
>
> Key: HBASE-20792
> URL: https://issues.apache.org/jira/browse/HBASE-20792
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Blocker
> Fix For: 3.0.0, 2.1.0, 2.2.0
>
> Attachments: HBASE-20792.patch, TestRegionMoveAndAbandon.java,
> hbase-hbase-master-ctr-e138-1518143905142-380753-01-04.hwx.site.log
>
>
> Next problem we've run into after HBASE-20752 and HBASE-20708
> After a rolling restart of a cluster, we'll see situations where a collection
> of regions will simply not be assigned out to the RS. I was able to reproduce
> this my mimic the restart patterns our tests do internally (ignore whether
> this is the best way to restart nodes for now :)). The general pattern is
> this:
> {code:java}
> for rs in regionservers:
> stop(server, rs, RS)
> for master in masters:
> stop(server, master, MASTER)
> sleep(15)
> for master in masters:
> start(server, master, MASTER)
> for rs in regionservers:
> start(server, rs, RS){code}
> Looking at meta, we can see why the Master is ignoring some regions:
> {noformat}
> test
> column=table:state, timestamp=1529871718998, value=\x08\x00
> test,,1529871718122.0297f680df6dc0166a44f9536346268e.
> column=info:regioninfo, timestamp=1529967103390, value={ENCODED =>
> 0297f680df6dc0166a44f9536346268e, NAME =>
> 'test,,1529871718122.0297f680df6dc0166a44f9536346268e.', STARTKEY
> => '', ENDKEY =>
> ''}
> test,,1529871718122.0297f680df6dc0166a44f9536346268e.
> column=info:seqnumDuringOpen, timestamp=1529967103390,
> value=\x00\x00\x00\x00\x00\x00\x00*
> test,,1529871718122.0297f680df6dc0166a44f9536346268e.
> column=info:server, timestamp=1529967103390,
> value=ctr-e138-1518143905142-378097-02-12.hwx.site:16020
> test,,1529871718122.0297f680df6dc0166a44f9536346268e.
> column=info:serverstartcode, timestamp=1529967103390, value=1529966776248
> test,,1529871718122.0297f680df6dc0166a44f9536346268e. column=info:sn,
> timestamp=1529967096482,
> value=ctr-e138-1518143905142-378097-02-06.hwx.site,16020,1529966755170
> test,,1529871718122.0297f680df6dc0166a44f9536346268e.
> column=info:state, timestamp=1529967103390, value=OPEN{noformat}
> The region is marked as {{OPEN}}. The master doesn't know any better.
> However, the interesting bit is that {{info:server}} and {{info:sn}} are
> inconsistent (which, according to the javadoc should not be possible for an
> {{OPEN}} region).{{}}
> This doesn't happen every time, but I caught it yesterday on the 2nd or 3rd
> attempt, so I'm hopeful it's not a bear to repro.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20914) Trim Master memory usage
[ https://issues.apache.org/jira/browse/HBASE-20914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20914: -- Attachment: Screen Shot 2018-07-19 at 1.38.56 PM.png > Trim Master memory usage > > > Key: HBASE-20914 > URL: https://issues.apache.org/jira/browse/HBASE-20914 > Project: HBase > Issue Type: Sub-task > Components: master >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.2 > > Attachments: Screen Shot 2018-07-19 at 1.20.23 PM.png, Screen Shot > 2018-07-19 at 1.38.56 PM.png, Screen Shot 2018-07-19 at 2.22.42 PM.png, > Screen Shot 2018-07-19 at 2.43.42 PM.png > > > While working on the parent issue, looking at a heap from a Master tha was > running ~650 servers and > 300k regions, I tripped over some silly items in > the heap: > 1. Balancer has a regions x server matrix which takes up 18% of the Master > heap according to jxray and 40% according to eclipse. Looks like the matrix > should be regions x racks which would be much smaller (Issue came in with > HBASE-18164 Fast locality computation in balancer -Added new > LocalityCostFunction and LocalityCandidateGenerator ..). > 2. Duplicate Strings make up ~5% of the Master heap. Of these, ServerName > seems to be the font. Interesting is report that there 54k instances of > ServerName in this heap though there are only 650 Servers. > 3. ArrayDequeue initializes its internal elements array with 16 elements. We > use this in a few places. In Procedures, of which there are many in this > heap, we near never make use of this array. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20914) Trim Master memory usage
[ https://issues.apache.org/jira/browse/HBASE-20914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20914: -- Attachment: Screen Shot 2018-07-19 at 1.20.23 PM.png > Trim Master memory usage > > > Key: HBASE-20914 > URL: https://issues.apache.org/jira/browse/HBASE-20914 > Project: HBase > Issue Type: Sub-task > Components: master >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.2 > > Attachments: Screen Shot 2018-07-19 at 1.20.23 PM.png, Screen Shot > 2018-07-19 at 1.38.56 PM.png, Screen Shot 2018-07-19 at 2.22.42 PM.png, > Screen Shot 2018-07-19 at 2.43.42 PM.png > > > While working on the parent issue, looking at a heap from a Master tha was > running ~650 servers and > 300k regions, I tripped over some silly items in > the heap: > 1. Balancer has a regions x server matrix which takes up 18% of the Master > heap according to jxray and 40% according to eclipse. Looks like the matrix > should be regions x racks which would be much smaller (Issue came in with > HBASE-18164 Fast locality computation in balancer -Added new > LocalityCostFunction and LocalityCandidateGenerator ..). > 2. Duplicate Strings make up ~5% of the Master heap. Of these, ServerName > seems to be the font. Interesting is report that there 54k instances of > ServerName in this heap though there are only 650 Servers. > 3. ArrayDequeue initializes its internal elements array with 16 elements. We > use this in a few places. In Procedures, of which there are many in this > heap, we near never make use of this array. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20914) Trim Master memory usage
[ https://issues.apache.org/jira/browse/HBASE-20914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20914: -- Attachment: Screen Shot 2018-07-19 at 2.43.42 PM.png > Trim Master memory usage > > > Key: HBASE-20914 > URL: https://issues.apache.org/jira/browse/HBASE-20914 > Project: HBase > Issue Type: Sub-task > Components: master >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.2 > > Attachments: Screen Shot 2018-07-19 at 2.22.42 PM.png, Screen Shot > 2018-07-19 at 2.43.42 PM.png > > > While working on the parent issue, looking at a heap from a Master tha was > running ~650 servers and > 300k regions, I tripped over some silly items in > the heap: > 1. Balancer has a regions x server matrix which takes up 18% of the Master > heap according to jxray and 40% according to eclipse. Looks like the matrix > should be regions x racks which would be much smaller (Issue came in with > HBASE-18164 Fast locality computation in balancer -Added new > LocalityCostFunction and LocalityCandidateGenerator ..). > 2. Duplicate Strings make up ~5% of the Master heap. Of these, ServerName > seems to be the font. Interesting is report that there 54k instances of > ServerName in this heap though there are only 650 Servers. > 3. ArrayDequeue initializes its internal elements array with 16 elements. We > use this in a few places. In Procedures, of which there are many in this > heap, we near never make use of this array. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20914) Trim Master memory usage
[ https://issues.apache.org/jira/browse/HBASE-20914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20914: -- Attachment: Screen Shot 2018-07-19 at 2.22.42 PM.png > Trim Master memory usage > > > Key: HBASE-20914 > URL: https://issues.apache.org/jira/browse/HBASE-20914 > Project: HBase > Issue Type: Sub-task > Components: master >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.2 > > Attachments: Screen Shot 2018-07-19 at 2.22.42 PM.png, Screen Shot > 2018-07-19 at 2.43.42 PM.png > > > While working on the parent issue, looking at a heap from a Master tha was > running ~650 servers and > 300k regions, I tripped over some silly items in > the heap: > 1. Balancer has a regions x server matrix which takes up 18% of the Master > heap according to jxray and 40% according to eclipse. Looks like the matrix > should be regions x racks which would be much smaller (Issue came in with > HBASE-18164 Fast locality computation in balancer -Added new > LocalityCostFunction and LocalityCandidateGenerator ..). > 2. Duplicate Strings make up ~5% of the Master heap. Of these, ServerName > seems to be the font. Interesting is report that there 54k instances of > ServerName in this heap though there are only 650 Servers. > 3. ArrayDequeue initializes its internal elements array with 16 elements. We > use this in a few places. In Procedures, of which there are many in this > heap, we near never make use of this array. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20914) Trim Master memory usage
stack created HBASE-20914: - Summary: Trim Master memory usage Key: HBASE-20914 URL: https://issues.apache.org/jira/browse/HBASE-20914 Project: HBase Issue Type: Sub-task Components: master Reporter: stack Assignee: stack Fix For: 2.0.2 While working on the parent issue, looking at a heap from a Master tha was running ~650 servers and > 300k regions, I tripped over some silly items in the heap: 1. Balancer has a regions x server matrix which takes up 18% of the Master heap according to jxray and 40% according to eclipse. Looks like the matrix should be regions x racks which would be much smaller (Issue came in with HBASE-18164 Fast locality computation in balancer -Added new LocalityCostFunction and LocalityCandidateGenerator ..). 2. Duplicate Strings make up ~5% of the Master heap. Of these, ServerName seems to be the font. Interesting is report that there 54k instances of ServerName in this heap though there are only 650 Servers. 3. ArrayDequeue initializes its internal elements array with 16 elements. We use this in a few places. In Procedures, of which there are many in this heap, we near never make use of this array. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20908) Infinite loop on regionserver if region replica are reduced
[
https://issues.apache.org/jira/browse/HBASE-20908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549932#comment-16549932
]
Ankit Singhal commented on HBASE-20908:
---
Thanks [~yuzhih...@gmail.com], have taken your review comments in the attached
patch.
> Infinite loop on regionserver if region replica are reduced
>
>
> Key: HBASE-20908
> URL: https://issues.apache.org/jira/browse/HBASE-20908
> Project: HBase
> Issue Type: Bug
> Components: read replicas
>Affects Versions: 1.2.0, 2.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HBASE-20908.patch, HBASE-20908_v1.patch,
> HBASE-20908_v3.patch
>
>
> Steps to reproduce
> {code}
> hbase(main):003:0> create 'myTable','cf',{REGION_REPLICATION=>3}
> hbase(main):003:0> put 'myTable','r1','cf:col1','1'
> 0 row(s) in 0.1230 seconds
> hbase(main):004:0> disable 'myTable'
> alter '0 row(s) in 2.3040 seconds
> hbase(main):005:0> alter 'myTable',{REGION_REPLICATION=>1}
> Updating all regions with the new schema...
> 1/1 regions updated.
> Done.
> 0 row(s) in 11.9550 seconds
> hbase(main):006:0> enable 'myTable'
> 0 row(s) in 1.2620 seconds
> hbase(main):007:0> put 'myTable1','r2','cf:col1','1'
> 0 row(s) in 0.0060 seconds
> {code}
> This is the replica region request which will not be present now in Meta but
> was there in cache. Server will say that he is not serving this region.
> {code}
> com.google.protobuf.ServiceException:
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.NotServingRegionException):
> org.apache.hadoop.hbase.NotServingRegionException: Region
> d997d9b47a106216b9b117617ec09015 is not online on
> 10.22.9.76,16020,1531341039091
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3124)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3106)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.replay(RSRpcServices.java:1714)
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22773)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2150)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167)
> {code}
> Eventually, when we will update our cache after looking into meta , we will
> get into an infinite loop as this event will not be replicated because the
> location of the replica will not appear again.
> {code}
> java.net.SocketTimeoutException: callTimeout=120, callDuration=2181316:
> Can't get the location null
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:170)
> at
> org.apache.hadoop.hbase.replication.regionserver.RegionReplicaReplicationEndpoint$RetryingRpcCallable.call(RegionReplicaReplicationEndpoint.java:606)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't
> get the location
> at
> org.apache.hadoop.hbase.client.RegionAdminServiceCallable.getRegionLocations(RegionAdminServiceCallable.java:178)
> at
> org.apache.hadoop.hbase.client.RegionAdminServiceCallable.getLocation(RegionAdminServiceCallable.java:105)
> at
> org.apache.hadoop.hbase.client.RegionAdminServiceCallable.prepare(RegionAdminServiceCallable.java:89)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:134)
> ... 5 more
> Caused by: java.io.IOException: HRegionInfo was null in myTable,
> row=keyvalues={myTable,,1531262022075.f2b68622cfd5851023be29d5599db6c9./info:regioninfo/1531262022425/Put/vlen=41/seqid=0,
>
> myTable,,1531262022075.f2b68622cfd5851023be29d5599db6c9./info:seqnumDuringOpen/1531341209944/Put/vlen=8/seqid=0,
>
> myTable,,1531262022075.f2b68622cfd5851023be29d5599db6c9./info:server/1531341209944/Put/vlen=16/seqid=0,
>
> myTable,,1531262022075.f2b68622cfd5851023be29d5599db6c9./info:serverstartcode/1531341209944/Put/vlen=8/seqid=0}
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1289)
> at
>
[jira] [Updated] (HBASE-20908) Infinite loop on regionserver if region replica are reduced
[
https://issues.apache.org/jira/browse/HBASE-20908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ankit Singhal updated HBASE-20908:
--
Attachment: HBASE-20908_v3.patch
> Infinite loop on regionserver if region replica are reduced
>
>
> Key: HBASE-20908
> URL: https://issues.apache.org/jira/browse/HBASE-20908
> Project: HBase
> Issue Type: Bug
> Components: read replicas
>Affects Versions: 1.2.0, 2.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HBASE-20908.patch, HBASE-20908_v1.patch,
> HBASE-20908_v3.patch
>
>
> Steps to reproduce
> {code}
> hbase(main):003:0> create 'myTable','cf',{REGION_REPLICATION=>3}
> hbase(main):003:0> put 'myTable','r1','cf:col1','1'
> 0 row(s) in 0.1230 seconds
> hbase(main):004:0> disable 'myTable'
> alter '0 row(s) in 2.3040 seconds
> hbase(main):005:0> alter 'myTable',{REGION_REPLICATION=>1}
> Updating all regions with the new schema...
> 1/1 regions updated.
> Done.
> 0 row(s) in 11.9550 seconds
> hbase(main):006:0> enable 'myTable'
> 0 row(s) in 1.2620 seconds
> hbase(main):007:0> put 'myTable1','r2','cf:col1','1'
> 0 row(s) in 0.0060 seconds
> {code}
> This is the replica region request which will not be present now in Meta but
> was there in cache. Server will say that he is not serving this region.
> {code}
> com.google.protobuf.ServiceException:
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.NotServingRegionException):
> org.apache.hadoop.hbase.NotServingRegionException: Region
> d997d9b47a106216b9b117617ec09015 is not online on
> 10.22.9.76,16020,1531341039091
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3124)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3106)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.replay(RSRpcServices.java:1714)
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22773)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2150)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167)
> {code}
> Eventually, when we will update our cache after looking into meta , we will
> get into an infinite loop as this event will not be replicated because the
> location of the replica will not appear again.
> {code}
> java.net.SocketTimeoutException: callTimeout=120, callDuration=2181316:
> Can't get the location null
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:170)
> at
> org.apache.hadoop.hbase.replication.regionserver.RegionReplicaReplicationEndpoint$RetryingRpcCallable.call(RegionReplicaReplicationEndpoint.java:606)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't
> get the location
> at
> org.apache.hadoop.hbase.client.RegionAdminServiceCallable.getRegionLocations(RegionAdminServiceCallable.java:178)
> at
> org.apache.hadoop.hbase.client.RegionAdminServiceCallable.getLocation(RegionAdminServiceCallable.java:105)
> at
> org.apache.hadoop.hbase.client.RegionAdminServiceCallable.prepare(RegionAdminServiceCallable.java:89)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:134)
> ... 5 more
> Caused by: java.io.IOException: HRegionInfo was null in myTable,
> row=keyvalues={myTable,,1531262022075.f2b68622cfd5851023be29d5599db6c9./info:regioninfo/1531262022425/Put/vlen=41/seqid=0,
>
> myTable,,1531262022075.f2b68622cfd5851023be29d5599db6c9./info:seqnumDuringOpen/1531341209944/Put/vlen=8/seqid=0,
>
> myTable,,1531262022075.f2b68622cfd5851023be29d5599db6c9./info:server/1531341209944/Put/vlen=16/seqid=0,
>
> myTable,,1531262022075.f2b68622cfd5851023be29d5599db6c9./info:serverstartcode/1531341209944/Put/vlen=8/seqid=0}
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1289)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1179)
> at
>
[jira] [Commented] (HBASE-20908) Infinite loop on regionserver if region replica are reduced
[
https://issues.apache.org/jira/browse/HBASE-20908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549922#comment-16549922
]
Ted Yu commented on HBASE-20908:
{code}
-private boolean requiresReplication(final TableName tableName, final
List entries)
+public boolean requiresReplication(final TableName tableName, final
List entries)
{code}
The method can be private, right ?
I don't see it referenced in additional places.
This line may need some additional comment :
{code}
+ && tableDescriptor.getRegionReplication() <= (replicaId +
1)) {
{code}
Looking above a few lines:
{code}
if (!RegionReplicaUtil.isDefaultReplica(replicaId)) {
{code}
Primary replica isn't added to {{tasks}}, hence the {{+1}} in the if condition.
{code}
-import org.apache.hbase.thirdparty.com.google.common.collect.Lists;
{code}
The above doesn't seem right - we should be referencing shaded artifact.
> Infinite loop on regionserver if region replica are reduced
>
>
> Key: HBASE-20908
> URL: https://issues.apache.org/jira/browse/HBASE-20908
> Project: HBase
> Issue Type: Bug
> Components: read replicas
>Affects Versions: 1.2.0, 2.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HBASE-20908.patch, HBASE-20908_v1.patch
>
>
> Steps to reproduce
> {code}
> hbase(main):003:0> create 'myTable','cf',{REGION_REPLICATION=>3}
> hbase(main):003:0> put 'myTable','r1','cf:col1','1'
> 0 row(s) in 0.1230 seconds
> hbase(main):004:0> disable 'myTable'
> alter '0 row(s) in 2.3040 seconds
> hbase(main):005:0> alter 'myTable',{REGION_REPLICATION=>1}
> Updating all regions with the new schema...
> 1/1 regions updated.
> Done.
> 0 row(s) in 11.9550 seconds
> hbase(main):006:0> enable 'myTable'
> 0 row(s) in 1.2620 seconds
> hbase(main):007:0> put 'myTable1','r2','cf:col1','1'
> 0 row(s) in 0.0060 seconds
> {code}
> This is the replica region request which will not be present now in Meta but
> was there in cache. Server will say that he is not serving this region.
> {code}
> com.google.protobuf.ServiceException:
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.NotServingRegionException):
> org.apache.hadoop.hbase.NotServingRegionException: Region
> d997d9b47a106216b9b117617ec09015 is not online on
> 10.22.9.76,16020,1531341039091
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3124)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3106)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.replay(RSRpcServices.java:1714)
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22773)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2150)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167)
> {code}
> Eventually, when we will update our cache after looking into meta , we will
> get into an infinite loop as this event will not be replicated because the
> location of the replica will not appear again.
> {code}
> java.net.SocketTimeoutException: callTimeout=120, callDuration=2181316:
> Can't get the location null
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:170)
> at
> org.apache.hadoop.hbase.replication.regionserver.RegionReplicaReplicationEndpoint$RetryingRpcCallable.call(RegionReplicaReplicationEndpoint.java:606)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't
> get the location
> at
> org.apache.hadoop.hbase.client.RegionAdminServiceCallable.getRegionLocations(RegionAdminServiceCallable.java:178)
> at
> org.apache.hadoop.hbase.client.RegionAdminServiceCallable.getLocation(RegionAdminServiceCallable.java:105)
> at
> org.apache.hadoop.hbase.client.RegionAdminServiceCallable.prepare(RegionAdminServiceCallable.java:89)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:134)
> ... 5 more
> Caused by: java.io.IOException: HRegionInfo was null in myTable,
>
[jira] [Updated] (HBASE-20913) List memstore direct memory/heap memory usage
[ https://issues.apache.org/jira/browse/HBASE-20913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huaxiang sun updated HBASE-20913: - Attachment: Screen Shot 2018-07-13 at 4.44.07 PM.png > List memstore direct memory/heap memory usage > - > > Key: HBASE-20913 > URL: https://issues.apache.org/jira/browse/HBASE-20913 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Attachments: Screen Shot 2018-07-13 at 4.44.07 PM.png > > > With offheap write path support, mslab can be allocated at offheap memory. > Currently, RS Server Metrics, only show Memstore Size and it does not list > the DM usage for memstore. It will be good that memstore's offheap memory > usage along with the heap memory usage be shown at the webpage so we will > know how much DM is used for memstore. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20913) List memstore direct memory/heap memory usage
huaxiang sun created HBASE-20913: Summary: List memstore direct memory/heap memory usage Key: HBASE-20913 URL: https://issues.apache.org/jira/browse/HBASE-20913 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 2.0.0 Reporter: huaxiang sun Assignee: huaxiang sun With offheap write path support, mslab can be allocated at offheap memory. Currently, RS Server Metrics, only show Memstore Size and it does not list the DM usage for memstore. It will be good that memstore's offheap memory usage along with the heap memory usage be shown at the webpage so we will know how much DM is used for memstore. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-19842) Cell ACLs v2
[ https://issues.apache.org/jira/browse/HBASE-19842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas D'Silva reassigned HBASE-19842: -- Assignee: (was: Thomas D'Silva) > Cell ACLs v2 > > > Key: HBASE-19842 > URL: https://issues.apache.org/jira/browse/HBASE-19842 > Project: HBase > Issue Type: New Feature > Components: security >Reporter: Andrew Purtell >Priority: Major > > Per cell ACLs as currently implemented (HBASE-7662) embed the serialized ACL > in a tag stored with each cell. This was done for performance. This has some > drawbacks, most significantly unnecessary duplication and to grant or revoke > requires a rewrite of every affected cell. We could implement them in a space > efficient (and management efficient) way at the cost of some performance like > so: > First, allow storage of cell level ACLs in the ACL table. Rowkey would be a > generic identifier of some kind that can be distinguished from existing > rowkeys that associate the ACL with a cf, or table, or namespace. Existing > code for cf/table/namespace ACLs should ignore rows that do not conform to > today's keying strategy. > Then provide the option for storing the rowkey of an entry in the ACL table > in the cell ACL tag instead of the complete serialization. Allocate a new > cell tag ID to distinguish v2 ACL references from v1 embedded ACL > serializations. > The advantages would be reduction of unnecessary duplication, and, like ACLs > at other granularities, a GRANT or REVOKE which updates the ACL table will > update access control rules for all affected cells. The disadvantage would be > in order to process the reference to the ACL for each cell with an ACL > reference in a tag we will need to look up the ACL in the ACL table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20910) Fix dev-support/submit-patch.py
[
https://issues.apache.org/jira/browse/HBASE-20910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549882#comment-16549882
]
Mike Drob commented on HBASE-20910:
---
Yea, I ran a few experiments locally and you're right that it's fine for
python2.
+1 on the patch.
> Fix dev-support/submit-patch.py
> ---
>
> Key: HBASE-20910
> URL: https://issues.apache.org/jira/browse/HBASE-20910
> Project: HBase
> Issue Type: Bug
> Components: scripts
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HBASE-20910.patch
>
>
> {code:python}
> hbase$ python3.6 dev-support/submit-patch.py
> INFO:submit-patch: Active branch: master
> INFO:submit-patch: Using tracking branch as base branch
> INFO:submit-patch: Base branch: origin/master
> INFO:submit-patch: Patch directory: /Users/asinghal/patches
> INFO:submit-patch: Patch name: master.patch
> Traceback (most recent call last):
> File "dev-support/submit-patch.py", line 253, in
> f.write(diff.encode('utf8'))
> TypeError: write() argument must be str, not bytes
> {code}
> [~te...@apache.org], FYI
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20910) Fix dev-support/submit-patch.py
[
https://issues.apache.org/jira/browse/HBASE-20910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549875#comment-16549875
]
Ankit Singhal commented on HBASE-20910:
---
bq. I think this will break python2?
bq. In python2, git.format_patch returns an object of type unicode, and encode
returns a str
[~mdrob], Python2 should be fine because it doesn't distinguish between str and
binary.
{code}
10082:hbase asinghal$ python2.7 dev-support/submit-patch.py
INFO:submit-patch: Active branch: master
INFO:submit-patch: Using tracking branch as base branch
INFO:submit-patch: Base branch: origin/master
INFO:submit-patch: Patch directory: /Users/asinghal/patches
INFO:submit-patch: Patch name: master.patch
{code}
> Fix dev-support/submit-patch.py
> ---
>
> Key: HBASE-20910
> URL: https://issues.apache.org/jira/browse/HBASE-20910
> Project: HBase
> Issue Type: Bug
> Components: scripts
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HBASE-20910.patch
>
>
> {code:python}
> hbase$ python3.6 dev-support/submit-patch.py
> INFO:submit-patch: Active branch: master
> INFO:submit-patch: Using tracking branch as base branch
> INFO:submit-patch: Base branch: origin/master
> INFO:submit-patch: Patch directory: /Users/asinghal/patches
> INFO:submit-patch: Patch name: master.patch
> Traceback (most recent call last):
> File "dev-support/submit-patch.py", line 253, in
> f.write(diff.encode('utf8'))
> TypeError: write() argument must be str, not bytes
> {code}
> [~te...@apache.org], FYI
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20401) Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable
[ https://issues.apache.org/jira/browse/HBASE-20401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak Lon (Stephen) Wu updated HBASE-20401: - Release Note: Add thread-level timeout and check interval configurations in LogCleaner and HFileCleaner, such that one can tune the file cleaners to match their environment. 1. hbase.oldwals.cleaner.thread.timeout.msec 2. hbase.oldwals.cleaner.thread.check.interval.msec 3. hbase.regionserver.hfilecleaner.thread.timeout.msec 4. hbase.regionserver.hfilecleaner.thread.check.interval.msec For example in the LogCleaner scenario, one might consider tuning hbase.oldwals.cleaner.thread.timeout.msec and hbase.oldwals.cleaner.thread.check.interval.msec 1. fs.delete never complete (strange but possible), then we need to wait for a max of 60 seconds. here, 60 seconds might be too long. or the opposite way is to increase more than 60 seconds in the use cases of slow oldwals delete. 2. getResult is waiting in the period of 500 milliseconds, but the fs.delete has completed and setFromCleaner is set but yet notify(). one might want to tune this 500 milliseconds to 200 or less . was: Add thread-level timeout and check interval configurations in LogCleaner and HFileCleaner, such that one can tune the file cleaners to match their environment. 1. hbase.oldwals.cleaner.thread.timeout.msec 2. hbase.oldwals.cleaner.thread.check.interval.msec 3. hbase.regionserver.hfilecleaner.thread.timeout.msec 4. hbase.regionserver.hfilecleaner.thread.check.interval.msec For example in the LogCleaner scenario, one might consider tuning hbase.oldwals.cleaner.thread.timeout.msec and hbase.oldwals.cleaner.thread.check.interval.msec 1. fs.delete never complete (strange but possible), then we need to wait for a max of 60 seconds. here, 60 seconds might be too long. or the opposite way is to increase more than 60 seconds in the use cases of slow oldwals delete. 2. getResult is waiting in the period of 500 milliseconds, but the fs.delete has completed and setFromClear is set but yet notify(). one might want to tune this 500 milliseconds to 200 or less . > Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable > -- > > Key: HBASE-20401 > URL: https://issues.apache.org/jira/browse/HBASE-20401 > Project: HBase > Issue Type: Improvement > Components: master >Affects Versions: 3.0.0, 1.5.0, 2.0.0-beta-1, 1.4.4, 2.0.0 >Reporter: Tak Lon (Stephen) Wu >Assignee: Tak Lon (Stephen) Wu >Priority: Minor > Labels: beginner > Attachments: HBASE-20401.branch-1.001.patch, > HBASE-20401.master.001.patch, HBASE-20401.master.002.patch, > HBASE-20401.master.003.patch, HBASE-20401.master.004.patch, > HBASE-20401.master.005.patch > > > When backporting HBASE-18309 in HBASE-20352, the deleteFiles calls > CleanerContext.java#getResult with a waitIfNotFinished timeout to wait for > notification (notify) from the fs.delete file thread. there might be two > situation need to tune the MAX_WAIT in CleanerContext or waitIfNotFinished > when LogClearner call getResult. > # fs.delete never complete (strange but possible), then we need to wait for > a max of 60 seconds. here, 60 seconds might be too long > # getResult is waiting in the period of 500 milliseconds, but the fs.delete > has completed and setFromClear is set but yet notify(). one might want to > tune this 500 milliseconds to 200 or less . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20401) Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable
[ https://issues.apache.org/jira/browse/HBASE-20401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549840#comment-16549840 ] Tak Lon (Stephen) Wu commented on HBASE-20401: -- [~yuzhih...@gmail.com] I have attached the branch-1 patch, do I also need to attach branch-2 patch (it's a clean cherry-pick) ? > Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable > -- > > Key: HBASE-20401 > URL: https://issues.apache.org/jira/browse/HBASE-20401 > Project: HBase > Issue Type: Improvement > Components: master >Affects Versions: 3.0.0, 1.5.0, 2.0.0-beta-1, 1.4.4, 2.0.0 >Reporter: Tak Lon (Stephen) Wu >Assignee: Tak Lon (Stephen) Wu >Priority: Minor > Labels: beginner > Attachments: HBASE-20401.branch-1.001.patch, > HBASE-20401.master.001.patch, HBASE-20401.master.002.patch, > HBASE-20401.master.003.patch, HBASE-20401.master.004.patch, > HBASE-20401.master.005.patch > > > When backporting HBASE-18309 in HBASE-20352, the deleteFiles calls > CleanerContext.java#getResult with a waitIfNotFinished timeout to wait for > notification (notify) from the fs.delete file thread. there might be two > situation need to tune the MAX_WAIT in CleanerContext or waitIfNotFinished > when LogClearner call getResult. > # fs.delete never complete (strange but possible), then we need to wait for > a max of 60 seconds. here, 60 seconds might be too long > # getResult is waiting in the period of 500 milliseconds, but the fs.delete > has completed and setFromClear is set but yet notify(). one might want to > tune this 500 milliseconds to 200 or less . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20401) Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable
[ https://issues.apache.org/jira/browse/HBASE-20401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549835#comment-16549835 ] Tak Lon (Stephen) Wu commented on HBASE-20401: -- [~mdrob] I modified the RN to include the use cases in LogCleaner. > Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable > -- > > Key: HBASE-20401 > URL: https://issues.apache.org/jira/browse/HBASE-20401 > Project: HBase > Issue Type: Improvement > Components: master >Affects Versions: 3.0.0, 1.5.0, 2.0.0-beta-1, 1.4.4, 2.0.0 >Reporter: Tak Lon (Stephen) Wu >Assignee: Tak Lon (Stephen) Wu >Priority: Minor > Labels: beginner > Attachments: HBASE-20401.branch-1.001.patch, > HBASE-20401.master.001.patch, HBASE-20401.master.002.patch, > HBASE-20401.master.003.patch, HBASE-20401.master.004.patch, > HBASE-20401.master.005.patch > > > When backporting HBASE-18309 in HBASE-20352, the deleteFiles calls > CleanerContext.java#getResult with a waitIfNotFinished timeout to wait for > notification (notify) from the fs.delete file thread. there might be two > situation need to tune the MAX_WAIT in CleanerContext or waitIfNotFinished > when LogClearner call getResult. > # fs.delete never complete (strange but possible), then we need to wait for > a max of 60 seconds. here, 60 seconds might be too long > # getResult is waiting in the period of 500 milliseconds, but the fs.delete > has completed and setFromClear is set but yet notify(). one might want to > tune this 500 milliseconds to 200 or less . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20401) Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable
[ https://issues.apache.org/jira/browse/HBASE-20401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak Lon (Stephen) Wu updated HBASE-20401: - Release Note: Add thread-level timeout and check interval configurations in LogCleaner and HFileCleaner, such that one can tune the file cleaners to match their environment. 1. hbase.oldwals.cleaner.thread.timeout.msec 2. hbase.oldwals.cleaner.thread.check.interval.msec 3. hbase.regionserver.hfilecleaner.thread.timeout.msec 4. hbase.regionserver.hfilecleaner.thread.check.interval.msec For example in the LogCleaner scenario, one might consider tuning hbase.oldwals.cleaner.thread.timeout.msec and hbase.oldwals.cleaner.thread.check.interval.msec 1. fs.delete never complete (strange but possible), then we need to wait for a max of 60 seconds. here, 60 seconds might be too long. or the opposite way is to increase more than 60 seconds in the use cases of slow oldwals delete. 2. getResult is waiting in the period of 500 milliseconds, but the fs.delete has completed and setFromClear is set but yet notify(). one might want to tune this 500 milliseconds to 200 or less . was: Add thread-level timeout and check interval configurations in LogCleaner and HFileCleaner, such that one can tune the file cleaners to match their environment. 1. hbase.oldwals.cleaner.thread.timeout.msec 1. hbase.oldwals.cleaner.thread.check.interval.msec 1. hbase.regionserver.hfilecleaner.thread.timeout.msec 1. hbase.regionserver.hfilecleaner.thread.check.interval.msec For example in the LogCleaner scenario, one might consider tuning hbase.oldwals.cleaner.thread.timeout.msec and hbase.oldwals.cleaner.thread.check.interval.msec 1. fs.delete never complete (strange but possible), then we need to wait for a max of 60 seconds. here, 60 seconds might be too long. or the opposite way is to increase more than 60 seconds in the use cases of slow oldwals delete. 2. getResult is waiting in the period of 500 milliseconds, but the fs.delete has completed and setFromClear is set but yet notify(). one might want to tune this 500 milliseconds to 200 or less . > Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable > -- > > Key: HBASE-20401 > URL: https://issues.apache.org/jira/browse/HBASE-20401 > Project: HBase > Issue Type: Improvement > Components: master >Affects Versions: 3.0.0, 1.5.0, 2.0.0-beta-1, 1.4.4, 2.0.0 >Reporter: Tak Lon (Stephen) Wu >Assignee: Tak Lon (Stephen) Wu >Priority: Minor > Labels: beginner > Attachments: HBASE-20401.branch-1.001.patch, > HBASE-20401.master.001.patch, HBASE-20401.master.002.patch, > HBASE-20401.master.003.patch, HBASE-20401.master.004.patch, > HBASE-20401.master.005.patch > > > When backporting HBASE-18309 in HBASE-20352, the deleteFiles calls > CleanerContext.java#getResult with a waitIfNotFinished timeout to wait for > notification (notify) from the fs.delete file thread. there might be two > situation need to tune the MAX_WAIT in CleanerContext or waitIfNotFinished > when LogClearner call getResult. > # fs.delete never complete (strange but possible), then we need to wait for > a max of 60 seconds. here, 60 seconds might be too long > # getResult is waiting in the period of 500 milliseconds, but the fs.delete > has completed and setFromClear is set but yet notify(). one might want to > tune this 500 milliseconds to 200 or less . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20401) Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable
[ https://issues.apache.org/jira/browse/HBASE-20401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak Lon (Stephen) Wu updated HBASE-20401: - Release Note: Add thread-level timeout and check interval configurations in LogCleaner and HFileCleaner, such that one can tune the file cleaners to match their environment. 1. hbase.oldwals.cleaner.thread.timeout.msec 1. hbase.oldwals.cleaner.thread.check.interval.msec 1. hbase.regionserver.hfilecleaner.thread.timeout.msec 1. hbase.regionserver.hfilecleaner.thread.check.interval.msec For example in the LogCleaner scenario, one might consider tuning hbase.oldwals.cleaner.thread.timeout.msec and hbase.oldwals.cleaner.thread.check.interval.msec 1. fs.delete never complete (strange but possible), then we need to wait for a max of 60 seconds. here, 60 seconds might be too long. or the opposite way is to increase more than 60 seconds in the use cases of slow oldwals delete. 2. getResult is waiting in the period of 500 milliseconds, but the fs.delete has completed and setFromClear is set but yet notify(). one might want to tune this 500 milliseconds to 200 or less . was: Add thread-level timeout and check interval configurations in LogCleaner and HFileCleaner, such that one can tune the file cleaners to match their environment. 1. hbase.oldwals.cleaner.thread.timeout.msec 2. hbase.oldwals.cleaner.thread.check.interval.msec 3. hbase.regionserver.hfilecleaner.thread.timeout.msec 4. hbase.regionserver.hfilecleaner.thread.check.interval.msec For example in the LogCleaner scenario, one might consider tuning hbase.oldwals.cleaner.thread.timeout.msec and hbase.oldwals.cleaner.thread.check.interval.msec 1. fs.delete never complete (strange but possible), then we need to wait for a max of 60 seconds. here, 60 seconds might be too long. or the opposite way is to increase more than 60 seconds in the use cases of slow oldwals delete. 2. getResult is waiting in the period of 500 milliseconds, but the fs.delete has completed and setFromClear is set but yet notify(). one might want to tune this 500 milliseconds to 200 or less . > Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable > -- > > Key: HBASE-20401 > URL: https://issues.apache.org/jira/browse/HBASE-20401 > Project: HBase > Issue Type: Improvement > Components: master >Affects Versions: 3.0.0, 1.5.0, 2.0.0-beta-1, 1.4.4, 2.0.0 >Reporter: Tak Lon (Stephen) Wu >Assignee: Tak Lon (Stephen) Wu >Priority: Minor > Labels: beginner > Attachments: HBASE-20401.branch-1.001.patch, > HBASE-20401.master.001.patch, HBASE-20401.master.002.patch, > HBASE-20401.master.003.patch, HBASE-20401.master.004.patch, > HBASE-20401.master.005.patch > > > When backporting HBASE-18309 in HBASE-20352, the deleteFiles calls > CleanerContext.java#getResult with a waitIfNotFinished timeout to wait for > notification (notify) from the fs.delete file thread. there might be two > situation need to tune the MAX_WAIT in CleanerContext or waitIfNotFinished > when LogClearner call getResult. > # fs.delete never complete (strange but possible), then we need to wait for > a max of 60 seconds. here, 60 seconds might be too long > # getResult is waiting in the period of 500 milliseconds, but the fs.delete > has completed and setFromClear is set but yet notify(). one might want to > tune this 500 milliseconds to 200 or less . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20401) Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable
[ https://issues.apache.org/jira/browse/HBASE-20401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak Lon (Stephen) Wu updated HBASE-20401: - Release Note: Add thread-level timeout and check interval configurations in LogCleaner and HFileCleaner, such that one can tune the file cleaners to match their environment. 1. hbase.oldwals.cleaner.thread.timeout.msec 2. hbase.oldwals.cleaner.thread.check.interval.msec 3. hbase.regionserver.hfilecleaner.thread.timeout.msec 4. hbase.regionserver.hfilecleaner.thread.check.interval.msec For example in the LogCleaner scenario, one might consider tuning hbase.oldwals.cleaner.thread.timeout.msec and hbase.oldwals.cleaner.thread.check.interval.msec 1. fs.delete never complete (strange but possible), then we need to wait for a max of 60 seconds. here, 60 seconds might be too long. or the opposite way is to increase more than 60 seconds in the use cases of slow oldwals delete. 2. getResult is waiting in the period of 500 milliseconds, but the fs.delete has completed and setFromClear is set but yet notify(). one might want to tune this 500 milliseconds to 200 or less . was: Add thread-level timeout and check interval configurations in LogCleaner and HFileCleaner, such that one can tune the file cleaners to match their environment. 1. hbase.oldwals.cleaner.thread.timeout.msec 2. hbase.oldwals.cleaner.thread.check.interval.msec 3. hbase.regionserver.hfilecleaner.thread.timeout.msec 4. hbase.regionserver.hfilecleaner.thread.check.interval.msec > Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable > -- > > Key: HBASE-20401 > URL: https://issues.apache.org/jira/browse/HBASE-20401 > Project: HBase > Issue Type: Improvement > Components: master >Affects Versions: 3.0.0, 1.5.0, 2.0.0-beta-1, 1.4.4, 2.0.0 >Reporter: Tak Lon (Stephen) Wu >Assignee: Tak Lon (Stephen) Wu >Priority: Minor > Labels: beginner > Attachments: HBASE-20401.branch-1.001.patch, > HBASE-20401.master.001.patch, HBASE-20401.master.002.patch, > HBASE-20401.master.003.patch, HBASE-20401.master.004.patch, > HBASE-20401.master.005.patch > > > When backporting HBASE-18309 in HBASE-20352, the deleteFiles calls > CleanerContext.java#getResult with a waitIfNotFinished timeout to wait for > notification (notify) from the fs.delete file thread. there might be two > situation need to tune the MAX_WAIT in CleanerContext or waitIfNotFinished > when LogClearner call getResult. > # fs.delete never complete (strange but possible), then we need to wait for > a max of 60 seconds. here, 60 seconds might be too long > # getResult is waiting in the period of 500 milliseconds, but the fs.delete > has completed and setFromClear is set but yet notify(). one might want to > tune this 500 milliseconds to 200 or less . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20901) Reducing region replica has no effect
[ https://issues.apache.org/jira/browse/HBASE-20901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-20901: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Thanks for the patch, Ankit. > Reducing region replica has no effect > - > > Key: HBASE-20901 > URL: https://issues.apache.org/jira/browse/HBASE-20901 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Ankit Singhal >Assignee: Ankit Singhal >Priority: Major > Labels: replica > Fix For: 2.2.0 > > Attachments: HBASE-20901.patch, HBASE-20901_v1.patch, > HBASE-20901_v2.patch > > > While reducing the region replica, server name(sn) and state column of the > replica are not getting deleted, resulting in assignment manager to think > that these regions are CLOSED and assign them again. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20401) Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable
[ https://issues.apache.org/jira/browse/HBASE-20401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak Lon (Stephen) Wu updated HBASE-20401: - Attachment: HBASE-20401.branch-1.001.patch > Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable > -- > > Key: HBASE-20401 > URL: https://issues.apache.org/jira/browse/HBASE-20401 > Project: HBase > Issue Type: Improvement > Components: master >Affects Versions: 3.0.0, 1.5.0, 2.0.0-beta-1, 1.4.4, 2.0.0 >Reporter: Tak Lon (Stephen) Wu >Assignee: Tak Lon (Stephen) Wu >Priority: Minor > Labels: beginner > Attachments: HBASE-20401.branch-1.001.patch, > HBASE-20401.master.001.patch, HBASE-20401.master.002.patch, > HBASE-20401.master.003.patch, HBASE-20401.master.004.patch, > HBASE-20401.master.005.patch > > > When backporting HBASE-18309 in HBASE-20352, the deleteFiles calls > CleanerContext.java#getResult with a waitIfNotFinished timeout to wait for > notification (notify) from the fs.delete file thread. there might be two > situation need to tune the MAX_WAIT in CleanerContext or waitIfNotFinished > when LogClearner call getResult. > # fs.delete never complete (strange but possible), then we need to wait for > a max of 60 seconds. here, 60 seconds might be too long > # getResult is waiting in the period of 500 milliseconds, but the fs.delete > has completed and setFromClear is set but yet notify(). one might want to > tune this 500 milliseconds to 200 or less . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20401) Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable
[ https://issues.apache.org/jira/browse/HBASE-20401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak Lon (Stephen) Wu updated HBASE-20401: - Attachment: (was: HBASE-20401.branch-1.001.patch) > Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable > -- > > Key: HBASE-20401 > URL: https://issues.apache.org/jira/browse/HBASE-20401 > Project: HBase > Issue Type: Improvement > Components: master >Affects Versions: 3.0.0, 1.5.0, 2.0.0-beta-1, 1.4.4, 2.0.0 >Reporter: Tak Lon (Stephen) Wu >Assignee: Tak Lon (Stephen) Wu >Priority: Minor > Labels: beginner > Attachments: HBASE-20401.master.001.patch, > HBASE-20401.master.002.patch, HBASE-20401.master.003.patch, > HBASE-20401.master.004.patch, HBASE-20401.master.005.patch > > > When backporting HBASE-18309 in HBASE-20352, the deleteFiles calls > CleanerContext.java#getResult with a waitIfNotFinished timeout to wait for > notification (notify) from the fs.delete file thread. there might be two > situation need to tune the MAX_WAIT in CleanerContext or waitIfNotFinished > when LogClearner call getResult. > # fs.delete never complete (strange but possible), then we need to wait for > a max of 60 seconds. here, 60 seconds might be too long > # getResult is waiting in the period of 500 milliseconds, but the fs.delete > has completed and setFromClear is set but yet notify(). one might want to > tune this 500 milliseconds to 200 or less . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20401) Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable
[ https://issues.apache.org/jira/browse/HBASE-20401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak Lon (Stephen) Wu updated HBASE-20401: - Attachment: (was: HBASE-20401.branch-1.002.patch) > Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable > -- > > Key: HBASE-20401 > URL: https://issues.apache.org/jira/browse/HBASE-20401 > Project: HBase > Issue Type: Improvement > Components: master >Affects Versions: 3.0.0, 1.5.0, 2.0.0-beta-1, 1.4.4, 2.0.0 >Reporter: Tak Lon (Stephen) Wu >Assignee: Tak Lon (Stephen) Wu >Priority: Minor > Labels: beginner > Attachments: HBASE-20401.master.001.patch, > HBASE-20401.master.002.patch, HBASE-20401.master.003.patch, > HBASE-20401.master.004.patch, HBASE-20401.master.005.patch > > > When backporting HBASE-18309 in HBASE-20352, the deleteFiles calls > CleanerContext.java#getResult with a waitIfNotFinished timeout to wait for > notification (notify) from the fs.delete file thread. there might be two > situation need to tune the MAX_WAIT in CleanerContext or waitIfNotFinished > when LogClearner call getResult. > # fs.delete never complete (strange but possible), then we need to wait for > a max of 60 seconds. here, 60 seconds might be too long > # getResult is waiting in the period of 500 milliseconds, but the fs.delete > has completed and setFromClear is set but yet notify(). one might want to > tune this 500 milliseconds to 200 or less . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20901) Reducing region replica has no effect
[ https://issues.apache.org/jira/browse/HBASE-20901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549824#comment-16549824 ] Ankit Singhal commented on HBASE-20901: --- bq. 20901_v2.patch is somehow bigger than patch v1. bq. Is this because of checkstyle fixes ? yes [~yuzhih...@gmail.com] , no code change. > Reducing region replica has no effect > - > > Key: HBASE-20901 > URL: https://issues.apache.org/jira/browse/HBASE-20901 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Ankit Singhal >Assignee: Ankit Singhal >Priority: Major > Labels: replica > Attachments: HBASE-20901.patch, HBASE-20901_v1.patch, > HBASE-20901_v2.patch > > > While reducing the region replica, server name(sn) and state column of the > replica are not getting deleted, resulting in assignment manager to think > that these regions are CLOSED and assign them again. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20401) Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable
[ https://issues.apache.org/jira/browse/HBASE-20401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak Lon (Stephen) Wu updated HBASE-20401: - Attachment: (was: HBASE-20401.branch-2.001.patch) > Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable > -- > > Key: HBASE-20401 > URL: https://issues.apache.org/jira/browse/HBASE-20401 > Project: HBase > Issue Type: Improvement > Components: master >Affects Versions: 3.0.0, 1.5.0, 2.0.0-beta-1, 1.4.4, 2.0.0 >Reporter: Tak Lon (Stephen) Wu >Assignee: Tak Lon (Stephen) Wu >Priority: Minor > Labels: beginner > Attachments: HBASE-20401.master.001.patch, > HBASE-20401.master.002.patch, HBASE-20401.master.003.patch, > HBASE-20401.master.004.patch, HBASE-20401.master.005.patch > > > When backporting HBASE-18309 in HBASE-20352, the deleteFiles calls > CleanerContext.java#getResult with a waitIfNotFinished timeout to wait for > notification (notify) from the fs.delete file thread. there might be two > situation need to tune the MAX_WAIT in CleanerContext or waitIfNotFinished > when LogClearner call getResult. > # fs.delete never complete (strange but possible), then we need to wait for > a max of 60 seconds. here, 60 seconds might be too long > # getResult is waiting in the period of 500 milliseconds, but the fs.delete > has completed and setFromClear is set but yet notify(). one might want to > tune this 500 milliseconds to 200 or less . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20910) Fix dev-support/submit-patch.py
[
https://issues.apache.org/jira/browse/HBASE-20910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549818#comment-16549818
]
Mike Drob commented on HBASE-20910:
---
I think this will break python2?
In python2, git.format_patch returns an object of type unicode, and encode
returns a str
In python3, we get str and then bytes
Will think a bit how to do this correctly
> Fix dev-support/submit-patch.py
> ---
>
> Key: HBASE-20910
> URL: https://issues.apache.org/jira/browse/HBASE-20910
> Project: HBase
> Issue Type: Bug
> Components: scripts
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HBASE-20910.patch
>
>
> {code:python}
> hbase$ python3.6 dev-support/submit-patch.py
> INFO:submit-patch: Active branch: master
> INFO:submit-patch: Using tracking branch as base branch
> INFO:submit-patch: Base branch: origin/master
> INFO:submit-patch: Patch directory: /Users/asinghal/patches
> INFO:submit-patch: Patch name: master.patch
> Traceback (most recent call last):
> File "dev-support/submit-patch.py", line 253, in
> f.write(diff.encode('utf8'))
> TypeError: write() argument must be str, not bytes
> {code}
> [~te...@apache.org], FYI
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
