[GitHub] [hbase] Apache-HBase commented on issue #674: HBASE-23095 Reuse FileStatus in StoreFileInfo
Apache-HBase commented on issue #674: HBASE-23095 Reuse FileStatus in StoreFileInfo URL: https://github.com/apache/hbase/pull/674#issuecomment-537302442 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | :blue_heart: | reexec | 1m 9s | Docker mode activated. | ||| _ Prechecks _ | | :green_heart: | dupname | 0m 0s | No case conflicting files found. | | :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ master Compile Tests _ | | :green_heart: | mvninstall | 5m 54s | master passed | | :green_heart: | compile | 0m 58s | master passed | | :green_heart: | checkstyle | 1m 30s | master passed | | :green_heart: | shadedjars | 5m 4s | branch has no errors when building our shaded downstream artifacts. | | :green_heart: | javadoc | 0m 38s | master passed | | :blue_heart: | spotbugs | 4m 27s | Used deprecated FindBugs config; considering switching to SpotBugs. | | :green_heart: | findbugs | 4m 24s | master passed | ||| _ Patch Compile Tests _ | | :green_heart: | mvninstall | 5m 28s | the patch passed | | :green_heart: | compile | 0m 57s | the patch passed | | :green_heart: | javac | 0m 57s | the patch passed | | :green_heart: | checkstyle | 1m 28s | the patch passed | | :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | :green_heart: | shadedjars | 5m 2s | patch has no errors when building our shaded downstream artifacts. | | :green_heart: | hadoopcheck | 17m 25s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | :green_heart: | javadoc | 0m 34s | the patch passed | | :green_heart: | findbugs | 4m 29s | the patch passed | ||| _ Other Tests _ | | :broken_heart: | unit | 248m 7s | hbase-server in the patch failed. | | :green_heart: | asflicense | 0m 28s | The patch does not generate ASF License warnings. | | | | 310m 30s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hbase.master.TestSplitWALManager | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-674/4/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/674 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 71e4e1221f58 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-674/out/precommit/personality/provided.sh | | git revision | master / 2ebdcbc3b0 | | Default Java | 1.8.0_181 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-674/4/artifact/out/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-674/4/testReport/ | | Max. process+thread count | 5080 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-674/4/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (HBASE-23107) Avoid temp byte array creation when doing cacheDataOnWrite
chenxu created HBASE-23107:
--
Summary: Avoid temp byte array creation when doing cacheDataOnWrite
Key: HBASE-23107
URL: https://issues.apache.org/jira/browse/HBASE-23107
Project: HBase
Issue Type: Improvement
Reporter: chenxu
Assignee: chenxu
code in HFileBlock.Writer.cloneUncompressedBufferWithHeader
{code:java}
ByteBuffer cloneUncompressedBufferWithHeader() {
expectState(State.BLOCK_READY);
byte[] uncompressedBlockBytesWithHeader = baosInMemory.toByteArray();
…
}
{code}
When cacheOnWrite feature enabled, a temp byte array was created in order to
copy block’s data, we can avoid this by use of ByteBuffAllocator. This can
improve GC performance in write heavy scenarios.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HBASE-22936) Close memStoreScanners in StoreScanner#updateReaders else memory leak
[
https://issues.apache.org/jira/browse/HBASE-22936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sandeep Pal reassigned HBASE-22936:
---
Assignee: Sandeep Pal
> Close memStoreScanners in StoreScanner#updateReaders else memory leak
> -
>
> Key: HBASE-22936
> URL: https://issues.apache.org/jira/browse/HBASE-22936
> Project: HBase
> Issue Type: Bug
>Reporter: Michael Stack
>Assignee: Sandeep Pal
>Priority: Major
> Fix For: 2.3.0, 2.1.7, 2.2.2
>
>
> Via [~aoxiang] from over on HBASE-22723
> {code}
> + if (!closeLock.tryLock()) {
> +// no lock acquired.
> +LOG.debug("StoreScanner already has the close lokc. There is no need
> to updateReaders");
> +return;
> + }
> + // lock acquired
> + updateReaders = true;
> + if (this.closing) {
> +LOG.debug("StoreScanner already closing. There is no need to
> updateReaders");
> +return;
> + }
> {code}
> We need to close memStoreScanners in StoreScanner#updateReaders before this
> two return, someone else can take over the task.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HBASE-22936) Close memStoreScanners in StoreScanner#updateReaders else memory leak
[
https://issues.apache.org/jira/browse/HBASE-22936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sandeep Pal reassigned HBASE-22936:
---
Assignee: (was: Sandeep Pal)
> Close memStoreScanners in StoreScanner#updateReaders else memory leak
> -
>
> Key: HBASE-22936
> URL: https://issues.apache.org/jira/browse/HBASE-22936
> Project: HBase
> Issue Type: Bug
>Reporter: Michael Stack
>Priority: Major
> Fix For: 2.3.0, 2.1.7, 2.2.2
>
>
> Via [~aoxiang] from over on HBASE-22723
> {code}
> + if (!closeLock.tryLock()) {
> +// no lock acquired.
> +LOG.debug("StoreScanner already has the close lokc. There is no need
> to updateReaders");
> +return;
> + }
> + // lock acquired
> + updateReaders = true;
> + if (this.closing) {
> +LOG.debug("StoreScanner already closing. There is no need to
> updateReaders");
> +return;
> + }
> {code}
> We need to close memStoreScanners in StoreScanner#updateReaders before this
> two return, someone else can take over the task.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HBASE-22936) Close memStoreScanners in StoreScanner#updateReaders else memory leak
[
https://issues.apache.org/jira/browse/HBASE-22936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sandeep Pal reassigned HBASE-22936:
---
Assignee: Sandeep Pal
> Close memStoreScanners in StoreScanner#updateReaders else memory leak
> -
>
> Key: HBASE-22936
> URL: https://issues.apache.org/jira/browse/HBASE-22936
> Project: HBase
> Issue Type: Bug
>Reporter: Michael Stack
>Assignee: Sandeep Pal
>Priority: Major
> Fix For: 2.3.0, 2.1.7, 2.2.2
>
>
> Via [~aoxiang] from over on HBASE-22723
> {code}
> + if (!closeLock.tryLock()) {
> +// no lock acquired.
> +LOG.debug("StoreScanner already has the close lokc. There is no need
> to updateReaders");
> +return;
> + }
> + // lock acquired
> + updateReaders = true;
> + if (this.closing) {
> +LOG.debug("StoreScanner already closing. There is no need to
> updateReaders");
> +return;
> + }
> {code}
> We need to close memStoreScanners in StoreScanner#updateReaders before this
> two return, someone else can take over the task.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Reopened] (HBASE-23054) Remove synchronization block from MetaTableMetrics and fix LossyCounting algorithm
[ https://issues.apache.org/jira/browse/HBASE-23054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell reopened HBASE-23054: - > Remove synchronization block from MetaTableMetrics and fix LossyCounting > algorithm > -- > > Key: HBASE-23054 > URL: https://issues.apache.org/jira/browse/HBASE-23054 > Project: HBase > Issue Type: Bug >Affects Versions: 2.1.5 >Reporter: Ankit Singhal >Assignee: Ankit Singhal >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.1.7, 2.2.2 > > Attachments: HBASE-23054.master.001.patch, > HBASE-23054.master.002.patch > > > While trying to use LossyCounting for HBASE-15519 , found following bugs in > current implementation > – Remove synchronization block from MetaTableMetrics to avoid congestion at > the code > – Fix license format > – Fix LossyCounting algorithm as per > [http://www.vldb.org/conf/2002/S10P03.pdf > |http://www.vldb.org/conf/2002/S10P03.pdf] > -- Avoid doing sweep on every insert in LossyCounting > – Remove extra redundant data structures from MetaTableMetrics. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23054) Remove synchronization block from MetaTableMetrics and fix LossyCounting algorithm
[ https://issues.apache.org/jira/browse/HBASE-23054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942412#comment-16942412 ] Andrew Kyle Purtell commented on HBASE-23054: - MetaTableMetrics and LossyCounting are in branch-1 too. This should be backported. > Remove synchronization block from MetaTableMetrics and fix LossyCounting > algorithm > -- > > Key: HBASE-23054 > URL: https://issues.apache.org/jira/browse/HBASE-23054 > Project: HBase > Issue Type: Bug >Affects Versions: 2.1.5 >Reporter: Ankit Singhal >Assignee: Ankit Singhal >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.1.7, 2.2.2 > > Attachments: HBASE-23054.master.001.patch, > HBASE-23054.master.002.patch > > > While trying to use LossyCounting for HBASE-15519 , found following bugs in > current implementation > – Remove synchronization block from MetaTableMetrics to avoid congestion at > the code > – Fix license format > – Fix LossyCounting algorithm as per > [http://www.vldb.org/conf/2002/S10P03.pdf > |http://www.vldb.org/conf/2002/S10P03.pdf] > -- Avoid doing sweep on every insert in LossyCounting > – Remove extra redundant data structures from MetaTableMetrics. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23055) Alter hbase:meta
[
https://issues.apache.org/jira/browse/HBASE-23055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942397#comment-16942397
]
Hudson commented on HBASE-23055:
Results for branch branch-2
[build #2308 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2308/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2308//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2308//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2308//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Alter hbase:meta
>
>
> Key: HBASE-23055
> URL: https://issues.apache.org/jira/browse/HBASE-23055
> Project: HBase
> Issue Type: Task
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0
>
>
> hbase:meta is currently hardcoded. Its schema cannot be change.
> This issue is about allowing edits to hbase:meta schema. It will allow our
> being able to set encodings such as the block-with-indexes which will help
> quell CPU usage on host carrying hbase:meta. A dynamic hbase:meta is first
> step on road to being able to split meta.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23100) Make a 3.1.0 hbase-thirdparty release
[ https://issues.apache.org/jira/browse/HBASE-23100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942395#comment-16942395 ] Michael Stack commented on HBASE-23100: --- Put up third RC this morning. > Make a 3.1.0 hbase-thirdparty release > - > > Key: HBASE-23100 > URL: https://issues.apache.org/jira/browse/HBASE-23100 > Project: HBase > Issue Type: Task > Components: hbase-thirdparty >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > > Make a hbase-thirdparty release. 3.0.0 broke out a gson module out of the > miscellaneous where it used live. 3.1.0 is about updating the dependencies in > hbase-thirdparty. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-22988) Backport HBASE-11062 "hbtop" to branch-1
[ https://issues.apache.org/jira/browse/HBASE-22988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942390#comment-16942390 ] Andrew Kyle Purtell commented on HBASE-22988: - Pushed the PR to branch-1. Ant checks passed. Java 7 compile passed. Unit tests passed. Launched all-in-one cluster and ltt in separate windows, hbtop in a third, switch between the modes, drilled down, looked good. > Backport HBASE-11062 "hbtop" to branch-1 > > > Key: HBASE-22988 > URL: https://issues.apache.org/jira/browse/HBASE-22988 > Project: HBase > Issue Type: Sub-task > Components: backport, hbtop >Reporter: Toshihiro Suzuki >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 1.5.0 > > Attachments: HBASE-22988-branch-1.patch > > > Backport parent issue to branch-1. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] apurtell closed pull request #647: HBASE-22988 Backport HBASE-11062 "hbtop" to branch-1
apurtell closed pull request #647: HBASE-22988 Backport HBASE-11062 "hbtop" to branch-1 URL: https://github.com/apache/hbase/pull/647 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] apurtell closed pull request #668: HBASE-22902: At regionserver start there's a request to roll the WAL
apurtell closed pull request #668: HBASE-22902: At regionserver start there's a request to roll the WAL URL: https://github.com/apache/hbase/pull/668 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-22988) Backport HBASE-11062 "hbtop" to branch-1
[ https://issues.apache.org/jira/browse/HBASE-22988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942373#comment-16942373 ] Andrew Kyle Purtell commented on HBASE-22988: - Testing PR#647 locally with Java 7 now. Will merge it if it looks good. (There's something going on with precommit where error-prone is being invoked for Java 7 builds, that won't work.) > Backport HBASE-11062 "hbtop" to branch-1 > > > Key: HBASE-22988 > URL: https://issues.apache.org/jira/browse/HBASE-22988 > Project: HBase > Issue Type: Sub-task > Components: backport, hbtop >Reporter: Toshihiro Suzuki >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 1.5.0 > > Attachments: HBASE-22988-branch-1.patch > > > Backport parent issue to branch-1. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] apurtell commented on issue #647: HBASE-22988 Backport HBASE-11062 "hbtop" to branch-1
apurtell commented on issue #647: HBASE-22988 Backport HBASE-11062 "hbtop" to branch-1 URL: https://github.com/apache/hbase/pull/647#issuecomment-537272742 Error prone should be disabled for Java 7 precommit, not sure what is going on with that. Let me test this locally with Java 7. If good I will merge it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (HBASE-23086) TestShell failing on branch-1 and branch-1.4
[ https://issues.apache.org/jira/browse/HBASE-23086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-23086: Hadoop Flags: Reviewed Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to branch-1 and branch-1.4 Confirmed TestShell passes on each branch. > TestShell failing on branch-1 and branch-1.4 > > > Key: HBASE-23086 > URL: https://issues.apache.org/jira/browse/HBASE-23086 > Project: HBase > Issue Type: Bug > Components: shell >Affects Versions: 1.5.0, 1.4.11 >Reporter: Sean Busbey >Assignee: Viraj Jasani >Priority: Blocker > Fix For: 1.5.0, 1.4.11 > > Attachments: HBASE-23086.branch-1.001.patch, > HBASE-23086.branch-1.001.patch, HBASE-23086.branch-1.002.patch > > > here's a failed run from branch-1.4 > > [https://builds.apache.org/job/HBase-Flaky-Tests/job/branch-1.4/3958/testReport/junit/org.apache.hadoop.hbase.client/TestShell/testRunShellTests/] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-22902) At regionserver start there's a request to roll the WAL
[
https://issues.apache.org/jira/browse/HBASE-22902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-22902:
Fix Version/s: 1.5.0
Hadoop Flags: Reviewed
Resolution: Fixed
Status: Resolved (was: Patch Available)
> At regionserver start there's a request to roll the WAL
> ---
>
> Key: HBASE-22902
> URL: https://issues.apache.org/jira/browse/HBASE-22902
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 3.0.0, 1.5.0, 2.3.0
>Reporter: David Manning
>Assignee: Sandeep Pal
>Priority: Minor
> Fix For: 1.5.0
>
> Attachments: HBASE-22902_branch-1_v1.patch
>
>
> See HBASE-22301 for logic that requests to roll the WAL if regionserver
> encounters a slow write pipeline. In the logs, during regionserver start, I
> see that the WAL is requested to roll once. It's strange that we roll the WAL
> because it wasn't a slow sync. It appears when this code executes, we haven't
> initialized the {{rollOnSyncNs}} variable to use for determining whether it's
> a slow sync. Current pipeline also shows empty in the logs.
> Disclaimer: I'm experiencing this after backporting this to 1.3.x and
> building it there - I haven't attempted in 1.5.x, though I'd expect similar
> results.
> Regionserver logs follow (notice *threshold=0 ms, current pipeline: []*):
> {noformat}
> Tue Aug 20 23:29:50 GMT 2019 Starting regionserver
> ...
> 2019-08-20 23:29:57,824 INFO wal.FSHLog - WAL configuration: blocksize=256
> MB, rollsize=243.20 MB, prefix=[truncated]%2C1566343792434, suffix=,
> logDir=hdfs://[truncated]/hbase/WALs/[truncated],1566343792434,
> archiveDir=hdfs://[truncated]/hbase/oldWALs
> 2019-08-20 23:29:58,104 INFO wal.FSHLog - Slow sync cost: 186 ms, current
> pipeline: []
> 2019-08-20 23:29:58,104 WARN wal.FSHLog - Requesting log roll because we
> exceeded slow sync threshold; time=186 ms, threshold=0 ms, current pipeline:
> []
> 2019-08-20 23:29:58,107 DEBUG regionserver.ReplicationSourceManager - Start
> tracking logs for wal group [truncated]%2C1566343792434 for peer 1
> 2019-08-20 23:29:58,107 INFO wal.FSHLog - New WAL
> /hbase/WALs/[truncated],1566343792434/[truncated]%2C1566343792434.1566343797824
> 2019-08-20 23:29:58,109 DEBUG regionserver.ReplicationSource - Starting up
> worker for wal group [truncated]%2C1566343792434{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] busbey commented on issue #661: HBASE-15519 Add per-user metrics with lossy counting
busbey commented on issue #661: HBASE-15519 Add per-user metrics with lossy counting URL: https://github.com/apache/hbase/pull/661#issuecomment-537239060 now that HBASE-23054 has landed, can you rebase onto the current master branch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Comment Edited] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942317#comment-16942317
]
Francis Christopher Liu edited comment on HBASE-11288 at 10/1/19 9:11 PM:
--
{quote}
I'd like to see old clients keep working though meta has split under them.
Perhaps the serverName in the MetaRegionServer znode proxies requests when the
region asked for is named 'hbase:meta,,1.1588230740' to actual meta table whose
regions could be elsewhere on the cluster. I suppose 'hbase:meta,,1.1588230740'
would have to be the old ROOT too?
{quote}
Oh interesting idea. How about instead of proxying when a requests for meta
hits the RS with the ROOT region, we could throw a RegionMovedException to
redirect the client to the proper meta region RS . On the RS holding the meta
region we translate the 'hbase:meta,,1.1588230740' region info to the
appropriate meta region info if split. This way we can avoid proxying which
seems to make things a bit more complicated?
{quote}
Lets do a one-pager.
{quote}
Sounds good.
was (Author: toffer):
{quote}
I'd like to see old clients keep working though meta has split under them.
Perhaps the serverName in the MetaRegionServer znode proxies requests when the
region asked for is named 'hbase:meta,,1.1588230740' to actual meta table whose
regions could be elsewhere on the cluster. I suppose 'hbase:meta,,1.1588230740'
would have to be the old ROOT too?
{quote}
Oh interesting idea. How about instead of proxying we could throw a
RegionMovedException to redirect to the proper meta region RS when a requests
for meta hits the RS with the ROOT region. On the RS holding the meta region we
translate the 'hbase:meta,,1.1588230740' region info to the appropriate meta
region info if split. This way we can avoid proxying which seems to make things
a bit more complicated?
{quote}
Lets do a one-pager.
{quote}
Sounds good.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Sub-task
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942317#comment-16942317
]
Francis Christopher Liu commented on HBASE-11288:
-
{quote}
I'd like to see old clients keep working though meta has split under them.
Perhaps the serverName in the MetaRegionServer znode proxies requests when the
region asked for is named 'hbase:meta,,1.1588230740' to actual meta table whose
regions could be elsewhere on the cluster. I suppose 'hbase:meta,,1.1588230740'
would have to be the old ROOT too?
{quote}
Oh interesting idea. How about instead of proxying we could throw a
RegionMovedException to the proper meta region destination when a requests for
meta hits the RS with the ROOT region. On the RS holding the meta region we
translate the 'hbase:meta,,1.1588230740' region info to the appropriate meta
region info if split. This way we can avoid proxying which seems to make things
a bit more complicated?
{quote}
Lets do a one-pager.
{quote}
Sounds good.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Sub-task
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942317#comment-16942317
]
Francis Christopher Liu edited comment on HBASE-11288 at 10/1/19 9:10 PM:
--
{quote}
I'd like to see old clients keep working though meta has split under them.
Perhaps the serverName in the MetaRegionServer znode proxies requests when the
region asked for is named 'hbase:meta,,1.1588230740' to actual meta table whose
regions could be elsewhere on the cluster. I suppose 'hbase:meta,,1.1588230740'
would have to be the old ROOT too?
{quote}
Oh interesting idea. How about instead of proxying we could throw a
RegionMovedException to redirect to the proper meta region RS when a requests
for meta hits the RS with the ROOT region. On the RS holding the meta region we
translate the 'hbase:meta,,1.1588230740' region info to the appropriate meta
region info if split. This way we can avoid proxying which seems to make things
a bit more complicated?
{quote}
Lets do a one-pager.
{quote}
Sounds good.
was (Author: toffer):
{quote}
I'd like to see old clients keep working though meta has split under them.
Perhaps the serverName in the MetaRegionServer znode proxies requests when the
region asked for is named 'hbase:meta,,1.1588230740' to actual meta table whose
regions could be elsewhere on the cluster. I suppose 'hbase:meta,,1.1588230740'
would have to be the old ROOT too?
{quote}
Oh interesting idea. How about instead of proxying we could throw a
RegionMovedException to the proper meta region destination when a requests for
meta hits the RS with the ROOT region. On the RS holding the meta region we
translate the 'hbase:meta,,1.1588230740' region info to the appropriate meta
region info if split. This way we can avoid proxying which seems to make things
a bit more complicated?
{quote}
Lets do a one-pager.
{quote}
Sounds good.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Sub-task
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-22817) Use hbase-shaded dependencies in hbase-spark
[
https://issues.apache.org/jira/browse/HBASE-22817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey resolved HBASE-22817.
-
Fix Version/s: connector-1.0.1
Release Note:
The HBase connector for working with Apache Spark now works with the shaded
client artifacts provided by the Apache HBase project and avoids adding
additional third party dependencies to the classpath.
Resolution: Fixed
> Use hbase-shaded dependencies in hbase-spark
>
>
> Key: HBASE-22817
> URL: https://issues.apache.org/jira/browse/HBASE-22817
> Project: HBase
> Issue Type: Improvement
> Components: hbase-connectors
>Affects Versions: connector-1.0.0
>Reporter: Balazs Meszaros
>Assignee: Balazs Meszaros
>Priority: Major
> Fix For: connector-1.0.1
>
>
> We changed the behavior of {{hbase mapredcp}} which returns shaded jars.
> So if we run
> {code:java}
> spark-shell --driver-class-path `hbase mapredcp`:hbase-spark.jar
> {code}
> then it should work fine, it shouldn't throw any exception when accessing
> hbase from spark.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-22874) Define a public interface for Canary and move existing implementation to LimitedPrivate
[ https://issues.apache.org/jira/browse/HBASE-22874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942288#comment-16942288 ] Rushabh Shah commented on HBASE-22874: -- [~zhangduo] [~busbey] Since the patch is in "Ready to Review" state, could I get some reviews to have this in next release ? Thank you ! > Define a public interface for Canary and move existing implementation to > LimitedPrivate > --- > > Key: HBASE-22874 > URL: https://issues.apache.org/jira/browse/HBASE-22874 > Project: HBase > Issue Type: Bug > Components: canary >Reporter: Duo Zhang >Assignee: Rushabh Shah >Priority: Critical > Fix For: 3.0.0, 1.5.0, 2.3.0, 1.3.6, 2.2.2, 1.4.12, 2.1.8 > > > It can be marked as IA.LimitedPrivate(TOOLS). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23106) [HBCK2/hbase-operator-tools] Add a WAL verifier
[ https://issues.apache.org/jira/browse/HBASE-23106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942281#comment-16942281 ] Michael Stack commented on HBASE-23106: --- Perhaps there is one. I couldn't find it (could have looked harder I suppose -- if it exists...need to add to refguide, etc.) > [HBCK2/hbase-operator-tools] Add a WAL verifier > --- > > Key: HBASE-23106 > URL: https://issues.apache.org/jira/browse/HBASE-23106 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Priority: Major > > We had a WALPlayer that loads edits up into hbase cluster but what I wanted > yesterday was a WAL verifier so I could find the bad WAL messing me up. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23106) [HBCK2/hbase-operator-tools] Add a WAL verifier
Michael Stack created HBASE-23106: - Summary: [HBCK2/hbase-operator-tools] Add a WAL verifier Key: HBASE-23106 URL: https://issues.apache.org/jira/browse/HBASE-23106 Project: HBase Issue Type: Sub-task Reporter: Michael Stack We had a WALPlayer that loads edits up into hbase cluster but what I wanted yesterday was a WAL verifier so I could find the bad WAL messing me up. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on issue #671: HBASE-23092 Make the RM tooling in dev-tools/create-release generic
Apache-HBase commented on issue #671: HBASE-23092 Make the RM tooling in dev-tools/create-release generic URL: https://github.com/apache/hbase/pull/671#issuecomment-537176087 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | :blue_heart: | reexec | 0m 33s | Docker mode activated. | ||| _ Prechecks _ | | :green_heart: | dupname | 0m 0s | No case conflicting files found. | | :blue_heart: | shelldocs | 0m 0s | Shelldocs was not available. | | :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | :blue_heart: | mvndep | 0m 36s | Maven dependency ordering for branch | ||| _ Patch Compile Tests _ | | :blue_heart: | mvndep | 0m 11s | Maven dependency ordering for patch | | :broken_heart: | shellcheck | 0m 5s | The patch generated 34 new + 111 unchanged - 59 fixed = 145 total (was 170) | | :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | ||| _ Other Tests _ | | :blue_heart: | asflicense | 0m 0s | ASF License check generated no output? | | | | 2m 23s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-671/8/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/671 | | Optional Tests | dupname asflicense shellcheck shelldocs | | uname | Linux 22999a00f992 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-671/out/precommit/personality/provided.sh | | git revision | master / 2ebdcbc3b0 | | shellcheck | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-671/8/artifact/out/diff-patch-shellcheck.txt | | Max. process+thread count | 52 (vs. ulimit of 1) | | modules | C: U: | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-671/8/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) shellcheck=0.7.0 | | Powered by | Apache Yetus 0.11.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (HBASE-23105) wangle/concurrent to folly/executors
Sudhir Babu Pothineni created HBASE-23105: - Summary: wangle/concurrent to folly/executors Key: HBASE-23105 URL: https://issues.apache.org/jira/browse/HBASE-23105 Project: HBase Issue Type: Sub-task Reporter: Sudhir Babu Pothineni I think wangle/concurrent moved to folly/executors, can somebody update this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on issue #671: HBASE-23092 Make the RM tooling in dev-tools/create-release generic
Apache-HBase commented on issue #671: HBASE-23092 Make the RM tooling in dev-tools/create-release generic URL: https://github.com/apache/hbase/pull/671#issuecomment-537155010 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | :blue_heart: | reexec | 0m 32s | Docker mode activated. | ||| _ Prechecks _ | | :green_heart: | dupname | 0m 1s | No case conflicting files found. | | :blue_heart: | shelldocs | 0m 1s | Shelldocs was not available. | | :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | :blue_heart: | mvndep | 0m 36s | Maven dependency ordering for branch | ||| _ Patch Compile Tests _ | | :blue_heart: | mvndep | 0m 11s | Maven dependency ordering for patch | | :broken_heart: | shellcheck | 0m 5s | The patch generated 38 new + 107 unchanged - 63 fixed = 145 total (was 170) | | :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | ||| _ Other Tests _ | | :blue_heart: | asflicense | 0m 1s | ASF License check generated no output? | | | | 2m 22s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-671/7/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/671 | | Optional Tests | dupname asflicense shellcheck shelldocs | | uname | Linux 5611bb4e8aba 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-671/out/precommit/personality/provided.sh | | git revision | master / 2ebdcbc3b0 | | shellcheck | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-671/7/artifact/out/diff-patch-shellcheck.txt | | Max. process+thread count | 52 (vs. ulimit of 1) | | modules | C: U: | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-671/7/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) shellcheck=0.7.0 | | Powered by | Apache Yetus 0.11.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #671: HBASE-23092 Make the RM tooling in dev-tools/create-release generic
Apache-HBase commented on issue #671: HBASE-23092 Make the RM tooling in dev-tools/create-release generic URL: https://github.com/apache/hbase/pull/671#issuecomment-537133119 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | :blue_heart: | reexec | 0m 35s | Docker mode activated. | ||| _ Prechecks _ | | :green_heart: | dupname | 0m 0s | No case conflicting files found. | | :blue_heart: | shelldocs | 0m 0s | Shelldocs was not available. | | :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | :blue_heart: | mvndep | 0m 37s | Maven dependency ordering for branch | ||| _ Patch Compile Tests _ | | :blue_heart: | mvndep | 0m 10s | Maven dependency ordering for patch | | :broken_heart: | shellcheck | 0m 6s | The patch generated 34 new + 111 unchanged - 59 fixed = 145 total (was 170) | | :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | ||| _ Other Tests _ | | :blue_heart: | asflicense | 0m 0s | ASF License check generated no output? | | | | 2m 25s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-671/6/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/671 | | Optional Tests | dupname asflicense shellcheck shelldocs | | uname | Linux 98f17fe08720 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-671/out/precommit/personality/provided.sh | | git revision | master / 2ebdcbc3b0 | | shellcheck | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-671/6/artifact/out/diff-patch-shellcheck.txt | | Max. process+thread count | 50 (vs. ulimit of 1) | | modules | C: U: | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-671/6/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) shellcheck=0.7.0 | | Powered by | Apache Yetus 0.11.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] saintstack commented on issue #671: HBASE-23092 Make the RM tooling in dev-tools/create-release generic
saintstack commented on issue #671: HBASE-23092 Make the RM tooling in dev-tools/create-release generic URL: https://github.com/apache/hbase/pull/671#issuecomment-537128282 Last push addresses @petersomogyi feedback. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] saintstack commented on issue #671: HBASE-23092 Make the RM tooling in dev-tools/create-release generic
saintstack commented on issue #671: HBASE-23092 Make the RM tooling in dev-tools/create-release generic URL: https://github.com/apache/hbase/pull/671#issuecomment-537127977 Thank you @petersomogyi for review. I was thinking it risky addressing shellcheck complaint w/o doing rull re-run. Thanks for interceding. I copied from hbase-operator-tools but should have done and update first. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #671: HBASE-23092 Make the RM tooling in dev-tools/create-release generic
saintstack commented on a change in pull request #671: HBASE-23092 Make the RM tooling in dev-tools/create-release generic URL: https://github.com/apache/hbase/pull/671#discussion_r330162619 ## File path: dev-support/create-release/do-release-docker.sh ## @@ -112,7 +118,7 @@ done GPG_KEY_FILE="$WORKDIR/gpg.key" fcreate_secure "$GPG_KEY_FILE" -$GPG --passphrase $GPG_PASSPHRASE --export-secret-key --armor "$GPG_KEY" > "$GPG_KEY_FILE" +"$GPG" --passphrase "$GPG_PASSPHRASE" --export-secret-key --armor "$GPG_KEY" > "$GPG_KEY_FILE" Review comment: Thanks @petersomogyi This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (HBASE-23101) Backport HBASE-22380 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-23101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-23101: Fix Version/s: 1.5.0 > Backport HBASE-22380 to branch-1 > > > Key: HBASE-23101 > URL: https://issues.apache.org/jira/browse/HBASE-23101 > Project: HBase > Issue Type: Sub-task >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23086) TestShell failing on branch-1 and branch-1.4
[ https://issues.apache.org/jira/browse/HBASE-23086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942134#comment-16942134 ] Andrew Kyle Purtell commented on HBASE-23086: - Let me try to commit this today. > TestShell failing on branch-1 and branch-1.4 > > > Key: HBASE-23086 > URL: https://issues.apache.org/jira/browse/HBASE-23086 > Project: HBase > Issue Type: Bug > Components: shell >Affects Versions: 1.5.0, 1.4.11 >Reporter: Sean Busbey >Assignee: Viraj Jasani >Priority: Blocker > Fix For: 1.5.0, 1.4.11 > > Attachments: HBASE-23086.branch-1.001.patch, > HBASE-23086.branch-1.001.patch, HBASE-23086.branch-1.002.patch > > > here's a failed run from branch-1.4 > > [https://builds.apache.org/job/HBase-Flaky-Tests/job/branch-1.4/3958/testReport/junit/org.apache.hadoop.hbase.client/TestShell/testRunShellTests/] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-22902) At regionserver start there's a request to roll the WAL
[
https://issues.apache.org/jira/browse/HBASE-22902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942135#comment-16942135
]
Andrew Kyle Purtell commented on HBASE-22902:
-
Let me try to commit this today.
> At regionserver start there's a request to roll the WAL
> ---
>
> Key: HBASE-22902
> URL: https://issues.apache.org/jira/browse/HBASE-22902
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 3.0.0, 1.5.0, 2.3.0
>Reporter: David Manning
>Assignee: Sandeep Pal
>Priority: Minor
> Attachments: HBASE-22902_branch-1_v1.patch
>
>
> See HBASE-22301 for logic that requests to roll the WAL if regionserver
> encounters a slow write pipeline. In the logs, during regionserver start, I
> see that the WAL is requested to roll once. It's strange that we roll the WAL
> because it wasn't a slow sync. It appears when this code executes, we haven't
> initialized the {{rollOnSyncNs}} variable to use for determining whether it's
> a slow sync. Current pipeline also shows empty in the logs.
> Disclaimer: I'm experiencing this after backporting this to 1.3.x and
> building it there - I haven't attempted in 1.5.x, though I'd expect similar
> results.
> Regionserver logs follow (notice *threshold=0 ms, current pipeline: []*):
> {noformat}
> Tue Aug 20 23:29:50 GMT 2019 Starting regionserver
> ...
> 2019-08-20 23:29:57,824 INFO wal.FSHLog - WAL configuration: blocksize=256
> MB, rollsize=243.20 MB, prefix=[truncated]%2C1566343792434, suffix=,
> logDir=hdfs://[truncated]/hbase/WALs/[truncated],1566343792434,
> archiveDir=hdfs://[truncated]/hbase/oldWALs
> 2019-08-20 23:29:58,104 INFO wal.FSHLog - Slow sync cost: 186 ms, current
> pipeline: []
> 2019-08-20 23:29:58,104 WARN wal.FSHLog - Requesting log roll because we
> exceeded slow sync threshold; time=186 ms, threshold=0 ms, current pipeline:
> []
> 2019-08-20 23:29:58,107 DEBUG regionserver.ReplicationSourceManager - Start
> tracking logs for wal group [truncated]%2C1566343792434 for peer 1
> 2019-08-20 23:29:58,107 INFO wal.FSHLog - New WAL
> /hbase/WALs/[truncated],1566343792434/[truncated]%2C1566343792434.1566343797824
> 2019-08-20 23:29:58,109 DEBUG regionserver.ReplicationSource - Starting up
> worker for wal group [truncated]%2C1566343792434{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-22988) Backport HBASE-11062 "hbtop" to branch-1
[ https://issues.apache.org/jira/browse/HBASE-22988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942133#comment-16942133 ] Andrew Kyle Purtell commented on HBASE-22988: - +1 Let me try to commit this today. > Backport HBASE-11062 "hbtop" to branch-1 > > > Key: HBASE-22988 > URL: https://issues.apache.org/jira/browse/HBASE-22988 > Project: HBase > Issue Type: Sub-task > Components: backport, hbtop >Reporter: Toshihiro Suzuki >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 1.5.0 > > Attachments: HBASE-22988-branch-1.patch > > > Backport parent issue to branch-1. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23101) Backport HBASE-22380 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-23101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942131#comment-16942131 ] Andrew Kyle Purtell commented on HBASE-23101: - Made this a blocker for 1.5.0. To release 1.5.0 anyway, first commit a patch that disables bulk load replication > Backport HBASE-22380 to branch-1 > > > Key: HBASE-23101 > URL: https://issues.apache.org/jira/browse/HBASE-23101 > Project: HBase > Issue Type: Sub-task >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Blocker > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23101) Backport HBASE-22380 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-23101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-23101: Priority: Blocker (was: Major) > Backport HBASE-22380 to branch-1 > > > Key: HBASE-23101 > URL: https://issues.apache.org/jira/browse/HBASE-23101 > Project: HBase > Issue Type: Sub-task >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Blocker > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23055) Alter hbase:meta
[ https://issues.apache.org/jira/browse/HBASE-23055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942122#comment-16942122 ] Michael Stack commented on HBASE-23055: --- Ok. Pushed a feature branch named HBASE-23055. The push includes fixes for [~zhangduo] feedback on PR. Outstanding is the survey of table state incidence checks against Master (sub-issue of this). > Alter hbase:meta > > > Key: HBASE-23055 > URL: https://issues.apache.org/jira/browse/HBASE-23055 > Project: HBase > Issue Type: Task >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0 > > > hbase:meta is currently hardcoded. Its schema cannot be change. > This issue is about allowing edits to hbase:meta schema. It will allow our > being able to set encodings such as the block-with-indexes which will help > quell CPU usage on host carrying hbase:meta. A dynamic hbase:meta is first > step on road to being able to split meta. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] saintstack commented on issue #667: HBASE-23055 Alter hbase:meta
saintstack commented on issue #667: HBASE-23055 Alter hbase:meta URL: https://github.com/apache/hbase/pull/667#issuecomment-537118045 Reverted from master and branch-2. Made feature branch HBASE-23055. The first commit on the feature branch includes change that addresses the @Apache9 feedback above. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (HBASE-23103) Survey incidence of table state queries
[ https://issues.apache.org/jira/browse/HBASE-23103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-23103: -- Priority: Blocker (was: Major) > Survey incidence of table state queries > --- > > Key: HBASE-23103 > URL: https://issues.apache.org/jira/browse/HBASE-23103 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Priority: Blocker > Fix For: 3.0.0 > > > Task that comes of parent issue. Parent makes it so we go via Master to > figure state of a table. It is the authority and since the parent issues adds > being able to enable/disable hbase:meta, table state is now in two places -- > in hbase:meta table... and elsewhere for the hbase:meta's state. Rather than > have client go to two locations dependent on which table is being asked > about, parent made it so we went to master. Parent allows that this puts more > load on the Master. [~zhangduo] brings up the valid concern that it might be > too much or that dependent on the Master for state puts Master too much > in-line with read/writes. > This issue is a survey to figure how much load and how much state-in-master > could mess up inline read/writes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-23104) Address review feedback on hbase:meta
[ https://issues.apache.org/jira/browse/HBASE-23104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack resolved HBASE-23104. --- Resolution: Incomplete Feedback addressed in first push to feature branch named for parent issue, HBASE-23055. This issue is not needed any more. > Address review feedback on hbase:meta > - > > Key: HBASE-23104 > URL: https://issues.apache.org/jira/browse/HBASE-23104 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Priority: Major > > Nice review by [~zhangduo]. Let me address in new issue because parent > already commited. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HBASE-23055) Alter hbase:meta
[ https://issues.apache.org/jira/browse/HBASE-23055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack reopened HBASE-23055: --- Let me reopen since reverting from master and branch-2 to make a feature branch while survey is on-going and because new review showed-up. > Alter hbase:meta > > > Key: HBASE-23055 > URL: https://issues.apache.org/jira/browse/HBASE-23055 > Project: HBase > Issue Type: Task >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0 > > > hbase:meta is currently hardcoded. Its schema cannot be change. > This issue is about allowing edits to hbase:meta schema. It will allow our > being able to set encodings such as the block-with-indexes which will help > quell CPU usage on host carrying hbase:meta. A dynamic hbase:meta is first > step on road to being able to split meta. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23055) Alter hbase:meta
[ https://issues.apache.org/jira/browse/HBASE-23055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942114#comment-16942114 ] Michael Stack commented on HBASE-23055: --- bq. I do not think this should go into branch-2 Ok. Let me revert from there. bq. I still suggest that we do this on a feature branch first. Ok. Will do. > Alter hbase:meta > > > Key: HBASE-23055 > URL: https://issues.apache.org/jira/browse/HBASE-23055 > Project: HBase > Issue Type: Task >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0 > > > hbase:meta is currently hardcoded. Its schema cannot be change. > This issue is about allowing edits to hbase:meta schema. It will allow our > being able to set encodings such as the block-with-indexes which will help > quell CPU usage on host carrying hbase:meta. A dynamic hbase:meta is first > step on road to being able to split meta. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23104) Address review feedback on hbase:meta
Michael Stack created HBASE-23104: - Summary: Address review feedback on hbase:meta Key: HBASE-23104 URL: https://issues.apache.org/jira/browse/HBASE-23104 Project: HBase Issue Type: Sub-task Reporter: Michael Stack Nice review by [~zhangduo]. Let me address in new issue because parent already commited. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] saintstack commented on a change in pull request #667: HBASE-23055 Alter hbase:meta
saintstack commented on a change in pull request #667: HBASE-23055 Alter hbase:meta URL: https://github.com/apache/hbase/pull/667#discussion_r330148173 ## File path: hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java ## @@ -157,6 +156,7 @@ import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil; + Review comment: Let me address review in a sub-issue. I want to get a test run in to make sure nothing broke. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #667: HBASE-23055 Alter hbase:meta
saintstack commented on a change in pull request #667: HBASE-23055 Alter
hbase:meta
URL: https://github.com/apache/hbase/pull/667#discussion_r330147431
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
##
@@ -123,30 +119,22 @@ public FSTableDescriptors(final Configuration conf,
final FileSystem fs, final P
* operations; i.e. on remove, we do not do delete in fs.
*/
public FSTableDescriptors(final Configuration conf, final FileSystem fs,
-final Path rootdir, final boolean fsreadonly,
final boolean usecache) throws IOException {
-this(conf, fs, rootdir, fsreadonly, usecache, null);
- }
-
- /**
- * @param fsreadonly True if we are read-only when it comes to filesystem
- * operations; i.e. on remove, we do not do delete in fs.
- * @param metaObserver Used by HMaster. It need to modify the
META_REPLICAS_NUM for meta table descriptor.
- * see HMaster#finishActiveMasterInitialization
- * TODO: This is a workaround. Should remove this ugly
code...
- */
- public FSTableDescriptors(final Configuration conf, final FileSystem fs,
-final Path rootdir, final boolean fsreadonly,
final boolean usecache,
-Function metaObserver) throws IOException {
+ final Path rootdir, final boolean fsreadonly, final boolean usecache)
throws IOException {
this.fs = fs;
this.rootdir = rootdir;
this.fsreadonly = fsreadonly;
this.usecache = usecache;
-this.metaTableDescriptor = metaObserver == null ?
createMetaTableDescriptor(conf)
- : metaObserver.apply(createMetaTableDescriptorBuilder(conf)).build();
+this.configuration = conf;
}
+ /**
+ * Should be private
+ * @deprecated Since 2.3.0. Should be for internal use only. Used by testing.
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #667: HBASE-23055 Alter hbase:meta
saintstack commented on a change in pull request #667: HBASE-23055 Alter hbase:meta URL: https://github.com/apache/hbase/pull/667#discussion_r330147822 ## File path: hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java ## @@ -157,6 +156,7 @@ import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil; + Review comment: Done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #667: HBASE-23055 Alter hbase:meta
saintstack commented on a change in pull request #667: HBASE-23055 Alter
hbase:meta
URL: https://github.com/apache/hbase/pull/667#discussion_r330139114
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/EnableTableProcedure.java
##
@@ -175,40 +166,32 @@ protected Flow executeFromState(final MasterProcedureEnv
env, final EnableTableS
default:
throw new UnsupportedOperationException("unhandled state=" + state);
}
-} catch (IOException e) {
+} catch (IOException | ExecutionException e) {
if (isRollbackSupported(state)) {
-setFailure("master-enable-table", e);
+setFailure("master-enable-table", getCause(e));
} else {
-LOG.warn(
- "Retriable error trying to enable table=" + tableName + " (in
state=" + state + ")", e);
+LOG.warn("Retryable error enabling {}, state={}", tableName, state,
getCause(e));
}
}
return Flow.HAS_MORE_STATE;
}
- private int getNumberOfReplicasFromMeta(Connection connection, int
regionReplicaCount,
- List regionsOfTable) throws IOException {
-Result r = getRegionFromMeta(connection, regionsOfTable);
-int replicasFound = 0;
-for (int i = 1; i < regionReplicaCount; i++) {
- // Since we have already added the entries to the META we will be
getting only that here
- List columnCells =
- r.getColumnCells(HConstants.CATALOG_FAMILY,
MetaTableAccessor.getServerColumn(i));
- if (!columnCells.isEmpty()) {
-replicasFound++;
- }
-}
-return replicasFound;
+ /**
+ * @return If ExecutionException, pull out the cause.
+ */
+ private Throwable getCause(Exception e) {
+return e instanceof ExecutionException?
((ExecutionException)e).getCause(): e;
}
- private Result getRegionFromMeta(Connection connection, List
regionsOfTable)
- throws IOException {
-byte[] metaKeyForRegion =
MetaTableAccessor.getMetaKeyForRegion(regionsOfTable.get(0));
-Get get = new Get(metaKeyForRegion);
-get.addFamily(HConstants.CATALOG_FAMILY);
-Table metaTable = MetaTableAccessor.getMetaHTable(connection);
-Result r = metaTable.get(get);
-return r;
+ /**
+ * @return If hbase;meta table, it goes to the registry implementation which
is what we want.
+ */
+ private int getReplicaCount(AsyncConnection connection, TableName tableName)
+ throws ExecutionException, InterruptedException {
+AsyncTable t = connection.getTable(TableName.META_TABLE_NAME);
+List rls =
+t.getRegionLocator().getRegionLocations(HConstants.EMPTY_START_ROW,
true).get();
Review comment:
Thanks. Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #667: HBASE-23055 Alter hbase:meta
saintstack commented on a change in pull request #667: HBASE-23055 Alter
hbase:meta
URL: https://github.com/apache/hbase/pull/667#discussion_r330138058
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/EnableTableProcedure.java
##
@@ -66,11 +69,12 @@ public EnableTableProcedure(MasterProcedureEnv env,
TableName tableName) {
/**
* Constructor
- * @param env MasterProcedureEnv
+ *
+ * @param env MasterProcedureEnv
* @param tableName the table to operate on
*/
public EnableTableProcedure(MasterProcedureEnv env, TableName tableName,
- ProcedurePrepareLatch syncLatch) {
+ ProcedurePrepareLatch syncLatch) {
Review comment:
Fixed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #667: HBASE-23055 Alter hbase:meta
saintstack commented on a change in pull request #667: HBASE-23055 Alter hbase:meta URL: https://github.com/apache/hbase/pull/667#discussion_r330137595 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java ## @@ -87,61 +98,6 @@ public void setTableState(TableName tableName, TableState.State newState) throws } } - /** - * Set table state to provided but only if table in specified states Caller should lock table on - * write. - * @param tableName table to change state for - * @param newState new state - * @param states states to check against - * @return null if succeed or table state if failed - */ - public TableState setTableStateIfInStates(TableName tableName, TableState.State newState, Review comment: Right. Not used. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (HBASE-23103) Survey incidence of table state queries
[ https://issues.apache.org/jira/browse/HBASE-23103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-23103: -- Fix Version/s: 3.0.0 > Survey incidence of table state queries > --- > > Key: HBASE-23103 > URL: https://issues.apache.org/jira/browse/HBASE-23103 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Priority: Major > Fix For: 3.0.0 > > > Task that comes of parent issue. Parent makes it so we go via Master to > figure state of a table. It is the authority and since the parent issues adds > being able to enable/disable hbase:meta, table state is now in two places -- > in hbase:meta table... and elsewhere for the hbase:meta's state. Rather than > have client go to two locations dependent on which table is being asked > about, parent made it so we went to master. Parent allows that this puts more > load on the Master. [~zhangduo] brings up the valid concern that it might be > too much or that dependent on the Master for state puts Master too much > in-line with read/writes. > This issue is a survey to figure how much load and how much state-in-master > could mess up inline read/writes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23103) Survey incidence of table state queries
Michael Stack created HBASE-23103: - Summary: Survey incidence of table state queries Key: HBASE-23103 URL: https://issues.apache.org/jira/browse/HBASE-23103 Project: HBase Issue Type: Sub-task Reporter: Michael Stack Task that comes of parent issue. Parent makes it so we go via Master to figure state of a table. It is the authority and since the parent issues adds being able to enable/disable hbase:meta, table state is now in two places -- in hbase:meta table... and elsewhere for the hbase:meta's state. Rather than have client go to two locations dependent on which table is being asked about, parent made it so we went to master. Parent allows that this puts more load on the Master. [~zhangduo] brings up the valid concern that it might be too much or that dependent on the Master for state puts Master too much in-line with read/writes. This issue is a survey to figure how much load and how much state-in-master could mess up inline read/writes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] saintstack commented on a change in pull request #667: HBASE-23055 Alter hbase:meta
saintstack commented on a change in pull request #667: HBASE-23055 Alter
hbase:meta
URL: https://github.com/apache/hbase/pull/667#discussion_r330137297
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawAsyncHBaseAdmin.java
##
@@ -666,42 +669,25 @@ public void run(PRESP resp) {
@Override
public CompletableFuture isTableEnabled(TableName tableName) {
-if (TableName.isMetaTableName(tableName)) {
- return CompletableFuture.completedFuture(true);
-}
-CompletableFuture future = new CompletableFuture<>();
-addListener(AsyncMetaTableAccessor.getTableState(metaTable, tableName),
(state, error) -> {
- if (error != null) {
-future.completeExceptionally(error);
-return;
- }
- if (state.isPresent()) {
-future.complete(state.get().inStates(TableState.State.ENABLED));
- } else {
-future.completeExceptionally(new TableNotFoundException(tableName));
- }
-});
-return future;
+return isTableState(tableName, TableState.State.ENABLED);
}
@Override
public CompletableFuture isTableDisabled(TableName tableName) {
-if (TableName.isMetaTableName(tableName)) {
- return CompletableFuture.completedFuture(false);
-}
-CompletableFuture future = new CompletableFuture<>();
-addListener(AsyncMetaTableAccessor.getTableState(metaTable, tableName),
(state, error) -> {
- if (error != null) {
-future.completeExceptionally(error);
-return;
- }
- if (state.isPresent()) {
-future.complete(state.get().inStates(TableState.State.DISABLED));
- } else {
-future.completeExceptionally(new TableNotFoundException(tableName));
- }
-});
-return future;
+return isTableState(tableName, TableState.State.DISABLED);
+ }
+
+ /**
+ * @return Future that calls Master getTableState and compares to
state
+ */
+ private CompletableFuture isTableState(TableName tableName,
TableState.State state) {
+return this. newMasterCaller().
Review comment:
HBASE-23103
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #667: HBASE-23055 Alter hbase:meta
saintstack commented on a change in pull request #667: HBASE-23055 Alter
hbase:meta
URL: https://github.com/apache/hbase/pull/667#discussion_r330134366
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawAsyncHBaseAdmin.java
##
@@ -666,42 +669,25 @@ public void run(PRESP resp) {
@Override
public CompletableFuture isTableEnabled(TableName tableName) {
-if (TableName.isMetaTableName(tableName)) {
- return CompletableFuture.completedFuture(true);
-}
-CompletableFuture future = new CompletableFuture<>();
-addListener(AsyncMetaTableAccessor.getTableState(metaTable, tableName),
(state, error) -> {
- if (error != null) {
-future.completeExceptionally(error);
-return;
- }
- if (state.isPresent()) {
-future.complete(state.get().inStates(TableState.State.ENABLED));
- } else {
-future.completeExceptionally(new TableNotFoundException(tableName));
- }
-});
-return future;
+return isTableState(tableName, TableState.State.ENABLED);
}
@Override
public CompletableFuture isTableDisabled(TableName tableName) {
-if (TableName.isMetaTableName(tableName)) {
- return CompletableFuture.completedFuture(false);
-}
-CompletableFuture future = new CompletableFuture<>();
-addListener(AsyncMetaTableAccessor.getTableState(metaTable, tableName),
(state, error) -> {
- if (error != null) {
-future.completeExceptionally(error);
-return;
- }
- if (state.isPresent()) {
-future.complete(state.get().inStates(TableState.State.DISABLED));
- } else {
-future.completeExceptionally(new TableNotFoundException(tableName));
- }
-});
-return future;
+return isTableState(tableName, TableState.State.DISABLED);
+ }
+
+ /**
+ * @return Future that calls Master getTableState and compares to
state
+ */
+ private CompletableFuture isTableState(TableName tableName,
TableState.State state) {
+return this. newMasterCaller().
Review comment:
Let me file subtask to look at the incidence of state lookups.
I was thinking admin enable/disable are infrequent and ok if they fail
because master is offline. Let me look at how often inline read/write does
table state. Thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #667: HBASE-23055 Alter hbase:meta
saintstack commented on a change in pull request #667: HBASE-23055 Alter
hbase:meta
URL: https://github.com/apache/hbase/pull/667#discussion_r330133645
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java
##
@@ -53,8 +52,20 @@
// TODO: Make this a guava Service
@InterfaceAudience.Private
public class TableStateManager {
-
private static final Logger LOG =
LoggerFactory.getLogger(TableStateManager.class);
+
+ /**
+ * All table state is kept in hbase:meta except that of hbase:meta itself.
+ * hbase:meta state is kept here locally in this in-memory variable. State
+ * for hbase:meta is not persistent. If this process dies, the hbase:meta
+ * state reverts to enabled. State is used so we can edit hbase:meta as we
+ * would any other table by disabling, altering, and then re-enabling. If
this
+ * process dies in the midst of an edit, the table reverts to enabled. Schema
+ * is read from the filesystem. It is changed atomically so if we die midway
+ * through an edit we should be good.
+ */
+ private TableState.State metaTableState = TableState.State.ENABLED;
Review comment:
hbase:meta being offline should be temporary state.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #667: HBASE-23055 Alter hbase:meta
saintstack commented on a change in pull request #667: HBASE-23055 Alter
hbase:meta
URL: https://github.com/apache/hbase/pull/667#discussion_r330131226
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java
##
@@ -1077,7 +1084,7 @@ public static RegionInfo getRegionInfo(final Result r,
byte [] qualifier) {
public static TableState getTableState(Connection conn, TableName tableName)
throws IOException {
if (tableName.equals(TableName.META_TABLE_NAME)) {
- return new TableState(tableName, TableState.State.ENABLED);
+ throw new IllegalAccessError("Go to the Master to find hbase:meta table
state, not here");
Review comment:
Nah. This is helpful figuring all the places where we are asking
MetaTableAccessor for hbase:meta state. It is an error if a client asks MTA for
state of hbase:meta.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #667: HBASE-23055 Alter hbase:meta
saintstack commented on a change in pull request #667: HBASE-23055 Alter hbase:meta URL: https://github.com/apache/hbase/pull/667#discussion_r330130358 ## File path: hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawAsyncHBaseAdmin.java ## @@ -90,6 +90,7 @@ import org.apache.hadoop.hbase.security.access.Permission; import org.apache.hadoop.hbase.security.access.ShadedAccessControlUtil; import org.apache.hadoop.hbase.security.access.UserPermission; + Review comment: Unintentional This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #667: HBASE-23055 Alter hbase:meta
saintstack commented on a change in pull request #667: HBASE-23055 Alter
hbase:meta
URL: https://github.com/apache/hbase/pull/667#discussion_r330129470
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java
##
@@ -53,8 +52,20 @@
// TODO: Make this a guava Service
@InterfaceAudience.Private
public class TableStateManager {
-
private static final Logger LOG =
LoggerFactory.getLogger(TableStateManager.class);
+
+ /**
+ * All table state is kept in hbase:meta except that of hbase:meta itself.
+ * hbase:meta state is kept here locally in this in-memory variable. State
+ * for hbase:meta is not persistent. If this process dies, the hbase:meta
+ * state reverts to enabled. State is used so we can edit hbase:meta as we
+ * would any other table by disabling, altering, and then re-enabling. If
this
+ * process dies in the midst of an edit, the table reverts to enabled. Schema
+ * is read from the filesystem. It is changed atomically so if we die midway
+ * through an edit we should be good.
+ */
+ private TableState.State metaTableState = TableState.State.ENABLED;
Review comment:
If the Master dies during enable/disable, I want it to default to being
online when new Master comes online. Lots of new conditions if hbase:meta is
offline.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #667: HBASE-23055 Alter hbase:meta
saintstack commented on a change in pull request #667: HBASE-23055 Alter hbase:meta URL: https://github.com/apache/hbase/pull/667#discussion_r330127699 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RejectReplicationRequestStateChecker.java ## @@ -1,4 +1,6 @@ -/** +/* + * Copyright The Apache Software Foundation Review comment: Because it is not javadoc. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (HBASE-23055) Alter hbase:meta
[ https://issues.apache.org/jira/browse/HBASE-23055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-23055: -- Fix Version/s: (was: 2.3.0) > Alter hbase:meta > > > Key: HBASE-23055 > URL: https://issues.apache.org/jira/browse/HBASE-23055 > Project: HBase > Issue Type: Task >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0 > > > hbase:meta is currently hardcoded. Its schema cannot be change. > This issue is about allowing edits to hbase:meta schema. It will allow our > being able to set encodings such as the block-with-indexes which will help > quell CPU usage on host carrying hbase:meta. A dynamic hbase:meta is first > step on road to being able to split meta. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-21981) MMaped bucket cache IOEngine does not work with persistence
[ https://issues.apache.org/jira/browse/HBASE-21981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-21981: -- Fix Version/s: (was: 2.1.7) 2.1.8 > MMaped bucket cache IOEngine does not work with persistence > --- > > Key: HBASE-21981 > URL: https://issues.apache.org/jira/browse/HBASE-21981 > Project: HBase > Issue Type: Bug > Components: BucketCache >Affects Versions: 2.1.3 >Reporter: ramkrishna.s.vasudevan >Assignee: Anoop Sam John >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.2.2, 2.1.8 > > Attachments: HBASE-21981.patch, HBASE-21981.patch > > > The MMap based IOEngines does not retrieve the data back if > 'hbase.bucketcache.persistent.path' is enabled. FileIOEngine works fine but > only the FileMMapEngine has this problem. > The reason is that we don't get the byte buffers in the proper order while > reading back from the file in case of persistence. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23082) Backport low-latency snapshot tracking for space quotas to 2.x
[ https://issues.apache.org/jira/browse/HBASE-23082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-23082: -- Fix Version/s: (was: 2.1.7) 2.1.8 > Backport low-latency snapshot tracking for space quotas to 2.x > -- > > Key: HBASE-23082 > URL: https://issues.apache.org/jira/browse/HBASE-23082 > Project: HBase > Issue Type: Improvement > Components: Quotas >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: 2.3.0, 2.2.2, 2.1.8 > > Attachments: HBASE-23082.001.patch, HBASE-23082.002.branch-2.patch, > HBASE-23082.002.patch, HBASE-23082.branch-2.1.002.patch > > > Some rebase'ing at $dayjob found that HBASE-18133 and HBASE-18135 never made > it to any branch-2's. > There's no good reason that I can think of as to why we shouldn't backport > these (will make quotas and snapshots much more accurate). As memory serves > me, they just were landing when we were branching for early 2.0.0 releases. > Let's get backports for HBASE-18133, HBASE-18135, and HBASE-20531 (and > implicitly HBASE-20439 and HBASE-20440) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23097) Set version as 2.1.7 in branch-2.1 in prep for first RC of 2.1.7
[ https://issues.apache.org/jira/browse/HBASE-23097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-23097: -- Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to branch-2.1. > Set version as 2.1.7 in branch-2.1 in prep for first RC of 2.1.7 > > > Key: HBASE-23097 > URL: https://issues.apache.org/jira/browse/HBASE-23097 > Project: HBase > Issue Type: Sub-task > Components: build >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.1.7 > > Attachments: HBASE-23097-branch-2.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23102) Improper Usage of Map putIfAbsent
[
https://issues.apache.org/jira/browse/HBASE-23102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942049#comment-16942049
]
Duo Zhang commented on HBASE-23102:
---
Some of them could be changed to computeIfAbsent but be careful that,
computeIfAbsent is much slower than get and then putIfAbsent if get wins for
most of the time. This is because that get is almost lock free and is much
faster than computeIfAbsent and putIfAbsent.
> Improper Usage of Map putIfAbsent
> -
>
> Key: HBASE-23102
> URL: https://issues.apache.org/jira/browse/HBASE-23102
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.3.0
>Reporter: David Mollitor
>Priority: Minor
> Labels: newbie, noob
>
> When using {{Map#putIfAbsent}}, the argument should not be a {{new}} object.
> Otherwise, if the item is present, the object that was instantiated is
> immediately thrown away. Instead, use {{Map#computeIfAbsent}} so that the
> object is only instantiated if it is needed.
> There exists a good example in the {{Map}} JavaDoc:
> https://docs.oracle.com/javase/8/docs/api/java/util/Map.html#computeIfAbsent-K-java.util.function.Function-
>
> h2. Locations
> https://github.com/apache/hbase/blob/9370347efea5b09e2fa8f4e5d82fa32491e1181b/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaCache.java#L227-L236
> https://github.com/apache/hbase/blob/025ddce868eb06b4072b5152c5ffae5a01e7ae30/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/throttle/StoreHotnessProtector.java#L124-L129
> https://github.com/apache/hbase/blob/1170f28122d9d36e511ba504a5263ec62e11ef6a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java#L555
> https://github.com/apache/hbase/blob/4ca760fe9dd373b8d8a4c48db15e42424920653c/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServer.java#L584-L586
> https://github.com/apache/hbase/blob/4ca760fe9dd373b8d8a4c48db15e42424920653c/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServer.java#L585
> https://github.com/apache/hbase/blob/5b01e613fbbb92e243e99a1d199b4ffbb21ed2d9/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java#L834
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23055) Alter hbase:meta
[ https://issues.apache.org/jira/browse/HBASE-23055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942040#comment-16942040 ] Duo Zhang commented on HBASE-23055: --- Left a few comments on the PR. And I do not think this should go into branch-2, this is really a big change. I still suggest that we do this on a feature branch first. As I said on the PR, after this change, we introduce master to the normal read/write path, this is a bit dangerous, as in the past we are safe to access the cluster even if master is dead for some time. > Alter hbase:meta > > > Key: HBASE-23055 > URL: https://issues.apache.org/jira/browse/HBASE-23055 > Project: HBase > Issue Type: Task >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0, 2.3.0 > > > hbase:meta is currently hardcoded. Its schema cannot be change. > This issue is about allowing edits to hbase:meta schema. It will allow our > being able to set encodings such as the block-with-indexes which will help > quell CPU usage on host carrying hbase:meta. A dynamic hbase:meta is first > step on road to being able to split meta. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache9 commented on a change in pull request #667: HBASE-23055 Alter hbase:meta
Apache9 commented on a change in pull request #667: HBASE-23055 Alter hbase:meta
URL: https://github.com/apache/hbase/pull/667#discussion_r330042034
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java
##
@@ -1077,7 +1084,7 @@ public static RegionInfo getRegionInfo(final Result r,
byte [] qualifier) {
public static TableState getTableState(Connection conn, TableName tableName)
throws IOException {
if (tableName.equals(TableName.META_TABLE_NAME)) {
- return new TableState(tableName, TableState.State.ENABLED);
+ throw new IllegalAccessError("Go to the Master to find hbase:meta table
state, not here");
Review comment:
IllegalArgumentException? An Error seems over kill.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache9 commented on a change in pull request #667: HBASE-23055 Alter hbase:meta
Apache9 commented on a change in pull request #667: HBASE-23055 Alter hbase:meta URL: https://github.com/apache/hbase/pull/667#discussion_r330041404 ## File path: hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawAsyncHBaseAdmin.java ## @@ -90,6 +90,7 @@ import org.apache.hadoop.hbase.security.access.Permission; import org.apache.hadoop.hbase.security.access.ShadedAccessControlUtil; import org.apache.hadoop.hbase.security.access.UserPermission; + Review comment: Why an empty line? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache9 commented on a change in pull request #667: HBASE-23055 Alter hbase:meta
Apache9 commented on a change in pull request #667: HBASE-23055 Alter hbase:meta
URL: https://github.com/apache/hbase/pull/667#discussion_r330096522
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/EnableTableProcedure.java
##
@@ -175,40 +166,32 @@ protected Flow executeFromState(final MasterProcedureEnv
env, final EnableTableS
default:
throw new UnsupportedOperationException("unhandled state=" + state);
}
-} catch (IOException e) {
+} catch (IOException | ExecutionException e) {
if (isRollbackSupported(state)) {
-setFailure("master-enable-table", e);
+setFailure("master-enable-table", getCause(e));
} else {
-LOG.warn(
- "Retriable error trying to enable table=" + tableName + " (in
state=" + state + ")", e);
+LOG.warn("Retryable error enabling {}, state={}", tableName, state,
getCause(e));
}
}
return Flow.HAS_MORE_STATE;
}
- private int getNumberOfReplicasFromMeta(Connection connection, int
regionReplicaCount,
- List regionsOfTable) throws IOException {
-Result r = getRegionFromMeta(connection, regionsOfTable);
-int replicasFound = 0;
-for (int i = 1; i < regionReplicaCount; i++) {
- // Since we have already added the entries to the META we will be
getting only that here
- List columnCells =
- r.getColumnCells(HConstants.CATALOG_FAMILY,
MetaTableAccessor.getServerColumn(i));
- if (!columnCells.isEmpty()) {
-replicasFound++;
- }
-}
-return replicasFound;
+ /**
+ * @return If ExecutionException, pull out the cause.
+ */
+ private Throwable getCause(Exception e) {
+return e instanceof ExecutionException?
((ExecutionException)e).getCause(): e;
}
- private Result getRegionFromMeta(Connection connection, List
regionsOfTable)
- throws IOException {
-byte[] metaKeyForRegion =
MetaTableAccessor.getMetaKeyForRegion(regionsOfTable.get(0));
-Get get = new Get(metaKeyForRegion);
-get.addFamily(HConstants.CATALOG_FAMILY);
-Table metaTable = MetaTableAccessor.getMetaHTable(connection);
-Result r = metaTable.get(get);
-return r;
+ /**
+ * @return If hbase;meta table, it goes to the registry implementation which
is what we want.
+ */
+ private int getReplicaCount(AsyncConnection connection, TableName tableName)
+ throws ExecutionException, InterruptedException {
+AsyncTable t = connection.getTable(TableName.META_TABLE_NAME);
+List rls =
+t.getRegionLocator().getRegionLocations(HConstants.EMPTY_START_ROW,
true).get();
Review comment:
Use FutureUtils.get? It will unwrap the ExecutionException.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache9 commented on a change in pull request #667: HBASE-23055 Alter hbase:meta
Apache9 commented on a change in pull request #667: HBASE-23055 Alter hbase:meta URL: https://github.com/apache/hbase/pull/667#discussion_r330097219 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RejectReplicationRequestStateChecker.java ## @@ -1,4 +1,6 @@ -/** +/* + * Copyright The Apache Software Foundation Review comment: Why this line? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache9 commented on a change in pull request #667: HBASE-23055 Alter hbase:meta
Apache9 commented on a change in pull request #667: HBASE-23055 Alter hbase:meta URL: https://github.com/apache/hbase/pull/667#discussion_r330094572 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java ## @@ -87,61 +98,6 @@ public void setTableState(TableName tableName, TableState.State newState) throws } } - /** - * Set table state to provided but only if table in specified states Caller should lock table on - * write. - * @param tableName table to change state for - * @param newState new state - * @param states states to check against - * @return null if succeed or table state if failed - */ - public TableState setTableStateIfInStates(TableName tableName, TableState.State newState, Review comment: Not used? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache9 commented on a change in pull request #667: HBASE-23055 Alter hbase:meta
Apache9 commented on a change in pull request #667: HBASE-23055 Alter hbase:meta
URL: https://github.com/apache/hbase/pull/667#discussion_r330043419
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawAsyncHBaseAdmin.java
##
@@ -666,42 +669,25 @@ public void run(PRESP resp) {
@Override
public CompletableFuture isTableEnabled(TableName tableName) {
-if (TableName.isMetaTableName(tableName)) {
- return CompletableFuture.completedFuture(true);
-}
-CompletableFuture future = new CompletableFuture<>();
-addListener(AsyncMetaTableAccessor.getTableState(metaTable, tableName),
(state, error) -> {
- if (error != null) {
-future.completeExceptionally(error);
-return;
- }
- if (state.isPresent()) {
-future.complete(state.get().inStates(TableState.State.ENABLED));
- } else {
-future.completeExceptionally(new TableNotFoundException(tableName));
- }
-});
-return future;
+return isTableState(tableName, TableState.State.ENABLED);
}
@Override
public CompletableFuture isTableDisabled(TableName tableName) {
-if (TableName.isMetaTableName(tableName)) {
- return CompletableFuture.completedFuture(false);
-}
-CompletableFuture future = new CompletableFuture<>();
-addListener(AsyncMetaTableAccessor.getTableState(metaTable, tableName),
(state, error) -> {
- if (error != null) {
-future.completeExceptionally(error);
-return;
- }
- if (state.isPresent()) {
-future.complete(state.get().inStates(TableState.State.DISABLED));
- } else {
-future.completeExceptionally(new TableNotFoundException(tableName));
- }
-});
-return future;
+return isTableState(tableName, TableState.State.DISABLED);
+ }
+
+ /**
+ * @return Future that calls Master getTableState and compares to
state
+ */
+ private CompletableFuture isTableState(TableName tableName,
TableState.State state) {
+return this. newMasterCaller().
Review comment:
Is this really a good degisn? In the normal read/write path, sometimes we
need to test whether a table is disabled. After this change, the normal
read/write path will also rely on master, while in the past we only need to
access meta table.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache9 commented on a change in pull request #667: HBASE-23055 Alter hbase:meta
Apache9 commented on a change in pull request #667: HBASE-23055 Alter hbase:meta
URL: https://github.com/apache/hbase/pull/667#discussion_r330095413
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/EnableTableProcedure.java
##
@@ -66,11 +69,12 @@ public EnableTableProcedure(MasterProcedureEnv env,
TableName tableName) {
/**
* Constructor
- * @param env MasterProcedureEnv
+ *
+ * @param env MasterProcedureEnv
* @param tableName the table to operate on
*/
public EnableTableProcedure(MasterProcedureEnv env, TableName tableName,
- ProcedurePrepareLatch syncLatch) {
+ ProcedurePrepareLatch syncLatch) {
Review comment:
Format issue?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache9 commented on a change in pull request #667: HBASE-23055 Alter hbase:meta
Apache9 commented on a change in pull request #667: HBASE-23055 Alter hbase:meta
URL: https://github.com/apache/hbase/pull/667#discussion_r330097961
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
##
@@ -123,30 +119,22 @@ public FSTableDescriptors(final Configuration conf,
final FileSystem fs, final P
* operations; i.e. on remove, we do not do delete in fs.
*/
public FSTableDescriptors(final Configuration conf, final FileSystem fs,
-final Path rootdir, final boolean fsreadonly,
final boolean usecache) throws IOException {
-this(conf, fs, rootdir, fsreadonly, usecache, null);
- }
-
- /**
- * @param fsreadonly True if we are read-only when it comes to filesystem
- * operations; i.e. on remove, we do not do delete in fs.
- * @param metaObserver Used by HMaster. It need to modify the
META_REPLICAS_NUM for meta table descriptor.
- * see HMaster#finishActiveMasterInitialization
- * TODO: This is a workaround. Should remove this ugly
code...
- */
- public FSTableDescriptors(final Configuration conf, final FileSystem fs,
-final Path rootdir, final boolean fsreadonly,
final boolean usecache,
-Function metaObserver) throws IOException {
+ final Path rootdir, final boolean fsreadonly, final boolean usecache)
throws IOException {
this.fs = fs;
this.rootdir = rootdir;
this.fsreadonly = fsreadonly;
this.usecache = usecache;
-this.metaTableDescriptor = metaObserver == null ?
createMetaTableDescriptor(conf)
- : metaObserver.apply(createMetaTableDescriptorBuilder(conf)).build();
+this.configuration = conf;
}
+ /**
+ * Should be private
+ * @deprecated Since 2.3.0. Should be for internal use only. Used by testing.
Review comment:
This class is IA.Private so we can remove this method at any time.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache9 commented on a change in pull request #667: HBASE-23055 Alter hbase:meta
Apache9 commented on a change in pull request #667: HBASE-23055 Alter hbase:meta
URL: https://github.com/apache/hbase/pull/667#discussion_r330094278
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java
##
@@ -53,8 +52,20 @@
// TODO: Make this a guava Service
@InterfaceAudience.Private
public class TableStateManager {
-
private static final Logger LOG =
LoggerFactory.getLogger(TableStateManager.class);
+
+ /**
+ * All table state is kept in hbase:meta except that of hbase:meta itself.
+ * hbase:meta state is kept here locally in this in-memory variable. State
+ * for hbase:meta is not persistent. If this process dies, the hbase:meta
+ * state reverts to enabled. State is used so we can edit hbase:meta as we
+ * would any other table by disabling, altering, and then re-enabling. If
this
+ * process dies in the midst of an edit, the table reverts to enabled. Schema
+ * is read from the filesystem. It is changed atomically so if we die midway
+ * through an edit we should be good.
+ */
+ private TableState.State metaTableState = TableState.State.ENABLED;
Review comment:
Why not store it on zk? Just like the location of meta table...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache9 commented on a change in pull request #667: HBASE-23055 Alter hbase:meta
Apache9 commented on a change in pull request #667: HBASE-23055 Alter hbase:meta
URL: https://github.com/apache/hbase/pull/667#discussion_r330095237
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DisableTableProcedure.java
##
@@ -242,10 +241,7 @@ public TableOperationType getTableOperationType() {
*/
private boolean prepareDisable(final MasterProcedureEnv env) throws
IOException {
boolean canTableBeDisabled = true;
-if (tableName.equals(TableName.META_TABLE_NAME)) {
- setFailure("master-disable-table", new ConstraintException("Cannot
disable catalog table"));
- canTableBeDisabled = false;
-} else if
(!MetaTableAccessor.tableExists(env.getMasterServices().getConnection(),
tableName)) {
+if
(!MetaTableAccessor.tableExists(env.getMasterServices().getConnection(),
tableName)) {
Review comment:
MetaTableAccessor can be used to test whether meta table exists?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache9 commented on a change in pull request #667: HBASE-23055 Alter hbase:meta
Apache9 commented on a change in pull request #667: HBASE-23055 Alter hbase:meta URL: https://github.com/apache/hbase/pull/667#discussion_r330099157 ## File path: hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java ## @@ -157,6 +156,7 @@ import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil; + Review comment: Nit: remove useless empty line. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-23066) Allow cache on write during compactions when prefetching is enabled
[ https://issues.apache.org/jira/browse/HBASE-23066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941877#comment-16941877 ] Jacob LeBlanc commented on HBASE-23066: --- [~ram_krish] the attached HBASE-23066.patch is for master. This is my first patch submission, can you please let me know next steps? > Allow cache on write during compactions when prefetching is enabled > --- > > Key: HBASE-23066 > URL: https://issues.apache.org/jira/browse/HBASE-23066 > Project: HBase > Issue Type: Improvement > Components: Compaction, regionserver >Affects Versions: 1.4.10 >Reporter: Jacob LeBlanc >Assignee: Jacob LeBlanc >Priority: Minor > Fix For: 1.5.0, 2.3.0 > > Attachments: HBASE-23066.patch, performance_results.png, > prefetchCompactedBlocksOnWrite.patch > > > In cases where users care a lot about read performance for tables that are > small enough to fit into a cache (or the cache is large enough), > prefetchOnOpen can be enabled to make the entire table available in cache > after the initial region opening is completed. Any new data can also be > guaranteed to be in cache with the cacheBlocksOnWrite setting. > However, the missing piece is when all blocks are evicted after a compaction. > We found very poor performance after compactions for tables under heavy read > load and a slower backing filesystem (S3). After a compaction the prefetching > threads need to compete with threads servicing read requests and get > constantly blocked as a result. > This is a proposal to introduce a new cache configuration option that would > cache blocks on write during compaction for any column family that has > prefetch enabled. This would virtually guarantee all blocks are kept in cache > after the initial prefetch on open is completed allowing for guaranteed > steady read performance despite a slow backing file system. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-22903) alter_status command is broken
[
https://issues.apache.org/jira/browse/HBASE-22903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941875#comment-16941875
]
Hudson commented on HBASE-22903:
Results for branch master
[build #1489 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1489/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1489//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1489//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1489//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> alter_status command is broken
> --
>
> Key: HBASE-22903
> URL: https://issues.apache.org/jira/browse/HBASE-22903
> Project: HBase
> Issue Type: Bug
> Components: metrics, shell
>Affects Versions: 3.0.0
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-22903.branch-2.000.patch,
> HBASE-22903.branch-2.1.000.patch
>
>
> This is applicable to master branch only:
> {code:java}
> > alter_status 't1'
> ERROR: undefined method `getAlterStatus' for
> #
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23055) Alter hbase:meta
[
https://issues.apache.org/jira/browse/HBASE-23055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941876#comment-16941876
]
Hudson commented on HBASE-23055:
Results for branch master
[build #1489 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1489/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1489//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1489//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1489//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Alter hbase:meta
>
>
> Key: HBASE-23055
> URL: https://issues.apache.org/jira/browse/HBASE-23055
> Project: HBase
> Issue Type: Task
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
>
> hbase:meta is currently hardcoded. Its schema cannot be change.
> This issue is about allowing edits to hbase:meta schema. It will allow our
> being able to set encodings such as the block-with-indexes which will help
> quell CPU usage on host carrying hbase:meta. A dynamic hbase:meta is first
> step on road to being able to split meta.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23102) Improper Usage of Map putIfAbsent
[
https://issues.apache.org/jira/browse/HBASE-23102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941850#comment-16941850
]
David Mollitor commented on HBASE-23102:
[~wchevreuil] I certainly can, but I won't be able to for at least a week.
> Improper Usage of Map putIfAbsent
> -
>
> Key: HBASE-23102
> URL: https://issues.apache.org/jira/browse/HBASE-23102
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.3.0
>Reporter: David Mollitor
>Priority: Minor
> Labels: newbie, noob
>
> When using {{Map#putIfAbsent}}, the argument should not be a {{new}} object.
> Otherwise, if the item is present, the object that was instantiated is
> immediately thrown away. Instead, use {{Map#computeIfAbsent}} so that the
> object is only instantiated if it is needed.
> There exists a good example in the {{Map}} JavaDoc:
> https://docs.oracle.com/javase/8/docs/api/java/util/Map.html#computeIfAbsent-K-java.util.function.Function-
>
> h2. Locations
> https://github.com/apache/hbase/blob/9370347efea5b09e2fa8f4e5d82fa32491e1181b/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaCache.java#L227-L236
> https://github.com/apache/hbase/blob/025ddce868eb06b4072b5152c5ffae5a01e7ae30/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/throttle/StoreHotnessProtector.java#L124-L129
> https://github.com/apache/hbase/blob/1170f28122d9d36e511ba504a5263ec62e11ef6a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java#L555
> https://github.com/apache/hbase/blob/4ca760fe9dd373b8d8a4c48db15e42424920653c/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServer.java#L584-L586
> https://github.com/apache/hbase/blob/4ca760fe9dd373b8d8a4c48db15e42424920653c/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServer.java#L585
> https://github.com/apache/hbase/blob/5b01e613fbbb92e243e99a1d199b4ffbb21ed2d9/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java#L834
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23102) Improper Usage of Map putIfAbsent
[
https://issues.apache.org/jira/browse/HBASE-23102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941818#comment-16941818
]
Wellington Chevreuil commented on HBASE-23102:
--
Thanks [~belugabehr], would you submit a PR with these fixes?
> Improper Usage of Map putIfAbsent
> -
>
> Key: HBASE-23102
> URL: https://issues.apache.org/jira/browse/HBASE-23102
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.3.0
>Reporter: David Mollitor
>Priority: Minor
> Labels: newbie, noob
>
> When using {{Map#putIfAbsent}}, the argument should not be a {{new}} object.
> Otherwise, if the item is present, the object that was instantiated is
> immediately thrown away. Instead, use {{Map#computeIfAbsent}} so that the
> object is only instantiated if it is needed.
> There exists a good example in the {{Map}} JavaDoc:
> https://docs.oracle.com/javase/8/docs/api/java/util/Map.html#computeIfAbsent-K-java.util.function.Function-
>
> h2. Locations
> https://github.com/apache/hbase/blob/9370347efea5b09e2fa8f4e5d82fa32491e1181b/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaCache.java#L227-L236
> https://github.com/apache/hbase/blob/025ddce868eb06b4072b5152c5ffae5a01e7ae30/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/throttle/StoreHotnessProtector.java#L124-L129
> https://github.com/apache/hbase/blob/1170f28122d9d36e511ba504a5263ec62e11ef6a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java#L555
> https://github.com/apache/hbase/blob/4ca760fe9dd373b8d8a4c48db15e42424920653c/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServer.java#L584-L586
> https://github.com/apache/hbase/blob/4ca760fe9dd373b8d8a4c48db15e42424920653c/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServer.java#L585
> https://github.com/apache/hbase/blob/5b01e613fbbb92e243e99a1d199b4ffbb21ed2d9/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java#L834
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase-filesystem] wchevreuil commented on a change in pull request #9: HBASE-23099 - [HBOSS] Raise hadoop dependency version to 3.2.1
wchevreuil commented on a change in pull request #9: HBASE-23099 - [HBOSS]
Raise hadoop dependency version to 3.2.1
URL: https://github.com/apache/hbase-filesystem/pull/9#discussion_r330048363
##
File path:
hbase-oss/src/test/java/org/apache/hadoop/hbase/oss/contract/TestHBOSSContractOpen.java
##
@@ -31,6 +33,14 @@ protected Configuration createConfiguration() {
@Override
protected AbstractFSContract createContract(Configuration conf) {
-return new HBOSSContract(conf);
+HBOSSContract contract = new HBOSSContract(conf);
+try {
+ TestUtils.getFileSystem(conf);
+} catch (Exception e) {
+ e.printStackTrace();
Review comment:
Removed within the overall refactoring, but noticed this is spread around
other points in hboss tests. Might need an additional jira for cleaning this
out.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase-filesystem] wchevreuil commented on a change in pull request #9: HBASE-23099 - [HBOSS] Raise hadoop dependency version to 3.2.1
wchevreuil commented on a change in pull request #9: HBASE-23099 - [HBOSS]
Raise hadoop dependency version to 3.2.1
URL: https://github.com/apache/hbase-filesystem/pull/9#discussion_r330047897
##
File path:
hbase-oss/src/test/java/org/apache/hadoop/hbase/oss/contract/TestHBOSSContractRootDirectory.java
##
@@ -34,6 +36,14 @@ protected Configuration createConfiguration() {
@Override
protected AbstractFSContract createContract(Configuration conf) {
-return new HBOSSContract(conf);
+HBOSSContract contract = new HBOSSContract(conf);
+try {
+ TestUtils.getFileSystem(conf);
+} catch (Exception e) {
+ e.printStackTrace();
+ fail("Exception configuring FS: " + e);
Review comment:
Right, moved this logic to a new method in TestUtils that's then called on
S3A specific contract tests.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (HBASE-23102) Improper Usage of Map putIfAbsent
[
https://issues.apache.org/jira/browse/HBASE-23102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HBASE-23102:
---
Labels: newbie noob (was: )
> Improper Usage of Map putIfAbsent
> -
>
> Key: HBASE-23102
> URL: https://issues.apache.org/jira/browse/HBASE-23102
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.3.0
>Reporter: David Mollitor
>Priority: Minor
> Labels: newbie, noob
>
> When using {{Map#putIfAbsent}}, the argument should not be a {{new}} object.
> Otherwise, if the item is present, the object that was instantiated is
> immediately thrown away. Instead, use {{Map#computeIfAbsent}} so that the
> object is only instantiated if it is needed.
> There exists a good example in the {{Map}} JavaDoc:
> https://docs.oracle.com/javase/8/docs/api/java/util/Map.html#computeIfAbsent-K-java.util.function.Function-
>
> h2. Locations
> https://github.com/apache/hbase/blob/9370347efea5b09e2fa8f4e5d82fa32491e1181b/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaCache.java#L227-L236
> https://github.com/apache/hbase/blob/025ddce868eb06b4072b5152c5ffae5a01e7ae30/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/throttle/StoreHotnessProtector.java#L124-L129
> https://github.com/apache/hbase/blob/1170f28122d9d36e511ba504a5263ec62e11ef6a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java#L555
> https://github.com/apache/hbase/blob/4ca760fe9dd373b8d8a4c48db15e42424920653c/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServer.java#L584-L586
> https://github.com/apache/hbase/blob/4ca760fe9dd373b8d8a4c48db15e42424920653c/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServer.java#L585
> https://github.com/apache/hbase/blob/5b01e613fbbb92e243e99a1d199b4ffbb21ed2d9/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java#L834
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-23102) Improper Usage of Map putIfAbsent
David Mollitor created HBASE-23102:
--
Summary: Improper Usage of Map putIfAbsent
Key: HBASE-23102
URL: https://issues.apache.org/jira/browse/HBASE-23102
Project: HBase
Issue Type: Improvement
Affects Versions: 2.3.0
Reporter: David Mollitor
When using {{Map#putIfAbsent}}, the argument should not be a {{new}} object.
Otherwise, if the item is present, the object that was instantiated is
immediately thrown away. Instead, use {{Map#computeIfAbsent}} so that the
object is only instantiated if it is needed.
There exists a good example in the {{Map}} JavaDoc:
https://docs.oracle.com/javase/8/docs/api/java/util/Map.html#computeIfAbsent-K-java.util.function.Function-
h2. Locations
https://github.com/apache/hbase/blob/9370347efea5b09e2fa8f4e5d82fa32491e1181b/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaCache.java#L227-L236
https://github.com/apache/hbase/blob/025ddce868eb06b4072b5152c5ffae5a01e7ae30/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/throttle/StoreHotnessProtector.java#L124-L129
https://github.com/apache/hbase/blob/1170f28122d9d36e511ba504a5263ec62e11ef6a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java#L555
https://github.com/apache/hbase/blob/4ca760fe9dd373b8d8a4c48db15e42424920653c/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServer.java#L584-L586
https://github.com/apache/hbase/blob/4ca760fe9dd373b8d8a4c48db15e42424920653c/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServer.java#L585
https://github.com/apache/hbase/blob/5b01e613fbbb92e243e99a1d199b4ffbb21ed2d9/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java#L834
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-23101) Backport HBASE-22380 to branch-1
Wellington Chevreuil created HBASE-23101: Summary: Backport HBASE-22380 to branch-1 Key: HBASE-23101 URL: https://issues.apache.org/jira/browse/HBASE-23101 Project: HBase Issue Type: Sub-task Reporter: Wellington Chevreuil Assignee: Wellington Chevreuil -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-22380) break circle replication when doing bulkload
[ https://issues.apache.org/jira/browse/HBASE-22380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil resolved HBASE-22380. -- Fix Version/s: (was: 1.4.12) (was: 1.5.0) Resolution: Fixed Actually, had discussed this internally with [~psomogyi], and we decided to add branch-1 backport as a subtask, so am closing this again, removing branch-1 related releases from fix versions. > break circle replication when doing bulkload > > > Key: HBASE-22380 > URL: https://issues.apache.org/jira/browse/HBASE-22380 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 3.0.0, 1.5.0, 2.2.0, 1.4.10, 2.0.5, 2.3.0, 2.1.5, 1.3.5 >Reporter: chenxu >Assignee: Wellington Chevreuil >Priority: Critical > Labels: bulkload > Fix For: 3.0.0, 2.3.0, 2.1.7, 2.2.2 > > Attachments: HBASE-22380.branch-2.1.0001.patch, > HBASE-22380.branch-2.1.0003.patch, HBASE-22380.branch-2.1.0004.patch > > > when enabled master-master bulkload replication, HFiles will be replicated > circularly between two clusters -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-22930) Set unique name to longCompactions/shortCompactions threads
[
https://issues.apache.org/jira/browse/HBASE-22930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941757#comment-16941757
]
Hudson commented on HBASE-22930:
Results for branch branch-1
[build #1091 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1091/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1091//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1091//JDK7_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1091//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Set unique name to longCompactions/shortCompactions threads
> ---
>
> Key: HBASE-22930
> URL: https://issues.apache.org/jira/browse/HBASE-22930
> Project: HBase
> Issue Type: Improvement
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Minor
> Fix For: 3.0.0, 1.5.0, 2.3.0, 1.3.6, 1.4.11, 2.1.7, 2.2.2
>
>
> Sometime observed multiple compaction threads have the same name which is
> misleading while analysing the log.
> Currently we just append the current timestamp which can be same for multiple
> threads,
> [https://github.com/apache/hbase/blob/ec68bf30ba91ea7484d3b56b0809ff5434f9406e/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplit.java#L144]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase-filesystem] wchevreuil commented on a change in pull request #9: HBASE-23099 - [HBOSS] Raise hadoop dependency version to 3.2.1
wchevreuil commented on a change in pull request #9: HBASE-23099 - [HBOSS]
Raise hadoop dependency version to 3.2.1
URL: https://github.com/apache/hbase-filesystem/pull/9#discussion_r330006697
##
File path:
hbase-oss/src/test/java/org/apache/hadoop/hbase/oss/contract/TestHBOSSContractRootDirectory.java
##
@@ -34,6 +36,14 @@ protected Configuration createConfiguration() {
@Override
protected AbstractFSContract createContract(Configuration conf) {
-return new HBOSSContract(conf);
+HBOSSContract contract = new HBOSSContract(conf);
+try {
+ TestUtils.getFileSystem(conf);
+} catch (Exception e) {
+ e.printStackTrace();
+ fail("Exception configuring FS: " + e);
Review comment:
Maybe make _TestUtils.getFileSystem(conf)_ throw _AssertionError_, since
it's a basic test requirement? That would simplify this check for the subset of
tests requiring it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase-filesystem] wchevreuil commented on a change in pull request #9: HBASE-23099 - [HBOSS] Raise hadoop dependency version to 3.2.1
wchevreuil commented on a change in pull request #9: HBASE-23099 - [HBOSS]
Raise hadoop dependency version to 3.2.1
URL: https://github.com/apache/hbase-filesystem/pull/9#discussion_r330006697
##
File path:
hbase-oss/src/test/java/org/apache/hadoop/hbase/oss/contract/TestHBOSSContractRootDirectory.java
##
@@ -34,6 +36,14 @@ protected Configuration createConfiguration() {
@Override
protected AbstractFSContract createContract(Configuration conf) {
-return new HBOSSContract(conf);
+HBOSSContract contract = new HBOSSContract(conf);
+try {
+ TestUtils.getFileSystem(conf);
+} catch (Exception e) {
+ e.printStackTrace();
+ fail("Exception configuring FS: " + e);
Review comment:
Maybe make _TestUtils.getFileSystem(conf);_ throw _AssertionError_, since
it's a basic test requirement? That would simplify this check for the subset of
tests requiring it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (HBASE-22514) Move rsgroup feature into core of HBase
[
https://issues.apache.org/jira/browse/HBASE-22514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941741#comment-16941741
]
Hudson commented on HBASE-22514:
Results for branch HBASE-22514
[build #133 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22514/133/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22514/133//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22514/133//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22514/133//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22514/133//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> Move rsgroup feature into core of HBase
> ---
>
> Key: HBASE-22514
> URL: https://issues.apache.org/jira/browse/HBASE-22514
> Project: HBase
> Issue Type: Umbrella
> Components: Admin, Client, rsgroup
>Reporter: Yechao Chen
>Assignee: Duo Zhang
>Priority: Major
> Attachments: HBASE-22514.master.001.patch,
> image-2019-05-31-18-25-38-217.png
>
>
> The class RSGroupAdminClient is not public
> we need to use java api RSGroupAdminClient to manager RSG
> so RSGroupAdminClient should be public
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work started] (HBASE-22380) break circle replication when doing bulkload
[ https://issues.apache.org/jira/browse/HBASE-22380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-22380 started by Wellington Chevreuil. > break circle replication when doing bulkload > > > Key: HBASE-22380 > URL: https://issues.apache.org/jira/browse/HBASE-22380 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 3.0.0, 1.5.0, 2.2.0, 1.4.10, 2.0.5, 2.3.0, 2.1.5, 1.3.5 >Reporter: chenxu >Assignee: Wellington Chevreuil >Priority: Critical > Labels: bulkload > Fix For: 3.0.0, 1.5.0, 2.3.0, 2.1.7, 2.2.2, 1.4.12 > > Attachments: HBASE-22380.branch-2.1.0001.patch, > HBASE-22380.branch-2.1.0003.patch, HBASE-22380.branch-2.1.0004.patch > > > when enabled master-master bulkload replication, HFiles will be replicated > circularly between two clusters -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HBASE-22380) break circle replication when doing bulkload
[ https://issues.apache.org/jira/browse/HBASE-22380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil reopened HBASE-22380: -- Thanks [~zhangduo]! Am still pending a branch-1 commit. The code base is too different, tried to cherry-pick commit from branch-2.1 into branch-1, but there are too many conflicts. Will try to submit a branch-1 PR soon. > break circle replication when doing bulkload > > > Key: HBASE-22380 > URL: https://issues.apache.org/jira/browse/HBASE-22380 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 3.0.0, 1.5.0, 2.2.0, 1.4.10, 2.0.5, 2.3.0, 2.1.5, 1.3.5 >Reporter: chenxu >Assignee: Wellington Chevreuil >Priority: Critical > Labels: bulkload > Fix For: 3.0.0, 1.5.0, 2.3.0, 2.1.7, 2.2.2, 1.4.12 > > Attachments: HBASE-22380.branch-2.1.0001.patch, > HBASE-22380.branch-2.1.0003.patch, HBASE-22380.branch-2.1.0004.patch > > > when enabled master-master bulkload replication, HFiles will be replicated > circularly between two clusters -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23066) Allow cache on write during compactions when prefetching is enabled
[ https://issues.apache.org/jira/browse/HBASE-23066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941735#comment-16941735 ] ramkrishna.s.vasudevan commented on HBASE-23066: bq.As time goes on, HFile will grow larger(because of Compaction), and it's data may get colder and colder, In some scenarios, only the latest time window data is accessed, so warmup the large HFile seems unnecessary. got it. thanks [~javaman_chen]. More of a size based threshold. > Allow cache on write during compactions when prefetching is enabled > --- > > Key: HBASE-23066 > URL: https://issues.apache.org/jira/browse/HBASE-23066 > Project: HBase > Issue Type: Improvement > Components: Compaction, regionserver >Affects Versions: 1.4.10 >Reporter: Jacob LeBlanc >Assignee: Jacob LeBlanc >Priority: Minor > Fix For: 1.5.0, 2.3.0 > > Attachments: HBASE-23066.patch, performance_results.png, > prefetchCompactedBlocksOnWrite.patch > > > In cases where users care a lot about read performance for tables that are > small enough to fit into a cache (or the cache is large enough), > prefetchOnOpen can be enabled to make the entire table available in cache > after the initial region opening is completed. Any new data can also be > guaranteed to be in cache with the cacheBlocksOnWrite setting. > However, the missing piece is when all blocks are evicted after a compaction. > We found very poor performance after compactions for tables under heavy read > load and a slower backing filesystem (S3). After a compaction the prefetching > threads need to compete with threads servicing read requests and get > constantly blocked as a result. > This is a proposal to introduce a new cache configuration option that would > cache blocks on write during compaction for any column family that has > prefetch enabled. This would virtually guarantee all blocks are kept in cache > after the initial prefetch on open is completed allowing for guaranteed > steady read performance despite a slow backing file system. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase-connectors] meszibalu merged pull request #42: HBASE-22817 Use hbase-shaded dependencies in hbase-spark
meszibalu merged pull request #42: HBASE-22817 Use hbase-shaded dependencies in hbase-spark URL: https://github.com/apache/hbase-connectors/pull/42 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase-connectors] meszibalu commented on issue #42: HBASE-22817 Use hbase-shaded dependencies in hbase-spark
meszibalu commented on issue #42: HBASE-22817 Use hbase-shaded dependencies in hbase-spark URL: https://github.com/apache/hbase-connectors/pull/42#issuecomment-536969156 Thanks @busbey, it compiles for me locally, let me push it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (HBASE-23093) Avoid Optional Anti-Patterns
[ https://issues.apache.org/jira/browse/HBASE-23093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viraj Jasani updated HBASE-23093: - Attachment: (was: HBASE-23093.master.000.patch) > Avoid Optional Anti-Patterns > > > Key: HBASE-23093 > URL: https://issues.apache.org/jira/browse/HBASE-23093 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0, 2.3.0, 1.6.0 >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Minor > Fix For: 3.0.0, 2.3.0, 1.6.0 > > > Optional should be used as a return type only. It's a neat solution for > handling data that might not be present. We should avoid using Optional > Anti-Patterns i.e. using it as a field or parameter type due to these reasons: > 1. Using Optional parameters causing conditional logic inside the methods is > not productive. > 2. Packing an argument in an Optional is suboptimal for the compiler and does > an unnecessary wrapping. > 3. Optional field is not serializable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] petersomogyi commented on a change in pull request #671: HBASE-23092 Make the RM tooling in dev-tools/create-release generic
petersomogyi commented on a change in pull request #671: HBASE-23092 Make the
RM tooling in dev-tools/create-release generic
URL: https://github.com/apache/hbase/pull/671#discussion_r329959354
##
File path: dev-support/create-release/vote.tmpl
##
@@ -1,13 +1,14 @@
-Please vote on this Apache HBase release candidate (RC), ${RELEASE_VERSION}.
+Please vote on this apache ${PROJECT_TEXT} release candidate,
+${PROJECT}-${RELEASE_TAG}
The VOTE will remain open for at least 72 hours.
-[ ] +1 Release this package as Apache HBase ${RELEASE_VERSION}
+[ ] +1 Release this package as apache ${PROJECT_TEXT} ${RELEASE_VERSION}
Review comment:
s/apache/Apache
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] petersomogyi commented on a change in pull request #671: HBASE-23092 Make the RM tooling in dev-tools/create-release generic
petersomogyi commented on a change in pull request #671: HBASE-23092 Make the
RM tooling in dev-tools/create-release generic
URL: https://github.com/apache/hbase/pull/671#discussion_r329959174
##
File path: dev-support/create-release/vote.tmpl
##
@@ -22,7 +23,8 @@ Artifacts were signed with the ${GPG_KEY} key which can be
found in:
https://dist.apache.org/repos/dist/release/hbase/KEYS
-To learn more about Apache HBase, please see http://hbase.apache.org/
+ To learn more about apache ${PROJECT_TEXT}, please see
Review comment:
s/apache/Apache
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] petersomogyi commented on a change in pull request #671: HBASE-23092 Make the RM tooling in dev-tools/create-release generic
petersomogyi commented on a change in pull request #671: HBASE-23092 Make the
RM tooling in dev-tools/create-release generic
URL: https://github.com/apache/hbase/pull/671#discussion_r329959298
##
File path: dev-support/create-release/vote.tmpl
##
@@ -1,13 +1,14 @@
-Please vote on this Apache HBase release candidate (RC), ${RELEASE_VERSION}.
+Please vote on this apache ${PROJECT_TEXT} release candidate,
Review comment:
s/apache/Apache
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
