[jira] [Commented] (HBASE-14614) Procedure v2: Core Assignment Manager
[ https://issues.apache.org/jira/browse/HBASE-14614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946565#comment-15946565 ] stack commented on HBASE-14614: --- Started up HBASE-17844 in effort at making this patch smaller (not very successful, but trying). > Procedure v2: Core Assignment Manager > - > > Key: HBASE-14614 > URL: https://issues.apache.org/jira/browse/HBASE-14614 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Stephen Yuan Jiang >Assignee: Matteo Bertozzi > Fix For: 2.0.0 > > Attachments: HBASE-14614.master.001.patch, > HBASE-14614.master.002.patch, HBASE-14614.master.003.patch, > HBASE-14614.master.004.patch, HBASE-14614.master.005.patch, > HBASE-14614.master.006.patch, HBASE-14614.master.007.patch, > HBASE-14614.master.008.patch, HBASE-14614.master.009.patch, > HBASE-14614.master.010.patch, HBASE-14614.master.011.patch, > HBASE-14614.master.012.patch, HBASE-14614.master.012.patch, > HBASE-14614.master.013.patch, HBASE-14614.master.014.patch, > HBASE-14614.master.015.patch, HBASE-14614.master.016.patch, > HBASE-14614.master.017.patch, HBASE-14614.master.018.patch, > HBASE-14614.master.019.patch, HBASE-14614.master.020.patch, > HBASE-14614.master.021.patch > > > New AssignmentManager implemented using proc-v2. > - AssignProcedure handle assignment operation > - UnassignProcedure handle unassign operation > - MoveRegionProcedure handle move/balance operation > Concurrent Assign operations are batched together and sent to the balancer > Concurrent Assign and Unassign operation ready to be sent to the RS are > batched together > This patch is an intermediate state where we add the new AM as > AssignmentManager2() to the master, to be reached by tests. but the new AM > will not be integrated with the rest of the system. Only new am unit-tests > will exercise the new assigment manager. The integration with the master code > is part of HBASE-14616 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17844) Subset of HBASE-14614, Procedure v2: Core Assignment Manager (non-critical related changes)
[ https://issues.apache.org/jira/browse/HBASE-17844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-17844: -- Attachment: HBASE-17844.master.001.patch > Subset of HBASE-14614, Procedure v2: Core Assignment Manager (non-critical > related changes) > --- > > Key: HBASE-17844 > URL: https://issues.apache.org/jira/browse/HBASE-17844 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Affects Versions: 2.0.0 >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: HBASE-17844.master.001.patch > > > Here is a patch that breaks out non-pertinent changes that made it into > HBASE-14614 in an attempt at shrinking its overall size. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17844) Subset of HBASE-14614, Procedure v2: Core Assignment Manager (non-critical related changes)
[ https://issues.apache.org/jira/browse/HBASE-17844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-17844: -- Summary: Subset of HBASE-14614, Procedure v2: Core Assignment Manager (non-critical related changes) (was: Subset of HBASE-14614, Procedure v2: Core Assignment Manager) > Subset of HBASE-14614, Procedure v2: Core Assignment Manager (non-critical > related changes) > --- > > Key: HBASE-17844 > URL: https://issues.apache.org/jira/browse/HBASE-17844 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Affects Versions: 2.0.0 >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > > Here is a patch that breaks out non-pertinent changes that made it into > HBASE-14614 in an attempt at shrinking its overall size. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HBASE-17844) Subset of HBASE-14614, Procedure v2: Core Assignment Manager
[ https://issues.apache.org/jira/browse/HBASE-17844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack reassigned HBASE-17844: - Assignee: stack Description: Here is a patch that breaks out non-pertinent changes that made it into HBASE-14614 in an attempt at shrinking its overall size. > Subset of HBASE-14614, Procedure v2: Core Assignment Manager > > > Key: HBASE-17844 > URL: https://issues.apache.org/jira/browse/HBASE-17844 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Affects Versions: 2.0.0 >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > > Here is a patch that breaks out non-pertinent changes that made it into > HBASE-14614 in an attempt at shrinking its overall size. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17844) Subset of HBASE-14614, Procedure v2: Core Assignment Manager
stack created HBASE-17844: - Summary: Subset of HBASE-14614, Procedure v2: Core Assignment Manager Key: HBASE-17844 URL: https://issues.apache.org/jira/browse/HBASE-17844 Project: HBase Issue Type: Bug Components: Region Assignment Affects Versions: 2.0.0 Reporter: stack Fix For: 2.0.0 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-14978) Don't allow Multi to retain too many blocks
[
https://issues.apache.org/jira/browse/HBASE-14978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946482#comment-15946482
]
Anoop Sam John commented on HBASE-14978:
Reading this part of code again
{code}
if (c instanceof ByteBufferCell) {
ByteBufferCell bbCell = (ByteBufferCell) c;
ByteBuffer bb = bbCell.getValueByteBuffer();
if (bb != lastBlock) {
context.incrementResponseBlockSize(bb.capacity());
lastBlock = bb;
}
}
{code}
This is specific case when the cell is read from BucketCache. The cell will be
backed by DBB in the Bucket Cache. Each of these BB in bucket cache is of
capacity 4 MB! It is possible to have multiple buckets and blocks within a BB.
The code above will take the response block size as 4 MB! This will make the
response to be send back immediately. This might not be correct IMO. If this
was counting 64 KB block size ya, that may be ok but 4 MB is too much. What do
you say [~eclark]?
cc [~saint@gmail.com], [~ram_krish]
> Don't allow Multi to retain too many blocks
> ---
>
> Key: HBASE-14978
> URL: https://issues.apache.org/jira/browse/HBASE-14978
> Project: HBase
> Issue Type: Improvement
> Components: io, IPC/RPC, regionserver
>Affects Versions: 2.0.0, 1.2.0, 1.3.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
>Priority: Blocker
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14978-branch-1.2.patch,
> HBASE-14978-branch-1.2.patch, HBASE-14978-branch-1.2.patch,
> HBASE-14978-branch-1.2.patch, HBASE-14978.patch, HBASE-14978-v1.patch,
> HBASE-14978-v2.patch, HBASE-14978-v3.patch, HBASE-14978-v4.patch
>
>
> Scans and Multi's have limits on the total size of cells that can be
> returned. However if those requests are not all pointing at the same blocks
> then the KeyValues can keep alive a lot more data than their size.
> Take the following example:
> A multi with a list of 1 gets to a fat row. Each column being returned in
> in a different block. Each column is small 32 bytes or so.
> So the total cell size will be 32 * 1 = ~320kb. However if each block is
> 128k then total retained heap size will be almost 2gigs.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-14417) Incremental backup and bulk loading
[ https://issues.apache.org/jira/browse/HBASE-14417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946469#comment-15946469 ] Hudson commented on HBASE-14417: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #2758 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2758/]) HBASE-14417 Incremental backup and bulk loading (tedyu: rev 0345fc87759a7d44ecc385327ebb586fc632fb65) * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/backup/BackupObserver.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/backup/impl/BackupManager.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/backup/TestIncrementalBackupWithBulkLoad.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/backup/TestBackupBase.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/backup/impl/BackupAdminImpl.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/backup/impl/RestoreTablesClient.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/backup/TestBackupHFileCleaner.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/backup/mapreduce/MapReduceRestoreJob.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFiles.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/backup/impl/IncrementalTableBackupClient.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/backup/impl/BackupSystemTable.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/backup/BackupHFileCleaner.java > Incremental backup and bulk loading > --- > > Key: HBASE-14417 > URL: https://issues.apache.org/jira/browse/HBASE-14417 > Project: HBase > Issue Type: New Feature >Reporter: Vladimir Rodionov >Assignee: Ted Yu >Priority: Blocker > Labels: backup > Fix For: 2.0 > > Attachments: 14417-tbl-ext.v10.txt, 14417-tbl-ext.v11.txt, > 14417-tbl-ext.v14.txt, 14417-tbl-ext.v18.txt, 14417-tbl-ext.v19.txt, > 14417-tbl-ext.v20.txt, 14417-tbl-ext.v21.txt, 14417-tbl-ext.v22.txt, > 14417-tbl-ext.v23.txt, 14417-tbl-ext.v24.txt, 14417-tbl-ext.v9.txt, > 14417.v11.txt, 14417.v13.txt, 14417.v1.txt, 14417.v21.txt, 14417.v23.txt, > 14417.v24.txt, 14417.v25.txt, 14417.v2.txt, 14417.v6.txt > > > Currently, incremental backup is based on WAL files. Bulk data loading > bypasses WALs for obvious reasons, breaking incremental backups. The only way > to continue backups after bulk loading is to create new full backup of a > table. This may not be feasible for customers who do bulk loading regularly > (say, every day). > Here is the review board (out of date): > https://reviews.apache.org/r/54258/ > In order not to miss the hfiles which are loaded into region directories in a > situation where postBulkLoadHFile() hook is not called (bulk load being > interrupted), we record hfile names thru preCommitStoreFile() hook. > At time of incremental backup, we check the presence of such hfiles. If they > are present, they become part of the incremental backup image. > Here is review board: > https://reviews.apache.org/r/57790/ > Google doc for design: > https://docs.google.com/document/d/1ACCLsecHDvzVSasORgqqRNrloGx4mNYIbvAU7lq5lJE -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17831) Support small scan in thrift2
[ https://issues.apache.org/jira/browse/HBASE-17831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946431#comment-15946431 ] Guangxu Cheng commented on HBASE-17831: --- [~yuzhih...@gmail.com] Thanks for reviewing. > Support small scan in thrift2 > - > > Key: HBASE-17831 > URL: https://issues.apache.org/jira/browse/HBASE-17831 > Project: HBase > Issue Type: Improvement > Components: Thrift >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng > Fix For: 1.4.0, 2.0 > > Attachments: HBASE-17831-branch-1.patch, > HBASE-17831-branch-1-v1.patch, HBASE-17831-master.patch, > HBASE-17831-master-v1.patch > > > Support small scan in thrift2 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17215) Separate small/large file delete threads in HFileCleaner to accelerate hfile cleanup speed
[
https://issues.apache.org/jira/browse/HBASE-17215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946421#comment-15946421
]
Yu Li commented on HBASE-17215:
---
Sure, let's revive this one, will prepare the patch in one day or two.
> Separate small/large file delete threads in HFileCleaner to accelerate hfile
> cleanup speed
> --
>
> Key: HBASE-17215
> URL: https://issues.apache.org/jira/browse/HBASE-17215
> Project: HBase
> Issue Type: Improvement
>Reporter: Yu Li
>Assignee: Yu Li
>
> When using PCIe-SSD the flush speed will be really quick, and although we
> have per CF flush, we still have the
> {{hbase.regionserver.optionalcacheflushinterval}} setting and some other
> mechanism to avoid data kept in memory for too long to flush small hfiles. In
> our online environment we found the single thread cleaner kept cleaning
> earlier flushed small files while large files got no chance, which caused
> disk full then many other problems.
> Deleting hfiles in parallel with too many threads will also increase the
> workload of namenode, so here we propose to separate large/small hfile
> cleaner threads just like we do for compaction, and it turned out to work
> well in our cluster.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17771) [C++] Classes required for implementation of BatchCallerBuilder
[ https://issues.apache.org/jira/browse/HBASE-17771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946385#comment-15946385 ] Enis Soztutar commented on HBASE-17771: --- Thanks Sudeep. Do you mind rebasing the patch to the current branch head. > [C++] Classes required for implementation of BatchCallerBuilder > --- > > Key: HBASE-17771 > URL: https://issues.apache.org/jira/browse/HBASE-17771 > Project: HBase > Issue Type: Sub-task >Reporter: Sudeep Sunthankar >Assignee: Sudeep Sunthankar > Fix For: HBASE-14850 > > Attachments: HBASE-17771.HBASE-14850.v1.patch, > HBASE-17771.HBASE-14850.v2.patch, HBASE-17771.HBASE-14850.v3.patch, > HBASE-17771.HBASE-14850.v4.patch, HBASE-17771.HBASE-14850.v5.patch > > > Separating depedencies of BatchCallerBuilder. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17598) new archetypes apparently not deployed to Maven Central Repository
[
https://issues.apache.org/jira/browse/HBASE-17598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946363#comment-15946363
]
Daniel Vimont commented on HBASE-17598:
---
[~mantonov]: I hope you don't mind that I added you as a "Watcher" of this
issue. Since you're the steward of all things 1.3.x, I thought this should be
brought to your attention. If my observations above are correct, then
apparently some kind of modification needs to be made to whatever script (or
other mechanism) is behind final deployment of artifacts to the Maven central
repository. I don't think I (as only a contributor) have access to that, but
please let me know if you need my assistance resolving this to get
{{hbase-client-project-archetype}} and
{{hbase-shaded-client-project-archetype}} available to the end-user community.
THANKS!
> new archetypes apparently not deployed to Maven Central Repository
> --
>
> Key: HBASE-17598
> URL: https://issues.apache.org/jira/browse/HBASE-17598
> Project: HBase
> Issue Type: Bug
> Components: Usability
>Affects Versions: 1.3.0, 1.3.1
>Reporter: Daniel Vimont
>
> On search.maven.org, one sees that most of the artifacts for the 1.3.0
> release have been successfully deployed, but neither of the two new
> archetypes can apparently be found there:
> {{hbase-client-project-archetype}}
> {{hbase-shaded-client-project-archetype}}
> We do see the projects used to *build* the archetypes (which actually
> probably don't need to be published via Maven), but we do not see the
> archetypes themselves.
> For a specific example, the following artifacts (POM and JAR) need to have
> been deployed for the first archetype listed above:
> {{hbase/hbase-archetypes/hbase-client-project/target/build-archetype/target/generated-sources/archetype/pom.xml}}
> {{hbase/hbase-archetypes/hbase-client-project/target/build-archetype/target/generated-sources/archetype/target/hbase-client-project-archetype-1.3.0.jar}}
> (The awkwardly deep directory structures are what the Maven
> archetype-generation tools generate during the build of hbase.)
> BTW, the Maven documentation on the deployment of archetypes amounts to a
> single sentence:
> "Once you are happy with the state of your archetype, you can deploy (or
> submit it to Maven Central) it as any other artifact and the archetype will
> then be available to any user of Maven."
> That sentence is found on this page:
> https://maven.apache.org/guides/mini/guide-creating-archetypes.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17771) [C++] Classes required for implementation of BatchCallerBuilder
[ https://issues.apache.org/jira/browse/HBASE-17771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946314#comment-15946314 ] Sudeep Sunthankar commented on HBASE-17771: --- Hi Enis, Lets commit this. Let me implement read and write locks at the AsyncBatchRpcRetryingCaller level. Also we would have the Row class in the repo which is required by HBASE-15894 Thanks. > [C++] Classes required for implementation of BatchCallerBuilder > --- > > Key: HBASE-17771 > URL: https://issues.apache.org/jira/browse/HBASE-17771 > Project: HBase > Issue Type: Sub-task >Reporter: Sudeep Sunthankar >Assignee: Sudeep Sunthankar > Fix For: HBASE-14850 > > Attachments: HBASE-17771.HBASE-14850.v1.patch, > HBASE-17771.HBASE-14850.v2.patch, HBASE-17771.HBASE-14850.v3.patch, > HBASE-17771.HBASE-14850.v4.patch, HBASE-17771.HBASE-14850.v5.patch > > > Separating depedencies of BatchCallerBuilder. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-14614) Procedure v2: Core Assignment Manager
[
https://issues.apache.org/jira/browse/HBASE-14614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946268#comment-15946268
]
Hadoop QA commented on HBASE-14614:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 79 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 32s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
22s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
17s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 58s
{color} | {color:red} hbase-protocol-shaded in master has 24 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 43s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
23s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 129 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 16s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m
53s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 49s
{color} | {color:red} hbase-server generated 1 new + 0 unchanged - 0 fixed = 1
total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 43s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 5s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 26s
{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 49s
{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 21s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 19s
{color} | {color:green} hbase-hadoop-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 23s
{color} | {color:green} hbase-hadoop2-compat in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 25m 10s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 15m 18s {color}
| {color:red} hbase-rsgroup in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
20s
[jira] [Comment Edited] (HBASE-16549) Procedure v2 - Add new AM metrics
[ https://issues.apache.org/jira/browse/HBASE-16549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946256#comment-15946256 ] Umesh Agashe edited comment on HBASE-16549 at 3/28/17 11:58 PM: I have summarized AMv2 metric changes in the doc: https://docs.google.com/document/d/1gJsjlacYRuFCSrp8q1zNRr8oWrel339rUtdmFuV7PxE/edit?usp=sharing. Grouping related details like batch size for remote dispatch etc should not be exposed through metrics. We can add TRACE level log messages for profiling and debugging. was (Author: uagashe): I have summarized AMv2 metric changes in the doc: https://docs.google.com/document/d/1gJsjlacYRuFCSrp8q1zNRr8oWrel339rUtdmFuV7PxE/edit?usp=sharing. Grouping related details like batch size for remote dispatch etc should not be exposed through metrics. We can add trace() messages for profiling and debugging. > Procedure v2 - Add new AM metrics > - > > Key: HBASE-16549 > URL: https://issues.apache.org/jira/browse/HBASE-16549 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, Region Assignment >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Umesh Agashe > Fix For: 2.0.0 > > > With the new AM we can add a bunch of metrics > - assign/unassign time > - server crash time > - grouping related metrics? (how many batch we do, and similar?) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16549) Procedure v2 - Add new AM metrics
[ https://issues.apache.org/jira/browse/HBASE-16549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946256#comment-15946256 ] Umesh Agashe commented on HBASE-16549: -- I have summarized AMv2 metric changes in the doc: https://docs.google.com/document/d/1gJsjlacYRuFCSrp8q1zNRr8oWrel339rUtdmFuV7PxE/edit?usp=sharing. Grouping related details like batch size for remote dispatch etc should not be exposed through metrics. We can add trace() messages for profiling and debugging. > Procedure v2 - Add new AM metrics > - > > Key: HBASE-16549 > URL: https://issues.apache.org/jira/browse/HBASE-16549 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, Region Assignment >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Umesh Agashe > Fix For: 2.0.0 > > > With the new AM we can add a bunch of metrics > - assign/unassign time > - server crash time > - grouping related metrics? (how many batch we do, and similar?) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17287) Master becomes a zombie if filesystem object closes
[
https://issues.apache.org/jira/browse/HBASE-17287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946243#comment-15946243
]
Enis Soztutar commented on HBASE-17287:
---
I've already +1'ed the patch above, no?
> Master becomes a zombie if filesystem object closes
> ---
>
> Key: HBASE-17287
> URL: https://issues.apache.org/jira/browse/HBASE-17287
> Project: HBase
> Issue Type: Bug
> Components: master
>Reporter: Clay B.
>Assignee: Ted Yu
> Fix For: 1.4.0, 2.0
>
> Attachments: 17287.branch-1.1.v4.txt, 17287.branch-1.v3.txt,
> 17287.branch-1.v4.txt, 17287.master.v2.txt, 17287.master.v3.txt,
> 17287.master.v4.txt, 17287.master.v5.txt, 17287.v2.txt
>
>
> We have seen an issue whereby if the HDFS is unstable and the HBase master's
> HDFS client is unable to stabilize before
> {{dfs.client.failover.max.attempts}} then the master's filesystem object
> closes. This seems to result in an HBase master which will continue to run
> (process and znode exists) but no meaningful work can be done (e.g. assigning
> meta).What we saw in our HBase master logs was:{code}2016-12-01 19:19:08,192
> ERROR org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler:
> Caught M_META_SERVER_SHUTDOWN, count=1java.io.IOException: failed log
> splitting for cluster-r5n12.bloomberg.com,60200,1480632863218, will retryat
> org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler.process(MetaServerShutdownHandler.java:84)at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at
> java.lang.Thread.run(Thread.java:745)Caused by: java.io.IOException:
> Filesystem closed{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17287) Master becomes a zombie if filesystem object closes
[
https://issues.apache.org/jira/browse/HBASE-17287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946218#comment-15946218
]
Ted Yu commented on HBASE-17287:
[~enis] [~clayb]:
Do you have more review comments ?
Thanks
> Master becomes a zombie if filesystem object closes
> ---
>
> Key: HBASE-17287
> URL: https://issues.apache.org/jira/browse/HBASE-17287
> Project: HBase
> Issue Type: Bug
> Components: master
>Reporter: Clay B.
>Assignee: Ted Yu
> Fix For: 1.4.0, 2.0
>
> Attachments: 17287.branch-1.1.v4.txt, 17287.branch-1.v3.txt,
> 17287.branch-1.v4.txt, 17287.master.v2.txt, 17287.master.v3.txt,
> 17287.master.v4.txt, 17287.master.v5.txt, 17287.v2.txt
>
>
> We have seen an issue whereby if the HDFS is unstable and the HBase master's
> HDFS client is unable to stabilize before
> {{dfs.client.failover.max.attempts}} then the master's filesystem object
> closes. This seems to result in an HBase master which will continue to run
> (process and znode exists) but no meaningful work can be done (e.g. assigning
> meta).What we saw in our HBase master logs was:{code}2016-12-01 19:19:08,192
> ERROR org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler:
> Caught M_META_SERVER_SHUTDOWN, count=1java.io.IOException: failed log
> splitting for cluster-r5n12.bloomberg.com,60200,1480632863218, will retryat
> org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler.process(MetaServerShutdownHandler.java:84)at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at
> java.lang.Thread.run(Thread.java:745)Caused by: java.io.IOException:
> Filesystem closed{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-14417) Incremental backup and bulk loading
[ https://issues.apache.org/jira/browse/HBASE-14417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14417: --- Resolution: Fixed Status: Resolved (was: Patch Available) Thanks for the reviews, Josh and Vlad. > Incremental backup and bulk loading > --- > > Key: HBASE-14417 > URL: https://issues.apache.org/jira/browse/HBASE-14417 > Project: HBase > Issue Type: New Feature >Reporter: Vladimir Rodionov >Assignee: Ted Yu >Priority: Blocker > Labels: backup > Fix For: 2.0 > > Attachments: 14417-tbl-ext.v10.txt, 14417-tbl-ext.v11.txt, > 14417-tbl-ext.v14.txt, 14417-tbl-ext.v18.txt, 14417-tbl-ext.v19.txt, > 14417-tbl-ext.v20.txt, 14417-tbl-ext.v21.txt, 14417-tbl-ext.v22.txt, > 14417-tbl-ext.v23.txt, 14417-tbl-ext.v24.txt, 14417-tbl-ext.v9.txt, > 14417.v11.txt, 14417.v13.txt, 14417.v1.txt, 14417.v21.txt, 14417.v23.txt, > 14417.v24.txt, 14417.v25.txt, 14417.v2.txt, 14417.v6.txt > > > Currently, incremental backup is based on WAL files. Bulk data loading > bypasses WALs for obvious reasons, breaking incremental backups. The only way > to continue backups after bulk loading is to create new full backup of a > table. This may not be feasible for customers who do bulk loading regularly > (say, every day). > Here is the review board (out of date): > https://reviews.apache.org/r/54258/ > In order not to miss the hfiles which are loaded into region directories in a > situation where postBulkLoadHFile() hook is not called (bulk load being > interrupted), we record hfile names thru preCommitStoreFile() hook. > At time of incremental backup, we check the presence of such hfiles. If they > are present, they become part of the incremental backup image. > Here is review board: > https://reviews.apache.org/r/57790/ > Google doc for design: > https://docs.google.com/document/d/1ACCLsecHDvzVSasORgqqRNrloGx4mNYIbvAU7lq5lJE -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-14417) Incremental backup and bulk loading
[
https://issues.apache.org/jira/browse/HBASE-14417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946136#comment-15946136
]
Hadoop QA commented on HBASE-14417:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 46s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 4s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for
instructions. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
37s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 11s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 103m 5s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 142m 37s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12860929/14417-tbl-ext.v24.txt
|
| JIRA Issue | HBASE-14417 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 649abde09dfb 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / cb4fac1 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| whitespace |
https://builds.apache.org/job/PreCommit-HBASE-Build/6246/artifact/patchprocess/whitespace-eol.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6246/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6246/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Incremental backup and bulk loading
>
[jira] [Commented] (HBASE-14614) Procedure v2: Core Assignment Manager
[
https://issues.apache.org/jira/browse/HBASE-14614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946074#comment-15946074
]
Hadoop QA commented on HBASE-14614:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 80 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 39s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
22s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 11s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
33s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
46s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 8s
{color} | {color:red} hbase-protocol-shaded in master has 24 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 24s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 32s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 18s
{color} | {color:red} hbase-rsgroup in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 31s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 18s
{color} | {color:red} hbase-rsgroup in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 31s {color} |
{color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 18s {color} |
{color:red} hbase-rsgroup in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 31s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 18s {color}
| {color:red} hbase-rsgroup in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
2s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
30s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 129 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m 36s
{color} | {color:red} The patch causes 20 errors with Hadoop v2.6.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 2m 56s
{color} | {color:red} The patch causes 20 errors with Hadoop v2.6.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 4m 22s
{color} | {color:red} The patch causes 20 errors with Hadoop v2.6.3. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m 47s
{color} | {color:red} The patch causes 20 errors with Hadoop v2.6.4. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 7m 13s
{color} | {color:red} The patch causes 20 errors with Hadoop v2.6.5. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 8m 40s
{color} | {color:red} The patch causes 20 errors with Hadoop v2.7.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 10m 7s
{color} | {color:red} The patch causes 20 errors with Hadoop v2.7.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 11m 32s
{color} | {color:red} The patch causes 20 errors with Hadoop v2.7.3. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 12m 50s
[jira] [Updated] (HBASE-14614) Procedure v2: Core Assignment Manager
[ https://issues.apache.org/jira/browse/HBASE-14614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14614: -- Attachment: HBASE-14614.master.021.patch > Procedure v2: Core Assignment Manager > - > > Key: HBASE-14614 > URL: https://issues.apache.org/jira/browse/HBASE-14614 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Stephen Yuan Jiang >Assignee: Matteo Bertozzi > Fix For: 2.0.0 > > Attachments: HBASE-14614.master.001.patch, > HBASE-14614.master.002.patch, HBASE-14614.master.003.patch, > HBASE-14614.master.004.patch, HBASE-14614.master.005.patch, > HBASE-14614.master.006.patch, HBASE-14614.master.007.patch, > HBASE-14614.master.008.patch, HBASE-14614.master.009.patch, > HBASE-14614.master.010.patch, HBASE-14614.master.011.patch, > HBASE-14614.master.012.patch, HBASE-14614.master.012.patch, > HBASE-14614.master.013.patch, HBASE-14614.master.014.patch, > HBASE-14614.master.015.patch, HBASE-14614.master.016.patch, > HBASE-14614.master.017.patch, HBASE-14614.master.018.patch, > HBASE-14614.master.019.patch, HBASE-14614.master.020.patch, > HBASE-14614.master.021.patch > > > New AssignmentManager implemented using proc-v2. > - AssignProcedure handle assignment operation > - UnassignProcedure handle unassign operation > - MoveRegionProcedure handle move/balance operation > Concurrent Assign operations are batched together and sent to the balancer > Concurrent Assign and Unassign operation ready to be sent to the RS are > batched together > This patch is an intermediate state where we add the new AM as > AssignmentManager2() to the master, to be reached by tests. but the new AM > will not be integrated with the rest of the system. Only new am unit-tests > will exercise the new assigment manager. The integration with the master code > is part of HBASE-14616 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17726) [C++] Move implementation from header to cc for request retry
[ https://issues.apache.org/jira/browse/HBASE-17726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-17726: -- Attachment: hbase-17726_v1.patch Straightforward patch. [~xiaobingo] let me know if you have concerns with the patch. > [C++] Move implementation from header to cc for request retry > - > > Key: HBASE-17726 > URL: https://issues.apache.org/jira/browse/HBASE-17726 > Project: HBase > Issue Type: Sub-task >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: hbase-17726_v1.patch > > > This is a follow up work related to HBASE-17465. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-14614) Procedure v2: Core Assignment Manager
[ https://issues.apache.org/jira/browse/HBASE-14614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14614: -- Attachment: HBASE-14614.master.020.patch > Procedure v2: Core Assignment Manager > - > > Key: HBASE-14614 > URL: https://issues.apache.org/jira/browse/HBASE-14614 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Stephen Yuan Jiang >Assignee: Matteo Bertozzi > Fix For: 2.0.0 > > Attachments: HBASE-14614.master.001.patch, > HBASE-14614.master.002.patch, HBASE-14614.master.003.patch, > HBASE-14614.master.004.patch, HBASE-14614.master.005.patch, > HBASE-14614.master.006.patch, HBASE-14614.master.007.patch, > HBASE-14614.master.008.patch, HBASE-14614.master.009.patch, > HBASE-14614.master.010.patch, HBASE-14614.master.011.patch, > HBASE-14614.master.012.patch, HBASE-14614.master.012.patch, > HBASE-14614.master.013.patch, HBASE-14614.master.014.patch, > HBASE-14614.master.015.patch, HBASE-14614.master.016.patch, > HBASE-14614.master.017.patch, HBASE-14614.master.018.patch, > HBASE-14614.master.019.patch, HBASE-14614.master.020.patch > > > New AssignmentManager implemented using proc-v2. > - AssignProcedure handle assignment operation > - UnassignProcedure handle unassign operation > - MoveRegionProcedure handle move/balance operation > Concurrent Assign operations are batched together and sent to the balancer > Concurrent Assign and Unassign operation ready to be sent to the RS are > batched together > This patch is an intermediate state where we add the new AM as > AssignmentManager2() to the master, to be reached by tests. but the new AM > will not be integrated with the rest of the system. Only new am unit-tests > will exercise the new assigment manager. The integration with the master code > is part of HBASE-14616 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-14614) Procedure v2: Core Assignment Manager
[ https://issues.apache.org/jira/browse/HBASE-14614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945956#comment-15945956 ] stack commented on HBASE-14614: --- Fix split test (required filling in split handling facility) > Procedure v2: Core Assignment Manager > - > > Key: HBASE-14614 > URL: https://issues.apache.org/jira/browse/HBASE-14614 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Stephen Yuan Jiang >Assignee: Matteo Bertozzi > Fix For: 2.0.0 > > Attachments: HBASE-14614.master.001.patch, > HBASE-14614.master.002.patch, HBASE-14614.master.003.patch, > HBASE-14614.master.004.patch, HBASE-14614.master.005.patch, > HBASE-14614.master.006.patch, HBASE-14614.master.007.patch, > HBASE-14614.master.008.patch, HBASE-14614.master.009.patch, > HBASE-14614.master.010.patch, HBASE-14614.master.011.patch, > HBASE-14614.master.012.patch, HBASE-14614.master.012.patch, > HBASE-14614.master.013.patch, HBASE-14614.master.014.patch, > HBASE-14614.master.015.patch, HBASE-14614.master.016.patch, > HBASE-14614.master.017.patch, HBASE-14614.master.018.patch, > HBASE-14614.master.019.patch > > > New AssignmentManager implemented using proc-v2. > - AssignProcedure handle assignment operation > - UnassignProcedure handle unassign operation > - MoveRegionProcedure handle move/balance operation > Concurrent Assign operations are batched together and sent to the balancer > Concurrent Assign and Unassign operation ready to be sent to the RS are > batched together > This patch is an intermediate state where we add the new AM as > AssignmentManager2() to the master, to be reached by tests. but the new AM > will not be integrated with the rest of the system. Only new am unit-tests > will exercise the new assigment manager. The integration with the master code > is part of HBASE-14616 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-14614) Procedure v2: Core Assignment Manager
[ https://issues.apache.org/jira/browse/HBASE-14614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14614: -- Attachment: HBASE-14614.master.019.patch > Procedure v2: Core Assignment Manager > - > > Key: HBASE-14614 > URL: https://issues.apache.org/jira/browse/HBASE-14614 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Stephen Yuan Jiang >Assignee: Matteo Bertozzi > Fix For: 2.0.0 > > Attachments: HBASE-14614.master.001.patch, > HBASE-14614.master.002.patch, HBASE-14614.master.003.patch, > HBASE-14614.master.004.patch, HBASE-14614.master.005.patch, > HBASE-14614.master.006.patch, HBASE-14614.master.007.patch, > HBASE-14614.master.008.patch, HBASE-14614.master.009.patch, > HBASE-14614.master.010.patch, HBASE-14614.master.011.patch, > HBASE-14614.master.012.patch, HBASE-14614.master.012.patch, > HBASE-14614.master.013.patch, HBASE-14614.master.014.patch, > HBASE-14614.master.015.patch, HBASE-14614.master.016.patch, > HBASE-14614.master.017.patch, HBASE-14614.master.018.patch, > HBASE-14614.master.019.patch > > > New AssignmentManager implemented using proc-v2. > - AssignProcedure handle assignment operation > - UnassignProcedure handle unassign operation > - MoveRegionProcedure handle move/balance operation > Concurrent Assign operations are batched together and sent to the balancer > Concurrent Assign and Unassign operation ready to be sent to the RS are > batched together > This patch is an intermediate state where we add the new AM as > AssignmentManager2() to the master, to be reached by tests. but the new AM > will not be integrated with the rest of the system. Only new am unit-tests > will exercise the new assigment manager. The integration with the master code > is part of HBASE-14616 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HBASE-16775) Flakey test with TestExportSnapshot#testExportRetry and TestMobExportSnapshot#testExportRetry
[
https://issues.apache.org/jira/browse/HBASE-16775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huaxiang sun reassigned HBASE-16775:
Assignee: Appy (was: huaxiang sun)
> Flakey test with TestExportSnapshot#testExportRetry and
> TestMobExportSnapshot#testExportRetry
> --
>
> Key: HBASE-16775
> URL: https://issues.apache.org/jira/browse/HBASE-16775
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: Appy
> Attachments: disable.patch, HBASE-16775.master.001.patch,
> HBASE-16775.master.002.patch, HBASE-16775.master.003.patch,
> HBASE-16775.master.004.patch
>
>
> The root cause is that conf.setInt("mapreduce.map.maxattempts", 10) is not
> taken by the mapper job, so the retry is actually 0. Debugging to see why
> this is the case.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-14141) HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits from backup tables
[
https://issues.apache.org/jira/browse/HBASE-14141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945949#comment-15945949
]
Ted Yu commented on HBASE-14141:
{code}
- public HashMap getIncrBackupLogFileList(Connection conn,
BackupInfo backupInfo)
+ public HashMap getIncrBackupLogFileMap(/*Connection conn ,
BackupInfo backupInfo*/)
{code}
Drop commented out code.
{code}
+ /**
+ * TODO
+ * @param conn
+ * @param backupInfo
+ * @return
+ * @throws IOException
+ */
+ public List getIncrBackupLogFileList()
{code}
Complete javadoc.
> HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits
> from backup tables
>
>
> Key: HBASE-14141
> URL: https://issues.apache.org/jira/browse/HBASE-14141
> Project: HBase
> Issue Type: New Feature
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Blocker
> Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-14141.HBASE-14123.v1.patch, HBASE-14141.v1.patch,
> HBASE-14141.v2.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (HBASE-17727) [C++] Make RespConverter work with RawAsyncTableImpl
[
https://issues.apache.org/jira/browse/HBASE-17727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar resolved HBASE-17727.
---
Resolution: Fixed
Fix Version/s: HBASE-14850
> [C++] Make RespConverter work with RawAsyncTableImpl
>
>
> Key: HBASE-17727
> URL: https://issues.apache.org/jira/browse/HBASE-17727
> Project: HBase
> Issue Type: Sub-task
>Reporter: Xiaobing Zhou
>Assignee: Enis Soztutar
> Fix For: HBASE-14850
>
> Attachments: hbase-17727-v1.patch
>
>
> This is a follow up work of HBASE-17465.
> There's a problem to dereference instance of RpcCallback when it's passed as
> function argument.
> {code}
> template typename S>
> using RespConverter = std::function;
> {code}
> {code}
> templatetypename PREQ,
>typename PRESP,
>typename RESP>
> folly::Future Call(
> std::shared_ptr rpc_client,
> std::shared_ptr controller,
> std::shared_ptr loc,
> const REQ& req,
> const ReqConverter&
> req_converter,
> const hbase::RpcCall& rpc_call,
> const RespConverter& resp_converter) {
> rpc_call(
> rpc_client,
> loc,
> controller,
> std::move(req_converter(req, loc->region_name(
> .then([&, this](std::unique_ptr presp) {
> // std::unique_ptr result = resp_converter(presp);
> std::unique_ptr result =
> hbase::ResponseConverter::FromGetResponse(*presp);
> promise_->setValue(std::move(*result));
> })
> .onError([this] (const std::exception& e)
> {promise_->setException(e);});
> return promise_->getFuture();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17727) [C++] Make RespConverter work with RawAsyncTableImpl
[
https://issues.apache.org/jira/browse/HBASE-17727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945945#comment-15945945
]
Enis Soztutar commented on HBASE-17727:
---
Pushed to branch.
> [C++] Make RespConverter work with RawAsyncTableImpl
>
>
> Key: HBASE-17727
> URL: https://issues.apache.org/jira/browse/HBASE-17727
> Project: HBase
> Issue Type: Sub-task
>Reporter: Xiaobing Zhou
>Assignee: Enis Soztutar
> Fix For: HBASE-14850
>
> Attachments: hbase-17727-v1.patch
>
>
> This is a follow up work of HBASE-17465.
> There's a problem to dereference instance of RpcCallback when it's passed as
> function argument.
> {code}
> template typename S>
> using RespConverter = std::function;
> {code}
> {code}
> templatetypename PREQ,
>typename PRESP,
>typename RESP>
> folly::Future Call(
> std::shared_ptr rpc_client,
> std::shared_ptr controller,
> std::shared_ptr loc,
> const REQ& req,
> const ReqConverter&
> req_converter,
> const hbase::RpcCall& rpc_call,
> const RespConverter& resp_converter) {
> rpc_call(
> rpc_client,
> loc,
> controller,
> std::move(req_converter(req, loc->region_name(
> .then([&, this](std::unique_ptr presp) {
> // std::unique_ptr result = resp_converter(presp);
> std::unique_ptr result =
> hbase::ResponseConverter::FromGetResponse(*presp);
> promise_->setValue(std::move(*result));
> })
> .onError([this] (const std::exception& e)
> {promise_->setException(e);});
> return promise_->getFuture();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-14141) HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits from backup tables
[ https://issues.apache.org/jira/browse/HBASE-14141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945922#comment-15945922 ] Ted Yu commented on HBASE-14141: I don't see new test(s). See if you can add test coverage. > HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits > from backup tables > > > Key: HBASE-14141 > URL: https://issues.apache.org/jira/browse/HBASE-14141 > Project: HBase > Issue Type: New Feature >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Blocker > Labels: backup > Fix For: 2.0.0 > > Attachments: HBASE-14141.HBASE-14123.v1.patch, HBASE-14141.v1.patch, > HBASE-14141.v2.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-14141) HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits from backup tables
[ https://issues.apache.org/jira/browse/HBASE-14141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945919#comment-15945919 ] Ted Yu commented on HBASE-14141: See if you can put the handling of missing WAL file(s) inside WALPlayer. Please put the next patch on reviewboard. Thanks > HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits > from backup tables > > > Key: HBASE-14141 > URL: https://issues.apache.org/jira/browse/HBASE-14141 > Project: HBase > Issue Type: New Feature >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Blocker > Labels: backup > Fix For: 2.0.0 > > Attachments: HBASE-14141.HBASE-14123.v1.patch, HBASE-14141.v1.patch, > HBASE-14141.v2.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-14417) Incremental backup and bulk loading
[ https://issues.apache.org/jira/browse/HBASE-14417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14417: --- Hadoop Flags: Reviewed > Incremental backup and bulk loading > --- > > Key: HBASE-14417 > URL: https://issues.apache.org/jira/browse/HBASE-14417 > Project: HBase > Issue Type: New Feature >Reporter: Vladimir Rodionov >Assignee: Ted Yu >Priority: Blocker > Labels: backup > Fix For: 2.0 > > Attachments: 14417-tbl-ext.v10.txt, 14417-tbl-ext.v11.txt, > 14417-tbl-ext.v14.txt, 14417-tbl-ext.v18.txt, 14417-tbl-ext.v19.txt, > 14417-tbl-ext.v20.txt, 14417-tbl-ext.v21.txt, 14417-tbl-ext.v22.txt, > 14417-tbl-ext.v23.txt, 14417-tbl-ext.v24.txt, 14417-tbl-ext.v9.txt, > 14417.v11.txt, 14417.v13.txt, 14417.v1.txt, 14417.v21.txt, 14417.v23.txt, > 14417.v24.txt, 14417.v25.txt, 14417.v2.txt, 14417.v6.txt > > > Currently, incremental backup is based on WAL files. Bulk data loading > bypasses WALs for obvious reasons, breaking incremental backups. The only way > to continue backups after bulk loading is to create new full backup of a > table. This may not be feasible for customers who do bulk loading regularly > (say, every day). > Here is the review board (out of date): > https://reviews.apache.org/r/54258/ > In order not to miss the hfiles which are loaded into region directories in a > situation where postBulkLoadHFile() hook is not called (bulk load being > interrupted), we record hfile names thru preCommitStoreFile() hook. > At time of incremental backup, we check the presence of such hfiles. If they > are present, they become part of the incremental backup image. > Here is review board: > https://reviews.apache.org/r/57790/ > Google doc for design: > https://docs.google.com/document/d/1ACCLsecHDvzVSasORgqqRNrloGx4mNYIbvAU7lq5lJE -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (HBASE-17842) Reference book - HBase application integration test is not correct
[ https://issues.apache.org/jira/browse/HBASE-17842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser resolved HBASE-17842. Resolution: Invalid Agreed, Jan. [~iraj.hedayati], please feel free to contact Cloudera to report any documentation issues on their website. This JIRA project is for Apache HBase. > Reference book - HBase application integration test is not correct > -- > > Key: HBASE-17842 > URL: https://issues.apache.org/jira/browse/HBASE-17842 > Project: HBase > Issue Type: Bug > Components: documentation >Reporter: Iraj Hedayati >Priority: Critical > > The directions in the following section are not correct. > https://archive.cloudera.com/cdh5/cdh/5/hbase-1.2.0-cdh5.8.0/book.html#_integration_testing_with_an_hbase_mini_cluster > It is pointing to HBase 0.98.3 and `HBaseTestingUtility` can not be resolved > in 1.2.0 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-14417) Incremental backup and bulk loading
[ https://issues.apache.org/jira/browse/HBASE-14417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14417: --- Attachment: 14417-tbl-ext.v24.txt Patch v24 creates TestIncrementalBackupWithBulkLoad for testing bulk load support. > Incremental backup and bulk loading > --- > > Key: HBASE-14417 > URL: https://issues.apache.org/jira/browse/HBASE-14417 > Project: HBase > Issue Type: New Feature >Reporter: Vladimir Rodionov >Assignee: Ted Yu >Priority: Blocker > Labels: backup > Fix For: 2.0 > > Attachments: 14417-tbl-ext.v10.txt, 14417-tbl-ext.v11.txt, > 14417-tbl-ext.v14.txt, 14417-tbl-ext.v18.txt, 14417-tbl-ext.v19.txt, > 14417-tbl-ext.v20.txt, 14417-tbl-ext.v21.txt, 14417-tbl-ext.v22.txt, > 14417-tbl-ext.v23.txt, 14417-tbl-ext.v24.txt, 14417-tbl-ext.v9.txt, > 14417.v11.txt, 14417.v13.txt, 14417.v1.txt, 14417.v21.txt, 14417.v23.txt, > 14417.v24.txt, 14417.v25.txt, 14417.v2.txt, 14417.v6.txt > > > Currently, incremental backup is based on WAL files. Bulk data loading > bypasses WALs for obvious reasons, breaking incremental backups. The only way > to continue backups after bulk loading is to create new full backup of a > table. This may not be feasible for customers who do bulk loading regularly > (say, every day). > Here is the review board (out of date): > https://reviews.apache.org/r/54258/ > In order not to miss the hfiles which are loaded into region directories in a > situation where postBulkLoadHFile() hook is not called (bulk load being > interrupted), we record hfile names thru preCommitStoreFile() hook. > At time of incremental backup, we check the presence of such hfiles. If they > are present, they become part of the incremental backup image. > Here is review board: > https://reviews.apache.org/r/57790/ > Google doc for design: > https://docs.google.com/document/d/1ACCLsecHDvzVSasORgqqRNrloGx4mNYIbvAU7lq5lJE -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HBASE-17215) Separate small/large file delete threads in HFileCleaner to accelerate hfile cleanup speed
[
https://issues.apache.org/jira/browse/HBASE-17215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945887#comment-15945887
]
huaxiang sun edited comment on HBASE-17215 at 3/28/17 8:25 PM:
---
Hi [~carp84], we are seeing a case that cleaner is not fast enough to free up
the disk space. Thought this jira can help in speeding up disk space clean. Do
you mind post the patch? Thanks!
was (Author: huaxiang):
Hi [~carp84], we are seeing a case that cleaner is fast enough to free up the
disk space. Thought this jira can help in speeding up disk space clean. Do you
mind post the patch? Thanks!
> Separate small/large file delete threads in HFileCleaner to accelerate hfile
> cleanup speed
> --
>
> Key: HBASE-17215
> URL: https://issues.apache.org/jira/browse/HBASE-17215
> Project: HBase
> Issue Type: Improvement
>Reporter: Yu Li
>Assignee: Yu Li
>
> When using PCIe-SSD the flush speed will be really quick, and although we
> have per CF flush, we still have the
> {{hbase.regionserver.optionalcacheflushinterval}} setting and some other
> mechanism to avoid data kept in memory for too long to flush small hfiles. In
> our online environment we found the single thread cleaner kept cleaning
> earlier flushed small files while large files got no chance, which caused
> disk full then many other problems.
> Deleting hfiles in parallel with too many threads will also increase the
> workload of namenode, so here we propose to separate large/small hfile
> cleaner threads just like we do for compaction, and it turned out to work
> well in our cluster.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17215) Separate small/large file delete threads in HFileCleaner to accelerate hfile cleanup speed
[
https://issues.apache.org/jira/browse/HBASE-17215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945887#comment-15945887
]
huaxiang sun commented on HBASE-17215:
--
Hi [~carp84], we are seeing a case that cleaner is fast enough to free up the
disk space. Thought this jira can help in speeding up disk space clean. Do you
mind post the patch? Thanks!
> Separate small/large file delete threads in HFileCleaner to accelerate hfile
> cleanup speed
> --
>
> Key: HBASE-17215
> URL: https://issues.apache.org/jira/browse/HBASE-17215
> Project: HBase
> Issue Type: Improvement
>Reporter: Yu Li
>Assignee: Yu Li
>
> When using PCIe-SSD the flush speed will be really quick, and although we
> have per CF flush, we still have the
> {{hbase.regionserver.optionalcacheflushinterval}} setting and some other
> mechanism to avoid data kept in memory for too long to flush small hfiles. In
> our online environment we found the single thread cleaner kept cleaning
> earlier flushed small files while large files got no chance, which caused
> disk full then many other problems.
> Deleting hfiles in parallel with too many threads will also increase the
> workload of namenode, so here we propose to separate large/small hfile
> cleaner threads just like we do for compaction, and it turned out to work
> well in our cluster.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17765) Reviving the merge possibility in the CompactingMemStore
[ https://issues.apache.org/jira/browse/HBASE-17765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945809#comment-15945809 ] Hudson commented on HBASE-17765: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #2756 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2756/]) HBASE-17765 Reviving the merge of the compacting pipeline (busbey: rev cb4fac1d18660094908afd6abaace6e441ec6a49) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactingMemStore.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestWalAndCompactingMemStoreFlush.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ImmutableSegment.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionPipeline.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreCompactor.java > Reviving the merge possibility in the CompactingMemStore > > > Key: HBASE-17765 > URL: https://issues.apache.org/jira/browse/HBASE-17765 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Fix For: 2.0.0 > > Attachments: HBASE-17765-V01.patch, HBASE-17765-V02.patch, > HBASE-17765-V03.patch, HBASE-17765-V04.patch, HBASE-17765-V05.patch > > > According to the new performance results presented in the HBASE-16417 we see > that the read latency of the 90th percentile of the BASIC policy is too big > due to the need to traverse through too many segments in the pipeline. In > this JIRA we correct the bug in the merge sizing calculations and allow > pipeline size threshold to be a configurable parameter. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17287) Master becomes a zombie if filesystem object closes
[
https://issues.apache.org/jira/browse/HBASE-17287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945753#comment-15945753
]
Ted Yu commented on HBASE-17287:
bq. with the default setup (master hosting meta table and regionserver) that
master abort is not causing the daemon to go down
There shouldn't be problem in the above scenario.
> Master becomes a zombie if filesystem object closes
> ---
>
> Key: HBASE-17287
> URL: https://issues.apache.org/jira/browse/HBASE-17287
> Project: HBase
> Issue Type: Bug
> Components: master
>Reporter: Clay B.
>Assignee: Ted Yu
> Fix For: 1.4.0, 2.0
>
> Attachments: 17287.branch-1.1.v4.txt, 17287.branch-1.v3.txt,
> 17287.branch-1.v4.txt, 17287.master.v2.txt, 17287.master.v3.txt,
> 17287.master.v4.txt, 17287.master.v5.txt, 17287.v2.txt
>
>
> We have seen an issue whereby if the HDFS is unstable and the HBase master's
> HDFS client is unable to stabilize before
> {{dfs.client.failover.max.attempts}} then the master's filesystem object
> closes. This seems to result in an HBase master which will continue to run
> (process and znode exists) but no meaningful work can be done (e.g. assigning
> meta).What we saw in our HBase master logs was:{code}2016-12-01 19:19:08,192
> ERROR org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler:
> Caught M_META_SERVER_SHUTDOWN, count=1java.io.IOException: failed log
> splitting for cluster-r5n12.bloomberg.com,60200,1480632863218, will retryat
> org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler.process(MetaServerShutdownHandler.java:84)at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at
> java.lang.Thread.run(Thread.java:745)Caused by: java.io.IOException:
> Filesystem closed{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17840) Update book
[
https://issues.apache.org/jira/browse/HBASE-17840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945742#comment-15945742
]
Hadoop QA commented on HBASE-17840:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
36s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
24s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 11s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
8s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 50s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 9s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 120m 44s
{color} | {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 164m 27s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12860895/HBASE-17840.002.patch
|
| JIRA Issue | HBASE-17840 |
| Optional Tests | asflicense javac javadoc unit |
| uname | Linux cbe036dbe92d 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / cb4fac1 |
| Default Java | 1.8.0_121 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/6245/artifact/patchprocess/patch-unit-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6245/testReport/ |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6245/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Update book
> ---
>
> Key: HBASE-17840
> URL: https://issues.apache.org/jira/browse/HBASE-17840
> Project: HBase
> Issue Type: Sub-task
>Reporter: Josh Elser
>Assignee: Josh Elser
> Attachments: HBASE-17840.001.patch, HBASE-17840.002.patch
>
>
> Need to update the book to include the new feature.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17771) [C++] Classes required for implementation of BatchCallerBuilder
[ https://issues.apache.org/jira/browse/HBASE-17771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945729#comment-15945729 ] Enis Soztutar commented on HBASE-17771: --- [~sudeeps] do you want me to commit this, and we do a follow up jira for the thread safety, or you want to do it in this patch. Either works, just let me know which is more convenient for you. > [C++] Classes required for implementation of BatchCallerBuilder > --- > > Key: HBASE-17771 > URL: https://issues.apache.org/jira/browse/HBASE-17771 > Project: HBase > Issue Type: Sub-task >Reporter: Sudeep Sunthankar >Assignee: Sudeep Sunthankar > Fix For: HBASE-14850 > > Attachments: HBASE-17771.HBASE-14850.v1.patch, > HBASE-17771.HBASE-14850.v2.patch, HBASE-17771.HBASE-14850.v3.patch, > HBASE-17771.HBASE-14850.v4.patch, HBASE-17771.HBASE-14850.v5.patch > > > Separating depedencies of BatchCallerBuilder. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17287) Master becomes a zombie if filesystem object closes
[
https://issues.apache.org/jira/browse/HBASE-17287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945725#comment-15945725
]
Ted Yu commented on HBASE-17287:
bq. The timeout of 60 secs
When I ran the new test, it took ~17 seconds. 60 should be long enough even on
a slow machine.
> Master becomes a zombie if filesystem object closes
> ---
>
> Key: HBASE-17287
> URL: https://issues.apache.org/jira/browse/HBASE-17287
> Project: HBase
> Issue Type: Bug
> Components: master
>Reporter: Clay B.
>Assignee: Ted Yu
> Fix For: 1.4.0, 2.0
>
> Attachments: 17287.branch-1.1.v4.txt, 17287.branch-1.v3.txt,
> 17287.branch-1.v4.txt, 17287.master.v2.txt, 17287.master.v3.txt,
> 17287.master.v4.txt, 17287.master.v5.txt, 17287.v2.txt
>
>
> We have seen an issue whereby if the HDFS is unstable and the HBase master's
> HDFS client is unable to stabilize before
> {{dfs.client.failover.max.attempts}} then the master's filesystem object
> closes. This seems to result in an HBase master which will continue to run
> (process and znode exists) but no meaningful work can be done (e.g. assigning
> meta).What we saw in our HBase master logs was:{code}2016-12-01 19:19:08,192
> ERROR org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler:
> Caught M_META_SERVER_SHUTDOWN, count=1java.io.IOException: failed log
> splitting for cluster-r5n12.bloomberg.com,60200,1480632863218, will retryat
> org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler.process(MetaServerShutdownHandler.java:84)at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at
> java.lang.Thread.run(Thread.java:745)Caused by: java.io.IOException:
> Filesystem closed{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-14141) HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits from backup tables
[ https://issues.apache.org/jira/browse/HBASE-14141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945710#comment-15945710 ] Vladimir Rodionov commented on HBASE-14141: --- [~te...@apache.org], can you take a look? > HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits > from backup tables > > > Key: HBASE-14141 > URL: https://issues.apache.org/jira/browse/HBASE-14141 > Project: HBase > Issue Type: New Feature >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Blocker > Labels: backup > Fix For: 2.0.0 > > Attachments: HBASE-14141.HBASE-14123.v1.patch, HBASE-14141.v1.patch, > HBASE-14141.v2.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-14141) HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits from backup tables
[ https://issues.apache.org/jira/browse/HBASE-14141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-14141: -- Fix Version/s: (was: HBASE-7912) 2.0.0 > HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits > from backup tables > > > Key: HBASE-14141 > URL: https://issues.apache.org/jira/browse/HBASE-14141 > Project: HBase > Issue Type: New Feature >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Blocker > Labels: backup > Fix For: 2.0.0 > > Attachments: HBASE-14141.HBASE-14123.v1.patch, HBASE-14141.v1.patch, > HBASE-14141.v2.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17287) Master becomes a zombie if filesystem object closes
[
https://issues.apache.org/jira/browse/HBASE-17287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945696#comment-15945696
]
Enis Soztutar commented on HBASE-17287:
---
bq. In patch v5, before starting the mini cluster, I set config for master not
to host meta region.
Ok makes sense. I've checked the test again, seems good. The timeout of 60 secs
is aggressive I think though. Let's bump that to 3 mins. In Jenkins things can
run super slow causing flakiness.
On the other issue, do we have a problem with the default setup (master hosting
meta table and regionserver) that master abort is not causing the daemon to go
down? If we are good there, +1 for the patch.
> Master becomes a zombie if filesystem object closes
> ---
>
> Key: HBASE-17287
> URL: https://issues.apache.org/jira/browse/HBASE-17287
> Project: HBase
> Issue Type: Bug
> Components: master
>Reporter: Clay B.
>Assignee: Ted Yu
> Fix For: 1.4.0, 2.0
>
> Attachments: 17287.branch-1.1.v4.txt, 17287.branch-1.v3.txt,
> 17287.branch-1.v4.txt, 17287.master.v2.txt, 17287.master.v3.txt,
> 17287.master.v4.txt, 17287.master.v5.txt, 17287.v2.txt
>
>
> We have seen an issue whereby if the HDFS is unstable and the HBase master's
> HDFS client is unable to stabilize before
> {{dfs.client.failover.max.attempts}} then the master's filesystem object
> closes. This seems to result in an HBase master which will continue to run
> (process and znode exists) but no meaningful work can be done (e.g. assigning
> meta).What we saw in our HBase master logs was:{code}2016-12-01 19:19:08,192
> ERROR org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler:
> Caught M_META_SERVER_SHUTDOWN, count=1java.io.IOException: failed log
> splitting for cluster-r5n12.bloomberg.com,60200,1480632863218, will retryat
> org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler.process(MetaServerShutdownHandler.java:84)at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at
> java.lang.Thread.run(Thread.java:745)Caused by: java.io.IOException:
> Filesystem closed{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17841) ServerCrashProcedure is not triggered when meta server with unflushed edits is aborted
[
https://issues.apache.org/jira/browse/HBASE-17841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945647#comment-15945647
]
Stephen Yuan Jiang commented on HBASE-17841:
Yeah, I think this is the root cause: in branch-1, meta is hosted in a RS; and
in master, meta is hosted in master.
> ServerCrashProcedure is not triggered when meta server with unflushed edits
> is aborted
> --
>
> Key: HBASE-17841
> URL: https://issues.apache.org/jira/browse/HBASE-17841
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0
>Reporter: Ted Yu
> Attachments: 17841.tst
>
>
> When writing unit test for HBASE-17287, I noticed that the wait for master to
> come down after hdfs enters safe mode times out (where meta server still has
> unflushed edits).
> The same test in branch-1 passes fine.
> Looking at
> org.apache.hadoop.hbase.master.procedure.TestSafemodeBringsDownMaster-output.txt
> , I don't see occurrence of ServerCrashProcedure.
> While in branch-1, there is something similar to the following:
> {code}
> at org.apache.hadoop.hdfs.DFSClient.rename(DFSClient.java:1661)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.rename(DistributedFileSystem.java:525)
> at
> org.apache.hadoop.hbase.master.MasterFileSystem.getLogDirs(MasterFileSystem.java:364)
> at
> org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:429)
> at
> org.apache.hadoop.hbase.master.MasterFileSystem.splitMetaLog(MasterFileSystem.java:343)
> at
> org.apache.hadoop.hbase.master.MasterFileSystem.splitMetaLog(MasterFileSystem.java:334)
> at
> org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure.processMeta(ServerCrashProcedure.java:351)
> at
> org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure.executeFromState(ServerCrashProcedure.java:239)
> at
> org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure.executeFromState(ServerCrashProcedure.java:73)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:139)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:506)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1152)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17842) Reference book - HBase application integration test is not correct
[ https://issues.apache.org/jira/browse/HBASE-17842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945636#comment-15945636 ] Jan Hentschel commented on HBASE-17842: --- It seems to be a problem with the Cloudera documentation. The current master only refers to the *hbase-testing-util* in the pom.xml example. The same is true for [branch-1.2|https://github.com/apache/hbase/blob/branch-1.2/src/main/asciidoc/_chapters/unit_testing.adoc]. > Reference book - HBase application integration test is not correct > -- > > Key: HBASE-17842 > URL: https://issues.apache.org/jira/browse/HBASE-17842 > Project: HBase > Issue Type: Bug > Components: documentation >Reporter: Iraj Hedayati >Priority: Critical > > The directions in the following section are not correct. > https://archive.cloudera.com/cdh5/cdh/5/hbase-1.2.0-cdh5.8.0/book.html#_integration_testing_with_an_hbase_mini_cluster > It is pointing to HBase 0.98.3 and `HBaseTestingUtility` can not be resolved > in 1.2.0 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17840) Update book
[ https://issues.apache.org/jira/browse/HBASE-17840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-17840: --- Attachment: HBASE-17840.002.patch .002 Goofed up the block-formatting > Update book > --- > > Key: HBASE-17840 > URL: https://issues.apache.org/jira/browse/HBASE-17840 > Project: HBase > Issue Type: Sub-task >Reporter: Josh Elser >Assignee: Josh Elser > Attachments: HBASE-17840.001.patch, HBASE-17840.002.patch > > > Need to update the book to include the new feature. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17765) Reviving the merge possibility in the CompactingMemStore
[ https://issues.apache.org/jira/browse/HBASE-17765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945442#comment-15945442 ] Sean Busbey commented on HBASE-17765: - I reverted c77e2135db07b6417f5fea4577c2c7ae8d6d7008 and repushed the change after correcting the commit message to reference this JIRA. Please ensure commit messages start with the JIRA ID of the issue they're addressing. > Reviving the merge possibility in the CompactingMemStore > > > Key: HBASE-17765 > URL: https://issues.apache.org/jira/browse/HBASE-17765 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Fix For: 2.0.0 > > Attachments: HBASE-17765-V01.patch, HBASE-17765-V02.patch, > HBASE-17765-V03.patch, HBASE-17765-V04.patch, HBASE-17765-V05.patch > > > According to the new performance results presented in the HBASE-16417 we see > that the read latency of the 90th percentile of the BASIC policy is too big > due to the need to traverse through too many segments in the pipeline. In > this JIRA we correct the bug in the merge sizing calculations and allow > pipeline size threshold to be a configurable parameter. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17834) close stale github PRs
[ https://issues.apache.org/jira/browse/HBASE-17834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-17834: Resolution: Fixed Status: Resolved (was: Patch Available) thanks for the review! BTW, it doesn't look like PR #40 reopened. > close stale github PRs > -- > > Key: HBASE-17834 > URL: https://issues.apache.org/jira/browse/HBASE-17834 > Project: HBase > Issue Type: Task > Components: community >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Minor > Fix For: 2.0.0 > > > time for another round of cleanup on PRs > please list potentials w/reasoning in comments. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17834) close stale github PRs
[ https://issues.apache.org/jira/browse/HBASE-17834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945418#comment-15945418 ] stack commented on HBASE-17834: --- +1 > close stale github PRs > -- > > Key: HBASE-17834 > URL: https://issues.apache.org/jira/browse/HBASE-17834 > Project: HBase > Issue Type: Task > Components: community >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Minor > Fix For: 2.0.0 > > > time for another round of cleanup on PRs > please list potentials w/reasoning in comments. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17834) close stale github PRs
[ https://issues.apache.org/jira/browse/HBASE-17834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945375#comment-15945375 ] Sean Busbey commented on HBASE-17834: - bump. Addendum look good to anyone? > close stale github PRs > -- > > Key: HBASE-17834 > URL: https://issues.apache.org/jira/browse/HBASE-17834 > Project: HBase > Issue Type: Task > Components: community >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Minor > Fix For: 2.0.0 > > > time for another round of cleanup on PRs > please list potentials w/reasoning in comments. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it
[ https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945040#comment-15945040 ] Anastasia Braginsky commented on HBASE-16438: - OK, I see from RB that many things are changed in the new patch. Let us see the new patch and review it again. In general, I am OK if we commit everything not related to performance here. Then all the issues with GC performance (as soft ref) we can do later. This commit should be correct, and performance issues we can deal with later. And BIG BIG BIG thanks to [~ram_krish] for doing amazing work here! And to [~anoop.hbase] for great ideas and reviews! > Create a cell type so that chunk id is embedded in it > - > > Key: HBASE-16438 > URL: https://issues.apache.org/jira/browse/HBASE-16438 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Attachments: HBASE-16438_1.patch, > HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, > HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, HBASE-16438.patch, > MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, > MemstoreChunkCell_trunk.patch > > > For CellChunkMap we may need a cell such that the chunk out of which it was > created, the id of the chunk be embedded in it so that when doing flattening > we can use the chunk id as a meta data. More details will follow once the > initial tasks are completed. > Why we need to embed the chunkid in the Cell is described by [~anastas] in > this remark over in parent issue > https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17125) Inconsistent result when use filter to read data
[
https://issues.apache.org/jira/browse/HBASE-17125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945009#comment-15945009
]
ramkrishna.s.vasudevan commented on HBASE-17125:
W.r.t to the change in the visibility controller it seems fine because instead
of getting the maxVersions that was set in the Get/scan you are passing it now
as the count param via the CPs..
bq.Set the maximum versions of each column will be check by filter. When use
filter, the get
257* maybe return less than maximum versions for each column.
Better to rephrase this like
'Sets the maximum versions of each column that will be checked by the filters.
When filters are in use, the get may return less than maximum versions for each
column'.
The rest of the patch need to see.
> Inconsistent result when use filter to read data
>
>
> Key: HBASE-17125
> URL: https://issues.apache.org/jira/browse/HBASE-17125
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: example.diff, HBASE-17125.master.001.patch,
> HBASE-17125.master.002.patch, HBASE-17125.master.002.patch,
> HBASE-17125.master.003.patch, HBASE-17125.master.004.patch,
> HBASE-17125.master.005.patch, HBASE-17125.master.006.patch
>
>
> Assume a cloumn's max versions is 3, then we write 4 versions of this column.
> The oldest version doesn't remove immediately. But from the user view, the
> oldest version has gone. When user use a filter to query, if the filter skip
> a new version, then the oldest version will be seen again. But after compact
> the region, then the oldest version will never been seen. So it is weird for
> user. The query will get inconsistent result before and after region
> compaction.
> The reason is matchColumn method of UserScanQueryMatcher. It first check the
> cell by filter, then check the number of versions needed. So if the filter
> skip the new version, then the oldest version will be seen again when it is
> not removed.
> Have a discussion offline with [~Apache9] and [~fenghh], now we have two
> solution for this problem. The first idea is check the number of versions
> first, then check the cell by filter. As the comment of setFilter, the filter
> is called after all tests for ttl, column match, deletes and max versions
> have been run.
> {code}
> /**
>* Apply the specified server-side filter when performing the Query.
>* Only {@link Filter#filterKeyValue(Cell)} is called AFTER all tests
>* for ttl, column match, deletes and max versions have been run.
>* @param filter filter to run on the server
>* @return this for invocation chaining
>*/
> public Query setFilter(Filter filter) {
> this.filter = filter;
> return this;
> }
> {code}
> But this idea has another problem, if a column's max version is 5 and the
> user query only need 3 versions. It first check the version's number, then
> check the cell by filter. So the cells number of the result may less than 3.
> But there are 2 versions which don't read anymore.
> So the second idea has three steps.
> 1. check by the max versions of this column
> 2. check the kv by filter
> 3. check the versions which user need.
> But this will lead the ScanQueryMatcher more complicated. And this will break
> the javadoc of Query.setFilter.
> Now we don't have a final solution for this problem. Suggestions are welcomed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HBASE-17436) Add facility to provide more information for Other Regions seen on Master UI
[ https://issues.apache.org/jira/browse/HBASE-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janos Gub reassigned HBASE-17436: - Assignee: Janos Gub > Add facility to provide more information for Other Regions seen on Master UI > > > Key: HBASE-17436 > URL: https://issues.apache.org/jira/browse/HBASE-17436 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Janos Gub > Labels: ui > > [~rpednekar] and I were looking at a case where the count displayed under > Other Regions was high (~1200). > Since the table page just maintains a count instead of List of region names, > it is very difficult for user to determine which regions belong to this > category. > We should provide facility to provide more details for this category (LOG, > JMX, etc). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17831) Support small scan in thrift2
[ https://issues.apache.org/jira/browse/HBASE-17831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944995#comment-15944995 ] Hudson commented on HBASE-17831: SUCCESS: Integrated in Jenkins build HBase-1.4 #682 (See [https://builds.apache.org/job/HBase-1.4/682/]) HBASE-17831 Support small scan in thrift2 (Guangxu Cheng) (tedyu: rev 6fe3b5e0f116d581039054d7e905d6bece23c24d) * (edit) hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/ThriftUtilities.java * (edit) hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TScan.java * (edit) hbase-thrift/src/test/java/org/apache/hadoop/hbase/thrift2/TestThriftHBaseServiceHandler.java * (edit) hbase-thrift/src/main/resources/org/apache/hadoop/hbase/thrift2/hbase.thrift > Support small scan in thrift2 > - > > Key: HBASE-17831 > URL: https://issues.apache.org/jira/browse/HBASE-17831 > Project: HBase > Issue Type: Improvement > Components: Thrift >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng > Fix For: 1.4.0, 2.0 > > Attachments: HBASE-17831-branch-1.patch, > HBASE-17831-branch-1-v1.patch, HBASE-17831-master.patch, > HBASE-17831-master-v1.patch > > > Support small scan in thrift2 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17817) Make Regionservers log which tables it removed coprocessors from when aborting
[
https://issues.apache.org/jira/browse/HBASE-17817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steen Manniche updated HBASE-17817:
---
Attachment: (was: HBASE-17817.master.002.patch)
> Make Regionservers log which tables it removed coprocessors from when aborting
> --
>
> Key: HBASE-17817
> URL: https://issues.apache.org/jira/browse/HBASE-17817
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors, regionserver
>Affects Versions: 1.1.2
>Reporter: Steen Manniche
> Labels: logging
> Attachments: HBASE-17817.master.001.patch
>
>
> When a coprocessor throws a runtime exception (e.g. NPE), the regionserver
> handles this according to {{hbase.coprocessor.abortonerror}}.
> If the coprocessor was loaded on a specific table, the output in the logs
> give no indication as to which table the coprocessor was removed from (or
> which version, or jarfile is the culprit). This causes longer debugging and
> recovery times.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17817) Make Regionservers log which tables it removed coprocessors from when aborting
[
https://issues.apache.org/jira/browse/HBASE-17817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steen Manniche updated HBASE-17817:
---
Attachment: HBASE-17817.master.002.patch
> Make Regionservers log which tables it removed coprocessors from when aborting
> --
>
> Key: HBASE-17817
> URL: https://issues.apache.org/jira/browse/HBASE-17817
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors, regionserver
>Affects Versions: 1.1.2
>Reporter: Steen Manniche
> Labels: logging
> Attachments: HBASE-17817.master.001.patch,
> HBASE-17817.master.002.patch
>
>
> When a coprocessor throws a runtime exception (e.g. NPE), the regionserver
> handles this according to {{hbase.coprocessor.abortonerror}}.
> If the coprocessor was loaded on a specific table, the output in the logs
> give no indication as to which table the coprocessor was removed from (or
> which version, or jarfile is the culprit). This causes longer debugging and
> recovery times.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17817) Make Regionservers log which tables it removed coprocessors from when aborting
[
https://issues.apache.org/jira/browse/HBASE-17817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steen Manniche updated HBASE-17817:
---
Attachment: HBASE-17817.master.001.patch
> Make Regionservers log which tables it removed coprocessors from when aborting
> --
>
> Key: HBASE-17817
> URL: https://issues.apache.org/jira/browse/HBASE-17817
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors, regionserver
>Affects Versions: 1.1.2
>Reporter: Steen Manniche
> Labels: logging
> Attachments: HBASE-17817.master.001.patch
>
>
> When a coprocessor throws a runtime exception (e.g. NPE), the regionserver
> handles this according to {{hbase.coprocessor.abortonerror}}.
> If the coprocessor was loaded on a specific table, the output in the logs
> give no indication as to which table the coprocessor was removed from (or
> which version, or jarfile is the culprit). This causes longer debugging and
> recovery times.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17138) Backport read-path offheap (HBASE-11425) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-17138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944960#comment-15944960 ] Yu Li commented on HBASE-17138: --- bq. how much benefit does HBASE-15756 contribute for the Singles day performane boost on top of this offheap feature? Please refer to [this comment|https://issues.apache.org/jira/browse/HBASE-15756?focusedCommentId=15743826=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15743826] in HBASE-15756. For the other question, I think the answer is "No", but better with [~haoran]'s confirmation. > Backport read-path offheap (HBASE-11425) to branch-1 > > > Key: HBASE-17138 > URL: https://issues.apache.org/jira/browse/HBASE-17138 > Project: HBase > Issue Type: Improvement >Reporter: Yu Li >Assignee: Yu Sun > Attachments: > 0001-fix-EHB-511-Resolve-client-compatibility-issue-introduced-by-offheap-change.patch, > 0001-to-EHB-446-offheap-hfile-format-should-keep-compatible-v3.patch, > 0001-to-EHB-456-Cell-should-be-compatible-with-branch-1.1.2.patch > > > From the > [thread|http://mail-archives.apache.org/mod_mbox/hbase-user/201611.mbox/%3CCAM7-19%2Bn7cEiY4H9iLQ3N9V0NXppOPduZwk-hhgNLEaJfiV3kA%40mail.gmail.com%3E] > of sharing our experience and performance data of read-path offheap usage in > Alibaba search, we could see people are positive to have HBASE-11425 in > branch-1, so I'd like to create a JIRA and move the discussion and decision > making here. > Echoing some comments from the mail thread: > Bryan: > Is the backported patch available anywhere? If it ends up not getting > officially backported to branch-1 due to 2.0 around the corner, some of us > who build our own deploy may want to integrate into our builds > Andrew: > Yes, please, the patches will be useful to the community even if we decide > not to backport into an official 1.x release. > Enis: > I don't see any reason why we cannot backport to branch-1. > Ted: > Opening a JIRA would be fine. This makes it easier for people to obtain the > patch(es) > Nick: > From the DISCUSS thread re: EOL of 1.1, it seems we'll continue to > support 1.x releases for some time... I would guess these will be > maintained until 2.2 at least. Therefore, offheap patches that have seen > production exposure seem like a reasonable candidate for backport, perhaps in > a 1.4 or 1.5 release timeframe. > Anoop: > Because of some compatibility issues, we decide that this will be done in 2.0 > only.. Ya as Andy said, it would be great to share the 1.x backported > patches. > The following is all the jira ids we have back ported: > HBASE-10930 Change Filters and GetClosestRowBeforeTracker to work with Cells > (Ram) > HBASE-13373 Squash HFileReaderV3 together with HFileReaderV2 and > AbstractHFileReader; ditto for Scanners and BlockReader, etc. > HBASE-13429 Remove deprecated seek/reseek methods from HFileScanner. > HBASE-13450 - Purge RawBytescomparator from the writers and readers for > HBASE-10800 (Ram) > HBASE-13501 - Deprecate/Remove getComparator() in HRegionInfo. > HBASE-12048 Remove deprecated APIs from Filter. > HBASE-10800 - Use CellComparator instead of KVComparator (Ram) > HBASE-13679 Change ColumnTracker and SQM to deal with Cell instead of byte[], > int, int. > HBASE-13642 Deprecate RegionObserver#postScannerFilterRow CP hook with > byte[],int,int args in favor of taking Cell arg. > HBASE-13641 Deperecate Filter#filterRowKey(byte[] buffer, int offset, int > length) in favor of filterRowKey(Cell firstRowCell). > HBASE-13827 Delayed scanner close in KeyValueHeap and StoreScanner. > HBASE-13871 Change RegionScannerImpl to deal with Cell instead of byte[], > int, int. > HBASE-11911 Break up tests into more fine grained categories (Alex Newman) > HBASE-12059 Create hbase-annotations module > HBASE-12106 Move test annotations to test artifact (Enis Soztutar) > HBASE-13916 Create MultiByteBuffer an aggregation of ByteBuffers. > HBASE-15679 Assertion on wrong variable in > TestReplicationThrottler#testThrottling > HBASE-13931 Move Unsafe based operations to UnsafeAccess. > HBASE-12345 Unsafe based ByteBuffer Comparator. > HBASE-13998 Remove CellComparator#compareRows(byte[] left, int loffset, int > llength, byte[] right, int roffset, int rlength). > HBASE-13998 Remove CellComparator#compareRows()- Addendum to fix javadoc warn > HBASE-13579 Avoid isCellTTLExpired() for NO-TAG cases (partially backport > this patch) > HBASE-13448 New Cell implementation with cached component offsets/lengths. > HBASE-13387 Add ByteBufferedCell an extension to Cell. > HBASE-13387 Add ByteBufferedCell an extension to Cell - addendum. > HBASE-12650 Move ServerName to hbase-common module (partially backport this > patch) > HBASE-12296 Filters should work with ByteBufferedCell. >
[jira] [Commented] (HBASE-17524) HBase 1.3.1 release
[ https://issues.apache.org/jira/browse/HBASE-17524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944945#comment-15944945 ] Yu Li commented on HBASE-17524: --- Thanks for the update sir! [~mantonov] > HBase 1.3.1 release > --- > > Key: HBASE-17524 > URL: https://issues.apache.org/jira/browse/HBASE-17524 > Project: HBase > Issue Type: Umbrella >Reporter: Mikhail Antonov > > Let's have this umbrella jira to track backports and bufgixes that we want to > go to 1.3.1 release. > Please add tasks and comment if you want something to be backported. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17831) Support small scan in thrift2
[ https://issues.apache.org/jira/browse/HBASE-17831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-17831: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.0 1.4.0 Status: Resolved (was: Patch Available) > Support small scan in thrift2 > - > > Key: HBASE-17831 > URL: https://issues.apache.org/jira/browse/HBASE-17831 > Project: HBase > Issue Type: Improvement > Components: Thrift >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng > Fix For: 1.4.0, 2.0 > > Attachments: HBASE-17831-branch-1.patch, > HBASE-17831-branch-1-v1.patch, HBASE-17831-master.patch, > HBASE-17831-master-v1.patch > > > Support small scan in thrift2 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17843) JUnit test timed out in TestRegionReplicaFailover.java
[ https://issues.apache.org/jira/browse/HBASE-17843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qilin Cao updated HBASE-17843: -- Description: I found junit test failed in TestRegionReplicaFailover.java, so I changed the testPrimaryRegionKill method test timeout to 24ms, and add sleep 5000ms for verify result. error logs: Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 285.221 sec <<< FAILURE! - in org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover testPrimaryRegionKill[0](org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover) Time elapsed: 125.963 sec <<< ERROR! org.junit.runners.model.TestTimedOutException: test timed out after 12 milliseconds at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:460) at java.util.concurrent.TimeUnit.timedWait(TimeUnit.java:348) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService.pollForSpecificCompletedTask(ResultBoundedCompletionService.java:258) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService.pollForFirstSuccessfullyCompletedTask(ResultBoundedCompletionService.java:214) at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.call(RpcRetryingCallerWithReadReplicas.java:209) at org.apache.hadoop.hbase.client.HTable.get(HTable.java:428) at org.apache.hadoop.hbase.client.HTable.get(HTable.java:392) at org.apache.hadoop.hbase.HBaseTestingUtility.verifyNumericRows(HBaseTestingUtility.java:2197) at org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover.verifyNumericRowsWithTimeout(TestRegionReplicaFailover.java:227) at org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover.testPrimaryRegionKill(TestRegionReplicaFailover.java:200) was: I found junit test failed in TestRegionReplicaFailover.java, so I changed the testPrimaryRegionKill method test timeout = 24 error logs: Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 285.221 sec <<< FAILURE! - in org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover testPrimaryRegionKill[0](org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover) Time elapsed: 125.963 sec <<< ERROR! org.junit.runners.model.TestTimedOutException: test timed out after 12 milliseconds at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:460) at java.util.concurrent.TimeUnit.timedWait(TimeUnit.java:348) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService.pollForSpecificCompletedTask(ResultBoundedCompletionService.java:258) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService.pollForFirstSuccessfullyCompletedTask(ResultBoundedCompletionService.java:214) at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.call(RpcRetryingCallerWithReadReplicas.java:209) at org.apache.hadoop.hbase.client.HTable.get(HTable.java:428) at org.apache.hadoop.hbase.client.HTable.get(HTable.java:392) at org.apache.hadoop.hbase.HBaseTestingUtility.verifyNumericRows(HBaseTestingUtility.java:2197) at org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover.verifyNumericRowsWithTimeout(TestRegionReplicaFailover.java:227) at org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover.testPrimaryRegionKill(TestRegionReplicaFailover.java:200) > JUnit test timed out in TestRegionReplicaFailover.java > -- > > Key: HBASE-17843 > URL: https://issues.apache.org/jira/browse/HBASE-17843 > Project: HBase > Issue Type: Improvement >Reporter: Qilin Cao >Priority: Minor > Attachments: HBASE-17843-v1.patch > > > I found junit test failed in TestRegionReplicaFailover.java, so I changed the > testPrimaryRegionKill method test timeout to 24ms, and add sleep 5000ms > for verify result. > error logs: > Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 285.221 sec > <<< FAILURE! - in > org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover > testPrimaryRegionKill[0](org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover) > Time elapsed: 125.963 sec <<< ERROR! > org.junit.runners.model.TestTimedOutException: test timed out after 12 > milliseconds > at java.lang.Object.wait(Native Method) > at java.lang.Object.wait(Object.java:460) > at java.util.concurrent.TimeUnit.timedWait(TimeUnit.java:348) > at > org.apache.hadoop.hbase.client.ResultBoundedCompletionService.pollForSpecificCompletedTask(ResultBoundedCompletionService.java:258) > at > org.apache.hadoop.hbase.client.ResultBoundedCompletionService.pollForFirstSuccessfullyCompletedTask(ResultBoundedCompletionService.java:214) > at >
[jira] [Updated] (HBASE-17843) JUnit test timed out in TestRegionReplicaFailover.java
[ https://issues.apache.org/jira/browse/HBASE-17843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qilin Cao updated HBASE-17843: -- Attachment: HBASE-17843-v1.patch > JUnit test timed out in TestRegionReplicaFailover.java > -- > > Key: HBASE-17843 > URL: https://issues.apache.org/jira/browse/HBASE-17843 > Project: HBase > Issue Type: Improvement >Reporter: Qilin Cao >Priority: Minor > Attachments: HBASE-17843-v1.patch > > > I found junit test failed in TestRegionReplicaFailover.java, so I changed the > testPrimaryRegionKill method test timeout = 24 > error logs: > Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 285.221 sec > <<< FAILURE! - in > org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover > testPrimaryRegionKill[0](org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover) > Time elapsed: 125.963 sec <<< ERROR! > org.junit.runners.model.TestTimedOutException: test timed out after 12 > milliseconds > at java.lang.Object.wait(Native Method) > at java.lang.Object.wait(Object.java:460) > at java.util.concurrent.TimeUnit.timedWait(TimeUnit.java:348) > at > org.apache.hadoop.hbase.client.ResultBoundedCompletionService.pollForSpecificCompletedTask(ResultBoundedCompletionService.java:258) > at > org.apache.hadoop.hbase.client.ResultBoundedCompletionService.pollForFirstSuccessfullyCompletedTask(ResultBoundedCompletionService.java:214) > at > org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.call(RpcRetryingCallerWithReadReplicas.java:209) > at org.apache.hadoop.hbase.client.HTable.get(HTable.java:428) > at org.apache.hadoop.hbase.client.HTable.get(HTable.java:392) > at > org.apache.hadoop.hbase.HBaseTestingUtility.verifyNumericRows(HBaseTestingUtility.java:2197) > at > org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover.verifyNumericRowsWithTimeout(TestRegionReplicaFailover.java:227) > at > org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover.testPrimaryRegionKill(TestRegionReplicaFailover.java:200) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17843) JUnit test timed out in TestRegionReplicaFailover.java
[ https://issues.apache.org/jira/browse/HBASE-17843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qilin Cao updated HBASE-17843: -- Attachment: (was: HBASE-17843-v1.patch) > JUnit test timed out in TestRegionReplicaFailover.java > -- > > Key: HBASE-17843 > URL: https://issues.apache.org/jira/browse/HBASE-17843 > Project: HBase > Issue Type: Improvement >Reporter: Qilin Cao >Priority: Minor > Attachments: HBASE-17843-v1.patch > > > I found junit test failed in TestRegionReplicaFailover.java, so I changed the > testPrimaryRegionKill method test timeout = 24 > error logs: > Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 285.221 sec > <<< FAILURE! - in > org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover > testPrimaryRegionKill[0](org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover) > Time elapsed: 125.963 sec <<< ERROR! > org.junit.runners.model.TestTimedOutException: test timed out after 12 > milliseconds > at java.lang.Object.wait(Native Method) > at java.lang.Object.wait(Object.java:460) > at java.util.concurrent.TimeUnit.timedWait(TimeUnit.java:348) > at > org.apache.hadoop.hbase.client.ResultBoundedCompletionService.pollForSpecificCompletedTask(ResultBoundedCompletionService.java:258) > at > org.apache.hadoop.hbase.client.ResultBoundedCompletionService.pollForFirstSuccessfullyCompletedTask(ResultBoundedCompletionService.java:214) > at > org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.call(RpcRetryingCallerWithReadReplicas.java:209) > at org.apache.hadoop.hbase.client.HTable.get(HTable.java:428) > at org.apache.hadoop.hbase.client.HTable.get(HTable.java:392) > at > org.apache.hadoop.hbase.HBaseTestingUtility.verifyNumericRows(HBaseTestingUtility.java:2197) > at > org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover.verifyNumericRowsWithTimeout(TestRegionReplicaFailover.java:227) > at > org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover.testPrimaryRegionKill(TestRegionReplicaFailover.java:200) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it
[ https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944707#comment-15944707 ] ramkrishna.s.vasudevan commented on HBASE-16438: [~anastas] If we are going with soft ref approach and then hardening it on creating CellChunkMap I can create a soft ref map version in ChunkCreator but the process of changing into hard ref has to happen only when CellChunkMap fllattening happens which will not be done by this JIRA. So I thought we can just proceed with existing version and then in the CellChunkMap JIRA update the map to convert from soft to hard. Even if that requires a change in ChunkCreator map then we will do it there. > Create a cell type so that chunk id is embedded in it > - > > Key: HBASE-16438 > URL: https://issues.apache.org/jira/browse/HBASE-16438 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Attachments: HBASE-16438_1.patch, > HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, > HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, HBASE-16438.patch, > MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, > MemstoreChunkCell_trunk.patch > > > For CellChunkMap we may need a cell such that the chunk out of which it was > created, the id of the chunk be embedded in it so that when doing flattening > we can use the chunk id as a meta data. More details will follow once the > initial tasks are completed. > Why we need to embed the chunkid in the Cell is described by [~anastas] in > this remark over in parent issue > https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17835) Spelling mistakes in the Java source
[
https://issues.apache.org/jira/browse/HBASE-17835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944682#comment-15944682
]
Hadoop QA commented on HBASE-17835:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 32s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
9s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
33s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
0s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 25s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 12s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 100m 57s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 147m 29s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12860803/HBASE-17835-001.patch
|
| JIRA Issue | HBASE-17835 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 1478b7a7f193 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 4b62a52 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/6244/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6244/testReport/ |
| modules | C: hbase-client hbase-server U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6244/console |
| Powered by | Apache Yetus 0.3.0
[jira] [Updated] (HBASE-17843) JUnit test timed out in TestRegionReplicaFailover.java
[ https://issues.apache.org/jira/browse/HBASE-17843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qilin Cao updated HBASE-17843: -- Priority: Minor (was: Trivial) > JUnit test timed out in TestRegionReplicaFailover.java > -- > > Key: HBASE-17843 > URL: https://issues.apache.org/jira/browse/HBASE-17843 > Project: HBase > Issue Type: Improvement >Reporter: Qilin Cao >Priority: Minor > Attachments: HBASE-17843-v1.patch > > > I found junit test failed in TestRegionReplicaFailover.java, so I changed the > testPrimaryRegionKill method test timeout = 24 > error logs: > Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 285.221 sec > <<< FAILURE! - in > org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover > testPrimaryRegionKill[0](org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover) > Time elapsed: 125.963 sec <<< ERROR! > org.junit.runners.model.TestTimedOutException: test timed out after 12 > milliseconds > at java.lang.Object.wait(Native Method) > at java.lang.Object.wait(Object.java:460) > at java.util.concurrent.TimeUnit.timedWait(TimeUnit.java:348) > at > org.apache.hadoop.hbase.client.ResultBoundedCompletionService.pollForSpecificCompletedTask(ResultBoundedCompletionService.java:258) > at > org.apache.hadoop.hbase.client.ResultBoundedCompletionService.pollForFirstSuccessfullyCompletedTask(ResultBoundedCompletionService.java:214) > at > org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.call(RpcRetryingCallerWithReadReplicas.java:209) > at org.apache.hadoop.hbase.client.HTable.get(HTable.java:428) > at org.apache.hadoop.hbase.client.HTable.get(HTable.java:392) > at > org.apache.hadoop.hbase.HBaseTestingUtility.verifyNumericRows(HBaseTestingUtility.java:2197) > at > org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover.verifyNumericRowsWithTimeout(TestRegionReplicaFailover.java:227) > at > org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover.testPrimaryRegionKill(TestRegionReplicaFailover.java:200) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HBASE-17835) Spelling mistakes in the Java source
[ https://issues.apache.org/jira/browse/HBASE-17835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944532#comment-15944532 ] Qilin Cao edited comment on HBASE-17835 at 3/28/17 6:25 AM: hi, please take a look at HBASE-17843, the failures may be not caused by the patch. was (Author: qilin cao): hi, please take a look at the junit tests, the failures may be not caused by the patch. > Spelling mistakes in the Java source > > > Key: HBASE-17835 > URL: https://issues.apache.org/jira/browse/HBASE-17835 > Project: HBase > Issue Type: Improvement >Reporter: Qilin Cao >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-17835-001.patch > > > I found spelling mistakes in the hbase java source files viz. recieved > instead of received, and SpanReciever instead of SpanReceiver -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17843) JUnit test timed out in TestRegionReplicaFailover.java
[ https://issues.apache.org/jira/browse/HBASE-17843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qilin Cao updated HBASE-17843: -- Attachment: HBASE-17843-v1.patch > JUnit test timed out in TestRegionReplicaFailover.java > -- > > Key: HBASE-17843 > URL: https://issues.apache.org/jira/browse/HBASE-17843 > Project: HBase > Issue Type: Improvement >Reporter: Qilin Cao >Priority: Trivial > Attachments: HBASE-17843-v1.patch > > > I found junit test failed in TestRegionReplicaFailover.java, so I changed the > testPrimaryRegionKill method test timeout = 24 > error logs: > Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 285.221 sec > <<< FAILURE! - in > org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover > testPrimaryRegionKill[0](org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover) > Time elapsed: 125.963 sec <<< ERROR! > org.junit.runners.model.TestTimedOutException: test timed out after 12 > milliseconds > at java.lang.Object.wait(Native Method) > at java.lang.Object.wait(Object.java:460) > at java.util.concurrent.TimeUnit.timedWait(TimeUnit.java:348) > at > org.apache.hadoop.hbase.client.ResultBoundedCompletionService.pollForSpecificCompletedTask(ResultBoundedCompletionService.java:258) > at > org.apache.hadoop.hbase.client.ResultBoundedCompletionService.pollForFirstSuccessfullyCompletedTask(ResultBoundedCompletionService.java:214) > at > org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.call(RpcRetryingCallerWithReadReplicas.java:209) > at org.apache.hadoop.hbase.client.HTable.get(HTable.java:428) > at org.apache.hadoop.hbase.client.HTable.get(HTable.java:392) > at > org.apache.hadoop.hbase.HBaseTestingUtility.verifyNumericRows(HBaseTestingUtility.java:2197) > at > org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover.verifyNumericRowsWithTimeout(TestRegionReplicaFailover.java:227) > at > org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover.testPrimaryRegionKill(TestRegionReplicaFailover.java:200) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17843) JUnit test timed out in TestRegionReplicaFailover.java
[ https://issues.apache.org/jira/browse/HBASE-17843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qilin Cao updated HBASE-17843: -- Description: I found junit test failed in TestRegionReplicaFailover.java, so I changed the testPrimaryRegionKill method test timeout = 24 error logs: Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 285.221 sec <<< FAILURE! - in org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover testPrimaryRegionKill[0](org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover) Time elapsed: 125.963 sec <<< ERROR! org.junit.runners.model.TestTimedOutException: test timed out after 12 milliseconds at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:460) at java.util.concurrent.TimeUnit.timedWait(TimeUnit.java:348) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService.pollForSpecificCompletedTask(ResultBoundedCompletionService.java:258) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService.pollForFirstSuccessfullyCompletedTask(ResultBoundedCompletionService.java:214) at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.call(RpcRetryingCallerWithReadReplicas.java:209) at org.apache.hadoop.hbase.client.HTable.get(HTable.java:428) at org.apache.hadoop.hbase.client.HTable.get(HTable.java:392) at org.apache.hadoop.hbase.HBaseTestingUtility.verifyNumericRows(HBaseTestingUtility.java:2197) at org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover.verifyNumericRowsWithTimeout(TestRegionReplicaFailover.java:227) at org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover.testPrimaryRegionKill(TestRegionReplicaFailover.java:200) was: I found junit test failed in TestRegionReplicaFailover.java, so I changed the testPrimaryRegionKill method test timeout = 24 error logs: Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 285.221 sec <<< FAILURE! - in org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover testPrimaryRegionKill[0](org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover) Time elapsed: 125.963 sec <<< ERROR! org.junit.runners.model.TestTimedOutException: test timed out after 12 milliseconds > JUnit test timed out in TestRegionReplicaFailover.java > -- > > Key: HBASE-17843 > URL: https://issues.apache.org/jira/browse/HBASE-17843 > Project: HBase > Issue Type: Improvement >Reporter: Qilin Cao >Priority: Trivial > > I found junit test failed in TestRegionReplicaFailover.java, so I changed the > testPrimaryRegionKill method test timeout = 24 > error logs: > Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 285.221 sec > <<< FAILURE! - in > org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover > testPrimaryRegionKill[0](org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover) > Time elapsed: 125.963 sec <<< ERROR! > org.junit.runners.model.TestTimedOutException: test timed out after 12 > milliseconds > at java.lang.Object.wait(Native Method) > at java.lang.Object.wait(Object.java:460) > at java.util.concurrent.TimeUnit.timedWait(TimeUnit.java:348) > at > org.apache.hadoop.hbase.client.ResultBoundedCompletionService.pollForSpecificCompletedTask(ResultBoundedCompletionService.java:258) > at > org.apache.hadoop.hbase.client.ResultBoundedCompletionService.pollForFirstSuccessfullyCompletedTask(ResultBoundedCompletionService.java:214) > at > org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.call(RpcRetryingCallerWithReadReplicas.java:209) > at org.apache.hadoop.hbase.client.HTable.get(HTable.java:428) > at org.apache.hadoop.hbase.client.HTable.get(HTable.java:392) > at > org.apache.hadoop.hbase.HBaseTestingUtility.verifyNumericRows(HBaseTestingUtility.java:2197) > at > org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover.verifyNumericRowsWithTimeout(TestRegionReplicaFailover.java:227) > at > org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover.testPrimaryRegionKill(TestRegionReplicaFailover.java:200) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17843) JUnit test timed out in TestRegionReplicaFailover.java
Qilin Cao created HBASE-17843: - Summary: JUnit test timed out in TestRegionReplicaFailover.java Key: HBASE-17843 URL: https://issues.apache.org/jira/browse/HBASE-17843 Project: HBase Issue Type: Improvement Reporter: Qilin Cao Priority: Trivial I found junit test failed in TestRegionReplicaFailover.java, so I changed the testPrimaryRegionKill method test timeout = 24 error logs: Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 285.221 sec <<< FAILURE! - in org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover testPrimaryRegionKill[0](org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover) Time elapsed: 125.963 sec <<< ERROR! org.junit.runners.model.TestTimedOutException: test timed out after 12 milliseconds -- This message was sent by Atlassian JIRA (v6.3.15#6346)
