[jira] [Commented] (HBASE-21981) MMaped bucket cache IOEngine does not work with persistence
[ https://issues.apache.org/jira/browse/HBASE-21981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17358128#comment-17358128 ] Duo Zhang commented on HBASE-21981: --- Any progress here? > MMaped bucket cache IOEngine does not work with persistence > --- > > Key: HBASE-21981 > URL: https://issues.apache.org/jira/browse/HBASE-21981 > Project: HBase > Issue Type: Bug > Components: BucketCache >Affects Versions: 2.1.3 >Reporter: ramkrishna.s.vasudevan >Assignee: Anoop Sam John >Priority: Major > Fix For: 3.0.0-alpha-1 > > Attachments: HBASE-21981.patch, HBASE-21981.patch > > > The MMap based IOEngines does not retrieve the data back if > 'hbase.bucketcache.persistent.path' is enabled. FileIOEngine works fine but > only the FileMMapEngine has this problem. > The reason is that we don't get the byte buffers in the proper order while > reading back from the file in case of persistence. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-21981) MMaped bucket cache IOEngine does not work with persistence
[
https://issues.apache.org/jira/browse/HBASE-21981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975065#comment-16975065
]
HBase QA commented on HBASE-21981:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} HBASE-21981 does not apply to master. Rebase required? Wrong
Branch? See
https://yetus.apache.org/documentation/in-progress/precommit-patchnames for
help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HBASE-21981 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12960889/HBASE-21981.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/1022/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
> MMaped bucket cache IOEngine does not work with persistence
> ---
>
> Key: HBASE-21981
> URL: https://issues.apache.org/jira/browse/HBASE-21981
> Project: HBase
> Issue Type: Bug
> Components: BucketCache
>Affects Versions: 2.1.3
>Reporter: ramkrishna.s.vasudevan
>Assignee: Anoop Sam John
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.2.3, 2.1.9
>
> Attachments: HBASE-21981.patch, HBASE-21981.patch
>
>
> The MMap based IOEngines does not retrieve the data back if
> 'hbase.bucketcache.persistent.path' is enabled. FileIOEngine works fine but
> only the FileMMapEngine has this problem.
> The reason is that we don't get the byte buffers in the proper order while
> reading back from the file in case of persistence.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-21981) MMaped bucket cache IOEngine does not work with persistence
[
https://issues.apache.org/jira/browse/HBASE-21981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876522#comment-16876522
]
HBase QA commented on HBASE-21981:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} HBASE-21981 does not apply to master. Rebase required? Wrong
Branch? See
https://yetus.apache.org/documentation/in-progress/precommit-patchnames for
help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HBASE-21981 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12960889/HBASE-21981.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/589/console |
| Powered by | Apache Yetus 0.9.0 http://yetus.apache.org |
This message was automatically generated.
> MMaped bucket cache IOEngine does not work with persistence
> ---
>
> Key: HBASE-21981
> URL: https://issues.apache.org/jira/browse/HBASE-21981
> Project: HBase
> Issue Type: Bug
> Components: BucketCache
>Affects Versions: 2.1.3
>Reporter: ramkrishna.s.vasudevan
>Assignee: Anoop Sam John
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.2.1, 2.1.7
>
> Attachments: HBASE-21981.patch, HBASE-21981.patch
>
>
> The MMap based IOEngines does not retrieve the data back if
> 'hbase.bucketcache.persistent.path' is enabled. FileIOEngine works fine but
> only the FileMMapEngine has this problem.
> The reason is that we don't get the byte buffers in the proper order while
> reading back from the file in case of persistence.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21981) MMaped bucket cache IOEngine does not work with persistence
[ https://issues.apache.org/jira/browse/HBASE-21981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842572#comment-16842572 ] stack commented on HBASE-21981: --- Moving out of 2.1.5 > MMaped bucket cache IOEngine does not work with persistence > --- > > Key: HBASE-21981 > URL: https://issues.apache.org/jira/browse/HBASE-21981 > Project: HBase > Issue Type: Bug > Components: BucketCache >Affects Versions: 2.1.3 >Reporter: ramkrishna.s.vasudevan >Assignee: Anoop Sam John >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.0.6, 2.2.1, 2.1.6 > > Attachments: HBASE-21981.patch, HBASE-21981.patch > > > The MMap based IOEngines does not retrieve the data back if > 'hbase.bucketcache.persistent.path' is enabled. FileIOEngine works fine but > only the FileMMapEngine has this problem. > The reason is that we don't get the byte buffers in the proper order while > reading back from the file in case of persistence. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21981) MMaped bucket cache IOEngine does not work with persistence
[ https://issues.apache.org/jira/browse/HBASE-21981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16783028#comment-16783028 ] Anoop Sam John commented on HBASE-21981: Wait let me do a reverse way... will report back soon Duo. Tks > MMaped bucket cache IOEngine does not work with persistence > --- > > Key: HBASE-21981 > URL: https://issues.apache.org/jira/browse/HBASE-21981 > Project: HBase > Issue Type: Bug > Components: BucketCache >Affects Versions: 2.1.3 >Reporter: ramkrishna.s.vasudevan >Assignee: Anoop Sam John >Priority: Major > Fix For: 3.0.0, 2.2.0, 2.0.5, 2.3.0, 2.1.4 > > Attachments: HBASE-21981.patch, HBASE-21981.patch > > > The MMap based IOEngines does not retrieve the data back if > 'hbase.bucketcache.persistent.path' is enabled. FileIOEngine works fine but > only the FileMMapEngine has this problem. > The reason is that we don't get the byte buffers in the proper order while > reading back from the file in case of persistence. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21981) MMaped bucket cache IOEngine does not work with persistence
[ https://issues.apache.org/jira/browse/HBASE-21981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16783026#comment-16783026 ] Anoop Sam John commented on HBASE-21981: It has to be either an index or a global offset number. (offset of this BB in the entire BC space). The multi threading way really helps when the size of the cache is big.. > MMaped bucket cache IOEngine does not work with persistence > --- > > Key: HBASE-21981 > URL: https://issues.apache.org/jira/browse/HBASE-21981 > Project: HBase > Issue Type: Bug > Components: BucketCache >Affects Versions: 2.1.3 >Reporter: ramkrishna.s.vasudevan >Assignee: Anoop Sam John >Priority: Major > Fix For: 3.0.0, 2.2.0, 2.0.5, 2.3.0, 2.1.4 > > Attachments: HBASE-21981.patch, HBASE-21981.patch > > > The MMap based IOEngines does not retrieve the data back if > 'hbase.bucketcache.persistent.path' is enabled. FileIOEngine works fine but > only the FileMMapEngine has this problem. > The reason is that we don't get the byte buffers in the proper order while > reading back from the file in case of persistence. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21981) MMaped bucket cache IOEngine does not work with persistence
[ https://issues.apache.org/jira/browse/HBASE-21981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16783000#comment-16783000 ] Duo Zhang commented on HBASE-21981: --- Having a index parameter for an allocator is weird... Do we have other solutions? > MMaped bucket cache IOEngine does not work with persistence > --- > > Key: HBASE-21981 > URL: https://issues.apache.org/jira/browse/HBASE-21981 > Project: HBase > Issue Type: Bug > Components: BucketCache >Affects Versions: 2.1.3 >Reporter: ramkrishna.s.vasudevan >Assignee: Anoop Sam John >Priority: Major > Fix For: 3.0.0, 2.2.0, 2.0.5, 2.3.0, 2.1.4 > > Attachments: HBASE-21981.patch, HBASE-21981.patch > > > The MMap based IOEngines does not retrieve the data back if > 'hbase.bucketcache.persistent.path' is enabled. FileIOEngine works fine but > only the FileMMapEngine has this problem. > The reason is that we don't get the byte buffers in the proper order while > reading back from the file in case of persistence. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21981) MMaped bucket cache IOEngine does not work with persistence
[
https://issues.apache.org/jira/browse/HBASE-21981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782549#comment-16782549
]
Hadoop QA commented on HBASE-21981:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
18s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
0s{color} | {color:blue} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
18s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 8s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
4s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}254m 15s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
51s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}310m 10s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.master.procedure.TestSCPWithReplicasWithoutZKCoordinated |
| | hadoop.hbase.master.TestSplitWALManager |
| | hadoop.hbase.master.procedure.TestSCPWithReplicas |
| | hadoop.hbase.client.TestAdmin1 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21981 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12960889/HBASE-21981.patch |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 4eb5df877c29 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HBASE-21981) MMaped bucket cache IOEngine does not work with persistence
[
https://issues.apache.org/jira/browse/HBASE-21981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782514#comment-16782514
]
Hadoop QA commented on HBASE-21981:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
9s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
16s{color} | {color:blue} hbase-server in master has 1 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
13s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
7m 51s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
47s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}131m
24s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
49s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}175m 33s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21981 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12960888/HBASE-21981.patch |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux d978236a81cd 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 841a45afb7 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default
[jira] [Commented] (HBASE-21981) MMaped bucket cache IOEngine does not work with persistence
[ https://issues.apache.org/jira/browse/HBASE-21981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782446#comment-16782446 ] Anoop Sam John commented on HBASE-21981: We have multi threaded way of creating Buffers for bucket cache. In case of file mmaped IOEngine, these buffers are obtained by mapping a file from a specific offset to length. When an RS starts and create the buffers by mapping the file, we get N buffers into ByteBufferArray at index 0 to N-1. In 'backingMap' we keep the blocks offset info considering these buffers index order. When RS restarts, the 'backingMap' is recreated. So each of the block refers to same old offsets. But this time the order of file mapping might be different. So we will not be really getting back the same block from file. > MMaped bucket cache IOEngine does not work with persistence > --- > > Key: HBASE-21981 > URL: https://issues.apache.org/jira/browse/HBASE-21981 > Project: HBase > Issue Type: Bug > Components: BucketCache >Affects Versions: 2.1.3 >Reporter: ramkrishna.s.vasudevan >Assignee: Anoop Sam John >Priority: Major > Fix For: 3.0.0, 2.2.0, 2.0.5, 2.3.0, 2.1.4 > > > The MMap based IOEngines does not retrieve the data back if > 'hbase.bucketcache.persistent.path' is enabled. FileIOEngine works fine but > only the FileMMapEngine has this problem. > The reason is that we don't get the byte buffers in the proper order while > reading back from the file in case of persistence. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
