[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[
https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16554244#comment-16554244
]
Hudson commented on HBASE-20565:
Results for branch master
[build #407 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/407/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/407//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/407//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/407//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect
> result since 1.4
> -
>
> Key: HBASE-20565
> URL: https://issues.apache.org/jira/browse/HBASE-20565
> Project: HBase
> Issue Type: Bug
> Components: Filters
>Affects Versions: 2.1.0, 1.4.4, 2.0.1
>Reporter: Jerry He
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.4.6, 2.0.2, 2.1.1
>
> Attachments: HBASE-20565.branch-1.v1.patch, HBASE-20565.v1.patch,
> HBASE-20565.v2.patch, HBASE-20565.v3.patch, debug.diff, debug.log,
> test-branch-1.4.patch
>
>
> When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see
> incorrect result.
> Here is a simple example.
> One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for
> range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset
> 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter,
> ColumnPaginationFilter).
> We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns
> are returned.
> In 1.2.x, the correct 5 columns are returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[
https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16554110#comment-16554110
]
Hudson commented on HBASE-20565:
Results for branch branch-2.1
[build #98 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/98/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/98//console].
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/98//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/98//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect
> result since 1.4
> -
>
> Key: HBASE-20565
> URL: https://issues.apache.org/jira/browse/HBASE-20565
> Project: HBase
> Issue Type: Bug
> Components: Filters
>Affects Versions: 2.1.0, 1.4.4, 2.0.1
>Reporter: Jerry He
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.4.6, 2.0.2, 2.1.1
>
> Attachments: HBASE-20565.branch-1.v1.patch, HBASE-20565.v1.patch,
> HBASE-20565.v2.patch, HBASE-20565.v3.patch, debug.diff, debug.log,
> test-branch-1.4.patch
>
>
> When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see
> incorrect result.
> Here is a simple example.
> One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for
> range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset
> 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter,
> ColumnPaginationFilter).
> We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns
> are returned.
> In 1.2.x, the correct 5 columns are returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[
https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16554075#comment-16554075
]
Hudson commented on HBASE-20565:
Results for branch branch-1
[build #391 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/391/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/391//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/391//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/391//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect
> result since 1.4
> -
>
> Key: HBASE-20565
> URL: https://issues.apache.org/jira/browse/HBASE-20565
> Project: HBase
> Issue Type: Bug
> Components: Filters
>Affects Versions: 2.1.0, 1.4.4, 2.0.1
>Reporter: Jerry He
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.4.6, 2.0.2, 2.1.1
>
> Attachments: HBASE-20565.branch-1.v1.patch, HBASE-20565.v1.patch,
> HBASE-20565.v2.patch, HBASE-20565.v3.patch, debug.diff, debug.log,
> test-branch-1.4.patch
>
>
> When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see
> incorrect result.
> Here is a simple example.
> One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for
> range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset
> 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter,
> ColumnPaginationFilter).
> We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns
> are returned.
> In 1.2.x, the correct 5 columns are returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[
https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16554078#comment-16554078
]
Hudson commented on HBASE-20565:
Results for branch branch-2.0
[build #586 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/586/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/586//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/586//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/586//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect
> result since 1.4
> -
>
> Key: HBASE-20565
> URL: https://issues.apache.org/jira/browse/HBASE-20565
> Project: HBase
> Issue Type: Bug
> Components: Filters
>Affects Versions: 2.1.0, 1.4.4, 2.0.1
>Reporter: Jerry He
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.4.6, 2.0.2, 2.1.1
>
> Attachments: HBASE-20565.branch-1.v1.patch, HBASE-20565.v1.patch,
> HBASE-20565.v2.patch, HBASE-20565.v3.patch, debug.diff, debug.log,
> test-branch-1.4.patch
>
>
> When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see
> incorrect result.
> Here is a simple example.
> One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for
> range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset
> 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter,
> ColumnPaginationFilter).
> We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns
> are returned.
> In 1.2.x, the correct 5 columns are returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[
https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16554048#comment-16554048
]
Hudson commented on HBASE-20565:
Results for branch branch-2
[build #1020 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1020/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1020//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1020//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1020//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect
> result since 1.4
> -
>
> Key: HBASE-20565
> URL: https://issues.apache.org/jira/browse/HBASE-20565
> Project: HBase
> Issue Type: Bug
> Components: Filters
>Affects Versions: 2.1.0, 1.4.4, 2.0.1
>Reporter: Jerry He
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.4.6, 2.0.2, 2.1.1
>
> Attachments: HBASE-20565.branch-1.v1.patch, HBASE-20565.v1.patch,
> HBASE-20565.v2.patch, HBASE-20565.v3.patch, debug.diff, debug.log,
> test-branch-1.4.patch
>
>
> When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see
> incorrect result.
> Here is a simple example.
> One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for
> range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset
> 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter,
> ColumnPaginationFilter).
> We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns
> are returned.
> In 1.2.x, the correct 5 columns are returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16553707#comment-16553707 ] Zheng Hu commented on HBASE-20565: -- Hadoop QA for branch-1 did not run, but I run under my local , it passed. Pushed this patch to branch-1.4 & branch-1 & branch-2.0 & branch-2.1 & master. Thanks all for reviewing. > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 2.1.0, 1.4.4, 2.0.1 >Reporter: Jerry He >Assignee: Zheng Hu >Priority: Major > Fix For: 1.5.0, 1.4.6, 2.0.2, 2.1.1 > > Attachments: HBASE-20565.branch-1.v1.patch, HBASE-20565.v1.patch, > HBASE-20565.v2.patch, HBASE-20565.v3.patch, debug.diff, debug.log, > test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16553515#comment-16553515 ] Andrew Purtell commented on HBASE-20565: +1 for branch-1 and branch-1.4 > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 2.1.0, 1.4.4, 2.0.1 >Reporter: Jerry He >Assignee: Zheng Hu >Priority: Major > Fix For: 1.5.0, 1.4.6, 2.0.2, 2.1.1 > > Attachments: HBASE-20565.branch-1.v1.patch, HBASE-20565.v1.patch, > HBASE-20565.v2.patch, HBASE-20565.v3.patch, debug.diff, debug.log, > test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[
https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16552373#comment-16552373
]
Hadoop QA commented on HBASE-20565:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
33s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} hbase-client: The patch generated 0 new + 22
unchanged - 11 fixed = 22 total (was 33) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
10s{color} | {color:green} The patch hbase-server passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
31s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 15s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
54s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}114m
35s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
43s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}165m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20565 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12932631/HBASE-20565.v3.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 76ed6fa32a7a 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16552243#comment-16552243 ] Zheng Hu commented on HBASE-20565: -- Thanks [~yuzhih...@gmail.com] & [~jinghe], I've uploaded the patch.v3 for 2.x branch ( fix checkstyle), and branch-1.v1.patch for branch-1.x branch... If Hadoop QA says OK, I'll commit to the relative branches. Thanks. > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 2.1.0, 1.4.4, 2.0.1 >Reporter: Jerry He >Assignee: Zheng Hu >Priority: Major > Fix For: 1.5.0, 1.4.6, 2.0.2, 2.1.1 > > Attachments: HBASE-20565.branch-1.v1.patch, HBASE-20565.v1.patch, > HBASE-20565.v2.patch, HBASE-20565.v3.patch, debug.diff, debug.log, > test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[
https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550535#comment-16550535
]
Ted Yu commented on HBASE-20565:
Zheng:
Can you attach patch for branch-1 ?
{code}
+result.listCells().forEach(results::add);
{code}
The above wouldn't compile by Java 1.7
> ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect
> result since 1.4
> -
>
> Key: HBASE-20565
> URL: https://issues.apache.org/jira/browse/HBASE-20565
> Project: HBase
> Issue Type: Bug
> Components: Filters
>Affects Versions: 2.1.0, 1.4.4, 2.0.1
>Reporter: Jerry He
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 2.1.0, 1.5.0, 1.4.6, 2.0.2
>
> Attachments: HBASE-20565.v1.patch, HBASE-20565.v2.patch, debug.diff,
> debug.log, test-branch-1.4.patch
>
>
> When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see
> incorrect result.
> Here is a simple example.
> One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for
> range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset
> 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter,
> ColumnPaginationFilter).
> We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns
> are returned.
> In 1.2.x, the correct 5 columns are returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[
https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550281#comment-16550281
]
Jerry He commented on HBASE-20565:
--
{quote}place the count-related filters at the last position
{quote}
{quote}ColumnPaginationFilter is order dependence filter
{quote}
Makes sense. I would think people use it last anyways. Thanks for the fix and
explanation.
+1
> ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect
> result since 1.4
> -
>
> Key: HBASE-20565
> URL: https://issues.apache.org/jira/browse/HBASE-20565
> Project: HBase
> Issue Type: Bug
> Components: Filters
>Affects Versions: 2.1.0, 1.4.4, 2.0.1
>Reporter: Jerry He
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 2.1.0, 1.5.0, 1.4.6, 2.0.2
>
> Attachments: HBASE-20565.v1.patch, HBASE-20565.v2.patch, debug.diff,
> debug.log, test-branch-1.4.patch
>
>
> When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see
> incorrect result.
> Here is a simple example.
> One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for
> range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset
> 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter,
> ColumnPaginationFilter).
> We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns
> are returned.
> In 1.2.x, the correct 5 columns are returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[
https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550244#comment-16550244
]
Ted Yu commented on HBASE-20565:
I ran the new test with ColumnPaginationFilter placed first in branch-1.3 where
there is no HBASE-18410 :
{code}
TestColumnRangeFilterWithColumnPaginationFilter(org.apache.hadoop.hbase.filter.TestColumnRangeFilter)
Time elapsed: 1.679 sec <<< FAILURE!
java.lang.AssertionError: expected:<5> but was:<4>
at
org.apache.hadoop.hbase.filter.TestColumnRangeFilter.TestColumnRangeFilterWithColumnPaginationFilter(TestColumnRangeFilter.java:278)
{code}
So the patch here is good by me.
> ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect
> result since 1.4
> -
>
> Key: HBASE-20565
> URL: https://issues.apache.org/jira/browse/HBASE-20565
> Project: HBase
> Issue Type: Bug
> Components: Filters
>Affects Versions: 2.1.0, 1.4.4, 2.0.1
>Reporter: Jerry He
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 2.1.0, 1.5.0, 1.4.6, 2.0.2
>
> Attachments: HBASE-20565.v1.patch, HBASE-20565.v2.patch, debug.diff,
> debug.log, test-branch-1.4.patch
>
>
> When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see
> incorrect result.
> Here is a simple example.
> One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for
> range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset
> 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter,
> ColumnPaginationFilter).
> We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns
> are returned.
> In 1.2.x, the correct 5 columns are returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550193#comment-16550193 ] Zheng Hu commented on HBASE-20565: -- bq. I am yet to see the relationship between SQL and FilterList. SQL is just an example to expression the semantics of ColumnPaginationFilter. I mean ColumnPaginationFilter is order dependence filter. so it's not worth us to eliminate order dependence in the upper layer (FilterList). [~anoop.hbase], [~jinghe], [~apurtell], If you have some time, please take a look. > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 2.1.0, 1.4.4, 2.0.1 >Reporter: Jerry He >Assignee: Zheng Hu >Priority: Major > Fix For: 2.1.0, 1.5.0, 1.4.6, 2.0.2 > > Attachments: HBASE-20565.v1.patch, HBASE-20565.v2.patch, debug.diff, > debug.log, test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550187#comment-16550187 ] Ted Yu commented on HBASE-20565: I am yet to see the relationship between SQL and FilterList. If you're confident in the current solution, I don't want to block it. Please get +1 from Anoop or Jerry. > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 2.1.0, 1.4.4, 2.0.1 >Reporter: Jerry He >Assignee: Zheng Hu >Priority: Major > Fix For: 2.1.0, 1.5.0, 1.4.6, 2.0.2 > > Attachments: HBASE-20565.v1.patch, HBASE-20565.v2.patch, debug.diff, > debug.log, test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550132#comment-16550132 ] Zheng Hu commented on HBASE-20565: -- Only for those filters who maintain some row-global state such as offset/count, it will be order dependence...otherwise, it's still order independence. Just as I said before, user will write SQL like : select * from test where x = 1 and y < 10 limit 0, 100 (For HBase, limit 0, 100 is also a Filter, ColumnPaginationFilter ex), If select * from test where limit 0, 100 and x = 1 and y < 10 has the same meaning as the former one, don't you think it's more confusing? > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 2.1.0, 1.4.4, 2.0.0 >Reporter: Jerry He >Assignee: Zheng Hu >Priority: Major > Fix For: 2.1.0, 1.5.0, 1.4.6, 2.0.2 > > Attachments: HBASE-20565.v1.patch, HBASE-20565.v2.patch, debug.diff, > debug.log, test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549477#comment-16549477 ] Ted Yu commented on HBASE-20565: I think filter order independence is important. MUST_PASS_ALL should produce same result regardless of the order of the Filter instances in the FilterList. It would be confusing if two queries produce different results where the only difference between the two is the order of the Filter instances in the FilterList. > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 2.1.0, 1.4.4, 2.0.0 >Reporter: Jerry He >Assignee: Zheng Hu >Priority: Major > Fix For: 2.1.0, 1.5.0, 1.4.6, 2.0.2 > > Attachments: HBASE-20565.v1.patch, HBASE-20565.v2.patch, debug.diff, > debug.log, test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549253#comment-16549253 ] Zheng Hu commented on HBASE-20565: -- bq. FilterList can make corresponding ordering adjustment without resorting to instanceof. What I want to say is: Is filter order independence really more better for user ? If order matter, CPF(5,1) AND CRF(1,9) means the cells[1..5] whose column is in 1..9 will be ommitted, CRF(1,9) AND CPF(5,1) means for the cells whose column is 1..9, all the first 6 cells except the first one will be ommitted. If order independence, we can only express one kind of semantics, which one will be more suitable ? IMO, I would like to keep the semantics just as the previous implementation before HBASE-18410. > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 2.1.0, 1.4.4, 2.0.0 >Reporter: Jerry He >Assignee: Zheng Hu >Priority: Major > Fix For: 2.1.0, 1.5.0, 1.4.6, 2.0.2 > > Attachments: HBASE-20565.v1.patch, HBASE-20565.v2.patch, debug.diff, > debug.log, test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549156#comment-16549156 ] Ted Yu commented on HBASE-20565: Can we introduce some method in Filter which tells FilterList that the underlying Filter class is count-related filter ? This way, FilterList can make corresponding ordering adjustment without resorting to instanceof. > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 2.1.0, 1.4.4, 2.0.0 >Reporter: Jerry He >Assignee: Zheng Hu >Priority: Major > Fix For: 2.1.0, 1.5.0, 1.4.6, 2.0.2 > > Attachments: HBASE-20565.v1.patch, HBASE-20565.v2.patch, debug.diff, > debug.log, test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549143#comment-16549143 ] Zheng Hu commented on HBASE-20565: -- Got your mind, but I don't think it's a good taste... > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 2.1.0, 1.4.4, 2.0.0 >Reporter: Jerry He >Assignee: Zheng Hu >Priority: Major > Fix For: 2.1.0, 1.5.0, 1.4.6, 2.0.2 > > Attachments: HBASE-20565.v1.patch, HBASE-20565.v2.patch, debug.diff, > debug.log, test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549138#comment-16549138 ] Ted Yu commented on HBASE-20565: What I meant is to use instanceof checks against ColumnRangeFilter and ColumnPaginationFilter classes when MUST_PASS_ALL is specified for FilterList. We can adjust the order of the column filters to conform to: bq. ColumnPaginationFilter should be placed after ColumnRangeFilter This way, bq. FilterList(MUST_PASS_ALL, ColumnPaginationFilter(5, 1), ColumnRangeFilter(1, true, 9, false)) would still work. > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 2.1.0, 1.4.4, 2.0.0 >Reporter: Jerry He >Assignee: Zheng Hu >Priority: Major > Fix For: 2.1.0, 1.5.0, 1.4.6, 2.0.2 > > Attachments: HBASE-20565.v1.patch, HBASE-20565.v2.patch, debug.diff, > debug.log, test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[
https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549135#comment-16549135
]
Zheng Hu commented on HBASE-20565:
--
[~yuzhih...@gmail.com], I think you mean
{code}
filterList = FilterList(MUST_PASS_ALL, ColumnPaginationFilter(5, 1),
ColumnRangeFilter(1, true, 9, false))
{code}
can still return the same result as
{code}
FilterList(MUST_PASS_ALL, ColumnRangeFilter(1, true, 9, false),
ColumnPaginationFilter(5, 1))
{code}
So for the former one, maintain the count variable in ColumnPaginationFilter
need to consider the following Filter's return codes... If not INCLUDE*, then
the count in ColumnPaginationFilter can not increment (If increment, UT in this
patch will fail.)... It will make the FilterList code very complicated... I
don't think it's worth us to implement this...
On the other hand, the offset in ColumnPaginationFilter is the relative offset
for emitted cells, So, from the user's point of view, it's easy to understand
that ColumnPaginationFilter should be placed after ColumnRangeFilter.
Thanks.
> ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect
> result since 1.4
> -
>
> Key: HBASE-20565
> URL: https://issues.apache.org/jira/browse/HBASE-20565
> Project: HBase
> Issue Type: Bug
> Components: Filters
>Affects Versions: 2.1.0, 1.4.4, 2.0.0
>Reporter: Jerry He
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 2.1.0, 1.5.0, 1.4.6, 2.0.2
>
> Attachments: HBASE-20565.v1.patch, HBASE-20565.v2.patch, debug.diff,
> debug.log, test-branch-1.4.patch
>
>
> When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see
> incorrect result.
> Here is a simple example.
> One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for
> range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset
> 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter,
> ColumnPaginationFilter).
> We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns
> are returned.
> In 1.2.x, the correct 5 columns are returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549103#comment-16549103 ] Ted Yu commented on HBASE-20565: Can we arrange the order of ColumnRangeFilter and ColumnPaginationFilter specifically in FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, ColumnPaginationFilter) scenario so that even if user passes the column filters in arbitrary order to FilterList, the fix in this JIRA is still effective ? > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 2.1.0, 1.4.4, 2.0.0 >Reporter: Jerry He >Assignee: Zheng Hu >Priority: Major > Fix For: 2.1.0, 1.5.0, 1.4.6, 2.0.2 > > Attachments: HBASE-20565.v1.patch, HBASE-20565.v2.patch, debug.diff, > debug.log, test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548776#comment-16548776 ] Zheng Hu commented on HBASE-20565: -- [~reidchan] This patch do not fix the problem in HBASE-20151... it's seems another diff problem. Will dig HBASE-20151 later .. > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 2.1.0, 1.4.4, 2.0.0 >Reporter: Jerry He >Assignee: Zheng Hu >Priority: Major > Fix For: 2.1.0, 1.5.0, 1.4.6, 2.0.2 > > Attachments: HBASE-20565.v1.patch, HBASE-20565.v2.patch, debug.diff, > debug.log, test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548702#comment-16548702 ] Reid Chan commented on HBASE-20565: --- Can you try your patch with the described test cases in HBASE-20151, just wondering since both issues happen on FilterAnd. It would be good if this patch is general enough to solve both. > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 2.1.0, 1.4.4, 2.0.0 >Reporter: Jerry He >Assignee: Zheng Hu >Priority: Major > Fix For: 2.1.0, 1.5.0, 1.4.6, 2.0.2 > > Attachments: HBASE-20565.v1.patch, HBASE-20565.v2.patch, debug.diff, > debug.log, test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548680#comment-16548680 ] Zheng Hu commented on HBASE-20565: -- Any other concerns ? [~yuzhih...@gmail.com], [~jinghe], [~anoop.hbase] Thanks. > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 2.1.0, 1.4.4, 2.0.0 >Reporter: Jerry He >Assignee: Zheng Hu >Priority: Major > Fix For: 2.1.0, 1.5.0, 1.4.6, 2.0.2 > > Attachments: HBASE-20565.v1.patch, HBASE-20565.v2.patch, debug.diff, > debug.log, test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548679#comment-16548679 ] Zheng Hu commented on HBASE-20565: -- bq. If the order of filters in the FilterList changes, e.g. It's the expected behavior. as I comment above. bq. Does this mean that the hint is not utilized with the fix ? This is because cell pass to filter4 will return NEXT_COL, so no need to pass this cell to filter5 & filter6, even though filter5/filter6 may jump to a farther place... > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 2.1.0, 1.4.4, 2.0.0 >Reporter: Jerry He >Assignee: Zheng Hu >Priority: Major > Fix For: 2.1.0, 1.5.0, 1.4.6, 2.0.2 > > Attachments: HBASE-20565.v1.patch, HBASE-20565.v2.patch, debug.diff, > debug.log, test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[
https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548415#comment-16548415
]
Hadoop QA commented on HBASE-20565:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
19s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
21s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
29s{color} | {color:red} hbase-client: The patch generated 1 new + 22 unchanged
- 11 fixed = 23 total (was 33) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
20s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 0s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
7s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}160m
18s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}212m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20565 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12932040/HBASE-20565.v2.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux ac7c75520e6f 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14
08:53:28 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 8c85763327 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe;
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[
https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16547639#comment-16547639
]
Ted Yu commented on HBASE-20565:
I am a little curious about the following change:
{code}
filterList = new FilterList(Operator.MUST_PASS_ALL, filter4, filter5,
filter6);
-assertEquals(ReturnCode.SEEK_NEXT_USING_HINT, filterList.filterCell(kv1));
+assertEquals(ReturnCode.NEXT_COL, filterList.filterCell(kv1));
{code}
Does this mean that the hint is not utilized with the fix ?
> ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect
> result since 1.4
> -
>
> Key: HBASE-20565
> URL: https://issues.apache.org/jira/browse/HBASE-20565
> Project: HBase
> Issue Type: Bug
> Components: Filters
>Affects Versions: 2.1.0, 1.4.4, 2.0.0
>Reporter: Jerry He
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 2.1.0, 1.5.0, 1.4.6, 2.0.2
>
> Attachments: HBASE-20565.v1.patch, HBASE-20565.v2.patch, debug.diff,
> debug.log, test-branch-1.4.patch
>
>
> When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see
> incorrect result.
> Here is a simple example.
> One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for
> range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset
> 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter,
> ColumnPaginationFilter).
> We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns
> are returned.
> In 1.2.x, the correct 5 columns are returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[
https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16547634#comment-16547634
]
Ted Yu commented on HBASE-20565:
If the order of filters in the FilterList changes, e.g.
{code}
assertEquals(5, cellsCount(ht, new FilterList(Operator.MUST_PASS_ALL,
filter2, filter1)));
{code}
I would get:
{code}
TestColumnRangeFilterWithColumnPaginationFilter(org.apache.hadoop.hbase.filter.TestColumnRangeFilter)
Time elapsed: 4.319 sec <<< FAILURE!
java.lang.AssertionError: expected:<5> but was:<4>
at
org.apache.hadoop.hbase.filter.TestColumnRangeFilter.TestColumnRangeFilterWithColumnPaginationFilter(TestColumnRangeFilter.java:275)
{code}
> ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect
> result since 1.4
> -
>
> Key: HBASE-20565
> URL: https://issues.apache.org/jira/browse/HBASE-20565
> Project: HBase
> Issue Type: Bug
> Components: Filters
>Affects Versions: 2.1.0, 1.4.4, 2.0.0
>Reporter: Jerry He
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 2.1.0, 1.5.0, 1.4.6, 2.0.2
>
> Attachments: HBASE-20565.v1.patch, HBASE-20565.v2.patch, debug.diff,
> debug.log, test-branch-1.4.patch
>
>
> When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see
> incorrect result.
> Here is a simple example.
> One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for
> range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset
> 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter,
> ColumnPaginationFilter).
> We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns
> are returned.
> In 1.2.x, the correct 5 columns are returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16547492#comment-16547492 ] Zheng Hu commented on HBASE-20565: -- Upload the patch.v2 to fix the UT. Thanks [~yuzhih...@gmail.com]. > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 2.1.0, 1.4.4, 2.0.0 >Reporter: Jerry He >Assignee: Zheng Hu >Priority: Major > Fix For: 2.1.0, 1.5.0, 1.4.6, 2.0.2 > > Attachments: HBASE-20565.v1.patch, HBASE-20565.v2.patch, debug.diff, > debug.log, test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[
https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16546711#comment-16546711
]
Hadoop QA commented on HBASE-20565:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
35s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
30s{color} | {color:red} hbase-client: The patch generated 1 new + 32 unchanged
- 1 fixed = 33 total (was 33) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
33s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 1s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
58s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 20m 10s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 69m 29s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.filter.TestFilterList |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20565 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12931938/HBASE-20565.v1.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux ad7290c50136 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 2997b6d071 |
| maven | version:
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[
https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16546709#comment-16546709
]
Ted Yu commented on HBASE-20565:
Running Filter tests with patch I got:
{code}
[ERROR] TestFilterList.testReversedFilterListWithMockSeekHintFilter:855
expected: but was:
[ERROR] TestFilterList.testTheMaximalRule:747 expected:
but was:
[ERROR] Errors:
[ERROR] TestFilterList.testHintPassThru:528 » NullPointer
{code}
Please fix failing tests.
> ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect
> result since 1.4
> -
>
> Key: HBASE-20565
> URL: https://issues.apache.org/jira/browse/HBASE-20565
> Project: HBase
> Issue Type: Bug
> Components: Filters
>Affects Versions: 2.1.0, 1.4.4, 2.0.0
>Reporter: Jerry He
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 2.1.0, 1.5.0, 1.4.6, 2.0.2
>
> Attachments: HBASE-20565.v1.patch, debug.diff, debug.log,
> test-branch-1.4.patch
>
>
> When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see
> incorrect result.
> Here is a simple example.
> One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for
> range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset
> 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter,
> ColumnPaginationFilter).
> We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns
> are returned.
> In 1.2.x, the correct 5 columns are returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16546584#comment-16546584 ] Zheng Hu commented on HBASE-20565: -- Upload the patch.v1, and pasted the discuss with [~anoop.hbase] ... > What if the order of filters be opposite way in FL? A good question, I think we need to tell the user explicitly to place the count-related filters at the last position. In SQL syntax, we accept the sql : select * from table where xxx and xxx limit 1, 100, the limit is at the end of the statement, sql such as: select * from table where xxx limit 1, 1000 and xx will not be accepted. I think it's meaningful to require the count-related filters put at the end of sub-filters. On Fri, May 25, 2018 at 6:25 PM, Anoop John wrote: > if a cell has been filtered out by filter-A, then no need to pass the cell to filter-B and filter-C, only the included cell set of filter-A should be passed to filter-B, and only the included cell set of filter-A & filter-B should be passed to filter-C ... U mean u propose such a change now? Then the order of filters matters right? Say the count based filter is coming second and the other (which can filter out some cells) come as 1st, it will work. What if the order of filters be opposite way in FL? -Anoop- On Fri, May 25, 2018 at 12:29 PM, OpenInx wrote: > I have to admit that my previous solution was one-sided... > Not only the ColumnPaginationFilter has the problem, other counter-related > filters also has the problem too. > >> We have 2 filters in a FL. We pass cell 1 and 2. First filter select cell1 >> but been filtered out by F2. Now we need to tell both filters that we >> have excludes this cell. This will be useful for filters which work on >> counting basis. It can reduce the counter which it would have advanced. >> Pls see the possibility. > > Assume that FilterList = filter-A AND ColumnCountGetFilter , if cell x > has been filtered out by filter-A, then what the expected return code do > the ColumnCountGetFilter#filterKeyValue shoud return ? > In theory, the count in ColumnCountGetFilter should not increment when > checking the cell x . So what is the purpose of passing the cell x to > ColumnCountGetFilter#filterKeyValue ? > To get the return code from ColumnCountGetFilter for max the forward step ? > > Now, I'm thinking that the implementation in branch-1.2 is more reasonable, > Assume that filterList = filter-A AND filter-B AND filter-C AND , > if a cell has been filtered out by filter-A, then no need to > pass the cell to filter-B and filter-C, only the included cell set of > filter-A should be passed to filter-B, and only the included cell set of > filter-A & filter-B should be passed to filter-C > > The max rule can still working, but only the include* return code should be > merged into a max return code. > > I think the semantic is more reasonable. > > > On Thu, May 24, 2018 at 4:31 PM, Anoop John wrote: >> >> The offset is the cell offset in a row na. This says we already fetched >> till there. So ya of there is another filter also along with this pagination >> filter, it must be hard for the pagination filter to decide the column >> offset for the next request. So ya ideally the column offset might work >> there. >> But the issue is we can not really generalize this. It depends on the way >> the col offset and column value offset is been implemented in pagination >> filter. >> >> I kind of thinking that we need a generic framework change now. If we pass >> all cells to all filters ( which is correct also) then there should be a way >> later with which we say all filters that we decided later that this cell is >> not included in result. >> >> We have 2 filters in a FL. We pass cell 1 and 2. First filter select cell1 >> but been filtered out by F2. Now we need to tell both filters that we have >> excludes this cell. This will be useful for filters which work on counting >> basis. It can reduce the counter which it would have advanced. Pls see the >> possibility. >> >> I think previously the issue was the order of filters in FL mattered as we >> wont pass all cells to all filters. Now that is not an issue. But the later >> filters possibly filtering out cells still an issue. WDYT? >> >> Anoop >> >> >> On Wednesday, May 23, 2018, OpenInx wrote: >>> >>> > That previously if we have A and >>> > B in FilterList with AND and if A is not including a cell, we were not >>> > passing that to B? (In 1.2 I mean) and in later versions we start >>> > passing it? >>> >>> Yes, you can see the code in branch-1.2 [1], and in master branch [2]. >>> >>> > I think it is a bug .. >>> Sure, it's a bug. >>> >>> > Say if we have another >>> > filter which might filter out some in between cells, then also we need >>> > to have 5 cells to be included. >>> If so , the offset is meaningless now, only the limit is
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544805#comment-16544805 ] Jerry He commented on HBASE-20565: -- [~openinx] Thanks for the update. I can twist the application to use ColumnRangeFilter (startColumn, endColumn), and ColumnPaginationFilter(limit, column-name-offset). It is ok to do it, and easy when starting from the beginning since column-name-offset is the startColumn. But then we will have to know the last retrieved column name to set the next round. > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 1.4.4 >Reporter: Jerry He >Assignee: Zheng Hu >Priority: Major > Attachments: debug.diff, debug.log, test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544730#comment-16544730 ] Zheng Hu commented on HBASE-20565: -- [~jinghe], I've talked with [~anoop.hbase], I think we have a basic idea to fix this bug. I'll upload the patch in these days. Thanks. > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 1.4.4 >Reporter: Jerry He >Assignee: Zheng Hu >Priority: Major > Attachments: debug.diff, debug.log, test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541187#comment-16541187 ] Jerry He commented on HBASE-20565: -- [~openinx] Is there any good news on this one? I think we need to fix it. Thanks. > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 1.4.4 >Reporter: Jerry He >Assignee: Zheng Hu >Priority: Major > Attachments: debug.diff, debug.log, test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480113#comment-16480113 ] Ted Yu commented on HBASE-20565: bq. so the x is unstable You mean the number of cells is unpredictable. > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 1.4.4 >Reporter: Jerry He >Assignee: Zheng Hu >Priority: Major > Attachments: debug.diff, debug.log, test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480099#comment-16480099 ] Zheng Hu commented on HBASE-20565: -- bq. Is the scenario limited to FilterListWithAND only ? No , FilterListWithOR has the similar problem too ... because currently the offset is neither an absolute offset nor relative offset, if we passed x cells to FilterList (WithAND or WithOR), then then the first offset cell will be abandoned, and start to include cell from the next , for ColumnPaginationFilter(offset, limit) but some times sub-filters return SEEK_NEXT_USING_HINT or some other RC, so the x is unstable, which means the offset is unstable (see [~jinghe]'s testing. ) > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 1.4.4 >Reporter: Jerry He >Assignee: Zheng Hu >Priority: Major > Attachments: debug.diff, debug.log, test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480083#comment-16480083 ] Ted Yu commented on HBASE-20565: Is the scenario limited to FilterListWithAND only ? If so, we can consider turning off the max step rule for FilterListWithAND. > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 1.4.4 >Reporter: Jerry He >Assignee: Zheng Hu >Priority: Major > Attachments: debug.diff, debug.log, test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480080#comment-16480080 ] Zheng Hu commented on HBASE-20565: -- bq. Is it possible to formalize the condition(s) where Max Step Rule doesn't apply ? No so easy ? unless we abstract a method named shouldAppyMaxStepRule (or something) for Filter, and handle this in FilterListWithAND. Seems that it's not a good practice. In fact, FilterList concated by sub-filter-X and filter which maintain a offset will hit this bug ? > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 1.4.4 >Reporter: Jerry He >Assignee: Zheng Hu >Priority: Major > Attachments: debug.diff, debug.log, test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480069#comment-16480069 ] Ted Yu commented on HBASE-20565: bq. FilterList concated by ColumnRangeFilter & ColumnPaginationFilter (offset, limit) is in conflict with the Max Step Rule Is it possible to formalize the condition(s) where Max Step Rule doesn't apply ? If so, Max Step Rule can be turned off in those scenarios. > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 1.4.4 >Reporter: Jerry He >Assignee: Zheng Hu >Priority: Major > Attachments: debug.diff, debug.log, test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480064#comment-16480064 ] Zheng Hu commented on HBASE-20565: -- FilterList concated by ColumnRangeFilter & ColumnPaginationFilter (offset, limit) is in conflict with the Max Step Rule. but FilterList concated by ColumnRangeFilter & ColumnPaginationFilter(limit) or ColumnPaginationFilter(byte[], limit) has no problem.Does it make sense that setting an offset & limit for ColumnPaginationFilter ? This scene can be completely replaced by ColumnPaginationFilter(byte[], limit) ? [~anoop.hbase], how do you think ? > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 1.4.4 >Reporter: Jerry He >Assignee: Zheng Hu >Priority: Major > Attachments: debug.diff, debug.log, test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473750#comment-16473750 ] Zheng Hu commented on HBASE-20565: -- In branch 1.2, if A's return code is not INCLUDE_*, then just return the code without passing the cell to B (so will not update B's column count), so no problem in this case, but ignore the global forward step. see [1] 1. https://github.com/apache/hbase/blob/branch-1.2/hbase-client/src/main/java/org/apache/hadoop/hbase/filter/FilterList.java#L272 > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 1.4.4 >Reporter: Jerry He >Assignee: Zheng Hu >Priority: Major > Attachments: debug.diff, debug.log, test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473738#comment-16473738 ] Zheng Hu commented on HBASE-20565: -- It's complex now...if a FilterListWithAND has two sub filters A & B, and B has a row-level global state (such as column limit...), after HBASE-18410, we optimze the forward step to be maximum, so we consider the return code from sub-filters at global level, which means that a cell may not be included the sub-filter A, but we still need to pass the cell to sub-filter B for calculating B' return code (for global return code purpose), finally B's row-level global state messed up ... > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 1.4.4 >Reporter: Jerry He >Assignee: Zheng Hu >Priority: Major > Attachments: debug.diff, debug.log, test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472775#comment-16472775 ] Jerry He commented on HBASE-20565: -- You can see the results from branch-1.2 makes more sense and have correctly identify the ColumnPaginationFilter's offset of either 0 or 1 relative to the ColumnRangeFilter's range. > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 1.4.4 >Reporter: Jerry He >Assignee: Zheng Hu >Priority: Major > Attachments: debug.diff, debug.log, test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472774#comment-16472774 ] Jerry He commented on HBASE-20565: -- Hi, [~openinx] Thanks for taking a look! i did a little more testing with the test patch. Here is what I got: Branch 1.4: --- ColumnRangeFilter StringRange(“1”, true, "9", false), ColumnPaginationFilter(5, 0) Result: 0, 1, 2, 3, 4 StringRange(“1”, true, "9", false), ColumnPaginationFilter(5, 0) Result: 1, 2, 3, 4 StringRange(“3”, true, "9", false), ColumnPaginationFilter(5, 0) Result: 3, 4, 5, 6 StringRange(“0”, true, "9", false), ColumnPaginationFilter(5, 1) Result: 1, 2, 3, 4, 5 StringRange(“1”, true, "9", false), ColumnPaginationFilter(5, 1) Result: 1, 2, 3, 4 , 5 StringRange(“3”, true, "9", false), ColumnPaginationFilter(5, 1) Result: 3, 4, 5, 6, 7 Branch 1.2 --- ColumnRangeFilter StringRange(“0”, true, "9", false), ColumnPaginationFilter(5, 0) Result: 0, 1, 2, 3, 4 StringRange(“1”, true, "9", false), ColumnPaginationFilter(5, 0) Result: 1, 2, 3, 4, 5 StringRange(“3”, true, "9", false), ColumnPaginationFilter(5, 0) Result: 3, 4, 5, 6, 7 StringRange(“0”, true, "9", false), ColumnPaginationFilter(5, 1) Result: 1, 2, 3, 4, 5 StringRange(“1”, true, "9", false), ColumnPaginationFilter(5, 1) Result: 2, 3, 4 , 5, 6 StringRange(“3”, true, "9", false), ColumnPaginationFilter(5, 1) Result: 4, 5, 6, 7, 8 > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 1.4.4 >Reporter: Jerry He >Assignee: Zheng Hu >Priority: Major > Attachments: debug.diff, debug.log, test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[
https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471531#comment-16471531
]
Zheng Hu commented on HBASE-20565:
--
Oh, change my mind now, it's not bug for branch-1.4, see below:
{code}
ColumnPaginationFilter
^
|---|
0 1 2 3 4 5 6 7 8 9
|---|
|
v
ColumnRangeFilter
{code}
The union set of ColumnPaginationFilter and ColumnRangeFilter should be
[1,2,3,4], so the size of results should be 4.
[~jinghe], I think the correct filterList to meet your requirement would be:
{code}
// Column range from 1 to 9.
StringRange stringRange = new StringRange("1", true, "9", false);
ColumnRangeFilter filter1 =
new ColumnRangeFilter(Bytes.toBytes(stringRange.getStart()),
stringRange.isStartInclusive(),
Bytes.toBytes(stringRange.getEnd()), stringRange.isEndInclusive());
// ColumnPaginationFilter limit 5
ColumnPaginationFilter filter2 = new ColumnPaginationFilter(5,
Bytes.toBytes("1")); // here, we use columnOffset instead of offset=0.
FilterList filterList = new FilterList(Operator.MUST_PASS_ALL, filter1,
filter2);
{code}
> ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect
> result since 1.4
> -
>
> Key: HBASE-20565
> URL: https://issues.apache.org/jira/browse/HBASE-20565
> Project: HBase
> Issue Type: Bug
> Components: Filters
>Affects Versions: 1.4.4
>Reporter: Jerry He
>Assignee: Zheng Hu
>Priority: Major
> Attachments: debug.diff, debug.log, test-branch-1.4.patch
>
>
> When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see
> incorrect result.
> Here is a simple example.
> One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for
> range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset
> 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter,
> ColumnPaginationFilter).
> We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns
> are returned.
> In 1.2.x, the correct 5 columns are returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[
https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471467#comment-16471467
]
Zheng Hu commented on HBASE-20565:
--
Add some log by debug.diff, and the result is shown in debug.log. I think the
bug is not from FilterList, but is from ColumnPaginationFilter#filterKeyValue.
{code}
public ReturnCode filterKeyValue(Cell v)
{
if (columnOffset != null) {
if (count >= limit) {
return ReturnCode.NEXT_ROW;
}
byte[] buffer = v.getQualifierArray();
if (buffer == null) {
return ReturnCode.SEEK_NEXT_USING_HINT;
}
int cmp = 0;
// Only compare if no KV's have been seen so far.
if (count == 0) {
cmp = Bytes.compareTo(buffer,
v.getQualifierOffset(),
v.getQualifierLength(),
this.columnOffset,
0,
this.columnOffset.length);
}
if (cmp < 0) {
return ReturnCode.SEEK_NEXT_USING_HINT;
} else {
count++;
return ReturnCode.INCLUDE_AND_NEXT_COL;
}
} else {
if (count >= offset + limit) {
return ReturnCode.NEXT_ROW;
}
ReturnCode code = count < offset ? ReturnCode.NEXT_COL :
ReturnCode.INCLUDE_AND_NEXT_COL;
count++; <--- the count increment even if return
NEXT_COL.
return code;
}
}
{code}
The count increment even if we return NEXT_COL, so after the filter checked
the column=Family:0, count=1, when filter is checking the column=Family:5,
the count is 5 now, so return a NEXT_ROW...
we should define our count as the number of included column in
ColumnPaginationFilter (just as the def in if (columnOffset != null) {} ),
rather than the index of column we encountered. ..
> ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect
> result since 1.4
> -
>
> Key: HBASE-20565
> URL: https://issues.apache.org/jira/browse/HBASE-20565
> Project: HBase
> Issue Type: Bug
> Components: Filters
>Affects Versions: 1.4.4
>Reporter: Jerry He
>Assignee: Zheng Hu
>Priority: Major
> Attachments: debug.diff, debug.log, test-branch-1.4.patch
>
>
> When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see
> incorrect result.
> Here is a simple example.
> One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for
> range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset
> 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter,
> ColumnPaginationFilter).
> We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns
> are returned.
> In 1.2.x, the correct 5 columns are returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471417#comment-16471417 ] Zheng Hu commented on HBASE-20565: -- [~jinghe], Thanks for your report. Seems like it's a bug. Let me dig this . > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 1.4.4 >Reporter: Jerry He >Priority: Major > Attachments: test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471024#comment-16471024 ] Jerry He commented on HBASE-20565: -- Some research shows it is caused by HBASE-18993. FYI . [~openinx], [~apurtell] > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 1.4.4 >Reporter: Jerry He >Priority: Major > Attachments: test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471010#comment-16471010 ] Jerry He commented on HBASE-20565: -- I attached a test case that can be used to re-create the problem. The test passes in branch 1.2, but fails in 1.4 and later. > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 1.4.4 >Reporter: Jerry He >Priority: Major > Attachments: test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
