[jira] [Commented] (PHOENIX-5753) Fix erroneous query result when RVC is clipped with desc column
[
https://issues.apache.org/jira/browse/PHOENIX-5753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17749879#comment-17749879
]
ASF GitHub Bot commented on PHOENIX-5753:
-
stoty closed pull request #719: PHOENIX-5753
URL: https://github.com/apache/phoenix/pull/719
> Fix erroneous query result when RVC is clipped with desc column
> ---
>
> Key: PHOENIX-5753
> URL: https://issues.apache.org/jira/browse/PHOENIX-5753
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 5.0.0, 4.15.0
>Reporter: chenglei
>Assignee: chenglei
>Priority: Major
> Labels: DESC
> Fix For: 5.1.0, 4.16.0
>
> Attachments: PHOENIX-5753_v2-4.x-HBase-1.4.patch,
> PHOENIX-5753_v5-4.x.patch, PHOENIX-5753_v5-master.patch,
> PHOENIX-5753_v6-master.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Given following table and data:
> {code:java}
>CREATE TABLE test
>(
> pk1 INTEGER NOT NULL ,
> pk2 INTEGER NOT NULL,
> pk3 INTEGER NOT NULL,
> pk4 INTEGER NOT NULL,
> v INTEGER, CONSTRAINT PK PRIMARY KEY(pk1,pk2,pk3 desc,pk4))
>)
> {code}
> Noticed pk3 is DESC.
> {code:java}
>UPSERT INTO test (pk1, pk2, pk3, pk4, v) VALUES (1,3,4,10,1)
> {code}
> If we execute the following sql:
> {code:java}
> select * from test
> where (pk1 >=1 and pk1<=2) and (pk2>=3 and pk2<=4) and (pk3,pk4) < (5,7)
> {code}
> the returned result is empty, but obviously, the above inserted row
> (1,3,4,10,1) should be returned.
> I think this problem is introduced by PHOENIX-3383 and PHOENIX-4841, when we
> clip the {{(pk3,pk4) < (5,7)}} because {{pk3}} is {{DESC}} by following line
> 260 in {{WhereOptimizer.pushKeyExpressionsToScan}} , {{(pk3,pk4) < (5,7)}} is
> clipped to {{pk3 <= 5}} and {{pk4 < 7}} .
> {code:java}
> 257List leftRanges = clipLeft(schema,
> slot.getPKPosition()
> 258+ slotOffset - clipLeftSpan, clipLeftSpan,
> keyRanges, ptr);
> 259keyRanges =
> 260clipRight(schema, slot.getPKPosition() +
> slotOffset - 1, keyRanges,
> 261leftRanges, ptr);
> 262if (prevSortOrder == SortOrder.DESC) {
> 263leftRanges = invertKeyRanges(leftRanges);
> 264}
> 265slotSpanArray[cnf.size()] = clipLeftSpan-1;
> 266cnf.add(leftRanges);
> 267clipLeftSpan = 0;
> 268prevSortOrder = sortOrder;
> 269// since we have to clip the portion with the same
> sort order, we can no longer
> 270// extract the nodes from the where clause
> 271// for eg. for the schema A VARCHAR DESC, B VARCHAR
> ASC and query
> 272// WHERE (A,B) < ('a','b')
> 273// the range (* - a\xFFb) is converted to (~a-*)(*-b)
> 274// so we still need to filter on A,B
> 275stopExtracting = true;
> 276}
> {code}
> Eventually after we completed the
> {{WhereOptimizer.pushKeyExpressionsToScan}}, the result
> {{ScanRanges.ranges}} is [[[1 - 2]], [[3 - 4]], [[~5 - *)], [(* - 7)]],
> {{ScanRanges.useSkipScanFilter}} is {{true}} and {{SkipScanFilter}} is also
> [[[1 - 2]], [[3 - 4]], [[~5 - *)], [(* - 7)]], so the the above inserted row
> (1,3,4,10,1) could not be retrieved.
> But as we know, {{(pk3,pk4) < (5,7)}} is not semantically equals to {{pk3 <=
> 5}} and {{pk4 < 7}} , we could only have
> {{pk3 <= 5}} but not {{pk4 < 7}}, so when we clipped {{(pk3,pk4) <
> (5,7)}} to {{pk3 <= 5}} , we could simply skip remaining columns of this
> RVC.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (PHOENIX-5753) Fix erroneous query result when RVC is clipped with desc column
[
https://issues.apache.org/jira/browse/PHOENIX-5753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17749878#comment-17749878
]
ASF GitHub Bot commented on PHOENIX-5753:
-
stoty commented on PR #719:
URL: https://github.com/apache/phoenix/pull/719#issuecomment-1660502677
Already merged.
> Fix erroneous query result when RVC is clipped with desc column
> ---
>
> Key: PHOENIX-5753
> URL: https://issues.apache.org/jira/browse/PHOENIX-5753
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 5.0.0, 4.15.0
>Reporter: chenglei
>Assignee: chenglei
>Priority: Major
> Labels: DESC
> Fix For: 5.1.0, 4.16.0
>
> Attachments: PHOENIX-5753_v2-4.x-HBase-1.4.patch,
> PHOENIX-5753_v5-4.x.patch, PHOENIX-5753_v5-master.patch,
> PHOENIX-5753_v6-master.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Given following table and data:
> {code:java}
>CREATE TABLE test
>(
> pk1 INTEGER NOT NULL ,
> pk2 INTEGER NOT NULL,
> pk3 INTEGER NOT NULL,
> pk4 INTEGER NOT NULL,
> v INTEGER, CONSTRAINT PK PRIMARY KEY(pk1,pk2,pk3 desc,pk4))
>)
> {code}
> Noticed pk3 is DESC.
> {code:java}
>UPSERT INTO test (pk1, pk2, pk3, pk4, v) VALUES (1,3,4,10,1)
> {code}
> If we execute the following sql:
> {code:java}
> select * from test
> where (pk1 >=1 and pk1<=2) and (pk2>=3 and pk2<=4) and (pk3,pk4) < (5,7)
> {code}
> the returned result is empty, but obviously, the above inserted row
> (1,3,4,10,1) should be returned.
> I think this problem is introduced by PHOENIX-3383 and PHOENIX-4841, when we
> clip the {{(pk3,pk4) < (5,7)}} because {{pk3}} is {{DESC}} by following line
> 260 in {{WhereOptimizer.pushKeyExpressionsToScan}} , {{(pk3,pk4) < (5,7)}} is
> clipped to {{pk3 <= 5}} and {{pk4 < 7}} .
> {code:java}
> 257List leftRanges = clipLeft(schema,
> slot.getPKPosition()
> 258+ slotOffset - clipLeftSpan, clipLeftSpan,
> keyRanges, ptr);
> 259keyRanges =
> 260clipRight(schema, slot.getPKPosition() +
> slotOffset - 1, keyRanges,
> 261leftRanges, ptr);
> 262if (prevSortOrder == SortOrder.DESC) {
> 263leftRanges = invertKeyRanges(leftRanges);
> 264}
> 265slotSpanArray[cnf.size()] = clipLeftSpan-1;
> 266cnf.add(leftRanges);
> 267clipLeftSpan = 0;
> 268prevSortOrder = sortOrder;
> 269// since we have to clip the portion with the same
> sort order, we can no longer
> 270// extract the nodes from the where clause
> 271// for eg. for the schema A VARCHAR DESC, B VARCHAR
> ASC and query
> 272// WHERE (A,B) < ('a','b')
> 273// the range (* - a\xFFb) is converted to (~a-*)(*-b)
> 274// so we still need to filter on A,B
> 275stopExtracting = true;
> 276}
> {code}
> Eventually after we completed the
> {{WhereOptimizer.pushKeyExpressionsToScan}}, the result
> {{ScanRanges.ranges}} is [[[1 - 2]], [[3 - 4]], [[~5 - *)], [(* - 7)]],
> {{ScanRanges.useSkipScanFilter}} is {{true}} and {{SkipScanFilter}} is also
> [[[1 - 2]], [[3 - 4]], [[~5 - *)], [(* - 7)]], so the the above inserted row
> (1,3,4,10,1) could not be retrieved.
> But as we know, {{(pk3,pk4) < (5,7)}} is not semantically equals to {{pk3 <=
> 5}} and {{pk4 < 7}} , we could only have
> {{pk3 <= 5}} but not {{pk4 < 7}}, so when we clipped {{(pk3,pk4) <
> (5,7)}} to {{pk3 <= 5}} , we could simply skip remaining columns of this
> RVC.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (PHOENIX-5753) Fix erroneous query result when RVC is clipped with desc column
[
https://issues.apache.org/jira/browse/PHOENIX-5753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17061947#comment-17061947
]
Hudson commented on PHOENIX-5753:
-
FAILURE: Integrated in Jenkins build PreCommit-PHOENIX-Build #3605 (See
[https://builds.apache.org/job/PreCommit-PHOENIX-Build/3605/])
PHOENIX-5753 Fix erroneous query result when RVC is clipped with desc
(chenglei: rev fb4f857696a29777a5ad9d68dd61fd51800f8c6b)
* (edit)
phoenix-core/src/main/java/org/apache/phoenix/compile/WhereOptimizer.java
* (edit)
phoenix-core/src/it/java/org/apache/phoenix/end2end/index/GlobalIndexOptimizationIT.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/query/KeyRange.java
* (edit)
phoenix-core/src/it/java/org/apache/phoenix/end2end/SkipScanQueryIT.java
* (edit)
phoenix-core/src/test/java/org/apache/phoenix/compile/WhereOptimizerTest.java
* (edit)
phoenix-core/src/test/java/org/apache/phoenix/compile/WhereCompilerTest.java
> Fix erroneous query result when RVC is clipped with desc column
> ---
>
> Key: PHOENIX-5753
> URL: https://issues.apache.org/jira/browse/PHOENIX-5753
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 5.0.0, 4.15.0
>Reporter: chenglei
>Assignee: chenglei
>Priority: Major
> Labels: DESC
> Fix For: 4.16.0
>
> Attachments: PHOENIX-5753_v2-4.x-HBase-1.4.patch,
> PHOENIX-5753_v5-4.x.patch, PHOENIX-5753_v5-master.patch,
> PHOENIX-5753_v6-master.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Given following table and data:
> {code:java}
>CREATE TABLE test
>(
> pk1 INTEGER NOT NULL ,
> pk2 INTEGER NOT NULL,
> pk3 INTEGER NOT NULL,
> pk4 INTEGER NOT NULL,
> v INTEGER, CONSTRAINT PK PRIMARY KEY(pk1,pk2,pk3 desc,pk4))
>)
> {code}
> Noticed pk3 is DESC.
> {code:java}
>UPSERT INTO test (pk1, pk2, pk3, pk4, v) VALUES (1,3,4,10,1)
> {code}
> If we execute the following sql:
> {code:java}
> select * from test
> where (pk1 >=1 and pk1<=2) and (pk2>=3 and pk2<=4) and (pk3,pk4) < (5,7)
> {code}
> the returned result is empty, but obviously, the above inserted row
> (1,3,4,10,1) should be returned.
> I think this problem is introduced by PHOENIX-3383 and PHOENIX-4841, when we
> clip the {{(pk3,pk4) < (5,7)}} because {{pk3}} is {{DESC}} by following line
> 260 in {{WhereOptimizer.pushKeyExpressionsToScan}} , {{(pk3,pk4) < (5,7)}} is
> clipped to {{pk3 <= 5}} and {{pk4 < 7}} .
> {code:java}
> 257List leftRanges = clipLeft(schema,
> slot.getPKPosition()
> 258+ slotOffset - clipLeftSpan, clipLeftSpan,
> keyRanges, ptr);
> 259keyRanges =
> 260clipRight(schema, slot.getPKPosition() +
> slotOffset - 1, keyRanges,
> 261leftRanges, ptr);
> 262if (prevSortOrder == SortOrder.DESC) {
> 263leftRanges = invertKeyRanges(leftRanges);
> 264}
> 265slotSpanArray[cnf.size()] = clipLeftSpan-1;
> 266cnf.add(leftRanges);
> 267clipLeftSpan = 0;
> 268prevSortOrder = sortOrder;
> 269// since we have to clip the portion with the same
> sort order, we can no longer
> 270// extract the nodes from the where clause
> 271// for eg. for the schema A VARCHAR DESC, B VARCHAR
> ASC and query
> 272// WHERE (A,B) < ('a','b')
> 273// the range (* - a\xFFb) is converted to (~a-*)(*-b)
> 274// so we still need to filter on A,B
> 275stopExtracting = true;
> 276}
> {code}
> Eventually after we completed the
> {{WhereOptimizer.pushKeyExpressionsToScan}}, the result
> {{ScanRanges.ranges}} is [[[1 - 2]], [[3 - 4]], [[~5 - *)], [(* - 7)]],
> {{ScanRanges.useSkipScanFilter}} is {{true}} and {{SkipScanFilter}} is also
> [[[1 - 2]], [[3 - 4]], [[~5 - *)], [(* - 7)]], so the the above inserted row
> (1,3,4,10,1) could not be retrieved.
> But as we know, {{(pk3,pk4) < (5,7)}} is not semantically equals to {{pk3 <=
> 5}} and {{pk4 < 7}} , we could only have
> {{pk3 <= 5}} but not {{pk4 < 7}}, so when we clipped {{(pk3,pk4) <
> (5,7)}} to {{pk3 <= 5}} , we could simply skip remaining columns of this
> RVC.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (PHOENIX-5753) Fix erroneous query result when RVC is clipped with desc column
[
https://issues.apache.org/jira/browse/PHOENIX-5753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17061116#comment-17061116
]
Hadoop QA commented on PHOENIX-5753:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12996932/PHOENIX-5753_v6-master.patch
against master branch at commit 981aa7bbd1ca01cb5607aa74b541403c3dd89135.
ATTACHMENT ID: 12996932
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+ " v INTEGER, CONSTRAINT PK PRIMARY
KEY(pk1,pk2,pk3 desc,pk4,pk5,pk6 desc,pk7,pk8))";;
+ " where (pk1 >=1 and pk1<=2) and (pk2>=3 and pk2<=4) and
(pk3,pk4) < (5,7) order by pk1,pk2,pk3";
+" where (pk1 >=1 and pk1<=2) and (pk2>=2 and pk2<=3) and
(pk3,pk4) < (5,7) and "+
+" where (pk1 >=1 and pk1<=2) and (pk2>=2 and pk2<=3) and
(pk3,pk4) in ((5,6),(2,10)) and "+
+ * This inner "_IDX_" + dataTableName use skipScan, and all the
whereExpressions are already in SkipScanFilter,
+ * the whereExpressions of current rowkey slot from the current
{@link SelectStatement#where},
+ * So we should stop extracting whereExpressions of current rowkey
slot once we encounter:
+ * so the endKey is computed as {@link
SchemaUtil#VAR_BINARY_SCHEMA},see {@link ScanRanges#create}.
+"where (OBJ.OBJECT_ID, OBJ.OBJECT_VERSION) in (('obj1',
''),('obj2', ''),('obj3', ''))";
+RowKeyComparisonFilter rowKeyComparisonFilter
=(RowKeyComparisonFilter) filterList.getFilters().get(1);
{color:red}-1 core tests{color}. The patch failed these unit tests:
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.AlterTableWithViewsIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.AlterMultiTenantTableWithViewsIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.DropIndexedColsIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.DropTableWithViewsIT
Test results:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/3600//testReport/
Console output:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/3600//console
This message is automatically generated.
> Fix erroneous query result when RVC is clipped with desc column
> ---
>
> Key: PHOENIX-5753
> URL: https://issues.apache.org/jira/browse/PHOENIX-5753
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 5.0.0, 4.15.0
>Reporter: chenglei
>Assignee: chenglei
>Priority: Major
> Labels: DESC
> Fix For: 4.16.0
>
> Attachments: PHOENIX-5753_v2-4.x-HBase-1.4.patch,
> PHOENIX-5753_v5-4.x.patch, PHOENIX-5753_v5-master.patch,
> PHOENIX-5753_v6-master.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Given following table and data:
> {code:java}
>CREATE TABLE test
>(
> pk1 INTEGER NOT NULL ,
> pk2 INTEGER NOT NULL,
> pk3 INTEGER NOT NULL,
> pk4 INTEGER NOT NULL,
> v INTEGER, CONSTRAINT PK PRIMARY KEY(pk1,pk2,pk3 desc,pk4))
>)
> {code}
> Noticed pk3 is DESC.
> {code:java}
>UPSERT INTO test (pk1, pk2, pk3, pk4, v) VALUES (1,3,4,10,1)
> {code}
> If we execute the following sql:
> {code:java}
> select * from test
> where (pk1 >=1 and pk1<=2) and (pk2>=3 and pk2<=4) and (pk3,pk4) < (5,7)
> {code}
> the returned result is empty, but obviously, the above inserted row
> (1,3,4,10,1) should be returned.
> I think this problem is introduced by PHOENIX-3383 and PHOENIX-4841, when we
> clip the {{(pk3,pk4) < (5,7)}} because {{pk3}} is {{DESC}} by following line
> 260 in {{WhereOptimizer.pushKeyExpressionsToScan}} , {{(pk3,pk4) < (5,7)}} is
> clipped to {{pk3 <= 5}} and {{pk4 < 7}} .
> {code:java}
> 257List leftRanges = clipLeft(schema,
> slot.getPKPosition()
> 258+ slotOffset - clipLeftSpan, clipLeftSpan,
> keyRanges, ptr);
> 259keyRanges =
> 260clipRight(schema, slot.getPKPosition() +
> slotOffset - 1, keyRanges,
> 261leftRanges, ptr);
> 262if
[jira] [Commented] (PHOENIX-5753) Fix erroneous query result when RVC is clipped with desc column
[
https://issues.apache.org/jira/browse/PHOENIX-5753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17060797#comment-17060797
]
Hadoop QA commented on PHOENIX-5753:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12996884/PHOENIX-5753_v5-master.patch
against master branch at commit 981aa7bbd1ca01cb5607aa74b541403c3dd89135.
ATTACHMENT ID: 12996884
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+ " v INTEGER, CONSTRAINT PK PRIMARY
KEY(pk1,pk2,pk3 desc,pk4,pk5,pk6 desc,pk7,pk8))";;
+ " where (pk1 >=1 and pk1<=2) and (pk2>=3 and pk2<=4) and
(pk3,pk4) < (5,7) order by pk1,pk2,pk3";
+" where (pk1 >=1 and pk1<=2) and (pk2>=2 and pk2<=3) and
(pk3,pk4) < (5,7) and "+
+" where (pk1 >=1 and pk1<=2) and (pk2>=2 and pk2<=3) and
(pk3,pk4) in ((5,6),(2,10)) and "+
+ * This inner "_IDX_" + dataTableName use skipScan, and all the
whereExpressions are already in SkipScanFilter,
+ * the whereExpressions of current rowkey slot from the current
{@link SelectStatement#where},
+ * So we should stop extracting whereExpressions of current rowkey
slot once we encounter:
+ * so the endKey is computed as {@link
SchemaUtil#VAR_BINARY_SCHEMA},see {@link ScanRanges#create}.
+"where (OBJ.OBJECT_ID, OBJ.OBJECT_VERSION) in (('obj1',
''),('obj2', ''),('obj3', ''))";
+RowKeyComparisonFilter rowKeyComparisonFilter
=(RowKeyComparisonFilter) filterList.getFilters().get(1);
{color:red}-1 core tests{color}. The patch failed these unit tests:
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.SystemCatalogCreationOnConnectionIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.PermissionsCacheIT
Test results:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/3598//testReport/
Console output:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/3598//console
This message is automatically generated.
> Fix erroneous query result when RVC is clipped with desc column
> ---
>
> Key: PHOENIX-5753
> URL: https://issues.apache.org/jira/browse/PHOENIX-5753
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 5.0.0, 4.15.0
>Reporter: chenglei
>Assignee: chenglei
>Priority: Major
> Labels: DESC
> Fix For: 4.16.0
>
> Attachments: PHOENIX-5753_v2-4.x-HBase-1.4.patch,
> PHOENIX-5753_v5-4.x.patch, PHOENIX-5753_v5-master.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Given following table and data:
> {code:java}
>CREATE TABLE test
>(
> pk1 INTEGER NOT NULL ,
> pk2 INTEGER NOT NULL,
> pk3 INTEGER NOT NULL,
> pk4 INTEGER NOT NULL,
> v INTEGER, CONSTRAINT PK PRIMARY KEY(pk1,pk2,pk3 desc,pk4))
>)
> {code}
> Noticed pk3 is DESC.
> {code:java}
>UPSERT INTO test (pk1, pk2, pk3, pk4, v) VALUES (1,3,4,10,1)
> {code}
> If we execute the following sql:
> {code:java}
> select * from test
> where (pk1 >=1 and pk1<=2) and (pk2>=3 and pk2<=4) and (pk3,pk4) < (5,7)
> {code}
> the returned result is empty, but obviously, the above inserted row
> (1,3,4,10,1) should be returned.
> I think this problem is introduced by PHOENIX-3383 and PHOENIX-4841, when we
> clip the {{(pk3,pk4) < (5,7)}} because {{pk3}} is {{DESC}} by following line
> 260 in {{WhereOptimizer.pushKeyExpressionsToScan}} , {{(pk3,pk4) < (5,7)}} is
> clipped to {{pk3 <= 5}} and {{pk4 < 7}} .
> {code:java}
> 257List leftRanges = clipLeft(schema,
> slot.getPKPosition()
> 258+ slotOffset - clipLeftSpan, clipLeftSpan,
> keyRanges, ptr);
> 259keyRanges =
> 260clipRight(schema, slot.getPKPosition() +
> slotOffset - 1, keyRanges,
> 261leftRanges, ptr);
> 262if (prevSortOrder == SortOrder.DESC) {
> 263leftRanges = invertKeyRanges(leftRanges);
> 264}
> 265slotSpanArray[cnf.size()] = clipLeftSpan-1;
> 266
[jira] [Commented] (PHOENIX-5753) Fix erroneous query result when RVC is clipped with desc column
[
https://issues.apache.org/jira/browse/PHOENIX-5753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17059427#comment-17059427
]
Hadoop QA commented on PHOENIX-5753:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12996741/PHOENIX-5753_v5-4.x.patch
against 4.x branch at commit 0442ccf03dd3932b7a178149ef65bfed09a1cb21.
ATTACHMENT ID: 12996741
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+ " v INTEGER, CONSTRAINT PK PRIMARY
KEY(pk1,pk2,pk3 desc,pk4,pk5,pk6 desc,pk7,pk8))";;
+ " where (pk1 >=1 and pk1<=2) and (pk2>=3 and pk2<=4) and
(pk3,pk4) < (5,7) order by pk1,pk2,pk3";
+" where (pk1 >=1 and pk1<=2) and (pk2>=2 and pk2<=3) and
(pk3,pk4) < (5,7) and "+
+" where (pk1 >=1 and pk1<=2) and (pk2>=2 and pk2<=3) and
(pk3,pk4) in ((5,6),(2,10)) and "+
+ * This inner "_IDX_" + dataTableName use skipScan, and all the
whereExpressions are already in SkipScanFilter,
+ * the whereExpressions of current rowkey slot from the current
{@link SelectStatement#where},
+ * So we should stop extracting whereExpressions of current rowkey
slot once we encounter:
+ * so the endKey is computed as {@link
SchemaUtil#VAR_BINARY_SCHEMA},see {@link ScanRanges#create}.
+"where (OBJ.OBJECT_ID, OBJ.OBJECT_VERSION) in (('obj1',
''),('obj2', ''),('obj3', ''))";
+RowKeyComparisonFilter rowKeyComparisonFilter
=(RowKeyComparisonFilter) filterList.getFilters().get(1);
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/3590//testReport/
Console output:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/3590//console
This message is automatically generated.
> Fix erroneous query result when RVC is clipped with desc column
> ---
>
> Key: PHOENIX-5753
> URL: https://issues.apache.org/jira/browse/PHOENIX-5753
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 5.0.0, 4.15.0
>Reporter: chenglei
>Assignee: chenglei
>Priority: Major
> Labels: DESC
> Fix For: 4.16.0
>
> Attachments: PHOENIX-5753_v2-4.x-HBase-1.4.patch,
> PHOENIX-5753_v5-4.x-HBase-1.4.patch, PHOENIX-5753_v5-4.x.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Given following table and data:
> {code:java}
>CREATE TABLE test
>(
> pk1 INTEGER NOT NULL ,
> pk2 INTEGER NOT NULL,
> pk3 INTEGER NOT NULL,
> pk4 INTEGER NOT NULL,
> v INTEGER, CONSTRAINT PK PRIMARY KEY(pk1,pk2,pk3 desc,pk4))
>)
> {code}
> Noticed pk3 is DESC.
> {code:java}
>UPSERT INTO test (pk1, pk2, pk3, pk4, v) VALUES (1,3,4,10,1)
> {code}
> If we execute the following sql:
> {code:java}
> select * from test
> where (pk1 >=1 and pk1<=2) and (pk2>=3 and pk2<=4) and (pk3,pk4) < (5,7)
> {code}
> the returned result is empty, but obviously, the above inserted row
> (1,3,4,10,1) should be returned.
> I think this problem is introduced by PHOENIX-3383 and PHOENIX-4841, when we
> clip the {{(pk3,pk4) < (5,7)}} because {{pk3}} is {{DESC}} by following line
> 260 in {{WhereOptimizer.pushKeyExpressionsToScan}} , {{(pk3,pk4) < (5,7)}} is
> clipped to {{pk3 <= 5}} and {{pk4 < 7}} .
> {code:java}
> 257List leftRanges = clipLeft(schema,
> slot.getPKPosition()
> 258+ slotOffset - clipLeftSpan, clipLeftSpan,
> keyRanges, ptr);
> 259keyRanges =
> 260clipRight(schema, slot.getPKPosition() +
> slotOffset - 1, keyRanges,
> 261leftRanges, ptr);
> 262if (prevSortOrder == SortOrder.DESC) {
> 263leftRanges = invertKeyRanges(leftRanges);
> 264}
> 265slotSpanArray[cnf.size()] = clipLeftSpan-1;
> 266cnf.add(leftRanges);
> 267clipLeftSpan = 0;
> 268prevSortOrder = sortOrder;
> 269// since we have to clip the portion with the
[jira] [Commented] (PHOENIX-5753) Fix erroneous query result when RVC is clipped with desc column
[
https://issues.apache.org/jira/browse/PHOENIX-5753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17058397#comment-17058397
]
Hadoop QA commented on PHOENIX-5753:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12996610/PHOENIX-5753_v3-4.x-HBase-1.4.patch
against 4.x-HBase-1.4 branch at commit
5bcd1ce19d6eda842d933c436623e329e2859e70.
ATTACHMENT ID: 12996610
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/3586//console
This message is automatically generated.
> Fix erroneous query result when RVC is clipped with desc column
> ---
>
> Key: PHOENIX-5753
> URL: https://issues.apache.org/jira/browse/PHOENIX-5753
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 5.0.0, 4.15.0
>Reporter: chenglei
>Assignee: chenglei
>Priority: Major
> Labels: DESC
> Fix For: 4.16.0
>
> Attachments: PHOENIX-5753_v2-4.x-HBase-1.4.patch,
> PHOENIX-5753_v3-4.x-HBase-1.4.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Given following table and data:
> {code:java}
>CREATE TABLE test
>(
> pk1 INTEGER NOT NULL ,
> pk2 INTEGER NOT NULL,
> pk3 INTEGER NOT NULL,
> pk4 INTEGER NOT NULL,
> v INTEGER, CONSTRAINT PK PRIMARY KEY(pk1,pk2,pk3 desc,pk4))
>)
> {code}
> Noticed pk3 is DESC.
> {code:java}
>UPSERT INTO test (pk1, pk2, pk3, pk4, v) VALUES (1,3,4,10,1)
> {code}
> If we execute the following sql:
> {code:java}
> select * from test
> where (pk1 >=1 and pk1<=2) and (pk2>=3 and pk2<=4) and (pk3,pk4) < (5,7)
> {code}
> the returned result is empty, but obviously, the above inserted row
> (1,3,4,10,1) should be returned.
> I think this problem is introduced by PHOENIX-3383 and PHOENIX-4841, when we
> clip the {{(pk3,pk4) < (5,7)}} because {{pk3}} is {{DESC}} by following line
> 260 in {{WhereOptimizer.pushKeyExpressionsToScan}} , {{(pk3,pk4) < (5,7)}} is
> clipped to {{pk3 <= 5}} and {{pk4 < 7}} .
> {code:java}
> 257List leftRanges = clipLeft(schema,
> slot.getPKPosition()
> 258+ slotOffset - clipLeftSpan, clipLeftSpan,
> keyRanges, ptr);
> 259keyRanges =
> 260clipRight(schema, slot.getPKPosition() +
> slotOffset - 1, keyRanges,
> 261leftRanges, ptr);
> 262if (prevSortOrder == SortOrder.DESC) {
> 263leftRanges = invertKeyRanges(leftRanges);
> 264}
> 265slotSpanArray[cnf.size()] = clipLeftSpan-1;
> 266cnf.add(leftRanges);
> 267clipLeftSpan = 0;
> 268prevSortOrder = sortOrder;
> 269// since we have to clip the portion with the same
> sort order, we can no longer
> 270// extract the nodes from the where clause
> 271// for eg. for the schema A VARCHAR DESC, B VARCHAR
> ASC and query
> 272// WHERE (A,B) < ('a','b')
> 273// the range (* - a\xFFb) is converted to (~a-*)(*-b)
> 274// so we still need to filter on A,B
> 275stopExtracting = true;
> 276}
> {code}
> Eventually after we completed the
> {{WhereOptimizer.pushKeyExpressionsToScan}}, the result
> {{ScanRanges.ranges}} is [[[1 - 2]], [[3 - 4]], [[~5 - *)], [(* - 7)]],
> {{ScanRanges.useSkipScanFilter}} is {{true}} and {{SkipScanFilter}} is also
> [[[1 - 2]], [[3 - 4]], [[~5 - *)], [(* - 7)]], so the the above inserted row
> (1,3,4,10,1) could not be retrieved.
> But as we know, {{(pk3,pk4) < (5,7)}} is not semantically equals to {{pk3 <=
> 5}} and {{pk4 < 7}} , we could only have
> {{pk3 <= 5}} but not {{pk4 < 7}}, so when we clipped {{(pk3,pk4) <
> (5,7)}} to {{pk3 <= 5}} , we could simply skip remaining columns of this
> RVC.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (PHOENIX-5753) Fix erroneous query result when RVC is clipped with desc column
[
https://issues.apache.org/jira/browse/PHOENIX-5753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17058389#comment-17058389
]
chenglei commented on PHOENIX-5753:
---
uploaded v3 patch，rebased with latest 4.x-HBase-1.4 branch.
> Fix erroneous query result when RVC is clipped with desc column
> ---
>
> Key: PHOENIX-5753
> URL: https://issues.apache.org/jira/browse/PHOENIX-5753
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 5.0.0, 4.15.0
>Reporter: chenglei
>Assignee: chenglei
>Priority: Major
> Labels: DESC
> Fix For: 4.16.0
>
> Attachments: PHOENIX-5753_v2-4.x-HBase-1.4.patch,
> PHOENIX-5753_v3-4.x-HBase-1.4.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Given following table and data:
> {code:java}
>CREATE TABLE test
>(
> pk1 INTEGER NOT NULL ,
> pk2 INTEGER NOT NULL,
> pk3 INTEGER NOT NULL,
> pk4 INTEGER NOT NULL,
> v INTEGER, CONSTRAINT PK PRIMARY KEY(pk1,pk2,pk3 desc,pk4))
>)
> {code}
> Noticed pk3 is DESC.
> {code:java}
>UPSERT INTO test (pk1, pk2, pk3, pk4, v) VALUES (1,3,4,10,1)
> {code}
> If we execute the following sql:
> {code:java}
> select * from test
> where (pk1 >=1 and pk1<=2) and (pk2>=3 and pk2<=4) and (pk3,pk4) < (5,7)
> {code}
> the returned result is empty, but obviously, the above inserted row
> (1,3,4,10,1) should be returned.
> I think this problem is introduced by PHOENIX-3383 and PHOENIX-4841, when we
> clip the {{(pk3,pk4) < (5,7)}} because {{pk3}} is {{DESC}} by following line
> 260 in {{WhereOptimizer.pushKeyExpressionsToScan}} , {{(pk3,pk4) < (5,7)}} is
> clipped to {{pk3 <= 5}} and {{pk4 < 7}} .
> {code:java}
> 257List leftRanges = clipLeft(schema,
> slot.getPKPosition()
> 258+ slotOffset - clipLeftSpan, clipLeftSpan,
> keyRanges, ptr);
> 259keyRanges =
> 260clipRight(schema, slot.getPKPosition() +
> slotOffset - 1, keyRanges,
> 261leftRanges, ptr);
> 262if (prevSortOrder == SortOrder.DESC) {
> 263leftRanges = invertKeyRanges(leftRanges);
> 264}
> 265slotSpanArray[cnf.size()] = clipLeftSpan-1;
> 266cnf.add(leftRanges);
> 267clipLeftSpan = 0;
> 268prevSortOrder = sortOrder;
> 269// since we have to clip the portion with the same
> sort order, we can no longer
> 270// extract the nodes from the where clause
> 271// for eg. for the schema A VARCHAR DESC, B VARCHAR
> ASC and query
> 272// WHERE (A,B) < ('a','b')
> 273// the range (* - a\xFFb) is converted to (~a-*)(*-b)
> 274// so we still need to filter on A,B
> 275stopExtracting = true;
> 276}
> {code}
> Eventually after we completed the
> {{WhereOptimizer.pushKeyExpressionsToScan}}, the result
> {{ScanRanges.ranges}} is [[[1 - 2]], [[3 - 4]], [[~5 - *)], [(* - 7)]],
> {{ScanRanges.useSkipScanFilter}} is {{true}} and {{SkipScanFilter}} is also
> [[[1 - 2]], [[3 - 4]], [[~5 - *)], [(* - 7)]], so the the above inserted row
> (1,3,4,10,1) could not be retrieved.
> But as we know, {{(pk3,pk4) < (5,7)}} is not semantically equals to {{pk3 <=
> 5}} and {{pk4 < 7}} , we could only have
> {{pk3 <= 5}} but not {{pk4 < 7}}, so when we clipped {{(pk3,pk4) <
> (5,7)}} to {{pk3 <= 5}} , we could simply skip remaining columns of this
> RVC.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (PHOENIX-5753) Fix erroneous query result when RVC is clipped with desc column
[
https://issues.apache.org/jira/browse/PHOENIX-5753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17055622#comment-17055622
]
Chinmay Kulkarni commented on PHOENIX-5753:
---
[~dbwong] [~yanxinyi] since this is similar to what you guys have been working
on, mind taking a look at [~chenglei]’s patch?
> Fix erroneous query result when RVC is clipped with desc column
> ---
>
> Key: PHOENIX-5753
> URL: https://issues.apache.org/jira/browse/PHOENIX-5753
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 5.0.0, 4.15.0
>Reporter: chenglei
>Assignee: chenglei
>Priority: Major
> Labels: DESC
> Fix For: 4.16.0
>
> Attachments: PHOENIX-5753_v2-4.x-HBase-1.4.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Given following table and data:
> {code:java}
>CREATE TABLE test
>(
> pk1 INTEGER NOT NULL ,
> pk2 INTEGER NOT NULL,
> pk3 INTEGER NOT NULL,
> pk4 INTEGER NOT NULL,
> v INTEGER, CONSTRAINT PK PRIMARY KEY(pk1,pk2,pk3 desc,pk4))
>)
> {code}
> Noticed pk3 is DESC.
> {code:java}
>UPSERT INTO test (pk1, pk2, pk3, pk4, v) VALUES (1,3,4,10,1)
> {code}
> If we execute the following sql:
> {code:java}
> select * from test
> where (pk1 >=1 and pk1<=2) and (pk2>=3 and pk2<=4) and (pk3,pk4) < (5,7)
> {code}
> the returned result is empty, but obviously, the above inserted row
> (1,3,4,10,1) should be returned.
> I think this problem is introduced by PHOENIX-3383 and PHOENIX-4841, when we
> clip the {{(pk3,pk4) < (5,7)}} because {{pk3}} is {{DESC}} by following line
> 260 in {{WhereOptimizer.pushKeyExpressionsToScan}} , {{(pk3,pk4) < (5,7)}} is
> clipped to {{pk3 <= 5}} and {{pk4 < 7}} .
> {code:java}
> 257List leftRanges = clipLeft(schema,
> slot.getPKPosition()
> 258+ slotOffset - clipLeftSpan, clipLeftSpan,
> keyRanges, ptr);
> 259keyRanges =
> 260clipRight(schema, slot.getPKPosition() +
> slotOffset - 1, keyRanges,
> 261leftRanges, ptr);
> 262if (prevSortOrder == SortOrder.DESC) {
> 263leftRanges = invertKeyRanges(leftRanges);
> 264}
> 265slotSpanArray[cnf.size()] = clipLeftSpan-1;
> 266cnf.add(leftRanges);
> 267clipLeftSpan = 0;
> 268prevSortOrder = sortOrder;
> 269// since we have to clip the portion with the same
> sort order, we can no longer
> 270// extract the nodes from the where clause
> 271// for eg. for the schema A VARCHAR DESC, B VARCHAR
> ASC and query
> 272// WHERE (A,B) < ('a','b')
> 273// the range (* - a\xFFb) is converted to (~a-*)(*-b)
> 274// so we still need to filter on A,B
> 275stopExtracting = true;
> 276}
> {code}
> Eventually after we completed the
> {{WhereOptimizer.pushKeyExpressionsToScan}}, the result
> {{ScanRanges.ranges}} is [[[1 - 2]], [[3 - 4]], [[~5 - *)], [(* - 7)]],
> {{ScanRanges.useSkipScanFilter}} is {{true}} and {{SkipScanFilter}} is also
> [[[1 - 2]], [[3 - 4]], [[~5 - *)], [(* - 7)]], so the the above inserted row
> (1,3,4,10,1) could not be retrieved.
> But as we know, {{(pk3,pk4) < (5,7)}} is not semantically equals to {{pk3 <=
> 5}} and {{pk4 < 7}} , we could only have
> {{pk3 <= 5}} but not {{pk4 < 7}}, so when we clipped {{(pk3,pk4) <
> (5,7)}} to {{pk3 <= 5}} , we could simply skip remaining columns of this
> RVC.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (PHOENIX-5753) Fix erroneous query result when RVC is clipped with desc column
[
https://issues.apache.org/jira/browse/PHOENIX-5753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048801#comment-17048801
]
Hadoop QA commented on PHOENIX-5753:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12995082/PHOENIX-5753_v2-4.x-HBase-1.4.patch
against 4.x-HBase-1.4 branch at commit
4c2f481f82bb1bcf5f62d009dd20854ef35b13a8.
ATTACHMENT ID: 12995082
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+ " v INTEGER, CONSTRAINT PK PRIMARY
KEY(pk1,pk2,pk3 desc,pk4,pk5,pk6 desc,pk7,pk8))";;
+ " where (pk1 >=1 and pk1<=2) and (pk2>=3 and pk2<=4) and
(pk3,pk4) < (5,7) order by pk1,pk2,pk3";
+" where (pk1 >=1 and pk1<=2) and (pk2>=2 and pk2<=3) and
(pk3,pk4) < (5,7) and "+
+" where (pk1 >=1 and pk1<=2) and (pk2>=2 and pk2<=3) and
(pk3,pk4) in ((5,6),(2,10)) and "+
+ * This inner "_IDX_" + dataTableName use skipScan, and all the
whereExpressions are already in SkipScanFilter,
+ * the whereExpressions of current rowkey slot from the current
{@link SelectStatement#where},
+ * So we should stop extracting whereExpressions of current rowkey
slot once we encounter:
+ * so the endKey is computed as {@link
SchemaUtil#VAR_BINARY_SCHEMA},see {@link ScanRanges#create}.
+"where (OBJ.OBJECT_ID, OBJ.OBJECT_VERSION) in (('obj1',
''),('obj2', ''),('obj3', ''))";
+RowKeyComparisonFilter rowKeyComparisonFilter
=(RowKeyComparisonFilter) filterList.getFilters().get(1);
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/3522//testReport/
Console output:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/3522//console
This message is automatically generated.
> Fix erroneous query result when RVC is clipped with desc column
> ---
>
> Key: PHOENIX-5753
> URL: https://issues.apache.org/jira/browse/PHOENIX-5753
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 5.0.0, 4.15.0
>Reporter: chenglei
>Assignee: chenglei
>Priority: Major
> Labels: DESC
> Fix For: 4.16.0
>
> Attachments: PHOENIX-5753_v2-4.x-HBase-1.4.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Given following table and data:
> {code:java}
>CREATE TABLE test
>(
> pk1 INTEGER NOT NULL ,
> pk2 INTEGER NOT NULL,
> pk3 INTEGER NOT NULL,
> pk4 INTEGER NOT NULL,
> v INTEGER, CONSTRAINT PK PRIMARY KEY(pk1,pk2,pk3 desc,pk4))
>)
> {code}
> Noticed pk3 is DESC.
> {code:java}
>UPSERT INTO test (pk1, pk2, pk3, pk4, v) VALUES (1,3,4,10,1)
> {code}
> If we execute the following sql:
> {code:java}
> select * from test
> where (pk1 >=1 and pk1<=2) and (pk2>=3 and pk2<=4) and (pk3,pk4) < (5,7)
> {code}
> the returned result is empty, but obviously, the above inserted row
> (1,3,4,10,1) should be returned.
> I think this problem is introduced by PHOENIX-3383 and PHOENIX-4841, when we
> clip the {{(pk3,pk4) < (5,7)}} because {{pk3}} is {{DESC}} by following line
> 260 in {{WhereOptimizer.pushKeyExpressionsToScan}} , {{(pk3,pk4) < (5,7)}} is
> clipped to {{pk3 <= 5}} and {{pk4 < 7}} .
> {code:java}
> 257List leftRanges = clipLeft(schema,
> slot.getPKPosition()
> 258+ slotOffset - clipLeftSpan, clipLeftSpan,
> keyRanges, ptr);
> 259keyRanges =
> 260clipRight(schema, slot.getPKPosition() +
> slotOffset - 1, keyRanges,
> 261leftRanges, ptr);
> 262if (prevSortOrder == SortOrder.DESC) {
> 263leftRanges = invertKeyRanges(leftRanges);
> 264}
> 265slotSpanArray[cnf.size()] = clipLeftSpan-1;
> 266cnf.add(leftRanges);
> 267clipLeftSpan = 0;
> 268prevSortOrder = sortOrder;
> 269// since we have to clip the portion with the same
> sort order, we can no longer
> 270
[jira] [Commented] (PHOENIX-5753) Fix erroneous query result when RVC is clipped with desc column
[
https://issues.apache.org/jira/browse/PHOENIX-5753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048746#comment-17048746
]
chenglei commented on PHOENIX-5753:
---
Uploaded my patch， the PR is [PR719|https://github.com/apache/phoenix/pull/719].
[~dbwong],[~ckulkarni], Would you help me review? thank you very much.
> Fix erroneous query result when RVC is clipped with desc column
> ---
>
> Key: PHOENIX-5753
> URL: https://issues.apache.org/jira/browse/PHOENIX-5753
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 5.0.0, 4.15.0
>Reporter: chenglei
>Assignee: chenglei
>Priority: Major
> Labels: DESC
> Fix For: 4.16.0
>
> Attachments: PHOENIX-5753_v2-4.x-HBase-1.4.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Given following table and data:
> {code:java}
>CREATE TABLE test
>(
> pk1 INTEGER NOT NULL ,
> pk2 INTEGER NOT NULL,
> pk3 INTEGER NOT NULL,
> pk4 INTEGER NOT NULL,
> v INTEGER, CONSTRAINT PK PRIMARY KEY(pk1,pk2,pk3 desc,pk4))
>)
> {code}
> Noticed pk3 is DESC.
> {code:java}
>UPSERT INTO test (pk1, pk2, pk3, pk4, v) VALUES (1,3,4,10,1)
> {code}
> If we execute the following sql:
> {code:java}
> select * from test
> where (pk1 >=1 and pk1<=2) and (pk2>=3 and pk2<=4) and (pk3,pk4) < (5,7)
> {code}
> the returned result is empty, but obviously, the above inserted row
> (1,3,4,10,1) should be returned.
> I think this problem is introduced by PHOENIX-3383 and PHOENIX-4841, when we
> clip the {{(pk3,pk4) < (5,7)}} because {{pk3}} is {{DESC}} by following line
> 260 in {{WhereOptimizer.pushKeyExpressionsToScan}} , {{(pk3,pk4) < (5,7)}} is
> clipped to {{pk3 <= 5}} and {{pk4 < 7}} .
> {code:java}
> 257List leftRanges = clipLeft(schema,
> slot.getPKPosition()
> 258+ slotOffset - clipLeftSpan, clipLeftSpan,
> keyRanges, ptr);
> 259keyRanges =
> 260clipRight(schema, slot.getPKPosition() +
> slotOffset - 1, keyRanges,
> 261leftRanges, ptr);
> 262if (prevSortOrder == SortOrder.DESC) {
> 263leftRanges = invertKeyRanges(leftRanges);
> 264}
> 265slotSpanArray[cnf.size()] = clipLeftSpan-1;
> 266cnf.add(leftRanges);
> 267clipLeftSpan = 0;
> 268prevSortOrder = sortOrder;
> 269// since we have to clip the portion with the same
> sort order, we can no longer
> 270// extract the nodes from the where clause
> 271// for eg. for the schema A VARCHAR DESC, B VARCHAR
> ASC and query
> 272// WHERE (A,B) < ('a','b')
> 273// the range (* - a\xFFb) is converted to (~a-*)(*-b)
> 274// so we still need to filter on A,B
> 275stopExtracting = true;
> 276}
> {code}
> Eventually after we completed the
> {{WhereOptimizer.pushKeyExpressionsToScan}}, the result
> {{ScanRanges.ranges}} is [[[1 - 2]], [[3 - 4]], [[~5 - *)], [(* - 7)]],
> {{ScanRanges.useSkipScanFilter}} is {{true}} and {{SkipScanFilter}} is also
> [[[1 - 2]], [[3 - 4]], [[~5 - *)], [(* - 7)]], so the the above inserted row
> (1,3,4,10,1) could not be retrieved.
> But as we know, {{(pk3,pk4) < (5,7)}} is not semantically equals to {{pk3 <=
> 5}} and {{pk4 < 7}} , we could only have
> {{pk3 <= 5}} but not {{pk4 < 7}}, so when we clipped {{(pk3,pk4) <
> (5,7)}} to {{pk3 <= 5}} , we could simply skip remaining columns of this
> RVC.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (PHOENIX-5753) Fix erroneous query result when RVC is clipped with desc column
[
https://issues.apache.org/jira/browse/PHOENIX-5753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048479#comment-17048479
]
Hadoop QA commented on PHOENIX-5753:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12995048/PHOENIX-5753_v1-4.x-HBase-1.4.patch
against 4.x-HBase-1.4 branch at commit
4c2f481f82bb1bcf5f62d009dd20854ef35b13a8.
ATTACHMENT ID: 12995048
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+ " v INTEGER, CONSTRAINT PK PRIMARY
KEY(pk1,pk2,pk3 desc,pk4,pk5,pk6 desc,pk7,pk8))";;
+" where (pk1 >=1 and pk1<=2) and (pk2>=2 and pk2<=3) and
(pk3,pk4) < (5,7) and "+
+ * This inner "_IDX_" + dataTableName use skipScan, and all the
whereExpressions are in SkipScanFilter,
+for (int i= currentRowKeyColumnPosition; i <
currentRowKeySlot.getPKPosition(); i++) {
+schema.getField(currentRowKeySlot.getPKPosition() +
slotOffset).getSortOrder();
++ slotOffset - clipLeftSpan, clipLeftSpan,
currentRowKeySlotRanges, ptr);
+clipRight(schema,
currentRowKeySlot.getPKPosition() + slotOffset - 1, currentRowKeySlotRanges,
+if (schema.getField(currentRowKeySlot.getPKPosition() + slotOffset
- 1).getSortOrder() == SortOrder.DESC) {
+ * the whereExpressions of current rowkey slot from the current
{@link SelectStatement#where},
+ * So we should stop extracting whereExpressions of current rowkey
slot once we encounter:
{color:red}-1 core tests{color}. The patch failed these unit tests:
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.IndexRebuildTaskIT
Test results:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/3521//testReport/
Console output:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/3521//console
This message is automatically generated.
> Fix erroneous query result when RVC is clipped with desc column
> ---
>
> Key: PHOENIX-5753
> URL: https://issues.apache.org/jira/browse/PHOENIX-5753
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 5.0.0, 4.15.0
>Reporter: chenglei
>Assignee: chenglei
>Priority: Major
> Labels: DESC
> Fix For: 4.16.0
>
> Attachments: PHOENIX-5753_v1-4.x-HBase-1.4.patch
>
>
> Given following table and data:
> {code:java}
>CREATE TABLE test
>(
> pk1 INTEGER NOT NULL ,
> pk2 INTEGER NOT NULL,
> pk3 INTEGER NOT NULL,
> pk4 INTEGER NOT NULL,
> v INTEGER, CONSTRAINT PK PRIMARY KEY(pk1,pk2,pk3 desc,pk4))
>)
> {code}
> Noticed pk3 is DESC.
> {code:java}
>UPSERT INTO test (pk1, pk2, pk3, pk4, v) VALUES (1,3,4,10,1)
> {code}
> If we execute the following sql:
> {code:java}
> select * from test
> where (pk1 >=1 and pk1<=2) and (pk2>=3 and pk2<=4) and (pk3,pk4) < (5,7)
> {code}
> the returned result is empty, but obviously, the above inserted row
> (1,3,4,10,1) should be returned.
> I think this problem is introduced by PHOENIX-3383 and PHOENIX-4841, when we
> clip the {{(pk3,pk4) < (5,7)}} because {{pk3}} is {{DESC}} by following line
> 260 in {{WhereOptimizer.pushKeyExpressionsToScan}} , {{(pk3,pk4) < (5,7)}} is
> clipped to {{pk3 <= 5}} and {{pk4 < 7}} .
> {code:java}
> 257List leftRanges = clipLeft(schema,
> slot.getPKPosition()
> 258+ slotOffset - clipLeftSpan, clipLeftSpan,
> keyRanges, ptr);
> 259keyRanges =
> 260clipRight(schema, slot.getPKPosition() +
> slotOffset - 1, keyRanges,
> 261leftRanges, ptr);
> 262if (prevSortOrder == SortOrder.DESC) {
> 263leftRanges = invertKeyRanges(leftRanges);
> 264}
> 265slotSpanArray[cnf.size()] = clipLeftSpan-1;
> 266cnf.add(leftRanges);
> 267clipLeftSpan = 0;
> 268prevSortOrder = sortOrder;
> 269// since we have to clip the portion with the same
> sort order, we can no longer
