[jira] [Commented] (PHOENIX-3705) SkipScanFilter may repeatedly copy rowKey Columns to startKey
[
https://issues.apache.org/jira/browse/PHOENIX-3705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15895507#comment-15895507
]
Hudson commented on PHOENIX-3705:
-
SUCCESS: Integrated in Jenkins build Phoenix-master #1576 (See
[https://builds.apache.org/job/Phoenix-master/1576/])
PHOENIX-3705 SkipScanFilter may repeatedly copy rowKey Columns to (chenglei:
rev c8612fa1b09f883726102951626798244f73db17)
* (edit)
phoenix-core/src/test/java/org/apache/phoenix/filter/SkipScanFilterTest.java
* (edit)
phoenix-core/src/main/java/org/apache/phoenix/filter/SkipScanFilter.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/schema/RowKeySchema.java
> SkipScanFilter may repeatedly copy rowKey Columns to startKey
> -
>
> Key: PHOENIX-3705
> URL: https://issues.apache.org/jira/browse/PHOENIX-3705
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.9.0
>Reporter: chenglei
>Assignee: chenglei
>Priority: Blocker
> Fix For: 4.10.0
>
> Attachments: PHOENIX-3705_v2.patch
>
>
> See following simple unit test first,the rowKey is composed of three PInteger
> columns,and the slots of SkipScanFilter are:
> [ [[1 - 4]], [5, 7], [[9 - 10]] ],
> When SkipScanFilter.filterKeyValue method is invoked on a KeyValue whose
> rowKey is [2,7,11], obviously SkipScanFilter.filterKeyValue
> returns ReturnCode.SEEK_NEXT_USING_HINT and SkipScanFilter.getNextCellHint
> returns [3,5,9], but unfortunately, SkipScanFilter.getNextCellHint actually
> returns [2,8,5,9] , a very strange value, the unit tests failed.
> {code}
> @Test
> public void testNavigate() {
> RowKeySchemaBuilder builder = new RowKeySchemaBuilder(3);
> for(int i=0;i<3;i++) {
> builder.addField(
> new PDatum() {
> @Override
> public boolean isNullable() {
> return false;
> }
> @Override
> public PDataType getDataType() {
> return PInteger.INSTANCE;
> }
> @Override
> public Integer getMaxLength() {
> return PInteger.INSTANCE.getMaxLength(null);
> }
> @Override
> public Integer getScale() {
> return PInteger.INSTANCE.getScale(null);
> }
> @Override
> public SortOrder getSortOrder() {
> return SortOrder.getDefault();
> }
> }, false, SortOrder.getDefault());
> }
>
> List rowKeyColumnRangesList=Arrays.asList(
> Arrays.asList(
>
> PInteger.INSTANCE.getKeyRange(PInteger.INSTANCE.toBytes(1), true,
> PInteger.INSTANCE.toBytes(4), true)),
> Arrays.asList(
> KeyRange.getKeyRange(PInteger.INSTANCE.toBytes(5)),
> KeyRange.getKeyRange(PInteger.INSTANCE.toBytes(7))),
> Arrays.asList(
>
> PInteger.INSTANCE.getKeyRange(PInteger.INSTANCE.toBytes(9), true,
> PInteger.INSTANCE.toBytes(10), true))
> );
>
> SkipScanFilter skipScanFilter=new
> SkipScanFilter(rowKeyColumnRangesList, builder.build());
>
> System.out.println(skipScanFilter);
>
> byte[] rowKey=ByteUtil.concat(
> PInteger.INSTANCE.toBytes(2),
> PInteger.INSTANCE.toBytes(7),
> PInteger.INSTANCE.toBytes(11));
> KeyValue keyValue=KeyValue.createFirstOnRow(rowKey);
> ReturnCode returnCode=skipScanFilter.filterKeyValue(keyValue);
> assertTrue(returnCode == ReturnCode.SEEK_NEXT_USING_HINT);
> Cell nextCellHint=skipScanFilter.getNextCellHint(keyValue);
>
> assertTrue(Bytes.toStringBinary(CellUtil.cloneRow(nextCellHint)).equals(
> "\\x80\\x00\\x00\\x03\\x80\\x00\\x00\\x05\\x80\\x00\\x00\\x09"));
> }
> {code}
> Let us see what's wrong, first column of rowKey [2,7,11 ] is 2, which is in
> SkipScanFilter's first slot range [1-4], so position[0] is 0 and we go to the
> second column 7, which match the second range [7] of SkipScanFilter's second
> slot [5, 7],so position[1] is 1 and we go to the third column 11, which is
> bigger than the third slot range [9 - 10],so position[2] is 0 and the
> {{SkipScanFilter.ptr}} which points to current column still stay on the third
> column. Now we begin to backtrack to second column, because the second range
> [7] of SkipScanFilter's second slot is singleKey and
[jira] [Commented] (PHOENIX-3705) SkipScanFilter may repeatedly copy rowKey Columns to startKey
[
https://issues.apache.org/jira/browse/PHOENIX-3705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15895471#comment-15895471
]
chenglei commented on PHOENIX-3705:
---
Thank you for the review, [~jamestaylor]. Pushed to the master, 4.x-HBase-0.98,
and 4.x-HBase-1.1 branches.
> SkipScanFilter may repeatedly copy rowKey Columns to startKey
> -
>
> Key: PHOENIX-3705
> URL: https://issues.apache.org/jira/browse/PHOENIX-3705
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.9.0
>Reporter: chenglei
>Priority: Blocker
> Fix For: 4.10.0
>
> Attachments: PHOENIX-3705_v2.patch
>
>
> See following simple unit test first,the rowKey is composed of three PInteger
> columns,and the slots of SkipScanFilter are:
> [ [[1 - 4]], [5, 7], [[9 - 10]] ],
> When SkipScanFilter.filterKeyValue method is invoked on a KeyValue whose
> rowKey is [2,7,11], obviously SkipScanFilter.filterKeyValue
> returns ReturnCode.SEEK_NEXT_USING_HINT and SkipScanFilter.getNextCellHint
> returns [3,5,9], but unfortunately, SkipScanFilter.getNextCellHint actually
> returns [2,8,5,9] , a very strange value, the unit tests failed.
> {code}
> @Test
> public void testNavigate() {
> RowKeySchemaBuilder builder = new RowKeySchemaBuilder(3);
> for(int i=0;i<3;i++) {
> builder.addField(
> new PDatum() {
> @Override
> public boolean isNullable() {
> return false;
> }
> @Override
> public PDataType getDataType() {
> return PInteger.INSTANCE;
> }
> @Override
> public Integer getMaxLength() {
> return PInteger.INSTANCE.getMaxLength(null);
> }
> @Override
> public Integer getScale() {
> return PInteger.INSTANCE.getScale(null);
> }
> @Override
> public SortOrder getSortOrder() {
> return SortOrder.getDefault();
> }
> }, false, SortOrder.getDefault());
> }
>
> List rowKeyColumnRangesList=Arrays.asList(
> Arrays.asList(
>
> PInteger.INSTANCE.getKeyRange(PInteger.INSTANCE.toBytes(1), true,
> PInteger.INSTANCE.toBytes(4), true)),
> Arrays.asList(
> KeyRange.getKeyRange(PInteger.INSTANCE.toBytes(5)),
> KeyRange.getKeyRange(PInteger.INSTANCE.toBytes(7))),
> Arrays.asList(
>
> PInteger.INSTANCE.getKeyRange(PInteger.INSTANCE.toBytes(9), true,
> PInteger.INSTANCE.toBytes(10), true))
> );
>
> SkipScanFilter skipScanFilter=new
> SkipScanFilter(rowKeyColumnRangesList, builder.build());
>
> System.out.println(skipScanFilter);
>
> byte[] rowKey=ByteUtil.concat(
> PInteger.INSTANCE.toBytes(2),
> PInteger.INSTANCE.toBytes(7),
> PInteger.INSTANCE.toBytes(11));
> KeyValue keyValue=KeyValue.createFirstOnRow(rowKey);
> ReturnCode returnCode=skipScanFilter.filterKeyValue(keyValue);
> assertTrue(returnCode == ReturnCode.SEEK_NEXT_USING_HINT);
> Cell nextCellHint=skipScanFilter.getNextCellHint(keyValue);
>
> assertTrue(Bytes.toStringBinary(CellUtil.cloneRow(nextCellHint)).equals(
> "\\x80\\x00\\x00\\x03\\x80\\x00\\x00\\x05\\x80\\x00\\x00\\x09"));
> }
> {code}
> Let us see what's wrong, first column of rowKey [2,7,11 ] is 2, which is in
> SkipScanFilter's first slot range [1-4], so position[0] is 0 and we go to the
> second column 7, which match the second range [7] of SkipScanFilter's second
> slot [5, 7],so position[1] is 1 and we go to the third column 11, which is
> bigger than the third slot range [9 - 10],so position[2] is 0 and the
> {{SkipScanFilter.ptr}} which points to current column still stay on the third
> column. Now we begin to backtrack to second column, because the second range
> [7] of SkipScanFilter's second slot is singleKey and there is no more
> range,so position[1] is 0 and we continue to backtrack to first column,
> because the first slot range [1-4] is not singleKey so we stop backtracking
> at first column.
> Now the problem comes, in following line 448 of {{SkipScanFilter.navigate}}
> method,{{SkipScanFilter.setStartKey}} method is invoked,first copy rowKey
> columns before {{SkipScanFilter.ptr}} to
[jira] [Commented] (PHOENIX-3705) SkipScanFilter may repeatedly copy rowKey Columns to startKey
[
https://issues.apache.org/jira/browse/PHOENIX-3705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894937#comment-15894937
]
James Taylor commented on PHOENIX-3705:
---
+1 on patch. Nice work, [~comnetwork]. Please commit to master and 4.x branches.
> SkipScanFilter may repeatedly copy rowKey Columns to startKey
> -
>
> Key: PHOENIX-3705
> URL: https://issues.apache.org/jira/browse/PHOENIX-3705
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.9.0
>Reporter: chenglei
>Priority: Blocker
> Fix For: 4.10.0
>
> Attachments: PHOENIX-3705_v2.patch
>
>
> See following simple unit test first,the rowKey is composed of three PInteger
> columns,and the slots of SkipScanFilter are:
> [ [[1 - 4]], [5, 7], [[9 - 10]] ],
> When SkipScanFilter.filterKeyValue method is invoked on a KeyValue whose
> rowKey is [2,7,11], obviously SkipScanFilter.filterKeyValue
> returns ReturnCode.SEEK_NEXT_USING_HINT and SkipScanFilter.getNextCellHint
> returns [3,5,9], but unfortunately, SkipScanFilter.getNextCellHint actually
> returns [2,8,5,9] , a very strange value, the unit tests failed.
> {code}
> @Test
> public void testNavigate() {
> RowKeySchemaBuilder builder = new RowKeySchemaBuilder(3);
> for(int i=0;i<3;i++) {
> builder.addField(
> new PDatum() {
> @Override
> public boolean isNullable() {
> return false;
> }
> @Override
> public PDataType getDataType() {
> return PInteger.INSTANCE;
> }
> @Override
> public Integer getMaxLength() {
> return PInteger.INSTANCE.getMaxLength(null);
> }
> @Override
> public Integer getScale() {
> return PInteger.INSTANCE.getScale(null);
> }
> @Override
> public SortOrder getSortOrder() {
> return SortOrder.getDefault();
> }
> }, false, SortOrder.getDefault());
> }
>
> List rowKeyColumnRangesList=Arrays.asList(
> Arrays.asList(
>
> PInteger.INSTANCE.getKeyRange(PInteger.INSTANCE.toBytes(1), true,
> PInteger.INSTANCE.toBytes(4), true)),
> Arrays.asList(
> KeyRange.getKeyRange(PInteger.INSTANCE.toBytes(5)),
> KeyRange.getKeyRange(PInteger.INSTANCE.toBytes(7))),
> Arrays.asList(
>
> PInteger.INSTANCE.getKeyRange(PInteger.INSTANCE.toBytes(9), true,
> PInteger.INSTANCE.toBytes(10), true))
> );
>
> SkipScanFilter skipScanFilter=new
> SkipScanFilter(rowKeyColumnRangesList, builder.build());
>
> System.out.println(skipScanFilter);
>
> byte[] rowKey=ByteUtil.concat(
> PInteger.INSTANCE.toBytes(2),
> PInteger.INSTANCE.toBytes(7),
> PInteger.INSTANCE.toBytes(11));
> KeyValue keyValue=KeyValue.createFirstOnRow(rowKey);
> ReturnCode returnCode=skipScanFilter.filterKeyValue(keyValue);
> assertTrue(returnCode == ReturnCode.SEEK_NEXT_USING_HINT);
> Cell nextCellHint=skipScanFilter.getNextCellHint(keyValue);
>
> assertTrue(Bytes.toStringBinary(CellUtil.cloneRow(nextCellHint)).equals(
> "\\x80\\x00\\x00\\x03\\x80\\x00\\x00\\x05\\x80\\x00\\x00\\x09"));
> }
> {code}
> Let us see what's wrong, first column of rowKey [2,7,11 ] is 2, which is in
> SkipScanFilter's first slot range [1-4], so position[0] is 0 and we go to the
> second column 7, which match the second range [7] of SkipScanFilter's second
> slot [5, 7],so position[1] is 1 and we go to the third column 11, which is
> bigger than the third slot range [9 - 10],so position[2] is 0 and the
> {{SkipScanFilter.ptr}} which points to current column still stay on the third
> column. Now we begin to backtrack to second column, because the second range
> [7] of SkipScanFilter's second slot is singleKey and there is no more
> range,so position[1] is 0 and we continue to backtrack to first column,
> because the first slot range [1-4] is not singleKey so we stop backtracking
> at first column.
> Now the problem comes, in following line 448 of {{SkipScanFilter.navigate}}
> method,{{SkipScanFilter.setStartKey}} method is invoked,first copy rowKey
> columns before {{SkipScanFilter.ptr}} to {{SkipScanFilter.startKey}}, because
>
[jira] [Commented] (PHOENIX-3705) SkipScanFilter may repeatedly copy rowKey Columns to startKey
[
https://issues.apache.org/jira/browse/PHOENIX-3705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894274#comment-15894274
]
Hadoop QA commented on PHOENIX-3705:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12855828/PHOENIX-3705_v2.patch
against master branch at commit cf65fb27edf6291500e3f7e7549c4b83240f.
ATTACHMENT ID: 12855828
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
45 warning messages.
{color:red}-1 release audit{color}. The applied patch generated 1 release
audit warnings (more than the master's current 0 warnings).
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+//for PHOENIX-3705, now ptr is still point to slot i,
we must make ptr point to slot j+1,
+//because following setStartKey method will copy rowKey
columns before ptr to startKey and
+//then copy the lower bound of slots from j+1, according
to position array, so if we do not
+//make ptr point to the first rowKey column of slot j,why
we need slotSpan[j] because for Row Value Constructor(RVC),
+//slot j may span multiple rowKey columns, so the length
of ptr must consider the slotSpan[j].
+schema.iterator(startKey, minOffset, maxOffset, ptr,
ScanUtil.getRowKeyPosition(slotSpan, j)+1,slotSpan[j]);
+public Boolean iterator(byte[] src, int srcOffset, int srcLength,
ImmutableBytesWritable ptr, int position,int extraColumnSpan) {
+public Boolean iterator(byte[] src, int srcOffset, int srcLength,
ImmutableBytesWritable ptr, int position) {
+public SkipScanFilterTest(List cnf, int[] widths, int[]
slotSpans,List expectations) {
+
PInteger.INSTANCE.getKeyRange(PInteger.INSTANCE.toBytes(1), true,
PInteger.INSTANCE.toBytes(4), true)
{color:red}-1 core tests{color}. The patch failed these unit tests:
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.index.MutableIndexFailureIT
Test results:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/791//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/791//artifact/patchprocess/patchReleaseAuditWarnings.txt
Javadoc warnings:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/791//artifact/patchprocess/patchJavadocWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/791//console
This message is automatically generated.
> SkipScanFilter may repeatedly copy rowKey Columns to startKey
> -
>
> Key: PHOENIX-3705
> URL: https://issues.apache.org/jira/browse/PHOENIX-3705
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.9.0
>Reporter: chenglei
>Priority: Blocker
> Fix For: 4.10.0
>
> Attachments: PHOENIX-3705_v2.patch
>
>
> See following simple unit test first,the rowKey is composed of three PInteger
> columns,and the slots of SkipScanFilter are:
> [ [[1 - 4]], [5, 7], [[9 - 10]] ],
> When SkipScanFilter.filterKeyValue method is invoked on a KeyValue whose
> rowKey is [2,7,11], obviously SkipScanFilter.filterKeyValue
> returns ReturnCode.SEEK_NEXT_USING_HINT and SkipScanFilter.getNextCellHint
> returns [3,5,9], but unfortunately, SkipScanFilter.getNextCellHint actually
> returns [2,8,5,9] , a very strange value, the unit tests failed.
> {code}
> @Test
> public void testNavigate() {
> RowKeySchemaBuilder builder = new RowKeySchemaBuilder(3);
> for(int i=0;i<3;i++) {
> builder.addField(
> new PDatum() {
> @Override
> public boolean isNullable() {
> return false;
> }
> @Override
> public PDataType getDataType() {
> return PInteger.INSTANCE;
> }
> @Override
> public Integer getMaxLength() {
> return PInteger.INSTANCE.getMaxLength(null);
> }
> @Override
> public Integer getScale() {
> return PInteger.INSTANCE.getScale(null);
> }
>
[jira] [Commented] (PHOENIX-3705) SkipScanFilter may repeatedly copy rowKey Columns to startKey
[
https://issues.apache.org/jira/browse/PHOENIX-3705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894167#comment-15894167
]
chenglei commented on PHOENIX-3705:
---
I uploaded my new patch, with additional changes for RVC and more tests for
RVC, please help me have a review.
> SkipScanFilter may repeatedly copy rowKey Columns to startKey
> -
>
> Key: PHOENIX-3705
> URL: https://issues.apache.org/jira/browse/PHOENIX-3705
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.9.0
>Reporter: chenglei
>Priority: Blocker
> Fix For: 4.10.0
>
> Attachments: PHOENIX-3705_v2.patch
>
>
> See following simple unit test first,the rowKey is composed of three PInteger
> columns,and the slots of SkipScanFilter are:
> [ [[1 - 4]], [5, 7], [[9 - 10]] ],
> When SkipScanFilter.filterKeyValue method is invoked on a KeyValue whose
> rowKey is [2,7,11], obviously SkipScanFilter.filterKeyValue
> returns ReturnCode.SEEK_NEXT_USING_HINT and SkipScanFilter.getNextCellHint
> returns [3,5,9], but unfortunately, SkipScanFilter.getNextCellHint actually
> returns [2,8,5,9] , a very strange value, the unit tests failed.
> {code}
> @Test
> public void testNavigate() {
> RowKeySchemaBuilder builder = new RowKeySchemaBuilder(3);
> for(int i=0;i<3;i++) {
> builder.addField(
> new PDatum() {
> @Override
> public boolean isNullable() {
> return false;
> }
> @Override
> public PDataType getDataType() {
> return PInteger.INSTANCE;
> }
> @Override
> public Integer getMaxLength() {
> return PInteger.INSTANCE.getMaxLength(null);
> }
> @Override
> public Integer getScale() {
> return PInteger.INSTANCE.getScale(null);
> }
> @Override
> public SortOrder getSortOrder() {
> return SortOrder.getDefault();
> }
> }, false, SortOrder.getDefault());
> }
>
> List rowKeyColumnRangesList=Arrays.asList(
> Arrays.asList(
>
> PInteger.INSTANCE.getKeyRange(PInteger.INSTANCE.toBytes(1), true,
> PInteger.INSTANCE.toBytes(4), true)),
> Arrays.asList(
> KeyRange.getKeyRange(PInteger.INSTANCE.toBytes(5)),
> KeyRange.getKeyRange(PInteger.INSTANCE.toBytes(7))),
> Arrays.asList(
>
> PInteger.INSTANCE.getKeyRange(PInteger.INSTANCE.toBytes(9), true,
> PInteger.INSTANCE.toBytes(10), true))
> );
>
> SkipScanFilter skipScanFilter=new
> SkipScanFilter(rowKeyColumnRangesList, builder.build());
>
> System.out.println(skipScanFilter);
>
> byte[] rowKey=ByteUtil.concat(
> PInteger.INSTANCE.toBytes(2),
> PInteger.INSTANCE.toBytes(7),
> PInteger.INSTANCE.toBytes(11));
> KeyValue keyValue=KeyValue.createFirstOnRow(rowKey);
> ReturnCode returnCode=skipScanFilter.filterKeyValue(keyValue);
> assertTrue(returnCode == ReturnCode.SEEK_NEXT_USING_HINT);
> Cell nextCellHint=skipScanFilter.getNextCellHint(keyValue);
>
> assertTrue(Bytes.toStringBinary(CellUtil.cloneRow(nextCellHint)).equals(
> "\\x80\\x00\\x00\\x03\\x80\\x00\\x00\\x05\\x80\\x00\\x00\\x09"));
> }
> {code}
> Let us see what's wrong, first column of rowKey [2,7,11 ] is 2, which is in
> SkipScanFilter's first slot range [1-4], so position[0] is 0 and we go to the
> second column 7, which match the second range [7] of SkipScanFilter's second
> slot [5, 7],so position[1] is 1 and we go to the third column 11, which is
> bigger than the third slot range [9 - 10],so position[2] is 0 and the
> {{SkipScanFilter.ptr}} which points to current column still stay on the third
> column. Now we begin to backtrack to second column, because the second range
> [7] of SkipScanFilter's second slot is singleKey and there is no more
> range,so position[1] is 0 and we continue to backtrack to first column,
> because the first slot range [1-4] is not singleKey so we stop backtracking
> at first column.
> Now the problem comes, in following line 448 of {{SkipScanFilter.navigate}}
> method,{{SkipScanFilter.setStartKey}} method is invoked,first copy rowKey
> columns before {{SkipScanFilter.ptr}} to
[jira] [Commented] (PHOENIX-3705) SkipScanFilter may repeatedly copy rowKey Columns to startKey
[
https://issues.apache.org/jira/browse/PHOENIX-3705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15892629#comment-15892629
]
chenglei commented on PHOENIX-3705:
---
I write following unit test for Row Value Constructor(RVC), ,the rowKey is
composed of four PInteger columns,and the slots of SkipScanFilter are:
[ [[(1,2) - (3,4)]], [5, 7], [[9 - 10]] ],
the first slot is RVC,when SkipScanFilter.filterKeyValue method is invoked on a
KeyValue whose rowKey is [2,3,7,11], SkipScanFilter.filterKeyValue should
return ReturnCode.SEEK_NEXT_USING_HINT and SkipScanFilter.getNextCellHint
should return [2,4,5,9].
{code}
@Test
public void testRVCSkipScanFilter() throws Exception
{
RowKeySchemaBuilder builder = new RowKeySchemaBuilder(4);
for(int i=0;i<4;i++)
{
builder.addField(
new PDatum()
{
@Override
public boolean isNullable()
{
return false;
}
@Override
public PDataType getDataType()
{
return
PInteger.INSTANCE;
}
@Override
public Integer getMaxLength()
{
return
PInteger.INSTANCE.getMaxLength(null);
}
@Override
public Integer getScale()
{
return
PInteger.INSTANCE.getScale(null);
}
@Override
public SortOrder getSortOrder()
{
return
SortOrder.getDefault();
}
}, false, SortOrder.getDefault());
}
List rowKeyColumnRangesList=Arrays.asList(
Arrays.asList(
RowKeyRange.createRowKeyRange(
ByteUtil.concat(PInteger.INSTANCE.toBytes(1),PInteger.INSTANCE.toBytes(2)),
true,
ByteUtil.concat(PInteger.INSTANCE.toBytes(3),PInteger.INSTANCE.toBytes(4)),
true)),
Arrays.asList(
RowKeyRange.createRowKeyRange(PInteger.INSTANCE.toBytes(5)),
RowKeyRange.createRowKeyRange(PInteger.INSTANCE.toBytes(7))),
Arrays.asList(
PInteger.INSTANCE.getKeyRange(PInteger.INSTANCE.toBytes(9), true,
PInteger.INSTANCE.toBytes(10), true))
);
SkipScanFilter skipScanFilter=new
SkipScanFilter(rowKeyColumnRangesList, new int[]{1,0,0},builder.build());
byte[] rowKey=ByteUtil.concat(
PInteger.INSTANCE.toBytes(2),
PInteger.INSTANCE.toBytes(3),
PInteger.INSTANCE.toBytes(7),
PInteger.INSTANCE.toBytes(11));
KeyValue keyValue=KeyValue.createFirstOnRow(rowKey);
ReturnCode returnCode=skipScanFilter.filterKeyValue(keyValue);
assertTrue(returnCode == ReturnCode.SEEK_NEXT_USING_HINT);
Cell nextCellHint=skipScanFilter.getNextCellHint(keyValue);
assertTrue(Arrays.equals(CellUtil.cloneRow(nextCellHint),
ByteUtil.concat(
PInteger.INSTANCE.toBytes(2),
PInteger.INSTANCE.toBytes(4),
PInteger.INSTANCE.toBytes(5),
PInteger.INSTANCE.toBytes(9))
));
}
{code}
Take the above unit test for example, we can not
[jira] [Commented] (PHOENIX-3705) SkipScanFilter may repeatedly copy rowKey Columns to startKey
[
https://issues.apache.org/jira/browse/PHOENIX-3705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15891527#comment-15891527
]
James Taylor commented on PHOENIX-3705:
---
Thanks, [~comnetwork]. Let me know what you find out about when a row value
constructor is used. I'm not sure yet what the right thing to do is there.
Really appreciate you digging into this - you're doing a great job!
> SkipScanFilter may repeatedly copy rowKey Columns to startKey
> -
>
> Key: PHOENIX-3705
> URL: https://issues.apache.org/jira/browse/PHOENIX-3705
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.9.0
>Reporter: chenglei
>Priority: Blocker
> Fix For: 4.10.0
>
> Attachments: PHOENIX-3705_v1.patch
>
>
> See following simple unit test first,the rowKey is composed of three PInteger
> columns,and the slots of SkipScanFilter are:
> [ [[1 - 4]], [5, 7], [[9 - 10]] ],
> When SkipScanFilter.filterKeyValue method is invoked on a KeyValue whose
> rowKey is [2,7,11], obviously SkipScanFilter.filterKeyValue
> returns ReturnCode.SEEK_NEXT_USING_HINT and SkipScanFilter.getNextCellHint
> returns [3,5,9], but unfortunately, SkipScanFilter.getNextCellHint actually
> returns [2,8,5,9] , a very strange value, the unit tests failed.
> {code}
> @Test
> public void testNavigate() {
> RowKeySchemaBuilder builder = new RowKeySchemaBuilder(3);
> for(int i=0;i<3;i++) {
> builder.addField(
> new PDatum() {
> @Override
> public boolean isNullable() {
> return false;
> }
> @Override
> public PDataType getDataType() {
> return PInteger.INSTANCE;
> }
> @Override
> public Integer getMaxLength() {
> return PInteger.INSTANCE.getMaxLength(null);
> }
> @Override
> public Integer getScale() {
> return PInteger.INSTANCE.getScale(null);
> }
> @Override
> public SortOrder getSortOrder() {
> return SortOrder.getDefault();
> }
> }, false, SortOrder.getDefault());
> }
>
> List rowKeyColumnRangesList=Arrays.asList(
> Arrays.asList(
>
> PInteger.INSTANCE.getKeyRange(PInteger.INSTANCE.toBytes(1), true,
> PInteger.INSTANCE.toBytes(4), true)),
> Arrays.asList(
> KeyRange.getKeyRange(PInteger.INSTANCE.toBytes(5)),
> KeyRange.getKeyRange(PInteger.INSTANCE.toBytes(7))),
> Arrays.asList(
>
> PInteger.INSTANCE.getKeyRange(PInteger.INSTANCE.toBytes(9), true,
> PInteger.INSTANCE.toBytes(10), true))
> );
>
> SkipScanFilter skipScanFilter=new
> SkipScanFilter(rowKeyColumnRangesList, builder.build());
>
> System.out.println(skipScanFilter);
>
> byte[] rowKey=ByteUtil.concat(
> PInteger.INSTANCE.toBytes(2),
> PInteger.INSTANCE.toBytes(7),
> PInteger.INSTANCE.toBytes(11));
> KeyValue keyValue=KeyValue.createFirstOnRow(rowKey);
> ReturnCode returnCode=skipScanFilter.filterKeyValue(keyValue);
> assertTrue(returnCode == ReturnCode.SEEK_NEXT_USING_HINT);
> Cell nextCellHint=skipScanFilter.getNextCellHint(keyValue);
>
> assertTrue(Bytes.toStringBinary(CellUtil.cloneRow(nextCellHint)).equals(
> "\\x80\\x00\\x00\\x03\\x80\\x00\\x00\\x05\\x80\\x00\\x00\\x09"));
> }
> {code}
> Let us see what's wrong, first column of rowKey [2,7,11 ] is 2, which is in
> SkipScanFilter's first slot range [1-4], so position[0] is 0 and we go to the
> second column 7, which match the second range [7] of SkipScanFilter's second
> slot [5, 7],so position[1] is 1 and we go to the third column 11, which is
> bigger than the third slot range [9 - 10],so position[2] is 0 and the
> {{SkipScanFilter.ptr}} which points to current column still stay on the third
> column. Now we begin to backtrack to second column, because the second range
> [7] of SkipScanFilter's second slot is singleKey and there is no more
> range,so position[1] is 0 and we continue to backtrack to first column,
> because the first slot range [1-4] is not singleKey so we stop backtracking
> at first column.
> Now the problem comes, in following line 448 of {{SkipScanFilter.navigate}}
>
[jira] [Commented] (PHOENIX-3705) SkipScanFilter may repeatedly copy rowKey Columns to startKey
[
https://issues.apache.org/jira/browse/PHOENIX-3705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15891484#comment-15891484
]
chenglei commented on PHOENIX-3705:
---
Thanks for review, [~jamestaylor],I will add more tests and modify following
your suggestion.
> SkipScanFilter may repeatedly copy rowKey Columns to startKey
> -
>
> Key: PHOENIX-3705
> URL: https://issues.apache.org/jira/browse/PHOENIX-3705
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.9.0
>Reporter: chenglei
>Priority: Blocker
> Fix For: 4.10.0
>
> Attachments: PHOENIX-3705_v1.patch
>
>
> See following simple unit test first,the rowKey is composed of three PInteger
> columns,and the slots of SkipScanFilter are:
> [ [[1 - 4]], [5, 7], [[9 - 10]] ],
> When SkipScanFilter.filterKeyValue method is invoked on a KeyValue whose
> rowKey is [2,7,11], obviously SkipScanFilter.filterKeyValue
> returns ReturnCode.SEEK_NEXT_USING_HINT and SkipScanFilter.getNextCellHint
> returns [3,5,9], but unfortunately, SkipScanFilter.getNextCellHint actually
> returns [2,8,5,9] , a very strange value, the unit tests failed.
> {code}
> @Test
> public void testNavigate() {
> RowKeySchemaBuilder builder = new RowKeySchemaBuilder(3);
> for(int i=0;i<3;i++) {
> builder.addField(
> new PDatum() {
> @Override
> public boolean isNullable() {
> return false;
> }
> @Override
> public PDataType getDataType() {
> return PInteger.INSTANCE;
> }
> @Override
> public Integer getMaxLength() {
> return PInteger.INSTANCE.getMaxLength(null);
> }
> @Override
> public Integer getScale() {
> return PInteger.INSTANCE.getScale(null);
> }
> @Override
> public SortOrder getSortOrder() {
> return SortOrder.getDefault();
> }
> }, false, SortOrder.getDefault());
> }
>
> List rowKeyColumnRangesList=Arrays.asList(
> Arrays.asList(
>
> PInteger.INSTANCE.getKeyRange(PInteger.INSTANCE.toBytes(1), true,
> PInteger.INSTANCE.toBytes(4), true)),
> Arrays.asList(
> KeyRange.getKeyRange(PInteger.INSTANCE.toBytes(5)),
> KeyRange.getKeyRange(PInteger.INSTANCE.toBytes(7))),
> Arrays.asList(
>
> PInteger.INSTANCE.getKeyRange(PInteger.INSTANCE.toBytes(9), true,
> PInteger.INSTANCE.toBytes(10), true))
> );
>
> SkipScanFilter skipScanFilter=new
> SkipScanFilter(rowKeyColumnRangesList, builder.build());
>
> System.out.println(skipScanFilter);
>
> byte[] rowKey=ByteUtil.concat(
> PInteger.INSTANCE.toBytes(2),
> PInteger.INSTANCE.toBytes(7),
> PInteger.INSTANCE.toBytes(11));
> KeyValue keyValue=KeyValue.createFirstOnRow(rowKey);
> ReturnCode returnCode=skipScanFilter.filterKeyValue(keyValue);
> assertTrue(returnCode == ReturnCode.SEEK_NEXT_USING_HINT);
> Cell nextCellHint=skipScanFilter.getNextCellHint(keyValue);
>
> assertTrue(Bytes.toStringBinary(CellUtil.cloneRow(nextCellHint)).equals(
> "\\x80\\x00\\x00\\x03\\x80\\x00\\x00\\x05\\x80\\x00\\x00\\x09"));
> }
> {code}
> Let us see what's wrong, first column of rowKey [2,7,11 ] is 2, which is in
> SkipScanFilter's first slot range [1-4], so position[0] is 0 and we go to the
> second column 7, which match the second range [7] of SkipScanFilter's second
> slot [5, 7],so position[1] is 1 and we go to the third column 11, which is
> bigger than the third slot range [9 - 10],so position[2] is 0 and the
> {{SkipScanFilter.ptr}} which points to current column still stay on the third
> column. Now we begin to backtrack to second column, because the second range
> [7] of SkipScanFilter's second slot is singleKey and there is no more
> range,so position[1] is 0 and we continue to backtrack to first column,
> because the first slot range [1-4] is not singleKey so we stop backtracking
> at first column.
> Now the problem comes, in following line 448 of {{SkipScanFilter.navigate}}
> method,{{SkipScanFilter.setStartKey}} method is invoked,first copy rowKey
> columns before {{SkipScanFilter.ptr}} to {{SkipScanFilter.startKey}}, because
[jira] [Commented] (PHOENIX-3705) SkipScanFilter may repeatedly copy rowKey Columns to startKey
[
https://issues.apache.org/jira/browse/PHOENIX-3705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15890854#comment-15890854
]
James Taylor commented on PHOENIX-3705:
---
Very nice, [~comnetwork]. It seems like this bug could affect the final query
result, perhaps skipping rows it shouldn't depending on the data, no? Anytime
the seek next hint is wrong, I think this could be the case.
Small question on whether perhaps one more change is required. I noticed you
used {{ScanUtil.getRowKeyPosition(slotSpan, j + 1)}} below in the call to
reposition, but further down (in existing code), we use
{{ScanUtil.getRowKeyPosition(slotSpan, j)+1}}. Should the latter be changed as
you've coded? Would be good to have a test case that uses Row Value Constructor
(i.e. has a slot span) to confirm:
{code}
schema.reposition(
ptr,
ScanUtil.getRowKeyPosition(slotSpan, i),
ScanUtil.getRowKeyPosition(slotSpan, j + 1),
minOffset,
maxOffset,
slotSpan[j + 1]);
int currentLength = setStartKey(ptr, minOffset, j+1,
nSlots, false);
// From here on, we use startKey as our buffer (resetting
minOffset and maxOffset)
// We've copied the part of the current key above that we
need into startKey
// Reinitialize the iterator to be positioned at previous
slot position
minOffset = 0;
maxOffset = startKeyLength;
schema.iterator(startKey, minOffset, maxOffset, ptr,
ScanUtil.getRowKeyPosition(slotSpan, j)+1);
{code}
> SkipScanFilter may repeatedly copy rowKey Columns to startKey
> -
>
> Key: PHOENIX-3705
> URL: https://issues.apache.org/jira/browse/PHOENIX-3705
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.9.0
>Reporter: chenglei
> Attachments: PHOENIX-3705_v1.patch
>
>
> See following simple unit test first,the rowKey is composed of three PInteger
> columns,and the slots of SkipScanFilter are:
> [ [[1 - 4]], [5, 7], [[9 - 10]] ],
> When SkipScanFilter.filterKeyValue method is invoked on a KeyValue whose
> rowKey is [2,7,11], obviously SkipScanFilter.filterKeyValue
> returns ReturnCode.SEEK_NEXT_USING_HINT and SkipScanFilter.getNextCellHint
> returns [3,5,9], but unfortunately, SkipScanFilter.getNextCellHint actually
> returns [2,8,5,9] , a very strange value, the unit tests failed.
> {code}
> @Test
> public void testNavigate() {
> RowKeySchemaBuilder builder = new RowKeySchemaBuilder(3);
> for(int i=0;i<3;i++) {
> builder.addField(
> new PDatum() {
> @Override
> public boolean isNullable() {
> return false;
> }
> @Override
> public PDataType getDataType() {
> return PInteger.INSTANCE;
> }
> @Override
> public Integer getMaxLength() {
> return PInteger.INSTANCE.getMaxLength(null);
> }
> @Override
> public Integer getScale() {
> return PInteger.INSTANCE.getScale(null);
> }
> @Override
> public SortOrder getSortOrder() {
> return SortOrder.getDefault();
> }
> }, false, SortOrder.getDefault());
> }
>
> List rowKeyColumnRangesList=Arrays.asList(
> Arrays.asList(
>
> PInteger.INSTANCE.getKeyRange(PInteger.INSTANCE.toBytes(1), true,
> PInteger.INSTANCE.toBytes(4), true)),
> Arrays.asList(
> KeyRange.getKeyRange(PInteger.INSTANCE.toBytes(5)),
> KeyRange.getKeyRange(PInteger.INSTANCE.toBytes(7))),
> Arrays.asList(
>
> PInteger.INSTANCE.getKeyRange(PInteger.INSTANCE.toBytes(9), true,
> PInteger.INSTANCE.toBytes(10), true))
> );
>
> SkipScanFilter skipScanFilter=new
> SkipScanFilter(rowKeyColumnRangesList, builder.build());
>
> System.out.println(skipScanFilter);
>
> byte[] rowKey=ByteUtil.concat(
> PInteger.INSTANCE.toBytes(2),
> PInteger.INSTANCE.toBytes(7),
> PInteger.INSTANCE.toBytes(11));
> KeyValue keyValue=KeyValue.createFirstOnRow(rowKey);
>
[jira] [Commented] (PHOENIX-3705) SkipScanFilter may repeatedly copy rowKey Columns to startKey
[
https://issues.apache.org/jira/browse/PHOENIX-3705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15889941#comment-15889941
]
chenglei commented on PHOENIX-3705:
---
I uploaded my first patch, please help me have a review.
> SkipScanFilter may repeatedly copy rowKey Columns to startKey
> -
>
> Key: PHOENIX-3705
> URL: https://issues.apache.org/jira/browse/PHOENIX-3705
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.9.0
>Reporter: chenglei
> Attachments: PHOENIX-3705_v1.patch
>
>
> See following simple unit test first,the rowKey is composed of three PInteger
> columns,and the slots of SkipScanFilter are:
> [ [[1 - 4]], [5, 7], [[9 - 10]] ],
> When SkipScanFilter.filterKeyValue method is invoked on a KeyValue whose
> rowKey is [2,7,11], obviously SkipScanFilter.filterKeyValue
> returns ReturnCode.SEEK_NEXT_USING_HINT and SkipScanFilter.getNextCellHint
> returns [3,5,9], but unfortunately, SkipScanFilter.getNextCellHint actually
> returns [2,8,5,9] , a very strange value, the unit tests failed.
> {code}
> @Test
> public void testNavigate() {
> RowKeySchemaBuilder builder = new RowKeySchemaBuilder(3);
> for(int i=0;i<3;i++) {
> builder.addField(
> new PDatum() {
> @Override
> public boolean isNullable() {
> return false;
> }
> @Override
> public PDataType getDataType() {
> return PInteger.INSTANCE;
> }
> @Override
> public Integer getMaxLength() {
> return PInteger.INSTANCE.getMaxLength(null);
> }
> @Override
> public Integer getScale() {
> return PInteger.INSTANCE.getScale(null);
> }
> @Override
> public SortOrder getSortOrder() {
> return SortOrder.getDefault();
> }
> }, false, SortOrder.getDefault());
> }
>
> List rowKeyColumnRangesList=Arrays.asList(
> Arrays.asList(
>
> PInteger.INSTANCE.getKeyRange(PInteger.INSTANCE.toBytes(1), true,
> PInteger.INSTANCE.toBytes(4), true)),
> Arrays.asList(
> KeyRange.getKeyRange(PInteger.INSTANCE.toBytes(5)),
> KeyRange.getKeyRange(PInteger.INSTANCE.toBytes(7))),
> Arrays.asList(
>
> PInteger.INSTANCE.getKeyRange(PInteger.INSTANCE.toBytes(9), true,
> PInteger.INSTANCE.toBytes(10), true))
> );
>
> SkipScanFilter skipScanFilter=new
> SkipScanFilter(rowKeyColumnRangesList, builder.build());
>
> System.out.println(skipScanFilter);
>
> byte[] rowKey=ByteUtil.concat(
> PInteger.INSTANCE.toBytes(2),
> PInteger.INSTANCE.toBytes(7),
> PInteger.INSTANCE.toBytes(11));
> KeyValue keyValue=KeyValue.createFirstOnRow(rowKey);
> ReturnCode returnCode=skipScanFilter.filterKeyValue(keyValue);
> assertTrue(returnCode == ReturnCode.SEEK_NEXT_USING_HINT);
> Cell nextCellHint=skipScanFilter.getNextCellHint(keyValue);
>
> assertTrue(Bytes.toStringBinary(CellUtil.cloneRow(nextCellHint)).equals(
> "\\x80\\x00\\x00\\x03\\x80\\x00\\x00\\x05\\x80\\x00\\x00\\x09"));
> }
> {code}
> Let us see what's wrong, first column of rowKey [2,7,11 ] is 2, which is in
> SkipScanFilter's first slot range [1-4], so position[0] is 0 and we go to the
> second column 7, which match the second range [7] of SkipScanFilter's second
> slot [5, 7],so position[1] is 1 and we go to the third column 11, which is
> bigger than the third slot range [9 - 10],so position[2] is 0 and we begin to
> backtrack to second column, because the second range [7] of SkipScanFilter's
> second slot is singleKey and there is no more range,so position[1] is 0 and
> we continue to backtrack to first column, because the first slot range [1-4]
> is not singleKey so we stop backtracking at first column.
> Now the problem comes, in following line 448 of SkipScanFilter.navigate
> method,SkipScanFilter.setStartKey method is invoked,first copy rowKey columns
> before ptr to SkipScanFilter.startKey, because now ptr still point to the
> third column, so copy the first and second columns to
> SkipScanFilter.startKey,SkipScanFilter.startKey is [2,7] after this step ,
> then setStartKey method copy the lower bound SkipScanFilter.slots
