subject:"\[jira\] \[Commented\] \(PHOENIX\-2628\) Ensure split when iterating through results handled correctly"

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-05-09 Thread James Taylor (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15276519#comment-15276519
 ] 

James Taylor commented on PHOENIX-2628:
---

[~rajeshbabu] - When will you commit this to 1.0 and 0.98 branches? We need to 
keep our branches in sync, otherwise we'll have no hope of consistent behavior 
across our releases. Diverging history will increase the developer effort 
required to commit a patch across all branches.

> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v10.patch, PHOENIX-2628_v11.patch, PHOENIX-2628_v7.patch, 
> PHOENIX-2628_v8.patch, PHOENIX-2628_v9.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-05-04 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15270992#comment-15270992
 ] 

Hudson commented on PHOENIX-2628:
-

FAILURE: Integrated in Phoenix-master #1214 (See 
[https://builds.apache.org/job/Phoenix-master/1214/])
PHOENIX-2628 Ensure split when iterating through results handled (rajeshbabu: 
rev d700c1f032a0f5d119c669100648caf040233ebe)
* phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java
* 
phoenix-core/src/main/java/org/apache/phoenix/iterate/DefaultTableResultIteratorFactory.java
* 
phoenix-core/src/main/java/org/apache/phoenix/iterate/SpoolingResultIterator.java
* phoenix-core/src/main/java/org/apache/phoenix/compile/QueryPlan.java
* 
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/BaseScannerRegionObserver.java
* phoenix-core/src/main/java/org/apache/phoenix/iterate/SerialIterators.java
* 
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/GroupedAggregateRegionObserver.java
* phoenix-core/src/main/java/org/apache/phoenix/compile/TraceQueryPlan.java
* phoenix-core/src/main/java/org/apache/phoenix/execute/CorrelatePlan.java
* 
phoenix-core/src/main/java/org/apache/hadoop/hbase/regionserver/IndexHalfStoreFileReader.java
* 
phoenix-hive/src/main/java/org/apache/phoenix/hive/mapreduce/PhoenixRecordReader.java
* phoenix-core/src/test/java/org/apache/phoenix/query/BaseTest.java
* phoenix-core/src/main/java/org/apache/phoenix/execute/AggregatePlan.java
* phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixStatement.java
* phoenix-core/src/main/java/org/apache/phoenix/execute/ClientAggregatePlan.java
* phoenix-core/src/main/java/org/apache/phoenix/execute/TupleProjectionPlan.java
* phoenix-core/src/main/java/org/apache/phoenix/execute/UnnestArrayPlan.java
* phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIterator.java
* phoenix-core/src/main/java/org/apache/phoenix/iterate/ParallelIterators.java
* phoenix-core/src/it/java/org/apache/phoenix/end2end/index/LocalIndexIT.java
* 
phoenix-core/src/main/java/org/apache/hadoop/hbase/regionserver/LocalIndexSplitter.java
* 
phoenix-core/src/it/java/org/apache/phoenix/iterate/DelayedTableResultIteratorFactory.java
* phoenix-core/src/main/java/org/apache/phoenix/execute/UnionPlan.java
* phoenix-core/src/main/java/org/apache/phoenix/compile/ListJarsQueryPlan.java
* phoenix-core/src/main/java/org/apache/phoenix/execute/DegenerateQueryPlan.java
* 
phoenix-core/src/main/java/org/apache/phoenix/iterate/ChunkedResultIterator.java
* 
phoenix-core/src/it/java/org/apache/phoenix/iterate/MockParallelIteratorFactory.java
* phoenix-core/src/main/java/org/apache/phoenix/execute/HashJoinPlan.java
* phoenix-core/src/main/java/org/apache/phoenix/compile/ScanRanges.java
* phoenix-core/src/main/java/org/apache/phoenix/execute/ClientScanPlan.java
* 
phoenix-core/src/main/java/org/apache/hadoop/hbase/regionserver/LocalIndexStoreFileScanner.java
* 
phoenix-core/src/main/java/org/apache/hadoop/hbase/regionserver/IndexHalfStoreFileReaderGenerator.java
* 
phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIteratorFactory.java
* 
phoenix-core/src/main/java/org/apache/phoenix/iterate/ParallelIteratorFactory.java
* phoenix-core/src/main/java/org/apache/phoenix/execute/ScanPlan.java
* phoenix-core/src/it/java/org/apache/phoenix/end2end/index/MutableIndexIT.java
* phoenix-core/src/main/java/org/apache/phoenix/util/ScanUtil.java
* phoenix-core/src/it/java/org/apache/phoenix/end2end/BaseViewIT.java
* 
phoenix-core/src/main/java/org/apache/phoenix/compile/MutatingParallelIteratorFactory.java
* phoenix-core/src/main/java/org/apache/phoenix/execute/BaseQueryPlan.java
* 
phoenix-core/src/test/java/org/apache/phoenix/query/ParallelIteratorsSplitTest.java
* 
phoenix-core/src/main/java/org/apache/phoenix/execute/LiteralResultIterationPlan.java
* phoenix-core/src/main/java/org/apache/phoenix/execute/SortMergeJoinPlan.java
* 
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/UngroupedAggregateRegionObserver.java
* 
phoenix-core/src/main/java/org/apache/phoenix/mapreduce/PhoenixRecordReader.java


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v10.patch, PHOENIX-2628_v11.patch, PHOENIX-2628_v7.patch, 
> PHOENIX-2628_v8.patch, PHOENIX-2628_v9.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-05-04 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15270625#comment-15270625
 ] 

Hadoop QA commented on PHOENIX-2628:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12802181/PHOENIX-2628_v11.patch
  against master branch at commit d700c1f032a0f5d119c669100648caf040233ebe.
  ATTACHMENT ID: 12802181

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 6 new 
or modified tests.

{color:red}-1 patch{color}.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-PHOENIX-Build/323//console

This message is automatically generated.

> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v10.patch, PHOENIX-2628_v11.patch, PHOENIX-2628_v7.patch, 
> PHOENIX-2628_v8.patch, PHOENIX-2628_v9.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-05-04 Thread Rajeshbabu Chintaguntla (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15270603#comment-15270603
 ] 

Rajeshbabu Chintaguntla commented on PHOENIX-2628:
--

{noformat}
org.apache.phoenix.end2end.QueryDatabaseMetaDataIT.testSchemaMetadataScan
{noformat}
Seems flaky it's passing in my local.
{noformat}
Running org.apache.phoenix.end2end.QueryDatabaseMetaDataIT
Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 115.212 sec - 
in org.apache.phoenix.end2end.QueryDatabaseMetaDataIT
{noformat} 

I am going to commit it.

> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v10.patch, PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch, 
> PHOENIX-2628_v9.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-05-04 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15270567#comment-15270567
 ] 

Hadoop QA commented on PHOENIX-2628:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12802167/PHOENIX-2628_v10.patch
  against master branch at commit 99713a61cd758a8486f1b49225c5e1c6766dc7b9.
  ATTACHMENT ID: 12802167

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 6 new 
or modified tests.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:red}-1 javadoc{color}.  The javadoc tool appears to have generated 
29 warning messages.

{color:red}-1 release audit{color}.  The applied patch generated 4 release 
audit warnings (more than the master's current 0 warnings).

{color:red}-1 lineLengths{color}.  The patch introduces the following lines 
longer than 100:
+ScanUtil.setLocalIndexAttributes(scan, 0, 
HConstants.EMPTY_BYTE_ARRAY, HConstants.EMPTY_BYTE_ARRAY, scan.getStartRow(), 
scan.getStopRow());
+String[] strings = 
{"a","b","c","d","e","f","g","h","i","j","k","l","m","n","o","p","q","r","s","t","u","v","w","x","y","z"};
+HBaseAdmin admin = driver.getConnectionQueryServices(getUrl(), 
TestUtil.TEST_PROPERTIES).getAdmin();
+if(admin.tableExists(localIndex? 
MetaDataUtil.getLocalIndexTableName(tableName): indexName)) {
+admin.disableTable(localIndex? 
MetaDataUtil.getLocalIndexTableName(tableName): indexName);
+admin.deleteTable(localIndex? 
MetaDataUtil.getLocalIndexTableName(tableName): indexName);
+private void createTableAndLoadData(Connection conn1, String[] strings, 
boolean isReverse) throws SQLException {
+"CREATE " + (localIndex ? "LOCAL" : "")+" INDEX " + indexName + " 
ON " + tableName + "(v1"+(isReverse?" DESC":"")+") include (k3)");
+HBaseAdmin admin = driver.getConnectionQueryServices(getUrl(), 
TestUtil.TEST_PROPERTIES).getAdmin();
+conn1.createStatement().execute("CREATE 
"+(localIndex?"LOCAL":"")+" INDEX " + indexName + " ON " + tableName + "(v1)" + 
(localIndex?"":" SPLIT ON ('e')"));

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
 
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.QueryDatabaseMetaDataIT

Test results: 
https://builds.apache.org/job/PreCommit-PHOENIX-Build/321//testReport/
Release audit warnings: 
https://builds.apache.org/job/PreCommit-PHOENIX-Build/321//artifact/patchprocess/patchReleaseAuditWarnings.txt
Javadoc warnings: 
https://builds.apache.org/job/PreCommit-PHOENIX-Build/321//artifact/patchprocess/patchJavadocWarnings.txt
Console output: 
https://builds.apache.org/job/PreCommit-PHOENIX-Build/321//console

This message is automatically generated.

> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v10.patch, PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch, 
> PHOENIX-2628_v9.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-20 Thread James Taylor (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15251142#comment-15251142
 ] 

James Taylor commented on PHOENIX-2628:
---

Pinging [~rajeshbabu] - were these passing for you locally? I'm seeing failures 
on my machine for these as well now.

> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch, PHOENIX-2628_v9.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-18 Thread James Taylor (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15246056#comment-15246056
 ] 

James Taylor commented on PHOENIX-2628:
---

[~rajeshbabu] - looks like you broke a number of unit tests. Please fix:
{code}
Test Result (22 failures / +21)
org.apache.phoenix.end2end.QueryDatabaseMetaDataIT.testSchemaMetadataScan
org.apache.phoenix.end2end.SaltedViewIT.testSaltedUpdatableViewWithLocalIndex[transactional
 = false]
org.apache.phoenix.end2end.SaltedViewIT.testSaltedUpdatableViewWithLocalIndex[transactional
 = true]
org.apache.phoenix.end2end.SubqueryIT.testAnyAllComparisonSubquery[0]
org.apache.phoenix.end2end.SubqueryIT.testAnyAllComparisonSubquery[1]
org.apache.phoenix.end2end.SubqueryIT.testAnyAllComparisonSubquery[2]
org.apache.phoenix.end2end.SubqueryUsingSortMergeJoinIT.testAnyAllComparisonSubquery[0]
org.apache.phoenix.end2end.SubqueryUsingSortMergeJoinIT.testAnyAllComparisonSubquery[1]
org.apache.phoenix.end2end.SubqueryUsingSortMergeJoinIT.testAnyAllComparisonSubquery[2]
org.apache.phoenix.end2end.ViewIT.testNonSaltedUpdatableViewWithLocalIndex[transactional
 = false]
org.apache.phoenix.end2end.ViewIT.testNonSaltedUpdatableViewWithLocalIndex[transactional
 = true]
org.apache.phoenix.end2end.index.LocalIndexIT.testLocalIndexScanAfterRegionSplit[isNamespaceMapped
 = false]
org.apache.phoenix.end2end.index.MutableIndexIT.testSplitDuringIndexScan[localIndex
 = true , transactional = false]
org.apache.phoenix.end2end.index.MutableIndexIT.testSplitDuringIndexReverseScan[localIndex
 = true , transactional = false]
org.apache.phoenix.end2end.index.MutableIndexIT.testCoveredColumns[localIndex = 
true , transactional = false]
org.apache.phoenix.end2end.index.MutableIndexIT.testCoveredColumnUpdates[localIndex
 = true , transactional = false]
org.apache.phoenix.end2end.index.MutableIndexIT.testIndexHalfStoreFileReader[localIndex
 = true , transactional = false]
org.apache.phoenix.end2end.index.MutableIndexIT.testSplitDuringIndexScan[localIndex
 = true , transactional = true]
org.apache.phoenix.end2end.index.MutableIndexIT.testSplitDuringIndexReverseScan[localIndex
 = true , transactional = true]
org.apache.phoenix.end2end.index.MutableIndexIT.testCoveredColumns[localIndex = 
true , transactional = true]
org.apache.phoenix.end2end.index.MutableIndexIT.testCoveredColumnUpdates[localIndex
 = true , transactional = true]
org.apache.phoenix.end2end.index.MutableIndexIT.testIndexHalfStoreFileReader[localIndex
 = true , transactional = true]
{code}

> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch, PHOENIX-2628_v9.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-18 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15245672#comment-15245672
 ] 

Hudson commented on PHOENIX-2628:
-

FAILURE: Integrated in Phoenix-master #1194 (See 
[https://builds.apache.org/job/Phoenix-master/1194/])
PHOENIX-2628 Ensure split when iterating through results handled (rajeshbabu: 
rev a31b70179cbc468cc3fe890bd615fc71e0bac5a2)
* 
phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIteratorFactory.java
* phoenix-core/src/main/java/org/apache/phoenix/execute/ScanPlan.java
* 
phoenix-core/src/it/java/org/apache/phoenix/iterate/MockParallelIteratorFactory.java
* phoenix-core/src/main/java/org/apache/phoenix/execute/ClientAggregatePlan.java
* 
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/GroupedAggregateRegionObserver.java
* phoenix-core/src/main/java/org/apache/phoenix/execute/AggregatePlan.java
* phoenix-core/src/main/java/org/apache/phoenix/execute/UnnestArrayPlan.java
* 
phoenix-core/src/main/java/org/apache/hadoop/hbase/regionserver/LocalIndexStoreFileScanner.java
* 
phoenix-core/src/test/java/org/apache/phoenix/query/ParallelIteratorsSplitTest.java
* 
phoenix-core/src/main/java/org/apache/hadoop/hbase/regionserver/IndexHalfStoreFileReader.java
* 
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/UngroupedAggregateRegionObserver.java
* phoenix-core/src/main/java/org/apache/phoenix/iterate/SerialIterators.java
* phoenix-core/src/main/java/org/apache/phoenix/compile/ScanRanges.java
* phoenix-core/src/main/java/org/apache/phoenix/execute/BaseQueryPlan.java
* phoenix-core/src/main/java/org/apache/phoenix/compile/QueryPlan.java
* phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixStatement.java
* phoenix-core/src/it/java/org/apache/phoenix/end2end/index/MutableIndexIT.java
* phoenix-core/src/main/java/org/apache/phoenix/execute/ClientScanPlan.java
* 
phoenix-core/src/main/java/org/apache/phoenix/iterate/DefaultTableResultIteratorFactory.java
* 
phoenix-core/src/it/java/org/apache/phoenix/iterate/DelayedTableResultIteratorFactory.java
* phoenix-core/src/it/java/org/apache/phoenix/end2end/index/LocalIndexIT.java
* phoenix-core/src/main/java/org/apache/phoenix/execute/CorrelatePlan.java
* 
phoenix-core/src/main/java/org/apache/phoenix/iterate/ParallelIteratorFactory.java
* phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java
* 
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/BaseScannerRegionObserver.java
* phoenix-core/src/main/java/org/apache/phoenix/util/ScanUtil.java
* phoenix-core/src/main/java/org/apache/phoenix/compile/TraceQueryPlan.java
* phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIterator.java
* 
phoenix-core/src/main/java/org/apache/phoenix/iterate/ChunkedResultIterator.java
* 
phoenix-core/src/main/java/org/apache/phoenix/mapreduce/PhoenixRecordReader.java
* 
phoenix-core/src/main/java/org/apache/phoenix/iterate/SpoolingResultIterator.java
* phoenix-core/src/main/java/org/apache/phoenix/execute/SortMergeJoinPlan.java
* 
phoenix-core/src/main/java/org/apache/phoenix/compile/MutatingParallelIteratorFactory.java
* phoenix-core/src/main/java/org/apache/phoenix/execute/HashJoinPlan.java
* phoenix-core/src/main/java/org/apache/phoenix/execute/TupleProjectionPlan.java
* phoenix-core/src/main/java/org/apache/phoenix/execute/DegenerateQueryPlan.java
* phoenix-core/src/test/java/org/apache/phoenix/query/BaseTest.java
* 
phoenix-core/src/main/java/org/apache/phoenix/execute/LiteralResultIterationPlan.java
* phoenix-core/src/main/java/org/apache/phoenix/iterate/ParallelIterators.java
* phoenix-core/src/main/java/org/apache/phoenix/compile/ListJarsQueryPlan.java
* phoenix-core/src/main/java/org/apache/phoenix/execute/UnionPlan.java
* 
phoenix-core/src/main/java/org/apache/hadoop/hbase/regionserver/LocalIndexSplitter.java
* 
phoenix-core/src/main/java/org/apache/hadoop/hbase/regionserver/IndexHalfStoreFileReaderGenerator.java


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch, PHOENIX-2628_v9.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-11 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15234567#comment-15234567
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user chrajeshbabu commented on the pull request:

https://github.com/apache/phoenix/pull/156#issuecomment-208184999
  
Thanks @JamesRTaylor  for review. Will rebase the patch and commit.


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-08 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15233263#comment-15233263
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user JamesRTaylor commented on the pull request:

https://github.com/apache/phoenix/pull/156#issuecomment-207677080
  
+1. Looks great, @chrajeshbabu. Thanks for the excellent work.


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-08 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15232836#comment-15232836
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user chrajeshbabu commented on the pull request:

https://github.com/apache/phoenix/pull/156#issuecomment-207582047
  
@JamesRTaylor  Thanks for the review. Committed the changes handling the 
review comments.
Refactored the code and added code comments where ever possible. 

Now not handling the special cases required for ChunkedResultItertor. 

bq. Another potential, different approach would be for Phoenix to 
universally handle the split during scan case (rather than letting the HBase 
client scanner handle it for non aggregate case and Phoenix handle it for the 
aggregate case). Would that simplify things?
Now throwing stale region bound exception all the cases when ever we get 
NSRE then phoenix client can handle the NSRE than hbase client handling with 
wrong region boundaries.

The ordered aggregate queries issue you are mentioning can be handled as 
part of other issue.

Please review the latest code.
Thanks.


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-08 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15232770#comment-15232770
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r59077569
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/BaseScannerRegionObserver.java
 ---
@@ -337,6 +384,22 @@ public boolean nextRaw(List result) throws 
IOException {
 arrayElementCell = 
result.get(arrayElementCellPosition);
 }
 if (ScanUtil.isLocalIndex(scan) && 
!ScanUtil.isAnalyzeTable(scan)) {
--- End diff --

We are getting the results after seek only. Will check once again whether 
we can handle this in local index scanner.


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-08 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15232780#comment-15232780
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r59078097
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/ChunkedResultIterator.java
 ---
@@ -56,6 +57,7 @@
 private final MutationState mutationState;
 private Scan scan;
 private PeekingResultIterator resultIterator;
+private QueryPlan plan;
 
 public static class ChunkedResultIteratorFactory implements 
ParallelIteratorFactory {
--- End diff --

Currently removed the changes required for ChunkedResultIterator. The 
changes in the chunked result iterator are just removing code not required.


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-08 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15232775#comment-15232775
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r59077866
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/GroupedAggregateRegionObserver.java
 ---
@@ -402,8 +405,8 @@ private RegionScanner 
scanUnordered(ObserverContext Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-08 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15232766#comment-15232766
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r59077319
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/BaseScannerRegionObserver.java
 ---
@@ -279,6 +301,31 @@ protected RegionScanner getWrappedScanner(final 
ObserverContext Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-05 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227561#comment-15227561
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58642550
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java 
---
@@ -556,35 +564,55 @@ private static String toString(List gps) {
 } else {
 endKey = regionBoundaries.get(regionIndex);
 }
-HRegionLocation regionLocation = 
regionLocations.get(regionIndex);
-if (isLocalIndex) {
-HRegionInfo regionInfo = 
regionLocation.getRegionInfo();
-endRegionKey = regionInfo.getEndKey();
-keyOffset = 
ScanUtil.getRowKeyOffset(regionInfo.getStartKey(), endRegionKey);
-}
-try {
-while (guideIndex < gpsSize && 
(currentGuidePost.compareTo(endKey) <= 0 || endKey.length == 0)) {
-Scan newScan = scanRanges.intersectScan(scan, 
currentKeyBytes, currentGuidePostBytes, keyOffset,
-false);
-estimatedRows += 
gps.getRowCounts().get(guideIndex);
-estimatedSize += 
gps.getByteCounts().get(guideIndex);
-scans = addNewScan(parallelScans, scans, newScan, 
currentGuidePostBytes, false, regionLocation);
-currentKeyBytes = currentGuidePost.copyBytes();
-currentGuidePost = PrefixByteCodec.decode(decoder, 
input);
-currentGuidePostBytes = 
currentGuidePost.copyBytes();
-guideIndex++;
-}
-} catch (EOFException e) {}
-Scan newScan = scanRanges.intersectScan(scan, 
currentKeyBytes, endKey, keyOffset, true);
-if (isLocalIndex) {
-if (newScan != null) {
-newScan.setAttribute(EXPECTED_UPPER_REGION_KEY, 
endRegionKey);
-} else if (!scans.isEmpty()) {
-
scans.get(scans.size()-1).setAttribute(EXPECTED_UPPER_REGION_KEY, endRegionKey);
-}
-}
-scans = addNewScan(parallelScans, scans, newScan, endKey, 
true, regionLocation);
-currentKeyBytes = endKey;
+if (Bytes.compareTo(scan.getStartRow(), 
context.getScan().getStartRow()) != 0
+ || Bytes.compareTo(scan.getStopRow(), 
context.getScan().getStopRow()) != 0) {
--- End diff --

getParalleScans is the entry point. Let me move it out.


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-05 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227554#comment-15227554
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58642188
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIterator.java 
---
@@ -107,8 +127,37 @@ public synchronized void close() throws SQLException {
 @Override
 public synchronized Tuple next() throws SQLException {
 initScanner();
-Tuple t = scanIterator.next();
-return t;
+try {
+lastTuple = scanIterator.next();
+if (lastTuple != null) {
+ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+lastTuple.getKey(ptr);
+}
+} catch (SQLException e) {
+try {
+throw ServerUtil.parseServerException(e);
+} catch(StaleRegionBoundaryCacheException e1) {
+if(scan.getAttribute(NON_AGGREGATE_QUERY)!=null) {
+Scan newScan = ScanUtil.newScan(scan);
+if(lastTuple != null) {
+lastTuple.getKey(ptr);
+byte[] startRowSuffix = 
ByteUtil.copyKeyBytesIfNecessary(ptr);
+if(ScanUtil.isLocalIndex(newScan)) {
+newScan.setAttribute(SCAN_START_ROW_SUFFIX, 
ByteUtil.nextKey(startRowSuffix));
+} else {
+
newScan.setStartRow(ByteUtil.nextKey(startRowSuffix));
+}
+}
+
plan.getContext().getConnection().getQueryServices().clearTableRegionCache(htable.getTableName());
+this.scanIterator =
+
plan.iterator(DefaultParallelScanGrouper.getInstance(), newScan);
--- End diff --

Agree with you James. I think I can raise another issue for this and work 
on it. Wdyt?


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-05 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227543#comment-15227543
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58641527
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java 
---
@@ -556,35 +564,55 @@ private static String toString(List gps) {
 } else {
 endKey = regionBoundaries.get(regionIndex);
 }
-HRegionLocation regionLocation = 
regionLocations.get(regionIndex);
-if (isLocalIndex) {
-HRegionInfo regionInfo = 
regionLocation.getRegionInfo();
-endRegionKey = regionInfo.getEndKey();
-keyOffset = 
ScanUtil.getRowKeyOffset(regionInfo.getStartKey(), endRegionKey);
-}
-try {
-while (guideIndex < gpsSize && 
(currentGuidePost.compareTo(endKey) <= 0 || endKey.length == 0)) {
-Scan newScan = scanRanges.intersectScan(scan, 
currentKeyBytes, currentGuidePostBytes, keyOffset,
-false);
-estimatedRows += 
gps.getRowCounts().get(guideIndex);
-estimatedSize += 
gps.getByteCounts().get(guideIndex);
-scans = addNewScan(parallelScans, scans, newScan, 
currentGuidePostBytes, false, regionLocation);
-currentKeyBytes = currentGuidePost.copyBytes();
-currentGuidePost = PrefixByteCodec.decode(decoder, 
input);
-currentGuidePostBytes = 
currentGuidePost.copyBytes();
-guideIndex++;
-}
-} catch (EOFException e) {}
-Scan newScan = scanRanges.intersectScan(scan, 
currentKeyBytes, endKey, keyOffset, true);
-if (isLocalIndex) {
-if (newScan != null) {
-newScan.setAttribute(EXPECTED_UPPER_REGION_KEY, 
endRegionKey);
-} else if (!scans.isEmpty()) {
-
scans.get(scans.size()-1).setAttribute(EXPECTED_UPPER_REGION_KEY, endRegionKey);
-}
-}
-scans = addNewScan(parallelScans, scans, newScan, endKey, 
true, regionLocation);
-currentKeyBytes = endKey;
+if (Bytes.compareTo(scan.getStartRow(), 
context.getScan().getStartRow()) != 0
+ || Bytes.compareTo(scan.getStopRow(), 
context.getScan().getStopRow()) != 0) {
--- End diff --

Seems like you've forked the bulk of the code completely. The only think 
you'd duplicate would be the loop and a bit of setup code up top (that could be 
factored out). What's the entry point for this? Is the problem that the entry 
point is the same?


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-05 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227541#comment-15227541
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58641320
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java 
---
@@ -556,35 +564,55 @@ private static String toString(List gps) {
 } else {
 endKey = regionBoundaries.get(regionIndex);
 }
-HRegionLocation regionLocation = 
regionLocations.get(regionIndex);
-if (isLocalIndex) {
-HRegionInfo regionInfo = 
regionLocation.getRegionInfo();
-endRegionKey = regionInfo.getEndKey();
-keyOffset = 
ScanUtil.getRowKeyOffset(regionInfo.getStartKey(), endRegionKey);
-}
-try {
-while (guideIndex < gpsSize && 
(currentGuidePost.compareTo(endKey) <= 0 || endKey.length == 0)) {
-Scan newScan = scanRanges.intersectScan(scan, 
currentKeyBytes, currentGuidePostBytes, keyOffset,
-false);
-estimatedRows += 
gps.getRowCounts().get(guideIndex);
-estimatedSize += 
gps.getByteCounts().get(guideIndex);
-scans = addNewScan(parallelScans, scans, newScan, 
currentGuidePostBytes, false, regionLocation);
-currentKeyBytes = currentGuidePost.copyBytes();
-currentGuidePost = PrefixByteCodec.decode(decoder, 
input);
-currentGuidePostBytes = 
currentGuidePost.copyBytes();
-guideIndex++;
-}
-} catch (EOFException e) {}
-Scan newScan = scanRanges.intersectScan(scan, 
currentKeyBytes, endKey, keyOffset, true);
-if (isLocalIndex) {
-if (newScan != null) {
-newScan.setAttribute(EXPECTED_UPPER_REGION_KEY, 
endRegionKey);
-} else if (!scans.isEmpty()) {
-
scans.get(scans.size()-1).setAttribute(EXPECTED_UPPER_REGION_KEY, endRegionKey);
-}
-}
-scans = addNewScan(parallelScans, scans, newScan, endKey, 
true, regionLocation);
-currentKeyBytes = endKey;
+if (Bytes.compareTo(scan.getStartRow(), 
context.getScan().getStartRow()) != 0
+ || Bytes.compareTo(scan.getStopRow(), 
context.getScan().getStopRow()) != 0) {
+Scan newScan = ScanUtil.newScan(scan);
+if(ScanUtil.isLocalIndex(scan)) {
+newScan.setStartRow(regionInfo.getStartKey());
+newScan.setAttribute(SCAN_ACTUAL_START_ROW, 
regionInfo.getStartKey());
+newScan.setStopRow(regionInfo.getEndKey());
+newScan.setAttribute(EXPECTED_UPPER_REGION_KEY, 
regionInfo.getEndKey());
+} else {
+if(Bytes.compareTo(scan.getStartRow(), 
regionInfo.getStartKey())<=0) {
+newScan.setAttribute(SCAN_ACTUAL_START_ROW, 
regionInfo.getStartKey());
+newScan.setStartRow(regionInfo.getStartKey());
+}
+if(scan.getStopRow().length == 0 || 
(regionInfo.getEndKey().length != 0 && Bytes.compareTo(scan.getStopRow(), 
regionInfo.getEndKey())>0)) {
+newScan.setStopRow(regionInfo.getEndKey());
+}
+}   
+scans = addNewScan(parallelScans, scans, newScan, 
endKey, true, regionLocation);
+ } else {
--- End diff --

Can we remove the if (isLocalIndex) checks below since the above if 
statement would have been entered instead?


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-05 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227540#comment-15227540
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58641238
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java 
---
@@ -556,35 +564,55 @@ private static String toString(List gps) {
 } else {
 endKey = regionBoundaries.get(regionIndex);
 }
-HRegionLocation regionLocation = 
regionLocations.get(regionIndex);
-if (isLocalIndex) {
-HRegionInfo regionInfo = 
regionLocation.getRegionInfo();
-endRegionKey = regionInfo.getEndKey();
-keyOffset = 
ScanUtil.getRowKeyOffset(regionInfo.getStartKey(), endRegionKey);
-}
-try {
-while (guideIndex < gpsSize && 
(currentGuidePost.compareTo(endKey) <= 0 || endKey.length == 0)) {
-Scan newScan = scanRanges.intersectScan(scan, 
currentKeyBytes, currentGuidePostBytes, keyOffset,
-false);
-estimatedRows += 
gps.getRowCounts().get(guideIndex);
-estimatedSize += 
gps.getByteCounts().get(guideIndex);
-scans = addNewScan(parallelScans, scans, newScan, 
currentGuidePostBytes, false, regionLocation);
-currentKeyBytes = currentGuidePost.copyBytes();
-currentGuidePost = PrefixByteCodec.decode(decoder, 
input);
-currentGuidePostBytes = 
currentGuidePost.copyBytes();
-guideIndex++;
-}
-} catch (EOFException e) {}
-Scan newScan = scanRanges.intersectScan(scan, 
currentKeyBytes, endKey, keyOffset, true);
-if (isLocalIndex) {
-if (newScan != null) {
-newScan.setAttribute(EXPECTED_UPPER_REGION_KEY, 
endRegionKey);
-} else if (!scans.isEmpty()) {
-
scans.get(scans.size()-1).setAttribute(EXPECTED_UPPER_REGION_KEY, endRegionKey);
-}
-}
-scans = addNewScan(parallelScans, scans, newScan, endKey, 
true, regionLocation);
-currentKeyBytes = endKey;
+if (Bytes.compareTo(scan.getStartRow(), 
context.getScan().getStartRow()) != 0
+ || Bytes.compareTo(scan.getStopRow(), 
context.getScan().getStopRow()) != 0) {
+Scan newScan = ScanUtil.newScan(scan);
+if(ScanUtil.isLocalIndex(scan)) {
+newScan.setStartRow(regionInfo.getStartKey());
+newScan.setAttribute(SCAN_ACTUAL_START_ROW, 
regionInfo.getStartKey());
--- End diff --

If we don't need EXPECTED_UPPER_REGION_KEY, let's remove it.


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-05 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227535#comment-15227535
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58641031
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java 
---
@@ -326,11 +325,12 @@ private static void 
optimizeProjection(StatementContext context, Scan scan, PTab
 }
 }
 
-public BaseResultIterators(QueryPlan plan, Integer perScanLimit, 
ParallelScanGrouper scanGrouper) throws SQLException {
+public BaseResultIterators(QueryPlan plan, Integer perScanLimit, 
ParallelScanGrouper scanGrouper, Scan scan) throws SQLException {
 super(plan.getContext(), plan.getTableRef(), plan.getGroupBy(), 
plan.getOrderBy(), plan.getStatement().getHint(), plan.getLimit());
 this.plan = plan;
 this.scanGrouper = scanGrouper;
 StatementContext context = plan.getContext();
+this.scan = scan == null ? context.getScan() : scan;
--- End diff --

This null check is not required will remove it.


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-05 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227534#comment-15227534
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58641009
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java 
---
@@ -465,7 +465,14 @@ private static String toString(List gps) {
 }
 
 private List getParallelScans() throws SQLException {
-return getParallelScans(EMPTY_BYTE_ARRAY, EMPTY_BYTE_ARRAY);
+if (scan == null
+|| (ScanUtil.isLocalIndex(scan)
+&& 
Bytes.compareTo(context.getScan().getStartRow(), scan.getStartRow()) == 0 && 
Bytes
+.compareTo(context.getScan().getStopRow(), 
scan.getStopRow()) == 0)) {
--- End diff --

Sure will move to ScanUtil


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-05 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227533#comment-15227533
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58640990
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java 
---
@@ -465,7 +465,14 @@ private static String toString(List gps) {
 }
 
 private List getParallelScans() throws SQLException {
-return getParallelScans(EMPTY_BYTE_ARRAY, EMPTY_BYTE_ARRAY);
+if (scan == null
+|| (ScanUtil.isLocalIndex(scan)
+&& 
Bytes.compareTo(context.getScan().getStartRow(), scan.getStartRow()) == 0 && 
Bytes
+.compareTo(context.getScan().getStopRow(), 
scan.getStopRow()) == 0)) {
+return getParallelScans(EMPTY_BYTE_ARRAY, EMPTY_BYTE_ARRAY);
--- End diff --

This check detects whether scan boundaries  are equal to context scan 
boundaries or not. If they are same we are going by getting all parallel scans 
for the table. Will document it.


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-05 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227527#comment-15227527
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58640786
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIterator.java 
---
@@ -107,8 +127,37 @@ public synchronized void close() throws SQLException {
 @Override
 public synchronized Tuple next() throws SQLException {
 initScanner();
-Tuple t = scanIterator.next();
-return t;
+try {
+lastTuple = scanIterator.next();
+if (lastTuple != null) {
+ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+lastTuple.getKey(ptr);
+}
+} catch (SQLException e) {
+try {
+throw ServerUtil.parseServerException(e);
+} catch(StaleRegionBoundaryCacheException e1) {
+if(scan.getAttribute(NON_AGGREGATE_QUERY)!=null) {
+Scan newScan = ScanUtil.newScan(scan);
+if(lastTuple != null) {
+lastTuple.getKey(ptr);
+byte[] startRowSuffix = 
ByteUtil.copyKeyBytesIfNecessary(ptr);
+if(ScanUtil.isLocalIndex(newScan)) {
+newScan.setAttribute(SCAN_START_ROW_SUFFIX, 
ByteUtil.nextKey(startRowSuffix));
+} else {
+
newScan.setStartRow(ByteUtil.nextKey(startRowSuffix));
--- End diff --

See other comment below, but how about we serialize the row key we found 
which is out-of-region in the StaleRegionBoundaryCacheException instead of 
mucking with the startRowSuffix?


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-05 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227528#comment-15227528
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58640818
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java 
---
@@ -556,35 +564,55 @@ private static String toString(List gps) {
 } else {
 endKey = regionBoundaries.get(regionIndex);
 }
-HRegionLocation regionLocation = 
regionLocations.get(regionIndex);
-if (isLocalIndex) {
-HRegionInfo regionInfo = 
regionLocation.getRegionInfo();
-endRegionKey = regionInfo.getEndKey();
-keyOffset = 
ScanUtil.getRowKeyOffset(regionInfo.getStartKey(), endRegionKey);
-}
-try {
-while (guideIndex < gpsSize && 
(currentGuidePost.compareTo(endKey) <= 0 || endKey.length == 0)) {
-Scan newScan = scanRanges.intersectScan(scan, 
currentKeyBytes, currentGuidePostBytes, keyOffset,
-false);
-estimatedRows += 
gps.getRowCounts().get(guideIndex);
-estimatedSize += 
gps.getByteCounts().get(guideIndex);
-scans = addNewScan(parallelScans, scans, newScan, 
currentGuidePostBytes, false, regionLocation);
-currentKeyBytes = currentGuidePost.copyBytes();
-currentGuidePost = PrefixByteCodec.decode(decoder, 
input);
-currentGuidePostBytes = 
currentGuidePost.copyBytes();
-guideIndex++;
-}
-} catch (EOFException e) {}
-Scan newScan = scanRanges.intersectScan(scan, 
currentKeyBytes, endKey, keyOffset, true);
-if (isLocalIndex) {
-if (newScan != null) {
-newScan.setAttribute(EXPECTED_UPPER_REGION_KEY, 
endRegionKey);
-} else if (!scans.isEmpty()) {
-
scans.get(scans.size()-1).setAttribute(EXPECTED_UPPER_REGION_KEY, endRegionKey);
-}
-}
-scans = addNewScan(parallelScans, scans, newScan, endKey, 
true, regionLocation);
-currentKeyBytes = endKey;
+if (Bytes.compareTo(scan.getStartRow(), 
context.getScan().getStartRow()) != 0
+ || Bytes.compareTo(scan.getStopRow(), 
context.getScan().getStopRow()) != 0) {
--- End diff --

If I move creating parallel scans for this special case I need to duplicate 
lot of code. That's why I have added special case as part of creating existing 
parallel scans only.


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-05 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227523#comment-15227523
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58640556
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIterator.java 
---
@@ -107,8 +127,37 @@ public synchronized void close() throws SQLException {
 @Override
 public synchronized Tuple next() throws SQLException {
 initScanner();
-Tuple t = scanIterator.next();
-return t;
+try {
+lastTuple = scanIterator.next();
+if (lastTuple != null) {
+ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+lastTuple.getKey(ptr);
+}
+} catch (SQLException e) {
+try {
+throw ServerUtil.parseServerException(e);
+} catch(StaleRegionBoundaryCacheException e1) {
+if(scan.getAttribute(NON_AGGREGATE_QUERY)!=null) {
+Scan newScan = ScanUtil.newScan(scan);
+if(lastTuple != null) {
+lastTuple.getKey(ptr);
+byte[] startRowSuffix = 
ByteUtil.copyKeyBytesIfNecessary(ptr);
+if(ScanUtil.isLocalIndex(newScan)) {
+newScan.setAttribute(SCAN_START_ROW_SUFFIX, 
ByteUtil.nextKey(startRowSuffix));
+} else {
+
newScan.setStartRow(ByteUtil.nextKey(startRowSuffix));
+}
+}
+
plan.getContext().getConnection().getQueryServices().clearTableRegionCache(htable.getTableName());
+this.scanIterator =
+
plan.iterator(DefaultParallelScanGrouper.getInstance(), newScan);
--- End diff --

Yes, exactly. Except for the ordered aggregation case, which needs to be 
handled as the non aggregate is handled (i.e. we'd still throw the stale region 
boundary exception, but the client would need to react to it as you've done for 
the non aggregate case. We need to know which exact row key we're on though 
(which the client won't know). Seems like the more feasible approach would be 
to include the row key in the exception and deserialize it on the client side.


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-05 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227518#comment-15227518
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58640455
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java 
---
@@ -556,35 +564,55 @@ private static String toString(List gps) {
 } else {
 endKey = regionBoundaries.get(regionIndex);
 }
-HRegionLocation regionLocation = 
regionLocations.get(regionIndex);
-if (isLocalIndex) {
-HRegionInfo regionInfo = 
regionLocation.getRegionInfo();
-endRegionKey = regionInfo.getEndKey();
-keyOffset = 
ScanUtil.getRowKeyOffset(regionInfo.getStartKey(), endRegionKey);
-}
-try {
-while (guideIndex < gpsSize && 
(currentGuidePost.compareTo(endKey) <= 0 || endKey.length == 0)) {
-Scan newScan = scanRanges.intersectScan(scan, 
currentKeyBytes, currentGuidePostBytes, keyOffset,
-false);
-estimatedRows += 
gps.getRowCounts().get(guideIndex);
-estimatedSize += 
gps.getByteCounts().get(guideIndex);
-scans = addNewScan(parallelScans, scans, newScan, 
currentGuidePostBytes, false, regionLocation);
-currentKeyBytes = currentGuidePost.copyBytes();
-currentGuidePost = PrefixByteCodec.decode(decoder, 
input);
-currentGuidePostBytes = 
currentGuidePost.copyBytes();
-guideIndex++;
-}
-} catch (EOFException e) {}
-Scan newScan = scanRanges.intersectScan(scan, 
currentKeyBytes, endKey, keyOffset, true);
-if (isLocalIndex) {
-if (newScan != null) {
-newScan.setAttribute(EXPECTED_UPPER_REGION_KEY, 
endRegionKey);
-} else if (!scans.isEmpty()) {
-
scans.get(scans.size()-1).setAttribute(EXPECTED_UPPER_REGION_KEY, endRegionKey);
-}
-}
-scans = addNewScan(parallelScans, scans, newScan, endKey, 
true, regionLocation);
-currentKeyBytes = endKey;
+if (Bytes.compareTo(scan.getStartRow(), 
context.getScan().getStartRow()) != 0
+ || Bytes.compareTo(scan.getStopRow(), 
context.getScan().getStopRow()) != 0) {
+Scan newScan = ScanUtil.newScan(scan);
+if(ScanUtil.isLocalIndex(scan)) {
+newScan.setStartRow(regionInfo.getStartKey());
+newScan.setAttribute(SCAN_ACTUAL_START_ROW, 
regionInfo.getStartKey());
--- End diff --

First one is actual start key of the scan second one is end key of region. 
After this patch we don't need EXPECTED_UPPER_REGION_KEY but kept it for 
compatibility.


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-05 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227514#comment-15227514
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58640276
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIterator.java 
---
@@ -121,8 +170,21 @@ public synchronized void initScanner() throws 
SQLException {
 this.scanIterator =
 new 
ScanningResultIterator(htable.getScanner(scan), scanMetrics);
 } catch (IOException e) {
-Closeables.closeQuietly(htable);
-throw ServerUtil.parseServerException(e);
+if(handleSplitRegionBoundaryFailureDuringInitialization) {
--- End diff --

This is required for ChunkedResultIterator if we are going to deprecate or 
not use it we don't need these changes. Other than that we might need to deal 
with it in PhoenixRecordReader(currently we skipping range check but for local 
indexes it's compulsory to handle otherwise we might miss one of the daughter 
records)


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-05 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227508#comment-15227508
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58639989
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIterator.java 
---
@@ -107,8 +127,37 @@ public synchronized void close() throws SQLException {
 @Override
 public synchronized Tuple next() throws SQLException {
 initScanner();
-Tuple t = scanIterator.next();
-return t;
+try {
+lastTuple = scanIterator.next();
+if (lastTuple != null) {
+ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+lastTuple.getKey(ptr);
+}
+} catch (SQLException e) {
+try {
+throw ServerUtil.parseServerException(e);
+} catch(StaleRegionBoundaryCacheException e1) {
--- End diff --

I will add.


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-05 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227506#comment-15227506
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58639956
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIterator.java 
---
@@ -107,8 +127,37 @@ public synchronized void close() throws SQLException {
 @Override
 public synchronized Tuple next() throws SQLException {
 initScanner();
-Tuple t = scanIterator.next();
-return t;
+try {
+lastTuple = scanIterator.next();
+if (lastTuple != null) {
+ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+lastTuple.getKey(ptr);
+}
+} catch (SQLException e) {
+try {
+throw ServerUtil.parseServerException(e);
+} catch(StaleRegionBoundaryCacheException e1) {
+if(scan.getAttribute(NON_AGGREGATE_QUERY)!=null) {
+Scan newScan = ScanUtil.newScan(scan);
+if(lastTuple != null) {
+lastTuple.getKey(ptr);
+byte[] startRowSuffix = 
ByteUtil.copyKeyBytesIfNecessary(ptr);
+if(ScanUtil.isLocalIndex(newScan)) {
+newScan.setAttribute(SCAN_START_ROW_SUFFIX, 
ByteUtil.nextKey(startRowSuffix));
+} else {
+
newScan.setStartRow(ByteUtil.nextKey(startRowSuffix));
--- End diff --

Will raise improvement action for this James.


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-05 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227507#comment-15227507
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58639974
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIterator.java 
---
@@ -107,8 +127,37 @@ public synchronized void close() throws SQLException {
 @Override
 public synchronized Tuple next() throws SQLException {
 initScanner();
-Tuple t = scanIterator.next();
-return t;
+try {
+lastTuple = scanIterator.next();
+if (lastTuple != null) {
+ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+lastTuple.getKey(ptr);
+}
+} catch (SQLException e) {
+try {
+throw ServerUtil.parseServerException(e);
+} catch(StaleRegionBoundaryCacheException e1) {
+if(scan.getAttribute(NON_AGGREGATE_QUERY)!=null) {
--- End diff --

Will move the check to ScanUtil.


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-05 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227501#comment-15227501
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58639845
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIterator.java 
---
@@ -80,13 +94,19 @@
 };
 
 
-public TableResultIterator(MutationState mutationState, TableRef 
tableRef, Scan scan, CombinableMetric scanMetrics, long renewLeaseThreshold) 
throws SQLException {
+public TableResultIterator(MutationState mutationState, Scan scan, 
CombinableMetric scanMetrics, long renewLeaseThreshold, QueryPlan plan) throws 
SQLException {
+this(mutationState, scan, scanMetrics, renewLeaseThreshold, plan, 
false);
+}
+
+public TableResultIterator(MutationState mutationState, Scan scan, 
CombinableMetric scanMetrics, long renewLeaseThreshold, QueryPlan plan, boolean 
handleSplitRegionBoundaryFailureDuringInitialization) throws SQLException {
 this.scan = scan;
 this.scanMetrics = scanMetrics;
-PTable table = tableRef.getTable();
+PTable table = plan.getTableRef().getTable();
 htable = mutationState.getHTable(table);
 this.scanIterator = UNINITIALIZED_SCANNER;
 this.renewLeaseThreshold = renewLeaseThreshold;
+this.plan = plan;
+this.handleSplitRegionBoundaryFailureDuringInitialization = 
handleSplitRegionBoundaryFailureDuringInitialization;
--- End diff --

handleSplitRegionBoundaryFailureDuringInitialization will be true for 
scanning intermediate chunks in ChunkedResultIterator. When we create scanner 
for new chunk we might see stale region boundaries in that case also we need to 
forcibly recreate the iterator with new boundaries. In normal case if we get 
StaleRegionBoundary exception while creating scanner BaseResultIterator handles 
it. 


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-05 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227496#comment-15227496
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58639358
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIterator.java 
---
@@ -107,8 +127,37 @@ public synchronized void close() throws SQLException {
 @Override
 public synchronized Tuple next() throws SQLException {
 initScanner();
-Tuple t = scanIterator.next();
-return t;
+try {
+lastTuple = scanIterator.next();
+if (lastTuple != null) {
+ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+lastTuple.getKey(ptr);
+}
+} catch (SQLException e) {
+try {
+throw ServerUtil.parseServerException(e);
+} catch(StaleRegionBoundaryCacheException e1) {
+if(scan.getAttribute(NON_AGGREGATE_QUERY)!=null) {
+Scan newScan = ScanUtil.newScan(scan);
+if(lastTuple != null) {
+lastTuple.getKey(ptr);
+byte[] startRowSuffix = 
ByteUtil.copyKeyBytesIfNecessary(ptr);
+if(ScanUtil.isLocalIndex(newScan)) {
+newScan.setAttribute(SCAN_START_ROW_SUFFIX, 
ByteUtil.nextKey(startRowSuffix));
+} else {
+
newScan.setStartRow(ByteUtil.nextKey(startRowSuffix));
+}
+}
+
plan.getContext().getConnection().getQueryServices().clearTableRegionCache(htable.getTableName());
+this.scanIterator =
+
plan.iterator(DefaultParallelScanGrouper.getInstance(), newScan);
--- End diff --

You mean we can check the in region boundary again in post scanner open and 
if it's out of range throw stale region boundary exception which will be 
handled by BaseResultIterators?


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-05 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227489#comment-15227489
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58638946
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/compile/ScanRanges.java ---
@@ -385,9 +387,25 @@ public Scan intersectScan(Scan scan, final byte[] 
originalStartKey, final byte[]
 if (scanStopKey.length > 0 && Bytes.compareTo(scanStartKey, 
scanStopKey) >= 0) { 
 return null; 
 }
-newScan.setAttribute(SCAN_ACTUAL_START_ROW, scanStartKey);
-newScan.setStartRow(scanStartKey);
-newScan.setStopRow(scanStopKey);
+if(ScanUtil.isLocalIndex(scan)) {
--- End diff --

I will try moving outside of this method.


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-05 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227488#comment-15227488
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58638911
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/GroupedAggregateRegionObserver.java
 ---
@@ -402,8 +405,8 @@ private RegionScanner 
scanUnordered(ObserverContext Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-05 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227486#comment-15227486
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58638661
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIterator.java 
---
@@ -107,8 +127,37 @@ public synchronized void close() throws SQLException {
 @Override
 public synchronized Tuple next() throws SQLException {
 initScanner();
-Tuple t = scanIterator.next();
-return t;
+try {
+lastTuple = scanIterator.next();
+if (lastTuple != null) {
+ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+lastTuple.getKey(ptr);
+}
+} catch (SQLException e) {
+try {
+throw ServerUtil.parseServerException(e);
+} catch(StaleRegionBoundaryCacheException e1) {
+if(scan.getAttribute(NON_AGGREGATE_QUERY)!=null) {
+Scan newScan = ScanUtil.newScan(scan);
+if(lastTuple != null) {
+lastTuple.getKey(ptr);
+byte[] startRowSuffix = 
ByteUtil.copyKeyBytesIfNecessary(ptr);
+if(ScanUtil.isLocalIndex(newScan)) {
+newScan.setAttribute(SCAN_START_ROW_SUFFIX, 
ByteUtil.nextKey(startRowSuffix));
+} else {
+
newScan.setStartRow(ByteUtil.nextKey(startRowSuffix));
+}
+}
+
plan.getContext().getConnection().getQueryServices().clearTableRegionCache(htable.getTableName());
+this.scanIterator =
+
plan.iterator(DefaultParallelScanGrouper.getInstance(), newScan);
--- End diff --

Ah, I misread that. The case of an ordered aggregate falls into the same 
category, however, as we're aggregating in-place. I don't think we need any 
additional code in the unordered aggregate code path in 
GroupedAggregateRegionObserver (other than the duplicate in-region check I 
mentioned above).



> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-05 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227483#comment-15227483
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58638568
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/GroupedAggregateRegionObserver.java
 ---
@@ -423,7 +426,14 @@ private RegionScanner 
scanUnordered(ObserverContext Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-05 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227463#comment-15227463
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58637933
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIterator.java 
---
@@ -107,8 +127,37 @@ public synchronized void close() throws SQLException {
 @Override
 public synchronized Tuple next() throws SQLException {
 initScanner();
-Tuple t = scanIterator.next();
-return t;
+try {
+lastTuple = scanIterator.next();
+if (lastTuple != null) {
+ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+lastTuple.getKey(ptr);
+}
+} catch (SQLException e) {
+try {
+throw ServerUtil.parseServerException(e);
+} catch(StaleRegionBoundaryCacheException e1) {
+if(scan.getAttribute(NON_AGGREGATE_QUERY)!=null) {
+Scan newScan = ScanUtil.newScan(scan);
+if(lastTuple != null) {
+lastTuple.getKey(ptr);
+byte[] startRowSuffix = 
ByteUtil.copyKeyBytesIfNecessary(ptr);
+if(ScanUtil.isLocalIndex(newScan)) {
+newScan.setAttribute(SCAN_START_ROW_SUFFIX, 
ByteUtil.nextKey(startRowSuffix));
+} else {
+
newScan.setStartRow(ByteUtil.nextKey(startRowSuffix));
+}
+}
+
plan.getContext().getConnection().getQueryServices().clearTableRegionCache(htable.getTableName());
+this.scanIterator =
+
plan.iterator(DefaultParallelScanGrouper.getInstance(), newScan);
--- End diff --

Yes aggregate queries are already handled properly. This code is only for 
handling splits when we are in the middle of non aggregate queries. If there 
are splits in the starting of the query then we are throwing out the stale 
region exception to BaseResultIterators which handles creating the proper 
parallel scans.
+if(scan.getAttribute(NON_AGGREGATE_QUERY)!=null) {



> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-05 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227454#comment-15227454
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58637402
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIterator.java 
---
@@ -107,8 +127,37 @@ public synchronized void close() throws SQLException {
 @Override
 public synchronized Tuple next() throws SQLException {
 initScanner();
-Tuple t = scanIterator.next();
-return t;
+try {
+lastTuple = scanIterator.next();
+if (lastTuple != null) {
+ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+lastTuple.getKey(ptr);
+}
+} catch (SQLException e) {
+try {
+throw ServerUtil.parseServerException(e);
+} catch(StaleRegionBoundaryCacheException e1) {
+if(scan.getAttribute(NON_AGGREGATE_QUERY)!=null) {
+Scan newScan = ScanUtil.newScan(scan);
+if(lastTuple != null) {
+lastTuple.getKey(ptr);
+byte[] startRowSuffix = 
ByteUtil.copyKeyBytesIfNecessary(ptr);
+if(ScanUtil.isLocalIndex(newScan)) {
+newScan.setAttribute(SCAN_START_ROW_SUFFIX, 
ByteUtil.nextKey(startRowSuffix));
+} else {
+
newScan.setStartRow(ByteUtil.nextKey(startRowSuffix));
+}
+}
+
plan.getContext().getConnection().getQueryServices().clearTableRegionCache(htable.getTableName());
+this.scanIterator =
+
plan.iterator(DefaultParallelScanGrouper.getInstance(), newScan);
--- End diff --

So we're re-running the scan, only if it's an aggregate scan? Does this 
apply only for an "ordered" group by (which is grouping by the leading part of 
the pk and doing so in-place)? In the unordered case, each scan is done during 
the post open and the results are cached on the RS, so a split won't impact the 
results. There may be a race condition between the pre open (where we check if 
we're in the region) and the post open (where we start the region operation), 
but I think that could be solved by doing the in-region check again). In these 
cases, we already handle this through the exception catching code in 
BaseResultIterators.


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-05 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227436#comment-15227436
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58636239
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/compile/ScanRanges.java ---
@@ -385,9 +387,25 @@ public Scan intersectScan(Scan scan, final byte[] 
originalStartKey, final byte[]
 if (scanStopKey.length > 0 && Bytes.compareTo(scanStartKey, 
scanStopKey) >= 0) { 
 return null; 
 }
-newScan.setAttribute(SCAN_ACTUAL_START_ROW, scanStartKey);
-newScan.setStartRow(scanStartKey);
-newScan.setStopRow(scanStopKey);
+if(ScanUtil.isLocalIndex(scan)) {
--- End diff --

How about moving the setting of these special attributes outside of this 
method?


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-05 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227368#comment-15227368
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58632305
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/compile/ScanRanges.java ---
@@ -385,9 +387,25 @@ public Scan intersectScan(Scan scan, final byte[] 
originalStartKey, final byte[]
 if (scanStopKey.length > 0 && Bytes.compareTo(scanStartKey, 
scanStopKey) >= 0) { 
 return null; 
 }
-newScan.setAttribute(SCAN_ACTUAL_START_ROW, scanStartKey);
-newScan.setStartRow(scanStartKey);
-newScan.setStopRow(scanStopKey);
+if(ScanUtil.isLocalIndex(scan)) {
--- End diff --

@JamesRTaylor  we cannot use keyoffset > 0 check always for considering 
local indexes scan because in special case like if a table has only one region 
then region start key length and end key length is zero so keyoffset also 
becoming zero. In that case we need to check local index scan or not to set the 
attributes properly. Wdyt? 


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-04 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225664#comment-15225664
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58485360
  
--- Diff: 
phoenix-core/src/main/java/org/apache/hadoop/hbase/regionserver/IndexHalfStoreFileReaderGenerator.java
 ---
@@ -157,6 +162,7 @@ public Reader 
preStoreFileReaderOpen(ObserverContext Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-04 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225663#comment-15225663
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58485341
  
--- Diff: 
phoenix-core/src/it/java/org/apache/phoenix/end2end/index/MutableIndexIT.java 
---
@@ -86,8 +98,8 @@ public static void doSetup() throws Exception {

@Parameters(name="localIndex = {0} , transactional = {1}")
 public static Collection data() {
-return Arrays.asList(new Boolean[][] { 
- { false, false }, { false, true }, { true, false }, { 
true, true }
--- End diff --

Yes James it's space only change. 


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-04 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225662#comment-15225662
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user chrajeshbabu commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58485324
  
--- Diff: 
phoenix-core/src/it/java/org/apache/phoenix/end2end/index/LocalIndexIT.java ---
@@ -727,7 +727,7 @@ public void testLocalIndexScanAfterRegionSplit() throws 
Exception {
 assertTrue(rs.next());
 
 HBaseAdmin admin = driver.getConnectionQueryServices(getUrl(), 
TestUtil.TEST_PROPERTIES).getAdmin();
-for (int i = 1; i < 5; i++) {
+for (int i = 1; i < 2; i++) {
--- End diff --

While debugging a failure changed it to 2. 5 is ok will keep it as it is.


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-04 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225661#comment-15225661
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user JamesRTaylor commented on the pull request:

https://github.com/apache/phoenix/pull/156#issuecomment-205637233
  
Thanks for the patch, @chrajeshbabu. This is a big improvement over your 
initial approach. It's still complex, though. There's an on going cost to 
complicating the code to this degree that would be good to minimize. 

One big thing missing is an overview of all the moving parts. Perhaps that 
could live in BaseResultIterators? For example - this exception is thrown under 
these conditions and this state is added to the scan to know how to adjust the 
scan ranges, etc. etc. Especially that hasReferences bit in the coprocessor - 
what's that all about?

I'd like to understand how ChunkedResultIterator complicates things too. I 
think it'd be ok to document that splits during aggregate queries aren't 
handled if your HBase version is less that XXX if it'd significantly simplify 
this patch (as that code is essentially deprecated). 

Another potential, different approach would be for Phoenix to universally 
handle the split during scan case (rather than letting the HBase client scanner 
handle it for non aggregate case and Phoenix handle it for the aggregate case). 
Would that simplify things?


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-04 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225654#comment-15225654
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58484498
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIterator.java 
---
@@ -121,8 +170,21 @@ public synchronized void initScanner() throws 
SQLException {
 this.scanIterator =
 new 
ScanningResultIterator(htable.getScanner(scan), scanMetrics);
 } catch (IOException e) {
-Closeables.closeQuietly(htable);
-throw ServerUtil.parseServerException(e);
+if(handleSplitRegionBoundaryFailureDuringInitialization) {
--- End diff --

Rather than have a handleSplitRegionBoundaryFailureDuringInitialization, 
would it work to always throw it and let the caller deal with it? When a 
variable name gets beyond 50 characters, it's a sign that it may be too special 
of a case. :-)


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-04 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225653#comment-15225653
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58484354
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIterator.java 
---
@@ -107,8 +127,37 @@ public synchronized void close() throws SQLException {
 @Override
 public synchronized Tuple next() throws SQLException {
 initScanner();
-Tuple t = scanIterator.next();
-return t;
+try {
+lastTuple = scanIterator.next();
+if (lastTuple != null) {
+ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+lastTuple.getKey(ptr);
+}
+} catch (SQLException e) {
+try {
+throw ServerUtil.parseServerException(e);
+} catch(StaleRegionBoundaryCacheException e1) {
--- End diff --

This needs code comments, please.


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-04 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225652#comment-15225652
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58484328
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIterator.java 
---
@@ -107,8 +127,37 @@ public synchronized void close() throws SQLException {
 @Override
 public synchronized Tuple next() throws SQLException {
 initScanner();
-Tuple t = scanIterator.next();
-return t;
+try {
+lastTuple = scanIterator.next();
+if (lastTuple != null) {
+ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+lastTuple.getKey(ptr);
+}
+} catch (SQLException e) {
+try {
+throw ServerUtil.parseServerException(e);
+} catch(StaleRegionBoundaryCacheException e1) {
+if(scan.getAttribute(NON_AGGREGATE_QUERY)!=null) {
--- End diff --

Minor nit: how about adding a ScanUtil.isAggregateQuery() method instead?


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-04 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225645#comment-15225645
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58484006
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIterator.java 
---
@@ -107,8 +127,37 @@ public synchronized void close() throws SQLException {
 @Override
 public synchronized Tuple next() throws SQLException {
 initScanner();
-Tuple t = scanIterator.next();
-return t;
+try {
+lastTuple = scanIterator.next();
+if (lastTuple != null) {
+ImmutableBytesWritable ptr = new ImmutableBytesWritable();
+lastTuple.getKey(ptr);
+}
+} catch (SQLException e) {
+try {
+throw ServerUtil.parseServerException(e);
+} catch(StaleRegionBoundaryCacheException e1) {
+if(scan.getAttribute(NON_AGGREGATE_QUERY)!=null) {
+Scan newScan = ScanUtil.newScan(scan);
+if(lastTuple != null) {
+lastTuple.getKey(ptr);
+byte[] startRowSuffix = 
ByteUtil.copyKeyBytesIfNecessary(ptr);
+if(ScanUtil.isLocalIndex(newScan)) {
+newScan.setAttribute(SCAN_START_ROW_SUFFIX, 
ByteUtil.nextKey(startRowSuffix));
+} else {
+
newScan.setStartRow(ByteUtil.nextKey(startRowSuffix));
--- End diff --

I'm not convinced that this nextKey will universally work. Probably better 
(if we really want to just get the next possible key) to add a 0 byte instead. 


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-04 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225643#comment-15225643
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58483863
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIterator.java 
---
@@ -80,13 +94,19 @@
 };
 
 
-public TableResultIterator(MutationState mutationState, TableRef 
tableRef, Scan scan, CombinableMetric scanMetrics, long renewLeaseThreshold) 
throws SQLException {
+public TableResultIterator(MutationState mutationState, Scan scan, 
CombinableMetric scanMetrics, long renewLeaseThreshold, QueryPlan plan) throws 
SQLException {
+this(mutationState, scan, scanMetrics, renewLeaseThreshold, plan, 
false);
+}
+
+public TableResultIterator(MutationState mutationState, Scan scan, 
CombinableMetric scanMetrics, long renewLeaseThreshold, QueryPlan plan, boolean 
handleSplitRegionBoundaryFailureDuringInitialization) throws SQLException {
 this.scan = scan;
 this.scanMetrics = scanMetrics;
-PTable table = tableRef.getTable();
+PTable table = plan.getTableRef().getTable();
 htable = mutationState.getHTable(table);
 this.scanIterator = UNINITIALIZED_SCANNER;
 this.renewLeaseThreshold = renewLeaseThreshold;
+this.plan = plan;
+this.handleSplitRegionBoundaryFailureDuringInitialization = 
handleSplitRegionBoundaryFailureDuringInitialization;
--- End diff --

When is handleSplitRegionBoundaryFailureDuringInitialization true versus 
false? Why would we *not* want to handle it?


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-04 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225637#comment-15225637
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58483665
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java 
---
@@ -556,35 +564,55 @@ private static String toString(List gps) {
 } else {
 endKey = regionBoundaries.get(regionIndex);
 }
-HRegionLocation regionLocation = 
regionLocations.get(regionIndex);
-if (isLocalIndex) {
-HRegionInfo regionInfo = 
regionLocation.getRegionInfo();
-endRegionKey = regionInfo.getEndKey();
-keyOffset = 
ScanUtil.getRowKeyOffset(regionInfo.getStartKey(), endRegionKey);
-}
-try {
-while (guideIndex < gpsSize && 
(currentGuidePost.compareTo(endKey) <= 0 || endKey.length == 0)) {
-Scan newScan = scanRanges.intersectScan(scan, 
currentKeyBytes, currentGuidePostBytes, keyOffset,
-false);
-estimatedRows += 
gps.getRowCounts().get(guideIndex);
-estimatedSize += 
gps.getByteCounts().get(guideIndex);
-scans = addNewScan(parallelScans, scans, newScan, 
currentGuidePostBytes, false, regionLocation);
-currentKeyBytes = currentGuidePost.copyBytes();
-currentGuidePost = PrefixByteCodec.decode(decoder, 
input);
-currentGuidePostBytes = 
currentGuidePost.copyBytes();
-guideIndex++;
-}
-} catch (EOFException e) {}
-Scan newScan = scanRanges.intersectScan(scan, 
currentKeyBytes, endKey, keyOffset, true);
-if (isLocalIndex) {
-if (newScan != null) {
-newScan.setAttribute(EXPECTED_UPPER_REGION_KEY, 
endRegionKey);
-} else if (!scans.isEmpty()) {
-
scans.get(scans.size()-1).setAttribute(EXPECTED_UPPER_REGION_KEY, endRegionKey);
-}
-}
-scans = addNewScan(parallelScans, scans, newScan, endKey, 
true, regionLocation);
-currentKeyBytes = endKey;
+if (Bytes.compareTo(scan.getStartRow(), 
context.getScan().getStartRow()) != 0
+ || Bytes.compareTo(scan.getStopRow(), 
context.getScan().getStopRow()) != 0) {
+Scan newScan = ScanUtil.newScan(scan);
+if(ScanUtil.isLocalIndex(scan)) {
+newScan.setStartRow(regionInfo.getStartKey());
+newScan.setAttribute(SCAN_ACTUAL_START_ROW, 
regionInfo.getStartKey());
--- End diff --

Is there some symmetry between SCAN_ACTUAL_START_ROW and 
EXPECTED_UPPER_REGION_KEY? Should they be named similarly?


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-04 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225636#comment-15225636
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58483591
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java 
---
@@ -556,35 +564,55 @@ private static String toString(List gps) {
 } else {
 endKey = regionBoundaries.get(regionIndex);
 }
-HRegionLocation regionLocation = 
regionLocations.get(regionIndex);
-if (isLocalIndex) {
-HRegionInfo regionInfo = 
regionLocation.getRegionInfo();
-endRegionKey = regionInfo.getEndKey();
-keyOffset = 
ScanUtil.getRowKeyOffset(regionInfo.getStartKey(), endRegionKey);
-}
-try {
-while (guideIndex < gpsSize && 
(currentGuidePost.compareTo(endKey) <= 0 || endKey.length == 0)) {
-Scan newScan = scanRanges.intersectScan(scan, 
currentKeyBytes, currentGuidePostBytes, keyOffset,
-false);
-estimatedRows += 
gps.getRowCounts().get(guideIndex);
-estimatedSize += 
gps.getByteCounts().get(guideIndex);
-scans = addNewScan(parallelScans, scans, newScan, 
currentGuidePostBytes, false, regionLocation);
-currentKeyBytes = currentGuidePost.copyBytes();
-currentGuidePost = PrefixByteCodec.decode(decoder, 
input);
-currentGuidePostBytes = 
currentGuidePost.copyBytes();
-guideIndex++;
-}
-} catch (EOFException e) {}
-Scan newScan = scanRanges.intersectScan(scan, 
currentKeyBytes, endKey, keyOffset, true);
-if (isLocalIndex) {
-if (newScan != null) {
-newScan.setAttribute(EXPECTED_UPPER_REGION_KEY, 
endRegionKey);
-} else if (!scans.isEmpty()) {
-
scans.get(scans.size()-1).setAttribute(EXPECTED_UPPER_REGION_KEY, endRegionKey);
-}
-}
-scans = addNewScan(parallelScans, scans, newScan, endKey, 
true, regionLocation);
-currentKeyBytes = endKey;
+if (Bytes.compareTo(scan.getStartRow(), 
context.getScan().getStartRow()) != 0
+ || Bytes.compareTo(scan.getStopRow(), 
context.getScan().getStopRow()) != 0) {
--- End diff --

Also, this code path is strange. What's the entry point? Does it need to be 
mixed up with the existing code? It feels like we've completely forked the flow 
here. Why not have it in it's own method called only when this special case is 
detected?


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-04 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225625#comment-15225625
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58483341
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java 
---
@@ -465,7 +465,14 @@ private static String toString(List gps) {
 }
 
 private List getParallelScans() throws SQLException {
-return getParallelScans(EMPTY_BYTE_ARRAY, EMPTY_BYTE_ARRAY);
+if (scan == null
+|| (ScanUtil.isLocalIndex(scan)
+&& 
Bytes.compareTo(context.getScan().getStartRow(), scan.getStartRow()) == 0 && 
Bytes
+.compareTo(context.getScan().getStopRow(), 
scan.getStopRow()) == 0)) {
+return getParallelScans(EMPTY_BYTE_ARRAY, EMPTY_BYTE_ARRAY);
--- End diff --

This seems strange - why do we call getParallelScans with empty byte arrays 
in this case? Please document.


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-04 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225622#comment-15225622
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58483244
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java 
---
@@ -326,11 +325,12 @@ private static void 
optimizeProjection(StatementContext context, Scan scan, PTab
 }
 }
 
-public BaseResultIterators(QueryPlan plan, Integer perScanLimit, 
ParallelScanGrouper scanGrouper) throws SQLException {
+public BaseResultIterators(QueryPlan plan, Integer perScanLimit, 
ParallelScanGrouper scanGrouper, Scan scan) throws SQLException {
 super(plan.getContext(), plan.getTableRef(), plan.getGroupBy(), 
plan.getOrderBy(), plan.getStatement().getHint(), plan.getLimit());
 this.plan = plan;
 this.scanGrouper = scanGrouper;
 StatementContext context = plan.getContext();
+this.scan = scan == null ? context.getScan() : scan;
--- End diff --

When is scan null?


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-04 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225621#comment-15225621
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58483157
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/execute/ScanPlan.java ---
@@ -209,7 +209,11 @@ protected ResultIterator 
newIterator(ParallelScanGrouper scanGrouper) throws SQL
 if (isOrdered) {
 scanner = new MergeSortTopNResultIterator(iterators, limit, 
orderBy.getOrderByExpressions());
 } else {
-if ((isSalted || table.getIndexType() == IndexType.LOCAL) && 
ScanUtil.shouldRowsBeInRowKeyOrder(orderBy, context)) {
+if ((isSalted || table.getIndexType() == IndexType.LOCAL)
+&& ScanUtil.shouldRowsBeInRowKeyOrder(orderBy, context)
+|| (table.getIndexType() == IndexType.LOCAL && 
(Bytes.compareTo(
+scan.getStartRow(), 
this.context.getScan().getStartRow()) != 0 || Bytes
+.compareTo(scan.getStopRow(), 
this.context.getScan().getStopRow()) != 0))) {
--- End diff --

Why is  a merge sort required for this new condition? Is this the case when 
the original start/stop row has changed? Is this purely for the chunked result 
iterator case? Let's document this and add a TODO to remove as the chunked 
result iterator is deprecated (and if you could consistently add this TODO for 
other parts of code only necessary due to this deprecated chunking mechanism, 
that'd be much appreciated).


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-04 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225600#comment-15225600
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58482201
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/execute/BaseQueryPlan.java ---
@@ -196,26 +196,31 @@ public Expression getDynamicFilter() {
 
 @Override
 public final ResultIterator iterator(ParallelScanGrouper scanGrouper) 
throws SQLException {
-return iterator(Collections.emptyList(), 
scanGrouper);
+return iterator(Collections.emptyList(), 
scanGrouper, this.context.getScan());
 }
-
+
+@Override
+public final ResultIterator iterator(ParallelScanGrouper scanGrouper, 
Scan scan) throws SQLException {
+return iterator(Collections.emptyList(), 
scanGrouper, scan);
+}
+
 @Override
 public final ResultIterator iterator() throws SQLException {
-return iterator(Collections.emptyList(), 
DefaultParallelScanGrouper.getInstance());
+return iterator(Collections.emptyList(), 
DefaultParallelScanGrouper.getInstance(), this.context.getScan());
 }
 
-public final ResultIterator iterator(final List dependencies, ParallelScanGrouper scanGrouper) throws 
SQLException {
+public final ResultIterator iterator(final List dependencies, ParallelScanGrouper scanGrouper, Scan s) throws 
SQLException {
 if (context.getScanRanges() == ScanRanges.NOTHING) {
 return ResultIterator.EMPTY_ITERATOR;
 }
 
 if (tableRef == TableRef.EMPTY_TABLE_REF) {
-return newIterator(scanGrouper);
+return newIterator(scanGrouper, s);
 }
 
 // Set miscellaneous scan attributes. This is the last chance to 
set them before we
 // clone the scan for each parallelized chunk.
-Scan scan = context.getScan();
+Scan scan = s == null ? context.getScan() : s;
--- End diff --

Is this null check required?


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-04 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225598#comment-15225598
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58482075
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/GroupedAggregateRegionObserver.java
 ---
@@ -402,8 +405,8 @@ private RegionScanner 
scanUnordered(ObserverContext Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-04 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225591#comment-15225591
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58481927
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/GroupedAggregateRegionObserver.java
 ---
@@ -423,7 +426,14 @@ private RegionScanner 
scanUnordered(ObserverContext Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-04 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225588#comment-15225588
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58481858
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/BaseScannerRegionObserver.java
 ---
@@ -337,6 +384,22 @@ public boolean nextRaw(List result) throws 
IOException {
 arrayElementCell = 
result.get(arrayElementCellPosition);
 }
 if (ScanUtil.isLocalIndex(scan) && 
!ScanUtil.isAnalyzeTable(scan)) {
--- End diff --

We'll need some code comments here. Are we skipping until we find the 
actualStartKey row? Would it be better/possible to do a seek? This seems like 
it'd be potentially expensive, no?


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-04 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225583#comment-15225583
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58481703
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/BaseScannerRegionObserver.java
 ---
@@ -279,6 +301,31 @@ protected RegionScanner getWrappedScanner(final 
ObserverContext=0) {
+return scan.getStartRow();
+} else {
+return startKey;
+}
+}
+return null;
+}
+
 @Override
 public boolean next(List results) throws IOException {
--- End diff --

Can we refactor the code between these two next(List, ...) 
implementations, as there's now a fair amount of code which is duplicated here?


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-04 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225579#comment-15225579
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58481542
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/BaseScannerRegionObserver.java
 ---
@@ -279,6 +301,31 @@ protected RegionScanner getWrappedScanner(final 
ObserverContext Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-04 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225574#comment-15225574
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58481362
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/BaseScannerRegionObserver.java
 ---
@@ -337,6 +384,22 @@ public boolean nextRaw(List result) throws 
IOException {
 arrayElementCell = 
result.get(arrayElementCellPosition);
 }
 if (ScanUtil.isLocalIndex(scan) && 
!ScanUtil.isAnalyzeTable(scan)) {
+if(hasReferences && actualStartKey!=null) {
+Cell firstCell = result.get(0);
+while(Bytes.compareTo(firstCell.getRowArray(), 
firstCell.getRowOffset(), firstCell.getRowLength(), actualStartKey, 0, 
actualStartKey.length)<0) {
+result.clear();
+next = s.nextRaw(result);
+if (result.isEmpty()) {
+return next;
+}
+if (arrayFuncRefs != null && 
arrayFuncRefs.length > 0 && arrayKVRefs.size() > 0) {
+replaceArrayIndexElement(arrayKVRefs, 
arrayFuncRefs, result);
+}
+firstCell = result.get(0);
+}
+}
+System.out.println("-< "+scan);
+System.out.println("-<< "+result);
--- End diff --

Remove System.out.println calls


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-04 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225545#comment-15225545
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58479273
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/BaseScannerRegionObserver.java
 ---
@@ -165,7 +183,7 @@ public RegionScanner preScannerOpen(final 
ObserverContext Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-04 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225539#comment-15225539
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58478944
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/compile/ScanRanges.java ---
@@ -385,9 +387,25 @@ public Scan intersectScan(Scan scan, final byte[] 
originalStartKey, final byte[]
 if (scanStopKey.length > 0 && Bytes.compareTo(scanStartKey, 
scanStopKey) >= 0) { 
 return null; 
 }
-newScan.setAttribute(SCAN_ACTUAL_START_ROW, scanStartKey);
-newScan.setStartRow(scanStartKey);
-newScan.setStopRow(scanStopKey);
+if(ScanUtil.isLocalIndex(scan)) {
--- End diff --

The code in this if statement can go in the if (keyOffset > 0) if statement 
instead. We're trying to avoid checks for local indexes. 


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-04 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225533#comment-15225533
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58478608
  
--- Diff: 
phoenix-core/src/main/java/org/apache/hadoop/hbase/regionserver/IndexHalfStoreFileReaderGenerator.java
 ---
@@ -157,6 +162,7 @@ public Reader 
preStoreFileReaderOpen(ObserverContext Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-04 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225523#comment-15225523
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r58477862
  
--- Diff: 
phoenix-core/src/it/java/org/apache/phoenix/end2end/index/MutableIndexIT.java 
---
@@ -86,8 +98,8 @@ public static void doSetup() throws Exception {

@Parameters(name="localIndex = {0} , transactional = {1}")
 public static Collection data() {
-return Arrays.asList(new Boolean[][] { 
- { false, false }, { false, true }, { true, false }, { 
true, true }
--- End diff --

Whitespace only change?


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-04 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225521#comment-15225521
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/156#discussion_r5845
  
--- Diff: 
phoenix-core/src/it/java/org/apache/phoenix/end2end/index/LocalIndexIT.java ---
@@ -727,7 +727,7 @@ public void testLocalIndexScanAfterRegionSplit() throws 
Exception {
 assertTrue(rs.next());
 
 HBaseAdmin admin = driver.getConnectionQueryServices(getUrl(), 
TestUtil.TEST_PROPERTIES).getAdmin();
-for (int i = 1; i < 5; i++) {
+for (int i = 1; i < 2; i++) {
--- End diff --

Any reason to go from 5 to 2 here? Maybe get rid of the loop then? Does it 
decrease the amount of testing being done?


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-04-04 Thread Rajeshbabu Chintaguntla (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225382#comment-15225382
 ] 

Rajeshbabu Chintaguntla commented on PHOENIX-2628:
--

Ping [~jamestaylor] [~enis] [~anoop.hbase] [~ram_krish]?

> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-03-29 Thread Andrew Purtell (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217188#comment-15217188
 ] 

Andrew Purtell commented on PHOENIX-2628:
-

bq. - We'll need someone like [~apurtell] to review and bless 
IndexHalfStoreFileReader and IndexHalfStoreFileReaderGenerator. For example, is 
it copacetic to derive from {{StoreFile.Reader}}?

IndexHalfStoreFileReader - looks ok, but extending StoreFile.Reader is a bit 
iffy. StoreFile is LP(coproc) but Reader is not tagged. You should submit a 
patch that tags Reader as also LP(coproc). 

IndexHalfStoreFileReaderGenerator - looks ok

LocalIndexStoreFileScanner - looks ok


> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-03-29 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216313#comment-15216313
 ] 

ASF GitHub Bot commented on PHOENIX-2628:
-

GitHub user chrajeshbabu opened a pull request:

https://github.com/apache/phoenix/pull/156

PHOENIX-2628 Ensure split when iterating through results handled corr…

The patch fixes issues with splits and merges while scanning local indexes. 
 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chrajeshbabu/phoenix master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/phoenix/pull/156.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #156


commit d2e4d166dc2b32ed725183fc4465378029bd7834
Author: Rajeshbabu Chintaguntla 
Date:   2016-03-29T16:45:02Z

PHOENIX-2628 Ensure split when iterating through results handled 
correctly(Rajeshbabu)




> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-03-29 Thread James Taylor (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15215642#comment-15215642
 ] 

James Taylor commented on PHOENIX-2628:
---

Thanks for the patch, [~rajeshbabu]. Would you mind spinning up a pull request 
so we can review it more easily?

> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch, 
> PHOENIX-2628_v7.patch, PHOENIX-2628_v8.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-2628) Ensure split when iterating through results handled correctly

2016-02-18 Thread James Taylor (JIRA)


[ 
https://issues.apache.org/jira/browse/PHOENIX-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15152962#comment-15152962
 ] 

James Taylor commented on PHOENIX-2628:
---

Thanks for the wip patch, [~rajeshbabu]. This is coming along nicely, but I 
still think it's better to cut a new RC for 4.7.0 tomorrow without these 
changes as they're big and potentially disruptive. Plus, there's more work to 
do and we want to make sure it all gets done "right". 

Here's some feedback:
- We'll need someone like [~apurtell] to review and bless 
IndexHalfStoreFileReader and IndexHalfStoreFileReaderGenerator. For example, is 
it copacetic to derive from {{StoreFile.Reader}}? Are public and/or evolving 
APIs being used exclusively? If not, we'll need to come to an agreement/plan 
with the HBase community for ones that aren't. It'd be a shame to do all this 
work and end up in the same place we're in now.
- It looks like a lot of the complication comes from the handling splits/merges 
during a query and that the only place I could find that reacts to this is in 
ChunkedResultIterator (which is essentially deprecated after PHOENIX-1428 for 
HBase 0.98.17+, 1.0.3+, 1.1.3+, and 1.2+). Is that correct and if so, is that 
because a split that occurs in the first scan is already handled correctly? Can 
a split occur while iterating through the rows from the first scan and if not, 
why not?
- For an aggregate query that is grouping on a subset of the leading PK 
columns, we're not holding the region lock for the entire scan. This would be 
similar to the above situation, I believe.
- Wouldn't it be simpler/better to just support this new local indexing for the 
above versions and simplify the implementation? Or is there some case where the 
StaleRegionBoundaryCacheException is being handled on the client-side that I'm 
not seeing?

I have more detailed feedback too for smaller things, like:
- This can be simpler, no? {{ScanUtil.isLocalIndex(scan)?true:false}}
- For the new TableResultIterator constructor, if you're including the 
QueryPlan as an argument, then you can likely remove the TableRef argument and 
get it from the QueryPlan instead.
- For the new QueryPlan method, instead of passing null through, why not pass 
context.getScan() here:
{code}
 @Override
 public ResultIterator iterator(ParallelScanGrouper scanGrouper) throws 
SQLException {
-ResultIterator iterator = new 
DelegateResultIterator(delegate.iterator(scanGrouper)) {
+return iterator(scanGrouper, null);
+}
+
+@Override
+public ResultIterator iterator(ParallelScanGrouper scanGrouper, Scan scan) 
throws SQLException {
+ResultIterator iterator = new 
DelegateResultIterator(delegate.iterator(scanGrouper, scan)) {
{code}
Then you can avoid a bunch of these kinds of checks:
{code}
+this.scan = scan == null ? context.getScan() : scan;
{code}

> Ensure split when iterating through results handled correctly
> -
>
> Key: PHOENIX-2628
> URL: https://issues.apache.org/jira/browse/PHOENIX-2628
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Rajeshbabu Chintaguntla
> Attachments: PHOENIX-2628-wip.patch, PHOENIX-2628.patch
>
>
> We should start with a test case to ensure this works correctly, both for 
> scans and aggregates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

72 matches

Mail list logo