[jira] [Commented] (PHOENIX-4066) Unused variable in MutationState.send()

2017-08-04 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/PHOENIX-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115136#comment-16115136
 ] 

Hudson commented on PHOENIX-4066:
-

FAILURE: Integrated in Jenkins build Phoenix-master #1727 (See 
[https://builds.apache.org/job/Phoenix-master/1727/])
PHOENIX-4066 - Unused variable in MutationState.send() (gjacoby: rev 
bf3c13b6d169e90e06c54a338a5931163a32d743)
* (edit) 
phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java


> Unused variable in MutationState.send()
> ---
>
> Key: PHOENIX-4066
> URL: https://issues.apache.org/jira/browse/PHOENIX-4066
> Project: Phoenix
>  Issue Type: Improvement
>Affects Versions: 4.11.0
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>Priority: Trivial
>  Labels: beginner
> Fix For: 4.12.0
>
> Attachments: PHOENIX-4066.patch
>
>
> In MutationState#send, we always allocate the following:
> List txTableRefs = 
> Lists.newArrayListWithExpectedSize(mutations.size());
> and if a table's transactional, we populate it:
> txTableRefs.add(origTableRef);`
> but this is never referred to again in either logic or logging, and it's a 
> local variable so we can't be using it elsewhere either. It should be removed 
> or used (and if used, it should only be allocated in the transactional use 
> case.)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (PHOENIX-4067) Fix flakiness in unit test runs

2017-08-04 Thread James Taylor (JIRA) 
James Taylor created PHOENIX-4067:
-

 Summary: Fix flakiness in unit test runs
 Key: PHOENIX-4067
 URL: https://issues.apache.org/jira/browse/PHOENIX-4067
 Project: Phoenix
  Issue Type: Bug
Reporter: James Taylor


We rarely have a clean test run. Lots of timeouts have crept in. We need to fix 
this so that a test failure means something again.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PHOENIX-4066) Unused variable in MutationState.send()

2017-08-04 Thread James Taylor (JIRA) 

[ 
https://issues.apache.org/jira/browse/PHOENIX-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16114947#comment-16114947
 ] 

James Taylor commented on PHOENIX-4066:
---

+1

> Unused variable in MutationState.send()
> ---
>
> Key: PHOENIX-4066
> URL: https://issues.apache.org/jira/browse/PHOENIX-4066
> Project: Phoenix
>  Issue Type: Improvement
>Affects Versions: 4.11.0
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>Priority: Trivial
>  Labels: beginner
> Fix For: 4.12.0
>
> Attachments: PHOENIX-4066.patch
>
>
> In MutationState#send, we always allocate the following:
> List txTableRefs = 
> Lists.newArrayListWithExpectedSize(mutations.size());
> and if a table's transactional, we populate it:
> txTableRefs.add(origTableRef);`
> but this is never referred to again in either logic or logging, and it's a 
> local variable so we can't be using it elsewhere either. It should be removed 
> or used (and if used, it should only be allocated in the transactional use 
> case.)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PHOENIX-4066) Unused variable in MutationState.send()

2017-08-04 Thread Geoffrey Jacoby (JIRA) 

[ 
https://issues.apache.org/jira/browse/PHOENIX-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16114931#comment-16114931
 ] 

Geoffrey Jacoby commented on PHOENIX-4066:
--

Test failures are two timeouts (QueryIT and QueryDatabaseMetaDataIT) and one 
mini-cluster-failed-to-start (AlterMultiTenantTableWithViewsIT, which doesn't 
seem to have anything to do with this change.)

> Unused variable in MutationState.send()
> ---
>
> Key: PHOENIX-4066
> URL: https://issues.apache.org/jira/browse/PHOENIX-4066
> Project: Phoenix
>  Issue Type: Improvement
>Affects Versions: 4.11.0
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>Priority: Trivial
>  Labels: beginner
> Fix For: 4.12.0
>
> Attachments: PHOENIX-4066.patch
>
>
> In MutationState#send, we always allocate the following:
> List txTableRefs = 
> Lists.newArrayListWithExpectedSize(mutations.size());
> and if a table's transactional, we populate it:
> txTableRefs.add(origTableRef);`
> but this is never referred to again in either logic or logging, and it's a 
> local variable so we can't be using it elsewhere either. It should be removed 
> or used (and if used, it should only be allocated in the transactional use 
> case.)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PHOENIX-4066) Unused variable in MutationState.send()

2017-08-04 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/PHOENIX-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16114923#comment-16114923
 ] 

Hadoop QA commented on PHOENIX-4066:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12880433/PHOENIX-4066.patch
  against master branch at commit 0bd43f536a13609b35629308c415aebc800d6799.
  ATTACHMENT ID: 12880433

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:red}-1 javadoc{color}.  The javadoc tool appears to have generated 
54 warning messages.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
 
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.QueryIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.AlterMultiTenantTableWithViewsIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.QueryDatabaseMetaDataIT

Test results: 
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1244//testReport/
Javadoc warnings: 
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1244//artifact/patchprocess/patchJavadocWarnings.txt
Console output: 
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1244//console

This message is automatically generated.

> Unused variable in MutationState.send()
> ---
>
> Key: PHOENIX-4066
> URL: https://issues.apache.org/jira/browse/PHOENIX-4066
> Project: Phoenix
>  Issue Type: Improvement
>Affects Versions: 4.11.0
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>Priority: Trivial
>  Labels: beginner
> Fix For: 4.12.0
>
> Attachments: PHOENIX-4066.patch
>
>
> In MutationState#send, we always allocate the following:
> List txTableRefs = 
> Lists.newArrayListWithExpectedSize(mutations.size());
> and if a table's transactional, we populate it:
> txTableRefs.add(origTableRef);`
> but this is never referred to again in either logic or logging, and it's a 
> local variable so we can't be using it elsewhere either. It should be removed 
> or used (and if used, it should only be allocated in the transactional use 
> case.)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (PHOENIX-4007) Surface last stats update in explain plan output

2017-08-04 Thread Samarth Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Samarth Jain reassigned PHOENIX-4007:
-

Assignee: Samarth Jain

> Surface last stats update in explain plan output
> 
>
> Key: PHOENIX-4007
> URL: https://issues.apache.org/jira/browse/PHOENIX-4007
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Samarth Jain
>Assignee: Samarth Jain
>
> As part of PHOENIX-3822, we surfaced byte and row estimates for queries in 
> explain plan. Since we collect this information through stats collection, it 
> would also be helpful to surface when this information was last updated to 
> reflect its freshness. We already store last_stats_update_time in 
> SYSTEM.STATS. So the task would be essentially surfacing 
> last_stats_update_time as another column in the explain plan result set.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PHOENIX-418) Support approximate COUNT DISTINCT

2017-08-04 Thread James Taylor (JIRA) 

[ 
https://issues.apache.org/jira/browse/PHOENIX-418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16114818#comment-16114818
 ] 

James Taylor commented on PHOENIX-418:
--

Thanks for the patch, [~aertoria]. This is a good new feature. Please revert 
the Phoenix grammar changes as they aren't needed since the new feature is 
surfaced as a new built-in aggregate function.

> Support approximate COUNT DISTINCT
> --
>
> Key: PHOENIX-418
> URL: https://issues.apache.org/jira/browse/PHOENIX-418
> Project: Phoenix
>  Issue Type: Task
>Reporter: James Taylor
>Assignee: Ethan Wang
>  Labels: gsoc2016
> Attachments: PHOENIX-418-v1.patch, PHOENIX-418-v2.patch
>
>
> Support an "approximation" of count distinct to prevent having to hold on to 
> all distinct values (since this will not scale well when the number of 
> distinct values is huge). The Apache Drill folks have had some interesting 
> discussions on this 
> [here](http://mail-archives.apache.org/mod_mbox/incubator-drill-dev/201306.mbox/%3CJIRA.12650169.1369931282407.88049.1370645900553%40arcas%3E).
>  They recommend using  [Welford's 
> method](http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance_Online_algorithm).
>  I'm open to having a config option that uses exact versus approximate. I 
> don't have experience implementing an approximate implementation, so I'm not 
> sure how much state is required to keep on the server and return to the 
> client (other than realizing it'd be much less that returning all distinct 
> values and their counts).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (PHOENIX-4066) Unused variable in MutationState.send()

2017-08-04 Thread Geoffrey Jacoby (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Geoffrey Jacoby reassigned PHOENIX-4066:


Assignee: Geoffrey Jacoby

> Unused variable in MutationState.send()
> ---
>
> Key: PHOENIX-4066
> URL: https://issues.apache.org/jira/browse/PHOENIX-4066
> Project: Phoenix
>  Issue Type: Improvement
>Affects Versions: 4.11.0
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>Priority: Trivial
>  Labels: beginner
> Fix For: 4.12.0
>
> Attachments: PHOENIX-4066.patch
>
>
> In MutationState#send, we always allocate the following:
> List txTableRefs = 
> Lists.newArrayListWithExpectedSize(mutations.size());
> and if a table's transactional, we populate it:
> txTableRefs.add(origTableRef);`
> but this is never referred to again in either logic or logging, and it's a 
> local variable so we can't be using it elsewhere either. It should be removed 
> or used (and if used, it should only be allocated in the transactional use 
> case.)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (PHOENIX-4066) Unused variable in MutationState.send()

2017-08-04 Thread Geoffrey Jacoby (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Geoffrey Jacoby updated PHOENIX-4066:
-
Attachment: PHOENIX-4066.patch

Removed the unused variable

> Unused variable in MutationState.send()
> ---
>
> Key: PHOENIX-4066
> URL: https://issues.apache.org/jira/browse/PHOENIX-4066
> Project: Phoenix
>  Issue Type: Improvement
>Affects Versions: 4.11.0
>Reporter: Geoffrey Jacoby
>Priority: Trivial
>  Labels: beginner
> Fix For: 4.12.0
>
> Attachments: PHOENIX-4066.patch
>
>
> In MutationState#send, we always allocate the following:
> List txTableRefs = 
> Lists.newArrayListWithExpectedSize(mutations.size());
> and if a table's transactional, we populate it:
> txTableRefs.add(origTableRef);`
> but this is never referred to again in either logic or logging, and it's a 
> local variable so we can't be using it elsewhere either. It should be removed 
> or used (and if used, it should only be allocated in the transactional use 
> case.)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (PHOENIX-4066) Unused variable in MutationState.send()

2017-08-04 Thread Geoffrey Jacoby (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Geoffrey Jacoby updated PHOENIX-4066:
-
Description: 
In MutationState#send, we always allocate the following:

List txTableRefs = 
Lists.newArrayListWithExpectedSize(mutations.size());

and if a table's transactional, we populate it:

txTableRefs.add(origTableRef);`

but this is never referred to again in either logic or logging, and it's a 
local variable so we can't be using it elsewhere either. It should be removed 
or used (and if used, it should only be allocated in the transactional use 
case.)

  was:
In MutationState#send, we always allocate the following:

```
List txTableRefs = 
Lists.newArrayListWithExpectedSize(mutations.size());
```
and if a table's transactional, we populate it:

```
txTableRefs.add(origTableRef);
```

but this is never referred to again in either logic or logging, and it's a 
local variable so we can't be using it elsewhere either. It should be removed 
or used (and if used, it should only be allocated in the transactional use 
case.)


> Unused variable in MutationState.send()
> ---
>
> Key: PHOENIX-4066
> URL: https://issues.apache.org/jira/browse/PHOENIX-4066
> Project: Phoenix
>  Issue Type: Improvement
>Affects Versions: 4.11.0
>Reporter: Geoffrey Jacoby
>Priority: Trivial
>  Labels: beginner
> Fix For: 4.12.0
>
>
> In MutationState#send, we always allocate the following:
> List txTableRefs = 
> Lists.newArrayListWithExpectedSize(mutations.size());
> and if a table's transactional, we populate it:
> txTableRefs.add(origTableRef);`
> but this is never referred to again in either logic or logging, and it's a 
> local variable so we can't be using it elsewhere either. It should be removed 
> or used (and if used, it should only be allocated in the transactional use 
> case.)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (PHOENIX-4066) Unused variable in MutationState.send()

2017-08-04 Thread Geoffrey Jacoby (JIRA) 
Geoffrey Jacoby created PHOENIX-4066:


 Summary: Unused variable in MutationState.send()
 Key: PHOENIX-4066
 URL: https://issues.apache.org/jira/browse/PHOENIX-4066
 Project: Phoenix
  Issue Type: Improvement
Affects Versions: 4.11.0
Reporter: Geoffrey Jacoby
Priority: Trivial
 Fix For: 4.12.0


In MutationState#send, we always allocate the following:

```
List txTableRefs = 
Lists.newArrayListWithExpectedSize(mutations.size());
```
and if a table's transactional, we populate it:

```
txTableRefs.add(origTableRef);
```

but this is never referred to again in either logic or logging, and it's a 
local variable so we can't be using it elsewhere either. It should be removed 
or used (and if used, it should only be allocated in the transactional use 
case.)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (PHOENIX-3525) Cap automatic index rebuilding to inactive timestamp.

2017-08-04 Thread James Taylor (JIRA) 

[ 
https://issues.apache.org/jira/browse/PHOENIX-3525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113128#comment-16113128
 ] 

James Taylor edited comment on PHOENIX-3525 at 8/4/17 5:02 PM:
---

Current plan is to eliminate simultaneous writes from the rebuilder and clients 
to prevent any race conditions by:
# introducing an INDEX_ACTIVATE_TIMESTAMP column that determines when 
incremental indexing will begin again. This timestamp will be set by the 
rebuilder to a time in the future (by a configurable delta) after all index 
regions are online.
# the INDEX_ACTIVATE_TIMESTAMP will be cleared when the 
INDEX_DISABLED_TIMESTAMP is set in the MetaDataEndPointImpl.setIndexState call. 
The rebuilder would then reset it according to the logic in (1), moving it out 
to a later time.
# the INDEX_ACTIVATE_TIMESTAMP will act as an upper bound on the rebuilder scan 
that replays mutations. Only after this timestamp plus some delta passes  (and 
the replaying is complete) will an index be marked as ACTIVE and the 
INDEX_ACTIVATE_TIMESTAMP and INDEX_DISABLED_TIMESTAMP be cleared.
# index maintenance will be prevented while server-based timestamp < 
INDEX_ACTIVATE_TIMESTAMP by having the clients not send the IndexMaintainer . 
The INDEX_ACTIVATE_TIMESTAMP will be included in PTable so that it makes its 
way to the clients.



was (Author: jamestaylor):
Current plan is to eliminate simultaneous writes from the rebuilder and clients 
to prevent any race conditions by:
* introducing a PENDING_ACTIVE index state. When in PENDING_ACTIVE state, an 
index will not be used by queries until the server-based timestamp >= 
INDEX_ACTIVATE_TIMESTAMP.
* introducing an INDEX_ACTIVATE_TIMESTAMP column that determines when an index 
will be reactivated. This timestamp will be set by the rebuilder to a time in 
the future (by a configurable amount of time) after all index regions are 
online. The index will be put either left in an ACTIVE state (depending on 
config) or moved to a PENDING_ACTIVE state.
* prevent index maintenance by not sending IndexMaintainer until server-based 
timestamp >= INDEX_ACTIVATE_TIMESTAMP.
* include INDEX_ACTIVATE_TIMESTAMP in PTable so that clients can use it to 
control whether index maintenance is performed.


> Cap automatic index rebuilding to inactive timestamp.
> -
>
> Key: PHOENIX-3525
> URL: https://issues.apache.org/jira/browse/PHOENIX-3525
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Ankit Singhal
>Assignee: James Taylor
> Attachments: PHOENIX-3525_wip2.patch, PHOENIX-3525_wip.patch
>
>
> From [~chrajeshbab...@gmail.com] review comment on 
> https://github.com/apache/phoenix/pull/210
> For automatic rebuilding ,DISABLED_TIMESTAMP is lower bound but there is no 
> upper bound so we are going rebuild all the new writes written after 
> DISABLED_TIMESTAMP even though indexes updated properly. So we can introduce 
> an upper bound of time where we are going to start a rebuild thread so we can 
> limit the data to rebuild. In case If there are frequent writes then we can 
> increment the rebuild period exponentially



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PHOENIX-418) Support approximate COUNT DISTINCT

2017-08-04 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/PHOENIX-418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16114526#comment-16114526
 ] 

Hadoop QA commented on PHOENIX-418:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12880344/PHOENIX-418-v2.patch
  against master branch at commit 0bd43f536a13609b35629308c415aebc800d6799.
  ATTACHMENT ID: 12880344

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:red}-1 javadoc{color}.  The javadoc tool appears to have generated 
54 warning messages.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 lineLengths{color}.  The patch introduces the following lines 
longer than 100:
+String tableName = initATableValues(null, tenantId, 
getDefaultSplits(tenantId), (Date)null, null, getUrl(), null);
+String tableName = initATableValues(null, tenantId, 
getDefaultSplits(tenantId), (Date)null, null, getUrl(), null);
+String query = "SELECT 
APPROX_COUNT_DISTINCT(ORGANIZATION_ID||A_STRING||B_STRING) FROM " + tableName;
+String tableName = initATableValues(null, tenantId, 
getDefaultSplits(tenantId), (Date)null, null, getUrl(), null);
+String query = "SELECT 
APPROX_COUNT_DISTINCT(a.ORGANIZATION_ID||a.A_STRING||b.ENTITY_ID) FROM " 
+   + tableName + " a, "+ tableName + " b where 
a.ORGANIZATION_ID=b.ORGANIZATION_ID and a.ENTITY_ID = b.ENTITY_ID";
+{ ParseContext context = contextStack.peek(); $ret = factory.select(u, 
order, l, approximate, o, getBindCount(), context.isAggregate()); }
+@BuiltInFunction(name=DistinctCountHyperLogLogAggregateFunction.NAME, 
nodeClass=DistinctCountHyperLogLogAggregateParseNode.class, args= {@Argument()} 
)
+public class DistinctCountHyperLogLogAggregateFunction extends 
DelegateConstantToCountAggregateFunction {
+public DistinctCountHyperLogLogAggregateFunction(List 
childExpressions, CountAggregateFunction delegate){

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
 
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.salted.SaltedTableVarLengthRowKeyIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.ConcurrentMutationsIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.NotQueryIT

Test results: 
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1243//testReport/
Javadoc warnings: 
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1243//artifact/patchprocess/patchJavadocWarnings.txt
Console output: 
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1243//console

This message is automatically generated.

> Support approximate COUNT DISTINCT
> --
>
> Key: PHOENIX-418
> URL: https://issues.apache.org/jira/browse/PHOENIX-418
> Project: Phoenix
>  Issue Type: Task
>Reporter: James Taylor
>Assignee: Ethan Wang
>  Labels: gsoc2016
> Attachments: PHOENIX-418-v1.patch, PHOENIX-418-v2.patch
>
>
> Support an "approximation" of count distinct to prevent having to hold on to 
> all distinct values (since this will not scale well when the number of 
> distinct values is huge). The Apache Drill folks have had some interesting 
> discussions on this 
> [here](http://mail-archives.apache.org/mod_mbox/incubator-drill-dev/201306.mbox/%3CJIRA.12650169.1369931282407.88049.1370645900553%40arcas%3E).
>  They recommend using  [Welford's 
> method](http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance_Online_algorithm).
>  I'm open to having a config option that uses exact versus approximate. I 
> don't have experience implementing an approximate implementation, so I'm not 
> sure how much state is required to keep on the server and return to the 
> client (other than realizing it'd be much less that returning all distinct 
> values and their counts).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Atomic Upsert salted table ERROR

2017-08-04 Thread James Taylor 
Please file a JIRA and include the Phoenix version.
Thanks,
James

On Fri, Aug 4, 2017 at 8:20 AM tian.yang  wrote:

> Hi,
>phoenix Atomic Upsert salted table ERROR
>
> 0: jdbc:phoenix:> drop table MY_TABLE;
> No rows affected (4.635 seconds)
> 0: jdbc:phoenix:> create table my_table(id varchar not null,id1 varchar
> not null, counter1 bigint, counter2 bigint CONSTRAINT pk PRIMARY KEY
> (id,id1));
> No rows affected (2.365 seconds)
> 0: jdbc:phoenix:> UPSERT INTO my_table(id,id1, counter1, counter2) VALUES
> ('abc','123', 0, 0) ON DUPLICATE KEY UPDATE counter1 = counter1 + 1,
> counter2 = counter2 + 1;
> 1 row affected (0.024 seconds)
> 0: jdbc:phoenix:> drop table MY_TABLE;
> No rows affected (4.62 seconds)
> 0: jdbc:phoenix:> create table my_table(id varchar not null,id1 varchar
> not null, counter1 bigint, counter2 bigint CONSTRAINT pk PRIMARY KEY
> (id,id1))SALT_BUCKETS = 6;
> No rows affected (4.378 seconds)
> 0: jdbc:phoenix:> UPSERT INTO my_table(id,id1, counter1, counter2) VALUES
> ('abc','123', 0, 0) ON DUPLICATE KEY UPDATE counter1 = counter1 + 1,
> counter2 = counter2 + 1;
> java.lang.NullPointerException: at index 3
> at
> com.google.common.collect.ObjectArrays.checkElementNotNull(ObjectArrays.java:191)
> at
> com.google.common.collect.ImmutableList.construct(ImmutableList.java:320)
> at
> com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:290)
> at org.apache.phoenix.schema.PTableImpl.init(PTableImpl.java:534)
> at org.apache.phoenix.schema.PTableImpl.(PTableImpl.java:408)
> at
> org.apache.phoenix.schema.PTableImpl.makePTable(PTableImpl.java:287)
> at
> org.apache.phoenix.compile.UpsertCompiler.compile(UpsertCompiler.java:979)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:611)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:597)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:351)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:341)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:339)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1511)
> at sqlline.Commands.execute(Commands.java:822)
> at sqlline.Commands.sql(Commands.java:732)
> at sqlline.SqlLine.dispatch(SqlLine.java:813)
> at sqlline.SqlLine.begin(SqlLine.java:686)
> at sqlline.SqlLine.start(SqlLine.java:398)
> at sqlline.SqlLine.main(SqlLine.java:291)
> 0: jdbc:phoenix:>
>

Atomic Upsert salted table ERROR

2017-08-04 Thread tian . yang 
Hi,
   phoenix Atomic Upsert salted table ERROR

0: jdbc:phoenix:> drop table MY_TABLE;
No rows affected (4.635 seconds)
0: jdbc:phoenix:> create table my_table(id varchar not null,id1 varchar not 
null, counter1 bigint, counter2 bigint CONSTRAINT pk PRIMARY KEY (id,id1));
No rows affected (2.365 seconds)
0: jdbc:phoenix:> UPSERT INTO my_table(id,id1, counter1, counter2) VALUES 
('abc','123', 0, 0) ON DUPLICATE KEY UPDATE counter1 = counter1 + 1, counter2 = 
counter2 + 1;
1 row affected (0.024 seconds)
0: jdbc:phoenix:> drop table MY_TABLE;
No rows affected (4.62 seconds)
0: jdbc:phoenix:> create table my_table(id varchar not null,id1 varchar not 
null, counter1 bigint, counter2 bigint CONSTRAINT pk PRIMARY KEY 
(id,id1))SALT_BUCKETS = 6;
No rows affected (4.378 seconds)
0: jdbc:phoenix:> UPSERT INTO my_table(id,id1, counter1, counter2) VALUES 
('abc','123', 0, 0) ON DUPLICATE KEY UPDATE counter1 = counter1 + 1, counter2 = 
counter2 + 1;
java.lang.NullPointerException: at index 3
at 
com.google.common.collect.ObjectArrays.checkElementNotNull(ObjectArrays.java:191)
at 
com.google.common.collect.ImmutableList.construct(ImmutableList.java:320)
at 
com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:290)
at org.apache.phoenix.schema.PTableImpl.init(PTableImpl.java:534)
at org.apache.phoenix.schema.PTableImpl.(PTableImpl.java:408)
at org.apache.phoenix.schema.PTableImpl.makePTable(PTableImpl.java:287)
at 
org.apache.phoenix.compile.UpsertCompiler.compile(UpsertCompiler.java:979)
at 
org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:611)
at 
org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:597)
at 
org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:351)
at 
org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:341)
at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
at 
org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:339)
at 
org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1511)
at sqlline.Commands.execute(Commands.java:822)
at sqlline.Commands.sql(Commands.java:732)
at sqlline.SqlLine.dispatch(SqlLine.java:813)
at sqlline.SqlLine.begin(SqlLine.java:686)
at sqlline.SqlLine.start(SqlLine.java:398)
at sqlline.SqlLine.main(SqlLine.java:291)
0: jdbc:phoenix:>

[jira] [Commented] (PHOENIX-3769) OnDuplicateKeyIT#testNewAndMultiDifferentUpdateOnSingleColumn fails on ppc64le

2017-08-04 Thread Josh Elser (JIRA) 

[ 
https://issues.apache.org/jira/browse/PHOENIX-3769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16114437#comment-16114437
 ] 

Josh Elser commented on PHOENIX-3769:
-

bq. There are two implementations of function compareTo. In case of x86, 
compareTo implemented by enum UnsafeComparer is executed whereas in case of 
ppc64le, compareTo implemented by enum PureJavaComparer is executed. Josh Elser 
looks like this is the root cause of this failure.

Ok, if that is broken, please spin out a JIRA issue for the HBASE project. 
{{Bytes}} is an HBase class.

bq. I found another workaround for this issue. I noticed that the respective 
lengths of the arrays are not passed to compareTo function. On making this 
change, the test passes on ppc64le as well as x86.

Good find, [~skanekar]! That looks like a copy-paste bug to me. Want to attach 
a new patch to this issue for this change only and see what the automated 
testing says?

> OnDuplicateKeyIT#testNewAndMultiDifferentUpdateOnSingleColumn fails on ppc64le
> --
>
> Key: PHOENIX-3769
> URL: https://issues.apache.org/jira/browse/PHOENIX-3769
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.11.0
> Environment: $ uname -a
> Linux 6945c232192e 3.16.0-30-generic #40~14.04.1-Ubuntu SMP Thu Jan 15 
> 17:42:36 UTC 2015 ppc64le ppc64le ppc64le GNU/Linux
> $ java -version
> openjdk version "1.8.0_111"
> OpenJDK Runtime Environment (build 1.8.0_111-8u111-b14-3~14.04.1-b14)
> OpenJDK 64-Bit Server VM (build 25.111-b14, mixed mode)
>Reporter: Sneha Kanekar
>  Labels: ppc64le
> Attachments: OnDuplicateKeyIT_Standard_output.txt, 
> PHOENIX-3769.patch, TEST-org.apache.phoenix.end2end.OnDuplicateKeyIT.xml
>
>
> The testcase 
> org.apache.phoenix.end2end.OnDuplicateKeyIT.testNewAndMultiDifferentUpdateOnSingleColumn
>  fails consistently on ppc64le architechture. The error message is as follows:
> {code: borderStyle=solid}
> java.lang.ArrayIndexOutOfBoundsException: 179
>   at 
> org.apache.phoenix.end2end.OnDuplicateKeyIT.testNewAndMultiDifferentUpdateOnSingleColumn(OnDuplicateKeyIT.java:392)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (PHOENIX-4065) NPE SELECT + WHERE + ORDER BY date

2017-08-04 Thread Konrad (JIRA) 
Konrad created PHOENIX-4065:
---

 Summary: NPE SELECT + WHERE + ORDER BY date
 Key: PHOENIX-4065
 URL: https://issues.apache.org/jira/browse/PHOENIX-4065
 Project: Phoenix
  Issue Type: Bug
Affects Versions: 4.11.0
 Environment: HBase 1.3.1, Phoenix 4.11.0
Reporter: Konrad
Priority: Critical


I found NPE while working with 4.11.0. Select statement was working for 4.9.0.

CREATE TABLE IF NOT EXISTS sample
(
 A UNSIGNED_LONG NOT NULL,
 B VARCHAR, 
 C DATE
 CONSTRAINT pk_A PRIMARY KEY (A)
);

upsert into sample (A, B, C) values (1, 'test', '2017-08-03');

select A FROM sample where B = 'test' ORDER BY C ;

org.apache.phoenix.exception.PhoenixIOException: 
org.apache.hadoop.hbase.DoNotRetryIOException: 
SAMPLE,,1501852502789.c348edcb5540cc47d91cb62c4721b689.: null
at 
org.apache.phoenix.util.ServerUtil.createIOException(ServerUtil.java:92)
at 
org.apache.phoenix.util.ServerUtil.throwIOException(ServerUtil.java:58)
at 
org.apache.phoenix.iterate.RegionScannerFactory$1.nextRaw(RegionScannerFactory.java:214)
at 
org.apache.phoenix.iterate.RegionScannerResultIterator.next(RegionScannerResultIterator.java:61)
at 
org.apache.phoenix.iterate.OrderedResultIterator.getResultIterator(OrderedResultIterator.java:255)
at 
org.apache.phoenix.iterate.OrderedResultIterator.next(OrderedResultIterator.java:199)
at 
org.apache.phoenix.iterate.NonAggregateRegionScannerFactory.getTopNScanner(NonAggregateRegionScannerFactory.java:319)
at 
org.apache.phoenix.iterate.NonAggregateRegionScannerFactory.getRegionScanner(NonAggregateRegionScannerFactory.java:161)
at 
org.apache.phoenix.coprocessor.ScanRegionObserver.doPostScannerOpen(ScanRegionObserver.java:72)
at 
org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.overrideDelegate(BaseScannerRegionObserver.java:221)
at 
org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.nextRaw(BaseScannerRegionObserver.java:266)
at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2633)
at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2837)
at 
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:34950)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
at 
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
at 
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
Caused by: java.lang.NullPointerException
at 
org.apache.phoenix.execute.TupleProjector.projectResults(TupleProjector.java:286)
at 
org.apache.phoenix.iterate.RegionScannerFactory$1.nextRaw(RegionScannerFactory.java:204)
... 15 more [SQL State=08000, DB Errorcode=101]

workaround:
select A, B FROM sample where B = 'test' ORDER BY C ;





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PHOENIX-3769) OnDuplicateKeyIT#testNewAndMultiDifferentUpdateOnSingleColumn fails on ppc64le

2017-08-04 Thread Sneha Kanekar (JIRA) 

[ 
https://issues.apache.org/jira/browse/PHOENIX-3769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16114210#comment-16114210
 ] 

Sneha Kanekar commented on PHOENIX-3769:


I found another workaround for this issue. I noticed that the respective 
lengths of the arrays are not passed to compareTo function. On making this 
change, the test passes on ppc64le as well as x86.
{code:borderStyle=solid}
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/index/PhoenixIndexBuilder.java 
b/phoenix-core/src/main/java/org/apache/phoenix/index/PhoenixIndexBuilder.java
index 5011245..acb139e 100644
--- 
a/phoenix-core/src/main/java/org/apache/phoenix/index/PhoenixIndexBuilder.java
+++ 
b/phoenix-core/src/main/java/org/apache/phoenix/index/PhoenixIndexBuilder.java
@@ -354,7 +354,7 @@ public class PhoenixIndexBuilder extends NonTxIndexBuilder {
 int repeating1 = oldInput.readShort();
 if (Bytes.compareTo(
 oldOnDupKeyBytes, ON_DUP_KEY_HEADER_BYTE_SIZE, 
oldOnDupKeyBytes.length - ON_DUP_KEY_HEADER_BYTE_SIZE,
-newOnDupKeyBytes, Bytes.SIZEOF_SHORT + 
Bytes.SIZEOF_BOOLEAN, oldOnDupKeyBytes.length - ON_DUP_KEY_HEADER_BYTE_SIZE) == 
0) {
+newOnDupKeyBytes, Bytes.SIZEOF_SHORT + 
Bytes.SIZEOF_BOOLEAN, newOnDupKeyBytes.length - ON_DUP_KEY_HEADER_BYTE_SIZE) == 
0) {
 // If both old and new ON DUPLICATE KEY UPDATE clauses match,
 // reduce the size of data we're sending over the wire.
 // TODO: optimization size of RPC more.
{code}

> OnDuplicateKeyIT#testNewAndMultiDifferentUpdateOnSingleColumn fails on ppc64le
> --
>
> Key: PHOENIX-3769
> URL: https://issues.apache.org/jira/browse/PHOENIX-3769
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.11.0
> Environment: $ uname -a
> Linux 6945c232192e 3.16.0-30-generic #40~14.04.1-Ubuntu SMP Thu Jan 15 
> 17:42:36 UTC 2015 ppc64le ppc64le ppc64le GNU/Linux
> $ java -version
> openjdk version "1.8.0_111"
> OpenJDK Runtime Environment (build 1.8.0_111-8u111-b14-3~14.04.1-b14)
> OpenJDK 64-Bit Server VM (build 25.111-b14, mixed mode)
>Reporter: Sneha Kanekar
>  Labels: ppc64le
> Attachments: OnDuplicateKeyIT_Standard_output.txt, 
> PHOENIX-3769.patch, TEST-org.apache.phoenix.end2end.OnDuplicateKeyIT.xml
>
>
> The testcase 
> org.apache.phoenix.end2end.OnDuplicateKeyIT.testNewAndMultiDifferentUpdateOnSingleColumn
>  fails consistently on ppc64le architechture. The error message is as follows:
> {code: borderStyle=solid}
> java.lang.ArrayIndexOutOfBoundsException: 179
>   at 
> org.apache.phoenix.end2end.OnDuplicateKeyIT.testNewAndMultiDifferentUpdateOnSingleColumn(OnDuplicateKeyIT.java:392)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)