[jira] [Created] (PHOENIX-3981) Ensure iterators are closed for join plans

[Samarth Jain (JIRA)]

Samarth Jain created PHOENIX-3981:
-

 Summary: Ensure iterators are closed for join plans
 Key: PHOENIX-3981
 URL: https://issues.apache.org/jira/browse/PHOENIX-3981
 Project: Phoenix
  Issue Type: Bug
Reporter: Samarth Jain
Assignee: Samarth Jain






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PHOENIX-3980) Wrap iterator returned by BaseQueryPlan so that it always closes dependencies

[Samarth Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-3980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065938#comment-16065938
 ] 

Samarth Jain commented on PHOENIX-3980:
---

Looks good, [~tdsilva]. +1. I think we should commit this to the 4.11 branches 
too.

> Wrap iterator returned by BaseQueryPlan so that it always closes dependencies
> -
>
> Key: PHOENIX-3980
> URL: https://issues.apache.org/jira/browse/PHOENIX-3980
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Thomas D'Silva
>Assignee: Thomas D'Silva
> Fix For: 4.12.0
>
> Attachments: PHOENIX-3980.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PHOENIX-3980) Wrap iterator returned by BaseQueryPlan so that it always closes dependencies

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-3980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065930#comment-16065930
 ] 

Hadoop QA commented on PHOENIX-3980:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12874798/PHOENIX-3980.patch
  against master branch at commit 3c68a87ef80d1443d2abfaa8e485a00f9780adbc.
  ATTACHMENT ID: 12874798

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:red}-1 javadoc{color}.  The javadoc tool appears to have generated 
50 warning messages.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
 
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.MutableQueryIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.CreateTableIT

Test results: 
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1138//testReport/
Javadoc warnings: 
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1138//artifact/patchprocess/patchJavadocWarnings.txt
Console output: 
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1138//console

This message is automatically generated.

> Wrap iterator returned by BaseQueryPlan so that it always closes dependencies
> -
>
> Key: PHOENIX-3980
> URL: https://issues.apache.org/jira/browse/PHOENIX-3980
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Thomas D'Silva
>Assignee: Thomas D'Silva
> Fix For: 4.12.0
>
> Attachments: PHOENIX-3980.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (PHOENIX-3980) Wrap iterator returned by BaseQueryPlan so that it always closes dependencies

[Thomas D'Silva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-3980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas D'Silva updated PHOENIX-3980:

Attachment: PHOENIX-3980.patch

[~samarthjain]

Can you please review?

> Wrap iterator returned by BaseQueryPlan so that it always closes dependencies
> -
>
> Key: PHOENIX-3980
> URL: https://issues.apache.org/jira/browse/PHOENIX-3980
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Thomas D'Silva
>Assignee: Thomas D'Silva
> Fix For: 4.12.0
>
> Attachments: PHOENIX-3980.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (PHOENIX-3971) Disable assertion in ConnectionQueryServicesTestImpl to get builds passing

[Thomas D'Silva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas D'Silva resolved PHOENIX-3971.
-
Resolution: Fixed

> Disable assertion in ConnectionQueryServicesTestImpl to get builds passing
> --
>
> Key: PHOENIX-3971
> URL: https://issues.apache.org/jira/browse/PHOENIX-3971
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Samarth Jain
>Assignee: Samarth Jain
> Attachments: PHOENIX-3971.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (PHOENIX-3980) Wrap iterator returned by BaseQueryPlan so that it always closes dependencies

[Thomas D'Silva (JIRA)]

Thomas D'Silva created PHOENIX-3980:
---

 Summary: Wrap iterator returned by BaseQueryPlan so that it always 
closes dependencies
 Key: PHOENIX-3980
 URL: https://issues.apache.org/jira/browse/PHOENIX-3980
 Project: Phoenix
  Issue Type: Bug
Reporter: Thomas D'Silva
Assignee: Thomas D'Silva
 Fix For: 4.12.0






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PHOENIX-3633) null pointer exception when subsquery for not exists returns empty result set

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-3633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065849#comment-16065849
 ] 

Jerry He commented on PHOENIX-3633:
---

I will be glad to add a test case.  Could you point to me where the best place 
to add this UT is, with similar test cases?

> null pointer exception when subsquery for not exists returns empty result set
> -
>
> Key: PHOENIX-3633
> URL: https://issues.apache.org/jira/browse/PHOENIX-3633
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.7.0
>Reporter: N Campbell
>Assignee: Jerry He
> Attachments: PHOENIX-3633.patch
>
>
> phoenix-4.7.0.2.5.3.0-37-client
> select RNUM, C1, C2 from TJOIN2 where not exists ( select C1 from TJOIN1 
> where C2=500)
> Error while executing SQL "select RNUM, C1, C2 from TJOIN2 where not exists ( 
> select C1 from TJOIN1 where C2=500)": Remote driver error: 
> NullPointerException: (null exception message)
> create table  if not exists TJOIN1 (RNUM integer  not null primary key , C1 
> integer, C2 integer);
> upsert into TJOIN1 (RNUM, C1, C2) values ( 0, 10, 15);
> upsert into TJOIN1 (RNUM, C1, C2) values ( 1, 20, 25);
> upsert into TJOIN1 (RNUM, C1, C2) values ( 2, NULL, 50);
> create table  if not exists TJOIN2 (RNUM integer  not null primary key , C1 
> integer, C2 char(2));
> upsert into TJOIN2 (RNUM, C1, C2) values ( 0, 10, 'BB');
> upsert into TJOIN2 (RNUM, C1, C2) values ( 1, 15, 'DD');
> upsert into TJOIN2 (RNUM, C1, C2) values ( 2, NULL, 'EE');
> upsert into TJOIN2 (RNUM, C1, C2) values ( 3, 10, 'FF');



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (PHOENIX-3732) Support for dynamic columns in UPSERT in Phoenix-Calcite

[Rajeshbabu Chintaguntla (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajeshbabu Chintaguntla reassigned PHOENIX-3732:


Assignee: Rajeshbabu Chintaguntla  (was: Kevin Liew)

> Support for dynamic columns in UPSERT in Phoenix-Calcite
> 
>
> Key: PHOENIX-3732
> URL: https://issues.apache.org/jira/browse/PHOENIX-3732
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Kevin Liew
>Assignee: Rajeshbabu Chintaguntla
>  Labels: calcite
> Attachments: PHOENIX-3732.patch
>
>
> https://phoenix.apache.org/dynamic_columns.html
> {quote}
> To upsert a row with dynamic columns:
> UPSERT INTO EventLog (eventId, eventTime, eventType, lastGCTime TIME, 
> usedMemory BIGINT, maxMemory BIGINT) VALUES(1, CURRENT_TIME(), ‘abc’, 
> CURRENT_TIME(), 512, 1024);
> {quote}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PHOENIX-3977) Region is not closed when moving region or balancing while writing to table with local index

[JeongMin Ju (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-3977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065726#comment-16065726
 ] 

JeongMin Ju commented on PHOENIX-3977:
--

Hi.
This problem is related only to the local index.
The global index problem can be solved by increasing 
hbase.regionserver.executor.openregion.threads, but this is not the case.
You can see this phenomenon by writing to a table with a local index and moving 
any region.
The problem with the global index is that the region is pending open and now 
the region is in the pending close state.

!screenshot-1.png!

> Region is not closed when moving region or balancing while writing to table 
> with local index
> 
>
> Key: PHOENIX-3977
> URL: https://issues.apache.org/jira/browse/PHOENIX-3977
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.10.0
>Reporter: JeongMin Ju
> Attachments: screenshot-1.png
>
>
> Region is not closed when moving region or balancing while writing to table 
> with local index.
> If the regionserver is forcibly killed and restart and then balancing is 
> performed during write operation perform using YCSB. The region is not moved 
> properly, so it becomes jammed.
> This is also true when moving a specific region.
> {panel:title=RegionServer1}
> 2017-06-26 17:18:49,096 INFO 
> org.apache.phoenix.hbase.index.util.IndexManagementUtil: Rethrowing 
> org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008 
> (INT10): Unable to find cached index metadata.  key=-4998016164816556219 
> region=PHOENIX_TABLE,&,1498464285648.df3f3da32306361ce5e38e5193f5e5d6.host=juke-cdh-36f531b4.s2.krane.9rum.cc,60020,1498464965204
>  Index update failed
> 2017-06-26 17:18:49,531 INFO 
> org.apache.phoenix.hbase.index.util.IndexManagementUtil: Rethrowing 
> org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008 
> (INT10): Unable to find cached index metadata.  key=6036765365681157624 
> region=PHOENIX_TABLE,&,1498464285648.df3f3da32306361ce5e38e5193f5e5d6.host=juke-cdh-36f531b4.s2.krane.9rum.cc,60020,1498464965204
>  Index update failed
> 2017-06-26 17:18:49,536 INFO 
> org.apache.phoenix.hbase.index.util.IndexManagementUtil: Rethrowing 
> org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008 
> (INT10): Unable to find cached index metadata.  key=-428088766346522675 
> region=PHOENIX_TABLE,&,1498464285648.df3f3da32306361ce5e38e5193f5e5d6.host=juke-cdh-36f531b4.s2.krane.9rum.cc,60020,1498464965204
>  Index update failed
> {panel}
> At other regionserver
> {panel:title=RegionServer2}
> 2017-06-26 17:19:49,755 INFO org.apache.hadoop.hbase.client.AsyncProcess: 
> #1476, waiting for 62  actions to finish on table: PHOENIX_TABLE
> 2017-06-26 17:19:49,755 INFO org.apache.hadoop.hbase.client.AsyncProcess: 
> #1476, waiting for 27  actions to finish on table: PHOENIX_TABLE
> 2017-06-26 17:19:49,755 INFO org.apache.hadoop.hbase.client.AsyncProcess: 
> #1476, waiting for 52  actions to finish on table: PHOENIX_TABLE
> {panel}
> phoenix.coprocessor.maxServerCacheTimeToLiveMs & 
> phoenix.coprocessor.maxMetaDataCacheSize was not effected.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (PHOENIX-3977) Region is not closed when moving region or balancing while writing to table with local index

[JeongMin Ju (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-3977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

JeongMin Ju updated PHOENIX-3977:
-
Attachment: screenshot-1.png

> Region is not closed when moving region or balancing while writing to table 
> with local index
> 
>
> Key: PHOENIX-3977
> URL: https://issues.apache.org/jira/browse/PHOENIX-3977
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.10.0
>Reporter: JeongMin Ju
> Attachments: screenshot-1.png
>
>
> Region is not closed when moving region or balancing while writing to table 
> with local index.
> If the regionserver is forcibly killed and restart and then balancing is 
> performed during write operation perform using YCSB. The region is not moved 
> properly, so it becomes jammed.
> This is also true when moving a specific region.
> {panel:title=RegionServer1}
> 2017-06-26 17:18:49,096 INFO 
> org.apache.phoenix.hbase.index.util.IndexManagementUtil: Rethrowing 
> org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008 
> (INT10): Unable to find cached index metadata.  key=-4998016164816556219 
> region=PHOENIX_TABLE,&,1498464285648.df3f3da32306361ce5e38e5193f5e5d6.host=juke-cdh-36f531b4.s2.krane.9rum.cc,60020,1498464965204
>  Index update failed
> 2017-06-26 17:18:49,531 INFO 
> org.apache.phoenix.hbase.index.util.IndexManagementUtil: Rethrowing 
> org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008 
> (INT10): Unable to find cached index metadata.  key=6036765365681157624 
> region=PHOENIX_TABLE,&,1498464285648.df3f3da32306361ce5e38e5193f5e5d6.host=juke-cdh-36f531b4.s2.krane.9rum.cc,60020,1498464965204
>  Index update failed
> 2017-06-26 17:18:49,536 INFO 
> org.apache.phoenix.hbase.index.util.IndexManagementUtil: Rethrowing 
> org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008 
> (INT10): Unable to find cached index metadata.  key=-428088766346522675 
> region=PHOENIX_TABLE,&,1498464285648.df3f3da32306361ce5e38e5193f5e5d6.host=juke-cdh-36f531b4.s2.krane.9rum.cc,60020,1498464965204
>  Index update failed
> {panel}
> At other regionserver
> {panel:title=RegionServer2}
> 2017-06-26 17:19:49,755 INFO org.apache.hadoop.hbase.client.AsyncProcess: 
> #1476, waiting for 62  actions to finish on table: PHOENIX_TABLE
> 2017-06-26 17:19:49,755 INFO org.apache.hadoop.hbase.client.AsyncProcess: 
> #1476, waiting for 27  actions to finish on table: PHOENIX_TABLE
> 2017-06-26 17:19:49,755 INFO org.apache.hadoop.hbase.client.AsyncProcess: 
> #1476, waiting for 52  actions to finish on table: PHOENIX_TABLE
> {panel}
> phoenix.coprocessor.maxServerCacheTimeToLiveMs & 
> phoenix.coprocessor.maxMetaDataCacheSize was not effected.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (PHOENIX-3979) Pherf - Support Timestamp datatype

[Mujtaba Chohan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-3979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mujtaba Chohan updated PHOENIX-3979:

Attachment: PHOENIX-3937.patch

> Pherf - Support Timestamp datatype
> --
>
> Key: PHOENIX-3979
> URL: https://issues.apache.org/jira/browse/PHOENIX-3979
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Mujtaba Chohan
>Assignee: Mujtaba Chohan
>Priority: Minor
> Attachments: PHOENIX-3937.patch
>
>
> Support TIMESTAMP datatype as supported types for data generation.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (PHOENIX-3979) Pherf - Support Timestamp datatype

[Mujtaba Chohan (JIRA)]

Mujtaba Chohan created PHOENIX-3979:
---

 Summary: Pherf - Support Timestamp datatype
 Key: PHOENIX-3979
 URL: https://issues.apache.org/jira/browse/PHOENIX-3979
 Project: Phoenix
  Issue Type: Improvement
Reporter: Mujtaba Chohan
Priority: Minor


Support TIMESTAMP datatype as supported types for data generation.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (PHOENIX-3979) Pherf - Support Timestamp datatype

[Mujtaba Chohan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-3979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mujtaba Chohan reassigned PHOENIX-3979:
---

Assignee: Mujtaba Chohan

> Pherf - Support Timestamp datatype
> --
>
> Key: PHOENIX-3979
> URL: https://issues.apache.org/jira/browse/PHOENIX-3979
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Mujtaba Chohan
>Assignee: Mujtaba Chohan
>Priority: Minor
>
> Support TIMESTAMP datatype as supported types for data generation.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PHOENIX-3598) Enable proxy access to Phoenix query server for third party on behalf of end users

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065615#comment-16065615
 ] 

ASF GitHub Bot commented on PHOENIX-3598:
-

Github user joshelser commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/265#discussion_r124414777
  
--- Diff: 
phoenix-queryserver/src/main/java/org/apache/phoenix/queryserver/server/QueryServer.java
 ---
@@ -273,6 +282,54 @@ public int run(String[] args) throws Exception {
 }
   }
 
+  // add remoteUserExtractor to builder if enabled
+  @VisibleForTesting
+  public void setRemoteUserExtractorIfNecessary(HttpServer.Builder 
builder, Configuration conf) {
+if 
(conf.getBoolean(QueryServices.QUERY_SERVER_WITH_REMOTEUSEREXTRACTOR_ATTRIB,
+
QueryServicesOptions.DEFAULT_QUERY_SERVER_WITH_REMOTEUSEREXTRACTOR)) {
+  builder.withRemoteUserExtractor(new 
PhoenixRemoteUserExtractor(conf));
+}
+  }
+
+  /**
+   * Use the correctly way to extract end user.
+   */
+
+  static class PhoenixRemoteUserExtractor implements RemoteUserExtractor{
+private final HttpQueryStringParameterRemoteUserExtractor 
paramRemoteUserExtractor;
+private final HttpRequestRemoteUserExtractor 
requestRemoteUserExtractor;
+private final String userExtractParam;
+
+public PhoenixRemoteUserExtractor(Configuration conf) {
+  this.requestRemoteUserExtractor = new 
HttpRequestRemoteUserExtractor();
+  this.userExtractParam = 
conf.get(QueryServices.QUERY_SERVER_REMOTEUSEREXTRACTOR_PARAM,
+  
QueryServicesOptions.DEFAULT_QUERY_SERVER_REMOTEUSEREXTRACTOR_PARAM);
+  this.paramRemoteUserExtractor = new 
HttpQueryStringParameterRemoteUserExtractor(userExtractParam);
+}
+
+@Override
+public String extract(HttpServletRequest request) throws 
RemoteUserExtractionException {
+  if (request.getParameter(userExtractParam) != null) {
+String extractedUser = paramRemoteUserExtractor.extract(request);
+UserGroupInformation ugi = 
UserGroupInformation.createRemoteUser(request.getRemoteUser());
+UserGroupInformation proxyUser = 
UserGroupInformation.createProxyUser(extractedUser, ugi);
--- End diff --

Agreed! I think the work you've put in would be nice to support for the 
non-Kerberos case, but let's not hold up this change for that.

I will try to write up a test case for PQS (mini-hbase, mini-kdc, and PQS) 
to validate your changes here before I commit.


> Enable proxy access to Phoenix query server for third party on behalf of end 
> users
> --
>
> Key: PHOENIX-3598
> URL: https://issues.apache.org/jira/browse/PHOENIX-3598
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Shi Wang
> Attachments: 0001-PHOENIX-3598.patch
>
>
> This JIRA tracks the follow-on work of CALCITE-1539 needed on Phoenix query 
> server side.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PHOENIX-3598) Enable proxy access to Phoenix query server for third party on behalf of end users

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065599#comment-16065599
 ] 

ASF GitHub Bot commented on PHOENIX-3598:
-

Github user Wancy commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/265#discussion_r124413815
  
--- Diff: 
phoenix-queryserver/src/main/java/org/apache/phoenix/queryserver/server/QueryServer.java
 ---
@@ -273,6 +282,54 @@ public int run(String[] args) throws Exception {
 }
   }
 
+  // add remoteUserExtractor to builder if enabled
+  @VisibleForTesting
+  public void setRemoteUserExtractorIfNecessary(HttpServer.Builder 
builder, Configuration conf) {
+if 
(conf.getBoolean(QueryServices.QUERY_SERVER_WITH_REMOTEUSEREXTRACTOR_ATTRIB,
+
QueryServicesOptions.DEFAULT_QUERY_SERVER_WITH_REMOTEUSEREXTRACTOR)) {
+  builder.withRemoteUserExtractor(new 
PhoenixRemoteUserExtractor(conf));
+}
+  }
+
+  /**
+   * Use the correctly way to extract end user.
+   */
+
+  static class PhoenixRemoteUserExtractor implements RemoteUserExtractor{
+private final HttpQueryStringParameterRemoteUserExtractor 
paramRemoteUserExtractor;
+private final HttpRequestRemoteUserExtractor 
requestRemoteUserExtractor;
+private final String userExtractParam;
+
+public PhoenixRemoteUserExtractor(Configuration conf) {
+  this.requestRemoteUserExtractor = new 
HttpRequestRemoteUserExtractor();
+  this.userExtractParam = 
conf.get(QueryServices.QUERY_SERVER_REMOTEUSEREXTRACTOR_PARAM,
+  
QueryServicesOptions.DEFAULT_QUERY_SERVER_REMOTEUSEREXTRACTOR_PARAM);
+  this.paramRemoteUserExtractor = new 
HttpQueryStringParameterRemoteUserExtractor(userExtractParam);
+}
+
+@Override
+public String extract(HttpServletRequest request) throws 
RemoteUserExtractionException {
+  if (request.getParameter(userExtractParam) != null) {
+String extractedUser = paramRemoteUserExtractor.extract(request);
+UserGroupInformation ugi = 
UserGroupInformation.createRemoteUser(request.getRemoteUser());
+UserGroupInformation proxyUser = 
UserGroupInformation.createProxyUser(extractedUser, ugi);
--- End diff --

Hi @joshelser,
I think I understand your concern of the edge cases. I originally wanna add 
it just for kerberos cases, but I thought user extract could be extended to 
simple auth as well in the future, but seems a lot more work needs to be done 
than I thought. Also like you said most people just want point1.

I think for this jira just add it for the kerberos case is more practical.


> Enable proxy access to Phoenix query server for third party on behalf of end 
> users
> --
>
> Key: PHOENIX-3598
> URL: https://issues.apache.org/jira/browse/PHOENIX-3598
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Shi Wang
> Attachments: 0001-PHOENIX-3598.patch
>
>
> This JIRA tracks the follow-on work of CALCITE-1539 needed on Phoenix query 
> server side.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PHOENIX-3598) Enable proxy access to Phoenix query server for third party on behalf of end users

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065602#comment-16065602
 ] 

ASF GitHub Bot commented on PHOENIX-3598:
-

Github user Wancy commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/265#discussion_r124413926
  
--- Diff: 
phoenix-queryserver/src/main/java/org/apache/phoenix/queryserver/server/QueryServer.java
 ---
@@ -228,7 +235,9 @@ public int run(String[] args) throws Exception {
 builder.withSpnego(ugi.getUserName(), additionalAllowedRealms)
 .withAutomaticLogin(keytab)
 .withImpersonation(new PhoenixDoAsCallback(ugi, getConf()));
+
   }
+  setRemoteUserExtractorIfNecessary(builder, getConf());
--- End diff --

Agree to put it inside the if-block for only kerberos case :)


> Enable proxy access to Phoenix query server for third party on behalf of end 
> users
> --
>
> Key: PHOENIX-3598
> URL: https://issues.apache.org/jira/browse/PHOENIX-3598
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Shi Wang
> Attachments: 0001-PHOENIX-3598.patch
>
>
> This JIRA tracks the follow-on work of CALCITE-1539 needed on Phoenix query 
> server side.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] phoenix pull request #265: PHOENIX-3598

[Wancy]

Github user Wancy commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/265#discussion_r124413926
  
--- Diff: 
phoenix-queryserver/src/main/java/org/apache/phoenix/queryserver/server/QueryServer.java
 ---
@@ -228,7 +235,9 @@ public int run(String[] args) throws Exception {
 builder.withSpnego(ugi.getUserName(), additionalAllowedRealms)
 .withAutomaticLogin(keytab)
 .withImpersonation(new PhoenixDoAsCallback(ugi, getConf()));
+
   }
+  setRemoteUserExtractorIfNecessary(builder, getConf());
--- End diff --

Agree to put it inside the if-block for only kerberos case :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] phoenix pull request #265: PHOENIX-3598

[Wancy]

Github user Wancy commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/265#discussion_r124413815
  
--- Diff: 
phoenix-queryserver/src/main/java/org/apache/phoenix/queryserver/server/QueryServer.java
 ---
@@ -273,6 +282,54 @@ public int run(String[] args) throws Exception {
 }
   }
 
+  // add remoteUserExtractor to builder if enabled
+  @VisibleForTesting
+  public void setRemoteUserExtractorIfNecessary(HttpServer.Builder 
builder, Configuration conf) {
+if 
(conf.getBoolean(QueryServices.QUERY_SERVER_WITH_REMOTEUSEREXTRACTOR_ATTRIB,
+
QueryServicesOptions.DEFAULT_QUERY_SERVER_WITH_REMOTEUSEREXTRACTOR)) {
+  builder.withRemoteUserExtractor(new 
PhoenixRemoteUserExtractor(conf));
+}
+  }
+
+  /**
+   * Use the correctly way to extract end user.
+   */
+
+  static class PhoenixRemoteUserExtractor implements RemoteUserExtractor{
+private final HttpQueryStringParameterRemoteUserExtractor 
paramRemoteUserExtractor;
+private final HttpRequestRemoteUserExtractor 
requestRemoteUserExtractor;
+private final String userExtractParam;
+
+public PhoenixRemoteUserExtractor(Configuration conf) {
+  this.requestRemoteUserExtractor = new 
HttpRequestRemoteUserExtractor();
+  this.userExtractParam = 
conf.get(QueryServices.QUERY_SERVER_REMOTEUSEREXTRACTOR_PARAM,
+  
QueryServicesOptions.DEFAULT_QUERY_SERVER_REMOTEUSEREXTRACTOR_PARAM);
+  this.paramRemoteUserExtractor = new 
HttpQueryStringParameterRemoteUserExtractor(userExtractParam);
+}
+
+@Override
+public String extract(HttpServletRequest request) throws 
RemoteUserExtractionException {
+  if (request.getParameter(userExtractParam) != null) {
+String extractedUser = paramRemoteUserExtractor.extract(request);
+UserGroupInformation ugi = 
UserGroupInformation.createRemoteUser(request.getRemoteUser());
+UserGroupInformation proxyUser = 
UserGroupInformation.createProxyUser(extractedUser, ugi);
--- End diff --

Hi @joshelser,
I think I understand your concern of the edge cases. I originally wanna add 
it just for kerberos cases, but I thought user extract could be extended to 
simple auth as well in the future, but seems a lot more work needs to be done 
than I thought. Also like you said most people just want point1.

I think for this jira just add it for the kerberos case is more practical.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (PHOENIX-3977) Region is not closed when moving region or balancing while writing to table with local index

[Sergey Soldatov (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-3977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065594#comment-16065594
 ] 

Sergey Soldatov commented on PHOENIX-3977:
--

[~mini666] Could you give some more details about the configuration you are 
using. Does table has only local index or global index as well? Such behavior 
usually happen when there is a global index on the scene and this problem is 
discussed in PHOENIX-3970 and related issues. It would be nice if you provide  
several jstacks for the stuck region server.

> Region is not closed when moving region or balancing while writing to table 
> with local index
> 
>
> Key: PHOENIX-3977
> URL: https://issues.apache.org/jira/browse/PHOENIX-3977
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.10.0
>Reporter: JeongMin Ju
>
> Region is not closed when moving region or balancing while writing to table 
> with local index.
> If the regionserver is forcibly killed and restart and then balancing is 
> performed during write operation perform using YCSB. The region is not moved 
> properly, so it becomes jammed.
> This is also true when moving a specific region.
> {panel:title=RegionServer1}
> 2017-06-26 17:18:49,096 INFO 
> org.apache.phoenix.hbase.index.util.IndexManagementUtil: Rethrowing 
> org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008 
> (INT10): Unable to find cached index metadata.  key=-4998016164816556219 
> region=PHOENIX_TABLE,&,1498464285648.df3f3da32306361ce5e38e5193f5e5d6.host=juke-cdh-36f531b4.s2.krane.9rum.cc,60020,1498464965204
>  Index update failed
> 2017-06-26 17:18:49,531 INFO 
> org.apache.phoenix.hbase.index.util.IndexManagementUtil: Rethrowing 
> org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008 
> (INT10): Unable to find cached index metadata.  key=6036765365681157624 
> region=PHOENIX_TABLE,&,1498464285648.df3f3da32306361ce5e38e5193f5e5d6.host=juke-cdh-36f531b4.s2.krane.9rum.cc,60020,1498464965204
>  Index update failed
> 2017-06-26 17:18:49,536 INFO 
> org.apache.phoenix.hbase.index.util.IndexManagementUtil: Rethrowing 
> org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008 
> (INT10): Unable to find cached index metadata.  key=-428088766346522675 
> region=PHOENIX_TABLE,&,1498464285648.df3f3da32306361ce5e38e5193f5e5d6.host=juke-cdh-36f531b4.s2.krane.9rum.cc,60020,1498464965204
>  Index update failed
> {panel}
> At other regionserver
> {panel:title=RegionServer2}
> 2017-06-26 17:19:49,755 INFO org.apache.hadoop.hbase.client.AsyncProcess: 
> #1476, waiting for 62  actions to finish on table: PHOENIX_TABLE
> 2017-06-26 17:19:49,755 INFO org.apache.hadoop.hbase.client.AsyncProcess: 
> #1476, waiting for 27  actions to finish on table: PHOENIX_TABLE
> 2017-06-26 17:19:49,755 INFO org.apache.hadoop.hbase.client.AsyncProcess: 
> #1476, waiting for 52  actions to finish on table: PHOENIX_TABLE
> {panel}
> phoenix.coprocessor.maxServerCacheTimeToLiveMs & 
> phoenix.coprocessor.maxMetaDataCacheSize was not effected.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PHOENIX-3598) Enable proxy access to Phoenix query server for third party on behalf of end users

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065567#comment-16065567
 ] 

ASF GitHub Bot commented on PHOENIX-3598:
-

Github user joshelser commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/265#discussion_r124402049
  
--- Diff: 
phoenix-queryserver/src/main/java/org/apache/phoenix/queryserver/server/QueryServer.java
 ---
@@ -273,6 +282,54 @@ public int run(String[] args) throws Exception {
 }
   }
 
+  // add remoteUserExtractor to builder if enabled
+  @VisibleForTesting
+  public void setRemoteUserExtractorIfNecessary(HttpServer.Builder 
builder, Configuration conf) {
+if 
(conf.getBoolean(QueryServices.QUERY_SERVER_WITH_REMOTEUSEREXTRACTOR_ATTRIB,
+
QueryServicesOptions.DEFAULT_QUERY_SERVER_WITH_REMOTEUSEREXTRACTOR)) {
+  builder.withRemoteUserExtractor(new 
PhoenixRemoteUserExtractor(conf));
+}
+  }
+
+  /**
+   * Use the correctly way to extract end user.
+   */
+
+  static class PhoenixRemoteUserExtractor implements RemoteUserExtractor{
+private final HttpQueryStringParameterRemoteUserExtractor 
paramRemoteUserExtractor;
+private final HttpRequestRemoteUserExtractor 
requestRemoteUserExtractor;
+private final String userExtractParam;
+
+public PhoenixRemoteUserExtractor(Configuration conf) {
+  this.requestRemoteUserExtractor = new 
HttpRequestRemoteUserExtractor();
+  this.userExtractParam = 
conf.get(QueryServices.QUERY_SERVER_REMOTEUSEREXTRACTOR_PARAM,
+  
QueryServicesOptions.DEFAULT_QUERY_SERVER_REMOTEUSEREXTRACTOR_PARAM);
+  this.paramRemoteUserExtractor = new 
HttpQueryStringParameterRemoteUserExtractor(userExtractParam);
+}
+
+@Override
+public String extract(HttpServletRequest request) throws 
RemoteUserExtractionException {
+  if (request.getParameter(userExtractParam) != null) {
--- End diff --

We should put a `requestRemoteUserExtractor.extract(request)` at the top of 
this method implementation. We should be using it in both branches of the 
conditional (replacing the `request.getRemoteUser()` call you have below)


> Enable proxy access to Phoenix query server for third party on behalf of end 
> users
> --
>
> Key: PHOENIX-3598
> URL: https://issues.apache.org/jira/browse/PHOENIX-3598
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Shi Wang
> Attachments: 0001-PHOENIX-3598.patch
>
>
> This JIRA tracks the follow-on work of CALCITE-1539 needed on Phoenix query 
> server side.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PHOENIX-3598) Enable proxy access to Phoenix query server for third party on behalf of end users

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065566#comment-16065566
 ] 

ASF GitHub Bot commented on PHOENIX-3598:
-

Github user joshelser commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/265#discussion_r124409282
  
--- Diff: 
phoenix-queryserver/src/main/java/org/apache/phoenix/queryserver/server/QueryServer.java
 ---
@@ -228,7 +235,9 @@ public int run(String[] args) throws Exception {
 builder.withSpnego(ugi.getUserName(), additionalAllowedRealms)
 .withAutomaticLogin(keytab)
 .withImpersonation(new PhoenixDoAsCallback(ugi, getConf()));
+
   }
+  setRemoteUserExtractorIfNecessary(builder, getConf());
--- End diff --

With respect to my long-winded comment below, if you're only looking to 
support Kerberos, we want to move this line into the above if-block.


> Enable proxy access to Phoenix query server for third party on behalf of end 
> users
> --
>
> Key: PHOENIX-3598
> URL: https://issues.apache.org/jira/browse/PHOENIX-3598
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Shi Wang
> Attachments: 0001-PHOENIX-3598.patch
>
>
> This JIRA tracks the follow-on work of CALCITE-1539 needed on Phoenix query 
> server side.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PHOENIX-3598) Enable proxy access to Phoenix query server for third party on behalf of end users

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065565#comment-16065565
 ] 

ASF GitHub Bot commented on PHOENIX-3598:
-

Github user joshelser commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/265#discussion_r124409157
  
--- Diff: 
phoenix-queryserver/src/main/java/org/apache/phoenix/queryserver/server/QueryServer.java
 ---
@@ -273,6 +282,54 @@ public int run(String[] args) throws Exception {
 }
   }
 
+  // add remoteUserExtractor to builder if enabled
+  @VisibleForTesting
+  public void setRemoteUserExtractorIfNecessary(HttpServer.Builder 
builder, Configuration conf) {
+if 
(conf.getBoolean(QueryServices.QUERY_SERVER_WITH_REMOTEUSEREXTRACTOR_ATTRIB,
+
QueryServicesOptions.DEFAULT_QUERY_SERVER_WITH_REMOTEUSEREXTRACTOR)) {
+  builder.withRemoteUserExtractor(new 
PhoenixRemoteUserExtractor(conf));
+}
+  }
+
+  /**
+   * Use the correctly way to extract end user.
+   */
+
+  static class PhoenixRemoteUserExtractor implements RemoteUserExtractor{
+private final HttpQueryStringParameterRemoteUserExtractor 
paramRemoteUserExtractor;
+private final HttpRequestRemoteUserExtractor 
requestRemoteUserExtractor;
+private final String userExtractParam;
+
+public PhoenixRemoteUserExtractor(Configuration conf) {
+  this.requestRemoteUserExtractor = new 
HttpRequestRemoteUserExtractor();
+  this.userExtractParam = 
conf.get(QueryServices.QUERY_SERVER_REMOTEUSEREXTRACTOR_PARAM,
+  
QueryServicesOptions.DEFAULT_QUERY_SERVER_REMOTEUSEREXTRACTOR_PARAM);
+  this.paramRemoteUserExtractor = new 
HttpQueryStringParameterRemoteUserExtractor(userExtractParam);
+}
+
+@Override
+public String extract(HttpServletRequest request) throws 
RemoteUserExtractionException {
+  if (request.getParameter(userExtractParam) != null) {
+String extractedUser = paramRemoteUserExtractor.extract(request);
+UserGroupInformation ugi = 
UserGroupInformation.createRemoteUser(request.getRemoteUser());
+UserGroupInformation proxyUser = 
UserGroupInformation.createProxyUser(extractedUser, ugi);
--- End diff --

In re-reading the above, I'm a little worried about the edge-cases. With 
PQS right now, we have the following cases "supported"

1) Kerberos+SPNEGO as the Kerberos user (els...@example.com authenticates 
to PQS and the PQS credentials are used to query Phoenix as els...@example.com)
2) Kerberos auth with HBase but no SPNEGO for PQS clients (legacy support 
for how things used to work before the SPNEGO auth was built -- PQS user does 
everything for users)
3) Without Kerberos, all queries run as the PQS user (out of the box).

Avatica supports more than what point 3 above does, but we haven't 
prioritized wiring that up as most people just want point 1. @Wancy, I had 
originally thought you were just trying to enable a PQS client with Kerberos 
credentials to say that they are someone else (extension of point 1 -- 
Credentials to PQS are for "elserj" but instead of querying Phoenix as 
"elserj", query as "bob").

Did you also want this to work for cases when Kerberos is not in the mix? I 
think that would require some additional changes as I don't think this will 
work as-is.


> Enable proxy access to Phoenix query server for third party on behalf of end 
> users
> --
>
> Key: PHOENIX-3598
> URL: https://issues.apache.org/jira/browse/PHOENIX-3598
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Shi Wang
> Attachments: 0001-PHOENIX-3598.patch
>
>
> This JIRA tracks the follow-on work of CALCITE-1539 needed on Phoenix query 
> server side.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] phoenix pull request #265: PHOENIX-3598

[joshelser]

Github user joshelser commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/265#discussion_r124409282
  
--- Diff: 
phoenix-queryserver/src/main/java/org/apache/phoenix/queryserver/server/QueryServer.java
 ---
@@ -228,7 +235,9 @@ public int run(String[] args) throws Exception {
 builder.withSpnego(ugi.getUserName(), additionalAllowedRealms)
 .withAutomaticLogin(keytab)
 .withImpersonation(new PhoenixDoAsCallback(ugi, getConf()));
+
   }
+  setRemoteUserExtractorIfNecessary(builder, getConf());
--- End diff --

With respect to my long-winded comment below, if you're only looking to 
support Kerberos, we want to move this line into the above if-block.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] phoenix pull request #265: PHOENIX-3598

[joshelser]

Github user joshelser commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/265#discussion_r124409157
  
--- Diff: 
phoenix-queryserver/src/main/java/org/apache/phoenix/queryserver/server/QueryServer.java
 ---
@@ -273,6 +282,54 @@ public int run(String[] args) throws Exception {
 }
   }
 
+  // add remoteUserExtractor to builder if enabled
+  @VisibleForTesting
+  public void setRemoteUserExtractorIfNecessary(HttpServer.Builder 
builder, Configuration conf) {
+if 
(conf.getBoolean(QueryServices.QUERY_SERVER_WITH_REMOTEUSEREXTRACTOR_ATTRIB,
+
QueryServicesOptions.DEFAULT_QUERY_SERVER_WITH_REMOTEUSEREXTRACTOR)) {
+  builder.withRemoteUserExtractor(new 
PhoenixRemoteUserExtractor(conf));
+}
+  }
+
+  /**
+   * Use the correctly way to extract end user.
+   */
+
+  static class PhoenixRemoteUserExtractor implements RemoteUserExtractor{
+private final HttpQueryStringParameterRemoteUserExtractor 
paramRemoteUserExtractor;
+private final HttpRequestRemoteUserExtractor 
requestRemoteUserExtractor;
+private final String userExtractParam;
+
+public PhoenixRemoteUserExtractor(Configuration conf) {
+  this.requestRemoteUserExtractor = new 
HttpRequestRemoteUserExtractor();
+  this.userExtractParam = 
conf.get(QueryServices.QUERY_SERVER_REMOTEUSEREXTRACTOR_PARAM,
+  
QueryServicesOptions.DEFAULT_QUERY_SERVER_REMOTEUSEREXTRACTOR_PARAM);
+  this.paramRemoteUserExtractor = new 
HttpQueryStringParameterRemoteUserExtractor(userExtractParam);
+}
+
+@Override
+public String extract(HttpServletRequest request) throws 
RemoteUserExtractionException {
+  if (request.getParameter(userExtractParam) != null) {
+String extractedUser = paramRemoteUserExtractor.extract(request);
+UserGroupInformation ugi = 
UserGroupInformation.createRemoteUser(request.getRemoteUser());
+UserGroupInformation proxyUser = 
UserGroupInformation.createProxyUser(extractedUser, ugi);
--- End diff --

In re-reading the above, I'm a little worried about the edge-cases. With 
PQS right now, we have the following cases "supported"

1) Kerberos+SPNEGO as the Kerberos user (els...@example.com authenticates 
to PQS and the PQS credentials are used to query Phoenix as els...@example.com)
2) Kerberos auth with HBase but no SPNEGO for PQS clients (legacy support 
for how things used to work before the SPNEGO auth was built -- PQS user does 
everything for users)
3) Without Kerberos, all queries run as the PQS user (out of the box).

Avatica supports more than what point 3 above does, but we haven't 
prioritized wiring that up as most people just want point 1. @Wancy, I had 
originally thought you were just trying to enable a PQS client with Kerberos 
credentials to say that they are someone else (extension of point 1 -- 
Credentials to PQS are for "elserj" but instead of querying Phoenix as 
"elserj", query as "bob").

Did you also want this to work for cases when Kerberos is not in the mix? I 
think that would require some additional changes as I don't think this will 
work as-is.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] phoenix pull request #265: PHOENIX-3598

[joshelser]

Github user joshelser commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/265#discussion_r124402049
  
--- Diff: 
phoenix-queryserver/src/main/java/org/apache/phoenix/queryserver/server/QueryServer.java
 ---
@@ -273,6 +282,54 @@ public int run(String[] args) throws Exception {
 }
   }
 
+  // add remoteUserExtractor to builder if enabled
+  @VisibleForTesting
+  public void setRemoteUserExtractorIfNecessary(HttpServer.Builder 
builder, Configuration conf) {
+if 
(conf.getBoolean(QueryServices.QUERY_SERVER_WITH_REMOTEUSEREXTRACTOR_ATTRIB,
+
QueryServicesOptions.DEFAULT_QUERY_SERVER_WITH_REMOTEUSEREXTRACTOR)) {
+  builder.withRemoteUserExtractor(new 
PhoenixRemoteUserExtractor(conf));
+}
+  }
+
+  /**
+   * Use the correctly way to extract end user.
+   */
+
+  static class PhoenixRemoteUserExtractor implements RemoteUserExtractor{
+private final HttpQueryStringParameterRemoteUserExtractor 
paramRemoteUserExtractor;
+private final HttpRequestRemoteUserExtractor 
requestRemoteUserExtractor;
+private final String userExtractParam;
+
+public PhoenixRemoteUserExtractor(Configuration conf) {
+  this.requestRemoteUserExtractor = new 
HttpRequestRemoteUserExtractor();
+  this.userExtractParam = 
conf.get(QueryServices.QUERY_SERVER_REMOTEUSEREXTRACTOR_PARAM,
+  
QueryServicesOptions.DEFAULT_QUERY_SERVER_REMOTEUSEREXTRACTOR_PARAM);
+  this.paramRemoteUserExtractor = new 
HttpQueryStringParameterRemoteUserExtractor(userExtractParam);
+}
+
+@Override
+public String extract(HttpServletRequest request) throws 
RemoteUserExtractionException {
+  if (request.getParameter(userExtractParam) != null) {
--- End diff --

We should put a `requestRemoteUserExtractor.extract(request)` at the top of 
this method implementation. We should be using it in both branches of the 
conditional (replacing the `request.getRemoteUser()` call you have below)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (PHOENIX-3779) Can not start Phoenix Tephra even if the tx properties are set in hbase-site.xml

[Sergey Soldatov (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-3779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065551#comment-16065551
 ] 

Sergey Soldatov commented on PHOENIX-3779:
--

I have strong feeling that tephra just should not import into classpath all 
jars from lib directory. If there is hbase in the path, it will take all 
required libraries from there, if there is no hbase, it will use those that are 
bundled in phoenix fat client jar. 
[~jamestaylor], [~tdsilva] do you have any thoughts?  

> Can not start Phoenix Tephra even if the tx properties are set in 
> hbase-site.xml
> 
>
> Key: PHOENIX-3779
> URL: https://issues.apache.org/jira/browse/PHOENIX-3779
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.8.1
>Reporter: Jerry He
>
> I have had a hard time getting Tephra to start even after I put all the 
> required tx properties in hbase-site.xml and have it in the classpath.
> After searching around, I think it is the same problem reported here:
> https://www.mail-archive.com/dev@phoenix.apache.org/msg23182.html
> There are hbase-site.xml in some of the test jars and pherf jar.  After 
> moving out these jars, Tephra server starts nicely.
> The test jars and pherf are included in the tarball distribution lib.
> I think this needs to be fixed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (PHOENIX-3978) Expose mutation failures in our metrics

[Samarth Jain (JIRA)]

Samarth Jain created PHOENIX-3978:
-

 Summary: Expose mutation failures in our metrics
 Key: PHOENIX-3978
 URL: https://issues.apache.org/jira/browse/PHOENIX-3978
 Project: Phoenix
  Issue Type: Bug
Reporter: Samarth Jain


We should be exposing whether a mutation has failed in our metrics system. This 
should be done both at a global as well as our request level metrics. 

The task basically boils down to:
1) Adding a new enum MUTATION_BATCH_FAILED_COUNTER in MetricType.
2) Adding a new enum GLOBAL_MUTATION_BATCH_FAILED_COUNTER in GlobalClientMetrics
3) Adding a new CombinableMetric member called mutationBatchFailed to 
MutationMetric class
4) Making sure that the two metrics are updated within the catch exception 
block of MutationState#send()
5) Unit test in PhoenixMetricsIT

FYI, [~tdsilva]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PHOENIX-3598) Enable proxy access to Phoenix query server for third party on behalf of end users

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065259#comment-16065259
 ] 

ASF GitHub Bot commented on PHOENIX-3598:
-

Github user Wancy commented on the issue:

https://github.com/apache/phoenix/pull/265
  
Hi @joshelser,
I made some changes according to your comments, please review, thanks.


> Enable proxy access to Phoenix query server for third party on behalf of end 
> users
> --
>
> Key: PHOENIX-3598
> URL: https://issues.apache.org/jira/browse/PHOENIX-3598
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Shi Wang
> Attachments: 0001-PHOENIX-3598.patch
>
>
> This JIRA tracks the follow-on work of CALCITE-1539 needed on Phoenix query 
> server side.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] phoenix issue #265: PHOENIX-3598

[Wancy]

Github user Wancy commented on the issue:

https://github.com/apache/phoenix/pull/265
  
Hi @joshelser,
I made some changes according to your comments, please review, thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (PHOENIX-3965) Github repo read me need to update

[Josh Elser (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-3965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065047#comment-16065047
 ] 

Josh Elser commented on PHOENIX-3965:
-

[~aertoria], any interest in putting together a patch/pull-request to update it?

https://raw.githubusercontent.com/apache/phoenix/master/README.md

> Github repo read me need to update
> --
>
> Key: PHOENIX-3965
> URL: https://issues.apache.org/jira/browse/PHOENIX-3965
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Ethan Wang
>Priority: Trivial
>
> Github repo read me need to update. Both logo and read me.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Fwd: [JENKINS] [IMPORTANT] - Jenkins Migration and Upgrade (And JDK7 deprecation)

[Josh Elser]

tl;dr Infra is upgrading Jenkins in 2 weeks and Java7 Maven jobs 
may/may-not work after this. See explanation below from [1]:



Users with jobs configured with the "Maven project" type may not be able 
to use Java 7 for their Maven jobs. The correct behavior is not 
guaranteed so proceed at your own risk. The Maven Project uses Jenkins 
Remoting to establish "interceptors" within the Maven executable. 
Because of this, Maven uses Remoting and other Jenkins core classes, and 
this behavior may break an update.



[1] https://jenkins.io/blog/2017/04/10/jenkins-has-upgraded-to-java-8/


 Forwarded Message 
Subject: [JENKINS]  [IMPORTANT] - Jenkins Migration and Upgrade (And 
JDK7 deprecation)

Date: Tue, 27 Jun 2017 17:03:13 +1000
From: Gavin McDonald 
Reply-To: bui...@apache.org, bui...@apache.org
To: bui...@apache.org
CC: ASF Operations 

ASF Jenkins Master Migration and Upgrade on :-


Location	Local Time	 
   Time Zone	UTC Offset
Melbourne (Australia - Victoria)	Sunday, 16 July 2017 at 10:00:00 am 
AEST	UTC+10 hours
New York (USA - New York)	Saturday, 15 July 2017 at 8:00:00 pm 
EDT	UTC-4 hours

Corresponding UTC (GMT) Sunday, 16 July 2017 at 00:00:00

Hi All,

A few things are going to happen in just over 2 weeks.

1. Migration of Jenkins to a new host. A Jenkins Master module and yaml 
have been puppetized and ready to go.
What we need to do to migrate the Master away from its current host 
is turn off the old service. Perform a final rsync of data and 
perform the migration tasks.

As we intend to preserve history for jobs this will take some time.
At the same time as doing this migration to a new host, all slave 
connections will be updated (see below.)
I have no current estimate of downtime, but it will run into 
several hours. We do plan to run this migration on a Sunday at the 
lowest part of Jenkins usual usage.


2. Upgrade of Jenkins - Jenkins project released a new LTS release, 
version 2.60.1. This is a major release and breaks Jenkins in terms 
of Maven jobs for JDK 7 in the same way that it happened for Maven and 
JDK 6 a few months back.


The infra team (mainly myself) got quite some feedback on not 
supplying advance notice of this breakage. That upgrade however was 
necessary due to security fixes that required our upgrade.  This email 
serves as advance warning of the upcoming upgrade of Jenkins, the 
downtime due to the migration of the service to a new host; and notice 
of the breakage to JDK 7 that the upgrade brings.


Please familiarise yourself with the Jenkins LTS upgrade notes at [1].
In particular please note:-

“…2.60.1 is the first Jenkins LTS release that requires Java 8 to 
run. If you're using the Maven Project type, please note that it needs 
to use a JDK capable of running Jenkins, i.e. JDK 8 or up. If you 
configure an older JDK in a Maven Project, Jenkins will attempt to find 
a newer JDK and use that automatically. If your SSH Slaves fail to start 
and you have the plugin install the JRE to run them, make sure to update 
SSH Slaves Plugin to at least version 1.17 (1.20 recommended).

Changes since 2.60:
Fix for NullPointerException while initiating some SSH connections 
(regression in 2.59). (issue 44120 
)

Notable changes since 2.46.3:
Jenkins (master and agents) now requires Java 8 to run. (issue 27624 
 <>, issue 42709 
 <>, pull 2802 
, announcement blog post 
)


…”

There are over 30 other enhancements/fixes since 2.46.2 which we 
currently run so please do take a note of those.


Recap: In just over 2 weeks, downtime for a migration AND upgrade is 
planned.
Please do not rely on Jenkins at all for that weekend if you use it in 
your release workflow.


Please do take this notice back to your dev lists.
Any questions or concerns please email back to bui...@apache.org 
 only.

Thanks

Gav…

[1] - https://jenkins.io/changelog-stable/

[jira] [Commented] (PHOENIX-3598) Enable proxy access to Phoenix query server for third party on behalf of end users

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065008#comment-16065008
 ] 

ASF GitHub Bot commented on PHOENIX-3598:
-

Github user joshelser commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/265#discussion_r124312271
  
--- Diff: 
phoenix-queryserver/src/test/java/org/apache/phoenix/queryserver/server/PhoenixRemoteUserExtractorTest.java
 ---
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.queryserver.server;
+
+import static org.junit.Assert.assertEquals;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import org.apache.calcite.avatica.server.RemoteUserExtractionException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.security.authorize.AuthorizationException;
+import org.apache.hadoop.security.authorize.ProxyUsers;
+import 
org.apache.phoenix.queryserver.server.QueryServer.PhoenixRemoteUserExtractor;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.servlet.http.HttpServletRequest;
+
+/**
+ * Tests for the RemoteUserExtractor Method Avatica provides for Phoenix 
to implement.
+ */
+public class PhoenixRemoteUserExtractorTest {
+  private static final Logger LOG = 
LoggerFactory.getLogger(PhoenixRemoteUserExtractorTest.class);
+
+  @Test
+  public void testUseDoAsSuccess() {
+HttpServletRequest request = mock(HttpServletRequest.class);
+when(request.getRemoteUser()).thenReturn("proxyserver");
+when(request.getParameter("doAs")).thenReturn("enduser");
+when(request.getRemoteAddr()).thenReturn("localhost:1234");
+
+Configuration conf = new Configuration(false);
+conf.set("hadoop.proxyuser.proxyserver.groups", "*");
+conf.set("hadoop.proxyuser.proxyserver.hosts", "*");
+conf.set("phoenix.queryserver.doAs.enabled", "true");
+ProxyUsers.refreshSuperUserGroupsConfiguration(conf);
+
+PhoenixRemoteUserExtractor extractor = new 
PhoenixRemoteUserExtractor(conf);
+try {
+  assertEquals("enduser", extractor.extract(request));
+} catch (RemoteUserExtractionException e) {
+  LOG.info(e.getMessage());
+}
+  }
+
+  @Test
+  public void testDoNotUseDoAs() {
--- End diff --

No, there is no getter on the builder to verify it's called. Instead you 
can use the `Mockito.verify(builder)` method. Something like:

```java
Configuration conf = createTestConfiguration();
Builder b = Mockito.mock(Builder.class);

Mockito.when(b.withRemoteUserExtractor(Mockito.any(PhoenixRemoteUserExtractor.class))).thenReturn(b);
setRemoteUserExtractorIfNecessary(b, conf);
Mockito.verify(b);
```

This should essentially verify that `withRemoteUserExtractor` was invoked 
by `setRemoteUserExtractorIfNecessary`


> Enable proxy access to Phoenix query server for third party on behalf of end 
> users
> --
>
> Key: PHOENIX-3598
> URL: https://issues.apache.org/jira/browse/PHOENIX-3598
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Shi Wang
> Attachments: 0001-PHOENIX-3598.patch
>
>
> This JIRA tracks the follow-on work of CALCITE-1539 needed on Phoenix query 
> server side.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] phoenix pull request #265: PHOENIX-3598

[joshelser]

Github user joshelser commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/265#discussion_r124312271
  
--- Diff: 
phoenix-queryserver/src/test/java/org/apache/phoenix/queryserver/server/PhoenixRemoteUserExtractorTest.java
 ---
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.queryserver.server;
+
+import static org.junit.Assert.assertEquals;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import org.apache.calcite.avatica.server.RemoteUserExtractionException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.security.authorize.AuthorizationException;
+import org.apache.hadoop.security.authorize.ProxyUsers;
+import 
org.apache.phoenix.queryserver.server.QueryServer.PhoenixRemoteUserExtractor;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.servlet.http.HttpServletRequest;
+
+/**
+ * Tests for the RemoteUserExtractor Method Avatica provides for Phoenix 
to implement.
+ */
+public class PhoenixRemoteUserExtractorTest {
+  private static final Logger LOG = 
LoggerFactory.getLogger(PhoenixRemoteUserExtractorTest.class);
+
+  @Test
+  public void testUseDoAsSuccess() {
+HttpServletRequest request = mock(HttpServletRequest.class);
+when(request.getRemoteUser()).thenReturn("proxyserver");
+when(request.getParameter("doAs")).thenReturn("enduser");
+when(request.getRemoteAddr()).thenReturn("localhost:1234");
+
+Configuration conf = new Configuration(false);
+conf.set("hadoop.proxyuser.proxyserver.groups", "*");
+conf.set("hadoop.proxyuser.proxyserver.hosts", "*");
+conf.set("phoenix.queryserver.doAs.enabled", "true");
+ProxyUsers.refreshSuperUserGroupsConfiguration(conf);
+
+PhoenixRemoteUserExtractor extractor = new 
PhoenixRemoteUserExtractor(conf);
+try {
+  assertEquals("enduser", extractor.extract(request));
+} catch (RemoteUserExtractionException e) {
+  LOG.info(e.getMessage());
+}
+  }
+
+  @Test
+  public void testDoNotUseDoAs() {
--- End diff --

No, there is no getter on the builder to verify it's called. Instead you 
can use the `Mockito.verify(builder)` method. Something like:

```java
Configuration conf = createTestConfiguration();
Builder b = Mockito.mock(Builder.class);

Mockito.when(b.withRemoteUserExtractor(Mockito.any(PhoenixRemoteUserExtractor.class))).thenReturn(b);
setRemoteUserExtractorIfNecessary(b, conf);
Mockito.verify(b);
```

This should essentially verify that `withRemoteUserExtractor` was invoked 
by `setRemoteUserExtractorIfNecessary`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (PHOENIX-3977) Region is not closed when moving region or balancing while writing to table with local index

[JeongMin Ju (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-3977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

JeongMin Ju updated PHOENIX-3977:
-
Description: 
Region is not closed when moving region or balancing while writing to table 
with local index.

If the regionserver is forcibly killed and restart and then balancing is 
performed during write operation perform using YCSB. The region is not moved 
properly, so it becomes jammed.
This is also true when moving a specific region.

{panel:title=RegionServer1}
2017-06-26 17:18:49,096 INFO 
org.apache.phoenix.hbase.index.util.IndexManagementUtil: Rethrowing 
org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008 
(INT10): Unable to find cached index metadata.  key=-4998016164816556219 
region=PHOENIX_TABLE,&,1498464285648.df3f3da32306361ce5e38e5193f5e5d6.host=juke-cdh-36f531b4.s2.krane.9rum.cc,60020,1498464965204
 Index update failed
2017-06-26 17:18:49,531 INFO 
org.apache.phoenix.hbase.index.util.IndexManagementUtil: Rethrowing 
org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008 
(INT10): Unable to find cached index metadata.  key=6036765365681157624 
region=PHOENIX_TABLE,&,1498464285648.df3f3da32306361ce5e38e5193f5e5d6.host=juke-cdh-36f531b4.s2.krane.9rum.cc,60020,1498464965204
 Index update failed
2017-06-26 17:18:49,536 INFO 
org.apache.phoenix.hbase.index.util.IndexManagementUtil: Rethrowing 
org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008 
(INT10): Unable to find cached index metadata.  key=-428088766346522675 
region=PHOENIX_TABLE,&,1498464285648.df3f3da32306361ce5e38e5193f5e5d6.host=juke-cdh-36f531b4.s2.krane.9rum.cc,60020,1498464965204
 Index update failed
{panel}

At other regionserver
{panel:title=RegionServer2}
2017-06-26 17:19:49,755 INFO org.apache.hadoop.hbase.client.AsyncProcess: 
#1476, waiting for 62  actions to finish on table: PHOENIX_TABLE
2017-06-26 17:19:49,755 INFO org.apache.hadoop.hbase.client.AsyncProcess: 
#1476, waiting for 27  actions to finish on table: PHOENIX_TABLE
2017-06-26 17:19:49,755 INFO org.apache.hadoop.hbase.client.AsyncProcess: 
#1476, waiting for 52  actions to finish on table: PHOENIX_TABLE
{panel}

phoenix.coprocessor.maxServerCacheTimeToLiveMs & 
phoenix.coprocessor.maxMetaDataCacheSize was not effected.


  was:
Region is not closed when moving region or balancing while writing to table 
with local index.

If the regionserver is forcibly killed and restart and then balancing is 
performed during write operation perform using YCSB. The region is not moved 
properly, so it becomes jammed.
This is also true when moving a specific region.

{panel:title=My title}
2017-06-26 17:18:49,096 INFO 
org.apache.phoenix.hbase.index.util.IndexManagementUtil: Rethrowing 
org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008 
(INT10): Unable to find cached index metadata.  key=-4998016164816556219 
region=PHOENIX_TABLE,&,1498464285648.df3f3da32306361ce5e38e5193f5e5d6.host=juke-cdh-36f531b4.s2.krane.9rum.cc,60020,1498464965204
 Index update failed
2017-06-26 17:18:49,531 INFO 
org.apache.phoenix.hbase.index.util.IndexManagementUtil: Rethrowing 
org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008 
(INT10): Unable to find cached index metadata.  key=6036765365681157624 
region=PHOENIX_TABLE,&,1498464285648.df3f3da32306361ce5e38e5193f5e5d6.host=juke-cdh-36f531b4.s2.krane.9rum.cc,60020,1498464965204
 Index update failed
2017-06-26 17:18:49,536 INFO 
org.apache.phoenix.hbase.index.util.IndexManagementUtil: Rethrowing 
org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008 
(INT10): Unable to find cached index metadata.  key=-428088766346522675 
region=PHOENIX_TABLE,&,1498464285648.df3f3da32306361ce5e38e5193f5e5d6.host=juke-cdh-36f531b4.s2.krane.9rum.cc,60020,1498464965204
 Index update failed
{panel}

At other regionserver
{panel:title=My title}
2017-06-26 17:19:49,755 INFO org.apache.hadoop.hbase.client.AsyncProcess: 
#1476, waiting for 62  actions to finish on table: PHOENIX_TABLE
2017-06-26 17:19:49,755 INFO org.apache.hadoop.hbase.client.AsyncProcess: 
#1476, waiting for 27  actions to finish on table: PHOENIX_TABLE
2017-06-26 17:19:49,755 INFO org.apache.hadoop.hbase.client.AsyncProcess: 
#1476, waiting for 52  actions to finish on table: PHOENIX_TABLE
{panel}

phoenix.coprocessor.maxServerCacheTimeToLiveMs & 
phoenix.coprocessor.maxMetaDataCacheSize was not effected.



> Region is not closed when moving region or balancing while writing to table 
> with local index
> 
>
> Key: PHOENIX-3977
> URL: https://issues.apache.org/jira/browse/PHOENIX-3977
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.10.0
>Reporter: JeongMin Ju
>
> Region is not closed 

[jira] [Created] (PHOENIX-3977) Region is not closed when moving region or balancing while writing to table with local index

[JeongMin Ju (JIRA)]

JeongMin Ju created PHOENIX-3977:


 Summary: Region is not closed when moving region or balancing 
while writing to table with local index
 Key: PHOENIX-3977
 URL: https://issues.apache.org/jira/browse/PHOENIX-3977
 Project: Phoenix
  Issue Type: Bug
Affects Versions: 4.10.0
Reporter: JeongMin Ju


Region is not closed when moving region or balancing while writing to table 
with local index.

If the regionserver is forcibly killed and restart and then balancing is 
performed during write operation perform using YCSB. The region is not moved 
properly, so it becomes jammed.
This is also true when moving a specific region.

{panel:title=My title}
2017-06-26 17:18:49,096 INFO 
org.apache.phoenix.hbase.index.util.IndexManagementUtil: Rethrowing 
org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008 
(INT10): Unable to find cached index metadata.  key=-4998016164816556219 
region=PHOENIX_TABLE,&,1498464285648.df3f3da32306361ce5e38e5193f5e5d6.host=juke-cdh-36f531b4.s2.krane.9rum.cc,60020,1498464965204
 Index update failed
2017-06-26 17:18:49,531 INFO 
org.apache.phoenix.hbase.index.util.IndexManagementUtil: Rethrowing 
org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008 
(INT10): Unable to find cached index metadata.  key=6036765365681157624 
region=PHOENIX_TABLE,&,1498464285648.df3f3da32306361ce5e38e5193f5e5d6.host=juke-cdh-36f531b4.s2.krane.9rum.cc,60020,1498464965204
 Index update failed
2017-06-26 17:18:49,536 INFO 
org.apache.phoenix.hbase.index.util.IndexManagementUtil: Rethrowing 
org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008 
(INT10): Unable to find cached index metadata.  key=-428088766346522675 
region=PHOENIX_TABLE,&,1498464285648.df3f3da32306361ce5e38e5193f5e5d6.host=juke-cdh-36f531b4.s2.krane.9rum.cc,60020,1498464965204
 Index update failed
{panel}

At other regionserver
{panel:title=My title}
2017-06-26 17:19:49,755 INFO org.apache.hadoop.hbase.client.AsyncProcess: 
#1476, waiting for 62  actions to finish on table: PHOENIX_TABLE
2017-06-26 17:19:49,755 INFO org.apache.hadoop.hbase.client.AsyncProcess: 
#1476, waiting for 27  actions to finish on table: PHOENIX_TABLE
2017-06-26 17:19:49,755 INFO org.apache.hadoop.hbase.client.AsyncProcess: 
#1476, waiting for 52  actions to finish on table: PHOENIX_TABLE
{panel}

phoenix.coprocessor.maxServerCacheTimeToLiveMs & 
phoenix.coprocessor.maxMetaDataCacheSize was not effected.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PHOENIX-3633) null pointer exception when subsquery for not exists returns empty result set

[Sergey Soldatov (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-3633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16064376#comment-16064376
 ] 

Sergey Soldatov commented on PHOENIX-3633:
--

[~jerryhe] Could you add a UT for this particular case? I've tested the patch 
against main branch and haven't found any issues, but it would be nice to have 
a separate UT for such kind of queries.

> null pointer exception when subsquery for not exists returns empty result set
> -
>
> Key: PHOENIX-3633
> URL: https://issues.apache.org/jira/browse/PHOENIX-3633
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.7.0
>Reporter: N Campbell
>Assignee: Jerry He
> Attachments: PHOENIX-3633.patch
>
>
> phoenix-4.7.0.2.5.3.0-37-client
> select RNUM, C1, C2 from TJOIN2 where not exists ( select C1 from TJOIN1 
> where C2=500)
> Error while executing SQL "select RNUM, C1, C2 from TJOIN2 where not exists ( 
> select C1 from TJOIN1 where C2=500)": Remote driver error: 
> NullPointerException: (null exception message)
> create table  if not exists TJOIN1 (RNUM integer  not null primary key , C1 
> integer, C2 integer);
> upsert into TJOIN1 (RNUM, C1, C2) values ( 0, 10, 15);
> upsert into TJOIN1 (RNUM, C1, C2) values ( 1, 20, 25);
> upsert into TJOIN1 (RNUM, C1, C2) values ( 2, NULL, 50);
> create table  if not exists TJOIN2 (RNUM integer  not null primary key , C1 
> integer, C2 char(2));
> upsert into TJOIN2 (RNUM, C1, C2) values ( 0, 10, 'BB');
> upsert into TJOIN2 (RNUM, C1, C2) values ( 1, 15, 'DD');
> upsert into TJOIN2 (RNUM, C1, C2) values ( 2, NULL, 'EE');
> upsert into TJOIN2 (RNUM, C1, C2) values ( 3, 10, 'FF');



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (PHOENIX-3966) Failed to move working directory snapshot { ss=_UPGRADING_TABLE_SYSTEM.MUTEX table=SYSTEM.MUTEX type=DISABLED }

[vishal (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-3966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16064367#comment-16064367
 ] 

vishal edited comment on PHOENIX-3966 at 6/27/17 6:50 AM:
--

Hi,
Actually raised that issue on HBase but someone moved that issue to phoenix 
thats why it created two same issues in phoenix.
But PHOENIX-3966 and PHOENIX-3962 are different issues right.



was (Author: vishalb.e...@gmail.com):
Hi,
Actually raised that issue on HBase but someone moved that issue to phoenix 
thats why it created two same issues in phoenix.


> Failed to move working directory snapshot { ss=_UPGRADING_TABLE_SYSTEM.MUTEX 
> table=SYSTEM.MUTEX type=DISABLED }
> ---
>
> Key: PHOENIX-3966
> URL: https://issues.apache.org/jira/browse/PHOENIX-3966
> Project: Phoenix
>  Issue Type: Bug
> Environment: HBase version : 1.2.6
> Phoenix Version : 4.10.0-HBase-1.2.0
>Reporter: vishal
>
> While creating schema or table I am getting this error:
> Failed to move working directory snapshot { ss=_UPGRADING_TABLE_SYSTEM.MUTEX 
> table=SYSTEM.MUTEX type=DISABLED }
> So because of this error it is not creating the schema or table.
> java-program:
> --
> {code:java}
> Properties connectionProps = new Properties();
> connectionProps.put("phoenix.schema.isNamespaceMappingEnabled", "true");
> connectionProps.put("phoenix.schema.mapSystemTablesToNamespace", "true");
> connection = 
> DriverManager.getConnection("jdbc:phoenix:localhost",connectionProps);
> statement = connection.createStatement();
> statement.executeUpdate("CREATE SCHEMA MYSCHEMA");
> connection.commit();
> {code}
> hdfs-site.xml
> 
> {code:java}
>  
>   phoenix.schema.isNamespaceMappingEnabled
>   true
>
>   
>   phoenix.schema.mapSystemTablesToNamespace
>   true
>
> {code}
> please help me.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PHOENIX-3966) Failed to move working directory snapshot { ss=_UPGRADING_TABLE_SYSTEM.MUTEX table=SYSTEM.MUTEX type=DISABLED }

[vishal (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-3966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16064367#comment-16064367
 ] 

vishal commented on PHOENIX-3966:
-

Hi,
Actually raised that issue on HBase but someone moved that issue to phoenix 
thats why it created two same issues in phoenix.


> Failed to move working directory snapshot { ss=_UPGRADING_TABLE_SYSTEM.MUTEX 
> table=SYSTEM.MUTEX type=DISABLED }
> ---
>
> Key: PHOENIX-3966
> URL: https://issues.apache.org/jira/browse/PHOENIX-3966
> Project: Phoenix
>  Issue Type: Bug
> Environment: HBase version : 1.2.6
> Phoenix Version : 4.10.0-HBase-1.2.0
>Reporter: vishal
>
> While creating schema or table I am getting this error:
> Failed to move working directory snapshot { ss=_UPGRADING_TABLE_SYSTEM.MUTEX 
> table=SYSTEM.MUTEX type=DISABLED }
> So because of this error it is not creating the schema or table.
> java-program:
> --
> {code:java}
> Properties connectionProps = new Properties();
> connectionProps.put("phoenix.schema.isNamespaceMappingEnabled", "true");
> connectionProps.put("phoenix.schema.mapSystemTablesToNamespace", "true");
> connection = 
> DriverManager.getConnection("jdbc:phoenix:localhost",connectionProps);
> statement = connection.createStatement();
> statement.executeUpdate("CREATE SCHEMA MYSCHEMA");
> connection.commit();
> {code}
> hdfs-site.xml
> 
> {code:java}
>  
>   phoenix.schema.isNamespaceMappingEnabled
>   true
>
>   
>   phoenix.schema.mapSystemTablesToNamespace
>   true
>
> {code}
> please help me.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (PHOENIX-3633) null pointer exception when subsquery for not exists returns empty result set

[Ankit Singhal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-3633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ankit Singhal reassigned PHOENIX-3633:
--

Assignee: Jerry He  (was: NIDHI GAMBHIR)

> null pointer exception when subsquery for not exists returns empty result set
> -
>
> Key: PHOENIX-3633
> URL: https://issues.apache.org/jira/browse/PHOENIX-3633
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.7.0
>Reporter: N Campbell
>Assignee: Jerry He
> Attachments: PHOENIX-3633.patch
>
>
> phoenix-4.7.0.2.5.3.0-37-client
> select RNUM, C1, C2 from TJOIN2 where not exists ( select C1 from TJOIN1 
> where C2=500)
> Error while executing SQL "select RNUM, C1, C2 from TJOIN2 where not exists ( 
> select C1 from TJOIN1 where C2=500)": Remote driver error: 
> NullPointerException: (null exception message)
> create table  if not exists TJOIN1 (RNUM integer  not null primary key , C1 
> integer, C2 integer);
> upsert into TJOIN1 (RNUM, C1, C2) values ( 0, 10, 15);
> upsert into TJOIN1 (RNUM, C1, C2) values ( 1, 20, 25);
> upsert into TJOIN1 (RNUM, C1, C2) values ( 2, NULL, 50);
> create table  if not exists TJOIN2 (RNUM integer  not null primary key , C1 
> integer, C2 char(2));
> upsert into TJOIN2 (RNUM, C1, C2) values ( 0, 10, 'BB');
> upsert into TJOIN2 (RNUM, C1, C2) values ( 1, 15, 'DD');
> upsert into TJOIN2 (RNUM, C1, C2) values ( 2, NULL, 'EE');
> upsert into TJOIN2 (RNUM, C1, C2) values ( 3, 10, 'FF');



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (PHOENIX-3959) Unable to create new schema

[Ankit Singhal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-3959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ankit Singhal resolved PHOENIX-3959.

Resolution: Invalid

You may refer the docs(https://phoenix.apache.org/namspace_mapping.html).
And, If you still need any help in using the feature, you can ask on user 
mailing list(u...@phoenix.apache.org)

> Unable to create new schema
> ---
>
> Key: PHOENIX-3959
> URL: https://issues.apache.org/jira/browse/PHOENIX-3959
> Project: Phoenix
>  Issue Type: Bug
> Environment: Phoenix verion :4.10.0-HBase-1.2
> HBase version :-HBase-1.2.6
>Reporter: vishal
>
> I am getting below error while creating new schema:  
> Inconsistent namespace mapping properites.. Ensure that config 
> phoenix.schema.isNamespaceMappingEnabled is consitent on client and server.
> What does mean by ensure on client and server ??  
> What changes i need to do ??  
> hbase-site.xml
> 
> 
> phoenix.schema.isNamespaceMappingEnabled
> true
>   
> Java code :
> 
> Connection connection = setupDbConnection();
> statement statement = connection.createStatement();
> int status = statement.executeUpdate("CREATE SCHEMA test");
> connection.commit();



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (PHOENIX-3966) Failed to move working directory snapshot { ss=_UPGRADING_TABLE_SYSTEM.MUTEX table=SYSTEM.MUTEX type=DISABLED }

[Ankit Singhal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-3966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ankit Singhal resolved PHOENIX-3966.

Resolution: Duplicate

> Failed to move working directory snapshot { ss=_UPGRADING_TABLE_SYSTEM.MUTEX 
> table=SYSTEM.MUTEX type=DISABLED }
> ---
>
> Key: PHOENIX-3966
> URL: https://issues.apache.org/jira/browse/PHOENIX-3966
> Project: Phoenix
>  Issue Type: Bug
> Environment: HBase version : 1.2.6
> Phoenix Version : 4.10.0-HBase-1.2.0
>Reporter: vishal
>
> While creating schema or table I am getting this error:
> Failed to move working directory snapshot { ss=_UPGRADING_TABLE_SYSTEM.MUTEX 
> table=SYSTEM.MUTEX type=DISABLED }
> So because of this error it is not creating the schema or table.
> java-program:
> --
> {code:java}
> Properties connectionProps = new Properties();
> connectionProps.put("phoenix.schema.isNamespaceMappingEnabled", "true");
> connectionProps.put("phoenix.schema.mapSystemTablesToNamespace", "true");
> connection = 
> DriverManager.getConnection("jdbc:phoenix:localhost",connectionProps);
> statement = connection.createStatement();
> statement.executeUpdate("CREATE SCHEMA MYSCHEMA");
> connection.commit();
> {code}
> hdfs-site.xml
> 
> {code:java}
>  
>   phoenix.schema.isNamespaceMappingEnabled
>   true
>
>   
>   phoenix.schema.mapSystemTablesToNamespace
>   true
>
> {code}
> please help me.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PHOENIX-3966) Failed to move working directory snapshot { ss=_UPGRADING_TABLE_SYSTEM.MUTEX table=SYSTEM.MUTEX type=DISABLED }

[Ankit Singhal (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-3966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16064322#comment-16064322
 ] 

Ankit Singhal commented on PHOENIX-3966:


Duplicate of PHOENIX-3962.
[~vishalb.e...@gmail.com], please avoid spamming the Jira. I'm seeing multiple 
Jira(PHOENIX-3967) for the same 


> Failed to move working directory snapshot { ss=_UPGRADING_TABLE_SYSTEM.MUTEX 
> table=SYSTEM.MUTEX type=DISABLED }
> ---
>
> Key: PHOENIX-3966
> URL: https://issues.apache.org/jira/browse/PHOENIX-3966
> Project: Phoenix
>  Issue Type: Bug
> Environment: HBase version : 1.2.6
> Phoenix Version : 4.10.0-HBase-1.2.0
>Reporter: vishal
>
> While creating schema or table I am getting this error:
> Failed to move working directory snapshot { ss=_UPGRADING_TABLE_SYSTEM.MUTEX 
> table=SYSTEM.MUTEX type=DISABLED }
> So because of this error it is not creating the schema or table.
> java-program:
> --
> {code:java}
> Properties connectionProps = new Properties();
> connectionProps.put("phoenix.schema.isNamespaceMappingEnabled", "true");
> connectionProps.put("phoenix.schema.mapSystemTablesToNamespace", "true");
> connection = 
> DriverManager.getConnection("jdbc:phoenix:localhost",connectionProps);
> statement = connection.createStatement();
> statement.executeUpdate("CREATE SCHEMA MYSCHEMA");
> connection.commit();
> {code}
> hdfs-site.xml
> 
> {code:java}
>  
>   phoenix.schema.isNamespaceMappingEnabled
>   true
>
>   
>   phoenix.schema.mapSystemTablesToNamespace
>   true
>
> {code}
> please help me.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PHOENIX-3967) Failed to move working directory snapshot { ss=_UPGRADING_TABLE_SYSTEM.MUTEX table=SYSTEM.MUTEX type=DISABLED }

[Ankit Singhal (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-3967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16064320#comment-16064320
 ] 

Ankit Singhal commented on PHOENIX-3967:


Duplicate of PHOENIX-3962.

> Failed to move working directory snapshot { ss=_UPGRADING_TABLE_SYSTEM.MUTEX 
> table=SYSTEM.MUTEX type=DISABLED }
> ---
>
> Key: PHOENIX-3967
> URL: https://issues.apache.org/jira/browse/PHOENIX-3967
> Project: Phoenix
>  Issue Type: Bug
> Environment: HBase version : 1.2.6
> Phoenix Version : 4.10.0-HBase-1.2.0
>Reporter: vishal
>
> While creating schema or table I am getting this error:
> Failed to move working directory snapshot { ss=_UPGRADING_TABLE_SYSTEM.MUTEX 
> table=SYSTEM.MUTEX type=DISABLED }
> So because of this error it is not creating the schema or table.
> java-program:
> --
> {code:java}
> Properties connectionProps = new Properties();
> connectionProps.put("phoenix.schema.isNamespaceMappingEnabled", "true");
> connectionProps.put("phoenix.schema.mapSystemTablesToNamespace", "true");
> connection = 
> DriverManager.getConnection("jdbc:phoenix:localhost",connectionProps);
> statement = connection.createStatement();
> statement.executeUpdate("CREATE SCHEMA MYSCHEMA");
> {code}
> hdfs-site.xml
> 
> {code:java}
>  
>   phoenix.schema.isNamespaceMappingEnabled
>   true
>
>   
>   phoenix.schema.mapSystemTablesToNamespace
>   true
>
> {code}
> please help me.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (PHOENIX-3967) Failed to move working directory snapshot { ss=_UPGRADING_TABLE_SYSTEM.MUTEX table=SYSTEM.MUTEX type=DISABLED }

[Ankit Singhal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-3967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ankit Singhal resolved PHOENIX-3967.

Resolution: Duplicate

> Failed to move working directory snapshot { ss=_UPGRADING_TABLE_SYSTEM.MUTEX 
> table=SYSTEM.MUTEX type=DISABLED }
> ---
>
> Key: PHOENIX-3967
> URL: https://issues.apache.org/jira/browse/PHOENIX-3967
> Project: Phoenix
>  Issue Type: Bug
> Environment: HBase version : 1.2.6
> Phoenix Version : 4.10.0-HBase-1.2.0
>Reporter: vishal
>
> While creating schema or table I am getting this error:
> Failed to move working directory snapshot { ss=_UPGRADING_TABLE_SYSTEM.MUTEX 
> table=SYSTEM.MUTEX type=DISABLED }
> So because of this error it is not creating the schema or table.
> java-program:
> --
> {code:java}
> Properties connectionProps = new Properties();
> connectionProps.put("phoenix.schema.isNamespaceMappingEnabled", "true");
> connectionProps.put("phoenix.schema.mapSystemTablesToNamespace", "true");
> connection = 
> DriverManager.getConnection("jdbc:phoenix:localhost",connectionProps);
> statement = connection.createStatement();
> statement.executeUpdate("CREATE SCHEMA MYSCHEMA");
> {code}
> hdfs-site.xml
> 
> {code:java}
>  
>   phoenix.schema.isNamespaceMappingEnabled
>   true
>
>   
>   phoenix.schema.mapSystemTablesToNamespace
>   true
>
> {code}
> please help me.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)