[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541179#comment-16541179
 ] 

Hadoop QA commented on PHOENIX-3534:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12931252/PHOENIX-3534-v3.patch
  against master branch at commit aee568beb02cdf983bb10889902c338ea016e6c9.
  ATTACHMENT ID: 12931252

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 29 new 
or modified tests.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:red}-1 release audit{color}.  The applied patch generated 3 release 
audit warnings (more than the master's current 0 warnings).

{color:red}-1 lineLengths{color}.  The patch introduces the following lines 
longer than 100:
+conn.createStatement().execute("CREATE LOCAL INDEX " + 
generateUniqueName() + " ON " + tableName + "(KV1)");
+assertEquals(4, getIndexOfPkColumn(phxConn, 
IndexUtil.getIndexColumnName(null, "k2"), fullView2IndexName));
+assertEquals(5, getIndexOfPkColumn(phxConn, 
IndexUtil.getIndexColumnName(null, "k3"), fullView2IndexName));
+assertEquals(4, getIndexOfPkColumn(phxConn, 
IndexUtil.getIndexColumnName(null, "k2"), fullView3IndexName));
+assertEquals(5, getIndexOfPkColumn(phxConn, 
IndexUtil.getIndexColumnName(null, "k3"), fullView3IndexName));
+String view1DDL = "CREATE VIEW " + view1 + " ( VIEW_COL1 
DECIMAL(10,2), VIEW_COL2 CHAR(256)) AS SELECT * FROM " + baseTable;
+String divergedViewDDL = "CREATE VIEW " + divergedView + " ( 
VIEW_COL1 DECIMAL(10,2), VIEW_COL2 CHAR(256)) AS SELECT * FROM " + baseTable;
+String indexDDL = "CREATE INDEX " + divergedViewIndex + " ON " + 
divergedView + " (V1) include (V3)";
+assertTableDefinition(tenant1Conn, view1, PTableType.VIEW, 
baseTable, 0, 7, 5,  "PK1", "V1", "V2", "V3", "KV", "PK2", "VIEW_COL1", 
"VIEW_COL2");
+assertTableDefinition(tenant2Conn, divergedView, PTableType.VIEW, 
baseTable, 1, 6, DIVERGED_VIEW_BASE_COLUMN_COUNT, "PK1", "V1", "V3", "PK2", 
"VIEW_COL1", "VIEW_COL2");

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
 
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.index.MutableIndexIT

 {color:red}-1 core zombie tests{color}.  There are 4 zombie test(s):   
at 
org.apache.phoenix.end2end.DeleteIT.testPointDeleteRowFromTableWithImmutableIndex(DeleteIT.java:403)
at 
org.apache.phoenix.end2end.DeleteIT.testPointDeleteRowFromTableWithImmutableIndex(DeleteIT.java:376)
at 
org.apache.phoenix.end2end.DefaultColumnValueIT.testDefaultColumnValue(DefaultColumnValueIT.java:66)

Test results: 
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1926//testReport/
Release audit warnings: 
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1926//artifact/patchprocess/patchReleaseAuditWarnings.txt
Console output: 
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1926//console

This message is automatically generated.

> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Fix For: 5.0.0, 4.15.0
>
> Attachments: PHOENIX-3534-v2.patch, PHOENIX-3534-v3.patch, 
> PHOENIX-3534.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region 
> based on the server-side row locks being held for operations that impact a 
> table and all of it's views. For example, adding/removing a column from a 
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), 
> when a new table is created we can do a lazy cleanup  of any rows that may be 
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with 
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on 
> checking our sequence numbers for each table/view before/after updating. Each 
> table/view row would be individually locked for their change (metadata for a 
> view or table cannot span regions due to our split policy), with the sequence 
> number being incremented under lock and then returned to the client.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (PHOENIX-4790) Simplify check for client side delete

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor updated PHOENIX-4790:
--
Attachment: PHOENIX-4790_addendum1.patch

> Simplify check for client side delete
> -
>
> Key: PHOENIX-4790
> URL: https://issues.apache.org/jira/browse/PHOENIX-4790
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: James Taylor
>Priority: Minor
> Fix For: 4.15.0, 5.1.0
>
> Attachments: PHOENIX-4790.patch, PHOENIX-4790_addendum1.patch
>
>
> We don't need to check every query plan for the existence of a where clause 
> to determine if we can do a client-side delete. Instead, we can simply check 
> if any non PK columns are projected.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4790) Simplify check for client side delete

[James Taylor (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541149#comment-16541149
 ] 

James Taylor commented on PHOENIX-4790:
---

Sorry about that, [~tdsilva]. I've attached an addendum which I'll commit 
unless I hear objections.

> Simplify check for client side delete
> -
>
> Key: PHOENIX-4790
> URL: https://issues.apache.org/jira/browse/PHOENIX-4790
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: James Taylor
>Priority: Minor
> Fix For: 4.15.0, 5.1.0
>
> Attachments: PHOENIX-4790.patch, PHOENIX-4790_addendum1.patch
>
>
> We don't need to check every query plan for the existence of a where clause 
> to determine if we can do a client-side delete. Instead, we can simply check 
> if any non PK columns are projected.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (PHOENIX-4811) Phoenix Secondary Index isn't be used

[Thomas D'Silva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas D'Silva resolved PHOENIX-4811.
-
Resolution: Not A Problem

The index stores data sorted on the indexed columns. If your query filters on 
name and age and you want to use the index create the index with (name, age) or 
(age, name)

> Phoenix Secondary Index isn't be used
> -
>
> Key: PHOENIX-4811
> URL: https://issues.apache.org/jira/browse/PHOENIX-4811
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.13.1
> Environment: Operating System : centos6.9 ;
> Phoenix version : 4.13.1;
> HBase version : 1.3.1;
>Reporter: 张延召
>Priority: Major
>  Labels: SecondaryIndex
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> I created two indexes (not Covered Indexes):
>  ^create table test(id varchar not null primary key,name varchar,age varchar 
> ,sex varchar);^
>  ^create local index if not exists val_test_name on test(name);^
>  ^create local index if not exists val_test_age on test(age);^
>  ^upsert into test(id,name,age,sex)values('myid','name','age','sex');^
> It is work to use one index:
>   ^explain select name from test where name = 'name';^
> But why can't hit index when using multiple indexes？
>   ^explain select name,age from test where name = 'name' and age = 'age';^
>  
>  
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (PHOENIX-4811) Phoenix Secondary Index isn't be used

[JIRA]

张延召 created PHOENIX-4811:


 Summary: Phoenix Secondary Index isn't be used
 Key: PHOENIX-4811
 URL: https://issues.apache.org/jira/browse/PHOENIX-4811
 Project: Phoenix
  Issue Type: Bug
Affects Versions: 4.13.1
 Environment: Operating System : centos6.9 ;

Phoenix version : 4.13.1;

HBase version : 1.3.1;
Reporter: 张延召


I created two indexes (not Covered Indexes):

 ^create table test(id varchar not null primary key,name varchar,age varchar 
,sex varchar);^
 ^create local index if not exists val_test_name on test(name);^
 ^create local index if not exists val_test_age on test(age);^
 ^upsert into test(id,name,age,sex)values('myid','name','age','sex');^

It is work to use one index:

  ^explain select name from test where name = 'name';^

But why can't hit index when using multiple indexes？

  ^explain select name,age from test where name = 'name' and age = 'age';^

 

 

 

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[DISCUSS] reduce notifications to phoenix-dev list

[Thomas D'Silva]

I think we should reduce the number of notifications that are sent to the
phoenix-dev list.
The notification scheme we use is located here : https://issues.apache.org/
jira/plugins/servlet/project-config/PHOENIX/notifications

Maybe we don't need to notify the dev list when an issue is updated, or a
comment is created/edited/deleted. People watching a particular issue would
still see all the changes.
This would be similar to how other projects like HBase and Calcite operate.

Thanks,
Thomas

[jira] [Updated] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table

[Thomas D'Silva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas D'Silva updated PHOENIX-3534:

Attachment: PHOENIX-3534-v3.patch

> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Fix For: 5.0.0, 4.15.0
>
> Attachments: PHOENIX-3534-v2.patch, PHOENIX-3534-v3.patch, 
> PHOENIX-3534.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region 
> based on the server-side row locks being held for operations that impact a 
> table and all of it's views. For example, adding/removing a column from a 
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), 
> when a new table is created we can do a lazy cleanup  of any rows that may be 
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with 
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on 
> checking our sequence numbers for each table/view before/after updating. Each 
> table/view row would be individually locked for their change (metadata for a 
> view or table cannot span regions due to our split policy), with the sequence 
> number being incremented under lock and then returned to the client.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (PHOENIX-4790) Simplify check for client side delete

[Thomas D'Silva (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540953#comment-16540953
 ] 

Thomas D'Silva edited comment on PHOENIX-4790 at 7/12/18 12:59 AM:
---

[~jamestaylor]

I think this patch is causing tests to fail :

BinaryRowKeyIT.testInsertPaddedBinaryValue() 
ArithmeticQueryIT.testRandomFunction() 

 


was (Author: tdsilva):
[~jamestaylor]

I think this patch is causing tests to fail : 
[BinaryRowKeyIT|eclipse-javadoc:%E2%98%82=phoenix-core/src%5C/it%5C/java%3Corg.apache.phoenix.end2end%7BBinaryRowKeyIT.java%E2%98%83BinaryRowKeyIT].testInsertPaddedBinaryValue()
 and 
[ArithmeticQueryIT|eclipse-javadoc:%E2%98%82=phoenix-core/src%5C/it%5C/java%3Corg.apache.phoenix.end2end%7BArithmeticQueryIT.java%E2%98%83ArithmeticQueryIT].testRandomFunction()

> Simplify check for client side delete
> -
>
> Key: PHOENIX-4790
> URL: https://issues.apache.org/jira/browse/PHOENIX-4790
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: James Taylor
>Priority: Minor
> Fix For: 4.15.0, 5.1.0
>
> Attachments: PHOENIX-4790.patch
>
>
> We don't need to check every query plan for the existence of a where clause 
> to determine if we can do a client-side delete. Instead, we can simply check 
> if any non PK columns are projected.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4790) Simplify check for client side delete

[Thomas D'Silva (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540953#comment-16540953
 ] 

Thomas D'Silva commented on PHOENIX-4790:
-

[~jamestaylor]

I think this patch is causing tests to fail : 
[BinaryRowKeyIT|eclipse-javadoc:%E2%98%82=phoenix-core/src%5C/it%5C/java%3Corg.apache.phoenix.end2end%7BBinaryRowKeyIT.java%E2%98%83BinaryRowKeyIT].testInsertPaddedBinaryValue()
 and 
[ArithmeticQueryIT|eclipse-javadoc:%E2%98%82=phoenix-core/src%5C/it%5C/java%3Corg.apache.phoenix.end2end%7BArithmeticQueryIT.java%E2%98%83ArithmeticQueryIT].testRandomFunction()

> Simplify check for client side delete
> -
>
> Key: PHOENIX-4790
> URL: https://issues.apache.org/jira/browse/PHOENIX-4790
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: James Taylor
>Priority: Minor
> Fix For: 4.15.0, 5.1.0
>
> Attachments: PHOENIX-4790.patch
>
>
> We don't need to check every query plan for the existence of a where clause 
> to determine if we can do a client-side delete. Instead, we can simply check 
> if any non PK columns are projected.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] stop minor releases for 0.98 and 1.1

[Thomas D'Silva]

+1

On Tue, Jul 10, 2018 at 1:12 PM, Ankit Singhal 
wrote:

> +1
>
> On Tue, Jul 10, 2018 at 12:22 PM Josh Elser  wrote:
>
> > +1
> >
> > On 7/9/18 7:03 PM, James Taylor wrote:
> > > +1
> > >
> > > On Mon, Jul 9, 2018 at 3:55 PM, Vincent Poon <
> vincent.poon...@gmail.com>
> > > wrote:
> > >
> > >> This is a call to vote on stopping maintenance of both the
> > 4.x-HBase-0.98
> > >> and 4.x-HBase-1.1 branches.
> > >> HBase 0.98 and 1.1 have been EOLed.  To commit currently, it's usually
> > 7 or
> > >> more branches, which is a burden on devs.
> > >>
> > >> Vote will be open for at least 72 hours. Please vote:
> > >>
> > >> [ ] +1 approve
> > >> [ ] +0 no opinion
> > >> [ ] -1 disapprove (and reason why)
> > >>
> > >> Thanks,
> > >> The Apache Phoenix Team
> > >>
> > >
> >
>

[jira] [Updated] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table

[Hadoop QA (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hadoop QA updated PHOENIX-3534:
---
Attachment: PHOENIX-3534-v2.patch

> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Fix For: 5.0.0, 4.15.0
>
> Attachments: PHOENIX-3534-v2.patch, PHOENIX-3534.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region 
> based on the server-side row locks being held for operations that impact a 
> table and all of it's views. For example, adding/removing a column from a 
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), 
> when a new table is created we can do a lazy cleanup  of any rows that may be 
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with 
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on 
> checking our sequence numbers for each table/view before/after updating. Each 
> table/view row would be individually locked for their change (metadata for a 
> view or table cannot span regions due to our split policy), with the sequence 
> number being incremented under lock and then returned to the client.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4805) Move Avatica version to 1.12 for PQS

[Karan Mehta (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540881#comment-16540881
 ] 

Karan Mehta commented on PHOENIX-4805:
--

Looks like [https://builds.apache.org/job/PreCommit-PHOENIX-Build/1917/console] 
has completed the tests successfully, buy it failed when trying to post Jira 
comment for it. Will commit to 4.x and master branches. [~elserj] any thing 
else over here?

> Move Avatica version to 1.12 for PQS
> 
>
> Key: PHOENIX-4805
> URL: https://issues.apache.org/jira/browse/PHOENIX-4805
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Karan Mehta
>Assignee: Karan Mehta
>Priority: Major
> Attachments: PHOENIX-4805.001.patch
>
>
> CALCITE-2330 is now resolved. We can use the new version for supporting new 
> features such as MTLS authentication and server customization in PQS.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4809) connectionQueue never cleared in ConnectionQueryServicesImpl when lease renewal is disabled/unsupported

[Ankit Singhal (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540666#comment-16540666
 ] 

Ankit Singhal commented on PHOENIX-4809:


+1 [~elserj], Thanks for the test as well :)

> connectionQueue never cleared in ConnectionQueryServicesImpl when lease 
> renewal is disabled/unsupported
> ---
>
> Key: PHOENIX-4809
> URL: https://issues.apache.org/jira/browse/PHOENIX-4809
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 4.15.0, 5.1.0
>
> Attachments: PHOENIX-4809.001.patch
>
>
> When we create a new {{PhoenixConnection}}, we update {{connectionQueues}} in 
> CQSI:
> {code:java}
>     @Override
>     public void addConnection(PhoenixConnection connection) throws 
> SQLException {
>     connectionQueues.get(getQueueIndex(connection)).add(new 
> WeakReference(connection));
>     if (returnSequenceValues) {
>     synchronized (connectionCountLock) {
>     connectionCount++;
>     }
>     }
>     }{code}
> We use connectionQueues to determine what needs lease renewal done.
> However, when the user closes a connection, this datastructure is never 
> cleaned up.
> {code:java}
>     @Override
>     public void removeConnection(PhoenixConnection connection) throws 
> SQLException {
>     if (returnSequenceValues) {
>     ConcurrentMap formerSequenceMap = null;
>     synchronized (connectionCountLock) {
>     if (--connectionCount <= 0) {
>     if (!this.sequenceMap.isEmpty()) {
>     formerSequenceMap = this.sequenceMap;
>     this.sequenceMap = Maps.newConcurrentMap();
>     }
>     }
>     if (connectionCount < 0) {
>     connectionCount = 0;
>     }
>     }
>     // Since we're using the former sequenceMap, we can do this 
> outside
>     // the lock.
>     if (formerSequenceMap != null) {
>     // When there are no more connections, attempt to return any 
> sequences
>     returnAllSequences(formerSequenceMap);
>     }
>     } else if (shouldThrottleNumConnections){ //still need to decrement 
> connection count
>     synchronized (connectionCountLock) {
>     if (connectionCount > 0) {
>     --connectionCount;
>     }
>     }
>     }
>     }{code}
> Running a test now, but seems to be the case on master.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4807) Document query behavior when statistics unavailable for some table regions.

[Sergey Soldatov (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540663#comment-16540663
 ] 

Sergey Soldatov commented on PHOENIX-4807:
--

I've updated the page with a known issues section. Not sure whether a real 
example is required, because the end user usually doesn't care about how it 
works under the hood.

> Document query behavior when statistics unavailable for some table regions.
> ---
>
> Key: PHOENIX-4807
> URL: https://issues.apache.org/jira/browse/PHOENIX-4807
> Project: Phoenix
>  Issue Type: Task
>Reporter: Guru Prateek Pinnadhari
>Assignee: Sergey Soldatov
>Priority: Major
>
> When troubleshooting an issue with retrieval of duplicate records for a 
> select query on same PK, the behavior seen was: query plan's collection of 
> Scan objects were such that they led to generation of overlapping scans on a 
> salted table's regions.
> A contributing factor was statistics being unavailable for the terminal 
> regions of the table. Filing Jira to add a note about this on 
> [https://phoenix.apache.org/update_statistics.html]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (PHOENIX-4809) connectionQueue never cleared in ConnectionQueryServicesImpl when lease renewal is disabled/unsupported

[Josh Elser (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh Elser updated PHOENIX-4809:

Attachment: PHOENIX-4809.001.patch

> connectionQueue never cleared in ConnectionQueryServicesImpl when lease 
> renewal is disabled/unsupported
> ---
>
> Key: PHOENIX-4809
> URL: https://issues.apache.org/jira/browse/PHOENIX-4809
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 4.15.0, 5.1.0
>
> Attachments: PHOENIX-4809.001.patch
>
>
> When we create a new {{PhoenixConnection}}, we update {{connectionQueues}} in 
> CQSI:
> {code:java}
>     @Override
>     public void addConnection(PhoenixConnection connection) throws 
> SQLException {
>     connectionQueues.get(getQueueIndex(connection)).add(new 
> WeakReference(connection));
>     if (returnSequenceValues) {
>     synchronized (connectionCountLock) {
>     connectionCount++;
>     }
>     }
>     }{code}
> We use connectionQueues to determine what needs lease renewal done.
> However, when the user closes a connection, this datastructure is never 
> cleaned up.
> {code:java}
>     @Override
>     public void removeConnection(PhoenixConnection connection) throws 
> SQLException {
>     if (returnSequenceValues) {
>     ConcurrentMap formerSequenceMap = null;
>     synchronized (connectionCountLock) {
>     if (--connectionCount <= 0) {
>     if (!this.sequenceMap.isEmpty()) {
>     formerSequenceMap = this.sequenceMap;
>     this.sequenceMap = Maps.newConcurrentMap();
>     }
>     }
>     if (connectionCount < 0) {
>     connectionCount = 0;
>     }
>     }
>     // Since we're using the former sequenceMap, we can do this 
> outside
>     // the lock.
>     if (formerSequenceMap != null) {
>     // When there are no more connections, attempt to return any 
> sequences
>     returnAllSequences(formerSequenceMap);
>     }
>     } else if (shouldThrottleNumConnections){ //still need to decrement 
> connection count
>     synchronized (connectionCountLock) {
>     if (connectionCount > 0) {
>     --connectionCount;
>     }
>     }
>     }
>     }{code}
> Running a test now, but seems to be the case on master.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] phoenix pull request #303: PHOENIX-3534 Support multi region SYSTEM.CATALOG ...

[twdsilva]

Github user twdsilva commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/303#discussion_r201835494
  
--- Diff: phoenix-core/src/it/java/org/apache/phoenix/end2end/ViewIT.java 
---
@@ -372,6 +378,31 @@ public void testViewAndTableAndDrop() throws Exception 
{
 // drop table cascade should succeed
 conn.createStatement().execute("DROP TABLE " + fullTableName + " 
CASCADE");
 
+validateViewDoesNotExist(conn, fullViewName1);
+validateViewDoesNotExist(conn, fullViewName2);
+
+}
+
+@Test
+public void testRecreateDroppedTableWithChildViews() throws Exception {
--- End diff --

We write the parent->child link first, then if the table uses column 
encoding we update the encoded column qualifiers on the parent table, and 
finally use mutateRowsWithLocks to write the view metadata atomically. 
We ignore views that can't be found (in case writing the child view 
metadata fails). 
If the metadata write fails and the table uses column encoding then we will 
lose a few column qualifiers. 
I'll add a test for this.



---

[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540649#comment-16540649
 ] 

ASF GitHub Bot commented on PHOENIX-3534:
-

Github user twdsilva commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/303#discussion_r201835494
  
--- Diff: phoenix-core/src/it/java/org/apache/phoenix/end2end/ViewIT.java 
---
@@ -372,6 +378,31 @@ public void testViewAndTableAndDrop() throws Exception 
{
 // drop table cascade should succeed
 conn.createStatement().execute("DROP TABLE " + fullTableName + " 
CASCADE");
 
+validateViewDoesNotExist(conn, fullViewName1);
+validateViewDoesNotExist(conn, fullViewName2);
+
+}
+
+@Test
+public void testRecreateDroppedTableWithChildViews() throws Exception {
--- End diff --

We write the parent->child link first, then if the table uses column 
encoding we update the encoded column qualifiers on the parent table, and 
finally use mutateRowsWithLocks to write the view metadata atomically. 
We ignore views that can't be found (in case writing the child view 
metadata fails). 
If the metadata write fails and the table uses column encoding then we will 
lose a few column qualifiers. 
I'll add a test for this.



> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Fix For: 5.0.0, 4.15.0
>
> Attachments: PHOENIX-3534.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region 
> based on the server-side row locks being held for operations that impact a 
> table and all of it's views. For example, adding/removing a column from a 
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), 
> when a new table is created we can do a lazy cleanup  of any rows that may be 
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with 
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on 
> checking our sequence numbers for each table/view before/after updating. Each 
> table/view row would be individually locked for their change (metadata for a 
> view or table cannot span regions due to our split policy), with the sequence 
> number being incremented under lock and then returned to the client.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4793) Backport PHOENIX-4303, PHOENIX-4305, PHOENIX-4321 to 4.x branches.

[James Taylor (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540609#comment-16540609
 ] 

James Taylor commented on PHOENIX-4793:
---

Would it be possible for you to do this one after PHOENIX-3534 makes it in in 
the next couple of days, [~chrajeshbab...@gmail.com] ?

> Backport PHOENIX-4303, PHOENIX-4305, PHOENIX-4321 to 4.x branches.
> --
>
> Key: PHOENIX-4793
> URL: https://issues.apache.org/jira/browse/PHOENIX-4793
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
>Priority: Major
> Fix For: 4.15.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Videos from HBaseCon and PhoenixCon 2018

[Josh Elser]

Looks like we missed one the first time around (sorry, Scott!)

https://www.youtube.com/watch?v=hbLYt4dUHp8

On 7/6/18 1:36 PM, Josh Elser wrote:

Hi all,

As enabled by one of the event sponsors (Bloomberg -- big thanks to 
them), we have some professional recordings of all of the talks that 
were given on June 18th in San Jose, CA. I've compiled them and uploaded 
them to Youtube:


https://www.youtube.com/playlist?list=PLdOhKvENNn6WH71iHXRNbEd1pTN2xY6P3

Each video is the presentation view and the audio from the speaker. We 
did not have cameras on each speaker during the talk. We also did not 
curate the slides that accompanied each talks -- please follow up with 
the speaker if you'd like a copy.


I mention that I personally compiled them to outline that I may have 
fat-fingered something :). Please let me know if you see any typos or 
similar. I'll be working on updating the HBase and Phoenix websites to 
reflect the content later.


- JOsh

[jira] [Updated] (PHOENIX-4809) connectionQueue never cleared in ConnectionQueryServicesImpl when lease renewal is disabled/unsupported

[Josh Elser (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh Elser updated PHOENIX-4809:

Summary: connectionQueue never cleared in ConnectionQueryServicesImpl when 
lease renewal is disabled/unsupported  (was: connectionQueue never cleared in 
ConnectionQueryServicesImpl)

> connectionQueue never cleared in ConnectionQueryServicesImpl when lease 
> renewal is disabled/unsupported
> ---
>
> Key: PHOENIX-4809
> URL: https://issues.apache.org/jira/browse/PHOENIX-4809
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 4.15.0, 5.1.0
>
>
> When we create a new {{PhoenixConnection}}, we update {{connectionQueues}} in 
> CQSI:
> {code:java}
>     @Override
>     public void addConnection(PhoenixConnection connection) throws 
> SQLException {
>     connectionQueues.get(getQueueIndex(connection)).add(new 
> WeakReference(connection));
>     if (returnSequenceValues) {
>     synchronized (connectionCountLock) {
>     connectionCount++;
>     }
>     }
>     }{code}
> We use connectionQueues to determine what needs lease renewal done.
> However, when the user closes a connection, this datastructure is never 
> cleaned up.
> {code:java}
>     @Override
>     public void removeConnection(PhoenixConnection connection) throws 
> SQLException {
>     if (returnSequenceValues) {
>     ConcurrentMap formerSequenceMap = null;
>     synchronized (connectionCountLock) {
>     if (--connectionCount <= 0) {
>     if (!this.sequenceMap.isEmpty()) {
>     formerSequenceMap = this.sequenceMap;
>     this.sequenceMap = Maps.newConcurrentMap();
>     }
>     }
>     if (connectionCount < 0) {
>     connectionCount = 0;
>     }
>     }
>     // Since we're using the former sequenceMap, we can do this 
> outside
>     // the lock.
>     if (formerSequenceMap != null) {
>     // When there are no more connections, attempt to return any 
> sequences
>     returnAllSequences(formerSequenceMap);
>     }
>     } else if (shouldThrottleNumConnections){ //still need to decrement 
> connection count
>     synchronized (connectionCountLock) {
>     if (connectionCount > 0) {
>     --connectionCount;
>     }
>     }
>     }
>     }{code}
> Running a test now, but seems to be the case on master.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4809) connectionQueue never cleared in ConnectionQueryServicesImpl

[Josh Elser (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540557#comment-16540557
 ] 

Josh Elser commented on PHOENIX-4809:
-

{quote}Maybe related to lease renewal not being on?
{quote}
Ah, yeah, that's the rub. If the RenewLeaseTask doesn't run, nothing drains 
these queues.

> connectionQueue never cleared in ConnectionQueryServicesImpl
> 
>
> Key: PHOENIX-4809
> URL: https://issues.apache.org/jira/browse/PHOENIX-4809
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 4.15.0, 5.1.0
>
>
> When we create a new {{PhoenixConnection}}, we update {{connectionQueues}} in 
> CQSI:
> {code:java}
>     @Override
>     public void addConnection(PhoenixConnection connection) throws 
> SQLException {
>     connectionQueues.get(getQueueIndex(connection)).add(new 
> WeakReference(connection));
>     if (returnSequenceValues) {
>     synchronized (connectionCountLock) {
>     connectionCount++;
>     }
>     }
>     }{code}
> We use connectionQueues to determine what needs lease renewal done.
> However, when the user closes a connection, this datastructure is never 
> cleaned up.
> {code:java}
>     @Override
>     public void removeConnection(PhoenixConnection connection) throws 
> SQLException {
>     if (returnSequenceValues) {
>     ConcurrentMap formerSequenceMap = null;
>     synchronized (connectionCountLock) {
>     if (--connectionCount <= 0) {
>     if (!this.sequenceMap.isEmpty()) {
>     formerSequenceMap = this.sequenceMap;
>     this.sequenceMap = Maps.newConcurrentMap();
>     }
>     }
>     if (connectionCount < 0) {
>     connectionCount = 0;
>     }
>     }
>     // Since we're using the former sequenceMap, we can do this 
> outside
>     // the lock.
>     if (formerSequenceMap != null) {
>     // When there are no more connections, attempt to return any 
> sequences
>     returnAllSequences(formerSequenceMap);
>     }
>     } else if (shouldThrottleNumConnections){ //still need to decrement 
> connection count
>     synchronized (connectionCountLock) {
>     if (connectionCount > 0) {
>     --connectionCount;
>     }
>     }
>     }
>     }{code}
> Running a test now, but seems to be the case on master.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4809) connectionQueue never cleared in ConnectionQueryServicesImpl

[Josh Elser (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540555#comment-16540555
 ] 

Josh Elser commented on PHOENIX-4809:
-

I ran a quick test locally, the odd thing is that I see fewer 
{{WeakReference}} objects than I'd expect..
{code:java}
public class PhoenixQuery {
  private static final String URL = "jdbc:phoenix:localhost:2181:/hbase-1.4";

  public static void main(String[] args) throws Exception {
    ExecutorService svc = Executors.newFixedThreadPool(8);
    try (Connection conn = DriverManager.getConnection(URL, "", "")) {
  PhoenixConnection phxConn = conn.unwrap(PhoenixConnection.class);
  ConnectionQueryServicesImpl queryServices = (ConnectionQueryServicesImpl) 
phxConn.getQueryServices();
  System.out.println(numConnectionsCached(queryServices));
    }
    CountDownLatch latch = new CountDownLatch(8);
    for (int i = 0; i < 8; i++) {
  svc.submit(new Runnable() {
    @Override public void run() {
  for (int j = 0; j < 1_000_000; j++) {
    try (Connection conn = DriverManager.getConnection(URL, "", "")) {
    conn.close();
    } catch (Exception e) {
  System.out.println("Caught error");
    }
  }
  latch.countDown();
    }
  });
    }
    latch.await();

    try (Connection conn = DriverManager.getConnection(URL, "", "")) {
  PhoenixConnection phxConn = conn.unwrap(PhoenixConnection.class);
  ConnectionQueryServicesImpl queryServices = (ConnectionQueryServicesImpl) 
phxConn.getQueryServices();
  System.out.println(numConnectionsCached(queryServices));
    }
  }

  private static long numConnectionsCached(ConnectionQueryServicesImpl cqs) {
    try {
  Field f = 
ConnectionQueryServicesImpl.class.getDeclaredField("connectionQueues");
  f.setAccessible(true);
  List>> list = 
(List>>) f.get(cqs);
  return 
list.stream().collect(Collectors.summingLong(LinkedBlockingQueue::size));
    } catch (Exception e) {
  e.printStackTrace(System.err);
  throw new RuntimeException(e);
    }
  }
}{code}
I'd expect to see 8M objects cached, but I'm actually seeing 615,696. Something 
must clean this up somehow.. Maybe related to lease renewal not being on?

> connectionQueue never cleared in ConnectionQueryServicesImpl
> 
>
> Key: PHOENIX-4809
> URL: https://issues.apache.org/jira/browse/PHOENIX-4809
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 4.15.0, 5.1.0
>
>
> When we create a new {{PhoenixConnection}}, we update {{connectionQueues}} in 
> CQSI:
> {code:java}
>     @Override
>     public void addConnection(PhoenixConnection connection) throws 
> SQLException {
>     connectionQueues.get(getQueueIndex(connection)).add(new 
> WeakReference(connection));
>     if (returnSequenceValues) {
>     synchronized (connectionCountLock) {
>     connectionCount++;
>     }
>     }
>     }{code}
> We use connectionQueues to determine what needs lease renewal done.
> However, when the user closes a connection, this datastructure is never 
> cleaned up.
> {code:java}
>     @Override
>     public void removeConnection(PhoenixConnection connection) throws 
> SQLException {
>     if (returnSequenceValues) {
>     ConcurrentMap formerSequenceMap = null;
>     synchronized (connectionCountLock) {
>     if (--connectionCount <= 0) {
>     if (!this.sequenceMap.isEmpty()) {
>     formerSequenceMap = this.sequenceMap;
>     this.sequenceMap = Maps.newConcurrentMap();
>     }
>     }
>     if (connectionCount < 0) {
>     connectionCount = 0;
>     }
>     }
>     // Since we're using the former sequenceMap, we can do this 
> outside
>     // the lock.
>     if (formerSequenceMap != null) {
>     // When there are no more connections, attempt to return any 
> sequences
>     returnAllSequences(formerSequenceMap);
>     }
>     } else if (shouldThrottleNumConnections){ //still need to decrement 
> connection count
>     synchronized (connectionCountLock) {
>     if (connectionCount > 0) {
>     --connectionCount;
>     }
>     }
>     }
>     }{code}
> Running a test now, but seems to be the case on master.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540504#comment-16540504
 ] 

ASF GitHub Bot commented on PHOENIX-3534:
-

Github user twdsilva commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/303#discussion_r201800952
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/util/UpgradeUtil.java ---
@@ -1893,26 +1981,45 @@ public static void upgradeTable(PhoenixConnection 
conn, String srcTable) throws
 conn.commit();
 }
 conn.getQueryServices().clearTableFromCache(
-conn.getTenantId() == null ? 
ByteUtil.EMPTY_BYTE_ARRAY : conn.getTenantId().getBytes(),
+tenantIdBytes,
 index.getSchemaName().getBytes(), 
index.getTableName().getBytes(),
 PhoenixRuntime.getCurrentScn(readOnlyProps));
 }
 updateIndexesSequenceIfPresent(conn, table);
 conn.commit();
-
 } else {
 throw new RuntimeException("Error: problem occured during 
upgrade. Table is not upgraded successfully");
 }
 if (table.getType() == PTableType.VIEW) {
 logger.info(String.format("Updating link information for 
view '%s' ..", table.getTableName()));
 updateLink(conn, oldPhysicalName, 
newPhysicalTablename,table.getSchemaName(),table.getTableName());
 conn.commit();
-
+
+// if the view is a first level child, then we need to 
create the PARENT_TABLE link
+// that was overwritten by the PHYSICAL_TABLE link 
--- End diff --

Yes this will make it so that the parent link row will be created correctly 
when upgrading tables to be namespace mapped. 


> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Fix For: 5.0.0, 4.15.0
>
> Attachments: PHOENIX-3534.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region 
> based on the server-side row locks being held for operations that impact a 
> table and all of it's views. For example, adding/removing a column from a 
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), 
> when a new table is created we can do a lazy cleanup  of any rows that may be 
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with 
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on 
> checking our sequence numbers for each table/view before/after updating. Each 
> table/view row would be individually locked for their change (metadata for a 
> view or table cannot span regions due to our split policy), with the sequence 
> number being incremented under lock and then returned to the client.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] phoenix pull request #303: PHOENIX-3534 Support multi region SYSTEM.CATALOG ...

[twdsilva]

Github user twdsilva commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/303#discussion_r201800952
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/util/UpgradeUtil.java ---
@@ -1893,26 +1981,45 @@ public static void upgradeTable(PhoenixConnection 
conn, String srcTable) throws
 conn.commit();
 }
 conn.getQueryServices().clearTableFromCache(
-conn.getTenantId() == null ? 
ByteUtil.EMPTY_BYTE_ARRAY : conn.getTenantId().getBytes(),
+tenantIdBytes,
 index.getSchemaName().getBytes(), 
index.getTableName().getBytes(),
 PhoenixRuntime.getCurrentScn(readOnlyProps));
 }
 updateIndexesSequenceIfPresent(conn, table);
 conn.commit();
-
 } else {
 throw new RuntimeException("Error: problem occured during 
upgrade. Table is not upgraded successfully");
 }
 if (table.getType() == PTableType.VIEW) {
 logger.info(String.format("Updating link information for 
view '%s' ..", table.getTableName()));
 updateLink(conn, oldPhysicalName, 
newPhysicalTablename,table.getSchemaName(),table.getTableName());
 conn.commit();
-
+
+// if the view is a first level child, then we need to 
create the PARENT_TABLE link
+// that was overwritten by the PHYSICAL_TABLE link 
--- End diff --

Yes this will make it so that the parent link row will be created correctly 
when upgrading tables to be namespace mapped. 


---

[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540498#comment-16540498
 ] 

ASF GitHub Bot commented on PHOENIX-3534:
-

Github user twdsilva commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/303#discussion_r201799117
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
 ---
@@ -1779,13 +2174,65 @@ public void createTable(RpcController controller, 
CreateTableRequest request,
 }
 }
 
+// The mutations to create a table are written in the 
following order:
+// 1. Write the child link as if the next two steps fail we
+// ignore missing children while processing a parent
+// 2. Update the encoded column qualifier for the parent 
table if its on a
+// different region server (for tables that use column 
qualifier encoding)
+// if the next step fails we end up wasting a few col 
qualifiers
+// 3. Finally write the mutations to create the table
+
+// From 4.15 the parent->child links are stored in a 
separate table SYSTEM.CHILD_LINK
+List childLinkMutations = 
MetaDataUtil.removeChildLinks(tableMetadata);
--- End diff --

I filed PHOENIX-4810 and added a comment to reference this jira.


> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Fix For: 5.0.0, 4.15.0
>
> Attachments: PHOENIX-3534.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region 
> based on the server-side row locks being held for operations that impact a 
> table and all of it's views. For example, adding/removing a column from a 
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), 
> when a new table is created we can do a lazy cleanup  of any rows that may be 
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with 
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on 
> checking our sequence numbers for each table/view before/after updating. Each 
> table/view row would be individually locked for their change (metadata for a 
> view or table cannot span regions due to our split policy), with the sequence 
> number being incremented under lock and then returned to the client.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] phoenix pull request #303: PHOENIX-3534 Support multi region SYSTEM.CATALOG ...

[twdsilva]

Github user twdsilva commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/303#discussion_r201799117
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
 ---
@@ -1779,13 +2174,65 @@ public void createTable(RpcController controller, 
CreateTableRequest request,
 }
 }
 
+// The mutations to create a table are written in the 
following order:
+// 1. Write the child link as if the next two steps fail we
+// ignore missing children while processing a parent
+// 2. Update the encoded column qualifier for the parent 
table if its on a
+// different region server (for tables that use column 
qualifier encoding)
+// if the next step fails we end up wasting a few col 
qualifiers
+// 3. Finally write the mutations to create the table
+
+// From 4.15 the parent->child links are stored in a 
separate table SYSTEM.CHILD_LINK
+List childLinkMutations = 
MetaDataUtil.removeChildLinks(tableMetadata);
--- End diff --

I filed PHOENIX-4810 and added a comment to reference this jira.


---

[jira] [Created] (PHOENIX-4810) Send parent->child link mutations to SYSTEM.CHILD_LINK table in MetdataClient.createTableInternal

[Thomas D'Silva (JIRA)]

Thomas D'Silva created PHOENIX-4810:
---

 Summary: Send parent->child link mutations to SYSTEM.CHILD_LINK 
table in MetdataClient.createTableInternal 
 Key: PHOENIX-4810
 URL: https://issues.apache.org/jira/browse/PHOENIX-4810
 Project: Phoenix
  Issue Type: Sub-task
Reporter: Thomas D'Silva


Instead of sending the parent->child link mutations to the 
MetadataEndpointImpl.createTable write them directly to SYSTEM.CHILD_LINK



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (PHOENIX-4809) connectionQueue never cleared in ConnectionQueryServicesImpl

[Josh Elser (JIRA)]

Josh Elser created PHOENIX-4809:
---

 Summary: connectionQueue never cleared in 
ConnectionQueryServicesImpl
 Key: PHOENIX-4809
 URL: https://issues.apache.org/jira/browse/PHOENIX-4809
 Project: Phoenix
  Issue Type: Bug
Reporter: Josh Elser
Assignee: Josh Elser
 Fix For: 4.15.0, 5.1.0


When we create a new {{PhoenixConnection}}, we update {{connectionQueues}} in 
CQSI:
{code:java}
    @Override
    public void addConnection(PhoenixConnection connection) throws SQLException 
{
    connectionQueues.get(getQueueIndex(connection)).add(new 
WeakReference(connection));
    if (returnSequenceValues) {
    synchronized (connectionCountLock) {
    connectionCount++;
    }
    }
    }{code}
We use connectionQueues to determine what needs lease renewal done.

However, when the user closes a connection, this datastructure is never cleaned 
up.
{code:java}
    @Override
    public void removeConnection(PhoenixConnection connection) throws 
SQLException {
    if (returnSequenceValues) {
    ConcurrentMap formerSequenceMap = null;
    synchronized (connectionCountLock) {
    if (--connectionCount <= 0) {
    if (!this.sequenceMap.isEmpty()) {
    formerSequenceMap = this.sequenceMap;
    this.sequenceMap = Maps.newConcurrentMap();
    }
    }
    if (connectionCount < 0) {
    connectionCount = 0;
    }
    }
    // Since we're using the former sequenceMap, we can do this outside
    // the lock.
    if (formerSequenceMap != null) {
    // When there are no more connections, attempt to return any 
sequences
    returnAllSequences(formerSequenceMap);
    }
    } else if (shouldThrottleNumConnections){ //still need to decrement 
connection count
    synchronized (connectionCountLock) {
    if (connectionCount > 0) {
    --connectionCount;
    }
    }
    }
    }{code}
Running a test now, but seems to be the case on master.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (PHOENIX-4808) Phoenix Spark Do not verify zKUrl in DefaultSource.scala

[Lev Bronshtein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lev Bronshtein updated PHOENIX-4808:

Description: 
Currently verifyParameters will throw an exception

_[zkUrl 
check|https://github.com/apache/phoenix/blob/master/phoenix-spark/src/main/scala/org/apache/phoenix/spark/DefaultSource.scala#L56]_

_if (parameters.get("zkUrl").isEmpty)_ however this is unnecessary as 
ultimately new PhoenixRelation(...)

[PhoenixRelation|https://github.com/apache/phoenix/blob/master/phoenix-spark/src/main/scala/org/apache/phoenix/spark/DefaultSource.scala#L29]

is called (see PhoenixRelation.scala) which in turn calls new PhoenixRDD(...) 
(See PhoenixRDD.scala).  PhoenixRDD constructor declares zkUrl as optional.  
PhoenixRDD will try to use zkUrl and fall back on HBase Configuration

 
{code:java}
// Override the Zookeeper URL if present. Throw exception if no address given.
zkUrl match {
case Some(url) => ConfigurationUtil.setZookeeperURL(config, url)
case _ => {
if(ConfigurationUtil.getZookeeperURL(config).isEmpty) {
throw new UnsupportedOperationException(
  s"One of zkUrl or '${HConstants.ZOOKEEPER_QUORUM}' config 
property must be provided"
)
}
}
{code}
 

I propose that this check is unnecessary and should be removed.  As we will 
still get an exception, but a little later

  was:
Currently verifyParameters will throw an exception

_[zkUrl 
check|https://github.com/apache/phoenix/blob/master/phoenix-spark/src/main/scala/org/apache/phoenix/spark/DefaultSource.scala#L56]_

_if (parameters.get("zkUrl").isEmpty)_ however this is unnecessary as 
ultimately new PhoenixRelation(...)

[PhoenixRelation|https://github.com/apache/phoenix/blob/master/phoenix-spark/src/main/scala/org/apache/phoenix/spark/DefaultSource.scala#L29]

is called (see PhoenixRelation.scala) which in turn calls new PhoenixRDD(...) 
(See PhoenixRDD.scala).  PhoenixRDD constructor declares zkUrl as optional.  
PhoenixRDD will try to use zkUrl and fall back on HBase Configuration

 
{code:java}
// Override the Zookeeper URL if present. Throw exception if no address given.
zkUrl match {
case Some(url) => ConfigurationUtil.setZookeeperURL(config, url)
case _ => {
if(ConfigurationUtil.getZookeeperURL(config).isEmpty) {
throw new UnsupportedOperationException(
s"One of zkUrl or '${HConstants.ZOOKEEPER_QUORUM}' config property must be 
provided"
)
}
}
{code}
 

I propose that this check is unnecessary and should be removed.  As we will 
still get an exception, but a little later


> Phoenix Spark Do not verify zKUrl in DefaultSource.scala
> 
>
> Key: PHOENIX-4808
> URL: https://issues.apache.org/jira/browse/PHOENIX-4808
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Lev Bronshtein
>Priority: Major
>
> Currently verifyParameters will throw an exception
> _[zkUrl 
> check|https://github.com/apache/phoenix/blob/master/phoenix-spark/src/main/scala/org/apache/phoenix/spark/DefaultSource.scala#L56]_
> _if (parameters.get("zkUrl").isEmpty)_ however this is unnecessary as 
> ultimately new PhoenixRelation(...)
> [PhoenixRelation|https://github.com/apache/phoenix/blob/master/phoenix-spark/src/main/scala/org/apache/phoenix/spark/DefaultSource.scala#L29]
> is called (see PhoenixRelation.scala) which in turn calls new PhoenixRDD(...) 
> (See PhoenixRDD.scala).  PhoenixRDD constructor declares zkUrl as optional.  
> PhoenixRDD will try to use zkUrl and fall back on HBase Configuration
>  
> {code:java}
> // Override the Zookeeper URL if present. Throw exception if no address given.
> zkUrl match {
> case Some(url) => ConfigurationUtil.setZookeeperURL(config, url)
> case _ => {
> if(ConfigurationUtil.getZookeeperURL(config).isEmpty) {
> throw new UnsupportedOperationException(
>   s"One of zkUrl or '${HConstants.ZOOKEEPER_QUORUM}' config 
> property must be provided"
> )
> }
> }
> {code}
>  
> I propose that this check is unnecessary and should be removed.  As we will 
> still get an exception, but a little later



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (PHOENIX-4808) Phoenix Spark Do not verify zKUrl in DefaultSource.scala

[Lev Bronshtein (JIRA)]

Lev Bronshtein created PHOENIX-4808:
---

 Summary: Phoenix Spark Do not verify zKUrl in DefaultSource.scala
 Key: PHOENIX-4808
 URL: https://issues.apache.org/jira/browse/PHOENIX-4808
 Project: Phoenix
  Issue Type: Improvement
Reporter: Lev Bronshtein


Currently verifyParameters will throw an exception

_[zkUrl 
check|https://github.com/apache/phoenix/blob/master/phoenix-spark/src/main/scala/org/apache/phoenix/spark/DefaultSource.scala#L56]_

_if (parameters.get("zkUrl").isEmpty)_ however this is unnecessary as 
ultimately new PhoenixRelation(...)

[PhoenixRelation|https://github.com/apache/phoenix/blob/master/phoenix-spark/src/main/scala/org/apache/phoenix/spark/DefaultSource.scala#L29]

is called (see PhoenixRelation.scala) which in turn calls new PhoenixRDD(...) 
(See PhoenixRDD.scala).  PhoenixRDD constructor declares zkUrl as optional.  
PhoenixRDD will try to use zkUrl and fall back on HBase Configuration

 
{code:java}
// Override the Zookeeper URL if present. Throw exception if no address given.
zkUrl match {
case Some(url) => ConfigurationUtil.setZookeeperURL(config, url)
case _ => {
if(ConfigurationUtil.getZookeeperURL(config).isEmpty) {
throw new UnsupportedOperationException(
s"One of zkUrl or '${HConstants.ZOOKEEPER_QUORUM}' config property must be 
provided"
)
}
}
{code}
 

I propose that this check is unnecessary and should be removed.  As we will 
still get an exception, but a little later



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] phoenix pull request #303: PHOENIX-3534 Support multi region SYSTEM.CATALOG ...

[twdsilva]

Github user twdsilva commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/303#discussion_r201776288
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
 ---
@@ -447,7 +447,7 @@
 static {
 Collections.sort(FUNCTION_KV_COLUMNS, KeyValue.COMPARATOR);
 }
-
+
--- End diff --

I modified the class level comment.


---

[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540416#comment-16540416
 ] 

ASF GitHub Bot commented on PHOENIX-3534:
-

Github user twdsilva commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/303#discussion_r201776089
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
 ---
@@ -1457,28 +1791,110 @@ private static void getSchemaTableNames(Mutation 
row, byte[][] schemaTableNames)
 schemaTableNames[2] = tName;
 }
 }
-
+
 @Override
 public void createTable(RpcController controller, CreateTableRequest 
request,
 RpcCallback done) {
 MetaDataResponse.Builder builder = MetaDataResponse.newBuilder();
 byte[][] rowKeyMetaData = new byte[3][];
 byte[] schemaName = null;
 byte[] tableName = null;
+String fullTableName = null;
 try {
 int clientVersion = request.getClientVersion();
 List tableMetadata = 
ProtobufUtil.getMutations(request);
 MetaDataUtil.getTenantIdAndSchemaAndTableName(tableMetadata, 
rowKeyMetaData);
 byte[] tenantIdBytes = 
rowKeyMetaData[PhoenixDatabaseMetaData.TENANT_ID_INDEX];
 schemaName = 
rowKeyMetaData[PhoenixDatabaseMetaData.SCHEMA_NAME_INDEX];
 tableName = 
rowKeyMetaData[PhoenixDatabaseMetaData.TABLE_NAME_INDEX];
+fullTableName = SchemaUtil.getTableName(schemaName, tableName);
+// TODO before creating a table we need to see if the table 
was previously created and then dropped
+// and clean up any parent->child links or child views
--- End diff --

Done.


> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Fix For: 5.0.0, 4.15.0
>
> Attachments: PHOENIX-3534.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region 
> based on the server-side row locks being held for operations that impact a 
> table and all of it's views. For example, adding/removing a column from a 
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), 
> when a new table is created we can do a lazy cleanup  of any rows that may be 
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with 
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on 
> checking our sequence numbers for each table/view before/after updating. Each 
> table/view row would be individually locked for their change (metadata for a 
> view or table cannot span regions due to our split policy), with the sequence 
> number being incremented under lock and then returned to the client.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540415#comment-16540415
 ] 

ASF GitHub Bot commented on PHOENIX-3534:
-

Github user twdsilva commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/303#discussion_r201776046
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
 ---
@@ -586,48 +590,359 @@ public void getTable(RpcController controller, 
GetTableRequest request,
 
builder.setMutationTime(minNonZerodisableIndexTimestamp - 1);
 }
 }
-
-if (table.getTimeStamp() != tableTimeStamp) {
+// the PTable of views and indexes on views might get updated 
because a column is added to one of
+// their parents (this won't change the timestamp)
+if (table.getType()!=PTableType.TABLE || table.getTimeStamp() 
!= tableTimeStamp) {
 builder.setTable(PTableImpl.toProto(table));
 }
 done.run(builder.build());
-return;
 } catch (Throwable t) {
 logger.error("getTable failed", t);
 ProtobufUtil.setControllerException(controller,
 
ServerUtil.createIOException(SchemaUtil.getTableName(schemaName, tableName), 
t));
 }
 }
 
+/**
+ * Used to add the columns present the ancestor hierarchy to the 
PTable of the given view or
+ * view index
+ * @param table PTable of the view or view index
+ * @param skipAddingIndexes if true the returned PTable won't include 
indexes
+ * @param skipAddingParentColumns if true the returned PTable won't 
include columns derived from
+ *ancestor tables
+ * @param lockedAncestorTable ancestor table table that is being 
mutated (as we won't be able to
+ *resolve this table as its locked)
+ */
+private Pair 
combineColumns(PTable table, long timestamp,
+int clientVersion, boolean skipAddingIndexes, boolean 
skipAddingParentColumns,
+PTable lockedAncestorTable) throws SQLException, IOException {
+boolean hasIndexId = table.getViewIndexId() != null;
+if (table.getType() != PTableType.VIEW && !hasIndexId) {
+return new Pair(table,
+MetaDataProtos.MutationCode.TABLE_ALREADY_EXISTS);
+}
+if (!skipAddingParentColumns) {
+table =
+addDerivedColumnsFromAncestors(table, timestamp, 
clientVersion,
+lockedAncestorTable);
+if (table==null) {
+return new Pair(table,
+MetaDataProtos.MutationCode.TABLE_NOT_FOUND);
+}
+// we need to resolve the indexes of views (to get ensure they 
also have all the columns
+// derived from their ancestors) 
+if (!skipAddingIndexes && !table.getIndexes().isEmpty()) {
+List indexes = 
Lists.newArrayListWithExpectedSize(table.getIndexes().size());
+for (PTable index : table.getIndexes()) {
+byte[] tenantIdBytes =
+index.getTenantId() == null ? 
ByteUtil.EMPTY_BYTE_ARRAY
+: index.getTenantId().getBytes();
+PTable latestIndex =
+doGetTable(tenantIdBytes, 
index.getSchemaName().getBytes(),
+index.getTableName().getBytes(), 
timestamp, null, clientVersion, true,
+false, lockedAncestorTable);
+if (latestIndex == null) {
+throw new TableNotFoundException(
+"Could not find index table while 
combining columns "
++ index.getTableName().getString() 
+ " with tenant id "
++ index.getTenantId());
+}
+indexes.add(latestIndex);
+}
+table = PTableImpl.makePTable(table, table.getTimeStamp(), 
indexes);
+}
+}
+
+MetaDataProtos.MutationCode mutationCode =
+table != null ? 
MetaDataProtos.MutationCode.TABLE_ALREADY_EXISTS
+: MetaDataProtos.MutationCode.TABLE_NOT_FOUND;
+return new Pair(table, 
mutationCode);
+}
+
+
+private PTable addDerivedColumnsFromAncestors(PTable table, long 
timestamp,
+int clientVersion, PTable 

[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540419#comment-16540419
 ] 

ASF GitHub Bot commented on PHOENIX-3534:
-

Github user twdsilva commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/303#discussion_r201776204
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
 ---
@@ -1809,180 +2256,97 @@ public void createTable(RpcController controller, 
CreateTableRequest request,
 } catch (Throwable t) {
 logger.error("createTable failed", t);
 ProtobufUtil.setControllerException(controller,
-
ServerUtil.createIOException(SchemaUtil.getTableName(schemaName, tableName), 
t));
+ServerUtil.createIOException(fullTableName, t));
 }
 }
 
+   private void dropChildMetadata(byte[] schemaName, byte[] tableName, 
byte[] tenantIdBytes)
+   throws IOException, SQLException, 
ClassNotFoundException {
+   TableViewFinderResult childViewsResult = new 
TableViewFinderResult();
+   findAllChildViews(tenantIdBytes, schemaName, tableName, 
childViewsResult);
+   if (childViewsResult.hasViews()) {
+   for (TableInfo viewInfo : 
childViewsResult.getResults()) {
+   byte[] viewTenantId = viewInfo.getTenantId();
+   byte[] viewSchemaName = 
viewInfo.getSchemaName();
+   byte[] viewName = viewInfo.getTableName();
+   Properties props = new Properties();
+   if (viewTenantId != null && viewTenantId.length 
!= 0)
+   
props.setProperty(PhoenixRuntime.TENANT_ID_ATTRIB, 
Bytes.toString(viewTenantId));
+   try (PhoenixConnection connection = 
QueryUtil.getConnectionOnServer(env.getConfiguration())
+   
.unwrap(PhoenixConnection.class)) {
+   MetaDataClient client = new 
MetaDataClient(connection);
+   org.apache.phoenix.parse.TableName 
viewTableName = org.apache.phoenix.parse.TableName
+   
.create(Bytes.toString(viewSchemaName), Bytes.toString(viewName));
+   client.dropTable(
+   new 
DropTableStatement(viewTableName, PTableType.VIEW, false, true, true));
+   }
+   }
+   }
+   }
+
 private boolean execeededIndexQuota(PTableType tableType, PTable 
parentTable) {
 return PTableType.INDEX == tableType && 
parentTable.getIndexes().size() >= maxIndexesPerTable;
 }
-
-private void findAllChildViews(Region region, byte[] tenantId, PTable 
table,
-TableViewFinder result, long clientTimeStamp, int 
clientVersion) throws IOException, SQLException {
-TableViewFinder currResult = findChildViews(region, tenantId, 
table, clientVersion, false);
-result.addResult(currResult);
-for (ViewInfo viewInfo : currResult.getViewInfoList()) {
-byte[] viewtenantId = viewInfo.getTenantId();
-byte[] viewSchema = viewInfo.getSchemaName();
-byte[] viewTable = viewInfo.getViewName();
-byte[] tableKey = SchemaUtil.getTableKey(viewtenantId, 
viewSchema, viewTable);
-ImmutableBytesPtr cacheKey = new ImmutableBytesPtr(tableKey);
-PTable view = loadTable(env, tableKey, cacheKey, 
clientTimeStamp, clientTimeStamp, clientVersion);
-if (view == null) {
-logger.warn("Found orphan tenant view row in 
SYSTEM.CATALOG with tenantId:"
-+ Bytes.toString(tenantId) + ", schema:"
-+ Bytes.toString(viewSchema) + ", table:"
-+ Bytes.toString(viewTable));
-continue;
-}
-findAllChildViews(region, viewtenantId, view, result, 
clientTimeStamp, clientVersion);
-}
-}
-
-// TODO use child link instead once splittable system catalog 
(PHOENIX-3534) is implemented
-// and we have a separate table for links.
-private TableViewFinder findChildViews_deprecated(Region region, 
byte[] tenantId, PTable table, byte[] linkTypeBytes, boolean stopAfterFirst) 
throws IOException {
-byte[] schemaName = table.getSchemaName().getBytes();
-byte[] tableName = table.getTableName().getBytes();
-boolean isMultiTenant = table.isMultiTenant();

[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540420#comment-16540420
 ] 

ASF GitHub Bot commented on PHOENIX-3534:
-

Github user twdsilva commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/303#discussion_r201776288
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
 ---
@@ -447,7 +447,7 @@
 static {
 Collections.sort(FUNCTION_KV_COLUMNS, KeyValue.COMPARATOR);
 }
-
+
--- End diff --

I modified the class level comment.


> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Fix For: 5.0.0, 4.15.0
>
> Attachments: PHOENIX-3534.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region 
> based on the server-side row locks being held for operations that impact a 
> table and all of it's views. For example, adding/removing a column from a 
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), 
> when a new table is created we can do a lazy cleanup  of any rows that may be 
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with 
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on 
> checking our sequence numbers for each table/view before/after updating. Each 
> table/view row would be individually locked for their change (metadata for a 
> view or table cannot span regions due to our split policy), with the sequence 
> number being incremented under lock and then returned to the client.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540417#comment-16540417
 ] 

ASF GitHub Bot commented on PHOENIX-3534:
-

Github user twdsilva commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/303#discussion_r201776136
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
 ---
@@ -1457,28 +1791,110 @@ private static void getSchemaTableNames(Mutation 
row, byte[][] schemaTableNames)
 schemaTableNames[2] = tName;
 }
 }
-
+
 @Override
 public void createTable(RpcController controller, CreateTableRequest 
request,
 RpcCallback done) {
 MetaDataResponse.Builder builder = MetaDataResponse.newBuilder();
 byte[][] rowKeyMetaData = new byte[3][];
 byte[] schemaName = null;
 byte[] tableName = null;
+String fullTableName = null;
 try {
 int clientVersion = request.getClientVersion();
 List tableMetadata = 
ProtobufUtil.getMutations(request);
 MetaDataUtil.getTenantIdAndSchemaAndTableName(tableMetadata, 
rowKeyMetaData);
 byte[] tenantIdBytes = 
rowKeyMetaData[PhoenixDatabaseMetaData.TENANT_ID_INDEX];
 schemaName = 
rowKeyMetaData[PhoenixDatabaseMetaData.SCHEMA_NAME_INDEX];
 tableName = 
rowKeyMetaData[PhoenixDatabaseMetaData.TABLE_NAME_INDEX];
+fullTableName = SchemaUtil.getTableName(schemaName, tableName);
+// TODO before creating a table we need to see if the table 
was previously created and then dropped
+// and clean up any parent->child links or child views
 boolean isNamespaceMapped = 
MetaDataUtil.isNameSpaceMapped(tableMetadata, GenericKeyValueBuilder.INSTANCE,
 new ImmutableBytesWritable());
 final IndexType indexType = 
MetaDataUtil.getIndexType(tableMetadata, GenericKeyValueBuilder.INSTANCE,
 new ImmutableBytesWritable());
+byte[] parentTenantId = null;
 byte[] parentSchemaName = null;
 byte[] parentTableName = null;
 PTableType tableType = 
MetaDataUtil.getTableType(tableMetadata, GenericKeyValueBuilder.INSTANCE, new 
ImmutableBytesWritable());
+ViewType viewType = MetaDataUtil.getViewType(tableMetadata, 
GenericKeyValueBuilder.INSTANCE, new ImmutableBytesWritable());
+
+// Load table to see if it already exists
+byte[] tableKey = SchemaUtil.getTableKey(tenantIdBytes, 
schemaName, tableName);
+ImmutableBytesPtr cacheKey = new ImmutableBytesPtr(tableKey);
+long clientTimeStamp = 
MetaDataUtil.getClientTimeStamp(tableMetadata);
+PTable table = null;
+   try {
+   // Get as of latest timestamp so we can detect 
if we have a newer table that already
+   // exists without making an additional query
+   table = loadTable(env, tableKey, cacheKey, 
clientTimeStamp, HConstants.LATEST_TIMESTAMP,
+   clientVersion);
+   } catch (ParentTableNotFoundException e) {
+   dropChildMetadata(e.getParentSchemaName(), 
e.getParentTableName(), e.getParentTenantId());
+   }
+if (table != null) {
+if (table.getTimeStamp() < clientTimeStamp) {
+// If the table is older than the client time stamp 
and it's deleted,
+// continue
+if (!isTableDeleted(table)) {
+
builder.setReturnCode(MetaDataProtos.MutationCode.TABLE_ALREADY_EXISTS);
+
builder.setMutationTime(EnvironmentEdgeManager.currentTimeMillis());
+builder.setTable(PTableImpl.toProto(table));
+done.run(builder.build());
+return;
+}
+} else {
+
builder.setReturnCode(MetaDataProtos.MutationCode.NEWER_TABLE_FOUND);
+
builder.setMutationTime(EnvironmentEdgeManager.currentTimeMillis());
+builder.setTable(PTableImpl.toProto(table));
+done.run(builder.build());
+return;
+}
+}
+
+   // check if the table was dropped, but had child views 
that were have not yet
+   // been cleaned up by compaction
+   if 

[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540418#comment-16540418
 ] 

ASF GitHub Bot commented on PHOENIX-3534:
-

Github user twdsilva commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/303#discussion_r201776174
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
 ---
@@ -1488,20 +1904,19 @@ public void createTable(RpcController controller, 
CreateTableRequest request,
 if (tableType == PTableType.VIEW) {
 byte[][] parentSchemaTableNames = new byte[3][];
 byte[][] parentPhysicalSchemaTableNames = new byte[3][];
-/*
- * For a view, we lock the base physical table row. For a 
mapped view, there is 
- * no link present to the physical table. So the 
viewPhysicalTableRow is null
- * in that case.
- */
+   /*
+* For a mapped view, there is no link present 
to the physical table. So the
+* viewPhysicalTableRow is null in that case.
+*/
--- End diff --

Fixed.


> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Fix For: 5.0.0, 4.15.0
>
> Attachments: PHOENIX-3534.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region 
> based on the server-side row locks being held for operations that impact a 
> table and all of it's views. For example, adding/removing a column from a 
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), 
> when a new table is created we can do a lazy cleanup  of any rows that may be 
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with 
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on 
> checking our sequence numbers for each table/view before/after updating. Each 
> table/view row would be individually locked for their change (metadata for a 
> view or table cannot span regions due to our split policy), with the sequence 
> number being incremented under lock and then returned to the client.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] phoenix pull request #303: PHOENIX-3534 Support multi region SYSTEM.CATALOG ...

[twdsilva]

Github user twdsilva commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/303#discussion_r201776204
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
 ---
@@ -1809,180 +2256,97 @@ public void createTable(RpcController controller, 
CreateTableRequest request,
 } catch (Throwable t) {
 logger.error("createTable failed", t);
 ProtobufUtil.setControllerException(controller,
-
ServerUtil.createIOException(SchemaUtil.getTableName(schemaName, tableName), 
t));
+ServerUtil.createIOException(fullTableName, t));
 }
 }
 
+   private void dropChildMetadata(byte[] schemaName, byte[] tableName, 
byte[] tenantIdBytes)
+   throws IOException, SQLException, 
ClassNotFoundException {
+   TableViewFinderResult childViewsResult = new 
TableViewFinderResult();
+   findAllChildViews(tenantIdBytes, schemaName, tableName, 
childViewsResult);
+   if (childViewsResult.hasViews()) {
+   for (TableInfo viewInfo : 
childViewsResult.getResults()) {
+   byte[] viewTenantId = viewInfo.getTenantId();
+   byte[] viewSchemaName = 
viewInfo.getSchemaName();
+   byte[] viewName = viewInfo.getTableName();
+   Properties props = new Properties();
+   if (viewTenantId != null && viewTenantId.length 
!= 0)
+   
props.setProperty(PhoenixRuntime.TENANT_ID_ATTRIB, 
Bytes.toString(viewTenantId));
+   try (PhoenixConnection connection = 
QueryUtil.getConnectionOnServer(env.getConfiguration())
+   
.unwrap(PhoenixConnection.class)) {
+   MetaDataClient client = new 
MetaDataClient(connection);
+   org.apache.phoenix.parse.TableName 
viewTableName = org.apache.phoenix.parse.TableName
+   
.create(Bytes.toString(viewSchemaName), Bytes.toString(viewName));
+   client.dropTable(
+   new 
DropTableStatement(viewTableName, PTableType.VIEW, false, true, true));
+   }
+   }
+   }
+   }
+
 private boolean execeededIndexQuota(PTableType tableType, PTable 
parentTable) {
 return PTableType.INDEX == tableType && 
parentTable.getIndexes().size() >= maxIndexesPerTable;
 }
-
-private void findAllChildViews(Region region, byte[] tenantId, PTable 
table,
-TableViewFinder result, long clientTimeStamp, int 
clientVersion) throws IOException, SQLException {
-TableViewFinder currResult = findChildViews(region, tenantId, 
table, clientVersion, false);
-result.addResult(currResult);
-for (ViewInfo viewInfo : currResult.getViewInfoList()) {
-byte[] viewtenantId = viewInfo.getTenantId();
-byte[] viewSchema = viewInfo.getSchemaName();
-byte[] viewTable = viewInfo.getViewName();
-byte[] tableKey = SchemaUtil.getTableKey(viewtenantId, 
viewSchema, viewTable);
-ImmutableBytesPtr cacheKey = new ImmutableBytesPtr(tableKey);
-PTable view = loadTable(env, tableKey, cacheKey, 
clientTimeStamp, clientTimeStamp, clientVersion);
-if (view == null) {
-logger.warn("Found orphan tenant view row in 
SYSTEM.CATALOG with tenantId:"
-+ Bytes.toString(tenantId) + ", schema:"
-+ Bytes.toString(viewSchema) + ", table:"
-+ Bytes.toString(viewTable));
-continue;
-}
-findAllChildViews(region, viewtenantId, view, result, 
clientTimeStamp, clientVersion);
-}
-}
-
-// TODO use child link instead once splittable system catalog 
(PHOENIX-3534) is implemented
-// and we have a separate table for links.
-private TableViewFinder findChildViews_deprecated(Region region, 
byte[] tenantId, PTable table, byte[] linkTypeBytes, boolean stopAfterFirst) 
throws IOException {
-byte[] schemaName = table.getSchemaName().getBytes();
-byte[] tableName = table.getTableName().getBytes();
-boolean isMultiTenant = table.isMultiTenant();
-Scan scan = new Scan();
-// If the table is multi-tenant, we need to check across all 
tenant_ids,
-// so we can't constrain the row key. Otherwise, any views would 
have
-// the same tenantId.
-  

[GitHub] phoenix pull request #303: PHOENIX-3534 Support multi region SYSTEM.CATALOG ...

[twdsilva]

Github user twdsilva commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/303#discussion_r201776174
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
 ---
@@ -1488,20 +1904,19 @@ public void createTable(RpcController controller, 
CreateTableRequest request,
 if (tableType == PTableType.VIEW) {
 byte[][] parentSchemaTableNames = new byte[3][];
 byte[][] parentPhysicalSchemaTableNames = new byte[3][];
-/*
- * For a view, we lock the base physical table row. For a 
mapped view, there is 
- * no link present to the physical table. So the 
viewPhysicalTableRow is null
- * in that case.
- */
+   /*
+* For a mapped view, there is no link present 
to the physical table. So the
+* viewPhysicalTableRow is null in that case.
+*/
--- End diff --

Fixed.


---

[GitHub] phoenix pull request #303: PHOENIX-3534 Support multi region SYSTEM.CATALOG ...

[twdsilva]

Github user twdsilva commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/303#discussion_r201776136
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
 ---
@@ -1457,28 +1791,110 @@ private static void getSchemaTableNames(Mutation 
row, byte[][] schemaTableNames)
 schemaTableNames[2] = tName;
 }
 }
-
+
 @Override
 public void createTable(RpcController controller, CreateTableRequest 
request,
 RpcCallback done) {
 MetaDataResponse.Builder builder = MetaDataResponse.newBuilder();
 byte[][] rowKeyMetaData = new byte[3][];
 byte[] schemaName = null;
 byte[] tableName = null;
+String fullTableName = null;
 try {
 int clientVersion = request.getClientVersion();
 List tableMetadata = 
ProtobufUtil.getMutations(request);
 MetaDataUtil.getTenantIdAndSchemaAndTableName(tableMetadata, 
rowKeyMetaData);
 byte[] tenantIdBytes = 
rowKeyMetaData[PhoenixDatabaseMetaData.TENANT_ID_INDEX];
 schemaName = 
rowKeyMetaData[PhoenixDatabaseMetaData.SCHEMA_NAME_INDEX];
 tableName = 
rowKeyMetaData[PhoenixDatabaseMetaData.TABLE_NAME_INDEX];
+fullTableName = SchemaUtil.getTableName(schemaName, tableName);
+// TODO before creating a table we need to see if the table 
was previously created and then dropped
+// and clean up any parent->child links or child views
 boolean isNamespaceMapped = 
MetaDataUtil.isNameSpaceMapped(tableMetadata, GenericKeyValueBuilder.INSTANCE,
 new ImmutableBytesWritable());
 final IndexType indexType = 
MetaDataUtil.getIndexType(tableMetadata, GenericKeyValueBuilder.INSTANCE,
 new ImmutableBytesWritable());
+byte[] parentTenantId = null;
 byte[] parentSchemaName = null;
 byte[] parentTableName = null;
 PTableType tableType = 
MetaDataUtil.getTableType(tableMetadata, GenericKeyValueBuilder.INSTANCE, new 
ImmutableBytesWritable());
+ViewType viewType = MetaDataUtil.getViewType(tableMetadata, 
GenericKeyValueBuilder.INSTANCE, new ImmutableBytesWritable());
+
+// Load table to see if it already exists
+byte[] tableKey = SchemaUtil.getTableKey(tenantIdBytes, 
schemaName, tableName);
+ImmutableBytesPtr cacheKey = new ImmutableBytesPtr(tableKey);
+long clientTimeStamp = 
MetaDataUtil.getClientTimeStamp(tableMetadata);
+PTable table = null;
+   try {
+   // Get as of latest timestamp so we can detect 
if we have a newer table that already
+   // exists without making an additional query
+   table = loadTable(env, tableKey, cacheKey, 
clientTimeStamp, HConstants.LATEST_TIMESTAMP,
+   clientVersion);
+   } catch (ParentTableNotFoundException e) {
+   dropChildMetadata(e.getParentSchemaName(), 
e.getParentTableName(), e.getParentTenantId());
+   }
+if (table != null) {
+if (table.getTimeStamp() < clientTimeStamp) {
+// If the table is older than the client time stamp 
and it's deleted,
+// continue
+if (!isTableDeleted(table)) {
+
builder.setReturnCode(MetaDataProtos.MutationCode.TABLE_ALREADY_EXISTS);
+
builder.setMutationTime(EnvironmentEdgeManager.currentTimeMillis());
+builder.setTable(PTableImpl.toProto(table));
+done.run(builder.build());
+return;
+}
+} else {
+
builder.setReturnCode(MetaDataProtos.MutationCode.NEWER_TABLE_FOUND);
+
builder.setMutationTime(EnvironmentEdgeManager.currentTimeMillis());
+builder.setTable(PTableImpl.toProto(table));
+done.run(builder.build());
+return;
+}
+}
+
+   // check if the table was dropped, but had child views 
that were have not yet
+   // been cleaned up by compaction
+   if 
(!Bytes.toString(schemaName).equals(QueryConstants.SYSTEM_SCHEMA_NAME)) {
+   dropChildMetadata(schemaName, tableName, 
tenantIdBytes);
+   }
--- End diff --

Fixed.


---

[GitHub] phoenix pull request #303: PHOENIX-3534 Support multi region SYSTEM.CATALOG ...

[twdsilva]

Github user twdsilva commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/303#discussion_r201776009
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
 ---
@@ -586,48 +590,359 @@ public void getTable(RpcController controller, 
GetTableRequest request,
 
builder.setMutationTime(minNonZerodisableIndexTimestamp - 1);
 }
 }
-
-if (table.getTimeStamp() != tableTimeStamp) {
+// the PTable of views and indexes on views might get updated 
because a column is added to one of
+// their parents (this won't change the timestamp)
+if (table.getType()!=PTableType.TABLE || table.getTimeStamp() 
!= tableTimeStamp) {
 builder.setTable(PTableImpl.toProto(table));
 }
 done.run(builder.build());
-return;
 } catch (Throwable t) {
 logger.error("getTable failed", t);
 ProtobufUtil.setControllerException(controller,
 
ServerUtil.createIOException(SchemaUtil.getTableName(schemaName, tableName), 
t));
 }
 }
 
+/**
+ * Used to add the columns present the ancestor hierarchy to the 
PTable of the given view or
+ * view index
+ * @param table PTable of the view or view index
+ * @param skipAddingIndexes if true the returned PTable won't include 
indexes
+ * @param skipAddingParentColumns if true the returned PTable won't 
include columns derived from
+ *ancestor tables
+ * @param lockedAncestorTable ancestor table table that is being 
mutated (as we won't be able to
+ *resolve this table as its locked)
+ */
+private Pair 
combineColumns(PTable table, long timestamp,
+int clientVersion, boolean skipAddingIndexes, boolean 
skipAddingParentColumns,
+PTable lockedAncestorTable) throws SQLException, IOException {
+boolean hasIndexId = table.getViewIndexId() != null;
+if (table.getType() != PTableType.VIEW && !hasIndexId) {
+return new Pair(table,
+MetaDataProtos.MutationCode.TABLE_ALREADY_EXISTS);
+}
+if (!skipAddingParentColumns) {
+table =
+addDerivedColumnsFromAncestors(table, timestamp, 
clientVersion,
+lockedAncestorTable);
+if (table==null) {
+return new Pair(table,
+MetaDataProtos.MutationCode.TABLE_NOT_FOUND);
+}
+// we need to resolve the indexes of views (to get ensure they 
also have all the columns
+// derived from their ancestors) 
+if (!skipAddingIndexes && !table.getIndexes().isEmpty()) {
+List indexes = 
Lists.newArrayListWithExpectedSize(table.getIndexes().size());
+for (PTable index : table.getIndexes()) {
+byte[] tenantIdBytes =
+index.getTenantId() == null ? 
ByteUtil.EMPTY_BYTE_ARRAY
+: index.getTenantId().getBytes();
+PTable latestIndex =
+doGetTable(tenantIdBytes, 
index.getSchemaName().getBytes(),
+index.getTableName().getBytes(), 
timestamp, null, clientVersion, true,
+false, lockedAncestorTable);
+if (latestIndex == null) {
+throw new TableNotFoundException(
+"Could not find index table while 
combining columns "
++ index.getTableName().getString() 
+ " with tenant id "
++ index.getTenantId());
+}
+indexes.add(latestIndex);
+}
+table = PTableImpl.makePTable(table, table.getTimeStamp(), 
indexes);
+}
+}
+
+MetaDataProtos.MutationCode mutationCode =
+table != null ? 
MetaDataProtos.MutationCode.TABLE_ALREADY_EXISTS
+: MetaDataProtos.MutationCode.TABLE_NOT_FOUND;
+return new Pair(table, 
mutationCode);
+}
+
+
+private PTable addDerivedColumnsFromAncestors(PTable table, long 
timestamp,
+int clientVersion, PTable lockedAncestorTable) throws 
IOException, SQLException, TableNotFoundException {
+// combine columns for view and view indexes
+byte[] tenantId =
+table.getTenantId() != null ? 
table.getTenantId().getBytes()
  

[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540414#comment-16540414
 ] 

ASF GitHub Bot commented on PHOENIX-3534:
-

Github user twdsilva commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/303#discussion_r201776009
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
 ---
@@ -586,48 +590,359 @@ public void getTable(RpcController controller, 
GetTableRequest request,
 
builder.setMutationTime(minNonZerodisableIndexTimestamp - 1);
 }
 }
-
-if (table.getTimeStamp() != tableTimeStamp) {
+// the PTable of views and indexes on views might get updated 
because a column is added to one of
+// their parents (this won't change the timestamp)
+if (table.getType()!=PTableType.TABLE || table.getTimeStamp() 
!= tableTimeStamp) {
 builder.setTable(PTableImpl.toProto(table));
 }
 done.run(builder.build());
-return;
 } catch (Throwable t) {
 logger.error("getTable failed", t);
 ProtobufUtil.setControllerException(controller,
 
ServerUtil.createIOException(SchemaUtil.getTableName(schemaName, tableName), 
t));
 }
 }
 
+/**
+ * Used to add the columns present the ancestor hierarchy to the 
PTable of the given view or
+ * view index
+ * @param table PTable of the view or view index
+ * @param skipAddingIndexes if true the returned PTable won't include 
indexes
+ * @param skipAddingParentColumns if true the returned PTable won't 
include columns derived from
+ *ancestor tables
+ * @param lockedAncestorTable ancestor table table that is being 
mutated (as we won't be able to
+ *resolve this table as its locked)
+ */
+private Pair 
combineColumns(PTable table, long timestamp,
+int clientVersion, boolean skipAddingIndexes, boolean 
skipAddingParentColumns,
+PTable lockedAncestorTable) throws SQLException, IOException {
+boolean hasIndexId = table.getViewIndexId() != null;
+if (table.getType() != PTableType.VIEW && !hasIndexId) {
+return new Pair(table,
+MetaDataProtos.MutationCode.TABLE_ALREADY_EXISTS);
+}
+if (!skipAddingParentColumns) {
+table =
+addDerivedColumnsFromAncestors(table, timestamp, 
clientVersion,
+lockedAncestorTable);
+if (table==null) {
+return new Pair(table,
+MetaDataProtos.MutationCode.TABLE_NOT_FOUND);
+}
+// we need to resolve the indexes of views (to get ensure they 
also have all the columns
+// derived from their ancestors) 
+if (!skipAddingIndexes && !table.getIndexes().isEmpty()) {
+List indexes = 
Lists.newArrayListWithExpectedSize(table.getIndexes().size());
+for (PTable index : table.getIndexes()) {
+byte[] tenantIdBytes =
+index.getTenantId() == null ? 
ByteUtil.EMPTY_BYTE_ARRAY
+: index.getTenantId().getBytes();
+PTable latestIndex =
+doGetTable(tenantIdBytes, 
index.getSchemaName().getBytes(),
+index.getTableName().getBytes(), 
timestamp, null, clientVersion, true,
+false, lockedAncestorTable);
+if (latestIndex == null) {
+throw new TableNotFoundException(
+"Could not find index table while 
combining columns "
++ index.getTableName().getString() 
+ " with tenant id "
++ index.getTenantId());
+}
+indexes.add(latestIndex);
+}
+table = PTableImpl.makePTable(table, table.getTimeStamp(), 
indexes);
+}
+}
+
+MetaDataProtos.MutationCode mutationCode =
+table != null ? 
MetaDataProtos.MutationCode.TABLE_ALREADY_EXISTS
+: MetaDataProtos.MutationCode.TABLE_NOT_FOUND;
+return new Pair(table, 
mutationCode);
+}
+
+
+private PTable addDerivedColumnsFromAncestors(PTable table, long 
timestamp,
+int clientVersion, PTable 

[GitHub] phoenix pull request #303: PHOENIX-3534 Support multi region SYSTEM.CATALOG ...

[twdsilva]

Github user twdsilva commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/303#discussion_r201776046
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
 ---
@@ -586,48 +590,359 @@ public void getTable(RpcController controller, 
GetTableRequest request,
 
builder.setMutationTime(minNonZerodisableIndexTimestamp - 1);
 }
 }
-
-if (table.getTimeStamp() != tableTimeStamp) {
+// the PTable of views and indexes on views might get updated 
because a column is added to one of
+// their parents (this won't change the timestamp)
+if (table.getType()!=PTableType.TABLE || table.getTimeStamp() 
!= tableTimeStamp) {
 builder.setTable(PTableImpl.toProto(table));
 }
 done.run(builder.build());
-return;
 } catch (Throwable t) {
 logger.error("getTable failed", t);
 ProtobufUtil.setControllerException(controller,
 
ServerUtil.createIOException(SchemaUtil.getTableName(schemaName, tableName), 
t));
 }
 }
 
+/**
+ * Used to add the columns present the ancestor hierarchy to the 
PTable of the given view or
+ * view index
+ * @param table PTable of the view or view index
+ * @param skipAddingIndexes if true the returned PTable won't include 
indexes
+ * @param skipAddingParentColumns if true the returned PTable won't 
include columns derived from
+ *ancestor tables
+ * @param lockedAncestorTable ancestor table table that is being 
mutated (as we won't be able to
+ *resolve this table as its locked)
+ */
+private Pair 
combineColumns(PTable table, long timestamp,
+int clientVersion, boolean skipAddingIndexes, boolean 
skipAddingParentColumns,
+PTable lockedAncestorTable) throws SQLException, IOException {
+boolean hasIndexId = table.getViewIndexId() != null;
+if (table.getType() != PTableType.VIEW && !hasIndexId) {
+return new Pair(table,
+MetaDataProtos.MutationCode.TABLE_ALREADY_EXISTS);
+}
+if (!skipAddingParentColumns) {
+table =
+addDerivedColumnsFromAncestors(table, timestamp, 
clientVersion,
+lockedAncestorTable);
+if (table==null) {
+return new Pair(table,
+MetaDataProtos.MutationCode.TABLE_NOT_FOUND);
+}
+// we need to resolve the indexes of views (to get ensure they 
also have all the columns
+// derived from their ancestors) 
+if (!skipAddingIndexes && !table.getIndexes().isEmpty()) {
+List indexes = 
Lists.newArrayListWithExpectedSize(table.getIndexes().size());
+for (PTable index : table.getIndexes()) {
+byte[] tenantIdBytes =
+index.getTenantId() == null ? 
ByteUtil.EMPTY_BYTE_ARRAY
+: index.getTenantId().getBytes();
+PTable latestIndex =
+doGetTable(tenantIdBytes, 
index.getSchemaName().getBytes(),
+index.getTableName().getBytes(), 
timestamp, null, clientVersion, true,
+false, lockedAncestorTable);
+if (latestIndex == null) {
+throw new TableNotFoundException(
+"Could not find index table while 
combining columns "
++ index.getTableName().getString() 
+ " with tenant id "
++ index.getTenantId());
+}
+indexes.add(latestIndex);
+}
+table = PTableImpl.makePTable(table, table.getTimeStamp(), 
indexes);
+}
+}
+
+MetaDataProtos.MutationCode mutationCode =
+table != null ? 
MetaDataProtos.MutationCode.TABLE_ALREADY_EXISTS
+: MetaDataProtos.MutationCode.TABLE_NOT_FOUND;
+return new Pair(table, 
mutationCode);
+}
+
+
+private PTable addDerivedColumnsFromAncestors(PTable table, long 
timestamp,
+int clientVersion, PTable lockedAncestorTable) throws 
IOException, SQLException, TableNotFoundException {
+// combine columns for view and view indexes
+byte[] tenantId =
+table.getTenantId() != null ? 
table.getTenantId().getBytes()
  

[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540284#comment-16540284
 ] 

ASF GitHub Bot commented on PHOENIX-3534:
-

Github user twdsilva commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/303#discussion_r201748315
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/schema/MetaDataClient.java ---
@@ -1957,6 +1968,17 @@ private PTable 
createTableInternal(CreateTableStatement statement, byte[][] spli
 linkStatement.setLong(6, parent.getSequenceNumber());
 linkStatement.setString(7, 
PTableType.INDEX.getSerializedValue());
 linkStatement.execute();
+
+// Add row linking index table to parent table for indexes 
on views
+if (parent.getType() == PTableType.VIEW) {
+   linkStatement = 
connection.prepareStatement(CREATE_VIEW_INDEX_PARENT_LINK);
+   linkStatement.setString(1, tenantIdStr);
+   linkStatement.setString(2, schemaName);
+   linkStatement.setString(3, tableName);
+   linkStatement.setString(4, 
parent.getName().getString());
+   linkStatement.setByte(5, 
LinkType.VIEW_INDEX_PARENT_TABLE.getSerializedValue());
+   linkStatement.execute();
+}
--- End diff --

I created PHOENIX-4766 for this, I will add a comment referencing this JIRA 
in createTableInternal().


> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Fix For: 5.0.0, 4.15.0
>
> Attachments: PHOENIX-3534.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region 
> based on the server-side row locks being held for operations that impact a 
> table and all of it's views. For example, adding/removing a column from a 
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), 
> when a new table is created we can do a lazy cleanup  of any rows that may be 
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with 
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on 
> checking our sequence numbers for each table/view before/after updating. Each 
> table/view row would be individually locked for their change (metadata for a 
> view or table cannot span regions due to our split policy), with the sequence 
> number being incremented under lock and then returned to the client.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540272#comment-16540272
 ] 

ASF GitHub Bot commented on PHOENIX-3534:
-

Github user twdsilva commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/303#discussion_r201744675
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
 ---
@@ -2573,307 +2897,139 @@ else if (pkCount <= COLUMN_NAME_INDEX
 return new 
MetaDataMutationResult(MutationCode.UNALLOWED_TABLE_MUTATION, 
EnvironmentEdgeManager.currentTimeMillis(), basePhysicalTable);
 }
 
-ColumnOrdinalPositionUpdateList ordinalPositionList = new 
ColumnOrdinalPositionUpdateList();
+//add the new columns to the child view
 List viewPkCols = new 
ArrayList<>(view.getPKColumns());
 boolean addingExistingPkCol = false;
-int numCols = view.getColumns().size();
-// add the new columns to the child view
-for (PutWithOrdinalPosition p : columnPutsForBaseTable) {
-Put baseTableColumnPut = p.put;
+for (Put columnToBeAdded : columnPutsForBaseTable) {
 PColumn existingViewColumn = null;
 byte[][] rkmd = new byte[5][];
-getVarChars(baseTableColumnPut.getRow(), rkmd);
+getVarChars(columnToBeAdded.getRow(), rkmd);
 String columnName = 
Bytes.toString(rkmd[COLUMN_NAME_INDEX]);
-String columnFamily = rkmd[FAMILY_NAME_INDEX] == null ? 
null : Bytes.toString(rkmd[FAMILY_NAME_INDEX]);
+String columnFamily =
+rkmd[FAMILY_NAME_INDEX] == null ? null
+: Bytes.toString(rkmd[FAMILY_NAME_INDEX]);
 try {
-existingViewColumn = columnFamily == null ? 
view.getColumnForColumnName(columnName) : view.getColumnFamily(
-
columnFamily).getPColumnForColumnName(columnName);
+existingViewColumn =
+columnFamily == null ? 
view.getColumnForColumnName(columnName)
+: view.getColumnFamily(columnFamily)
+
.getPColumnForColumnName(columnName);
 } catch (ColumnFamilyNotFoundException e) {
-// ignore since it means that the column family is not 
present for the column to be added.
+// ignore since it means that the column family is not 
present for the column to
+// be added.
 } catch (ColumnNotFoundException e) {
 // ignore since it means the column is not present in 
the view
 }
-
-boolean isPkCol = columnFamily == null;
-byte[] columnKey = getColumnKey(viewKey, columnName, 
columnFamily);
+
+boolean isColumnToBeAddPkCol = columnFamily == null;
 if (existingViewColumn != null) {
-MetaDataMutationResult result = 
validateColumnForAddToBaseTable(existingViewColumn, baseTableColumnPut, 
basePhysicalTable, isPkCol, view);
-if (result != null) {
-return result;
+if 
(EncodedColumnsUtil.usesEncodedColumnNames(basePhysicalTable)
+&& !SchemaUtil.isPKColumn(existingViewColumn)) 
{
--- End diff --

The race condition with adding the same column to the base table and a 
child view will be covered in PHOENIX-4799.


> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Fix For: 5.0.0, 4.15.0
>
> Attachments: PHOENIX-3534.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region 
> based on the server-side row locks being held for operations that impact a 
> table and all of it's views. For example, adding/removing a column from a 
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), 
> when a new table is created we can do a lazy cleanup  of any rows that may be 
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with 
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 
> so that we 

[GitHub] phoenix pull request #303: PHOENIX-3534 Support multi region SYSTEM.CATALOG ...

[twdsilva]

Github user twdsilva commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/303#discussion_r201744675
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
 ---
@@ -2573,307 +2897,139 @@ else if (pkCount <= COLUMN_NAME_INDEX
 return new 
MetaDataMutationResult(MutationCode.UNALLOWED_TABLE_MUTATION, 
EnvironmentEdgeManager.currentTimeMillis(), basePhysicalTable);
 }
 
-ColumnOrdinalPositionUpdateList ordinalPositionList = new 
ColumnOrdinalPositionUpdateList();
+//add the new columns to the child view
 List viewPkCols = new 
ArrayList<>(view.getPKColumns());
 boolean addingExistingPkCol = false;
-int numCols = view.getColumns().size();
-// add the new columns to the child view
-for (PutWithOrdinalPosition p : columnPutsForBaseTable) {
-Put baseTableColumnPut = p.put;
+for (Put columnToBeAdded : columnPutsForBaseTable) {
 PColumn existingViewColumn = null;
 byte[][] rkmd = new byte[5][];
-getVarChars(baseTableColumnPut.getRow(), rkmd);
+getVarChars(columnToBeAdded.getRow(), rkmd);
 String columnName = 
Bytes.toString(rkmd[COLUMN_NAME_INDEX]);
-String columnFamily = rkmd[FAMILY_NAME_INDEX] == null ? 
null : Bytes.toString(rkmd[FAMILY_NAME_INDEX]);
+String columnFamily =
+rkmd[FAMILY_NAME_INDEX] == null ? null
+: Bytes.toString(rkmd[FAMILY_NAME_INDEX]);
 try {
-existingViewColumn = columnFamily == null ? 
view.getColumnForColumnName(columnName) : view.getColumnFamily(
-
columnFamily).getPColumnForColumnName(columnName);
+existingViewColumn =
+columnFamily == null ? 
view.getColumnForColumnName(columnName)
+: view.getColumnFamily(columnFamily)
+
.getPColumnForColumnName(columnName);
 } catch (ColumnFamilyNotFoundException e) {
-// ignore since it means that the column family is not 
present for the column to be added.
+// ignore since it means that the column family is not 
present for the column to
+// be added.
 } catch (ColumnNotFoundException e) {
 // ignore since it means the column is not present in 
the view
 }
-
-boolean isPkCol = columnFamily == null;
-byte[] columnKey = getColumnKey(viewKey, columnName, 
columnFamily);
+
+boolean isColumnToBeAddPkCol = columnFamily == null;
 if (existingViewColumn != null) {
-MetaDataMutationResult result = 
validateColumnForAddToBaseTable(existingViewColumn, baseTableColumnPut, 
basePhysicalTable, isPkCol, view);
-if (result != null) {
-return result;
+if 
(EncodedColumnsUtil.usesEncodedColumnNames(basePhysicalTable)
+&& !SchemaUtil.isPKColumn(existingViewColumn)) 
{
--- End diff --

The race condition with adding the same column to the base table and a 
child view will be covered in PHOENIX-4799.


---

[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540261#comment-16540261
 ] 

ASF GitHub Bot commented on PHOENIX-3534:
-

Github user twdsilva commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/303#discussion_r201741221
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
 ---
@@ -2147,46 +2566,29 @@ private MetaDataMutationResult doDropTable(byte[] 
key, byte[] tenantId, byte[] s
 }
 
 if (tableType == PTableType.TABLE || tableType == 
PTableType.SYSTEM) {
-// Handle any child views that exist
-TableViewFinder tableViewFinderResult = 
findChildViews(region, tenantId, table, clientVersion, !isCascade);
-if (tableViewFinderResult.hasViews()) {
-if (isCascade) {
-if 
(tableViewFinderResult.allViewsInMultipleRegions()) {
-// We don't yet support deleting a table with 
views where SYSTEM.CATALOG has split and the
-// view metadata spans multiple regions
-return new 
MetaDataMutationResult(MutationCode.UNALLOWED_TABLE_MUTATION,
-
EnvironmentEdgeManager.currentTimeMillis(), null);
-} else if 
(tableViewFinderResult.allViewsInSingleRegion()) {
-// Recursively delete views - safe as all the 
views as all in the same region
-for (ViewInfo viewInfo : 
tableViewFinderResult.getViewInfoList()) {
-byte[] viewTenantId = 
viewInfo.getTenantId();
-byte[] viewSchemaName = 
viewInfo.getSchemaName();
-byte[] viewName = viewInfo.getViewName();
-byte[] viewKey = 
SchemaUtil.getTableKey(viewTenantId, viewSchemaName, viewName);
-Delete delete = new Delete(viewKey, 
clientTimeStamp);
-rowsToDelete.add(delete);
-acquireLock(region, viewKey, locks);
-MetaDataMutationResult result = 
doDropTable(viewKey, viewTenantId, viewSchemaName,
-viewName, null, PTableType.VIEW, 
rowsToDelete, invalidateList, locks,
-tableNamesToDelete, 
sharedTablesToDelete, false, clientVersion);
-if (result.getMutationCode() != 
MutationCode.TABLE_ALREADY_EXISTS) { return result; }
-}
-}
-} else {
+// check to see if the table has any child views
+try (Table hTable =
+env.getTable(SchemaUtil.getPhysicalTableName(
+
PhoenixDatabaseMetaData.SYSTEM_CHILD_LINK_NAME_BYTES,
+env.getConfiguration( {
+boolean hasChildViews =
+ViewFinder.hasChildViews(hTable, tenantId, 
schemaName, tableName,
+clientTimeStamp);
+if (hasChildViews && !isCascade) {
--- End diff --

I think we can handle the race condition in a similar way to how we handle 
conflicting columns using checkAndMutate. I have updated PHOENIX-4799 to 
include this case.


> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Fix For: 5.0.0, 4.15.0
>
> Attachments: PHOENIX-3534.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region 
> based on the server-side row locks being held for operations that impact a 
> table and all of it's views. For example, adding/removing a column from a 
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), 
> when a new table is created we can do a lazy cleanup  of any rows that may be 
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with 
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on 
> checking our sequence numbers for each table/view before/after 

[GitHub] phoenix pull request #303: PHOENIX-3534 Support multi region SYSTEM.CATALOG ...

[twdsilva]

Github user twdsilva commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/303#discussion_r201741221
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
 ---
@@ -2147,46 +2566,29 @@ private MetaDataMutationResult doDropTable(byte[] 
key, byte[] tenantId, byte[] s
 }
 
 if (tableType == PTableType.TABLE || tableType == 
PTableType.SYSTEM) {
-// Handle any child views that exist
-TableViewFinder tableViewFinderResult = 
findChildViews(region, tenantId, table, clientVersion, !isCascade);
-if (tableViewFinderResult.hasViews()) {
-if (isCascade) {
-if 
(tableViewFinderResult.allViewsInMultipleRegions()) {
-// We don't yet support deleting a table with 
views where SYSTEM.CATALOG has split and the
-// view metadata spans multiple regions
-return new 
MetaDataMutationResult(MutationCode.UNALLOWED_TABLE_MUTATION,
-
EnvironmentEdgeManager.currentTimeMillis(), null);
-} else if 
(tableViewFinderResult.allViewsInSingleRegion()) {
-// Recursively delete views - safe as all the 
views as all in the same region
-for (ViewInfo viewInfo : 
tableViewFinderResult.getViewInfoList()) {
-byte[] viewTenantId = 
viewInfo.getTenantId();
-byte[] viewSchemaName = 
viewInfo.getSchemaName();
-byte[] viewName = viewInfo.getViewName();
-byte[] viewKey = 
SchemaUtil.getTableKey(viewTenantId, viewSchemaName, viewName);
-Delete delete = new Delete(viewKey, 
clientTimeStamp);
-rowsToDelete.add(delete);
-acquireLock(region, viewKey, locks);
-MetaDataMutationResult result = 
doDropTable(viewKey, viewTenantId, viewSchemaName,
-viewName, null, PTableType.VIEW, 
rowsToDelete, invalidateList, locks,
-tableNamesToDelete, 
sharedTablesToDelete, false, clientVersion);
-if (result.getMutationCode() != 
MutationCode.TABLE_ALREADY_EXISTS) { return result; }
-}
-}
-} else {
+// check to see if the table has any child views
+try (Table hTable =
+env.getTable(SchemaUtil.getPhysicalTableName(
+
PhoenixDatabaseMetaData.SYSTEM_CHILD_LINK_NAME_BYTES,
+env.getConfiguration( {
+boolean hasChildViews =
+ViewFinder.hasChildViews(hTable, tenantId, 
schemaName, tableName,
+clientTimeStamp);
+if (hasChildViews && !isCascade) {
--- End diff --

I think we can handle the race condition in a similar way to how we handle 
conflicting columns using checkAndMutate. I have updated PHOENIX-4799 to 
include this case.


---

[jira] [Updated] (PHOENIX-4799) Write cells using checkAndMutate to prevent conflicting changes

[Thomas D'Silva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas D'Silva updated PHOENIX-4799:

Description: 
In order to prevent race conditions when multiple client try to mutate the same 
column before sending the request to mutate the column to the server do a 
checkAndMutate with the column name being added/dropped. Also:
 1. When a view is created do a checkAndMutate with the columns in the view 
where clause.
 2. When an index on a view is created do a checkAndMutate with the indexed 
columns.

 

To prevent a race condition in the DROP TABLE CASCADE case, when a table is 
dropped do a checkAndMutate with the rowkey of the base table name. If a view 
is created it also does a checkAndMutate with the same rowkey. 

  was:
In order to prevent race conditions when multiple client try to mutate the same 
column before sending the request to mutate the column to the server do a 
checkAndMutate with the column name being added/dropped. Also:
1. When a view is created  do a checkAndMutate with the columns in the view 
where clause.
2. When an index on a view is created do a checkAndMutate with the indexed 
columns.






> Write cells using checkAndMutate to prevent conflicting changes
> ---
>
> Key: PHOENIX-4799
> URL: https://issues.apache.org/jira/browse/PHOENIX-4799
> Project: Phoenix
>  Issue Type: Sub-task
>Reporter: Thomas D'Silva
>Priority: Major
>
> In order to prevent race conditions when multiple client try to mutate the 
> same column before sending the request to mutate the column to the server do 
> a checkAndMutate with the column name being added/dropped. Also:
>  1. When a view is created do a checkAndMutate with the columns in the view 
> where clause.
>  2. When an index on a view is created do a checkAndMutate with the indexed 
> columns.
>  
> To prevent a race condition in the DROP TABLE CASCADE case, when a table is 
> dropped do a checkAndMutate with the rowkey of the base table name. If a view 
> is created it also does a checkAndMutate with the same rowkey. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540256#comment-16540256
 ] 

ASF GitHub Bot commented on PHOENIX-3534:
-

Github user twdsilva commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/303#discussion_r201739480
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
 ---
@@ -1809,180 +2256,97 @@ public void createTable(RpcController controller, 
CreateTableRequest request,
 } catch (Throwable t) {
 logger.error("createTable failed", t);
 ProtobufUtil.setControllerException(controller,
-
ServerUtil.createIOException(SchemaUtil.getTableName(schemaName, tableName), 
t));
+ServerUtil.createIOException(fullTableName, t));
 }
 }
 
+   private void dropChildMetadata(byte[] schemaName, byte[] tableName, 
byte[] tenantIdBytes)
+   throws IOException, SQLException, 
ClassNotFoundException {
+   TableViewFinderResult childViewsResult = new 
TableViewFinderResult();
+   findAllChildViews(tenantIdBytes, schemaName, tableName, 
childViewsResult);
+   if (childViewsResult.hasViews()) {
+   for (TableInfo viewInfo : 
childViewsResult.getResults()) {
+   byte[] viewTenantId = viewInfo.getTenantId();
+   byte[] viewSchemaName = 
viewInfo.getSchemaName();
+   byte[] viewName = viewInfo.getTableName();
+   Properties props = new Properties();
+   if (viewTenantId != null && viewTenantId.length 
!= 0)
+   
props.setProperty(PhoenixRuntime.TENANT_ID_ATTRIB, 
Bytes.toString(viewTenantId));
+   try (PhoenixConnection connection = 
QueryUtil.getConnectionOnServer(env.getConfiguration())
+   
.unwrap(PhoenixConnection.class)) {
+   MetaDataClient client = new 
MetaDataClient(connection);
+   org.apache.phoenix.parse.TableName 
viewTableName = org.apache.phoenix.parse.TableName
+   
.create(Bytes.toString(viewSchemaName), Bytes.toString(viewName));
+   client.dropTable(
+   new 
DropTableStatement(viewTableName, PTableType.VIEW, false, true, true));
+   }
+   }
+   }
+   }
+
 private boolean execeededIndexQuota(PTableType tableType, PTable 
parentTable) {
 return PTableType.INDEX == tableType && 
parentTable.getIndexes().size() >= maxIndexesPerTable;
 }
-
-private void findAllChildViews(Region region, byte[] tenantId, PTable 
table,
-TableViewFinder result, long clientTimeStamp, int 
clientVersion) throws IOException, SQLException {
-TableViewFinder currResult = findChildViews(region, tenantId, 
table, clientVersion, false);
-result.addResult(currResult);
-for (ViewInfo viewInfo : currResult.getViewInfoList()) {
-byte[] viewtenantId = viewInfo.getTenantId();
-byte[] viewSchema = viewInfo.getSchemaName();
-byte[] viewTable = viewInfo.getViewName();
-byte[] tableKey = SchemaUtil.getTableKey(viewtenantId, 
viewSchema, viewTable);
-ImmutableBytesPtr cacheKey = new ImmutableBytesPtr(tableKey);
-PTable view = loadTable(env, tableKey, cacheKey, 
clientTimeStamp, clientTimeStamp, clientVersion);
-if (view == null) {
-logger.warn("Found orphan tenant view row in 
SYSTEM.CATALOG with tenantId:"
-+ Bytes.toString(tenantId) + ", schema:"
-+ Bytes.toString(viewSchema) + ", table:"
-+ Bytes.toString(viewTable));
-continue;
-}
-findAllChildViews(region, viewtenantId, view, result, 
clientTimeStamp, clientVersion);
-}
-}
-
-// TODO use child link instead once splittable system catalog 
(PHOENIX-3534) is implemented
-// and we have a separate table for links.
-private TableViewFinder findChildViews_deprecated(Region region, 
byte[] tenantId, PTable table, byte[] linkTypeBytes, boolean stopAfterFirst) 
throws IOException {
-byte[] schemaName = table.getSchemaName().getBytes();
-byte[] tableName = table.getTableName().getBytes();
-boolean isMultiTenant = table.isMultiTenant();

[GitHub] phoenix pull request #303: PHOENIX-3534 Support multi region SYSTEM.CATALOG ...

[twdsilva]

Github user twdsilva commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/303#discussion_r201739480
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
 ---
@@ -1809,180 +2256,97 @@ public void createTable(RpcController controller, 
CreateTableRequest request,
 } catch (Throwable t) {
 logger.error("createTable failed", t);
 ProtobufUtil.setControllerException(controller,
-
ServerUtil.createIOException(SchemaUtil.getTableName(schemaName, tableName), 
t));
+ServerUtil.createIOException(fullTableName, t));
 }
 }
 
+   private void dropChildMetadata(byte[] schemaName, byte[] tableName, 
byte[] tenantIdBytes)
+   throws IOException, SQLException, 
ClassNotFoundException {
+   TableViewFinderResult childViewsResult = new 
TableViewFinderResult();
+   findAllChildViews(tenantIdBytes, schemaName, tableName, 
childViewsResult);
+   if (childViewsResult.hasViews()) {
+   for (TableInfo viewInfo : 
childViewsResult.getResults()) {
+   byte[] viewTenantId = viewInfo.getTenantId();
+   byte[] viewSchemaName = 
viewInfo.getSchemaName();
+   byte[] viewName = viewInfo.getTableName();
+   Properties props = new Properties();
+   if (viewTenantId != null && viewTenantId.length 
!= 0)
+   
props.setProperty(PhoenixRuntime.TENANT_ID_ATTRIB, 
Bytes.toString(viewTenantId));
+   try (PhoenixConnection connection = 
QueryUtil.getConnectionOnServer(env.getConfiguration())
+   
.unwrap(PhoenixConnection.class)) {
+   MetaDataClient client = new 
MetaDataClient(connection);
+   org.apache.phoenix.parse.TableName 
viewTableName = org.apache.phoenix.parse.TableName
+   
.create(Bytes.toString(viewSchemaName), Bytes.toString(viewName));
+   client.dropTable(
+   new 
DropTableStatement(viewTableName, PTableType.VIEW, false, true, true));
+   }
+   }
+   }
+   }
+
 private boolean execeededIndexQuota(PTableType tableType, PTable 
parentTable) {
 return PTableType.INDEX == tableType && 
parentTable.getIndexes().size() >= maxIndexesPerTable;
 }
-
-private void findAllChildViews(Region region, byte[] tenantId, PTable 
table,
-TableViewFinder result, long clientTimeStamp, int 
clientVersion) throws IOException, SQLException {
-TableViewFinder currResult = findChildViews(region, tenantId, 
table, clientVersion, false);
-result.addResult(currResult);
-for (ViewInfo viewInfo : currResult.getViewInfoList()) {
-byte[] viewtenantId = viewInfo.getTenantId();
-byte[] viewSchema = viewInfo.getSchemaName();
-byte[] viewTable = viewInfo.getViewName();
-byte[] tableKey = SchemaUtil.getTableKey(viewtenantId, 
viewSchema, viewTable);
-ImmutableBytesPtr cacheKey = new ImmutableBytesPtr(tableKey);
-PTable view = loadTable(env, tableKey, cacheKey, 
clientTimeStamp, clientTimeStamp, clientVersion);
-if (view == null) {
-logger.warn("Found orphan tenant view row in 
SYSTEM.CATALOG with tenantId:"
-+ Bytes.toString(tenantId) + ", schema:"
-+ Bytes.toString(viewSchema) + ", table:"
-+ Bytes.toString(viewTable));
-continue;
-}
-findAllChildViews(region, viewtenantId, view, result, 
clientTimeStamp, clientVersion);
-}
-}
-
-// TODO use child link instead once splittable system catalog 
(PHOENIX-3534) is implemented
-// and we have a separate table for links.
-private TableViewFinder findChildViews_deprecated(Region region, 
byte[] tenantId, PTable table, byte[] linkTypeBytes, boolean stopAfterFirst) 
throws IOException {
-byte[] schemaName = table.getSchemaName().getBytes();
-byte[] tableName = table.getTableName().getBytes();
-boolean isMultiTenant = table.isMultiTenant();
-Scan scan = new Scan();
-// If the table is multi-tenant, we need to check across all 
tenant_ids,
-// so we can't constrain the row key. Otherwise, any views would 
have
-// the same tenantId.
-  

[jira] [Commented] (PHOENIX-4688) Add kerberos authentication to python-phoenixdb

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540255#comment-16540255
 ] 

ASF GitHub Bot commented on PHOENIX-4688:
-

Github user pu239ppy commented on the issue:

https://github.com/apache/phoenix/pull/307
  
@joshelser If you are still unable to get .deactivate to work, remove it 
from the shell script for now and we can revisit it.  Again I am being overly 
pedantic here, but a the shell exits, we really do not need to clean up any 
environment.


> Add kerberos authentication to python-phoenixdb
> ---
>
> Key: PHOENIX-4688
> URL: https://issues.apache.org/jira/browse/PHOENIX-4688
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Lev Bronshtein
>Priority: Minor
>
> In its current state python-phoenixdv does not support support kerberos 
> authentication.  Using a modern python http library such as requests or 
> urllib it would be simple (if not trivial) to add this support.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] phoenix issue #307: PHOENIX-4688 Kerberize python phoenixdb

[pu239ppy]

Github user pu239ppy commented on the issue:

https://github.com/apache/phoenix/pull/307
  
@joshelser If you are still unable to get .deactivate to work, remove it 
from the shell script for now and we can revisit it.  Again I am being overly 
pedantic here, but a the shell exits, we really do not need to clean up any 
environment.


---

[jira] [Resolved] (PHOENIX-4587) Store child links in separate table from system catalog

[Thomas D'Silva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas D'Silva resolved PHOENIX-4587.
-
Resolution: Duplicate

> Store child links in separate table from system catalog
> ---
>
> Key: PHOENIX-4587
> URL: https://issues.apache.org/jira/browse/PHOENIX-4587
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: James Taylor
>Priority: Major
>
> Because there can be so many child links (in particular from a global table 
> or view), we should store them in a separate table without a split policy 
> that keeps them together.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-3955) Ensure KEEP_DELETED_CELLS, REPLICATION_SCOPE, and TTL properties stay in sync between the physical data table and index tables

[James Taylor (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-3955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540238#comment-16540238
 ] 

James Taylor commented on PHOENIX-3955:
---

You've made great progress on this one, [~ckulkarni]. Will you be able to see 
this one through? It's pretty high on the prioritized list here: 
https://docs.google.com/document/d/16BWU73qlnCQdxCflM3LScyIxE_OGi76ZYT2WHUfewiE

> Ensure KEEP_DELETED_CELLS, REPLICATION_SCOPE, and TTL properties stay in sync 
> between the physical data table and index tables
> --
>
> Key: PHOENIX-3955
> URL: https://issues.apache.org/jira/browse/PHOENIX-3955
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Samarth Jain
>Assignee: Chinmay Kulkarni
>Priority: Major
>
> We need to make sure that indexes inherit the REPLICATION_SCOPE, 
> KEEP_DELETED_CELLS and TTL properties from the base table. Otherwise we can 
> run into situations where the data was removed (or not removed) from the data 
> table but was removed (or not removed) from the index. Or vice-versa. We also 
> need to make sure that any ALTER TABLE SET TTL or ALTER TABLE SET 
> KEEP_DELETED_CELLS statements propagate the properties to the indexes too.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Help: setting hbase row timestamp in phoenix upserts ?

[Pedro Boado]

I hadn't seen this Jira. Yes that is essentially it.

On Wed, 11 Jul 2018, 15:49 James Taylor,  wrote:

> I think the answer is PHOENIX-4552. There's an outline of the work involved
> on the JIRA. I think passing through data like that for hints would get
> unwieldy quickly.
>
> On Tue, Jul 10, 2018 at 1:31 PM, Pedro Boado 
> wrote:
>
> > Hi guys, just a refloat from the @user list.
> >
> > May it be of interest having this functionality for defining HBase
> > timestamps in a per row basis as part of an UPSERT VALUES?
> >
> > For a table defined as
> > CREATE TABLE T0001 ( k VARCHAR PRIMARY KEY, v INTEGER)
> >
> > Allow a hint to extract and override hbase put timestamp through a
> > "virtual" column?
> > UPSERT /*+ ROW_TIMESTAMP(ts) */ INTO T0001(k,v,ts) VALUES
> > ('a',1, 1531253959043)
> >
> > If the column existed and had appropiate type it would also be populated
> > with the same value.
> >
> > Thanks,
> > Pedro.
> >
> >
> > On Fri, 1 Dec 2017 at 07:15, James Taylor 
> wrote:
> >
> > > The only way I can think of accomplishing this is by using the raw
> HBase
> > > APIs to write the data but using our utilities to write it in a Phoenix
> > > compatible manner. For example, you could run an UPSERT VALUES
> statement,
> > > use the PhoenixRuntime.getUncommittedDataIterator()method to get the
> > Cells
> > > that would have been written, update the Cell timestamp as needed, and
> do
> > > an htable.batch() call to commit them.
> > >
> > > On Wed, Nov 29, 2017 at 11:46 AM Pedro Boado 
> > > wrote:
> > >
> > >> Hi,
> > >>
> > >> I'm looking for a little bit of help trying to get some light over
> > >> ROW_TIMESTAMP.
> > >>
> > >> Some background over the problem ( simplified ) : I'm working in a
> > >> project that needs to create a "enriched" replica of a RBDMS table
> > based on
> > >> a stream of cdc changes off that table.
> > >>
> > >> Each cdc event contains the timestamp of the change plus all the
> column
> > >> values 'before' and 'after' the change . And each event is pushed to a
> > >> kafka topic.  Because of certain "non-negotiable" design decisions
> kafka
> > >> guarantees delivering each event at least once, but doesn't guarantee
> > >> ordering for changes over the same row in the source table.
> > >>
> > >> The final step of the kafka-based flow is sinking the information into
> > >> HBase/Phoenix.
> > >>
> > >> As I cannot get in order delivery guarantee from Kafka I need to use
> the
> > >> cdc event timestamp to ensure that HBase keeps the latest change over
> a
> > row.
> > >>
> > >> This fits perfectly well with an HBase table design with VERSIONS=1
> and
> > >> using the source event timestamp as HBase row/cells timestamp
> > >>
> > >> The thing is that I cannot find a way to define the value of the HBase
> > >> cell from a Phoenix upsert.
> > >>
> > >> I came across the ROW_TIMESTAMP functionality, but I've just found (
> I'm
> > >> devastated now ) that the ROW_TIMESTAMP columns store the date in both
> > >> hbase's cell timestamp and in the primary key, meaning that I cannot
> > >> leverage that functionality to keep only the latest change.
> > >>
> > >> Is there a way of defining hbase's row timestamp when doing the
> UPSERT -
> > >> even by setting it through some obscure hidden jdbc property - ?
> > >>
> > >> I want to avoid by all means doing a checkAndPut as the volume of
> > changes
> > >> is going to be quite bug.
> > >>
> > >>
> > >>
> > >> --
> > >> Un saludo.
> > >> Pedro Boado.
> > >>
> > >
> >
> > --
> > Un saludo.
> > Pedro Boado.
> >
>

Re: Help: setting hbase row timestamp in phoenix upserts ?

[James Taylor]

I think the answer is PHOENIX-4552. There's an outline of the work involved
on the JIRA. I think passing through data like that for hints would get
unwieldy quickly.

On Tue, Jul 10, 2018 at 1:31 PM, Pedro Boado  wrote:

> Hi guys, just a refloat from the @user list.
>
> May it be of interest having this functionality for defining HBase
> timestamps in a per row basis as part of an UPSERT VALUES?
>
> For a table defined as
> CREATE TABLE T0001 ( k VARCHAR PRIMARY KEY, v INTEGER)
>
> Allow a hint to extract and override hbase put timestamp through a
> "virtual" column?
> UPSERT /*+ ROW_TIMESTAMP(ts) */ INTO T0001(k,v,ts) VALUES
> ('a',1, 1531253959043)
>
> If the column existed and had appropiate type it would also be populated
> with the same value.
>
> Thanks,
> Pedro.
>
>
> On Fri, 1 Dec 2017 at 07:15, James Taylor  wrote:
>
> > The only way I can think of accomplishing this is by using the raw HBase
> > APIs to write the data but using our utilities to write it in a Phoenix
> > compatible manner. For example, you could run an UPSERT VALUES statement,
> > use the PhoenixRuntime.getUncommittedDataIterator()method to get the
> Cells
> > that would have been written, update the Cell timestamp as needed, and do
> > an htable.batch() call to commit them.
> >
> > On Wed, Nov 29, 2017 at 11:46 AM Pedro Boado 
> > wrote:
> >
> >> Hi,
> >>
> >> I'm looking for a little bit of help trying to get some light over
> >> ROW_TIMESTAMP.
> >>
> >> Some background over the problem ( simplified ) : I'm working in a
> >> project that needs to create a "enriched" replica of a RBDMS table
> based on
> >> a stream of cdc changes off that table.
> >>
> >> Each cdc event contains the timestamp of the change plus all the column
> >> values 'before' and 'after' the change . And each event is pushed to a
> >> kafka topic.  Because of certain "non-negotiable" design decisions kafka
> >> guarantees delivering each event at least once, but doesn't guarantee
> >> ordering for changes over the same row in the source table.
> >>
> >> The final step of the kafka-based flow is sinking the information into
> >> HBase/Phoenix.
> >>
> >> As I cannot get in order delivery guarantee from Kafka I need to use the
> >> cdc event timestamp to ensure that HBase keeps the latest change over a
> row.
> >>
> >> This fits perfectly well with an HBase table design with VERSIONS=1 and
> >> using the source event timestamp as HBase row/cells timestamp
> >>
> >> The thing is that I cannot find a way to define the value of the HBase
> >> cell from a Phoenix upsert.
> >>
> >> I came across the ROW_TIMESTAMP functionality, but I've just found ( I'm
> >> devastated now ) that the ROW_TIMESTAMP columns store the date in both
> >> hbase's cell timestamp and in the primary key, meaning that I cannot
> >> leverage that functionality to keep only the latest change.
> >>
> >> Is there a way of defining hbase's row timestamp when doing the UPSERT -
> >> even by setting it through some obscure hidden jdbc property - ?
> >>
> >> I want to avoid by all means doing a checkAndPut as the volume of
> changes
> >> is going to be quite bug.
> >>
> >>
> >>
> >> --
> >> Un saludo.
> >> Pedro Boado.
> >>
> >
>
> --
> Un saludo.
> Pedro Boado.
>

[jira] [Commented] (PHOENIX-4688) Add kerberos authentication to python-phoenixdb

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540185#comment-16540185
 ] 

ASF GitHub Bot commented on PHOENIX-4688:
-

Github user pu239ppy commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/307#discussion_r201717820
  
--- Diff: 
phoenix-queryserver/src/it/java/org/apache/phoenix/end2end/SecureQueryServerPhoenixDBIT.java
 ---
@@ -0,0 +1,423 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.end2end;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Maps;
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.HBaseTestingUtility;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.LocalHBaseCluster;
+import org.apache.hadoop.hbase.coprocessor.CoprocessorHost;
+import org.apache.hadoop.hbase.http.ssl.KeyStoreTestUtil;
+import org.apache.hadoop.hbase.security.HBaseKerberosUtils;
+import org.apache.hadoop.hbase.security.token.TokenProvider;
+import org.apache.hadoop.hbase.util.FSUtils;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.http.HttpConfig;
+import org.apache.hadoop.minikdc.MiniKdc;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.security.authentication.util.KerberosName;
+import org.apache.phoenix.query.ConfigurationFactory;
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.queryserver.client.ThinClientUtil;
+import org.apache.phoenix.queryserver.server.QueryServer;
+import org.apache.phoenix.util.InstanceResolver;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+import java.io.*;
+import java.lang.reflect.Field;
+import java.security.PrivilegedAction;
+import java.security.PrivilegedExceptionAction;
+import java.sql.DriverManager;
+import java.sql.ResultSet;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+import java.nio.file.Paths;
+import java.util.Map;
+
+import static org.junit.Assert.*;
+
+@Category(NeedsOwnMiniClusterTest.class)
+public class SecureQueryServerPhoenixDBIT {
+private static final Log LOG = 
LogFactory.getLog(SecureQueryServerPhoenixDBIT.class);
+
+private static final File TEMP_DIR = new File(getTempDirForClass());
+private static final File KEYTAB_DIR = new File(TEMP_DIR, "keytabs");
+private static final List USER_KEYTAB_FILES = new ArrayList<>();
+
+private static final String SPNEGO_PRINCIPAL = "HTTP/localhost";
+private static final String PQS_PRINCIPAL = "phoenixqs/localhost";
+private static final String SERVICE_PRINCIPAL = 
"securecluster/localhost";
+private static File KEYTAB;
+
+private static MiniKdc KDC;
+private static HBaseTestingUtility UTIL = new HBaseTestingUtility();
+private static LocalHBaseCluster HBASE_CLUSTER;
+private static int NUM_CREATED_USERS;
+
+private static ExecutorService PQS_EXECUTOR;
+private static QueryServer PQS;
+private static int PQS_PORT;
+private static String PQS_URL;
+
+private static String getTempDirForClass() {
+StringBuilder sb = new StringBuilder(32);
+sb.append(System.getProperty("user.dir")).append(File.separator);
+sb.append("target").append(File.separator);
+

[jira] [Updated] (PHOENIX-4790) Simplify check for client side delete

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor updated PHOENIX-4790:
--
Fix Version/s: 5.1.0
   4.15.0

> Simplify check for client side delete
> -
>
> Key: PHOENIX-4790
> URL: https://issues.apache.org/jira/browse/PHOENIX-4790
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: James Taylor
>Priority: Minor
> Fix For: 4.15.0, 5.1.0
>
> Attachments: PHOENIX-4790.patch
>
>
> We don't need to check every query plan for the existence of a where clause 
> to determine if we can do a client-side delete. Instead, we can simply check 
> if any non PK columns are projected.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (PHOENIX-4790) Simplify check for client side delete

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor resolved PHOENIX-4790.
---
Resolution: Fixed

> Simplify check for client side delete
> -
>
> Key: PHOENIX-4790
> URL: https://issues.apache.org/jira/browse/PHOENIX-4790
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: James Taylor
>Priority: Minor
> Fix For: 4.15.0, 5.1.0
>
> Attachments: PHOENIX-4790.patch
>
>
> We don't need to check every query plan for the existence of a where clause 
> to determine if we can do a client-side delete. Instead, we can simply check 
> if any non PK columns are projected.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] phoenix pull request #307: PHOENIX-4688 Kerberize python phoenixdb

[pu239ppy]

Github user pu239ppy commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/307#discussion_r201717820
  
--- Diff: 
phoenix-queryserver/src/it/java/org/apache/phoenix/end2end/SecureQueryServerPhoenixDBIT.java
 ---
@@ -0,0 +1,423 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.end2end;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Maps;
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.HBaseTestingUtility;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.LocalHBaseCluster;
+import org.apache.hadoop.hbase.coprocessor.CoprocessorHost;
+import org.apache.hadoop.hbase.http.ssl.KeyStoreTestUtil;
+import org.apache.hadoop.hbase.security.HBaseKerberosUtils;
+import org.apache.hadoop.hbase.security.token.TokenProvider;
+import org.apache.hadoop.hbase.util.FSUtils;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.http.HttpConfig;
+import org.apache.hadoop.minikdc.MiniKdc;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.security.authentication.util.KerberosName;
+import org.apache.phoenix.query.ConfigurationFactory;
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.queryserver.client.ThinClientUtil;
+import org.apache.phoenix.queryserver.server.QueryServer;
+import org.apache.phoenix.util.InstanceResolver;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+import java.io.*;
+import java.lang.reflect.Field;
+import java.security.PrivilegedAction;
+import java.security.PrivilegedExceptionAction;
+import java.sql.DriverManager;
+import java.sql.ResultSet;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+import java.nio.file.Paths;
+import java.util.Map;
+
+import static org.junit.Assert.*;
+
+@Category(NeedsOwnMiniClusterTest.class)
+public class SecureQueryServerPhoenixDBIT {
+private static final Log LOG = 
LogFactory.getLog(SecureQueryServerPhoenixDBIT.class);
+
+private static final File TEMP_DIR = new File(getTempDirForClass());
+private static final File KEYTAB_DIR = new File(TEMP_DIR, "keytabs");
+private static final List USER_KEYTAB_FILES = new ArrayList<>();
+
+private static final String SPNEGO_PRINCIPAL = "HTTP/localhost";
+private static final String PQS_PRINCIPAL = "phoenixqs/localhost";
+private static final String SERVICE_PRINCIPAL = 
"securecluster/localhost";
+private static File KEYTAB;
+
+private static MiniKdc KDC;
+private static HBaseTestingUtility UTIL = new HBaseTestingUtility();
+private static LocalHBaseCluster HBASE_CLUSTER;
+private static int NUM_CREATED_USERS;
+
+private static ExecutorService PQS_EXECUTOR;
+private static QueryServer PQS;
+private static int PQS_PORT;
+private static String PQS_URL;
+
+private static String getTempDirForClass() {
+StringBuilder sb = new StringBuilder(32);
+sb.append(System.getProperty("user.dir")).append(File.separator);
+sb.append("target").append(File.separator);
+sb.append(SecureQueryServerPhoenixDBIT.class.getSimpleName());
+return sb.toString();
+}
+
+private static void updateDefaultRealm() throws Exception {
+// (at least) one other phoenix test triggers the caching of this 
field before the KDC is up
+  

[jira] [Commented] (PHOENIX-4688) Add kerberos authentication to python-phoenixdb

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540181#comment-16540181
 ] 

ASF GitHub Bot commented on PHOENIX-4688:
-

Github user pu239ppy commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/307#discussion_r201716375
  
--- Diff: python/phoenixdb-module/phoenixdb/__init__.py ---
@@ -1,11 +1,10 @@
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements.  See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License.  You may obtain a copy of the License at
+# Copyright 2015 Lukas Lalinsky
--- End diff --

I am not sure how that went back in, it is possible that I may have copied 
__init__.py from the time I was doing this work on my own before I found out 
that this has been moved to phoenix.  I will change the header


> Add kerberos authentication to python-phoenixdb
> ---
>
> Key: PHOENIX-4688
> URL: https://issues.apache.org/jira/browse/PHOENIX-4688
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Lev Bronshtein
>Priority: Minor
>
> In its current state python-phoenixdv does not support support kerberos 
> authentication.  Using a modern python http library such as requests or 
> urllib it would be simple (if not trivial) to add this support.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] phoenix pull request #307: PHOENIX-4688 Kerberize python phoenixdb

[pu239ppy]

Github user pu239ppy commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/307#discussion_r201716375
  
--- Diff: python/phoenixdb-module/phoenixdb/__init__.py ---
@@ -1,11 +1,10 @@
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements.  See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License.  You may obtain a copy of the License at
+# Copyright 2015 Lukas Lalinsky
--- End diff --

I am not sure how that went back in, it is possible that I may have copied 
__init__.py from the time I was doing this work on my own before I found out 
that this has been moved to phoenix.  I will change the header


---

[jira] [Commented] (PHOENIX-3383) Comparison between descending row keys used in RVC is reverse

[James Taylor (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-3383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540165#comment-16540165
 ] 

James Taylor commented on PHOENIX-3383:
---

Thanks for the feedback, [~tdsilva]. I've incorporated that and fixed the test 
failures with v13 patch: 
[https://builds.apache.org/job/PreCommit-PHOENIX-Build/1921/.|https://builds.apache.org/job/PreCommit-PHOENIX-Build/1921/]
 Will go ahead and commit that version.

> Comparison between descending row keys used in RVC is reverse
> -
>
> Key: PHOENIX-3383
> URL: https://issues.apache.org/jira/browse/PHOENIX-3383
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: James Taylor
>Priority: Major
>  Labels: DESC
> Fix For: 4.15.0
>
> Attachments: PHOENIX-3383-wip1.patch, PHOENIX-3383-wip5.patch, 
> PHOENIX-3383-wip6.patch, PHOENIX-3383-wip7.patch, PHOENIX-3383_v1.patch, 
> PHOENIX-3383_v10.patch, PHOENIX-3383_v11.patch, PHOENIX-3383_v12.patch, 
> PHOENIX-3383_v13.patch, PHOENIX-3383_v2.patch, PHOENIX-3383_v3.patch, 
> PHOENIX-3383_v4.patch, PHOENIX-3383_v5.patch, PHOENIX-3383_v6.patch, 
> PHOENIX-3383_v7.patch, PHOENIX-3383_v8.patch, PHOENIX-3383_v9.patch, 
> PHOENIX-3383_wip.patch, PHOENIX-3383_wip2.patch, PHOENIX-3383_wip3.patch
>
>
> See PHOENIX-3382, but the comparison for RVC with descending row key columns 
> is the reverse of what it should be.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (PHOENIX-3383) Comparison between descending row keys used in RVC is reverse

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-3383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor updated PHOENIX-3383:
--
Attachment: PHOENIX-3383_v13.patch

> Comparison between descending row keys used in RVC is reverse
> -
>
> Key: PHOENIX-3383
> URL: https://issues.apache.org/jira/browse/PHOENIX-3383
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: James Taylor
>Priority: Major
>  Labels: DESC
> Fix For: 4.15.0
>
> Attachments: PHOENIX-3383-wip1.patch, PHOENIX-3383-wip5.patch, 
> PHOENIX-3383-wip6.patch, PHOENIX-3383-wip7.patch, PHOENIX-3383_v1.patch, 
> PHOENIX-3383_v10.patch, PHOENIX-3383_v11.patch, PHOENIX-3383_v12.patch, 
> PHOENIX-3383_v13.patch, PHOENIX-3383_v2.patch, PHOENIX-3383_v3.patch, 
> PHOENIX-3383_v4.patch, PHOENIX-3383_v5.patch, PHOENIX-3383_v6.patch, 
> PHOENIX-3383_v7.patch, PHOENIX-3383_v8.patch, PHOENIX-3383_v9.patch, 
> PHOENIX-3383_wip.patch, PHOENIX-3383_wip2.patch, PHOENIX-3383_wip3.patch
>
>
> See PHOENIX-3382, but the comparison for RVC with descending row key columns 
> is the reverse of what it should be.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (PHOENIX-3383) Comparison between descending row keys used in RVC is reverse

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-3383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor updated PHOENIX-3383:
--
Attachment: (was: PHOENIX-3383_v13.patch)

> Comparison between descending row keys used in RVC is reverse
> -
>
> Key: PHOENIX-3383
> URL: https://issues.apache.org/jira/browse/PHOENIX-3383
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: James Taylor
>Priority: Major
>  Labels: DESC
> Fix For: 4.15.0
>
> Attachments: PHOENIX-3383-wip1.patch, PHOENIX-3383-wip5.patch, 
> PHOENIX-3383-wip6.patch, PHOENIX-3383-wip7.patch, PHOENIX-3383_v1.patch, 
> PHOENIX-3383_v10.patch, PHOENIX-3383_v11.patch, PHOENIX-3383_v12.patch, 
> PHOENIX-3383_v2.patch, PHOENIX-3383_v3.patch, PHOENIX-3383_v4.patch, 
> PHOENIX-3383_v5.patch, PHOENIX-3383_v6.patch, PHOENIX-3383_v7.patch, 
> PHOENIX-3383_v8.patch, PHOENIX-3383_v9.patch, PHOENIX-3383_wip.patch, 
> PHOENIX-3383_wip2.patch, PHOENIX-3383_wip3.patch
>
>
> See PHOENIX-3382, but the comparison for RVC with descending row key columns 
> is the reverse of what it should be.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)