[jira] [Commented] (PHOENIX-4534) upsert/delete/upsert for the same row corrupts the indexes

[Rajeshbabu Chintaguntla (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335462#comment-16335462
 ] 

Rajeshbabu Chintaguntla commented on PHOENIX-4534:
--

[~sergey.soldatov] uploaded final patch with the tests including for multiple 
column families as well. Please review.

> upsert/delete/upsert for the same row corrupts the indexes
> --
>
> Key: PHOENIX-4534
> URL: https://issues.apache.org/jira/browse/PHOENIX-4534
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 5.0
>Reporter: Romil Choksi
>Assignee: Rajeshbabu Chintaguntla
>Priority: Critical
>  Labels: HBase-2.0
> Fix For: 5.0
>
> Attachments: PHOENIX-4534.patch, PHOENIX-4534_v2.patch, 
> PHOENIX-4534_v3.patch
>
>
> If we delete and upsert again the same row, the corresponding index has a 
> null value. 
> {noformat}
> 0: jdbc:phoenix:> create table a (id integer primary key, f float);
> No rows affected (2.272 seconds)
> 0: jdbc:phoenix:> create index i1 on a (f);
> No rows affected (5.769 seconds)
> 0: jdbc:phoenix:> upsert into a values (1,0.5);
> 1 row affected (0.021 seconds)
> 0: jdbc:phoenix:> select * from i1;
> +--+--+
> | 0:F  | :ID  |
> +--+--+
> | 0.5  | 1|
> +--+--+
> 1 row selected (0.016 seconds)
> 0: jdbc:phoenix:> delete from a where id = 1;
> 1 row affected (0.009 seconds)
> 0: jdbc:phoenix:> select * from i1;
> +--+--+
> | 0:F  | :ID  |
> +--+--+
> +--+--+
> No rows selected (0.015 seconds)
> 0: jdbc:phoenix:> upsert into a values (1,0.5);
> 1 row affected (0.008 seconds)
> 0: jdbc:phoenix:> select * from i1;
> +---+--+
> |  0:F  | :ID  |
> +---+--+
> | null  | 1|
> +---+--+
> 1 row selected (0.013 seconds)
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (PHOENIX-4534) upsert/delete/upsert for the same row corrupts the indexes

[Rajeshbabu Chintaguntla (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajeshbabu Chintaguntla updated PHOENIX-4534:
-
Attachment: PHOENIX-4534_v3.patch

> upsert/delete/upsert for the same row corrupts the indexes
> --
>
> Key: PHOENIX-4534
> URL: https://issues.apache.org/jira/browse/PHOENIX-4534
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 5.0
>Reporter: Romil Choksi
>Assignee: Rajeshbabu Chintaguntla
>Priority: Critical
>  Labels: HBase-2.0
> Fix For: 5.0
>
> Attachments: PHOENIX-4534.patch, PHOENIX-4534_v2.patch, 
> PHOENIX-4534_v3.patch
>
>
> If we delete and upsert again the same row, the corresponding index has a 
> null value. 
> {noformat}
> 0: jdbc:phoenix:> create table a (id integer primary key, f float);
> No rows affected (2.272 seconds)
> 0: jdbc:phoenix:> create index i1 on a (f);
> No rows affected (5.769 seconds)
> 0: jdbc:phoenix:> upsert into a values (1,0.5);
> 1 row affected (0.021 seconds)
> 0: jdbc:phoenix:> select * from i1;
> +--+--+
> | 0:F  | :ID  |
> +--+--+
> | 0.5  | 1|
> +--+--+
> 1 row selected (0.016 seconds)
> 0: jdbc:phoenix:> delete from a where id = 1;
> 1 row affected (0.009 seconds)
> 0: jdbc:phoenix:> select * from i1;
> +--+--+
> | 0:F  | :ID  |
> +--+--+
> +--+--+
> No rows selected (0.015 seconds)
> 0: jdbc:phoenix:> upsert into a values (1,0.5);
> 1 row affected (0.008 seconds)
> 0: jdbc:phoenix:> select * from i1;
> +---+--+
> |  0:F  | :ID  |
> +---+--+
> | null  | 1|
> +---+--+
> 1 row selected (0.013 seconds)
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4550) Allow declaration of max columns on base physical table

[James Taylor (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335385#comment-16335385
 ] 

James Taylor commented on PHOENIX-4550:
---

bq. This will mainly help for the case when views declare all their columns at 
creation time, right?
Yep, that's exactly the case we'd like to optimize.

bq. You would need to store the minimum view column encoding counter in the 
view metadata. You also have to change SingleCellColumnExpression.evaluate() to 
calculate the index into the array.
Agreed - that sounds like what'd be required.

> Allow declaration of max columns on base physical table
> ---
>
> Key: PHOENIX-4550
> URL: https://issues.apache.org/jira/browse/PHOENIX-4550
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: James Taylor
>Priority: Major
>
> By declaring the max number of columns on a base table, we can optimize the 
> storage for SINGLE_CELL_ARRAY_WITH_OFFSETS by not storing null values for the 
> columns preceding the initial column of a view. This will make a huge 
> difference in storage when you have a base table with many views. For example:
> {code}
> -- Declare that the base table will have no more than 10 columns
> CREATE IMMUTABLE TABLE base (k1 VARCHAR, prefix CHAR(3) v1 DATE,
> CONSTRAINT pk PRIMARY KEY (k1, prefix))
> MULTI_TENANT = true,
> MAX_COLUMNS = 10;
> CREATE VIEW v1(k2 VARCHAR PRIMARY KEY, v2 VARCHAR, v3 VARCHAR)
> AS SELECT * FROM base WHERE prefix = 'A00';
> CREATE VIEW v2(k2 VARCHAR PRIMARY KEY, v2 VARCHAR, v3 VARCHAR);
> AS SELECT * FROM base WHERE prefix = 'A10';
> ...
> {code}
> As the number of views grow, the difference between the base table column 
> encoding (column #1) and the starting column number of the view (since the 
> starting offset is determined by an incrementing value on the base table) 
> will increase. This bloats the storage as we need to store null values for 
> column encodings between the base table column and the starting column of the 
> view.
> Instead, we'll pass through the MAX_COLUMNS value for queries and anything 
> column encoding less than this we know it'll be at the start. Anything 
> greater and we'll start the search from  -  column encoding>.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4550) Allow declaration of max columns on base physical table

[Thomas D'Silva (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335378#comment-16335378
 ] 

Thomas D'Silva commented on PHOENIX-4550:
-

[~jamestaylor] 

This will mainly help for the case when views declare all their columns at 
creation time, right?

You would need to store the minimum view column encoding counter in the view 
metadata. You also have to change SingleCellColumnExpression.evaluate() to 
calculate the index into the array.

> Allow declaration of max columns on base physical table
> ---
>
> Key: PHOENIX-4550
> URL: https://issues.apache.org/jira/browse/PHOENIX-4550
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: James Taylor
>Priority: Major
>
> By declaring the max number of columns on a base table, we can optimize the 
> storage for SINGLE_CELL_ARRAY_WITH_OFFSETS by not storing null values for the 
> columns preceding the initial column of a view. This will make a huge 
> difference in storage when you have a base table with many views. For example:
> {code}
> -- Declare that the base table will have no more than 10 columns
> CREATE IMMUTABLE TABLE base (k1 VARCHAR, prefix CHAR(3) v1 DATE,
> CONSTRAINT pk PRIMARY KEY (k1, prefix))
> MULTI_TENANT = true,
> MAX_COLUMNS = 10;
> CREATE VIEW v1(k2 VARCHAR PRIMARY KEY, v2 VARCHAR, v3 VARCHAR)
> AS SELECT * FROM base WHERE prefix = 'A00';
> CREATE VIEW v2(k2 VARCHAR PRIMARY KEY, v2 VARCHAR, v3 VARCHAR);
> AS SELECT * FROM base WHERE prefix = 'A10';
> ...
> {code}
> As the number of views grow, the difference between the base table column 
> encoding (column #1) and the starting column number of the view (since the 
> starting offset is determined by an incrementing value on the base table) 
> will increase. This bloats the storage as we need to store null values for 
> column encodings between the base table column and the starting column of the 
> view.
> Instead, we'll pass through the MAX_COLUMNS value for queries and anything 
> column encoding less than this we know it'll be at the start. Anything 
> greater and we'll start the search from  -  column encoding>.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4550) Allow declaration of max columns on base physical table

[James Taylor (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335340#comment-16335340
 ] 

James Taylor commented on PHOENIX-4550:
---

FYI, [~shehzaadn].

> Allow declaration of max columns on base physical table
> ---
>
> Key: PHOENIX-4550
> URL: https://issues.apache.org/jira/browse/PHOENIX-4550
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: James Taylor
>Priority: Major
>
> By declaring the max number of columns on a base table, we can optimize the 
> storage for SINGLE_CELL_ARRAY_WITH_OFFSETS by not storing null values for the 
> columns preceding the initial column of a view. This will make a huge 
> difference in storage when you have a base table with many views. For example:
> {code}
> -- Declare that the base table will have no more than 10 columns
> CREATE IMMUTABLE TABLE base (k1 VARCHAR, prefix CHAR(3) v1 DATE,
> CONSTRAINT pk PRIMARY KEY (k1, prefix))
> MULTI_TENANT = true,
> MAX_COLUMNS = 10;
> CREATE VIEW v1(k2 VARCHAR PRIMARY KEY, v2 VARCHAR, v3 VARCHAR)
> AS SELECT * FROM base WHERE prefix = 'A00';
> CREATE VIEW v2(k2 VARCHAR PRIMARY KEY, v2 VARCHAR, v3 VARCHAR);
> AS SELECT * FROM base WHERE prefix = 'A10';
> ...
> {code}
> As the number of views grow, the difference between the base table column 
> encoding (column #1) and the starting column number of the view (since the 
> starting offset is determined by an incrementing value on the base table) 
> will increase. This bloats the storage as we need to store null values for 
> column encodings between the base table column and the starting column of the 
> view.
> Instead, we'll pass through the MAX_COLUMNS value for queries and anything 
> column encoding less than this we know it'll be at the start. Anything 
> greater and we'll start the search from  -  column encoding>.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (PHOENIX-4550) Allow declaration of max columns on base physical table

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor updated PHOENIX-4550:
--
Issue Type: Improvement  (was: Bug)

> Allow declaration of max columns on base physical table
> ---
>
> Key: PHOENIX-4550
> URL: https://issues.apache.org/jira/browse/PHOENIX-4550
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: James Taylor
>Priority: Major
>
> By declaring the max number of columns on a base table, we can optimize the 
> storage for SINGLE_CELL_ARRAY_WITH_OFFSETS by not storing null values for the 
> columns preceding the initial column of a view. This will make a huge 
> difference in storage when you have a base table with many views. For example:
> {code}
> -- Declare that the base table will have no more than 10 columns
> CREATE IMMUTABLE TABLE base (k1 VARCHAR, prefix CHAR(3) v1 DATE,
> CONSTRAINT pk PRIMARY KEY (k1, prefix))
> MULTI_TENANT = true,
> MAX_COLUMNS = 10;
> CREATE VIEW v1(k2 VARCHAR PRIMARY KEY, v2 VARCHAR, v3 VARCHAR)
> AS SELECT * FROM base WHERE prefix = 'A00';
> CREATE VIEW v2(k2 VARCHAR PRIMARY KEY, v2 VARCHAR, v3 VARCHAR);
> AS SELECT * FROM base WHERE prefix = 'A10';
> ...
> {code}
> As the number of views grow, the difference between the base table column 
> encoding (column #1) and the starting column number of the view (since the 
> starting offset is determined by an incrementing value on the base table) 
> will increase. This bloats the storage as we need to store null values for 
> column encodings between the base table column and the starting column of the 
> view.
> Instead, we'll pass through the MAX_COLUMNS value for queries and anything 
> column encoding less than this we know it'll be at the start. Anything 
> greater and we'll start the search from  -  column encoding>.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (PHOENIX-4550) Allow declaration of max columns on base physical table

[James Taylor (JIRA)]

James Taylor created PHOENIX-4550:
-

 Summary: Allow declaration of max columns on base physical table
 Key: PHOENIX-4550
 URL: https://issues.apache.org/jira/browse/PHOENIX-4550
 Project: Phoenix
  Issue Type: Bug
Reporter: James Taylor


By declaring the max number of columns on a base table, we can optimize the 
storage for SINGLE_CELL_ARRAY_WITH_OFFSETS by not storing null values for the 
columns preceding the initial column of a view. This will make a huge 
difference in storage when you have a base table with many views. For example:

{code}
-- Declare that the base table will have no more than 10 columns
CREATE IMMUTABLE TABLE base (k1 VARCHAR, prefix CHAR(3) v1 DATE,
CONSTRAINT pk PRIMARY KEY (k1, prefix))
MULTI_TENANT = true,
MAX_COLUMNS = 10;
CREATE VIEW v1(k2 VARCHAR PRIMARY KEY, v2 VARCHAR, v3 VARCHAR)
AS SELECT * FROM base WHERE prefix = 'A00';
CREATE VIEW v2(k2 VARCHAR PRIMARY KEY, v2 VARCHAR, v3 VARCHAR);
AS SELECT * FROM base WHERE prefix = 'A10';
...
{code}

As the number of views grow, the difference between the base table column 
encoding (column #1) and the starting column number of the view (since the 
starting offset is determined by an incrementing value on the base table) will 
increase. This bloats the storage as we need to store null values for column 
encodings between the base table column and the starting column of the view.

Instead, we'll pass through the MAX_COLUMNS value for queries and anything 
column encoding less than this we know it'll be at the start. Anything greater 
and we'll start the search from  - .



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4529) Users should only require RX access to SYSTEM.SEQUENCE table

[Thomas D'Silva (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335309#comment-16335309
 ] 

Thomas D'Silva commented on PHOENIX-4529:
-

Thats good to know, thanks [~apurtell].

> Users should only require RX access to SYSTEM.SEQUENCE table
> 
>
> Key: PHOENIX-4529
> URL: https://issues.apache.org/jira/browse/PHOENIX-4529
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Karan Mehta
>Assignee: Thomas D'Silva
>Priority: Major
>
> Currently, users don't need to have Write access to {{SYSTEM.CATALOG}} and 
> other tables, since the code is run on the server side as login user. However 
> for {{SYSTEM.SEQUENCE}}, write permission is still needed. This is a 
> potential security concern, since it allows anyone to modify the sequences 
> created by others. This JIRA is to discuss how we can improve the security of 
> this table. 
> Potential options include
> 1. Usage of HBase Cell Level Permissions (works only with HFile version 3 and 
> above)
> 2. AccessControl at Phoenix Layer by addition of user column in the 
> {{SYSTEM.SEQUENCE}} table and use it for access control (Can be error-prone 
> for complex scenarios like sequence sharing)
> Please advice.
> [~tdsilva] [~jamestaylor] [~apurtell] [~an...@apache.org] [~elserj]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4537) RegionServer initiating compaction can trigger schema migration and deadlock the system

[Josh Elser (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335235#comment-16335235
 ] 

Josh Elser commented on PHOENIX-4537:
-

bq. I'm hoping it could use the same general mechanism - check the 
PhoenixConnection for this "special" property and don't initiate the migration 
if it's set.

Ya, perhaps just requires a game of whack-a-mole to address the areas, 
server-side, to keep using the un-mapped table names (instead of bubbling up an 
Exception like happens client-side)?

> RegionServer initiating compaction can trigger schema migration and deadlock 
> the system
> ---
>
> Key: PHOENIX-4537
> URL: https://issues.apache.org/jira/browse/PHOENIX-4537
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Romil Choksi
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 5.0.0, 4.14.0
>
> Attachments: PHOENIX-4537.001.patch
>
>
> [~sergey.soldatov] has been doing some great digging around a test failure 
> we've been seeing at $dayjob. The situation goes like this.
> 0. Run some arbitrary load
> 1. Stop HBase
> 2. Enable schema mapping ({{phoenix.schema.isNamespaceMappingEnabled=true}} 
> and {{phoenix.schema.mapSystemTablesToNamespace=true}} in hbase-site.xml)
> 3. Start HBase
> 4. Circumstantially, have the SYSTEM.CATALOG table need a compaction to run 
> before a client first connects
> When the RegionServer initiates the compaction, it will end up running 
> {{UngroupedAggregateRegionObserver.clearTsOnDisabledIndexes}} which opens a 
> Phoenix connection. While the RegionServer won't upgrade system tables, it 
> *will* try to migrate them into the schema mapped variants (e.g. 
> SYSTEM.CATALOG to SYSTEM:CATALOG).
> However, one of the first steps in the schema migration is to disable the 
> SYSTEM.CATALOG table. However, the SYSTEM.CATALOG table can't be disabled 
> until the region is CLOSED, and the region cannot be CLOSED until the 
> compaction is finished. *deadlock*
> The "obvious" fix is to avoid RegionServers from triggering system table 
> migrations, but Sergey and [~elserj] both think that this will end badly 
> (RegionServers falling over because they expect the tables to be migrated and 
> they aren't).
> Thoughts? [~ankit.singhal], [~jamestaylor], any others?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4537) RegionServer initiating compaction can trigger schema migration and deadlock the system

[James Taylor (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335228#comment-16335228
 ] 

James Taylor commented on PHOENIX-4537:
---

[~sergey.soldatov] - good point. PHOENIX-4530  would just prevent one 
occurrence. Can we brainstorm on what it'd take to ensure that this migration 
doesn't happen on a server connection? I'm hoping it could use the same general 
mechanism - check the PhoenixConnection for this "special" property and don't 
initiate the migration if it's set.

> RegionServer initiating compaction can trigger schema migration and deadlock 
> the system
> ---
>
> Key: PHOENIX-4537
> URL: https://issues.apache.org/jira/browse/PHOENIX-4537
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Romil Choksi
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 5.0.0, 4.14.0
>
> Attachments: PHOENIX-4537.001.patch
>
>
> [~sergey.soldatov] has been doing some great digging around a test failure 
> we've been seeing at $dayjob. The situation goes like this.
> 0. Run some arbitrary load
> 1. Stop HBase
> 2. Enable schema mapping ({{phoenix.schema.isNamespaceMappingEnabled=true}} 
> and {{phoenix.schema.mapSystemTablesToNamespace=true}} in hbase-site.xml)
> 3. Start HBase
> 4. Circumstantially, have the SYSTEM.CATALOG table need a compaction to run 
> before a client first connects
> When the RegionServer initiates the compaction, it will end up running 
> {{UngroupedAggregateRegionObserver.clearTsOnDisabledIndexes}} which opens a 
> Phoenix connection. While the RegionServer won't upgrade system tables, it 
> *will* try to migrate them into the schema mapped variants (e.g. 
> SYSTEM.CATALOG to SYSTEM:CATALOG).
> However, one of the first steps in the schema migration is to disable the 
> SYSTEM.CATALOG table. However, the SYSTEM.CATALOG table can't be disabled 
> until the region is CLOSED, and the region cannot be CLOSED until the 
> compaction is finished. *deadlock*
> The "obvious" fix is to avoid RegionServers from triggering system table 
> migrations, but Sergey and [~elserj] both think that this will end badly 
> (RegionServers falling over because they expect the tables to be migrated and 
> they aren't).
> Thoughts? [~ankit.singhal], [~jamestaylor], any others?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (PHOENIX-4529) Users should only require RX access to SYSTEM.SEQUENCE table

[Andrew Purtell (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335206#comment-16335206
 ] 

Andrew Purtell edited comment on PHOENIX-4529 at 1/23/18 12:33 AM:
---

Cell level ACLs have limitations you should be aware of. The most significant 
is they cannot override ACLs which grant permissions at a coarser granularity, 
like cf or table or namespace. (This is generally true of ACLs, btw: if 
namespace grants, but table ACL does not, namespace grant is where we stop and 
grant access anyway.) The next is the only way to change a cell ACL is to 
rewrite the cell with a new ACL. Say you have a table with a million entries. 
If you set a CF level or coarser ACL, this is managed in the ACL table and one 
REVOKE will do what you expect. If you have a million entries with per cell 
ACLs and you want to revoke, you'll have to rewrite a million entries. 

Edit: This could be more efficient after HBASE-19842 


was (Author: apurtell):
Cell level ACLs have limitations you should be aware of. The most significant 
is they cannot override ACLs which grant permissions at a coarser granularity, 
like cf or table or namespace. (This is generally true of ACLs, btw: if 
namespace grants, but table ACL does not, namespace grant is where we stop and 
grant access anyway.) The next is the only way to change a cell ACL is to 
rewrite the cell with a new ACL. Say you have a table with a million entries. 
If you set a CF level or coarser ACL, this is managed in the ACL table and one 
REVOKE will do what you expect. If you have a million entries with per cell 
ACLs and you want to revoke, you'll have to rewrite a million entries. 

> Users should only require RX access to SYSTEM.SEQUENCE table
> 
>
> Key: PHOENIX-4529
> URL: https://issues.apache.org/jira/browse/PHOENIX-4529
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Karan Mehta
>Assignee: Thomas D'Silva
>Priority: Major
>
> Currently, users don't need to have Write access to {{SYSTEM.CATALOG}} and 
> other tables, since the code is run on the server side as login user. However 
> for {{SYSTEM.SEQUENCE}}, write permission is still needed. This is a 
> potential security concern, since it allows anyone to modify the sequences 
> created by others. This JIRA is to discuss how we can improve the security of 
> this table. 
> Potential options include
> 1. Usage of HBase Cell Level Permissions (works only with HFile version 3 and 
> above)
> 2. AccessControl at Phoenix Layer by addition of user column in the 
> {{SYSTEM.SEQUENCE}} table and use it for access control (Can be error-prone 
> for complex scenarios like sequence sharing)
> Please advice.
> [~tdsilva] [~jamestaylor] [~apurtell] [~an...@apache.org] [~elserj]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] 5.0.0-beta release before month's end?

[Josh Elser]

I don't think so. We (I use that loosely -- I'd hardly lump my 
contributions to this effort in the same room as the contributions of 
the others) have been moving fast to get back to functional state. This 
have definitely lapsed.


It is a blocker to make sure 5.x isn't missing stuff from 4.x and 
vice-versa before we consider a 5.0.0 "stable" release. I don't think 
this needs to block the alpha/beta I was hoping to get to this week.


On 1/22/18 3:10 PM, James Taylor wrote:

Have you guys back ported the removal of deprecated APIs patches to the 4.x
branches? That'll probably help minimize the merge conflicts we see going
forward.

On Thu, Jan 18, 2018 at 1:03 PM, James Taylor 
wrote:


Awesome! That's great work!!

On Thu, Jan 18, 2018 at 1:00 PM, Josh Elser  wrote:


Hah, funny you should ask! I was just thinking that I should send out a
note.

* Rajeshbabu and Sergey are trying to track down a nasty issue that the
IndexScrutiny tool has caught where there are dangling index records
(PHOENIX-4534)
* Rajeshbabu is also looking into some local index failures
* Ankit has done some testing of the phoenix-hive integration, I think
phoenix-spark is also looking OK

Last I chatted with folks, they were seeing >90% pass rate of the test
suite which seems pretty good to me. My plan was to work on a beta release
soon now that we have a beta from HBase. Things are generally functional as
of now -- I think getting it into the hands of folks to get more people
poking it would be great.

Maybe rc0 next week?

- Josh


On 1/18/18 1:11 PM, James Taylor wrote:


How are things looking with the 5.0.0 alpha/beta on HBase 2.x?

On Thu, Jan 4, 2018 at 10:57 AM, Josh Elser  wrote:

Good point. Perhaps "alpha" would be a better label?


IIUC, the issue is that we need the HBase release, and then a Tephra
release, and then we can get Tephra fixed for Phoenix5. Perhaps Ankit
can
provide some more color to the situation.


On 1/4/18 12:07 PM, Nick Dimiduk wrote:

Isn't Tephra integration mandatory for transaction support? What happens

to
a user who has TRANSACTIONAL=true tables when they upgrade? This can't
really fail gracefully. I guess transaction support is still marked
'beta',
but still, this would be a regression of functionality in "base
Phoenix".

On Thu, Jan 4, 2018 at 8:34 AM Josh Elser  wrote:

Talked to Rajeshbabu and Ankit offline this morning.



Sounds like there are a few integration points which are still
lacking:

* phoenix-hive: PHOENIX-4423
* phoenix-spark: untested (probably broken against newest Spark)
* phoenix-kafka: untested (probably broken against newest Kafka -- see
PHOENIX-4515 PHOENIX-4516)
* Tephra integration: Needs a new release of Tephra with some fixes
Ankit helped with.

I plan to not consider these 5.0.0-alpha/beta release blockers, we'll
just call those out which we don't get tested/fixed.

On 1/2/18 1:08 PM, Josh Elser wrote:

Happy New Year folks!


I'd like to test the waters: what do people think about trying to
get a
5.0.0 "beta" release out to the community before the end of January?

HBase is doing the same right now with 2.0.0. My thinking is that if
things are stable "enough", getting a base for people to use a 5.0
Phoenix release more easily, we can catch more bugs and get a better
product out the door.

Thoughts/concerns? I'm happy to RM.

- Josh

[jira] [Commented] (PHOENIX-4529) Users should only require RX access to SYSTEM.SEQUENCE table

[Andrew Purtell (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335206#comment-16335206
 ] 

Andrew Purtell commented on PHOENIX-4529:
-

Cell level ACLs have limitations you should be aware of. The most significant 
is they cannot override ACLs which grant permissions at a coarser granularity, 
like cf or table or namespace. (This is generally true of ACLs, btw: if 
namespace grants, but table ACL does not, namespace grant is where we stop and 
grant access anyway.) The next is the only way to change a cell ACL is to 
rewrite the cell with a new ACL. Say you have a table with a million entries. 
If you set a CF level or coarser ACL, this is managed in the ACL table and one 
REVOKE will do what you expect. If you have a million entries with per cell 
ACLs and you want to revoke, you'll have to rewrite a million entries. 

> Users should only require RX access to SYSTEM.SEQUENCE table
> 
>
> Key: PHOENIX-4529
> URL: https://issues.apache.org/jira/browse/PHOENIX-4529
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Karan Mehta
>Assignee: Thomas D'Silva
>Priority: Major
>
> Currently, users don't need to have Write access to {{SYSTEM.CATALOG}} and 
> other tables, since the code is run on the server side as login user. However 
> for {{SYSTEM.SEQUENCE}}, write permission is still needed. This is a 
> potential security concern, since it allows anyone to modify the sequences 
> created by others. This JIRA is to discuss how we can improve the security of 
> this table. 
> Potential options include
> 1. Usage of HBase Cell Level Permissions (works only with HFile version 3 and 
> above)
> 2. AccessControl at Phoenix Layer by addition of user column in the 
> {{SYSTEM.SEQUENCE}} table and use it for access control (Can be error-prone 
> for complex scenarios like sequence sharing)
> Please advice.
> [~tdsilva] [~jamestaylor] [~apurtell] [~an...@apache.org] [~elserj]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4537) RegionServer initiating compaction can trigger schema migration and deadlock the system

[Sergey Soldatov (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335197#comment-16335197
 ] 

Sergey Soldatov commented on PHOENIX-4537:
--

I doubt that PHOENIX-4530 would make any difference. It's not the only place 
where we create server-server phoenix connection (stats collection, the handler 
for local indexes are two other places where we use it). To properly implement 
it we need to handle the migration in the same way as the upgrade, but the main 
problem that I see there is that in our code we mostly rely on the 
configuration property, but not the actual state of the system tables, so I 
expect that something else may not work correctly. 

> RegionServer initiating compaction can trigger schema migration and deadlock 
> the system
> ---
>
> Key: PHOENIX-4537
> URL: https://issues.apache.org/jira/browse/PHOENIX-4537
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Romil Choksi
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 5.0.0, 4.14.0
>
> Attachments: PHOENIX-4537.001.patch
>
>
> [~sergey.soldatov] has been doing some great digging around a test failure 
> we've been seeing at $dayjob. The situation goes like this.
> 0. Run some arbitrary load
> 1. Stop HBase
> 2. Enable schema mapping ({{phoenix.schema.isNamespaceMappingEnabled=true}} 
> and {{phoenix.schema.mapSystemTablesToNamespace=true}} in hbase-site.xml)
> 3. Start HBase
> 4. Circumstantially, have the SYSTEM.CATALOG table need a compaction to run 
> before a client first connects
> When the RegionServer initiates the compaction, it will end up running 
> {{UngroupedAggregateRegionObserver.clearTsOnDisabledIndexes}} which opens a 
> Phoenix connection. While the RegionServer won't upgrade system tables, it 
> *will* try to migrate them into the schema mapped variants (e.g. 
> SYSTEM.CATALOG to SYSTEM:CATALOG).
> However, one of the first steps in the schema migration is to disable the 
> SYSTEM.CATALOG table. However, the SYSTEM.CATALOG table can't be disabled 
> until the region is CLOSED, and the region cannot be CLOSED until the 
> compaction is finished. *deadlock*
> The "obvious" fix is to avoid RegionServers from triggering system table 
> migrations, but Sergey and [~elserj] both think that this will end badly 
> (RegionServers falling over because they expect the tables to be migrated and 
> they aren't).
> Thoughts? [~ankit.singhal], [~jamestaylor], any others?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (PHOENIX-4529) Users should only require RX access to SYSTEM.SEQUENCE table

[Thomas D'Silva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas D'Silva reassigned PHOENIX-4529:
---

Assignee: Thomas D'Silva

> Users should only require RX access to SYSTEM.SEQUENCE table
> 
>
> Key: PHOENIX-4529
> URL: https://issues.apache.org/jira/browse/PHOENIX-4529
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Karan Mehta
>Assignee: Thomas D'Silva
>Priority: Major
>
> Currently, users don't need to have Write access to {{SYSTEM.CATALOG}} and 
> other tables, since the code is run on the server side as login user. However 
> for {{SYSTEM.SEQUENCE}}, write permission is still needed. This is a 
> potential security concern, since it allows anyone to modify the sequences 
> created by others. This JIRA is to discuss how we can improve the security of 
> this table. 
> Potential options include
> 1. Usage of HBase Cell Level Permissions (works only with HFile version 3 and 
> above)
> 2. AccessControl at Phoenix Layer by addition of user column in the 
> {{SYSTEM.SEQUENCE}} table and use it for access control (Can be error-prone 
> for complex scenarios like sequence sharing)
> Please advice.
> [~tdsilva] [~jamestaylor] [~apurtell] [~an...@apache.org] [~elserj]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4548) UpgradeUtil.mapChildViewsToNamespace does not handle multi-tenant views that have the same name.

[Thomas D'Silva (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335192#comment-16335192
 ] 

Thomas D'Silva commented on PHOENIX-4548:
-

[~an...@apache.org] can you please take a look ?

> UpgradeUtil.mapChildViewsToNamespace does not handle multi-tenant views that 
> have the same name.
> 
>
> Key: PHOENIX-4548
> URL: https://issues.apache.org/jira/browse/PHOENIX-4548
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Thomas D'Silva
>Assignee: Thomas D'Silva
>Priority: Major
> Fix For: 4.14.0
>
> Attachments: PHOENIX-4548.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (PHOENIX-4537) RegionServer initiating compaction can trigger schema migration and deadlock the system

[Josh Elser (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335181#comment-16335181
 ] 

Josh Elser edited comment on PHOENIX-4537 at 1/22/18 11:55 PM:
---

bq. The upgrade should not happen from a server-side connection. We have code 
in place for this already. See QueryUtil.getConnectionOnServer():

[~jamestaylor], again, I think these are two separate things.

System table _movement_ to the SYSTEM schema happens irregardless of whether it 
comes from client or server. I have explicitly looked at this code in 
ConnectionQueryServicesImpl and confirmed that this is how it works ("intent" 
of how it is meant to work aside). We are in agreement that schema migration of 
the System tables does not happen on Connections initiated from the server.

(edit: s/migration/movement/ because I think this terminology is causing the 
confusion)

bq. If we're going to go this route, can't we prevent this migration in a 
similar manner?

This is what Sergey was hinting at, that the real smell is that we should not 
be moving System tables into the SYSTEM schema from a server-side connection.

If the consensus is that we don't want to fix this "now" and work towards a fix 
that addresses the server-side SYSTEM schema movement, that's fine. However, 
this is blocking me at $dayjob and I need a fix for it now.

bq. What about the idea I outlined above of doing PHOENIX-4530 and removing the 
clearTsOnDisabledIndexes altogether?

To be honest, I understand the words you've written there, but am at a loss as 
to how to go about implementing it :). If we prevent movement of system tables 
to the SYSTEM schema by server-side Connections, that lowers the "need" to 
implement your suggestion in PHOENIX-4530, right?


was (Author: elserj):
bq. The upgrade should not happen from a server-side connection. We have code 
in place for this already. See QueryUtil.getConnectionOnServer():

[~jamestaylor], again, I think these are two separate things.

System table _migration_ to the SYSTEM schema happens irregardless of whether 
it comes from client or server. I have explicitly looked at this code in 
ConnectionQueryServicesImpl and confirmed that this is how it works ("intent" 
of how it is meant to work aside). We are in agreement that schema migration of 
the System tables does not happen on Connections initiated from the server.

bq. If we're going to go this route, can't we prevent this migration in a 
similar manner?

This is what Sergey was hinting at, that the real smell is that we should not 
be moving System tables into the SYSTEM schema from a server-side connection.

If the consensus is that we don't want to fix this "now" and work towards a fix 
that addresses the server-side SYSTEM schema movement, that's fine. However, 
this is blocking me at $dayjob and I need a fix for it now.

bq. What about the idea I outlined above of doing PHOENIX-4530 and removing the 
clearTsOnDisabledIndexes altogether?

To be honest, I understand the words you've written there, but am at a loss as 
to how to go about implementing it :). If we prevent movement of system tables 
to the SYSTEM schema by server-side Connections, that lowers the "need" to 
implement your suggestion in PHOENIX-4530, right?

> RegionServer initiating compaction can trigger schema migration and deadlock 
> the system
> ---
>
> Key: PHOENIX-4537
> URL: https://issues.apache.org/jira/browse/PHOENIX-4537
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Romil Choksi
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 5.0.0, 4.14.0
>
> Attachments: PHOENIX-4537.001.patch
>
>
> [~sergey.soldatov] has been doing some great digging around a test failure 
> we've been seeing at $dayjob. The situation goes like this.
> 0. Run some arbitrary load
> 1. Stop HBase
> 2. Enable schema mapping ({{phoenix.schema.isNamespaceMappingEnabled=true}} 
> and {{phoenix.schema.mapSystemTablesToNamespace=true}} in hbase-site.xml)
> 3. Start HBase
> 4. Circumstantially, have the SYSTEM.CATALOG table need a compaction to run 
> before a client first connects
> When the RegionServer initiates the compaction, it will end up running 
> {{UngroupedAggregateRegionObserver.clearTsOnDisabledIndexes}} which opens a 
> Phoenix connection. While the RegionServer won't upgrade system tables, it 
> *will* try to migrate them into the schema mapped variants (e.g. 
> SYSTEM.CATALOG to SYSTEM:CATALOG).
> However, one of the first steps in the schema migration is to disable the 
> SYSTEM.CATALOG table. However, the SYSTEM.CATALOG table can't be disabled 
> until the region is CLOSED, and the region cannot be CLOSED until the 
> compaction is finished. *deadlock*
> The "obvious" fix is to avoid 

[jira] [Commented] (PHOENIX-4537) RegionServer initiating compaction can trigger schema migration and deadlock the system

[Josh Elser (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335181#comment-16335181
 ] 

Josh Elser commented on PHOENIX-4537:
-

bq. The upgrade should not happen from a server-side connection. We have code 
in place for this already. See QueryUtil.getConnectionOnServer():

[~jamestaylor], again, I think these are two separate things.

System table _migration_ to the SYSTEM schema happens irregardless of whether 
it comes from client or server. I have explicitly looked at this code in 
ConnectionQueryServicesImpl and confirmed that this is how it works ("intent" 
of how it is meant to work aside). We are in agreement that schema migration of 
the System tables does not happen on Connections initiated from the server.

bq. If we're going to go this route, can't we prevent this migration in a 
similar manner?

This is what Sergey was hinting at, that the real smell is that we should not 
be moving System tables into the SYSTEM schema from a server-side connection.

If the consensus is that we don't want to fix this "now" and work towards a fix 
that addresses the server-side SYSTEM schema movement, that's fine. However, 
this is blocking me at $dayjob and I need a fix for it now.

bq. What about the idea I outlined above of doing PHOENIX-4530 and removing the 
clearTsOnDisabledIndexes altogether?

To be honest, I understand the words you've written there, but am at a loss as 
to how to go about implementing it :). If we prevent movement of system tables 
to the SYSTEM schema by server-side Connections, that lowers the "need" to 
implement your suggestion in PHOENIX-4530, right?

> RegionServer initiating compaction can trigger schema migration and deadlock 
> the system
> ---
>
> Key: PHOENIX-4537
> URL: https://issues.apache.org/jira/browse/PHOENIX-4537
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Romil Choksi
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 5.0.0, 4.14.0
>
> Attachments: PHOENIX-4537.001.patch
>
>
> [~sergey.soldatov] has been doing some great digging around a test failure 
> we've been seeing at $dayjob. The situation goes like this.
> 0. Run some arbitrary load
> 1. Stop HBase
> 2. Enable schema mapping ({{phoenix.schema.isNamespaceMappingEnabled=true}} 
> and {{phoenix.schema.mapSystemTablesToNamespace=true}} in hbase-site.xml)
> 3. Start HBase
> 4. Circumstantially, have the SYSTEM.CATALOG table need a compaction to run 
> before a client first connects
> When the RegionServer initiates the compaction, it will end up running 
> {{UngroupedAggregateRegionObserver.clearTsOnDisabledIndexes}} which opens a 
> Phoenix connection. While the RegionServer won't upgrade system tables, it 
> *will* try to migrate them into the schema mapped variants (e.g. 
> SYSTEM.CATALOG to SYSTEM:CATALOG).
> However, one of the first steps in the schema migration is to disable the 
> SYSTEM.CATALOG table. However, the SYSTEM.CATALOG table can't be disabled 
> until the region is CLOSED, and the region cannot be CLOSED until the 
> compaction is finished. *deadlock*
> The "obvious" fix is to avoid RegionServers from triggering system table 
> migrations, but Sergey and [~elserj] both think that this will end badly 
> (RegionServers falling over because they expect the tables to be migrated and 
> they aren't).
> Thoughts? [~ankit.singhal], [~jamestaylor], any others?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (PHOENIX-4537) RegionServer initiating compaction can trigger schema migration and deadlock the system

[James Taylor (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335178#comment-16335178
 ] 

James Taylor edited comment on PHOENIX-4537 at 1/22/18 11:47 PM:
-

The upgrade *should not* happen from a server-side connection. We have code in 
place for this already. See QueryUtil.getConnectionOnServer():
{code}
public static Connection getConnectionOnServer(Properties props, 
Configuration conf)
throws ClassNotFoundException,
SQLException {
UpgradeUtil.doNotUpgradeOnFirstConnection(props);
return getConnection(props, conf);
}
{code}
If we're going to go this route, can't we prevent this migration in a similar 
manner?

Sounds like we need a bit more discussion on this one before committing given 
the comfort level, [~elserj]. What about the idea I outlined above of doing 
PHOENIX-4530 and removing the clearTsOnDisabledIndexes altogether?



was (Author: jamestaylor):
The upgrade *should not* happen from a server-side connection. We have code in 
place for this already. See QueryUtil.getConnectionOnServer():
{code}
public static Connection getConnectionOnServer(Properties props, 
Configuration conf)
throws ClassNotFoundException,
SQLException {
UpgradeUtil.doNotUpgradeOnFirstConnection(props);
return getConnection(props, conf);
}
{code}

Sounds like we need a bit more discussion on this one before committing given 
the comfort level, [~elserj]. What about the idea I outlined above of doing 
PHOENIX-4530 and removing the clearTsOnDisabledIndexes altogether?


> RegionServer initiating compaction can trigger schema migration and deadlock 
> the system
> ---
>
> Key: PHOENIX-4537
> URL: https://issues.apache.org/jira/browse/PHOENIX-4537
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Romil Choksi
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 5.0.0, 4.14.0
>
> Attachments: PHOENIX-4537.001.patch
>
>
> [~sergey.soldatov] has been doing some great digging around a test failure 
> we've been seeing at $dayjob. The situation goes like this.
> 0. Run some arbitrary load
> 1. Stop HBase
> 2. Enable schema mapping ({{phoenix.schema.isNamespaceMappingEnabled=true}} 
> and {{phoenix.schema.mapSystemTablesToNamespace=true}} in hbase-site.xml)
> 3. Start HBase
> 4. Circumstantially, have the SYSTEM.CATALOG table need a compaction to run 
> before a client first connects
> When the RegionServer initiates the compaction, it will end up running 
> {{UngroupedAggregateRegionObserver.clearTsOnDisabledIndexes}} which opens a 
> Phoenix connection. While the RegionServer won't upgrade system tables, it 
> *will* try to migrate them into the schema mapped variants (e.g. 
> SYSTEM.CATALOG to SYSTEM:CATALOG).
> However, one of the first steps in the schema migration is to disable the 
> SYSTEM.CATALOG table. However, the SYSTEM.CATALOG table can't be disabled 
> until the region is CLOSED, and the region cannot be CLOSED until the 
> compaction is finished. *deadlock*
> The "obvious" fix is to avoid RegionServers from triggering system table 
> migrations, but Sergey and [~elserj] both think that this will end badly 
> (RegionServers falling over because they expect the tables to be migrated and 
> they aren't).
> Thoughts? [~ankit.singhal], [~jamestaylor], any others?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4537) RegionServer initiating compaction can trigger schema migration and deadlock the system

[James Taylor (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335178#comment-16335178
 ] 

James Taylor commented on PHOENIX-4537:
---

The upgrade *should not* happen from a server-side connection. We have code in 
place for this already. See QueryUtil.getConnectionOnServer():
{code}
public static Connection getConnectionOnServer(Properties props, 
Configuration conf)
throws ClassNotFoundException,
SQLException {
UpgradeUtil.doNotUpgradeOnFirstConnection(props);
return getConnection(props, conf);
}
{code}

Sounds like we need a bit more discussion on this one before committing given 
the comfort level, [~elserj]. What about the idea I outlined above of doing 
PHOENIX-4530 and removing the clearTsOnDisabledIndexes altogether?


> RegionServer initiating compaction can trigger schema migration and deadlock 
> the system
> ---
>
> Key: PHOENIX-4537
> URL: https://issues.apache.org/jira/browse/PHOENIX-4537
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Romil Choksi
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 5.0.0, 4.14.0
>
> Attachments: PHOENIX-4537.001.patch
>
>
> [~sergey.soldatov] has been doing some great digging around a test failure 
> we've been seeing at $dayjob. The situation goes like this.
> 0. Run some arbitrary load
> 1. Stop HBase
> 2. Enable schema mapping ({{phoenix.schema.isNamespaceMappingEnabled=true}} 
> and {{phoenix.schema.mapSystemTablesToNamespace=true}} in hbase-site.xml)
> 3. Start HBase
> 4. Circumstantially, have the SYSTEM.CATALOG table need a compaction to run 
> before a client first connects
> When the RegionServer initiates the compaction, it will end up running 
> {{UngroupedAggregateRegionObserver.clearTsOnDisabledIndexes}} which opens a 
> Phoenix connection. While the RegionServer won't upgrade system tables, it 
> *will* try to migrate them into the schema mapped variants (e.g. 
> SYSTEM.CATALOG to SYSTEM:CATALOG).
> However, one of the first steps in the schema migration is to disable the 
> SYSTEM.CATALOG table. However, the SYSTEM.CATALOG table can't be disabled 
> until the region is CLOSED, and the region cannot be CLOSED until the 
> compaction is finished. *deadlock*
> The "obvious" fix is to avoid RegionServers from triggering system table 
> migrations, but Sergey and [~elserj] both think that this will end badly 
> (RegionServers falling over because they expect the tables to be migrated and 
> they aren't).
> Thoughts? [~ankit.singhal], [~jamestaylor], any others?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4546) Projected immutable table cannot be read through ProjectedColumnExpression

[Sergey Soldatov (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335167#comment-16335167
 ] 

Sergey Soldatov commented on PHOENIX-4546:
--

I'm not sure whether it's a valid case. According to our documentation 
{noformat}
Deletes of rows in immutable tables are allowed with some restrictions if there 
are indexes on the table.
{noformat}
Doesn't it sound like we can't delete from immutable tables if there are any 
index table? [~jamestaylor] could you please confirm that I understand it 
correctly?

This particular patch would break even a simple select on the table with 
immutable rows:
{noformat}
0: jdbc:phoenix:> CREATE TABLE IF NOT EXISTS  A (ID INTEGER PRIMARY 
KEY,double_id DOUBLE,varchar_id VARCHAR (30));
No rows affected (1.277 seconds)
0: jdbc:phoenix:> ALTER TABLE A  set IMMUTABLE_ROWS=true;
No rows affected (0.094 seconds)
0: jdbc:phoenix:>
0: jdbc:phoenix:> CREATE INDEX IF NOT EXISTS index_column_varchar_id ON A 
(varchar_id);
No rows affected (7.298 seconds)
0: jdbc:phoenix:> CREATE INDEX IF NOT EXISTS index_column_double_id ON A 
(double_id);
No rows affected (6.28 seconds)
0: jdbc:phoenix:> UPSERT INTO A VALUES (900,0.5,'Sample text extra');
1 row affected (0.067 seconds)
0: jdbc:phoenix:> select * from a;
+--++-+
|ID| DOUBLE_ID  | VARCHAR_ID  |
+--++-+
| 900  | null   | |
+--++-+
1 row selected (0.042 seconds)
{noformat}
because instead of returning projected column in ColumnRef now we return 
expressions for double and varchar. 

> Projected immutable table cannot be read through ProjectedColumnExpression
> --
>
> Key: PHOENIX-4546
> URL: https://issues.apache.org/jira/browse/PHOENIX-4546
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Romil Choksi
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 5.0.0, 4.14.0
>
> Attachments: PHOENIX-4546.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4537) RegionServer initiating compaction can trigger schema migration and deadlock the system

[Josh Elser (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335134#comment-16335134
 ] 

Josh Elser commented on PHOENIX-4537:
-

bq. honestly speaking I'm not very happy that we perform migration due 
server-side connection (and it's still possible if the compaction happens for 
any user table during cluster restart). But since I'm not sure whether we have 
a better option atm, so we may live with this workaround for a while.

Acknowledged and I agree with you. Treating as separate issues seem best to me.

Let me add a comment to this change in the Observer and then I'll commit. 
Thanks!

> RegionServer initiating compaction can trigger schema migration and deadlock 
> the system
> ---
>
> Key: PHOENIX-4537
> URL: https://issues.apache.org/jira/browse/PHOENIX-4537
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Romil Choksi
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 5.0.0, 4.14.0
>
> Attachments: PHOENIX-4537.001.patch
>
>
> [~sergey.soldatov] has been doing some great digging around a test failure 
> we've been seeing at $dayjob. The situation goes like this.
> 0. Run some arbitrary load
> 1. Stop HBase
> 2. Enable schema mapping ({{phoenix.schema.isNamespaceMappingEnabled=true}} 
> and {{phoenix.schema.mapSystemTablesToNamespace=true}} in hbase-site.xml)
> 3. Start HBase
> 4. Circumstantially, have the SYSTEM.CATALOG table need a compaction to run 
> before a client first connects
> When the RegionServer initiates the compaction, it will end up running 
> {{UngroupedAggregateRegionObserver.clearTsOnDisabledIndexes}} which opens a 
> Phoenix connection. While the RegionServer won't upgrade system tables, it 
> *will* try to migrate them into the schema mapped variants (e.g. 
> SYSTEM.CATALOG to SYSTEM:CATALOG).
> However, one of the first steps in the schema migration is to disable the 
> SYSTEM.CATALOG table. However, the SYSTEM.CATALOG table can't be disabled 
> until the region is CLOSED, and the region cannot be CLOSED until the 
> compaction is finished. *deadlock*
> The "obvious" fix is to avoid RegionServers from triggering system table 
> migrations, but Sergey and [~elserj] both think that this will end badly 
> (RegionServers falling over because they expect the tables to be migrated and 
> they aren't).
> Thoughts? [~ankit.singhal], [~jamestaylor], any others?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4537) RegionServer initiating compaction can trigger schema migration and deadlock the system

[Sergey Soldatov (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335132#comment-16335132
 ] 

Sergey Soldatov commented on PHOENIX-4537:
--

[~elserj] honestly speaking I'm not very happy that we perform migration due 
server-side connection (and it's still possible if the compaction happens for 
any user table during cluster restart). But since I'm not sure whether we have 
a better option atm, so we may live with this workaround for a while. 

> RegionServer initiating compaction can trigger schema migration and deadlock 
> the system
> ---
>
> Key: PHOENIX-4537
> URL: https://issues.apache.org/jira/browse/PHOENIX-4537
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Romil Choksi
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 5.0.0, 4.14.0
>
> Attachments: PHOENIX-4537.001.patch
>
>
> [~sergey.soldatov] has been doing some great digging around a test failure 
> we've been seeing at $dayjob. The situation goes like this.
> 0. Run some arbitrary load
> 1. Stop HBase
> 2. Enable schema mapping ({{phoenix.schema.isNamespaceMappingEnabled=true}} 
> and {{phoenix.schema.mapSystemTablesToNamespace=true}} in hbase-site.xml)
> 3. Start HBase
> 4. Circumstantially, have the SYSTEM.CATALOG table need a compaction to run 
> before a client first connects
> When the RegionServer initiates the compaction, it will end up running 
> {{UngroupedAggregateRegionObserver.clearTsOnDisabledIndexes}} which opens a 
> Phoenix connection. While the RegionServer won't upgrade system tables, it 
> *will* try to migrate them into the schema mapped variants (e.g. 
> SYSTEM.CATALOG to SYSTEM:CATALOG).
> However, one of the first steps in the schema migration is to disable the 
> SYSTEM.CATALOG table. However, the SYSTEM.CATALOG table can't be disabled 
> until the region is CLOSED, and the region cannot be CLOSED until the 
> compaction is finished. *deadlock*
> The "obvious" fix is to avoid RegionServers from triggering system table 
> migrations, but Sergey and [~elserj] both think that this will end badly 
> (RegionServers falling over because they expect the tables to be migrated and 
> they aren't).
> Thoughts? [~ankit.singhal], [~jamestaylor], any others?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4537) RegionServer initiating compaction can trigger schema migration and deadlock the system

[Josh Elser (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335094#comment-16335094
 ] 

Josh Elser commented on PHOENIX-4537:
-

I've been unsuccessful in making an IT for this, but I have reproduced the 
problem locally and verified that this fix addresses the issue.

[~jamestaylor], [~sergey.soldatov], [~an...@apache.org], you guys think this is 
good to go?

> RegionServer initiating compaction can trigger schema migration and deadlock 
> the system
> ---
>
> Key: PHOENIX-4537
> URL: https://issues.apache.org/jira/browse/PHOENIX-4537
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Romil Choksi
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 5.0.0, 4.14.0
>
> Attachments: PHOENIX-4537.001.patch
>
>
> [~sergey.soldatov] has been doing some great digging around a test failure 
> we've been seeing at $dayjob. The situation goes like this.
> 0. Run some arbitrary load
> 1. Stop HBase
> 2. Enable schema mapping ({{phoenix.schema.isNamespaceMappingEnabled=true}} 
> and {{phoenix.schema.mapSystemTablesToNamespace=true}} in hbase-site.xml)
> 3. Start HBase
> 4. Circumstantially, have the SYSTEM.CATALOG table need a compaction to run 
> before a client first connects
> When the RegionServer initiates the compaction, it will end up running 
> {{UngroupedAggregateRegionObserver.clearTsOnDisabledIndexes}} which opens a 
> Phoenix connection. While the RegionServer won't upgrade system tables, it 
> *will* try to migrate them into the schema mapped variants (e.g. 
> SYSTEM.CATALOG to SYSTEM:CATALOG).
> However, one of the first steps in the schema migration is to disable the 
> SYSTEM.CATALOG table. However, the SYSTEM.CATALOG table can't be disabled 
> until the region is CLOSED, and the region cannot be CLOSED until the 
> compaction is finished. *deadlock*
> The "obvious" fix is to avoid RegionServers from triggering system table 
> migrations, but Sergey and [~elserj] both think that this will end badly 
> (RegionServers falling over because they expect the tables to be migrated and 
> they aren't).
> Thoughts? [~ankit.singhal], [~jamestaylor], any others?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (PHOENIX-4549) Pherf - Column override and sequenced index creation support

[Mujtaba Chohan (JIRA)]

Mujtaba Chohan created PHOENIX-4549:
---

 Summary: Pherf - Column override and sequenced index creation 
support
 Key: PHOENIX-4549
 URL: https://issues.apache.org/jira/browse/PHOENIX-4549
 Project: Phoenix
  Issue Type: Improvement
Reporter: Mujtaba Chohan
Assignee: Mujtaba Chohan
 Attachments: PHOENIX-4549.patch

Support for column level override for upserting view specific data along with 
sequenced index creation either before or after data is loaded. Patch also 
contains additional datatype support (TIMESTAMP, VARCHAR_ARRAY. VARBINARY AND 
UNSIGNED_LONG)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (PHOENIX-4549) Pherf - Column override and sequenced index creation support

[Mujtaba Chohan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mujtaba Chohan updated PHOENIX-4549:

Attachment: PHOENIX-4549.patch

> Pherf - Column override and sequenced index creation support
> 
>
> Key: PHOENIX-4549
> URL: https://issues.apache.org/jira/browse/PHOENIX-4549
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Mujtaba Chohan
>Assignee: Mujtaba Chohan
>Priority: Major
> Attachments: PHOENIX-4549.patch
>
>
> Support for column level override for upserting view specific data along with 
> sequenced index creation either before or after data is loaded. Patch also 
> contains additional datatype support (TIMESTAMP, VARCHAR_ARRAY. VARBINARY AND 
> UNSIGNED_LONG)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (PHOENIX-4382) Immutable table SINGLE_CELL_ARRAY_WITH_OFFSETS values starting with separator byte return null in query results

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor updated PHOENIX-4382:
--
Fix Version/s: 4.13.2-cdh5.11.2

> Immutable table SINGLE_CELL_ARRAY_WITH_OFFSETS values starting with separator 
> byte return null in query results
> ---
>
> Key: PHOENIX-4382
> URL: https://issues.apache.org/jira/browse/PHOENIX-4382
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.14.0
>Reporter: Vincent Poon
>Assignee: Vincent Poon
>Priority: Major
> Fix For: 5.0, 4.14.0, 4.13.2, 4.13.2-cdh5.11.2
>
> Attachments: PHOENIX-4382.v1.master.patch, 
> PHOENIX-4382.v2.master.patch, PHOENIX-4382.v3.master.patch, 
> PHOENIX-4382.v4.master.patch, UpsertBigValuesIT.java
>
>
> For immutable tables, upsert of some values like Short.MAX_VALUE results in a 
> null value in query resultsets.  Mutable tables are not affected.  I tried 
> with BigInt and got the same problem.
> For Short, the breaking point seems to be 32512.
> This is happening because of the way we serialize nulls.  For nulls, we write 
> out [separatorByte, #_of_nulls].  However, some data values, like 
> Short.MAX_VALUE, start with separatorByte, we can't distinguish between a 
> null and these values.  Currently the code assumes it's a null when it sees a 
> leading separatorByte, hence the incorrect query results.
> See attached test - testShort() , testBigInt()



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (PHOENIX-4382) Immutable table SINGLE_CELL_ARRAY_WITH_OFFSETS values starting with separator byte return null in query results

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor updated PHOENIX-4382:
--
Fix Version/s: 4.13.2

> Immutable table SINGLE_CELL_ARRAY_WITH_OFFSETS values starting with separator 
> byte return null in query results
> ---
>
> Key: PHOENIX-4382
> URL: https://issues.apache.org/jira/browse/PHOENIX-4382
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.14.0
>Reporter: Vincent Poon
>Assignee: Vincent Poon
>Priority: Major
> Fix For: 5.0, 4.14.0, 4.13.2, 4.13.2-cdh5.11.2
>
> Attachments: PHOENIX-4382.v1.master.patch, 
> PHOENIX-4382.v2.master.patch, PHOENIX-4382.v3.master.patch, 
> PHOENIX-4382.v4.master.patch, UpsertBigValuesIT.java
>
>
> For immutable tables, upsert of some values like Short.MAX_VALUE results in a 
> null value in query resultsets.  Mutable tables are not affected.  I tried 
> with BigInt and got the same problem.
> For Short, the breaking point seems to be 32512.
> This is happening because of the way we serialize nulls.  For nulls, we write 
> out [separatorByte, #_of_nulls].  However, some data values, like 
> Short.MAX_VALUE, start with separatorByte, we can't distinguish between a 
> null and these values.  Currently the code assumes it's a null when it sees a 
> leading separatorByte, hence the incorrect query results.
> See attached test - testShort() , testBigInt()



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4531) Delete on a table with a global mutable index can issue client-side deletes against the index

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334878#comment-16334878
 ] 

Hudson commented on PHOENIX-4531:
-

FAILURE: Integrated in Jenkins build Phoenix-master #1917 (See 
[https://builds.apache.org/job/Phoenix-master/1917/])
PHOENIX-4531 Delete on a table with a global mutable index can issue 
(vincentpoon: rev 195f82bae2835a6bfc9481205bad7e8f21944471)
* (edit) 
phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java
* (edit) 
phoenix-core/src/main/java/org/apache/phoenix/optimize/QueryOptimizer.java
* (edit) 
phoenix-core/src/it/java/org/apache/phoenix/end2end/index/BaseIndexIT.java
* (edit) 
phoenix-core/src/it/java/org/apache/phoenix/end2end/index/PartialIndexRebuilderIT.java
* (edit) 
phoenix-core/src/test/java/org/apache/phoenix/compile/QueryOptimizerTest.java


> Delete on a table with a global mutable index can issue client-side deletes 
> against the index
> -
>
> Key: PHOENIX-4531
> URL: https://issues.apache.org/jira/browse/PHOENIX-4531
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.13.0
> Environment:  
>Reporter: Vincent Poon
>Assignee: Vincent Poon
>Priority: Major
> Fix For: 4.14.0
>
> Attachments: PHOENIX-4531.v1.master.patch, 
> PHOENIX-4531.v3.master.patch, PHOENIX-4531.v4.master.patch, 
> PHOENIX-4531.v5.master.patch, PHOENIX-4531_v1.patch, PHOENIX-4531_v2.patch, 
> PartialIndexRebuilderIT.java
>
>
> For a table with a global mutable index, I found the following result in 
> client-side deletes against both the data table and index table.
> "DELETE FROM data_table" 
> "DELETE FROM data_table WHERE indexed_col='v'"
> We only need the delete to be issued against the data table, because
> 1) It's redundant since a delete against the index will be issued on the 
> server side when we process the delete of the data table row
> 2) Deletes issued from the client-side won't have the index failure policy



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4531) Delete on a table with a global mutable index can issue client-side deletes against the index

[Josh Elser (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334861#comment-16334861
 ] 

Josh Elser commented on PHOENIX-4531:
-

Also, I have it on my list to go back through the changes and get things synced 
up again between the 4.x and 5.x stuff. One more tacked onto that list isn't 
the end of the world.

> Delete on a table with a global mutable index can issue client-side deletes 
> against the index
> -
>
> Key: PHOENIX-4531
> URL: https://issues.apache.org/jira/browse/PHOENIX-4531
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.13.0
> Environment:  
>Reporter: Vincent Poon
>Assignee: Vincent Poon
>Priority: Major
> Fix For: 4.14.0
>
> Attachments: PHOENIX-4531.v1.master.patch, 
> PHOENIX-4531.v3.master.patch, PHOENIX-4531.v4.master.patch, 
> PHOENIX-4531.v5.master.patch, PHOENIX-4531_v1.patch, PHOENIX-4531_v2.patch, 
> PartialIndexRebuilderIT.java
>
>
> For a table with a global mutable index, I found the following result in 
> client-side deletes against both the data table and index table.
> "DELETE FROM data_table" 
> "DELETE FROM data_table WHERE indexed_col='v'"
> We only need the delete to be issued against the data table, because
> 1) It's redundant since a delete against the index will be issued on the 
> server side when we process the delete of the data table row
> 2) Deletes issued from the client-side won't have the index failure policy



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4531) Delete on a table with a global mutable index can issue client-side deletes against the index

[Josh Elser (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334858#comment-16334858
 ] 

Josh Elser commented on PHOENIX-4531:
-

cc/ [~sergey.soldatov] [~rajeshbabu]

> Delete on a table with a global mutable index can issue client-side deletes 
> against the index
> -
>
> Key: PHOENIX-4531
> URL: https://issues.apache.org/jira/browse/PHOENIX-4531
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.13.0
> Environment:  
>Reporter: Vincent Poon
>Assignee: Vincent Poon
>Priority: Major
> Fix For: 4.14.0
>
> Attachments: PHOENIX-4531.v1.master.patch, 
> PHOENIX-4531.v3.master.patch, PHOENIX-4531.v4.master.patch, 
> PHOENIX-4531.v5.master.patch, PHOENIX-4531_v1.patch, PHOENIX-4531_v2.patch, 
> PartialIndexRebuilderIT.java
>
>
> For a table with a global mutable index, I found the following result in 
> client-side deletes against both the data table and index table.
> "DELETE FROM data_table" 
> "DELETE FROM data_table WHERE indexed_col='v'"
> We only need the delete to be issued against the data table, because
> 1) It's redundant since a delete against the index will be issued on the 
> server side when we process the delete of the data table row
> 2) Deletes issued from the client-side won't have the index failure policy



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4531) Delete on a table with a global mutable index can issue client-side deletes against the index

[James Taylor (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334853#comment-16334853
 ] 

James Taylor commented on PHOENIX-4531:
---

Thanks, [~vincentpoon]. Looks like 4.x-HBase-1.1 was missing some previous 
commits. I've caught it up and cherry-picked your patch there.

We'll may need some help from [~elserj] & team for the 5.x branch. I don't have 
a good idea of what's missing there since they created the branch.

> Delete on a table with a global mutable index can issue client-side deletes 
> against the index
> -
>
> Key: PHOENIX-4531
> URL: https://issues.apache.org/jira/browse/PHOENIX-4531
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.13.0
> Environment:  
>Reporter: Vincent Poon
>Assignee: Vincent Poon
>Priority: Major
> Fix For: 4.14.0
>
> Attachments: PHOENIX-4531.v1.master.patch, 
> PHOENIX-4531.v3.master.patch, PHOENIX-4531.v4.master.patch, 
> PHOENIX-4531.v5.master.patch, PHOENIX-4531_v1.patch, PHOENIX-4531_v2.patch, 
> PartialIndexRebuilderIT.java
>
>
> For a table with a global mutable index, I found the following result in 
> client-side deletes against both the data table and index table.
> "DELETE FROM data_table" 
> "DELETE FROM data_table WHERE indexed_col='v'"
> We only need the delete to be issued against the data table, because
> 1) It's redundant since a delete against the index will be issued on the 
> server side when we process the delete of the data table row
> 2) Deletes issued from the client-side won't have the index failure policy



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (PHOENIX-4531) Delete on a table with a global mutable index can issue client-side deletes against the index

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor updated PHOENIX-4531:
--
Fix Version/s: 4.14.0

> Delete on a table with a global mutable index can issue client-side deletes 
> against the index
> -
>
> Key: PHOENIX-4531
> URL: https://issues.apache.org/jira/browse/PHOENIX-4531
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.13.0
> Environment:  
>Reporter: Vincent Poon
>Assignee: Vincent Poon
>Priority: Major
> Fix For: 4.14.0
>
> Attachments: PHOENIX-4531.v1.master.patch, 
> PHOENIX-4531.v3.master.patch, PHOENIX-4531.v4.master.patch, 
> PHOENIX-4531.v5.master.patch, PHOENIX-4531_v1.patch, PHOENIX-4531_v2.patch, 
> PartialIndexRebuilderIT.java
>
>
> For a table with a global mutable index, I found the following result in 
> client-side deletes against both the data table and index table.
> "DELETE FROM data_table" 
> "DELETE FROM data_table WHERE indexed_col='v'"
> We only need the delete to be issued against the data table, because
> 1) It's redundant since a delete against the index will be issued on the 
> server side when we process the delete of the data table row
> 2) Deletes issued from the client-side won't have the index failure policy



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] 5.0.0-beta release before month's end?

[James Taylor]

Have you guys back ported the removal of deprecated APIs patches to the 4.x
branches? That'll probably help minimize the merge conflicts we see going
forward.

On Thu, Jan 18, 2018 at 1:03 PM, James Taylor 
wrote:

> Awesome! That's great work!!
>
> On Thu, Jan 18, 2018 at 1:00 PM, Josh Elser  wrote:
>
>> Hah, funny you should ask! I was just thinking that I should send out a
>> note.
>>
>> * Rajeshbabu and Sergey are trying to track down a nasty issue that the
>> IndexScrutiny tool has caught where there are dangling index records
>> (PHOENIX-4534)
>> * Rajeshbabu is also looking into some local index failures
>> * Ankit has done some testing of the phoenix-hive integration, I think
>> phoenix-spark is also looking OK
>>
>> Last I chatted with folks, they were seeing >90% pass rate of the test
>> suite which seems pretty good to me. My plan was to work on a beta release
>> soon now that we have a beta from HBase. Things are generally functional as
>> of now -- I think getting it into the hands of folks to get more people
>> poking it would be great.
>>
>> Maybe rc0 next week?
>>
>> - Josh
>>
>>
>> On 1/18/18 1:11 PM, James Taylor wrote:
>>
>>> How are things looking with the 5.0.0 alpha/beta on HBase 2.x?
>>>
>>> On Thu, Jan 4, 2018 at 10:57 AM, Josh Elser  wrote:
>>>
>>> Good point. Perhaps "alpha" would be a better label?

 IIUC, the issue is that we need the HBase release, and then a Tephra
 release, and then we can get Tephra fixed for Phoenix5. Perhaps Ankit
 can
 provide some more color to the situation.


 On 1/4/18 12:07 PM, Nick Dimiduk wrote:

 Isn't Tephra integration mandatory for transaction support? What happens
> to
> a user who has TRANSACTIONAL=true tables when they upgrade? This can't
> really fail gracefully. I guess transaction support is still marked
> 'beta',
> but still, this would be a regression of functionality in "base
> Phoenix".
>
> On Thu, Jan 4, 2018 at 8:34 AM Josh Elser  wrote:
>
> Talked to Rajeshbabu and Ankit offline this morning.
>
>>
>> Sounds like there are a few integration points which are still
>> lacking:
>>
>> * phoenix-hive: PHOENIX-4423
>> * phoenix-spark: untested (probably broken against newest Spark)
>> * phoenix-kafka: untested (probably broken against newest Kafka -- see
>> PHOENIX-4515 PHOENIX-4516)
>> * Tephra integration: Needs a new release of Tephra with some fixes
>> Ankit helped with.
>>
>> I plan to not consider these 5.0.0-alpha/beta release blockers, we'll
>> just call those out which we don't get tested/fixed.
>>
>> On 1/2/18 1:08 PM, Josh Elser wrote:
>>
>> Happy New Year folks!
>>>
>>> I'd like to test the waters: what do people think about trying to
>>> get a
>>> 5.0.0 "beta" release out to the community before the end of January?
>>>
>>> HBase is doing the same right now with 2.0.0. My thinking is that if
>>> things are stable "enough", getting a base for people to use a 5.0
>>> Phoenix release more easily, we can catch more bugs and get a better
>>> product out the door.
>>>
>>> Thoughts/concerns? I'm happy to RM.
>>>
>>> - Josh
>>>
>>>
>>
>>
>
>>>
>

[jira] [Commented] (PHOENIX-4531) Delete on a table with a global mutable index can issue client-side deletes against the index

[Vincent Poon (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334817#comment-16334817
 ] 

Vincent Poon commented on PHOENIX-4531:
---

[~jamestaylor] could you create a version of your patch for 5.x-HBase-2.0, 
4.x-cdh5.11.2 and 4.x-HBase-1.1 ?

There are merge conflicts in DeleteCompiler.  If you can resolve those, I can 
commit to those branches

I've already committed to master, 4.x-HBase-1.3, 4.x-HBase-1.2, 4.x-HBase-0.98

Thanks!

> Delete on a table with a global mutable index can issue client-side deletes 
> against the index
> -
>
> Key: PHOENIX-4531
> URL: https://issues.apache.org/jira/browse/PHOENIX-4531
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.13.0
> Environment:  
>Reporter: Vincent Poon
>Assignee: Vincent Poon
>Priority: Major
> Attachments: PHOENIX-4531.v1.master.patch, 
> PHOENIX-4531.v3.master.patch, PHOENIX-4531.v4.master.patch, 
> PHOENIX-4531.v5.master.patch, PHOENIX-4531_v1.patch, PHOENIX-4531_v2.patch, 
> PartialIndexRebuilderIT.java
>
>
> For a table with a global mutable index, I found the following result in 
> client-side deletes against both the data table and index table.
> "DELETE FROM data_table" 
> "DELETE FROM data_table WHERE indexed_col='v'"
> We only need the delete to be issued against the data table, because
> 1) It's redundant since a delete against the index will be issued on the 
> server side when we process the delete of the data table row
> 2) Deletes issued from the client-side won't have the index failure policy



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

RE: [ANNOUNCE] Apache Phoenix 4.13.2 for CDH 5.11.2 released

[Sawant, Chandramohan]

Is there any documentation on cloudera site where it is announced about this 
news.
I checked with cloudera team and they are still not aware of this.


Regards,
CM
+1 201 763 1656

-Original Message-
From: James Taylor [mailto:jamestay...@apache.org] 
Sent: Saturday, January 20, 2018 6:02 PM
To: annou...@apache.org; u...@phoenix.apache.org
Cc: d...@hbase.apache.org; dev@phoenix.apache.org; u...@hbase.apache.org
Subject: Re: [ANNOUNCE] Apache Phoenix 4.13.2 for CDH 5.11.2 released

On Sat, Jan 20, 2018 at 12:29 PM Pedro Boado  wrote:

> The Apache Phoenix team is pleased to announce the immediate availability
> of the 4.13.2 release for CDH 5.11.2. Apache Phoenix enables SQL-based OLTP
> and operational analytics for Apache Hadoop using Apache HBase as its
> backing store and providing integration with other projects in the Apache
> ecosystem such as Spark, Hive, Pig, Flume, and MapReduce. This release is
> compatible with CDH 5.11.2
>
> Highlights of the release include:
>
> * For the first time, support for CDH due to community ask
> * More than 10 fixes over release 4.13.1-HBase-1.2
>
> Source and binary downloads are available here [1]. Folder 'parcels' can
> be directly used as parcel repository from Cloudera Manager ( follow [2] to
> one of Apache mirrors ) .
>
> Thanks,
> Pedro (on behalf of the Apache Phoenix team)
>
> [1] 
> https://urldefense.proofpoint.com/v2/url?u=http-3A__phoenix.apache.org_download.html=DwIBaQ=j-EkbjBYwkAB4f8ZbVn1Fw=_LbUM-zocIAZ2IaTRY8Qi3cPxGOZ63yD_eWhd5YhdP0=BNkA1jvifU56iaJTRPod7dpkbrxZgOjy5gy6rIUfnTo=0VdmvOaL1rAaG94Rnz3fTLN7y-yOyGRLWBSHkloXmk0=
> [2]
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.apache.org_dyn_closer.lua_phoenix_apache-2Dphoenix-2D4.13.2-2Dcdh5.11.2_parcels_=DwIBaQ=j-EkbjBYwkAB4f8ZbVn1Fw=_LbUM-zocIAZ2IaTRY8Qi3cPxGOZ63yD_eWhd5YhdP0=BNkA1jvifU56iaJTRPod7dpkbrxZgOjy5gy6rIUfnTo=umx9TTBDTbnbYrjBZ7vQcCkrM5HNzqhuA4DEPCVWgFc=
> 
>

[jira] [Updated] (PHOENIX-4548) UpgradeUtil.mapChildViewsToNamespace does not handle multi-tenant views that have the same name.

[Thomas D'Silva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas D'Silva updated PHOENIX-4548:

Attachment: PHOENIX-4548.patch

> UpgradeUtil.mapChildViewsToNamespace does not handle multi-tenant views that 
> have the same name.
> 
>
> Key: PHOENIX-4548
> URL: https://issues.apache.org/jira/browse/PHOENIX-4548
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Thomas D'Silva
>Assignee: Thomas D'Silva
>Priority: Major
> Fix For: 4.14.0
>
> Attachments: PHOENIX-4548.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (PHOENIX-4548) UpgradeUtil.mapChildViewsToNamespace does not handle multi-tenant views that have the same name.

[Thomas D'Silva (JIRA)]

Thomas D'Silva created PHOENIX-4548:
---

 Summary: UpgradeUtil.mapChildViewsToNamespace does not handle 
multi-tenant views that have the same name.
 Key: PHOENIX-4548
 URL: https://issues.apache.org/jira/browse/PHOENIX-4548
 Project: Phoenix
  Issue Type: Bug
Reporter: Thomas D'Silva
Assignee: Thomas D'Silva
 Fix For: 4.14.0






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table

[James Taylor (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334603#comment-16334603
 ] 

James Taylor commented on PHOENIX-3534:
---

FYI, the patch is now on the system-catalog feature branch.

> Support multi region SYSTEM.CATALOG table
> -
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Thomas D'Silva
>Priority: Major
> Attachments: PHOENIX-3534-wip.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region 
> based on the server-side row locks being held for operations that impact a 
> table and all of it's views. For example, adding/removing a column from a 
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), 
> when a new table is created we can do a lazy cleanup  of any rows that may be 
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with 
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051 
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on 
> checking our sequence numbers for each table/view before/after updating. Each 
> table/view row would be individually locked for their change (metadata for a 
> view or table cannot span regions due to our split policy), with the sequence 
> number being incremented under lock and then returned to the client.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4547) Use hbase 2.0.0-beta-1

[Josh Elser (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334572#comment-16334572
 ] 

Josh Elser commented on PHOENIX-4547:
-

Thanks, Rajesh!

> Use hbase 2.0.0-beta-1
> --
>
> Key: PHOENIX-4547
> URL: https://issues.apache.org/jira/browse/PHOENIX-4547
> Project: Phoenix
>  Issue Type: Task
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 5.0.0
>
> Attachments: PHOENIX-4547.5.x-HBase-2.0.patch
>
>
> Use the beta release.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)