[jira] [Comment Edited] (PHOENIX-3291) Do not throw return value of Throwables#propagate call

[James Taylor (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-3291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15498278#comment-15498278
 ] 

James Taylor edited comment on PHOENIX-3291 at 9/17/16 5:59 AM:


[~samarthjain] - simple patch - would you mind reviewing? This was driving me 
crazy as I was getting a NPE test failure with no stack trace. This change 
fixes that.


was (Author: jamestaylor):
[~samarthjain] - simple patch - would you mind reviewing? This was driving me 
crazy as I was getting a NPE test failure with no stack trace. This changed 
fixes that.

> Do not throw return value of Throwables#propagate call
> --
>
> Key: PHOENIX-3291
> URL: https://issues.apache.org/jira/browse/PHOENIX-3291
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: James Taylor
> Fix For: 4.9.0, 4.8.1
>
> Attachments: PHOENIX-3291.patch
>
>
> Several places in the code are doing the following which is wrong:
> {code}
> throw Throwables.propagate(e);
> {code}
> This seems to get rid of any stack trace.  Instead, it should be:
> {code}
> Throwables.propagate(e);
> throw new IllegalStateException(); // won't happen as above call throws
> {code}
> This preserves the stack trace.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Decode rowkey

[Anil]

HI,

Please use the attached utils. let me know if you see any issue. thanks.

Regards,
Anil



On 16 September 2016 at 22:45, Krishna  wrote:

> Hi,
>
> Does Phoenix have API for converting a rowkey (made up of multiple
> columns) and in ImmutableBytesRow format to split into primary key columns?
> I am performing a scan directly from HBase and would like to convert the
> rowkey into column values. We used Phoenix standard JDBC API while writing
> to the table.
>
> Thanks
>

[jira] [Updated] (PHOENIX-3290) Move and/or combine as many NeedsOwnCluster tests to bring down test run time

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor updated PHOENIX-3290:
--
Attachment: PHOENIX-3290_v5.patch

> Move and/or combine as many NeedsOwnCluster tests to bring down test run time
> -
>
> Key: PHOENIX-3290
> URL: https://issues.apache.org/jira/browse/PHOENIX-3290
> Project: Phoenix
>  Issue Type: Sub-task
>Reporter: James Taylor
> Attachments: PHOENIX-3290_WIP.patch, PHOENIX-3290_v2.patch, 
> PHOENIX-3290_v3.patch, PHOENIX-3290_v4.patch, PHOENIX-3290_v5.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (PHOENIX-3290) Move and/or combine as many NeedsOwnCluster tests to bring down test run time

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor updated PHOENIX-3290:
--
Attachment: PHOENIX-3290_v4.patch

> Move and/or combine as many NeedsOwnCluster tests to bring down test run time
> -
>
> Key: PHOENIX-3290
> URL: https://issues.apache.org/jira/browse/PHOENIX-3290
> Project: Phoenix
>  Issue Type: Sub-task
>Reporter: James Taylor
> Attachments: PHOENIX-3290_WIP.patch, PHOENIX-3290_v2.patch, 
> PHOENIX-3290_v3.patch, PHOENIX-3290_v4.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (PHOENIX-3290) Move and/or combine as many NeedsOwnCluster tests to bring down test run time

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor updated PHOENIX-3290:
--
Attachment: PHOENIX-3290_v3.patch

> Move and/or combine as many NeedsOwnCluster tests to bring down test run time
> -
>
> Key: PHOENIX-3290
> URL: https://issues.apache.org/jira/browse/PHOENIX-3290
> Project: Phoenix
>  Issue Type: Sub-task
>Reporter: James Taylor
> Attachments: PHOENIX-3290_WIP.patch, PHOENIX-3290_v2.patch, 
> PHOENIX-3290_v3.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (PHOENIX-3290) Move and/or combine as many NeedsOwnCluster tests to bring down test run time

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor updated PHOENIX-3290:
--
Attachment: (was: PHOENIX-3290_WIP.patch)

> Move and/or combine as many NeedsOwnCluster tests to bring down test run time
> -
>
> Key: PHOENIX-3290
> URL: https://issues.apache.org/jira/browse/PHOENIX-3290
> Project: Phoenix
>  Issue Type: Sub-task
>Reporter: James Taylor
> Attachments: PHOENIX-3290_WIP.patch, PHOENIX-3290_v2.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (PHOENIX-3290) Move and/or combine as many NeedsOwnCluster tests to bring down test run time

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor updated PHOENIX-3290:
--
Attachment: PHOENIX-3290_v2.patch

> Move and/or combine as many NeedsOwnCluster tests to bring down test run time
> -
>
> Key: PHOENIX-3290
> URL: https://issues.apache.org/jira/browse/PHOENIX-3290
> Project: Phoenix
>  Issue Type: Sub-task
>Reporter: James Taylor
> Attachments: PHOENIX-3290_WIP.patch, PHOENIX-3290_v2.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (PHOENIX-3291) Do not throw return value of Throwables#propagate call

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-3291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor updated PHOENIX-3291:
--
Attachment: PHOENIX-3291.patch

> Do not throw return value of Throwables#propagate call
> --
>
> Key: PHOENIX-3291
> URL: https://issues.apache.org/jira/browse/PHOENIX-3291
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: James Taylor
> Fix For: 4.9.0, 4.8.1
>
> Attachments: PHOENIX-3291.patch
>
>
> Several places in the code are doing the following which is wrong:
> {code}
> throw Throwables.propagate(e);
> {code}
> This seems to get rid of any stack trace.  Instead, it should be:
> {code}
> Throwables.propagate(e);
> throw new IllegalStateException(); // won't happen as above call throws
> {code}
> This preserves the stack trace.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (PHOENIX-3291) Do not throw return value of Throwables#propagate call

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-3291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor updated PHOENIX-3291:
--
Summary: Do not throw return value of Throwables#propagate call  (was: Do 
not throw return value of Throwables#propagate calls)

> Do not throw return value of Throwables#propagate call
> --
>
> Key: PHOENIX-3291
> URL: https://issues.apache.org/jira/browse/PHOENIX-3291
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: James Taylor
> Fix For: 4.9.0, 4.8.1
>
>
> Several places in the code are doing the following which is wrong:
> {code}
> throw Throwables.propagate(e);
> {code}
> This seems to get rid of any stack trace.  Instead, it should be:
> {code}
> Throwables.propagate(e);
> throw new IllegalStateException(); // won't happen as above call throws
> {code}
> This preserves the stack trace.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (PHOENIX-3291) Do not throw return value of Throwables#propagate calls

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-3291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor updated PHOENIX-3291:
--
Summary: Do not throw return value of Throwables#propagate calls  (was: Do 
not throw return value of Throwables.propagate(e))

> Do not throw return value of Throwables#propagate calls
> ---
>
> Key: PHOENIX-3291
> URL: https://issues.apache.org/jira/browse/PHOENIX-3291
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: James Taylor
> Fix For: 4.9.0, 4.8.1
>
>
> Several places in the code are doing the following which is wrong:
> {code}
> throw Throwables.propagate(e);
> {code}
> This seems to get rid of any stack trace.  Instead, it should be:
> {code}
> Throwables.propagate(e);
> throw new IllegalStateException(); // won't happen as above call throws
> {code}
> This preserves the stack trace.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (PHOENIX-3291) Do not throw return value of Throwables.propagate(e)

[James Taylor (JIRA)]

James Taylor created PHOENIX-3291:
-

 Summary: Do not throw return value of Throwables.propagate(e)
 Key: PHOENIX-3291
 URL: https://issues.apache.org/jira/browse/PHOENIX-3291
 Project: Phoenix
  Issue Type: Bug
Reporter: James Taylor
Assignee: James Taylor


Several places in the code are doing the following which is wrong:
{code}
throw Throwables.propagate(e);
{code}
This seems to get rid of any stack trace.  Instead, it should be:
{code}
Throwables.propagate(e);
throw new IllegalStateException(); // won't happen as above call throws
{code}
This preserves the stack trace.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (PHOENIX-3291) Do not throw return value of Throwables.propagate(e)

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-3291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor updated PHOENIX-3291:
--
Fix Version/s: 4.8.1
   4.9.0

> Do not throw return value of Throwables.propagate(e)
> 
>
> Key: PHOENIX-3291
> URL: https://issues.apache.org/jira/browse/PHOENIX-3291
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: James Taylor
> Fix For: 4.9.0, 4.8.1
>
>
> Several places in the code are doing the following which is wrong:
> {code}
> throw Throwables.propagate(e);
> {code}
> This seems to get rid of any stack trace.  Instead, it should be:
> {code}
> Throwables.propagate(e);
> throw new IllegalStateException(); // won't happen as above call throws
> {code}
> This preserves the stack trace.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (PHOENIX-3174) Make minor upgrade a manual step

[Samarth Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Samarth Jain updated PHOENIX-3174:
--
Attachment: PHOENIX-3174_v4_master.patch

Thanks for the review, [~jamestaylor]. Attached is the updated patch with tests 
added in UpgradeIT. I was also able to add a test for the concurrent upgrade 
scenario where an UpgradeInProgress exception is thrown. I also brought back 
the "do not upgrade" property since we do not want our server side code to 
unintentionally trigger upgrade when the autoupgrade property is set to true.



> Make minor upgrade a manual step
> 
>
> Key: PHOENIX-3174
> URL: https://issues.apache.org/jira/browse/PHOENIX-3174
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Samarth Jain
> Fix For: 4.9.0
>
> Attachments: PHOENIX-3174.patch, PHOENIX-3174_v2.patch, 
> PHOENIX-3174_v3_master.patch, PHOENIX-3174_v4_master.patch
>
>
> Instead of automatically performing minor release upgrades to system tables 
> in an automated manner (on first connection), we should make upgrade a 
> separate manual step. If a newer client attempts to connect to a newer server 
> without the upgrade step having occurred, we'd fail the connection.
> Though not as automated, this would give users more control and visibility 
> into when an upgrade over system tables occurs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Decode rowkey

[Jonathan Leech]

This would be really useful. The use case I have that is similar is to map 
Phoenix data to Hive (but the subset of Hive that Impala understands). I 
imagine it could work by reading the System.catalog table, or connection 
metadata, and generating Hive create table statements. There would need to be 
UDFs to split apart row keys and transform data, e.g. flipping the 1st byte of 
numeric types. You could use the same logic in the UDFs to read the data from a 
standalone hbase client.

> On Sep 16, 2016, at 11:15 AM, Krishna  wrote:
> 
> Hi,
> 
> Does Phoenix have API for converting a rowkey (made up of multiple columns) 
> and in ImmutableBytesRow format to split into primary key columns? I am 
> performing a scan directly from HBase and would like to convert the rowkey 
> into column values. We used Phoenix standard JDBC API while writing to the 
> table. 
> 
> Thanks

[jira] [Commented] (PHOENIX-476) Support declaration of DEFAULT in CREATE statement

[Julian Hyde (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15497639#comment-15497639
 ] 

Julian Hyde commented on PHOENIX-476:
-

I agree with [~kliew]. The result should be (1, NULL). {{DEFAULT}} kicks in 
only in INSERT or UPSERT, and only when the column is not mentioned. If you 
insert a NULL value, the value becomes NULL. 

The other way to get a default value is to mention the column explicitly and 
use DEFAULT, e.g. {{INSERT INTO testdefault (pk, test) VALUES (1, DEFAULT)}}.

> Support declaration of DEFAULT in CREATE statement
> --
>
> Key: PHOENIX-476
> URL: https://issues.apache.org/jira/browse/PHOENIX-476
> Project: Phoenix
>  Issue Type: Task
>Affects Versions: 3.0-Release
>Reporter: James Taylor
>Assignee: Kevin Liew
>  Labels: enhancement
> Attachments: PHOENIX-476.2.patch, PHOENIX-476.patch
>
>
> Support the declaration of a default value in the CREATE TABLE/VIEW statement 
> like this:
> CREATE TABLE Persons (
> Pid int NOT NULL PRIMARY KEY,
> LastName varchar(255) NOT NULL,
> FirstName varchar(255),
> Address varchar(255),
> City varchar(255) DEFAULT 'Sandnes'
> )
> To implement this, we'd need to:
> 1. add a new DEFAULT_VALUE key value column in SYSTEM.TABLE and pass through 
> the value when the table is created (in MetaDataClient).
> 2. always set NULLABLE to ResultSetMetaData.columnNoNulls if a default value 
> is present, since the column will never be null.
> 3. add a getDefaultValue() accessor in PColumn
> 4.  for a row key column, during UPSERT use the default value if no value was 
> specified for that column. This could be done in the PTableImpl.newKey method.
> 5.  for a key value column with a default value, we can get away without 
> incurring any storage cost. Although a little bit of extra effort than if we 
> persisted the default value on an UPSERT for key value columns, this approach 
> has the benefit of not incurring any storage cost for a default value.
> * serialize any default value into KeyValueColumnExpression
> * in the evaluate method of KeyValueColumnExpression, conditionally use 
> the default value if the column value is not present. If doing partial 
> evaluation, you should not yet return the default value, as we may not have 
> encountered the the KeyValue for the column yet (since a filter evaluates 
> each time it sees each KeyValue, and there may be more than one KeyValue 
> referenced in the expression). Partial evaluation is determined by calling 
> Tuple.isImmutable(), where false means it is NOT doing partial evaluation, 
> while true means it is.
> * modify EvaluateOnCompletionVisitor by adding a visitor method for 
> RowKeyColumnExpression and KeyValueColumnExpression to set 
> evaluateOnCompletion to true if they have a default value specified. This 
> will cause filter evaluation to execute one final time after all KeyValues 
> for a row have been seen, since it's at this time we know we should use the 
> default value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-476) Support declaration of DEFAULT in CREATE statement

[Kevin Liew (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15497545#comment-15497545
 ] 

Kevin Liew commented on PHOENIX-476:


The SQL92 spec states:
{noformat}
4) Let Q be the result of that .

Case:

a) If Q is empty, then no row is inserted and a completion con-
  dition is raised: no data.

b) Otherwise, for each row R of Q:

  i) A candidate row of B is effectively created in which the
 value of each column is its default value, as specified in
 the General Rules of Subclause 11.5, "".
 The candidate row includes every column of B.

 ii) For every object column in the candidate row, the value of
 the object column identified by the i-th  in
 the  is replaced by the i-th value of
 R.

iii) Let C be a column that is represented in the candidate row
 and let SV be its value in the candidate row. The General
 Rules of Subclause 9.2, "Store assignment", are applied to
 C and SV as TARGET and VALUE, respectively.

 iv) The candidate row is inserted into B.

 Note: The data values allowable in the candidate row may be
 constrained by a WITH CHECK OPTION constraint. The effect
 of a WITH CHECK OPTION constraint is defined in the General
 Rules of Subclause 11.19, "".
{noformat}

In step `b.i`, a candidate row with all default values will be constructed. 
Step `b.ii` is a bit ambiguous but I'd interpret that step such that a default 
value value could be overwritten with a null value. 

I tested with MySQL and Postgres and they both allow overwriting a default with 
a null value. Any input [~julianhyde]?

> Support declaration of DEFAULT in CREATE statement
> --
>
> Key: PHOENIX-476
> URL: https://issues.apache.org/jira/browse/PHOENIX-476
> Project: Phoenix
>  Issue Type: Task
>Affects Versions: 3.0-Release
>Reporter: James Taylor
>Assignee: Kevin Liew
>  Labels: enhancement
> Attachments: PHOENIX-476.2.patch, PHOENIX-476.patch
>
>
> Support the declaration of a default value in the CREATE TABLE/VIEW statement 
> like this:
> CREATE TABLE Persons (
> Pid int NOT NULL PRIMARY KEY,
> LastName varchar(255) NOT NULL,
> FirstName varchar(255),
> Address varchar(255),
> City varchar(255) DEFAULT 'Sandnes'
> )
> To implement this, we'd need to:
> 1. add a new DEFAULT_VALUE key value column in SYSTEM.TABLE and pass through 
> the value when the table is created (in MetaDataClient).
> 2. always set NULLABLE to ResultSetMetaData.columnNoNulls if a default value 
> is present, since the column will never be null.
> 3. add a getDefaultValue() accessor in PColumn
> 4.  for a row key column, during UPSERT use the default value if no value was 
> specified for that column. This could be done in the PTableImpl.newKey method.
> 5.  for a key value column with a default value, we can get away without 
> incurring any storage cost. Although a little bit of extra effort than if we 
> persisted the default value on an UPSERT for key value columns, this approach 
> has the benefit of not incurring any storage cost for a default value.
> * serialize any default value into KeyValueColumnExpression
> * in the evaluate method of KeyValueColumnExpression, conditionally use 
> the default value if the column value is not present. If doing partial 
> evaluation, you should not yet return the default value, as we may not have 
> encountered the the KeyValue for the column yet (since a filter evaluates 
> each time it sees each KeyValue, and there may be more than one KeyValue 
> referenced in the expression). Partial evaluation is determined by calling 
> Tuple.isImmutable(), where false means it is NOT doing partial evaluation, 
> while true means it is.
> * modify EvaluateOnCompletionVisitor by adding a visitor method for 
> RowKeyColumnExpression and KeyValueColumnExpression to set 
> evaluateOnCompletion to true if they have a default value specified. This 
> will cause filter evaluation to execute one final time after all KeyValues 
> for a row have been seen, since it's at this time we know we should use the 
> default value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (PHOENIX-3290) Move and/or combine as many NeedsOwnCluster tests to bring down test run time

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor updated PHOENIX-3290:
--
Attachment: PHOENIX-3290_WIP.patch

> Move and/or combine as many NeedsOwnCluster tests to bring down test run time
> -
>
> Key: PHOENIX-3290
> URL: https://issues.apache.org/jira/browse/PHOENIX-3290
> Project: Phoenix
>  Issue Type: Sub-task
>Reporter: James Taylor
> Attachments: PHOENIX-3290_WIP.patch, PHOENIX-3290_WIP.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (PHOENIX-3290) Move and/or combine as many NeedsOwnCluster tests to bring down test run time

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor updated PHOENIX-3290:
--
Attachment: PHOENIX-3290_WIP.patch

> Move and/or combine as many NeedsOwnCluster tests to bring down test run time
> -
>
> Key: PHOENIX-3290
> URL: https://issues.apache.org/jira/browse/PHOENIX-3290
> Project: Phoenix
>  Issue Type: Sub-task
>Reporter: James Taylor
> Attachments: PHOENIX-3290_WIP.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (PHOENIX-3290) Move and/or combine as many NeedsOwnCluster tests to bring down test run time

[James Taylor (JIRA)]

James Taylor created PHOENIX-3290:
-

 Summary: Move and/or combine as many NeedsOwnCluster tests to 
bring down test run time
 Key: PHOENIX-3290
 URL: https://issues.apache.org/jira/browse/PHOENIX-3290
 Project: Phoenix
  Issue Type: Sub-task
Reporter: James Taylor






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-476) Support declaration of DEFAULT in CREATE statement

[James Taylor (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15497244#comment-15497244
 ] 

James Taylor commented on PHOENIX-476:
--

If you set a column value with a default back to null, then you should get the 
default value when you query it.

> Support declaration of DEFAULT in CREATE statement
> --
>
> Key: PHOENIX-476
> URL: https://issues.apache.org/jira/browse/PHOENIX-476
> Project: Phoenix
>  Issue Type: Task
>Affects Versions: 3.0-Release
>Reporter: James Taylor
>Assignee: Kevin Liew
>  Labels: enhancement
> Attachments: PHOENIX-476.2.patch, PHOENIX-476.patch
>
>
> Support the declaration of a default value in the CREATE TABLE/VIEW statement 
> like this:
> CREATE TABLE Persons (
> Pid int NOT NULL PRIMARY KEY,
> LastName varchar(255) NOT NULL,
> FirstName varchar(255),
> Address varchar(255),
> City varchar(255) DEFAULT 'Sandnes'
> )
> To implement this, we'd need to:
> 1. add a new DEFAULT_VALUE key value column in SYSTEM.TABLE and pass through 
> the value when the table is created (in MetaDataClient).
> 2. always set NULLABLE to ResultSetMetaData.columnNoNulls if a default value 
> is present, since the column will never be null.
> 3. add a getDefaultValue() accessor in PColumn
> 4.  for a row key column, during UPSERT use the default value if no value was 
> specified for that column. This could be done in the PTableImpl.newKey method.
> 5.  for a key value column with a default value, we can get away without 
> incurring any storage cost. Although a little bit of extra effort than if we 
> persisted the default value on an UPSERT for key value columns, this approach 
> has the benefit of not incurring any storage cost for a default value.
> * serialize any default value into KeyValueColumnExpression
> * in the evaluate method of KeyValueColumnExpression, conditionally use 
> the default value if the column value is not present. If doing partial 
> evaluation, you should not yet return the default value, as we may not have 
> encountered the the KeyValue for the column yet (since a filter evaluates 
> each time it sees each KeyValue, and there may be more than one KeyValue 
> referenced in the expression). Partial evaluation is determined by calling 
> Tuple.isImmutable(), where false means it is NOT doing partial evaluation, 
> while true means it is.
> * modify EvaluateOnCompletionVisitor by adding a visitor method for 
> RowKeyColumnExpression and KeyValueColumnExpression to set 
> evaluateOnCompletion to true if they have a default value specified. This 
> will cause filter evaluation to execute one final time after all KeyValues 
> for a row have been seen, since it's at this time we know we should use the 
> default value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (PHOENIX-476) Support declaration of DEFAULT in CREATE statement

[Kevin Liew (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15497128#comment-15497128
 ] 

Kevin Liew edited comment on PHOENIX-476 at 9/16/16 7:15 PM:
-

If we consider the following case:

{noformat}
create table testdefault (pk integer primary key, test integer default 5);

upsert into testdefault values (1, null);

select * from testdefault;
{noformat}

We expect `1, ` but if we use `COALESCE` while retrieving the second 
column, would we get `1, 5`? 

I suppose CoalesceFunction will differentiate between null parse nodes and null 
values and that would give us the expected result?


was (Author: kliew):
If we consider the following case:

{noformat}
create table testdefault (pk integer primary key, test integer default 5);

upsert into testdefault values (1, null);

select * from testdefault;
{noformat}

We expect `1, ` but if we use `COALESCE` while retrieving the second 
column, would we get `1, 5`? 

I suppose CoalesceFunction will differentiate between null parse nodes and null 
values.

> Support declaration of DEFAULT in CREATE statement
> --
>
> Key: PHOENIX-476
> URL: https://issues.apache.org/jira/browse/PHOENIX-476
> Project: Phoenix
>  Issue Type: Task
>Affects Versions: 3.0-Release
>Reporter: James Taylor
>Assignee: Kevin Liew
>  Labels: enhancement
> Attachments: PHOENIX-476.2.patch, PHOENIX-476.patch
>
>
> Support the declaration of a default value in the CREATE TABLE/VIEW statement 
> like this:
> CREATE TABLE Persons (
> Pid int NOT NULL PRIMARY KEY,
> LastName varchar(255) NOT NULL,
> FirstName varchar(255),
> Address varchar(255),
> City varchar(255) DEFAULT 'Sandnes'
> )
> To implement this, we'd need to:
> 1. add a new DEFAULT_VALUE key value column in SYSTEM.TABLE and pass through 
> the value when the table is created (in MetaDataClient).
> 2. always set NULLABLE to ResultSetMetaData.columnNoNulls if a default value 
> is present, since the column will never be null.
> 3. add a getDefaultValue() accessor in PColumn
> 4.  for a row key column, during UPSERT use the default value if no value was 
> specified for that column. This could be done in the PTableImpl.newKey method.
> 5.  for a key value column with a default value, we can get away without 
> incurring any storage cost. Although a little bit of extra effort than if we 
> persisted the default value on an UPSERT for key value columns, this approach 
> has the benefit of not incurring any storage cost for a default value.
> * serialize any default value into KeyValueColumnExpression
> * in the evaluate method of KeyValueColumnExpression, conditionally use 
> the default value if the column value is not present. If doing partial 
> evaluation, you should not yet return the default value, as we may not have 
> encountered the the KeyValue for the column yet (since a filter evaluates 
> each time it sees each KeyValue, and there may be more than one KeyValue 
> referenced in the expression). Partial evaluation is determined by calling 
> Tuple.isImmutable(), where false means it is NOT doing partial evaluation, 
> while true means it is.
> * modify EvaluateOnCompletionVisitor by adding a visitor method for 
> RowKeyColumnExpression and KeyValueColumnExpression to set 
> evaluateOnCompletion to true if they have a default value specified. This 
> will cause filter evaluation to execute one final time after all KeyValues 
> for a row have been seen, since it's at this time we know we should use the 
> default value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (PHOENIX-476) Support declaration of DEFAULT in CREATE statement

[Kevin Liew (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15497128#comment-15497128
 ] 

Kevin Liew edited comment on PHOENIX-476 at 9/16/16 7:16 PM:
-

If we consider the following case:

{noformat}
create table testdefault (pk integer primary key, test integer default 5);

upsert into testdefault values (1, null);

select * from testdefault;
{noformat}

We expect `1, ` but if we use `COALESCE` while retrieving the second 
column, would we get `1, 5`? 

I suppose CoalesceFunction will differentiate between null `Expression` and 
null values and that would give us the expected result?


was (Author: kliew):
If we consider the following case:

{noformat}
create table testdefault (pk integer primary key, test integer default 5);

upsert into testdefault values (1, null);

select * from testdefault;
{noformat}

We expect `1, ` but if we use `COALESCE` while retrieving the second 
column, would we get `1, 5`? 

I suppose CoalesceFunction will differentiate between null parse nodes and null 
values and that would give us the expected result?

> Support declaration of DEFAULT in CREATE statement
> --
>
> Key: PHOENIX-476
> URL: https://issues.apache.org/jira/browse/PHOENIX-476
> Project: Phoenix
>  Issue Type: Task
>Affects Versions: 3.0-Release
>Reporter: James Taylor
>Assignee: Kevin Liew
>  Labels: enhancement
> Attachments: PHOENIX-476.2.patch, PHOENIX-476.patch
>
>
> Support the declaration of a default value in the CREATE TABLE/VIEW statement 
> like this:
> CREATE TABLE Persons (
> Pid int NOT NULL PRIMARY KEY,
> LastName varchar(255) NOT NULL,
> FirstName varchar(255),
> Address varchar(255),
> City varchar(255) DEFAULT 'Sandnes'
> )
> To implement this, we'd need to:
> 1. add a new DEFAULT_VALUE key value column in SYSTEM.TABLE and pass through 
> the value when the table is created (in MetaDataClient).
> 2. always set NULLABLE to ResultSetMetaData.columnNoNulls if a default value 
> is present, since the column will never be null.
> 3. add a getDefaultValue() accessor in PColumn
> 4.  for a row key column, during UPSERT use the default value if no value was 
> specified for that column. This could be done in the PTableImpl.newKey method.
> 5.  for a key value column with a default value, we can get away without 
> incurring any storage cost. Although a little bit of extra effort than if we 
> persisted the default value on an UPSERT for key value columns, this approach 
> has the benefit of not incurring any storage cost for a default value.
> * serialize any default value into KeyValueColumnExpression
> * in the evaluate method of KeyValueColumnExpression, conditionally use 
> the default value if the column value is not present. If doing partial 
> evaluation, you should not yet return the default value, as we may not have 
> encountered the the KeyValue for the column yet (since a filter evaluates 
> each time it sees each KeyValue, and there may be more than one KeyValue 
> referenced in the expression). Partial evaluation is determined by calling 
> Tuple.isImmutable(), where false means it is NOT doing partial evaluation, 
> while true means it is.
> * modify EvaluateOnCompletionVisitor by adding a visitor method for 
> RowKeyColumnExpression and KeyValueColumnExpression to set 
> evaluateOnCompletion to true if they have a default value specified. This 
> will cause filter evaluation to execute one final time after all KeyValues 
> for a row have been seen, since it's at this time we know we should use the 
> default value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (PHOENIX-476) Support declaration of DEFAULT in CREATE statement

[Kevin Liew (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15497128#comment-15497128
 ] 

Kevin Liew edited comment on PHOENIX-476 at 9/16/16 7:15 PM:
-

If we consider the following case:

{noformat}
create table testdefault (pk integer primary key, test integer default 5);

upsert into testdefault values (1, null);

select * from testdefault;
{noformat}

We expect `1, ` but if we use `COALESCE` while retrieving the second 
column, would we get `1, 5`? 

I suppose CoalesceFunction will differentiate between null parse nodes and null 
values.


was (Author: kliew):
If we consider the following case:

{noformat}
create table testdefault (pk integer primary key, test integer default 5);

upsert into testdefault values (1, null);

select * from testdefault;
{noformat}

We expect `1, ` but if we use `COALESCE` while retrieving the second 
column, would we get `1, 5`?

> Support declaration of DEFAULT in CREATE statement
> --
>
> Key: PHOENIX-476
> URL: https://issues.apache.org/jira/browse/PHOENIX-476
> Project: Phoenix
>  Issue Type: Task
>Affects Versions: 3.0-Release
>Reporter: James Taylor
>Assignee: Kevin Liew
>  Labels: enhancement
> Attachments: PHOENIX-476.2.patch, PHOENIX-476.patch
>
>
> Support the declaration of a default value in the CREATE TABLE/VIEW statement 
> like this:
> CREATE TABLE Persons (
> Pid int NOT NULL PRIMARY KEY,
> LastName varchar(255) NOT NULL,
> FirstName varchar(255),
> Address varchar(255),
> City varchar(255) DEFAULT 'Sandnes'
> )
> To implement this, we'd need to:
> 1. add a new DEFAULT_VALUE key value column in SYSTEM.TABLE and pass through 
> the value when the table is created (in MetaDataClient).
> 2. always set NULLABLE to ResultSetMetaData.columnNoNulls if a default value 
> is present, since the column will never be null.
> 3. add a getDefaultValue() accessor in PColumn
> 4.  for a row key column, during UPSERT use the default value if no value was 
> specified for that column. This could be done in the PTableImpl.newKey method.
> 5.  for a key value column with a default value, we can get away without 
> incurring any storage cost. Although a little bit of extra effort than if we 
> persisted the default value on an UPSERT for key value columns, this approach 
> has the benefit of not incurring any storage cost for a default value.
> * serialize any default value into KeyValueColumnExpression
> * in the evaluate method of KeyValueColumnExpression, conditionally use 
> the default value if the column value is not present. If doing partial 
> evaluation, you should not yet return the default value, as we may not have 
> encountered the the KeyValue for the column yet (since a filter evaluates 
> each time it sees each KeyValue, and there may be more than one KeyValue 
> referenced in the expression). Partial evaluation is determined by calling 
> Tuple.isImmutable(), where false means it is NOT doing partial evaluation, 
> while true means it is.
> * modify EvaluateOnCompletionVisitor by adding a visitor method for 
> RowKeyColumnExpression and KeyValueColumnExpression to set 
> evaluateOnCompletion to true if they have a default value specified. This 
> will cause filter evaluation to execute one final time after all KeyValues 
> for a row have been seen, since it's at this time we know we should use the 
> default value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-476) Support declaration of DEFAULT in CREATE statement

[Kevin Liew (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15497128#comment-15497128
 ] 

Kevin Liew commented on PHOENIX-476:


If we consider the following case:

{noformat}
create table testdefault (pk integer primary key, test integer default 5);

upsert into testdefault values (1, null);

select * from testdefault;
{noformat}

We expect `1, ` but if we use `COALESCE` while retrieving the second 
column, would we get `1, 5`?

> Support declaration of DEFAULT in CREATE statement
> --
>
> Key: PHOENIX-476
> URL: https://issues.apache.org/jira/browse/PHOENIX-476
> Project: Phoenix
>  Issue Type: Task
>Affects Versions: 3.0-Release
>Reporter: James Taylor
>Assignee: Kevin Liew
>  Labels: enhancement
> Attachments: PHOENIX-476.2.patch, PHOENIX-476.patch
>
>
> Support the declaration of a default value in the CREATE TABLE/VIEW statement 
> like this:
> CREATE TABLE Persons (
> Pid int NOT NULL PRIMARY KEY,
> LastName varchar(255) NOT NULL,
> FirstName varchar(255),
> Address varchar(255),
> City varchar(255) DEFAULT 'Sandnes'
> )
> To implement this, we'd need to:
> 1. add a new DEFAULT_VALUE key value column in SYSTEM.TABLE and pass through 
> the value when the table is created (in MetaDataClient).
> 2. always set NULLABLE to ResultSetMetaData.columnNoNulls if a default value 
> is present, since the column will never be null.
> 3. add a getDefaultValue() accessor in PColumn
> 4.  for a row key column, during UPSERT use the default value if no value was 
> specified for that column. This could be done in the PTableImpl.newKey method.
> 5.  for a key value column with a default value, we can get away without 
> incurring any storage cost. Although a little bit of extra effort than if we 
> persisted the default value on an UPSERT for key value columns, this approach 
> has the benefit of not incurring any storage cost for a default value.
> * serialize any default value into KeyValueColumnExpression
> * in the evaluate method of KeyValueColumnExpression, conditionally use 
> the default value if the column value is not present. If doing partial 
> evaluation, you should not yet return the default value, as we may not have 
> encountered the the KeyValue for the column yet (since a filter evaluates 
> each time it sees each KeyValue, and there may be more than one KeyValue 
> referenced in the expression). Partial evaluation is determined by calling 
> Tuple.isImmutable(), where false means it is NOT doing partial evaluation, 
> while true means it is.
> * modify EvaluateOnCompletionVisitor by adding a visitor method for 
> RowKeyColumnExpression and KeyValueColumnExpression to set 
> evaluateOnCompletion to true if they have a default value specified. This 
> will cause filter evaluation to execute one final time after all KeyValues 
> for a row have been seen, since it's at this time we know we should use the 
> default value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (PHOENIX-476) Support declaration of DEFAULT in CREATE statement

[Kevin Liew (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496816#comment-15496816
 ] 

Kevin Liew edited comment on PHOENIX-476 at 9/16/16 7:05 PM:
-

Thanks for the feedback [~jamestaylor].

I attached a patch with  values implemented for primary keys. The  expression 
node is compiled and cached in CreateTableCompiler and accessed in PTableImpl. 
Will the compiled expression persist between server restarts? Do I need to 
recompile the  expression in MetadataClient so that the  values of existing 
tables are cached on server-restart?

* For non-pk columns, if we evaluate the  expression once on  and wrap the 
column in `COALESCE` to replace `NULL` values with the  expression, the user 
can explicitly set values (for columns with default values) to `NULL` but the 
values will always be  as the  value. We can only make this optimization for 
columns that have a `NOT NULL` constraint.

* For nullable columns and stateful default expressions (or non-literal 
expressions), we will probably have to  the value during the `UPSERT` 
execution. In my previous patch, did I make changes in the right locations 
(UpsertCompiler) for these specific cases?


was (Author: kliew):
Thanks for the feedback [~jamestaylor].

I attached a patch with  values implemented for primary keys. The  expression 
node is compiled and cached in CreateTableCompiler and accessed in PTableImpl. 
Will the compiled expression persist between server restarts? Do I need to 
recompile the  expression in MetadataClient so that the  values of existing 
tables are cached on server-restart?

* For non-pk columns, if we evaluate the  expression once on  and wrap the 
column in `COALESCE` to replace `NULL` values with the  expression, the user 
can explicitly set values (for columns with  values) to `NULL` but the values 
will always be  as the  value. We can only make this optimization for columns 
that have a `NOT NULL` constraint.

* For nullable columns and stateful default expressions (or non-literal 
expressions), we will probably have to  the value during the `UPSERT` 
execution. In my previous patch, did I make changes in the right locations 
(UpsertCompiler) for these specific cases?

> Support declaration of DEFAULT in CREATE statement
> --
>
> Key: PHOENIX-476
> URL: https://issues.apache.org/jira/browse/PHOENIX-476
> Project: Phoenix
>  Issue Type: Task
>Affects Versions: 3.0-Release
>Reporter: James Taylor
>Assignee: Kevin Liew
>  Labels: enhancement
> Attachments: PHOENIX-476.2.patch, PHOENIX-476.patch
>
>
> Support the declaration of a default value in the CREATE TABLE/VIEW statement 
> like this:
> CREATE TABLE Persons (
> Pid int NOT NULL PRIMARY KEY,
> LastName varchar(255) NOT NULL,
> FirstName varchar(255),
> Address varchar(255),
> City varchar(255) DEFAULT 'Sandnes'
> )
> To implement this, we'd need to:
> 1. add a new DEFAULT_VALUE key value column in SYSTEM.TABLE and pass through 
> the value when the table is created (in MetaDataClient).
> 2. always set NULLABLE to ResultSetMetaData.columnNoNulls if a default value 
> is present, since the column will never be null.
> 3. add a getDefaultValue() accessor in PColumn
> 4.  for a row key column, during UPSERT use the default value if no value was 
> specified for that column. This could be done in the PTableImpl.newKey method.
> 5.  for a key value column with a default value, we can get away without 
> incurring any storage cost. Although a little bit of extra effort than if we 
> persisted the default value on an UPSERT for key value columns, this approach 
> has the benefit of not incurring any storage cost for a default value.
> * serialize any default value into KeyValueColumnExpression
> * in the evaluate method of KeyValueColumnExpression, conditionally use 
> the default value if the column value is not present. If doing partial 
> evaluation, you should not yet return the default value, as we may not have 
> encountered the the KeyValue for the column yet (since a filter evaluates 
> each time it sees each KeyValue, and there may be more than one KeyValue 
> referenced in the expression). Partial evaluation is determined by calling 
> Tuple.isImmutable(), where false means it is NOT doing partial evaluation, 
> while true means it is.
> * modify EvaluateOnCompletionVisitor by adding a visitor method for 
> RowKeyColumnExpression and KeyValueColumnExpression to set 
> evaluateOnCompletion to true if they have a default value specified. This 
> will cause filter evaluation to execute one final time after all KeyValues 
> for a row have been seen, since it's at this time we know we should use the 
> default value.



--
This message was sent 

[jira] [Commented] (PHOENIX-476) Support declaration of DEFAULT in CREATE statement

[James Taylor (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15497020#comment-15497020
 ] 

James Taylor commented on PHOENIX-476:
--

Thanks for the updated patch, [~kliew].

bq. For non-pk columns, if we evaluate the expression once on and wrap the 
column in `COALESCE` to replace `NULL` values with the expression
See my bullet point above starting with "For other, non PK columns, you don't 
need to store the value at all". You wouldn't evaluate the expression. You're 
replacing an expression of {{col1}} with {{COALESCE(col1,)}}. 
This will do the right thing. No need to write the default value for non pk 
columns.



> Support declaration of DEFAULT in CREATE statement
> --
>
> Key: PHOENIX-476
> URL: https://issues.apache.org/jira/browse/PHOENIX-476
> Project: Phoenix
>  Issue Type: Task
>Affects Versions: 3.0-Release
>Reporter: James Taylor
>Assignee: Kevin Liew
>  Labels: enhancement
> Attachments: PHOENIX-476.2.patch, PHOENIX-476.patch
>
>
> Support the declaration of a default value in the CREATE TABLE/VIEW statement 
> like this:
> CREATE TABLE Persons (
> Pid int NOT NULL PRIMARY KEY,
> LastName varchar(255) NOT NULL,
> FirstName varchar(255),
> Address varchar(255),
> City varchar(255) DEFAULT 'Sandnes'
> )
> To implement this, we'd need to:
> 1. add a new DEFAULT_VALUE key value column in SYSTEM.TABLE and pass through 
> the value when the table is created (in MetaDataClient).
> 2. always set NULLABLE to ResultSetMetaData.columnNoNulls if a default value 
> is present, since the column will never be null.
> 3. add a getDefaultValue() accessor in PColumn
> 4.  for a row key column, during UPSERT use the default value if no value was 
> specified for that column. This could be done in the PTableImpl.newKey method.
> 5.  for a key value column with a default value, we can get away without 
> incurring any storage cost. Although a little bit of extra effort than if we 
> persisted the default value on an UPSERT for key value columns, this approach 
> has the benefit of not incurring any storage cost for a default value.
> * serialize any default value into KeyValueColumnExpression
> * in the evaluate method of KeyValueColumnExpression, conditionally use 
> the default value if the column value is not present. If doing partial 
> evaluation, you should not yet return the default value, as we may not have 
> encountered the the KeyValue for the column yet (since a filter evaluates 
> each time it sees each KeyValue, and there may be more than one KeyValue 
> referenced in the expression). Partial evaluation is determined by calling 
> Tuple.isImmutable(), where false means it is NOT doing partial evaluation, 
> while true means it is.
> * modify EvaluateOnCompletionVisitor by adding a visitor method for 
> RowKeyColumnExpression and KeyValueColumnExpression to set 
> evaluateOnCompletion to true if they have a default value specified. This 
> will cause filter evaluation to execute one final time after all KeyValues 
> for a row have been seen, since it's at this time we know we should use the 
> default value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (PHOENIX-476) Support declaration of DEFAULT in CREATE statement

[Kevin Liew (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496816#comment-15496816
 ] 

Kevin Liew edited comment on PHOENIX-476 at 9/16/16 6:21 PM:
-

Thanks for the feedback [~jamestaylor].

I attached a patch with  values implemented for primary keys. The  expression 
node is compiled and cached in CreateTableCompiler and accessed in PTableImpl. 
Will the compiled expression persist between server restarts? Do I need to 
recompile the  expression in MetadataClient so that the  values of existing 
tables are cached on server-restart?

* For non-pk columns, if we evaluate the  expression once on  and wrap the 
column in `COALESCE` to replace `NULL` values with the  expression, the user 
can explicitly set values (for columns with  values) to `NULL` but the values 
will always be  as the  value. We can only make this optimization for columns 
that have a `NOT NULL` constraint.

* For nullable columns and stateful default expressions (or non-literal 
expressions), we will probably have to  the value during the `UPSERT` 
execution. In my previous patch, did I make changes in the right locations 
(UpsertCompiler) for these specific cases?


was (Author: kliew):
Thanks for the feedback [~jamestaylor].

I attached a patch with default values implemented for primary keys. The 
default expression node is compiled and cached in CreateTableCompiler and 
accessed in PTableImpl. Will the compiled expression persist between server 
restarts? Do I need to recompile the default expression in MetadataClient so 
that the default values of existing tables are cached on server-restart?

* For non-pk columns, if we evaluate the default expression once on read and 
wrap the column in `COALESCE` to replace `NULL` values with the default 
expression, the user can explicitly set values (for columns with default 
values) to `NULL` but the values will always be read as the default value. We 
can only make this optimization for columns that have a `NOT NULL` constraint.

* For nullable columns and stateful default expressions, we will probably have 
to store the value during the `UPSERT` execution. In my previous patch, did I 
make changes in the right locations (UpsertCompiler) for these specific cases?

> Support declaration of DEFAULT in CREATE statement
> --
>
> Key: PHOENIX-476
> URL: https://issues.apache.org/jira/browse/PHOENIX-476
> Project: Phoenix
>  Issue Type: Task
>Affects Versions: 3.0-Release
>Reporter: James Taylor
>Assignee: Kevin Liew
>  Labels: enhancement
> Attachments: PHOENIX-476.2.patch, PHOENIX-476.patch
>
>
> Support the declaration of a default value in the CREATE TABLE/VIEW statement 
> like this:
> CREATE TABLE Persons (
> Pid int NOT NULL PRIMARY KEY,
> LastName varchar(255) NOT NULL,
> FirstName varchar(255),
> Address varchar(255),
> City varchar(255) DEFAULT 'Sandnes'
> )
> To implement this, we'd need to:
> 1. add a new DEFAULT_VALUE key value column in SYSTEM.TABLE and pass through 
> the value when the table is created (in MetaDataClient).
> 2. always set NULLABLE to ResultSetMetaData.columnNoNulls if a default value 
> is present, since the column will never be null.
> 3. add a getDefaultValue() accessor in PColumn
> 4.  for a row key column, during UPSERT use the default value if no value was 
> specified for that column. This could be done in the PTableImpl.newKey method.
> 5.  for a key value column with a default value, we can get away without 
> incurring any storage cost. Although a little bit of extra effort than if we 
> persisted the default value on an UPSERT for key value columns, this approach 
> has the benefit of not incurring any storage cost for a default value.
> * serialize any default value into KeyValueColumnExpression
> * in the evaluate method of KeyValueColumnExpression, conditionally use 
> the default value if the column value is not present. If doing partial 
> evaluation, you should not yet return the default value, as we may not have 
> encountered the the KeyValue for the column yet (since a filter evaluates 
> each time it sees each KeyValue, and there may be more than one KeyValue 
> referenced in the expression). Partial evaluation is determined by calling 
> Tuple.isImmutable(), where false means it is NOT doing partial evaluation, 
> while true means it is.
> * modify EvaluateOnCompletionVisitor by adding a visitor method for 
> RowKeyColumnExpression and KeyValueColumnExpression to set 
> evaluateOnCompletion to true if they have a default value specified. This 
> will cause filter evaluation to execute one final time after all KeyValues 
> for a row have been seen, since it's at this time we know we should use the 
> default 

Re: Decode rowkey

[James Taylor]

See
http://search-hadoop.com/m/9UY0h2ZCvra1NlgtC1=Re+Extracting+column+values+from+Phoenix+composite+primary+key

On Fri, Sep 16, 2016 at 10:46 AM, Jonathan Leech  wrote:

> This would be really useful. The use case I have that is similar is to map
> Phoenix data to Hive (but the subset of Hive that Impala understands). I
> imagine it could work by reading the System.catalog table, or connection
> metadata, and generating Hive create table statements. There would need to
> be UDFs to split apart row keys and transform data, e.g. flipping the 1st
> byte of numeric types. You could use the same logic in the UDFs to read the
> data from a standalone hbase client.
>
> On Sep 16, 2016, at 11:15 AM, Krishna  wrote:
>
> Hi,
>
> Does Phoenix have API for converting a rowkey (made up of multiple
> columns) and in ImmutableBytesRow format to split into primary key columns?
> I am performing a scan directly from HBase and would like to convert the
> rowkey into column values. We used Phoenix standard JDBC API while writing
> to the table.
>
> Thanks
>
>

Decode rowkey

[Krishna]

Hi,

Does Phoenix have API for converting a rowkey (made up of multiple columns)
and in ImmutableBytesRow format to split into primary key columns? I am
performing a scan directly from HBase and would like to convert the rowkey
into column values. We used Phoenix standard JDBC API while writing to the
table.

Thanks

[jira] [Created] (PHOENIX-3289) Region servers crashing randomly with "Native memory allocation (malloc) failed to allocate 12288 bytes for committing reserved memory"

[Ankit Singhal (JIRA)]

Ankit Singhal created PHOENIX-3289:
--

 Summary: Region servers crashing randomly with "Native memory 
allocation (malloc) failed to allocate 12288 bytes for committing reserved 
memory"
 Key: PHOENIX-3289
 URL: https://issues.apache.org/jira/browse/PHOENIX-3289
 Project: Phoenix
  Issue Type: Bug
Reporter: Ankit Singhal


for GROUP BY case, we try to keep the data in memory but if we exceed the total 
memory given for global cache(phoenix.query.maxGlobalMemorySize), then we start 
spilling the data to the disk. We spill and map them with MappedByteBuffer  and 
add bloom filter so that they can be accessed faster.

MappedByteBuffer doesn't release memory immediately on close(),  GC needs to 
collect them when it find such buffers with no references. Though, we are 
closing the channel and file properly and deleting the file at the end, it's 
possible that GCs has not run for a while and this map count is increasing.

PFB, parent ticket talking about the memory leak with the FileChannel.map() 
function.
http://bugs.java.com/view_bug.do?bug_id=4724038

And ,related ticket
http://bugs.java.com/view_bug.do?bug_id=6558368

But now the problems is that we are keeping page size of 4KB only for each 
mmap, which seems to be very small. I don't know the rationale behind keeping 
it so low, may be for performance to get a single key all the time but it can 
result in multiple mmap for even small file , so we can thought of keeping it 
as 64KB page size or more as default but we need to see the impact on 
performance.

Workaround to mask the issue, increasing the phoenix.query.maxGlobalMemorySize 
and increasing vm.max_map_count is a solution for get going 

thanks [~rmaruthiyodan] for reducing the territory of the problem by analyzing 
VM hostspot error file and detecting the increase in mmap count during GroupBy 
queries.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (PHOENIX-476) Support declaration of DEFAULT in CREATE statement

[Kevin Liew (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Liew updated PHOENIX-476:
---
Attachment: PHOENIX-476.2.patch

Thanks for the feedback [~jamestaylor].

I attached a patch with default values implemented for primary keys. The 
default expression node is compiled and cached in CreateTableCompiler and 
accessed in PTableImpl. Will the compiled expression persist between server 
restarts? Do I need to recompile the default expression in MetadataClient so 
that the default values of existing tables are cached on server-restart?

* For non-pk columns, if we evaluate the default expression once on read and 
wrap the column in `COALESCE` to replace `NULL` values with the default 
expression, the user can explicitly set values (for columns with default 
values) to `NULL` but the values will always be read as the default value. We 
can only make this optimization for columns that have a `NOT NULL` constraint.

* For nullable columns and stateful default expressions, we will probably have 
to store the value during the `UPSERT` execution. In my previous patch, did I 
make changes in the right locations (UpsertCompiler) for these specific cases?

> Support declaration of DEFAULT in CREATE statement
> --
>
> Key: PHOENIX-476
> URL: https://issues.apache.org/jira/browse/PHOENIX-476
> Project: Phoenix
>  Issue Type: Task
>Affects Versions: 3.0-Release
>Reporter: James Taylor
>Assignee: Kevin Liew
>  Labels: enhancement
> Attachments: PHOENIX-476.2.patch, PHOENIX-476.patch
>
>
> Support the declaration of a default value in the CREATE TABLE/VIEW statement 
> like this:
> CREATE TABLE Persons (
> Pid int NOT NULL PRIMARY KEY,
> LastName varchar(255) NOT NULL,
> FirstName varchar(255),
> Address varchar(255),
> City varchar(255) DEFAULT 'Sandnes'
> )
> To implement this, we'd need to:
> 1. add a new DEFAULT_VALUE key value column in SYSTEM.TABLE and pass through 
> the value when the table is created (in MetaDataClient).
> 2. always set NULLABLE to ResultSetMetaData.columnNoNulls if a default value 
> is present, since the column will never be null.
> 3. add a getDefaultValue() accessor in PColumn
> 4.  for a row key column, during UPSERT use the default value if no value was 
> specified for that column. This could be done in the PTableImpl.newKey method.
> 5.  for a key value column with a default value, we can get away without 
> incurring any storage cost. Although a little bit of extra effort than if we 
> persisted the default value on an UPSERT for key value columns, this approach 
> has the benefit of not incurring any storage cost for a default value.
> * serialize any default value into KeyValueColumnExpression
> * in the evaluate method of KeyValueColumnExpression, conditionally use 
> the default value if the column value is not present. If doing partial 
> evaluation, you should not yet return the default value, as we may not have 
> encountered the the KeyValue for the column yet (since a filter evaluates 
> each time it sees each KeyValue, and there may be more than one KeyValue 
> referenced in the expression). Partial evaluation is determined by calling 
> Tuple.isImmutable(), where false means it is NOT doing partial evaluation, 
> while true means it is.
> * modify EvaluateOnCompletionVisitor by adding a visitor method for 
> RowKeyColumnExpression and KeyValueColumnExpression to set 
> evaluateOnCompletion to true if they have a default value specified. This 
> will cause filter evaluation to execute one final time after all KeyValues 
> for a row have been seen, since it's at this time we know we should use the 
> default value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-3162) TableNotFoundException might be thrown when an index dropped while upserting.

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496796#comment-15496796
 ] 

Hudson commented on PHOENIX-3162:
-

SUCCESS: Integrated in Jenkins build Phoenix-4.8-HBase-1.2 #25 (See 
[https://builds.apache.org/job/Phoenix-4.8-HBase-1.2/25/])
PHOENIX-3162 TableNotFoundException might be thrown when an index (rajeshbabu: 
rev 614f870bdeff324ce2236a64d6c5465ced649884)
* (edit) 
phoenix-core/src/main/java/org/apache/phoenix/index/PhoenixIndexFailurePolicy.java
* (edit) 
phoenix-core/src/it/java/org/apache/phoenix/end2end/index/MutableIndexFailureIT.java


> TableNotFoundException might be thrown when an index dropped while upserting.
> -
>
> Key: PHOENIX-3162
> URL: https://issues.apache.org/jira/browse/PHOENIX-3162
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.8.1
>
> Attachments: PHOENIX-3162.patch, PHOENIX-3162_v2.patch, 
> PHOENIX-3162_v3.patch
>
>
> If a table has mix of global and local indexes and one of them is dropped 
> while upserting data then there is a chance that the query might fail with 
> TableNotFoundException. Usually when an index dropped we skip writing to the 
> dropped index on failure.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-3159) CachingHTableFactory may close HTable during eviction even if it is getting used for writing by another thread.

[Devaraj Das (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496617#comment-15496617
 ] 

Devaraj Das commented on PHOENIX-3159:
--

Will get to it today or over the weekend James

> CachingHTableFactory may close HTable during eviction even if it is getting 
> used for writing by another thread.
> ---
>
> Key: PHOENIX-3159
> URL: https://issues.apache.org/jira/browse/PHOENIX-3159
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
> Fix For: 4.8.1
>
> Attachments: PHOENIX-3159.patch, PHOENIX-3159_v1.patch, 
> PHOENIX-3159_v2.patch, PHOENIX-3159_v3.patch
>
>
> CachingHTableFactory may close HTable during eviction even if it is getting 
> used for writing by another thread which results in writing thread to fail 
> and index is disabled.
> LRU eviction closing HTable or underlying connection when cache is full and 
> new HTable is requested.
> {code}
> 2016-08-04 13:45:21,109 DEBUG 
> [nat-s11-4-ioss-phoenix-1-5.openstacklocal,16020,1470297472814-index-writer--pool11-t35]
>  client.ConnectionManager$HConnectionImplementation: Closing HConnection 
> (debugging purposes only)
> java.lang.Exception
> at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.internalClose(ConnectionManager.java:2423)
> at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.close(ConnectionManager.java:2447)
> at 
> org.apache.hadoop.hbase.client.CoprocessorHConnection.close(CoprocessorHConnection.java:41)
> at 
> org.apache.hadoop.hbase.client.HTableWrapper.internalClose(HTableWrapper.java:91)
> at 
> org.apache.hadoop.hbase.client.HTableWrapper.close(HTableWrapper.java:107)
> at 
> org.apache.phoenix.hbase.index.table.CachingHTableFactory$HTableInterfaceLRUMap.removeLRU(CachingHTableFactory.java:61)
> at 
> org.apache.commons.collections.map.LRUMap.addMapping(LRUMap.java:256)
> at 
> org.apache.commons.collections.map.AbstractHashedMap.put(AbstractHashedMap.java:284)
> at 
> org.apache.phoenix.hbase.index.table.CachingHTableFactory.getTable(CachingHTableFactory.java:100)
> at 
> org.apache.phoenix.hbase.index.write.ParallelWriterIndexCommitter$1.call(ParallelWriterIndexCommitter.java:160)
> at 
> org.apache.phoenix.hbase.index.write.ParallelWriterIndexCommitter$1.call(ParallelWriterIndexCommitter.java:136)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> But the IndexWriter was using this old connection to write to the table which 
> was closed during LRU eviction
> {code}
> 016-08-04 13:44:59,553 ERROR [htable-pool659-t1] client.AsyncProcess: Cannot 
> get replica 0 location for 
> {"totalColumns":1,"row":"\\xC7\\x03\\x04\\x06X\\x1C)\\x00\\x80\\x07\\xB0X","families":{"0":[{"qualifier":"_0","vlen":2,"tag":[],"timestamp":1470318296425}]}}
> java.io.IOException: hconnection-0x21f468be closed
> at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1153)
> at 
> org.apache.hadoop.hbase.client.CoprocessorHConnection.locateRegion(CoprocessorHConnection.java:41)
> at 
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncProcess.java:949)
> at 
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:866)
> at 
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1195)
> at 
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveGlobalFailure(AsyncProcess.java:1162)
> at 
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1100(AsyncProcess.java:584)
> at 
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:727)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> Although the workaround is to the cache size(index.tablefactory.cache.size). 
> But still we should 

[jira] [Updated] (PHOENIX-3174) Make minor upgrade a manual step

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor updated PHOENIX-3174:
--
Fix Version/s: (was: 4.8.1)

> Make minor upgrade a manual step
> 
>
> Key: PHOENIX-3174
> URL: https://issues.apache.org/jira/browse/PHOENIX-3174
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Samarth Jain
> Fix For: 4.9.0
>
> Attachments: PHOENIX-3174.patch, PHOENIX-3174_v2.patch, 
> PHOENIX-3174_v3_master.patch
>
>
> Instead of automatically performing minor release upgrades to system tables 
> in an automated manner (on first connection), we should make upgrade a 
> separate manual step. If a newer client attempts to connect to a newer server 
> without the upgrade step having occurred, we'd fail the connection.
> Though not as automated, this would give users more control and visibility 
> into when an upgrade over system tables occurs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-3159) CachingHTableFactory may close HTable during eviction even if it is getting used for writing by another thread.

[James Taylor (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496599#comment-15496599
 ] 

James Taylor commented on PHOENIX-3159:
---

[~devaraj] - would it be possible to give this a final review as we'd like to 
get this into the 4.8.1 RC we plan to cut on Monday?

> CachingHTableFactory may close HTable during eviction even if it is getting 
> used for writing by another thread.
> ---
>
> Key: PHOENIX-3159
> URL: https://issues.apache.org/jira/browse/PHOENIX-3159
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
> Fix For: 4.8.1
>
> Attachments: PHOENIX-3159.patch, PHOENIX-3159_v1.patch, 
> PHOENIX-3159_v2.patch, PHOENIX-3159_v3.patch
>
>
> CachingHTableFactory may close HTable during eviction even if it is getting 
> used for writing by another thread which results in writing thread to fail 
> and index is disabled.
> LRU eviction closing HTable or underlying connection when cache is full and 
> new HTable is requested.
> {code}
> 2016-08-04 13:45:21,109 DEBUG 
> [nat-s11-4-ioss-phoenix-1-5.openstacklocal,16020,1470297472814-index-writer--pool11-t35]
>  client.ConnectionManager$HConnectionImplementation: Closing HConnection 
> (debugging purposes only)
> java.lang.Exception
> at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.internalClose(ConnectionManager.java:2423)
> at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.close(ConnectionManager.java:2447)
> at 
> org.apache.hadoop.hbase.client.CoprocessorHConnection.close(CoprocessorHConnection.java:41)
> at 
> org.apache.hadoop.hbase.client.HTableWrapper.internalClose(HTableWrapper.java:91)
> at 
> org.apache.hadoop.hbase.client.HTableWrapper.close(HTableWrapper.java:107)
> at 
> org.apache.phoenix.hbase.index.table.CachingHTableFactory$HTableInterfaceLRUMap.removeLRU(CachingHTableFactory.java:61)
> at 
> org.apache.commons.collections.map.LRUMap.addMapping(LRUMap.java:256)
> at 
> org.apache.commons.collections.map.AbstractHashedMap.put(AbstractHashedMap.java:284)
> at 
> org.apache.phoenix.hbase.index.table.CachingHTableFactory.getTable(CachingHTableFactory.java:100)
> at 
> org.apache.phoenix.hbase.index.write.ParallelWriterIndexCommitter$1.call(ParallelWriterIndexCommitter.java:160)
> at 
> org.apache.phoenix.hbase.index.write.ParallelWriterIndexCommitter$1.call(ParallelWriterIndexCommitter.java:136)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> But the IndexWriter was using this old connection to write to the table which 
> was closed during LRU eviction
> {code}
> 016-08-04 13:44:59,553 ERROR [htable-pool659-t1] client.AsyncProcess: Cannot 
> get replica 0 location for 
> {"totalColumns":1,"row":"\\xC7\\x03\\x04\\x06X\\x1C)\\x00\\x80\\x07\\xB0X","families":{"0":[{"qualifier":"_0","vlen":2,"tag":[],"timestamp":1470318296425}]}}
> java.io.IOException: hconnection-0x21f468be closed
> at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1153)
> at 
> org.apache.hadoop.hbase.client.CoprocessorHConnection.locateRegion(CoprocessorHConnection.java:41)
> at 
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncProcess.java:949)
> at 
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:866)
> at 
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1195)
> at 
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveGlobalFailure(AsyncProcess.java:1162)
> at 
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1100(AsyncProcess.java:584)
> at 
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:727)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> Although the 

[jira] [Commented] (PHOENIX-3174) Make minor upgrade a manual step

[James Taylor (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496595#comment-15496595
 ] 

James Taylor commented on PHOENIX-3174:
---

Also, would it be possible to add some tests in UpgradeIT, 
[~samarth.j...@gmail.com]? In particular around verifying that no SQL is 
allowed to be executed when the client and server are new but an upgrade is 
required.

> Make minor upgrade a manual step
> 
>
> Key: PHOENIX-3174
> URL: https://issues.apache.org/jira/browse/PHOENIX-3174
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Samarth Jain
> Fix For: 4.9.0, 4.8.1
>
> Attachments: PHOENIX-3174.patch, PHOENIX-3174_v2.patch, 
> PHOENIX-3174_v3_master.patch
>
>
> Instead of automatically performing minor release upgrades to system tables 
> in an automated manner (on first connection), we should make upgrade a 
> separate manual step. If a newer client attempts to connect to a newer server 
> without the upgrade step having occurred, we'd fail the connection.
> Though not as automated, this would give users more control and visibility 
> into when an upgrade over system tables occurs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-3287) SecureUserConnectionsTest is flapping

[Josh Elser (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-3287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496559#comment-15496559
 ] 

Josh Elser commented on PHOENIX-3287:
-

Oh funny. I'll take a look. Thanks, James!

> SecureUserConnectionsTest is flapping
> -
>
> Key: PHOENIX-3287
> URL: https://issues.apache.org/jira/browse/PHOENIX-3287
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Josh Elser
>Priority: Minor
>
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 99.229 sec 
> <<< FAILURE! - in org.apache.phoenix.jdbc.SecureUserConnectionsTest
> org.apache.phoenix.jdbc.SecureUserConnectionsTest  Time elapsed: 99.229 sec  
> <<< ERROR!
> java.net.BindException: Address already in use
> See https://builds.apache.org/job/Phoenix-master/1404/changes for example 
> (and another earlier one). Wild guess, but maybe the port needs to be 
> randomized? Can it be?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-3261) Phoenix includes a JAXB version that conflicts with CXF 3.0.x dependencies

[Doug Holubek (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-3261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496383#comment-15496383
 ] 

Doug Holubek commented on PHOENIX-3261:
---

This issue can be closed.

For anyone who might be interested: my issue was I was using the 
phoenix-4.4.0.2.3.4.0-3485-client.jar exported from the hortonworks cluster I'm 
using.  The fix was to use the core jar from the horton works repo and exclude 
several packages.  Below are the changes I made to my pom.xml file to get the 
driver working in a java web app that has in Jersey 2 and CXF 3 without 
classpath conflicts.

4.4.0.2.3.4.0-3485
...

org.apache.phoenix
phoenix-core
${phoenix.version}


org.slf4j
slf4j-log4j12


jetty-util
org.mortbay.jetty


jetty
org.mortbay.jetty


jetty-sslengine
org.mortbay.jetty


jersey-core
com.sun.jersey


jersey-server
com.sun.jersey


jersey-json
com.sun.jersey


jaxb-impl
com.sun.xml.bind



...
 

hortonworks

http://repo.hortonworks.com/content/repositories/releases

true




> Phoenix includes a JAXB version that conflicts with CXF 3.0.x dependencies
> --
>
> Key: PHOENIX-3261
> URL: https://issues.apache.org/jira/browse/PHOENIX-3261
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.4.0
>Reporter: Doug Holubek
>Priority: Blocker
>
> Phoenix jars seem to bundle an older version of JAXB.  The DEPENDENCIES file 
> in the .jar's META-INF seems to show that Phoenix is including 
> jaxb-impl-2.2.3-1.  According to This CXF jira issue 
> https://issues.apache.org/jira/browse/CXF-5894
> (with same exception stack trace) CXF needs 2.10.x or higher.  Since Phoenix 
> is built with the classes inside the jar I can't exclude Phoenix's older 
> version of JAXB via Maven.
> (exert from DEPEDENCIES file inside Phoenix jar - seems to be the same across 
> many Phoenix versions.)
> From: 'Oracle Corporation' (http://www.oracle.com/)
>   - jersey-client (https://jersey.java.net/jersey-client/) 
> com.sun.jersey:jersey-client:bundle:1.9
> License: CDDL 1.1  (http://glassfish.java.net/public/CDDL+GPL_1_1.html)   
>  License: GPL2 w/ CPE  (http://glassfish.java.net/public/CDDL+GPL_1_1.html)
>   - jersey-core (https://jersey.java.net/jersey-core/) 
> com.sun.jersey:jersey-core:bundle:1.9
> License: CDDL 1.1  (http://glassfish.java.net/public/CDDL+GPL_1_1.html)   
>  License: GPL2 w/ CPE  (http://glassfish.java.net/public/CDDL+GPL_1_1.html)
>   - jersey-json (https://jersey.java.net/jersey-json/) 
> com.sun.jersey:jersey-json:bundle:1.9
> License: CDDL 1.1  (http://glassfish.java.net/public/CDDL+GPL_1_1.html)   
>  License: GPL2 w/ CPE  (http://glassfish.java.net/public/CDDL+GPL_1_1.html)
>   - jersey-server (https://jersey.java.net/jersey-server/) 
> com.sun.jersey:jersey-server:bundle:1.9
> License: CDDL 1.1  (http://glassfish.java.net/public/CDDL+GPL_1_1.html)   
>  License: GPL2 w/ CPE  (http://glassfish.java.net/public/CDDL+GPL_1_1.html)
>   - jersey-guice (https://jersey.java.net/jersey-contribs/jersey-guice/) 
> com.sun.jersey.contribs:jersey-guice:jar:1.9
> License: CDDL 1.1  (http://glassfish.java.net/public/CDDL+GPL_1_1.html)   
>  License: GPL2 w/ CPE  (http://glassfish.java.net/public/CDDL+GPL_1_1.html)
>   - JAXB RI (http://jaxb.java.net/) com.sun.xml.bind:jaxb-impl:jar:2.2.3-1
> License: CDDL 1.1  

[jira] [Created] (PHOENIX-3288) we should be able to fetch Namespace mapping properties from server so that every client doesn't need to provide that

[Ankit Singhal (JIRA)]

Ankit Singhal created PHOENIX-3288:
--

 Summary: we should be able to fetch Namespace mapping properties 
from server so that every client doesn't need to provide that
 Key: PHOENIX-3288
 URL: https://issues.apache.org/jira/browse/PHOENIX-3288
 Project: Phoenix
  Issue Type: Improvement
Reporter: Ankit Singhal
Assignee: Ankit Singhal






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: I want to provide a module that provides the unshaded version of the jdbc client.

[Marco Villalobos]

TL;DR; Publish a non-shaded version of phoenix client to a maven repository 
because it would much easier for complex systems to utilize since it is meant 
to be used as a library.

I'll elaborate, because both approaches have their benefits when done 
correctly. 

I want to constructively say that it was not done correctly in this project.

CURRENTLY

Currently, the phoenix-client jar is shaded and it only repackages (changes the 
package name structure of dependencies) for some of its dependencies.
Currently, the phoenix-client jar is NOT available in the most popular maven 
repositories.

However, most organizations use a dependency management tool such as Maven, 
Ivy, or Gradle. Typically, an organize simply declares an artifact as a 
dependency.
Their build system will then pull ALL the required dependencies and package 
them in the proper location for an application.

WHAT IF CLIENT WAS PUBLISHED IN MAVEN REPOSITORY?

Now, let's say phoenix-client jar was published in a maven repository. 

FIRST BENEFIT

This would help the adoption rate since it would be easier for organizations to 
integrate into their projects via dependency management.

BUT THERE ARE CLASS LOADING PROBLEMS BECAUSE IT SHADED

However, providing only a shaded jar has less benefit and utility, and in fact, 
probably introduces a greater chance that it will be in conflict during the 
runtime with class loading issues. 

If it was not shaded then all of those runtime class loading issues disappear, 
and build tools such as maven could easily manage the required dependencies.

If you listed the all of the non-phoenix packages in the phoenix client:

jar tvf phoenix-4.8.0-HBase-1.2-client.jar | awk '{if 
(!match($8,/^org\/apache\/phoenix/)) {print $8}}' 

It would list these packages (a sample):

org/apache/hadoop/
org/apache/hadoop/hbase/
org/apache/tephra/
com/google/gson/
com/google/inject
sqlline
com/google/common
com/google/protobuf
org/apache/commons/logging
org/apache/log4j/
org/apache/htrace/
org/apache/commons/csv
javax/annotation/
org/apache/hadoop/hbase/client/
org/apache/hadoop/hbase/zookeeper/
tables/
org/apache/curator/
com/sun/jersey/server
javax/ws/
com/sun/appserv/
com/sun/el/parser
javax/servlet/
org/owasp/esapi
bsh/
org/apache/batik
org/w3c/dom/svg
org/cyberneko/html/
org/apache/hadoop/hdfs
org/apache/xerces/
javax/xml/
org/w3c/dom
org/xml/sax
com/sun/jersey/json
com/sun/xml/bind/
org/slf4j/impl/
javax/activation
com/sun/jersey/api/client/
com/google/inject
org/apache/flume
org/apache/pig/

That listing alone would prevent a person from using the client that has a 
different version of Servlet, Guava, Protocol Buffers, Jersey (jax-rs), SLF-4j, 
Jersey Client.

Those are some of the most useful popular libraries available to the public.

Now let's say that it was shaded properly.  There could still be some problems 
in certain environments, such as one that uses JAX-RS because classes that were 
annotated with @Provider would be loaded during a web application!

So really, this brings us back to square one.  An unshaded version of the 
client should be made available so that users can just declare it as a maven 
dependency.

THE BENEFITS OF A SHADED JAR

Now, there are a few benefits to providing a shaded jar:

It is easier to MANUALLY install on a system that has NONE of the dependencies 
that are shaded into it.

Its okay to to provide shaded jars for executable applications.

REBUTTAL

Most systems use Maven for dependency management.  Phoenix client is not an 
application, it is a library.

CONCLUSION

I killed this, didn't I?  I hope my point is taken that a non-shaded version of 
the client should be provided by now. And a more complete effort to repackage 
its dependencies completely should be taken.

Libraries should never shaded jars.  


> On Sep 15, 2016, at 1:10 PM, Nick Dimiduk  wrote:
> 
> So how is no shading better than this partial shading? You'd end up with
> the same conflicts, no?
> 
> As far as I know, a JDBC implementation exposes nothing outside java.sql.*.
> Anything else exposed by the shaded jar is probably a bug. Can you provide
> more details on what you're seeing, or -- better still -- file a JIRA?
> 
> On Thu, Sep 15, 2016 at 11:45 AM, Marco Villalobos <
> mvillalo...@kineteque.com> wrote:
> 
>> Not all of its dependencies are repackaged though, which leads to
>> class loading conflicts.  When is that ever a good thing?
>> 
>> On Thu, Sep 15, 2016 at 9:28 AM, Nick Dimiduk  wrote:
>>> Maybe I'm missing something, but...
>>> 
>>> The whole point of providing a shaded client jar is to prevent exposing
>>> Phoenix implementation details to the applications that consume it --
>>> effectively allowing people to manage their own dependencies. Using a
>>> shaded client jar means you don't have to worry about dependency conflict
>>> because by definition there's only one dependency: the shaded client.
>> What
>>> are you able to achieve now with, say, the