[jira] [Commented] (PHOENIX-476) Support declaration of DEFAULT in CREATE statement
[ https://issues.apache.org/jira/browse/PHOENIX-476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15564469#comment-15564469 ] James Taylor commented on PHOENIX-476: -- Couple more important tests around secondary indexes: - create an index with a covered column that has a default value and make sure that the default value takes affect. - create an index on a column with a default and make sure the index rows are correct. > Support declaration of DEFAULT in CREATE statement > -- > > Key: PHOENIX-476 > URL: https://issues.apache.org/jira/browse/PHOENIX-476 > Project: Phoenix > Issue Type: Task >Affects Versions: 3.0-Release >Reporter: James Taylor >Assignee: Kevin Liew > Labels: enhancement > Fix For: 4.9.0 > > Attachments: PHOENIX-476.2.patch, PHOENIX-476.3.patch, > PHOENIX-476.4.patch, PHOENIX-476.5.patch, PHOENIX-476.6.patch, > PHOENIX-476.7.patch, PHOENIX-476.patch > > > Support the declaration of a default value in the CREATE TABLE/VIEW statement > like this: > CREATE TABLE Persons ( > Pid int NOT NULL PRIMARY KEY, > LastName varchar(255) NOT NULL, > FirstName varchar(255), > Address varchar(255), > City varchar(255) DEFAULT 'Sandnes' > ) > To implement this, we'd need to: > 1. add a new DEFAULT_VALUE key value column in SYSTEM.TABLE and pass through > the value when the table is created (in MetaDataClient). > 2. always set NULLABLE to ResultSetMetaData.columnNoNulls if a default value > is present, since the column will never be null. > 3. add a getDefaultValue() accessor in PColumn > 4. for a row key column, during UPSERT use the default value if no value was > specified for that column. This could be done in the PTableImpl.newKey method. > 5. for a key value column with a default value, we can get away without > incurring any storage cost. Although a little bit of extra effort than if we > persisted the default value on an UPSERT for key value columns, this approach > has the benefit of not incurring any storage cost for a default value. > * serialize any default value into KeyValueColumnExpression > * in the evaluate method of KeyValueColumnExpression, conditionally use > the default value if the column value is not present. If doing partial > evaluation, you should not yet return the default value, as we may not have > encountered the the KeyValue for the column yet (since a filter evaluates > each time it sees each KeyValue, and there may be more than one KeyValue > referenced in the expression). Partial evaluation is determined by calling > Tuple.isImmutable(), where false means it is NOT doing partial evaluation, > while true means it is. > * modify EvaluateOnCompletionVisitor by adding a visitor method for > RowKeyColumnExpression and KeyValueColumnExpression to set > evaluateOnCompletion to true if they have a default value specified. This > will cause filter evaluation to execute one final time after all KeyValues > for a row have been seen, since it's at this time we know we should use the > default value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3240) ClassCastException from Pig loader
[
https://issues.apache.org/jira/browse/PHOENIX-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15564361#comment-15564361
]
Josh Elser commented on PHOENIX-3240:
-
Thanks [~mujtabachohan] and [~warwithin]! Will push this in tmrw.
> ClassCastException from Pig loader
> --
>
> Key: PHOENIX-3240
> URL: https://issues.apache.org/jira/browse/PHOENIX-3240
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.8.0
>Reporter: YoungWoo Kim
>Assignee: Josh Elser
> Fix For: 4.9.0, 4.8.2
>
>
> I'm loading data from Hive table to Phoenix table in using Phoenix-Pig
> integration. my pig script looks like following:
> {code}
> T = LOAD 'mydb.$TBL' USING org.apache.hive.hcatalog.pig.HCatLoader();
> STORE T into 'hbase://MYSCHEMA.$TBL' using
> org.apache.phoenix.pig.PhoenixHBaseStorage('i004,i005,i006','-batchSize
> 1000');
> {code}
> If the source table has timestamp, MapReduce job for Pig script does not work:
> {noformat}
> ERROR 0: java.lang.ClassCastException: org.joda.time.DateTime cannot be cast
> to org.apache.phoenix.shaded.org.joda.time.DateTime
> org.apache.pig.backend.executionengine.ExecException: ERROR 0:
> java.lang.ClassCastException: org.joda.time.DateTime cannot be cast to
> org.apache.phoenix.shaded.org.joda.time.DateTime
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.getStats(MapReduceLauncher.java:822)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:452)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:304)
> at org.apache.pig.PigServer.launchPlan(PigServer.java:1390)
> at
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1375)
> at org.apache.pig.PigServer.execute(PigServer.java:1364)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:415)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:398)
> at
> org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:171)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:234)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> at org.apache.pig.Main.run(Main.java:502)
> at org.apache.pig.Main.main(Main.java:177)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.lang.ClassCastException: org.joda.time.DateTime cannot be
> cast to org.apache.phoenix.shaded.org.joda.time.DateTime
> at
> org.apache.phoenix.pig.util.TypeUtil.castPigTypeToPhoenix(TypeUtil.java:201)
> at
> org.apache.phoenix.pig.PhoenixHBaseStorage.putNext(PhoenixHBaseStorage.java:189)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:136)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:95)
> at
> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:658)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map.collect(PigMapOnly.java:48)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:260)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-476) Support declaration of DEFAULT in CREATE statement
[ https://issues.apache.org/jira/browse/PHOENIX-476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15564303#comment-15564303 ] James Taylor commented on PHOENIX-476: -- Let's get array support in and the other revisions I mentioned, but do the other work in follow up JIRAs. It looks like this can make 4.9.0 which is great! > Support declaration of DEFAULT in CREATE statement > -- > > Key: PHOENIX-476 > URL: https://issues.apache.org/jira/browse/PHOENIX-476 > Project: Phoenix > Issue Type: Task >Affects Versions: 3.0-Release >Reporter: James Taylor >Assignee: Kevin Liew > Labels: enhancement > Fix For: 4.9.0 > > Attachments: PHOENIX-476.2.patch, PHOENIX-476.3.patch, > PHOENIX-476.4.patch, PHOENIX-476.5.patch, PHOENIX-476.6.patch, > PHOENIX-476.7.patch, PHOENIX-476.patch > > > Support the declaration of a default value in the CREATE TABLE/VIEW statement > like this: > CREATE TABLE Persons ( > Pid int NOT NULL PRIMARY KEY, > LastName varchar(255) NOT NULL, > FirstName varchar(255), > Address varchar(255), > City varchar(255) DEFAULT 'Sandnes' > ) > To implement this, we'd need to: > 1. add a new DEFAULT_VALUE key value column in SYSTEM.TABLE and pass through > the value when the table is created (in MetaDataClient). > 2. always set NULLABLE to ResultSetMetaData.columnNoNulls if a default value > is present, since the column will never be null. > 3. add a getDefaultValue() accessor in PColumn > 4. for a row key column, during UPSERT use the default value if no value was > specified for that column. This could be done in the PTableImpl.newKey method. > 5. for a key value column with a default value, we can get away without > incurring any storage cost. Although a little bit of extra effort than if we > persisted the default value on an UPSERT for key value columns, this approach > has the benefit of not incurring any storage cost for a default value. > * serialize any default value into KeyValueColumnExpression > * in the evaluate method of KeyValueColumnExpression, conditionally use > the default value if the column value is not present. If doing partial > evaluation, you should not yet return the default value, as we may not have > encountered the the KeyValue for the column yet (since a filter evaluates > each time it sees each KeyValue, and there may be more than one KeyValue > referenced in the expression). Partial evaluation is determined by calling > Tuple.isImmutable(), where false means it is NOT doing partial evaluation, > while true means it is. > * modify EvaluateOnCompletionVisitor by adding a visitor method for > RowKeyColumnExpression and KeyValueColumnExpression to set > evaluateOnCompletion to true if they have a default value specified. This > will cause filter evaluation to execute one final time after all KeyValues > for a row have been seen, since it's at this time we know we should use the > default value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3240) ClassCastException from Pig loader
[
https://issues.apache.org/jira/browse/PHOENIX-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15564290#comment-15564290
]
YoungWoo Kim commented on PHOENIX-3240:
---
Nice work! [~elserj] Looks good to me and a couple of examples work fine on my
end. Thanks!
> ClassCastException from Pig loader
> --
>
> Key: PHOENIX-3240
> URL: https://issues.apache.org/jira/browse/PHOENIX-3240
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.8.0
>Reporter: YoungWoo Kim
>Assignee: Josh Elser
> Fix For: 4.9.0, 4.8.2
>
>
> I'm loading data from Hive table to Phoenix table in using Phoenix-Pig
> integration. my pig script looks like following:
> {code}
> T = LOAD 'mydb.$TBL' USING org.apache.hive.hcatalog.pig.HCatLoader();
> STORE T into 'hbase://MYSCHEMA.$TBL' using
> org.apache.phoenix.pig.PhoenixHBaseStorage('i004,i005,i006','-batchSize
> 1000');
> {code}
> If the source table has timestamp, MapReduce job for Pig script does not work:
> {noformat}
> ERROR 0: java.lang.ClassCastException: org.joda.time.DateTime cannot be cast
> to org.apache.phoenix.shaded.org.joda.time.DateTime
> org.apache.pig.backend.executionengine.ExecException: ERROR 0:
> java.lang.ClassCastException: org.joda.time.DateTime cannot be cast to
> org.apache.phoenix.shaded.org.joda.time.DateTime
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.getStats(MapReduceLauncher.java:822)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:452)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:304)
> at org.apache.pig.PigServer.launchPlan(PigServer.java:1390)
> at
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1375)
> at org.apache.pig.PigServer.execute(PigServer.java:1364)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:415)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:398)
> at
> org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:171)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:234)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> at org.apache.pig.Main.run(Main.java:502)
> at org.apache.pig.Main.main(Main.java:177)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.lang.ClassCastException: org.joda.time.DateTime cannot be
> cast to org.apache.phoenix.shaded.org.joda.time.DateTime
> at
> org.apache.phoenix.pig.util.TypeUtil.castPigTypeToPhoenix(TypeUtil.java:201)
> at
> org.apache.phoenix.pig.PhoenixHBaseStorage.putNext(PhoenixHBaseStorage.java:189)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:136)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:95)
> at
> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:658)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map.collect(PigMapOnly.java:48)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:260)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-476) Support declaration of DEFAULT in CREATE statement
[ https://issues.apache.org/jira/browse/PHOENIX-476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15564286#comment-15564286 ] Kevin Liew commented on PHOENIX-476: Oh, right - I was thinking about backward compatibility from the server pov but the user pov sees broken compatibility if they have to use a new name for the default schema. The tests are all passing locally and I think [~hadoopqa] will post results in a little bit. I will revise based on your feedback and also take another look at support for arrays. I made some changes to type coercion in DefaultValueExpression for the data type test cases and this may resolve the coercion issues I was seeing with arrays. If there is no urgency in getting this committed, we could commit array support with this patch. We could also branch array support into a later JIRA, as well as stateful expressions and additional DDL support. And support for `DEFAULT` in `UPSERT` statements as Julian had mentioned. > Support declaration of DEFAULT in CREATE statement > -- > > Key: PHOENIX-476 > URL: https://issues.apache.org/jira/browse/PHOENIX-476 > Project: Phoenix > Issue Type: Task >Affects Versions: 3.0-Release >Reporter: James Taylor >Assignee: Kevin Liew > Labels: enhancement > Fix For: 4.9.0 > > Attachments: PHOENIX-476.2.patch, PHOENIX-476.3.patch, > PHOENIX-476.4.patch, PHOENIX-476.5.patch, PHOENIX-476.6.patch, > PHOENIX-476.7.patch, PHOENIX-476.patch > > > Support the declaration of a default value in the CREATE TABLE/VIEW statement > like this: > CREATE TABLE Persons ( > Pid int NOT NULL PRIMARY KEY, > LastName varchar(255) NOT NULL, > FirstName varchar(255), > Address varchar(255), > City varchar(255) DEFAULT 'Sandnes' > ) > To implement this, we'd need to: > 1. add a new DEFAULT_VALUE key value column in SYSTEM.TABLE and pass through > the value when the table is created (in MetaDataClient). > 2. always set NULLABLE to ResultSetMetaData.columnNoNulls if a default value > is present, since the column will never be null. > 3. add a getDefaultValue() accessor in PColumn > 4. for a row key column, during UPSERT use the default value if no value was > specified for that column. This could be done in the PTableImpl.newKey method. > 5. for a key value column with a default value, we can get away without > incurring any storage cost. Although a little bit of extra effort than if we > persisted the default value on an UPSERT for key value columns, this approach > has the benefit of not incurring any storage cost for a default value. > * serialize any default value into KeyValueColumnExpression > * in the evaluate method of KeyValueColumnExpression, conditionally use > the default value if the column value is not present. If doing partial > evaluation, you should not yet return the default value, as we may not have > encountered the the KeyValue for the column yet (since a filter evaluates > each time it sees each KeyValue, and there may be more than one KeyValue > referenced in the expression). Partial evaluation is determined by calling > Tuple.isImmutable(), where false means it is NOT doing partial evaluation, > while true means it is. > * modify EvaluateOnCompletionVisitor by adding a visitor method for > RowKeyColumnExpression and KeyValueColumnExpression to set > evaluateOnCompletion to true if they have a default value specified. This > will cause filter evaluation to execute one final time after all KeyValues > for a row have been seen, since it's at this time we know we should use the > default value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-476) Support declaration of DEFAULT in CREATE statement
[
https://issues.apache.org/jira/browse/PHOENIX-476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15564198#comment-15564198
]
James Taylor commented on PHOENIX-476:
--
Thanks for the quick turnaround, [~kliew]. I think we're almost there. Do the
tests pass now?
A few changes are needed, though.
- To maintain backwards compatibility (as your change would break usages of USE
SCHEMA DEFAULT), you can make this change to PhoenixSQL.g:
{code}
use_schema_node returns [UseSchemaStatement ret]
: USE (DEFAULT | s=identifier)
{ret = factory.useSchema(s); }
;
{code}
And then in UseSchemaStatement:
{code}
public class UseSchemaStatement extends MutableStatement {
private final String schemaName;
public UseSchemaStatement(String schemaName) {
this.schemaName = schemaName == null ? StringUtil.EMPTY_STRING :
schemaName;
}
{code}
And then finally in SchemaUtil, make this change:
{code}
public MutationState useSchema(UseSchemaStatement useSchemaStatement)
throws SQLException {
// As we allow default namespace mapped to empty schema, so this is to
reset schema in connection
if (useSchemaStatement.getSchemaName().equals(StringUtil.EMPTY_STRING) {
connection.setSchema(null);
} else {
{code}
- Using null in UngroupedAggregateRegionObserver will be a problem. Add a test
for a table with a default value for one of the PK columns and do an UPSERT
SELECT into it.
{code}
diff --git
a/phoenix-core/src/main/java/org/apache/phoenix/coprocessor/UngroupedAggregateRegionObserver.java
b/phoenix-core/src/main/java/org/apache/phoenix/coprocessor/UngroupedAggregateRegionObserver.java
index d8d313d..c30a44b 100644
---
a/phoenix-core/src/main/java/org/apache/phoenix/coprocessor/UngroupedAggregateRegionObserver.java
+++
b/phoenix-core/src/main/java/org/apache/phoenix/coprocessor/UngroupedAggregateRegionObserver.java
@@ -435,7 +435,7 @@ public class UngroupedAggregateRegionObserver extends
BaseScannerRegionObserver
}
values[i] = ptr.copyBytes();
}
-writeToTable.newKey(ptr, values);
+writeToTable.newKey(null, ptr, values);
{code}
Rather than requiring a StatementContext, let's do the following:
- In CreateTableCompiler, when you compile the defaultExpression, you'll end up
with a LiteralExpression (i.e. DEFAULT 2 + 7 would be turned into a
LiteralExpression of 9). Create a new CreateTableStatement copy constructor
that passes in a createTableStatement and an expressionStr. Use this to create
a new CreateTableStatement using defaultExpression.toString() to form the
literal expression string which you'll pass through to the MetaDataClient to
guarantee that the default expression is always a literal.
- Then modify the usage of the default value back to the way you had it where
you didn't need a StatementContext since you'll know you have a literal.
We'll may have to tweak this once we want to support NEXT VALUE FOR and
CURRENT_DATE expressions (or we may not need to as these values are generated
on the client and turned into literals anyway). But for now, this will work
fine.
- Also, instead of deriving DefaultColumnValueIT from BaseClientManagedTimeIT,
derive it from ParallelStatsDisabledIT. Then just change hardcoded references
of T_DEFAULT to a new fullTableName member variable which you set in an @Before
method using the static generateUniqueName() method. This allows your tests to
run in parallel (since the table names are independent) and enables our tests
to run much faster. The tests in BaseClientManagedTimeIT are the only ones
waiting to be converted like this as well, so it's essentially deprecated. You
also don't need to use nextTimestamp() and open new connections every time -
just use the same connection throughout your test.
> Support declaration of DEFAULT in CREATE statement
> --
>
> Key: PHOENIX-476
> URL: https://issues.apache.org/jira/browse/PHOENIX-476
> Project: Phoenix
> Issue Type: Task
>Affects Versions: 3.0-Release
>Reporter: James Taylor
>Assignee: Kevin Liew
> Labels: enhancement
> Fix For: 4.9.0
>
> Attachments: PHOENIX-476.2.patch, PHOENIX-476.3.patch,
> PHOENIX-476.4.patch, PHOENIX-476.5.patch, PHOENIX-476.6.patch,
> PHOENIX-476.7.patch, PHOENIX-476.patch
>
>
> Support the declaration of a default value in the CREATE TABLE/VIEW statement
> like this:
> CREATE TABLE Persons (
> Pid int NOT NULL PRIMARY KEY,
> LastName varchar(255) NOT NULL,
> FirstName varchar(255),
> Address varchar(255),
> City varchar(255) DEFAULT 'Sandnes'
> )
> To implement this, we'd
[jira] [Updated] (PHOENIX-476) Support declaration of DEFAULT in CREATE statement
[ https://issues.apache.org/jira/browse/PHOENIX-476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Liew updated PHOENIX-476: --- Attachment: PHOENIX-476.7.patch Thanks for the feedback [~jamestaylor]. I attached a another revision. This time, we pass StatementContext as a parameter to `ColumnRef.newColumnExpression` and `PTable.newKey` so that we can compile arbitrary expressions. `SCHEMA_FOR_DEFAULT_NAMESPACE` is defined as `DEFAULT` which is now a reserved keyword. This causes parsing errors in various tests involving the schema tables so I changed it to 'DEFAULT_SCHEMA'. It doesn't look like it will cause backward compatibility issues. > Support declaration of DEFAULT in CREATE statement > -- > > Key: PHOENIX-476 > URL: https://issues.apache.org/jira/browse/PHOENIX-476 > Project: Phoenix > Issue Type: Task >Affects Versions: 3.0-Release >Reporter: James Taylor >Assignee: Kevin Liew > Labels: enhancement > Fix For: 4.9.0 > > Attachments: PHOENIX-476.2.patch, PHOENIX-476.3.patch, > PHOENIX-476.4.patch, PHOENIX-476.5.patch, PHOENIX-476.6.patch, > PHOENIX-476.7.patch, PHOENIX-476.patch > > > Support the declaration of a default value in the CREATE TABLE/VIEW statement > like this: > CREATE TABLE Persons ( > Pid int NOT NULL PRIMARY KEY, > LastName varchar(255) NOT NULL, > FirstName varchar(255), > Address varchar(255), > City varchar(255) DEFAULT 'Sandnes' > ) > To implement this, we'd need to: > 1. add a new DEFAULT_VALUE key value column in SYSTEM.TABLE and pass through > the value when the table is created (in MetaDataClient). > 2. always set NULLABLE to ResultSetMetaData.columnNoNulls if a default value > is present, since the column will never be null. > 3. add a getDefaultValue() accessor in PColumn > 4. for a row key column, during UPSERT use the default value if no value was > specified for that column. This could be done in the PTableImpl.newKey method. > 5. for a key value column with a default value, we can get away without > incurring any storage cost. Although a little bit of extra effort than if we > persisted the default value on an UPSERT for key value columns, this approach > has the benefit of not incurring any storage cost for a default value. > * serialize any default value into KeyValueColumnExpression > * in the evaluate method of KeyValueColumnExpression, conditionally use > the default value if the column value is not present. If doing partial > evaluation, you should not yet return the default value, as we may not have > encountered the the KeyValue for the column yet (since a filter evaluates > each time it sees each KeyValue, and there may be more than one KeyValue > referenced in the expression). Partial evaluation is determined by calling > Tuple.isImmutable(), where false means it is NOT doing partial evaluation, > while true means it is. > * modify EvaluateOnCompletionVisitor by adding a visitor method for > RowKeyColumnExpression and KeyValueColumnExpression to set > evaluateOnCompletion to true if they have a default value specified. This > will cause filter evaluation to execute one final time after all KeyValues > for a row have been seen, since it's at this time we know we should use the > default value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (PHOENIX-6) Support ON DUPLICATE KEY construct
[
https://issues.apache.org/jira/browse/PHOENIX-6?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15547135#comment-15547135
]
James Taylor edited comment on PHOENIX-6 at 10/11/16 12:33 AM:
---
Would you mind filing a separate JIRA for your use case? If I understand it
correctly, you'd like to be able to have multiple atomic insert/update
statements for the same row in the same commit batch and have them sequentially
execute? Would the atomicity be for the entire commit batch (for all rows being
operated on) or if not, how would potentially overlapping rows from other
batches be handled?
For your question, I think you meant to use the same table name for both
statements, right? For non transactional tables, the UPSERT SELECT wouldn't see
the data from the UPSERT VALUES. When auto commit is off, the data is kept in
memory on the client until the commit occurs. The UPSERT VALUES would cause a
new row to be cached on the client (in memory) and the UPSERT SELECT would
cause N rows to be cached on the client (where N is the number of rows
currently in the table). The commit essentially generates the HBase Puts and
does a batch mutate.
For transactional tables, you do see your own updates. In that case, the UPSERT
SELECT would see the row from the UPSERT VALUES call.
One more thought I had, [~cameron.hatfield]. If we allowed multiple UPSERT
VALUES ... ON DUPLICATE KEY statements for the same row key, it might work as
you'd like. This could potentially be accomplished by having a different
version for each Cell. In that case, each statement would operate under its own
row lock on the latest value. Is that more of what you're looking for?
was (Author: jamestaylor):
Would you mind filing a separate JIRA for your use case? If I understand it
correctly, you'd like to be able to have multiple atomic insert/update
statements for the same row in the same commit batch and have them sequentially
execute? Would the atomicity be for the entire commit batch (for all rows being
operated on) or if not, how would potentially overlapping rows from other
batches be handled?
For your question, I think you meant to use the same table name for both
statements, right? For non transactional tables, the UPSERT SELECT wouldn't see
the data from the UPSERT VALUES. When auto commit is off, the data is kept in
memory on the client until the commit occurs. The UPSERT VALUES would cause a
new row to be cached on the client (in memory) and the UPSERT SELECT would
cause N rows to be cached on the client (where N is the number of rows
currently in the table). The commit essentially generates the HBase Puts and
does a batch mutate.
For transactional tables, you do see your own updates. In that case, the UPSERT
SELECT would see the row from the UPSERT VALUES call.
> Support ON DUPLICATE KEY construct
> --
>
> Key: PHOENIX-6
> URL: https://issues.apache.org/jira/browse/PHOENIX-6
> Project: Phoenix
> Issue Type: New Feature
>Reporter: James Taylor
>Assignee: James Taylor
> Fix For: 4.9.0
>
>
> To support inserting a new row only if it doesn't already exist, we should
> support the "on duplicate key" construct for UPSERT. With this construct, the
> UPSERT VALUES statement would run atomically and would thus require a read
> before write which would obviously have a negative impact on performance. For
> an example of similar syntax , see MySQL documentation at
> http://dev.mysql.com/doc/refman/5.7/en/insert-on-duplicate.html
> See this discussion for more detail:
> https://groups.google.com/d/msg/phoenix-hbase-user/Bof-TLrbTGg/68bnc8ZcWe0J.
> A related discussion is on PHOENIX-2909.
> Initially we'd support the following:
> # This would prevent the setting of VAL to 0 if the row already exists:
> {code}
> UPSERT INTO T (PK, VAL) VALUES ('a',0)
> ON DUPLICATE KEY IGNORE;
> {code}
> # This would increment the valueS of COUNTER1 and COUNTER2 if the row already
> exists and otherwise initialize them to 0:
> {code}
> UPSERT INTO T (PK, COUNTER1, COUNTER2) VALUES ('a',0,0)
> ON DUPLICATE KEY COUNTER1 = COUNTER1 + 1, COUNTER2 = COUNTER2 + 1;
> {code}
> So the general form is:
> {code}
> UPSERT ... VALUES ... [ ON DUPLICATE KEY [IGNORE | UPDATE
> =, ...] ]
> {code}
> The following restrictions will apply:
> - The may not be part of the primary key constraint - only KeyValue
> columns will be allowed.
> To handle the maintenance of immutable indexes, we'll need to push the
> maintenance to the server side.
> This is because the mutations for indexes on immutable tables are calculated
> on the client-side, while this new syntax would potentially modify the value
> on the server-side.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3181) Run test methods in parallel to reduce test suite run time
[
https://issues.apache.org/jira/browse/PHOENIX-3181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563998#comment-15563998
]
Hadoop QA commented on PHOENIX-3181:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12832567/PHOENIX-3181_v2.patch
against master branch at commit eaf99f23da47b86e7577837c9c73ad47ad9ef0d1.
ATTACHMENT ID: 12832567
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
42 warning messages.
{color:red}-1 release audit{color}. The applied patch generated 1 release
audit warnings (more than the master's current 0 warnings).
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 core tests{color}. The patch failed these unit tests:
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.DistinctPrefixFilterIT
Test results:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/621//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/621//artifact/patchprocess/patchReleaseAuditWarnings.txt
Javadoc warnings:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/621//artifact/patchprocess/patchJavadocWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/621//console
This message is automatically generated.
> Run test methods in parallel to reduce test suite run time
> --
>
> Key: PHOENIX-3181
> URL: https://issues.apache.org/jira/browse/PHOENIX-3181
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: James Taylor
> Attachments: PHOENIX-3181_v1.patch, PHOENIX-3181_v2.patch
>
>
> With PHOENIX-3036, the tests within a test class can be executed in parallel
> since the table names won't conflict. This should greatly reduce the time
> taken to run all of our tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3359) Update JDBC urls to match connection type
[
https://issues.apache.org/jira/browse/PHOENIX-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563954#comment-15563954
]
Maryann Xue commented on PHOENIX-3359:
--
Think this wouldn't be a difficult change. It's only that since we are skipping
EXPLAIN plan check for a lot of other test cases due to the different plans
generated by Phoenix and Calcite, I'm wondering if we can skip this check as
well. We can first focus on the functional support and query results and come
back later for plan checks. Anyway, I'm good either way.
> Update JDBC urls to match connection type
> -
>
> Key: PHOENIX-3359
> URL: https://issues.apache.org/jira/browse/PHOENIX-3359
> Project: Phoenix
> Issue Type: Sub-task
>Reporter: Eric Lomore
>Assignee: Eric Lomore
>
> Connection's that are intended to be unwrapped into PhoenixConnection's need
> to be created using the old JDBC url,
> {{DriverManager.getConnection(getOldUrl())}}
> {{PhoenixConnection pc = conn.unwrap(PhoenixConnection.class);}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3359) Update JDBC urls to match connection type
[
https://issues.apache.org/jira/browse/PHOENIX-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563940#comment-15563940
]
Maryann Xue commented on PHOENIX-3359:
--
Yes, I agree that we should eventually have a way to display a plan with more
Phoenix details or display the Phoenix plan with a switch/different grammar.
But this is a different issue here, in the example [~lomoree] has given here,
we are trying to verify that we will be doing an ordered aggregate. We can
easily verify this with the current Calcite EXPLAIN plan. So I was suggesting
we verify directly from the EXPLAIN plan instead of using internal Phoenix
interfaces, since we are going through this code path now in Calcite-Phoenix.
> Update JDBC urls to match connection type
> -
>
> Key: PHOENIX-3359
> URL: https://issues.apache.org/jira/browse/PHOENIX-3359
> Project: Phoenix
> Issue Type: Sub-task
>Reporter: Eric Lomore
>Assignee: Eric Lomore
>
> Connection's that are intended to be unwrapped into PhoenixConnection's need
> to be created using the old JDBC url,
> {{DriverManager.getConnection(getOldUrl())}}
> {{PhoenixConnection pc = conn.unwrap(PhoenixConnection.class);}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3359) Update JDBC urls to match connection type
[
https://issues.apache.org/jira/browse/PHOENIX-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563852#comment-15563852
]
James Taylor commented on PHOENIX-3359:
---
Would be great if we could come up with a way of getting the Phoenix explain
plan instead of the Calcite one. Two ideas:
- Create a new detail level for EXPLAIN PLAN in Calcite and have this generate
the Phoenix explain plan. This would be the cleanest way, I think.
- Take advantage of the subtle difference in syntax (i.e. EXPLAIN PLAN FOR
versus EXPLAIN ) and have the latter generate the Phoenix
explain plan.
If we can make a general change across the board for the Calcite branch to
generate Phoenix explain plans, I think it's probably worth it. If they're just
one-off changes, though, I agree, we should move on to other stuff.
> Update JDBC urls to match connection type
> -
>
> Key: PHOENIX-3359
> URL: https://issues.apache.org/jira/browse/PHOENIX-3359
> Project: Phoenix
> Issue Type: Sub-task
>Reporter: Eric Lomore
>Assignee: Eric Lomore
>
> Connection's that are intended to be unwrapped into PhoenixConnection's need
> to be created using the old JDBC url,
> {{DriverManager.getConnection(getOldUrl())}}
> {{PhoenixConnection pc = conn.unwrap(PhoenixConnection.class);}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3361) Collect stats correctly for local indexes
[ https://issues.apache.org/jira/browse/PHOENIX-3361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563847#comment-15563847 ] Hudson commented on PHOENIX-3361: - FAILURE: Integrated in Jenkins build Phoenix-master #1434 (See [https://builds.apache.org/job/Phoenix-master/1434/]) PHOENIX-3361 Collect stats correct for local indexes (jamestaylor: rev eaf99f23da47b86e7577837c9c73ad47ad9ef0d1) * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/index/ViewIndexIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServices.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/schema/MetaDataClient.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/schema/stats/GuidePostsInfo.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/schema/stats/StatisticsUtil.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/schema/stats/NoOpStatisticsCollector.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/StatsCollectorIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/query/QueryConstants.java * (edit) phoenix-core/src/test/java/org/apache/phoenix/query/PhoenixStatsCacheRemovalListenerTest.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/RoundFloorCeilFuncIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixDatabaseMetaData.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/schema/stats/StatisticsCollector.java * (delete) phoenix-core/src/main/java/org/apache/phoenix/schema/stats/PTableStatsImpl.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/index/MutableIndexIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/coprocessor/UngroupedAggregateRegionObserver.java * (edit) phoenix-core/src/test/java/org/apache/phoenix/filter/SkipScanBigFilterTest.java * (delete) phoenix-core/src/main/java/org/apache/phoenix/query/TableStatsCache.java * (edit) phoenix-core/src/test/java/org/apache/phoenix/util/TestUtil.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/schema/stats/GuidePostsInfoBuilder.java * (add) phoenix-core/src/main/java/org/apache/phoenix/schema/stats/GuidePostsKey.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionlessQueryServicesImpl.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/IndexExtendedIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/util/MetaDataUtil.java * (delete) phoenix-core/src/main/java/org/apache/phoenix/schema/stats/PTableStats.java * (add) phoenix-core/src/main/java/org/apache/phoenix/query/GuidePostsCache.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/schema/stats/DefaultStatisticsCollector.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/index/MutableIndexFailureIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/query/DelegateConnectionQueryServices.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/schema/stats/StatisticsWriter.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/GroupByCaseIT.java > Collect stats correctly for local indexes > - > > Key: PHOENIX-3361 > URL: https://issues.apache.org/jira/browse/PHOENIX-3361 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: James Taylor > Fix For: 4.9.0 > > Attachments: PHOENIX-3361.patch, PHOENIX-3361_wip.patch, > PHOENIX-3361_wip2.patch > > > Stats are not being correctly collected for local indexes and tables with > local indexes, because the logic that deletes the old stats is based on all > store families of a table. Thus, when stats are collected for a table with > local indexes, the local index stats would be deleted and when the stats for > a local index are collected, the stats for the table will be deleted. > Instead, if we cache stats per column family instead of across entire table, > plus detect if we're collecting stats for a local index versus the data > table, we can fix this issue. Caching stats per column family makes more > sense anyway, because depending on the query, we may only use the stats > associated with the default column family. Also, we wouldn't want to have to > pull over stats for local index with the data table stats. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2675) Allow stats to be configured on a table-by-table basis
[ https://issues.apache.org/jira/browse/PHOENIX-2675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563846#comment-15563846 ] Hudson commented on PHOENIX-2675: - FAILURE: Integrated in Jenkins build Phoenix-master #1434 (See [https://builds.apache.org/job/Phoenix-master/1434/]) PHOENIX-2675 Allow stats to be configured on a table-by-table basis (jamestaylor: rev 711d7f0639d294974c2f56c7d604bc37c22e0a95) * (edit) phoenix-core/src/main/java/org/apache/phoenix/schema/stats/StatisticsUtil.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/util/MetaDataUtil.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/index/ViewIndexIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/schema/TableProperty.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/schema/stats/StatisticsCollectorFactory.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/schema/stats/NoOpStatisticsCollector.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataProtocol.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/compile/PostDDLCompiler.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/util/IndexUtil.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/schema/stats/StatisticsScanner.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/coprocessor/UngroupedAggregateRegionObserver.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/schema/MetaDataClient.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/schema/stats/StatisticsCollector.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/schema/stats/DefaultStatisticsCollector.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/query/QueryConstants.java * (edit) phoenix-core/src/test/java/org/apache/phoenix/compile/QueryCompilerTest.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/GroupByCaseIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/util/UpgradeUtil.java > Allow stats to be configured on a table-by-table basis > -- > > Key: PHOENIX-2675 > URL: https://issues.apache.org/jira/browse/PHOENIX-2675 > Project: Phoenix > Issue Type: Improvement >Reporter: James Taylor >Assignee: James Taylor > Fix For: 4.9.0 > > Attachments: PHOENIX-2675.patch, PHOENIX-2675_wip.patch, > PHOENIX-2675_wip2.patch > > > Currently stats are controlled and collected at a global level. We should > allow them to be configured on a table-by-table basis. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3359) Update JDBC urls to match connection type
[
https://issues.apache.org/jira/browse/PHOENIX-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563821#comment-15563821
]
Maryann Xue commented on PHOENIX-3359:
--
So expect these test cases should have a different test method now. Maybe we
can get the EXPLAIN plan for these queries and see if they contain certain
strings, for example, you can find test cases in CalciteIT containing a string
like this
{code}
PhoenixServerAggregate(group=[{0}], EXPR$1=[COUNT($3)], isOrdered=[true])
{code}
> Update JDBC urls to match connection type
> -
>
> Key: PHOENIX-3359
> URL: https://issues.apache.org/jira/browse/PHOENIX-3359
> Project: Phoenix
> Issue Type: Sub-task
>Reporter: Eric Lomore
>Assignee: Eric Lomore
>
> Connection's that are intended to be unwrapped into PhoenixConnection's need
> to be created using the old JDBC url,
> {{DriverManager.getConnection(getOldUrl())}}
> {{PhoenixConnection pc = conn.unwrap(PhoenixConnection.class);}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (PHOENIX-3181) Run test methods in parallel to reduce test suite run time
[ https://issues.apache.org/jira/browse/PHOENIX-3181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated PHOENIX-3181: -- Attachment: PHOENIX-3181_v2.patch > Run test methods in parallel to reduce test suite run time > -- > > Key: PHOENIX-3181 > URL: https://issues.apache.org/jira/browse/PHOENIX-3181 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: James Taylor > Attachments: PHOENIX-3181_v1.patch, PHOENIX-3181_v2.patch > > > With PHOENIX-3036, the tests within a test class can be executed in parallel > since the table names won't conflict. This should greatly reduce the time > taken to run all of our tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-3181) Run test methods in parallel to reduce test suite run time
[ https://issues.apache.org/jira/browse/PHOENIX-3181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated PHOENIX-3181: -- Attachment: (was: PHOENIX-3181_v2.patch) > Run test methods in parallel to reduce test suite run time > -- > > Key: PHOENIX-3181 > URL: https://issues.apache.org/jira/browse/PHOENIX-3181 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: James Taylor > Attachments: PHOENIX-3181_v1.patch > > > With PHOENIX-3036, the tests within a test class can be executed in parallel > since the table names won't conflict. This should greatly reduce the time > taken to run all of our tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (PHOENIX-3369) Update Phoenix-Pig integration documentatino
Josh Elser created PHOENIX-3369: --- Summary: Update Phoenix-Pig integration documentatino Key: PHOENIX-3369 URL: https://issues.apache.org/jira/browse/PHOENIX-3369 Project: Phoenix Issue Type: Sub-task Reporter: Josh Elser Assignee: Josh Elser Priority: Minor I noticed when trying to test out the changes here, the documentation on how to invoke Pig with the Phoenix JAR was extremely sparse (https://phoenix.apache.org/pig_integration.html). We could do some work to spruce this up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3240) ClassCastException from Pig loader
[
https://issues.apache.org/jira/browse/PHOENIX-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563725#comment-15563725
]
James Taylor commented on PHOENIX-3240:
---
[~mujtabachohan] - can you take a quick look?
> ClassCastException from Pig loader
> --
>
> Key: PHOENIX-3240
> URL: https://issues.apache.org/jira/browse/PHOENIX-3240
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.8.0
>Reporter: YoungWoo Kim
>Assignee: Josh Elser
> Fix For: 4.9.0, 4.8.2
>
>
> I'm loading data from Hive table to Phoenix table in using Phoenix-Pig
> integration. my pig script looks like following:
> {code}
> T = LOAD 'mydb.$TBL' USING org.apache.hive.hcatalog.pig.HCatLoader();
> STORE T into 'hbase://MYSCHEMA.$TBL' using
> org.apache.phoenix.pig.PhoenixHBaseStorage('i004,i005,i006','-batchSize
> 1000');
> {code}
> If the source table has timestamp, MapReduce job for Pig script does not work:
> {noformat}
> ERROR 0: java.lang.ClassCastException: org.joda.time.DateTime cannot be cast
> to org.apache.phoenix.shaded.org.joda.time.DateTime
> org.apache.pig.backend.executionengine.ExecException: ERROR 0:
> java.lang.ClassCastException: org.joda.time.DateTime cannot be cast to
> org.apache.phoenix.shaded.org.joda.time.DateTime
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.getStats(MapReduceLauncher.java:822)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:452)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:304)
> at org.apache.pig.PigServer.launchPlan(PigServer.java:1390)
> at
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1375)
> at org.apache.pig.PigServer.execute(PigServer.java:1364)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:415)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:398)
> at
> org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:171)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:234)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> at org.apache.pig.Main.run(Main.java:502)
> at org.apache.pig.Main.main(Main.java:177)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.lang.ClassCastException: org.joda.time.DateTime cannot be
> cast to org.apache.phoenix.shaded.org.joda.time.DateTime
> at
> org.apache.phoenix.pig.util.TypeUtil.castPigTypeToPhoenix(TypeUtil.java:201)
> at
> org.apache.phoenix.pig.PhoenixHBaseStorage.putNext(PhoenixHBaseStorage.java:189)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:136)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:95)
> at
> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:658)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map.collect(PigMapOnly.java:48)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:260)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3359) Update JDBC urls to match connection type
[
https://issues.apache.org/jira/browse/PHOENIX-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563705#comment-15563705
]
Eric Lomore commented on PHOENIX-3359:
--
There's a bunch of examples in QueryCompilerTest.java where we want to use
phoenix's query planner.
{code}public void testGroupByOrderPreserving() throws Exception {
Connection conn = DriverManager.getConnection(getUrl());
conn.createStatement().execute("CREATE TABLE t (k1 date not null, k2
date not null, k3 date not null, v varchar, constraint pk primary
key(k1,k2,k3))");
String[] queries = {
"SELECT 1 FROM T GROUP BY k3, (k1,k2)",
"SELECT 1 FROM T GROUP BY k2,k1,k3",
"SELECT 1 FROM T GROUP BY k1,k2",
"SELECT 1 FROM T GROUP BY k1",
"SELECT 1 FROM T GROUP BY CAST(k1 AS TIMESTAMP)",
"SELECT 1 FROM T GROUP BY (k1,k2,k3)",
"SELECT 1 FROM T GROUP BY TRUNC(k2, 'DAY'), CEIL(k1, 'HOUR')",
};
String query;
for (int i = 0; i < queries.length; i++) {
query = queries[i];
QueryPlan plan =
conn.createStatement().unwrap(PhoenixStatement.class).compileQuery(query);
assertTrue("Expected group by to be order preserving: " + query,
plan.getGroupBy().isOrderPreserving());
}
}{code}
> Update JDBC urls to match connection type
> -
>
> Key: PHOENIX-3359
> URL: https://issues.apache.org/jira/browse/PHOENIX-3359
> Project: Phoenix
> Issue Type: Sub-task
>Reporter: Eric Lomore
>Assignee: Eric Lomore
>
> Connection's that are intended to be unwrapped into PhoenixConnection's need
> to be created using the old JDBC url,
> {{DriverManager.getConnection(getOldUrl())}}
> {{PhoenixConnection pc = conn.unwrap(PhoenixConnection.class);}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3240) ClassCastException from Pig loader
[
https://issues.apache.org/jira/browse/PHOENIX-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563696#comment-15563696
]
ASF GitHub Bot commented on PHOENIX-3240:
-
GitHub user joshelser opened a pull request:
https://github.com/apache/phoenix/pull/217
PHOENIX-3240 Create phoenix-$VERSION-pig shaded jar
Running inside of Pig's environment creates some issues where Pig is already
providing some versions of dependencies (e.g. joda-time). Create a custom
JAR specifically to be used with Pig.
Testing approach:
1. `create table foobar(a varchar, b date not null, constraint pk primary
key (a, b));`
2. `cd /usr/local/lib/pig-0.16.0`
3. `PIG_HOME=$(pwd)
PIG_CLASSPATH=/usr/local/lib/phoenix/phoenix-4.9.0-HBase-1.2-SNAPSHOT-pig.jar
./bin/pig` (or alternatively, `register
/usr/local/lib/phoenix/phoenix-4.9.0-HBase-1.2-SNAPSHOT-pig.jar` in the pig
interpreter)
4. Create a two-column CSV file in HDFS (e.g. `a1,2016-10-10 17:05:00`)
5. `A = load 'times' using PigStorage(',') as (a:chararray, b:datetime);`
6. `STORE A into 'hbase://FOOBAR' using
org.apache.phoenix.pig.PhoenixHBaseStorage('localhost:2181:/hbase-1.2','-batchSize
100');`
7. `select * from FOOBAR;`
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/joshelser/phoenix 3240-shaded-pig
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/phoenix/pull/217.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #217
commit ee0101d175708d6abe1079b36d07bd4a9cbada16
Author: Josh Elser
Date: 2016-10-10T22:03:44Z
PHOENIX-3240 Create phoenix-$VERSION-pig shaded jar
Running inside of Pig's environment creates some issues where Pig is already
providing some versions of dependencies (e.g. joda-time). Create a custom
JAR specifically to be used with Pig.
> ClassCastException from Pig loader
> --
>
> Key: PHOENIX-3240
> URL: https://issues.apache.org/jira/browse/PHOENIX-3240
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.8.0
>Reporter: YoungWoo Kim
>Assignee: Josh Elser
> Fix For: 4.9.0
>
>
> I'm loading data from Hive table to Phoenix table in using Phoenix-Pig
> integration. my pig script looks like following:
> {code}
> T = LOAD 'mydb.$TBL' USING org.apache.hive.hcatalog.pig.HCatLoader();
> STORE T into 'hbase://MYSCHEMA.$TBL' using
> org.apache.phoenix.pig.PhoenixHBaseStorage('i004,i005,i006','-batchSize
> 1000');
> {code}
> If the source table has timestamp, MapReduce job for Pig script does not work:
> {noformat}
> ERROR 0: java.lang.ClassCastException: org.joda.time.DateTime cannot be cast
> to org.apache.phoenix.shaded.org.joda.time.DateTime
> org.apache.pig.backend.executionengine.ExecException: ERROR 0:
> java.lang.ClassCastException: org.joda.time.DateTime cannot be cast to
> org.apache.phoenix.shaded.org.joda.time.DateTime
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.getStats(MapReduceLauncher.java:822)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:452)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:304)
> at org.apache.pig.PigServer.launchPlan(PigServer.java:1390)
> at
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1375)
> at org.apache.pig.PigServer.execute(PigServer.java:1364)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:415)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:398)
> at
> org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:171)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:234)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> at org.apache.pig.Main.run(Main.java:502)
> at org.apache.pig.Main.main(Main.java:177)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.lang.ClassCastException: org.joda.time.DateTime cannot be
> cast to
[GitHub] phoenix pull request #217: PHOENIX-3240 Create phoenix-$VERSION-pig shaded j...
GitHub user joshelser opened a pull request:
https://github.com/apache/phoenix/pull/217
PHOENIX-3240 Create phoenix-$VERSION-pig shaded jar
Running inside of Pig's environment creates some issues where Pig is already
providing some versions of dependencies (e.g. joda-time). Create a custom
JAR specifically to be used with Pig.
Testing approach:
1. `create table foobar(a varchar, b date not null, constraint pk primary
key (a, b));`
2. `cd /usr/local/lib/pig-0.16.0`
3. `PIG_HOME=$(pwd)
PIG_CLASSPATH=/usr/local/lib/phoenix/phoenix-4.9.0-HBase-1.2-SNAPSHOT-pig.jar
./bin/pig` (or alternatively, `register
/usr/local/lib/phoenix/phoenix-4.9.0-HBase-1.2-SNAPSHOT-pig.jar` in the pig
interpreter)
4. Create a two-column CSV file in HDFS (e.g. `a1,2016-10-10 17:05:00`)
5. `A = load 'times' using PigStorage(',') as (a:chararray, b:datetime);`
6. `STORE A into 'hbase://FOOBAR' using
org.apache.phoenix.pig.PhoenixHBaseStorage('localhost:2181:/hbase-1.2','-batchSize
100');`
7. `select * from FOOBAR;`
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/joshelser/phoenix 3240-shaded-pig
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/phoenix/pull/217.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #217
commit ee0101d175708d6abe1079b36d07bd4a9cbada16
Author: Josh Elser
Date: 2016-10-10T22:03:44Z
PHOENIX-3240 Create phoenix-$VERSION-pig shaded jar
Running inside of Pig's environment creates some issues where Pig is already
providing some versions of dependencies (e.g. joda-time). Create a custom
JAR specifically to be used with Pig.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (PHOENIX-3242) Support UDF in Phoenix-Calcite Integration
[ https://issues.apache.org/jira/browse/PHOENIX-3242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563673#comment-15563673 ] Maryann Xue commented on PHOENIX-3242: -- BTW, would you mind completing the default parameter support before doing other patches? That way we can close this issue. > Support UDF in Phoenix-Calcite Integration > -- > > Key: PHOENIX-3242 > URL: https://issues.apache.org/jira/browse/PHOENIX-3242 > Project: Phoenix > Issue Type: Task >Reporter: Rajeshbabu Chintaguntla >Assignee: Rajeshbabu Chintaguntla > Labels: calcite > Attachments: PHOENIX-3242-WIP-2.patch, PHOENIX-3242-wip.patch, > PHOENIX-3242_select-wip.patch, PHOENIX-3242_v1.patch, PHOENIX-3242_v2.patch, > PHOENIX-3242_v3.patch, PHOENIX-3242_v4.patch, PHOENIX-3242_v5.patch, > PHOENIX-3242_v6.patch, PHOENIX-3242_v7.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3242) Support UDF in Phoenix-Calcite Integration
[ https://issues.apache.org/jira/browse/PHOENIX-3242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563670#comment-15563670 ] Maryann Xue commented on PHOENIX-3242: -- Thanks, [~rajeshbabu]! Just wanted to double check if we need to more work for PHOENIX-3355. But not sure which takes precedence, built-in or UDF. Let's continue this discussion on that JIRA then. > Support UDF in Phoenix-Calcite Integration > -- > > Key: PHOENIX-3242 > URL: https://issues.apache.org/jira/browse/PHOENIX-3242 > Project: Phoenix > Issue Type: Task >Reporter: Rajeshbabu Chintaguntla >Assignee: Rajeshbabu Chintaguntla > Labels: calcite > Attachments: PHOENIX-3242-WIP-2.patch, PHOENIX-3242-wip.patch, > PHOENIX-3242_select-wip.patch, PHOENIX-3242_v1.patch, PHOENIX-3242_v2.patch, > PHOENIX-3242_v3.patch, PHOENIX-3242_v4.patch, PHOENIX-3242_v5.patch, > PHOENIX-3242_v6.patch, PHOENIX-3242_v7.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-3242) Support UDF in Phoenix-Calcite Integration
[ https://issues.apache.org/jira/browse/PHOENIX-3242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maryann Xue updated PHOENIX-3242: - Summary: Support UDF in Phoenix-Calcite Integration (was: Support CREATE OR REPLACE FUNCTION in Phoenix-Calcite Integration) > Support UDF in Phoenix-Calcite Integration > -- > > Key: PHOENIX-3242 > URL: https://issues.apache.org/jira/browse/PHOENIX-3242 > Project: Phoenix > Issue Type: Sub-task >Reporter: Rajeshbabu Chintaguntla >Assignee: Rajeshbabu Chintaguntla > Labels: calcite > Attachments: PHOENIX-3242-WIP-2.patch, PHOENIX-3242-wip.patch, > PHOENIX-3242_select-wip.patch, PHOENIX-3242_v1.patch, PHOENIX-3242_v2.patch, > PHOENIX-3242_v3.patch, PHOENIX-3242_v4.patch, PHOENIX-3242_v5.patch, > PHOENIX-3242_v6.patch, PHOENIX-3242_v7.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3359) Update JDBC urls to match connection type
[
https://issues.apache.org/jira/browse/PHOENIX-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563616#comment-15563616
]
Maryann Xue commented on PHOENIX-3359:
--
Definitely we are not to use the same objects for each occurrence. It's just I
didn't realize both the connection object and the statement object are needed.
Anyway, can you show an example? I'd like to see why a PhoenixStatement
instance is needed.
> Update JDBC urls to match connection type
> -
>
> Key: PHOENIX-3359
> URL: https://issues.apache.org/jira/browse/PHOENIX-3359
> Project: Phoenix
> Issue Type: Sub-task
>Reporter: Eric Lomore
>Assignee: Eric Lomore
>
> Connection's that are intended to be unwrapped into PhoenixConnection's need
> to be created using the old JDBC url,
> {{DriverManager.getConnection(getOldUrl())}}
> {{PhoenixConnection pc = conn.unwrap(PhoenixConnection.class);}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3242) Support CREATE OR REPLACE FUNCTION in Phoenix-Calcite Integration
[ https://issues.apache.org/jira/browse/PHOENIX-3242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563612#comment-15563612 ] Rajeshbabu Chintaguntla commented on PHOENIX-3242: -- bq. does our FunctionResolver resolve built-in functions as well? No but we can tweak the code in PhoenixSchema to first check for built in functions and then resolve for UDFs. So that we need not worry about registering the methods as UDFs. bq.You also need to check how well view extension columns are supported in Calcite-Phoenix. If the scope is too large, let's simply open another JIRA for view upsert. I think we need to chance create view parsing as well better to work as separate JIRA [~maryannxue]. Will raise and work on it. > Support CREATE OR REPLACE FUNCTION in Phoenix-Calcite Integration > - > > Key: PHOENIX-3242 > URL: https://issues.apache.org/jira/browse/PHOENIX-3242 > Project: Phoenix > Issue Type: Sub-task >Reporter: Rajeshbabu Chintaguntla >Assignee: Rajeshbabu Chintaguntla > Labels: calcite > Attachments: PHOENIX-3242-WIP-2.patch, PHOENIX-3242-wip.patch, > PHOENIX-3242_select-wip.patch, PHOENIX-3242_v1.patch, PHOENIX-3242_v2.patch, > PHOENIX-3242_v3.patch, PHOENIX-3242_v4.patch, PHOENIX-3242_v5.patch, > PHOENIX-3242_v6.patch, PHOENIX-3242_v7.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3345) SQLException code's not propagating as expected
[
https://issues.apache.org/jira/browse/PHOENIX-3345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563604#comment-15563604
]
ASF GitHub Bot commented on PHOENIX-3345:
-
Github user lomoree commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/215#discussion_r82688259
--- Diff:
phoenix-core/src/main/java/org/apache/calcite/jdbc/PhoenixCalciteFactory.java
---
@@ -161,6 +162,29 @@ public Object apply(
}));
}
+public CalciteStatement createStatement(String sql, int
resultSetType, int resultSetConcurrency, int resultSetHoldability) throws
SQLException {
+try {
+return super.createStatement(resultSetType,
resultSetConcurrency, resultSetHoldability);
+} catch (SQLException e) {
+if (e.getCause().getCause() instanceof SQLException) {
+throw (SQLException) e.getCause().getCause();
--- End diff --
I wondered about this myself. While I didn't encounter any cases where
exception levels varied, it's a good idea to account for it. Will make these
changes, thanks @maryannxue
> SQLException code's not propagating as expected
> ---
>
> Key: PHOENIX-3345
> URL: https://issues.apache.org/jira/browse/PHOENIX-3345
> Project: Phoenix
> Issue Type: Sub-task
>Reporter: Eric Lomore
>Assignee: Eric Lomore
> Attachments: variablestate.png
>
>
> Perhaps this is intended by Calcite, but when errors are thrown by execute()
> functions the error code that is initially thrown (say 1000) does not make
> its way to the final SQLException on top.
> This is prevalent in multiple tests throughout QueryCompilerTest.java. One
> such example is included below.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] phoenix pull request #215: PHOENIX-3345 SQLException code's not propagating ...
Github user lomoree commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/215#discussion_r82688259
--- Diff:
phoenix-core/src/main/java/org/apache/calcite/jdbc/PhoenixCalciteFactory.java
---
@@ -161,6 +162,29 @@ public Object apply(
}));
}
+public CalciteStatement createStatement(String sql, int
resultSetType, int resultSetConcurrency, int resultSetHoldability) throws
SQLException {
+try {
+return super.createStatement(resultSetType,
resultSetConcurrency, resultSetHoldability);
+} catch (SQLException e) {
+if (e.getCause().getCause() instanceof SQLException) {
+throw (SQLException) e.getCause().getCause();
--- End diff --
I wondered about this myself. While I didn't encounter any cases where
exception levels varied, it's a good idea to account for it. Will make these
changes, thanks @maryannxue
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (PHOENIX-3345) SQLException code's not propagating as expected
[
https://issues.apache.org/jira/browse/PHOENIX-3345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563582#comment-15563582
]
ASF GitHub Bot commented on PHOENIX-3345:
-
Github user maryannxue commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/215#discussion_r82686793
--- Diff:
phoenix-core/src/main/java/org/apache/calcite/jdbc/PhoenixCalciteFactory.java
---
@@ -161,6 +162,29 @@ public Object apply(
}));
}
+public CalciteStatement createStatement(String sql, int
resultSetType, int resultSetConcurrency, int resultSetHoldability) throws
SQLException {
+try {
+return super.createStatement(resultSetType,
resultSetConcurrency, resultSetHoldability);
+} catch (SQLException e) {
+if (e.getCause().getCause() instanceof SQLException) {
+throw (SQLException) e.getCause().getCause();
--- End diff --
Can we can have a utility method that unwraps the Exception? And as we
might different number of levels of Exception wrapping, can we just look down
one level at a time till we hit an instance of SQLException?
> SQLException code's not propagating as expected
> ---
>
> Key: PHOENIX-3345
> URL: https://issues.apache.org/jira/browse/PHOENIX-3345
> Project: Phoenix
> Issue Type: Sub-task
>Reporter: Eric Lomore
>Assignee: Eric Lomore
> Attachments: variablestate.png
>
>
> Perhaps this is intended by Calcite, but when errors are thrown by execute()
> functions the error code that is initially thrown (say 1000) does not make
> its way to the final SQLException on top.
> This is prevalent in multiple tests throughout QueryCompilerTest.java. One
> such example is included below.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] phoenix pull request #215: PHOENIX-3345 SQLException code's not propagating ...
Github user maryannxue commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/215#discussion_r82686793
--- Diff:
phoenix-core/src/main/java/org/apache/calcite/jdbc/PhoenixCalciteFactory.java
---
@@ -161,6 +162,29 @@ public Object apply(
}));
}
+public CalciteStatement createStatement(String sql, int
resultSetType, int resultSetConcurrency, int resultSetHoldability) throws
SQLException {
+try {
+return super.createStatement(resultSetType,
resultSetConcurrency, resultSetHoldability);
+} catch (SQLException e) {
+if (e.getCause().getCause() instanceof SQLException) {
+throw (SQLException) e.getCause().getCause();
--- End diff --
Can we can have a utility method that unwraps the Exception? And as we
might different number of levels of Exception wrapping, can we just look down
one level at a time till we hit an instance of SQLException?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (PHOENIX-3265) Surround columns named date with double quotes
[
https://issues.apache.org/jira/browse/PHOENIX-3265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563573#comment-15563573
]
Maryann Xue commented on PHOENIX-3265:
--
So this is supposed to go into master branch, right? Could you please open a
new PR based on master branch?
> Surround columns named date with double quotes
> --
>
> Key: PHOENIX-3265
> URL: https://issues.apache.org/jira/browse/PHOENIX-3265
> Project: Phoenix
> Issue Type: Sub-task
>Reporter: James Taylor
>Assignee: Eric Lomore
>
> In Phoenix {{date}} is not a reserved word, but perhaps in Calcite it is? If
> that's the case, we should surround occurrences of columns named {{date}} in
> double quotes to prevent the parser error.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3242) Support CREATE OR REPLACE FUNCTION in Phoenix-Calcite Integration
[ https://issues.apache.org/jira/browse/PHOENIX-3242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563568#comment-15563568 ] Maryann Xue commented on PHOENIX-3242: -- You also need to check how well view extension columns are supported in Calcite-Phoenix. If the scope is too large, let's simply open another JIRA for view upsert. > Support CREATE OR REPLACE FUNCTION in Phoenix-Calcite Integration > - > > Key: PHOENIX-3242 > URL: https://issues.apache.org/jira/browse/PHOENIX-3242 > Project: Phoenix > Issue Type: Sub-task >Reporter: Rajeshbabu Chintaguntla >Assignee: Rajeshbabu Chintaguntla > Labels: calcite > Attachments: PHOENIX-3242-WIP-2.patch, PHOENIX-3242-wip.patch, > PHOENIX-3242_select-wip.patch, PHOENIX-3242_v1.patch, PHOENIX-3242_v2.patch, > PHOENIX-3242_v3.patch, PHOENIX-3242_v4.patch, PHOENIX-3242_v5.patch, > PHOENIX-3242_v6.patch, PHOENIX-3242_v7.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3242) Support CREATE OR REPLACE FUNCTION in Phoenix-Calcite Integration
[ https://issues.apache.org/jira/browse/PHOENIX-3242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563560#comment-15563560 ] Maryann Xue commented on PHOENIX-3242: -- Thank you very much, [~rajeshbabu]! One question though, does our FunctionResolver resolve built-in functions as well? If yes, PHOENIX-3355 should already be done. > Support CREATE OR REPLACE FUNCTION in Phoenix-Calcite Integration > - > > Key: PHOENIX-3242 > URL: https://issues.apache.org/jira/browse/PHOENIX-3242 > Project: Phoenix > Issue Type: Sub-task >Reporter: Rajeshbabu Chintaguntla >Assignee: Rajeshbabu Chintaguntla > Labels: calcite > Attachments: PHOENIX-3242-WIP-2.patch, PHOENIX-3242-wip.patch, > PHOENIX-3242_select-wip.patch, PHOENIX-3242_v1.patch, PHOENIX-3242_v2.patch, > PHOENIX-3242_v3.patch, PHOENIX-3242_v4.patch, PHOENIX-3242_v5.patch, > PHOENIX-3242_v6.patch, PHOENIX-3242_v7.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3240) ClassCastException from Pig loader
[
https://issues.apache.org/jira/browse/PHOENIX-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563534#comment-15563534
]
Josh Elser commented on PHOENIX-3240:
-
Oh, and right, this also explains the issue. Pig is putting JodaTime on the
classpath too. We try to hand off a relocated DateTime instance and the Pig
code doesn't know how to handle that. We should just use version of JodaTime
from Pig.
Going back to the original plan, we should just create a new shaded jar just
for Pig that doesn't include JodaTime (since PIg would be providing it).
> ClassCastException from Pig loader
> --
>
> Key: PHOENIX-3240
> URL: https://issues.apache.org/jira/browse/PHOENIX-3240
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.8.0
>Reporter: YoungWoo Kim
>Assignee: Josh Elser
> Fix For: 4.9.0
>
>
> I'm loading data from Hive table to Phoenix table in using Phoenix-Pig
> integration. my pig script looks like following:
> {code}
> T = LOAD 'mydb.$TBL' USING org.apache.hive.hcatalog.pig.HCatLoader();
> STORE T into 'hbase://MYSCHEMA.$TBL' using
> org.apache.phoenix.pig.PhoenixHBaseStorage('i004,i005,i006','-batchSize
> 1000');
> {code}
> If the source table has timestamp, MapReduce job for Pig script does not work:
> {noformat}
> ERROR 0: java.lang.ClassCastException: org.joda.time.DateTime cannot be cast
> to org.apache.phoenix.shaded.org.joda.time.DateTime
> org.apache.pig.backend.executionengine.ExecException: ERROR 0:
> java.lang.ClassCastException: org.joda.time.DateTime cannot be cast to
> org.apache.phoenix.shaded.org.joda.time.DateTime
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.getStats(MapReduceLauncher.java:822)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:452)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:304)
> at org.apache.pig.PigServer.launchPlan(PigServer.java:1390)
> at
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1375)
> at org.apache.pig.PigServer.execute(PigServer.java:1364)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:415)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:398)
> at
> org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:171)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:234)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> at org.apache.pig.Main.run(Main.java:502)
> at org.apache.pig.Main.main(Main.java:177)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.lang.ClassCastException: org.joda.time.DateTime cannot be
> cast to org.apache.phoenix.shaded.org.joda.time.DateTime
> at
> org.apache.phoenix.pig.util.TypeUtil.castPigTypeToPhoenix(TypeUtil.java:201)
> at
> org.apache.phoenix.pig.PhoenixHBaseStorage.putNext(PhoenixHBaseStorage.java:189)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:136)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:95)
> at
> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:658)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map.collect(PigMapOnly.java:48)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:260)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at
[jira] [Commented] (PHOENIX-3240) ClassCastException from Pig loader
[
https://issues.apache.org/jira/browse/PHOENIX-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563520#comment-15563520
]
Josh Elser commented on PHOENIX-3240:
-
Ok, I got around that issue by removing org.apache.pig:pig from the
phoenix-client jar. Seems wrong to me to be packaging that (the user would be
using the pig-withouthadoop jar)
{noformat}
2016-10-10 17:09:04,069 WARN [main] org.apache.hadoop.mapred.YarnChild:
Exception running child : java.io.IOException: java.lang.ClassCastException:
org.joda.time.DateTime cannot be cast to
org.apache.phoenix.shaded.org.joda.time.DateTime
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.StoreFuncDecorator.putNext(StoreFuncDecorator.java:83)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:144)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:97)
at
org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:658)
at
org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
at
org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map.collect(PigMapOnly.java:48)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:282)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:275)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:65)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.ClassCastException: org.joda.time.DateTime cannot be cast
to org.apache.phoenix.shaded.org.joda.time.DateTime
at
org.apache.phoenix.pig.util.TypeUtil.castPigTypeToPhoenix(TypeUtil.java:199)
at
org.apache.phoenix.pig.PhoenixHBaseStorage.putNext(PhoenixHBaseStorage.java:189)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.StoreFuncDecorator.putNext(StoreFuncDecorator.java:75)
... 17 more
{noformat}
Anyways, I was able to repro this.
> ClassCastException from Pig loader
> --
>
> Key: PHOENIX-3240
> URL: https://issues.apache.org/jira/browse/PHOENIX-3240
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.8.0
>Reporter: YoungWoo Kim
>Assignee: Josh Elser
> Fix For: 4.9.0
>
>
> I'm loading data from Hive table to Phoenix table in using Phoenix-Pig
> integration. my pig script looks like following:
> {code}
> T = LOAD 'mydb.$TBL' USING org.apache.hive.hcatalog.pig.HCatLoader();
> STORE T into 'hbase://MYSCHEMA.$TBL' using
> org.apache.phoenix.pig.PhoenixHBaseStorage('i004,i005,i006','-batchSize
> 1000');
> {code}
> If the source table has timestamp, MapReduce job for Pig script does not work:
> {noformat}
> ERROR 0: java.lang.ClassCastException: org.joda.time.DateTime cannot be cast
> to org.apache.phoenix.shaded.org.joda.time.DateTime
> org.apache.pig.backend.executionengine.ExecException: ERROR 0:
> java.lang.ClassCastException: org.joda.time.DateTime cannot be cast to
> org.apache.phoenix.shaded.org.joda.time.DateTime
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.getStats(MapReduceLauncher.java:822)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:452)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:304)
> at org.apache.pig.PigServer.launchPlan(PigServer.java:1390)
> at
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1375)
> at org.apache.pig.PigServer.execute(PigServer.java:1364)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:415)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:398)
> at
> org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:171)
> at
>
[jira] [Commented] (PHOENIX-2935) IndexMetaData cache can expire when a delete and or query running on server
[ https://issues.apache.org/jira/browse/PHOENIX-2935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563474#comment-15563474 ] Enis Soztutar commented on PHOENIX-2935: bq. Problem could be a size of scan object become little large(by 2KB+ with 15-30 indexes) and RPC could be little slow which would be aggravated with large number of guidePosts or region in range for the query but not sure if this is huge side effect against failing the query on cache expiry. Agreed, however, it should be acceptable I think given that this is for UPSERT SELECT kind of queries where the expected runtime of the query is already high to offset the extra cost in serialization of index metadata. +1 for the patch. > IndexMetaData cache can expire when a delete and or query running on server > --- > > Key: PHOENIX-2935 > URL: https://issues.apache.org/jira/browse/PHOENIX-2935 > Project: Phoenix > Issue Type: Bug >Reporter: Ankit Singhal >Assignee: Ankit Singhal > Labels: index > Fix For: 4.9.0 > > Attachments: PHOENIX-2935.patch > > > IndexMetaData cache can expire when a delete or upsert query is running on > server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (PHOENIX-3242) Support CREATE OR REPLACE FUNCTION in Phoenix-Calcite Integration
[ https://issues.apache.org/jira/browse/PHOENIX-3242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563444#comment-15563444 ] Rajeshbabu Chintaguntla edited comment on PHOENIX-3242 at 10/10/16 8:50 PM: [~maryannxue] Thanks for review. I have separated the patch into PHOENIX-3242 and PHOENIX-3351 committed PHOENIX-3242. Handled review comments like reverted unnecessary changes. [~lomoree] you can continue making built-in methods as udfs. bq. Also, I think what Julian Hyde mentioned as the named parameter and default parameter interface this morning is in PhoenixScalarFunction already, which is an implementation class for Calcite's ScalarFunction. I will check this [~maryannxue]. For PHOENIX-3351 need to handle a case of writes for views additional columns. The tests related them are failing once I fix it will upload patch there. Thanks was (Author: rajeshbabu): [~maryannxue] Thanks for review. I have separated the patch into PHOENIX-3242 and PHOENIX-3351 committed PHOENIX-3242. Handled review comments like reverted unnecessary changes. [~lomoree] you can continue making built-in methods as udfs. bq. Also, I think what Julian Hyde mentioned as the named parameter and default parameter interface this morning is in PhoenixScalarFunction already, which is an implementation class for Calcite's ScalarFunction. I will check this [~maryannxue]. Thanks > Support CREATE OR REPLACE FUNCTION in Phoenix-Calcite Integration > - > > Key: PHOENIX-3242 > URL: https://issues.apache.org/jira/browse/PHOENIX-3242 > Project: Phoenix > Issue Type: Sub-task >Reporter: Rajeshbabu Chintaguntla >Assignee: Rajeshbabu Chintaguntla > Labels: calcite > Attachments: PHOENIX-3242-WIP-2.patch, PHOENIX-3242-wip.patch, > PHOENIX-3242_select-wip.patch, PHOENIX-3242_v1.patch, PHOENIX-3242_v2.patch, > PHOENIX-3242_v3.patch, PHOENIX-3242_v4.patch, PHOENIX-3242_v5.patch, > PHOENIX-3242_v6.patch, PHOENIX-3242_v7.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (PHOENIX-3244) Support adding/listing udf jars in Phoenix-Calcite Integration
[ https://issues.apache.org/jira/browse/PHOENIX-3244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajeshbabu Chintaguntla resolved PHOENIX-3244. -- Resolution: Fixed Fixed as part of PHOENIX-3242. > Support adding/listing udf jars in Phoenix-Calcite Integration > -- > > Key: PHOENIX-3244 > URL: https://issues.apache.org/jira/browse/PHOENIX-3244 > Project: Phoenix > Issue Type: Sub-task >Reporter: Rajeshbabu Chintaguntla >Assignee: Rajeshbabu Chintaguntla > Labels: calcite > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (PHOENIX-3359) Update JDBC urls to match connection type
[
https://issues.apache.org/jira/browse/PHOENIX-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563434#comment-15563434
]
Eric Lomore edited comment on PHOENIX-3359 at 10/10/16 8:47 PM:
I'm not sure I see the reasoning behind a utility method. Both the connection
and statement objects will be needed, instantiation will vary (some are
prepared), and it's not like we would re-use the same objects on every test.
Shouldn't I just find cases of {{unwrap(PhoenixStatement.class)}} and
{{unwrap(PhoenixPreparedStatement.class}} and spot fix those? Thanks!
was (Author: lomoree):
I'm not sure I see the reasoning behind a utility method. Both the connection
and statement objects will be needed in the test context, and it's not like we
would re-use the same objects on every test.
Shouldn't I just find cases of {{unwrap(PhoenixStatement.class)}} and spot fix
those? Thanks!
> Update JDBC urls to match connection type
> -
>
> Key: PHOENIX-3359
> URL: https://issues.apache.org/jira/browse/PHOENIX-3359
> Project: Phoenix
> Issue Type: Sub-task
>Reporter: Eric Lomore
>Assignee: Eric Lomore
>
> Connection's that are intended to be unwrapped into PhoenixConnection's need
> to be created using the old JDBC url,
> {{DriverManager.getConnection(getOldUrl())}}
> {{PhoenixConnection pc = conn.unwrap(PhoenixConnection.class);}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (PHOENIX-3244) Support adding/listing udf jars in Phoenix-Calcite Integration
[ https://issues.apache.org/jira/browse/PHOENIX-3244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajeshbabu Chintaguntla updated PHOENIX-3244: - Labels: calcite (was: ) > Support adding/listing udf jars in Phoenix-Calcite Integration > -- > > Key: PHOENIX-3244 > URL: https://issues.apache.org/jira/browse/PHOENIX-3244 > Project: Phoenix > Issue Type: Sub-task >Reporter: Rajeshbabu Chintaguntla >Assignee: Rajeshbabu Chintaguntla > Labels: calcite > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (PHOENIX-3243) Support DROP FUNCTION in Phoenix-Calcite Integration
[ https://issues.apache.org/jira/browse/PHOENIX-3243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajeshbabu Chintaguntla resolved PHOENIX-3243. -- Resolution: Fixed Fixed as part of PHOENIX-3242. > Support DROP FUNCTION in Phoenix-Calcite Integration > > > Key: PHOENIX-3243 > URL: https://issues.apache.org/jira/browse/PHOENIX-3243 > Project: Phoenix > Issue Type: Sub-task >Reporter: Rajeshbabu Chintaguntla >Assignee: Rajeshbabu Chintaguntla > Labels: calcite > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3287) SecureUserConnectionsTest is flapping
[ https://issues.apache.org/jira/browse/PHOENIX-3287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563449#comment-15563449 ] Hudson commented on PHOENIX-3287: - FAILURE: Integrated in Jenkins build Phoenix-4.8-HBase-1.2 #35 (See [https://builds.apache.org/job/Phoenix-4.8-HBase-1.2/35/]) PHOENIX-3287 Retry starting KDC on BindException (elserj: rev af89f1655423d081eff300db13dc76bab0fed79f) * (edit) phoenix-core/src/test/java/org/apache/phoenix/jdbc/SecureUserConnectionsTest.java > SecureUserConnectionsTest is flapping > - > > Key: PHOENIX-3287 > URL: https://issues.apache.org/jira/browse/PHOENIX-3287 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.8.1 >Reporter: James Taylor >Assignee: Josh Elser >Priority: Minor > Fix For: 4.9.0, 4.8.2 > > Attachments: PHOENIX-3287.001.patch > > > Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 99.229 sec > <<< FAILURE! - in org.apache.phoenix.jdbc.SecureUserConnectionsTest > org.apache.phoenix.jdbc.SecureUserConnectionsTest Time elapsed: 99.229 sec > <<< ERROR! > java.net.BindException: Address already in use > See https://builds.apache.org/job/Phoenix-master/1404/changes for example > (and another earlier one). Wild guess, but maybe the port needs to be > randomized? Can it be? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-3243) Support DROP FUNCTION in Phoenix-Calcite Integration
[ https://issues.apache.org/jira/browse/PHOENIX-3243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajeshbabu Chintaguntla updated PHOENIX-3243: - Labels: calcite (was: ) > Support DROP FUNCTION in Phoenix-Calcite Integration > > > Key: PHOENIX-3243 > URL: https://issues.apache.org/jira/browse/PHOENIX-3243 > Project: Phoenix > Issue Type: Sub-task >Reporter: Rajeshbabu Chintaguntla >Assignee: Rajeshbabu Chintaguntla > Labels: calcite > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3242) Support CREATE OR REPLACE FUNCTION in Phoenix-Calcite Integration
[ https://issues.apache.org/jira/browse/PHOENIX-3242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563444#comment-15563444 ] Rajeshbabu Chintaguntla commented on PHOENIX-3242: -- [~maryannxue] Thanks for review. I have separated the patch into PHOENIX-3242 and PHOENIX-3351 committed PHOENIX-3242. Handled review comments like reverted unnecessary changes. [~lomoree] you can continue making built-in methods as udfs. bq. Also, I think what Julian Hyde mentioned as the named parameter and default parameter interface this morning is in PhoenixScalarFunction already, which is an implementation class for Calcite's ScalarFunction. I will check this [~maryannxue]. Thanks > Support CREATE OR REPLACE FUNCTION in Phoenix-Calcite Integration > - > > Key: PHOENIX-3242 > URL: https://issues.apache.org/jira/browse/PHOENIX-3242 > Project: Phoenix > Issue Type: Sub-task >Reporter: Rajeshbabu Chintaguntla >Assignee: Rajeshbabu Chintaguntla > Labels: calcite > Attachments: PHOENIX-3242-WIP-2.patch, PHOENIX-3242-wip.patch, > PHOENIX-3242_select-wip.patch, PHOENIX-3242_v1.patch, PHOENIX-3242_v2.patch, > PHOENIX-3242_v3.patch, PHOENIX-3242_v4.patch, PHOENIX-3242_v5.patch, > PHOENIX-3242_v6.patch, PHOENIX-3242_v7.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3359) Update JDBC urls to match connection type
[
https://issues.apache.org/jira/browse/PHOENIX-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563434#comment-15563434
]
Eric Lomore commented on PHOENIX-3359:
--
I'm not sure I see the reasoning behind a utility method. Both the connection
and statement objects will be needed in the test context, and it's not like we
would re-use the same objects on every test.
Shouldn't I just find cases of {{unwrap(PhoenixStatement.class)}} and spot fix
those? Thanks!
> Update JDBC urls to match connection type
> -
>
> Key: PHOENIX-3359
> URL: https://issues.apache.org/jira/browse/PHOENIX-3359
> Project: Phoenix
> Issue Type: Sub-task
>Reporter: Eric Lomore
>Assignee: Eric Lomore
>
> Connection's that are intended to be unwrapped into PhoenixConnection's need
> to be created using the old JDBC url,
> {{DriverManager.getConnection(getOldUrl())}}
> {{PhoenixConnection pc = conn.unwrap(PhoenixConnection.class);}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (PHOENIX-3368) Default GUIDE_POSTS_WIDTH of an index to value from its table
James Taylor created PHOENIX-3368: - Summary: Default GUIDE_POSTS_WIDTH of an index to value from its table Key: PHOENIX-3368 URL: https://issues.apache.org/jira/browse/PHOENIX-3368 Project: Phoenix Issue Type: Bug Reporter: James Taylor Fix For: 4.9.0 Currently the GUIDE_POSTS_WIDTH property on an index is independent from its value for its table. We should default the index to the same value as the table when the index is created. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3357) Issue with column re-ordering in column family support for Calcite-Phoenix
[
https://issues.apache.org/jira/browse/PHOENIX-3357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563329#comment-15563329
]
Maryann Xue commented on PHOENIX-3357:
--
Yes, this idea sounds good to me. Basically there are two places where we need
this interface, one is for SELECT *, the other for UPSERT. It seems easy for
UPSERT by adding a new interface to Table, and I'll see for SELECT *. I'll
update the title of CALCITE-1426 accordingly once we've reached consensus on
the interface.
> Issue with column re-ordering in column family support for Calcite-Phoenix
> --
>
> Key: PHOENIX-3357
> URL: https://issues.apache.org/jira/browse/PHOENIX-3357
> Project: Phoenix
> Issue Type: Sub-task
>Reporter: Maryann Xue
>Assignee: Maryann Xue
> Labels: calcite
>
> Found another issue here. Right now we re-order our columns so that columns
> from the same family can be grouped together and that column family can be
> taken as a structured type. So OrderByIT, there's a test regression in
> https://builds.apache.org/job/Phoenix-calcite/20/#showFailuresLink, because
> it has a table definition like:
> {code}
> CREATE TABLE t
> (a_string varchar not null, cf1.a integer, cf1.b varchar, col1 integer,
> cf2.c varchar, cf2.d integer, col2 integer
> CONSTRAINT pk PRIMARY KEY (a_string))
> {code}
> So when we implement PhoenixTable.getRowType(), the table definition is
> actually changed from "a_string, cf1.a, cf1.b, col1, cf2.c, cf2.d, col2" into
> "a_string, cf1.a, cf1.b, col1, col2 cf2.c, cf2.d". As a result "UPSERT INTO
> T" expects a different column order (data type order) for parameters, and we
> got an Exception. This happens just because the column re-ordering and has
> nothing to do with the changes for PHOENIX-2679 yet. UPSERT will get more
> complicated once we apply PHOENIX-2679 changes, which means there's more work
> to do for UPSERT for column family support. But regarding this problem alone,
> would you think it would make sense to add a check in Phoenix DDL that
> requires users to put column definitions from the same family together, like
> in this case, only allows "a_string, cf1.a, cf1.b, col1, col2 cf2.c, cf2.d",
> [~jamestaylor]?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3240) ClassCastException from Pig loader
[
https://issues.apache.org/jira/browse/PHOENIX-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563313#comment-15563313
]
Josh Elser commented on PHOENIX-3240:
-
I'm trying out pig-0.13.0 with 4.9.0-SNAPSHOT and not having much success. Pig
0.13 seems to be expecting jline-1.0, whereas we bundle the relocated
jline-2.11. For some reason, a jline 1.0 class is trying to be loaded using the
relocated package name (which obviously doesn't exist). I'm.. not quite sure
how to address that one.
> ClassCastException from Pig loader
> --
>
> Key: PHOENIX-3240
> URL: https://issues.apache.org/jira/browse/PHOENIX-3240
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.8.0
>Reporter: YoungWoo Kim
>Assignee: Josh Elser
> Fix For: 4.9.0
>
>
> I'm loading data from Hive table to Phoenix table in using Phoenix-Pig
> integration. my pig script looks like following:
> {code}
> T = LOAD 'mydb.$TBL' USING org.apache.hive.hcatalog.pig.HCatLoader();
> STORE T into 'hbase://MYSCHEMA.$TBL' using
> org.apache.phoenix.pig.PhoenixHBaseStorage('i004,i005,i006','-batchSize
> 1000');
> {code}
> If the source table has timestamp, MapReduce job for Pig script does not work:
> {noformat}
> ERROR 0: java.lang.ClassCastException: org.joda.time.DateTime cannot be cast
> to org.apache.phoenix.shaded.org.joda.time.DateTime
> org.apache.pig.backend.executionengine.ExecException: ERROR 0:
> java.lang.ClassCastException: org.joda.time.DateTime cannot be cast to
> org.apache.phoenix.shaded.org.joda.time.DateTime
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.getStats(MapReduceLauncher.java:822)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:452)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:304)
> at org.apache.pig.PigServer.launchPlan(PigServer.java:1390)
> at
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1375)
> at org.apache.pig.PigServer.execute(PigServer.java:1364)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:415)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:398)
> at
> org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:171)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:234)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> at org.apache.pig.Main.run(Main.java:502)
> at org.apache.pig.Main.main(Main.java:177)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.lang.ClassCastException: org.joda.time.DateTime cannot be
> cast to org.apache.phoenix.shaded.org.joda.time.DateTime
> at
> org.apache.phoenix.pig.util.TypeUtil.castPigTypeToPhoenix(TypeUtil.java:201)
> at
> org.apache.phoenix.pig.PhoenixHBaseStorage.putNext(PhoenixHBaseStorage.java:189)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:136)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:95)
> at
> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:658)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map.collect(PigMapOnly.java:48)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:260)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at
[jira] [Commented] (PHOENIX-3265) Surround columns named date with double quotes
[
https://issues.apache.org/jira/browse/PHOENIX-3265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563286#comment-15563286
]
Maryann Xue commented on PHOENIX-3265:
--
Yes, sure. I thought you were waiting for [~jamestaylor]'s approval though. I
was syncing Calcite branch with master branch and thought the DATE column
quoting had been there before, so I committed for Join related ITs in order not
to introduce any regression. I knew it would cause some conflict for this
check-in, sorry about that. Anyway, I'll review and commit for you.
For PHOENIX-3345, somehow I missed the pull request since the github bot name
instead of your name appeared in the message. Sorry about that too.
> Surround columns named date with double quotes
> --
>
> Key: PHOENIX-3265
> URL: https://issues.apache.org/jira/browse/PHOENIX-3265
> Project: Phoenix
> Issue Type: Sub-task
>Reporter: James Taylor
>Assignee: Eric Lomore
>
> In Phoenix {{date}} is not a reserved word, but perhaps in Calcite it is? If
> that's the case, we should surround occurrences of columns named {{date}} in
> double quotes to prevent the parser error.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3361) Collect stats correctly for local indexes
[ https://issues.apache.org/jira/browse/PHOENIX-3361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563281#comment-15563281 ] James Taylor commented on PHOENIX-3361: --- That's true about TestUtil#dumpTable, but I keep having to write that again when I need it, so I thought I'd just leave it in TestUtil instead. > Collect stats correctly for local indexes > - > > Key: PHOENIX-3361 > URL: https://issues.apache.org/jira/browse/PHOENIX-3361 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: James Taylor > Fix For: 4.9.0 > > Attachments: PHOENIX-3361.patch, PHOENIX-3361_wip.patch, > PHOENIX-3361_wip2.patch > > > Stats are not being correctly collected for local indexes and tables with > local indexes, because the logic that deletes the old stats is based on all > store families of a table. Thus, when stats are collected for a table with > local indexes, the local index stats would be deleted and when the stats for > a local index are collected, the stats for the table will be deleted. > Instead, if we cache stats per column family instead of across entire table, > plus detect if we're collecting stats for a local index versus the data > table, we can fix this issue. Caching stats per column family makes more > sense anyway, because depending on the query, we may only use the stats > associated with the default column family. Also, we wouldn't want to have to > pull over stats for local index with the data table stats. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3357) Issue with column re-ordering in column family support for Calcite-Phoenix
[
https://issues.apache.org/jira/browse/PHOENIX-3357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563275#comment-15563275
]
Julian Hyde commented on PHOENIX-3357:
--
I think the ordered list of columns should be a property of the table. Let's
not try to expand star by recursively expanding each of the column families. I
think that means we don't have to modify RelDataType. We might need to add an
extra interface to Table.
> Issue with column re-ordering in column family support for Calcite-Phoenix
> --
>
> Key: PHOENIX-3357
> URL: https://issues.apache.org/jira/browse/PHOENIX-3357
> Project: Phoenix
> Issue Type: Sub-task
>Reporter: Maryann Xue
>Assignee: Maryann Xue
> Labels: calcite
>
> Found another issue here. Right now we re-order our columns so that columns
> from the same family can be grouped together and that column family can be
> taken as a structured type. So OrderByIT, there's a test regression in
> https://builds.apache.org/job/Phoenix-calcite/20/#showFailuresLink, because
> it has a table definition like:
> {code}
> CREATE TABLE t
> (a_string varchar not null, cf1.a integer, cf1.b varchar, col1 integer,
> cf2.c varchar, cf2.d integer, col2 integer
> CONSTRAINT pk PRIMARY KEY (a_string))
> {code}
> So when we implement PhoenixTable.getRowType(), the table definition is
> actually changed from "a_string, cf1.a, cf1.b, col1, cf2.c, cf2.d, col2" into
> "a_string, cf1.a, cf1.b, col1, col2 cf2.c, cf2.d". As a result "UPSERT INTO
> T" expects a different column order (data type order) for parameters, and we
> got an Exception. This happens just because the column re-ordering and has
> nothing to do with the changes for PHOENIX-2679 yet. UPSERT will get more
> complicated once we apply PHOENIX-2679 changes, which means there's more work
> to do for UPSERT for column family support. But regarding this problem alone,
> would you think it would make sense to add a check in Phoenix DDL that
> requires users to put column definitions from the same family together, like
> in this case, only allows "a_string, cf1.a, cf1.b, col1, col2 cf2.c, cf2.d",
> [~jamestaylor]?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3367) Replace usage of bind variables in BaseTest.createTestTable()
[ https://issues.apache.org/jira/browse/PHOENIX-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563277#comment-15563277 ] James Taylor commented on PHOENIX-3367: --- I've been trying hard to get rid of BaseTest.createTestTable(). Most usages are gone already. Each test should just manage their own DDL rather than having it funnel through there. If the SPLIT ON is done differently for different tests, is it still an issue? Then you can just construct the SPLIT ON clause differently as needed (i.e. CREATE TABLE ... SPLIT ON (2,'foo') ). > Replace usage of bind variables in BaseTest.createTestTable() > - > > Key: PHOENIX-3367 > URL: https://issues.apache.org/jira/browse/PHOENIX-3367 > Project: Phoenix > Issue Type: Sub-task >Reporter: Maryann Xue >Priority: Minor > Labels: calcite > > We don't support bind variables in DDL yet, due to > https://issues.apache.org/jira/browse/CALCITE-1319. So we can probably > replace usage of bind variables in BaseTest.createTestTable() with sql string > construction. In a previous check-in, I commented out > BaseTest.getDefaultSplits() so that this wouldn't have big impact on the pass > rate, but once this issue is solved, we can uncomment those lines. Anyway, do > you think this is the right thing to do for now, [~jamestaylor]? or should we > wait on CALCITE-1319? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3359) Update JDBC urls to match connection type
[
https://issues.apache.org/jira/browse/PHOENIX-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563266#comment-15563266
]
Maryann Xue commented on PHOENIX-3359:
--
Yes, let's just make this stay in test utility classes, since they are for test
purpose only. Wrap these lines in a utility function and put it in BaseTest or
anywhere you think might be more appropriate.
> Update JDBC urls to match connection type
> -
>
> Key: PHOENIX-3359
> URL: https://issues.apache.org/jira/browse/PHOENIX-3359
> Project: Phoenix
> Issue Type: Sub-task
>Reporter: Eric Lomore
>Assignee: Eric Lomore
>
> Connection's that are intended to be unwrapped into PhoenixConnection's need
> to be created using the old JDBC url,
> {{DriverManager.getConnection(getOldUrl())}}
> {{PhoenixConnection pc = conn.unwrap(PhoenixConnection.class);}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (PHOENIX-3359) Update JDBC urls to match connection type
[
https://issues.apache.org/jira/browse/PHOENIX-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563255#comment-15563255
]
Eric Lomore edited comment on PHOENIX-3359 at 10/10/16 7:41 PM:
Upon further examination, the issue is actually in unwrapping the statement,
not the connection.
// PhoenixConnection pc = conn.unwrap(PhoenixConnection.class); executes fine
// PhoenixStatement stmt =
conn.createStatement().unwrap(PhoenixStatement.class); throws an error.
Unless we want to move things around, this seems as simple as:
{code}
Connection conn = DriverManager.getConnection(getUrl());
PhoenixConnection pc = conn.unwrap(PhoenixConnection.class);
PhoenixStatement stmt = (PhoenixStatement) pc.createStatement();{code}
Of course, we could also move this into an unwrap function in
PhoenixCalciteConnection but a utility casting AvaticaStatement <--->
PhoenixStatement requires 2 object creations.
That said, [~maryannxue] should I simply use the above fix and replace the old
code in test cases?
was (Author: lomoree):
Upon further examination, the issue is actually in unwrapping the statement,
not the connection.
// PhoenixConnection pc = conn.unwrap(PhoenixConnection.class); executes fine
// PhoenixStatement stmt =
conn.createStatement().unwrap(PhoenixStatement.class); throws an error.
Unless we want to move things around, this seems as simple as:
{code}
Connection conn = DriverManager.getConnection(getUrl());
PhoenixConnection pc = conn.unwrap(PhoenixConnection.class);
PhoenixStatement stmt = (PhoenixStatement) pc.createStatement();{code}
Of course, we could also move this into an unwrap function in
PhoenixCalciteConnection but a utility casting AvaticaStatement <--->
PhoenixStatement requires 2 object creations.
That said, should I simply use the above fix and replace the old code in test
cases?
> Update JDBC urls to match connection type
> -
>
> Key: PHOENIX-3359
> URL: https://issues.apache.org/jira/browse/PHOENIX-3359
> Project: Phoenix
> Issue Type: Sub-task
>Reporter: Eric Lomore
>Assignee: Eric Lomore
>
> Connection's that are intended to be unwrapped into PhoenixConnection's need
> to be created using the old JDBC url,
> {{DriverManager.getConnection(getOldUrl())}}
> {{PhoenixConnection pc = conn.unwrap(PhoenixConnection.class);}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3361) Collect stats correctly for local indexes
[ https://issues.apache.org/jira/browse/PHOENIX-3361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563264#comment-15563264 ] Samarth Jain commented on PHOENIX-3361: --- +1, patch looks great, [~jamestaylor]. Just one minor nit - I don't see the TestUtil#dumpTable method used anywhere. > Collect stats correctly for local indexes > - > > Key: PHOENIX-3361 > URL: https://issues.apache.org/jira/browse/PHOENIX-3361 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: James Taylor > Fix For: 4.9.0 > > Attachments: PHOENIX-3361.patch, PHOENIX-3361_wip.patch, > PHOENIX-3361_wip2.patch > > > Stats are not being correctly collected for local indexes and tables with > local indexes, because the logic that deletes the old stats is based on all > store families of a table. Thus, when stats are collected for a table with > local indexes, the local index stats would be deleted and when the stats for > a local index are collected, the stats for the table will be deleted. > Instead, if we cache stats per column family instead of across entire table, > plus detect if we're collecting stats for a local index versus the data > table, we can fix this issue. Caching stats per column family makes more > sense anyway, because depending on the query, we may only use the stats > associated with the default column family. Also, we wouldn't want to have to > pull over stats for local index with the data table stats. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-6) Support ON DUPLICATE KEY construct
[
https://issues.apache.org/jira/browse/PHOENIX-6?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563263#comment-15563263
]
James Taylor commented on PHOENIX-6:
I'm exploring an alternate syntax (with equivalent functionality) so that when
we move to Calcite there will be no need for users to make any changes. Calcite
supports the MERGE statement which is a superset of this functionality. I'm
trying to come up with the minimal support for MERGE we'd need.
{code}
MERGE MyTable AS t
USING (VALUES('a','b',0,0)) AS s(pk1,pk2,counter1,counter2)
ON (t.pk1,t.pk2) = (s.pk1, s.pk2)
WHEN MATCHED THEN UPDATE SET t.counter1 = s.counter1 + 1, t.counter2 =
s.counter2 + 1
WHEN NOT MATCHED THEN INSERT(pk1,pk2,counter1,counter2) VALUES('a', 'b', 0, 0)
{code}
Not sure how restrict/scope down the ON clause as the only construct allowed
there would be a PK match. The verbosity of the MERGE statement is not ideal
for this simple use case, as the ON DUPLICATE KEY construct only has to define
the WHEN MATCHED part. Also, it's unfortunate that the VALUES clause has to be
repeated. Is there any way around that, [~julianhyde]?
IMHO, the above is harder to read and will be harder to implement too. Perhaps
we can support the ON DUPLICATE KEY syntax and generate a MERGE statement from
it when the Phoenix/Calcite integration is complete.
Thoughts?
> Support ON DUPLICATE KEY construct
> --
>
> Key: PHOENIX-6
> URL: https://issues.apache.org/jira/browse/PHOENIX-6
> Project: Phoenix
> Issue Type: New Feature
>Reporter: James Taylor
>Assignee: James Taylor
> Fix For: 4.9.0
>
>
> To support inserting a new row only if it doesn't already exist, we should
> support the "on duplicate key" construct for UPSERT. With this construct, the
> UPSERT VALUES statement would run atomically and would thus require a read
> before write which would obviously have a negative impact on performance. For
> an example of similar syntax , see MySQL documentation at
> http://dev.mysql.com/doc/refman/5.7/en/insert-on-duplicate.html
> See this discussion for more detail:
> https://groups.google.com/d/msg/phoenix-hbase-user/Bof-TLrbTGg/68bnc8ZcWe0J.
> A related discussion is on PHOENIX-2909.
> Initially we'd support the following:
> # This would prevent the setting of VAL to 0 if the row already exists:
> {code}
> UPSERT INTO T (PK, VAL) VALUES ('a',0)
> ON DUPLICATE KEY IGNORE;
> {code}
> # This would increment the valueS of COUNTER1 and COUNTER2 if the row already
> exists and otherwise initialize them to 0:
> {code}
> UPSERT INTO T (PK, COUNTER1, COUNTER2) VALUES ('a',0,0)
> ON DUPLICATE KEY COUNTER1 = COUNTER1 + 1, COUNTER2 = COUNTER2 + 1;
> {code}
> So the general form is:
> {code}
> UPSERT ... VALUES ... [ ON DUPLICATE KEY [IGNORE | UPDATE
> =, ...] ]
> {code}
> The following restrictions will apply:
> - The may not be part of the primary key constraint - only KeyValue
> columns will be allowed.
> To handle the maintenance of immutable indexes, we'll need to push the
> maintenance to the server side.
> This is because the mutations for indexes on immutable tables are calculated
> on the client-side, while this new syntax would potentially modify the value
> on the server-side.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3265) Surround columns named date with double quotes
[
https://issues.apache.org/jira/browse/PHOENIX-3265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563260#comment-15563260
]
Eric Lomore commented on PHOENIX-3265:
--
[~maryannxue] looks like you did a commit that's a subset of this. Mind taking
a look at the PR I submitted for this? (and the SQLException PR)
> Surround columns named date with double quotes
> --
>
> Key: PHOENIX-3265
> URL: https://issues.apache.org/jira/browse/PHOENIX-3265
> Project: Phoenix
> Issue Type: Sub-task
>Reporter: James Taylor
>Assignee: Eric Lomore
>
> In Phoenix {{date}} is not a reserved word, but perhaps in Calcite it is? If
> that's the case, we should surround occurrences of columns named {{date}} in
> double quotes to prevent the parser error.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3367) Replace usage of bind variables in BaseTest.createTestTable()
[ https://issues.apache.org/jira/browse/PHOENIX-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563256#comment-15563256 ] Maryann Xue commented on PHOENIX-3367: -- Yes, I know. But they are all passed into createTestTable() in byte[] form, so is it ok that in the constructed sql string we just use the byte array? > Replace usage of bind variables in BaseTest.createTestTable() > - > > Key: PHOENIX-3367 > URL: https://issues.apache.org/jira/browse/PHOENIX-3367 > Project: Phoenix > Issue Type: Sub-task >Reporter: Maryann Xue >Priority: Minor > Labels: calcite > > We don't support bind variables in DDL yet, due to > https://issues.apache.org/jira/browse/CALCITE-1319. So we can probably > replace usage of bind variables in BaseTest.createTestTable() with sql string > construction. In a previous check-in, I commented out > BaseTest.getDefaultSplits() so that this wouldn't have big impact on the pass > rate, but once this issue is solved, we can uncomment those lines. Anyway, do > you think this is the right thing to do for now, [~jamestaylor]? or should we > wait on CALCITE-1319? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3359) Update JDBC urls to match connection type
[
https://issues.apache.org/jira/browse/PHOENIX-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563255#comment-15563255
]
Eric Lomore commented on PHOENIX-3359:
--
Upon further examination, the issue is actually in unwrapping the statement,
not the connection.
// PhoenixConnection pc = conn.unwrap(PhoenixConnection.class); executes fine
// PhoenixStatement stmt =
conn.createStatement().unwrap(PhoenixStatement.class); throws an error.
Unless we want to move things around, this seems as simple as:
{code}
Connection conn = DriverManager.getConnection(getUrl());
PhoenixConnection pc = conn.unwrap(PhoenixConnection.class);
PhoenixStatement stmt = (PhoenixStatement) pc.createStatement();{code}
Of course, we could also move this into an unwrap function in
PhoenixCalciteConnection but a utility casting AvaticaStatement <--->
PhoenixStatement requires 2 object creations.
That said, should I simply use the above fix and replace the old code in test
cases?
> Update JDBC urls to match connection type
> -
>
> Key: PHOENIX-3359
> URL: https://issues.apache.org/jira/browse/PHOENIX-3359
> Project: Phoenix
> Issue Type: Sub-task
>Reporter: Eric Lomore
>Assignee: Eric Lomore
>
> Connection's that are intended to be unwrapped into PhoenixConnection's need
> to be created using the old JDBC url,
> {{DriverManager.getConnection(getOldUrl())}}
> {{PhoenixConnection pc = conn.unwrap(PhoenixConnection.class);}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3367) Replace usage of bind variables in BaseTest.createTestTable()
[ https://issues.apache.org/jira/browse/PHOENIX-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563246#comment-15563246 ] James Taylor commented on PHOENIX-3367: --- I think it's fine to update the tests, [~maryannxue]. FWIW, the SPLIT ON bind variables can be of any type (VARCHAR, BIGINT, etc). > Replace usage of bind variables in BaseTest.createTestTable() > - > > Key: PHOENIX-3367 > URL: https://issues.apache.org/jira/browse/PHOENIX-3367 > Project: Phoenix > Issue Type: Sub-task >Reporter: Maryann Xue >Priority: Minor > Labels: calcite > > We don't support bind variables in DDL yet, due to > https://issues.apache.org/jira/browse/CALCITE-1319. So we can probably > replace usage of bind variables in BaseTest.createTestTable() with sql string > construction. In a previous check-in, I commented out > BaseTest.getDefaultSplits() so that this wouldn't have big impact on the pass > rate, but once this issue is solved, we can uncomment those lines. Anyway, do > you think this is the right thing to do for now, [~jamestaylor]? or should we > wait on CALCITE-1319? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3295) Remove ReplaceArrayColumnWithKeyValueColumnExpressionVisitor
[
https://issues.apache.org/jira/browse/PHOENIX-3295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563238#comment-15563238
]
James Taylor commented on PHOENIX-3295:
---
Thanks for the explanation. Let change ByteBuffer to ImmutableBytesPtr then. I
don't think we need the empty key value column in the array itself, since every
row will have the single key value. Seems to me it'd be simpler to just use
QueryConstants.EMPTY_COLUMN_BYTES as the column qualifier of the single key
value. Since our encoding scheme is only for new tables, we should have a
variant of QueryConstants.EMPTY_COLUMN_BYTES which is only a single character
too. I'd have to look closer at how the getKeyValueExpression is used while
adding parent indexes to child views. Not sure, but I we may be able to tweak
the IndexExpressionParseNodeRewriter so that we don't need that call any longer.
> Remove ReplaceArrayColumnWithKeyValueColumnExpressionVisitor
> -
>
> Key: PHOENIX-3295
> URL: https://issues.apache.org/jira/browse/PHOENIX-3295
> Project: Phoenix
> Issue Type: Improvement
>Reporter: Thomas D'Silva
>Assignee: Thomas D'Silva
> Attachments: PHOENIX-3295-v2.patch, PHOENIX-3295.patch
>
>
> ReplaceArrayColumnWithKeyValueColumnExpressionVisitor is only used in one
> place in IndexUtil.generateIndexData because we use a ValueGetter to get the
> value of the data table column using the original data table column
> reference. This is also why ArrayColumnExpression needs to keep track of the
> original key value column expression.
> If we don't replace the array column expression with the original column
> expression when it looks up the column by the qualifier it won't find it.
> {code}
> ValueGetter valueGetter = new ValueGetter() {
>
> @Override
> public byte[] getRowKey() {
> return dataMutation.getRow();
> }
>
> @Override
> public ImmutableBytesWritable
> getLatestValue(ColumnReference ref) {
> // Always return null for our empty key value, as
> this will cause the index
> // maintainer to always treat this Put as a new
> row.
> if (isEmptyKeyValue(table, ref)) {
> return null;
> }
> byte[] family = ref.getFamily();
> byte[] qualifier = ref.getQualifier();
> RowMutationState rowMutationState =
> valuesMap.get(ptr);
> PColumn column = null;
> try {
> column =
> table.getColumnFamily(family).getPColumnForColumnQualifier(qualifier);
> } catch (ColumnNotFoundException e) {
> } catch (ColumnFamilyNotFoundException e) {
> }
> if (rowMutationState!=null && column!=null) {
> byte[] value =
> rowMutationState.getColumnValues().get(column);
> ImmutableBytesPtr ptr = new
> ImmutableBytesPtr();
> ptr.set(value==null ?
> ByteUtil.EMPTY_BYTE_ARRAY : value);
>
> SchemaUtil.padData(table.getName().getString(), column, ptr);
> return ptr;
> }
> return null;
> }
>
> };
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (PHOENIX-3367) Replace usage of bind variables in BaseTest.createTestTable()
Maryann Xue created PHOENIX-3367: Summary: Replace usage of bind variables in BaseTest.createTestTable() Key: PHOENIX-3367 URL: https://issues.apache.org/jira/browse/PHOENIX-3367 Project: Phoenix Issue Type: Sub-task Reporter: Maryann Xue Priority: Minor We don't support bind variables in DDL yet, due to https://issues.apache.org/jira/browse/CALCITE-1319. So we can probably replace usage of bind variables in BaseTest.createTestTable() with sql string construction. In a previous check-in, I commented out BaseTest.getDefaultSplits() so that this wouldn't have big impact on the pass rate, but once this issue is solved, we can uncomment those lines. Anyway, do you think this is the right thing to do for now, [~jamestaylor]? or should we wait on CALCITE-1319? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3240) ClassCastException from Pig loader
[
https://issues.apache.org/jira/browse/PHOENIX-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563188#comment-15563188
]
Josh Elser commented on PHOENIX-3240:
-
bq. The phoenix-pig artifact should not expect to use bundled dependencies from
the phoenix-client jar. The isolation the shaded client jar is intended to give
us just backfired on us
Circling around: this looks like.. it should already be OK. phoenix-pig is
included in the phoenix-$VERSION-client.jar which means it should know where to
get joda-time from. I'll have to poke at this. Maybe I'm just missing something.
> ClassCastException from Pig loader
> --
>
> Key: PHOENIX-3240
> URL: https://issues.apache.org/jira/browse/PHOENIX-3240
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.8.0
>Reporter: YoungWoo Kim
>Assignee: Josh Elser
> Fix For: 4.9.0
>
>
> I'm loading data from Hive table to Phoenix table in using Phoenix-Pig
> integration. my pig script looks like following:
> {code}
> T = LOAD 'mydb.$TBL' USING org.apache.hive.hcatalog.pig.HCatLoader();
> STORE T into 'hbase://MYSCHEMA.$TBL' using
> org.apache.phoenix.pig.PhoenixHBaseStorage('i004,i005,i006','-batchSize
> 1000');
> {code}
> If the source table has timestamp, MapReduce job for Pig script does not work:
> {noformat}
> ERROR 0: java.lang.ClassCastException: org.joda.time.DateTime cannot be cast
> to org.apache.phoenix.shaded.org.joda.time.DateTime
> org.apache.pig.backend.executionengine.ExecException: ERROR 0:
> java.lang.ClassCastException: org.joda.time.DateTime cannot be cast to
> org.apache.phoenix.shaded.org.joda.time.DateTime
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.getStats(MapReduceLauncher.java:822)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:452)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:304)
> at org.apache.pig.PigServer.launchPlan(PigServer.java:1390)
> at
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1375)
> at org.apache.pig.PigServer.execute(PigServer.java:1364)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:415)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:398)
> at
> org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:171)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:234)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> at org.apache.pig.Main.run(Main.java:502)
> at org.apache.pig.Main.main(Main.java:177)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.lang.ClassCastException: org.joda.time.DateTime cannot be
> cast to org.apache.phoenix.shaded.org.joda.time.DateTime
> at
> org.apache.phoenix.pig.util.TypeUtil.castPigTypeToPhoenix(TypeUtil.java:201)
> at
> org.apache.phoenix.pig.PhoenixHBaseStorage.putNext(PhoenixHBaseStorage.java:189)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:136)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:95)
> at
> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:658)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map.collect(PigMapOnly.java:48)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:260)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at
[jira] [Commented] (PHOENIX-3357) Issue with column re-ordering in column family support for Calcite-Phoenix
[
https://issues.apache.org/jira/browse/PHOENIX-3357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563171#comment-15563171
]
Maryann Xue commented on PHOENIX-3357:
--
[~jamestaylor], [~julianhyde], I've created CALCITE-1426 to address the extra
support we'd like to have in Calcite to solve this problem. Not sure if this
would be the best interface, so [~julianhyde] please let us know your thoughts.
> Issue with column re-ordering in column family support for Calcite-Phoenix
> --
>
> Key: PHOENIX-3357
> URL: https://issues.apache.org/jira/browse/PHOENIX-3357
> Project: Phoenix
> Issue Type: Sub-task
>Reporter: Maryann Xue
>Assignee: Maryann Xue
> Labels: calcite
>
> Found another issue here. Right now we re-order our columns so that columns
> from the same family can be grouped together and that column family can be
> taken as a structured type. So OrderByIT, there's a test regression in
> https://builds.apache.org/job/Phoenix-calcite/20/#showFailuresLink, because
> it has a table definition like:
> {code}
> CREATE TABLE t
> (a_string varchar not null, cf1.a integer, cf1.b varchar, col1 integer,
> cf2.c varchar, cf2.d integer, col2 integer
> CONSTRAINT pk PRIMARY KEY (a_string))
> {code}
> So when we implement PhoenixTable.getRowType(), the table definition is
> actually changed from "a_string, cf1.a, cf1.b, col1, cf2.c, cf2.d, col2" into
> "a_string, cf1.a, cf1.b, col1, col2 cf2.c, cf2.d". As a result "UPSERT INTO
> T" expects a different column order (data type order) for parameters, and we
> got an Exception. This happens just because the column re-ordering and has
> nothing to do with the changes for PHOENIX-2679 yet. UPSERT will get more
> complicated once we apply PHOENIX-2679 changes, which means there's more work
> to do for UPSERT for column family support. But regarding this problem alone,
> would you think it would make sense to add a check in Phoenix DDL that
> requires users to put column definitions from the same family together, like
> in this case, only allows "a_string, cf1.a, cf1.b, col1, col2 cf2.c, cf2.d",
> [~jamestaylor]?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3295) Remove ReplaceArrayColumnWithKeyValueColumnExpressionVisitor
[
https://issues.apache.org/jira/browse/PHOENIX-3295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563032#comment-15563032
]
Samarth Jain commented on PHOENIX-3295:
---
bq. Is the column qualifier name of the single key value the family name? Does
that create any issues for Samarth Jain's column encoding scheme and would it
be easier to use a known/static column qualifier name like
QueryConstants.EMPTY_COLUMN_BYTES? More of a question, as I think Samarth can
probably change if need be.
We use QueryConstants.EMPTY_COLUMN_BYTES as the qualifier for our "empty key
value" column. In this case the empty key value column will be part of the
array itself. Using just the column family name here should be fine since we
won't be using the optimization (replacing binary search with O(1) lookup) for
the COLUMNS_STORED_IN_SINGLE_CELL scheme.
> Remove ReplaceArrayColumnWithKeyValueColumnExpressionVisitor
> -
>
> Key: PHOENIX-3295
> URL: https://issues.apache.org/jira/browse/PHOENIX-3295
> Project: Phoenix
> Issue Type: Improvement
>Reporter: Thomas D'Silva
>Assignee: Thomas D'Silva
> Attachments: PHOENIX-3295-v2.patch, PHOENIX-3295.patch
>
>
> ReplaceArrayColumnWithKeyValueColumnExpressionVisitor is only used in one
> place in IndexUtil.generateIndexData because we use a ValueGetter to get the
> value of the data table column using the original data table column
> reference. This is also why ArrayColumnExpression needs to keep track of the
> original key value column expression.
> If we don't replace the array column expression with the original column
> expression when it looks up the column by the qualifier it won't find it.
> {code}
> ValueGetter valueGetter = new ValueGetter() {
>
> @Override
> public byte[] getRowKey() {
> return dataMutation.getRow();
> }
>
> @Override
> public ImmutableBytesWritable
> getLatestValue(ColumnReference ref) {
> // Always return null for our empty key value, as
> this will cause the index
> // maintainer to always treat this Put as a new
> row.
> if (isEmptyKeyValue(table, ref)) {
> return null;
> }
> byte[] family = ref.getFamily();
> byte[] qualifier = ref.getQualifier();
> RowMutationState rowMutationState =
> valuesMap.get(ptr);
> PColumn column = null;
> try {
> column =
> table.getColumnFamily(family).getPColumnForColumnQualifier(qualifier);
> } catch (ColumnNotFoundException e) {
> } catch (ColumnFamilyNotFoundException e) {
> }
> if (rowMutationState!=null && column!=null) {
> byte[] value =
> rowMutationState.getColumnValues().get(column);
> ImmutableBytesPtr ptr = new
> ImmutableBytesPtr();
> ptr.set(value==null ?
> ByteUtil.EMPTY_BYTE_ARRAY : value);
>
> SchemaUtil.padData(table.getName().getString(), column, ptr);
> return ptr;
> }
> return null;
> }
>
> };
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3295) Remove ReplaceArrayColumnWithKeyValueColumnExpressionVisitor
[
https://issues.apache.org/jira/browse/PHOENIX-3295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563006#comment-15563006
]
Thomas D'Silva commented on PHOENIX-3295:
-
Yes the column qualifier is the same as the family name.
getKeyValueExpression() iis used in one place in IndexMaintainer (needed while
adding parent indexes to child views)
Yes we can use ImmutableByteWritable instead of ByteBuffer.
I removed the original key value col expression member variable , but I guess
it can just be initialized based on the index and cf of the ArrayColExpression
> Remove ReplaceArrayColumnWithKeyValueColumnExpressionVisitor
> -
>
> Key: PHOENIX-3295
> URL: https://issues.apache.org/jira/browse/PHOENIX-3295
> Project: Phoenix
> Issue Type: Improvement
>Reporter: Thomas D'Silva
>Assignee: Thomas D'Silva
> Attachments: PHOENIX-3295-v2.patch, PHOENIX-3295.patch
>
>
> ReplaceArrayColumnWithKeyValueColumnExpressionVisitor is only used in one
> place in IndexUtil.generateIndexData because we use a ValueGetter to get the
> value of the data table column using the original data table column
> reference. This is also why ArrayColumnExpression needs to keep track of the
> original key value column expression.
> If we don't replace the array column expression with the original column
> expression when it looks up the column by the qualifier it won't find it.
> {code}
> ValueGetter valueGetter = new ValueGetter() {
>
> @Override
> public byte[] getRowKey() {
> return dataMutation.getRow();
> }
>
> @Override
> public ImmutableBytesWritable
> getLatestValue(ColumnReference ref) {
> // Always return null for our empty key value, as
> this will cause the index
> // maintainer to always treat this Put as a new
> row.
> if (isEmptyKeyValue(table, ref)) {
> return null;
> }
> byte[] family = ref.getFamily();
> byte[] qualifier = ref.getQualifier();
> RowMutationState rowMutationState =
> valuesMap.get(ptr);
> PColumn column = null;
> try {
> column =
> table.getColumnFamily(family).getPColumnForColumnQualifier(qualifier);
> } catch (ColumnNotFoundException e) {
> } catch (ColumnFamilyNotFoundException e) {
> }
> if (rowMutationState!=null && column!=null) {
> byte[] value =
> rowMutationState.getColumnValues().get(column);
> ImmutableBytesPtr ptr = new
> ImmutableBytesPtr();
> ptr.set(value==null ?
> ByteUtil.EMPTY_BYTE_ARRAY : value);
>
> SchemaUtil.padData(table.getName().getString(), column, ptr);
> return ptr;
> }
> return null;
> }
>
> };
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-2675) Allow stats to be configured on a table-by-table basis
[ https://issues.apache.org/jira/browse/PHOENIX-2675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563001#comment-15563001 ] Samarth Jain commented on PHOENIX-2675: --- Ok, will do. > Allow stats to be configured on a table-by-table basis > -- > > Key: PHOENIX-2675 > URL: https://issues.apache.org/jira/browse/PHOENIX-2675 > Project: Phoenix > Issue Type: Improvement >Reporter: James Taylor >Assignee: James Taylor > Fix For: 4.9.0 > > Attachments: PHOENIX-2675.patch, PHOENIX-2675_wip.patch, > PHOENIX-2675_wip2.patch > > > Currently stats are controlled and collected at a global level. We should > allow them to be configured on a table-by-table basis. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3355) Register Phoenix built-in functions as Calcite functions
[ https://issues.apache.org/jira/browse/PHOENIX-3355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15562997#comment-15562997 ] Maryann Xue commented on PHOENIX-3355: -- Yes, DDLs have a completely execution path from other statements (queries and DMLs). DDLs only go through Calcite's parser (not even validation) and then get translated into Phoenix AST and executed directly in Phoenix, whereas other statements go through Calcite's parser, validator, query planner, and then be translated into Phoenix QueryPlans, which is also when functions and other expressions are translated into Phoenix classes. So you could imagine, for a built-in function in DDL, e.g. TO_DATE(), it will be parsed by Calcite parser, but will not be validated by Calcite, which means it doesn't matter whether Calcite knows of this function signature. And even afterwards when it comes to the execution of the CREATE VIEW statement in Phoenix, the TO_DATE() function will only exist in its ParseNode form and won't be executed anyway. However I don't believe we can get away with it when we try to reference this VIEW in a query, coz that's when Calcite will try converting the view definition into a Rel. Another thing is that if you look at CalciteUtils' translation of other functions, there are a few built-in functions like SQRT, ABS, etc, that are support by both Calcite and Phoenix, and we simply translate them the same way as we do for other expressions. But in case we have functions with a different signature than Calcite, the UDF approach would work instead. > Register Phoenix built-in functions as Calcite functions > > > Key: PHOENIX-3355 > URL: https://issues.apache.org/jira/browse/PHOENIX-3355 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: Eric Lomore > Labels: calcite > > We should register all Phoenix built-in functions that don't exist in Calcite. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2675) Allow stats to be configured on a table-by-table basis
[ https://issues.apache.org/jira/browse/PHOENIX-2675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15562999#comment-15562999 ] James Taylor commented on PHOENIX-2675: --- Can you just review from the patch on PHOENIX-3361? It's this one plus that one together. They're kind of linked together. > Allow stats to be configured on a table-by-table basis > -- > > Key: PHOENIX-2675 > URL: https://issues.apache.org/jira/browse/PHOENIX-2675 > Project: Phoenix > Issue Type: Improvement >Reporter: James Taylor >Assignee: James Taylor > Fix For: 4.9.0 > > Attachments: PHOENIX-2675.patch, PHOENIX-2675_wip.patch, > PHOENIX-2675_wip2.patch > > > Currently stats are controlled and collected at a global level. We should > allow them to be configured on a table-by-table basis. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2675) Allow stats to be configured on a table-by-table basis
[
https://issues.apache.org/jira/browse/PHOENIX-2675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15562986#comment-15562986
]
Samarth Jain commented on PHOENIX-2675:
---
[~jamestaylor] - it looks like your latest patch has commits from other JIRAs
too. Would you mind attaching a patch with just this JIRA's changes as it would
make it easier for me to review. Also, it looks like there is some commented
code included in this patch. For ex :
{code}
+//TestUtil.analyzeTable(conn, fullTableName);
+//List guideposts = TestUtil.getAllSplits(conn,
fullTableName);
+//assertEquals(1, guideposts.size());
+//assertEquals(KeyRange.EVERYTHING_RANGE, guideposts.get(0));
+//
+//conn.createStatement().execute("ALTER TABLE " + fullTableName + "
SET GUIDE_POST_WIDTH=20");
+//
+//TestUtil.analyzeTable(conn, fullTableName);
+//guideposts = TestUtil.getAllSplits(conn, fullTableName);
+//assertEquals(5, guideposts.size());
+//
+//// Confirm that when view index used, the GUIDE_POST_WIDTH from the
data physical table
+//// was used
+//sql = "SELECT * FROM " + viewName + " WHERE v2 > 100";
+//stmt = conn1.prepareStatement(sql);
+//stmt.executeQuery();
+//QueryPlan plan = stmt.unwrap(PhoenixStatement.class).getQueryPlan();
+//assertEquals(5, plan.getSplits().size());
{code}
> Allow stats to be configured on a table-by-table basis
> --
>
> Key: PHOENIX-2675
> URL: https://issues.apache.org/jira/browse/PHOENIX-2675
> Project: Phoenix
> Issue Type: Improvement
>Reporter: James Taylor
>Assignee: James Taylor
> Fix For: 4.9.0
>
> Attachments: PHOENIX-2675.patch, PHOENIX-2675_wip.patch,
> PHOENIX-2675_wip2.patch
>
>
> Currently stats are controlled and collected at a global level. We should
> allow them to be configured on a table-by-table basis.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-2935) IndexMetaData cache can expire when a delete and or query running on server
[ https://issues.apache.org/jira/browse/PHOENIX-2935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15562958#comment-15562958 ] James Taylor commented on PHOENIX-2935: --- +1. I like this approach. Nice find, [~an...@apache.org]. > IndexMetaData cache can expire when a delete and or query running on server > --- > > Key: PHOENIX-2935 > URL: https://issues.apache.org/jira/browse/PHOENIX-2935 > Project: Phoenix > Issue Type: Bug >Reporter: Ankit Singhal >Assignee: Ankit Singhal > Labels: index > Fix For: 4.9.0 > > Attachments: PHOENIX-2935.patch > > > IndexMetaData cache can expire when a delete or upsert query is running on > server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3289) Region servers crashing randomly with "Native memory allocation (malloc) failed to allocate 12288 bytes for committing reserved memory"
[ https://issues.apache.org/jira/browse/PHOENIX-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15562946#comment-15562946 ] James Taylor commented on PHOENIX-3289: --- bq. But now the problems is that we are keeping page size of 4KB only for each mmap Agreed - we fixed a similar issue for ORDER BY by increasing this. I think it's just an oversight that the value is configured so low. Let's definitely increase it. Also, there has been some work by [~maryannxue] and one of the GSoC students to try replacing Memory Mapped file usage with just regular files (the thinking being that the OS does a good job of caching these already anyway). See PHOENIX-2405. Unfortunately, this work never made it far enough to verify the hypothesis and get into the code base. > Region servers crashing randomly with "Native memory allocation (malloc) > failed to allocate 12288 bytes for committing reserved memory" > --- > > Key: PHOENIX-3289 > URL: https://issues.apache.org/jira/browse/PHOENIX-3289 > Project: Phoenix > Issue Type: Bug >Reporter: Ankit Singhal >Assignee: Ankit Singhal > > for GROUP BY case, we try to keep the data in memory but if we exceed the > total memory given for global cache(phoenix.query.maxGlobalMemorySize), then > we start spilling the data to the disk. We spill and map them with > MappedByteBuffer and add bloom filter so that they can be accessed faster. > MappedByteBuffer doesn't release memory immediately on close(), GC needs to > collect them when it find such buffers with no references. Though, we are > closing the channel and file properly and deleting the file at the end, it's > possible that GCs has not run for a while and this map count is increasing. > PFB, parent ticket talking about the memory leak with the FileChannel.map() > function. > http://bugs.java.com/view_bug.do?bug_id=4724038 > And ,related ticket > http://bugs.java.com/view_bug.do?bug_id=6558368 > But now the problems is that we are keeping page size of 4KB only for each > mmap, which seems to be very small. I don't know the rationale behind keeping > it so low, may be for performance to get a single key all the time but it can > result in multiple mmap for even small file , so we can thought of keeping it > as 64KB page size or more as default but we need to see the impact on > performance. > Workaround to mask the issue, increasing the > phoenix.query.maxGlobalMemorySize and increasing vm.max_map_count is a > solution for get going > thanks [~rmaruthiyodan] for reducing the territory of the problem by > analyzing VM hostspot error file and detecting the increase in mmap count > during GroupBy queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-3366) Local index queries fail with many guideposts if split occurs
[ https://issues.apache.org/jira/browse/PHOENIX-3366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated PHOENIX-3366: -- Fix Version/s: 4.9.0 > Local index queries fail with many guideposts if split occurs > - > > Key: PHOENIX-3366 > URL: https://issues.apache.org/jira/browse/PHOENIX-3366 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: Rajeshbabu Chintaguntla > Fix For: 4.9.0 > > > In getting guideposts to be correctly built for local indexes (PHOENIX-3361), > I accidentally configured many guideposts in the > IndexExtendedIT.testLocalIndexScanAfterRegionSplit() test and the test > started failing. We should confirm there's not a general issue lurking here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (PHOENIX-3366) Local index queries fail with many guideposts if split occurs
James Taylor created PHOENIX-3366: - Summary: Local index queries fail with many guideposts if split occurs Key: PHOENIX-3366 URL: https://issues.apache.org/jira/browse/PHOENIX-3366 Project: Phoenix Issue Type: Bug Reporter: James Taylor Assignee: Rajeshbabu Chintaguntla In getting guideposts to be correctly built for local indexes (PHOENIX-3361), I accidentally configured many guideposts in the IndexExtendedIT.testLocalIndexScanAfterRegionSplit() test and the test started failing. We should confirm there's not a general issue lurking here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3361) Collect stats correctly for local indexes
[ https://issues.apache.org/jira/browse/PHOENIX-3361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15562869#comment-15562869 ] James Taylor commented on PHOENIX-3361: --- [~samarthjain] or [~tdsilva]? Please review - I'd like to check this in today. > Collect stats correctly for local indexes > - > > Key: PHOENIX-3361 > URL: https://issues.apache.org/jira/browse/PHOENIX-3361 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: James Taylor > Fix For: 4.9.0 > > Attachments: PHOENIX-3361.patch, PHOENIX-3361_wip.patch, > PHOENIX-3361_wip2.patch > > > Stats are not being correctly collected for local indexes and tables with > local indexes, because the logic that deletes the old stats is based on all > store families of a table. Thus, when stats are collected for a table with > local indexes, the local index stats would be deleted and when the stats for > a local index are collected, the stats for the table will be deleted. > Instead, if we cache stats per column family instead of across entire table, > plus detect if we're collecting stats for a local index versus the data > table, we can fix this issue. Caching stats per column family makes more > sense anyway, because depending on the query, we may only use the stats > associated with the default column family. Also, we wouldn't want to have to > pull over stats for local index with the data table stats. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2675) Allow stats to be configured on a table-by-table basis
[ https://issues.apache.org/jira/browse/PHOENIX-2675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15562867#comment-15562867 ] James Taylor commented on PHOENIX-2675: --- [~samarthjain] or [~tdsilva]? Please review - I'd like to check this in today. > Allow stats to be configured on a table-by-table basis > -- > > Key: PHOENIX-2675 > URL: https://issues.apache.org/jira/browse/PHOENIX-2675 > Project: Phoenix > Issue Type: Improvement >Reporter: James Taylor >Assignee: James Taylor > Fix For: 4.9.0 > > Attachments: PHOENIX-2675.patch, PHOENIX-2675_wip.patch, > PHOENIX-2675_wip2.patch > > > Currently stats are controlled and collected at a global level. We should > allow them to be configured on a table-by-table basis. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-476) Support declaration of DEFAULT in CREATE statement
[
https://issues.apache.org/jira/browse/PHOENIX-476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15562842#comment-15562842
]
James Taylor commented on PHOENIX-476:
--
Thanks, [~kliew]. Looking very good. Couple of comments:
- Why do you say that without caching expression we can only compile literals?
The SQLParser.parseExpression() call will happily parse any expression and the
ExpressionCompiler will translate from the ParseNode to an Expression. The
caching of expressions was just an optimization, not a requirement. Also, we
don't parse negative literals because it leads to problems with certain
expressions. For example, QueryParserTest.testSubtractionInSelect doesn't pass
with your patch applied.
- Minor, but I think the change to this file can be reverted:
{code}
diff --git
a/phoenix-core/src/main/java/org/apache/phoenix/schema/MetaDataClient.java
b/phoenix-core/src/main/java/org/apache/phoenix/schema/MetaDataClient.java
index 1e635d8..f617f92 100644
--- a/phoenix-core/src/main/java/org/apache/phoenix/schema/MetaDataClient.java
+++ b/phoenix-core/src/main/java/org/apache/phoenix/schema/MetaDataClient.java
@@ -431,7 +431,7 @@ public class MetaDataClient {
/**
* Update the cache with the latest as of the connection scn.
- * @param functioNames
+ * @param functionNames
* @return the timestamp from the server, negative if the function was
added to the cache and positive otherwise
* @throws SQLException
*/
{code}
If you can undo the changes for the parser for negative literals, use
SQLParser.parseExpression() instead of parseLiteral(), add a few of your test
back that use expressions instead of literals, and ensure that all unit tests
pass (i.e. run {{mvn verify}}), then I think we'll be good to go.
> Support declaration of DEFAULT in CREATE statement
> --
>
> Key: PHOENIX-476
> URL: https://issues.apache.org/jira/browse/PHOENIX-476
> Project: Phoenix
> Issue Type: Task
>Affects Versions: 3.0-Release
>Reporter: James Taylor
>Assignee: Kevin Liew
> Labels: enhancement
> Attachments: PHOENIX-476.2.patch, PHOENIX-476.3.patch,
> PHOENIX-476.4.patch, PHOENIX-476.5.patch, PHOENIX-476.6.patch,
> PHOENIX-476.patch
>
>
> Support the declaration of a default value in the CREATE TABLE/VIEW statement
> like this:
> CREATE TABLE Persons (
> Pid int NOT NULL PRIMARY KEY,
> LastName varchar(255) NOT NULL,
> FirstName varchar(255),
> Address varchar(255),
> City varchar(255) DEFAULT 'Sandnes'
> )
> To implement this, we'd need to:
> 1. add a new DEFAULT_VALUE key value column in SYSTEM.TABLE and pass through
> the value when the table is created (in MetaDataClient).
> 2. always set NULLABLE to ResultSetMetaData.columnNoNulls if a default value
> is present, since the column will never be null.
> 3. add a getDefaultValue() accessor in PColumn
> 4. for a row key column, during UPSERT use the default value if no value was
> specified for that column. This could be done in the PTableImpl.newKey method.
> 5. for a key value column with a default value, we can get away without
> incurring any storage cost. Although a little bit of extra effort than if we
> persisted the default value on an UPSERT for key value columns, this approach
> has the benefit of not incurring any storage cost for a default value.
> * serialize any default value into KeyValueColumnExpression
> * in the evaluate method of KeyValueColumnExpression, conditionally use
> the default value if the column value is not present. If doing partial
> evaluation, you should not yet return the default value, as we may not have
> encountered the the KeyValue for the column yet (since a filter evaluates
> each time it sees each KeyValue, and there may be more than one KeyValue
> referenced in the expression). Partial evaluation is determined by calling
> Tuple.isImmutable(), where false means it is NOT doing partial evaluation,
> while true means it is.
> * modify EvaluateOnCompletionVisitor by adding a visitor method for
> RowKeyColumnExpression and KeyValueColumnExpression to set
> evaluateOnCompletion to true if they have a default value specified. This
> will cause filter evaluation to execute one final time after all KeyValues
> for a row have been seen, since it's at this time we know we should use the
> default value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (PHOENIX-3355) Register Phoenix built-in functions as Calcite functions
[
https://issues.apache.org/jira/browse/PHOENIX-3355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15562837#comment-15562837
]
Eric Lomore edited comment on PHOENIX-3355 at 10/10/16 5:09 PM:
Thanks [~maryannxue], the UDF approach makes sense.
{{"CREATE VIEW v as SELECT * FROM t WHERE k1=TO_DATE('2016-06-06')";}} executes
successfully
was (Author: lomoree):
{{"CREATE VIEW v as SELECT * FROM t WHERE k1=TO_DATE('2016-06-06')";}} executes
successfully
> Register Phoenix built-in functions as Calcite functions
>
>
> Key: PHOENIX-3355
> URL: https://issues.apache.org/jira/browse/PHOENIX-3355
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Eric Lomore
> Labels: calcite
>
> We should register all Phoenix built-in functions that don't exist in Calcite.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3355) Register Phoenix built-in functions as Calcite functions
[
https://issues.apache.org/jira/browse/PHOENIX-3355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15562837#comment-15562837
]
Eric Lomore commented on PHOENIX-3355:
--
{{"CREATE VIEW v as SELECT * FROM t WHERE k1=TO_DATE('2016-06-06')";}} executes
successfully
> Register Phoenix built-in functions as Calcite functions
>
>
> Key: PHOENIX-3355
> URL: https://issues.apache.org/jira/browse/PHOENIX-3355
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Eric Lomore
> Labels: calcite
>
> We should register all Phoenix built-in functions that don't exist in Calcite.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3355) Register Phoenix built-in functions as Calcite functions
[ https://issues.apache.org/jira/browse/PHOENIX-3355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15562823#comment-15562823 ] Maryann Xue commented on PHOENIX-3355: -- First of all, I don't think we need two sets of classes for these Phoenix built-in functions if we just register them as Calcite UDFs. So we need to wait for [~rajeshbabu]'s check-in on PHOENIX-3242. And [~rajeshbabu], I have approved your patch for UDF part. What you have right now is good enough to get [~lomoree] started on this issue. Would you mind checking in ASAP? You can continue to work on the default parameter support and other issues related to UserDefinedFunctionIT. And [~lomoree], once the UDF support is in, all we need for this issue is to add our built-in functions in implementation of PhoenixSchema.getFunctions(String name). Also, can you give an example of DDL statement you talked about, [~lomoree]? > Register Phoenix built-in functions as Calcite functions > > > Key: PHOENIX-3355 > URL: https://issues.apache.org/jira/browse/PHOENIX-3355 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: Eric Lomore > Labels: calcite > > We should register all Phoenix built-in functions that don't exist in Calcite. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3355) Register Phoenix built-in functions as Calcite functions
[ https://issues.apache.org/jira/browse/PHOENIX-3355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15562765#comment-15562765 ] James Taylor commented on PHOENIX-3355: --- In general, the execution will be handled by Phoenix, but the parsing, validation, compilation, and planning should be done by Calcite (and the Phoenix/Calcite integration code). For built-in functions, we'll want to stick exclusively to using Phoenix built-in functions as otherwise when their execution is pushed to the server (namely when a function is used in a WHERE clause), the Calcite-based built-in functions wouldn't work as there's no mechanism to generate code on the server side. For DDL statements, can you give an example of one that's not being validated? Is it when you execute a CREATE VIEW statement? It may be that the validation occurs in Calcite when the view statement is used rather than when it's created. [~maryannxue] or [~julianhyde] would know. > Register Phoenix built-in functions as Calcite functions > > > Key: PHOENIX-3355 > URL: https://issues.apache.org/jira/browse/PHOENIX-3355 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: Eric Lomore > Labels: calcite > > We should register all Phoenix built-in functions that don't exist in Calcite. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3355) Register Phoenix built-in functions as Calcite functions
[
https://issues.apache.org/jira/browse/PHOENIX-3355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15562592#comment-15562592
]
Eric Lomore commented on PHOENIX-3355:
--
Question for [~maryannxue] and [~jamestaylor],
Currently, DDL statements leverage the phoenix runtime, and hence use phoenix
registered builtin functions. Is there ever a plan to fully move away from
phoenix execution?
If we don't then it appears we need each function to be implemented in Calcite
and Phoenix. In the DDL case, using the phoenix runtime, and for all other
queries, calcite runtime.
Therefore, are we looking to make each builtin function registered once but
usable by both, or, duplicate the functions have one implementation in Calcite,
and another in Phoenix?
Making these functions dual compatible may be difficult to the vastly different
structures they implement.
Ex//
{{Phoenix : CurrentDateFunction.java}}
{{Calcite : SqlCurrentDateFunction.java}} -- extends SqlFunction
> Register Phoenix built-in functions as Calcite functions
>
>
> Key: PHOENIX-3355
> URL: https://issues.apache.org/jira/browse/PHOENIX-3355
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Eric Lomore
> Labels: calcite
>
> We should register all Phoenix built-in functions that don't exist in Calcite.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (PHOENIX-3328) Skip scan optimization failed for multi pk columns
[
https://issues.apache.org/jira/browse/PHOENIX-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Taylor updated PHOENIX-3328:
--
Assignee: William Yang
> Skip scan optimization failed for multi pk columns
> --
>
> Key: PHOENIX-3328
> URL: https://issues.apache.org/jira/browse/PHOENIX-3328
> Project: Phoenix
> Issue Type: Bug
>Reporter: William Yang
>Assignee: William Yang
>Priority: Minor
> Fix For: 4.9.0, 4.8.2
>
> Attachments: PHOENIX-3328.patch
>
>
> {code:sql}
> create table t1 (pk integer primary key, a integer);
> create table t2 (pk1 integer not null, pk2 integer not null, a integer
> constraint pk primary key (pk1, pk2));
> explain select * from t1 where (pk > 10 and pk < 20) or (pk > 30 and pk < 40);
> explain select * from t2 where (pk1 > 10 and pk1 < 20) or (pk1 > 30 and pk1 <
> 40);
> {code}
> The first SELECT statement will use skip scan filter for two ranges, so as
> the second one. But actually the WhereOptimizer failed doing so and using a
> full table scan instead. This happens for tables have multi PK columns.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (PHOENIX-3328) Skip scan optimization failed for multi pk columns
[
https://issues.apache.org/jira/browse/PHOENIX-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Taylor updated PHOENIX-3328:
--
Fix Version/s: 4.8.2
4.9.0
> Skip scan optimization failed for multi pk columns
> --
>
> Key: PHOENIX-3328
> URL: https://issues.apache.org/jira/browse/PHOENIX-3328
> Project: Phoenix
> Issue Type: Bug
>Reporter: William Yang
>Priority: Minor
> Fix For: 4.9.0, 4.8.2
>
> Attachments: PHOENIX-3328.patch
>
>
> {code:sql}
> create table t1 (pk integer primary key, a integer);
> create table t2 (pk1 integer not null, pk2 integer not null, a integer
> constraint pk primary key (pk1, pk2));
> explain select * from t1 where (pk > 10 and pk < 20) or (pk > 30 and pk < 40);
> explain select * from t2 where (pk1 > 10 and pk1 < 20) or (pk1 > 30 and pk1 <
> 40);
> {code}
> The first SELECT statement will use skip scan filter for two ranges, so as
> the second one. But actually the WhereOptimizer failed doing so and using a
> full table scan instead. This happens for tables have multi PK columns.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (PHOENIX-3289) Region servers crashing randomly with "Native memory allocation (malloc) failed to allocate 12288 bytes for committing reserved memory"
[ https://issues.apache.org/jira/browse/PHOENIX-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Singhal reassigned PHOENIX-3289: -- Assignee: Ankit Singhal > Region servers crashing randomly with "Native memory allocation (malloc) > failed to allocate 12288 bytes for committing reserved memory" > --- > > Key: PHOENIX-3289 > URL: https://issues.apache.org/jira/browse/PHOENIX-3289 > Project: Phoenix > Issue Type: Bug >Reporter: Ankit Singhal >Assignee: Ankit Singhal > > for GROUP BY case, we try to keep the data in memory but if we exceed the > total memory given for global cache(phoenix.query.maxGlobalMemorySize), then > we start spilling the data to the disk. We spill and map them with > MappedByteBuffer and add bloom filter so that they can be accessed faster. > MappedByteBuffer doesn't release memory immediately on close(), GC needs to > collect them when it find such buffers with no references. Though, we are > closing the channel and file properly and deleting the file at the end, it's > possible that GCs has not run for a while and this map count is increasing. > PFB, parent ticket talking about the memory leak with the FileChannel.map() > function. > http://bugs.java.com/view_bug.do?bug_id=4724038 > And ,related ticket > http://bugs.java.com/view_bug.do?bug_id=6558368 > But now the problems is that we are keeping page size of 4KB only for each > mmap, which seems to be very small. I don't know the rationale behind keeping > it so low, may be for performance to get a single key all the time but it can > result in multiple mmap for even small file , so we can thought of keeping it > as 64KB page size or more as default but we need to see the impact on > performance. > Workaround to mask the issue, increasing the > phoenix.query.maxGlobalMemorySize and increasing vm.max_map_count is a > solution for get going > thanks [~rmaruthiyodan] for reducing the territory of the problem by > analyzing VM hostspot error file and detecting the increase in mmap count > during GroupBy queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2935) IndexMetaData cache can expire when a delete and or query running on server
[
https://issues.apache.org/jira/browse/PHOENIX-2935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15562204#comment-15562204
]
Hadoop QA commented on PHOENIX-2935:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12832459/PHOENIX-2935.patch
against master branch at commit ad99952792b8b119085499d8d0d734c3d2dc053b.
ATTACHMENT ID: 12832459
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
38 warning messages.
{color:red}-1 release audit{color}. The applied patch generated 1 release
audit warnings (more than the master's current 0 warnings).
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+
context.getScan().setAttribute(PhoenixIndexCodec.INDEX_MD, ptr.get());
+
context.getScan().setAttribute(BaseScannerRegionObserver.TX_STATE, txState);
+final long mutationCount =
(Long)aggProjector.getColumnProjector(0).getValue(row,
+long blockingMemstoreSize, byte[] indexMaintainersPtr, byte[]
txState) throws IOException {
+commitBatch(region, mutations, indexUUID,
blockingMemStoreSize, indexMaintainersPtr, txState);
+commitBatch(region, indexMutations, null,
blockingMemStoreSize, null, txState);
+commitBatch(region,mutations, indexUUID,
blockingMemStoreSize, indexMaintainersPtr, txState);
+String msg = "key=" + ServerCacheClient.idToString(uuid) + "
region=" + env.getRegion() + "host="
+SQLException e = new
SQLExceptionInfo.Builder(SQLExceptionCode.INDEX_METADATA_NOT_FOUND).setMessage(msg)
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/620//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/620//artifact/patchprocess/patchReleaseAuditWarnings.txt
Javadoc warnings:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/620//artifact/patchprocess/patchJavadocWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/620//console
This message is automatically generated.
> IndexMetaData cache can expire when a delete and or query running on server
> ---
>
> Key: PHOENIX-2935
> URL: https://issues.apache.org/jira/browse/PHOENIX-2935
> Project: Phoenix
> Issue Type: Bug
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
> Labels: index
> Fix For: 4.9.0
>
> Attachments: PHOENIX-2935.patch
>
>
> IndexMetaData cache can expire when a delete or upsert query is running on
> server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-2935) IndexMetaData cache can expire when a delete and or query running on server
[ https://issues.apache.org/jira/browse/PHOENIX-2935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15562050#comment-15562050 ] Ankit Singhal commented on PHOENIX-2935: [~jamestaylor], in case of DELETE on server or UPSERT SELECT on server, how about sending indexMetaDataPtr in scan attribute(PFA, for the same) . Problem could be a size of scan object become little large(by 2KB+ with 15-30 indexes) and RPC could be little slow which would be aggravated with large number of guidePosts or region in range for the query but not sure if this is huge side effect against failing the query on cache expiry. > IndexMetaData cache can expire when a delete and or query running on server > --- > > Key: PHOENIX-2935 > URL: https://issues.apache.org/jira/browse/PHOENIX-2935 > Project: Phoenix > Issue Type: Bug >Reporter: Ankit Singhal > Attachments: PHOENIX-2935.patch > > > IndexMetaData cache can expire when a delete or upsert query is running on > server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-2935) IndexMetaData cache can expire when a delete and or query running on server
[ https://issues.apache.org/jira/browse/PHOENIX-2935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Singhal updated PHOENIX-2935: --- Attachment: PHOENIX-2935.patch > IndexMetaData cache can expire when a delete and or query running on server > --- > > Key: PHOENIX-2935 > URL: https://issues.apache.org/jira/browse/PHOENIX-2935 > Project: Phoenix > Issue Type: Bug >Reporter: Ankit Singhal > Attachments: PHOENIX-2935.patch > > > IndexMetaData cache can expire when a delete or upsert query is running on > server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (PHOENIX-3049) saving dataframe when use Spark cluster model throw NoClassDefFoundError: co/cask/tephra/TxConstants
[
https://issues.apache.org/jira/browse/PHOENIX-3049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lichenglin resolved PHOENIX-3049.
-
Resolution: Won't Fix
> saving dataframe when use Spark cluster model throw NoClassDefFoundError:
> co/cask/tephra/TxConstants
> -
>
> Key: PHOENIX-3049
> URL: https://issues.apache.org/jira/browse/PHOENIX-3049
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.7.0
>Reporter: lichenglin
>
> I'm using spark1.6.2, phoenix 4.7.0,hbase1.1.2
> When I trying to save a dataframe to phoenix in cluster mode.
> It throws NoClassDefFoundError: co/cask/tephra/TxConstants
> but local model work well.
> I'm sure that I have add phoenix-4.7.0-HBase-1.1-client-spark.jar to the
> worker classpath.
> First I think it's caused by tephra's verion conflict.
> Then I delete all the tephra class in
> phoenix-4.7.0-HBase-1.1-client-spark.jar.
> But the exception turns into classnotfound.
> So It is not jar version conflict.
> It troubles me all the day.
> Here is the log
> {code}
> Exception in thread "main" org.apache.spark.SparkException: Job aborted due
> to stage failure: Task 9 in stage 0.0 failed 4 times, most recent failure:
> Lost task 9.3 in stage 0.0 (TID 13, namenode): java.lang.RuntimeException:
> Exception while committing to database.
> at
> org.apache.phoenix.mapreduce.PhoenixRecordWriter.write(PhoenixRecordWriter.java:85)
> at
> org.apache.phoenix.mapreduce.PhoenixRecordWriter.write(PhoenixRecordWriter.java:39)
> at
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply$mcV$sp(PairRDDFunctions.scala:1113)
> at
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:)
> at
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:)
> at
> org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1277)
> at
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1119)
> at
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1091)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
> at org.apache.spark.scheduler.Task.run(Task.scala:89)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.phoenix.execute.CommitException:
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed
> 1994 actions:
> org.apache.phoenix.hbase.index.builder.IndexBuildingFailureException: Failed
> to build index for unexpected reason!
> at
> org.apache.phoenix.hbase.index.util.IndexManagementUtil.rethrowIndexingException(IndexManagementUtil.java:180)
> at
> org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:205)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$35.call(RegionCoprocessorHost.java:991)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1673)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1748)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1705)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:987)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3027)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2801)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2743)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:692)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:654)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2031)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
> at
>
[jira] [Created] (PHOENIX-3365) Surport spark 2.x.y
lichenglin created PHOENIX-3365:
---
Summary: Surport spark 2.x.y
Key: PHOENIX-3365
URL: https://issues.apache.org/jira/browse/PHOENIX-3365
Project: Phoenix
Issue Type: New Feature
Affects Versions: 4.8.0
Reporter: lichenglin
Spark change a lot in 2.x.y.
support like this
{code}
CREATE TEMPORARY VIEW phoenixTable
USING org.apache.phoenix.spark
OPTIONS (
. table, "TABLE1"
zkUrl, "localhost:2181"
)
SELECT * FROM phoenixTable
{code}
The sql above should be worked
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (PHOENIX-476) Support declaration of DEFAULT in CREATE statement
[ https://issues.apache.org/jira/browse/PHOENIX-476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Liew updated PHOENIX-476: --- Attachment: PHOENIX-476.6.patch I attached another revision with bugfixes and test coverage. Without caching the expression, we can only compile literals, not arbitrary (stateless) expressions. So I made a change to the parser grammar so that `expression` would parse negative values as literals rather than expressions. ie. `-100` instead of `(100 * -1)`. Arrays are a WIP due to issues with coercion in DefaultValueExpression. > Support declaration of DEFAULT in CREATE statement > -- > > Key: PHOENIX-476 > URL: https://issues.apache.org/jira/browse/PHOENIX-476 > Project: Phoenix > Issue Type: Task >Affects Versions: 3.0-Release >Reporter: James Taylor >Assignee: Kevin Liew > Labels: enhancement > Attachments: PHOENIX-476.2.patch, PHOENIX-476.3.patch, > PHOENIX-476.4.patch, PHOENIX-476.5.patch, PHOENIX-476.6.patch, > PHOENIX-476.patch > > > Support the declaration of a default value in the CREATE TABLE/VIEW statement > like this: > CREATE TABLE Persons ( > Pid int NOT NULL PRIMARY KEY, > LastName varchar(255) NOT NULL, > FirstName varchar(255), > Address varchar(255), > City varchar(255) DEFAULT 'Sandnes' > ) > To implement this, we'd need to: > 1. add a new DEFAULT_VALUE key value column in SYSTEM.TABLE and pass through > the value when the table is created (in MetaDataClient). > 2. always set NULLABLE to ResultSetMetaData.columnNoNulls if a default value > is present, since the column will never be null. > 3. add a getDefaultValue() accessor in PColumn > 4. for a row key column, during UPSERT use the default value if no value was > specified for that column. This could be done in the PTableImpl.newKey method. > 5. for a key value column with a default value, we can get away without > incurring any storage cost. Although a little bit of extra effort than if we > persisted the default value on an UPSERT for key value columns, this approach > has the benefit of not incurring any storage cost for a default value. > * serialize any default value into KeyValueColumnExpression > * in the evaluate method of KeyValueColumnExpression, conditionally use > the default value if the column value is not present. If doing partial > evaluation, you should not yet return the default value, as we may not have > encountered the the KeyValue for the column yet (since a filter evaluates > each time it sees each KeyValue, and there may be more than one KeyValue > referenced in the expression). Partial evaluation is determined by calling > Tuple.isImmutable(), where false means it is NOT doing partial evaluation, > while true means it is. > * modify EvaluateOnCompletionVisitor by adding a visitor method for > RowKeyColumnExpression and KeyValueColumnExpression to set > evaluateOnCompletion to true if they have a default value specified. This > will cause filter evaluation to execute one final time after all KeyValues > for a row have been seen, since it's at this time we know we should use the > default value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
