[jira] [Updated] (PHOENIX-2160) Projection of specific array index does not work
[ https://issues.apache.org/jira/browse/PHOENIX-2160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated PHOENIX-2160: Fix Version/s: 4.5.0 4.4.0 > Projection of specific array index does not work > > > Key: PHOENIX-2160 > URL: https://issues.apache.org/jira/browse/PHOENIX-2160 > Project: Phoenix > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Assignee: Dumindu Buddhika > Fix For: 4.4.0, 4.5.0 > > Attachments: PHOENIX-2160.patch, PHOENIX-2160_4.5_4.x.patch, > PHOENIX-2160_4.5_4.x.patch, PHOENIX-2160_v2.patch, PHOENIX-2160_v3.patch, > PHOENIX-2160_v4.patch, PHOENIX-2160_v5.patch, PHOENIX-2160_v6.patch, > PHOENIX-2160_v7.patch > > > PHOENIX-10 that allowed projection of specific array index does not work now. > Was looking into the code for some thing and found this issue. Let me know if > am missing something. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2160) Projection of specific array index does not work
[ https://issues.apache.org/jira/browse/PHOENIX-2160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14730353#comment-14730353 ] ramkrishna.s.vasudevan commented on PHOENIX-2160: - Pushed to 4.5 and 4.x branch. > Projection of specific array index does not work > > > Key: PHOENIX-2160 > URL: https://issues.apache.org/jira/browse/PHOENIX-2160 > Project: Phoenix > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Assignee: Dumindu Buddhika > Fix For: 4.4.0, 4.5.0 > > Attachments: PHOENIX-2160.patch, PHOENIX-2160_4.5_4.x.patch, > PHOENIX-2160_4.5_4.x.patch, PHOENIX-2160_v2.patch, PHOENIX-2160_v3.patch, > PHOENIX-2160_v4.patch, PHOENIX-2160_v5.patch, PHOENIX-2160_v6.patch, > PHOENIX-2160_v7.patch > > > PHOENIX-10 that allowed projection of specific array index does not work now. > Was looking into the code for some thing and found this issue. Let me know if > am missing something. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2231) Support CREATE/DROP SEQUENCE in Phoenix/Calcite Integration
[ https://issues.apache.org/jira/browse/PHOENIX-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14730281#comment-14730281 ] Maryann Xue commented on PHOENIX-2231: -- [~maghamraviki...@gmail.com] Would you be interested in taking this one? The skeleton of CREATE VIEW is already there, the related files are mainly: phoenix-core/src/main/codegen/data/Parser.tdd phoenix-core/src/main/codegen/includes/parserImpls.ftl phoenix-core/src/main/java/org/apache/phoenix/calcite/jdbc/PhoenixPrepareImpl.java phoenix-core/src/main/java/org/apache/phoenix/calcite/parse/*.java > Support CREATE/DROP SEQUENCE in Phoenix/Calcite Integration > --- > > Key: PHOENIX-2231 > URL: https://issues.apache.org/jira/browse/PHOENIX-2231 > Project: Phoenix > Issue Type: Sub-task >Reporter: Maryann Xue > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (PHOENIX-2230) Support CREATE/DROP INDEX in Phoenix-Calcite Integration
Maryann Xue created PHOENIX-2230: Summary: Support CREATE/DROP INDEX in Phoenix-Calcite Integration Key: PHOENIX-2230 URL: https://issues.apache.org/jira/browse/PHOENIX-2230 Project: Phoenix Issue Type: Sub-task Reporter: Maryann Xue -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (PHOENIX-2231) Support CREATE/DROP SEQUENCE in Phoenix/Calcite Integration
Maryann Xue created PHOENIX-2231: Summary: Support CREATE/DROP SEQUENCE in Phoenix/Calcite Integration Key: PHOENIX-2231 URL: https://issues.apache.org/jira/browse/PHOENIX-2231 Project: Phoenix Issue Type: Sub-task Reporter: Maryann Xue -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (PHOENIX-2229) Support CREATE/DROP VIEW in Phoenix-Calcite Integration
Maryann Xue created PHOENIX-2229: Summary: Support CREATE/DROP VIEW in Phoenix-Calcite Integration Key: PHOENIX-2229 URL: https://issues.apache.org/jira/browse/PHOENIX-2229 Project: Phoenix Issue Type: Sub-task Reporter: Maryann Xue -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (PHOENIX-2228) Support CREATE/DROP TABLE in Phoenix-Calcite Integration
Maryann Xue created PHOENIX-2228: Summary: Support CREATE/DROP TABLE in Phoenix-Calcite Integration Key: PHOENIX-2228 URL: https://issues.apache.org/jira/browse/PHOENIX-2228 Project: Phoenix Issue Type: Sub-task Reporter: Maryann Xue -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-2199) Support DDL in Phoenix-Calcite integration
[ https://issues.apache.org/jira/browse/PHOENIX-2199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maryann Xue updated PHOENIX-2199: - Issue Type: Task (was: New Feature) > Support DDL in Phoenix-Calcite integration > -- > > Key: PHOENIX-2199 > URL: https://issues.apache.org/jira/browse/PHOENIX-2199 > Project: Phoenix > Issue Type: Task >Reporter: James Taylor > > The existing Phoenix compiler classes are of the form Create type>Compiler. For example CreateTableCompiler handles both VIEW and TABLE > compilation, CreateSequenceCompiler is for CREATE SEQEUNCE, and > CreateIndexCompiler is for CREATE INDEX. The compiler class does most of the > validation and then calls a MetaDataClient method to update the meta data > repository. > As a general implementation strategy, we'd transfer over the validation code > from the compiler methods to new classes plugged into the JavaCC compilation > of Calcite and then have these new classes call the same, somewhat modified > MetaDataClient APIs. A slightly more detailed breakdown of the work includes: > - Creating new JavaCC rules using the same templating technique as we did for > CREATE VIEW (see https://github.com/apache/phoenix/pull/113/commits), marking > the parse node produced as DDL. > - Producing the information required while the Calcite parser is running to > enable the call to the appropriate MetaDataClient API to update the metadata > repository. > - Tweaking the MetaDataClient APIs to not depend on the objects created at > parse time, but instead pass through the constituent parts. For example, > instead of passing through a CreateTableStatement object in the > MetaDataClient.createTable() call, only the necessary member variables would > be passed through. The idea would be to break the dependency on parse-time > object after compilation is complete. > - Changing ParseNode references used in MetaDataClient, as we don't want to > continue maintaining these. By ParseNode, I mean the parts in the grammar > that match an expression and produce a ParseNode instance. For DDL, this is > the CreateIndexStatement object, which stores a list of ParseNodes as the > indexed expressions. These are currently processed in MetaDataClient, so > these could be changed to either Expressions or RexNodes. > - Collecting at parse time the list of UDFs that need to be resolved. I see > this in CreateIndexStatement, but not CreateTableStatement (which I believe > is an oversight - probably wouldn't work to reference a UDF in the WHERE > clause of a view). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2198) Support correlate variable
[ https://issues.apache.org/jira/browse/PHOENIX-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14730254#comment-14730254 ] Maryann Xue commented on PHOENIX-2198: -- bq. Is it not currently possible to use the CorrelatedPlan in Phoenix, but only from the Calcite branch? Correct. In most cases, nested-loop join would have the worse performance, unless we know in advance (from stats) that the outer relation is only going to produce a very limited number of rows and executing the inner relation is cheap (for example a point lookup). So I think this would only be worthwhile in Phoenix + Calcite, where we can cost it. CorrelatedPlan is also used where correlated subqueries that cannot be de-correlated (not very common though), and Phoenix does not have this front-end support yet. > Support correlate variable > -- > > Key: PHOENIX-2198 > URL: https://issues.apache.org/jira/browse/PHOENIX-2198 > Project: Phoenix > Issue Type: New Feature >Reporter: Maryann Xue >Assignee: Maryann Xue > Attachments: PHOENIX-2198.patch > > Original Estimate: 240h > Remaining Estimate: 240h > > This will enable the outer query to set a correlate variable as a parameter > to restart the inner query for each iteration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-2227) [Pherf] Add ability to execute a DDL statement at the scenario level before any workloads are run
[ https://issues.apache.org/jira/browse/PHOENIX-2227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jan Fernando updated PHOENIX-2227: -- Attachment: PHOENIX-2227.patch [~cody.mar...@gmail.com] Here's a possible patch for what I had in mind for this. Let me know what you think. > [Pherf] Add ability to execute a DDL statement at the scenario level before > any workloads are run > - > > Key: PHOENIX-2227 > URL: https://issues.apache.org/jira/browse/PHOENIX-2227 > Project: Phoenix > Issue Type: Improvement >Reporter: Jan Fernando >Assignee: Jan Fernando > Attachments: PHOENIX-2227.patch > > > In order to support Pherf scenarios that write and read from multi-tenant > views it would be useful to add ddl attribute at the scenario level. This DDL > would get run before the workload ran. This would provide a nice hook to > create a mulit-tenant view that we are going to read and write from. This is > analogous to how Pherf allows ddl to be created at the querySet level for > enabling multi-tenant reads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2198) Support correlate variable
[ https://issues.apache.org/jira/browse/PHOENIX-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14730112#comment-14730112 ] Maryann Xue commented on PHOENIX-2198: -- In terms of performance, the number of iterations is fixed, but we could probably combine the network requests or even HBase scans together for several iterations of the inner relation, but this would be a really case-by-case optimization depending on what type of QueryPlan the inner relation is. Basically it can only be a simple ScanPlan, and if it turns out that the dynamic filter compiles into an HBase Get, we can batch these Get into one network request. But if the inner plan is a range scan or a full scan with a filter, it becomes much trickier, we might have to combine the filter (requires some compile-time work) with OR logic and push them into one scan (otherwise it wouldn't make much sense), and then after the scan returns, we'll have to figure out on the client side which results maps to which correlate variable value, which is like executing the inner query again over a much smaller result set. Anyway, I think we'll first just implement the literal semantics and make sure things all work well with this basic version. And later on, we can think of optimizing CorrelatePlan bit by bit. Yes, VIEWs should just work fine. In Calcite VIEWs are replaced with its definition algebra in the real query, so the filter of the VIEW will be automatically added to the relational expression tree, and all those rules dealing with Filter (filter-combine-rule, filter-push-down-to-scan-rule, etc) will kick in to make this work. But good reminder, James! I'll add a test on the calcite branch to verify. Global index hint doesn't work in Phoenix/Calcite integration yet. It depends on CALCITE-772. But on the Phoenix side, it's a more general question of migrating our child/parent join optimization onto calcite branch, which is logged as PHOENIX-1785. And this dynamic filter added in BaseQueryPlan is just a good facility that can be made use of later on for this issue. bq. Sometimes this can happen if the size of the row changes, as you'll send up with more guideposts than before. Can you explain a little more? How will the size of the row change in this test case? BTW, I can't repro any more now. > Support correlate variable > -- > > Key: PHOENIX-2198 > URL: https://issues.apache.org/jira/browse/PHOENIX-2198 > Project: Phoenix > Issue Type: New Feature >Reporter: Maryann Xue >Assignee: Maryann Xue > Attachments: PHOENIX-2198.patch > > Original Estimate: 240h > Remaining Estimate: 240h > > This will enable the outer query to set a correlate variable as a parameter > to restart the inner query for each iteration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (PHOENIX-2227) [Pherf] Add ability to execute a DDL statement at the scenario level before any workloads are run
Jan Fernando created PHOENIX-2227: - Summary: [Pherf] Add ability to execute a DDL statement at the scenario level before any workloads are run Key: PHOENIX-2227 URL: https://issues.apache.org/jira/browse/PHOENIX-2227 Project: Phoenix Issue Type: Improvement Reporter: Jan Fernando Assignee: Jan Fernando In order to support Pherf scenarios that write and read from multi-tenant views it would be useful to add ddl attribute at the scenario level. This DDL would get run before the workload ran. This would provide a nice hook to create a mulit-tenant view that we are going to read and write from. This is analogous to how Pherf allows ddl to be created at the querySet level for enabling multi-tenant reads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2207) Load scanner caches in parallel when using stats and round robin iterator
[
https://issues.apache.org/jira/browse/PHOENIX-2207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14729992#comment-14729992
]
James Taylor commented on PHOENIX-2207:
---
Make sure the KeyValue your creating for the Tuple is valid against your
schema. You're using a VARCHAR for the key and an INTEGER for the value, so try
this schema instead:
{code}
CREATE TABLE T (k VARCHAR PRIMARY KEY, v INTEGER) SALT_BUCKETS = 8
{code}
I don't think the column family and column qualifier not matching (S.S) will
make any difference, so probably no need to change your KeyValue construction
for that.
> Load scanner caches in parallel when using stats and round robin iterator
> -
>
> Key: PHOENIX-2207
> URL: https://issues.apache.org/jira/browse/PHOENIX-2207
> Project: Phoenix
> Issue Type: Bug
>Reporter: Samarth Jain
>Assignee: Samarth Jain
> Attachments: PHOENIX-2207.patch, PHOENIX-2207_v2.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-2207) Load scanner caches in parallel when using stats and round robin iterator
[
https://issues.apache.org/jira/browse/PHOENIX-2207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14729982#comment-14729982
]
Samarth Jain commented on PHOENIX-2207:
---
I was running into problems returning the ID as Tuple in the next() method.
This is what I did:
{code}
@Override
public Tuple next() throws SQLException {
byte[] value = PInteger.INSTANCE.toBytes(id);
KeyValue keyValue = KeyValueUtil.newKeyValue(Bytes.toBytes("abcd"),
SINGLE_COLUMN_FAMILY, SINGLE_COLUMN, HConstants.LATEST_TIMESTAMP, value, 0,
value.length);
return new SingleKeyValueTuple(keyValue);
}
{code}
When I try doing rs.getInt(1) it fails with the exception:
{code}
java.sql.SQLException: ERROR 201 (22000): Illegal data. Value
-2206091586288004225 cannot be cast to Integer without changing its value
at
org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:389)
at
org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:145)
at
org.apache.phoenix.schema.types.PDataType.newIllegalDataException(PDataType.java:290)
at
org.apache.phoenix.schema.types.PLong$LongCodec.decodeInt(PLong.java:254)
at org.apache.phoenix.schema.types.PInteger.toObject(PInteger.java:81)
at org.apache.phoenix.schema.types.PInteger.toObject(PInteger.java:28)
at
org.apache.phoenix.schema.types.PDataType.toObject(PDataType.java:983)
at
org.apache.phoenix.compile.ExpressionProjector.getValue(ExpressionProjector.java:75)
at
org.apache.phoenix.jdbc.PhoenixResultSet.getInt(PhoenixResultSet.java:449)
{code}
So I am not sure if I can just return a SingleKeyValueTuple here.
My table schema is:
CREATE TABLE T (k BIGINT PRIMARY KEY, v VARCHAR) SALT_BUCKETS = 8
And the query I am executing is:
select K from T
> Load scanner caches in parallel when using stats and round robin iterator
> -
>
> Key: PHOENIX-2207
> URL: https://issues.apache.org/jira/browse/PHOENIX-2207
> Project: Phoenix
> Issue Type: Bug
>Reporter: Samarth Jain
>Assignee: Samarth Jain
> Attachments: PHOENIX-2207.patch, PHOENIX-2207_v2.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-2198) Support correlate variable
[ https://issues.apache.org/jira/browse/PHOENIX-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14729916#comment-14729916 ] James Taylor commented on PHOENIX-2198: --- Thanks for the patch, [~maryannxue]. This is a very nice addition. What will be the performance characteristics of a query that uses the new CorrelatedPlan as a nested loop join? What kind of batching are we doing and what controls that? Is it not currently possible to use the CorrelatedPlan in Phoenix, but only from the Calcite branch? Not sure about the failure, [~maryannxue] - I haven't seen that before. Sometimes this can happen if the size of the row changes, as you'll send up with more guideposts than before. For this new dynamic filter that gets passed through, are there any potential interactions when a VIEW is used? In this case, we grab the WHERE clause off of the view and dynamically apply it when we compile the query. Will that work together with this? Also, what about the skip scan we dynamically do when a global index is hinted. Probably a good idea to add some testing around these scenarios. > Support correlate variable > -- > > Key: PHOENIX-2198 > URL: https://issues.apache.org/jira/browse/PHOENIX-2198 > Project: Phoenix > Issue Type: New Feature >Reporter: Maryann Xue >Assignee: Maryann Xue > Attachments: PHOENIX-2198.patch > > Original Estimate: 240h > Remaining Estimate: 240h > > This will enable the outer query to set a correlate variable as a parameter > to restart the inner query for each iteration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2207) Load scanner caches in parallel when using stats and round robin iterator
[
https://issues.apache.org/jira/browse/PHOENIX-2207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14729896#comment-14729896
]
James Taylor commented on PHOENIX-2207:
---
Thanks for the revisions, [~samarthjain]. Why not just have the Tuple use the
iterator ID as the value? Then you'd need no static state or casting.
{code}
+@Override
+public Tuple next() throws SQLException {
+iteratorOrder.add(id);
+return dummyTuple; // create new Tuple each time with value of id
+}
+
{code}
Too bad the TableResultIterator isn't created inside of the
iteratorFactory.newIterator() method, as then I think we'd be able to do it.
That might be a good change to make down the road, but having an IT test for
now seems ok.
> Load scanner caches in parallel when using stats and round robin iterator
> -
>
> Key: PHOENIX-2207
> URL: https://issues.apache.org/jira/browse/PHOENIX-2207
> Project: Phoenix
> Issue Type: Bug
>Reporter: Samarth Jain
>Assignee: Samarth Jain
> Attachments: PHOENIX-2207.patch, PHOENIX-2207_v2.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-2116) phoenix-flume: Sink/Serializer should be extendable
[ https://issues.apache.org/jira/browse/PHOENIX-2116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14729399#comment-14729399 ] Hudson commented on PHOENIX-2116: - SUCCESS: Integrated in Phoenix-master #890 (See [https://builds.apache.org/job/Phoenix-master/890/]) PHOENIX-2116: phoenix-flume - Sink/Serializer should be extendable (jmahonin: rev 6ada2ae039b717a35056b1b8df4e4f9e1145ace5) * phoenix-flume/src/main/java/org/apache/phoenix/flume/sink/PhoenixSink.java * phoenix-flume/src/it/java/org/apache/phoenix/flume/serializer/CustomSerializer.java * phoenix-flume/pom.xml * phoenix-flume/src/it/java/org/apache/phoenix/flume/PhoenixSinkIT.java * phoenix-flume/src/it/java/org/apache/phoenix/flume/sink/NullPhoenixSink.java > phoenix-flume: Sink/Serializer should be extendable > --- > > Key: PHOENIX-2116 > URL: https://issues.apache.org/jira/browse/PHOENIX-2116 > Project: Phoenix > Issue Type: Improvement >Affects Versions: 4.5.0, 4.4.1 >Reporter: Josh Mahonin >Assignee: Josh Mahonin > Fix For: 4.6.0 > > Attachments: PHOENIX-2116-v2.patch, PHOENIX-2116.patch > > > When using flume, often times custom serializers are necessary to transform > data before sending to a sink. The existing Phoenix implementation however > makes it difficult to extend and add new functionality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-2160) Projection of specific array index does not work
[ https://issues.apache.org/jira/browse/PHOENIX-2160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dumindu Buddhika updated PHOENIX-2160: -- Attachment: PHOENIX-2160_4.5_4.x.patch Oh. I am sorry about that. Here is an updated patch. > Projection of specific array index does not work > > > Key: PHOENIX-2160 > URL: https://issues.apache.org/jira/browse/PHOENIX-2160 > Project: Phoenix > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Assignee: Dumindu Buddhika > Attachments: PHOENIX-2160.patch, PHOENIX-2160_4.5_4.x.patch, > PHOENIX-2160_4.5_4.x.patch, PHOENIX-2160_v2.patch, PHOENIX-2160_v3.patch, > PHOENIX-2160_v4.patch, PHOENIX-2160_v5.patch, PHOENIX-2160_v6.patch, > PHOENIX-2160_v7.patch > > > PHOENIX-10 that allowed projection of specific array index does not work now. > Was looking into the code for some thing and found this issue. Let me know if > am missing something. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (PHOENIX-2116) phoenix-flume: Sink/Serializer should be extendable
[ https://issues.apache.org/jira/browse/PHOENIX-2116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Mahonin resolved PHOENIX-2116. --- Resolution: Fixed Fix Version/s: 4.6.0 Fixed in: https://github.com/apache/phoenix/commit/6ada2ae039b717a35056b1b8df4e4f9e1145ace5 > phoenix-flume: Sink/Serializer should be extendable > --- > > Key: PHOENIX-2116 > URL: https://issues.apache.org/jira/browse/PHOENIX-2116 > Project: Phoenix > Issue Type: Improvement >Affects Versions: 4.5.0, 4.4.1 >Reporter: Josh Mahonin >Assignee: Josh Mahonin > Fix For: 4.6.0 > > Attachments: PHOENIX-2116-v2.patch, PHOENIX-2116.patch > > > When using flume, often times custom serializers are necessary to transform > data before sending to a sink. The existing Phoenix implementation however > makes it difficult to extend and add new functionality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [DISCUSS] drop 4.x-HBase-1.1 branch
Just an FYI, branch 4.x-HBase-1.1 seems to have re-appeared in both Github and Apache upstream. On Wed, Jul 1, 2015 at 2:37 PM, James Taylor wrote: > Thanks, Nick. I'll send out a reminder to folks. > > On Wed, Jul 1, 2015 at 11:34 AM, Nick Dimiduk wrote: > > > Hmm. Looks like i'm not the only one who's pushed stuff there. Okay, it's > > gone. > > > > On Wed, Jul 1, 2015 at 11:26 AM, James Taylor > > wrote: > > > > > Yes, it resurrected itself. :-) That'd be good if you could delete that > > > branch to avoid any confusion. > > > > > > On Wed, Jul 1, 2015 at 11:21 AM, Nick Dimiduk > > wrote: > > > > > > > On Tue, Jun 30, 2015 at 7:07 PM, James Taylor < > jamestay...@apache.org> > > > > wrote: > > > > > > > > > Nick - looks like the 4.x-HBase-1.1 branch is back. Do you think we > > > need > > > > > this branch currently? > > > > > > > > > > > > > Oops, is that my mistake? No, I think we agreed that master branch is > > the > > > > equivalent of 4.x-hbase-1.1. > > > > > > > > On Mon, Jun 29, 2015 at 7:04 PM, Enis Söztutar > > wrote: > > > > > > > > > > > Should we also drop 1.0 branch and support only HBase-1.1+ in > > Phoenix > > > > > 4.5+ > > > > > > ? > > > > > > > > > > > > HBase-1.0 users have 4.4.x releases to use. It should be fair to > > > > require > > > > > a > > > > > > minor HBase upgrade for a minor Phoenix version upgrade. > > > > > > > > > > > > Enis > > > > > > > > > > > > On Mon, Jun 22, 2015 at 9:18 AM, James Taylor < > > > jamestay...@apache.org> > > > > > > wrote: > > > > > > > > > > > > > Ok, I'll drop the branch tomorrow then. Thanks, > > > > > > > > > > > > > > James > > > > > > > > > > > > > > On Mon, Jun 22, 2015 at 9:16 AM, Nick Dimiduk < > > ndimi...@gmail.com> > > > > > > wrote: > > > > > > > > Yeah, I guess it's fine, +1. > > > > > > > > > > > > > > > > On Mon, Jun 22, 2015 at 8:50 AM, Cody Marcel < > > > > cmar...@salesforce.com > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > >> +1 to removing. > > > > > > > >> > > > > > > > >> On Sun, Jun 21, 2015 at 2:14 PM, Samarth Jain < > > > sama...@apache.org > > > > > > > > > > > > wrote: > > > > > > > >> > > > > > > > >> > +1 to removing 4.x-HBase-1.1 branch and recreating it > later > > > from > > > > > > > master, > > > > > > > >> if > > > > > > > >> > needed. Another feature that isn't on that branch because > > > other > > > > > > > commits > > > > > > > >> > were missing is PHOENIX-1504. > > > > > > > >> > > > > > > > > >> > On Sunday, June 21, 2015, James Taylor < > > > jamestay...@apache.org> > > > > > > > wrote: > > > > > > > >> > > > > > > > > >> > > Nick - based on your last email, I wasn't sure if you > > meant > > > > > you're > > > > > > > ok > > > > > > > >> > > or not with the 4.x-HBase-1.1 branch being dropped. It's > > > > getting > > > > > > > >> > > further and further behind master and we can always > > > re-create > > > > it > > > > > > > from > > > > > > > >> > > master if we need it down-the-road. > > > > > > > >> > > > > > > > > > >> > > On Thu, Jun 18, 2015 at 1:28 PM, Nick Dimiduk < > > > > > ndimi...@gmail.com > > > > > > > >> > > > wrote: > > > > > > > >> > > > Fair point, commits should be starting at master and > > > working > > > > > > their > > > > > > > >> way > > > > > > > >> > > > back. We'll still need to verify that all the 4.x > > branches > > > > > have > > > > > > > the > > > > > > > >> > same > > > > > > > >> > > > patches as master. > > > > > > > >> > > > > > > > > > > >> > > > On Thu, Jun 18, 2015 at 12:09 PM, James Taylor < > > > > > > > >> jamestay...@apache.org > > > > > > > >> > > > > > > > > > > >> > > > wrote: > > > > > > > >> > > > > > > > > > > >> > > >> I don't know how to verify they have the same > patches, > > > but > > > > > why > > > > > > > would > > > > > > > >> > > >> master be missing commits from the 4.x-HBase-1.1 > > branch? > > > > > > > >> > > >> > > > > > > > >> > > >> I do see commits missing from the 4.x-HBase-1.1 > branch, > > > > > though. > > > > > > > >> When I > > > > > > > >> > > >> run the unit tests on this branch after adding some > of > > > the > > > > > > > missing > > > > > > > >> > > >> commits, I get failures while the tests pass fine in > > > > master. > > > > > > > >> > > >> > > > > > > > >> > > >> I don't see any value in spending time figuring this > > out > > > as > > > > > > > there's > > > > > > > >> > > >> really no need to have the 4.x-HBase-1.1 branch at > this > > > > point > > > > > > in > > > > > > > >> time. > > > > > > > >> > > >> This branch is only getting further behind master at > > this > > > > > > point. > > > > > > > >> > > >> > > > > > > > >> > > >> If you want to spend time figuring it out, then I'll > > wait > > > > to > > > > > > drop > > > > > > > >> the > > > > > > > >> > > >> branch. Just let me know when you're ok with it. > > > > > > > >> > > >> > > > > > > > >> > > >> Thanks, > > > > > > > >> > > >> James > > > > > > > >> > > >> > > > > > > > >> > > >> On Thu, Jun 18, 2015 at 11:45 AM, Nick Dimiduk
[jira] [Commented] (PHOENIX-2116) phoenix-flume: Sink/Serializer should be extendable
[ https://issues.apache.org/jira/browse/PHOENIX-2116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14729194#comment-14729194 ] James Taylor commented on PHOENIX-2116: --- +1 for master and 4.x branches. > phoenix-flume: Sink/Serializer should be extendable > --- > > Key: PHOENIX-2116 > URL: https://issues.apache.org/jira/browse/PHOENIX-2116 > Project: Phoenix > Issue Type: Improvement >Affects Versions: 4.5.0, 4.4.1 >Reporter: Josh Mahonin >Assignee: Josh Mahonin > Attachments: PHOENIX-2116-v2.patch, PHOENIX-2116.patch > > > When using flume, often times custom serializers are necessary to transform > data before sending to a sink. The existing Phoenix implementation however > makes it difficult to extend and add new functionality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2196) phoenix-spark should automatically convert DataFrame field names to all caps
[ https://issues.apache.org/jira/browse/PHOENIX-2196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14729195#comment-14729195 ] Josh Mahonin commented on PHOENIX-2196: --- Oh, yeah that's definitely a gap with the table name normalization. I'm OK with including it in this JIRA since it's pretty similar issue, so we might as well do it in one shot. I think all we need is to invoke one of the SchemaUtil helper functions to normalize the table name, and handle schema names as well: https://github.com/apache/phoenix/blob/master/phoenix-core/src/main/java/org/apache/phoenix/util/SchemaUtil.java I don't think I'll be able to get to it before Tuesday at the earliest. If you have time to tackle it before then, I can review. I think all we'd need is that new normalize call before the following line, and a few unit tests to validate it: https://github.com/apache/phoenix/blob/master/phoenix-spark/src/main/scala/org/apache/phoenix/spark/PhoenixRDD.scala#L73 > phoenix-spark should automatically convert DataFrame field names to all caps > > > Key: PHOENIX-2196 > URL: https://issues.apache.org/jira/browse/PHOENIX-2196 > Project: Phoenix > Issue Type: Improvement >Reporter: Randy Gelhausen >Assignee: Josh Mahonin >Priority: Minor > Attachments: PHOENIX-2196.patch > > > phoenix-spark will fail to save a DF into a Phoenix table if the DataFrame's > fields are not all capitalized. Since Phoenix internally capitalizes all > column names, the DataFrame.save method should automatically capitalize DF > field names as a convenience to the end user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2196) phoenix-spark should automatically convert DataFrame field names to all caps
[ https://issues.apache.org/jira/browse/PHOENIX-2196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14729178#comment-14729178 ] Randy Gelhausen commented on PHOENIX-2196: -- Makes sense for the column rules. I built an application (https://github.com/randerzander/CSV-to-Phoenix) for allowing users to turn CSV files into tables. However, users don't know to capitalize their table _names_. Shouldn't the same normalization logic be applied to the user supplied _table name_ as well as to the column names in the CSV headers? If so, can we do that on this JIRA or do we need to create another? > phoenix-spark should automatically convert DataFrame field names to all caps > > > Key: PHOENIX-2196 > URL: https://issues.apache.org/jira/browse/PHOENIX-2196 > Project: Phoenix > Issue Type: Improvement >Reporter: Randy Gelhausen >Assignee: Josh Mahonin >Priority: Minor > Attachments: PHOENIX-2196.patch > > > phoenix-spark will fail to save a DF into a Phoenix table if the DataFrame's > fields are not all capitalized. Since Phoenix internally capitalizes all > column names, the DataFrame.save method should automatically capitalize DF > field names as a convenience to the end user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2196) phoenix-spark should automatically convert DataFrame field names to all caps
[ https://issues.apache.org/jira/browse/PHOENIX-2196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14729121#comment-14729121 ] Josh Mahonin commented on PHOENIX-2196: --- So essentially what happens is the field names go through the same normalization process the query parser uses. Without surrounding quotes, the column name will get capitalized, but with surrounding quotes, the column will be passed through as-is. For your specific use case, I think if your CSV headers are surrounded by double quotes, they should get passed through with the case kept in-tact. > phoenix-spark should automatically convert DataFrame field names to all caps > > > Key: PHOENIX-2196 > URL: https://issues.apache.org/jira/browse/PHOENIX-2196 > Project: Phoenix > Issue Type: Improvement >Reporter: Randy Gelhausen >Assignee: Josh Mahonin >Priority: Minor > Attachments: PHOENIX-2196.patch > > > phoenix-spark will fail to save a DF into a Phoenix table if the DataFrame's > fields are not all capitalized. Since Phoenix internally capitalizes all > column names, the DataFrame.save method should automatically capitalize DF > field names as a convenience to the end user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2196) phoenix-spark should automatically convert DataFrame field names to all caps
[ https://issues.apache.org/jira/browse/PHOENIX-2196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14729081#comment-14729081 ] Randy Gelhausen commented on PHOENIX-2196: -- Apologies all, haven't had the chance to do a test with a 4.6 instance of Phoenix. I did notice a similar issue with recognizing lowercase tablenames. Is it possible to do the same conversion rules there? > phoenix-spark should automatically convert DataFrame field names to all caps > > > Key: PHOENIX-2196 > URL: https://issues.apache.org/jira/browse/PHOENIX-2196 > Project: Phoenix > Issue Type: Improvement >Reporter: Randy Gelhausen >Assignee: Josh Mahonin >Priority: Minor > Attachments: PHOENIX-2196.patch > > > phoenix-spark will fail to save a DF into a Phoenix table if the DataFrame's > fields are not all capitalized. Since Phoenix internally capitalizes all > column names, the DataFrame.save method should automatically capitalize DF > field names as a convenience to the end user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (PHOENIX-2226) Support Semi-Join in Phoenix/Calcite Integration
Maryann Xue created PHOENIX-2226: Summary: Support Semi-Join in Phoenix/Calcite Integration Key: PHOENIX-2226 URL: https://issues.apache.org/jira/browse/PHOENIX-2226 Project: Phoenix Issue Type: Task Reporter: Maryann Xue Assignee: Maryann Xue -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (PHOENIX-2225) Support Correlate (Nested-loop Join) in Phoenix/Calcite Integration
Maryann Xue created PHOENIX-2225: Summary: Support Correlate (Nested-loop Join) in Phoenix/Calcite Integration Key: PHOENIX-2225 URL: https://issues.apache.org/jira/browse/PHOENIX-2225 Project: Phoenix Issue Type: Task Reporter: Maryann Xue Assignee: Maryann Xue -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2196) phoenix-spark should automatically convert DataFrame field names to all caps
[ https://issues.apache.org/jira/browse/PHOENIX-2196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14729003#comment-14729003 ] Josh Mahonin commented on PHOENIX-2196: --- Thanks [~maghamraviki...@gmail.com] No response from [~rgelhau], but I think this satisfies the issue. Is this OK to go in, and if so, what branches? > phoenix-spark should automatically convert DataFrame field names to all caps > > > Key: PHOENIX-2196 > URL: https://issues.apache.org/jira/browse/PHOENIX-2196 > Project: Phoenix > Issue Type: Improvement >Reporter: Randy Gelhausen >Assignee: Josh Mahonin >Priority: Minor > Attachments: PHOENIX-2196.patch > > > phoenix-spark will fail to save a DF into a Phoenix table if the DataFrame's > fields are not all capitalized. Since Phoenix internally capitalizes all > column names, the DataFrame.save method should automatically capitalize DF > field names as a convenience to the end user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2116) phoenix-flume: Sink/Serializer should be extendable
[ https://issues.apache.org/jira/browse/PHOENIX-2116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14729001#comment-14729001 ] Josh Mahonin commented on PHOENIX-2116: --- Thanks [~maghamraviki...@gmail.com]. [~jamestaylor] is this OK to go in? And if so, what branches? > phoenix-flume: Sink/Serializer should be extendable > --- > > Key: PHOENIX-2116 > URL: https://issues.apache.org/jira/browse/PHOENIX-2116 > Project: Phoenix > Issue Type: Improvement >Affects Versions: 4.5.0, 4.4.1 >Reporter: Josh Mahonin >Assignee: Josh Mahonin > Attachments: PHOENIX-2116-v2.patch, PHOENIX-2116.patch > > > When using flume, often times custom serializers are necessary to transform > data before sending to a sink. The existing Phoenix implementation however > makes it difficult to extend and add new functionality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2160) Projection of specific array index does not work
[ https://issues.apache.org/jira/browse/PHOENIX-2160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14728902#comment-14728902 ] ramkrishna.s.vasudevan commented on PHOENIX-2160: - [~Dumindux] The new files are missing in this 4.5 and 4.x branch patch? > Projection of specific array index does not work > > > Key: PHOENIX-2160 > URL: https://issues.apache.org/jira/browse/PHOENIX-2160 > Project: Phoenix > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Assignee: Dumindu Buddhika > Attachments: PHOENIX-2160.patch, PHOENIX-2160_4.5_4.x.patch, > PHOENIX-2160_v2.patch, PHOENIX-2160_v3.patch, PHOENIX-2160_v4.patch, > PHOENIX-2160_v5.patch, PHOENIX-2160_v6.patch, PHOENIX-2160_v7.patch > > > PHOENIX-10 that allowed projection of specific array index does not work now. > Was looking into the code for some thing and found this issue. Let me know if > am missing something. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-2207) Load scanner caches in parallel when using stats and round robin iterator
[ https://issues.apache.org/jira/browse/PHOENIX-2207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samarth Jain updated PHOENIX-2207: -- Attachment: PHOENIX-2207_v2.patch [~jamestaylor], attached is the updated patch with tests for RoundRobinIterator with salted tables and tables with stats. I wasn't able to extend BaseConnectionLessQueryTest because the ConnectionLessQueryServicesImpl.getTable() throws a UnsupportedOperationException when creating TableResultIterator in ParallelIterators. So I had to resort to writing the test as an IT test. Please review when you get a chance. Thanks! > Load scanner caches in parallel when using stats and round robin iterator > - > > Key: PHOENIX-2207 > URL: https://issues.apache.org/jira/browse/PHOENIX-2207 > Project: Phoenix > Issue Type: Bug >Reporter: Samarth Jain >Assignee: Samarth Jain > Attachments: PHOENIX-2207.patch, PHOENIX-2207_v2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
