Re: [VOTE] Apache Phoenix 5.0.0-alpha for HBase 2.0 rc0

[James Taylor]

Josh - should we hold off check-ins to 5.x branch until the vote passes?

On Mon, Feb 5, 2018 at 4:30 PM, James Taylor  wrote:

> That list in the vote thread is good. IMHO, it'd be ok to add release
> notes on the website once the vote passes. Couple of other minor features
> I'm not sure about: Does atomic upsert work (UPSERT VALUES ... ON DUPLICATE
> KEY ...)? Looks like point-in-time queries don't work correctly if your
> table is taking writes (PHOENIX-4378).
>
> One possible way to confirm any other missing feature would be to look at
> newly ignored tests and any failing unit tests in 5.0.
>
>
> On Mon, Feb 5, 2018 at 3:22 PM, Josh Elser  wrote:
>
>> Tried to include what I knew off the top of my head in the VOTE, but will
>> add it to the website if you think that will be helpful.
>>
>>
>> On 2/5/18 5:37 PM, James Taylor wrote:
>>
>>> Would be useful to write up release notes (typically here[1]) for what
>>> doesn't work yet. I think this will help folks know what to test and not
>>> test in the next RC.
>>>
>>> [1] https://phoenix.apache.org/release_notes.html
>>>
>>> On Mon, Feb 5, 2018 at 2:32 PM, Josh Elser  wrote:
>>>
>>> Just put up a patch to 4582.

 Any other comments from folks before I re-roll?


 On 2/5/18 3:35 PM, James Taylor wrote:

 -1. Need to fix PHOENIX-4582 or 4.x clients would attempt to connect to
> 5.x
> server.
>
> On Fri, Feb 2, 2018 at 8:45 PM, Artem Ervits 
> wrote:
>
> -1 (nonbinding)
>
>>
>> md5 binary: OK
>>
>> Centos 7.4
>> jdk: 1.8.0_161
>> hadoop 2.7.5
>> hbase 2.0
>>
>> ran example
>> $PHOENIX_HOME/bin/psql.py localhost:2181:/hbase-unsecure
>> $PHOENIX_HOME/examples/WEB_STAT.sql $PHOENIX_HOME/examples/WEB_STA
>> T.csv
>> $PHOENIX_HOME/examples/WEB_STAT_QUERIES.sql
>>
>> OK
>>
>> ran example in https://phoenix.apache.org/Pho
>> enix-in-15-minutes-or-less.
>> html
>>
>> OK
>>
>> tried to run performance.py script
>>
>> getting the same results as in
>> https://issues.apache.org/jira/browse/PHOENIX-4510
>>
>> [vagrant@hadoop ~]$ $PHOENIX_HOME/bin/performance.py
>> localhost:2181:/hbase-unsecure 1000
>> Phoenix Performance Evaluation Script 1.0
>> -
>>
>> Creating performance table...
>> 18/02/03 04:23:48 WARN util.NativeCodeLoader: Unable to load
>> native-hadoop
>> library for your platform... using builtin-java classes where
>> applicable
>> no rows upserted
>> Time: 4.59 sec(s)
>>
>> Query # 1 - Count - SELECT COUNT(1) FROM PERFORMANCE_1000;
>> Query # 2 - Group By First PK - SELECT HOST FROM PERFORMANCE_1000
>> GROUP
>> BY
>> HOST;
>> Query # 3 - Group By Second PK - SELECT DOMAIN FROM PERFORMANCE_1000
>> GROUP
>> BY DOMAIN;
>> Query # 4 - Truncate + Group By - SELECT TRUNC(DATE,'DAY') DAY FROM
>> PERFORMANCE_1000 GROUP BY TRUNC(DATE,'DAY');
>> Query # 5 - Filter + Count - SELECT COUNT(1) FROM PERFORMANCE_1000
>> WHERE
>> CORE<10;
>>
>> Generating and upserting data...
>> Error: Invalid or corrupt jarfile /tmp/data_akuell.csv
>>
>>
>> NOT OK
>>
>>
>>
>> sqlline still shows version 4.13
>>
>> Connected to: Phoenix (version 4.13)
>> Driver: PhoenixEmbeddedDriver (version 4.13)
>>
>> NOT OK
>>
>> On Fri, Feb 2, 2018 at 10:58 AM, Josh Elser 
>> wrote:
>>
>> Hello Everyone,
>>
>>>
>>> This is a call for a vote on Apache Phoenix 5.0.0-alpha rc0. This
>>> release
>>> is only targeting Apache HBase 2.0 and is known to have lacking
>>> functionality as compared to previous releases (e.g. transactional
>>>
>>> tables,
>>
>> Hive integration, full local indexing support). It is presented as-is
>>> in
>>>
>>> an
>>
>> attempt to encourage the community at large to get involved.
>>>
>>> The RC is available at the standard location:
>>>
>>> https://dist.apache.org/repos/dist/dev/phoenix/apache-phoeni
>>> x-5.0.0-alpha-HBase-2.0-rc0
>>>
>>> RC0 is based on the following commit: a2053c1d3b64a9cc2f35b1f83faa54
>>> e421bb20f1
>>>
>>> Signed with my key: 9E62822F4668F17B0972ADD9B7D5CD454677D66C,
>>> http://pgp.mit.edu/pks/lookup?op=get=0xB7D5CD454677D66C
>>>
>>> Vote will be open for at least 72 hours (2018/02/05 1600GMT). Please
>>>
>>> vote:
>>
>>
>>> [ ] +1 approve
>>> [ ] +0 no opinion
>>> [ ] -1 disapprove (and reason why)
>>>
>>> Thanks,
>>> The Apache Phoenix Team
>>>
>>>
>>>
>>
>
>>>
>

[jira] [Resolved] (PHOENIX-3941) Filter regions to scan for local indexes based on data table leading pk filter conditions

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-3941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor resolved PHOENIX-3941.
---
Resolution: Fixed

> Filter regions to scan for local indexes based on data table leading pk 
> filter conditions
> -
>
> Key: PHOENIX-3941
> URL: https://issues.apache.org/jira/browse/PHOENIX-3941
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: James Taylor
>Priority: Major
>  Labels: SFDC, localIndex
> Fix For: 4.14.0
>
> Attachments: PHOENIX-3941_v1.patch, PHOENIX-3941_v2.patch, 
> PHOENIX-3941_v3.patch, PHOENIX-3941_v4.patch
>
>
> Had a good offline conversation with [~ndimiduk] at PhoenixCon about local 
> indexes. Depending on the query, we can often times prune the regions we need 
> to scan over based on the where conditions against the data table pk. For 
> example, with a multi-tenant table, we only need to scan the regions that are 
> prefixed by the tenant ID.
> We can easily get this information from the compilation of the query against 
> the data table (which we always do), through the 
> statementContext.getScanRanges() structure. We'd just want to keep a pointer 
> to the data table QueryPlan from the local index QueryPlan.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4546) Projected table cannot be read through ProjectedColumnExpression

[Ankit Singhal (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356580#comment-16356580
 ] 

Ankit Singhal commented on PHOENIX-4546:


Thanks, [~jamestaylor] and [~sergey.soldatov] for the review. Committed in 5.x, 
master and 4.x branches.

> Projected table cannot be read through ProjectedColumnExpression
> 
>
> Key: PHOENIX-4546
> URL: https://issues.apache.org/jira/browse/PHOENIX-4546
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Romil Choksi
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 5.0.0, 4.14.0
>
> Attachments: PHOENIX-4546.patch, PHOENIX-4546_v1.patch, 
> PHOENIX-4546_v2.patch, PHOENIX-4546_v3.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4549) Pherf - Column override and sequenced index creation support

[James Taylor (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356572#comment-16356572
 ] 

James Taylor commented on PHOENIX-4549:
---

Looks like you only checked this into the 0.98 branch, [~mujtabachohan]. Also, 
please make sure to set the fixVersion appropriately and resolve as fixed once 
checked into all 4.x, master, and 5.x branches.

> Pherf - Column override and sequenced index creation support
> 
>
> Key: PHOENIX-4549
> URL: https://issues.apache.org/jira/browse/PHOENIX-4549
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Mujtaba Chohan
>Assignee: Mujtaba Chohan
>Priority: Major
> Attachments: PHOENIX-4549.patch
>
>
> Support for column level override for upserting view specific data along with 
> sequenced index creation either before or after data is loaded. Patch also 
> contains additional datatype support (TIMESTAMP, VARCHAR_ARRAY. VARBINARY AND 
> UNSIGNED_LONG)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-3941) Filter regions to scan for local indexes based on data table leading pk filter conditions

[James Taylor (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-3941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356568#comment-16356568
 ] 

James Taylor commented on PHOENIX-3941:
---

Attached final patch (with unit test fix for case where columns are in common 
but there aren't that many matching filters).

> Filter regions to scan for local indexes based on data table leading pk 
> filter conditions
> -
>
> Key: PHOENIX-3941
> URL: https://issues.apache.org/jira/browse/PHOENIX-3941
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: James Taylor
>Priority: Major
>  Labels: SFDC, localIndex
> Fix For: 4.14.0
>
> Attachments: PHOENIX-3941_v1.patch, PHOENIX-3941_v2.patch, 
> PHOENIX-3941_v3.patch, PHOENIX-3941_v4.patch
>
>
> Had a good offline conversation with [~ndimiduk] at PhoenixCon about local 
> indexes. Depending on the query, we can often times prune the regions we need 
> to scan over based on the where conditions against the data table pk. For 
> example, with a multi-tenant table, we only need to scan the regions that are 
> prefixed by the tenant ID.
> We can easily get this information from the compilation of the query against 
> the data table (which we always do), through the 
> statementContext.getScanRanges() structure. We'd just want to keep a pointer 
> to the data table QueryPlan from the local index QueryPlan.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (PHOENIX-3941) Filter regions to scan for local indexes based on data table leading pk filter conditions

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-3941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor updated PHOENIX-3941:
--
Attachment: PHOENIX-3941_v4.patch

> Filter regions to scan for local indexes based on data table leading pk 
> filter conditions
> -
>
> Key: PHOENIX-3941
> URL: https://issues.apache.org/jira/browse/PHOENIX-3941
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: James Taylor
>Priority: Major
>  Labels: SFDC, localIndex
> Fix For: 4.14.0
>
> Attachments: PHOENIX-3941_v1.patch, PHOENIX-3941_v2.patch, 
> PHOENIX-3941_v3.patch, PHOENIX-3941_v4.patch
>
>
> Had a good offline conversation with [~ndimiduk] at PhoenixCon about local 
> indexes. Depending on the query, we can often times prune the regions we need 
> to scan over based on the where conditions against the data table pk. For 
> example, with a multi-tenant table, we only need to scan the regions that are 
> prefixed by the tenant ID.
> We can easily get this information from the compilation of the query against 
> the data table (which we always do), through the 
> statementContext.getScanRanges() structure. We'd just want to keep a pointer 
> to the data table QueryPlan from the local index QueryPlan.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (PHOENIX-4589) Split up OrderByIT into multiple tests

[James Taylor (JIRA)]

James Taylor created PHOENIX-4589:
-

 Summary: Split up OrderByIT into multiple tests
 Key: PHOENIX-4589
 URL: https://issues.apache.org/jira/browse/PHOENIX-4589
 Project: Phoenix
  Issue Type: Improvement
Reporter: James Taylor


There are too many physical tables created by OrderByIT. We need to break it up 
into multiple tests so it doesn't impact the overall test run.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (PHOENIX-4576) Fix LocalIndexSplitMergeIT tests failing in master branch

[Rajeshbabu Chintaguntla (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356515#comment-16356515
 ] 

Rajeshbabu Chintaguntla edited comment on PHOENIX-4576 at 2/8/18 5:54 AM:
--

Currently what's happening is we are getting duplicate results because we are 
not skipping the child regions scanners. We are doing that properly at 
BaseResultIterators#getIterators on stale region exception. So as part of fix 
if the the exception thrown during scanner initialization throwing back to 
BaseResultIterators#getIterators there we handle the merge case properly.
[~jamestaylor] Please review.

In the patch I have enabled the test cases disabled at PHOENIX-4273 which are 
passing locally. Next time will take look at them if they failed as currently 
not able to get full info whats happening.
FYI [~samarthjain].


was (Author: rajeshbabu):
[~jamestaylor] Patch fixes the Local index merge failure test case. I have 
enabled the test cases disabled PHOENIX-4273 passing locally. Next time will 
take look at them if they failed as currently not able to get full info whats 
happening.
Please review.

> Fix LocalIndexSplitMergeIT tests failing in master branch
> -
>
> Key: PHOENIX-4576
> URL: https://issues.apache.org/jira/browse/PHOENIX-4576
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
>Priority: Major
> Fix For: 4.14.0
>
> Attachments: PHOENIX-4576.patch
>
>
> Currenty LocalIndexSplitMergeIT#testLocalIndexScanAfterRegionsMerge is 
> failing in master branch. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4576) Fix LocalIndexSplitMergeIT tests failing in master branch

[Rajeshbabu Chintaguntla (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356515#comment-16356515
 ] 

Rajeshbabu Chintaguntla commented on PHOENIX-4576:
--

[~jamestaylor] Patch fixes the Local index merge failure test case. I have 
enabled the test cases disabled PHOENIX-4273 passing locally. Next time will 
take look at them if they failed as currently not able to get full info whats 
happening.
Please review.

> Fix LocalIndexSplitMergeIT tests failing in master branch
> -
>
> Key: PHOENIX-4576
> URL: https://issues.apache.org/jira/browse/PHOENIX-4576
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
>Priority: Major
> Fix For: 4.14.0
>
> Attachments: PHOENIX-4576.patch
>
>
> Currenty LocalIndexSplitMergeIT#testLocalIndexScanAfterRegionsMerge is 
> failing in master branch. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (PHOENIX-4576) Fix LocalIndexSplitMergeIT tests failing in master branch

[Rajeshbabu Chintaguntla (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajeshbabu Chintaguntla updated PHOENIX-4576:
-
Attachment: PHOENIX-4576.patch

> Fix LocalIndexSplitMergeIT tests failing in master branch
> -
>
> Key: PHOENIX-4576
> URL: https://issues.apache.org/jira/browse/PHOENIX-4576
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
>Priority: Major
> Fix For: 4.14.0
>
> Attachments: PHOENIX-4576.patch
>
>
> Currenty LocalIndexSplitMergeIT#testLocalIndexScanAfterRegionsMerge is 
> failing in master branch. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4278) Implement pure client side transactional index maintenance

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356383#comment-16356383
 ] 

ASF GitHub Bot commented on PHOENIX-4278:
-

Github user JamesRTaylor commented on the issue:

https://github.com/apache/phoenix/pull/291
  
This looks really good! Couple of minor comments, but overall it looks fine.


> Implement pure client side transactional index maintenance
> --
>
> Key: PHOENIX-4278
> URL: https://issues.apache.org/jira/browse/PHOENIX-4278
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: James Taylor
>Assignee: Ohad Shacham
>Priority: Major
>
> The index maintenance for transactions follows the same model as non 
> transactional tables - coprocessor based on data table updates that looks up 
> previous row value to perform maintenance. This is necessary for non 
> transactional tables to ensure the rows are locked so that a consistent view 
> may be obtained. However, for transactional tables, the time stamp oracle 
> ensures uniqueness of time stamps (via transaction IDs) and the filtering 
> handles a scan seeing the "true" last committed value for a row. Thus, 
> there's no hard dependency to perform this on the server side.
> Moving the index maintenance to the client side would prevent any RS->RS RPC 
> calls (which have proved to be troublesome for HBase). It would require 
> returning more data to the client (i.e. the prior row value), but this seems 
> like a reasonable tradeoff.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] phoenix issue #291: [PHOENIX-4278] Implement pure client side transactional ...

[JamesRTaylor]

Github user JamesRTaylor commented on the issue:

https://github.com/apache/phoenix/pull/291
  
This looks really good! Couple of minor comments, but overall it looks fine.


---

[jira] [Commented] (PHOENIX-4278) Implement pure client side transactional index maintenance

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356380#comment-16356380
 ] 

ASF GitHub Bot commented on PHOENIX-4278:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/291#discussion_r166816653
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/execute/PhoenixTxnIndexMutationGenerator.java
 ---
@@ -0,0 +1,505 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.execute;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
+import com.google.common.primitives.Longs;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.hbase.*;
+import org.apache.hadoop.hbase.client.*;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.regionserver.MiniBatchOperationInProgress;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.compile.ScanRanges;
+import org.apache.phoenix.coprocessor.BaseScannerRegionObserver;
+import org.apache.phoenix.filter.SkipScanFilter;
+import org.apache.phoenix.hbase.index.MultiMutation;
+import org.apache.phoenix.hbase.index.ValueGetter;
+import org.apache.phoenix.hbase.index.covered.IndexMetaData;
+import org.apache.phoenix.hbase.index.covered.IndexUpdate;
+import org.apache.phoenix.hbase.index.covered.TableState;
+import org.apache.phoenix.hbase.index.covered.update.ColumnReference;
+import org.apache.phoenix.hbase.index.covered.update.ColumnTracker;
+import org.apache.phoenix.hbase.index.covered.update.IndexedColumnGroup;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.index.PhoenixIndexCodec;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.KeyRange;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.types.PVarbinary;
+import org.apache.phoenix.transaction.PhoenixTransactionContext;
+import 
org.apache.phoenix.transaction.PhoenixTransactionContext.PhoenixVisibilityLevel;
+import org.apache.phoenix.util.ScanUtil;
+import org.apache.phoenix.util.SchemaUtil;
+
+import java.io.IOException;
+import java.sql.SQLException;
+import java.util.*;
+
+
+public class PhoenixTxnIndexMutationGenerator {
+
+private static final Log LOG = 
LogFactory.getLog(PhoenixTxnIndexMutationGenerator.class);
+
+private final PhoenixConnection connection;
+private final PhoenixTransactionContext phoenixTransactionContext;
+
+PhoenixTxnIndexMutationGenerator(PhoenixConnection connection, 
PhoenixTransactionContext phoenixTransactionContext) {
+this.phoenixTransactionContext = phoenixTransactionContext;
+this.connection = connection;
+}
+
+private static void addMutation(Map 
mutations, ImmutableBytesPtr row, Mutation m) {
+MultiMutation stored = mutations.get(row);
+// we haven't seen this row before, so add it
+if (stored == null) {
+stored = new MultiMutation(row);
+mutations.put(row, stored);
+}
+stored.addAll(m);
+}
+
+public List getIndexUpdates(final PTable table, PTable 
index, List dataMutations) throws IOException, SQLException {
+
+if (dataMutations.isEmpty()) {
+return new ArrayList();
+}
+
+Map updateAttributes = 
dataMutations.get(0).getAttributesMap();
+boolean replyWrite = 

[GitHub] phoenix pull request #291: [PHOENIX-4278] Implement pure client side transac...

[JamesRTaylor]

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/291#discussion_r166816653
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/execute/PhoenixTxnIndexMutationGenerator.java
 ---
@@ -0,0 +1,505 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.execute;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
+import com.google.common.primitives.Longs;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.hbase.*;
+import org.apache.hadoop.hbase.client.*;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.regionserver.MiniBatchOperationInProgress;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.compile.ScanRanges;
+import org.apache.phoenix.coprocessor.BaseScannerRegionObserver;
+import org.apache.phoenix.filter.SkipScanFilter;
+import org.apache.phoenix.hbase.index.MultiMutation;
+import org.apache.phoenix.hbase.index.ValueGetter;
+import org.apache.phoenix.hbase.index.covered.IndexMetaData;
+import org.apache.phoenix.hbase.index.covered.IndexUpdate;
+import org.apache.phoenix.hbase.index.covered.TableState;
+import org.apache.phoenix.hbase.index.covered.update.ColumnReference;
+import org.apache.phoenix.hbase.index.covered.update.ColumnTracker;
+import org.apache.phoenix.hbase.index.covered.update.IndexedColumnGroup;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.index.PhoenixIndexCodec;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.KeyRange;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.types.PVarbinary;
+import org.apache.phoenix.transaction.PhoenixTransactionContext;
+import 
org.apache.phoenix.transaction.PhoenixTransactionContext.PhoenixVisibilityLevel;
+import org.apache.phoenix.util.ScanUtil;
+import org.apache.phoenix.util.SchemaUtil;
+
+import java.io.IOException;
+import java.sql.SQLException;
+import java.util.*;
+
+
+public class PhoenixTxnIndexMutationGenerator {
+
+private static final Log LOG = 
LogFactory.getLog(PhoenixTxnIndexMutationGenerator.class);
+
+private final PhoenixConnection connection;
+private final PhoenixTransactionContext phoenixTransactionContext;
+
+PhoenixTxnIndexMutationGenerator(PhoenixConnection connection, 
PhoenixTransactionContext phoenixTransactionContext) {
+this.phoenixTransactionContext = phoenixTransactionContext;
+this.connection = connection;
+}
+
+private static void addMutation(Map 
mutations, ImmutableBytesPtr row, Mutation m) {
+MultiMutation stored = mutations.get(row);
+// we haven't seen this row before, so add it
+if (stored == null) {
+stored = new MultiMutation(row);
+mutations.put(row, stored);
+}
+stored.addAll(m);
+}
+
+public List getIndexUpdates(final PTable table, PTable 
index, List dataMutations) throws IOException, SQLException {
+
+if (dataMutations.isEmpty()) {
+return new ArrayList();
+}
+
+Map updateAttributes = 
dataMutations.get(0).getAttributesMap();
+boolean replyWrite = 
(BaseScannerRegionObserver.ReplayWrite.fromBytes(updateAttributes.get(BaseScannerRegionObserver.REPLAY_WRITES))
 != null);
+byte[] txRollbackAttribute = 
updateAttributes.get(PhoenixTransactionContext.TX_ROLLBACK_ATTRIBUTE_KEY);
+
+IndexMaintainer maintainer = 

[jira] [Commented] (PHOENIX-4278) Implement pure client side transactional index maintenance

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356377#comment-16356377
 ] 

ASF GitHub Bot commented on PHOENIX-4278:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/291#discussion_r166816466
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/execute/PhoenixTxnIndexMutationGenerator.java
 ---
@@ -0,0 +1,505 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.execute;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
+import com.google.common.primitives.Longs;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.hbase.*;
+import org.apache.hadoop.hbase.client.*;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.regionserver.MiniBatchOperationInProgress;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.compile.ScanRanges;
+import org.apache.phoenix.coprocessor.BaseScannerRegionObserver;
+import org.apache.phoenix.filter.SkipScanFilter;
+import org.apache.phoenix.hbase.index.MultiMutation;
+import org.apache.phoenix.hbase.index.ValueGetter;
+import org.apache.phoenix.hbase.index.covered.IndexMetaData;
+import org.apache.phoenix.hbase.index.covered.IndexUpdate;
+import org.apache.phoenix.hbase.index.covered.TableState;
+import org.apache.phoenix.hbase.index.covered.update.ColumnReference;
+import org.apache.phoenix.hbase.index.covered.update.ColumnTracker;
+import org.apache.phoenix.hbase.index.covered.update.IndexedColumnGroup;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.index.PhoenixIndexCodec;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.KeyRange;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.types.PVarbinary;
+import org.apache.phoenix.transaction.PhoenixTransactionContext;
+import 
org.apache.phoenix.transaction.PhoenixTransactionContext.PhoenixVisibilityLevel;
+import org.apache.phoenix.util.ScanUtil;
+import org.apache.phoenix.util.SchemaUtil;
+
+import java.io.IOException;
+import java.sql.SQLException;
+import java.util.*;
+
+
+public class PhoenixTxnIndexMutationGenerator {
--- End diff --

I reviewed this, but it's difficult to determine if anything has changed. I 
see that you've removed coprocessor specific stuff like 
RegionCoprocessorEnvironment and the serialized PhoenixIndexMetaData which you 
don't need any more. Also, attributes aren't pass through TxTableState, but it 
seems like we don't rely on this anywhere that I can see. Any other changes 
that you remember that you think I should look at?


> Implement pure client side transactional index maintenance
> --
>
> Key: PHOENIX-4278
> URL: https://issues.apache.org/jira/browse/PHOENIX-4278
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: James Taylor
>Assignee: Ohad Shacham
>Priority: Major
>
> The index maintenance for transactions follows the same model as non 
> transactional tables - coprocessor based on data table updates that looks up 
> previous row value to perform maintenance. This is necessary for non 
> transactional tables to ensure the rows are locked so that a consistent view 
> may be obtained. However, for transactional tables, the time stamp oracle 
> ensures uniqueness of time stamps (via transaction IDs) and the filtering 
> handles a scan seeing the "true" last 

[GitHub] phoenix pull request #291: [PHOENIX-4278] Implement pure client side transac...

[JamesRTaylor]

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/291#discussion_r166816466
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/execute/PhoenixTxnIndexMutationGenerator.java
 ---
@@ -0,0 +1,505 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.execute;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
+import com.google.common.primitives.Longs;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.hbase.*;
+import org.apache.hadoop.hbase.client.*;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.regionserver.MiniBatchOperationInProgress;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.compile.ScanRanges;
+import org.apache.phoenix.coprocessor.BaseScannerRegionObserver;
+import org.apache.phoenix.filter.SkipScanFilter;
+import org.apache.phoenix.hbase.index.MultiMutation;
+import org.apache.phoenix.hbase.index.ValueGetter;
+import org.apache.phoenix.hbase.index.covered.IndexMetaData;
+import org.apache.phoenix.hbase.index.covered.IndexUpdate;
+import org.apache.phoenix.hbase.index.covered.TableState;
+import org.apache.phoenix.hbase.index.covered.update.ColumnReference;
+import org.apache.phoenix.hbase.index.covered.update.ColumnTracker;
+import org.apache.phoenix.hbase.index.covered.update.IndexedColumnGroup;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.index.PhoenixIndexCodec;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.KeyRange;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.types.PVarbinary;
+import org.apache.phoenix.transaction.PhoenixTransactionContext;
+import 
org.apache.phoenix.transaction.PhoenixTransactionContext.PhoenixVisibilityLevel;
+import org.apache.phoenix.util.ScanUtil;
+import org.apache.phoenix.util.SchemaUtil;
+
+import java.io.IOException;
+import java.sql.SQLException;
+import java.util.*;
+
+
+public class PhoenixTxnIndexMutationGenerator {
--- End diff --

I reviewed this, but it's difficult to determine if anything has changed. I 
see that you've removed coprocessor specific stuff like 
RegionCoprocessorEnvironment and the serialized PhoenixIndexMetaData which you 
don't need any more. Also, attributes aren't pass through TxTableState, but it 
seems like we don't rely on this anywhere that I can see. Any other changes 
that you remember that you think I should look at?


---

[jira] [Commented] (PHOENIX-4586) UPSERT SELECT doesn't take in account comparison operators for subqueries.

[Sergey Soldatov (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356346#comment-16356346
 ] 

Sergey Soldatov commented on PHOENIX-4586:
--

+ from me as well. Great work, [~maryannxue]

> UPSERT SELECT doesn't take in account comparison operators for subqueries.
> --
>
> Key: PHOENIX-4586
> URL: https://issues.apache.org/jira/browse/PHOENIX-4586
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.14.0
>Reporter: Sergey Soldatov
>Assignee: Maryann Xue
>Priority: Critical
> Fix For: 4.14.0
>
> Attachments: PHOENIX-4586.patch
>
>
> If upsert select has a where condition that is using any comparison operator 
> (including ANY/SOME/etc), the whole WHERE clause just ignored. Table:
> {noformat}
> create table T (id integer primary key, i1 integer);
> upsert into T values (1,1);
> upsert into T values (2,2);
> {noformat}
> Query that should not upsert anything because we have a condition in where 
> that I1 should be greater than any value we already have as well as not 
> existing ID:
> {noformat}
> 0: jdbc:phoenix:> upsert into T select id, 4 from T where id = 3 AND i1 > 
> (select i1 from T);
> 2 rows affected (0.02 seconds)
> 0: jdbc:phoenix:> select * from T;
> +-+-+
> | ID  | I1  |
> +-+-+
> | 1   | 4   |
> | 2   | 4   |
> +-+-+
> 2 rows selected (0.014 seconds)
> {noformat}
> Now with ANY.  Should not upsert anything as well because ID is [1,2], while 
> I1 are all '4':
> {noformat}
> 0: jdbc:phoenix:> upsert into T select id, 5 from T where id = 2 AND i1 = ANY 
> (select ID from T);
> 2 rows affected (0.016 seconds)
> 0: jdbc:phoenix:> select * from T;
> +-+-+
> | ID  | I1  |
> +-+-+
> | 1   | 5   |
> | 2   | 5   |
> +-+-+
> 2 rows selected (0.013 seconds)
> {noformat}
> A similar query with IN works just fine:
> {noformat}
> 0: jdbc:phoenix:> upsert into T select id, 6 from T where id = 2 AND i1 IN 
> (select ID from T);
> No rows affected (0.094 seconds)
> 0: jdbc:phoenix:> select * from T;
> +-+-+
> | ID  | I1  |
> +-+-+
> | 1   | 5   |
> | 2   | 5   |
> +-+-+
> 2 rows selected (0.014 seconds)
> {noformat}
> The reason for this behavior is that for IN we convert subselect to semi-join 
> and execute upsert on the client side.  For comparisons, we don't perform any 
> transformations and query is considered flat and finally executed on the 
> server side.  Not sure why, but we also completely ignore the second 
> condition in WHERE clause as well and that may lead to a serious data loss. 
> [~jamestaylor], [~maryannxue] any thoughts or suggestions how to fix that are 
> really appreciated. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4278) Implement pure client side transactional index maintenance

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356336#comment-16356336
 ] 

ASF GitHub Bot commented on PHOENIX-4278:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/291#discussion_r166809191
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/schema/PTableImpl.java ---
@@ -1038,32 +1038,21 @@ public void setValue(PColumn column, byte[] 
byteValue) {
 @Override
 public void delete() {
 newMutations();
-// we're using the Tephra column family delete marker here to 
prevent the translation 
-// of deletes to puts by the Tephra's TransactionProcessor
-if (PTableImpl.this.isTransactional()) {
-Put put = new Put(key);
-if (families.isEmpty()) {
-
put.add(SchemaUtil.getEmptyColumnFamily(PTableImpl.this), 
TransactionFactory.getTransactionFactory().getTransactionContext().getFamilyDeleteMarker(),
 ts,
-HConstants.EMPTY_BYTE_ARRAY);
-} else {
-for (PColumnFamily colFamily : families) {
-put.add(colFamily.getName().getBytes(), 
TransactionFactory.getTransactionFactory().getTransactionContext().getFamilyDeleteMarker(),
 ts,
-HConstants.EMPTY_BYTE_ARRAY);
-}
-}
-deleteRow = put;
--- End diff --

Does the code in IndexMaintainer here need to change too (i.e. check for 
getFamilyDeleteMarker()), or is it fine?

private DeleteType getDeleteTypeOrNull(Collection 
pendingUpdates, int nCFs) {
int nDeleteCF = 0;
int nDeleteVersionCF = 0;
for (Cell kv : pendingUpdates) {
if (kv.getTypeByte() == 
KeyValue.Type.DeleteFamilyVersion.getCode()) {
nDeleteVersionCF++;
}
else if (kv.getTypeByte() == 
KeyValue.Type.DeleteFamily.getCode()
// Since we don't include the index rows in the 
change set for txn tables, we need to detect row deletes that have transformed 
by TransactionProcessor
|| (CellUtil.matchingQualifier(kv, 
TransactionFactory.getTransactionFactory().getTransactionContext().getFamilyDeleteMarker())
 && CellUtil.matchingValue(kv, HConstants.EMPTY_BYTE_ARRAY))) {
nDeleteCF++;
}
}



> Implement pure client side transactional index maintenance
> --
>
> Key: PHOENIX-4278
> URL: https://issues.apache.org/jira/browse/PHOENIX-4278
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: James Taylor
>Assignee: Ohad Shacham
>Priority: Major
>
> The index maintenance for transactions follows the same model as non 
> transactional tables - coprocessor based on data table updates that looks up 
> previous row value to perform maintenance. This is necessary for non 
> transactional tables to ensure the rows are locked so that a consistent view 
> may be obtained. However, for transactional tables, the time stamp oracle 
> ensures uniqueness of time stamps (via transaction IDs) and the filtering 
> handles a scan seeing the "true" last committed value for a row. Thus, 
> there's no hard dependency to perform this on the server side.
> Moving the index maintenance to the client side would prevent any RS->RS RPC 
> calls (which have proved to be troublesome for HBase). It would require 
> returning more data to the client (i.e. the prior row value), but this seems 
> like a reasonable tradeoff.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] phoenix pull request #291: [PHOENIX-4278] Implement pure client side transac...

[JamesRTaylor]

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/291#discussion_r166809191
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/schema/PTableImpl.java ---
@@ -1038,32 +1038,21 @@ public void setValue(PColumn column, byte[] 
byteValue) {
 @Override
 public void delete() {
 newMutations();
-// we're using the Tephra column family delete marker here to 
prevent the translation 
-// of deletes to puts by the Tephra's TransactionProcessor
-if (PTableImpl.this.isTransactional()) {
-Put put = new Put(key);
-if (families.isEmpty()) {
-
put.add(SchemaUtil.getEmptyColumnFamily(PTableImpl.this), 
TransactionFactory.getTransactionFactory().getTransactionContext().getFamilyDeleteMarker(),
 ts,
-HConstants.EMPTY_BYTE_ARRAY);
-} else {
-for (PColumnFamily colFamily : families) {
-put.add(colFamily.getName().getBytes(), 
TransactionFactory.getTransactionFactory().getTransactionContext().getFamilyDeleteMarker(),
 ts,
-HConstants.EMPTY_BYTE_ARRAY);
-}
-}
-deleteRow = put;
--- End diff --

Does the code in IndexMaintainer here need to change too (i.e. check for 
getFamilyDeleteMarker()), or is it fine?

private DeleteType getDeleteTypeOrNull(Collection 
pendingUpdates, int nCFs) {
int nDeleteCF = 0;
int nDeleteVersionCF = 0;
for (Cell kv : pendingUpdates) {
if (kv.getTypeByte() == 
KeyValue.Type.DeleteFamilyVersion.getCode()) {
nDeleteVersionCF++;
}
else if (kv.getTypeByte() == 
KeyValue.Type.DeleteFamily.getCode()
// Since we don't include the index rows in the 
change set for txn tables, we need to detect row deletes that have transformed 
by TransactionProcessor
|| (CellUtil.matchingQualifier(kv, 
TransactionFactory.getTransactionFactory().getTransactionContext().getFamilyDeleteMarker())
 && CellUtil.matchingValue(kv, HConstants.EMPTY_BYTE_ARRAY))) {
nDeleteCF++;
}
}



---

[jira] [Commented] (PHOENIX-4278) Implement pure client side transactional index maintenance

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356329#comment-16356329
 ] 

ASF GitHub Bot commented on PHOENIX-4278:
-

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/291#discussion_r166808341
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
 ---
@@ -850,19 +849,12 @@ private void addCoprocessors(byte[] tableName, 
HTableDescriptor descriptor, PTab
 && !SchemaUtil.isMetaTable(tableName)
 && !SchemaUtil.isStatsTable(tableName)) {
 if (isTransactional) {
-if 
(!descriptor.hasCoprocessor(PhoenixTransactionalIndexer.class.getName())) {
-
descriptor.addCoprocessor(PhoenixTransactionalIndexer.class.getName(), null, 
priority, null);
-}
--- End diff --

This handles the case of the creation of new tables, but we'd need to 
handle the case of existing tables as well. This is somewhat trickier because 
ideally we'd want to handle the case in which the server jar has been upgraded, 
but the client jar hasn't been. Until the client is upgraded (in which case 
index maintenance will occur on the client side), we'd need to continue 
performing index maintenance through this coprocessor. The somewhat clunky way 
of doing that is to pass the client Phoenix version through the 
PhoenixIndexMetaData. This would be somewhat painful.

Since we've labelled transactions as "beta", perhaps we can punt on this. 
In this case, we'd require both the client and server jar to be upgraded at the 
same time (if transactional tables are already being used). With this approach, 
we'd want to modify PhoenixTransactionalIndexer to be an empty class derived 
still from BaseRegionObserver.


> Implement pure client side transactional index maintenance
> --
>
> Key: PHOENIX-4278
> URL: https://issues.apache.org/jira/browse/PHOENIX-4278
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: James Taylor
>Assignee: Ohad Shacham
>Priority: Major
>
> The index maintenance for transactions follows the same model as non 
> transactional tables - coprocessor based on data table updates that looks up 
> previous row value to perform maintenance. This is necessary for non 
> transactional tables to ensure the rows are locked so that a consistent view 
> may be obtained. However, for transactional tables, the time stamp oracle 
> ensures uniqueness of time stamps (via transaction IDs) and the filtering 
> handles a scan seeing the "true" last committed value for a row. Thus, 
> there's no hard dependency to perform this on the server side.
> Moving the index maintenance to the client side would prevent any RS->RS RPC 
> calls (which have proved to be troublesome for HBase). It would require 
> returning more data to the client (i.e. the prior row value), but this seems 
> like a reasonable tradeoff.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] phoenix pull request #291: [PHOENIX-4278] Implement pure client side transac...

[JamesRTaylor]

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/291#discussion_r166808341
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
 ---
@@ -850,19 +849,12 @@ private void addCoprocessors(byte[] tableName, 
HTableDescriptor descriptor, PTab
 && !SchemaUtil.isMetaTable(tableName)
 && !SchemaUtil.isStatsTable(tableName)) {
 if (isTransactional) {
-if 
(!descriptor.hasCoprocessor(PhoenixTransactionalIndexer.class.getName())) {
-
descriptor.addCoprocessor(PhoenixTransactionalIndexer.class.getName(), null, 
priority, null);
-}
--- End diff --

This handles the case of the creation of new tables, but we'd need to 
handle the case of existing tables as well. This is somewhat trickier because 
ideally we'd want to handle the case in which the server jar has been upgraded, 
but the client jar hasn't been. Until the client is upgraded (in which case 
index maintenance will occur on the client side), we'd need to continue 
performing index maintenance through this coprocessor. The somewhat clunky way 
of doing that is to pass the client Phoenix version through the 
PhoenixIndexMetaData. This would be somewhat painful.

Since we've labelled transactions as "beta", perhaps we can punt on this. 
In this case, we'd require both the client and server jar to be upgraded at the 
same time (if transactional tables are already being used). With this approach, 
we'd want to modify PhoenixTransactionalIndexer to be an empty class derived 
still from BaseRegionObserver.


---

Re: Anonymous survey: Apache HBase 1.x Usage

[Andrew Purtell]

Response to the survey so far. I think it confirms our expectations.
Multiple choice was allowed so percentage will not add up to 100%.

1.0: 8%
1.1: 21%
1.2: 47%
1.3: 24%
1.4: 8%
1.5: 5%

[image: Inline image 1]

On Fri, Feb 2, 2018 at 3:40 PM, Andrew Purtell  wrote:

> Please take this anonymous survey
> ​ to ​
> let us know what version of Apache HBase 1.x you are using in production
> now or are planning to use in production in the next year or so.
>
> Multiple choices are allowed.
>
> ​There is no "I'm not using 1.x" choice. Consider upgrading! (smile)
>
> https://www.surveymonkey.com/r/8WQ8QY6
>

[jira] [Commented] (PHOENIX-4529) Users should only require RX access to SYSTEM.SEQUENCE table

[Thomas D'Silva (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356297#comment-16356297
 ] 

Thomas D'Silva commented on PHOENIX-4529:
-

[~jamestaylor] 

I spoke to [~apurtell] offline. I think we should just wait for  HBASE-19842, 
after which we only have to set the acl tag on the cells of the sequence row 
when they are created. This acl tag is stored in the hbase acl table. When a 
user is granted (or revoked) access to a schema or table, we also update this 
acl tag. We won't have to rewrite all cells of all sequences in the schema. 



> Users should only require RX access to SYSTEM.SEQUENCE table
> 
>
> Key: PHOENIX-4529
> URL: https://issues.apache.org/jira/browse/PHOENIX-4529
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Karan Mehta
>Assignee: Thomas D'Silva
>Priority: Major
>
> Currently, users don't need to have Write access to {{SYSTEM.CATALOG}} and 
> other tables, since the code is run on the server side as login user. However 
> for {{SYSTEM.SEQUENCE}}, write permission is still needed. This is a 
> potential security concern, since it allows anyone to modify the sequences 
> created by others. This JIRA is to discuss how we can improve the security of 
> this table. 
> Potential options include
> 1. Usage of HBase Cell Level Permissions (works only with HFile version 3 and 
> above)
> 2. AccessControl at Phoenix Layer by addition of user column in the 
> {{SYSTEM.SEQUENCE}} table and use it for access control (Can be error-prone 
> for complex scenarios like sequence sharing)
> Please advice.
> [~tdsilva] [~jamestaylor] [~apurtell] [~an...@apache.org] [~elserj]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4586) UPSERT SELECT doesn't take in account comparison operators for subqueries.

[James Taylor (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356292#comment-16356292
 ] 

James Taylor commented on PHOENIX-4586:
---

+1 on the fix. Thanks for the quick turnaround, [~maryannxue].

> UPSERT SELECT doesn't take in account comparison operators for subqueries.
> --
>
> Key: PHOENIX-4586
> URL: https://issues.apache.org/jira/browse/PHOENIX-4586
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.14.0
>Reporter: Sergey Soldatov
>Assignee: Maryann Xue
>Priority: Critical
> Fix For: 4.14.0
>
> Attachments: PHOENIX-4586.patch
>
>
> If upsert select has a where condition that is using any comparison operator 
> (including ANY/SOME/etc), the whole WHERE clause just ignored. Table:
> {noformat}
> create table T (id integer primary key, i1 integer);
> upsert into T values (1,1);
> upsert into T values (2,2);
> {noformat}
> Query that should not upsert anything because we have a condition in where 
> that I1 should be greater than any value we already have as well as not 
> existing ID:
> {noformat}
> 0: jdbc:phoenix:> upsert into T select id, 4 from T where id = 3 AND i1 > 
> (select i1 from T);
> 2 rows affected (0.02 seconds)
> 0: jdbc:phoenix:> select * from T;
> +-+-+
> | ID  | I1  |
> +-+-+
> | 1   | 4   |
> | 2   | 4   |
> +-+-+
> 2 rows selected (0.014 seconds)
> {noformat}
> Now with ANY.  Should not upsert anything as well because ID is [1,2], while 
> I1 are all '4':
> {noformat}
> 0: jdbc:phoenix:> upsert into T select id, 5 from T where id = 2 AND i1 = ANY 
> (select ID from T);
> 2 rows affected (0.016 seconds)
> 0: jdbc:phoenix:> select * from T;
> +-+-+
> | ID  | I1  |
> +-+-+
> | 1   | 5   |
> | 2   | 5   |
> +-+-+
> 2 rows selected (0.013 seconds)
> {noformat}
> A similar query with IN works just fine:
> {noformat}
> 0: jdbc:phoenix:> upsert into T select id, 6 from T where id = 2 AND i1 IN 
> (select ID from T);
> No rows affected (0.094 seconds)
> 0: jdbc:phoenix:> select * from T;
> +-+-+
> | ID  | I1  |
> +-+-+
> | 1   | 5   |
> | 2   | 5   |
> +-+-+
> 2 rows selected (0.014 seconds)
> {noformat}
> The reason for this behavior is that for IN we convert subselect to semi-join 
> and execute upsert on the client side.  For comparisons, we don't perform any 
> transformations and query is considered flat and finally executed on the 
> server side.  Not sure why, but we also completely ignore the second 
> condition in WHERE clause as well and that may lead to a serious data loss. 
> [~jamestaylor], [~maryannxue] any thoughts or suggestions how to fix that are 
> really appreciated. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4549) Pherf - Column override and sequenced index creation support

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356228#comment-16356228
 ] 

Hudson commented on PHOENIX-4549:
-

FAILURE: Integrated in Jenkins build Phoenix-master #1927 (See 
[https://builds.apache.org/job/Phoenix-master/1927/])
PHOENIX-4549 Pherf - Column override and sequenced index creation (mujtaba: rev 
073392e35d740e06748500b554ee60bf755de349)
* (edit) phoenix-pherf/src/test/resources/scenario/test_scenario.xml
* (edit) 
phoenix-pherf/src/main/java/org/apache/phoenix/pherf/configuration/Column.java
* (edit) 
phoenix-pherf/src/main/java/org/apache/phoenix/pherf/result/QueryResult.java
* (edit) 
phoenix-pherf/src/test/java/org/apache/phoenix/pherf/RuleGeneratorTest.java
* (edit) 
phoenix-pherf/src/main/java/org/apache/phoenix/pherf/configuration/Query.java
* (edit) 
phoenix-pherf/src/main/java/org/apache/phoenix/pherf/workload/MultiThreadedRunner.java
* (edit) 
phoenix-pherf/src/main/java/org/apache/phoenix/pherf/result/ResultUtil.java
* (edit) 
phoenix-pherf/src/main/java/org/apache/phoenix/pherf/configuration/Scenario.java
* (edit) phoenix-pherf/src/test/java/org/apache/phoenix/pherf/ColumnTest.java
* (edit) 
phoenix-pherf/src/main/resources/scenario/prod_test_unsalted_scenario.xml
* (edit) phoenix-pherf/src/test/resources/datamodel/test_schema.sql
* (edit) 
phoenix-pherf/src/main/java/org/apache/phoenix/pherf/configuration/DataTypeMapping.java
* (edit) 
phoenix-pherf/src/main/java/org/apache/phoenix/pherf/result/ResultManager.java
* (edit) 
phoenix-pherf/src/main/java/org/apache/phoenix/pherf/workload/QueryExecutor.java
* (edit) 
phoenix-pherf/src/main/java/org/apache/phoenix/pherf/rules/RulesApplier.java
* (edit) 
phoenix-pherf/src/main/java/org/apache/phoenix/pherf/util/PhoenixUtil.java
* (edit) 
phoenix-pherf/src/test/java/org/apache/phoenix/pherf/ConfigurationParserTest.java
* (edit) 
phoenix-pherf/src/main/java/org/apache/phoenix/pherf/schema/SchemaReader.java
* (edit) 
phoenix-pherf/src/main/java/org/apache/phoenix/pherf/workload/WriteWorkload.java


> Pherf - Column override and sequenced index creation support
> 
>
> Key: PHOENIX-4549
> URL: https://issues.apache.org/jira/browse/PHOENIX-4549
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Mujtaba Chohan
>Assignee: Mujtaba Chohan
>Priority: Major
> Attachments: PHOENIX-4549.patch
>
>
> Support for column level override for upserting view specific data along with 
> sequenced index creation either before or after data is loaded. Patch also 
> contains additional datatype support (TIMESTAMP, VARCHAR_ARRAY. VARBINARY AND 
> UNSIGNED_LONG)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4579) Add a config to conditionally create Phoenix meta tables on first client connection

[James Taylor (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356224#comment-16356224
 ] 

James Taylor commented on PHOENIX-4579:
---

I think having this be purely driven from a property is a bit dangerous. If 
that property was set to false and a new client and server jar were deployed 
which required some upgrade, it wouldn't be performed, but instead stuff would 
just start to break. In addition to that (though less important), Phoenix 
already has a lot of configuration properties so if we can get away without 
adding another one, that'd be best.

Instead, I'd recommend we do the following:
- In ConnectionQueryServicesImpl.checkClientServerCompatibility(), add a new 
piece of information that gets returned from the getVersion RPC. In that call, 
we can (on the RS side) query the SYSTEM.CATALOG for the version information 
and return that back to the client.
- In ConnectionQueryServicesImpl.init(), if the actual system catalog version 
(i.e. the one from the server) is equal to 
MetaDataProtocol.MIN_SYSTEM_TABLE_TIMESTAMP, then we know no upgrade is 
required and we can skip the create table call for system catalog. We should 
think about the other system tables - maybe they each need their own 
MIN_SYSTEM_TABLE_TIMESTAMP value too? Maybe that could be follow on work.
- If we do change the hbase metadata of system catalog, then we'd have to make 
sure to increment MIN_SYSTEM_TABLE_TIMESTAMP, so that'd be kind of a new case. 
The empty key value of the SYSTEM.CATALOG row in the SYSTEM.CATALOG table could 
be updated in this case
- orthogonal to this change, it'd be good if we turned off 
{{phoenix.autoupgrade.enabled}} and required an EXECUTE UPGRADE to be run 
manually.

If we do the above, then we'd still catch the case where we have a new client 
against a new server where an upgrade is required.



> Add a config to conditionally create Phoenix meta tables on first client 
> connection
> ---
>
> Key: PHOENIX-4579
> URL: https://issues.apache.org/jira/browse/PHOENIX-4579
> Project: Phoenix
>  Issue Type: New Feature
>Reporter: Mujtaba Chohan
>Assignee: Mujtaba Chohan
>Priority: Major
> Attachments: PHOENIX-4579.patch
>
>
> Currently we create/modify Phoenix meta tables on first client connection. 
> Adding a property to make it configurable (with default true as it is 
> currently implemented).
> With this property set to false, it will avoid lockstep upgrade requirement 
> for all clients when changing meta properties using PHOENIX-4575 as this 
> property can be flipped back on once all the clients are upgraded.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4575) Phoenix metadata KEEP_DELETED_CELLS and VERSIONS should be property driven

[James Taylor (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356198#comment-16356198
 ] 

James Taylor commented on PHOENIX-4575:
---

Looks good, except for this change, keep the call to 
{{scan.setMaxVersions();}}, but just use the empty argument version of it (b/c 
we want to process all versions):
{code}
--- a/phoenix-core/src/main/java/org/apache/phoenix/util/UpgradeUtil.java
+++ b/phoenix-core/src/main/java/org/apache/phoenix/util/UpgradeUtil.java
@@ -223,7 +223,6 @@ public class UpgradeUtil {
 
 Scan scan = new Scan();
 scan.setRaw(true);
-scan.setMaxVersions(MetaDataProtocol.DEFAULT_MAX_META_DATA_VERSIONS);
 ResultScanner scanner = null;
 HTableInterface source = null;
 HTableInterface target = null;
@@ -696,7 +695,6 @@ public class UpgradeUtil {
 boolean success = false;
 Scan scan = new Scan();
 scan.setRaw(true);
-
scan.setMaxVersions(MetaDataProtocol.DEFAULT_MAX_META_DATA_VERSIONS);
 HTableInterface seqTable = 
conn.getQueryServices().getTable(PhoenixDatabaseMetaData.SYSTEM_SEQUENCE_NAME_BYTES);
 try {
 boolean committed = false;
{code}

> Phoenix metadata KEEP_DELETED_CELLS and VERSIONS should be property driven
> --
>
> Key: PHOENIX-4575
> URL: https://issues.apache.org/jira/browse/PHOENIX-4575
> Project: Phoenix
>  Issue Type: New Feature
>Reporter: Mujtaba Chohan
>Assignee: Mujtaba Chohan
>Priority: Major
> Attachments: PHOENIX-4575.patch, PHOENIX-4575_v2.patch
>
>
> This is to cater for circumstances where we need to alter state of 
> KEEP_DELETED_CELLS/VERSION on Phoenix meta tables.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4586) UPSERT SELECT doesn't take in account comparison operators for subqueries.

[Sergey Soldatov (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356184#comment-16356184
 ] 

Sergey Soldatov commented on PHOENIX-4586:
--

[~maryannxue] Wow, that was quick :) I'll test it. Actually, I was thinking 
about adding such kind of check to avoid server-side execution but was not sure 
whether it's enough (comparing to the case with IN, which is similar to = ANY,  
where we generate semi-join).

> UPSERT SELECT doesn't take in account comparison operators for subqueries.
> --
>
> Key: PHOENIX-4586
> URL: https://issues.apache.org/jira/browse/PHOENIX-4586
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.14.0
>Reporter: Sergey Soldatov
>Assignee: Maryann Xue
>Priority: Critical
> Fix For: 4.14.0
>
> Attachments: PHOENIX-4586.patch
>
>
> If upsert select has a where condition that is using any comparison operator 
> (including ANY/SOME/etc), the whole WHERE clause just ignored. Table:
> {noformat}
> create table T (id integer primary key, i1 integer);
> upsert into T values (1,1);
> upsert into T values (2,2);
> {noformat}
> Query that should not upsert anything because we have a condition in where 
> that I1 should be greater than any value we already have as well as not 
> existing ID:
> {noformat}
> 0: jdbc:phoenix:> upsert into T select id, 4 from T where id = 3 AND i1 > 
> (select i1 from T);
> 2 rows affected (0.02 seconds)
> 0: jdbc:phoenix:> select * from T;
> +-+-+
> | ID  | I1  |
> +-+-+
> | 1   | 4   |
> | 2   | 4   |
> +-+-+
> 2 rows selected (0.014 seconds)
> {noformat}
> Now with ANY.  Should not upsert anything as well because ID is [1,2], while 
> I1 are all '4':
> {noformat}
> 0: jdbc:phoenix:> upsert into T select id, 5 from T where id = 2 AND i1 = ANY 
> (select ID from T);
> 2 rows affected (0.016 seconds)
> 0: jdbc:phoenix:> select * from T;
> +-+-+
> | ID  | I1  |
> +-+-+
> | 1   | 5   |
> | 2   | 5   |
> +-+-+
> 2 rows selected (0.013 seconds)
> {noformat}
> A similar query with IN works just fine:
> {noformat}
> 0: jdbc:phoenix:> upsert into T select id, 6 from T where id = 2 AND i1 IN 
> (select ID from T);
> No rows affected (0.094 seconds)
> 0: jdbc:phoenix:> select * from T;
> +-+-+
> | ID  | I1  |
> +-+-+
> | 1   | 5   |
> | 2   | 5   |
> +-+-+
> 2 rows selected (0.014 seconds)
> {noformat}
> The reason for this behavior is that for IN we convert subselect to semi-join 
> and execute upsert on the client side.  For comparisons, we don't perform any 
> transformations and query is considered flat and finally executed on the 
> server side.  Not sure why, but we also completely ignore the second 
> condition in WHERE clause as well and that may lead to a serious data loss. 
> [~jamestaylor], [~maryannxue] any thoughts or suggestions how to fix that are 
> really appreciated. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (PHOENIX-4586) UPSERT SELECT doesn't take in account comparison operators for subqueries.

[Maryann Xue (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maryann Xue updated PHOENIX-4586:
-
Attachment: PHOENIX-4586.patch

> UPSERT SELECT doesn't take in account comparison operators for subqueries.
> --
>
> Key: PHOENIX-4586
> URL: https://issues.apache.org/jira/browse/PHOENIX-4586
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.14.0
>Reporter: Sergey Soldatov
>Assignee: Maryann Xue
>Priority: Critical
> Fix For: 4.14.0
>
> Attachments: PHOENIX-4586.patch
>
>
> If upsert select has a where condition that is using any comparison operator 
> (including ANY/SOME/etc), the whole WHERE clause just ignored. Table:
> {noformat}
> create table T (id integer primary key, i1 integer);
> upsert into T values (1,1);
> upsert into T values (2,2);
> {noformat}
> Query that should not upsert anything because we have a condition in where 
> that I1 should be greater than any value we already have as well as not 
> existing ID:
> {noformat}
> 0: jdbc:phoenix:> upsert into T select id, 4 from T where id = 3 AND i1 > 
> (select i1 from T);
> 2 rows affected (0.02 seconds)
> 0: jdbc:phoenix:> select * from T;
> +-+-+
> | ID  | I1  |
> +-+-+
> | 1   | 4   |
> | 2   | 4   |
> +-+-+
> 2 rows selected (0.014 seconds)
> {noformat}
> Now with ANY.  Should not upsert anything as well because ID is [1,2], while 
> I1 are all '4':
> {noformat}
> 0: jdbc:phoenix:> upsert into T select id, 5 from T where id = 2 AND i1 = ANY 
> (select ID from T);
> 2 rows affected (0.016 seconds)
> 0: jdbc:phoenix:> select * from T;
> +-+-+
> | ID  | I1  |
> +-+-+
> | 1   | 5   |
> | 2   | 5   |
> +-+-+
> 2 rows selected (0.013 seconds)
> {noformat}
> A similar query with IN works just fine:
> {noformat}
> 0: jdbc:phoenix:> upsert into T select id, 6 from T where id = 2 AND i1 IN 
> (select ID from T);
> No rows affected (0.094 seconds)
> 0: jdbc:phoenix:> select * from T;
> +-+-+
> | ID  | I1  |
> +-+-+
> | 1   | 5   |
> | 2   | 5   |
> +-+-+
> 2 rows selected (0.014 seconds)
> {noformat}
> The reason for this behavior is that for IN we convert subselect to semi-join 
> and execute upsert on the client side.  For comparisons, we don't perform any 
> transformations and query is considered flat and finally executed on the 
> server side.  Not sure why, but we also completely ignore the second 
> condition in WHERE clause as well and that may lead to a serious data loss. 
> [~jamestaylor], [~maryannxue] any thoughts or suggestions how to fix that are 
> really appreciated. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (PHOENIX-4586) UPSERT SELECT doesn't take in account comparison operators for subqueries.

[Maryann Xue (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maryann Xue reassigned PHOENIX-4586:


Assignee: Maryann Xue

> UPSERT SELECT doesn't take in account comparison operators for subqueries.
> --
>
> Key: PHOENIX-4586
> URL: https://issues.apache.org/jira/browse/PHOENIX-4586
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.14.0
>Reporter: Sergey Soldatov
>Assignee: Maryann Xue
>Priority: Critical
> Fix For: 4.14.0
>
> Attachments: PHOENIX-4586.patch
>
>
> If upsert select has a where condition that is using any comparison operator 
> (including ANY/SOME/etc), the whole WHERE clause just ignored. Table:
> {noformat}
> create table T (id integer primary key, i1 integer);
> upsert into T values (1,1);
> upsert into T values (2,2);
> {noformat}
> Query that should not upsert anything because we have a condition in where 
> that I1 should be greater than any value we already have as well as not 
> existing ID:
> {noformat}
> 0: jdbc:phoenix:> upsert into T select id, 4 from T where id = 3 AND i1 > 
> (select i1 from T);
> 2 rows affected (0.02 seconds)
> 0: jdbc:phoenix:> select * from T;
> +-+-+
> | ID  | I1  |
> +-+-+
> | 1   | 4   |
> | 2   | 4   |
> +-+-+
> 2 rows selected (0.014 seconds)
> {noformat}
> Now with ANY.  Should not upsert anything as well because ID is [1,2], while 
> I1 are all '4':
> {noformat}
> 0: jdbc:phoenix:> upsert into T select id, 5 from T where id = 2 AND i1 = ANY 
> (select ID from T);
> 2 rows affected (0.016 seconds)
> 0: jdbc:phoenix:> select * from T;
> +-+-+
> | ID  | I1  |
> +-+-+
> | 1   | 5   |
> | 2   | 5   |
> +-+-+
> 2 rows selected (0.013 seconds)
> {noformat}
> A similar query with IN works just fine:
> {noformat}
> 0: jdbc:phoenix:> upsert into T select id, 6 from T where id = 2 AND i1 IN 
> (select ID from T);
> No rows affected (0.094 seconds)
> 0: jdbc:phoenix:> select * from T;
> +-+-+
> | ID  | I1  |
> +-+-+
> | 1   | 5   |
> | 2   | 5   |
> +-+-+
> 2 rows selected (0.014 seconds)
> {noformat}
> The reason for this behavior is that for IN we convert subselect to semi-join 
> and execute upsert on the client side.  For comparisons, we don't perform any 
> transformations and query is considered flat and finally executed on the 
> server side.  Not sure why, but we also completely ignore the second 
> condition in WHERE clause as well and that may lead to a serious data loss. 
> [~jamestaylor], [~maryannxue] any thoughts or suggestions how to fix that are 
> really appreciated. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4575) Phoenix metadata KEEP_DELETED_CELLS and VERSIONS should be property driven

[Mujtaba Chohan (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356121#comment-16356121
 ] 

Mujtaba Chohan commented on PHOENIX-4575:
-

[~jamestaylor]

> Phoenix metadata KEEP_DELETED_CELLS and VERSIONS should be property driven
> --
>
> Key: PHOENIX-4575
> URL: https://issues.apache.org/jira/browse/PHOENIX-4575
> Project: Phoenix
>  Issue Type: New Feature
>Reporter: Mujtaba Chohan
>Assignee: Mujtaba Chohan
>Priority: Major
> Attachments: PHOENIX-4575.patch, PHOENIX-4575_v2.patch
>
>
> This is to cater for circumstances where we need to alter state of 
> KEEP_DELETED_CELLS/VERSION on Phoenix meta tables.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4579) Add a config to conditionally create Phoenix meta tables on first client connection

[Mujtaba Chohan (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356120#comment-16356120
 ] 

Mujtaba Chohan commented on PHOENIX-4579:
-

[~jamestaylor]

> Add a config to conditionally create Phoenix meta tables on first client 
> connection
> ---
>
> Key: PHOENIX-4579
> URL: https://issues.apache.org/jira/browse/PHOENIX-4579
> Project: Phoenix
>  Issue Type: New Feature
>Reporter: Mujtaba Chohan
>Assignee: Mujtaba Chohan
>Priority: Major
> Attachments: PHOENIX-4579.patch
>
>
> Currently we create/modify Phoenix meta tables on first client connection. 
> Adding a property to make it configurable (with default true as it is 
> currently implemented).
> With this property set to false, it will avoid lockstep upgrade requirement 
> for all clients when changing meta properties using PHOENIX-4575 as this 
> property can be flipped back on once all the clients are upgraded.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4549) Pherf - Column override and sequenced index creation support

[Thomas D'Silva (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355953#comment-16355953
 ] 

Thomas D'Silva commented on PHOENIX-4549:
-

+1

> Pherf - Column override and sequenced index creation support
> 
>
> Key: PHOENIX-4549
> URL: https://issues.apache.org/jira/browse/PHOENIX-4549
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Mujtaba Chohan
>Assignee: Mujtaba Chohan
>Priority: Major
> Attachments: PHOENIX-4549.patch
>
>
> Support for column level override for upserting view specific data along with 
> sequenced index creation either before or after data is loaded. Patch also 
> contains additional datatype support (TIMESTAMP, VARCHAR_ARRAY. VARBINARY AND 
> UNSIGNED_LONG)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4546) Projected table cannot be read through ProjectedColumnExpression

[James Taylor (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355680#comment-16355680
 ] 

James Taylor commented on PHOENIX-4546:
---

+1 on the patch. Great work, [~an...@apache.org] and good find, 
[~sergey.soldatov].

> Projected table cannot be read through ProjectedColumnExpression
> 
>
> Key: PHOENIX-4546
> URL: https://issues.apache.org/jira/browse/PHOENIX-4546
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Romil Choksi
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 5.0.0, 4.14.0
>
> Attachments: PHOENIX-4546.patch, PHOENIX-4546_v1.patch, 
> PHOENIX-4546_v2.patch, PHOENIX-4546_v3.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4588) Clone expression also if it's children have Determinism.PER_INVOCATION

[James Taylor (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355669#comment-16355669
 ] 

James Taylor commented on PHOENIX-4588:
---

The largest Determinism of a child expression should be propagated to its 
parent. So if a child has Determinism.PER_INVOCATION, its parent should have it 
as well. Are you finding that not to be the case, [~an...@apache.org]?

> Clone expression also if it's children have Determinism.PER_INVOCATION
> --
>
> Key: PHOENIX-4588
> URL: https://issues.apache.org/jira/browse/PHOENIX-4588
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 4.14.0
>
> Attachments: PHOENIX-4588.patch
>
>
> In case of parallel UPSERT for non-aggregate SELECT, we should be cloning the 
> expression too if any of it's children needs to be cloned. Otherwise, queries 
> will fail with IllegalDataException
>  
> {code:java}
> Error: ERROR 201 (22000): Illegal data. ERROR 201 (22000): Illegal data. 
> Expected length of at least 32 bytes, but had 31 (state=22000,code=201)
> java.sql.SQLException: ERROR 201 (22000): Illegal data. ERROR 201 (22000): 
> Illegal data. Expected length of at least 32 bytes, but had 31
> at 
> org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:443)
> at 
> org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:145)
> at 
> org.apache.phoenix.util.ServerUtil.parseRemoteException(ServerUtil.java:141)
> at 
> org.apache.phoenix.util.ServerUtil.parseServerExceptionOrNull(ServerUtil.java:125)
> at 
> org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:111)
> at 
> org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:778)
> at 
> org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:719)
> at 
> org.apache.phoenix.iterate.ConcatResultIterator.getIterators(ConcatResultIterator.java:50)
> at 
> org.apache.phoenix.iterate.ConcatResultIterator.currentIterator(ConcatResultIterator.java:97)
> at 
> org.apache.phoenix.iterate.ConcatResultIterator.next(ConcatResultIterator.java:117)
> at 
> org.apache.phoenix.iterate.DelegateResultIterator.next(DelegateResultIterator.java:44)
> at 
> org.apache.phoenix.compile.UpsertCompiler$2.execute(UpsertCompiler.java:797)
> at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:343)
> at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:331)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at 
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:330)
> at 
> org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1440)
> at sqlline.Commands.execute(Commands.java:822)
> at sqlline.Commands.sql(Commands.java:732)
> at sqlline.SqlLine.dispatch(SqlLine.java:808)
> at sqlline.SqlLine.begin(SqlLine.java:681)
> at sqlline.SqlLine.start(SqlLine.java:398)
> at sqlline.SqlLine.main(SqlLine.java:292)
> Caused by: java.sql.SQLException: ERROR 201 (22000): Illegal data. Expected 
> length of at least 32 bytes, but had 31
> at 
> org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:443)
> at 
> org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:145)
> at org.apache.phoenix.schema.KeyValueSchema.next(KeyValueSchema.java:213)
> at 
> org.apache.phoenix.expression.ProjectedColumnExpression.evaluate(ProjectedColumnExpression.java:115)
> at 
> org.apache.phoenix.expression.CoerceExpression.evaluate(CoerceExpression.java:148)
> at 
> org.apache.phoenix.compile.ExpressionProjector.getValue(ExpressionProjector.java:69)
> at 
> org.apache.phoenix.jdbc.PhoenixResultSet.getObject(PhoenixResultSet.java:524)
> at 
> org.apache.phoenix.compile.UpsertCompiler.upsertSelect(UpsertCompiler.java:194)
> at 
> org.apache.phoenix.compile.UpsertCompiler.access$000(UpsertCompiler.java:110)
> at 
> org.apache.phoenix.compile.UpsertCompiler$UpsertingParallelIteratorFactory.mutate(UpsertCompiler.java:254)
> at 
> org.apache.phoenix.compile.MutatingParallelIteratorFactory.newIterator(MutatingParallelIteratorFactory.java:59)
> at 
> org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:115)
> at 
> org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:106)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at 

[jira] [Comment Edited] (PHOENIX-4056) java.lang.IllegalArgumentException: Can not create a Path from an empty string

[Josh Mahonin (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355485#comment-16355485
 ] 

Josh Mahonin edited comment on PHOENIX-4056 at 2/7/18 2:11 PM:
---

I don't have much time to do more than a cursory review right now, 
unfortunately. Overall I like the approach, but:

a) What happens if 'mapred.output.dir' isn't set? Or is that always the case? 
org.apache.hadoop.conf.Configuration::get() will return a null in that case. Is 
Spark OK with that? Maybe another idea is to wrap it up in a scala Option, 
something like:
 {{Option(conf.get("mapred.output.dir")).getOrElse("")}}

a) Any ideas on if this is backwards compatible down to Spark 2.0? It maybe an 
impossible goal, but it would be nice to stay binary compatible with the whole 
2.x range, if possible.


was (Author: jmahonin):
I don't have much time to do more than a cursory review right now, 
unfortunately. Overall I like the approach, but:

a) What happens if 'mapred.output.dir' isn't set? Or is that always the case? 
org.apache.hadoop.conf.Configuration::get() will return a null in that case. Is 
Spark OK with that? Maybe another idea is to wrap it up in a scala Option, 
something like:
{{Option(conf.get('mapred.output.dir')).getOrElse("")}}

a) Any ideas on if this is backwards compatible down to Spark 2.0? It maybe an 
impossible goal, but it would be nice to stay binary compatible with the whole 
2.x range, if possible.

> java.lang.IllegalArgumentException: Can not create a Path from an empty string
> --
>
> Key: PHOENIX-4056
> URL: https://issues.apache.org/jira/browse/PHOENIX-4056
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.11.0
> Environment: CDH5.12
> Phoenix:4.11
> HBase:1.2
> Spark: 2.2.0
> phoenix-spark.version:4.11.0-HBase-1.2
>Reporter: Jepson
>Priority: Major
>  Labels: features, patch, test
> Attachments: PHOENIX-4056.patch
>
>   Original Estimate: 12h
>  Remaining Estimate: 12h
>
> 1.use the configuration of server and client(scala project)
>  
> phoenix.schema.isNamespaceMappingEnabled
> true
>   
>   
> phoenix.schema.mapSystemTablesToNamespace
> true
>   
> 2.The Code:
> {code:java}
> resultDF.write
>  .format("org.apache.phoenix.spark")
>  .mode(SaveMode.Overwrite)
>  .option("table", "JYDW.ADDRESS_ORDERCOUNT")
>  .option("zkUrl","192.168.1.40,192.168.1.41,192.168.1.42:2181")
>  .save()
> {code}
> 3.Throw this error,help to fix it,thankyou :
> 7/08/02 01:07:25 INFO DAGScheduler: Job 6 finished: runJob at 
> SparkHadoopMapReduceWriter.scala:88, took 7.990715 s
> 17/08/02 01:07:25 ERROR SparkHadoopMapReduceWriter: Aborting job 
> job_20170802010717_0079.
> {color:#59afe1}*java.lang.IllegalArgumentException: Can not create a Path 
> from an empty string*{color}
>   at org.apache.hadoop.fs.Path.checkPathArg(Path.java:126)
>   at org.apache.hadoop.fs.Path.(Path.java:134)
>   at org.apache.hadoop.fs.Path.(Path.java:88)
>   at 
> org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.absPathStagingDir(HadoopMapReduceCommitProtocol.scala:58)
>   at 
> org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.commitJob(HadoopMapReduceCommitProtocol.scala:132)
>   at 
> org.apache.spark.internal.io.SparkHadoopMapReduceWriter$.write(SparkHadoopMapReduceWriter.scala:101)
>   at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply$mcV$sp(PairRDDFunctions.scala:1085)
>   at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1085)
>   at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1085)
>   at 
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
>   at 
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
>   at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
>   at 
> org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:1084)
>   at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply$mcV$sp(PairRDDFunctions.scala:1003)
>   at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply(PairRDDFunctions.scala:994)
>   at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply(PairRDDFunctions.scala:994)
>   at 
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
>   at 
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
>   at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
>   at 
> 

[jira] [Commented] (PHOENIX-4056) java.lang.IllegalArgumentException: Can not create a Path from an empty string

[Josh Mahonin (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355485#comment-16355485
 ] 

Josh Mahonin commented on PHOENIX-4056:
---

I don't have much time to do more than a cursory review right now, 
unfortunately. Overall I like the approach, but:

a) What happens if 'mapred.output.dir' isn't set? Or is that always the case? 
org.apache.hadoop.conf.Configuration::get() will return a null in that case. Is 
Spark OK with that? Maybe another idea is to wrap it up in a scala Option, 
something like:
{{Option(conf.get('mapred.output.dir')).getOrElse("")}}

a) Any ideas on if this is backwards compatible down to Spark 2.0? It maybe an 
impossible goal, but it would be nice to stay binary compatible with the whole 
2.x range, if possible.

> java.lang.IllegalArgumentException: Can not create a Path from an empty string
> --
>
> Key: PHOENIX-4056
> URL: https://issues.apache.org/jira/browse/PHOENIX-4056
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.11.0
> Environment: CDH5.12
> Phoenix:4.11
> HBase:1.2
> Spark: 2.2.0
> phoenix-spark.version:4.11.0-HBase-1.2
>Reporter: Jepson
>Priority: Major
>  Labels: features, patch, test
> Attachments: PHOENIX-4056.patch
>
>   Original Estimate: 12h
>  Remaining Estimate: 12h
>
> 1.use the configuration of server and client(scala project)
>  
> phoenix.schema.isNamespaceMappingEnabled
> true
>   
>   
> phoenix.schema.mapSystemTablesToNamespace
> true
>   
> 2.The Code:
> {code:java}
> resultDF.write
>  .format("org.apache.phoenix.spark")
>  .mode(SaveMode.Overwrite)
>  .option("table", "JYDW.ADDRESS_ORDERCOUNT")
>  .option("zkUrl","192.168.1.40,192.168.1.41,192.168.1.42:2181")
>  .save()
> {code}
> 3.Throw this error,help to fix it,thankyou :
> 7/08/02 01:07:25 INFO DAGScheduler: Job 6 finished: runJob at 
> SparkHadoopMapReduceWriter.scala:88, took 7.990715 s
> 17/08/02 01:07:25 ERROR SparkHadoopMapReduceWriter: Aborting job 
> job_20170802010717_0079.
> {color:#59afe1}*java.lang.IllegalArgumentException: Can not create a Path 
> from an empty string*{color}
>   at org.apache.hadoop.fs.Path.checkPathArg(Path.java:126)
>   at org.apache.hadoop.fs.Path.(Path.java:134)
>   at org.apache.hadoop.fs.Path.(Path.java:88)
>   at 
> org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.absPathStagingDir(HadoopMapReduceCommitProtocol.scala:58)
>   at 
> org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.commitJob(HadoopMapReduceCommitProtocol.scala:132)
>   at 
> org.apache.spark.internal.io.SparkHadoopMapReduceWriter$.write(SparkHadoopMapReduceWriter.scala:101)
>   at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply$mcV$sp(PairRDDFunctions.scala:1085)
>   at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1085)
>   at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1085)
>   at 
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
>   at 
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
>   at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
>   at 
> org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:1084)
>   at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply$mcV$sp(PairRDDFunctions.scala:1003)
>   at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply(PairRDDFunctions.scala:994)
>   at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply(PairRDDFunctions.scala:994)
>   at 
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
>   at 
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
>   at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
>   at 
> org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:994)
>   at 
> org.apache.phoenix.spark.DataFrameFunctions.saveToPhoenix(DataFrameFunctions.scala:59)
>   at 
> org.apache.phoenix.spark.DataFrameFunctions.saveToPhoenix(DataFrameFunctions.scala:28)
>   at 
> org.apache.phoenix.spark.DefaultSource.createRelation(DefaultSource.scala:47)
>   at 
> org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:472)
>   at 
> org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:48)
>   at 
> 

Re: 4.13.0-HBase-1.1 not released?

[Xavier Jodoin]

Hi Stefan,

4.13.1-HBase-1.1 is available.

Xavier


On 2018-02-07 02:54 AM, Stepan Migunov wrote:

Hi James,
I have submitted patch on the JIRA. I've never done this before, so please
excuse me if something is wrong. Thanks,
Stepan.

-Original Message-
From: James Taylor [mailto:jamestay...@apache.org]
Sent: Monday, November 20, 2017 6:13 PM
To: dev@phoenix.apache.org
Subject: Re: 4.13.0-HBase-1.1 not released?

Hi Stepan,
Please submit a patch on the JIRA.
Thanks,
James

On Mon, Nov 20, 2017 at 1:38 AM Stepan Migunov <
stepan.migu...@firstlinesoftware.com> wrote:


Good news, thank you.

Btw, do you know if https://issues.apache.org/jira/browse/PHOENIX-4056
still unresolved? That means that Phoenix is not compatible with spark
2.2.
I see  saveToPhoenix contains the follwing line:
phxRDD.saveAsNewAPIHadoopFile("", ...). But spark 2.2 doesn't work if
path is empty.

It whould be great if this param will be changed to something like
phxRDD.saveAsNewAPIHadoopFile(conf.get("phoenix.tempPath"),), then
we could be able to set param "phoenix.tempPath" to some temp path as
workaround.

Regards,
Stepan.

On 2017-11-18 23:22, James Taylor  wrote:

FYI, we'll do one final release for Phoenix on HBase 1.1 (look for a

4.13.1

release soon). It looks like HBase 1.1 itself is nearing
end-of-life, so probably good to move off of it. If someone is
interested in being the RM for continued Phoenix HBase 1.1 releases,
please volunteer.

On Mon, Nov 13, 2017 at 10:24 AM, James R. Taylor <

jamestay...@apache.org>

wrote:


Hi Xavier,
Please see these threads for some discussion. Would be great if
you

could

volunteer to be the release manager for Phoenix released on HBase 1.1.

https://lists.apache.org/thread.html/8a73efa27edb70ea5cbc89b
43c312faefaf2b78751c9459834523b81@%3Cuser.phoenix.apache.org%3E
https://lists.apache.org/thread.html/04de7c47724d8ef2ed7414d
5bdc51325b2a0eecd324556d9e83f3718@%3Cdev.phoenix.apache.org%3E
https://lists.apache.org/thread.html/ae13def3c024603ce3cdde8
71223cbdbae0219b4efe93ed4e48f55d5@%3Cdev.phoenix.apache.org%3E

Thanks,
James

On 2017-11-13 07:51, Xavier Jodoin  wrote:

Hi,

I would like to know if there is a reason why phoenix wasn't
released for hbase 1.1?

Thanks

Xavier Jodoin

[jira] [Commented] (PHOENIX-4546) Projected table cannot be read through ProjectedColumnExpression

[Ankit Singhal (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355304#comment-16355304
 ] 

Ankit Singhal commented on PHOENIX-4546:


{quote}Is this an issue for 4.x too? 
{quote}
 yes, this is an issue with 4.x too
{quote}Is it specific to local indexes?
{quote}
No, it is not specific to local indexes, this is observed when deleting data 
from an Immutable table having multiple immutable indexes.

During delete, maintenance of immutable indexes at client executes a select 
query by adding all columns involved in all immutable indexes and projected 
table is created out of it, KeyValue schema will be built considering the order 
of columns in projected table but TupleProjectorCompiler doesn't maintain the 
order as per the SELECT projection, result in wrong ProjectedColumnExpression 
used in IndexMaintainers to read the value of the columns. 

This is observed in this case only because others use RowProjector which take 
care of indexes of the columns correctly and use the right 
ProjectedColumnExpression.

 

> Projected table cannot be read through ProjectedColumnExpression
> 
>
> Key: PHOENIX-4546
> URL: https://issues.apache.org/jira/browse/PHOENIX-4546
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Romil Choksi
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 5.0.0, 4.14.0
>
> Attachments: PHOENIX-4546.patch, PHOENIX-4546_v1.patch, 
> PHOENIX-4546_v2.patch, PHOENIX-4546_v3.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (PHOENIX-4588) Clone expression also if it's children have Determinism.PER_INVOCATION

[Ankit Singhal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ankit Singhal updated PHOENIX-4588:
---
Attachment: PHOENIX-4588.patch

> Clone expression also if it's children have Determinism.PER_INVOCATION
> --
>
> Key: PHOENIX-4588
> URL: https://issues.apache.org/jira/browse/PHOENIX-4588
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 4.14.0
>
> Attachments: PHOENIX-4588.patch
>
>
> In case of parallel UPSERT for non-aggregate SELECT, we should be cloning the 
> expression too if any of it's children needs to be cloned. Otherwise, queries 
> will fail with IllegalDataException
>  
> {code:java}
> Error: ERROR 201 (22000): Illegal data. ERROR 201 (22000): Illegal data. 
> Expected length of at least 32 bytes, but had 31 (state=22000,code=201)
> java.sql.SQLException: ERROR 201 (22000): Illegal data. ERROR 201 (22000): 
> Illegal data. Expected length of at least 32 bytes, but had 31
> at 
> org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:443)
> at 
> org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:145)
> at 
> org.apache.phoenix.util.ServerUtil.parseRemoteException(ServerUtil.java:141)
> at 
> org.apache.phoenix.util.ServerUtil.parseServerExceptionOrNull(ServerUtil.java:125)
> at 
> org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:111)
> at 
> org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:778)
> at 
> org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:719)
> at 
> org.apache.phoenix.iterate.ConcatResultIterator.getIterators(ConcatResultIterator.java:50)
> at 
> org.apache.phoenix.iterate.ConcatResultIterator.currentIterator(ConcatResultIterator.java:97)
> at 
> org.apache.phoenix.iterate.ConcatResultIterator.next(ConcatResultIterator.java:117)
> at 
> org.apache.phoenix.iterate.DelegateResultIterator.next(DelegateResultIterator.java:44)
> at 
> org.apache.phoenix.compile.UpsertCompiler$2.execute(UpsertCompiler.java:797)
> at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:343)
> at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:331)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at 
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:330)
> at 
> org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1440)
> at sqlline.Commands.execute(Commands.java:822)
> at sqlline.Commands.sql(Commands.java:732)
> at sqlline.SqlLine.dispatch(SqlLine.java:808)
> at sqlline.SqlLine.begin(SqlLine.java:681)
> at sqlline.SqlLine.start(SqlLine.java:398)
> at sqlline.SqlLine.main(SqlLine.java:292)
> Caused by: java.sql.SQLException: ERROR 201 (22000): Illegal data. Expected 
> length of at least 32 bytes, but had 31
> at 
> org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:443)
> at 
> org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:145)
> at org.apache.phoenix.schema.KeyValueSchema.next(KeyValueSchema.java:213)
> at 
> org.apache.phoenix.expression.ProjectedColumnExpression.evaluate(ProjectedColumnExpression.java:115)
> at 
> org.apache.phoenix.expression.CoerceExpression.evaluate(CoerceExpression.java:148)
> at 
> org.apache.phoenix.compile.ExpressionProjector.getValue(ExpressionProjector.java:69)
> at 
> org.apache.phoenix.jdbc.PhoenixResultSet.getObject(PhoenixResultSet.java:524)
> at 
> org.apache.phoenix.compile.UpsertCompiler.upsertSelect(UpsertCompiler.java:194)
> at 
> org.apache.phoenix.compile.UpsertCompiler.access$000(UpsertCompiler.java:110)
> at 
> org.apache.phoenix.compile.UpsertCompiler$UpsertingParallelIteratorFactory.mutate(UpsertCompiler.java:254)
> at 
> org.apache.phoenix.compile.MutatingParallelIteratorFactory.newIterator(MutatingParallelIteratorFactory.java:59)
> at 
> org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:115)
> at 
> org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:106)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748){code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4588) Clone expression also if it's children have Determinism.PER_INVOCATION

[Ankit Singhal (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355180#comment-16355180
 ] 

Ankit Singhal commented on PHOENIX-4588:


Attaching the fix and verified on the cluster but It's getting difficult to 
create a unit test case for the same.

> Clone expression also if it's children have Determinism.PER_INVOCATION
> --
>
> Key: PHOENIX-4588
> URL: https://issues.apache.org/jira/browse/PHOENIX-4588
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 4.14.0
>
>
> In case of parallel UPSERT for non-aggregate SELECT, we should be cloning the 
> expression too if any of it's children needs to be cloned. Otherwise, queries 
> will fail with IllegalDataException
>  
> {code:java}
> Error: ERROR 201 (22000): Illegal data. ERROR 201 (22000): Illegal data. 
> Expected length of at least 32 bytes, but had 31 (state=22000,code=201)
> java.sql.SQLException: ERROR 201 (22000): Illegal data. ERROR 201 (22000): 
> Illegal data. Expected length of at least 32 bytes, but had 31
> at 
> org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:443)
> at 
> org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:145)
> at 
> org.apache.phoenix.util.ServerUtil.parseRemoteException(ServerUtil.java:141)
> at 
> org.apache.phoenix.util.ServerUtil.parseServerExceptionOrNull(ServerUtil.java:125)
> at 
> org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:111)
> at 
> org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:778)
> at 
> org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:719)
> at 
> org.apache.phoenix.iterate.ConcatResultIterator.getIterators(ConcatResultIterator.java:50)
> at 
> org.apache.phoenix.iterate.ConcatResultIterator.currentIterator(ConcatResultIterator.java:97)
> at 
> org.apache.phoenix.iterate.ConcatResultIterator.next(ConcatResultIterator.java:117)
> at 
> org.apache.phoenix.iterate.DelegateResultIterator.next(DelegateResultIterator.java:44)
> at 
> org.apache.phoenix.compile.UpsertCompiler$2.execute(UpsertCompiler.java:797)
> at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:343)
> at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:331)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at 
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:330)
> at 
> org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1440)
> at sqlline.Commands.execute(Commands.java:822)
> at sqlline.Commands.sql(Commands.java:732)
> at sqlline.SqlLine.dispatch(SqlLine.java:808)
> at sqlline.SqlLine.begin(SqlLine.java:681)
> at sqlline.SqlLine.start(SqlLine.java:398)
> at sqlline.SqlLine.main(SqlLine.java:292)
> Caused by: java.sql.SQLException: ERROR 201 (22000): Illegal data. Expected 
> length of at least 32 bytes, but had 31
> at 
> org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:443)
> at 
> org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:145)
> at org.apache.phoenix.schema.KeyValueSchema.next(KeyValueSchema.java:213)
> at 
> org.apache.phoenix.expression.ProjectedColumnExpression.evaluate(ProjectedColumnExpression.java:115)
> at 
> org.apache.phoenix.expression.CoerceExpression.evaluate(CoerceExpression.java:148)
> at 
> org.apache.phoenix.compile.ExpressionProjector.getValue(ExpressionProjector.java:69)
> at 
> org.apache.phoenix.jdbc.PhoenixResultSet.getObject(PhoenixResultSet.java:524)
> at 
> org.apache.phoenix.compile.UpsertCompiler.upsertSelect(UpsertCompiler.java:194)
> at 
> org.apache.phoenix.compile.UpsertCompiler.access$000(UpsertCompiler.java:110)
> at 
> org.apache.phoenix.compile.UpsertCompiler$UpsertingParallelIteratorFactory.mutate(UpsertCompiler.java:254)
> at 
> org.apache.phoenix.compile.MutatingParallelIteratorFactory.newIterator(MutatingParallelIteratorFactory.java:59)
> at 
> org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:115)
> at 
> org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:106)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748){code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (PHOENIX-4588) Clone expression also if it's children have Determinism.PER_INVOCATION

[Ankit Singhal (JIRA)]

Ankit Singhal created PHOENIX-4588:
--

 Summary: Clone expression also if it's children have 
Determinism.PER_INVOCATION
 Key: PHOENIX-4588
 URL: https://issues.apache.org/jira/browse/PHOENIX-4588
 Project: Phoenix
  Issue Type: Bug
Reporter: Ankit Singhal
Assignee: Ankit Singhal
 Fix For: 4.14.0


In case of parallel UPSERT for non-aggregate SELECT, we should be cloning the 
expression too if any of it's children needs to be cloned. Otherwise, queries 
will fail with IllegalDataException

 
{code:java}

Error: ERROR 201 (22000): Illegal data. ERROR 201 (22000): Illegal data. 
Expected length of at least 32 bytes, but had 31 (state=22000,code=201)
java.sql.SQLException: ERROR 201 (22000): Illegal data. ERROR 201 (22000): 
Illegal data. Expected length of at least 32 bytes, but had 31
at 
org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:443)
at 
org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:145)
at org.apache.phoenix.util.ServerUtil.parseRemoteException(ServerUtil.java:141)
at 
org.apache.phoenix.util.ServerUtil.parseServerExceptionOrNull(ServerUtil.java:125)
at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:111)
at 
org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:778)
at 
org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:719)
at 
org.apache.phoenix.iterate.ConcatResultIterator.getIterators(ConcatResultIterator.java:50)
at 
org.apache.phoenix.iterate.ConcatResultIterator.currentIterator(ConcatResultIterator.java:97)
at 
org.apache.phoenix.iterate.ConcatResultIterator.next(ConcatResultIterator.java:117)
at 
org.apache.phoenix.iterate.DelegateResultIterator.next(DelegateResultIterator.java:44)
at org.apache.phoenix.compile.UpsertCompiler$2.execute(UpsertCompiler.java:797)
at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:343)
at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:331)
at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
at 
org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:330)
at org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1440)
at sqlline.Commands.execute(Commands.java:822)
at sqlline.Commands.sql(Commands.java:732)
at sqlline.SqlLine.dispatch(SqlLine.java:808)
at sqlline.SqlLine.begin(SqlLine.java:681)
at sqlline.SqlLine.start(SqlLine.java:398)
at sqlline.SqlLine.main(SqlLine.java:292)
Caused by: java.sql.SQLException: ERROR 201 (22000): Illegal data. Expected 
length of at least 32 bytes, but had 31
at 
org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:443)
at 
org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:145)
at org.apache.phoenix.schema.KeyValueSchema.next(KeyValueSchema.java:213)
at 
org.apache.phoenix.expression.ProjectedColumnExpression.evaluate(ProjectedColumnExpression.java:115)
at 
org.apache.phoenix.expression.CoerceExpression.evaluate(CoerceExpression.java:148)
at 
org.apache.phoenix.compile.ExpressionProjector.getValue(ExpressionProjector.java:69)
at org.apache.phoenix.jdbc.PhoenixResultSet.getObject(PhoenixResultSet.java:524)
at 
org.apache.phoenix.compile.UpsertCompiler.upsertSelect(UpsertCompiler.java:194)
at org.apache.phoenix.compile.UpsertCompiler.access$000(UpsertCompiler.java:110)
at 
org.apache.phoenix.compile.UpsertCompiler$UpsertingParallelIteratorFactory.mutate(UpsertCompiler.java:254)
at 
org.apache.phoenix.compile.MutatingParallelIteratorFactory.newIterator(MutatingParallelIteratorFactory.java:59)
at 
org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:115)
at 
org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:106)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748){code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4278) Implement pure client side transactional index maintenance

[Ohad Shacham (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355152#comment-16355152
 ] 

Ohad Shacham commented on PHOENIX-4278:
---

Thanks, I will try jdk 7 then. On another case, I used lambda expressions in 
Omid , which was introduced in 8, I will discard it to be 7 compatible.

 

> Implement pure client side transactional index maintenance
> --
>
> Key: PHOENIX-4278
> URL: https://issues.apache.org/jira/browse/PHOENIX-4278
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: James Taylor
>Assignee: Ohad Shacham
>Priority: Major
>
> The index maintenance for transactions follows the same model as non 
> transactional tables - coprocessor based on data table updates that looks up 
> previous row value to perform maintenance. This is necessary for non 
> transactional tables to ensure the rows are locked so that a consistent view 
> may be obtained. However, for transactional tables, the time stamp oracle 
> ensures uniqueness of time stamps (via transaction IDs) and the filtering 
> handles a scan seeing the "true" last committed value for a row. Thus, 
> there's no hard dependency to perform this on the server side.
> Moving the index maintenance to the client side would prevent any RS->RS RPC 
> calls (which have proved to be troublesome for HBase). It would require 
> returning more data to the client (i.e. the prior row value), but this seems 
> like a reasonable tradeoff.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4056) java.lang.IllegalArgumentException: Can not create a Path from an empty string

[James Taylor (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355106#comment-16355106
 ] 

James Taylor commented on PHOENIX-4056:
---

[~jmahonin] - would you some free cycles to review and commit if it looks ok? 
Any b/w compat issues to worry about?

> java.lang.IllegalArgumentException: Can not create a Path from an empty string
> --
>
> Key: PHOENIX-4056
> URL: https://issues.apache.org/jira/browse/PHOENIX-4056
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.11.0
> Environment: CDH5.12
> Phoenix:4.11
> HBase:1.2
> Spark: 2.2.0
> phoenix-spark.version:4.11.0-HBase-1.2
>Reporter: Jepson
>Priority: Major
>  Labels: features, patch, test
> Attachments: PHOENIX-4056.patch
>
>   Original Estimate: 12h
>  Remaining Estimate: 12h
>
> 1.use the configuration of server and client(scala project)
>  
> phoenix.schema.isNamespaceMappingEnabled
> true
>   
>   
> phoenix.schema.mapSystemTablesToNamespace
> true
>   
> 2.The Code:
> {code:java}
> resultDF.write
>  .format("org.apache.phoenix.spark")
>  .mode(SaveMode.Overwrite)
>  .option("table", "JYDW.ADDRESS_ORDERCOUNT")
>  .option("zkUrl","192.168.1.40,192.168.1.41,192.168.1.42:2181")
>  .save()
> {code}
> 3.Throw this error,help to fix it,thankyou :
> 7/08/02 01:07:25 INFO DAGScheduler: Job 6 finished: runJob at 
> SparkHadoopMapReduceWriter.scala:88, took 7.990715 s
> 17/08/02 01:07:25 ERROR SparkHadoopMapReduceWriter: Aborting job 
> job_20170802010717_0079.
> {color:#59afe1}*java.lang.IllegalArgumentException: Can not create a Path 
> from an empty string*{color}
>   at org.apache.hadoop.fs.Path.checkPathArg(Path.java:126)
>   at org.apache.hadoop.fs.Path.(Path.java:134)
>   at org.apache.hadoop.fs.Path.(Path.java:88)
>   at 
> org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.absPathStagingDir(HadoopMapReduceCommitProtocol.scala:58)
>   at 
> org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.commitJob(HadoopMapReduceCommitProtocol.scala:132)
>   at 
> org.apache.spark.internal.io.SparkHadoopMapReduceWriter$.write(SparkHadoopMapReduceWriter.scala:101)
>   at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply$mcV$sp(PairRDDFunctions.scala:1085)
>   at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1085)
>   at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1085)
>   at 
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
>   at 
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
>   at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
>   at 
> org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:1084)
>   at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply$mcV$sp(PairRDDFunctions.scala:1003)
>   at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply(PairRDDFunctions.scala:994)
>   at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply(PairRDDFunctions.scala:994)
>   at 
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
>   at 
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
>   at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
>   at 
> org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:994)
>   at 
> org.apache.phoenix.spark.DataFrameFunctions.saveToPhoenix(DataFrameFunctions.scala:59)
>   at 
> org.apache.phoenix.spark.DataFrameFunctions.saveToPhoenix(DataFrameFunctions.scala:28)
>   at 
> org.apache.phoenix.spark.DefaultSource.createRelation(DefaultSource.scala:47)
>   at 
> org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:472)
>   at 
> org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:48)
>   at 
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58)
>   at 
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56)
>   at 
> org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)
>   at 
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
>   at 
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
>   at 
>