date:20180716

[jira] [Commented] (PHOENIX-4797) file not found or file exist exception when create global index use -snaopshot option

2018-07-16 Thread sailingYang (JIRA)



[ 
https://issues.apache.org/jira/browse/PHOENIX-4797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16545989#comment-16545989
 ] 

sailingYang commented on PHOENIX-4797:
--

[~jamestaylor] hi，my pull request in github had been approved by 
[~karanmehta93], when this pull request can be merge. Do I need to ask other 
committer?

> file not found or file exist exception when create global index use 
> -snaopshot option
> -
>
> Key: PHOENIX-4797
> URL: https://issues.apache.org/jira/browse/PHOENIX-4797
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.13.2-cdh5.11.2
>Reporter: sailingYang
>Priority: Major
>
> when use indextool with -snapshot option and if the mapreduce create multi 
> mapper.this will cause the hdfs file not found or  hdfs file exist 
> exception。finally the mapreduce task must be failed. because the mapper use 
> the same restore work dir.
> {code:java}
> Error: java.io.IOException: java.util.concurrent.ExecutionException: 
> java.io.IOException: The specified region already exists on disk: 
> hdfs://m12v1.mlamp.cn:8020/tmp/index-snapshot-dir/restore-dir/e738c85b-2394-43fc-b9de-b8280bc329ca/data/default/SCOPA.CETUS_EVENT_ZY_SCOPA_31_0516_TRAIN_EVENT/2ab2c1d73d2e31bb5a5e2b394da564f8
> at 
> org.apache.hadoop.hbase.util.ModifyRegionUtils.createRegions(ModifyRegionUtils.java:186)
> at 
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.cloneHdfsRegions(RestoreSnapshotHelper.java:578)
> at 
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:249)
> at 
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:171)
> at 
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.copySnapshotForScanner(RestoreSnapshotHelper.java:814)
> at 
> org.apache.phoenix.iterate.TableSnapshotResultIterator.init(TableSnapshotResultIterator.java:77)
> at 
> org.apache.phoenix.iterate.TableSnapshotResultIterator.(TableSnapshotResultIterator.java:73)
> at 
> org.apache.phoenix.mapreduce.PhoenixRecordReader.initialize(PhoenixRecordReader.java:126)
> at 
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:548)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:786)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.util.concurrent.ExecutionException: java.io.IOException: The 
> specified region already exists on disk: 
> hdfs://m12v1.mlamp.cn:8020/tmp/index-snapshot-dir/restore-dir/e738c85b-2394-43fc-b9de-b8280bc329ca/data/default/SCOPA.CETUS_EVENT_ZY_SCOPA_31_0516_TRAIN_EVENT/2ab2c1d73d2e31bb5a5e2b394da564f8
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:188)
> at 
> org.apache.hadoop.hbase.util.ModifyRegionUtils.createRegions(ModifyRegionUtils.java:180)
> ... 15 more
> Caused by: java.io.IOException: The specified region already exists on disk: 
> hdfs://m12v1.mlamp.cn:8020/tmp/index-snapshot-dir/restore-dir/e738c85b-2394-43fc-b9de-b8280bc329ca/data/default/SCOPA.CETUS_EVENT_ZY_SCOPA_31_0516_TRAIN_EVENT/2ab2c1d73d2e31bb5a5e2b394da564f8
> at 
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.createRegionOnFileSystem(HRegionFileSystem.java:877)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.createHRegion(HRegion.java:6252)
> at 
> org.apache.hadoop.hbase.util.ModifyRegionUtils.createRegion(ModifyRegionUtils.java:205)
> at 
> org.apache.hadoop.hbase.util.ModifyRegionUtils$1.call(ModifyRegionUtils.java:173)
> at 
> org.apache.hadoop.hbase.util.ModifyRegionUtils$1.call(ModifyRegionUtils.java:170)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 2018-06-28 15:01:55 70909 [main] INFO org.apache.hadoop.mapreduce.Job - Task 
> Id : attempt_1530004808977_0011_m_01_0, Status : FAILED
> Error: java.io.IOException: java.util.concurrent.ExecutionException: 
> java.io.IOException: The specified region already exists on disk: 
>

[jira] [Commented] (PHOENIX-2405) Improve performance and stability of server side sort for ORDER BY

2018-07-16 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/PHOENIX-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16545926#comment-16545926
 ] 

ASF GitHub Bot commented on PHOENIX-2405:
-

Github user solzy commented on the issue:

https://github.com/apache/phoenix/pull/308
  
*@JamesRTaylor * Thanks for your clarify.



   Yun Zhang
   Best regards!


2018-07-17 6:18 GMT+08:00 James Taylor :

> *@JamesRTaylor* commented on this pull request.
> --
>
> In phoenix-core/src/main/java/org/apache/phoenix/iterate/
> ClientHashAggregatingResultIterator.java
> :
>
> > +protected ImmutableBytesWritable getGroupingKey(Tuple tuple, 
ImmutableBytesWritable ptr) throws SQLException {
> +try {
> +ImmutableBytesWritable key = 
TupleUtil.getConcatenatedValue(tuple, groupByExpressions);
> +ptr.set(key.get(), key.getOffset(), key.getLength());
> +return ptr;
> +} catch (IOException e) {
> +throw new SQLException(e);
> +}
> +}
> +
> +// Copied from ClientGroupedAggregatingResultIterator
> +protected Tuple wrapKeyValueAsResult(KeyValue keyValue) {
> +return new MultiKeyValueTuple(Collections. 
singletonList(keyValue));
> +}
> +
> +private void populateHash() throws SQLException {
>
> @geraldss  - memory management is tracked by
> the GlobalMemoryManager. Operations that potentially use memory allocate
> (and eventually free) a set of MemoryChunk instances. You can see an
> example of this in GroupedAggregateRegionObserver (the runtime code for
> aggregation). If the memory used goes over a threshold 
(phoenix.query.maxGlobalMemoryPercentage
> and phoenix.query.maxTenantMemoryPercentage as the allowed percentage of
> Java heap across all queries that is allowed to be used), then the query
> will fail. Most typically, this mechanism is used on the server side as we
> don't typically use a lot of memory on the client-side (as we're mostly
> doing merge joins). One example where we use this on the client side is 
for
> our broadcast join implementation (see HashCacheClient) to track memory
> held onto for Hash Join caches.
>
> Classes you may want to look at (or perhaps you already have?):
> OrderedResultIterator and MappedByteBufferSortedQueue. Above a certain
> configurable threshold (phoenix.query.spoolThresholdBytes defaults to
> 20MB), we output results into memory mapped files. Have you tried
> decreasing that threshold?
>
> Couple of JIRAs you may want to take a look at: PHOENIX-2405 (unclear if
> this is still an issue) and PHOENIX-3289. Are you running into issues with
> too many memory mapped files?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



> Improve performance and stability of server side sort for ORDER BY
> --
>
> Key: PHOENIX-2405
> URL: https://issues.apache.org/jira/browse/PHOENIX-2405
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Assignee: Haoran Zhang
>Priority: Major
>  Labels: gsoc2016
>
> We currently use memory mapped files to buffer data as it's being sorted in 
> an ORDER BY (see MappedByteBufferQueue). The following types of exceptions 
> have been seen to occur:
> {code}
> Caused by: java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method)
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:904)
> {code}
> [~apurtell] has read that memory mapped files are not cleaned up after very 
> well in Java:
> {quote}
> "Map failed" means the JVM ran out of virtual address space. If you search 
> around stack overflow for suggestions on what to do when your app (in this 
> case Phoenix) encounters this issue when using mapped buffers, the answers 
> tend toward manually cleaning up the mapped buffers or explicitly triggering 
> a full GC. See 
> http://stackoverflow.com/questions/8553158/prevent-outofmemory-when-using-java-nio-mappedbytebuffer
>  for example. There are apparently long standing JVM/JRE problems with 
> reclamation of mapped buffers. I think we may want to explore in Phoenix a 
> different way to achieve what the current

[GitHub] phoenix issue #308: Client-side hash aggregation

2018-07-16 Thread solzy

Github user solzy commented on the issue:

https://github.com/apache/phoenix/pull/308
  
*@JamesRTaylor * Thanks for your clarify.



   Yun Zhang
   Best regards!


2018-07-17 6:18 GMT+08:00 James Taylor :

> *@JamesRTaylor* commented on this pull request.
> --
>
> In phoenix-core/src/main/java/org/apache/phoenix/iterate/
> ClientHashAggregatingResultIterator.java
> :
>
> > +protected ImmutableBytesWritable getGroupingKey(Tuple tuple, 
ImmutableBytesWritable ptr) throws SQLException {
> +try {
> +ImmutableBytesWritable key = 
TupleUtil.getConcatenatedValue(tuple, groupByExpressions);
> +ptr.set(key.get(), key.getOffset(), key.getLength());
> +return ptr;
> +} catch (IOException e) {
> +throw new SQLException(e);
> +}
> +}
> +
> +// Copied from ClientGroupedAggregatingResultIterator
> +protected Tuple wrapKeyValueAsResult(KeyValue keyValue) {
> +return new MultiKeyValueTuple(Collections. 
singletonList(keyValue));
> +}
> +
> +private void populateHash() throws SQLException {
>
> @geraldss  - memory management is tracked by
> the GlobalMemoryManager. Operations that potentially use memory allocate
> (and eventually free) a set of MemoryChunk instances. You can see an
> example of this in GroupedAggregateRegionObserver (the runtime code for
> aggregation). If the memory used goes over a threshold 
(phoenix.query.maxGlobalMemoryPercentage
> and phoenix.query.maxTenantMemoryPercentage as the allowed percentage of
> Java heap across all queries that is allowed to be used), then the query
> will fail. Most typically, this mechanism is used on the server side as we
> don't typically use a lot of memory on the client-side (as we're mostly
> doing merge joins). One example where we use this on the client side is 
for
> our broadcast join implementation (see HashCacheClient) to track memory
> held onto for Hash Join caches.
>
> Classes you may want to look at (or perhaps you already have?):
> OrderedResultIterator and MappedByteBufferSortedQueue. Above a certain
> configurable threshold (phoenix.query.spoolThresholdBytes defaults to
> 20MB), we output results into memory mapped files. Have you tried
> decreasing that threshold?
>
> Couple of JIRAs you may want to take a look at: PHOENIX-2405 (unclear if
> this is still an issue) and PHOENIX-3289. Are you running into issues with
> too many memory mapped files?
>
> â
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



---

[jira] [Updated] (PHOENIX-4815) support alter table modify column

2018-07-16 Thread Jaanai Zhang (JIRA)



 [ 
https://issues.apache.org/jira/browse/PHOENIX-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jaanai Zhang updated PHOENIX-4815:
--
Description: 
if we want to change max length or scale of  fields of  variable length type(  
example for :varchar, char and decimal type etc),  we can not drop column to 
recreate new column when the table has massive data,  which may affects online 
service，meanwhile, it is also very expensive. so sometimes this function is 
very useful.

Taking ORACLE dialect as an reference 

{code:java}
alter table
   table_name
modify
   column_name  datatype;
{code}


  was:
if we want to change max length or scale of  fields of  variable length type(  
example for :varchar, char and decimal type etc),  we can not drop column to 
recreate new column when the table has massive data,  which may affects online 
service，meanwhile, it is also very expensive. so sometimes this function is 
very useful.

Taking ORACLE dialect as reference 

{code:java}
alter table
   table_name
modify
   column_name  datatype;
{code}



> support alter table modify column 
> --
>
> Key: PHOENIX-4815
> URL: https://issues.apache.org/jira/browse/PHOENIX-4815
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Jaanai Zhang
>Priority: Major
>
> if we want to change max length or scale of  fields of  variable length type( 
>  example for :varchar, char and decimal type etc),  we can not drop column to 
> recreate new column when the table has massive data,  which may affects 
> online service，meanwhile, it is also very expensive. so sometimes this 
> function is very useful.
> Taking ORACLE dialect as an reference 
> {code:java}
> alter table
>table_name
> modify
>column_name  datatype;
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] phoenix pull request #308: Client-side hash aggregation

2018-07-16 Thread JamesRTaylor

Github user JamesRTaylor commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/308#discussion_r202842452
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/ClientHashAggregatingResultIterator.java
 ---
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.iterate;
+
+import static org.apache.phoenix.query.QueryConstants.AGG_TIMESTAMP;
+import static org.apache.phoenix.query.QueryConstants.SINGLE_COLUMN;
+import static org.apache.phoenix.query.QueryConstants.SINGLE_COLUMN_FAMILY;
+
+import java.io.IOException;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.KeyValue;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.phoenix.expression.Expression;
+import org.apache.phoenix.expression.aggregator.Aggregator;
+import org.apache.phoenix.expression.aggregator.Aggregators;
+import org.apache.phoenix.schema.tuple.MultiKeyValueTuple;
+import org.apache.phoenix.schema.tuple.Tuple;
+import org.apache.phoenix.util.KeyValueUtil;
+import org.apache.phoenix.util.TupleUtil;
+
+/**
+ * 
+ * This class implements client-side hash aggregation in memory.
+ * Issue https://issues.apache.org/jira/browse/PHOENIX-4751.
+ * 
+ */
+public class ClientHashAggregatingResultIterator
+implements AggregatingResultIterator {
+
+private static final int HASH_AGG_INIT_SIZE = 64*1024;
+private static final byte[] UNITIALIZED_KEY_BUFFER = new byte[0];
+private final ResultIterator resultIterator;
+private final Aggregators aggregators;
+private final List groupByExpressions;
+private HashMap hash;
+private List keyList;
+private Iterator keyIterator;
+
+public ClientHashAggregatingResultIterator(ResultIterator 
resultIterator, Aggregators aggregators, List groupByExpressions) {
+if (resultIterator == null) throw new NullPointerException();
+if (aggregators == null) throw new NullPointerException();
+if (groupByExpressions == null) throw new NullPointerException();
+this.resultIterator = resultIterator;
+this.aggregators = aggregators;
+this.groupByExpressions = groupByExpressions;
+}
+
+@Override
+public Tuple next() throws SQLException {
+if (keyIterator == null) {
+populateHash();
+sortKeys();
+keyIterator = keyList.iterator();
+}
+
+if (!keyIterator.hasNext()) {
+return null;
+}
+
+ImmutableBytesWritable key = keyIterator.next();
+Aggregator[] rowAggregators = hash.get(key);
+byte[] value = aggregators.toBytes(rowAggregators);
+Tuple tuple = wrapKeyValueAsResult(KeyValueUtil.newKeyValue(key, 
SINGLE_COLUMN_FAMILY, SINGLE_COLUMN, AGG_TIMESTAMP, value, 0, value.length));
+return tuple;
+}
+
+@Override
+public void close() throws SQLException {
+keyIterator = null;
+keyList = null;
+hash = null;
+resultIterator.close();
+}
+
+@Override
+public Aggregator[] aggregate(Tuple result) {
+Aggregator[] rowAggregators = aggregators.getAggregators();
+aggregators.reset(rowAggregators);
+aggregators.aggregate(rowAggregators, result);
+return rowAggregators;
+}
+
+@Override
+public void explain(List planSteps) {
+resultIterator.explain(planSteps);
+}
+
+@Override
+public String toString() {
+return "ClientHashAggregatingResultIterator [resultIterator=" 
++

[ANNOUNCE] No more releases for Phoenix 0.98 and 1.1 branches

2018-07-16 Thread Vincent Poon

This is an announcement that 4.x-HBase-0.98 and 4.x-HBase-1.1 will no
longer be maintained, as per the vote[1].

Thanks,
The Apache Phoenix Team

[1]
http://mail-archives.apache.org/mod_mbox/phoenix-dev/201807.mbox/%3CCAHBoBrUALp0f3aSA%2BrFJ7c6mjEhgooVZyqpkRdvO9mRXE70jbw%40mail.gmail.com%3E

Re: [VOTE] stop minor releases for 0.98 and 1.1

2018-07-16 Thread Vincent Poon

The vote is now closed and passes with 5 binding +1s and no 0 or -1 votes.

Phoenix 4.x-HBase-0.98 and 4.x-HBase-1.1 are now EOL and will no longer be
maintained going forward.

On Mon, Jul 16, 2018 at 11:18 AM, Vincent Poon 
wrote:

> +1
>
> On Wed, Jul 11, 2018 at 4:59 PM, Thomas D'Silva 
> wrote:
>
>> +1
>>
>> On Tue, Jul 10, 2018 at 1:12 PM, Ankit Singhal 
>> wrote:
>>
>> > +1
>> >
>> > On Tue, Jul 10, 2018 at 12:22 PM Josh Elser  wrote:
>> >
>> > > +1
>> > >
>> > > On 7/9/18 7:03 PM, James Taylor wrote:
>> > > > +1
>> > > >
>> > > > On Mon, Jul 9, 2018 at 3:55 PM, Vincent Poon <
>> > vincent.poon...@gmail.com>
>> > > > wrote:
>> > > >
>> > > >> This is a call to vote on stopping maintenance of both the
>> > > 4.x-HBase-0.98
>> > > >> and 4.x-HBase-1.1 branches.
>> > > >> HBase 0.98 and 1.1 have been EOLed.  To commit currently, it's
>> usually
>> > > 7 or
>> > > >> more branches, which is a burden on devs.
>> > > >>
>> > > >> Vote will be open for at least 72 hours. Please vote:
>> > > >>
>> > > >> [ ] +1 approve
>> > > >> [ ] +0 no opinion
>> > > >> [ ] -1 disapprove (and reason why)
>> > > >>
>> > > >> Thanks,
>> > > >> The Apache Phoenix Team
>> > > >>
>> > > >
>> > >
>> >
>>
>
>

Re: [VOTE] stop minor releases for 0.98 and 1.1

2018-07-16 Thread Vincent Poon

+1

On Wed, Jul 11, 2018 at 4:59 PM, Thomas D'Silva 
wrote:

> +1
>
> On Tue, Jul 10, 2018 at 1:12 PM, Ankit Singhal 
> wrote:
>
> > +1
> >
> > On Tue, Jul 10, 2018 at 12:22 PM Josh Elser  wrote:
> >
> > > +1
> > >
> > > On 7/9/18 7:03 PM, James Taylor wrote:
> > > > +1
> > > >
> > > > On Mon, Jul 9, 2018 at 3:55 PM, Vincent Poon <
> > vincent.poon...@gmail.com>
> > > > wrote:
> > > >
> > > >> This is a call to vote on stopping maintenance of both the
> > > 4.x-HBase-0.98
> > > >> and 4.x-HBase-1.1 branches.
> > > >> HBase 0.98 and 1.1 have been EOLed.  To commit currently, it's
> usually
> > > 7 or
> > > >> more branches, which is a burden on devs.
> > > >>
> > > >> Vote will be open for at least 72 hours. Please vote:
> > > >>
> > > >> [ ] +1 approve
> > > >> [ ] +0 no opinion
> > > >> [ ] -1 disapprove (and reason why)
> > > >>
> > > >> Thanks,
> > > >> The Apache Phoenix Team
> > > >>
> > > >
> > >
> >
>

[jira] [Commented] (PHOENIX-3955) Ensure KEEP_DELETED_CELLS, REPLICATION_SCOPE, and TTL properties stay in sync between the physical data table and index tables

2018-07-16 Thread Chinmay Kulkarni (JIRA)



[ 
https://issues.apache.org/jira/browse/PHOENIX-3955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16545547#comment-16545547
 ] 

Chinmay Kulkarni commented on PHOENIX-3955:
---

[~jamestaylor] I definitely plan to..just been caught up with a few things at 
work. Will update this soon. Thanks!

> Ensure KEEP_DELETED_CELLS, REPLICATION_SCOPE, and TTL properties stay in sync 
> between the physical data table and index tables
> --
>
> Key: PHOENIX-3955
> URL: https://issues.apache.org/jira/browse/PHOENIX-3955
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Samarth Jain
>Assignee: Chinmay Kulkarni
>Priority: Major
>
> We need to make sure that indexes inherit the REPLICATION_SCOPE, 
> KEEP_DELETED_CELLS and TTL properties from the base table. Otherwise we can 
> run into situations where the data was removed (or not removed) from the data 
> table but was removed (or not removed) from the index. Or vice-versa. We also 
> need to make sure that any ALTER TABLE SET TTL or ALTER TABLE SET 
> KEEP_DELETED_CELLS statements propagate the properties to the indexes too.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (PHOENIX-4815) support alter table modify column

2018-07-16 Thread Jaanai Zhang (JIRA)

Jaanai Zhang created PHOENIX-4815:
-

 Summary: support alter table modify column 
 Key: PHOENIX-4815
 URL: https://issues.apache.org/jira/browse/PHOENIX-4815
 Project: Phoenix
  Issue Type: Improvement
Reporter: Jaanai Zhang


if we want to change max length or scale of  fields of  variable length type(  
example for :varchar, char and decimal type etc),  we can not drop column to 
recreate new column when the table has massive data,  which may affects online 
service，meanwhile, it is also very expensive. so sometimes this function is 
very useful.

Taking ORACLE dialect as reference 

{code:java}
alter table
   table_name
modify
   column_name  datatype;
{code}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] phoenix pull request #308: Client-side hash aggregation

2018-07-16 Thread solzy

Github user solzy commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/308#discussion_r202593990
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/iterate/ClientHashAggregatingResultIterator.java
 ---
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.iterate;
+
+import static org.apache.phoenix.query.QueryConstants.AGG_TIMESTAMP;
+import static org.apache.phoenix.query.QueryConstants.SINGLE_COLUMN;
+import static org.apache.phoenix.query.QueryConstants.SINGLE_COLUMN_FAMILY;
+
+import java.io.IOException;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.KeyValue;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.phoenix.expression.Expression;
+import org.apache.phoenix.expression.aggregator.Aggregator;
+import org.apache.phoenix.expression.aggregator.Aggregators;
+import org.apache.phoenix.schema.tuple.MultiKeyValueTuple;
+import org.apache.phoenix.schema.tuple.Tuple;
+import org.apache.phoenix.util.KeyValueUtil;
+import org.apache.phoenix.util.TupleUtil;
+
+/**
+ * 
+ * This class implements client-side hash aggregation in memory.
+ * Issue https://issues.apache.org/jira/browse/PHOENIX-4751.
+ * 
+ */
+public class ClientHashAggregatingResultIterator
+implements AggregatingResultIterator {
+
+private static final int HASH_AGG_INIT_SIZE = 64*1024;
+private static final byte[] UNITIALIZED_KEY_BUFFER = new byte[0];
+private final ResultIterator resultIterator;
+private final Aggregators aggregators;
+private final List groupByExpressions;
+private HashMap hash;
+private List keyList;
+private Iterator keyIterator;
+
+public ClientHashAggregatingResultIterator(ResultIterator 
resultIterator, Aggregators aggregators, List groupByExpressions) {
+if (resultIterator == null) throw new NullPointerException();
+if (aggregators == null) throw new NullPointerException();
+if (groupByExpressions == null) throw new NullPointerException();
+this.resultIterator = resultIterator;
+this.aggregators = aggregators;
+this.groupByExpressions = groupByExpressions;
+}
+
+@Override
+public Tuple next() throws SQLException {
+if (keyIterator == null) {
+populateHash();
+sortKeys();
+keyIterator = keyList.iterator();
+}
+
+if (!keyIterator.hasNext()) {
+return null;
+}
+
+ImmutableBytesWritable key = keyIterator.next();
+Aggregator[] rowAggregators = hash.get(key);
+byte[] value = aggregators.toBytes(rowAggregators);
+Tuple tuple = wrapKeyValueAsResult(KeyValueUtil.newKeyValue(key, 
SINGLE_COLUMN_FAMILY, SINGLE_COLUMN, AGG_TIMESTAMP, value, 0, value.length));
+return tuple;
+}
+
+@Override
+public void close() throws SQLException {
+keyIterator = null;
+keyList = null;
+hash = null;
+resultIterator.close();
+}
+
+@Override
+public Aggregator[] aggregate(Tuple result) {
+Aggregator[] rowAggregators = aggregators.getAggregators();
+aggregators.reset(rowAggregators);
+aggregators.aggregate(rowAggregators, result);
+return rowAggregators;
+}
+
+@Override
+public void explain(List planSteps) {
+resultIterator.explain(planSteps);
+}
+
+@Override
+public String toString() {
+return "ClientHashAggregatingResultIterator [resultIterator=" 
++

[jira] [Commented] (PHOENIX-4797) file not found or file exist exception when create global index use -snaopshot option

[jira] [Commented] (PHOENIX-2405) Improve performance and stability of server side sort for ORDER BY

[GitHub] phoenix issue #308: Client-side hash aggregation

[jira] [Updated] (PHOENIX-4815) support alter table modify column

[GitHub] phoenix pull request #308: Client-side hash aggregation

[ANNOUNCE] No more releases for Phoenix 0.98 and 1.1 branches

Re: [VOTE] stop minor releases for 0.98 and 1.1

Re: [VOTE] stop minor releases for 0.98 and 1.1

[jira] [Commented] (PHOENIX-3955) Ensure KEEP_DELETED_CELLS, REPLICATION_SCOPE, and TTL properties stay in sync between the physical data table and index tables

[jira] [Created] (PHOENIX-4815) support alter table modify column

[GitHub] phoenix pull request #308: Client-side hash aggregation

11 matches

Site Navigation

Mail list logo

Footer information