[jira] [Updated] (PHOENIX-3591) Group by is slower for immutable encoded table

2017-01-11 Thread Mujtaba Chohan (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PHOENIX-3591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mujtaba Chohan updated PHOENIX-3591:

Description: 
{noformat}
 select /*+ SERIAL NO_CACHE*/ c0,sum(k1) from $T group by c0
{noformat}

This is >2X slower when compared to non-encoded table. There are 100 distinct 
values in c0 column.

For data and schema see data generator script in 
https://issues.apache.org/jira/browse/PHOENIX-3560

  was:
{noformat}
 select /*+ SERIAL NO_CACHE*/ c0,sum(k1) from $T group by c0
{noformat}

There are 100 distinct values in c0 column.

For data and schema see data generator script in 
https://issues.apache.org/jira/browse/PHOENIX-3560


> Group by is slower for immutable encoded table
> --
>
> Key: PHOENIX-3591
> URL: https://issues.apache.org/jira/browse/PHOENIX-3591
> Project: Phoenix
>  Issue Type: Sub-task
>Reporter: Mujtaba Chohan
>
> {noformat}
>  select /*+ SERIAL NO_CACHE*/ c0,sum(k1) from $T group by c0
> {noformat}
> This is >2X slower when compared to non-encoded table. There are 100 distinct 
> values in c0 column.
> For data and schema see data generator script in 
> https://issues.apache.org/jira/browse/PHOENIX-3560



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (PHOENIX-3591) Group by is slower for immutable encoded table

2017-01-11 Thread Mujtaba Chohan (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PHOENIX-3591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mujtaba Chohan updated PHOENIX-3591:

Assignee: Samarth Jain

> Group by is slower for immutable encoded table
> --
>
> Key: PHOENIX-3591
> URL: https://issues.apache.org/jira/browse/PHOENIX-3591
> Project: Phoenix
>  Issue Type: Sub-task
>Reporter: Mujtaba Chohan
>Assignee: Samarth Jain
>
> {noformat}
>  select /*+ SERIAL NO_CACHE*/ c0,sum(k1) from $T group by c0
> {noformat}
> This is >2X slower when compared to non-encoded table. There are 100 distinct 
> values in c0 column.
> For data and schema see data generator script in 
> https://issues.apache.org/jira/browse/PHOENIX-3560



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (PHOENIX-3591) Group by is slower for immutable encoded table

2017-01-11 Thread Mujtaba Chohan (JIRA) 
Mujtaba Chohan created PHOENIX-3591:
---

 Summary: Group by is slower for immutable encoded table
 Key: PHOENIX-3591
 URL: https://issues.apache.org/jira/browse/PHOENIX-3591
 Project: Phoenix
  Issue Type: Sub-task
Reporter: Mujtaba Chohan


{noformat}
 select /*+ SERIAL NO_CACHE*/ c0,sum(k1) from $T group by c0
{noformat}

There are 100 distinct values in c0 column.

For data and schema see data generator script in 
https://issues.apache.org/jira/browse/PHOENIX-3560



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (PHOENIX-3590) Filter on value column for mutable encoded table is > 3X slower compared to non encoded table

2017-01-11 Thread Mujtaba Chohan (JIRA) 
Mujtaba Chohan created PHOENIX-3590:
---

 Summary: Filter on value column for mutable encoded table is > 3X 
slower compared to non encoded table
 Key: PHOENIX-3590
 URL: https://issues.apache.org/jira/browse/PHOENIX-3590
 Project: Phoenix
  Issue Type: Sub-task
Reporter: Mujtaba Chohan
Assignee: Samarth Jain


{noformat}
select /*+ SERIAL NO_CACHE*/ k2,c0,c1000,c2000,c3000 from $T where c4000='50' 
limit 1
{noformat}

Query get progressively slower if the column which is filtered is the last 
column of the table.

For data and schema see data generator script in 
https://issues.apache.org/jira/browse/PHOENIX-3560



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-3588) Update Phoenix-Spark integration tests to use Spark 2.0 features instead of 1.x deprecated methods

2017-01-11 Thread Nico Pappagianis (JIRA) 

[ 
https://issues.apache.org/jira/browse/PHOENIX-3588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819114#comment-15819114
 ] 

Nico Pappagianis commented on PHOENIX-3588:
---

[~jmahonin] Any objections to swapping out deprecated methods in the 
phoenix-spark integration tests to their 2.0 counterparts for 4.10?

> Update Phoenix-Spark integration tests to use Spark 2.0 features instead of 
> 1.x deprecated methods
> --
>
> Key: PHOENIX-3588
> URL: https://issues.apache.org/jira/browse/PHOENIX-3588
> Project: Phoenix
>  Issue Type: Improvement
>Affects Versions: 4.10.0
>Reporter: Nico Pappagianis
>Priority: Minor
>
> Update the integration test to use Spark 2.0 features like SessionBuilder.
> Swap out deprecated methods in favor of 2.0 supported methods.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (PHOENIX-3588) Update Phoenix-Spark integration tests to use Spark 2.0 features instead of 1.x deprecated methods

2017-01-11 Thread Nico Pappagianis (JIRA) 
Nico Pappagianis created PHOENIX-3588:
-

 Summary: Update Phoenix-Spark integration tests to use Spark 2.0 
features instead of 1.x deprecated methods
 Key: PHOENIX-3588
 URL: https://issues.apache.org/jira/browse/PHOENIX-3588
 Project: Phoenix
  Issue Type: Improvement
Affects Versions: 4.10.0
Reporter: Nico Pappagianis
Priority: Minor


Update the integration test to use Spark 2.0 features like SessionBuilder.
Swap out deprecated methods in favor of 2.0 supported methods.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (PHOENIX-3587) Support FETCH PREV|n ROWS from Cursor

2017-01-11 Thread Biju Nair (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PHOENIX-3587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Biju Nair updated PHOENIX-3587:
---
Summary: Support FETCH PREV|n ROWS from Cursor  (was: Support FETCH PREV|n 
ROWS from cursor)

> Support FETCH PREV|n ROWS from Cursor
> -
>
> Key: PHOENIX-3587
> URL: https://issues.apache.org/jira/browse/PHOENIX-3587
> Project: Phoenix
>  Issue Type: Sub-task
>Reporter: Biju Nair
>
> Implement required changes to support
> - query {{FETCH PREV | n ROWS}} from the cursor



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (PHOENIX-3587) Support FETCH PREV|n ROWS from cursor

2017-01-11 Thread Biju Nair (JIRA) 
Biju Nair created PHOENIX-3587:
--

 Summary: Support FETCH PREV|n ROWS from cursor
 Key: PHOENIX-3587
 URL: https://issues.apache.org/jira/browse/PHOENIX-3587
 Project: Phoenix
  Issue Type: Sub-task
Reporter: Biju Nair


Implement required changes to support
- query {{FETCH PREV | n ROWS}} from the cursor



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Re: Why SkipScanFilter can not support OrderBy.REV_ROW_KEY_ORDER_BY?

2017-01-11 Thread James Taylor 
Thanks - sounds like you're on to the root cause. Would you mind commenting
on the JIRA, please? Patches are most welcome too.
Thanks,
James

On Wed, Jan 11, 2017 at 12:46 AM 程磊  wrote:

>
>
>
>
> Thank you, James Taylor，I just found this problem when I looked into
> PHOENIX-3578，PHOENIX-3578 may be caused by the fact that the Join SQL is
> using SkipScanFilter after dynamic filtering  but the sql is also
> OrderBy.REV_ROW_KEY_ORDER_BY.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> At 2017-01-10 23:47:08, "James Taylor"  wrote:
>
> >Because no one has implemented it. It would be a welcome addition and
>
> >probably not too difficult.
>
> >Thanks,
>
> >James
>
> >
>
> >On Tue, Jan 10, 2017 at 7:00 AM 程磊  wrote:
>
> >
>
> >> Hi,when I  read the following code in OrderBy.complie method, in line
>
> >> 160,it seems that SkipScanFilter can not support
>
> >> OrderBy.REV_ROW_KEY_ORDER_BY,
>
> >>
>
> >> SkipScanFilter still could not support OrderBy.REV_ROW_KEY_ORDER_BY now?
>
> >> and why? :
>
> >>
>
> >>
>
> >>
>
> >>
>
> >>
>
> >> 155  if (isInRowKeyOrder && tracker.isOrderPreserving()) {
>
> >>
>
> >> 156if (tracker.isReverse()) {
>
> >>
>
> >> 157// Don't use reverse scan if we're using a skip scan,
>
> >> as our skip scan doesn't support this yet.
>
> >>
>
> >> 158// REV_ROW_KEY_ORDER_BY scan would not take effect
> for
>
> >> a projected table, so don't return it for such table types.
>
> >>
>
> >> 159if
>
> >>
> (context.getConnection().getQueryServices().getProps().getBoolean(QueryServices.USE_REVERSE_SCAN_ATTRIB,
>
> >> QueryServicesOptions.DEFAULT_USE_REVERSE_SCAN)
>
> >>
>
> >> 160&&
> !context.getScanRanges().useSkipScanFilter()
>
> >>
>
> >> 161&&
>
> >> context.getCurrentTable().getTable().getType() != PTableType.PROJECTED
>
> >>
>
> >> 162&&
>
> >> context.getCurrentTable().getTable().getType() != PTableType.SUBQUERY) {
>
> >>
>
> >> 163return OrderBy.REV_ROW_KEY_ORDER_BY;
>
> >>
>
> >> 164}
>
>

[jira] [Updated] (PHOENIX-3576) ParallelWriterIndexCommitter should fail fast when writing the local index to the closing table

2017-01-11 Thread ChiaPing Tsai (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PHOENIX-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ChiaPing Tsai updated PHOENIX-3576:
---
Attachment: hbase-root-regionserver-node06.ncku.edu.tw.log

This issue requires the support of HBASE-15600..
Or we need to pre-split the table for avoiding the splitting when updating the 
local index.

> ParallelWriterIndexCommitter should fail fast when writing the local index to 
> the closing table
> ---
>
> Key: PHOENIX-3576
> URL: https://issues.apache.org/jira/browse/PHOENIX-3576
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.9.0
>Reporter: ChiaPing Tsai
> Attachments: hbase-root-regionserver-node06.ncku.edu.tw.log
>
>
> ParallelWriterIndexCommitter uses the AP to update the local index after the 
> region#batchMutate fails. If the region is splitting, all later updates will 
> fail (RegionTooBusyException) due to the write lock. The AP will retry the 
> failed updates until it reaches the retry threshold, so the read lock of 
> region is released after a long time. The read lock blocks the HRegion#close, 
> so the region splitting is also blocked.
> {noformat}
> try {
> if (allowLocalUpdates
> && env != null
> && tableReference.getTableName().equals(
> 
> env.getRegion().getTableDesc().getNameAsString())) {
> try {
> throwFailureIfDone();
> IndexUtil.writeLocalUpdates(env.getRegion(), 
> mutations, true);
> return null;
> } catch (IOException ignord) {
> // when it's failed we fall back to the 
> standard & slow way
> if (LOG.isDebugEnabled()) {
> LOG.debug("indexRegion.batchMutate failed 
> and fall back to HTable.batch(). Got error="
> + ignord);
> }
> }
> }
> table = factory.getTable(tableReference.get());
> throwFailureIfDone();
> table.batch(mutations);
> } catch (SingleIndexWriteFailureException e) {
> throw e;
> } catch (IOException e) {
> throw new 
> SingleIndexWriteFailureException(tableReference.toString(), mutations, e);
> } catch (InterruptedException e) {
> // reset the interrupt status on the thread
> Thread.currentThread().interrupt();
> throw new 
> SingleIndexWriteFailureException(tableReference.toString(), mutations, e);
> }
> finally{
> if (table != null) {
> table.close();
> }
> }
> return null;
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PHOENIX-3578) Incorrect query results when applying inner join and orderby desc

2017-01-11 Thread chenglei (JIRA) 

[ 
https://issues.apache.org/jira/browse/PHOENIX-3578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15817695#comment-15817695
 ] 

chenglei commented on PHOENIX-3578:
---

It can also be reproduced under 4.9.0, it may be caused by the fact that the 
Join SQL is using SkipScanFilter after dynamic filtering but the sql is also 
OrderBy.REV_ROW_KEY_ORDER_BY.

> Incorrect query results when applying inner join and orderby desc
> -
>
> Key: PHOENIX-3578
> URL: https://issues.apache.org/jira/browse/PHOENIX-3578
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.8.0
> Environment: hbase-1.1.2
>Reporter: sungmin.cho
>
> Step to reproduce:
> h4. 1. Create two tables
> {noformat}
> CREATE TABLE IF NOT EXISTS master (
>   id integer not null,
>   col1 varchar,
>   constraint pk_master primary key(id)
> );
> CREATE TABLE IF NOT EXISTS detail (
>   id integer not null,
>   seq integer not null,
>   col2 varchar,
>   constraint pk_master primary key(id, seq)
> );
> {noformat}
> h4. 2. Upsert values
> {noformat}
> upsert into master values(1, 'A1');
> upsert into master values(2, 'A2');
> upsert into master values(3, 'A3');
> upsert into detail values(1, 1, 'B1');
> upsert into detail values(1, 2, 'B2');
> upsert into detail values(2, 1, 'B1');
> upsert into detail values(2, 2, 'B2');
> upsert into detail values(3, 1, 'B1');
> upsert into detail values(3, 2, 'B2');
> upsert into detail values(3, 3, 'B3');
> {noformat}
> h4. 3. Execute query
> {noformat}
> select m.id, m.col1, d.seq, d.col2
> from master m, detail d
> where m.id = d.id
>   and d.id between 1 and 2
> order by m.id desc
> {noformat}
> h4. (/) Expected result
> {noformat}
> +---+-++-+
> | M.ID  | M.COL1  | D.SEQ  | D.COL2  |
> +---+-++-+
> | 2 | A2  | 1  | B1  |
> | 2 | A2  | 2  | B2  |
> | 1 | A1  | 1  | B1  |
> | 1 | A1  | 2  | B2  |
> +---+-++-+
> {noformat}
> h4. (!) Incorrect result
> {noformat}
> +---+-++-+
> | M.ID  | M.COL1  | D.SEQ  | D.COL2  |
> +---+-++-+
> | 1 | A1  | 1  | B1  |
> | 1 | A1  | 2  | B2  |
> +---+-++-+
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re:Re: Why SkipScanFilter can not support OrderBy.REV_ROW_KEY_ORDER_BY?

2017-01-11 Thread 程磊 


Thank you, James Taylor，I just found this problem when I looked into 
PHOENIX-3578，PHOENIX-3578 may be caused by the fact that the Join SQL is using 
SkipScanFilter after dynamic filtering  but the sql is also 
OrderBy.REV_ROW_KEY_ORDER_BY.







At 2017-01-10 23:47:08, "James Taylor"  wrote:
>Because no one has implemented it. It would be a welcome addition and
>probably not too difficult.
>Thanks,
>James
>
>On Tue, Jan 10, 2017 at 7:00 AM 程磊  wrote:
>
>> Hi,when I  read the following code in OrderBy.complie method, in line
>> 160,it seems that SkipScanFilter can not support
>> OrderBy.REV_ROW_KEY_ORDER_BY,
>>
>> SkipScanFilter still could not support OrderBy.REV_ROW_KEY_ORDER_BY now?
>> and why? :
>>
>>
>>
>>
>>
>> 155  if (isInRowKeyOrder && tracker.isOrderPreserving()) {
>>
>> 156if (tracker.isReverse()) {
>>
>> 157// Don't use reverse scan if we're using a skip scan,
>> as our skip scan doesn't support this yet.
>>
>> 158// REV_ROW_KEY_ORDER_BY scan would not take effect for
>> a projected table, so don't return it for such table types.
>>
>> 159if
>> (context.getConnection().getQueryServices().getProps().getBoolean(QueryServices.USE_REVERSE_SCAN_ATTRIB,
>> QueryServicesOptions.DEFAULT_USE_REVERSE_SCAN)
>>
>> 160&& !context.getScanRanges().useSkipScanFilter()
>>
>> 161&&
>> context.getCurrentTable().getTable().getType() != PTableType.PROJECTED
>>
>> 162&&
>> context.getCurrentTable().getTable().getType() != PTableType.SUBQUERY) {
>>
>> 163return OrderBy.REV_ROW_KEY_ORDER_BY;
>>
>> 164}