[jira] [Commented] (PHOENIX-1035) Performance test local indexing against global indexing
[ https://issues.apache.org/jira/browse/PHOENIX-1035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977736#comment-15977736 ] James Taylor commented on PHOENIX-1035: --- We need to test reads for a variety of queries on a large cluster against a large table (2k regions) with a variety of client thread pool sizes (20 - 1000). [~lhofhansl] > Performance test local indexing against global indexing > --- > > Key: PHOENIX-1035 > URL: https://issues.apache.org/jira/browse/PHOENIX-1035 > Project: Phoenix > Issue Type: Sub-task >Reporter: James Taylor >Assignee: Mujtaba Chohan > > Local indexing is close to being in. We should perf test it both for reads > and writes against global secondary indexing. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (PHOENIX-1035) Performance test local indexing against global indexing
[ https://issues.apache.org/jira/browse/PHOENIX-1035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977637#comment-15977637 ] Lars Hofhansl commented on PHOENIX-1035: Is this still an issue? Close? > Performance test local indexing against global indexing > --- > > Key: PHOENIX-1035 > URL: https://issues.apache.org/jira/browse/PHOENIX-1035 > Project: Phoenix > Issue Type: Sub-task >Reporter: James Taylor >Assignee: Mujtaba Chohan > > Local indexing is close to being in. We should perf test it both for reads > and writes against global secondary indexing. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (PHOENIX-1035) Performance test local indexing against global indexing
[ https://issues.apache.org/jira/browse/PHOENIX-1035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314826#comment-14314826 ] James Taylor commented on PHOENIX-1035: --- [~mujtabachohan] - anything left to do here? Performance test local indexing against global indexing --- Key: PHOENIX-1035 URL: https://issues.apache.org/jira/browse/PHOENIX-1035 Project: Phoenix Issue Type: Sub-task Reporter: James Taylor Assignee: Mujtaba Chohan Local indexing is close to being in. We should perf test it both for reads and writes against global secondary indexing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-1035) Performance test local indexing against global indexing
[
https://issues.apache.org/jira/browse/PHOENIX-1035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14241275#comment-14241275
]
rajeshbabu commented on PHOENIX-1035:
-
[~mujtabachohan]
I have setup pseudo cluster and loaded the data. Then when I ran count query
it's not taking much time.
{code}
0: jdbc:phoenix:localhost explain select count(*) from
LOCAL_INDEXED_TABLE_WIDE;
+--+
| PLAN |
+--+
| CLIENT 32-CHUNK PARALLEL 32-WAY RANGE SCAN OVER
_LOCAL_IDX_LOCAL_INDEXED_TABLE_WIDE [-32768] |
| SERVER AGGREGATE INTO SINGLE ROW |
+--+
2 rows selected (0.089 seconds)
0: jdbc:phoenix:localhost select count(*) from LOCAL_INDEXED_TABLE_WIDE;
+--+
| COUNT(1) |
+--+
| 300 |
+--+
1 row selected (20.688 seconds)
0: jdbc:phoenix:localhost select /*NO_INDEX*/count(*) from
LOCAL_INDEXED_TABLE_WIDE;
+--+
| COUNT(1) |
+--+
| 300 |
+--+
{code}
When splitting is in progress I ran explain query then it's going to data table
as for the fix.
{code}
0: jdbc:phoenix:localhost explain select count(*) from
LOCAL_INDEXED_TABLE_WIDE;
+--+
| PLAN |
+--+
| CLIENT 8-CHUNK PARALLEL 1-WAY FULL SCAN OVER LOCAL_INDEXED_TABLE_WIDE |
| SERVER AGGREGATE INTO SINGLE ROW |
+--+
2 rows selected (0.064 seconds)
{code}
Am I missing anything?
Performance test local indexing against global indexing
---
Key: PHOENIX-1035
URL: https://issues.apache.org/jira/browse/PHOENIX-1035
Project: Phoenix
Issue Type: Sub-task
Reporter: James Taylor
Assignee: Mujtaba Chohan
Local indexing is close to being in. We should perf test it both for reads
and writes against global secondary indexing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-1035) Performance test local indexing against global indexing
[
https://issues.apache.org/jira/browse/PHOENIX-1035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14241465#comment-14241465
]
Mujtaba Chohan commented on PHOENIX-1035:
-
[~rajeshbabu] There is + sign missing in no index hint query. Try this and the
warm query that is not using local index will be under a second compared to
query that is using local index.
{code}
select /*+NO_INDEX*/ count(*) from LOCAL_INDEXED_TABLE_WIDE;
{code}
Performance test local indexing against global indexing
---
Key: PHOENIX-1035
URL: https://issues.apache.org/jira/browse/PHOENIX-1035
Project: Phoenix
Issue Type: Sub-task
Reporter: James Taylor
Assignee: Mujtaba Chohan
Local indexing is close to being in. We should perf test it both for reads
and writes against global secondary indexing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-1035) Performance test local indexing against global indexing
[ https://issues.apache.org/jira/browse/PHOENIX-1035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14239766#comment-14239766 ] rajeshbabu commented on PHOENIX-1035: - Thanks [~mujtabachohan] tonight or tomorrow I will test and try to find the root cause for this. Performance test local indexing against global indexing --- Key: PHOENIX-1035 URL: https://issues.apache.org/jira/browse/PHOENIX-1035 Project: Phoenix Issue Type: Sub-task Reporter: James Taylor Assignee: Mujtaba Chohan Local indexing is close to being in. We should perf test it both for reads and writes against global secondary indexing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-1035) Performance test local indexing against global indexing
[ https://issues.apache.org/jira/browse/PHOENIX-1035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14239795#comment-14239795 ] Mujtaba Chohan commented on PHOENIX-1035: - [~rajesh23] Here is the CSV for LOCAL_INDEXED_TABLE_WIDE if you want to test over the same data http://1drv.ms/1ugF57z Performance test local indexing against global indexing --- Key: PHOENIX-1035 URL: https://issues.apache.org/jira/browse/PHOENIX-1035 Project: Phoenix Issue Type: Sub-task Reporter: James Taylor Assignee: Mujtaba Chohan Local indexing is close to being in. We should perf test it both for reads and writes against global secondary indexing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-1035) Performance test local indexing against global indexing
[ https://issues.apache.org/jira/browse/PHOENIX-1035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14238173#comment-14238173 ] Mujtaba Chohan commented on PHOENIX-1035: - Sure. I'll test this out and will let you know results soon. Performance test local indexing against global indexing --- Key: PHOENIX-1035 URL: https://issues.apache.org/jira/browse/PHOENIX-1035 Project: Phoenix Issue Type: Sub-task Reporter: James Taylor Assignee: Mujtaba Chohan Local indexing is close to being in. We should perf test it both for reads and writes against global secondary indexing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-1035) Performance test local indexing against global indexing
[
https://issues.apache.org/jira/browse/PHOENIX-1035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14238677#comment-14238677
]
Mujtaba Chohan commented on PHOENIX-1035:
-
[~jamestaylor] Performance issue is still there with local index. Performance
is ~50X slower for count(*) query compared to mutable index and ~25X slower a
min or two after data load.
Here is the schema I used for testing with 3M rows upserted.
{code}
CREATE TABLE LOCAL_INDEXED_TABLE_WIDE (HOST CHAR(2) NOT NULL,DOMAIN CHAR(200)
NOT NULL,
FEATURE VARCHAR NOT NULL,DATE DATE NOT NULL,USAGE.CORE BIGINT,USAGE.DB
BIGINT,STATS.ACTIVE_VISITOR INTEGER, STATS.NONINDEXED
INTEGER CONSTRAINT PK PRIMARY KEY (HOST, DOMAIN, FEATURE, DATE))
MAX_FILESIZE=30485760;
CREATE LOCAL INDEX lidx1 ON LOCAL_INDEXED_TABLE_WIDE (CORE);
CREATE LOCAL INDEX lidx2 ON LOCAL_INDEXED_TABLE_WIDE (DB);
CREATE LOCAL INDEX lidx3 ON LOCAL_INDEXED_TABLE_WIDE (DB,ACTIVE_VISITOR);
CREATE LOCAL INDEX lidx4 ON LOCAL_INDEXED_TABLE_WIDE (CORE,DB,ACTIVE_VISITOR);
{code}
Performance numbers with Local Index:
{code}
select count(*) from LOCAL_INDEXED_TABLE_WIDE - query time:[33645ms]
EXPLAIN: CLIENT 102-CHUNK PARALLEL 32-WAY RANGE SCAN OVER
_LOCAL_IDX_LOCAL_INDEXED_TABLE_WIDE [-32768]
SERVER AGGREGATE INTO SINGLE ROW
{code}
Performance numbers with Mutable Index:
{code}
select count(*) from MUTABLE_INDEXED_TABLE_WIDE - query time:[582ms]
EXPLAIN: CLIENT 107-CHUNK PARALLEL 1-WAY FULL SCAN OVER MIDX1
SERVER AGGREGATE INTO SINGLE ROW
{code}
2. Another issue is that local index table did not end up on the same region
server as the base table.
HBase v0.98.7-hadoop2
Phoenix v4.2.2 (commit hash: 5c6fc2f02d01805255fff335abb675ece07d07d0)
Performance test local indexing against global indexing
---
Key: PHOENIX-1035
URL: https://issues.apache.org/jira/browse/PHOENIX-1035
Project: Phoenix
Issue Type: Sub-task
Reporter: James Taylor
Assignee: Mujtaba Chohan
Local indexing is close to being in. We should perf test it both for reads
and writes against global secondary indexing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-1035) Performance test local indexing against global indexing
[ https://issues.apache.org/jira/browse/PHOENIX-1035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14238833#comment-14238833 ] James Taylor commented on PHOENIX-1035: --- [~rajesh23] Performance test local indexing against global indexing --- Key: PHOENIX-1035 URL: https://issues.apache.org/jira/browse/PHOENIX-1035 Project: Phoenix Issue Type: Sub-task Reporter: James Taylor Assignee: Mujtaba Chohan Local indexing is close to being in. We should perf test it both for reads and writes against global secondary indexing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-1035) Performance test local indexing against global indexing
[ https://issues.apache.org/jira/browse/PHOENIX-1035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14238847#comment-14238847 ] rajeshbabu commented on PHOENIX-1035: - [~mujtabachohan] Can you just check how much time it is taking scanning from main table after load(with NO_INDEX hint)? Just want to see how much time it's taking. Thanks. Performance test local indexing against global indexing --- Key: PHOENIX-1035 URL: https://issues.apache.org/jira/browse/PHOENIX-1035 Project: Phoenix Issue Type: Sub-task Reporter: James Taylor Assignee: Mujtaba Chohan Local indexing is close to being in. We should perf test it both for reads and writes against global secondary indexing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-1035) Performance test local indexing against global indexing
[ https://issues.apache.org/jira/browse/PHOENIX-1035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14238861#comment-14238861 ] Anoop Sam John commented on PHOENIX-1035: - Any chance to test when only one index is there over the table LOCAL_INDEXED_TABLE_WIDE (?) Performance test local indexing against global indexing --- Key: PHOENIX-1035 URL: https://issues.apache.org/jira/browse/PHOENIX-1035 Project: Phoenix Issue Type: Sub-task Reporter: James Taylor Assignee: Mujtaba Chohan Local indexing is close to being in. We should perf test it both for reads and writes against global secondary indexing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-1035) Performance test local indexing against global indexing
[ https://issues.apache.org/jira/browse/PHOENIX-1035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14238871#comment-14238871 ] James Taylor commented on PHOENIX-1035: --- I'm sure that's doable. [~rajesh23] - you can use the bin/performance.py script to generate 3M rows very quickly. Then you can add these indexes and run these queries to get to the bottom of it. Performance test local indexing against global indexing --- Key: PHOENIX-1035 URL: https://issues.apache.org/jira/browse/PHOENIX-1035 Project: Phoenix Issue Type: Sub-task Reporter: James Taylor Assignee: Mujtaba Chohan Local indexing is close to being in. We should perf test it both for reads and writes against global secondary indexing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
