[jira] [Commented] (PHOENIX-933) Local index support to Phoenix
[ https://issues.apache.org/jira/browse/PHOENIX-933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173135#comment-14173135 ] Hudson commented on PHOENIX-933: SUCCESS: Integrated in Phoenix | 3.0 | Hadoop1 #259 (See [https://builds.apache.org/job/Phoenix-3.0-hadoop1/259/]) Backport fixes for PHOENIX-1149 (view index ignored if no where clause) and PHOENIX-933 (IndexMaintainer fixes for view index) (jtaylor: rev abffd3fc7614ca70e040cdc4c752ee90e4b3b270) * phoenix-core/src/main/java/org/apache/phoenix/index/IndexMaintainer.java * phoenix-core/src/main/java/org/apache/phoenix/compile/WhereOptimizer.java * phoenix-core/src/it/java/org/apache/phoenix/end2end/BaseViewIT.java Local index support to Phoenix -- Key: PHOENIX-933 URL: https://issues.apache.org/jira/browse/PHOENIX-933 Project: Phoenix Issue Type: New Feature Reporter: rajeshbabu Assignee: rajeshbabu Attachments: PHOENIX-933-addendum_2.patch, PHOENIX-933.patch, PHOENIX-933_4.0.patch, PHOENIX-933_addendum.patch Hindex(https://github.com/Huawei-Hadoop/hindex) provides local indexing support to HBase. It stores region level index in a separate table, and co-locates the user and index table regions with a custom load balancer. See http://goo.gl/phkhwC and http://goo.gl/EswlxC for more information. This JIRA addresses the local indexing solution integration to phoenix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-933) Local index support to Phoenix
[ https://issues.apache.org/jira/browse/PHOENIX-933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14116438#comment-14116438 ] ASF GitHub Bot commented on PHOENIX-933: Github user asfgit closed the pull request at: https://github.com/apache/phoenix/pull/1 Local index support to Phoenix -- Key: PHOENIX-933 URL: https://issues.apache.org/jira/browse/PHOENIX-933 Project: Phoenix Issue Type: New Feature Reporter: rajeshbabu Assignee: rajeshbabu Attachments: PHOENIX-933-addendum_2.patch, PHOENIX-933.patch, PHOENIX-933_4.0.patch, PHOENIX-933_addendum.patch Hindex(https://github.com/Huawei-Hadoop/hindex) provides local indexing support to HBase. It stores region level index in a separate table, and co-locates the user and index table regions with a custom load balancer. See http://goo.gl/phkhwC and http://goo.gl/EswlxC for more information. This JIRA addresses the local indexing solution integration to phoenix. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (PHOENIX-933) Local index support to Phoenix
[ https://issues.apache.org/jira/browse/PHOENIX-933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14064594#comment-14064594 ] Anoop Sam John commented on PHOENIX-933: Committing this now. Local index support to Phoenix -- Key: PHOENIX-933 URL: https://issues.apache.org/jira/browse/PHOENIX-933 Project: Phoenix Issue Type: New Feature Reporter: rajeshbabu Assignee: rajeshbabu Attachments: PHOENIX-933-addendum_2.patch, PHOENIX-933.patch, PHOENIX-933_4.0.patch, PHOENIX-933_addendum.patch Hindex(https://github.com/Huawei-Hadoop/hindex) provides local indexing support to HBase. It stores region level index in a separate table, and co-locates the user and index table regions with a custom load balancer. See http://goo.gl/phkhwC and http://goo.gl/EswlxC for more information. This JIRA addresses the local indexing solution integration to phoenix. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (PHOENIX-933) Local index support to Phoenix
[ https://issues.apache.org/jira/browse/PHOENIX-933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14064596#comment-14064596 ] Anoop Sam John commented on PHOENIX-933: Addendum 2 committed to 4.0 and master. Thanks Rajesh Local index support to Phoenix -- Key: PHOENIX-933 URL: https://issues.apache.org/jira/browse/PHOENIX-933 Project: Phoenix Issue Type: New Feature Reporter: rajeshbabu Assignee: rajeshbabu Attachments: PHOENIX-933-addendum_2.patch, PHOENIX-933.patch, PHOENIX-933_4.0.patch, PHOENIX-933_addendum.patch Hindex(https://github.com/Huawei-Hadoop/hindex) provides local indexing support to HBase. It stores region level index in a separate table, and co-locates the user and index table regions with a custom load balancer. See http://goo.gl/phkhwC and http://goo.gl/EswlxC for more information. This JIRA addresses the local indexing solution integration to phoenix. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (PHOENIX-933) Local index support to Phoenix
[ https://issues.apache.org/jira/browse/PHOENIX-933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062376#comment-14062376 ] James Taylor commented on PHOENIX-933: -- Excellent, thanks! [~ramkrishna] - still up for committing this? Local index support to Phoenix -- Key: PHOENIX-933 URL: https://issues.apache.org/jira/browse/PHOENIX-933 Project: Phoenix Issue Type: New Feature Reporter: rajeshbabu Assignee: rajeshbabu Attachments: PHOENIX-933.patch, PHOENIX-933_4.0.patch Hindex(https://github.com/Huawei-Hadoop/hindex) provides local indexing support to HBase. It stores region level index in a separate table, and co-locates the user and index table regions with a custom load balancer. See http://goo.gl/phkhwC and http://goo.gl/EswlxC for more information. This JIRA addresses the local indexing solution integration to phoenix. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (PHOENIX-933) Local index support to Phoenix
[ https://issues.apache.org/jira/browse/PHOENIX-933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14063125#comment-14063125 ] ramkrishna.s.vasudevan commented on PHOENIX-933: Can this issue be resolved now? Local index support to Phoenix -- Key: PHOENIX-933 URL: https://issues.apache.org/jira/browse/PHOENIX-933 Project: Phoenix Issue Type: New Feature Reporter: rajeshbabu Assignee: rajeshbabu Attachments: PHOENIX-933.patch, PHOENIX-933_4.0.patch, PHOENIX-933_addendum.patch Hindex(https://github.com/Huawei-Hadoop/hindex) provides local indexing support to HBase. It stores region level index in a separate table, and co-locates the user and index table regions with a custom load balancer. See http://goo.gl/phkhwC and http://goo.gl/EswlxC for more information. This JIRA addresses the local indexing solution integration to phoenix. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (PHOENIX-933) Local index support to Phoenix
[ https://issues.apache.org/jira/browse/PHOENIX-933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14063132#comment-14063132 ] rajeshbabu commented on PHOENIX-933: [~ramkrishna] Thank you very much for committing patch. Please commit the addendum to 4.0 also. This issue has some more tasks to be handled. So can be resolved later. Local index support to Phoenix -- Key: PHOENIX-933 URL: https://issues.apache.org/jira/browse/PHOENIX-933 Project: Phoenix Issue Type: New Feature Reporter: rajeshbabu Assignee: rajeshbabu Attachments: PHOENIX-933.patch, PHOENIX-933_4.0.patch, PHOENIX-933_addendum.patch Hindex(https://github.com/Huawei-Hadoop/hindex) provides local indexing support to HBase. It stores region level index in a separate table, and co-locates the user and index table regions with a custom load balancer. See http://goo.gl/phkhwC and http://goo.gl/EswlxC for more information. This JIRA addresses the local indexing solution integration to phoenix. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (PHOENIX-933) Local index support to Phoenix
[ https://issues.apache.org/jira/browse/PHOENIX-933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14063135#comment-14063135 ] ramkrishna.s.vasudevan commented on PHOENIX-933: Committed the addendum also to 4.0. Local index support to Phoenix -- Key: PHOENIX-933 URL: https://issues.apache.org/jira/browse/PHOENIX-933 Project: Phoenix Issue Type: New Feature Reporter: rajeshbabu Assignee: rajeshbabu Attachments: PHOENIX-933.patch, PHOENIX-933_4.0.patch, PHOENIX-933_addendum.patch Hindex(https://github.com/Huawei-Hadoop/hindex) provides local indexing support to HBase. It stores region level index in a separate table, and co-locates the user and index table regions with a custom load balancer. See http://goo.gl/phkhwC and http://goo.gl/EswlxC for more information. This JIRA addresses the local indexing solution integration to phoenix. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (PHOENIX-933) Local index support to Phoenix
[ https://issues.apache.org/jira/browse/PHOENIX-933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14063139#comment-14063139 ] Hudson commented on PHOENIX-933: FAILURE: Integrated in Phoenix | Master | Hadoop1 #274 (See [https://builds.apache.org/job/Phoenix-master-hadoop1/274/]) Phoenix-933 Addendum for failing test(Rajesh via Ram) (ramkrishna: rev 4e19c8cfb2413f426b46158d3bfa0a8774157bf7) * phoenix-core/src/main/java/org/apache/phoenix/index/IndexMaintainer.java Local index support to Phoenix -- Key: PHOENIX-933 URL: https://issues.apache.org/jira/browse/PHOENIX-933 Project: Phoenix Issue Type: New Feature Reporter: rajeshbabu Assignee: rajeshbabu Attachments: PHOENIX-933.patch, PHOENIX-933_4.0.patch, PHOENIX-933_addendum.patch Hindex(https://github.com/Huawei-Hadoop/hindex) provides local indexing support to HBase. It stores region level index in a separate table, and co-locates the user and index table regions with a custom load balancer. See http://goo.gl/phkhwC and http://goo.gl/EswlxC for more information. This JIRA addresses the local indexing solution integration to phoenix. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (PHOENIX-933) Local index support to Phoenix
[
https://issues.apache.org/jira/browse/PHOENIX-933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14060544#comment-14060544
]
ASF GitHub Bot commented on PHOENIX-933:
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/3#discussion_r14873050
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/iterate/ParallelIterators.java ---
@@ -238,7 +240,12 @@ private void
doColumnProjectionOptimization(StatementContext context, Scan scan,
minMaxRange =
SaltingUtil.addSaltByte(split.getLowerRange(), minMaxRange);
split = split.intersect(minMaxRange);
}
-}
+} else if (localIndex) {
--- End diff --
In a follow up commit, we should look to commonize the above salting logic
with this logic for local indexes. It's doing the same thing, it's just adding
one byte for the salt byte while the local indexing is adding the region start
key.
Local index support to Phoenix
--
Key: PHOENIX-933
URL: https://issues.apache.org/jira/browse/PHOENIX-933
Project: Phoenix
Issue Type: New Feature
Reporter: rajeshbabu
Hindex(https://github.com/Huawei-Hadoop/hindex) provides local indexing
support to HBase. It stores region level index in a separate table, and
co-locates the user and index table regions with a custom load balancer.
See http://goo.gl/phkhwC and http://goo.gl/EswlxC for more information.
This JIRA addresses the local indexing solution integration to phoenix.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (PHOENIX-933) Local index support to Phoenix
[ https://issues.apache.org/jira/browse/PHOENIX-933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14060832#comment-14060832 ] James Taylor commented on PHOENIX-933: -- [~anoop.hbase] - would you mind committing this? The pull is here: https://github.com/apache/phoenix/pull/3 Local index support to Phoenix -- Key: PHOENIX-933 URL: https://issues.apache.org/jira/browse/PHOENIX-933 Project: Phoenix Issue Type: New Feature Reporter: rajeshbabu Hindex(https://github.com/Huawei-Hadoop/hindex) provides local indexing support to HBase. It stores region level index in a separate table, and co-locates the user and index table regions with a custom load balancer. See http://goo.gl/phkhwC and http://goo.gl/EswlxC for more information. This JIRA addresses the local indexing solution integration to phoenix. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (PHOENIX-933) Local index support to Phoenix
[ https://issues.apache.org/jira/browse/PHOENIX-933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14060868#comment-14060868 ] ASF GitHub Bot commented on PHOENIX-933: Github user ramkrish86 commented on the pull request: https://github.com/apache/phoenix/pull/3#issuecomment-48927901 I think I don't have write access to do this merge. Only those with write access to this repository can merge pull requests. Local index support to Phoenix -- Key: PHOENIX-933 URL: https://issues.apache.org/jira/browse/PHOENIX-933 Project: Phoenix Issue Type: New Feature Reporter: rajeshbabu Hindex(https://github.com/Huawei-Hadoop/hindex) provides local indexing support to HBase. It stores region level index in a separate table, and co-locates the user and index table regions with a custom load balancer. See http://goo.gl/phkhwC and http://goo.gl/EswlxC for more information. This JIRA addresses the local indexing solution integration to phoenix. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (PHOENIX-933) Local index support to Phoenix
[ https://issues.apache.org/jira/browse/PHOENIX-933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14061041#comment-14061041 ] ASF GitHub Bot commented on PHOENIX-933: Github user chrajeshbabu commented on the pull request: https://github.com/apache/phoenix/pull/3#issuecomment-48945719 bq. Just add the check to disable creating local indexes on a table with immutable rows and then let's check this in. Changed pull request to disallow local index on immutable rows and added some test cases. Local index support to Phoenix -- Key: PHOENIX-933 URL: https://issues.apache.org/jira/browse/PHOENIX-933 Project: Phoenix Issue Type: New Feature Reporter: rajeshbabu Hindex(https://github.com/Huawei-Hadoop/hindex) provides local indexing support to HBase. It stores region level index in a separate table, and co-locates the user and index table regions with a custom load balancer. See http://goo.gl/phkhwC and http://goo.gl/EswlxC for more information. This JIRA addresses the local indexing solution integration to phoenix. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (PHOENIX-933) Local index support to Phoenix
[
https://issues.apache.org/jira/browse/PHOENIX-933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14061053#comment-14061053
]
ASF GitHub Bot commented on PHOENIX-933:
Github user chrajeshbabu commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/3#discussion_r14897250
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/schema/MetaDataClient.java ---
@@ -568,8 +629,13 @@ public MutationState createIndex(CreateIndexStatement
statement, byte[][] splits
allPkColumns.add(new PairColumnName,
SortOrder(colName, col.getSortOrder()));
columnDefs.add(FACTORY.columnDef(colName,
dataType.getSqlTypeName(), col.isNullable(), col.getMaxLength(),
col.getScale(), false, SortOrder.getDefault()));
}
-if (dataTable.getType() == PTableType.VIEW
dataTable.getViewType() != ViewType.MAPPED) {
-allocateViewIndexId = true;
+/*
+ * Allocate an index ID in two circumstances:
+ * 1) for a local index, as all local indexes will reside
in the same HBase table
+ * 2) for a view on an index.
+ */
+if (statement.getIndexType() == IndexType.LOCAL ||
(dataTable.getType() == PTableType.VIEW dataTable.getViewType() !=
ViewType.MAPPED)) {
+allocateIndexId = true;
--- End diff --
Done.
Local index support to Phoenix
--
Key: PHOENIX-933
URL: https://issues.apache.org/jira/browse/PHOENIX-933
Project: Phoenix
Issue Type: New Feature
Reporter: rajeshbabu
Hindex(https://github.com/Huawei-Hadoop/hindex) provides local indexing
support to HBase. It stores region level index in a separate table, and
co-locates the user and index table regions with a custom load balancer.
See http://goo.gl/phkhwC and http://goo.gl/EswlxC for more information.
This JIRA addresses the local indexing solution integration to phoenix.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (PHOENIX-933) Local index support to Phoenix
[
https://issues.apache.org/jira/browse/PHOENIX-933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14061060#comment-14061060
]
ASF GitHub Bot commented on PHOENIX-933:
Github user chrajeshbabu commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/3#discussion_r14897277
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/compile/IndexStatementRewriter.java
---
@@ -96,6 +96,11 @@ public ParseNode visit(ColumnParseNode node) throws
SQLException {
String indexColName = IndexUtil.getIndexColumnName(dataCol);
// Same alias as before, but use the index column name instead of
the data column name
+// ColumnParseNode cannot be resolved. When this occurs, add the
dataColRef to a list
--- End diff --
Removed.
Local index support to Phoenix
--
Key: PHOENIX-933
URL: https://issues.apache.org/jira/browse/PHOENIX-933
Project: Phoenix
Issue Type: New Feature
Reporter: rajeshbabu
Hindex(https://github.com/Huawei-Hadoop/hindex) provides local indexing
support to HBase. It stores region level index in a separate table, and
co-locates the user and index table regions with a custom load balancer.
See http://goo.gl/phkhwC and http://goo.gl/EswlxC for more information.
This JIRA addresses the local indexing solution integration to phoenix.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (PHOENIX-933) Local index support to Phoenix
[
https://issues.apache.org/jira/browse/PHOENIX-933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14061074#comment-14061074
]
ASF GitHub Bot commented on PHOENIX-933:
Github user chrajeshbabu commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/3#discussion_r14897862
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/compile/JoinCompiler.java ---
@@ -1134,8 +1135,15 @@ public static SelectStatement
optimize(PhoenixStatement statement, SelectStateme
ListOrderByNode orderBy = tableRef.equals(orderByTableRef) ?
select.getOrderBy() : null;
SelectStatement stmt =
getSubqueryForOptimizedPlan(select.getHint(), table.getDynamicColumns(),
tableRef, join.getColumnRefs(), table.getPreFiltersCombined(), groupBy,
orderBy, table.isWildCardSelect());
QueryPlan plan =
statement.getConnection().getQueryServices().getOptimizer().optimize(statement,
stmt);
+boolean localIndex =
plan.getContext().getCurrentTable().getTable().getIndexType()==IndexType.LOCAL;
if (!plan.getTableRef().equals(tableRef)) {
-replacement.put(tableRef, plan.getTableRef());
+// Use local index plan only when all the columns to
project are available in index.
--- End diff --
This change is not required in the master branch James. In local-index
branch join queries have failed with ColumnNotFoundException with local indexes
when join query has columns other than index. So I have added the check.
Currently join back from data table is not supported in join queries.
Local index support to Phoenix
--
Key: PHOENIX-933
URL: https://issues.apache.org/jira/browse/PHOENIX-933
Project: Phoenix
Issue Type: New Feature
Reporter: rajeshbabu
Hindex(https://github.com/Huawei-Hadoop/hindex) provides local indexing
support to HBase. It stores region level index in a separate table, and
co-locates the user and index table regions with a custom load balancer.
See http://goo.gl/phkhwC and http://goo.gl/EswlxC for more information.
This JIRA addresses the local indexing solution integration to phoenix.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (PHOENIX-933) Local index support to Phoenix
[ https://issues.apache.org/jira/browse/PHOENIX-933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14061075#comment-14061075 ] ASF GitHub Bot commented on PHOENIX-933: Github user chrajeshbabu commented on a diff in the pull request: https://github.com/apache/phoenix/pull/3#discussion_r14897879 --- Diff: phoenix-core/src/main/java/org/apache/phoenix/expression/ExpressionType.java --- @@ -169,6 +170,7 @@ SQLViewTypeFunction(SQLViewTypeFunction.class), ExternalSqlTypeIdFunction(ExternalSqlTypeIdFunction.class), ConvertTimezoneFunction(ConvertTimezoneFunction.class), +SQLIndexTypeFunction(SQLIndexTypeFunction.class), --- End diff -- done. Local index support to Phoenix -- Key: PHOENIX-933 URL: https://issues.apache.org/jira/browse/PHOENIX-933 Project: Phoenix Issue Type: New Feature Reporter: rajeshbabu Hindex(https://github.com/Huawei-Hadoop/hindex) provides local indexing support to HBase. It stores region level index in a separate table, and co-locates the user and index table regions with a custom load balancer. See http://goo.gl/phkhwC and http://goo.gl/EswlxC for more information. This JIRA addresses the local indexing solution integration to phoenix. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (PHOENIX-933) Local index support to Phoenix
[
https://issues.apache.org/jira/browse/PHOENIX-933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14061091#comment-14061091
]
ASF GitHub Bot commented on PHOENIX-933:
Github user chrajeshbabu commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/3#discussion_r14898383
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/iterate/DefaultParallelIteratorRegionSplitter.java
---
@@ -140,7 +142,14 @@ public boolean apply(HRegionLocation location) {
// distributed across regions, using this scheme compensates for
regions that
// have more rows than others, by applying tighter splits and
therefore spawning
// off more scans over the overloaded regions.
-int splitsPerRegion = regions.size() = targetConcurrency ? 1 :
(regions.size() targetConcurrency / 2 ? maxConcurrency : targetConcurrency) /
regions.size();
+PTable table = tableRef.getTable();
--- End diff --
bq. getSplitsPerRegion method to ParallelIteratorRegionSplitter so we can
move this special case to your new implementation for local indexes.
Current patch added getSplitsPerRegion method to
ParallelIteratorRegionSplitter and returning one in
LocalIndexParallelIteratorRegionSplitter and removed the changes in other
splitter.
Local index support to Phoenix
--
Key: PHOENIX-933
URL: https://issues.apache.org/jira/browse/PHOENIX-933
Project: Phoenix
Issue Type: New Feature
Reporter: rajeshbabu
Hindex(https://github.com/Huawei-Hadoop/hindex) provides local indexing
support to HBase. It stores region level index in a separate table, and
co-locates the user and index table regions with a custom load balancer.
See http://goo.gl/phkhwC and http://goo.gl/EswlxC for more information.
This JIRA addresses the local indexing solution integration to phoenix.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (PHOENIX-933) Local index support to Phoenix
[ https://issues.apache.org/jira/browse/PHOENIX-933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14061105#comment-14061105 ] ASF GitHub Bot commented on PHOENIX-933: Github user chrajeshbabu commented on the pull request: https://github.com/apache/phoenix/pull/3#issuecomment-48949296 Resolved the conflicts after PHOENIX-1002 also. Local index support to Phoenix -- Key: PHOENIX-933 URL: https://issues.apache.org/jira/browse/PHOENIX-933 Project: Phoenix Issue Type: New Feature Reporter: rajeshbabu Hindex(https://github.com/Huawei-Hadoop/hindex) provides local indexing support to HBase. It stores region level index in a separate table, and co-locates the user and index table regions with a custom load balancer. See http://goo.gl/phkhwC and http://goo.gl/EswlxC for more information. This JIRA addresses the local indexing solution integration to phoenix. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (PHOENIX-933) Local index support to Phoenix
[
https://issues.apache.org/jira/browse/PHOENIX-933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14061103#comment-14061103
]
ASF GitHub Bot commented on PHOENIX-933:
Github user chrajeshbabu commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/3#discussion_r14898689
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/iterate/ParallelIterators.java ---
@@ -238,7 +240,12 @@ private void
doColumnProjectionOptimization(StatementContext context, Scan scan,
minMaxRange =
SaltingUtil.addSaltByte(split.getLowerRange(), minMaxRange);
split = split.intersect(minMaxRange);
}
-}
+} else if (localIndex) {
--- End diff --
one main difference is there between the two methods James. in addSaltByte
if keyrange upper or lower bound is unbound we don't set salting byte. in case
of local index if both start or end keys are unbound then only we don't prefix
the bytes. So not able to commonize both.
Local index support to Phoenix
--
Key: PHOENIX-933
URL: https://issues.apache.org/jira/browse/PHOENIX-933
Project: Phoenix
Issue Type: New Feature
Reporter: rajeshbabu
Hindex(https://github.com/Huawei-Hadoop/hindex) provides local indexing
support to HBase. It stores region level index in a separate table, and
co-locates the user and index table regions with a custom load balancer.
See http://goo.gl/phkhwC and http://goo.gl/EswlxC for more information.
This JIRA addresses the local indexing solution integration to phoenix.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (PHOENIX-933) Local index support to Phoenix
[
https://issues.apache.org/jira/browse/PHOENIX-933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14061116#comment-14061116
]
ASF GitHub Bot commented on PHOENIX-933:
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/3#discussion_r14899221
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/iterate/ParallelIterators.java ---
@@ -238,7 +240,12 @@ private void
doColumnProjectionOptimization(StatementContext context, Scan scan,
minMaxRange =
SaltingUtil.addSaltByte(split.getLowerRange(), minMaxRange);
split = split.intersect(minMaxRange);
}
-}
+} else if (localIndex) {
--- End diff --
Let's discuss in a separate JIRA. It's not a big deal.
Local index support to Phoenix
--
Key: PHOENIX-933
URL: https://issues.apache.org/jira/browse/PHOENIX-933
Project: Phoenix
Issue Type: New Feature
Reporter: rajeshbabu
Hindex(https://github.com/Huawei-Hadoop/hindex) provides local indexing
support to HBase. It stores region level index in a separate table, and
co-locates the user and index table regions with a custom load balancer.
See http://goo.gl/phkhwC and http://goo.gl/EswlxC for more information.
This JIRA addresses the local indexing solution integration to phoenix.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (PHOENIX-933) Local index support to Phoenix
[
https://issues.apache.org/jira/browse/PHOENIX-933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14061127#comment-14061127
]
ASF GitHub Bot commented on PHOENIX-933:
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/3#discussion_r14899559
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/iterate/DefaultParallelIteratorRegionSplitter.java
---
@@ -140,7 +142,14 @@ public boolean apply(HRegionLocation location) {
// distributed across regions, using this scheme compensates for
regions that
// have more rows than others, by applying tighter splits and
therefore spawning
// off more scans over the overloaded regions.
-int splitsPerRegion = regions.size() = targetConcurrency ? 1 :
(regions.size() targetConcurrency / 2 ? maxConcurrency : targetConcurrency) /
regions.size();
+PTable table = tableRef.getTable();
--- End diff --
That's what the splitsPerRegion variable and subsequent logic in
ParallelIterators does - it creates additional split points within the range so
that multiple scans get run over a single region. We'd want to prefix each of
these with the same start region key. I'll open a separate JIRA for this too.
It's not a big deal - most of the time the parallelization slots would be used
up by having to do a scan in each region anyway.
Local index support to Phoenix
--
Key: PHOENIX-933
URL: https://issues.apache.org/jira/browse/PHOENIX-933
Project: Phoenix
Issue Type: New Feature
Reporter: rajeshbabu
Hindex(https://github.com/Huawei-Hadoop/hindex) provides local indexing
support to HBase. It stores region level index in a separate table, and
co-locates the user and index table regions with a custom load balancer.
See http://goo.gl/phkhwC and http://goo.gl/EswlxC for more information.
This JIRA addresses the local indexing solution integration to phoenix.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (PHOENIX-933) Local index support to Phoenix
[ https://issues.apache.org/jira/browse/PHOENIX-933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14061653#comment-14061653 ] ramkrishna.s.vasudevan commented on PHOENIX-933: Anyway to edit the message? Local index support to Phoenix -- Key: PHOENIX-933 URL: https://issues.apache.org/jira/browse/PHOENIX-933 Project: Phoenix Issue Type: New Feature Reporter: rajeshbabu Assignee: rajeshbabu Attachments: PHOENIX-933.patch Hindex(https://github.com/Huawei-Hadoop/hindex) provides local indexing support to HBase. It stores region level index in a separate table, and co-locates the user and index table regions with a custom load balancer. See http://goo.gl/phkhwC and http://goo.gl/EswlxC for more information. This JIRA addresses the local indexing solution integration to phoenix. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (PHOENIX-933) Local index support to Phoenix
[ https://issues.apache.org/jira/browse/PHOENIX-933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14061659#comment-14061659 ] Hudson commented on PHOENIX-933: FAILURE: Integrated in Phoenix | Master | Hadoop1 #271 (See [https://builds.apache.org/job/Phoenix-master-hadoop1/271/]) PHOENIX-933 Local index support to Phoenix (Ram) (ramkrishna: rev 247ac8eee8ca604ca1616ba4bcf91fec6475ea88) * phoenix-core/src/main/java/org/apache/phoenix/query/QueryConstants.java * phoenix-core/src/main/java/org/apache/phoenix/index/IndexMaintainer.java * phoenix-core/src/it/java/org/apache/phoenix/end2end/index/LocalIndexIT.java * phoenix-core/src/main/java/org/apache/phoenix/expression/visitor/RowKeyExpressionVisitor.java * phoenix-core/src/main/java/org/apache/phoenix/iterate/ParallelIteratorRegionSplitter.java * phoenix-core/src/main/java/org/apache/phoenix/compile/CreateIndexCompiler.java * phoenix-core/src/it/java/org/apache/phoenix/hbase/index/balancer/TestIndexLoadBalancer.java * phoenix-core/src/test/java/org/apache/phoenix/query/BaseTest.java * phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixDatabaseMetaData.java * phoenix-core/src/main/java/org/apache/phoenix/coprocessor/BaseScannerRegionObserver.java * phoenix-core/src/it/java/org/apache/phoenix/end2end/AlterTableIT.java * phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java * phoenix-core/src/main/java/org/apache/phoenix/hbase/index/balancer/IndexLoadBalancer.java * phoenix-core/src/main/java/org/apache/phoenix/iterate/LocalIndexParallelIteratorRegionSplitter.java * phoenix-core/src/main/java/org/apache/phoenix/expression/RowKeyColumnExpression.java * phoenix-core/src/it/java/org/apache/phoenix/end2end/DeleteIT.java * phoenix-core/src/main/java/org/apache/phoenix/expression/ExpressionType.java * phoenix-core/src/main/antlr3/PhoenixSQL.g * phoenix-core/src/it/java/org/apache/phoenix/end2end/TenantSpecificViewIndexSaltedIT.java * phoenix-core/src/main/java/org/apache/phoenix/join/TupleProjector.java * phoenix-core/src/main/java/org/apache/phoenix/index/PhoenixIndexCodec.java * phoenix-core/src/main/java/org/apache/phoenix/coprocessor/generated/PTableProtos.java * phoenix-core/src/main/java/org/apache/phoenix/util/IndexUtil.java * phoenix-core/src/main/java/org/apache/phoenix/execute/BasicQueryPlan.java * phoenix-core/src/main/java/org/apache/phoenix/parse/ParseNodeFactory.java * phoenix-core/src/main/java/org/apache/phoenix/hbase/index/master/IndexMasterObserver.java * phoenix-core/src/main/java/org/apache/phoenix/execute/ScanPlan.java * phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataProtocol.java * phoenix-core/src/it/java/org/apache/phoenix/end2end/BaseViewIT.java * phoenix-core/src/main/java/org/apache/phoenix/coprocessor/UngroupedAggregateRegionObserver.java * phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java * phoenix-core/src/main/java/org/apache/phoenix/coprocessor/ScanRegionObserver.java * phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java * phoenix-core/src/main/java/org/apache/phoenix/coprocessor/GroupedAggregateRegionObserver.java * phoenix-core/src/it/java/org/apache/phoenix/end2end/index/MutableIndexIT.java * phoenix-core/src/main/java/org/apache/phoenix/compile/ExpressionCompiler.java * phoenix-core/src/it/java/org/apache/phoenix/end2end/TenantSpecificViewIndexIT.java * phoenix-core/src/it/java/org/apache/phoenix/end2end/ViewIT.java * phoenix-core/src/main/java/org/apache/phoenix/expression/KeyValueColumnExpression.java * phoenix-core/src/main/java/org/apache/phoenix/hbase/index/util/IndexManagementUtil.java * phoenix-core/src/main/java/org/apache/phoenix/schema/PTableImpl.java * phoenix-core/src/it/java/org/apache/phoenix/end2end/SaltedViewIT.java * phoenix-core/src/main/java/org/apache/phoenix/schema/MetaDataClient.java * phoenix-core/src/main/java/org/apache/phoenix/schema/LocalIndexDataColumnRef.java * phoenix-core/src/main/java/org/apache/phoenix/compile/WhereCompiler.java * phoenix-core/src/test/java/org/apache/phoenix/index/IndexMaintainerTest.java * phoenix-core/src/main/java/org/apache/phoenix/parse/CreateIndexStatement.java * phoenix-core/src/it/java/org/apache/phoenix/end2end/HashJoinIT.java * phoenix-core/src/main/java/org/apache/phoenix/schema/ColumnRef.java * phoenix-core/src/main/java/org/apache/phoenix/filter/SkipScanFilter.java * phoenix-core/src/main/java/org/apache/phoenix/iterate/ParallelIterators.java * phoenix-core/src/main/java/org/apache/phoenix/hbase/index/IndexRegionSplitPolicy.java * phoenix-core/src/main/java/org/apache/phoenix/exception/SQLExceptionCode.java * phoenix-core/src/main/java/org/apache/phoenix/iterate/DefaultParallelIteratorRegionSplitter.java * phoenix-core/src/main/java/org/apache/phoenix/parse/ParseNodeRewriter.java *
[jira] [Commented] (PHOENIX-933) Local index support to Phoenix
[ https://issues.apache.org/jira/browse/PHOENIX-933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14061670#comment-14061670 ] ASF GitHub Bot commented on PHOENIX-933: Github user ramkrish86 commented on the pull request: https://github.com/apache/phoenix/pull/3#issuecomment-48991061 bq. just add a .patch to the url for the pull request and you'll have the patch file to apply against the normal/updatable repo That's nice. I tried it, it works. :) Local index support to Phoenix -- Key: PHOENIX-933 URL: https://issues.apache.org/jira/browse/PHOENIX-933 Project: Phoenix Issue Type: New Feature Reporter: rajeshbabu Assignee: rajeshbabu Attachments: PHOENIX-933.patch Hindex(https://github.com/Huawei-Hadoop/hindex) provides local indexing support to HBase. It stores region level index in a separate table, and co-locates the user and index table regions with a custom load balancer. See http://goo.gl/phkhwC and http://goo.gl/EswlxC for more information. This JIRA addresses the local indexing solution integration to phoenix. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (PHOENIX-933) Local index support to Phoenix
[
https://issues.apache.org/jira/browse/PHOENIX-933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14060094#comment-14060094
]
ASF GitHub Bot commented on PHOENIX-933:
Github user chrajeshbabu commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/1#discussion_r14858124
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/iterate/SkipRangeParallelIteratorRegionSplitter.java
---
@@ -54,7 +55,8 @@ protected
SkipRangeParallelIteratorRegionSplitter(StatementContext context, Tabl
public ListHRegionLocation filterRegions(ListHRegionLocation
allTableRegions, final ScanRanges ranges) {
IterableHRegionLocation regions;
-if (ranges == ScanRanges.EVERYTHING) {
+if (ranges == ScanRanges.EVERYTHING
--- End diff --
Even with the change skip scan will be used James. The change is required
because the key ranges generated by compiler won't be in the local index
regions key range because local index rows have prefixed region start key
extra. Without the change mostly no region will be selected for scanning.
QueryIT#testSimpleInListStatement is the test case verifies the same.
Here is the explain query result.
CLIENT PARALLEL 4-WAY SKIP SCAN ON 2 KEYS OVER _LOCAL_IDX_ATABLE [-32768,2]
- [-32768,4]
SERVER FILTER BY FIRST KEY ONLY AND ORGANIZATION_ID = '00D3XHP'
CLIENT MERGE SORT
Local index support to Phoenix
--
Key: PHOENIX-933
URL: https://issues.apache.org/jira/browse/PHOENIX-933
Project: Phoenix
Issue Type: New Feature
Reporter: rajeshbabu
Hindex(https://github.com/Huawei-Hadoop/hindex) provides local indexing
support to HBase. It stores region level index in a separate table, and
co-locates the user and index table regions with a custom load balancer.
See http://goo.gl/phkhwC and http://goo.gl/EswlxC for more information.
This JIRA addresses the local indexing solution integration to phoenix.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (PHOENIX-933) Local index support to Phoenix
[ https://issues.apache.org/jira/browse/PHOENIX-933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14060096#comment-14060096 ] ASF GitHub Bot commented on PHOENIX-933: Github user chrajeshbabu commented on the pull request: https://github.com/apache/phoenix/pull/1#issuecomment-48839233 bq. Cleanest might be to just implement a simple ParallelIteratorRegionSplitter for use when a local index is used that just returns all regions: I will add new ParallelIteratorRegionSplitter for local index and remove the unnecessary changes in SkipRangeParallelIteratorRegionSplitter/DefaultParallelIteratorRegionSplitter. Then I will submit another pull request. Thanks @JamesRTaylor Local index support to Phoenix -- Key: PHOENIX-933 URL: https://issues.apache.org/jira/browse/PHOENIX-933 Project: Phoenix Issue Type: New Feature Reporter: rajeshbabu Hindex(https://github.com/Huawei-Hadoop/hindex) provides local indexing support to HBase. It stores region level index in a separate table, and co-locates the user and index table regions with a custom load balancer. See http://goo.gl/phkhwC and http://goo.gl/EswlxC for more information. This JIRA addresses the local indexing solution integration to phoenix. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (PHOENIX-933) Local index support to Phoenix
[ https://issues.apache.org/jira/browse/PHOENIX-933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14060189#comment-14060189 ] ASF GitHub Bot commented on PHOENIX-933: GitHub user chrajeshbabu opened a pull request: https://github.com/apache/phoenix/pull/3 PHOENIX-933 Local index support to Phoenix Updated pull request after resolving conflicts and handling James review comments. You can merge this pull request into a Git repository by running: $ git pull https://github.com/chrajeshbabu/phoenix master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/phoenix/pull/3.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3 commit 014ec218b151ef960347b875b373828ad8fe8681 Author: James Taylor jamestay...@apache.org Date: 2014-04-17T01:28:18Z Add local index to grammar and metadata commit 1855a8d1c6b2064d81f5fffc27c18342599cf7b0 Author: James Taylor jamestay...@apache.org Date: 2014-04-18T23:31:29Z PHOENIX-936 Custom load balancer to colocate user table regions and index table regions (Rajeshbabu) commit 1bc8b57801e299635ac601aea5dc7ba8d4915a2f Author: James Taylor jamestay...@apache.org Date: 2014-04-26T04:36:03Z PHOENIX-935 create local index table with the same split keys of user table (Rajeshbabu) commit b02df4f054f78a866f25d9beb4e7fc2242df2a1f Author: James Taylor jamestay...@apache.org Date: 2014-04-28T05:26:27Z PHOENIX-935 create local index table with the same split keys of user table (Rajeshbabu) commit 8f3f8100210538d0c162cbbc5e02efc63db52d31 Author: James Taylor jamestay...@apache.org Date: 2014-04-28T16:31:31Z PHOENIX-955 Skip region start key at beginning of local index rows (JamesTaylor) commit 8e35bc99068da33db4e2019d9bc65d8d78bee4d2 Author: James Taylor jtay...@salesforce.com Date: 2014-05-08T06:23:37Z PHOENIX-937 Handle puts on local index table (Rajeshbabu) commit cf78def0579f51a5342bcfbff802902478052d25 Author: James Taylor jtay...@salesforce.com Date: 2014-05-23T03:48:01Z PHOENIX-994 Handle scans on local index table in case any best fit covering local index available (Rajeshbabu) commit 835bcf675db9435da1335bbfb6d55d6d7edd86b3 Author: James Taylor jtay...@salesforce.com Date: 2014-05-27T19:51:56Z PHOENIX-1004 'drop index' should delete index data from local index table (Rajeshbabu) commit 2edd2b6a19deef0847086da2c7b30250ed2907c7 Author: James Taylor jtay...@salesforce.com Date: 2014-06-09T18:11:35Z PHOENIX-1038 Dynamically add INDEX_TYPE column to SYSTEM.CATALOG if not already there commit 0408b333780f4ce428e5d9b018e5cd13934d0f84 Author: Ramkrishna ramkrishna.s.vasude...@intel.com Date: 2014-07-04T10:19:41Z PHOENIX-1015 Support joining back to data table row from local index when query condition involves leading columns in local index (Rajeshbabu) commit 3df832e73e138320a30e5c5d0c80680b79de6358 Author: Rajeshbabu Chintaguntla rajeshbabu.chintagun...@huawei.com Date: 2014-07-12T17:14:44Z PHOENIX-1015 Support joining back to data table row from local index when query condition involves leading columns in local index commit 84097757a3173adc2a3b464901b79f35bc9d07cb Author: Rajeshbabu Chintaguntla rajeshbabu.chintagun...@huawei.com Date: 2014-07-13T17:26:13Z PHOENIX-1015 Support joining back to data table row from local index when query condition involves leading columns in local index Local index support to Phoenix -- Key: PHOENIX-933 URL: https://issues.apache.org/jira/browse/PHOENIX-933 Project: Phoenix Issue Type: New Feature Reporter: rajeshbabu Hindex(https://github.com/Huawei-Hadoop/hindex) provides local indexing support to HBase. It stores region level index in a separate table, and co-locates the user and index table regions with a custom load balancer. See http://goo.gl/phkhwC and http://goo.gl/EswlxC for more information. This JIRA addresses the local indexing solution integration to phoenix. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (PHOENIX-933) Local index support to Phoenix
[
https://issues.apache.org/jira/browse/PHOENIX-933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14060225#comment-14060225
]
ASF GitHub Bot commented on PHOENIX-933:
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/3#discussion_r14860610
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/iterate/DefaultParallelIteratorRegionSplitter.java
---
@@ -140,7 +142,14 @@ public boolean apply(HRegionLocation location) {
// distributed across regions, using this scheme compensates for
regions that
// have more rows than others, by applying tighter splits and
therefore spawning
// off more scans over the overloaded regions.
-int splitsPerRegion = regions.size() = targetConcurrency ? 1 :
(regions.size() targetConcurrency / 2 ? maxConcurrency : targetConcurrency) /
regions.size();
+PTable table = tableRef.getTable();
--- End diff --
Please add TODO to add getSplitsPerRegion method to
ParallelIteratorRegionSplitter so we can move this special case to your new
implementation for local indexes.
Local index support to Phoenix
--
Key: PHOENIX-933
URL: https://issues.apache.org/jira/browse/PHOENIX-933
Project: Phoenix
Issue Type: New Feature
Reporter: rajeshbabu
Hindex(https://github.com/Huawei-Hadoop/hindex) provides local indexing
support to HBase. It stores region level index in a separate table, and
co-locates the user and index table regions with a custom load balancer.
See http://goo.gl/phkhwC and http://goo.gl/EswlxC for more information.
This JIRA addresses the local indexing solution integration to phoenix.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (PHOENIX-933) Local index support to Phoenix
[
https://issues.apache.org/jira/browse/PHOENIX-933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14059703#comment-14059703
]
ASF GitHub Bot commented on PHOENIX-933:
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/1#discussion_r14852129
--- Diff:
phoenix-core/src/it/java/org/apache/phoenix/end2end/index/MutableIndexIT.java
---
@@ -51,21 +51,38 @@ public static void doSetup() throws Exception {
props.put(QueryServices.MAX_INTRA_REGION_PARALLELIZATION_ATTRIB,
Integer.toString(1));
// Forces server cache to be used
props.put(QueryServices.INDEX_MUTATE_BATCH_SIZE_THRESHOLD_ATTRIB,
Integer.toString(2));
+props.put(QueryServices.DROP_METADATA_ATTRIB,
Boolean.toString(true));
// Must update config before starting server
startServer(getUrl(), new
ReadOnlyProps(props.entrySet().iterator()));
}
-
+
@Test
public void testIndexWithNullableFixedWithCols() throws Exception {
+testIndexWithNullableFixedWithCols(false);
+}
+
+@Test
+public void testLocalIndexWithNullableFixedWithCols() throws Exception
{
+testIndexWithNullableFixedWithCols(true);
+}
+
+private void testIndexWithNullableFixedWithCols(boolean localIndex)
throws Exception {
Properties props = new Properties(TEST_PROPERTIES);
Connection conn = DriverManager.getConnection(getUrl(), props);
conn.setAutoCommit(false);
try {
createTestTable();
populateTestTable();
-String ddl = CREATE INDEX + INDEX_TABLE_NAME + ON +
DATA_TABLE_FULL_NAME
-+ (char_col1 ASC, int_col1 ASC)
-+ INCLUDE (long_col1, long_col2);
+String ddl = null;
+if(localIndex){
+ddl = CREATE INDEX + INDEX_TABLE_NAME + ON +
DATA_TABLE_FULL_NAME
--- End diff --
Looks like the if and the else branch match. If that's expected/correct,
can you get rid of the if statement?
Local index support to Phoenix
--
Key: PHOENIX-933
URL: https://issues.apache.org/jira/browse/PHOENIX-933
Project: Phoenix
Issue Type: New Feature
Reporter: rajeshbabu
Hindex(https://github.com/Huawei-Hadoop/hindex) provides local indexing
support to HBase. It stores region level index in a separate table, and
co-locates the user and index table regions with a custom load balancer.
See http://goo.gl/phkhwC and http://goo.gl/EswlxC for more information.
This JIRA addresses the local indexing solution integration to phoenix.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (PHOENIX-933) Local index support to Phoenix
[
https://issues.apache.org/jira/browse/PHOENIX-933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14059826#comment-14059826
]
ASF GitHub Bot commented on PHOENIX-933:
Github user chrajeshbabu commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/1#discussion_r14853783
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/ScanRegionObserver.java
---
@@ -100,7 +109,7 @@ public static void serializeIntoScan(Scan scan, int
thresholdBytes, int limit, L
}
}
-public static OrderedResultIterator deserializeFromScan(Scan scan,
RegionScanner s) {
+public static OrderedResultIterator deserializeFromScan(Scan scan,
RegionScanner s, int offset) {
--- End diff --
bq. There's a bit more you need to do to handle ORDER BY correctly. It'd be
for the case in which a data column was referenced in the ORDER BY while the
index table is being used to satisfy the query.
This is working fine James. I have added test case.
Local index support to Phoenix
--
Key: PHOENIX-933
URL: https://issues.apache.org/jira/browse/PHOENIX-933
Project: Phoenix
Issue Type: New Feature
Reporter: rajeshbabu
Hindex(https://github.com/Huawei-Hadoop/hindex) provides local indexing
support to HBase. It stores region level index in a separate table, and
co-locates the user and index table regions with a custom load balancer.
See http://goo.gl/phkhwC and http://goo.gl/EswlxC for more information.
This JIRA addresses the local indexing solution integration to phoenix.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (PHOENIX-933) Local index support to Phoenix
[
https://issues.apache.org/jira/browse/PHOENIX-933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14059828#comment-14059828
]
ASF GitHub Bot commented on PHOENIX-933:
Github user chrajeshbabu commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/1#discussion_r14853806
--- Diff:
phoenix-core/src/it/java/org/apache/phoenix/end2end/index/MutableIndexIT.java
---
@@ -51,21 +51,38 @@ public static void doSetup() throws Exception {
props.put(QueryServices.MAX_INTRA_REGION_PARALLELIZATION_ATTRIB,
Integer.toString(1));
// Forces server cache to be used
props.put(QueryServices.INDEX_MUTATE_BATCH_SIZE_THRESHOLD_ATTRIB,
Integer.toString(2));
+props.put(QueryServices.DROP_METADATA_ATTRIB,
Boolean.toString(true));
// Must update config before starting server
startServer(getUrl(), new
ReadOnlyProps(props.entrySet().iterator()));
}
-
+
@Test
public void testIndexWithNullableFixedWithCols() throws Exception {
+testIndexWithNullableFixedWithCols(false);
+}
+
+@Test
+public void testLocalIndexWithNullableFixedWithCols() throws Exception
{
+testIndexWithNullableFixedWithCols(true);
+}
+
+private void testIndexWithNullableFixedWithCols(boolean localIndex)
throws Exception {
Properties props = new Properties(TEST_PROPERTIES);
Connection conn = DriverManager.getConnection(getUrl(), props);
conn.setAutoCommit(false);
try {
createTestTable();
populateTestTable();
-String ddl = CREATE INDEX + INDEX_TABLE_NAME + ON +
DATA_TABLE_FULL_NAME
-+ (char_col1 ASC, int_col1 ASC)
-+ INCLUDE (long_col1, long_col2);
+String ddl = null;
+if(localIndex){
+ddl = CREATE INDEX + INDEX_TABLE_NAME + ON +
DATA_TABLE_FULL_NAME
--- End diff --
This is the actual code. Earlier by mistake I missed to add LOCAL in if
branch so if else branches are same.
if (localIndex) {
ddl = CREATE LOCAL INDEX + INDEX_TABLE_NAME + ON +
DATA_TABLE_FULL_NAME + (date_col);
} else {
ddl = CREATE INDEX + INDEX_TABLE_NAME + ON +
DATA_TABLE_FULL_NAME + (date_col);
}
Now I have corrected it.
Local index support to Phoenix
--
Key: PHOENIX-933
URL: https://issues.apache.org/jira/browse/PHOENIX-933
Project: Phoenix
Issue Type: New Feature
Reporter: rajeshbabu
Hindex(https://github.com/Huawei-Hadoop/hindex) provides local indexing
support to HBase. It stores region level index in a separate table, and
co-locates the user and index table regions with a custom load balancer.
See http://goo.gl/phkhwC and http://goo.gl/EswlxC for more information.
This JIRA addresses the local indexing solution integration to phoenix.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (PHOENIX-933) Local index support to Phoenix
[
https://issues.apache.org/jira/browse/PHOENIX-933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14059852#comment-14059852
]
ASF GitHub Bot commented on PHOENIX-933:
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/1#discussion_r14854253
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/ScanRegionObserver.java
---
@@ -100,7 +109,7 @@ public static void serializeIntoScan(Scan scan, int
thresholdBytes, int limit, L
}
}
-public static OrderedResultIterator deserializeFromScan(Scan scan,
RegionScanner s) {
+public static OrderedResultIterator deserializeFromScan(Scan scan,
RegionScanner s, int offset) {
--- End diff --
Excellent. Would you mind updating this pull request? I think it's ready to
get committed, no?
Local index support to Phoenix
--
Key: PHOENIX-933
URL: https://issues.apache.org/jira/browse/PHOENIX-933
Project: Phoenix
Issue Type: New Feature
Reporter: rajeshbabu
Hindex(https://github.com/Huawei-Hadoop/hindex) provides local indexing
support to HBase. It stores region level index in a separate table, and
co-locates the user and index table regions with a custom load balancer.
See http://goo.gl/phkhwC and http://goo.gl/EswlxC for more information.
This JIRA addresses the local indexing solution integration to phoenix.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (PHOENIX-933) Local index support to Phoenix
[
https://issues.apache.org/jira/browse/PHOENIX-933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14059899#comment-14059899
]
ASF GitHub Bot commented on PHOENIX-933:
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/1#discussion_r14855032
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/iterate/SkipRangeParallelIteratorRegionSplitter.java
---
@@ -54,7 +55,8 @@ protected
SkipRangeParallelIteratorRegionSplitter(StatementContext context, Tabl
public ListHRegionLocation filterRegions(ListHRegionLocation
allTableRegions, final ScanRanges ranges) {
IterableHRegionLocation regions;
-if (ranges == ScanRanges.EVERYTHING) {
+if (ranges == ScanRanges.EVERYTHING
--- End diff --
This shouldn't be necessary and will cause the skip scan not to be used
(which will be horrible for point queries). It should work fine to do the skip
scan for each region (much better than doing a full region scan for every
region).
Can you remove it and add a test that does a point lookup with multiple
values (for example, a IN clause with multiple values for an indexed column)?
Local index support to Phoenix
--
Key: PHOENIX-933
URL: https://issues.apache.org/jira/browse/PHOENIX-933
Project: Phoenix
Issue Type: New Feature
Reporter: rajeshbabu
Hindex(https://github.com/Huawei-Hadoop/hindex) provides local indexing
support to HBase. It stores region level index in a separate table, and
co-locates the user and index table regions with a custom load balancer.
See http://goo.gl/phkhwC and http://goo.gl/EswlxC for more information.
This JIRA addresses the local indexing solution integration to phoenix.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (PHOENIX-933) Local index support to Phoenix
[
https://issues.apache.org/jira/browse/PHOENIX-933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14059948#comment-14059948
]
ASF GitHub Bot commented on PHOENIX-933:
Github user JamesRTaylor commented on the pull request:
https://github.com/apache/phoenix/pull/1#issuecomment-48825943
Related to the revert of the ScanRanges change, you'll need to make this
change to prevent the SkipRangeParallelIteratorRegionSplitter from being used
(as you always need to scan all regions for a local index). Cleanest might be
to just implement a simple ParallelIteratorRegionSplitter for use when a local
index is used that just returns all regions:
public class ParallelIteratorRegionSplitterFactory {
public static ParallelIteratorRegionSplitter
getSplitter(StatementContext context, TableRef table, HintNode hintNode) throws
SQLException {
if (!isLocalIndex
context.getScanRanges().useSkipScanFilter()) {
return
SkipRangeParallelIteratorRegionSplitter.getInstance(context, table, hintNode);
}
return
DefaultParallelIteratorRegionSplitter.getInstance(context, table, hintNode);
}
}
Local index support to Phoenix
--
Key: PHOENIX-933
URL: https://issues.apache.org/jira/browse/PHOENIX-933
Project: Phoenix
Issue Type: New Feature
Reporter: rajeshbabu
Hindex(https://github.com/Huawei-Hadoop/hindex) provides local indexing
support to HBase. It stores region level index in a separate table, and
co-locates the user and index table regions with a custom load balancer.
See http://goo.gl/phkhwC and http://goo.gl/EswlxC for more information.
This JIRA addresses the local indexing solution integration to phoenix.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (PHOENIX-933) Local index support to Phoenix
[
https://issues.apache.org/jira/browse/PHOENIX-933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14059119#comment-14059119
]
ASF GitHub Bot commented on PHOENIX-933:
Github user chrajeshbabu commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/1#discussion_r14834905
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/compile/ExpressionCompiler.java
---
@@ -320,7 +322,21 @@ public Expression visitLeave(FunctionParseNode node,
ListExpression children)
* @throws SQLException if the column expression node does not refer
to a known/unambiguous column
*/
protected ColumnRef resolveColumn(ColumnParseNode node) throws
SQLException {
-ColumnRef ref =
context.getResolver().resolveColumn(node.getSchemaName(), node.getTableName(),
node.getName());
+ColumnRef ref = null;
+try {
+ref =
context.getResolver().resolveColumn(node.getSchemaName(), node.getTableName(),
node.getName());
+} catch (ColumnNotFoundException e) {
+// If local index table (need to test join case here)
--- End diff --
Added the comment.
Local index support to Phoenix
--
Key: PHOENIX-933
URL: https://issues.apache.org/jira/browse/PHOENIX-933
Project: Phoenix
Issue Type: New Feature
Reporter: rajeshbabu
Hindex(https://github.com/Huawei-Hadoop/hindex) provides local indexing
support to HBase. It stores region level index in a separate table, and
co-locates the user and index table regions with a custom load balancer.
See http://goo.gl/phkhwC and http://goo.gl/EswlxC for more information.
This JIRA addresses the local indexing solution integration to phoenix.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (PHOENIX-933) Local index support to Phoenix
[ https://issues.apache.org/jira/browse/PHOENIX-933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14059126#comment-14059126 ] ASF GitHub Bot commented on PHOENIX-933: Github user chrajeshbabu commented on a diff in the pull request: https://github.com/apache/phoenix/pull/1#discussion_r14835161 --- Diff: phoenix-core/src/main/java/org/apache/phoenix/compile/TrackOrderPreservingExpressionCompiler.java --- @@ -69,6 +70,7 @@ boolean isSharedViewIndex = table.getViewIndexId() != null; // TODO: util for this offset, as it's computed in numerous places positionOffset = (isSalted ? 1 : 0) + (isMultiTenant ? 1 : 0) + (isSharedViewIndex ? 1 : 0); +this.isOrderPreserving = table.getIndexType() != IndexType.LOCAL; --- End diff -- Thanks for pointing this James. Yes merge sort is fine. Done the changes locally. Local index support to Phoenix -- Key: PHOENIX-933 URL: https://issues.apache.org/jira/browse/PHOENIX-933 Project: Phoenix Issue Type: New Feature Reporter: rajeshbabu Hindex(https://github.com/Huawei-Hadoop/hindex) provides local indexing support to HBase. It stores region level index in a separate table, and co-locates the user and index table regions with a custom load balancer. See http://goo.gl/phkhwC and http://goo.gl/EswlxC for more information. This JIRA addresses the local indexing solution integration to phoenix. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (PHOENIX-933) Local index support to Phoenix
[
https://issues.apache.org/jira/browse/PHOENIX-933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14057420#comment-14057420
]
ASF GitHub Bot commented on PHOENIX-933:
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/1#discussion_r14762643
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/compile/TrackOrderPreservingExpressionCompiler.java
---
@@ -69,6 +70,7 @@
boolean isSharedViewIndex = table.getViewIndexId() != null;
// TODO: util for this offset, as it's computed in numerous places
positionOffset = (isSalted ? 1 : 0) + (isMultiTenant ? 1 : 0) +
(isSharedViewIndex ? 1 : 0);
+this.isOrderPreserving = table.getIndexType() != IndexType.LOCAL;
--- End diff --
One thing that's necessary, though, to maintain rows in row key order is to
modify ScanPlan.java:118 to do a merge sort instead of a concat:
if ((isSalted || isLocalIndex)
(context.getConnection().getQueryServices().getProps().getBoolean(
QueryServices.ROW_KEY_ORDER_SALTED_TABLE_ATTRIB,
QueryServicesOptions.DEFAULT_ROW_KEY_ORDER_SALTED_TABLE) ||
orderBy == OrderBy.FWD_ROW_KEY_ORDER_BY ||
orderBy == OrderBy.REV_ROW_KEY_ORDER_BY)) { // ORDER
BY was optimized out b/c query is in row key order
scanner = new MergeSortRowKeyResultIterator(iterators,
SaltingUtil.NUM_SALTING_BYTES, orderBy == OrderBy.REV_ROW_KEY_ORDER_BY);
} else {
scanner = new ConcatResultIterator(iterators);
}
Local indexes are similar to salted tables in that the parallel scans will
all be within a region, ordered correctly. As long as we do a merge sort across
the results of these scans, the rows will be ordered correctly.
Local index support to Phoenix
--
Key: PHOENIX-933
URL: https://issues.apache.org/jira/browse/PHOENIX-933
Project: Phoenix
Issue Type: New Feature
Reporter: rajeshbabu
Hindex(https://github.com/Huawei-Hadoop/hindex) provides local indexing
support to HBase. It stores region level index in a separate table, and
co-locates the user and index table regions with a custom load balancer.
See http://goo.gl/phkhwC and http://goo.gl/EswlxC for more information.
This JIRA addresses the local indexing solution integration to phoenix.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
