[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14539730#comment-14539730 ] Anoop Sam John commented on HBASE-9203: --- As discussed above and in mail list, it is not possible for a global solution which satisfy all sort of needs. So to HBase core code we might not add any indexing. As you said, Phoenix have local indexing solution. It also has global indexing solution. Depending on the need and data size user can opt for any. The local indexing try to reduce the write time hit and make things 100% strong consistent model. bq.But hey! What do I know. I'm not a committer ... ;-P One idea or comment is treated as per the quality of it and not based on whether it is from committer or not. I have not come across some non committers complaining that their comments/ideas are just ignored. Just saying. > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Reporter: rajeshbabu >Assignee: rajeshbabu > Attachments: SecondaryIndex Design.pdf, SecondaryIndex > Design_Updated.pdf, SecondaryIndex Design_Updated_2.pdf > > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14092057#comment-14092057 ] Nathan M commented on HBASE-9203: - I'm keen to see this. What is the progress. Anything I can do to help? > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Reporter: rajeshbabu >Assignee: rajeshbabu > Attachments: SecondaryIndex Design.pdf, SecondaryIndex > Design_Updated.pdf, SecondaryIndex Design_Updated_2.pdf > > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13909992#comment-13909992 ] hoar commented on HBASE-9203: - thank you very much ,rajeshbabu. > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Reporter: rajeshbabu >Assignee: rajeshbabu > Attachments: SecondaryIndex Design.pdf, SecondaryIndex > Design_Updated.pdf, SecondaryIndex Design_Updated_2.pdf > > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[
https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13906880#comment-13906880
]
rajeshbabu commented on HBASE-9203:
---
If you have some data then you can use
org.apache.hadoop.hbase.index.mapreduce.TableIndexer mapreduce job to create
index for existing data.
Are you refering IndexedHTableDescriptor from
https://github.com/Huawei-Hadoop/hindex/blob/master/secondaryindex/src/main/java/org/apache/hadoop/hbase/index/IndexedHTableDescriptor.java?
If yes, currently you need to create IndexHTableDescriptor and set columns and
properties of htabledescritor to IndexTableDescriptor object and add new
indices(its little ugly). We have in plan to support add and drop indices
dynamically.
In current work we are setting indices details as meta data to
HTableDescriptor(see HTableDescriptor.java#setValue(byte[], byte[])) so need
not have special table descritor(IndexedTableDescriptor).
For ex:
{code}
TableIndices tableIndices = new TableIndices();
tableIndices.addIndex(iSpec);
htd.setValue(Constants.INDEX_SPEC_KEY, tableIndices.toByteArray());
{code}
> Secondary index support through coprocessors
>
>
> Key: HBASE-9203
> URL: https://issues.apache.org/jira/browse/HBASE-9203
> Project: HBase
> Issue Type: New Feature
>Reporter: rajeshbabu
>Assignee: rajeshbabu
> Attachments: SecondaryIndex Design.pdf, SecondaryIndex
> Design_Updated.pdf, SecondaryIndex Design_Updated_2.pdf
>
>
> We have been working on implementing secondary index in HBase and open
> sourced on hbase 0.94.8 version.
> The project is available on github.
> https://github.com/Huawei-Hadoop/hindex
> This Jira is to support secondary index on trunk(0.98).
> Following features will be supported.
> - multiple indexes on table,
> - multi column index,
> - index based on part of a column value,
> - equals and range condition scans using index, and
> - bulk loading data to indexed table (Indexing done with bulk load)
> Most of the kernel changes needed for secondary index is available in trunk.
> Very minimal changes needed for it.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13906872#comment-13906872 ] hoar commented on HBASE-9203: - hi rajeshbabu , how do i add a new index on existing table. I get the HTableDescriptor using HBaseAdmin.getTableDescriptor(table name ), and how to convert it to IndexedHTableDescriptor? > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Reporter: rajeshbabu >Assignee: rajeshbabu > Attachments: SecondaryIndex Design.pdf, SecondaryIndex > Design_Updated.pdf, SecondaryIndex Design_Updated_2.pdf > > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13897580#comment-13897580 ] rajeshbabu commented on HBASE-9203: --- bq. What abt the Master and collocation and balancer side changes? As of now there are two APIs to update/remove region location in the custom balancer. These APIs should be called when region is online or offline. > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Reporter: rajeshbabu >Assignee: rajeshbabu > Attachments: SecondaryIndex Design.pdf, SecondaryIndex > Design_Updated.pdf, SecondaryIndex Design_Updated_2.pdf > > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13897572#comment-13897572 ] ramkrishna.s.vasudevan commented on HBASE-9203: --- May be only the apis need to go in HBase and the other one will have in Phoenix for now. Later can merge to hbase also (i mean the actual customized load balancer). > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Reporter: rajeshbabu >Assignee: rajeshbabu > Attachments: SecondaryIndex Design.pdf, SecondaryIndex > Design_Updated.pdf, SecondaryIndex Design_Updated_2.pdf > > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13897569#comment-13897569 ] Anoop Sam John commented on HBASE-9203: --- What all core changes still we need to get in HBase? I know some changes in HRegion. What abt the Master and collocation and balancer side changes? Suggest raising those as subtasks to this JIra. Even the load balancer also raise as a Subtask here. Later we can raise in Phoenix also and copy the patch to there. > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Reporter: rajeshbabu >Assignee: rajeshbabu > Attachments: SecondaryIndex Design.pdf, SecondaryIndex > Design_Updated.pdf, SecondaryIndex Design_Updated_2.pdf > > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13897556#comment-13897556 ] rajeshbabu commented on HBASE-9203: --- Yes most of the code outside of HBase.The are few dependencies required to handle master restart scenarios and split scenarios. The changes are not present in HBase code base. > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Reporter: rajeshbabu >Assignee: rajeshbabu > Attachments: SecondaryIndex Design.pdf, SecondaryIndex > Design_Updated.pdf, SecondaryIndex Design_Updated_2.pdf > > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13897547#comment-13897547 ] James Taylor commented on HBASE-9203: - Wow, I didn't realize so much of the custom load balancer was outside of HBase. Would there be interest in submitting a patch to Phoenix for this? It could be done on the 4.0 branch that works with 0.98. Is the other 5% already in the HBase code base? > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Reporter: rajeshbabu >Assignee: rajeshbabu > Attachments: SecondaryIndex Design.pdf, SecondaryIndex > Design_Updated.pdf, SecondaryIndex Design_Updated_2.pdf > > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13897526#comment-13897526 ] ramkrishna.s.vasudevan commented on HBASE-9203: --- >>The only part that Phoenix would need is the custom load balancer that keeps >>two regions co-located. If that functionality was available This is like 95% outside the hbase code. I think some small part we had a dependency on the hbase code. Rajesh could clarify that as for the current status. We just wrote a custom balancer and all the logic of colocating the regions was in that balancer. So should be easy for Phoenix too. > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Reporter: rajeshbabu >Assignee: rajeshbabu > Attachments: SecondaryIndex Design.pdf, SecondaryIndex > Design_Updated.pdf, SecondaryIndex Design_Updated_2.pdf > > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13897450#comment-13897450 ] James Taylor commented on HBASE-9203: - Is it possible to break this JIRA down into smaller pieces? The only part that Phoenix would need is the custom load balancer that keeps two regions co-located. If that functionality was available, I believe that Phoenix could support a local index option to compliment its global index feature. Would it be possible for that to be a separate JIRA that is tackled first? > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Reporter: rajeshbabu >Assignee: rajeshbabu > Attachments: SecondaryIndex Design.pdf, SecondaryIndex > Design_Updated.pdf, SecondaryIndex Design_Updated_2.pdf > > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13868831#comment-13868831 ] ramkrishna.s.vasudevan commented on HBASE-9203: --- >>bout scan(e.g. there's a filter on the indexed column) performance. how to >>decide or evaluate when we do the query into user table directly and when we >>do the query into index table first then do the (multi-)get into user table ? You mean the dynamic decision whether to use index or not? Should that be user decision? > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Reporter: rajeshbabu >Assignee: rajeshbabu > Attachments: SecondaryIndex Design.pdf, SecondaryIndex > Design_Updated.pdf, SecondaryIndex Design_Updated_2.pdf > > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13868812#comment-13868812 ] Anoop Sam John commented on HBASE-9203: --- I think instead of the padding approach, can change this to having a separator byte. (0 byte) .. That should work out. > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Reporter: rajeshbabu >Assignee: rajeshbabu > Attachments: SecondaryIndex Design.pdf, SecondaryIndex > Design_Updated.pdf, SecondaryIndex Design_Updated_2.pdf > > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13868759#comment-13868759 ] Liang Xie commented on HBASE-9203: -- another problem: about scan(e.g. there's a filter on the indexed column) performance. how to decide or evaluate when we do the query into user table directly and when we do the query into index table first then do the (multi-)get into user table ? > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Reporter: rajeshbabu >Assignee: rajeshbabu > Attachments: SecondaryIndex Design.pdf, SecondaryIndex > Design_Updated.pdf, SecondaryIndex Design_Updated_2.pdf > > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13868757#comment-13868757 ] Liang Xie commented on HBASE-9203: -- to me, one main drawback of this design is the required padding mechanism, for "index name" section inside index table's row key, maybe we can give a doc to educate end-user, but other "indexed column(s) value(s)", are totally depend on real user scenario, e.g. most of indexed column value probably very short, say "a", and a few long value, say "abode...z", then even for the short "a" value, based on current design, we still need to pad to sth like "a000..0", am i correct? i don't have a better improvement idea through... > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Reporter: rajeshbabu >Assignee: rajeshbabu > Attachments: SecondaryIndex Design.pdf, SecondaryIndex > Design_Updated.pdf, SecondaryIndex Design_Updated_2.pdf > > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[
https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13860165#comment-13860165
]
rajeshbabu commented on HBASE-9203:
---
[~yuzhih...@gmail.com] Thanks for review.
bq. In figure CreateTableWithSplits, there is an arrow from CreateTableHandler
to IndexMasterObserver labeled CreateIndexTableCameout. I don't find such
callback in the patch. What does the arrow represent ?
Just wanted to represent coming back to IndexMasterObserver from
CreateTableHandler. There are no such call backs in the code, the arrow should
start from CreateTableHandler(by mistake drawn from HMaster).
bq. Can you explain the second sentence in more detail ?
Since the rowkey for the index table put is like this
{code}
startkey of index region + index name + indexed column(s) value(s) + user table
rowkey
{code}
The last two bytes represents starting position of actual rowkey.
bq. IndexLoadIncrementalHFile is the utility that does the loading into index
table.
Yes it should be IndexLoadIncrementalHFile utility. Nice catch Ted.
bq. Is IndexTsvImporterMapper this new class ?
Yes its IndexTsvImporterMapper. It will be used to prepare puts for both user
table index table from raw data.
IndexCreationMapper will be used to prepare index table puts from user table
data. I will add javadoc.
bq. there is a dummy CF under .indexTable directory. What's its purpose ?
Actually it's column family in index table name. Didn't decide the name at that
time so given some name.
> Secondary index support through coprocessors
>
>
> Key: HBASE-9203
> URL: https://issues.apache.org/jira/browse/HBASE-9203
> Project: HBase
> Issue Type: New Feature
>Reporter: rajeshbabu
>Assignee: rajeshbabu
> Attachments: SecondaryIndex Design.pdf, SecondaryIndex
> Design_Updated.pdf
>
>
> We have been working on implementing secondary index in HBase and open
> sourced on hbase 0.94.8 version.
> The project is available on github.
> https://github.com/Huawei-Hadoop/hindex
> This Jira is to support secondary index on trunk(0.98).
> Following features will be supported.
> - multiple indexes on table,
> - multi column index,
> - index based on part of a column value,
> - equals and range condition scans using index, and
> - bulk loading data to indexed table (Indexing done with bulk load)
> Most of the kernel changes needed for secondary index is available in trunk.
> Very minimal changes needed for it.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13860016#comment-13860016 ] Ted Yu commented on HBASE-9203: --- bq. First 2 bytes is the length of the start key of the index region and last 2 bytes is the offset to the actual table’s rowkey within the index rowkey. Can you explain the second sentence in more detail ? In section 5: bq. LoadIncrementalHFiles utility can be used for this. IndexLoadIncrementalHFile is the utility that does the loading into index table. For section 6. Bulk load data to indexed table: bq. Create a new mapper class which Is IndexTsvImporterMapper this new class ? There is another mapper, IndexCreationMapper, in the patch. Neither of them has class javadoc. In the figure for section 6, there is a dummy CF under .indexTable directory. What's its purpose ? > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Reporter: rajeshbabu >Assignee: rajeshbabu > Attachments: SecondaryIndex Design.pdf, SecondaryIndex > Design_Updated.pdf > > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13860012#comment-13860012 ] Ted Yu commented on HBASE-9203: --- In figure CreateTableWithSplits, there is an arrow from CreateTableHandler to IndexMasterObserver labeled CreateIndexTableCameout. I don't find such callback in the patch. What does the arrow represent ? > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Reporter: rajeshbabu >Assignee: rajeshbabu > Attachments: SecondaryIndex Design.pdf, SecondaryIndex > Design_Updated.pdf > > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13836456#comment-13836456 ] Jyothi Mandava commented on HBASE-9203: --- bq. Yes this is supported. Can bulk load data to user table. The ImportTSV like tool will then create index data for the index table Please use org.apache.hadoop.hbase.index.mapreduce.IndexImportTSV tool for updating index along with the user table data. > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Reporter: rajeshbabu >Assignee: rajeshbabu > Attachments: SecondaryIndex Design.pdf > > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13835616#comment-13835616 ] rajeshbabu commented on HBASE-9203: --- bq. This was also done. That code is available in Git Rajesh? Yes Anoop. Its available. We can use org.apache.hadoop.hbase.index.mapreduce.TableIndexer mapreduce job to create index for existing data. > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Reporter: rajeshbabu >Assignee: rajeshbabu > Attachments: SecondaryIndex Design.pdf > > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13835610#comment-13835610 ] Anoop Sam John commented on HBASE-9203: --- bq.1.Do you have verify tool to check user table and secondary index table 's data, this can ensure the data consistency? like VerifyReplication. It was there in the plan. Not yet done. +1 for having some tool for this bq.2.This solution may be contain secondary index bulkload, load the data to user table and secondary index table same time. Yes this is supported. Can bulk load data to user table. The ImportTSV like tool will then create index data for the index table bq.3.When I have user table and have used some time (abundant data in the table), I want to create index for some column. Is this scenario supported? This was also done. That code is available in Git Rajesh? > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Reporter: rajeshbabu >Assignee: rajeshbabu > Attachments: SecondaryIndex Design.pdf > > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13835598#comment-13835598 ] andylee commented on HBASE-9203: Your desgin documention are very good and explicit. I have some question: 1.Do you have verify tool to check user table and secondary index table 's data, this can ensure the data consistency? like VerifyReplication. 2.This solution may be contain secondary index bulkload, load the data to user table and secondary index table same time. 3.When I have user table and have used some time (abundant data in the table), I want to create index for some column. Is this scenario supported? > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Reporter: rajeshbabu >Assignee: rajeshbabu > Attachments: SecondaryIndex Design.pdf > > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13834207#comment-13834207 ] Andrew Purtell commented on HBASE-9203: --- I'm thinking of branching 0.98 on 12/1. However, there's no reason we can't take a patch for trunk and the branch (the delta would be very small) both soon thereafter if this is about ready. Definitely we can take the "kernel changes" right? > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Reporter: rajeshbabu >Assignee: rajeshbabu > Attachments: SecondaryIndex Design.pdf > > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13833676#comment-13833676 ] rajeshbabu commented on HBASE-9203: --- We wanted to target 0.98 branch but I am afraid that it might be too late as it will take at least couple of weeks to come up with patch. > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Reporter: rajeshbabu >Assignee: rajeshbabu > Attachments: SecondaryIndex Design.pdf > > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13833610#comment-13833610 ] Nicolas Liochon commented on HBASE-9203: Great! Do you target the 0.98 branch, even if it ? > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Reporter: rajeshbabu >Assignee: rajeshbabu > Attachments: SecondaryIndex Design.pdf > > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13833608#comment-13833608 ] rajeshbabu commented on HBASE-9203: --- Yes. I am working on this [~nkeywal]. First contributing the kernel changes needed for it and working on trunk patch. Thanks. > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Reporter: rajeshbabu >Assignee: rajeshbabu > Attachments: SecondaryIndex Design.pdf > > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13833603#comment-13833603 ] Nicolas Liochon commented on HBASE-9203: Any life here, btw? > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Reporter: rajeshbabu >Assignee: rajeshbabu > Attachments: SecondaryIndex Design.pdf > > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13744970#comment-13744970 ] ramkrishna.s.vasudevan commented on HBASE-9203: --- For the indexhalf store file reader, Yes, as i said in the above comment bq.Mostly it will be the same data but the thing is we would read the HFile twice to create the top and bottom files for the index region. For the namespaces I think we could move index table to a namespace so that it is not directly workable for the user something like a system table. > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.98.0 >Reporter: rajeshbabu >Assignee: rajeshbabu > Attachments: SecondaryIndex Design.pdf > > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13744928#comment-13744928 ] Anoop Sam John commented on HBASE-9203: --- bq.Since the midpoint for index table region may not be chosen for the split, it is possible that the daughter regions of index region may have (quite) different amount of data. How can we mitigate this effect ? I think this wont happen. The daughter regions of index region will have similar size proportions as that of the actual table region. For an actual table region there are 10 entries in that and now its is split as 6,4. Consider there one index for the data. The index region before the split will contain 10 entries in it and after the spilt the daugthers will have 6,4 entries each. Only diff will be the way the half file reading will happen. In case of normal table there is a clear split point wrt RK and the readers can readup split point/ read from split point. But for the index region, both the daugther region readers need to start from the begin position and check whether each entry belongs to it or not and traverse. After a split the compaction will happen using the HalfFileReader and split it into 2 physical files. So the reader overhead is only temporal. > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.98.0 >Reporter: rajeshbabu >Assignee: rajeshbabu > Attachments: SecondaryIndex Design.pdf > > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13744906#comment-13744906 ] rajeshbabu commented on HBASE-9203: --- bq. What if user specifies two indices on the same set of columns but in different order ? The two indices will be considered as independent. Lets suppose c1,c2 are the columns and indices are [idx1,c1,c2],[idx2,c2,c1] For some queries where c1 has equal condition and c2 has range condition then idx1 will considered optimal index. Similarly if c2 has equal condition then idx2 will be considered as optimal index. bq. Was there a typo above (w.r.t. the parameter name) ? Presently not given any configuration for max index name length. I will add a configuration for this. This is different from maxValueLength. bq. If the max length is changed after index table is created and written to, would index row key parsing be affected ? I assume you mean max value length? Yes index rowkey parsing will have effect, We need to validate this properly and should not allow to modify maxValueLength. bq. What if the maxValueLength cannot cover the actual value for indexed column from the user query ? That record wont come in result. bq. Since the midpoint for index table region may not be chosen for the split, it is possible that the daughter regions of index region may have (quite) different amount of data. How can we mitigate this effect ? Index region should split along with user region split only, we are avoiding splitting index region from external split requests. There should not be a problem. > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.98.0 >Reporter: rajeshbabu >Assignee: rajeshbabu > Attachments: SecondaryIndex Design.pdf > > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13744677#comment-13744677 ] Ted Yu commented on HBASE-9203: --- bq. Yes - I think we can mitigate this (with namespaces in place now). Can you elaborate a bit more on the above ? Index table would be created in its own namespace ? > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.98.0 >Reporter: rajeshbabu >Assignee: rajeshbabu > Attachments: SecondaryIndex Design.pdf > > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13744672#comment-13744672 ] ramkrishna.s.vasudevan commented on HBASE-9203: --- >>The actual load balancer would support favored nodes feature. We should >>consider the placement not just for one replica of HFiles but the second and >>third replica as well. Okie. So the HFiles of main and index table should be placed on the same node. >>The above implementation may result in the size of index table region being >>larger than the size of user table region. I assume split policy would be >>customized for index table. Currently there is no customization here. Mostly it will be the same data but the thing is we would read the HFile twice to create the top and bottom files for the index region. Your concern seems valid, need to check those instances. >>Using fixed table name suffix is not flexible. Yes - I think we can mitigate this (with namespaces in place now). >>here is second createTable call issued by Master. What if this second call >>has some IOException ? overall there is some lack of mechanism to know what happens on the CP calls. Most of the master operations are now happening on the RPC handler thread rather than the main threadmainly to reduce the impact of this problem. > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.98.0 >Reporter: rajeshbabu >Assignee: rajeshbabu > Attachments: SecondaryIndex Design.pdf > > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13744170#comment-13744170 ] Ted Yu commented on HBASE-9203: --- bq. As the user table and index table data is with different row keys, we won’t be able to split the index region same way as user region at some middle point. Since the midpoint for index table region may not be chosen for the split, it is possible that the daughter regions of index region may have (quite) different amount of data. How can we mitigate this effect ? bq. Now what is different with the split of index region is the role of HalfStoreFileReader. Typo above: different with the split of user table region > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.98.0 >Reporter: rajeshbabu >Assignee: rajeshbabu > Attachments: SecondaryIndex Design.pdf > > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13744090#comment-13744090 ] Ted Yu commented on HBASE-9203: --- bq. When the index is on more than one column we need to take values of all the columns and add them in the rowkey in the same order of columns added to the index specifier. What if user specifies two indices on the same set of columns but in different order ? bq. This index name max length is configurable using the parameter <<>>. Was there a typo above (w.r.t. the parameter name) ? If the max length is changed after index table is created and written to, would index row key parsing be affected ? bq. For this we take the maxValueLength for an indexed column when table is created. What if the maxValueLength cannot cover the actual value for indexed column from the user query ? bq. In order to make integers compare correctly, we need to flip the sign bit. OrderedBytes is in 0.95 and trunk. It should help with this use case. > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.98.0 >Reporter: rajeshbabu >Assignee: rajeshbabu > Attachments: SecondaryIndex Design.pdf > > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13744051#comment-13744051 ] Ted Yu commented on HBASE-9203: --- bq. We will hide the index table regions from this actual load balancer. The actual load balancer would support favored nodes feature. We should consider the placement not just for one replica of HFiles but the second and third replica as well. bq. The index table can be created with a name which is the actual table name suffixed with a predefined pattern ‘_idx’. Using fixed table name suffix is not flexible. For CreateTableWithSplits diagram, the flow is different from that of TableCreate diagram - there is second createTable call issued by Master. What if this second call has some IOException ? Would the user table be dropped in that case ? bq. When the user table is having N number of indices, N entries need to get added into the index region for this user table row. The above implementation may result in the size of index table region being larger than the size of user table region. I assume split policy would be customized for index table. > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.98.0 >Reporter: rajeshbabu >Assignee: rajeshbabu > Attachments: SecondaryIndex Design.pdf > > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13741915#comment-13741915 ] Anoop Sam John commented on HBASE-9203: --- bq.If the number of index tables increases, what would the put performance be like ? Just to make it clear, There will be only one index table per actual table, irrespective of the #indices. Yes need to calculate the put performance throughput with say 2,3,5 indices also (?) > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.98.0 >Reporter: rajeshbabu >Assignee: rajeshbabu > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13741904#comment-13741904 ] rajeshbabu commented on HBASE-9203: --- bq. The trunk patch would put new classes for secondaryindex in a new module, I assume. Yes Ted. > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.98.0 >Reporter: rajeshbabu >Assignee: rajeshbabu > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13741875#comment-13741875 ] ramkrishna.s.vasudevan commented on HBASE-9203: --- bq.What potential bug may exist in IndexHalfStoreFileReader ? Some apis were not implemented. Those things needs to be revisited. >>For TestIndexRegionObserver, what does testHDP2938 cover ? Yes, renaming would be necessary. Some internal github issue has been raised for the same. [~lhofhansl] bq.Even if we add these there would be some other kernel changes, though, right? Yes there are few more. In the kernel patch attached check for HRegion, SplitTransaction, and some client side changes. > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.98.0 >Reporter: rajeshbabu >Assignee: rajeshbabu > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[
https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13741521#comment-13741521
]
Ted Yu commented on HBASE-9203:
---
The trunk patch would put new classes for secondaryindex in a new module, I
assume.
On page 15 of Anoop's presentation, one index table was created, right ?
If the number of index tables increases, what would the put performance be like
?
{code}
// TODO check some implementations here. Seems buggy!!
public class IndexHalfStoreFileReader extends StoreFile.Reader {
{code}
What potential bug may exist in IndexHalfStoreFileReader ?
For TestIndexRegionObserver, what does testHDP2938 cover ?
> Secondary index support through coprocessors
>
>
> Key: HBASE-9203
> URL: https://issues.apache.org/jira/browse/HBASE-9203
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 0.98.0
>Reporter: rajeshbabu
>Assignee: rajeshbabu
>
> We have been working on implementing secondary index in HBase and open
> sourced on hbase 0.94.8 version.
> The project is available on github.
> https://github.com/Huawei-Hadoop/hindex
> This Jira is to support secondary index on trunk(0.98).
> Following features will be supported.
> - multiple indexes on table,
> - multi column index,
> - index based on part of a column value,
> - equals and range condition scans using index, and
> - bulk loading data to indexed table (Indexing done with bulk load)
> Most of the kernel changes needed for secondary index is available in trunk.
> Very minimal changes needed for it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13741264#comment-13741264 ] Lars Hofhansl commented on HBASE-9203: -- [~rajesh23], I looked through the two patches. They are innocuous - just adding new coproc hooks and defaulting their actions in BaseRegionObserver (all observer should be derived from that class, although I assume it is possible that somebody implemented RegionObserver directly). Even if we add these there would be some other kernel changes, though, right? > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.98.0 >Reporter: rajeshbabu >Assignee: rajeshbabu > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13741153#comment-13741153 ] Ted Yu commented on HBASE-9203: --- bq. we can add index tables to system namespaces How do you deal with permissions ? Suppose user A has access to tables in namespace A. Would you grant permission on hbase:A-index to user A ? I think using metadata across releases would be more consistent. > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.98.0 >Reporter: rajeshbabu >Assignee: rajeshbabu > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13741141#comment-13741141 ] rajeshbabu commented on HBASE-9203: --- bq. What if some existing table(s) at customer site already uses the above suffix ? How about using metadata in HTableDescriptor for this purpose ? In trunk we can add index tables to system namespaces,so we distinguish user tables and index tables. In 94 its better to use metadata. I will handle the above case and other comments in trunk patch. For github raised an issue: https://github.com/Huawei-Hadoop/hindex/issues/16 > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.98.0 >Reporter: rajeshbabu >Assignee: rajeshbabu > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13741050#comment-13741050 ] ramkrishna.s.vasudevan commented on HBASE-9203: --- We need not back port I think. May be new hooks we can contribute to trunk. > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.98.0 >Reporter: rajeshbabu >Assignee: rajeshbabu > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13740996#comment-13740996 ] rajeshbabu commented on HBASE-9203: --- These two issues made kernel changes patch bigger. These are not available in 0.94 version so we can back port them. HBASE-6633(Adding new hooks to the split flow - For roll backs and one final hook after split is completed either successfully or failed) HBASE-5584(Coprocessor hooks can be called in the respective handlers) > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.98.0 >Reporter: rajeshbabu >Assignee: rajeshbabu > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[
https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13740511#comment-13740511
]
Ted Yu commented on HBASE-9203:
---
First batch of comments.
{code}
+ public static boolean isIndexTable(String tableName) {
+return tableName.endsWith(INDEX_TABLE_SUFFIX);
+ }
{code}
What if some existing table(s) at customer site already uses the above suffix ?
How about using metadata in HTableDescriptor for this purpose ?
{code}
+boolean indexEnabled = this.conf.getBoolean("hbase.use.secondary.index",
false);
{code}
Please introduce constant for the above config.
Looking at changes in BaseMasterObserver.java :
{code}
+ public void preCreateTableHandler(final
ObserverContext ctx,
+ HTableDescriptor desc, HRegionInfo[] regions) throws IOException {
+ }
{code}
It would be nice if the above changes are extracted from the big patch into a
small one which adds additional hooks to coprocessor API - this way it would be
easier for the small patch to be accepted.
{code}
+ * Copyright 2011 The Apache Software Foundation
{code}
Year is not needed.
For MasterObserverExt.java:
{code}
* This is an extension for the MasterObserver interface. The APIs added into
this interface are not
+ * exposed by HBase. This is internally being used by CMWH HBase. Customer
should not make use of
+ * this interface points.
{code}
Please modify the above comment - it is not consistent with Apache HBase.
> Secondary index support through coprocessors
>
>
> Key: HBASE-9203
> URL: https://issues.apache.org/jira/browse/HBASE-9203
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 0.98.0
>Reporter: rajeshbabu
>Assignee: rajeshbabu
>
> We have been working on implementing secondary index in HBase and open
> sourced on hbase 0.94.8 version.
> The project is available on github.
> https://github.com/Huawei-Hadoop/hindex
> This Jira is to support secondary index on trunk(0.98).
> Following features will be supported.
> - multiple indexes on table,
> - multi column index,
> - index based on part of a column value,
> - equals and range condition scans using index, and
> - bulk loading data to indexed table (Indexing done with bulk load)
> Most of the kernel changes needed for secondary index is available in trunk.
> Very minimal changes needed for it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13740049#comment-13740049 ] Lars Hofhansl commented on HBASE-9203: -- I'd certainly be interested to put the plumbing (additional coprocessor hooks, etc) into 0.94 - barring any risks of course, such that the indexing stuff would be purely additional code, external to HBase. > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.98.0 >Reporter: rajeshbabu >Assignee: rajeshbabu > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13739889#comment-13739889 ] ramkrishna.s.vasudevan commented on HBASE-9203: --- When there are more than one index we would definitely have some hit on the write through put but how much matters. Given the readings we can definitely improve on that. Cross region transaction, supporting WAlEdits for different regions in a single Edit are some areas that would improve the write performance. > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.98.0 >Reporter: rajeshbabu >Assignee: rajeshbabu > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13739872#comment-13739872 ] Anoop Sam John commented on HBASE-9203: --- > impact of creating 1, 2 or more index tables There will be only one index table associated with base table for 1 or more indices on base table... JFYI. Yes the write throughput to the base table when there are 1,2 or more index are there, to be measured. > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.98.0 >Reporter: rajeshbabu >Assignee: rajeshbabu > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13739860#comment-13739860 ] Ted Yu commented on HBASE-9203: --- Given a base table, what is the impact of creating 1, 2 or more index tables associated with the base table ? The impact would first be measured by write performance to the base table. Thanks > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.98.0 >Reporter: rajeshbabu >Assignee: rajeshbabu > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13739331#comment-13739331 ] rajeshbabu commented on HBASE-9203: --- Sorry Anoop, our comments crossed. Yes we have few more issues need to committed. Then we can still reduce kernel changes. > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.98.0 >Reporter: rajeshbabu >Assignee: rajeshbabu > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13739323#comment-13739323 ] rajeshbabu commented on HBASE-9203: --- Lars, 0.94 there core classes changed are high only. But in trunk, there are very few changes in kernel. 0.94 kernel changes patch available here https://github.com/Huawei-Hadoop/hindex/blob/master/dev-support/SecondaryIndexKernelChanges.patch Is it fine or you want me to upload here? > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.98.0 >Reporter: rajeshbabu >Assignee: rajeshbabu > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13739318#comment-13739318 ] Anoop Sam John commented on HBASE-9203: --- There are still some more CP hooks which are not contributed back. Also same changes to make the across region (within a RS) transaction kind of stuff. I guess this is there in our future roadmap to support a cross region transaction(regions within same RS). I opened one JIRA for that. Based on the patch will get inputs for making a patch for that issue. Will work on that later if it sounds interesting generally. > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.98.0 >Reporter: rajeshbabu >Assignee: rajeshbabu > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13739313#comment-13739313 ] Anoop Sam John commented on HBASE-9203: --- https://github.com/Huawei-Hadoop/hindex/blob/master/dev-support/SecondaryIndexKernelChanges.patch > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.98.0 >Reporter: rajeshbabu >Assignee: rajeshbabu > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13739307#comment-13739307 ] Lars Hofhansl commented on HBASE-9203: -- Is it possible to provide the 0.94 changes as a patch as well? Looks like this would be a large patch with many of the core classes changed. > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.98.0 >Reporter: rajeshbabu >Assignee: rajeshbabu > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13739228#comment-13739228 ] rajeshbabu commented on HBASE-9203: --- Ted, I will upload by tomorrow. > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.98.0 >Reporter: rajeshbabu >Assignee: rajeshbabu > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13738301#comment-13738301 ] Ted Yu commented on HBASE-9203: --- A high level design doc would help reviewers understand this feature. > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Reporter: rajeshbabu >Assignee: rajeshbabu > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13738042#comment-13738042 ] linwukang commented on HBASE-9203: -- Nice! 3Q Anoop! > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Reporter: rajeshbabu >Assignee: rajeshbabu > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13738040#comment-13738040 ] Anoop Sam John commented on HBASE-9203: --- You are correct. This can be done. https://github.com/Huawei-Hadoop/hindex/issues/6 Thanks > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Reporter: rajeshbabu >Assignee: rajeshbabu > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13738035#comment-13738035 ] linwukang commented on HBASE-9203: -- @Anoop yes, that what i exactly means. I read the ppt that published on the HBTC, It seems not address to this issue. > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Reporter: rajeshbabu >Assignee: rajeshbabu > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13738026#comment-13738026 ] Anoop Sam John commented on HBASE-9203: --- You mean the index usage for a query like col LIKE "ab%" ? > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: New Feature >Reporter: rajeshbabu >Assignee: rajeshbabu > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13738015#comment-13738015 ] linwukang commented on HBASE-9203: -- One more question, is this implement support wildcard query with the index table? > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: Bug >Reporter: rajeshbabu >Assignee: rajeshbabu > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13738014#comment-13738014 ] linwukang commented on HBASE-9203: -- Well done! Why this jira is tag as a bug? tag as 'New Feature' may be better. > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: Bug >Reporter: rajeshbabu >Assignee: rajeshbabu > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9203) Secondary index support through coprocessors
[ https://issues.apache.org/jira/browse/HBASE-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13737916#comment-13737916 ] rajeshbabu commented on HBASE-9203: --- I am working on this. I will upload the patch in couple of days. > Secondary index support through coprocessors > > > Key: HBASE-9203 > URL: https://issues.apache.org/jira/browse/HBASE-9203 > Project: HBase > Issue Type: Bug >Reporter: rajeshbabu >Assignee: rajeshbabu > > We have been working on implementing secondary index in HBase and open > sourced on hbase 0.94.8 version. > The project is available on github. > https://github.com/Huawei-Hadoop/hindex > This Jira is to support secondary index on trunk(0.98). > Following features will be supported. > - multiple indexes on table, > - multi column index, > - index based on part of a column value, > - equals and range condition scans using index, and > - bulk loading data to indexed table (Indexing done with bulk load) > Most of the kernel changes needed for secondary index is available in trunk. > Very minimal changes needed for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
