[jira] [Commented] (HBASE-5723) Simple Design of Secondary Index
[ https://issues.apache.org/jira/browse/HBASE-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249907#comment-13249907 ] Zhihong Yu commented on HBASE-5723: --- @Todd: Do you think the attached proposal belongs to HBASE-3340 ? Simple Design of Secondary Index Key: HBASE-5723 URL: https://issues.apache.org/jira/browse/HBASE-5723 Project: HBase Issue Type: New Feature Components: coprocessors Reporter: ShiXing Priority: Minor Attachments: Simple Design of HBase SecondaryIndex.pdf Use coprocessor to create index. And primary tables' compaction to purge the stale data. Attach file is the Design of the Seconday Index. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5723) Simple Design of Secondary Index
[ https://issues.apache.org/jira/browse/HBASE-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249927#comment-13249927 ] Todd Lipcon commented on HBASE-5723: This seems to give even fewer guarantees than HBASE-3340 (which as I understood it used the WAL to ensure reasonably timely consistency of the index table). I'm still of the opinion that these things should start as separate projects, rather than inside HBase core, and only if we see a high demand for them and good maintenance, we should pull it in. I don't think there is much demand for non-consistent indexes. Simple Design of Secondary Index Key: HBASE-5723 URL: https://issues.apache.org/jira/browse/HBASE-5723 Project: HBase Issue Type: New Feature Components: coprocessors Reporter: ShiXing Priority: Minor Attachments: Simple Design of HBase SecondaryIndex.pdf Use coprocessor to create index. And primary tables' compaction to purge the stale data. Attach file is the Design of the Seconday Index. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5723) Simple Design of Secondary Index
[ https://issues.apache.org/jira/browse/HBASE-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247127#comment-13247127 ] ShiXing commented on HBASE-5723: Difficulty and risk 1. During primary table compact, the StoreScanner just read family.getMaxVersions() versions' data, the old data will be skiped, but the old data's indices should be deleted, So we must read all version data for indices' delete, and family.getMaxVersions() version for store's compaction. Also the timestamp of columns that we want to delete the indices should also have not TimeRange, now it is System.CurrentTimeMillis() - ttl; Simple Design of Secondary Index Key: HBASE-5723 URL: https://issues.apache.org/jira/browse/HBASE-5723 Project: HBase Issue Type: New Feature Components: coprocessors Reporter: ShiXing Priority: Minor Attachments: Simple Design of HBase SecondaryIndex.pdf Use coprocessor to create index. And primary tables' compaction to purge the stale data. Attach file is the Design of the Seconday Index. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5723) Simple Design of Secondary Index
[ https://issues.apache.org/jira/browse/HBASE-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247438#comment-13247438 ] Todd Lipcon commented on HBASE-5723: The design doc doesn't make any reference to consistency. How do you ensure consistency of the index? To do proper indexes you really need cross-table transactions, or at least a mechanism for eventual consistency if that's all you plan on providing. Simple Design of Secondary Index Key: HBASE-5723 URL: https://issues.apache.org/jira/browse/HBASE-5723 Project: HBase Issue Type: New Feature Components: coprocessors Reporter: ShiXing Priority: Minor Attachments: Simple Design of HBase SecondaryIndex.pdf Use coprocessor to create index. And primary tables' compaction to purge the stale data. Attach file is the Design of the Seconday Index. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5723) Simple Design of Secondary Index
[ https://issues.apache.org/jira/browse/HBASE-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13248021#comment-13248021 ] Xing Shi commented on HBASE-5723: - Todd, you mean the consistency of index table? Because the index table is also a HBase table, so it will use the mvcc to ensure the consistency of the index table. Mainly there are 2 steps when do put. Step1 : It will insert data into index table. Step2 : It will insert data into primary table. After step1, the client can read the index, but may be the step2 has not done. So client will find there is no data are associated with the index, and it will read the old version index. Here it is weak consistency, and at last the index and data will be eventual consistency except as follows:. If the step2 failed, we will not delete the new index， the index is not correct(there is no row in the primary table associated with this index by the rowkey, the value, and the same timestamp) ant it is Weak consistency. For these stale indices, we will not delete it forever(may be we can delete them by period job to detect the not correct indice). Simple Design of Secondary Index Key: HBASE-5723 URL: https://issues.apache.org/jira/browse/HBASE-5723 Project: HBase Issue Type: New Feature Components: coprocessors Reporter: ShiXing Priority: Minor Attachments: Simple Design of HBase SecondaryIndex.pdf Use coprocessor to create index. And primary tables' compaction to purge the stale data. Attach file is the Design of the Seconday Index. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5723) Simple Design of Secondary Index
[ https://issues.apache.org/jira/browse/HBASE-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13248024#comment-13248024 ] Todd Lipcon commented on HBASE-5723: If this implementation isn't giving you any consistency, then why is it any better than just having the client do two puts? I think for us to consider adding an indexing library to HBase proper, it needs to have stronger semantics. The idea that an index may be incorrect in the case of failures does not seem like something we should ship. Simple Design of Secondary Index Key: HBASE-5723 URL: https://issues.apache.org/jira/browse/HBASE-5723 Project: HBase Issue Type: New Feature Components: coprocessors Reporter: ShiXing Priority: Minor Attachments: Simple Design of HBase SecondaryIndex.pdf Use coprocessor to create index. And primary tables' compaction to purge the stale data. Attach file is the Design of the Seconday Index. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
