[
https://issues.apache.org/jira/browse/PHOENIX-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
binlijin updated PHOENIX-1740:
------------------------------
Description:
I want to share a idea for implment global secondary index, it is eventually
consistent, the index table will out of sync with the primary table for a short
periods. We do add a notify family for the primary table, when the primary
table a row modified, we will add a corresponding qualifier for the row and the
time to record the change(capture change), and write with the data. A daemon
will continue scanning the notify family and get the data to write to the
index table, when write success we will delete the corresponding qualifier in
notify family, so the index table will be a little behind the primary table.
The idea inspired by Percolator, you can read the idea in
http://blog.cloudera.com/blog/2014/11/how-cerner-uses-cdh-with-apache-kafka/.
We improve the scan notifications performance because we store the
notifications in coprocessor and it is in memstore, and the hbase notify family
for persistent storage.
was:
I want to share a idea for implment global secondary index, it is eventually
consistent, the index table will out of sync with the primary table for a short
periods. We do add a notify family for the primary table, when the primary
table a row modified, we will add a corresponding qualifier for the row and the
time to record the change, and write with the data. A daemon will continue
scanning the notify family and get the data to write to the index table, when
write success we will delete the corresponding qualifier in notify family, so
the index table will be a little behind the primary table. The idea inspired by
Percolator, you can read the idea in
http://blog.cloudera.com/blog/2014/11/how-cerner-uses-cdh-with-apache-kafka/.
We improve the scan notifications performance because we store the
notifications in coprocessor and it is in memstore, and the hbase notify family
for persistent storage.
> A Eventually Consistent Secondary Index solution for Global Indexing
> --------------------------------------------------------------------
>
> Key: PHOENIX-1740
> URL: https://issues.apache.org/jira/browse/PHOENIX-1740
> Project: Phoenix
> Issue Type: New Feature
> Reporter: binlijin
>
> I want to share a idea for implment global secondary index, it is eventually
> consistent, the index table will out of sync with the primary table for a
> short periods. We do add a notify family for the primary table, when the
> primary table a row modified, we will add a corresponding qualifier for the
> row and the time to record the change(capture change), and write with the
> data. A daemon will continue scanning the notify family and get the data to
> write to the index table, when write success we will delete the corresponding
> qualifier in notify family, so the index table will be a little behind the
> primary table. The idea inspired by Percolator, you can read the idea in
> http://blog.cloudera.com/blog/2014/11/how-cerner-uses-cdh-with-apache-kafka/.
> We improve the scan notifications performance because we store the
> notifications in coprocessor and it is in memstore, and the hbase notify
> family for persistent storage.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
