Hi all,
To maintain consistency between data table and its index tables, we have to do 
a transactional update cross regions on different region servers. For 
non-transactional table, we cannot guarantee this consistency for mutable 
global secondary index. Here are the problems of existing solutions:
1. disable index write
  a) update system.catalog to change index status, and set timestamp, may lead 
to chain failures
  b) partially rebuild index may not be a good solution for production env, 
because:
     b1) may execute for a long time for large table (several TBs)
     b2) there might be only a few inconsistent data which needs to be caught 
up but we have to do a full table time-ranged scan over the data table
     b3) if there are deletes/updates and a major compaction took place, it'll 
leave dirty data in index tables
  c) selects that hits the disabled index will degenerate to full table scan 
against data table which may quickly exhausts the read ability of the whole 
cluster


2. disable data table write
  a) selects that hits index still works
  b) actually data table write is not disabled, but raise an exception. So  
still needs to rebuild index tables when index regions are back online, which 
has the same issues in 1.b
  c) as index rebuild is needed, system.catalog still needs to be updated, so 
chain failure may still happen.


What should be guaranteed:
1. absolutely no chain failure
2. absolutely no inconsistency no matter what happened
3. selects that hit the index will not degenerate


New solution:
1. When update index failed, retry forever until succeed
2. Do the same retry when replaying WAL
3. No need to update catalog table to avoid potential chain failures
4. This index failure policy is an option that can be switched on/off


About this solution:
1. Simple
2. When update index failed, we give up the write ability to maintain 
consistency and read ability. This is acceptable for mutable global index as 
its read ability is more important.
3. No need to rebuild index afterwards, as long as the pending retries 
complete, indexes will be in sync.
4. In worst case, some or all of the RS will not be able to write.
5. We cannot handle index updates failure elegantly because we are not doing 
real transactions. So this solution is a simple but effective way to achieve 
consistency without transactions, though there is a price.


What does everybody think?


Thanks
William

Reply via email to