Hello, we are thinking about using Hbase table as a simple queue which
will dispatch the work for a mapreduce job, as well as real time
fetching of data to present to end user. In simple terms, suppose you
had a data source table and a queue table. The queue table has a
smaller set of Rows that point to Values which in turn point to
Perma-set table, which has large collection of Rows. (so Queue{Row,
Value} -> Perma-Set {Row, Value}). Or Q-Value -> P-Row. Our Goal is
to look up which Rows to retrieve from the Perma-Set table by looking
through the Queue. Once the lookup into the Queue is done, the Row
from the Queue must be deleted to avoid the same process of Perma-Set
lookup be done twice; We expect many concurrent lookups to happen, so
I assume the first thing we need to do is to have a client that does
the work is acquire a lock on the Queue Row, process the work, then
Remove the Queue Row.
Has anyone done something similar before? Any gotchas we should be away of?
Thanks.
-Jack
