Rick,
Sorry for the late answer, I just figured I misunderstood your example in
the first place. When I saw:
static final Text col2Id = new Text("c2");
I thought this was your family and not a "family:qualifier" (but I now
understand that you meant ) which had me wondered why you did not use column
keys at all. But you are.
Regards your question at the end, the Bigtable paper states (at the end of
page 2 and beginning of page 3) that applications may need to avoid
collisions and this is also the case in HBase. If two puts are so close in
time that they are inserted at the same timestamp, the first one that is
written will simply be overwritten by the second one. It means that you
cannot have more versions than the number of timestamps for a given cell
(row+column key) since it's the same thing.
Hope this helps and again, sorry for the misunderstanding.
Regards,
J-D
On Mon, Jul 21, 2008 at 5:54 PM, Rick Hangartner <[EMAIL PROTECTED]>
wrote:
> Hi Jean-Daniel,
>
> Thanks for the info on the HBase transaction model.
>
> I'm not sure I quite understand your subsequent question though, and I'm
> interested in learning whether you are suggesting a preferred alternative to
> what we are doing.
>
> To set the background, I should begin by noting that by the nature of our
> application, the table in this case will always be dense. We probably are
> not using HBase optimally in that regard, since such data could be stored in
> an RDBMs. However, we mainly want to use HBase because of the scale, and to
> run map-reduce across the data set for different computations where RDBMs
> data-mining tools would be both overkill in some cases and inefficient in
> others.
>
> We have used the transaction model described for several reasons. By
> creating a column family ("key"), which has numerous column members
> ("key:c1", "key:c2", etc), we have a complex key for a column family
> ("value") that we can filter on in the map phase of map-reduce computations.
> Each row in the HBase table corresponds to a data set for an entity in our
> system, which it seems would be stored relatively efficiently in the HFS.
> Timestamps and versions become relevant as indices for data whose temporal
> course is of interest. (Figure 1 in the seminal Google paper "Bigtable, A
> Distributed Storage System for Structured Data" is in fact a good picture of
> what we are doing --- except our rows are dense in the sense that every
> column has an entry for every timestamp because of the nature of our data.)
>
> The ability to insure that all columns written in a single transaction are
> returned together also allows us to retrieve data sequences of interest to
> us in non-map-reduce environments. In this case, we have used the
> transaction model described because of the issue of apparently not being
> able to easily retrieve all versions of two different columns in a row,
> along with timestamps, with the get(), getRow(), and obtainScanner(),
> methods the HTable v0.1.x native client. By including a "key:timestamp"
> column in the "key" column-family, we can get an explicit timestamp value
> that can be combined with the row key value as a key for fast retrieval of
> column values in the "value" column-family. In that sense we are using a
> column key (the "timestamp" column), which just happens to be a timestamp,
> to access other other column values in the row.
>
> Which actually brings me to another question. I think our model highlights
> something about "timestamps" and "versions" which is not quite clear.
> Namely, the Google figure and the model I've described sort of implicitly
> assumes that row key + timestamp forms a unique key for an entry in each
> column. In everything else I can find, "versions" generally refer to a set
> of entries in a column (row-column cell) where each value has a unique
> timestamp. But if timestamp is not stored with sufficient resolution, e.g.
> 1 sec resolution, one can postulate that a situation in which two put()s
> close enough in time that two entries in a cells could in theory have the
> same row key + timestamp. This suggests that set of versions of a cell are
> not 1-1 with the set of unique timestamps on the cell entries. Put another
> way, could one retrieve X versions of a row-column cell, but only find Y < X
> unique timestamps on those X versions of the cell? Is there any additional
> explanation about versions viz a viz timestamps you can point me to that
> helps sort this out better?
>
> So finally, given all this, are you thinking about other ways to use column
> keys rather than timestamps we should be considering?
>
> Thanks,
> Rick
>
>
> On Jul 21, 2008, at 8:23 AM, Jean-Daniel Cryans wrote:
>
> Rick,
>>
>> Yes and yes, but why not using the column keys instead of the timestamps?
>>
>> J-D
>>
>> On Sat, Jul 19, 2008 at 1:47 AM, Rick Hangartner <[EMAIL PROTECTED]>
>> wrote:
>>
>> Hi,
>>>
>>> This is a question about the HBase transaction model.
>>>
>>> Suppose I have a table with two columns "c1" and "c2". Now assume for
>>> each
>>> timestamp in each row, I have a entry in each column. That is, assume
>>> the
>>> table is ALWAYS written such that it is "dense" (e.g. a complete
>>> relation)
>>> rather than sparse (e.g. a partial relation) using the transaction
>>> semantics:
>>>
>>> HTable table = new HTable(conf, new Text("test"));
>>> static final Text rowId = new Text("row_num");
>>> static final Text col1Id = new Text("c1");
>>> static final Text col2Id = new Text("c2");
>>>
>>> long lockid = table.startUpdate(rowId);
>>>
>>> // always write all columns in a row
>>> table.put( lockid, col1Id, val );
>>> table.put( lockid, col2Id, val );
>>>
>>> table.commit( lockid, timestamp );
>>>
>>> Also assume that the two columns are read as something like:
>>>
>>> byte[][] c1Vals = table.get( rowId, col1Id, versions );
>>> byte[][] c2Vals = table.get( rowId, col2Id, versions );
>>>
>>>
>>> Is it guaranteed that for each index value i, c1Vals[i][] and
>>> c2Vals[i][]
>>> are the two column entries originally written with the same timestamp?
>>>
>>> Also, is something like:
>>>
>>> byte[][] c1Vals = table.get( rowId, col1Id, MAX_VALUE );
>>>
>>> sufficient to guarantee all versions are returned in the "get"
>>> operations?
>>>
>>> Thanks,
>>> Rick
>>>
>>>
>>>
>>>
>
