Hi hadoop developers,
I'm looking for a hint or inspiration for a problem I would love to
solve with the hadoop platform but it is not map reduce related.
My data structure is builded from rows and each row has a set of
columns and column values.
For example row key: cnn.com column keys: user1, user2 ... values: 2,
1... etc.
I plan to realize the structure using writable's to be able storing
my data in dfs.
My problem is now that I would love to read very fast one rowkey-
columnKey- (may be timestamp) value.
My first idea was to have a row writable that contains all column-
value tuples. This can be written into a mapfile and find using a row
key.
However it require me to load all column-values into the memory to
find a given column value by column key. In case the I have to many
columns this would not be very fast and memory consuming.
The second idea was to write the rowkey and some meta data to the map
file followed by a set of column key column value tuples.
The problem here is that a map file only accepts sorted keys, means
my column -key required to build form a row key - column key tuple.
This solution would blow up my data very much since I have to write
the row key as many times I have columns. :-/
Any hints how I can reuse the existing writers and readers to solve
me problem or should I implement my own reader and writer?
Thank you very much for any hints.
Stefan
