There should be a static setRanges(Configuration, Collection<Range>) method somewhere in the type hierarchy of AccumuloInputFormat which lets you specify the Range[s].

Not using the TimestampFilter (not being able to use the timestamp for this filtering), you have two options to perform row-filtering based on the value for a column.


1) Perform the filter on the client side. If you have significant amounts of data, this will be slow. Even with MapReduce, this may present a significant overhead in processing.

2) Implement a custom Accumulo Iterator which can perform this filtering in Accumulo itself. I would recommend using the WholeRowIterator in conjunction with this filter you would implement.

At a high level, configure the WholeRowIterator to aggregate all of the Keys in one row into a single Key-Value pair. Then, implement and configure a custom Iterator (ideally, extend the abstract Filter iterator) which deserializes that single key-value pair back into many, extract the createdDate column, and make a decision as to whether or not the row should be returned to the client.

On the client, you would then unpack the serialized row into many key-value pairs again.

Bob Cook wrote:
Josh,

Thanks. I was able to get TimestampFilter to works for my needs.  But I
originally wanted "createdDate" as our application creates that date
which is known to the user
and may be different than accumulo timestamp due to when the data
actually got processed into accumulo.

So if I wanted to use the ColumnFamily "createdDate" and it's value,
  what java code would I have to write?

I looked at the AccumuloInputFormat class, but confused on how to
specify the "range" for the date range that I'm interested in..

So would I use the TimestampFilter Class similar to how I'm using it in
the "scanner.addScanIterator", but instead using
"AcculoInputFormat.addIterator(job, is), as below.

IteratorSetting is = newIteratorSetting(30, TimestampFilter.class);
TimestampFilter.setRange(is, startDate, endDate);
AccumuloInputFormat.addIterator(job, is);

Or could I use
is.addOption("start", startDate);
is.addOption("end", endDate);

NOTE: for me "TimestempFilter.setRange"  nor "TimestampFilter.setStart
and TimestampFilter.setEnd didn't seem to work.

On Sun, Oct 16, 2016 at 2:05 PM, Josh Elser <josh.el...@gmail.com
<mailto:josh.el...@gmail.com>> wrote:

    The TimestampFilter will return only the Keys whose timestamp fall
    in the range you specify. The timestamp is an attribute on every
    Key, a long value which, when not set by the client at write time,
    is the number of millis since the epoch. You specify the numeric
    range of timestamps you want. This is a post-filter operation --
    Accumulo must still read all of the data in the table.

    You need to tell *us* what the time component you're actually
    filtering on: the timestamp on each Key, or the createdDate column
    in each row.

    MapReduce is likely more efficient to do this batch processing (as
    MapReduce is a batch processing system). See the AccumuloInputFormat
    class.

    Bob Cook wrote:

        All,

        I'm new to accumulo and inherited this project to extract all
        data from
        accumulo (assembled as a "document" by RowID) into another web
        service.

        So I started with SimpleReadClient.java to "scan" all data, and
        built a
        "document" based on the RowID, ColumnFamily and Value. Sending
        this "document" to the service.
        Example data.
        ID CF CV
        RowID_1 createdDate "2015-01-01:00:00:01 UTC"
        RowID_1 data "this is a test"
        RowID_1 title "My test title"

        RowID_2 createdDate "2015-01-01:12:01:01 UTC"
        RowID_2 data "this is test 2"
        RowID_2 title "My test2 title"

        ...

        So my table is pretty simple,  RowID, ColumnFamily and Value (no
        ColumnQualifier)

        I need to process one Billion "OLD" unique RowIDs (a years worth of
        data) on a live system that is ingesting "new data" at a rate of
        about a
        4million RowIds a day.
        i.e. I need to process data from September 2015 - September
        2016, not
        worrying about new data coming in.

        So I'm thinking I need to run multiple processes to extract ALL
        the data
        in this "data range" to be more efficient.
        Also, it may allow me to run the processes at a lower priority
        and at
        off-hours of the day when traffic is less.

        My issue is how do I specify the "range" to scan, and how do I
        specify.

        1. Is using the "createdDate" a good idea, if so how would I
        specify the
        range for it.

        2. How about the TimestampFilter?   If I specify my start to end to
        "equal" a day (about 4 Million unique RowIDs),
        Will this get me all ColumnFamily and Values for a given RowID?  Or
        could I miss something because it's timestamp
        was the next day.  I don't really understand Timestamps wrt
        Accumulo.

        3. Does a map-reduce job make sense.  If so, how would I specify.


        Thanks,

        Bob


Reply via email to