Again, can we please bring this discussion back from discussions of
implementations to security?
Does the fact that you three were discussing implementations imply that
you do not think this invalidates one of the core tenets (security
first) of Accumulo?
Keith, Russ, myself (and possible others) were discussing this at the
hackathon after the Accumulo Summit, and I think our consensus were
We need a generic pluggable mechanism for injecting arbitrary user counters
into the RFiles. We can then use these counters in custom compaction
strategies, or other analysis. We can aggregate these counters at the
tablet, and table levels, and expose them in the API.
These counters could store information about visibility frequencies, number
of delete entries, etc.
The interface might just be a Function<Entry<Key,Value>,Map<String, Long>>.
In the discussion, there were lots of variations on the theme, though. So,
the actual implementation could vary. But, having something like this could
support a large number of use cases beyond just the histogram case.
On Tue, Oct 11, 2016 at 10:06 PM Josh Elser<josh.el...@gmail.com> wrote:
Trivially. We could do something more intelligent like also cache it in
metadata (updating with compactions). Don't read too much into the
implementation at this point; it was just the first idea I had about how we
could do it :). I'm more concerned with the idea and its security
implications right now.
In general, it seems like people are ok with it protected by a new
permission role. Do you have more to add, Mike? Was your comment based on
your interpretation of how Accumulo works or more a concern about
implementing such a feature?
On Oct 11, 2016 21:29,<dlmar...@comcast.net> wrote:
So, to get the set of visibilities used in a table, we would have to open
all of the rfiles?
From: Dylan Hutchison [mailto:dhutc...@cs.washington.edu]
Sent: Tuesday, October 11, 2016 3:43 PM
To: Accumulo Dev List
Subject: Re: [DISCUSS] Would a visibility histogram on a table be
Interesting idea. It begs the question: should we allow any custom
the RFile level? If RFile indexes were user-extensible, then a
would be something any developer could write. That said, we can still
include such an index as an example, and if we did it could be used by
The RFile-level sampling followed this path. I would support further
similar to it, though I admit I don't know how difficult a job it
Bonus points if the index information could be accessed from iterators
same way that sampled data can.
I can't speak to the appropriateness of visibility histograms on the
*by default*, but it would be a strictly useful feature if it could be
a conf option.
On Tue, Oct 11, 2016 at 12:20 PM, Josh Elser<josh.el...@gmail.com>
Today at Accumulo Summit, our own Russ Weeks gave a talk. One topic
mentioned was the lack of insight into the distribution of data
with certain visibilities in a table. He presented an example similar
Image a hypothetical system backed by Accumulo which stores medical
information. There are three labels in the system: PRIVATE,
ANONYMIZED, and PUBLIC. PRIVATE data is that which could reasonably
considered to identify the individual. ANONYMIZED data is some
version of the attribute that retains some portion of the original
value, but is missing enough context to not identify the individual
(e.g. converting the name "Josh Elser" to "J E"). PUBLIC data is for
attributes which are cannot identify the individual.
Doctors would be able to read the PRIVATE data, while researchers
could only read the ANONYMIZED and PUBLIC data. This leads to a
question: how much of each kind of data is in the system? Without
knowing how much data is in the system, how can some application
developer (who does not have the ability to read all of the PRIVATE
data) know that their application is returning an reasonably correct
amount of data? (there are many examples of questions which could be
answer on this data alone)
Concretely, this histogram would look like (50 records with PRIVATE,
50 with ANONYMIZED, and 20 with PUBLIC; 120 records total):
Technically, I think this would actually be relatively simple to
implement. Inside of each RFile, we could maintain some histogram of
the visibilities observed in that file. This would allow us to very
easily report how much data in each table has each visibility label.
However, would this feature be harmful to one of the core tenants of
Accumulo? Or, is acknowledging the existence of data in Accumulo with
a certain visibility acceptable? Would a new permission to use such
API to access this information be sufficient to protect the data?