Re: [DISCUSS] What to do about encryption at rest?

On Wed, Nov 4, 2015 at 7:38 PM, William Slacum wrote: > @Adam, column family level encryption can be useful for multi-tenant > environments, and I think it maps pretty well to the document > partitioning/sharding/wikisearch style tables. Things are trickier in > Accumulo than

Re: [DISCUSS] What to do about encryption at rest?

Can we file some JIRAs to build out a suite to test this and run the necessary tests? On Thu, Nov 5, 2015 at 11:17 AM, Christopher wrote: > My main concern using HDFS encryption vs. built-in Accumulo implementation > is possibly performance with respect to seeks. If we

Re: [DISCUSS] What to do about encryption at rest?

+1 I think this is the right step. My hunch is that some of the common data access patterns that we have in Accumulo (over HBase) is that the per-colfam encryption isn't quick as common a design pattern as it is for HBase (please tell me I'm wrong if anyone disagrees -- this is mostly a gut

Re: [DISCUSS] What to do about encryption at rest?

My main concern using HDFS encryption vs. built-in Accumulo implementation is possibly performance with respect to seeks. If we encrypt our indexed blocks independently (as we do now), I suspect our seeks would be more performant than relying on HDFS encryption, whose encrypted blocks may not fall

Re: [DISCUSS] What to do about encryption at rest?

JIRAs are fine, but I thought this thread was mostly addressing the fact that there doesn't seem to be a sustained interest in actually working on any of the JIRAs addressing that area of code. Am I wrong? Is there willingness from anybody to expend effort on this code? Even if not, we can still

Re: [DISCUSS] What to do about encryption at rest?

I think you have misidentified the two camps. There is a camp that believes we should phase out the code in favour of the HDFS encryption, and a camp that believes the code is sufficiently mature. I don't think there is a group that is interested in improving the state of things. On Thu, Nov 5,

Re: [DISCUSS] What to do about encryption at rest?

On Thu, Nov 5, 2015 at 12:17 PM, Christopher wrote: > My main concern using HDFS encryption vs. built-in Accumulo implementation > is possibly performance with respect to seeks. If we encrypt our indexed > blocks independently (as we do now), I suspect our seeks would be

Re: [DISCUSS] What to do about encryption at rest?

> Does anybody have a good diagram showing the architecture of HDFS encryption? Related: Can we collect the digram and design docs from the various implementation JIRAs and put them up on the Accumulo website? Every time that I've needed to reference them it's been a giant pain to go find them.

Re: [DISCUSS] What to do about encryption at rest?

Perhaps. I had interpreted some of Adam's comments ("The only thing that doesn't get encrypted is a temporary WAL recovery file. That is a project we should take on..."), as favoring improvements to the current state of things. As that has also been the focus of previous conversations about the

Re: [DISCUSS] What to do about encryption at rest?

With regards to adding features, it probably makes sense to talk about adding table/namespace crypto configuration separately from column-level encryption. Column-level encryption would require big changes to how we partition data, how we organize configuration information, and how we handle

Re: [DISCUSS] What to do about encryption at rest?

Just to moonwalk back a bit, I see a few things happening concurrently now. First is trying to get a consensus on where we want to go with the encryption at rest story in Accumulo. I see us having established that what we have is scoped down to working for WALs and RFiles, and if you happen to

Re: [DISCUSS] What to do about encryption at rest?

SGTM William Slacum wrote: Just to moonwalk back a bit, I see a few things happening concurrently now. First is trying to get a consensus on where we want to go with the encryption at rest story in Accumulo. I see us having established that what we have is scoped down to working for WALs and

Re: [DISCUSS] What to do about encryption at rest?

Bill, Do you envision one of the following as the driver behind finer-grained encryption?: 1. We would only encrypt certain columns in order to get better performance; 2. We would use different keys on different columns in order to revoke access to a column via the key store; 3. We would only

[DISCUSS] What to do about encryption at rest?

@Adam, column family level encryption can be useful for multi-tenant environments, and I think it maps pretty well to the document partitioning/sharding/wikisearch style tables. Things are trickier in Accumulo than in HBase since there isn't a 1:1 mapping between column families and files. The

Re: [DISCUSS] What to do about encryption at rest?

Thanks for exposing the issues on this. I had equated 'stale' with incomplete, but I was missing the point entirely. In this case, 'stale' equates to complete, working and stable (but not changing). On Sat, Oct 31, 2015 at 4:22 PM, Josef Roehrl - PHEMI wrote: > For this

Re: [DISCUSS] What to do about encryption at rest?

On Tue, Nov 3, 2015 at 11:10 AM Josh Elser wrote: > Josef Roehrl - PHEMI wrote: > > Thanks for exposing the issues on this. I had equated 'stale' with > > incomplete, but I was missing the point entirely. In this case, 'stale' > > equates to complete, working and stable

Re: [DISCUSS] What to do about encryption at rest?

On Mon, Nov 2, 2015 at 1:37 PM, Keith Turner wrote: > > > On Mon, Nov 2, 2015 at 12:27 PM, William Slacum wrote: > >> Is "the code being 'at rest'" you making a funny about active development? >> Making sure I haven't lost my ability to get jokes :) >> >> I

Re: [DISCUSS] What to do about encryption at rest?

Josef Roehrl - PHEMI wrote: Thanks for exposing the issues on this. I had equated 'stale' with incomplete, but I was missing the point entirely. In this case, 'stale' equates to complete, working and stable (but not changing). (pedantically) minus the intermediate-WAL recovery files not

Re: [DISCUSS] What to do about encryption at rest?

There's another way to look at the state of Accumulo's encryption at rest: 1. Encryption at rest works great for what it does, and the code being "at rest" isn't necessarily a problem 2. Several organizations are using Accumulo's encryption at rest effectively in operations 3. Encryption at rest

Re: [DISCUSS] What to do about encryption at rest?

Responses inline. Adam On Nov 1, 2015 9:58 AM, "Christopher" wrote: > > 1. I'm not sure I'd call an incomplete solution 'great'. What it does is > provide partial encryption-at-rest protection (unless you're running > without walogs, and have good integration with some

Re: [DISCUSS] What to do about encryption at rest?

1. I'm not sure I'd call an incomplete solution 'great'. What it does is provide partial encryption-at-rest protection (unless you're running without walogs, and have good integration with some external secure key management faculty, and then it's probably fine). 2. I'm concerned that anybody

Re: [DISCUSS] What to do about encryption at rest?

+1 on #2 if anyone wants to pick it back up later, we can always pull it back out of the git history. how would implementation work? I know it's not in the public API, but if there are folks relying on it we'd essentially be locking them out of upgrades. would we provide migration tools? On

Re: [DISCUSS] What to do about encryption at rest?

For this reason, we were just thinking of waiting for Encryption at Rest with HDFS. Presumably, Accumulo could optimize encryption if it implemented encryption itself with a few trade-offs. On Fri, Oct 30, 2015 at 10:22 PM, William Slacum wrote: > So I've been looking into

Re: [DISCUSS] What to do about encryption at rest?

William Slacum wrote: So I've been looking into options for providing encryption at rest, and it seems like what Accumulo has is abandonware from a project perspective. There is no official documentation on how to perform encryption at rest, and the best information from its status comes from

[DISCUSS] What to do about encryption at rest?

So I've been looking into options for providing encryption at rest, and it seems like what Accumulo has is abandonware from a project perspective. There is no official documentation on how to perform encryption at rest, and the best information from its status comes from year (or greater) old

Re: [DISCUSS] What to do about encryption at rest?

Some colleagues have expressed interest in examining the current state of our rfile encryption with the expectation to suggest improvements or contributions to close the gap. I don't know a timeline for any of that, if that interest even bears out in terms of concrete action, so I don't know how