Does EMRFS extend Hadoop FileSystem? If so, it seems like you would be able to do this by configuring the FileSystem plugin to use emrfs.
On Tue, Apr 7, 2015 at 3:29 PM, Ted Dunning <[email protected]> wrote: > Looking at the link that you provided, it appears that you are encrypting > entire data files. That probably makes it better to implement this as a > layer in the file access path. > > Drill doesn't do this just now, but it would be relatively easy to add, I > think. > > > > On Tue, Apr 7, 2015 at 3:26 PM, Ted Dunning <[email protected]> wrote: > > > > > Ahh... > > > > There is no magic that will handle decryption that you can plug into (at > > this time). > > > > > > > > On Tue, Apr 7, 2015 at 3:02 PM, Ganesha Muthuraman < > [email protected] > > > wrote: > > > >> The situation is this: > >> There is client side encrypted data on S3. There is an EMR cluster that > >> uses this as EMRFS. The EMR client reaches out to a custom java class > for > >> decrypting it. EMR does it using the envelope encryption method, > documented > >> on AWS. > >> > http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-plan-cse.html > >> My question was, is there a way that I can use the custom java module > >> that I have (aka EncryptionProvider) to work with Drill so that I can > >> achieve the same kind of envelope decryption that EMR does? Or does it > have > >> to be a completely new UDF that I use that in turn calls a custom Java > >> module that can decrypt this data? Apologies if my message is confusing. > >> -Ganesh > >> > Subject: Re: Drill to query Client-side encrypted data from S3 > >> > From: [email protected] > >> > Date: Tue, 7 Apr 2015 14:47:39 -0700 > >> > To: [email protected] > >> > > >> > Ganesh, > >> > > >> > When you say the keys are “custom controlled”, does that mean that > only > >> special logic within your Java application allows the data to be > properly > >> accessed ? There are several mechanisms within the S3 API such that > >> encryption/decryption occur transparently to the application. If your > >> data is accessible in that manner, it’s likely that simply setting the > >> correct properties and jar files for your Drill environment will allow > your > >> queries to access the data. > >> > > >> > — David > >> > > >> > On Apr 7, 2015, at 2:41 PM, Ganesha Muthuraman < > [email protected]> > >> wrote: > >> > > >> > > I am trying to use Drill to read from Amazon S3 where the data is > >> Client-side encrypted, meaning the keys to decrypt the data are custom > >> controlled. Is there a way I can use drill with this data given that I > have > >> a java module that can be called that will provide the master key to > >> decrypt the data on the fly? > >> > > My situation: A lot of the use cases that we have might work well > >> with the new approach of S3 client-side encryption, but for using drill > to > >> explore that data. So any pointers/help here will be much appreciated. > >> > > Thanks! > >> > > -Ganesh > >> > > >> > >> > > > > > -- Steven Phillips Software Engineer mapr.com
