Re: Adding KMS support to generic filesystem interface

2019-01-28 Thread Ismaël Mejía
Hello Udi, I took a look at your PR and let some coments.Please take a look at them because I think you can achieve what you want in a simpler way and without radical changes to the FileSystem (and File-related) interface(s). Let's continue the discussion in the PR and then we bring back the subj

Re: Adding KMS support to generic filesystem interface

2019-01-18 Thread Udi Meiri
Hi Ismaël, I'd like your feedback, especially from the AWS perspective. I wasn't aware of BEAM-3821, but I did create a JIRA for Cloud KMS support on GCS: https://issues.apache.org/jira/browse/BEAM-5959 Some details of my plan for KMS support: 1. Add KMS settings to sources and sinks. 2. Add a --k

Re: Adding KMS support to generic filesystem interface

2019-01-18 Thread Ismaël Mejía
Hello Udi, I implemented the support for KMS in Amazon and I am really interested in check your PR. However I won't have time to do it until next monday. I hope waiting a bit is ok with you if you want some feedback from me. I am curious if you considered or are aware of this issue: BEAM-3821 Sup

Adding KMS support to generic filesystem interface

2019-01-17 Thread Udi Meiri
Hi, I'd like to add support for creating files using a cloud Key Management System. A KMS allows you to audit, create, rotate, and disable encryption keys. Both AWS and GCP have such a service.. I wanted to show the community what I've been working on and see if there are any comments or objection