Thanks Lukasz for revert. a) Have a further question. I may not be understanding it right. If I embed this logic in ParDo then won't logic be in a way that it limits file read to only one file as directly writing in ParDo or will it still be a parallel operation .If it is parallel operation then please elaborate more.
b) Is it possible to extend TextIO or AvroIO to decrypt keys (customer supplied) , if yes then request to elaborate how to do it. Thanks Aniruddh Thanks Aniruddh On Mon, Aug 6, 2018 at 8:30 PM, Lukasz Cwik <[email protected]> wrote: > I'm assuming your talking about using the Java SDK. Additional details > about which SDK language, filesystem, and KMS you want to access would be > useful if the approach below doesn't work for you. > > Your best bet would be use FileIO[1] followed by a ParDo that accesses the > KMS to get the decryption key and wraps the readable channel with a > decryptor. > Note that you'll need to apply your own file parsing logic to pull out the > records as you won't be able to use things like AvroIO/TextIO to do the > record parsing for you. > > 1: https://beam.apache.org/documentation/sdks/javadoc/2. > 5.0/org/apache/beam/sdk/io/FileIO.html > > > > On Tue, Jul 31, 2018 at 10:27 AM [email protected] < > [email protected]> wrote: > >> >> what is the best suggested way to to read a file encrypted with a >> customer key which is also wrapped in KMS >> >
