a) FileIO produces elements which are file descriptors, each descriptor represents one file. If you have many files to read then you will get parallel processing since multiple files will be read concurrently. You could get parallel reading in a single file but you would need to be able to split the decrypted file by knowing how to randomly seek in the encrypted stream and start reading the decrypted stream without having to decrypt everything before that point.
b) You need to add support to the underlying filesystem like S3[1] or GCS[2] to support file decryption. Note for solution a) and b), you can only get efficient parallel reading within a single file if you can efficiently seek within the "decrypted" stream. Many encryption methods do not allow for this and typically require you to decrypt everything up to where you want to read; making parallel reading of a single file extremely inefficient. This is a common problem for compressed files as well, you can't choose an arbitrary spot in the compressed file and start decompressing without decompressing everything before it. 1: https://github.com/apache/beam/blob/master/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/s3/S3FileSystem.java 2: https://github.com/apache/beam/blob/master/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/storage/GcsFileSystem.java On Tue, Aug 7, 2018 at 1:06 PM Aniruddh Sharma <[email protected]> wrote: > Thanks Lukasz for revert. > a) Have a further question. I may not be understanding it right. If I > embed this logic in ParDo then won't logic be in a way that it limits file > read to only one file as directly writing in ParDo or will it still be a > parallel operation .If it is parallel operation then please elaborate more. > > b) Is it possible to extend TextIO or AvroIO to decrypt keys (customer > supplied) , if yes then request to elaborate how to do it. > > Thanks > Aniruddh > > > Thanks > Aniruddh > > On Mon, Aug 6, 2018 at 8:30 PM, Lukasz Cwik <[email protected]> wrote: > >> I'm assuming your talking about using the Java SDK. Additional details >> about which SDK language, filesystem, and KMS you want to access would be >> useful if the approach below doesn't work for you. >> >> Your best bet would be use FileIO[1] followed by a ParDo that accesses >> the KMS to get the decryption key and wraps the readable channel with a >> decryptor. >> Note that you'll need to apply your own file parsing logic to pull out >> the records as you won't be able to use things like AvroIO/TextIO to do the >> record parsing for you. >> >> 1: >> https://beam.apache.org/documentation/sdks/javadoc/2.5.0/org/apache/beam/sdk/io/FileIO.html >> >> >> >> On Tue, Jul 31, 2018 at 10:27 AM [email protected] < >> [email protected]> wrote: >> >>> >>> what is the best suggested way to to read a file encrypted with a >>> customer key which is also wrapped in KMS >>> >> >
