Chandni, The reason I proposed this is because, currently we don't have any way to have the streaming data in encrypted format. What Tim mentioned is also a good idea and the DAG level attribute can decide for it.
But I feel its a useful utility for one who wants to keep the data secure at every stage of processing. -Chinmay. ~ Chinmay. On Tue, Dec 15, 2015 at 1:45 PM, Chandni Singh <[email protected]> wrote: > The processing of data, which is the entire pipeline, will run usually in a > secured cluster. The data may need to be encrypted only when it is done > processing by the DAG and leaves the cluster. > > > > On Tue, Dec 15, 2015 at 12:10 AM, Timothy Farkas <[email protected]> > wrote: > > > I think encryption of data sent across the wire and operator logic are > > orthogonal. The user should just have to set DAG level attribute to > > enable/disable encryption, without having to write any encryption related > > code. I think this would require changes to the Buffer Server publisher > and > > subscriber though. > > > > On Mon, Dec 14, 2015 at 11:27 PM, Chandni Singh <[email protected] > > > > wrote: > > > > > When we are dealing with secured data, the usual scenarios are that you > > get > > > encrypted data. > > > This data need to decrypt and then perform other functions on it. The > > > output of the dag is then encrypted. > > > > > > In the past we have solved these use cases by performing > > > decryption/encryption in the operator. > > > IMO the operator approach works better because these processes may > > require > > > invoking utilities and also operators can be configured easily using > > > properties. > > > > > > Chandni > > > > > > On Mon, Dec 14, 2015 at 10:34 PM, Sandesh Hegde < > [email protected] > > > > > > wrote: > > > > > > > Well we have committers from bank, their feedback will be really > > > valuable. > > > > > > > > On Mon, Dec 14, 2015 at 10:30 PM Priyanka Gugale < > > > [email protected] > > > > > > > > > wrote: > > > > > > > > > Sounds good. This is good feature for banks and security domain. > > > > > One suggestion: We can do key management ourself at application > (may > > be > > > > by > > > > > providing default keys) and there should be an option to override > > keys > > > if > > > > > user really want to do so. > > > > > > > > > > -Priyanka > > > > > > > > > > On Tue, Dec 15, 2015 at 11:37 AM, Chinmay Kolhatkar < > > > > > [email protected] > > > > > > wrote: > > > > > > > > > > > Hi All, > > > > > > > > > > > > I wanted to propose an idea using which one can have encrypted > > stream > > > > > > flowing in a DAG. > > > > > > > > > > > > Basically, the idea is to create a new EncryptedInputPort which > > will > > > > > extend > > > > > > from DefaultInputPort and will return a StreamCodec object which > > will > > > > > take > > > > > > care of encryption/decryption. > > > > > > As the same StreamCodec object will be used at OutputPort, the > > > > encryption > > > > > > can be done in toByteArray method at Output port and decryption > can > > > be > > > > > done > > > > > > in fromByteArray at Input port. > > > > > > > > > > > > By default we can support some basic encryption algorithms like > RSA > > > and > > > > > DSA > > > > > > where user need to provide the key(s) to EncryptedInputPort. > > > > > > > > > > > > Any thoughts? > > > > > > > > > > > > ~ Chinmay. > > > > > > > > > > > > > > > > > > > > >
