Agreed Jorn. Disk encryption is one option that will help to secure data but how do I know at which location Spark is spilling temp file, shuffle data and application data ?
Thanks Shashi On Fri, Apr 28, 2017 at 3:54 PM, Jörn Franke <jornfra...@gmail.com> wrote: > You can use disk encryption as provided by the operating system. > Additionally, you may think about shredding disks after they are not used > anymore. > > > On 28. Apr 2017, at 14:45, Shashi Vishwakarma <shashi.vish...@gmail.com> > wrote: > > > > Hi All > > > > I was dealing with one the spark requirement here where Client (like > Banking Client where security is major concern) needs all spark processing > should happen securely. > > > > For example all communication happening between spark client and server > ( driver & executor communication) should be on secure channel. Even when > spark spills on disk based on storage level (Mem+Disk), it should not be > written in un-encrypted format on local disk or there should be some > workaround to prevent spill. > > > > I did some research but could not get any concrete solution.Let me know > if someone has done this. > > > > Any guidance would be a great help. > > > > Thanks > > Shashi >
