> does that depend on setting up Kerberos Careful - Kerberos and data-in-motion encryption are orthogonal. If you use kerberos without also setting up TLS then the payload will be in the clear.
(At least that's my understanding of how they work together.) Bear On Mon, Jan 6, 2020 at 9:45 PM Hariharan <hariharan...@gmail.com> wrote: > For 1) you can set up transparent encryption at the root directory level > for HDFS. However this works at file level and not volume level. For volume > level encryption you will have to use something like LUKS only. > > For 2), in addition to the steps mentioned in "data confidentiality", you > may also need to set up Encrypted Shuffle > <https://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/EncryptedShuffle.html>, > depending on your use-cases for Hadoop. > > Thanks, > Hariharan > > On Tue, Jan 7, 2020 at 1:16 AM Daniel Howard <danny...@toldme.com> wrote: > > > > Hello, > > > > I am working on getting Hadoop running within our organization. Our > high-level use case is to be able to say we're running with end-to-end > encryption. It looks like there are two major strategies for getting this > done in Hadoop: > > > > A) HDFS Transparent Encryption: > https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html > > B) Secure Mode: > https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SecureMode.html > > > > In our case, we are less concerned with Kerberos' user-level > authentication, but we want node-to-node encryption and encryption-at-rest. > With cluster applications, I typically achieve encryption-at-rest with LUKS > and then enable an application's TLS settings to achieve > encryption-in-motion. > > > > What is my best strategy for Hadoop? Here are a couple of questions: > > > > 1) The docs say I have to create a new directory, but can I configure > HDFS Transparent Encryption to operate across an entire volume? > > 2) If I just need encrypted-in-motion, can I do just the "Data > confidentiality" part of the Secure Mode doc, or does that depend on > setting up Kerberos? > > > > Thank You! > > -danny > > > > -- > > http://dannyman.toldme.com > > > On Tue, Jan 7, 2020 at 1:16 AM Daniel Howard <danny...@toldme.com> wrote: > >> Hello, >> >> I am working on getting Hadoop running within our organization. Our >> high-level use case is to be able to say we're running with end-to-end >> encryption. It looks like there are two major strategies for getting this >> done in Hadoop: >> >> A) HDFS Transparent Encryption: >> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html >> B) Secure Mode: >> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SecureMode.html >> >> In our case, we are less concerned with Kerberos' user-level >> authentication, but we want node-to-node encryption and encryption-at-rest. >> With cluster applications, I typically achieve encryption-at-rest with LUKS >> and then enable an application's TLS settings to achieve >> encryption-in-motion. >> >> What is my best strategy for Hadoop? Here are a couple of questions: >> >> 1) The docs say I have to create a new directory, but can I configure >> HDFS Transparent Encryption to operate across an entire volume? >> 2) If I just need encrypted-in-motion, can I do just the "Data >> confidentiality" part of the Secure Mode doc, or does that depend on >> setting up Kerberos? >> >> Thank You! >> -danny >> >> -- >> http://dannyman.toldme.com >> > -- Bear Giles Sr. Software Engineer bgi...@snaplogic.com Mobile: 720-749-7876 <http://www.snaplogic.com/about-us/jobs> *SnapLogic Inc | 1825 South Grant Street | San Mateo CA | USA * This message is confidential. It may also be privileged or otherwise protected by work product immunity or other legal rules. If you have received it by mistake, please let us know by e-mail reply and delete it from your system; you may not copy this message or disclose its contents to anyone. The integrity and security of this message cannot be guaranteed on the Internet.
