Re: In-transit Data Encryption in EMR
Hi Vinay, Apologies for the inactivity on this thread, I was occupied with some critical fixes for 1.3.1. 1. Can anyone please explain me how do you test if SSL is working correctly ? Currently I am just relying on the logs. AFAIK, if any of the SSL configuration settings are enabled (*.ssl.enabled) and your job is running fine, then everything should be functioning. 2. Wild Card is not working with the keytool command, can you please let me know what is the issue with the following command: The wildcard option only works for wildcarding subdomains. For example, SAN=*.domain.com On 9 June 2017 at 4:33:46 PM, vinay patil (vinay18.pa...@gmail.com) wrote: Hi Guys, Can anyone please provide me solution to my queries. On Jun 8, 2017 11:30 PM, "Vinay Patil" <[hidden email]> wrote: Hi Guys, I am able to setup SSL correctly, however the following command does not work correctly and results in the error I had mailed earlier flink run -m yarn-cluster -yt deploy-keys/ TestJob.jar Few Doubts: 1. Can anyone please explain me how do you test if SSL is working correctly ? Currently I am just relying on the logs. 2. Wild Card is not working with the keytool command, can you please let me know what is the issue with the following command: keytool -genkeypair -alias ca -keystore: -ext SAN=dns:node1.* Regards, Vinay Patil On Mon, Jun 5, 2017 at 8:43 PM, vinay patil [via Apache Flink User Mailing List archive.] <[hidden email]> wrote: Hi Gordon, The yarn session gets created when I try to run the following command: yarn-session.sh -n 4 -s 2 -jm 1024 -tm 3000 -d --ship deploy-keys/ However when I try to access the Job Manager UI, it gives me exception as : javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target I am able to see the Job Manager UI when I imported the CA certificate to java truststore on EMR master node : keytool -keystore /etc/alternatives/jre/lib/security/cacerts -importcert -alias FLINKSSL -file ca.cer Does this mean that SSL is configured correctly ? I can see in the Job Manager configurations and also in th e logs. Is there any other way to verify ? Also the keystore and truststore password should be masked in the logs which is not case. 2017-06-05 14:51:31,135 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: security.ssl.enabled, true 2017-06-05 14:51:31,136 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: security.ssl.keystore, deploy-keys/ca.keystore 2017-06-05 14:51:31,136 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: security.ssl.keystore-password, password 2017-06-05 14:51:31,136 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: security.ssl.key-password, password 2017-06-05 14:51:31,136 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: security.ssl.truststore, deploy-keys/ca.truststore 2017-06-05 14:51:31,136 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: security.ssl.truststore-password, password Regards, Vinay Patil If you reply to this email, your message will be added to the discussion below: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/In-transit-Data-Encryption-in-EMR-tp13455p13490.html To start a new topic under Apache Flink User Mailing List archive., email [hidden email] To unsubscribe from Apache Flink User Mailing List archive., click here. NAML View this message in context: Re: In-transit Data Encryption in EMR Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: In-transit Data Encryption in EMR
Hi Guys, Can anyone please provide me solution to my queries. On Jun 8, 2017 11:30 PM, "Vinay Patil" <vinay18.pa...@gmail.com> wrote: > Hi Guys, > > I am able to setup SSL correctly, however the following command does not > work correctly and results in the error I had mailed earlier > > flink run -m yarn-cluster -yt deploy-keys/ TestJob.jar > > > Few Doubts: > 1. Can anyone please explain me how do you test if SSL is working > correctly ? Currently I am just relying on the logs. > > 2. Wild Card is not working with the keytool command, can you please let > me know what is the issue with the following command: > keytool -genkeypair -alias ca -keystore: -ext SAN=dns:node1.* > > > Regards, > Vinay Patil > > On Mon, Jun 5, 2017 at 8:43 PM, vinay patil [via Apache Flink User Mailing > List archive.] <ml+s2336050n13490...@n4.nabble.com> wrote: > >> Hi Gordon, >> >> The yarn session gets created when I try to run the following command: >> yarn-session.sh -n 4 -s 2 -jm 1024 -tm 3000 -d --ship deploy-keys/ >> >> However when I try to access the Job Manager UI, it gives me exception as >> : >> javax.net.ssl.SSLHandshakeException: >> sun.security.validator.ValidatorException: >> PKIX path building failed: >> sun.security.provider.certpath.SunCertPathBuilderException: >> unable to find valid certification path to requested target >> >> I am able to see the Job Manager UI when I imported the CA certificate >> to java truststore on EMR master node : >> keytool -keystore /etc/alternatives/jre/lib/security/cacerts -importcert >> -alias FLINKSSL -file ca.cer >> >> >> Does this mean that SSL is configured correctly ? I can see in the Job >> Manager configurations and also in th e logs. Is there any other way to >> verify ? >> >> Also the keystore and truststore password should be masked in the logs >> which is not case. >> >> >> >> >> >> >> *2017-06-05 14:51:31,135 INFO >> org.apache.flink.configuration.GlobalConfiguration- Loading >> configuration property: security.ssl.enabled, true 2017-06-05 14:51:31,136 >> INFO org.apache.flink.configuration.GlobalConfiguration- >> Loading configuration property: security.ssl.keystore, >> deploy-keys/ca.keystore 2017-06-05 14:51:31,136 INFO >> org.apache.flink.configuration.GlobalConfiguration- Loading >> configuration property: security.ssl.keystore-password, password 2017-06-05 >> 14:51:31,136 INFO org.apache.flink.configuration.GlobalConfiguration >> - Loading configuration property: security.ssl.key-password, password >> 2017-06-05 14:51:31,136 INFO >> org.apache.flink.configuration.GlobalConfiguration- Loading >> configuration property: security.ssl.truststore, deploy-keys/ca.truststore >> 2017-06-05 14:51:31,136 INFO >> org.apache.flink.configuration.GlobalConfiguration- Loading >> configuration property: security.ssl.truststore-password, password* >> >> >> Regards, >> Vinay Patil >> >> >> -- >> If you reply to this email, your message will be added to the discussion >> below: >> http://apache-flink-user-mailing-list-archive.2336050.n4. >> nabble.com/In-transit-Data-Encryption-in-EMR-tp13455p13490.html >> To start a new topic under Apache Flink User Mailing List archive., email >> ml+s2336050n1...@n4.nabble.com >> To unsubscribe from Apache Flink User Mailing List archive., click here >> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code=1=dmluYXkxOC5wYXRpbEBnbWFpbC5jb218MXwxODExMDE2NjAx> >> . >> NAML >> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer=instant_html%21nabble%3Aemail.naml=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >> > > -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/In-transit-Data-Encryption-in-EMR-tp13455p13609.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: In-transit Data Encryption in EMR
Hi Guys, I am able to setup SSL correctly, however the following command does not work correctly and results in the error I had mailed earlier flink run -m yarn-cluster -yt deploy-keys/ TestJob.jar Few Doubts: 1. Can anyone please explain me how do you test if SSL is working correctly ? Currently I am just relying on the logs. 2. Wild Card is not working with the keytool command, can you please let me know what is the issue with the following command: keytool -genkeypair -alias ca -keystore: -ext SAN=dns:node1.* Regards, Vinay Patil On Mon, Jun 5, 2017 at 8:43 PM, vinay patil [via Apache Flink User Mailing List archive.] <ml+s2336050n13490...@n4.nabble.com> wrote: > Hi Gordon, > > The yarn session gets created when I try to run the following command: > yarn-session.sh -n 4 -s 2 -jm 1024 -tm 3000 -d --ship deploy-keys/ > > However when I try to access the Job Manager UI, it gives me exception as > : > javax.net.ssl.SSLHandshakeException: > sun.security.validator.ValidatorException: > PKIX path building failed: > sun.security.provider.certpath.SunCertPathBuilderException: > unable to find valid certification path to requested target > > I am able to see the Job Manager UI when I imported the CA certificate to > java truststore on EMR master node : > keytool -keystore /etc/alternatives/jre/lib/security/cacerts -importcert > -alias FLINKSSL -file ca.cer > > > Does this mean that SSL is configured correctly ? I can see in the Job > Manager configurations and also in th e logs. Is there any other way to > verify ? > > Also the keystore and truststore password should be masked in the logs > which is not case. > > > > > > > *2017-06-05 14:51:31,135 INFO > org.apache.flink.configuration.GlobalConfiguration- Loading > configuration property: security.ssl.enabled, true 2017-06-05 14:51:31,136 > INFO org.apache.flink.configuration.GlobalConfiguration- > Loading configuration property: security.ssl.keystore, > deploy-keys/ca.keystore 2017-06-05 14:51:31,136 INFO > org.apache.flink.configuration.GlobalConfiguration- Loading > configuration property: security.ssl.keystore-password, password 2017-06-05 > 14:51:31,136 INFO org.apache.flink.configuration.GlobalConfiguration > - Loading configuration property: security.ssl.key-password, password > 2017-06-05 14:51:31,136 INFO > org.apache.flink.configuration.GlobalConfiguration- Loading > configuration property: security.ssl.truststore, deploy-keys/ca.truststore > 2017-06-05 14:51:31,136 INFO > org.apache.flink.configuration.GlobalConfiguration- Loading > configuration property: security.ssl.truststore-password, password* > > > Regards, > Vinay Patil > > > ------ > If you reply to this email, your message will be added to the discussion > below: > http://apache-flink-user-mailing-list-archive.2336050. > n4.nabble.com/In-transit-Data-Encryption-in-EMR-tp13455p13490.html > To start a new topic under Apache Flink User Mailing List archive., email > ml+s2336050n1...@n4.nabble.com > To unsubscribe from Apache Flink User Mailing List archive., click here > <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code=1=dmluYXkxOC5wYXRpbEBnbWFpbC5jb218MXwxODExMDE2NjAx> > . > NAML > <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer=instant_html%21nabble%3Aemail.naml=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/In-transit-Data-Encryption-in-EMR-tp13455p13598.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: In-transit Data Encryption in EMR
Hi Gordon, The yarn session gets created when I try to run the following command: yarn-session.sh -n 4 -s 2 -jm 1024 -tm 3000 -d --ship deploy-keys/ However when I try to access the Job Manager UI, it gives me exception as : javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target I am able to see the Job Manager UI when I imported the CA certificate to java truststore on EMR master node : keytool -keystore /etc/alternatives/jre/lib/security/cacerts -importcert -alias FLINKSSL -file ca.cer Does this mean that SSL is configured correctly ? I can see in the Job Manager configurations and also in th e logs. Is there any other way to verify ? Also the keystore and truststore password should be masked in the logs which is not case. /*2017-06-05 14:51:31,135 INFO org.apache.flink.configuration.GlobalConfiguration- Loading configuration property: security.ssl.enabled, true 2017-06-05 14:51:31,136 INFO org.apache.flink.configuration.GlobalConfiguration- Loading configuration property: security.ssl.keystore, deploy-keys/ca.keystore 2017-06-05 14:51:31,136 INFO org.apache.flink.configuration.GlobalConfiguration- Loading configuration property: security.ssl.keystore-password, password 2017-06-05 14:51:31,136 INFO org.apache.flink.configuration.GlobalConfiguration- Loading configuration property: security.ssl.key-password, password 2017-06-05 14:51:31,136 INFO org.apache.flink.configuration.GlobalConfiguration- Loading configuration property: security.ssl.truststore, deploy-keys/ca.truststore 2017-06-05 14:51:31,136 INFO org.apache.flink.configuration.GlobalConfiguration- Loading configuration property: security.ssl.truststore-password, password*/ Regards, Vinay Patil -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/In-transit-Data-Encryption-in-EMR-tp13455p13490.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: In-transit Data Encryption in EMR
Hi Gordan, Thank you for your response. I have done the necessary configurations by adding all the node ip's from Resource Manager , is this correct ? Also I will try to check if wildcard works as all our hostname begins with a same pattern. For ex : SAN=dns:ip-192-168.* should work , right ? Facing a weird issue when I try to submit the job using the following command: flink run -m yarn-cluster -yn 4 -ys 4 -yjm 1024 -ytm 4000 -yt deploy-keys/ testFlinkSSL.jar --configFileName conf.yaml Error is : java.lang.IllegalArgumentException: Wrong FS: hdfs://:8020/user/hadoop/.flink/application_1496660166576_0001/flink-dist_2.10-1.2.0.jar, expected: file:/// I see a JIRA ticket regarding the same but did not find any solution to this. Regards, Vinay Patil -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/In-transit-Data-Encryption-in-EMR-tp13455p13489.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: In-transit Data Encryption in EMR
Hi Vinay! 1. Will the existing functionality provided by Amazon to configure in-transit data encrytion work for Flink as well. This is explained here: http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-encryption-enable-security-configuration.html http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-data-encryption-options.html#emr-encryption-intransit I don’t think so. AFAIK, the AWS security configurations needs to be integrated for per-platform’s specific security features, and as of now, there doesn’t seem to be an integration for Flink SSL encryption features, yet. 2. Using Flink SSL Setup: as we know only the IP address of master node on EMR , should we pass only its ip address in the SAN list as given here ? (I think it should work as the yarn-cli command will distribute the truststore and keystore to each TM ) https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/security-ssl.html#use-yarn-cli-to-deploy-the-keystores-and-truststore The generated certificate needs to cover all nodes (hostname and IP address). Is it possible for you to use wildcard subdomain names to generate the certificate? I’m not entirely sure of the subdomain patterns of EMR nodes, but this should be possible. Cheers, Gordon On 5 June 2017 at 12:56:45 PM, vinay patil (vinay18.pa...@gmail.com) wrote: Thank you Till. Gordon can you please help. Regards, Vinay Patil On Fri, Jun 2, 2017 at 9:10 PM, Till Rohrmann [via Apache Flink User Mailing List archive.] <[hidden email]> wrote: Hi Vinay, I've pulled my colleague Gordon into the conversation who can probably tell you more about Flink's security features. Cheers, Till On Fri, Jun 2, 2017 at 2:22 PM, vinay patil <[hidden email]> wrote: Hi, Currently I am looking into configuring in-transit data encryption either using Flink SSL Setup or directly using EMR. Few Doubts: 1. Will the existing functionality provided by Amazon to configure in-transit data encrytion work for Flink as well. This is explained here: http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-encryption-enable-security-configuration.html http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-data-encryption-options.html#emr-encryption-intransit 2. Using Flink SSL Setup: as we know only the IP address of master node on EMR , should we pass only its ip address in the SAN list as given here ? (I think it should work as the yarn-cli command will distribute the truststore and keystore to each TM ) https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/security-ssl.html#use-yarn-cli-to-deploy-the-keystores-and-truststore Regards, Vinay Patil -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/In-transit-Data-Encryption-in-EMR-tp13455.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com. If you reply to this email, your message will be added to the discussion below: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/In-transit-Data-Encryption-in-EMR-tp13455p13459.html To start a new topic under Apache Flink User Mailing List archive., email [hidden email] To unsubscribe from Apache Flink User Mailing List archive., click here. NAML View this message in context: Re: In-transit Data Encryption in EMR Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: In-transit Data Encryption in EMR
Thank you Till. Gordon can you please help. Regards, Vinay Patil On Fri, Jun 2, 2017 at 9:10 PM, Till Rohrmann [via Apache Flink User Mailing List archive.] <ml+s2336050n13459...@n4.nabble.com> wrote: > Hi Vinay, > > I've pulled my colleague Gordon into the conversation who can probably > tell you more about Flink's security features. > > Cheers, > Till > > On Fri, Jun 2, 2017 at 2:22 PM, vinay patil <[hidden email] > <http:///user/SendEmail.jtp?type=node=13459=0>> wrote: > >> Hi, >> >> Currently I am looking into configuring in-transit data encryption either >> using Flink SSL Setup or directly using EMR. >> >> Few Doubts: >>1. Will the existing functionality provided by Amazon to configure >> in-transit data encrytion work for Flink as well. This is explained here: >> http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-encry >> ption-enable-security-configuration.html >> http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-data- >> encryption-options.html#emr-encryption-intransit >> >>2. Using Flink SSL Setup: as we know only the IP address of master node >> on EMR , should we pass only its ip address in the SAN list as given here >> ? >> (I think it should work as the yarn-cli command will distribute the >> truststore and keystore to each TM ) >> https://ci.apache.org/projects/flink/flink-docs-release-1.3/ >> setup/security-ssl.html#use-yarn-cli-to-deploy-the- >> keystores-and-truststore >> >> Regards, >> Vinay Patil >> >> >> >> -- >> View this message in context: http://apache-flink-user-maili >> ng-list-archive.2336050.n4.nabble.com/In-transit-Data-Enc >> ryption-in-EMR-tp13455.html >> Sent from the Apache Flink User Mailing List archive. mailing list >> archive at Nabble.com. >> > > > > -- > If you reply to this email, your message will be added to the discussion > below: > http://apache-flink-user-mailing-list-archive.2336050. > n4.nabble.com/In-transit-Data-Encryption-in-EMR-tp13455p13459.html > To start a new topic under Apache Flink User Mailing List archive., email > ml+s2336050n1...@n4.nabble.com > To unsubscribe from Apache Flink User Mailing List archive., click here > <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code=1=dmluYXkxOC5wYXRpbEBnbWFpbC5jb218MXwxODExMDE2NjAx> > . > NAML > <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer=instant_html%21nabble%3Aemail.naml=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/In-transit-Data-Encryption-in-EMR-tp13455p13486.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: In-transit Data Encryption in EMR
Hi Vinay, I've pulled my colleague Gordon into the conversation who can probably tell you more about Flink's security features. Cheers, Till On Fri, Jun 2, 2017 at 2:22 PM, vinay patil <vinay18.pa...@gmail.com> wrote: > Hi, > > Currently I am looking into configuring in-transit data encryption either > using Flink SSL Setup or directly using EMR. > > Few Doubts: >1. Will the existing functionality provided by Amazon to configure > in-transit data encrytion work for Flink as well. This is explained here: > http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr- > encryption-enable-security-configuration.html > http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr- > data-encryption-options.html#emr-encryption-intransit > >2. Using Flink SSL Setup: as we know only the IP address of master node > on EMR , should we pass only its ip address in the SAN list as given here ? > (I think it should work as the yarn-cli command will distribute the > truststore and keystore to each TM ) > https://ci.apache.org/projects/flink/flink-docs- > release-1.3/setup/security-ssl.html#use-yarn-cli-to- > deploy-the-keystores-and-truststore > > Regards, > Vinay Patil > > > > -- > View this message in context: http://apache-flink-user- > mailing-list-archive.2336050.n4.nabble.com/In-transit-Data- > Encryption-in-EMR-tp13455.html > Sent from the Apache Flink User Mailing List archive. mailing list archive > at Nabble.com. >
In-transit Data Encryption in EMR
Hi, Currently I am looking into configuring in-transit data encryption either using Flink SSL Setup or directly using EMR. Few Doubts: 1. Will the existing functionality provided by Amazon to configure in-transit data encrytion work for Flink as well. This is explained here: http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-encryption-enable-security-configuration.html http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-data-encryption-options.html#emr-encryption-intransit 2. Using Flink SSL Setup: as we know only the IP address of master node on EMR , should we pass only its ip address in the SAN list as given here ? (I think it should work as the yarn-cli command will distribute the truststore and keystore to each TM ) https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/security-ssl.html#use-yarn-cli-to-deploy-the-keystores-and-truststore Regards, Vinay Patil -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/In-transit-Data-Encryption-in-EMR-tp13455.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
