By following Chesney's recommendation we will hopefully uncover an SSL error that is being masked. Another thing to try is to disable hostname verification (it is enabled by default) to see whether the certificate is being rejected.
On Wed, Oct 4, 2017 at 5:15 AM, Chesnay Schepler <ches...@apache.org> wrote: > something that would also help us narrow down the problematic area is to > enable SSL for one component at a time and see > which one causesd the job to fail. > > > On 04.10.2017 14:11, Chesnay Schepler wrote: > > The configuration looks reasonable. Just to be sure, are the paths > accessible by all nodes? > > As a first step, could you set the logging level to DEBUG (by modifying > the 'conf/log4j.properties' file), resubmit the job (after a cluster > restart) and check the Job- and TaskManager logs for any exception? > > On 04.10.2017 03:15, Aniket Deshpande wrote: > > Background: We have a setup of Flink 1.3.1 along with a secure MAPR > cluster (Flink is running on mapr client nodes). We run this flink cluster > via flink-jobmanager.sh foreground and flink-taskmanager.sh foreground command > via Marathon. In order for us to make this work, we had to add > -Djavax.net.ssl.trustStore="$JAVA_HOME/jre/lib/security/cacerts" in > flink-console.sh as extra JVM arg (otherwise, flink was taking MAPR's > ssl_truststore as default truststore and then we were facing issues for any > 3rd party jars like aws_sdk etc.). This entire setup was working fine as it > is and we could submit our jars and the pipelines ran without any problem > > > Problem: We started experimenting with enabling ssl for all communication > for Flink. For this, we followed https://ci.apache. > org/projects/flink/flink-docs-release-1.3/setup/security-ssl.html for > generating CA and keystore. I added the following properties to > flink-conf.yaml: > > > security.ssl.enabled: true > security.ssl.keystore: /opt/flink/certs/node1.keystore > security.ssl.keystore-password: <password> > security.ssl.key-password: <password> > security.ssl.truststore: /opt/flink/certs/ca.truststore > security.ssl.truststore-password: <password> > jobmanager.web.ssl.enabled: true > taskmanager.data.ssl.enabled: true > blob.service.ssl.enabled: true > akka.ssl.enabled: true > > > We then spin up a cluster and tried submitting the same job which was > working before. We get the following erros: > org.apache.flink.streaming.runtime.tasks.StreamTaskException: Cannot load > user class: org.apache.flink.streaming.connectors.kafka. > FlinkKafkaConsumer09 > ClassLoader info: URL ClassLoader: > Class not resolvable through given classloader. > at org.apache.flink.streaming.api.graph.StreamConfig. > getStreamOperator(StreamConfig.java:229) > at org.apache.flink.streaming.runtime.tasks.OperatorChain.< > init>(OperatorChain.java:95) > at org.apache.flink.streaming.runtime.tasks.StreamTask. > invoke(StreamTask.java:230) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702) > at java.lang.Thread.run(Thread.java:748) > > > This error disappears when we remove the ssl config properties i.e run > flink cluster without ssl enabled. > > > So, did we miss any steps for enabling ssl? > > > P.S.: We tried removing the extra JVm arg mentioned above, but still get > the same error. > > -- > > Aniket > > > >
