Hi,
Tried following a couple of blog posts about this [1], [2], but neither of
these refer to using NiFi in clustered environment with SSL and I suspect this
is where I am hitting problems (but don’t know where).
The blogs state that using an output port (in the root process group I.e. on
main canvas) which I have done and tried to connect thus..
System.setProperty("javax.net.ssl.keyStore", "/spark-processor.jks");
System.setProperty("javax.net.ssl.keyStorePassword", “*****");
System.setProperty("javax.net.ssl.trustStore", “/cacerts.jks");
SiteToSiteClientConfig config = new SiteToSiteClient.Builder()
.url("https://yarn-cm1.mis-cds.local:9090/nifi";)
.portName("Spark test out")
.buildConfig();
SparkConf sparkConf = new SparkConf().setMaster("local[2]").setAppName("NiFi
Spark Log Processor");
JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, new
Duration(5000));
JavaReceiverInputDStream<NiFiDataPacket> packetStream = jssc.receiverStream(new
NiFiReceiver(config, StorageLevel.MEMORY_ONLY()));
JavaDStream text = packetStream.map(dataPacket -> new
String(dataPacket.getContent(), StandardCharsets.UTF_8));
text.print();
jssc.start();
jssc.awaitTermination();
The error I am getting is
16/05/19 16:39:03 WARN ReceiverSupervisorImpl: Restarting receiver with delay
2000 ms: Failed to receive data from NiFi
java.io.IOException: Server returned HTTP response code: 401 for URL:
https://yarn-cm1.mis-cds.local:9090/nifi-api/controller
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at
sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1889)
at
sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1884)
at java.security.AccessController.doPrivileged(Native Method)
at
sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1883)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1456)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
at
sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
at
org.apache.nifi.remote.util.NiFiRestApiUtil.getController(NiFiRestApiUtil.java:69)
at
org.apache.nifi.remote.client.socket.EndpointConnectionPool.refreshRemoteInfo(EndpointConnectionPool.java:891)
at
org.apache.nifi.remote.client.socket.EndpointConnectionPool.getPortIdentifier(EndpointConnectionPool.java:878)
at
org.apache.nifi.remote.client.socket.EndpointConnectionPool.getOutputPortIdentifier(EndpointConnectionPool.java:862)
at
org.apache.nifi.remote.client.socket.SocketClient.getPortIdentifier(SocketClient.java:81)
at
org.apache.nifi.remote.client.socket.SocketClient.createTransaction(SocketClient.java:123)
at org.apache.nifi.spark.NiFiReceiver$ReceiveRunnable.run(NiFiReceiver.java:149)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Server returned HTTP response code: 401 for
URL: https://yarn-cm1.mis-cds.local:9090/nifi-api/controller
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1839)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at
sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
at
org.apache.nifi.remote.util.NiFiRestApiUtil.getController(NiFiRestApiUtil.java:66)
... 7 more
Any pointers would be helpful in getting this working. I don’t know if I have
to set up a remote process group with the output port (not sure how this
works), or what. When I go to
https://yarn-cm1.mis-cds.local:9090/nifi-api/controller in the browser, I get
an access denied error.
I have created keystore and signed by the RootCA used to sign all the self
signed certs for the cluster.
Running 0.6.1, 6 node cluster.
Thanks
Conrad
[1[ -
https://community.hortonworks.com/articles/12708/nifi-feeding-data-to-spark-streaming.html
[2] - https://blogs.apache.org/nifi/entry/stream_processing_nifi_and_spark
SecureData, combating cyber threats
______________________________________________________________________
The information contained in this message or any of its attachments may be
privileged and confidential and intended for the exclusive use of the intended
recipient. If you are not the intended recipient any disclosure, reproduction,
distribution or other dissemination or use of this communications is strictly
prohibited. The views expressed in this email are those of the individual and
not necessarily of SecureData Europe Ltd. Any prices quoted are only valid if
followed up by a formal written quote.
SecureData Europe Limited. Registered in England & Wales 04365896. Registered
Address: SecureData House, Hermitage Court, Hermitage Lane, Maidstone, Kent,
ME16 9NT
