Placement and specification of certificates for StandardRestrictedSSLContextService
I'm trying to set up TLS for a service using /InvokeHTTP/ against an
external-to-NiFi Tomcat-based service and I have configured
/StandardRestrictedSSLContextService/ thus:
https://www.javahotchocolate.com/notes/nifi-images/mdmi-standard-ssl-context-service.png
...which results in the errors shown here:
https://www.javahotchocolate.com/notes/nifi-images/s-sslcontextservice.png
Do the NiFi errors mean that "changeit" can't be used as a password?
At the risk of over-simplifying their placement, I dropped them into
/${NIFI_ROOT}/conf/.
~/dev/nifi/nifi-1.15.0/conf $ *ll mdmi**
-rw-rw-r-- 1 russ russ 899 Jul 20 15:40 mdmi-keystore.crt
-rw-rw-r-- 1 russ russ 2725 Jul 20 15:39 *mdmi-keystore.jks*
-rw-rw-r-- 1 russ russ 1255 Jul 20 15:53 *mdmi-truststore.jks*
/mdmi-keystore.crt/ is self-signed for now and (for now) I have used
"changeit":
~/dev/nifi/nifi-1.15.0/conf $ *keytool -genkey -keyalg RSA -alias
mdmi -keystore mdmi-keystore.jks -validity 365 -keysize 2048*
Enter keystore password: changeit
Re-enter new password: changeit
What is your first and last name?
...
~/dev/nifi/nifi-1.15.0/conf $ *keytool -export -alias mdmi -file
mdmi-**keystore.crt -keystore mdmi-keystore.jks -storepass changeit*
Certificate stored in file
~/dev/nifi/nifi-1.15.0/conf $ *keytool -import -noprompt
-trustcacerts**-alias mdmi -file mdmi-keystore.crt -keystore
mdmi-truststore.jks**-storepass changeit*
Certificate was added to keystore
This all works fine via curl or Postman outside of NiFi for hitting the
service (I put the keytool artifacts into /${CATALINA_BASE}/conf/and
note this in /${CATALINA_BASE}/conf/server.xml/).
When it comes to TLS in NiFi, this is my first rodeo. I'm open to
suggestions on any other this. Thanks.
Re: retry issue
Greg, Whether the output goes to “nonzero status” or “output stream”, the original FlowFile should still be sent to the “original” relationship. Now, if you set retries to say 3 for “nonzero status” and a FlowFile is routed to that relationship, what will happen is that the FlowFile is retried. Nothing is routed to “original” or to “nonzero status”. It then tries again. This time, let’s say it goes to “nonzero status” again. Adds it back to the input queue. It then tries a third time. If it goes to “nonzero status”, then the original FlowFile should still go to “original” and the result of the command goes to “nonzero status.” If it instead goes to “output stream” then the original FlowFiles goes to “original” and the result of the command goes to “output stream.” In either case, the contents going to the “original” relationship should be the incoming FlowFile, unmodified. Am I misunderstanding what you’re trying to achieve? Thanks -Mark > On Jul 20, 2022, at 5:37 PM, Gregory M. Foreman > wrote: > > Mark: > > Thank you for the explanation, yes it helps. I made the changes and > everything worked with one exception: preserving the original content. The > non-zero status will replace the flowfile content with whatever came through > the output stream, so the original content is lost. When the file is > processed correctly, two flowfiles are created for the original and the > output stream relationships. Connecting the original to a routeonattribute > processor that inspects the execution.status code and traps original files > that are errors is an option. Are there any options besides using the second > processor? > > Thanks, > Greg > >> On Jul 20, 2022, at 9:22 AM, Mark Payne wrote: >> >> Greg, >> >> You wouldn’t want to retry the “original” relationship. The processor has 3 >> relationships: original, output stream, and nonzero status. It should always >> send the incoming FlowFile to original. So if you retry that relationship >> you’ll always retry the flowfile, regardless of whether it was successful or >> not. >> >> Instead, you should retry the “nonzero status” relationship. If a FlowFIle >> is routed to this relationship, it will instead be re-queued and processed >> again. Nothing will go to original in this case, because the processing is >> atomic. >> >> But if a FlowFile is routed to the “output stream” relationship (because >> it’s not retried) the FlowFile will continue on normally. And the original >> FlowFile will go to ‘original’. >> >> Additionally, if all retries are completed and it still is routed to >> “nonzero status” then at that point, the FlowFile will go to “nonzero >> status” and the original will be transferred to whatever connection (if any) >> you have for the “original” relationship. >> >> Does that help? >> >> Thanks >> -Mark >> >>> On Jul 20, 2022, at 9:12 AM, Gregory M. Foreman >>> wrote: >>> >>> Hello: >>> >>> The ExecuteStreamCommand processor has 3 relationships. To trap processing >>> failures, I capture flowfiles from the original relationship, inspect the >>> cmd exit status, and reroute to a RetryFlowFile processor if the exit >>> status is not 0. I wanted to see if this could be simplified with the new >>> retry feature in 1.16. When I enable retry on the original relationship >>> and execute a failing cmd, the flowfiles remain penalized in the previous >>> success queue, but never exit to the original relationship. Is this >>> scenario supposed to work as I have it setup? >>> >>> Thanks, >>> Greg >> >
Re: retry issue
Mark: Thank you for the explanation, yes it helps. I made the changes and everything worked with one exception: preserving the original content. The non-zero status will replace the flowfile content with whatever came through the output stream, so the original content is lost. When the file is processed correctly, two flowfiles are created for the original and the output stream relationships. Connecting the original to a routeonattribute processor that inspects the execution.status code and traps original files that are errors is an option. Are there any options besides using the second processor? Thanks, Greg > On Jul 20, 2022, at 9:22 AM, Mark Payne wrote: > > Greg, > > You wouldn’t want to retry the “original” relationship. The processor has 3 > relationships: original, output stream, and nonzero status. It should always > send the incoming FlowFile to original. So if you retry that relationship > you’ll always retry the flowfile, regardless of whether it was successful or > not. > > Instead, you should retry the “nonzero status” relationship. If a FlowFIle is > routed to this relationship, it will instead be re-queued and processed > again. Nothing will go to original in this case, because the processing is > atomic. > > But if a FlowFile is routed to the “output stream” relationship (because it’s > not retried) the FlowFile will continue on normally. And the original > FlowFile will go to ‘original’. > > Additionally, if all retries are completed and it still is routed to “nonzero > status” then at that point, the FlowFile will go to “nonzero status” and the > original will be transferred to whatever connection (if any) you have for the > “original” relationship. > > Does that help? > > Thanks > -Mark > >> On Jul 20, 2022, at 9:12 AM, Gregory M. Foreman >> wrote: >> >> Hello: >> >> The ExecuteStreamCommand processor has 3 relationships. To trap processing >> failures, I capture flowfiles from the original relationship, inspect the >> cmd exit status, and reroute to a RetryFlowFile processor if the exit status >> is not 0. I wanted to see if this could be simplified with the new retry >> feature in 1.16. When I enable retry on the original relationship and >> execute a failing cmd, the flowfiles remain penalized in the previous >> success queue, but never exit to the original relationship. Is this >> scenario supposed to work as I have it setup? >> >> Thanks, >> Greg >
Re: retry issue
Greg, You wouldn’t want to retry the “original” relationship. The processor has 3 relationships: original, output stream, and nonzero status. It should always send the incoming FlowFile to original. So if you retry that relationship you’ll always retry the flowfile, regardless of whether it was successful or not. Instead, you should retry the “nonzero status” relationship. If a FlowFIle is routed to this relationship, it will instead be re-queued and processed again. Nothing will go to original in this case, because the processing is atomic. But if a FlowFile is routed to the “output stream” relationship (because it’s not retried) the FlowFile will continue on normally. And the original FlowFile will go to ‘original’. Additionally, if all retries are completed and it still is routed to “nonzero status” then at that point, the FlowFile will go to “nonzero status” and the original will be transferred to whatever connection (if any) you have for the “original” relationship. Does that help? Thanks -Mark > On Jul 20, 2022, at 9:12 AM, Gregory M. Foreman > wrote: > > Hello: > > The ExecuteStreamCommand processor has 3 relationships. To trap processing > failures, I capture flowfiles from the original relationship, inspect the cmd > exit status, and reroute to a RetryFlowFile processor if the exit status is > not 0. I wanted to see if this could be simplified with the new retry > feature in 1.16. When I enable retry on the original relationship and > execute a failing cmd, the flowfiles remain penalized in the previous success > queue, but never exit to the original relationship. Is this scenario > supposed to work as I have it setup? > > Thanks, > Greg
retry issue
Hello: The ExecuteStreamCommand processor has 3 relationships. To trap processing failures, I capture flowfiles from the original relationship, inspect the cmd exit status, and reroute to a RetryFlowFile processor if the exit status is not 0. I wanted to see if this could be simplified with the new retry feature in 1.16. When I enable retry on the original relationship and execute a failing cmd, the flowfiles remain penalized in the previous success queue, but never exit to the original relationship. Is this scenario supposed to work as I have it setup? Thanks, Greg
Re: Use Kafka processors with MSK/IAM
Hi Max, Having a quick look through the PR, it looks like it was closed due to inactivity. All that probably needs to happen is for someone to check that the change is still good and then re-open the PR. If someone can pick this up, the contributor guidelines, ( https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=57904904#content/view/57904904), should cover what needs to be done. Edward On Wed, 20 Jul 2022, 06:52 LEZIER Maxime (ITNOVEM), wrote: > Hello, > > > > I have to consume an MSK (kafka from aws) wich use IAM assume role for > authentication. > > > > this does not seem to be supported by NiFi. > > > > I see there is an pull request for this. : > > https://github.com/apache/nifi/pull/5291#discussion_r724428821 > > > > but it look to be closed and the feature not implemented. > > > > Someone could tell me if this work will be finished ? > > > > Also, is there any workaround I could use for connect a kafka nifi > consumer to an msk with iam assume role authentication ? > > > > Any help will be very usefull . > > > > Thanks . > > > > Max L. > > > > Interne > > Interne > --- > Ce message et toutes les pièces jointes sont établis à l'intention > exclusive de ses destinataires et sont confidentiels. L'intégrité de ce > message n'étant pas assurée sur Internet, la SNCF ne peut être tenue > responsable des altérations qui pourraient se produire sur son contenu. > Toute publication, utilisation, reproduction, ou diffusion, même partielle, > non autorisée préalablement par la SNCF, est strictement interdite. Si vous > n'êtes pas le destinataire de ce message, merci d'en avertir immédiatement > l'expéditeur et de le détruire. > --- > This message and any attachments are intended solely for the addressees > and are confidential. SNCF may not be held responsible for their contents > whose accuracy and completeness cannot be guaranteed over the Internet. > Unauthorized use, disclosure, distribution, copying, or any part thereof is > strictly prohibited. If you are not the intended recipient of this message, > please notify the sender immediately and delete it. >
