Re: additional credential needed in S3 connection

[Peter Turcsanyi]

Hi Glenda,

Based on the AWS documentation, AWS_CA_BUNDLE is not a client credential
but it references the SSL certificate of the AWS service.
*AWS_CA_BUNDLE: Specifies the path to a certificate bundle to use for HTTPS
certificate validation.*

If I understand your use case correctly, you are trying to connect to your
"internal" bucket which is hosted by a custom S3 compatible service. Not
the public AWS service with the public certificate. That's why you need to
specify a custom certificate for AWS CLI via the AWS_CA_BUNDLE environment
variable.
If this is the case, you can configure it using the SSL Context Service
property of PutS3Object. You will need to create a keystore with the
certificate first, and set it in the Truststore* properties of SSL Context
Service.

Regards,
Peter Turcsanyi

On Tue, Feb 13, 2024 at 9:57 PM Glenda Ramos  wrote:

> Hi,
>
>
>
> We are currently testing Apache NiFi 1.23.0 specifically the "PutS3Object"
> processor to automatically transfer data from remote site to an "internal"
> S3 bucket. The standard AWSCLI commands work and able to connect the bucket
> because we are able to pass in additional credentials like the
> AWS_CA_BUNDLE environment variable. That being said, we need AWS_CA_BUNDLE
> property added in order to connect successfully to our S3 bucket using
> Apache NiFi's "PutS3Object". I have tried passing this as a Parameter and
> made it available to the flow but it's not being used by the "PutS3Object"
> processor to connect. Is there a way we can add an additional property for
> the AWS credentials/config file? It looks like it's only looking for
> "accessKey" and "secretKey".
>
>
>
> Any help would be greatly appreciated.
>
>
>
> Respectfully,
>
> Glenda Ramos
>
> Application Engineer
>

Re: additional credential needed in S3 connection

[Shane Ardell]

Hi Glenda,

Looking at the documentation, an approach you could take is to create an
AWSCredentialsProviderControllerService[1] as your AWS Credentials Provider
Service property value in the PutS3Object processor configuration
properties. You can assign a value to the access key and secret key
properties of the controller service. I have not done this myself, so
hopefully others can chime in to verify if this is the correct approach or
not. Hope that helps.

Best,
Shane

[1]
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-aws-nar/1.23.0/org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderControllerService/index.html

On Tue, Feb 13, 2024 at 3:59 PM Glenda Ramos  wrote:

> Hi,
>
>
>
> We are currently testing Apache NiFi 1.23.0 specifically the "PutS3Object"
> processor to automatically transfer data from remote site to an "internal"
> S3 bucket. The standard AWSCLI commands work and able to connect the bucket
> because we are able to pass in additional credentials like the
> AWS_CA_BUNDLE environment variable. That being said, we need AWS_CA_BUNDLE
> property added in order to connect successfully to our S3 bucket using
> Apache NiFi's "PutS3Object". I have tried passing this as a Parameter and
> made it available to the flow but it's not being used by the "PutS3Object"
> processor to connect. Is there a way we can add an additional property for
> the AWS credentials/config file? It looks like it's only looking for
> "accessKey" and "secretKey".
>
>
>
> Any help would be greatly appreciated.
>
>
>
> Respectfully,
>
> Glenda Ramos
>
> Application Engineer
>

additional credential needed in S3 connection

[Glenda Ramos]

Hi,



We are currently testing Apache NiFi 1.23.0 specifically the "PutS3Object"
processor to automatically transfer data from remote site to an "internal"
S3 bucket. The standard AWSCLI commands work and able to connect the bucket
because we are able to pass in additional credentials like the
AWS_CA_BUNDLE environment variable. That being said, we need AWS_CA_BUNDLE
property added in order to connect successfully to our S3 bucket using
Apache NiFi's "PutS3Object". I have tried passing this as a Parameter and
made it available to the flow but it's not being used by the "PutS3Object"
processor to connect. Is there a way we can add an additional property for
the AWS credentials/config file? It looks like it's only looking for
"accessKey" and "secretKey".



Any help would be greatly appreciated.



Respectfully,

Glenda Ramos

Application Engineer

Re: Streaming processor / primary node only

[LEZIER Maxime (ITNOVEM)]

Really thanks for your answer !

ML

De : Lehel Boér 
Date : mardi, 13 février 2024 à 15:18
À : dev@nifi.apache.org 
Objet : Re: Streaming processor / primary node only
Hi, The @OnPrimaryNodeStateChange annotation can be utilized to manage the 
behavior of processors based on changes in the primary node status. This 
annotation allows you to define actions to be taken when the primary node 
changes, such as starting


Hi,



The @OnPrimaryNodeStateChange annotation can be utilized to manage the behavior 
of processors based on changes in the primary node status. This annotation 
allows you to define actions to be taken when the primary node changes, such as 
starting or stopping message consumption.



I recommend checking the GetAzureEventHub processor, as it likely implements 
similar functionality with regards to handling primary node changes.



Best Regards,

Lehel


Interne




From: LEZIER Maxime (ITNOVEM) 

Sent: Tuesday, February 13, 2024 6:33

To: dev@nifi.apache.org 

Subject: Streaming processor / primary node only





Hello,



I have to dev a streaming processor to consume stomp messages.

The code make the job but I have a problem with de primary node change.

By default, the processor is configured on primary node only to handle 
duplicate message.



What I find is that when there's a change of primary node, the consumer on the 
previous primary continues to work, so I end up with duplicates.

the processor on the previous primary should stop when a switch is made.



I've found the "@OnPrimaryNodeStateChange" annotation, but I'm not sure how to 
use it.

Do you have any examples of processors that use it to switch message 
consumption from one primary to another?



Thanks for your help.



ML







Interne



---

Ce message et toutes les pièces jointes sont établis à l'intention exclusive de 
ses destinataires et sont confidentiels. L'intégrité de ce message n'étant pas 
assurée sur Internet, la SNCF ne peut être tenue responsable des altérations 
qui pourraient se produire sur son contenu. Toute publication, utilisation, 
reproduction, ou diffusion, même partielle, non autorisée préalablement par la 
SNCF, est strictement interdite. Si vous n'êtes pas le destinataire de ce 
message, merci d'en avertir immédiatement l'expéditeur et de le détruire.

---

This message and any attachments are intended solely for the addressees and are 
confidential. SNCF may not be held responsible for their contents whose 
accuracy and completeness cannot be guaranteed over the Internet. Unauthorized 
use, disclosure, distribution, copying, or any part thereof is strictly 
prohibited. If you are not the intended recipient of this message, please 
notify the sender immediately and delete it.

Re: Major bug in Apache NiFi 2.0.0-M2/2.0.0-M1

[Ferenc Kis]

Hi Ajay,

thanks for submitting the change.
Will review the PR as soon as possible and share the feedback via PR
comments.

Regards
Ferenc

On Tue, Feb 13, 2024 at 2:36 PM Premarajan, Ajay
 wrote:

> Hi Team,
>
> I have created a pull request to address this issue by adding a new method
> "transform" in minifi toolkit to convert json flow file from Nifi to MiNifi
> flow json file.
> I have also updated the issue in JIRA to my name.
>
> https://github.com/apache/nifi/pull/8393
> https://issues.apache.org/jira/browse/NIFI-12644
>
> Regards,
> Ajay Premarajan
> ---
> Intelizign Lifecycle Services Pvt Ltd
> No.9 Murrays Gate Road (1st floor),
> Alwarpet, Chennai - 600 018
> Mobile: +91-9677916373
> website: www.intelizign.com
>
>
> DISCLAIMER: This Electronic Mail (e-mail) contains confidential and
> privileged information intended only for the use of individual only for the
> use of the individual or entity to which is sent. If the reader of this
> message is not the intended recipient, or the employee or agent responsible
> for delivery to the intended recipient, you are hereby notified that any
> dissemination, distribution, or copying of this communication is STRICTLY
> PROHIBITED. If you have received this communication in error, please
> immediately notify the sender by reply e-mail or telephone
> 
> From: Premarajan, Ajay
> Sent: Wednesday, January 31, 2024 2:54 PM
> To: dev@nifi.apache.org 
> Subject: Major bug in Apache NiFi 2.0.0-M2/2.0.0-M1
>
> Hi Team,
>
> We aim to integrate the most recent versions, M.2.0-M1 or M.2.0.M2, of
> Apache NiFi and MiNiFi into our production environment. However, we are
> encountering difficulties deploying flow files from NiFi to MiNiFi. A bug
> has been reported for this issue. Please find the link below for more
> details.
> https://issues.apache.org/jira/browse/NIFI-12644
>
> Kindly provide an estimated timeline for this issue to be addressed, or
> direct me to the individual responsible for developing the module so that I
> can contribute to the fixes as well.
>
> Regards,
> Ajay Premarajan
> ---
> Intelizign Lifecycle Services Pvt Ltd
> No.9 Murrays Gate Road (1st floor),
> Alwarpet, Chennai - 600 018
> Mobile: +91-9677916373
> website: www.intelizign.com
>
>
> DISCLAIMER: This Electronic Mail (e-mail) contains confidential and
> privileged information intended only for the use of individual only for the
> use of the individual or entity to which is sent. If the reader of this
> message is not the intended recipient, or the employee or agent responsible
> for delivery to the intended recipient, you are hereby notified that any
> dissemination, distribution, or copying of this communication is STRICTLY
> PROHIBITED. If you have received this communication in error, please
> immediately notify the sender by reply e-mail or telephone
>

Re: Streaming processor / primary node only

[Lehel Boér]

Hi,

The @OnPrimaryNodeStateChange annotation can be utilized to manage the behavior 
of processors based on changes in the primary node status. This annotation 
allows you to define actions to be taken when the primary node changes, such as 
starting or stopping message consumption.

I recommend checking the GetAzureEventHub processor, as it likely implements 
similar functionality with regards to handling primary node changes.

Best Regards,
Lehel

From: LEZIER Maxime (ITNOVEM) 
Sent: Tuesday, February 13, 2024 6:33
To: dev@nifi.apache.org 
Subject: Streaming processor / primary node only


Hello,

I have to dev a streaming processor to consume stomp messages.
The code make the job but I have a problem with de primary node change.
By default, the processor is configured on primary node only to handle 
duplicate message.

What I find is that when there's a change of primary node, the consumer on the 
previous primary continues to work, so I end up with duplicates.
the processor on the previous primary should stop when a switch is made.

I've found the "@OnPrimaryNodeStateChange" annotation, but I'm not sure how to 
use it.
Do you have any examples of processors that use it to switch message 
consumption from one primary to another?

Thanks for your help.

ML



Interne

---
Ce message et toutes les pièces jointes sont établis à l'intention exclusive de 
ses destinataires et sont confidentiels. L'intégrité de ce message n'étant pas 
assurée sur Internet, la SNCF ne peut être tenue responsable des altérations 
qui pourraient se produire sur son contenu. Toute publication, utilisation, 
reproduction, ou diffusion, même partielle, non autorisée préalablement par la 
SNCF, est strictement interdite. Si vous n'êtes pas le destinataire de ce 
message, merci d'en avertir immédiatement l'expéditeur et de le détruire.
---
This message and any attachments are intended solely for the addressees and are 
confidential. SNCF may not be held responsible for their contents whose 
accuracy and completeness cannot be guaranteed over the Internet. Unauthorized 
use, disclosure, distribution, copying, or any part thereof is strictly 
prohibited. If you are not the intended recipient of this message, please 
notify the sender immediately and delete it.

Re: Major bug in Apache NiFi 2.0.0-M2/2.0.0-M1

[Premarajan, Ajay]

Hi Team,

I have created a pull request to address this issue by adding a new method 
"transform" in minifi toolkit to convert json flow file from Nifi to MiNifi 
flow json file.
I have also updated the issue in JIRA to my name.

https://github.com/apache/nifi/pull/8393
https://issues.apache.org/jira/browse/NIFI-12644

Regards,
Ajay Premarajan
---
Intelizign Lifecycle Services Pvt Ltd
No.9 Murrays Gate Road (1st floor),
Alwarpet, Chennai - 600 018
Mobile: +91-9677916373
website: www.intelizign.com


DISCLAIMER: This Electronic Mail (e-mail) contains confidential and privileged 
information intended only for the use of individual only for the use of the 
individual or entity to which is sent. If the reader of this message is not the 
intended recipient, or the employee or agent responsible for delivery to the 
intended recipient, you are hereby notified that any dissemination, 
distribution, or copying of this communication is STRICTLY PROHIBITED. If you 
have received this communication in error, please immediately notify the sender 
by reply e-mail or telephone

From: Premarajan, Ajay
Sent: Wednesday, January 31, 2024 2:54 PM
To: dev@nifi.apache.org 
Subject: Major bug in Apache NiFi 2.0.0-M2/2.0.0-M1

Hi Team,

We aim to integrate the most recent versions, M.2.0-M1 or M.2.0.M2, of Apache 
NiFi and MiNiFi into our production environment. However, we are encountering 
difficulties deploying flow files from NiFi to MiNiFi. A bug has been reported 
for this issue. Please find the link below for more details.
https://issues.apache.org/jira/browse/NIFI-12644

Kindly provide an estimated timeline for this issue to be addressed, or direct 
me to the individual responsible for developing the module so that I can 
contribute to the fixes as well.

Regards,
Ajay Premarajan
---
Intelizign Lifecycle Services Pvt Ltd
No.9 Murrays Gate Road (1st floor),
Alwarpet, Chennai - 600 018
Mobile: +91-9677916373
website: www.intelizign.com


DISCLAIMER: This Electronic Mail (e-mail) contains confidential and privileged 
information intended only for the use of individual only for the use of the 
individual or entity to which is sent. If the reader of this message is not the 
intended recipient, or the employee or agent responsible for delivery to the 
intended recipient, you are hereby notified that any dissemination, 
distribution, or copying of this communication is STRICTLY PROHIBITED. If you 
have received this communication in error, please immediately notify the sender 
by reply e-mail or telephone

Streaming processor / primary node only

[LEZIER Maxime (ITNOVEM)]

Hello,

I have to dev a streaming processor to consume stomp messages.
The code make the job but I have a problem with de primary node change.
By default, the processor is configured on primary node only to handle 
duplicate message.

What I find is that when there's a change of primary node, the consumer on the 
previous primary continues to work, so I end up with duplicates.
the processor on the previous primary should stop when a switch is made.

I've found the "@OnPrimaryNodeStateChange" annotation, but I'm not sure how to 
use it.
Do you have any examples of processors that use it to switch message 
consumption from one primary to another?

Thanks for your help.

ML



Interne

---
Ce message et toutes les pièces jointes sont établis à l'intention exclusive de 
ses destinataires et sont confidentiels. L'intégrité de ce message n'étant pas 
assurée sur Internet, la SNCF ne peut être tenue responsable des altérations 
qui pourraient se produire sur son contenu. Toute publication, utilisation, 
reproduction, ou diffusion, même partielle, non autorisée préalablement par la 
SNCF, est strictement interdite. Si vous n'êtes pas le destinataire de ce 
message, merci d'en avertir immédiatement l'expéditeur et de le détruire.
---
This message and any attachments are intended solely for the addressees and are 
confidential. SNCF may not be held responsible for their contents whose 
accuracy and completeness cannot be guaranteed over the Internet. Unauthorized 
use, disclosure, distribution, copying, or any part thereof is strictly 
prohibited. If you are not the intended recipient of this message, please 
notify the sender immediately and delete it.