Re: Unsubscribe

[Peter Turcsanyi]

Please send an email to users-unsubscr...@nifi.apache.org !

More info: https://nifi.apache.org/community/contact/


On Thu, Feb 15, 2024 at 4:54 PM Martin, Stanley L <
stanley.mar...@hexagonusfederal.com> wrote:

>
>
>
>
>
>

Re: ConsumeMQTT with custom path for Broker URI

[Peter Turcsanyi]

Hi Marek,

It has just been fixed in NIFI-12576 [1] which will be available in the
next releases (1.25.0 / 2.0.0).

Regards,
Peter Turcsanyi

[1] - https://issues.apache.org/jira/browse/NIFI-12576

On Thu, Nov 23, 2023 at 10:47 PM  wrote:

> Hello,
> has anyone solved how to change the path for Broker URI in ConsumeMQTT for
> Websocket over SSL ?
>
> I need:   *wss://[server]:[port]/ws *
> But ConsumeMQTT works with:  *wss://[server]:[port] *only.
>
>
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-mqtt-nar/1.5.0/org.apache.nifi.processors.mqtt.ConsumeMQTT/
>
> P.S.
> I could solve it elsewhere with apache httpd as wss reverse proxy, but
> it's not possible here :-(
>
> Thank you and kind Regards
> Marek
>

Re: Azure managed identity in NiFi

[Peter Turcsanyi]

There is no separate credential controller service for PutAzureEventHub.
The credential properties can be configured on the processor itself.
Setting "Use Azure Managed Identity" processor property to "true" should
work, assuming the Azure config is correct. I tested it with NiFi 1.23.0
running on an Azure VM.

Azure side config:
 - System assigned managed identity must be turned on for the VM running
NiFi.
 - Assign "Azure Event Hubs Data Sender" or "Azure Event Hubs Data Owner"
role to the VM identity. It can be granted at Event Hub Namespace or Event
Hub level.

After the Azure config, NiFi restart may be needed if the processor was
started earlier with different settings (it seems the azure client library
can cache tokens issued with different credentials).

Regards,
Peter Turcsanyi

On Thu, Aug 17, 2023 at 1:25 AM Sean Davis  wrote:

> I have but the processor is not working, and it's not giving an error
> message.
> I saw controller services like ADLSCredentialsControllerService to
> configure a managed identity for PutADLS, was wondering if there is an
> equivalent controller service for PutEventHub, where I could configure a
> system-assigned managed identity within NiFi.
> If not, I'm guessing the issue is on the Azure side.
>
> Thanks,
> Sean.
>
> On Wed, Aug 16, 2023 at 1:40 PM Peter Turcsanyi 
> wrote:
>
>> Hi Sean,
>>
>> PutAzureEventHub processor has "Use Azure Managed Identity" property.
>> Have you tried it?
>>
>> Regards,
>> Peter Turcsanyi
>>
>> On Wed, Aug 16, 2023 at 8:24 PM Sean Davis  wrote:
>>
>>> Hello,
>>>
>>> I'm trying to configure a PutAzureEventHub processor using
>>> system-assigned managed identity.
>>> I don't see a controller service or a different processor that would
>>> allow me to configure the managed identity in NiFi.
>>> I was wondering if there is any way to configure this in NiFi?
>>>
>>> Thanks,
>>> Sean.
>>>
>>

Re: Azure managed identity in NiFi

[Peter Turcsanyi]

Hi Sean,

PutAzureEventHub processor has "Use Azure Managed Identity" property. Have
you tried it?

Regards,
Peter Turcsanyi

On Wed, Aug 16, 2023 at 8:24 PM Sean Davis  wrote:

> Hello,
>
> I'm trying to configure a PutAzureEventHub processor using system-assigned
> managed identity.
> I don't see a controller service or a different processor that would allow
> me to configure the managed identity in NiFi.
> I was wondering if there is any way to configure this in NiFi?
>
> Thanks,
> Sean.
>

Re: Unsubscribe

[Peter Turcsanyi]

Jorge,

Please send a mail to users-unsubscr...@nifi.apache.org.

Further info:
https://nifi.apache.org/mailing_lists.html

Regards,
Peter Turcsanyi

On Mon, May 15, 2023 at 7:50 PM Jorge Machado  wrote:

>

Re: Snowflake processors 1.19.0 release

[Peter Turcsanyi]

Hi Dwane,

Snowflake and Iceberg processors are not part of the standard NiFi
distribution due to bundle size constraints. But they are built during
the release process and the nar files can be downloaded from Maven
Central for example.

Links for Snowflake nars:
https://repo1.maven.org/maven2/org/apache/nifi/nifi-snowflake-services-api-nar/1.20.0/nifi-snowflake-services-api-nar-1.20.0.nar
https://repo1.maven.org/maven2/org/apache/nifi/nifi-snowflake-services-nar/1.20.0/nifi-snowflake-services-nar-1.20.0.nar
https://repo1.maven.org/maven2/org/apache/nifi/nifi-snowflake-processors-nar/1.20.0/nifi-snowflake-processors-nar-1.20.0.nar

You'll need to download the nar files and copy them to the 'lib' or
'extension' folder of your NiFi installation (on all nodes if you have
a cluster). In case of 'extension' no restart is needed.

Regards,
Peter Turcsanyi

On Mon, Feb 13, 2023 at 1:14 PM Dwane Hall  wrote:
>
> Hey Nifi community. In the 1.19 release notes I noticed the addition of a a 
> couple of processors for managing data in snowflake using snowpipe.
>
> Release note highlights (1.19.0)
> "Provided a new processor to Put data to Snowflake using Snowpipe Ingest 
> called PutSnowflake"
>
> Pierre also has a great demo of their use in his YouTube channel. Are these 
> processors intended to be included in the default nifi distro or added as 
> custom .nars?  The reason I ask is I can see the processor code in the nifi 
> git repo but they don't seem to appear in the doco or distributions. This 
> also looks to be the case for the puticeberg processor from the same release. 
>  Is anyone able to see these processors in the doco or distribution or were 
> they accidentally missed in the deployment? (I don't think I'm going crazy I 
> can't seem to find them)
>
> Cheers,
>
> Dwane

Re: Unsubscribe

[Peter Turcsanyi]

Dave,

Please send a mail to users-unsubscr...@nifi.apache.org.

https://nifi.apache.org/mailing_lists.html

Regards,
Peter Turcsanyi

On Thu, Dec 15, 2022 at 10:39 PM Maza, Dave (NIH/CC/BTRIS) [C] via
users  wrote:
>
> unsubscribe
>
>
>
> _
>
> Dave Maza (Contractor)
>
> BTRIS System Administrator
>
> Fedpoint Systems LLC, SBA certified 8(a), EDWOSB
>
> Phone: (301) 451-4715
>
> Email: dave.m...@nih.gov
>
>

Re: DistributedMapCacheServer

[Peter Turcsanyi]

Embedded Hazelcast can also be an option. In that case, there is no
need to set up an external cache but the Hazelcast instances are
running on the NiFi nodes (in the same JVM as NiFi).
Please note: no security/authentication is supported in embedded mode.

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-hazelcast-services-nar/1.18.0/org.apache.nifi.hazelcast.services.cacheclient.HazelcastMapCacheClient/index.html
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-hazelcast-services-nar/1.18.0/org.apache.nifi.hazelcast.services.cachemanager.EmbeddedHazelcastCacheManager/index.html

On Wed, Nov 2, 2022 at 5:10 PM Bryan Bende  wrote:
>
> The DMC Server does not support high availability. If you have a 3
> node nifi cluster, each node will have a DMC client and DMC server,
> but the clients all have to point at only one of the servers, and if
> the node where that server is running goes down, there is no
> replication or failover to another node. So it is really up to you to
> decide if that is acceptable for your use case. If its not then you
> need to use a different DMC client implementation that can communicate
> with an external HA cache, like Redis.
>
> On Wed, Nov 2, 2022 at 11:38 AM Greene (US), Geoffrey N
>  wrote:
> >
> > I make heavy use of DistributedMapCacheServer in my nifi flows (one node; 
> > not clustered).
> >
> >
> >
> > I seem to remember reading that the DistributedMapCacheServer is not to be 
> > used in production; it’s a reference implementation only, and it is not 
> > really recommended for production.
> >
> >
> >
> > Unfortunately, I can no longer find the reference saying that 
> > DistributedMapCacheServer is not trustworthy for prod.
> >
> > I don’t have an HDFS implementation anywhere, but I do need the cacheing 
> > part.
> >
> >
> >
> > Can someone explain?  Can I use DistributedMapCacheServer in my production 
> > flows?
> >
> >
> >
> >
> >
> >

Re: unsubscribe

[Peter Turcsanyi]

Atul,

Please send an email to users-unsubscr...@nifi.apache.org

Further info: https://nifi.apache.org/mailing_lists.html

On Fri, Mar 25, 2022 at 7:44 AM Atul Jadhav 
wrote:

>

Re: [AWS][1.15.3] - Unable to use vpc endpoint

[Peter Turcsanyi]

Hi Maxime,

It seems to be a similar issue as NIFI-8662 [1] where the AWS
client library cannot parse the Region from a VPC-style endpoint and we
need to get it from the Region property.
I will look into it in more detail and get back to you.

Regards,
Peter Turcsanyi

[1] https://issues.apache.org/jira/browse/NIFI-8662

On Tue, Mar 8, 2022 at 4:05 PM LEZIER Maxime (ITNOVEM) 
wrote:

> Hello,
>
>
>
> I would like to ask you again about this subject, would you have an answer
> for me please?
>
>
>
> *ML*
>
>
>
>
>
> *De : *LEZIER Maxime (ITNOVEM) 
> *Date : *vendredi, 4 mars 2022 à 08:47
> *À : *users@nifi.apache.org 
> *Objet : *[AWS][1.15.3] - Unable to use vpc endpoint
>
> Hello,
>
>
>
> I use nifi in 1.15.3 version.
>
>
>
> I have to reach aws ressources (S3,SQS..) which have VPCE in front.
>
>
>
> My test pipeline for this usecase is a simple ListS3 with AwsCredentials
> controller.
>
>
>
> In the listS3 i have set the bucket’s name, the region, point it to the
> credentials controller and set an endpoint override url with a url like
> this :
>
> https://bucketname.vpce-xxx-xxx.s3.eu-west-3.vpce.amazonaws.com
>
>
>
> For the controller, i set up access id/secret id, the role’s arn i have to
> assume, the role assume session name, and the sts vpce endpoint url which
> have this form :
>
> vpce-xxx-xxx.sts.eu-west-3.vpce.amazonaws.com (tried with and without
> https:// in front)d
>
>
>
> When i start this flow i’ve got this error :
>
>
>
> 2022-03-04 08:42:08,067 ERROR [Timer-Driven Process Thread-16]
> org.apache.nifi.processors.aws.s3.ListS3
> ListS3[id=501cd602-017f-1000--9f28cb25] Failed to list contents of
> bucket due to
> com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:
> Credential should be scoped to a valid region, not 'vpce'. (Service:
> AWSSecurityTokenService; Status Code: 403; Error Code:
> SignatureDoesNotMatch; Request ID: d57a4d34-7e59-41ba-9e71-4505dd801e75;
> Proxy: null):
> com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:
> *Credential should be scoped to a valid region, not 'vpce'*. (Service:
> AWSSecurityTokenService; Status Code: 403; Error Code:
> SignatureDoesNotMatch; Request ID: d57a4d34-7e59-41ba-9e71-4505dd801e75;
> Proxy: null)
>
> com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:
> Credential should be scoped to a valid region, not 'vpce'. (Service:
> AWSSecurityTokenService; Status Code: 403; Error Code:
> SignatureDoesNotMatch; Request ID: d57a4d34-7e59-41ba-9e71-4505dd801e75;
> Proxy: null)
>
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1819)
>
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1403)
>
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1372)
>
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
>
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
>
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
>
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
>
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
>
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
>
> at
> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
>
> at
> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
>
> at
> com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.doInvoke(AWSSecurityTokenServiceClient.java:1728)
>
> at
> com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1695)
>
> at
> com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1684)
>
> at
> com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.executeAssumeRole(AWSSecurityTokenServiceClient.java:488)
>
> at
> com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.assumeRole(AWSSecurityTokenServiceClient.java:457)
>
> at
> com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.newSession(STSAssumeRoleSessionCrede

Re: ADLSCredentialsControllerService with user-assigned managed identity

[Peter Turcsanyi]

Hi Isha,

Only system-assigned managed identity is supported currently.
Feel free to file a jira ticket for supporting ClientID / user-assigned
identity too. (https://issues.apache.org/jira, project: Apache NiFi / NIFI).

Best regards,
Peter Turcsanyi

On Fri, Feb 25, 2022 at 12:18 PM Isha Lamboo 
wrote:

> Hi all,
>
>
>
> We have an Azure VM setup with both system-assigned and user-assigned
> identities, and when we set “Use Azure Managed Identity” to true, the
> Controller appears to get one or the other randomly.
>
> We would like to always get the user-assigned identity, but I don’t see a
> way to supply the ClientID for this purpose, only for Service Principal
> authentication.
>
>
>
> Does the ADLSCredentialsControllerService support user-assigned managed
> identities at all?
>
>
>
> Kind regards,
>
>
>
> Isha Lamboo
>

Re: nifi 1.15.3 - GenerateFlowFile and ReplaceText

[Peter Turcsanyi]

Hi Michal,

Most probably, it happens because the content repo is filling up your disk.
The default setting is quite conservative (nifi.properties):
nifi.content.repository.archive.max.usage.percentage=50%

Please try to set it higher, e.g. 80%

Best,
Peter Turcsanyi

On Fri, Jan 21, 2022 at 4:35 PM Michal Tomaszewski <
michal.tomaszew...@cca.pl> wrote:

> Find enclosed pictures with setting for ReplaceText
>
>
>
> *From:* Michal Tomaszewski
>
>
>
> Hi,
>
> We found next issues in 1.15.3:
>
>
>
> #1
>
> GenrateFlowFile, when anything is set in custom text (e.g. 1,2,3,4) having
> all other standard settings – flow is not generated, component hangs doing
> nothing with nothing in logs.
>
> When custom text is not filled-in – it generates flow without any problem.
>
>
>
> #2
>
> ReplaceText works the same – when you fill anything in “replacement value”
> not touching any other options – processor hangs with no information in
> logs.
>
>
>
>
>
> Regards,
>
> Mike
>
>
>

Re: unsubscribe

[Peter Turcsanyi]

You will need to send an email to users-unsubscr...@nifi.apache.org

Further info: https://nifi.apache.org/mailing_lists.html

On Sat, Dec 18, 2021 at 6:53 AM Maxime.Lézier  wrote:

> unsubscribe
>
>
>

Re: CSVRecordSetWriter - Date format

[Peter Turcsanyi]

Hi Asmath,

If I understand your flow correctly, you have ExecuteSQLRecord
(or QueryDatabaseTableRecord) with CSVRecordSetWriter and the expected
output format for Oracle DATE type would be DD-MM- in the CSV.

To achieve this, you'll need to configure CSVRecordSetWriter's "Timestamp
Format" property as DD-MM- and also set the processor's "Use Avro
Logical Types" property to true. The output format of your timestamp
columns will not be affected by this.

Explanation / technical background info on date/timestamp handling in NiFi
in case of Oracle DB:
For DATE data type, the Oracle driver returns a java.sql.Timestamp object
by default. That's why the user needs to configure "Timestamp Format"
instead of "Date Format".
For TIMESTAMP data type, the Oracle driver returns an oracle.sql.TIMESTAMP
object by default. It is not a regular java timestamp and no formatting
will be applied to it. It will be added in its default format in the CSV
(eg.: 2021-03-02 16:30:41.0).
Other RDBMSs (typically) return DATE/TIMESTAMP values as java.sql.Date
and java.sql.Timestamp respectively and the proper format (from "Date
Format" vs "Timestamp Format" properties) can be applied to them in the
NiFi flow.

Hope it helps. Let us know if you are able to configure your flow to
produce the date format you want.

Best,
Peter

On Mon, Mar 1, 2021 at 7:35 PM KhajaAsmath Mohammed 
wrote:

> Hi Peter,
>
> This issue is occuring for the DATE field on oracle. There are other
> fields which are timestampe in oracle for the same table.
>
> Sample Data in Oracle: Date Field type --> 01-NOV-20
>
> May I know the field name and format that I need to use in csvrecordwriter
> now?
>
>
> Thanks,
> Asmath
>
> On Mon, Mar 1, 2021 at 12:20 PM Peter Turcsanyi 
> wrote:
>
>> Hi Asmath,
>>
>> I would try to specify the format in the Timestamp Format property.
>> Oracle's DATE type can contain HH:mm:ss part and I believe it is
>> converted to a Timestamp and the Timestamp Format is applied.
>>
>> Regards,
>> Peter
>>
>> On Mon, Mar 1, 2021 at 6:42 PM KhajaAsmath Mohammed <
>> mdkhajaasm...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I have an issue where the csvrecordwriter is automatically converting
>>> data from date to number. how to resolve this?
>>>
>>> any suggestions to change this?
>>>
>>> [image: image.png]
>>>
>>> Source : Oracle with Date format
>>>
>>> [image: image.png]
>>>
>>> Target: Sql-server into Date format
>>>
>>>
>>> Thanks,
>>> Asmath
>>>
>>

Re: CSVRecordSetWriter - Date format

[Peter Turcsanyi]

Hi Asmath,

I would try to specify the format in the Timestamp Format property.
Oracle's DATE type can contain HH:mm:ss part and I believe it is converted
to a Timestamp and the Timestamp Format is applied.

Regards,
Peter

On Mon, Mar 1, 2021 at 6:42 PM KhajaAsmath Mohammed 
wrote:

> Hi,
>
> I have an issue where the csvrecordwriter is automatically converting data
> from date to number. how to resolve this?
>
> any suggestions to change this?
>
> [image: image.png]
>
> Source : Oracle with Date format
>
> [image: image.png]
>
> Target: Sql-server into Date format
>
>
> Thanks,
> Asmath
>

Re: Special Characters - Table name - Putddatabase record

[Peter Turcsanyi]

Hi Asmath,

Scenarios 2 and 3 should work.
When using quotes, the table / columns names are typically case sensitive
in the database (though I don't know SAP).
Are you sure your table exists in the database with the exact same name (in
terms of lower/upper case)?

Regards,
Peter

Re: Special Characters - Table name - Putddatabase record

[Peter Turcsanyi]

Asmath,

Setting the "Quote Table Identifiers" to true should work (no explicit
quotes needed in "Table Name").
Please attach the exception stack trace if you get an error with these
settings.

Regards,
Peter

On Wed, Jan 20, 2021 at 12:10 AM KhajaAsmath Mohammed <
mdkhajaasm...@gmail.com> wrote:

> Hi,
>
> I am having hard time inserting data into the table which has special
> characters in the table name. I am able to test the same connection without
> special characters in the table name. Any suggestions on how to resolve
> this?
>
> I tried quotes too for the table name but it didn't help me.  Any
> suggestions please?
> [image: image.png]
>
> Thanks,
> Asmath
>

Re: Need help setting up NiFi PutS3Object processor

[Peter Turcsanyi]

Dennis,

You can add user-defined metadata via dynamic properties of PutS3Object
itself (no UpdateAttribute needed). Plus (+) icon at the top-right corner
on the processor properties tab.
The name of the dynamic property will be the key (or name) of the
user-defined metadata.
The value of the dynamic property will be the value of the user-defined
metadata. You can use expression language for the value (eg. referencing FF
attributes or now() for the timestamp).

Hope it helps.

Best,
Peter Turcsanyi

On Thu, Dec 10, 2020 at 11:27 PM Dennis N Brown  wrote:

> Running NiFi v1.12.1 , attempting to copy files from existing filesystem
> to S3.
>
> Trying to set user-defined metadata on the file objects, but can't seem to
> figure out the correct format.
>
> All the doc suggests it can be done with dynamic properties, so have an
> UpdateAttribute processor before the PutS3Object processor and have
> attempted everything I can think of to pass the correct format, but no
> metadata gets created.
>
> I have been able to set metadata with python using something like:
>
> *s3.upload_file(Filename, bucketname, key, ExtraArgs={"Metadata":
> {"upload_ts":"2020-12-02 10:57:00.34343","path":"put/some/path/here"}})*
>
> so I know it can be done.
>
>
>
> Can someone provide an example of the correct format to use.
>
>
>
> Thanks!
>
>
>
>
>
> Regards,
>
>
>
> Dennis N Brown
>
>
>
>
>
>
>
>
>
>
>

Re: How to disconnect rabbitmq in PublishAMQP

[Peter Turcsanyi]

Hi Franco,

If I understand your use case correctly, you do not ingest data
continuously and would like PublishAMQP to disconnect after an idle period
(when there are no messages to send to RabbitMQ).
PublishAMQP is rather optimized for continuous / frequent data load and for
this reason it pools the connection(s) and only disconnects when it gets
stopped. Configuring idle timeout is not possible currently.
The processor holds 1 connection per concurrent task so you can minimize
the open connections by setting Concurrent Tasks = 1 (default) on the
processor.

Best,
Peter

On Thu, Dec 10, 2020 at 3:48 AM walter franco  wrote:

> Hello,
> I have a question and I need your help:
>
> I have a project (PoC) using apache nifi with PublishAMQP processor,
> writing messages in RabbitMQ, and I see that the connections stay ACTIVE
> always, don´t disconnect never (only stop the PublishAMQP processor). The
> RabbitMQ has limite of the connections, so I need that diconnect in X
> seconds or minutes.
>
> Is this possible disconnect? or How is it done to free the connections?
>
> Thanks
>
> Best regards,
> Franco
>

Re: NiFi 1.11.4 HDFS/HBASE Processor Errors After Kerberos Ticket Expires

[Peter Turcsanyi]

Are there any exception stack traces in the log when the processors fail /
before that?

On Thu, Oct 22, 2020 at 12:28 AM  wrote:

> Hello!
>
>
>
> We’re running into a problem with NiFi 1.11.4.
>
>
>
> Our HBASE/HDFS/Parquet processors are all configured with a master
> KeytabCredentialsService that is pointing to a Kerberos principal and
> keytab file.
>
>
>
> The environment’s /etc/krb5.conf file has the line renew_lifetime = 7d
> commented out due to an issue with Java-OpenJDK (that is apparently fixed
> but still shows up) causing “MESSAGE STREAM MODIFIED (41)” errors to appear
> whenever we have it uncommented.
>
>
>
> When NiFi starts, it is able to kinit with the Kerberos KDC and is issued
> a 24 hour ticket. Everything works fine right up until that ticket expires.
> Once the ticket expires, all of our HDFS/HBASE/Parquet processors start
> failing.
>
>
>
> I haven’t been able to find anything in our logs around the timeframe, but
> I can’t turn on debug logging for this because the logs are tremendously
> large when we do that (approximately 100-200 MB per minute and the problem
> only occurs at the 24 hour mark).
>
>
>
> How would we go about troubleshooting this issue?
>
>
>
> Environment:
>
> Red Hat Enterprise Linux 7.9
>
> Apache NiFi 1.11.4
>
> java-11-openjdk 11.0.8.10-1.el7
>
>
>
> Thanks!
>
>
>
>
>

Re: IBM MQ message selector

[Peter Turcsanyi]

Hi Marc,

Great, thanks for the feedback.

Best,
Peter

On Mon, Aug 3, 2020 at 3:16 PM Marc Recht  wrote:

> Hi Peter,
>
> I tested it now quite a bit and it works like a charm! Thanks for
> implementing it so swiftly.
>
> Best,
>
> Marc
> On 27/07/2020 13:40, Peter Turcsanyi wrote:
>
> Hi Marc,
>
> ConsumeJMS message selector support has been implemented and will be
> released in the upcoming 1.12 version.
>
> Best,
> Peter
>
> On Fri, Jul 24, 2020 at 7:40 AM Marc Recht  wrote:
>
>> Hi Peter,
>>
>> OK, thanks! Will do.
>>
>> Best,
>>
>> Marc
>> On 23/07/2020 20:30, Peter Turcsanyi wrote:
>>
>> Hi Marc,
>>
>> Message selectors are not supported by ConsumeJMS currently.
>> Feel free to file a jira ticket (https://issues.apache.org/jira) to
>> request for this improvement.
>>
>> Regards,
>> Peter
>>
>> On Wed, Jul 22, 2020 at 5:51 PM Marc Recht 
>> wrote:
>>
>>>
>>>
>>> Hi,
>>>
>>> I try to use a message selector with NiFi and IBM MQ and the ConsumeJMS
>>> processor. Unfortunately, I am not sure how... Is there an example
>>> available somewhere how to set this up?
>>>
>>> Any help appreciated,
>>> Marc
>>>
>>>

Re: IBM MQ message selector

[Peter Turcsanyi]

Hi Marc,

ConsumeJMS message selector support has been implemented and will be
released in the upcoming 1.12 version.

Best,
Peter

On Fri, Jul 24, 2020 at 7:40 AM Marc Recht  wrote:

> Hi Peter,
>
> OK, thanks! Will do.
>
> Best,
>
> Marc
> On 23/07/2020 20:30, Peter Turcsanyi wrote:
>
> Hi Marc,
>
> Message selectors are not supported by ConsumeJMS currently.
> Feel free to file a jira ticket (https://issues.apache.org/jira) to
> request for this improvement.
>
> Regards,
> Peter
>
> On Wed, Jul 22, 2020 at 5:51 PM Marc Recht  wrote:
>
>>
>>
>> Hi,
>>
>> I try to use a message selector with NiFi and IBM MQ and the ConsumeJMS
>> processor. Unfortunately, I am not sure how... Is there an example
>> available somewhere how to set this up?
>>
>> Any help appreciated,
>> Marc
>>
>>

Re: IBM MQ message selector

[Peter Turcsanyi]

Hi Marc,

Message selectors are not supported by ConsumeJMS currently.
Feel free to file a jira ticket (https://issues.apache.org/jira) to request
for this improvement.

Regards,
Peter

On Wed, Jul 22, 2020 at 5:51 PM Marc Recht  wrote:

>
>
> Hi,
>
> I try to use a message selector with NiFi and IBM MQ and the ConsumeJMS
> processor. Unfortunately, I am not sure how... Is there an example
> available somewhere how to set this up?
>
> Any help appreciated,
> Marc
>
>

Re: Session state in cluster HandleHttpRequest and HandleHttpReponse

[Peter Turcsanyi]

Hi Jeremy,

I don't think you can accept the request in one node and send back the
response from another node.
There is an open HTTP connection between the client and the NiFi node while
the HandleHttpRequest -> ... -> HandleHttpResponse flow is running.
Even if we passed the request/response context object between the NiFi
nodes via a distributed cache, it would not be possible to send back the
HTTP response to a client that originally connected and sent the request to
another node. At least I don't know any solution to it in/outside NiFi.

Best,
Peter

On Tue, Jun 30, 2020 at 6:04 AM Jeremy Pemberton-Pigott <
fuzzych...@gmail.com> wrote:

> Hi,
>
> I have a cluster of 3 nodes and the incoming request on one node's
> HandleHttpRequest may be replied to by a different node's
> HandleHttpResponse, in between there is a Spark streaming job process.  Is
> there any example how to do that, maybe with DistributeMapCacheService?  So
> that I can still reply back to the client with the necessary data in the
> response.
>
> Jeremy
>

Re: NIFI JMS consuler : AMQP 1.0 trouble

[Peter Turcsanyi]

Olivier,

The problematic property is the JNDI Provider URL. It seems the amqp:// url
cannot be specified here directly (though I believe I managed to do this
earlier somehow, but I cannot reproduce it now).

The Qpid JMS documentation [1] says that you need to create a properties
file (eg. jndi.properties but it can have any arbitrary name):
##
connectionfactory.ConnectionFactory = amqp://localhost:5672
##

and then reference this file in the JNDI Provider URL property via file://
url or OS filesystem path.

It works for me properly. My processor config:

JNDI Initial Context Factory
Class: org.apache.qpid.jms.jndi.JmsInitialContextFactory
JNDI Provider URL: ${NIFI_HOME}/ext/qpid/jndi.properties
JNDI Name of the Connection Factory: ConnectionFactory
JNDI / JMS Client Libraries: ${NIFI_HOME}/ext/qpid/lib

Please note that you must configure the same name in the JNDI Name of the
Connection Factory property as in the properties file.

Hope it will work for you too.

Best,
Peter

[1] http://qpid.apache.org/releases/qpid-jms-0.52.0/docs/index.html

On Mon, Jun 22, 2020 at 2:56 PM Mark Payne  wrote:

> Olivier,
>
> I don’t know much of anything about qpid. But I did notice that the JNDI
> Name that you provided is topicConnectionFactry rather than
> topicConnectionFactory. I.e., “Factory” is misspelled. Would recommend you
> double-check that.
>
> Thanks
> -Mark
>
> On Jun 22, 2020, at 5:02 AM, GAQUIERE, Olivier <
> olivier.gaqui...@capgemini.com> wrote:
>
> Hi again Peter
>
> Here is my processor configuration
>
> 
>
> *Olivier GAQUIERE*
>
> Architect Solution Managing | CSD-Practice | Space/Public/ATM
> Capgemini France - Toulouse
> Tel.: *+33 582 28 34 61*
> 
> Si vous recevez ce courriel pendant vos temps non travaillés, il ne
> requiert
> ni prise de connaissance ni réponse immédiate de votre part.
>
> *De :* Peter Turcsanyi 
> *Envoyé :* vendredi 19 juin 2020 14:59
> *À :* users@nifi.apache.org
> *Objet :* Re: NIFI JMS consuler : AMQP 1.0 trouble
>
> Hi Olivier,
>
> You can use Apache Qpid Proton AMQP client library [1] as recommended in
> the ActiveMQ AMQP configuration docs [2].
> You'll want to download Qpid JMS (AMQP 1.0) and use the jars from the lib
> folder of the tarball.
> The JNDI Initial Context Factory Class needs to be configured:
> org.apache.qpid.jms.jndi.JmsInitialContextFactory
>
>
> Regards,
> Peter
>
> [1] https://qpid.apache.org/proton/
> [2] https://activemq.apache.org/amqp
>
> On Fri, Jun 19, 2020 at 10:51 AM GAQUIERE, Olivier <
> olivier.gaqui...@capgemini.com> wrote:
>
> Hi all
>
> I try to use NIFI to consume AMQP 1.0 messages from a ActiveMQ Topic.
> Did anyone manage to implement the JMS consumer to consume AMQP messages ?
> If yes, which client librairies did you use ?
>
> Thanks in advance
>
> Regards
>
> Olivier
> This message contains information that may be privileged or confidential
> and is the property of the Capgemini Group. It is intended only for the
> person to whom it is addressed. If you are not the intended recipient, you
> are not authorized to read, print, retain, copy, disseminate, distribute,
> or use this message or any part thereof. If you receive this message in
> error, please notify the sender immediately and delete all copies of this
> message.
>
>
>

Re: NIFI JMS consuler : AMQP 1.0 trouble

[Peter Turcsanyi]

Hi Olivier,

You can use Apache Qpid Proton AMQP client library [1] as recommended in
the ActiveMQ AMQP configuration docs [2].
You'll want to download Qpid JMS (AMQP 1.0) and use the jars from the lib
folder of the tarball.
The JNDI Initial Context Factory Class needs to be configured:
org.apache.qpid.jms.jndi.JmsInitialContextFactory


Regards,
Peter

[1] https://qpid.apache.org/proton/
[2] https://activemq.apache.org/amqp

On Fri, Jun 19, 2020 at 10:51 AM GAQUIERE, Olivier <
olivier.gaqui...@capgemini.com> wrote:

> Hi all
>
>
>
> I try to use NIFI to consume AMQP 1.0 messages from a ActiveMQ Topic.
>
> Did anyone manage to implement the JMS consumer to consume AMQP messages ?
>
> If yes, which client librairies did you use ?
>
>
>
> Thanks in advance
>
>
>
> Regards
>
>
>
> Olivier
> This message contains information that may be privileged or confidential
> and is the property of the Capgemini Group. It is intended only for the
> person to whom it is addressed. If you are not the intended recipient, you
> are not authorized to read, print, retain, copy, disseminate, distribute,
> or use this message or any part thereof. If you receive this message in
> error, please notify the sender immediately and delete all copies of this
> message.
>

Re: AWS custom processor build failing on upgrading nifi version to 1.11 from 1.8

[Peter Turcsanyi]

Hi Sanjeet,

IMO this topic is rather for the dev mailing list.

Could you please attach the pom files you are using in your custom module?

Regards,
Peter

On Fri, Jun 5, 2020 at 6:22 PM sanjeet rath  wrote:

> Hi,
>
> I have created a  custom aws controler service by creating a new java
> project. Where i have created a new java file is same as
> awsCredentialProviderControlerService  with some of my changes.
>
> Public class awsCredentialProviderControlerService extends
> abstractcontrolerservice implements awsCredentialProviderService
> {
> code
> }
>
> In my project i have added nifi-aws-processor 1.8 as pom depedency to make
> it work and build nar file and this was working fine.
>
> However when i am changing the pom version to 1.11.4 of nifi-aws-processor.
>
> It gives me Compilation error on awsCredentialProviderService.
>
> When i am explicitily adding Nifi-aws-service-api as my pom file as a
> depdency in my new project along with nifi-aws-processor it is working fine.
>
> My question is, Nifi-aws-service-api
> Is already added as pom dependency in nifi-aws-processor's pom file.
>
> So only addding nifi-aws-processor in pom file as a dependency should work
> and it was working in 1.8 version.
>
>
> Could you please help me to understand,
> Is there in build related changes there in 1.11.4 version.
>
> Thanks and regards
> Sanjeet
>

Re: Filtering Tweets by User Name

[Peter Turcsanyi]

Hi Kaya,

You can set Filter Endpoint for the Twitter Endpoint property and then
specify one or more user ids in the "IDs to Follow" property. It only works
with the id, so you will need to look up the id for the username before
configure it.
Another option would be to filter the received tweets based on the metadata
in the json: extract username with EvaluateJsonPath, then use
RouteOnAttribute.

Hope it helps.

Regards,
Peter Turcsanyi

On Fri, Nov 15, 2019 at 2:06 AM Kayak28  wrote:

> Hello, Comunity:
>
> I am a new user of Nifi, and currently using Nifi 1.10.
> I use sequences of processors as follows.
> My current configurations are:
> 1. GetTwitter
> - Twitter Endpoint = Sample Endpoint
> - MaxClient Error try = 5
> - Language = ja
> - 4 of the Twitter API key.
>
> 2, EvaluateJsonPath
> - Destination = flowfile-attribute
> - Return Type = auto-detect
> - Path Not Found Behavior = ignore
> - Null Value Representation = empty string
> - twitter.created_at = $.created_at
> - twitter.id = $.id_str
> - twitter.lang = $.lang
> - twitter.text = $.text
>
> 3.  PutSolrContentStream
> - f.1 = id:/id
> - f.2 = twitter_text_t:/text
> - f.3 = twitter_username_s:/user/name
> - f.4 = twitter_created_at_s:/created_at
> - f.5 = twitter_timestamp_ms_tl:/timestamp_ms
> - f.6 = twitter_user_loc:/user/location
> - f.7 = twitter_favorited:/favorited
> - f.8 = twitter_url:/url
> - f.9 = twitter_favorite_count:/favorite_count
>
> By using the above settings, I successfully get tons of tweets from any
> users.
> For the first step, I would like to get tweets from a certain user,
> say @xx.
> For the second step, I would like to get tweets that are liked by the
> user, @x
>
> Is it a possible thing by Nifi?
> if so, how can I configure the processor to achieve the first step?
>
>
> I would really appreciate any input on the matter.
>
> Sincerely,
> Kaya Ota
>
>

Re: Default Retry mechanism for NIFI puts3Object processor

[Peter Turcsanyi]

Hi Sanjeet,

There is an open issue [1] about retry handling in AWS processors with a
pull request available [2] that might be interesting for you / solve your
problem. Unfortunately it has not been merged yet.

This would be a more generic solution for all AWS processors which also
adds an option to configure the retry policy.

Regards,
Peter

[1] https://issues.apache.org/jira/browse/NIFI-6486
[2] https://github.com/apache/nifi/pull/3612

On Mon, Nov 11, 2019 at 6:15 PM sanjeet rath  wrote:

> Hi Team,
>
> I am using puts3Object processor of the nifi , to uploading object from
> onprem to AWS s3 bucket. i believe we have 2 types of uploading , single
> part upload and multipart upload as per the threshold value defined for
> multipart.
>
> for multipart , 3 steps are followed
> 1)s3.nitiateMultipartUpload , 2)s3.uploadPart 3)s3.completeMultipartUpload
>
> while checking the code i found , in s3.completeMultipartUpload method, if
> there is any server side exception(5**), then it is retrying 3 times (as in
> CompleteMultipartUploadRetryCondition class of AWS SDK,  MAX_RETRY_ATTEMPTS
> is constant variable of value 3) using a do while loop .
>
> I have 2 questions
>
> a) This default retry mechanism (value is 3)is only used in
> s3.completeMultipartUpload method ? as i don't find any code for retry used
> in single object upload.
>
> b) if am going to changes MaxErrorRetry value AWS ClientConfiguration,
> does this will change it retry count if there is S3exception(5**)  as per
> value i have set, as its a constant value of 3. Please confirm.
>
> c)If B answer is YES. Then only
> ClientConfiguration.MaxErrorRetry(myCostumValue) will work or
>
> I have to add bellow code for retry policy also.
>
> ClientConfiguration.setRetryPolicy(new
> RetryPolicy(config.getRetryPolicy().getRetryCondition(),config.getRetryPolicy().getBackoffStrategy(),
> myCostumValue, true).
>
>
> Thanks ,
>
> Sanjeet
>
>
>
>

Re: Influence about removing RequiresInstanceClassLoading from AbstractHadoopProcessor processor

[Peter Turcsanyi]

Hi Hai Luo,

@RequiresInstanceClassLoading makes possible to configure separate /
isolated "Additional Classpath Resources" settings on your HDFS processors
(eg. S3 storage driver on one of your PutHDFS and Azure Blob on the other).

Is there any specific reason / use case why you are considering to remove
it?

Regards,
Peter Turcsanyi

On Mon, Nov 11, 2019 at 3:30 PM abellnotring  wrote:

> Hi,all
>  I’m considering removing the RequiresInstanceClassLoading annotation
> from class AbstractHadoopProcessor,
>  Does anybody know the potential Influence?
>
> Thanks
> By Hai Luo
>

Re: HandleHTTPRequest 503 error issue

[Peter Turcsanyi]

Hi Wyllys, sorry for my late reply. 1.10 release in progress, expected soon.

Regards,
Peter Turcsanyi

On Fri, Oct 4, 2019 at 10:22 PM Wyllys Ingersoll <
wyllys.ingers...@keepertech.com> wrote:

>
> Yes, thanks.  Is there an ETA for 1.10?
>
> On Fri, Oct 4, 2019 at 3:05 PM Peter Turcsanyi 
> wrote:
>
>> Hi Wyllys,
>>
>> It has been fixed in NIFI-6317
>> <https://issues.apache.org/jira/browse/NIFI-6317> which will be shipped
>> in the upcoming 1.10 release.
>> Could you please retest your workload on that version when it is
>> available?
>>
>> Regards,
>> Peter Turcsanyi
>>
>> On Fri, Oct 4, 2019 at 6:28 PM Wyllys Ingersoll <
>> wyllys.ingers...@keepertech.com> wrote:
>>
>>>
>>> I believe this issue: https://issues.apache.org/jira/browse/NIFI-5522
>>> is a regression in Nifi 1.9.2, it was marked as fixed in 1.8.0, but Im
>>> seeing it a lot in 1.9.2
>>>
>>> I can recreate it fairly regularly when I am hitting the endpoint pretty
>>> heavily and an error is encountered.  The process return 503 error (or
>>> nothing at all) and cannot be stopped or restarted without rebooting the
>>> entire node.
>>>
>>> -Wyllys Ingersoll
>>>
>>>

Re: DBCPConnectionPoolLookup

[Peter Turcsanyi]

Hi John,

You need to add dynamic properties to DBCPConnectionPoolLookup controller
service. The name of the property corresponds to database.name attribute,
the value is the DBCPConnectionPool that can be looked up (by the name of
the property).
So the attribute database.name ties back to the name of the dynamic
property, not to the name of the DBCPConnectionPool component.

Please find attached my simple example. It contains only one
DBCPConnectionPool that can be looked up, but a second one can be added in
the same way.

Regards,
Peter Turcsanyi

On Fri, Oct 4, 2019 at 10:13 PM John Fak  wrote:

> Does someone have a basic example of using this ?
> I really dont understand it. And it seems useful.
>
> Does the attribute database.name tie back to the "Name" of the
> DBCPConnectionPool ?
>
> How do you pass that dynamically ?  Via some updateattibute processor ?
>
> Be great to see a very basic flow on connecting to two or more sources
> dynamically.
>
>
>
>



6a4bf1bf-016b-1000-1fda-2e79e336695e
DBCP


47e5dacd-f1e4-30b4--
bd34a285-0243-32c3--

0.0
0.0




be2b62c1-c19c-3dec--
47e5dacd-f1e4-30b4--
1 GB
1

47e5dacd-f1e4-30b4--
5f285237-9d31-3f8b--
PROCESSOR

0 sec
1
DO_NOT_COMPRESS

LOAD_BALANCE_NOT_CONFIGURED
DO_NOT_LOAD_BALANCE

success

47e5dacd-f1e4-30b4--
32dcf1a4-9cea-3cd0--
PROCESSOR

0


e14cd5b8-85ba-3fe2--
47e5dacd-f1e4-30b4--
1 GB
1

47e5dacd-f1e4-30b4--
c7e7e246-88f5-3b4d--
PROCESSOR

0 sec
1
DO_NOT_COMPRESS

LOAD_BALANCE_NOT_CONFIGURED
DO_NOT_LOAD_BALANCE

success

47e5dacd-f1e4-30b4--
5f285237-9d31-3f8b--
PROCESSOR

0


042a173c-50cd-3376--
47e5dacd-f1e4-30b4--

nifi-record-serialization-services-nar
org.apache.nifi
1.10.0-SNAPSHOT




Schema Write Strategy

Schema Write Strategy



schema-cache

org.apache.nifi.serialization.RecordSchemaCacheService
schema-cache



schema-access-strategy

schema-access-strategy



schema-registry

org.apache.nifi.schemaregistry.services.SchemaRegistry
schema-registry



schema-name

schema-name



schema-version

schema-version



schema-branch

schema-branch



schema-text

schema-text



Date Format

Date Format

Re: HandleHTTPRequest 503 error issue

[Peter Turcsanyi]

Hi Wyllys,

It has been fixed in NIFI-6317
<https://issues.apache.org/jira/browse/NIFI-6317> which will be shipped in
the upcoming 1.10 release.
Could you please retest your workload on that version when it is available?

Regards,
Peter Turcsanyi

On Fri, Oct 4, 2019 at 6:28 PM Wyllys Ingersoll <
wyllys.ingers...@keepertech.com> wrote:

>
> I believe this issue: https://issues.apache.org/jira/browse/NIFI-5522  is
> a regression in Nifi 1.9.2, it was marked as fixed in 1.8.0, but Im seeing
> it a lot in 1.9.2
>
> I can recreate it fairly regularly when I am hitting the endpoint pretty
> heavily and an error is encountered.  The process return 503 error (or
> nothing at all) and cannot be stopped or restarted without rebooting the
> entire node.
>
> -Wyllys Ingersoll
>
>

Re: NIFI - PUTSQL - sql.args.1.type

[Peter Turcsanyi]

If you navigate to Constant Field Values

in
the javadoc, you can see the integer constants.

On Tue, Sep 24, 2019 at 6:10 AM KhajaAsmath Mohammed <
mdkhajaasm...@gmail.com> wrote:

> This link doesn’t have any value to integer or other types.  It has
> comparison of data types between sql and java.
>
> Sent from my iPhone
>
> On Sep 23, 2019, at 8:20 PM, Juan Pablo Gardella <
> gardellajuanpa...@gmail.com> wrote:
>
> The values of the constants defined at
> https://docs.oracle.com/javase/8/docs/api/java/sql/Types.html
>
> On Mon, Sep 23, 2019, 8:10 PM Wesley C. Dias de Oliveira <
> wcdolive...@gmail.com> wrote:
>
>> Hi, KhajaAsmath.
>>
>> I've search it too but found nothing.
>>
>> I think it's more easy to observe the type based on your query. "mapping"
>> the arguments with the query fields.
>>
>> Em seg, 23 de set de 2019 às 20:02, KhajaAsmath Mohammed <
>> mdkhajaasm...@gmail.com> escreveu:
>>
>>> Hi,
>>>
>>> I have existing flow and trying to understand what is sql.args.1.type ?
>>>
>>> sql.args.1.type =11
>>> sql.args.1.type=13
>>>
>>> I understood that they are matched to column data type for
>>> sql.args.1.value.
>>>
>>> what is data type for 11 and 13 ? May I know what are all the options
>>> availabe for String, Integer, double ,date etc
>>>
>>> Thanks,
>>> Asmath
>>>
>>
>>
>> --
>> Grato,
>> Wesley C. Dias de Oliveira.
>>
>> Linux User nº 576838.
>>
>

Re: Variables to start a NiFi flow

[Peter Turcsanyi]

Hi Bimal,

With Variable Registry, you can implement it in the following way:
Put your flow into a Process Group. Use variable references in your
processors (eg. ${db.table}) and define the variables at the process group
level. Then copy the process group (by simply copying it or creating a
template from it first) and set the variables to the proper values in each
process group. You can also configure separate scheduling in each process
group.
The drawback that you need to multiply your flow.

Another approach:
Defined your flow only once and use FlowFile attributes instead of
variables in variable registry.
Use GenerateFlowFile and add the FlowFile attributes via the dynamic
properties of this processor. Configure a separate GenerateFlowFile for
each of your source tables and connect them to the same "SQL" processor
(which was the entry point earlier). Configure the scheduling on these
GenerateFlowFile-s.
The problem is that not all "SQL" processors support flowfile input. You
can use ExecuteSQL(Record) or GenerateTableFetch in this way, but not
QueryDatabaseTable.

Regards,
Peter

On Mon, Aug 26, 2019 at 6:30 PM Bimal Mehta  wrote:

> Hi,
>
> We have a data flow which extracts data from source database table and
> loads into target hive table. This flow needs to  run several times in a
> day to get delta records from source table and also for multiple tables .
> Now we need to replicate this same process for all the different source
> tables. So rather than creating multiple data flows for each separate
> table, can I use the existing flow and pass parameters like source table
> name to that flow and the flow starts. Basically looking for an interface
> where the user can pass the table names that we want to load at a given
> point in time  and the flow is triggered for that table. Variable Registry
> comes to mind, but I am not sure how to make it work for this use case. We
> are using NiFi 1.9.0  as part of CDF bundle.
>
> Thanks
> Bimal Mehta
>

Re: java.lang.IllegalStateException: AsyncContext completed and/or Request lifecycle recycled ???

[Peter Turcsanyi]

The request has a timeout configured on HttpContextMap. If there is some
delay in the flow and the response is being processed around the expiration
time, it can happen that the clean-up mechanism in HttpContextMap closes
the expired request and sends 503 response to the client (while the
response thread is still processing). You get this error when the response
processor tries to complete the AsyncContext but it is already completed
(AsyncContext is the context that wraps the asynchronous request / reponse).
It is a kind of race condition between the response processor and the
clean-up threads. Recently I was thinking about adding some synchronisation
logic for these threads, but in the end I decided not to do so because the
examples I found ([1], [2]) do not handle it. The exception seems to me
harmless, it just signs that there was a race condition.

[1]
https://www.logicbig.com/tutorials/java-ee-tutorial/java-servlet/servlet-async.html
[2] https://www.journaldev.com/2008/async-servlet-example

On Fri, May 24, 2019 at 5:59 PM William Gosse 
wrote:

> Does anyone know what this exception is about:
>
> 2019-05-23 19:09:07,102 ERROR [Timer-Driven Process Thread-2]
> o.a.n.p.standard.HandleHttpResponse
> HandleHttpResponse[id=d7b2feb7-4a1e-38c8-07b9-e3d9448bc92b] Failed to
> complete HTTP Transaction for
> StandardFlowFileRecord[uuid=423f4435-95ed-43fa-b03d-d1798a40f21e,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1558638515089-4, container=default,
> section=4], offset=127, length=129],offset=0,name=1500212603.pdf,size=129]
> due to java.lang.IllegalStateException: AsyncContext completed and/or
> Request lifecycle recycled: java.lang.IllegalStateException: AsyncContext
> completed and/or Request lifecycle recycled
> java.lang.IllegalStateException: AsyncContext completed and/or Request
> lifecycle recycled
> at
> org.eclipse.jetty.server.AsyncContextState.state(AsyncContextState.java:54)
> at
> org.eclipse.jetty.server.AsyncContextState.complete(AsyncContextState.java:99)
> at
> org.apache.nifi.http.StandardHttpContextMap.complete(StandardHttpContextMap.java:142)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:87)
> at com.sun.proxy.$Proxy87.complete(Unknown Source)
> at
> org.apache.nifi.processors.standard.HandleHttpResponse.onTrigger(HandleHttpResponse.java:184)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:209)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>

Re: PutS3Object failing when using non-Latin characters in filename

[Peter Turcsanyi]

PR is available with the fix: https://github.com/apache/nifi/pull/3373

On Thu, Mar 7, 2019 at 11:47 PM Peter Turcsanyi 
wrote:

> I managed to reproduce the S3 Put error, but FetchFile works fine in my
> local dev env, so I think the two issues are unrelated.
>
> I looked into the code of PutS3Object and also checked the HTTP request
> sent to S3:
> The filename is being set in the Content-Disposition HTTP header and if it
> contains national characters, then the encoding will be wrong. It seems it
> needs to be URL encoded (related RFC: RFC 6266
> <https://tools.ietf.org/html/rfc6266>, however I didn't dig into it in
> detail). I've checked it with S3 and it works fine.
> You can find my proposed fix here
> <https://github.com/apache/nifi/compare/master...turcsanyip:s3_put_i18n?expand=1>.
> If there are no objections, I'll file an issue / open a pull request
> tomorrow.
>
> Regards,
> Peter
>
> On Thu, Mar 7, 2019 at 7:47 PM Andy LoPresto  wrote:
>
>> The fact that the signatures don’t match may indicate some kind of
>> character normalization or encoding difference with the way AWS handles the
>> input. There is an existing Jira for handling filenames with orthographic
>> marks in FetchFile [1].
>>
>> [1] https://issues.apache.org/jira/browse/NIFI-6051
>>
>>
>> Andy LoPresto
>> alopre...@apache.org
>> *alopresto.apa...@gmail.com *
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>
>> On Mar 7, 2019, at 8:04 AM, Mike Thomsen  wrote:
>>
>> I kept the default for the object key, which is ${filename} and some of
>> our files have non-Latin characters. The error from AWS is:
>>
>> > The request signature we calculated does not match the signature you
>> provided. Check your key and signing method. (Service: Amazon S3; Status
>> Code: 403; Error Code: SignatureDoesNotMatch; Request ID: ; S3
>> Extended Request ID: )
>>
>> There are no obvious encoding issues on the NiFi end. It renders the
>> characters just fine in the flowfile viewer. Is it something with UTF8
>> characters being problematic here? Any mitigation suggestions?
>>
>> Thanks,
>>
>> Mike
>>
>>
>>

Re: couldn't set up logback to log to database

[Peter Turcsanyi]

Hi Kirill,

What I don't understand is the duplicated  tags at the beginning
of your config snippet:

 ...

Is it a copy-paste error in the mail (or in the config)?

Could you please send over the exact / whole logback.xml file as you are
trying to use it?

Regards,
Peter

On Tue, Mar 5, 2019 at 11:43 AM K Makeev  wrote:

> Oracle DB is already a central part of our production setup. It stores ETL
> metadata, including etl task logs. I can't change it to some other storage.
>
> Going to do TailFile and further app.log parsing to put it then in Oracle
>
> anyway thanks for suggestions
>
> Kirill
>
> On Mon, Mar 4, 2019 at 9:20 PM Mike Thomsen 
> wrote:
>
>> Kirill,
>>
>> This is a good place to get started with ELK:
>>
>> https://www.elastic.co/solutions/logging
>>
>> It uses ElasticSearch for storage. Check out some of their demos to get a
>> feel for what they offer. It's all open source and runs on very modest
>> hardware. It's also external to NiFi and something you can hook into every
>> part of your deployment rather than changing anything with NiFi.
>>
>> Mike
>>
>> On Mon, Mar 4, 2019 at 11:06 AM K Makeev  wrote:
>>
>>> My use case is that i need to put ExecuteSQL ERROS in database.
>>>
>>> Logback says that logging to database is their standard feature.
>>>
>>> Never used ELK. Could it accomplish such a task?
>>> Where should i start if i want to setup it with NIFI?
>>>
>>> Thanks
>>> Kirill
>>>
>>>
>>> On Mon, Mar 4, 2019 at 6:42 PM Mike Thomsen 
>>> wrote:
>>>
 If this is a standard log analysis use case, I would recommend the ELK
 stack instead (File Beat + Logstash + Elastic and Kibana).

 On Mon, Mar 4, 2019 at 10:21 AM K Makeev  wrote:

> Hi all
>
> Using Nifi 1.5
> I want my certain processors Erorrs to be logged to Oracle database
>
> So i set up separate dbaappender in logback.xml.
> I also created tables in database where logs should go with these
> scripts
>
> https://github.com/qos-ch/logback/blob/master/logback-classic/src/main/resources/ch/qos/logback/classic/db/script/oracle.sql
>
> Those are my additional entries to logback.xml
>
> 
> 
>  class="ch.qos.logback.core.db.DriverManagerConnectionSource">
> oracle.jdbc.driver.OracleDriver
> jdbc:oracle:thin:@myhost:1521/myservice_name
> myuser
> mypassword
> 
> 
>
>   level="ERROR"/>
> 
> 
>
> I've restarted NIFI. But it still logs ExecuteSQL errors to
> nifi-app.log.
>
> May be i should check some logback errors in its own log. But i don't
> know where it is.
> May be it doesn't see Oracle driver or smth. Nifi itself connects to
> these database properly.
>
> How should i set up database logging properly?
>
> Kirill
>