RE: Content Claims Filling Disk - Best practice for small files?

[Williams, Jim]

Ryan,

Is this this maybe a case of exhausting inodes on the filesystem rather than 
exhausting the space available?  If you do a ‘df -I’ on the system what do you 
see for inode usage?

Warm regards,

[cid:image001.jpg@01D68CDD.FC463950]
Jim Williams | Manager, Site Reliability Engineering
O: +1 713.341.7812 | C: +1 919.523.8767 | 
jwilli...@alertlogic.com | 
alertlogic.com [cid:image002.png@01D68CDD.FC463950] 
 [cid:image003.png@01D68CDD.FC463950] 


[cid:image004.png@01D68CDD.FC463950]

From: Joe Witt 
Sent: Thursday, September 17, 2020 10:19 AM
To: users@nifi.apache.org
Subject: Re: Content Claims Filling Disk - Best practice for small files?

can you share your flow.xml.gz?

On Thu, Sep 17, 2020 at 8:08 AM Ryan Hendrickson 
mailto:ryan.andrew.hendrick...@gmail.com>> 
wrote:
1.12.0

Thanks,
Ryan

On Thu, Sep 17, 2020 at 11:04 AM Joe Witt 
mailto:joe.w...@gmail.com>> wrote:
Ryan

What version are you using? I do think we had an issue that kept items around 
longer than intended that has been addressed.

Thanks

On Thu, Sep 17, 2020 at 7:58 AM Ryan Hendrickson 
mailto:ryan.andrew.hendrick...@gmail.com>> 
wrote:
Hello,
I've got ~15 million FlowFiles, each roughly 4KB, totally in about 55GB of data 
on my canvas.

However, the content repository (on it's own partition) is completely full with 
350GB of data.  I'm pretty certain the way Content Claims store the data is 
responsible for this.  In previous experience, we've had files that are larger, 
and haven't seen this as much.

My guess is that as data was streaming through and being added to a claim, it 
isn't always released as the small files leaves the canvas.

We've run into this issue enough times that I figure there's probably a "best 
practice for small files" for the content claims settings.

These are our current settings:
nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
nifi.content.claim.max.appendable.size=1 MB
nifi.content.claim.max.flow.files=100
nifi.content.repository.directory.default=/var/nifi/repositories/content
nifi.content.repository.archive.max.retention.period=12 hours
nifi.content.repository.archive.max.usage.percentage=50%
nifi.content.repository.archive.enabled=true
nifi.content.repository.always.sync=false

https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#content-repository

There's 1024 folders on the disk (0-1023) for the Content Claims.
Each file inside the folders are roughly  2MB to 8 MB (Which is odd because I 
thought the max appendable size would make this no larger than 1MB.)

Is there a way to expand the number of folders and/or reduce the amount of 
individual FlowFiles that are stored in the claims?

I'm hoping there might be a best practice out there though.

Thanks,
Ryan

Confidentiality Notice | This email and any included attachments may be 
privileged, confidential and/or otherwise protected from disclosure. Access to 
this email by anyone other than the intended recipient is unauthorized. If you 
believe you have received this email in error, please contact the sender 
immediately and delete all copies. If you are not the intended recipient, you 
are notified that disclosing, copying, distributing or taking any action in 
reliance on the contents of this information is strictly prohibited.

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast, a leader in email security and cyber 
resilience. Mimecast integrates email defenses with brand protection, security 
awareness training, web security, compliance and other essential capabilities. 
Mimecast helps protect large and small organizations from malicious activity, 
human error and technology failure; and to lead the movement toward building a 
more resilient world. To find out more, visit our website.

RE: NiFi cluster heavy cpu usage

[Williams, Jim]

Lei,

 

I have seen this on 1.9.2 standalone and cluster deployments.  Every time I
have seen this so far, the reason has been an input port in a process group
which is not connected by an output port.  The input port can be identified
in the user interface by drilling down to the active thread.  Disabling or
removing the input port resolves the high CPU usage.

 

 

Warm regards,

 


  

Jim Williams | Principal Database Developer


O: +1 713.341.7812 | C: +1 919.523.8767 | jwilli...@alertlogic.com |
 alertlogic.com

 


 



 

From: wangl...@geekplus.com.cn  
Sent: Monday, September 16, 2019 4:53 AM
To: users 
Cc: dev 
Subject: NiFi cluster heavy cpu usage

 

 

I deployed a nifi cluster with two virtual machines,  each  4 core, 16 GB
memroy

The cpu is more than 100% even there's no processor running.

Often the cpu is about 300% after i start some processors. 

 

[root@iZuf6agrzcolqeja3if7kbZ ~]# ps -mp 3429  -o THREAD,tid,time | sort -k2
-n -r |less 

root  171   -- - -  - - 07:08:43

root 16.1  19- futex_-  -  3598 00:40:18

root 16.1  19- futex_-  -  3569 00:40:29

root 16.0  19- futex_-  -  3597 00:40:02

root 15.9  19- futex_-  -  3603 00:39:51

root 15.9  19- futex_-  -  3601 00:39:57

root 15.9  19- - -  -  3604 00:39:57

root 15.8  19- - -  -  3713 00:39:40

root 15.7  19- - -  -  3600 00:39:27

root 15.6  19- futex_-  -  3712 00:39:00

root 15.6  19- futex_-  -  3593 00:39:05

 

 

There's some threads consumes cpu. I pick one and jstack it, often it is in
waiting state.

Any insight on this?

 

Thanks, 

Lei

 

  _  

wangl...@geekplus.com.cn  



smime.p7s
Description: S/MIME cryptographic signature

RE: Kafka to parquet to s3

[Williams, Jim]

Dweep,

 

The data I am moving into S3 is already some fairly large sets of files, since 
they are a bulk export from a SaaS application.  Thus, the number of files 
which were being PUT to S3 was not a huge consideration.  However, since the 
Parquet files are to be consumed by Redshift Spectrum I had an interest in 
consolidating flow files containing like objects into a single flow file prior 
to Parquet conversion.  I used the MergeRecord processor [1] to do this.

 

So, to amplify on the flow, it really looks more like this:

 

(Get stuff in JSON format) --> ConvertRecord --> MergeRecord --> 
ConvertAvroToParquet --> PutS3

 

 

This is not really a “real-time streaming flow” it’s more batch-oriented.  
There is a delay in the flow (which is acceptable to us) for the MergeRecord 
processor to collect and merge possibly several flow files into a bigger flow 
file.

 

 

[1] - 
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.2/org.apache.nifi.processors.standard.MergeRecord/index.html

 

 

 

Warm regards,

 


 <https://www.alertlogic.com/> 

Jim Williams | Principal Database Developer


O: +1 713.341.7812 | C: +1 919.523.8767 | jwilli...@alertlogic.com |  
<http://www.alertlogic.com/> alertlogic.com  <https://twitter.com/alertlogic>  
<https://www.linkedin.com/company/alert-logic> 


 



 

From: Dweep Sharma  
Sent: Sunday, July 14, 2019 1:07 AM
To: users@nifi.apache.org
Subject: Re: Kafka to parquet to s3

 

Thanks Jim for the insights on advantages, this worked for me as well. 

 

Any thoughts on partitioning and filesize so the S3 PUT costs are not too high?

 

I do not see options on the convertavrotoparquet for this

 

-Dweep

 

On Mon, Jul 8, 2019 at 6:09 PM Williams, Jim mailto:jwilli...@alertlogic.com> > wrote:

Dweep,

 

I have been working on a project where Parquet files are being written to S3.  
I’ve had the liberty to use the most up-to-date version of Nifi, so I have 
implemented this on 1.9.2.

 

The approach I have taken is something like this:

 

(Get stuff in JSON format) --> ConvertRecord --> ConvertAvroToParquet --> PutS3

 

The ConvertRecord [1] processor changes the flow files from JSON to Avro.  
Although it is possible to use schema inference with this processor, it is 
something we have not leveraged yet.  The ConvertAvroToParquet [2] converts the 
flow file, but does not write it out to a local or HDFS file system like the 
PutParquet [3] processor would.

 

Implementing the flow in this way gives a couple advantages:

 

1.  We do not need to use the PutParquet processor

a.  Extra configuration on cluster nodes is avoided for writing directly to 
S3 with this processor
b.  Writing to a local or HDFS filesystem and then copying to S3 is avoided

2.  We can use the native authentication methods which come with the S3 
processor

a.  Roles associated with EC2 instances are leveraged, which makes cluster 
deployment much simpler

 

We have been happy using this pattern for the past couple months.  I am 
watching for progress on Nifi-6089 [4] for a Parquet Record Reader/Writer with 
interest.

 

 

 

[1] - 
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.2/org.apache.nifi.processors.standard.ConvertRecord/index.html
 
<https://urldefense.proofpoint.com/v2/url?u=https-3A__nifi.apache.org_docs_nifi-2Ddocs_components_org.apache.nifi_nifi-2Dstandard-2Dnar_1.9.2_org.apache.nifi.processors.standard.ConvertRecord_index.html=DwMFaQ=L_h2OePR2UWWefmqrezxOsP9Uqw55rRfX5bRtw9S4KY=8BKCOHGXeuGDgPbW9jE4jktuFFiof_whsQaGaYqyyjs=d_kR1GWPNss4CLyS4-44v2ghw-2JOU_7mHjFqZNMzoQ=-xRMuIesOHDc-Qh9SKoVvE7x9EZVnHvR5oTmpj_9ccM=>
 

[2] - 
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-parquet-nar/1.9.2/org.apache.nifi.processors.parquet.ConvertAvroToParquet/index.html
 
<https://urldefense.proofpoint.com/v2/url?u=https-3A__nifi.apache.org_docs_nifi-2Ddocs_components_org.apache.nifi_nifi-2Dparquet-2Dnar_1.9.2_org.apache.nifi.processors.parquet.ConvertAvroToParquet_index.html=DwMFaQ=L_h2OePR2UWWefmqrezxOsP9Uqw55rRfX5bRtw9S4KY=8BKCOHGXeuGDgPbW9jE4jktuFFiof_whsQaGaYqyyjs=d_kR1GWPNss4CLyS4-44v2ghw-2JOU_7mHjFqZNMzoQ=H7S26aP4FKFn_xmTAuMgfgn9Wqf0haoIWTLS1T9b-qE=>
 

[3] - 
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-parquet-nar/1.9.2/org.apache.nifi.processors.parquet.PutParquet/index.html
 
<https://urldefense.proofpoint.com/v2/url?u=https-3A__nifi.apache.org_docs_nifi-2Ddocs_components_org.apache.nifi_nifi-2Dparquet-2Dnar_1.9.2_org.apache.nifi.processors.parquet.PutParquet_index.html=DwMFaQ=L_h2OePR2UWWefmqrezxOsP9Uqw55rRfX5bRtw9S4KY=8BKCOHGXeuGDgPbW9jE4jktuFFiof_whsQaGaYqyyjs=d_kR1GWPNss4CLyS4-44v2ghw-2JOU_7mHjFqZNMzoQ=umNhX0Fi2wpJhFqSFsVNDml65cyD02cLMxVK4k6mneg=>
 

[4] - https://issues.apache.org/jira/browse/NIFI-6089 
<https://urldefense.proofpoint

RE: Kafka to parquet to s3

[Williams, Jim]

Dweep,

 

I have been working on a project where Parquet files are being written to S3.  
I’ve had the liberty to use the most up-to-date version of Nifi, so I have 
implemented this on 1.9.2.

 

The approach I have taken is something like this:

 

(Get stuff in JSON format) --> ConvertRecord --> ConvertAvroToParquet --> PutS3

 

The ConvertRecord [1] processor changes the flow files from JSON to Avro.  
Although it is possible to use schema inference with this processor, it is 
something we have not leveraged yet.  The ConvertAvroToParquet [2] converts the 
flow file, but does not write it out to a local or HDFS file system like the 
PutParquet [3] processor would.

 

Implementing the flow in this way gives a couple advantages:

 

1.  We do not need to use the PutParquet processor

a.  Extra configuration on cluster nodes is avoided for writing directly to 
S3 with this processor
b.  Writing to a local or HDFS filesystem and then copying to S3 is avoided

2.  We can use the native authentication methods which come with the S3 
processor

a.  Roles associated with EC2 instances are leveraged, which makes cluster 
deployment much simpler

 

We have been happy using this pattern for the past couple months.  I am 
watching for progress on Nifi-6089 [4] for a Parquet Record Reader/Writer with 
interest.

 

 

 

[1] - 
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.2/org.apache.nifi.processors.standard.ConvertRecord/index.html

[2] - 
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-parquet-nar/1.9.2/org.apache.nifi.processors.parquet.ConvertAvroToParquet/index.html

[3] - 
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-parquet-nar/1.9.2/org.apache.nifi.processors.parquet.PutParquet/index.html

[4] - https://issues.apache.org/jira/browse/NIFI-6089

 

 

Warm regards,

 


  

Jim Williams | Principal Database Developer


O: +1 713.341.7812 | C: +1 919.523.8767 | jwilli...@alertlogic.com |  
 alertlogic.com    
 


 



 

From: Bryan Bende  
Sent: Saturday, July 6, 2019 11:49 AM
To: users@nifi.apache.org
Subject: Re: Kafka to parquet to s3

 

Currently put and fetch parquet are tied to the Hadoop API so they need the 
config files. As mentioned you can create a core-site with a local file system, 
and then you could use another part of the flow to pick up the file using 
ListFile -> FetchFile -> PutS3Object.

 

There is a way to write to S3 directly from PutParquet and PutHdfs, but it 
requires additional jars and config, and is honestly harder to setup then just 
using the above approach.

 

There is also a JIRA to implement a parquet record reader and writer which 
would then let you use ConvertRecord to go from JSON to parquet, and then to 
PutS3Object.

 

I think the error mentioned means you have the same field name at different 
levels in your JSON and that is not allowed in an Avro schema.

 

On Sat, Jul 6, 2019 at 9:21 AM Dweep Sharma mailto:dweep.sha...@redbus.com> > wrote:

Thanks Shanker, 

 

But I do not see options in putparquet for s3 bucket/credential, I am assuming 
I need to push this to a local store and then add the puts3object processor on 
top of that ?

 

Also, as record reader in putparquet I am using the JsonTreeReader with 
defaults (InferSchema)  and I get the error "Failed to write due to can't 
redefine: org.apache.nifi.addresstype: 

 

Some files however do get written. Are the default settings good or am I 
missing something ?

 

 

-Dweep

 

 

 

On Fri, Jul 5, 2019 at 11:43 PM Andrew Grande mailto:apere...@gmail.com> > wrote:

Interestingly enough, the ORC processor in NiFi can just use defaults if hadoop 
configs aren't provided, no additional config steps required. Is it something 
which can be improved for PutParquet maybe?

 

Andrew

 

 

 

On Fri, Jul 5, 2019, 4:18 AM Shanker Sneh  wrote:

Hello Dweep,

 

In putparquet processor you can set the attribute 'Hadoop Configuration 
Resources' to a core-site.xml file whose content can be somewhat like below:

 





fs.defaultFS

file:///reservoir-dl  





 

Here the file:///reservoir-dl could be your path where in-transit parquet files 
have be written to -- before being pushed to S3.

More importantly, you do not need Hadoop to be installed. You can just place 
the core-site.xml file on your NiFi nodes and get started.

 

 

On Fri, Jul 5, 2019 at 1:54 PM Dweep Sharma mailto:dweep.sha...@redbus.com> > wrote:

Hi, 

 

I have been trying to move some JSON data to S3 in Parquet format. 

 

>From Kafka to s3 is straight forward but I cannot seem to find the right 
>processor to convert JSON to parquet and move it to s3.

 

putparquet does not take s3 bucket or credentials and requires hadoop to be 
installed. 

 

Can someone please share a 

Re: DistributeLoad across a NiFi cluster

[Williams, Jim]

Edward,

Documentation on this feature may be found here: 
https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#settings



Warm regards,
Jim Williams

From: Edward Armes 
Sent: Thursday, July 4, 2019 5:54 AM
To: users@nifi.apache.org
Subject: Re: DistributeLoad across a NiFi cluster

Hi Andrew,

Is this functionality documented anywhere do you know? As I've had a quick look 
through the documentation and I haven't seen this.

Edward

On Tue, Jul 2, 2019 at 5:33 PM James McMahon 
mailto:jsmcmah...@gmail.com>> wrote:
Excellent - thanks very much Andrew. This is my first crack at working with a 
clustered configuration, and I guess that shows by my question. Outstanding - 
thanks again.

On Tue, Jul 2, 2019 at 12:29 PM Andrew Grande 
mailto:apere...@gmail.com>> wrote:
Jim,

There's a better solution in NiFi. Right click on the connection between 
ListFile and FetchFile and select a cluster distribution strategy in options. 
That's it :)

Andrew

On Tue, Jul 2, 2019, 7:37 AM James McMahon 
mailto:jsmcmah...@gmail.com>> wrote:
We would like to employ a DistributeLoad processor, restricted to run on the 
primary node of our cluster. Is there a recommended approach employed to 
efficiently distribute across nodes in the cluster?

As I understand it, and using a FetchFile running in "all nodes" as the first 
processor following the DistributeLoad, I can have it distribute by round 
robin, next available, or load distribution service.  Can anyone provide a link 
to an example that employs the load distribution service? Is that the 
recommended distribution approach when running in clustered mode?

I am interested in maintaining load balance across my cluster nodes when 
running at high flowfile volumes. Flow files will vary greatly in contents, so 
I'd like to design with an approach that helps me balance processing 
distribution.

Thanks very much in advance. -Jim
Confidentiality Notice | This email and any included attachments may be 
privileged, confidential and/or otherwise protected from disclosure. Access to 
this email by anyone other than the intended recipient is unauthorized. If you 
believe you have received this email in error, please contact the sender 
immediately and delete all copies. If you are not the intended recipient, you 
are notified that disclosing, copying, distributing or taking any action in 
reliance on the contents of this information is strictly prohibited.

Re: Empty "nifi users" page.

[Williams, Jim]

Matt,

I think the JIRA you filed may be substantially similar to NIFI-5636 I filed a 
little while ago.  I'm not sure if there is a process for de-duplication, if 
they do turn out to be the same?


https://issues.apache.org/jira/browse/NIFI-5636


Warm Regards,
Jim Williams


From: Matt Gilman 
Sent: Thursday, March 28, 2019 4:06 PM
To: users@nifi.apache.org
Subject: Re: Empty "nifi users" page.

Yes, I believe the issue is that you have a user and a group with the same 
name. The UUID's are determined deterministically based on the user identity 
and group name. This is done so that we can ensure the same data model across a 
cluster. While not ideal, this doesn't present any issue managing them or the 
policies. However, the UI attempts to present them in a single table using that 
UUID as an identifier. In this case, they are not unique. We should likely do 
something to ensure that users and group have unique identifiers regardless if 
there are name conflicts. I have filed a JIRA for this [1]. If possible, you 
could work around this by ensuring that your users and groups have unique names.

Matt

[1] 
https://issues.apache.org/jira/browse/NIFI-6161

On Thu, Mar 28, 2019 at 12:38 PM DEHAY Aurelien 
mailto:aurelien.de...@faurecia.com>> wrote:
Well, I can compare a lot of things I don’t really know where to search.

What are to be unique? It’s the UUID (id field) present in users/userGroups 
endpoints? The identity (short names)?
I may have some duplicated between names between user and groups, I’m going to 
check that point.

Thanks.

From: Matt Gilman mailto:matt.c.gil...@gmail.com>>
Sent: mercredi 27 mars 2019 19:32
To: users@nifi.apache.org
Subject: Re: Empty "nifi users" page.

Is it possible that you have a user and a group with the same name/identity? 
Can you compare what is in your users.xml file and what is being loaded from 
your directory server? To see what's being loaded from your directory server, 
if your running 1.9.x you should be able to enable debug level logging on

org.apache.nifi.ldap.tenants.LdapUserGroupProvider
Thanks

Matt

On Wed, Mar 27, 2019 at 5:46 AM DEHAY Aurelien 
mailto:aurelien.de...@faurecia.com>> wrote:
Thanks for you answer.

Like Bryan and you made me investigate on the browser side, I’ve check cache 
and dev tools.

Indeed the data are correctly sent to the browser. The users & groups api are 
correctly called and results are good.

So I suppose it’s something on browser side (tested with FF and chrome with 
same issue).

I got these errors in the console without guarantee that it’s linked to the 
issue:

Content Security Policy: « x-frame-options » ignoré en raison de la directive « 
frame-ancestors ».
CURRENT_SITE:  par01prdedge1.soft.fau:18443/nifi/users KeeperFill-var.js:30:935
uncaught exception: Each data element must implement a unique 'id' property

Thanks.

From: Matt Gilman mailto:matt.c.gil...@gmail.com>>
Sent: mardi 26 mars 2019 17:49
To: users@nifi.apache.org
Subject: Re: Empty "nifi users" page.

Hi.

Could you open up your browsers Developer Tools and check the Console for any 
errors and look at the Network requests and ensure that they are returning 
correctly. The behavior your describing indicate one of two things. Either the 
request is not returning from the server or there is a front end code bug.

We have addressed a couple of issues recently that are scheduled to land in 
1.10.0 where users are removed (from LDAP externally) and they are still 
referenced in NiFi policies or groups. However, I don't think the behavior is 
exactly how you describe.

Also, are you able to build current master and try that out? If your running 
into the same problem, it's possible its already addressed.

Thanks

Matt

On Tue, Mar 26, 2019 at 10:29 AM Bryan Bende 
mailto:bbe...@gmail.com>> wrote:
What browser and browser version are you using?

Have you tried clearing your browser cache just to make sure the page
is loading properly?

On Tue, Mar 26, 2019 at 10:21 AM DEHAY Aurelien
mailto:aurelien.de...@faurecia.com>> wrote:
>
> Hello.
>
> I expect to see at least the local users with associated rights (e.g. content 
> of users.xml and authorizations.xml), I've got 3 local users and a bunch of 
> associated rights with ldap users/groups.
>
> I'll try the search filter, but the list is ok when modifying the policies of 
> objects, I can see groups 
>
> Thanks.
>
> -Original Message-
> From: Kevin Doran mailto:kdo...@apache.org>>
> Sent: samedi 23 mars 2019 18:52
> To: users@nifi.apache.org
> Subject: Re: Empty "nifi users" page.
>
> How many users and 

RedisConnectionPoolService - Incorrectly Attempts to Connect to localhost

[Williams, Jim]

Hello,

 

I'm trying to set up the
RedisConnectionPoolService/RedisDistributedMapCacheClientService.

 

Some basic observations:

*   This is a standalone Nifi 1.8.0 server
*   SELinux is disabled on the server
*   There are no iptables rules configured for blocking on the server
*   I am able to resolve the hostname of the Redis server to an IP
address on the Nifi server
*   I can connect to the Redis server to the Nifi server using telnet

 

The stack trace I see when the services are started is:

2018-11-29 21:16:03,527 WARN [Timer-Driven Process Thread-8]
o.a.n.controller.tasks.ConnectableTask Administratively Yielding
PutDistributedMapCache[id=0167105c-4a54-1adf-cb8d-1b45de7f0c99] due to
uncaught Exception: org.springframework.data.redis.RedisConnection

FailureException: Cannot get Jedis connection; nested exception is
redis.clients.jedis.exceptions.JedisConnectionException: Could not get a
resource from the pool

org.springframework.data.redis.RedisConnectionFailureException: Cannot get
Jedis connection; nested exception is
redis.clients.jedis.exceptions.JedisConnectionException: Could not get a
resource from the pool

at
org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetch
JedisConnector(JedisConnectionFactory.java:281)

at
org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getCo
nnection(JedisConnectionFactory.java:464)

at
org.apache.nifi.redis.service.RedisConnectionPoolService.getConnection(Redis
ConnectionPoolService.java:89)

at sun.reflect.GeneratedMethodAccessor580.invoke(Unknown Source)

at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at
org.apache.nifi.controller.service.StandardControllerServiceInvocationHandle
r.invoke(StandardControllerServiceInvocationHandler.java:84)

at com.sun.proxy.$Proxy98.getConnection(Unknown Source)

at
org.apache.nifi.redis.service.RedisDistributedMapCacheClientService.withConn
ection(RedisDistributedMapCacheClientService.java:343)

at
org.apache.nifi.redis.service.RedisDistributedMapCacheClientService.put(Redi
sDistributedMapCacheClientService.java:189)

at sun.reflect.GeneratedMethodAccessor579.invoke(Unknown Source)

at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at
org.apache.nifi.controller.service.StandardControllerServiceInvocationHandle
r.invoke(StandardControllerServiceInvocationHandler.java:84)

at com.sun.proxy.$Proxy96.put(Unknown Source)

at
org.apache.nifi.processors.standard.PutDistributedMapCache.onTrigger(PutDist
ributedMapCache.java:202)

at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java
:27)

at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessor
Node.java:1165)

at
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java
:203)

at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(Timer
DrivenSchedulingAgent.java:117)

at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)

at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$
301(ScheduledThreadPoolExecutor.java:180)

at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Sch
eduledThreadPoolExecutor.java:294)

at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
49)

at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
24)

at java.lang.Thread.run(Thread.java:748)

Caused by: redis.clients.jedis.exceptions.JedisConnectionException: Could
not get a resource from the pool

at redis.clients.util.Pool.getResource(Pool.java:53)

at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226)

at redis.clients.jedis.JedisPool.getResource(JedisPool.java:16)

at
org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetch
JedisConnector(JedisConnectionFactory.java:271)

... 26 common frames omitted

Caused by: redis.clients.jedis.exceptions.JedisConnectionException:
java.net.ConnectException: Connection refused (Connection refused)

at redis.clients.jedis.Connection.connect(Connection.java:207)

at redis.clients.jedis.BinaryClient.connect(BinaryClient.java:93)

at redis.clients.jedis.BinaryJedis.connect(BinaryJedis.java:1767)

at
redis.clients.jedis.JedisFactory.makeObject(JedisFactory.java:106)

at
org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.jav
a:868)

at

RE: Problem Debugging InvokeHTTP Processor in Nifi 1.8.0

[Williams, Jim]

Mark,

 

Thanks for the observation and the hint!  I’ll change the flow and see if I get 
a better result.

 

 

Warm regards,

 


 <https://www.alertlogic.com/> 

Jim Williams | Principal Database Developer


O: +1 713.341.7812 | C: +1 919.523.8767 | jwilli...@alertlogic.com |  
<http://www.alertlogic.com/> alertlogic.com  <https://twitter.com/alertlogic>  
<https://www.linkedin.com/company/alert-logic> 


 



 

From: Mark Payne  
Sent: Friday, November 16, 2018 3:14 PM
To: users@nifi.apache.org
Subject: Re: Problem Debugging InvokeHTTP Processor in Nifi 1.8.0

 

Jim, 

 

Thanks, that's enough to understand what's happening. As it is configured, the 
InvokeHTTP processor has no incoming connection.

However, it is configured to use the HTTP POST method. It doesn't really make 
sense to perform a POST with no incoming data,

so the Processor just returns. If you have no incoming connection, you cannot 
use the Processor to perform a POST, PUT, or PATCH

request.

 

You could force it to occur, if you want to, because the service does something 
interesting with an empty POST, by using a GenerateFlowFile

processor ahead of it and generating a 0-byte FlowFile, then sending that to 
the InvokeHTTP processor.

 

Thanks

-Mark

 

 





On Nov 16, 2018, at 3:29 PM, Williams, Jim mailto:jwilli...@alertlogic.com> > wrote:

 

Hey Mark,

 

Replication is problematic, since the HTTP server I am accessing is internal to 
my company.  However, I have attached a template of the (rather simple) flow.

 

 

Warm regards,

 


 <https://www.alertlogic.com/> 

Jim Williams | Principal Database Developer


O:   +1 713.341.7812 | C:   +1 
919.523.8767 |  <mailto:jwilli...@alertlogic.com> jwilli...@alertlogic.com |  
<http://www.alertlogic.com/> alertlogic.com  
<https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_alertlogic=DwMGaQ=L_h2OePR2UWWefmqrezxOsP9Uqw55rRfX5bRtw9S4KY=8BKCOHGXeuGDgPbW9jE4jktuFFiof_whsQaGaYqyyjs=13mmQmOyilSae-MGpp3n8sGR_dlBGlPONs657tD53KM=DZ3ro4bxRxFa1X8Ch7qXHm0cqWDW9EPY0EXGobLGdjs=>
  
<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_alert-2Dlogic=DwMGaQ=L_h2OePR2UWWefmqrezxOsP9Uqw55rRfX5bRtw9S4KY=8BKCOHGXeuGDgPbW9jE4jktuFFiof_whsQaGaYqyyjs=13mmQmOyilSae-MGpp3n8sGR_dlBGlPONs657tD53KM=hv3BdhKofz8u169QQKemxYQlZVQTaIPpldPYlUDfu5s=>
 


 



 

From: Mark Payne mailto:marka...@hotmail.com> > 
Sent: Friday, November 16, 2018 2:14 PM
To: users@nifi.apache.org <mailto:users@nifi.apache.org> 
Subject: Re: Problem Debugging InvokeHTTP Processor in Nifi 1.8.0

 

Hi Jim, 

 

Can you build a template of your flow and share that? If so, that's usually the 
easiest way to try to

replicate the behavior and to understand exactly how your flow is configured.

 

Thanks

-Mark






On Nov 16, 2018, at 2:58 PM, Williams, Jim < <mailto:jwilli...@alertlogic.com> 
jwilli...@alertlogic.com> wrote:

 

Hello,

 

I’m having an issue where the InvokeHTTP processor is apparently not producing 
a flow file, and is also not throwing any errors.  This is occurring for a 
particular site, but I have tested and found it to work for other sites.

 

Some observations:

 

*   It was attempted to send all relationships to a PutFile processor, but 
no files were generated 

*   The ‘Always Output Response’ setting was set to “true”, but still no 
files were generated

*   The processor is not generating any provenance events
*   The bulletin level was set to DEBUG, but no bulletins were produced
*   Debugging was added to the conf/logback.xml file after the ‘root’ 
entry, but no debugging information was seen in logs/nifi-app.log :

 







 







 

 

Does someone have a suggestion of how we may get further information from this 
processor to debug what we are seeing?

 

 

Warm regards,

 


 <https://www.alertlogic.com/> 

Jim Williams | Principal Database Developer


O:   +1 713.341.7812 | C:   +1 
919.523.8767 |  <mailto:jwilli...@alertlogic.com> jwilli...@alertlogic.com |  
<http://www.alertlogic.com/> alertlogic.com  
<https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_alertlogic=DwMGaQ=L_h2OePR2UWWefmqrezxOsP9Uqw55rRfX5bRtw9S4KY=8BKCOHGXeuGDgPbW9jE4jktuFFiof_whsQaGaYqyyjs=x-wAa94k2BSEEhmcwCQMcnSH5gNlbTF7hxnLddLupv0=OQhUqh_o2_0BmYfe_Mv5D6pRAFa42n8rJObmUpZSGuE=>
  
<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_alert-2Dlogic=DwMGaQ=L_h2OePR2UWWefmqrezxOsP9Uqw55rRfX5bRtw9S4KY=8BKCOHGXeuGDgPbW9jE4jktuFFiof_whsQaGaYqyyjs=x-wAa94k2BSEEhmcwCQMcnSH5gNlbTF7hxnLddLupv0=k1rniGCDMSDCanKifFM6Heh4PJSWHAu9MhnOKdaST5Y=>
 


 



 



 



smime.p7s
Description: S/MIME cryptographic signature