Re: Empty Queue - but UI shows messages

[Mark Payne]

Joe,

The problem is that NiFi is not able to load balance the data. What errors do 
you see in the logs around load balancing?

Thanks
-Mark

On Nov 22, 2022, at 9:15 AM, Joe Obernberger 
mailto:joseph.obernber...@gmail.com>> wrote:


Hi Joe - this is happening only in queues that do round robin load balancing:



When I list queue on that queue it shows no flow files.  If I stop the 
processors, and change the queue to no longer load balance, then I can not only 
list the queue, but the 22k messages are then processed.

I did the dump as suggested - and it nifi-bootstrap.log file is attached.

-Joe


On 11/9/2022 5:09 PM, Joe Witt wrote:
Please get back to the bad state and then do a thread dump.  Please share

./bin/nifi.sh dump

Only if LB appears stuck.
Thanks

On Wed, Nov 9, 2022 at 3:05 PM Joe Obernberger 
mailto:joseph.obernber...@gmail.com>> wrote:

If I stop the RouteOnContent and InvokeHTTP processors, and then change the 
queue to no longer load balance, then the messages appear and the processor 
start to run.  I can also then list the messages in the queue.

-Joe

On 11/9/2022 12:04 PM, Joe Obernberger wrote:

The processor consuming from the queue runs on all nodes and is a InokeHTTP 
processor.  That processor is currently idle.

I can stop/start InvokeHTTP, but the queue size remains.  I can't list what is 
in the queue.  This seems like a bug?  What I'm confused about is - do I have 
data to process or not?  I can empty the queue, and something is removed, but I 
can't see what...so I don't know if I lost messages or not.

<6aueDc0O2l8gSHvp.png>

-Joe

On 11/9/2022 11:50 AM, Joe Witt wrote:
Hello

This likely means the processor consuming from this queue has the flowfiles 
held being processed.  MergeContent is a common processor that would do this 
but others certainly can.  What processor do you have there?

If you stop the target processor then delete it should always work though you 
generally should not need to do so but normally deleting queue content is a 
debug thing so you can.

Thanks

On Wed, Nov 9, 2022 at 9:47 AM Joe Obernberger 
mailto:joseph.obernber...@gmail.com>> wrote:

Hi - I'm using NiFi 1.18.0 in a three node cluster using internal zookeeper.  
Occasionally, I will see a queue showing queued messages, but when I list the 
queue, the UI says that there are no flow files.  Also the consumer of this 
queue seems to be idle.  I can then empty the queue and it reports that n 
messages were deleted.

Any idea what is happening here?






-Joe


[https://s-install.avcdn.net/ipm/preview/icons/icon-envelope-tick-green-avg-v1.png]
 
Virus-free.www.avg.com

PrometheusReportingTask authorisation?

[Garland, Michael R]

Hi,

I've noticed that whilst the PrometheusReportingTask can be configured for 
client authentication, there is no means to authorise connections to the 
/metrics endpoint.  Given the /metrics endpoint can expose processor group and 
processor names, which could be sensitive in nature, should exposing endpoints 
like this also be authorised, for example by using the global access policies 
functionality?

This strikes me as being something that is maybe niche, but nevertheless 
without authorisation, could undermine the security aspects of NiFi by 
providing access to information to say an insider threat (valid client 
certificate, but not authorised).

Would be interested to know other's thoughts on this?

Michael

Re: Nifi unable to list/empty queue.

[Mark Payne]

Hello,

For access to data, because it’s considered more sensitive than the flow 
definition, both the user and the nifi node accessing the data must be granted 
permissions to view and modify data. Did you give the nodes permissions to view 
and modify data?

Also moving this from security@ to users@ mailing list, as this is more of a 
use-based question. The security@ mailing list should be used for sensitive 
topics such as potential vulnerabilities, etc.

Thanks
Mark


Sent from my iPhone

On Nov 22, 2022, at 4:58 AM, lemontree <715733...@qq.com> wrote:



Hello

we have secured Nifi cluster in 1.1.7 with 3 nodes. When we click to list or 
empty queue on connection, there is error message



Insufficient Permissions

Node 192.168.106.5:9443 is unable to fulfill this request due to: Unable to 
modify the data for Processor with ID 682706ca-08e4-3d90-9b6a-5f845573299f. 
Contact the system administrator. Contact the system administrator.


the error request:

  1.
请求 URL:
https://192.168.106.5:9443/nifi-api/flowfile-queues/f7b68394-8e95-3e87-902f-90a74d3d8a42/listing-requests
  2.
请求方法:
POST
  3.
状态代码:
403 Forbidden



We grant user policy to view and modify data, but no success. Admin user got 
the same error.message


we use the managed-authorizer configed as this : 
nifi.security.user.authorizer=managed-authorizer;

very strangely, other action policy work normally, such as create 
processor...





Regards

RE: ODBC with unixODBC in Linux

[Isha Lamboo]

Hi Jeremy,

Have you tried the JDBC client? NiFi, being a Java application works with JDBC 
drivers, not ODBC.

https://code.kx.com/q/interfaces/jdbc-client-for-kdb/

After building the driver jar or obtaining a binary somehow, you can upload it 
to the nifi server in an accessible directory and create a DBCP ConnectionPool 
with the location of the jar file in the driver location and class name (I’m 
not sure what the class name is supposed to be, it looks like just “jdbc”).

Regards,

Isha

Van: Jeremy Pemberton-Pigott 
Verzonden: dinsdag 22 november 2022 09:22
Aan: users@nifi.apache.org
Onderwerp: ODBC with unixODBC in Linux

Hi, has anyone had success using unixODBC to make an ODBC connection to a DB 
that supports this mode of connection in Linux (CentOS/Debian).  We are trying 
to connect NiFi to Kx's kdb+ and this is the initial feedback that they gave to 
connect to the time series database.  Or has anyone connected to kdb from NiFi 
on Linux?

Reference page: 
https://code.kx.com/q/interfaces/q-client-for-odbc/

Jeremy

ODBC with unixODBC in Linux

[Jeremy Pemberton-Pigott]

Hi, has anyone had success using unixODBC to make an ODBC connection to a
DB that supports this mode of connection in Linux (CentOS/Debian).  We are
trying to connect NiFi to Kx's kdb+ and this is the initial feedback that
they gave to connect to the time series database.  Or has anyone connected
to kdb from NiFi on Linux?

Reference page: https://code.kx.com/q/interfaces/q-client-for-odbc/

Jeremy