[ https://issues.apache.org/jira/browse/NIFI-6801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16958868#comment-16958868 ]
ASF subversion and git services commented on NIFI-6801: ------------------------------------------------------- Commit 147365285cbe385a85e58371dd9570dc01683936 in nifi's branch refs/heads/master from Yolanda M. Davis [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=1473652 ] NIFI-6801 - fix to ensure unique model instance is created for each connection. Also increased default query interval. Signed-off-by: Matthew Burgess <mattyb...@apache.org> This closes #3838 > Connection analytics models should be distinct for connections > -------------------------------------------------------------- > > Key: NIFI-6801 > URL: https://issues.apache.org/jira/browse/NIFI-6801 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework > Affects Versions: 1.10.0 > Reporter: Yolanda M. Davis > Assignee: Yolanda M. Davis > Priority: Major > Fix For: 1.11.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Connections should have Status Analytics Model instances that are unique for > each connection such that calculations for the model can be referred to or > updated when new observations are available. Currently the model appears to > be shared across connection objects which leads to erroneous results, > especially when a connection does not have enough observations to update the > model (since the older values of a previous connection may still be present). > This behavior can be seen when debugging is enabled for analytics and noting > differences between a Timer Driven Thread execution of predictions vs the > NiFi Web Server thread predictions for multiple connections (a flow with one > connection would not exhibit this issue). The Time Driven thread may not > retrieve enough connection status observations to refresh the model, and when > that occurs, it may still contain calculations from a previous model and use > that for predictions. Observers may see behavior where web server and timer > driven predictions only have similar or matching predictions for one > connection but others may be mismatched, such as in the below (where > connection BBBB has similar prediction between threads but AAAA, and CCCC do > not): > {code:java} > 2019-10-22 08:01:34,539 INFO [NiFi Web Server-23] > o.a.nifi.reporting.StandardEventAccess >>>> AAAA: predicted time to COUNT > backpressure = 00:44:13 / 2653416 > 2019-10-22 08:01:34,539 INFO [NiFi Web Server-23] > o.a.nifi.reporting.StandardEventAccess >>>> AAAA: predicted time to BYTES > backpressure = 01:09:13 / 4153511 > 2019-10-22 08:01:34,540 INFO [NiFi Web Server-23] > o.a.nifi.reporting.StandardEventAccess >>>> CCCC: predicted time to COUNT > backpressure = 00:32:31 / 1951609 > 2019-10-22 08:01:34,540 INFO [NiFi Web Server-23] > o.a.nifi.reporting.StandardEventAccess >>>> CCCC: predicted time to BYTES > backpressure = 00:32:28 / 1948504 > 2019-10-22 08:01:34,541 INFO [NiFi Web Server-23] > o.a.nifi.reporting.StandardEventAccess >>>> BBBB: predicted time to COUNT > backpressure = 07:29:14 / 26954837 > 2019-10-22 08:01:34,541 INFO [NiFi Web Server-23] > o.a.nifi.reporting.StandardEventAccess >>>> BBBB: predicted time to BYTES > backpressure = 05:24:56 / 19496954 > 2019-10-22 08:02:02,767 INFO [Timer-Driven Process Thread-2] > o.a.nifi.reporting.StandardEventAccess >>>> AAAA: predicted time to COUNT > backpressure = 04:08:36 / 14916150 > 2019-10-22 08:02:02,767 INFO [Timer-Driven Process Thread-2] > o.a.nifi.reporting.StandardEventAccess >>>> AAAA: predicted time to BYTES > backpressure = 04:17:49 / 15469243 > 2019-10-22 08:02:02,769 INFO [Timer-Driven Process Thread-2] > o.a.nifi.reporting.StandardEventAccess >>>> CCCC: predicted time to COUNT > backpressure = 01:13:42 / 4422829 > 2019-10-22 08:02:02,769 INFO [Timer-Driven Process Thread-2] > o.a.nifi.reporting.StandardEventAccess >>>> CCCC: predicted time to BYTES > backpressure = 00:57:15 / 3435109 > 2019-10-22 08:02:02,770 INFO [Timer-Driven Process Thread-2] > o.a.nifi.reporting.StandardEventAccess >>>> BBBB: predicted time to COUNT > backpressure = 07:28:46 / 26926608 > 2019-10-22 08:02:02,770 INFO [Timer-Driven Process Thread-2] > o.a.nifi.reporting.StandardEventAccess >>>> BBBB: predicted time to BYTES > backpressure = 05:24:28 / 19468725 > {code} > A workaround for this is to increase the nifi.analytics.query.interval > property or decrease the nifi.components.status.snapshot.frequency property > to ensure obtaining enough observations for a refresh. However an appropriate > fix is to ensure model instances are unique for each connection and to ensure > that, by default, all threads will obtain enough observations for predictions. > -- This message was sent by Atlassian Jira (v8.3.4#803005)