I can confirm the sate is being managed by zookeeper. I've been doing some more testing today and I haven't been able to make the ListXXX processes fail by manually causing the primary node to change (just using tcpdump to drop zookeeper traffic, causing a node change). So I'm not sure at the moment if it's because I'm in a cluster or not. I'm setting up a seperate nifi node that will only perform ListXXX operations without being in a cluster.
Though even if this is a single node, will it need a state manager configured? On Thu, Sep 21, 2017 at 10:58 AM, Bryan Bende <[email protected]> wrote: > This should be correct behavior... the cache approach was used before the > state manager got introduced, and the cache was left there to migrate old > state, but we do need to pick a release and remove those lingering cache > properties. > > Do you have the cluster state manager configured in state managers xml? > > It should be using zoo keeper when in a cluster. > > On Wed, Sep 20, 2017 at 6:23 PM Gino Lisignoli <[email protected]> > wrote: > >> Hi Joe >> >> The distributed cache service property of ListXXX doesn't seem to matter >> anymore. Which was quite a surprise to me. It just gets handled by the >> state manager: https://github.com/apache/nifi/blob/ >> e68ff153e81ddb82d1136d44a96bdb7a70da86d1/nifi-nar-bundles/ >> nifi-extension-utils/nifi-processor-utils/src/main/java/ >> org/apache/nifi/processor/util/list/AbstractListProcessor.java#L264 >> >> You can test this via setting the Distributed Cache Service to a redis >> server and watch no connections ever being made, which was how I found out. >> Not sure if this is intended or not. >> >> >> On Wed, Sep 20, 2017 at 12:42 PM, Joe Witt <[email protected]> wrote: >> >>> Gino >>> >>> When the primary node shifts the new primary node should take over the >>> function of doing the listings for you. You'll want to be using the >>> distributed cache property of the ListFTP/etc.. processors to help >>> keep state in such cases. >>> >>> Thanks >>> >>> On Tue, Sep 19, 2017 at 4:50 PM, Gino Lisignoli <[email protected]> >>> wrote: >>> > Hi >>> > >>> > I can confirm that the cause of my problem is the primary node >>> shifting to a >>> > new node in my cluster. This causes all of the ListXXX processors to >>> stop >>> > working (as they are set to only run on my primary node). I'm not sure >>> what >>> > is causing my primary node to shift, or why the processors are unable >>> to >>> > recover when the node shifts. If I find any more info I'll raise a >>> ticket. >>> > >>> > On Thu, Sep 14, 2017 at 8:08 AM, Joe Witt <[email protected]> wrote: >>> >> >>> >> Thanks Gino. If you can confirm/replicate the behavior please do >>> file a >>> >> jira. >>> >> >>> >> Thanks >>> >> >>> >> On Wed, Sep 13, 2017 at 4:06 PM, Gino Lisignoli <[email protected] >>> > >>> >> wrote: >>> >> > In 1.4.0-SNAPSHOT, the ListFTP processor fails to add new files >>> after >>> >> > several hours. >>> >> > >>> >> > At the moment I'm testing this on a cluster of 2. But I'm looking to >>> >> > replicate my issue on a single instance as well. >>> >> > >>> >> > I'm not sure if this is a cluster state problem, a processor state >>> >> > problem >>> >> > or the new ListXXX commits >>> >> > >>> >> > (https://github.com/apache/nifi/commit/ >>> e68ff153e81ddb82d1136d44a96bdb7a70da86d1 >>> >> > and >>> >> > >>> >> > https://github.com/apache/nifi/commit/ >>> 28ee70222b892fb799f5f74a31a9de678d9fb629). >>> >> > >>> >> > I'll be looking at those commits first around the ListFTP process. >>> >> > Unless >>> >> > theres any advice can be offered. >>> >> > >>> >> > Should I submit this to Jira issues, or wait until I can >>> >> > confirm/replicate >>> >> > the behaviour? >>> > >>> > >>> >> >> -- > Sent from Gmail Mobile >
