A single node still needs a state manager to survive restarts of the application, but there is no more distributed state at that point, so the local state manager is backed by a write-ahead-log store in ./state under NiFi’s home directory.
- Bryan > On Sep 20, 2017, at 8:06 PM, Gino Lisignoli <[email protected]> wrote: > > I can confirm the sate is being managed by zookeeper. I've been doing some > more testing today and I haven't been able to make the ListXXX processes fail > by manually causing the primary node to change (just using tcpdump to drop > zookeeper traffic, causing a node change). So I'm not sure at the moment if > it's because I'm in a cluster or not. I'm setting up a seperate nifi node > that will only perform ListXXX operations without being in a cluster. > > Though even if this is a single node, will it need a state manager configured? > > On Thu, Sep 21, 2017 at 10:58 AM, Bryan Bende <[email protected] > <mailto:[email protected]>> wrote: > This should be correct behavior... the cache approach was used before the > state manager got introduced, and the cache was left there to migrate old > state, but we do need to pick a release and remove those lingering cache > properties. > > Do you have the cluster state manager configured in state managers xml? > > It should be using zoo keeper when in a cluster. > > On Wed, Sep 20, 2017 at 6:23 PM Gino Lisignoli <[email protected] > <mailto:[email protected]>> wrote: > Hi Joe > > The distributed cache service property of ListXXX doesn't seem to matter > anymore. Which was quite a surprise to me. It just gets handled by the state > manager: > https://github.com/apache/nifi/blob/e68ff153e81ddb82d1136d44a96bdb7a70da86d1/nifi-nar-bundles/nifi-extension-utils/nifi-processor-utils/src/main/java/org/apache/nifi/processor/util/list/AbstractListProcessor.java#L264 > > <https://github.com/apache/nifi/blob/e68ff153e81ddb82d1136d44a96bdb7a70da86d1/nifi-nar-bundles/nifi-extension-utils/nifi-processor-utils/src/main/java/org/apache/nifi/processor/util/list/AbstractListProcessor.java#L264> > > You can test this via setting the Distributed Cache Service to a redis server > and watch no connections ever being made, which was how I found out. Not sure > if this is intended or not. > > > On Wed, Sep 20, 2017 at 12:42 PM, Joe Witt <[email protected] > <mailto:[email protected]>> wrote: > Gino > > When the primary node shifts the new primary node should take over the > function of doing the listings for you. You'll want to be using the > distributed cache property of the ListFTP/etc.. processors to help > keep state in such cases. > > Thanks > > On Tue, Sep 19, 2017 at 4:50 PM, Gino Lisignoli <[email protected] > <mailto:[email protected]>> wrote: > > Hi > > > > I can confirm that the cause of my problem is the primary node shifting to a > > new node in my cluster. This causes all of the ListXXX processors to stop > > working (as they are set to only run on my primary node). I'm not sure what > > is causing my primary node to shift, or why the processors are unable to > > recover when the node shifts. If I find any more info I'll raise a ticket. > > > > On Thu, Sep 14, 2017 at 8:08 AM, Joe Witt <[email protected] > > <mailto:[email protected]>> wrote: > >> > >> Thanks Gino. If you can confirm/replicate the behavior please do file a > >> jira. > >> > >> Thanks > >> > >> On Wed, Sep 13, 2017 at 4:06 PM, Gino Lisignoli <[email protected] > >> <mailto:[email protected]>> > >> wrote: > >> > In 1.4.0-SNAPSHOT, the ListFTP processor fails to add new files after > >> > several hours. > >> > > >> > At the moment I'm testing this on a cluster of 2. But I'm looking to > >> > replicate my issue on a single instance as well. > >> > > >> > I'm not sure if this is a cluster state problem, a processor state > >> > problem > >> > or the new ListXXX commits > >> > > >> > (https://github.com/apache/nifi/commit/e68ff153e81ddb82d1136d44a96bdb7a70da86d1 > >> > > >> > <https://github.com/apache/nifi/commit/e68ff153e81ddb82d1136d44a96bdb7a70da86d1> > >> > and > >> > > >> > https://github.com/apache/nifi/commit/28ee70222b892fb799f5f74a31a9de678d9fb629 > >> > > >> > <https://github.com/apache/nifi/commit/28ee70222b892fb799f5f74a31a9de678d9fb629>). > >> > > >> > I'll be looking at those commits first around the ListFTP process. > >> > Unless > >> > theres any advice can be offered. > >> > > >> > Should I submit this to Jira issues, or wait until I can > >> > confirm/replicate > >> > the behaviour? > > > > > > -- > Sent from Gmail Mobile >
signature.asc
Description: Message signed with OpenPGP
