Hello, Is there anyone who can help with the resolution of this issue? I have looked at the JIRA[1], and the referenced PR is closed.
There seems to be no other way or details available to fix or bypass this issue. This is kind of a blocker in upgrading the Nifi version, and Any help on this would be greatly appreciated. [1] https://issues.apache.org/jira/browse/NIFI-14392 Kind Regards -- Arvind Singh Tomar On Fri, Jul 12, 2024 at 1:15 AM Jim Steinebrey <[email protected]> wrote: > Hi Peter, > Thanks for trying that and getting back to me. At least we have confirmed > the root cause. > I am sorry that suggested workaround does not seem feasible. > I explored the code for other workarounds and none proved usable. > > Does anyone else have a suggested workaround for Peter’s issue? > > Given the uncertainty of a workaround, Peter, I suggest you enter a Jira > ticket describing the situation and what you have learned about root cause,. > Then hopefully someone will make a change before the final NiFi 2.0 > release. > > Best, > Jim Steinebrey > > > On Jul 11, 2024, at 12:44 PM, Sharp, Peter <[email protected]> wrote: > > Hi Jim, > > Thanks for the response. > > You are correct the local state management referenced in the > state-management.xml has partitions set. > > The local state directory is partitioned to about 20 partitions, each with > its own .journal file. > > I went ahead and concatenated all of the .journal files from the partition > directories along with the .journal file under /state/local/journals/ and > renamed the new file to follow the numbered sequence of journal files in > /state/local/journals. > > I tried placing the new .journal file first in /state/local and restarting > nifi, it was not picked up. I then placed it in /state/local/journals where > it was picked up but is throwing errors stating the .journal file does not > appear to be a valid journal file. > > My guess is theres some metadata or header information that is being > corrupted when the files are combined. Hence not sure if this is the best > approach for restoring the processor state for the ListSFTP processors. > > Any further suggestions would be greatly appreciated! > > Cheers ~ > > Peter > > *From:* Jim Steinebrey <[email protected]> > *Sent:* Wednesday, July 10, 2024 11:00 AM > *To:* [email protected] > *Subject:* Re: ListSFTP Processors losing state after upgrading from > 1.26.0 to 2.0.0-M3 > > > > *IMPORTANT!! - External Content - Please use caution.* > > Hi Peter, > Can you review the configuration for your local state provider from > state-management.xml for 1.26.0 and 2.0.0-M3 configurations? > > From comparing the code in 1.26.0 vs 2.x, > My working theory is that you have partitions set to the default of 16 for > both old and new Nifi configurations. > NiFI 1.26.0 uses > <property name="Partitions">16</property> > to decide how many partition subdirectories to create under your state > base path (./state/local) directory. > Partition directories for 1.26.0 are in your ./state/local/ and begin with > prefix partition- > > > But NiFi 2.x ignores the number specified by > <property name="Partitions">16</property> > and never creates any partition subdirectories as far I can see in the > code. > > > Therefore NiFI 2.x does NOT look for any partition subdirectories. > > > This partition directory change in WriteAheadLocalStateProvider.java was a > side effect of this change. > MinimalLockingWriteAheadLog.java was the deprecated class that was removed. > MinimalLockingWriteAheadLog had the logic for interacting with partition > directories. > [NIFI-11833] Remove deprecated classes in nifi-commons - ASF JIRA > <https://issues.apache.org/jira/browse/NIFI-11833> > issues.apache.org <https://issues.apache.org/jira/browse/NIFI-11833> > <image001.png> <https://issues.apache.org/jira/browse/NIFI-11833> > > Not sure if this is easy or not because I do not have local state > partition directories handy to inspect, but as a workaround can you > manually move/combine all the state information from the partition > directories' files and merge them into a single file that you place > directly in ./state/local > > Please let know what you find out, Peter, > Jim Steinebrey > > > > > On Jul 9, 2024, at 4:13 PM, Sharp, Peter <[email protected]> wrote: > > Hello, > > I am reaching out with an issue I am facing when attempting to upgrade > Nifi from 1.26.0 to 2.0.0-M3. > > After migrating from 1.26.0 to 2.0.0-M3 all of the ListSFTP processors > lose their stored state. > > We are running Nifi as a standalone instance and are using local state > management. > > What I did and what has worked for me in the past with minor version > upgrades (1.23.2 to 1.26.0) is copying over the existing /state/local > directory to the new version. > > I validated that the state-mangement.xml file is referencing the local > state provider with the correct path to the state directory. I also > validated the state properties in the nifi.properties file are consistent > with the old version. > > When checking the app logs between the old running version (1.26.0) and > the new running version (2.0.0-M3) I am seeing some differences between > where its trying to load state. (see below screenshots of logs) > > 1.26.0 nifi-app-log - > <image002.png> > > 2.0.0-M3 nifi-app-log - > <image003.png> > > It appears that its unable to pick the snapshot file successfully even > though it exists in the /state/local directory. > > Any help on this would be greatly appreciated! > > Thanks ~ > > *Peter Sharp* > *Software Engineer* > *Data & Analytics | ICE Mortgage Technology* > M: +1 760-390-4895 > [email protected] > www.ice.com > <image001.png> > > The information contained in this message is proprietary and/or > confidential. If you are not the intended recipient, please: (i) delete the > message and all copies; (ii) do not disclose, distribute or use the message > in any manner; and (iii) notify the sender immediately. In addition, please > be aware that any message addressed to our domain is subject to archiving > and review by persons other than the intended recipient. Thank you. > > > The information contained in this message is proprietary and/or > confidential. If you are not the intended recipient, please: (i) delete the > message and all copies; (ii) do not disclose, distribute or use the message > in any manner; and (iii) notify the sender immediately. In addition, please > be aware that any message addressed to our domain is subject to archiving > and review by persons other than the intended recipient. Thank you. > > >
