John MergeContent cannot reliably work well with cron scheduling. That component is designed to get threads consistently so it can perform its bin packing function and time and size based kick out functions.
If it ever worked with cron scheduling that was mostly on accident I'd say. Thanks Joe On Tue, Feb 21, 2023 at 1:26 PM John McGinn via users <[email protected]> wrote: > > I've created an XML template as a GitHub Gist of the possible MergeContent > cron scheduling bug. The template was created in 1.12.0, and uploaded to > 1.18.0. Worked in 1.12, didn't work in 1.18.0. I downloaded 1.20.0 locally, > and it doesn't work there either. I have the initial GenerateFlowFile and the > MergeContent set for cron scheduling with a 25 minute break in between the > two. (I know the file extension is .js, but it's an XML file.) > > https://gist.github.com/Figgie123/245e49ca29135ef6e4db50a7b4f5d5b7.js > On Friday, February 17, 2023, 01:39:17 PM EST, John McGinn > <[email protected]> wrote: > > > Hello, > > NiFi 1.19.0 - I need some help in trying to make my idea work, or figure out > the better way to do this. > > I've got a flow that retrieves data from a few data sources, enhances > individual flow files, converts attributes to CSV and then merges that all > back into a single file. It takes roughly 20 minutes for the process to run > from start to the MergeContent part, so when I do it manually, I stop the > MergeContent processor until all flowfiles are in the queue waiting, and then > I start the MergeContent processor. (Run One Time doesn't work for some > reason.) That works fine, manually. > > When I try to put cron scheduling in, it never kicks off. For instance, the > initial processor in the flow has a cron schedule of the top of the hour. (0 > 0 * * * ?) I then put 25 past the hour for Merge Content (0 25 * * * ?). When > I start the flow, the flowfiles are generated and queue up in front of > MergeContent by 25 minutes past the hour, but the MergeContent never kicks > off. > > I added a correlation attribute recently and removed the cron entry, but the > MergeContent just creates small bunches of merged files. > > I even attempted to put a cron on the AttributesToCSV with a maximum bin age > on the Merge Content, since it takes less than a minute for the > AttribuesToCSV to process the flowfiles at that point, but the cron didn't > kick off there either. > > Any ideas on how to get this to work? > > Thanks, > John
