Re: FetchFile Completion Strategy
It is possible the directory could get created and then something else fails, but personally I can't think of any downside to that, and seems like a better trade-off than routing to success, but if anyone else has thoughts please let me know. On Wed, Oct 17, 2018 at 12:07 PM Shawn Weeks wrote: > > Took a look at the pull request and that should handle the issue I was > seeing. Is there going to be an issue with the directory being left there if > something else fails? For my case the empty directory is fine. > > > Thanks > > Shawn Weeks > > > From: Bryan Bende > Sent: Wednesday, October 17, 2018 10:48:20 AM > To: users@nifi.apache.org > Subject: Re: FetchFile Completion Strategy > > Hi Shawn, > > The way the processor works is the following... > > 1) Perform checks to try and ensure the completion strategy will work later > 2) Perform the fetch and if successful then transfer to success and > commit the session > 3) Perform the completion strategy > > The issue is if the completion fails in step 3, it is too late to > route to failure because in step 2 the session was already committed. > > The reason it is done this way is because if we fetched the file > without committing the session and then executed the completion > strategy, it would be possible NiFi could crash at this moment before > committing the session, and now the source file could be gone from the > completion strategy and the data is also gone from NiFi since it was > never committed, so there would be data loss. > > All that being said, I think I was able to figure out an improvement > to protect against the case where there is a permission problem... > > https://github.com/apache/nifi/pull/3088 > > Thanks, > > Bryan > > On Mon, Oct 15, 2018 at 4:10 PM Shawn Weeks wrote: > > > > I was just wondering if it's expected behavior for the FetchFile Processor > > to eat errors related to the completion strategy. If for example your > > completion strategy is move and you specify a directory that doesn't exist > > or that NiFi doesn't have permissions on I'd have expected the Processor to > > route to failure not success. > > > > > > Thanks > > > > Shawn Weeks
Re: FetchFile Completion Strategy
Took a look at the pull request and that should handle the issue I was seeing. Is there going to be an issue with the directory being left there if something else fails? For my case the empty directory is fine. Thanks Shawn Weeks From: Bryan Bende Sent: Wednesday, October 17, 2018 10:48:20 AM To: users@nifi.apache.org Subject: Re: FetchFile Completion Strategy Hi Shawn, The way the processor works is the following... 1) Perform checks to try and ensure the completion strategy will work later 2) Perform the fetch and if successful then transfer to success and commit the session 3) Perform the completion strategy The issue is if the completion fails in step 3, it is too late to route to failure because in step 2 the session was already committed. The reason it is done this way is because if we fetched the file without committing the session and then executed the completion strategy, it would be possible NiFi could crash at this moment before committing the session, and now the source file could be gone from the completion strategy and the data is also gone from NiFi since it was never committed, so there would be data loss. All that being said, I think I was able to figure out an improvement to protect against the case where there is a permission problem... https://github.com/apache/nifi/pull/3088 Thanks, Bryan On Mon, Oct 15, 2018 at 4:10 PM Shawn Weeks wrote: > > I was just wondering if it's expected behavior for the FetchFile Processor to > eat errors related to the completion strategy. If for example your completion > strategy is move and you specify a directory that doesn't exist or that NiFi > doesn't have permissions on I'd have expected the Processor to route to > failure not success. > > > Thanks > > Shawn Weeks
Re: FetchFile Completion Strategy
Hi Shawn, The way the processor works is the following... 1) Perform checks to try and ensure the completion strategy will work later 2) Perform the fetch and if successful then transfer to success and commit the session 3) Perform the completion strategy The issue is if the completion fails in step 3, it is too late to route to failure because in step 2 the session was already committed. The reason it is done this way is because if we fetched the file without committing the session and then executed the completion strategy, it would be possible NiFi could crash at this moment before committing the session, and now the source file could be gone from the completion strategy and the data is also gone from NiFi since it was never committed, so there would be data loss. All that being said, I think I was able to figure out an improvement to protect against the case where there is a permission problem... https://github.com/apache/nifi/pull/3088 Thanks, Bryan On Mon, Oct 15, 2018 at 4:10 PM Shawn Weeks wrote: > > I was just wondering if it's expected behavior for the FetchFile Processor to > eat errors related to the completion strategy. If for example your completion > strategy is move and you specify a directory that doesn't exist or that NiFi > doesn't have permissions on I'd have expected the Processor to route to > failure not success. > > > Thanks > > Shawn Weeks