[jira] [Commented] (NIFI-5892) Wait timestamp lingers, potentially messing up downstream wait-notify pairs
[ https://issues.apache.org/jira/browse/NIFI-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729985#comment-16729985 ] ASF subversion and git services commented on NIFI-5892: --- Commit 1330b92cfa01f9dd04bade7a05a047ad6f080c4b in nifi's branch refs/heads/master from Otto Fowler [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=1330b92 ] NIFI-5892 Wait timestamp lingers, potentially messing up downstream wait-notify pairs Clear the wait timestamp when transferring to failur or success replace explicit attribute clear with function call, refactor and integrate into existing tests per review This closes #3233. Signed-off-by: Koji Kawamura > Wait timestamp lingers, potentially messing up downstream wait-notify pairs > --- > > Key: NIFI-5892 > URL: https://issues.apache.org/jira/browse/NIFI-5892 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Lars Birger Aasheim >Assignee: Otto Fowler >Priority: Minor > Labels: patch-available > Attachments: TestLingeringWaitTimestamps.xml > > Time Spent: 50m > Remaining Estimate: 0h > > The Wait processor makes use of the attribute "wait.start.timestamp" to keep > track of when it first encountered a flowfile. The observed behaviour is that > this attribute is set only if it does not exist, if else it is just checked > against the current time. With several wait-notify pairs in succession, there > is a likelihood that a flowfile will be routed directly to "expired" in one > of the downstream Wait processors due to a timestamp set upstream of it. I > suggest allowing an option of deleting the timestamp once a file is routed to > success (and perhaps also to expired). Currently I do this manually with an > UpdateAttribute processor. > > Attached is a template with necessary components and also an explanation of > how to reproduce the issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5892) Wait timestamp lingers, potentially messing up downstream wait-notify pairs
[ https://issues.apache.org/jira/browse/NIFI-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729395#comment-16729395 ] Koji Kawamura commented on NIFI-5892: - Thanks [~ottobackwards] for the PR. I'm reviewing it now. Will post few comments. [~laashe] , adding another processor property to control whether to clear wait.* attributes would be convenient. But the default value should be 'false'. Because some use-cases depend on the result wait.counter attribute values to analyze context at the downstream flow, especially when using multiple counter names. [~ottobackwards] I don't see the 'clear-wait-attribute' property added in the PR. Are you going to add it? I'm fine to not add the new 'clear-wait-attributes' property because UpdateAttribute can do the job if needed. The simpler each processor the better I guess. > Wait timestamp lingers, potentially messing up downstream wait-notify pairs > --- > > Key: NIFI-5892 > URL: https://issues.apache.org/jira/browse/NIFI-5892 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Lars Birger Aasheim >Assignee: Otto Fowler >Priority: Minor > Labels: patch-available > Attachments: TestLingeringWaitTimestamps.xml > > Time Spent: 10m > Remaining Estimate: 0h > > The Wait processor makes use of the attribute "wait.start.timestamp" to keep > track of when it first encountered a flowfile. The observed behaviour is that > this attribute is set only if it does not exist, if else it is just checked > against the current time. With several wait-notify pairs in succession, there > is a likelihood that a flowfile will be routed directly to "expired" in one > of the downstream Wait processors due to a timestamp set upstream of it. I > suggest allowing an option of deleting the timestamp once a file is routed to > success (and perhaps also to expired). Currently I do this manually with an > UpdateAttribute processor. > > Attached is a template with necessary components and also an explanation of > how to reproduce the issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5892) Wait timestamp lingers, potentially messing up downstream wait-notify pairs
[ https://issues.apache.org/jira/browse/NIFI-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16728504#comment-16728504 ] Lars Birger Aasheim commented on NIFI-5892: --- Great, thanks! > Wait timestamp lingers, potentially messing up downstream wait-notify pairs > --- > > Key: NIFI-5892 > URL: https://issues.apache.org/jira/browse/NIFI-5892 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Lars Birger Aasheim >Assignee: Otto Fowler >Priority: Minor > Labels: patch-available > Attachments: TestLingeringWaitTimestamps.xml > > Time Spent: 10m > Remaining Estimate: 0h > > The Wait processor makes use of the attribute "wait.start.timestamp" to keep > track of when it first encountered a flowfile. The observed behaviour is that > this attribute is set only if it does not exist, if else it is just checked > against the current time. With several wait-notify pairs in succession, there > is a likelihood that a flowfile will be routed directly to "expired" in one > of the downstream Wait processors due to a timestamp set upstream of it. I > suggest allowing an option of deleting the timestamp once a file is routed to > success (and perhaps also to expired). Currently I do this manually with an > UpdateAttribute processor. > > Attached is a template with necessary components and also an explanation of > how to reproduce the issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5892) Wait timestamp lingers, potentially messing up downstream wait-notify pairs
[ https://issues.apache.org/jira/browse/NIFI-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16726812#comment-16726812 ] Otto Fowler commented on NIFI-5892: --- [~laashe], [~ijokarumawak] patch pr submitted > Wait timestamp lingers, potentially messing up downstream wait-notify pairs > --- > > Key: NIFI-5892 > URL: https://issues.apache.org/jira/browse/NIFI-5892 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Lars Birger Aasheim >Assignee: Otto Fowler >Priority: Minor > Labels: patch-available > Attachments: TestLingeringWaitTimestamps.xml > > Time Spent: 10m > Remaining Estimate: 0h > > The Wait processor makes use of the attribute "wait.start.timestamp" to keep > track of when it first encountered a flowfile. The observed behaviour is that > this attribute is set only if it does not exist, if else it is just checked > against the current time. With several wait-notify pairs in succession, there > is a likelihood that a flowfile will be routed directly to "expired" in one > of the downstream Wait processors due to a timestamp set upstream of it. I > suggest allowing an option of deleting the timestamp once a file is routed to > success (and perhaps also to expired). Currently I do this manually with an > UpdateAttribute processor. > > Attached is a template with necessary components and also an explanation of > how to reproduce the issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5892) Wait timestamp lingers, potentially messing up downstream wait-notify pairs
[ https://issues.apache.org/jira/browse/NIFI-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16725653#comment-16725653 ] Lars Birger Aasheim commented on NIFI-5892: --- To solve the described problem, yes, but I guess there is no reason not to remove all wait.* attributes upon routing to the success relationship. This could be optional though, controlled by a true/false option in the processor. Personally I think such an option should be set to true by default (remove wait.* attributes). > Wait timestamp lingers, potentially messing up downstream wait-notify pairs > --- > > Key: NIFI-5892 > URL: https://issues.apache.org/jira/browse/NIFI-5892 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Lars Birger Aasheim >Assignee: Otto Fowler >Priority: Minor > Attachments: TestLingeringWaitTimestamps.xml > > > The Wait processor makes use of the attribute "wait.start.timestamp" to keep > track of when it first encountered a flowfile. The observed behaviour is that > this attribute is set only if it does not exist, if else it is just checked > against the current time. With several wait-notify pairs in succession, there > is a likelihood that a flowfile will be routed directly to "expired" in one > of the downstream Wait processors due to a timestamp set upstream of it. I > suggest allowing an option of deleting the timestamp once a file is routed to > success (and perhaps also to expired). Currently I do this manually with an > UpdateAttribute processor. > > Attached is a template with necessary components and also an explanation of > how to reproduce the issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5892) Wait timestamp lingers, potentially messing up downstream wait-notify pairs
[ https://issues.apache.org/jira/browse/NIFI-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16725131#comment-16725131 ] Otto Fowler commented on NIFI-5892: --- Ok, just to sum up: removing the timestamp alone is what is required, everything else can work as documented? > Wait timestamp lingers, potentially messing up downstream wait-notify pairs > --- > > Key: NIFI-5892 > URL: https://issues.apache.org/jira/browse/NIFI-5892 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Lars Birger Aasheim >Assignee: Otto Fowler >Priority: Minor > Attachments: TestLingeringWaitTimestamps.xml > > > The Wait processor makes use of the attribute "wait.start.timestamp" to keep > track of when it first encountered a flowfile. The observed behaviour is that > this attribute is set only if it does not exist, if else it is just checked > against the current time. With several wait-notify pairs in succession, there > is a likelihood that a flowfile will be routed directly to "expired" in one > of the downstream Wait processors due to a timestamp set upstream of it. I > suggest allowing an option of deleting the timestamp once a file is routed to > success (and perhaps also to expired). Currently I do this manually with an > UpdateAttribute processor. > > Attached is a template with necessary components and also an explanation of > how to reproduce the issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5892) Wait timestamp lingers, potentially messing up downstream wait-notify pairs
[ https://issues.apache.org/jira/browse/NIFI-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16724635#comment-16724635 ] Koji Kawamura commented on NIFI-5892: - [~ottobackwards] I'd leave default name blank. And if not specified, Wait processor works just the same. My idea is adding specified prefix to any wait related attribute so that the single FlowFile can be waited by few different Wait processors. E.g. using 3 Wait processors to wait for 'validation', 'registration' and 'reporting' phases ... etc. Having said that, while I was writing this comment, I'm just reminded that this use-case is already supported by 'Signal Counter Name'. Now I think deleting 'wait.start.timestamp' is the best approach, as [~laashe] suggested. wait.counters.* can be left as is because if subsequent flow uses Wait processor again, 'Signal Counter Name' can be used to wait for different notification. > Wait timestamp lingers, potentially messing up downstream wait-notify pairs > --- > > Key: NIFI-5892 > URL: https://issues.apache.org/jira/browse/NIFI-5892 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Lars Birger Aasheim >Assignee: Otto Fowler >Priority: Minor > Attachments: TestLingeringWaitTimestamps.xml > > > The Wait processor makes use of the attribute "wait.start.timestamp" to keep > track of when it first encountered a flowfile. The observed behaviour is that > this attribute is set only if it does not exist, if else it is just checked > against the current time. With several wait-notify pairs in succession, there > is a likelihood that a flowfile will be routed directly to "expired" in one > of the downstream Wait processors due to a timestamp set upstream of it. I > suggest allowing an option of deleting the timestamp once a file is routed to > success (and perhaps also to expired). Currently I do this manually with an > UpdateAttribute processor. > > Attached is a template with necessary components and also an explanation of > how to reproduce the issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5892) Wait timestamp lingers, potentially messing up downstream wait-notify pairs
[ https://issues.apache.org/jira/browse/NIFI-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16724229#comment-16724229 ] Otto Fowler commented on NIFI-5892: --- Thanks [~bende], I've been trying to keep that stuff in mind > Wait timestamp lingers, potentially messing up downstream wait-notify pairs > --- > > Key: NIFI-5892 > URL: https://issues.apache.org/jira/browse/NIFI-5892 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Lars Birger Aasheim >Assignee: Otto Fowler >Priority: Minor > Attachments: TestLingeringWaitTimestamps.xml > > > The Wait processor makes use of the attribute "wait.start.timestamp" to keep > track of when it first encountered a flowfile. The observed behaviour is that > this attribute is set only if it does not exist, if else it is just checked > against the current time. With several wait-notify pairs in succession, there > is a likelihood that a flowfile will be routed directly to "expired" in one > of the downstream Wait processors due to a timestamp set upstream of it. I > suggest allowing an option of deleting the timestamp once a file is routed to > success (and perhaps also to expired). Currently I do this manually with an > UpdateAttribute processor. > > Attached is a template with necessary components and also an explanation of > how to reproduce the issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5892) Wait timestamp lingers, potentially messing up downstream wait-notify pairs
[ https://issues.apache.org/jira/browse/NIFI-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16724209#comment-16724209 ] Bryan Bende commented on NIFI-5892: --- We can't assume everyone is using NiFi Registry so we can add properties to processors like we always have. Even with registry, it is a fairly specific scenario... if you had two environments like dev and prod, and you upgraded dev NiFi and it has a processor in a versioned flow with new properties you would then need to save the versioned flow to registry to capture the new properties. If you then went to prod it would show an upgrade on the flow which you could do even though you haven't upgrade prod NiFi yet, so if you do the upgrade then your processor becomes invalid because it doesn't have those properties. If you then delete those properties, now you made local changes again. > Wait timestamp lingers, potentially messing up downstream wait-notify pairs > --- > > Key: NIFI-5892 > URL: https://issues.apache.org/jira/browse/NIFI-5892 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Lars Birger Aasheim >Assignee: Otto Fowler >Priority: Minor > Attachments: TestLingeringWaitTimestamps.xml > > > The Wait processor makes use of the attribute "wait.start.timestamp" to keep > track of when it first encountered a flowfile. The observed behaviour is that > this attribute is set only if it does not exist, if else it is just checked > against the current time. With several wait-notify pairs in succession, there > is a likelihood that a flowfile will be routed directly to "expired" in one > of the downstream Wait processors due to a timestamp set upstream of it. I > suggest allowing an option of deleting the timestamp once a file is routed to > success (and perhaps also to expired). Currently I do this manually with an > UpdateAttribute processor. > > Attached is a template with necessary components and also an explanation of > how to reproduce the issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5892) Wait timestamp lingers, potentially messing up downstream wait-notify pairs
[ https://issues.apache.org/jira/browse/NIFI-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16724113#comment-16724113 ] Otto Fowler commented on NIFI-5892: --- adding new properties may cause issues for registry manage flows won't it? [~bbende]? What would you imagine the default name to be if not configured [~ijokarumawak]? They would have to be unique, but consistent. > Wait timestamp lingers, potentially messing up downstream wait-notify pairs > --- > > Key: NIFI-5892 > URL: https://issues.apache.org/jira/browse/NIFI-5892 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Lars Birger Aasheim >Assignee: Otto Fowler >Priority: Minor > Attachments: TestLingeringWaitTimestamps.xml > > > The Wait processor makes use of the attribute "wait.start.timestamp" to keep > track of when it first encountered a flowfile. The observed behaviour is that > this attribute is set only if it does not exist, if else it is just checked > against the current time. With several wait-notify pairs in succession, there > is a likelihood that a flowfile will be routed directly to "expired" in one > of the downstream Wait processors due to a timestamp set upstream of it. I > suggest allowing an option of deleting the timestamp once a file is routed to > success (and perhaps also to expired). Currently I do this manually with an > UpdateAttribute processor. > > Attached is a template with necessary components and also an explanation of > how to reproduce the issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5892) Wait timestamp lingers, potentially messing up downstream wait-notify pairs
[ https://issues.apache.org/jira/browse/NIFI-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723732#comment-16723732 ] Koji Kawamura commented on NIFI-5892: - I think all Wait related attributes should be cleared to process the same FlowFile with another Wait processor. Adding a boolean property something like 'Clear Successful Wait Attributes' with description to note that whether to clear wait related attributes when the FlowFiles wait activity is done, meaning when it gets routed to 'success'. An alternative approach is adding a new optional processor property to specify 'Wait Attribute Prefix'. This way, multiple Wait processors can be used for the same FlowFile without clearing previous attributes by separating attribute name spaces. Being able to keep attributes may be helpful at the subsequent flow. I think this approach will provide more flexibility. > Wait timestamp lingers, potentially messing up downstream wait-notify pairs > --- > > Key: NIFI-5892 > URL: https://issues.apache.org/jira/browse/NIFI-5892 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Lars Birger Aasheim >Assignee: Otto Fowler >Priority: Minor > Attachments: TestLingeringWaitTimestamps.xml > > > The Wait processor makes use of the attribute "wait.start.timestamp" to keep > track of when it first encountered a flowfile. The observed behaviour is that > this attribute is set only if it does not exist, if else it is just checked > against the current time. With several wait-notify pairs in succession, there > is a likelihood that a flowfile will be routed directly to "expired" in one > of the downstream Wait processors due to a timestamp set upstream of it. I > suggest allowing an option of deleting the timestamp once a file is routed to > success (and perhaps also to expired). Currently I do this manually with an > UpdateAttribute processor. > > Attached is a template with necessary components and also an explanation of > how to reproduce the issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5892) Wait timestamp lingers, potentially messing up downstream wait-notify pairs
[ https://issues.apache.org/jira/browse/NIFI-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16719714#comment-16719714 ] Otto Fowler commented on NIFI-5892: --- I think the wait.counters.* would need to be cleared as well. [~bbende]? [~joewitt]? > Wait timestamp lingers, potentially messing up downstream wait-notify pairs > --- > > Key: NIFI-5892 > URL: https://issues.apache.org/jira/browse/NIFI-5892 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Lars Birger Aasheim >Assignee: Otto Fowler >Priority: Minor > Attachments: TestLingeringWaitTimestamps.xml > > > The Wait processor makes use of the attribute "wait.start.timestamp" to keep > track of when it first encountered a flowfile. The observed behaviour is that > this attribute is set only if it does not exist, if else it is just checked > against the current time. With several wait-notify pairs in succession, there > is a likelihood that a flowfile will be routed directly to "expired" in one > of the downstream Wait processors due to a timestamp set upstream of it. I > suggest allowing an option of deleting the timestamp once a file is routed to > success (and perhaps also to expired). Currently I do this manually with an > UpdateAttribute processor. > > Attached is a template with necessary components and also an explanation of > how to reproduce the issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
