[jira] [Commented] (NIFI-4805) allow delayed transfer
[ https://issues.apache.org/jira/browse/NIFI-4805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655640#comment-16655640 ] ASF GitHub Bot commented on NIFI-4805: -- Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3078 @joewitt It's not done, more of a rough draft, but take a look at my latest commit and let me know if this is more in line with what you were thinking. > allow delayed transfer > -- > > Key: NIFI-4805 > URL: https://issues.apache.org/jira/browse/NIFI-4805 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Martin Mucha >Assignee: Peter Wicks >Priority: Minor > Attachments: retry.xml > > > Nifi has concept of penalization, but this penalization has fixed delay, and > there isn't way how to change it dynamically. > If we want to implement retry flow, where FlowFile flows in loop, we can > either lower performance of Processor via yielding it, or we can do active > waiting. And this is actually recommended as a correct way how to do that. > It seems, that we can easily implement better RetryProcessor, all we missing > is `session.penalize` which accepts `penalizationPeriod`. Processor then can > gradually prolong waiting time after each failure. > > Would it be possible to make such method visible? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4805) allow delayed transfer
[ https://issues.apache.org/jira/browse/NIFI-4805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654161#comment-16654161 ] ASF GitHub Bot commented on NIFI-4805: -- Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/3078 @patricker yeah sorry - i agree. I just meant that once you pull a flowfile from the queue and find it isn't time (and perhaps not even close to time) to send it to success you need to do something with it. That something could be penalize it and route to a 'not ready' process so you're not constantly interrogating the same file and others can go through > allow delayed transfer > -- > > Key: NIFI-4805 > URL: https://issues.apache.org/jira/browse/NIFI-4805 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Martin Mucha >Assignee: Peter Wicks >Priority: Minor > Attachments: retry.xml > > > Nifi has concept of penalization, but this penalization has fixed delay, and > there isn't way how to change it dynamically. > If we want to implement retry flow, where FlowFile flows in loop, we can > either lower performance of Processor via yielding it, or we can do active > waiting. And this is actually recommended as a correct way how to do that. > It seems, that we can easily implement better RetryProcessor, all we missing > is `session.penalize` which accepts `penalizationPeriod`. Processor then can > gradually prolong waiting time after each failure. > > Would it be possible to make such method visible? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4805) allow delayed transfer
[ https://issues.apache.org/jira/browse/NIFI-4805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654143#comment-16654143 ] ASF GitHub Bot commented on NIFI-4805: -- Github user bdesert commented on the issue: https://github.com/apache/nifi/pull/3078 @joewitt , @patricker While working on implementations, I faced a problem of penalization on failure. Some processors are penalizing FF on failure, some do only on rollback, some gives you a choice whether to penalize or not on failure. Use Case: FetchFile -> transform -> PutHiveStreaming On PutHiveStreaming failure -> 1) wait 1 min + retry, 2) create a table (puthiveql will make you overwrite content, so you cannot do that sequentially). There are more use, fo now I do like this:  > allow delayed transfer > -- > > Key: NIFI-4805 > URL: https://issues.apache.org/jira/browse/NIFI-4805 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Martin Mucha >Assignee: Peter Wicks >Priority: Minor > Attachments: retry.xml > > > Nifi has concept of penalization, but this penalization has fixed delay, and > there isn't way how to change it dynamically. > If we want to implement retry flow, where FlowFile flows in loop, we can > either lower performance of Processor via yielding it, or we can do active > waiting. And this is actually recommended as a correct way how to do that. > It seems, that we can easily implement better RetryProcessor, all we missing > is `session.penalize` which accepts `penalizationPeriod`. Processor then can > gradually prolong waiting time after each failure. > > Would it be possible to make such method visible? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4805) allow delayed transfer
[ https://issues.apache.org/jira/browse/NIFI-4805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654133#comment-16654133 ] ASF GitHub Bot commented on NIFI-4805: -- Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3078 I'm confused by your message @joewitt. I understand, "Yes, make it a delay processor". But then you mention `penalize` a lot of times. My thought was if it's a delay processor then it wouldn't ever call Penalize, but just not pass FlowFile's through to the next queue until the appropriate amount of time had passed. > allow delayed transfer > -- > > Key: NIFI-4805 > URL: https://issues.apache.org/jira/browse/NIFI-4805 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Martin Mucha >Assignee: Peter Wicks >Priority: Minor > Attachments: retry.xml > > > Nifi has concept of penalization, but this penalization has fixed delay, and > there isn't way how to change it dynamically. > If we want to implement retry flow, where FlowFile flows in loop, we can > either lower performance of Processor via yielding it, or we can do active > waiting. And this is actually recommended as a correct way how to do that. > It seems, that we can easily implement better RetryProcessor, all we missing > is `session.penalize` which accepts `penalizationPeriod`. Processor then can > gradually prolong waiting time after each failure. > > Would it be possible to make such method visible? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4805) allow delayed transfer
[ https://issues.apache.org/jira/browse/NIFI-4805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654122#comment-16654122 ] ASF GitHub Bot commented on NIFI-4805: -- Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/3078 I do think so to be honest and i say that knowing it would be a good bit more work. You can penalize flowfiles though that are 'not ready' - we might need to allow penalization values on a per flowfile basis (add an api call for it i mean). I think it makes a lot more sense to the user and would be more powerful/configurable. > allow delayed transfer > -- > > Key: NIFI-4805 > URL: https://issues.apache.org/jira/browse/NIFI-4805 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Martin Mucha >Assignee: Peter Wicks >Priority: Minor > Attachments: retry.xml > > > Nifi has concept of penalization, but this penalization has fixed delay, and > there isn't way how to change it dynamically. > If we want to implement retry flow, where FlowFile flows in loop, we can > either lower performance of Processor via yielding it, or we can do active > waiting. And this is actually recommended as a correct way how to do that. > It seems, that we can easily implement better RetryProcessor, all we missing > is `session.penalize` which accepts `penalizationPeriod`. Processor then can > gradually prolong waiting time after each failure. > > Would it be possible to make such method visible? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4805) allow delayed transfer
[ https://issues.apache.org/jira/browse/NIFI-4805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654116#comment-16654116 ] ASF GitHub Bot commented on NIFI-4805: -- Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3078 @joewitt Originally I was going to write a processor that looked at how long files had been queued and only moved them forward after the appropriate time had elapsed, a true Delay processor. But when I found this existing JIRA ticket, and since two other developers had discussed it quite a bit, I went with building a Penalize processor. The use case is absolutely causing a delay in the Flow. Would you prefer I go back to something more like my original approach and only let FlowFile's through based on their Queued times? This way would be more configurable, and we could use an optional attribute/EL to decide how long they should be forced to delay. > allow delayed transfer > -- > > Key: NIFI-4805 > URL: https://issues.apache.org/jira/browse/NIFI-4805 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Martin Mucha >Assignee: Peter Wicks >Priority: Minor > Attachments: retry.xml > > > Nifi has concept of penalization, but this penalization has fixed delay, and > there isn't way how to change it dynamically. > If we want to implement retry flow, where FlowFile flows in loop, we can > either lower performance of Processor via yielding it, or we can do active > waiting. And this is actually recommended as a correct way how to do that. > It seems, that we can easily implement better RetryProcessor, all we missing > is `session.penalize` which accepts `penalizationPeriod`. Processor then can > gradually prolong waiting time after each failure. > > Would it be possible to make such method visible? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4805) allow delayed transfer
[ https://issues.apache.org/jira/browse/NIFI-4805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654086#comment-16654086 ] ASF GitHub Bot commented on NIFI-4805: -- Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/3078 We should avoid calling this processor a 'PenalizeFlowFile' processor. First, we have a notion of penalization in the framework already and it has a very specific meaning and it kicks in when processors reach certain failure conditions that are believed to be specific to a given flowfile or set of flowfiles and that by penalizing them the condition might improve on its own. Second, the use for this processor in a flow might not be for the purposes of penalizing that flowfile but rather might simply be to cause a given wait or hold to occur. I'd recommend calling it DelayFlowFile. <...then I went and looked at the code> It is literally just calling penalize flow file so it makes sense why the name was given. What use case is this for again? If we're really wanting to create a DelayFlowFile processor, which is what it sounds like was the intent, we should do that because penalization means something else. > allow delayed transfer > -- > > Key: NIFI-4805 > URL: https://issues.apache.org/jira/browse/NIFI-4805 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Martin Mucha >Assignee: Peter Wicks >Priority: Minor > Attachments: retry.xml > > > Nifi has concept of penalization, but this penalization has fixed delay, and > there isn't way how to change it dynamically. > If we want to implement retry flow, where FlowFile flows in loop, we can > either lower performance of Processor via yielding it, or we can do active > waiting. And this is actually recommended as a correct way how to do that. > It seems, that we can easily implement better RetryProcessor, all we missing > is `session.penalize` which accepts `penalizationPeriod`. Processor then can > gradually prolong waiting time after each failure. > > Would it be possible to make such method visible? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4805) allow delayed transfer
[ https://issues.apache.org/jira/browse/NIFI-4805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654052#comment-16654052 ] ASF GitHub Bot commented on NIFI-4805: -- Github user bdesert commented on the issue: https://github.com/apache/nifi/pull/3078 +1 LGTM Pulled the changes, tested on local env, all looks good to go. @markap14 , @alopresto , any additional comments? > allow delayed transfer > -- > > Key: NIFI-4805 > URL: https://issues.apache.org/jira/browse/NIFI-4805 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Martin Mucha >Assignee: Peter Wicks >Priority: Minor > Attachments: retry.xml > > > Nifi has concept of penalization, but this penalization has fixed delay, and > there isn't way how to change it dynamically. > If we want to implement retry flow, where FlowFile flows in loop, we can > either lower performance of Processor via yielding it, or we can do active > waiting. And this is actually recommended as a correct way how to do that. > It seems, that we can easily implement better RetryProcessor, all we missing > is `session.penalize` which accepts `penalizationPeriod`. Processor then can > gradually prolong waiting time after each failure. > > Would it be possible to make such method visible? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4805) allow delayed transfer
[ https://issues.apache.org/jira/browse/NIFI-4805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16653958#comment-16653958 ] ASF GitHub Bot commented on NIFI-4805: -- Github user bdesert commented on a diff in the pull request: https://github.com/apache/nifi/pull/3078#discussion_r226029040 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PenalizeFlowFile.java --- @@ -0,0 +1,74 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.processors.standard; +import org.apache.nifi.annotation.behavior.EventDriven; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.SideEffectFree; +import org.apache.nifi.annotation.behavior.SupportsBatching; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.components.PropertyDescriptor; --- End diff -- ```suggestion ``` > allow delayed transfer > -- > > Key: NIFI-4805 > URL: https://issues.apache.org/jira/browse/NIFI-4805 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Martin Mucha >Assignee: Peter Wicks >Priority: Minor > Attachments: retry.xml > > > Nifi has concept of penalization, but this penalization has fixed delay, and > there isn't way how to change it dynamically. > If we want to implement retry flow, where FlowFile flows in loop, we can > either lower performance of Processor via yielding it, or we can do active > waiting. And this is actually recommended as a correct way how to do that. > It seems, that we can easily implement better RetryProcessor, all we missing > is `session.penalize` which accepts `penalizationPeriod`. Processor then can > gradually prolong waiting time after each failure. > > Would it be possible to make such method visible? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4805) allow delayed transfer
[ https://issues.apache.org/jira/browse/NIFI-4805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16653959#comment-16653959 ] ASF GitHub Bot commented on NIFI-4805: -- Github user bdesert commented on a diff in the pull request: https://github.com/apache/nifi/pull/3078#discussion_r226029162 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PenalizeFlowFile.java --- @@ -0,0 +1,74 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.processors.standard; +import org.apache.nifi.annotation.behavior.EventDriven; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.SideEffectFree; +import org.apache.nifi.annotation.behavior.SupportsBatching; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.processor.AbstractProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.ProcessorInitializationContext; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; + +import java.util.Collections; +import java.util.HashSet; +import java.util.List; --- End diff -- ```suggestion ``` > allow delayed transfer > -- > > Key: NIFI-4805 > URL: https://issues.apache.org/jira/browse/NIFI-4805 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Martin Mucha >Assignee: Peter Wicks >Priority: Minor > Attachments: retry.xml > > > Nifi has concept of penalization, but this penalization has fixed delay, and > there isn't way how to change it dynamically. > If we want to implement retry flow, where FlowFile flows in loop, we can > either lower performance of Processor via yielding it, or we can do active > waiting. And this is actually recommended as a correct way how to do that. > It seems, that we can easily implement better RetryProcessor, all we missing > is `session.penalize` which accepts `penalizationPeriod`. Processor then can > gradually prolong waiting time after each failure. > > Would it be possible to make such method visible? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4805) allow delayed transfer
[ https://issues.apache.org/jira/browse/NIFI-4805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16653956#comment-16653956 ] ASF GitHub Bot commented on NIFI-4805: -- Github user bdesert commented on a diff in the pull request: https://github.com/apache/nifi/pull/3078#discussion_r226028831 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PenalizeFlowFile.java --- @@ -0,0 +1,74 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.processors.standard; +import org.apache.nifi.annotation.behavior.EventDriven; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.SideEffectFree; +import org.apache.nifi.annotation.behavior.SupportsBatching; +import org.apache.nifi.annotation.behavior.WritesAttribute; --- End diff -- ```suggestion ``` > allow delayed transfer > -- > > Key: NIFI-4805 > URL: https://issues.apache.org/jira/browse/NIFI-4805 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Martin Mucha >Assignee: Peter Wicks >Priority: Minor > Attachments: retry.xml > > > Nifi has concept of penalization, but this penalization has fixed delay, and > there isn't way how to change it dynamically. > If we want to implement retry flow, where FlowFile flows in loop, we can > either lower performance of Processor via yielding it, or we can do active > waiting. And this is actually recommended as a correct way how to do that. > It seems, that we can easily implement better RetryProcessor, all we missing > is `session.penalize` which accepts `penalizationPeriod`. Processor then can > gradually prolong waiting time after each failure. > > Would it be possible to make such method visible? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4805) allow delayed transfer
[
https://issues.apache.org/jira/browse/NIFI-4805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16653957#comment-16653957
]
ASF GitHub Bot commented on NIFI-4805:
--
Github user bdesert commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3078#discussion_r226035394
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PenalizeFlowFile.java
---
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"penalty", "penalize", "flowfile"})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Penalizes a FlowFile.")
--- End diff --
```suggestion
@CapabilityDescription("This processor provides capability to penalize flow
files. "
+ "Every flow file will be penalized as per 'Penalty Duration'
property of the processor.")
```
> allow delayed transfer
> --
>
> Key: NIFI-4805
> URL: https://issues.apache.org/jira/browse/NIFI-4805
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: Martin Mucha
>Assignee: Peter Wicks
>Priority: Minor
> Attachments: retry.xml
>
>
> Nifi has concept of penalization, but this penalization has fixed delay, and
> there isn't way how to change it dynamically.
> If we want to implement retry flow, where FlowFile flows in loop, we can
> either lower performance of Processor via yielding it, or we can do active
> waiting. And this is actually recommended as a correct way how to do that.
> It seems, that we can easily implement better RetryProcessor, all we missing
> is `session.penalize` which accepts `penalizationPeriod`. Processor then can
> gradually prolong waiting time after each failure.
>
> Would it be possible to make such method visible?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-4805) allow delayed transfer
[ https://issues.apache.org/jira/browse/NIFI-4805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16653955#comment-16653955 ] ASF GitHub Bot commented on NIFI-4805: -- Github user bdesert commented on a diff in the pull request: https://github.com/apache/nifi/pull/3078#discussion_r226028907 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PenalizeFlowFile.java --- @@ -0,0 +1,74 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.processors.standard; +import org.apache.nifi.annotation.behavior.EventDriven; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.SideEffectFree; +import org.apache.nifi.annotation.behavior.SupportsBatching; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; --- End diff -- ```suggestion ``` > allow delayed transfer > -- > > Key: NIFI-4805 > URL: https://issues.apache.org/jira/browse/NIFI-4805 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Martin Mucha >Assignee: Peter Wicks >Priority: Minor > Attachments: retry.xml > > > Nifi has concept of penalization, but this penalization has fixed delay, and > there isn't way how to change it dynamically. > If we want to implement retry flow, where FlowFile flows in loop, we can > either lower performance of Processor via yielding it, or we can do active > waiting. And this is actually recommended as a correct way how to do that. > It seems, that we can easily implement better RetryProcessor, all we missing > is `session.penalize` which accepts `penalizationPeriod`. Processor then can > gradually prolong waiting time after each failure. > > Would it be possible to make such method visible? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4805) allow delayed transfer
[ https://issues.apache.org/jira/browse/NIFI-4805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16653843#comment-16653843 ] ASF GitHub Bot commented on NIFI-4805: -- Github user bdesert commented on the issue: https://github.com/apache/nifi/pull/3078 @patricker , the code looks good. I'll test them locally later today and will provide my feedback. > allow delayed transfer > -- > > Key: NIFI-4805 > URL: https://issues.apache.org/jira/browse/NIFI-4805 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Martin Mucha >Assignee: Peter Wicks >Priority: Minor > Attachments: retry.xml > > > Nifi has concept of penalization, but this penalization has fixed delay, and > there isn't way how to change it dynamically. > If we want to implement retry flow, where FlowFile flows in loop, we can > either lower performance of Processor via yielding it, or we can do active > waiting. And this is actually recommended as a correct way how to do that. > It seems, that we can easily implement better RetryProcessor, all we missing > is `session.penalize` which accepts `penalizationPeriod`. Processor then can > gradually prolong waiting time after each failure. > > Would it be possible to make such method visible? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4805) allow delayed transfer
[
https://issues.apache.org/jira/browse/NIFI-4805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16653838#comment-16653838
]
ASF GitHub Bot commented on NIFI-4805:
--
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3078#discussion_r226012968
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PenalizeFlowFile.java
---
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"penalty", "penalize", "flowfile"})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Penalizes a FlowFile.")
+@WritesAttributes({
+@WritesAttribute(attribute = "penalization.count.{processor
uuid}", description = "How many times this processor has penalized this
FlowFile.")
+})
+
+public class PenalizeFlowFile extends AbstractProcessor {
+public static final Relationship REL_SUCCESS = new
Relationship.Builder().name("success")
+.description("Successfully penalized FlowFile").build();
+
+private List properties;
+private Set relationships;
+
+@Override
+protected void init(final ProcessorInitializationContext context) {
+final Set relationships = new HashSet<>();
+relationships.add(REL_SUCCESS);
+this.relationships = Collections.unmodifiableSet(relationships);
+}
+@Override
+public Set getRelationships() {
+return relationships;
+}
+
+@Override
+public void onTrigger(ProcessContext context, ProcessSession session)
throws ProcessException {
+FlowFile flowFile = session.get();
+if (flowFile == null) {
+return;
+}
+
+// Track how many times a FlowFile passes through this processor
to better support the Retry use case
+final String retryAttrName = "penalization.count." +
this.getIdentifier();
+final String initialCount = flowFile.getAttribute(retryAttrName);
+long cnt = 0;
+if(initialCount != null) {
+cnt = Long.parseLong(initialCount);
+}
+
+cnt++;
+
+flowFile = session.putAttribute(flowFile, retryAttrName,
Long.toString(cnt));
--- End diff --
I agree. I've updated the PR. All of those other features related to retry
should be handled under NIFI-3792. It sounds like there is a processor setup,
he's planning to PR it soon.
> allow delayed transfer
> --
>
> Key: NIFI-4805
> URL: https://issues.apache.org/jira/browse/NIFI-4805
> Project: Apache NiFi
> Issue Type: Bug
>
[jira] [Commented] (NIFI-4805) allow delayed transfer
[
https://issues.apache.org/jira/browse/NIFI-4805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16653803#comment-16653803
]
ASF GitHub Bot commented on NIFI-4805:
--
Github user bdesert commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3078#discussion_r226004572
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PenalizeFlowFile.java
---
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"penalty", "penalize", "flowfile"})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Penalizes a FlowFile.")
+@WritesAttributes({
+@WritesAttribute(attribute = "penalization.count.{processor
uuid}", description = "How many times this processor has penalized this
FlowFile.")
+})
+
+public class PenalizeFlowFile extends AbstractProcessor {
+public static final Relationship REL_SUCCESS = new
Relationship.Builder().name("success")
+.description("Successfully penalized FlowFile").build();
+
+private List properties;
+private Set relationships;
+
+@Override
+protected void init(final ProcessorInitializationContext context) {
+final Set relationships = new HashSet<>();
+relationships.add(REL_SUCCESS);
+this.relationships = Collections.unmodifiableSet(relationships);
+}
+@Override
+public Set getRelationships() {
+return relationships;
+}
+
+@Override
+public void onTrigger(ProcessContext context, ProcessSession session)
throws ProcessException {
+FlowFile flowFile = session.get();
+if (flowFile == null) {
+return;
+}
+
+// Track how many times a FlowFile passes through this processor
to better support the Retry use case
+final String retryAttrName = "penalization.count." +
this.getIdentifier();
+final String initialCount = flowFile.getAttribute(retryAttrName);
+long cnt = 0;
+if(initialCount != null) {
+cnt = Long.parseLong(initialCount);
+}
+
+cnt++;
+
+flowFile = session.putAttribute(flowFile, retryAttrName,
Long.toString(cnt));
--- End diff --
as discussed under JIRA, if this processor goes for penalizing only
(without re-try functionality), then I'd remove lines 77-86 at all. retry
capabilities should be implemented then in separate processor.
> allow delayed transfer
> --
>
> Key: NIFI-4805
> URL: https://issues.apache.org/jira/browse/NIFI-4805
> Project: Apache NiFi
> Issue Type: Bug
[jira] [Commented] (NIFI-4805) allow delayed transfer
[ https://issues.apache.org/jira/browse/NIFI-4805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16653790#comment-16653790 ] Ed Berezitsky commented on NIFI-4805: - [~patricker], if the you go with two processors solution - then I would suggest to remove "retryAttrName" from PenalizeFlowFile Processor implementation. Because you indeed do not implement retry. So that was misleading in the code. > allow delayed transfer > -- > > Key: NIFI-4805 > URL: https://issues.apache.org/jira/browse/NIFI-4805 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Martin Mucha >Assignee: Peter Wicks >Priority: Minor > Attachments: retry.xml > > > Nifi has concept of penalization, but this penalization has fixed delay, and > there isn't way how to change it dynamically. > If we want to implement retry flow, where FlowFile flows in loop, we can > either lower performance of Processor via yielding it, or we can do active > waiting. And this is actually recommended as a correct way how to do that. > It seems, that we can easily implement better RetryProcessor, all we missing > is `session.penalize` which accepts `penalizationPeriod`. Processor then can > gradually prolong waiting time after each failure. > > Would it be possible to make such method visible? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4805) allow delayed transfer
[ https://issues.apache.org/jira/browse/NIFI-4805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16653767#comment-16653767 ] Peter Wicks commented on NIFI-4805: --- [~bdesert] I don't disagree that there is a strong use case for a Retry processor. The funny thing is, in my case the FlowFile hasn't even failed... I just really want to delay it before it runs. I think there is a need for two distinct processors. One that lets you penalize any FlowFile, in line with what I've written for the PenalizeFlowFile, and a Retry processor (NIFI-3792?). If nobody minds, I'll link NIFI-3792 to this ticket, and add the Retry processor as well. [~bdesert], I think you've provided a really good list of requirements, and I'll work from that. > allow delayed transfer > -- > > Key: NIFI-4805 > URL: https://issues.apache.org/jira/browse/NIFI-4805 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Martin Mucha >Assignee: Peter Wicks >Priority: Minor > Attachments: retry.xml > > > Nifi has concept of penalization, but this penalization has fixed delay, and > there isn't way how to change it dynamically. > If we want to implement retry flow, where FlowFile flows in loop, we can > either lower performance of Processor via yielding it, or we can do active > waiting. And this is actually recommended as a correct way how to do that. > It seems, that we can easily implement better RetryProcessor, all we missing > is `session.penalize` which accepts `penalizationPeriod`. Processor then can > gradually prolong waiting time after each failure. > > Would it be possible to make such method visible? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4805) allow delayed transfer
[ https://issues.apache.org/jira/browse/NIFI-4805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16653454#comment-16653454 ] Ed Berezitsky commented on NIFI-4805: - Some time ago I have implemented such functionality using scripted processor. I've designed it as following: * attribute for counter with unique ID (similar to Mark's suggestion) * max number of retries (null/empty for infinite) * input property for sequence of times to penalize. For instance, "1,60, 3600" - means: first time wait 1 sec, seconds time - 1 min, third time - 1 hour (and same for each additional retry till max). I didn't want to make it exponential infinite because at some point you don't want to gradually increase the waiting time, it just doesn't make sense. My relationships were defined as following: * retry - continue (aka success) after penalization * expired - continue after max retry count reached (also was dropping counter to 0 to avoid wrapping. This implementation also gave me a simple "PenalizeFlowFile" functionality by simply not looping, or specifying max count to 0 My reasons were to minimize number of processors for a retry flow, and make it simple as much as possible. [~alopresto], [~alfonz], [~markap14], [~patricker], what do you think about this design? (tagged [~patricker] as he implemented initial version, which I think needs to be enhanced to include at least features that have been discussed above by Andy, Martin and Mark). > allow delayed transfer > -- > > Key: NIFI-4805 > URL: https://issues.apache.org/jira/browse/NIFI-4805 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Martin Mucha >Assignee: Peter Wicks >Priority: Minor > Attachments: retry.xml > > > Nifi has concept of penalization, but this penalization has fixed delay, and > there isn't way how to change it dynamically. > If we want to implement retry flow, where FlowFile flows in loop, we can > either lower performance of Processor via yielding it, or we can do active > waiting. And this is actually recommended as a correct way how to do that. > It seems, that we can easily implement better RetryProcessor, all we missing > is `session.penalize` which accepts `penalizationPeriod`. Processor then can > gradually prolong waiting time after each failure. > > Would it be possible to make such method visible? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4805) allow delayed transfer
[ https://issues.apache.org/jira/browse/NIFI-4805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16653438#comment-16653438 ] ASF GitHub Bot commented on NIFI-4805: -- Github user bdesert commented on the issue: https://github.com/apache/nifi/pull/3078 @patricker, this PR doesn't implement all the features requested originally in NIFI-4805, and discussed throughout the thread. Please take a look on comments from Andy, Mark and Martin. Missing features: gradual penalization on re-try, support of EL for defining penalization period > allow delayed transfer > -- > > Key: NIFI-4805 > URL: https://issues.apache.org/jira/browse/NIFI-4805 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Martin Mucha >Assignee: Peter Wicks >Priority: Minor > Attachments: retry.xml > > > Nifi has concept of penalization, but this penalization has fixed delay, and > there isn't way how to change it dynamically. > If we want to implement retry flow, where FlowFile flows in loop, we can > either lower performance of Processor via yielding it, or we can do active > waiting. And this is actually recommended as a correct way how to do that. > It seems, that we can easily implement better RetryProcessor, all we missing > is `session.penalize` which accepts `penalizationPeriod`. Processor then can > gradually prolong waiting time after each failure. > > Would it be possible to make such method visible? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4805) allow delayed transfer
[ https://issues.apache.org/jira/browse/NIFI-4805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16651070#comment-16651070 ] ASF GitHub Bot commented on NIFI-4805: -- GitHub user patricker opened a pull request: https://github.com/apache/nifi/pull/3078 NIFI-4805 Allow Delayed Transfer Went looking for this processor, found a work around, and found that others had already discussed building one throughout 2018 (see ticket/email thread from today, October 15th). Happy to make changes, feedback welcome. ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/patricker/nifi NIFI-4805 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3078.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3078 commit badfe8651651e5cefe1d2447458ff43be7ba0cdc Author: patricker Date: 2018-10-16T02:55:20Z NIFI-4805 Allow Delayed Transfer > allow delayed transfer > -- > > Key: NIFI-4805 > URL: https://issues.apache.org/jira/browse/NIFI-4805 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Martin Mucha >Assignee: Peter Wicks >Priority: Minor > Attachments: retry.xml > > > Nifi has concept of penalization, but this penalization has fixed delay, and > there isn't way how to change it dynamically. > If we want to implement retry flow, where FlowFile flows in loop, we can > either lower performance of Processor via yielding it, or we can do active > waiting. And this is actually recommended as a correct way how to do that. > It seems, that we can easily implement better RetryProcessor, all we missing > is `session.penalize` which accepts `penalizationPeriod`. Processor then can > gradually prolong waiting time after each failure. > > Would it be possible to make such method visible? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4805) allow delayed transfer
[
https://issues.apache.org/jira/browse/NIFI-4805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581050#comment-16581050
]
Mark Payne commented on NIFI-4805:
--
[~alopresto] I agree - as I mentioned in the PR, after I thought about this
more, I concluded that it was a pretty reasonable ask. I do think it should be
named {{Penalize}} or {{PenalizeFlowFile}}, as that's the NiFi terminology that
we use. I would be sure to store that 'counter' as an attribute instead of
attempting to maintain any state in the processor. That can get a bit tricky,
though, because we want to ensure that we could have a Penalize processor at
one point in the flow and another one down the line. So the attribute may need
to be named something like 'penalization.count.\{processor uuid}'. Even then,
though, there are likely some corner cases to consider... just something to
keep in mind.
> allow delayed transfer
> --
>
> Key: NIFI-4805
> URL: https://issues.apache.org/jira/browse/NIFI-4805
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: Martin Mucha
>Priority: Minor
> Attachments: retry.xml
>
>
> Nifi has concept of penalization, but this penalization has fixed delay, and
> there isn't way how to change it dynamically.
> If we want to implement retry flow, where FlowFile flows in loop, we can
> either lower performance of Processor via yielding it, or we can do active
> waiting. And this is actually recommended as a correct way how to do that.
> It seems, that we can easily implement better RetryProcessor, all we missing
> is `session.penalize` which accepts `penalizationPeriod`. Processor then can
> gradually prolong waiting time after each failure.
>
> Would it be possible to make such method visible?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-4805) allow delayed transfer
[
https://issues.apache.org/jira/browse/NIFI-4805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579122#comment-16579122
]
Andy LoPresto commented on NIFI-4805:
-
I almost opened what I think is a duplicate of this today; I was going to
propose a {{Sleep}} processor which takes a configurable property as the delay
time and just waits to release the flowfile. I think it should support
Expression Language and in the context of the penalization, maybe a counter of
the number of times this flowfile has been penalized so that exponential
backoff like Martin described above could work.
> allow delayed transfer
> --
>
> Key: NIFI-4805
> URL: https://issues.apache.org/jira/browse/NIFI-4805
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: Martin Mucha
>Priority: Minor
> Attachments: retry.xml
>
>
> Nifi has concept of penalization, but this penalization has fixed delay, and
> there isn't way how to change it dynamically.
> If we want to implement retry flow, where FlowFile flows in loop, we can
> either lower performance of Processor via yielding it, or we can do active
> waiting. And this is actually recommended as a correct way how to do that.
> It seems, that we can easily implement better RetryProcessor, all we missing
> is `session.penalize` which accepts `penalizationPeriod`. Processor then can
> gradually prolong waiting time after each failure.
>
> Would it be possible to make such method visible?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-4805) allow delayed transfer
[ https://issues.apache.org/jira/browse/NIFI-4805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343956#comment-16343956 ] Martin Mucha commented on NIFI-4805: I understand what you're saying and we can do that. Somewhat. The problem is, that retry 10 seconds is extremely long for first re-connection attempt and extremely short for fifth re-connection attempt. Using tools at hand, we can at most use penalization property as a 'waiting-basis' and multiply it: we will have processor with self loop, where I will loop only each second to get reasonable number of loops. And we have to keep track for each flowfile via nonsensical flowfile attributes(how long it already waited, how long it should wait before next retry) and ... it still waste cpu and it's ugly and laborious processor users considering otherwise trivial task. > allow delayed transfer > -- > > Key: NIFI-4805 > URL: https://issues.apache.org/jira/browse/NIFI-4805 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Martin Mucha >Priority: Minor > Attachments: retry.xml > > > Nifi has concept of penalization, but this penalization has fixed delay, and > there isn't way how to change it dynamically. > If we want to implement retry flow, where FlowFile flows in loop, we can > either lower performance of Processor via yielding it, or we can do active > waiting. And this is actually recommended as a correct way how to do that. > It seems, that we can easily implement better RetryProcessor, all we missing > is `session.penalize` which accepts `penalizationPeriod`. Processor then can > gradually prolong waiting time after each failure. > > Would it be possible to make such method visible? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4805) allow delayed transfer
[ https://issues.apache.org/jira/browse/NIFI-4805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343536#comment-16343536 ] Mark Payne commented on NIFI-4805: -- [~alfonz] - I'm sorry, I don't think I was clear on my concern. I am totally okay with having a PenalizeFlowFile processor (in fact, that's exactly what i would recommend calling it, in order to adhere to the standard naming conventions. Or, perhaps, just Penalize). The only concern that I have is with the the concept of introducing a new method overload to ProcessSession.penalize(). I think we should keep the existing penalize() method that exists and then have the new processor just penalize the FlowFile and route to 'success'. Unless I am missing something, I think this does still give you what you're looking for (the ability to delay processing of the flowfile without introducing artificial delays with yield or 'active waiting'), no? > allow delayed transfer > -- > > Key: NIFI-4805 > URL: https://issues.apache.org/jira/browse/NIFI-4805 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Martin Mucha >Priority: Minor > Attachments: retry.xml > > > Nifi has concept of penalization, but this penalization has fixed delay, and > there isn't way how to change it dynamically. > If we want to implement retry flow, where FlowFile flows in loop, we can > either lower performance of Processor via yielding it, or we can do active > waiting. And this is actually recommended as a correct way how to do that. > It seems, that we can easily implement better RetryProcessor, all we missing > is `session.penalize` which accepts `penalizationPeriod`. Processor then can > gradually prolong waiting time after each failure. > > Would it be possible to make such method visible? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4805) allow delayed transfer
[ https://issues.apache.org/jira/browse/NIFI-4805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341415#comment-16341415 ] Martin Mucha commented on NIFI-4805: I understand that, but retry-loop is valid usecase. There are event 'how-to's on web implemented via just looping without any slowing down (active waiting). Please give me resolution on this. I don't want active waiting loop. I don't want to implement my own scheduler either with persisting FlowFiles until their retry come. IIUC penalization is the simplest correct way, once it exists. Or would you recomment something else? (and if some devel abuses feature, it's his users who should suffer, not other devels/users, no?) > allow delayed transfer > -- > > Key: NIFI-4805 > URL: https://issues.apache.org/jira/browse/NIFI-4805 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Martin Mucha >Priority: Minor > Attachments: retry.xml > > > Nifi has concept of penalization, but this penalization has fixed delay, and > there isn't way how to change it dynamically. > If we want to implement retry flow, where FlowFile flows in loop, we can > either lower performance of Processor via yielding it, or we can do active > waiting. And this is actually recommended as a correct way how to do that. > It seems, that we can easily implement better RetryProcessor, all we missing > is `session.penalize` which accepts `penalizationPeriod`. Processor then can > gradually prolong waiting time after each failure. > > Would it be possible to make such method visible? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4805) allow delayed transfer
[ https://issues.apache.org/jira/browse/NIFI-4805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341218#comment-16341218 ] Mark Payne commented on NIFI-4805: -- [~alfonz] the delay that is imposed by the session penalization is controlled in the Processor's Settings configuration dialog. We did used to have a session.penalize method that allowed the developer to indicate a penalization period. As a result, developers typically would expose this configuration via a processor property. Or sometimes multiple properties - "Penalization Period for ABC", "Penalization Period for XYZ", "Penalization Period for This Other Thing". This ended up becoming confusing for users, because it was just more complicated to configure this, and every processor configured this differently. As a result, it was eventually removed all together and placed just on the Settings pane for the processor. > allow delayed transfer > -- > > Key: NIFI-4805 > URL: https://issues.apache.org/jira/browse/NIFI-4805 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Martin Mucha >Priority: Minor > Attachments: retry.xml > > > Nifi has concept of penalization, but this penalization has fixed delay, and > there isn't way how to change it dynamically. > If we want to implement retry flow, where FlowFile flows in loop, we can > either lower performance of Processor via yielding it, or we can do active > waiting. And this is actually recommended as a correct way how to do that. > It seems, that we can easily implement better RetryProcessor, all we missing > is `session.penalize` which accepts `penalizationPeriod`. Processor then can > gradually prolong waiting time after each failure. > > Would it be possible to make such method visible? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
