Re: [PR] NIFI-12918 Fix Stateless NullPointerException on versioned sub-process groups - main branch [nifi]
exceptionfactory commented on code in PR #8536:
URL: https://github.com/apache/nifi/pull/8536#discussion_r1550854635
##
nifi-stateless/nifi-stateless-bundle/nifi-stateless-engine/src/test/java/org/apache/nifi/stateless/core/TestRegistryUtil.java:
##
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.stateless.core;
+
+import org.apache.nifi.flow.VersionedFlowCoordinates;
+import org.apache.nifi.flow.VersionedProcessGroup;
+import org.apache.nifi.registry.client.FlowSnapshotClient;
+import org.apache.nifi.registry.client.NiFiRegistryClient;
+import org.apache.nifi.registry.client.NiFiRegistryException;
+import org.apache.nifi.registry.flow.VersionedFlowSnapshot;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.UUID;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestRegistryUtil {
+private static final String registryUrl = "http://localhost:18080;;
+private static final String rootBucketId = UUID.randomUUID().toString();
+private static final String rootFlowId = UUID.randomUUID().toString();
+private static final int rootVersion = 1;
+
+private static final String childBucketId = UUID.randomUUID().toString();
+private static final String childFlowId = UUID.randomUUID().toString();
+private static final int childVersion = 1;
+
+@Test
+public void testRegistryUrlCreation() throws NiFiRegistryException,
IOException {
+NiFiRegistryClient registryClient = mock(NiFiRegistryClient.class);
+FlowSnapshotClient flowSnapshotClient = mock(FlowSnapshotClient.class);
+
+RegistryUtil registryUtil = new RegistryUtil(registryClient,
registryUrl, null);
+
+
when(registryClient.getFlowSnapshotClient()).thenReturn(flowSnapshotClient);
+when(flowSnapshotClient.get(rootBucketId, rootFlowId,
rootVersion)).thenAnswer(
+invocation -> buildVfs());
+when(flowSnapshotClient.get(childBucketId, childFlowId,
childVersion)).thenAnswer(
+invocation -> buildVfs());
+
+VersionedFlowSnapshot snapshot =
registryUtil.getFlowContents(rootBucketId, rootFlowId, rootVersion, true, null);
+VersionedFlowCoordinates coordinates =
snapshot.getFlowContents().getVersionedFlowCoordinates();
+
+String formattedUrl =
registryUtil.getBaseRegistryUrl(coordinates.getStorageLocation());
+assertEquals(formattedUrl, registryUrl);
+}
+
+private VersionedFlowSnapshot buildVfs(){
+VersionedFlowSnapshot vfs = new VersionedFlowSnapshot();
+VersionedProcessGroup vpg1 = new VersionedProcessGroup();
+VersionedProcessGroup vpg2 = new VersionedProcessGroup();
+VersionedFlowCoordinates vfc1 = new VersionedFlowCoordinates();
+VersionedFlowCoordinates vfc2 = new VersionedFlowCoordinates();
+
+String storageLocation1 =
String.format(registryUrl+"/nifi-registry-api/buckets/%s/flows/%s/versions/%s",
rootBucketId, rootFlowId, rootVersion);
Review Comment:
The `registryUrl` parameter should be moved to a format element:
```suggestion
String storageLocation1 =
String.format("%s/nifi-registry-api/buckets/%s/flows/%s/versions/%s",
registryUrl, rootBucketId, rootFlowId, rootVersion);
```
##
nifi-stateless/nifi-stateless-bundle/nifi-stateless-engine/src/main/java/org/apache/nifi/stateless/core/RegistryUtil.java:
##
@@ -119,6 +126,10 @@ public VersionedFlowSnapshot getFlowContents(final String
bucketId, final String
return flowSnapshot;
}
+public String getBaseRegistryUrl(String storageLocation) throws
MalformedURLException {
+URL url = new URL(storageLocation);
Review Comment:
Recommend using `URI.create()` instead of `new URL()` to avoid the checked
`MalformedURLException`
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/flow/synchronization/StandardVersionedComponentSynchronizer.java:
##
@@ -509,6 +509,8 @@ private String
Re: [PR] NIFI-12982 Extend test suite of MockProcessSession [nifi]
EndzeitBegins commented on PR #8589: URL: https://github.com/apache/nifi/pull/8589#issuecomment-2036148323 You're welcome. Thank you for the review and merge, @exceptionfactory. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12970: Generate documentation for Python processors [nifi]
exceptionfactory commented on code in PR #8579:
URL: https://github.com/apache/nifi/pull/8579#discussion_r1550834981
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-documentation/src/main/java/org/apache/nifi/documentation/html/AbstractHtmlDocumentationWriter.java:
##
@@ -0,0 +1,356 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.documentation.html;
+
+import org.apache.nifi.documentation.DocumentationWriter;
+import org.apache.nifi.util.StringUtils;
+
+import javax.xml.stream.FactoryConfigurationError;
+import javax.xml.stream.XMLOutputFactory;
+import javax.xml.stream.XMLStreamException;
+import javax.xml.stream.XMLStreamWriter;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+abstract class AbstractHtmlDocumentationWriter implements
DocumentationWriter {
+
+/**
+ * The filename where additional user specified information may be stored.
+ */
+public static final String ADDITIONAL_DETAILS_HTML =
"additionalDetails.html";
+
+static final String NO_DESCRIPTION = "No description provided.";
+static final String NO_TAGS = "No tags provided.";
+static final String NO_PROPERTIES = "This component has no required or
optional properties.";
+
+static final String H2 = "h2";
+static final String H3 = "h3";
+static final String H4 = "h4";
+static final String P = "p";
+static final String BR = "br";
+static final String SPAN = "span";
+static final String STRONG = "strong";
+static final String TABLE = "table";
+static final String TH = "th";
+static final String TR = "tr";
+static final String TD = "td";
+static final String UL = "ul";
+static final String LI = "li";
+static final String ID = "id";
+
+@Override
+public void write(final T component, final OutputStream streamToWriteTo,
final boolean includesAdditionalDocumentation) throws IOException {
+try {
+XMLStreamWriter xmlStreamWriter =
XMLOutputFactory.newInstance().createXMLStreamWriter(streamToWriteTo, "UTF-8");
+xmlStreamWriter.writeDTD("");
+xmlStreamWriter.writeStartElement("html");
+xmlStreamWriter.writeAttribute("lang", "en");
+writeHead(component, xmlStreamWriter);
+writeBody(component, xmlStreamWriter,
includesAdditionalDocumentation);
+xmlStreamWriter.writeEndElement();
+xmlStreamWriter.close();
+} catch (XMLStreamException | FactoryConfigurationError e) {
+throw new IOException("Unable to create XMLOutputStream", e);
+}
+}
+
+/**
+ * Writes the head portion of the HTML documentation.
+ *
+ * @param component the component to describe
+ * @param xmlStreamWriter the stream to write to
+ * @throws XMLStreamException thrown if there was a problem writing to the
stream
+ */
+protected void writeHead(final T component, final XMLStreamWriter
xmlStreamWriter) throws XMLStreamException {
+xmlStreamWriter.writeStartElement("head");
+xmlStreamWriter.writeStartElement("meta");
+xmlStreamWriter.writeAttribute("charset", "utf-8");
+xmlStreamWriter.writeEndElement();
+writeSimpleElement(xmlStreamWriter, "title", getTitle(component));
+
+xmlStreamWriter.writeStartElement("link");
+xmlStreamWriter.writeAttribute("rel", "stylesheet");
+xmlStreamWriter.writeAttribute("href",
"../../../../../css/component-usage.css");
+xmlStreamWriter.writeAttribute("type", "text/css");
+xmlStreamWriter.writeEndElement();
+xmlStreamWriter.writeEndElement();
+
+xmlStreamWriter.writeStartElement("script");
+xmlStreamWriter.writeAttribute("type", "text/javascript");
+xmlStreamWriter.writeCharacters("window.onload =
function(){if(self==top) { " +
+"document.getElementById('nameHeader').style.display =
\"inherit\"; } }");
+xmlStreamWriter.writeEndElement();
+}
+
+/**
+ * Gets the class name of the component.
+ *
+ * @param component the component to describe
+ *
[PR] Revert "Bump webpack-dev-middleware and karma-webpack (#8547)" [nifi]
sardell opened a new pull request, #8603: URL: https://github.com/apache/nifi/pull/8603 This PR reverts commit 69f1a3be4dd0ede7912ee8cfd67d39bdc3d6eca0 (PR #8547). This commit on `main` is currently causing local builds of NiFi to fail due to a dependency in nifi-registry's package-lock.json requiring a version of Node.js higher than the one we use to build nifi-registry. https://github.com/apache/nifi/assets/7304869/712a8326-4751-4bd1-b383-81cfa968f3f3;> https://github.com/apache/nifi/assets/7304869/83eb590a-fab1-400e-829b-a92462d08867;> # Summary [NIFI-0](https://issues.apache.org/jira/browse/NIFI-0) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [ ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [ ] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [ ] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [ ] Pull Request based on current revision of the `main` branch - [ ] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 21 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-12875) Add Response Handling Strategy to RestLookupService
[ https://issues.apache.org/jira/browse/NIFI-12875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-12875: Fix Version/s: 2.0.0-M3 Assignee: Gregory M. Foreman Resolution: Fixed Status: Resolved (was: Patch Available) > Add Response Handling Strategy to RestLookupService > --- > > Key: NIFI-12875 > URL: https://issues.apache.org/jira/browse/NIFI-12875 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 2.0.0-M2 >Reporter: Gregory M. Foreman >Assignee: Gregory M. Foreman >Priority: Major > Fix For: 2.0.0-M3 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > The RestLookupService service does not handle HTTP error codes correctly. If > the web service being consumed returns a 4xx or 5xx error code, the > RestLookupService service ignores the error and passes the response content > to the record reader, causing unexpected exception reporting. > Example: a web service that communicates with clients via JSON is behind a > proxy. The proxy returns a 502 error and responds with HTML content > describing the error. The RestLookupService uses a JsonTreeReader reader and > attempts to parse the HTML characters. This leads to it throwing an > exception that it was unable to parse the '<' character. The real reason is > the proxy is sending back a 502 error, but this detail is hidden. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-7851) RestLookupService does not throw a LookupFailureException when the rest endpoint response-code is not successful.
[ https://issues.apache.org/jira/browse/NIFI-7851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann resolved NIFI-7851. Fix Version/s: 2.0.0-M3 Resolution: Fixed > RestLookupService does not throw a LookupFailureException when the rest > endpoint response-code is not successful. > - > > Key: NIFI-7851 > URL: https://issues.apache.org/jira/browse/NIFI-7851 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Esteban Capoccetti >Assignee: Nadeem >Priority: Major > Fix For: 2.0.0-M3 > > > RestLookupService does not throw a LookupFailureException when the rest > endpoint response-code is not successful. > Lookup Processor using a "Lookup Service" of type "RestLookupService" do not > send the flowFile into a Failure Connections when the rest service > http-response-code is not a successful one; it sends those flowfiles to the > "success/matched" connections instead. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-7851) RestLookupService does not throw a LookupFailureException when the rest endpoint response-code is not successful.
[ https://issues.apache.org/jira/browse/NIFI-7851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-7851: --- Issue Type: Improvement (was: Bug) > RestLookupService does not throw a LookupFailureException when the rest > endpoint response-code is not successful. > - > > Key: NIFI-7851 > URL: https://issues.apache.org/jira/browse/NIFI-7851 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Esteban Capoccetti >Assignee: Nadeem >Priority: Major > > RestLookupService does not throw a LookupFailureException when the rest > endpoint response-code is not successful. > Lookup Processor using a "Lookup Service" of type "RestLookupService" do not > send the flowFile into a Failure Connections when the rest service > http-response-code is not a successful one; it sends those flowfiles to the > "success/matched" connections instead. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12875) Add Response Handling Strategy to RestLookupService
[ https://issues.apache.org/jira/browse/NIFI-12875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-12875: Summary: Add Response Handling Strategy to RestLookupService (was: RestLookupService not handling HTTP 4xx/5xx error codes) > Add Response Handling Strategy to RestLookupService > --- > > Key: NIFI-12875 > URL: https://issues.apache.org/jira/browse/NIFI-12875 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 2.0.0-M2 >Reporter: Gregory M. Foreman >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > > The RestLookupService service does not handle HTTP error codes correctly. If > the web service being consumed returns a 4xx or 5xx error code, the > RestLookupService service ignores the error and passes the response content > to the record reader, causing unexpected exception reporting. > Example: a web service that communicates with clients via JSON is behind a > proxy. The proxy returns a 502 error and responds with HTML content > describing the error. The RestLookupService uses a JsonTreeReader reader and > attempts to parse the HTML characters. This leads to it throwing an > exception that it was unable to parse the '<' character. The real reason is > the proxy is sending back a 502 error, but this detail is hidden. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12875) RestLookupService not handling HTTP 4xx/5xx error codes
[ https://issues.apache.org/jira/browse/NIFI-12875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-12875: Issue Type: Improvement (was: Bug) > RestLookupService not handling HTTP 4xx/5xx error codes > --- > > Key: NIFI-12875 > URL: https://issues.apache.org/jira/browse/NIFI-12875 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 2.0.0-M2 >Reporter: Gregory M. Foreman >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > > The RestLookupService service does not handle HTTP error codes correctly. If > the web service being consumed returns a 4xx or 5xx error code, the > RestLookupService service ignores the error and passes the response content > to the record reader, causing unexpected exception reporting. > Example: a web service that communicates with clients via JSON is behind a > proxy. The proxy returns a 502 error and responds with HTML content > describing the error. The RestLookupService uses a JsonTreeReader reader and > attempts to parse the HTML characters. This leads to it throwing an > exception that it was unable to parse the '<' character. The real reason is > the proxy is sending back a 502 error, but this detail is hidden. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12875) RestLookupService not handling HTTP 4xx/5xx error codes
[ https://issues.apache.org/jira/browse/NIFI-12875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17833805#comment-17833805 ] ASF subversion and git services commented on NIFI-12875: Commit 84df025ccfe68bf9eefc0a43ad14fdc727e3723d in nifi's branch refs/heads/main from Gregory M. Foreman [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=84df025ccf ] NIFI-12875 Added Response Handling Strategy to RestLookupService This closes #8484 Signed-off-by: David Handermann > RestLookupService not handling HTTP 4xx/5xx error codes > --- > > Key: NIFI-12875 > URL: https://issues.apache.org/jira/browse/NIFI-12875 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 2.0.0-M2 >Reporter: Gregory M. Foreman >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > > The RestLookupService service does not handle HTTP error codes correctly. If > the web service being consumed returns a 4xx or 5xx error code, the > RestLookupService service ignores the error and passes the response content > to the record reader, causing unexpected exception reporting. > Example: a web service that communicates with clients via JSON is behind a > proxy. The proxy returns a 502 error and responds with HTML content > describing the error. The RestLookupService uses a JsonTreeReader reader and > attempts to parse the HTML characters. This leads to it throwing an > exception that it was unable to parse the '<' character. The real reason is > the proxy is sending back a 502 error, but this detail is hidden. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12875: Add unsuccessful HTTP status code handling logic [nifi]
exceptionfactory closed pull request #8484: NIFI-12875: Add unsuccessful HTTP status code handling logic URL: https://github.com/apache/nifi/pull/8484 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-12982) Extend test suite of MockProcessSession
[
https://issues.apache.org/jira/browse/NIFI-12982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17833801#comment-17833801
]
ASF subversion and git services commented on NIFI-12982:
Commit 6939ffc08d6a6d6ce6a932637fde8523fb2811cd in nifi's branch
refs/heads/main from EndzeitBegins
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=6939ffc08d ]
NIFI-12982 Extend test suite of MockProcessSession
This closes #8589
Signed-off-by: David Handermann
> Extend test suite of MockProcessSession
> ---
>
> Key: NIFI-12982
> URL: https://issues.apache.org/jira/browse/NIFI-12982
> Project: Apache NiFi
> Issue Type: Sub-task
>Reporter: endzeit
>Assignee: endzeit
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> As part of NIFI-12971 (or a subtask thereof) most likely changes will be
> introduced to both {{StandardProcessorTestRunner}} and {{MockProcessSession}}.
>
> To reduce the risk of regressions introduced by those changes, the test
> suite of {{MockProcessSession}} should be extended to account for more
> scenarios, regarding both expected successful and failing interactions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (NIFI-12982) Extend test suite of MockProcessSession
[
https://issues.apache.org/jira/browse/NIFI-12982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann resolved NIFI-12982.
-
Fix Version/s: 2.0.0-M3
Resolution: Fixed
> Extend test suite of MockProcessSession
> ---
>
> Key: NIFI-12982
> URL: https://issues.apache.org/jira/browse/NIFI-12982
> Project: Apache NiFi
> Issue Type: Sub-task
>Reporter: endzeit
>Assignee: endzeit
>Priority: Major
> Fix For: 2.0.0-M3
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> As part of NIFI-12971 (or a subtask thereof) most likely changes will be
> introduced to both {{StandardProcessorTestRunner}} and {{MockProcessSession}}.
>
> To reduce the risk of regressions introduced by those changes, the test
> suite of {{MockProcessSession}} should be extended to account for more
> scenarios, regarding both expected successful and failing interactions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] NIFI-12982 Extend test suite of MockProcessSession [nifi]
exceptionfactory closed pull request #8589: NIFI-12982 Extend test suite of MockProcessSession URL: https://github.com/apache/nifi/pull/8589 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12982 Extend test suite of MockProcessSession [nifi]
exceptionfactory commented on code in PR #8589:
URL: https://github.com/apache/nifi/pull/8589#discussion_r1550787557
##
nifi-mock/src/test/java/org/apache/nifi/util/TestMockProcessSession.java:
##
@@ -45,189 +52,588 @@
public class TestMockProcessSession {
-@Test
-public void testReadWithoutCloseThrowsExceptionOnCommit() throws
IOException {
-final Processor processor = new PoorlyBehavedProcessor();
-final MockProcessSession session = new MockProcessSession(new
SharedSessionState(processor, new AtomicLong(0L)), processor, true, new
MockStateManager(processor), true);
-FlowFile flowFile = session.createFlowFile("hello, world".getBytes());
-final InputStream in = session.read(flowFile);
-final byte[] buffer = new byte[12];
-fillBuffer(in, buffer);
+private final Processor processor = new TestProcessor();
+private final SharedSessionState sharedState = new
SharedSessionState(processor, new AtomicLong(0L));
+private final MockStateManager stateManager = new
MockStateManager(processor);
+private final MockProcessSession session = new
MockProcessSession(sharedState, processor, stateManager);
+
+private final Processor statefulProcessor = new StatefulTestProcessor();
+private final SharedSessionState sharedStateOfStatefulProcessor = new
SharedSessionState(statefulProcessor, new AtomicLong(0L));
+private final MockStateManager stateManagerOfStatefulProcessor = new
MockStateManager(statefulProcessor);
+private final MockProcessSession sessionOfStatefulProcessor = new
MockProcessSession(sharedStateOfStatefulProcessor, statefulProcessor,
stateManagerOfStatefulProcessor);
Review Comment:
Thanks for the helpful reply @EndzeitBegins, very good point regarding the
standard per-method behavior. In light of this, and the fact that there are no
Mockito-based mock objects, the current approach makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-12959: Support loading python processors from NARs [nifi]
exceptionfactory commented on code in PR #8573:
URL: https://github.com/apache/nifi/pull/8573#discussion_r1550777551
##
nifi-nar-bundles/nifi-py4j-bundle/nifi-py4j-bridge/src/main/java/org/apache/nifi/py4j/PythonProcess.java:
##
@@ -224,6 +226,10 @@ private String generateAuthToken() {
return Base64.getEncoder().encodeToString(bytes);
}
+private boolean isUseVirtualEnv() {
Review Comment:
It looks like the only use of this method is a negated reference. Can this
be removed and replaced with `if (packagedWithDependencies)`?
##
nifi-nar-bundles/nifi-py4j-bundle/nifi-py4j-bridge/src/main/java/org/apache/nifi/py4j/PythonProcess.java:
##
@@ -234,8 +240,18 @@ private Process launchPythonProcess(final int
listeningPort, final String authTo
final List commands = new ArrayList<>();
commands.add(pythonCommand);
+if (!isUseVirtualEnv()) {
+// If not using venv, we will not launch a separate virtual
environment, so we need to use the -S
+// flag in order to prevent the Python process from using the
installation's site-packages. This provides
+// proper dependency isolation to the Python process.
+commands.add("-S");
+}
String pythonPath = pythonApiDirectory.getAbsolutePath();
+final String absolutePath = virtualEnvHome.getAbsolutePath();
+final File dependenciesDir = new File(new File(absolutePath),
"NAR-INF/bundled-dependencies");
Review Comment:
Is the intent to add `NAR-INF/bundled-dependencies` in all cases? It seems
like it should be conditional on whether the Processor is packaged with
dependencies.
##
nifi-nar-bundles/nifi-py4j-bundle/nifi-py4j-bridge/src/main/java/org/apache/nifi/py4j/PythonProcess.java:
##
@@ -287,6 +309,13 @@ String resolvePythonCommand() throws IOException {
private void setupEnvironment() throws IOException {
+// Environment creation is only necessary if using PIP. Otherwise, the
Process requires no outside dependencies, other than those
+// provided in the package and thus we can simply include those
packages in the PYTHON_PATH.
+if (!isUseVirtualEnv()) {
+logger.debug("Will not create Python Virtual Environment because
PIP is disabled in nifi.properties");
Review Comment:
Very minor, but general usage of `pip` is all lowercase and opposed to all
uppercase. More importantly though, the message does not reflect the behavior,
as it is only based on the the presence of packaged dependencies.
```suggestion
logger.debug("Python Virtual Environment not created: Python
Processor packaged with dependencies");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Resolved] (NIFI-12996) nifi-shared-bom construct broke classloading for optional dependencies of common-compress including zstd
[
https://issues.apache.org/jira/browse/NIFI-12996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann resolved NIFI-12996.
-
Resolution: Fixed
> nifi-shared-bom construct broke classloading for optional dependencies of
> common-compress including zstd
>
>
> Key: NIFI-12996
> URL: https://issues.apache.org/jira/browse/NIFI-12996
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.24.0, 1.25.0, 2.0.0-M2
>Reporter: Joe Witt
>Assignee: Joe Witt
>Priority: Critical
> Fix For: 2.0.0-M3, 1.26.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> In Apache Slack a user reported this was broken for them in NiFi 1.25.0 but
> used to work in 1.19.1.
> On apache 2.0 m2+latest I can reproduce this by simply trying to use it at
> all. First question is do we not have unit tests but anyway this needs to be
> evaluated as to why it isn't working.
> {noformat}
> 2024-04-02 13:30:12,514 ERROR [Timer-Driven Process Thread-3]
> o.a.n.p.standard.CompressContent
> CompressContent[id=a07f7ae5-018e-1000-c82c-2e2143f9c320] Processing halted:
> yielding [1 sec]
> java.lang.NoClassDefFoundError: com/github/luben/zstd/ZstdOutputStream
> at
> org.apache.commons.compress.compressors.zstandard.ZstdCompressorOutputStream.(ZstdCompressorOutputStream.java:57)
> at
> org.apache.nifi.processors.standard.CompressContent$1.process(CompressContent.java:353)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:3426)
> at
> org.apache.nifi.processors.standard.CompressContent.onTrigger(CompressContent.java:300)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1274)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:244)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:102)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
> at
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> Caused by: java.lang.ClassNotFoundException:
> com.github.luben.zstd.ZstdOutputStream
> at
> java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:593)
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)
> ... 15 common frames omitted
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-12996) nifi-shared-bom construct broke classloading for optional dependencies of common-compress including zstd
[
https://issues.apache.org/jira/browse/NIFI-12996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17833794#comment-17833794
]
ASF subversion and git services commented on NIFI-12996:
Commit d3ea429d25db6f339ddd0701d2bb2616f054eed6 in nifi's branch
refs/heads/support/nifi-1.x from Joe Witt
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=d3ea429d25 ]
NIFI-12996 Moved zstd-jni to standard-shared-nar
- zstd-jni must be present in the same NAR as commons-compress
This closes #8598
Signed-off-by: David Handermann
(cherry picked from commit 98c4061cfee78590a7d391e53a835c5b87ca4bed)
> nifi-shared-bom construct broke classloading for optional dependencies of
> common-compress including zstd
>
>
> Key: NIFI-12996
> URL: https://issues.apache.org/jira/browse/NIFI-12996
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.24.0, 1.25.0, 2.0.0-M2
>Reporter: Joe Witt
>Assignee: Joe Witt
>Priority: Critical
> Fix For: 2.0.0-M3, 1.26.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> In Apache Slack a user reported this was broken for them in NiFi 1.25.0 but
> used to work in 1.19.1.
> On apache 2.0 m2+latest I can reproduce this by simply trying to use it at
> all. First question is do we not have unit tests but anyway this needs to be
> evaluated as to why it isn't working.
> {noformat}
> 2024-04-02 13:30:12,514 ERROR [Timer-Driven Process Thread-3]
> o.a.n.p.standard.CompressContent
> CompressContent[id=a07f7ae5-018e-1000-c82c-2e2143f9c320] Processing halted:
> yielding [1 sec]
> java.lang.NoClassDefFoundError: com/github/luben/zstd/ZstdOutputStream
> at
> org.apache.commons.compress.compressors.zstandard.ZstdCompressorOutputStream.(ZstdCompressorOutputStream.java:57)
> at
> org.apache.nifi.processors.standard.CompressContent$1.process(CompressContent.java:353)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:3426)
> at
> org.apache.nifi.processors.standard.CompressContent.onTrigger(CompressContent.java:300)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1274)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:244)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:102)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
> at
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> Caused by: java.lang.ClassNotFoundException:
> com.github.luben.zstd.ZstdOutputStream
> at
> java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:593)
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)
> ... 15 common frames omitted
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] WIP NIFI-12996 removing commons-compress from shared nar/bom and ensuring… [nifi]
exceptionfactory closed pull request #8598: WIP NIFI-12996 removing commons-compress from shared nar/bom and ensuring… URL: https://github.com/apache/nifi/pull/8598 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-12996) nifi-shared-bom construct broke classloading for optional dependencies of common-compress including zstd
[
https://issues.apache.org/jira/browse/NIFI-12996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17833793#comment-17833793
]
ASF subversion and git services commented on NIFI-12996:
Commit 98c4061cfee78590a7d391e53a835c5b87ca4bed in nifi's branch
refs/heads/main from Joe Witt
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=98c4061cfe ]
NIFI-12996 Moved zstd-jni to standard-shared-nar
- zstd-jni must be present in the same NAR as commons-compress
This closes #8598
Signed-off-by: David Handermann
> nifi-shared-bom construct broke classloading for optional dependencies of
> common-compress including zstd
>
>
> Key: NIFI-12996
> URL: https://issues.apache.org/jira/browse/NIFI-12996
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.24.0, 1.25.0, 2.0.0-M2
>Reporter: Joe Witt
>Assignee: Joe Witt
>Priority: Critical
> Fix For: 2.0.0-M3, 1.26.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> In Apache Slack a user reported this was broken for them in NiFi 1.25.0 but
> used to work in 1.19.1.
> On apache 2.0 m2+latest I can reproduce this by simply trying to use it at
> all. First question is do we not have unit tests but anyway this needs to be
> evaluated as to why it isn't working.
> {noformat}
> 2024-04-02 13:30:12,514 ERROR [Timer-Driven Process Thread-3]
> o.a.n.p.standard.CompressContent
> CompressContent[id=a07f7ae5-018e-1000-c82c-2e2143f9c320] Processing halted:
> yielding [1 sec]
> java.lang.NoClassDefFoundError: com/github/luben/zstd/ZstdOutputStream
> at
> org.apache.commons.compress.compressors.zstandard.ZstdCompressorOutputStream.(ZstdCompressorOutputStream.java:57)
> at
> org.apache.nifi.processors.standard.CompressContent$1.process(CompressContent.java:353)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:3426)
> at
> org.apache.nifi.processors.standard.CompressContent.onTrigger(CompressContent.java:300)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1274)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:244)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:102)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
> at
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> Caused by: java.lang.ClassNotFoundException:
> com.github.luben.zstd.ZstdOutputStream
> at
> java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:593)
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)
> ... 15 common frames omitted
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12998) maxmind geoip2 does not belong in nifi-nar-bundle pom
[ https://issues.apache.org/jira/browse/NIFI-12998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Witt updated NIFI-12998: Fix Version/s: (was: 2.0.0-M3) > maxmind geoip2 does not belong in nifi-nar-bundle pom > - > > Key: NIFI-12998 > URL: https://issues.apache.org/jira/browse/NIFI-12998 > Project: Apache NiFi > Issue Type: Task >Reporter: Joe Witt >Assignee: Joe Witt >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > found in nifi-nar-bundles pom > > com.maxmind.geoip2 > geoip2 > 4.2.0 > > This should not be here. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12998) maxmind geoip2 does not belong in nifi-nar-bundle pom
[ https://issues.apache.org/jira/browse/NIFI-12998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Witt updated NIFI-12998: Affects Version/s: 2.0.0-M2 > maxmind geoip2 does not belong in nifi-nar-bundle pom > - > > Key: NIFI-12998 > URL: https://issues.apache.org/jira/browse/NIFI-12998 > Project: Apache NiFi > Issue Type: Task >Affects Versions: 2.0.0-M2 >Reporter: Joe Witt >Assignee: Joe Witt >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > found in nifi-nar-bundles pom > > com.maxmind.geoip2 > geoip2 > 4.2.0 > > This should not be here. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12998) maxmind geoip2 does not belong in nifi-nar-bundle pom
[ https://issues.apache.org/jira/browse/NIFI-12998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Witt updated NIFI-12998: Status: Patch Available (was: Open) > maxmind geoip2 does not belong in nifi-nar-bundle pom > - > > Key: NIFI-12998 > URL: https://issues.apache.org/jira/browse/NIFI-12998 > Project: Apache NiFi > Issue Type: Task >Reporter: Joe Witt >Assignee: Joe Witt >Priority: Major > Fix For: 2.0.0-M3 > > Time Spent: 10m > Remaining Estimate: 0h > > found in nifi-nar-bundles pom > > com.maxmind.geoip2 > geoip2 > 4.2.0 > > This should not be here. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] NIFI-12998 clean up version reference [nifi]
joewitt opened a new pull request, #8602: URL: https://github.com/apache/nifi/pull/8602 # Summary [NIFI-0](https://issues.apache.org/jira/browse/NIFI-0) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [ ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [ ] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [ ] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [ ] Pull Request based on current revision of the `main` branch - [ ] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 21 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-12969) Under heavy load, nifi node unable to rejoin cluster, graph modified with temp funnel
[
https://issues.apache.org/jira/browse/NIFI-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe Witt updated NIFI-12969:
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Under heavy load, nifi node unable to rejoin cluster, graph modified with
> temp funnel
> -
>
> Key: NIFI-12969
> URL: https://issues.apache.org/jira/browse/NIFI-12969
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.24.0, 2.0.0-M2
>Reporter: Nissim Shiman
>Assignee: Mark Payne
>Priority: Critical
> Fix For: 2.0.0-M3, 1.26.0
>
> Attachments: nifi-app.log, simple_flow.png,
> simple_flow_with_temp-funnel.png
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Under heavy load, if a node leaves the cluster (due to heartbeat time out),
> many times it is unable to rejoin the cluster.
> The nodes' graph will have been modified with a temp-funnel as well.
> Appears to be some sort of [timing
> issue|https://github.com/apache/nifi/blob/rel/nifi-2.0.0-M2/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/connectable/StandardConnection.java#L298]
> # To reproduce, on a nifi cluster of three nodes, set up:
> 2 GenerateFlowFile processors -> PG
> Inside PG:
> inputPort -> UpdateAttribute
> # Keep all defaults except for the following:
> For UpdateAttribute terminate the success relationship
> One of the GenerateFlowFile processors can be disabled,
> the other one should have Run Schedule to be 0 min (this will allow for the
> heavy load)
> # In nifi.properties (on all 3 nodes) to allow for nodes to fall out of the
> cluster, set: nifi.cluster.protocol.heartbeat.interval=2 sec (default is 5)
> nifi.cluster.protocol.heartbeat.missable.max=1 (default is 8)
> Restart nifi. Start flow. The nodes will quickly fall out and rejoin cluster.
> After a few minutes one will likely not be able to rejoin. The graph for
> that node will have the disabled GenerateFlowFile now pointing to a funnel (a
> temp-funnel) instead of the PG
> Stack trace on that nodes nifi-app.log will look like this: (this is from
> 2.0.0-M2):
> {code:java}
> 2024-03-28 13:55:19,395 INFO [Reconnect to Cluster]
> o.a.nifi.controller.StandardFlowService Node disconnected due to Failed to
> properly handle Reconnection request due to org.apache.nifi.control
> ler.serialization.FlowSynchronizationException: Failed to connect node to
> cluster because local flow controller partially updated. Administrator should
> disconnect node and review flow for corrup
> tion.
> 2024-03-28 13:55:19,395 ERROR [Reconnect to Cluster]
> o.a.nifi.controller.StandardFlowService Handling reconnection request failed
> due to: org.apache.nifi.controller.serialization.FlowSynchroniza
> tionException: Failed to connect node to cluster because local flow
> controller partially updated. Administrator should disconnect node and review
> flow for corruption.
> org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed
> to connect node to cluster because local flow controller partially updated.
> Administrator should disconnect node and
> review flow for corruption.
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:985)
> at
> org.apache.nifi.controller.StandardFlowService.handleReconnectionRequest(StandardFlowService.java:655)
> at
> org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:384)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> Caused by:
> org.apache.nifi.controller.serialization.FlowSynchronizationException:
> java.lang.IllegalStateException: Cannot change destination of Connection
> because FlowFiles from this Connection
> are currently held by LocalPort[id=99213c00-78ca-4848-112f-5454cc20656b,
> type=INPUT_PORT, name=inputPort, group=innerPG]
> at
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:472)
> at
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.sync(VersionedFlowSynchronizer.java:223)
> at
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1740)
> at
> org.apache.nifi.persistence.StandardFlowConfigurationDAO.load(StandardFlowConfigurationDAO.java:91)
> at
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:805)
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:954)
> ... 3 common frames omitted
> Caused by: java.lang.IllegalStateException: Cannot change destination of
> Connection because FlowFiles from
[jira] [Commented] (NIFI-12969) Under heavy load, nifi node unable to rejoin cluster, graph modified with temp funnel
[
https://issues.apache.org/jira/browse/NIFI-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17833780#comment-17833780
]
ASF subversion and git services commented on NIFI-12969:
Commit 212763dd79b3652c9870ff6a59db28c232113c55 in nifi's branch
refs/heads/support/nifi-1.x from Mark Payne
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=212763dd79 ]
NIFI-12969: Fixed a typo in the #isTempDestinationNecessary method, where we
were comparing existingConnection.getProcessGroup() to
newDestination.getProcessGroup(), instead of comparing
existingConnection.getDestination().getProcessGroup() to
newDestination.getProcessGroup(). I.e., we were comparing the Destination's PG
to the Connection's PG instead of comparing Destination's PG to Destination's
PG. This resulted in using a temporary funnel when we shouldn't. And as a
result it attempted to change a Connection's Destination while the destination
was running.
Signed-off-by: Joseph Witt
> Under heavy load, nifi node unable to rejoin cluster, graph modified with
> temp funnel
> -
>
> Key: NIFI-12969
> URL: https://issues.apache.org/jira/browse/NIFI-12969
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.24.0, 2.0.0-M2
>Reporter: Nissim Shiman
>Assignee: Mark Payne
>Priority: Critical
> Fix For: 2.0.0-M3, 1.26.0
>
> Attachments: nifi-app.log, simple_flow.png,
> simple_flow_with_temp-funnel.png
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Under heavy load, if a node leaves the cluster (due to heartbeat time out),
> many times it is unable to rejoin the cluster.
> The nodes' graph will have been modified with a temp-funnel as well.
> Appears to be some sort of [timing
> issue|https://github.com/apache/nifi/blob/rel/nifi-2.0.0-M2/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/connectable/StandardConnection.java#L298]
> # To reproduce, on a nifi cluster of three nodes, set up:
> 2 GenerateFlowFile processors -> PG
> Inside PG:
> inputPort -> UpdateAttribute
> # Keep all defaults except for the following:
> For UpdateAttribute terminate the success relationship
> One of the GenerateFlowFile processors can be disabled,
> the other one should have Run Schedule to be 0 min (this will allow for the
> heavy load)
> # In nifi.properties (on all 3 nodes) to allow for nodes to fall out of the
> cluster, set: nifi.cluster.protocol.heartbeat.interval=2 sec (default is 5)
> nifi.cluster.protocol.heartbeat.missable.max=1 (default is 8)
> Restart nifi. Start flow. The nodes will quickly fall out and rejoin cluster.
> After a few minutes one will likely not be able to rejoin. The graph for
> that node will have the disabled GenerateFlowFile now pointing to a funnel (a
> temp-funnel) instead of the PG
> Stack trace on that nodes nifi-app.log will look like this: (this is from
> 2.0.0-M2):
> {code:java}
> 2024-03-28 13:55:19,395 INFO [Reconnect to Cluster]
> o.a.nifi.controller.StandardFlowService Node disconnected due to Failed to
> properly handle Reconnection request due to org.apache.nifi.control
> ler.serialization.FlowSynchronizationException: Failed to connect node to
> cluster because local flow controller partially updated. Administrator should
> disconnect node and review flow for corrup
> tion.
> 2024-03-28 13:55:19,395 ERROR [Reconnect to Cluster]
> o.a.nifi.controller.StandardFlowService Handling reconnection request failed
> due to: org.apache.nifi.controller.serialization.FlowSynchroniza
> tionException: Failed to connect node to cluster because local flow
> controller partially updated. Administrator should disconnect node and review
> flow for corruption.
> org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed
> to connect node to cluster because local flow controller partially updated.
> Administrator should disconnect node and
> review flow for corruption.
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:985)
> at
> org.apache.nifi.controller.StandardFlowService.handleReconnectionRequest(StandardFlowService.java:655)
> at
> org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:384)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> Caused by:
> org.apache.nifi.controller.serialization.FlowSynchronizationException:
> java.lang.IllegalStateException: Cannot change destination of Connection
> because FlowFiles from this Connection
> are currently held by LocalPort[id=99213c00-78ca-4848-112f-5454cc20656b,
> type=INPUT_PORT, name=inputPort, group=innerPG]
> at
>
Re: [PR] NIFI-12969: Fixed a typo in the #isTempDestinationNecessary method, w… [nifi]
asfgit closed pull request #8600: NIFI-12969: Fixed a typo in the #isTempDestinationNecessary method, w… URL: https://github.com/apache/nifi/pull/8600 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-12969) Under heavy load, nifi node unable to rejoin cluster, graph modified with temp funnel
[
https://issues.apache.org/jira/browse/NIFI-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17833773#comment-17833773
]
ASF subversion and git services commented on NIFI-12969:
Commit 3a03caa14968d0c9a2e6790373871e886b632f09 in nifi's branch
refs/heads/main from Mark Payne
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=3a03caa149 ]
NIFI-12969: Fixed a typo in the #isTempDestinationNecessary method, where we
were comparing existingConnection.getProcessGroup() to
newDestination.getProcessGroup(), instead of comparing
existingConnection.getDestination().getProcessGroup() to
newDestination.getProcessGroup(). I.e., we were comparing the Destination's PG
to the Connection's PG instead of comparing Destination's PG to Destination's
PG. This resulted in using a temporary funnel when we shouldn't. And as a
result it attempted to change a Connection's Destination while the destination
was running.
This closes #8600.
Signed-off-by: Joseph Witt
> Under heavy load, nifi node unable to rejoin cluster, graph modified with
> temp funnel
> -
>
> Key: NIFI-12969
> URL: https://issues.apache.org/jira/browse/NIFI-12969
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.24.0, 2.0.0-M2
>Reporter: Nissim Shiman
>Assignee: Mark Payne
>Priority: Critical
> Fix For: 2.0.0-M3, 1.26.0
>
> Attachments: nifi-app.log, simple_flow.png,
> simple_flow_with_temp-funnel.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Under heavy load, if a node leaves the cluster (due to heartbeat time out),
> many times it is unable to rejoin the cluster.
> The nodes' graph will have been modified with a temp-funnel as well.
> Appears to be some sort of [timing
> issue|https://github.com/apache/nifi/blob/rel/nifi-2.0.0-M2/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/connectable/StandardConnection.java#L298]
> # To reproduce, on a nifi cluster of three nodes, set up:
> 2 GenerateFlowFile processors -> PG
> Inside PG:
> inputPort -> UpdateAttribute
> # Keep all defaults except for the following:
> For UpdateAttribute terminate the success relationship
> One of the GenerateFlowFile processors can be disabled,
> the other one should have Run Schedule to be 0 min (this will allow for the
> heavy load)
> # In nifi.properties (on all 3 nodes) to allow for nodes to fall out of the
> cluster, set: nifi.cluster.protocol.heartbeat.interval=2 sec (default is 5)
> nifi.cluster.protocol.heartbeat.missable.max=1 (default is 8)
> Restart nifi. Start flow. The nodes will quickly fall out and rejoin cluster.
> After a few minutes one will likely not be able to rejoin. The graph for
> that node will have the disabled GenerateFlowFile now pointing to a funnel (a
> temp-funnel) instead of the PG
> Stack trace on that nodes nifi-app.log will look like this: (this is from
> 2.0.0-M2):
> {code:java}
> 2024-03-28 13:55:19,395 INFO [Reconnect to Cluster]
> o.a.nifi.controller.StandardFlowService Node disconnected due to Failed to
> properly handle Reconnection request due to org.apache.nifi.control
> ler.serialization.FlowSynchronizationException: Failed to connect node to
> cluster because local flow controller partially updated. Administrator should
> disconnect node and review flow for corrup
> tion.
> 2024-03-28 13:55:19,395 ERROR [Reconnect to Cluster]
> o.a.nifi.controller.StandardFlowService Handling reconnection request failed
> due to: org.apache.nifi.controller.serialization.FlowSynchroniza
> tionException: Failed to connect node to cluster because local flow
> controller partially updated. Administrator should disconnect node and review
> flow for corruption.
> org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed
> to connect node to cluster because local flow controller partially updated.
> Administrator should disconnect node and
> review flow for corruption.
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:985)
> at
> org.apache.nifi.controller.StandardFlowService.handleReconnectionRequest(StandardFlowService.java:655)
> at
> org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:384)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> Caused by:
> org.apache.nifi.controller.serialization.FlowSynchronizationException:
> java.lang.IllegalStateException: Cannot change destination of Connection
> because FlowFiles from this Connection
> are currently held by LocalPort[id=99213c00-78ca-4848-112f-5454cc20656b,
> type=INPUT_PORT, name=inputPort, group=innerPG]
> at
>
Re: [PR] WIP NIFI-12996 removing commons-compress from shared nar/bom and ensuring… [nifi]
joewitt commented on PR #8598: URL: https://github.com/apache/nifi/pull/8598#issuecomment-2035807614 @exceptionfactory Ok I believe I've updated to what you've described. Tested on a live install and it works as well. Checked where commons-compress and zstd libs are now as well and checks out -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFI-12998) maxmind geoip2 does not belong in nifi-nar-bundle pom
Joe Witt created NIFI-12998: --- Summary: maxmind geoip2 does not belong in nifi-nar-bundle pom Key: NIFI-12998 URL: https://issues.apache.org/jira/browse/NIFI-12998 Project: Apache NiFi Issue Type: Task Reporter: Joe Witt Assignee: Joe Witt Fix For: 2.0.0-M3 found in nifi-nar-bundles pom com.maxmind.geoip2 geoip2 4.2.0 This should not be here. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12972) UI - Improve listing of relationships for a connection
[ https://issues.apache.org/jira/browse/NIFI-12972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17833764#comment-17833764 ] ASF subversion and git services commented on NIFI-12972: Commit 644d086ff2d8d98bb8c31e714e1a254493f9daf3 in nifi's branch refs/heads/main from Pierre Villard [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=644d086ff2 ] NIFI-12972 - Only show selected relationships in read-only connection details (#8582) * NIFI-12972 - Only show selected relationships in read-only connection details * review This closes #8582 > UI - Improve listing of relationships for a connection > -- > > Key: NIFI-12972 > URL: https://issues.apache.org/jira/browse/NIFI-12972 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > Consider a RouteOnAttribute processor with "Route to Property name" with many > dynamic properties. This will lead to many relationships (r1, r2, r3, etc). > Now consider an UpdateAttribute processor. > Connect RouteOnAttribute to this UpdateAttribute processor and select r1 and > r3. > If the two processors are stopped, opening the configuration of the > connection will show an alphabetically sorted list of the ALL the > relationships (of the RouteOnAttribute processor) with checkboxes and r1/r3 > will show as checked. > Now, if one of the two processors is started, opening the configuration of > the connection will show an alphabetically sorted list of the ALL the > relationships (of the RouteOnAttribute processor) and the only distinction > will be that r1 and r3 will be in bold. > When having many connections (think 50+), just showing the selected > relationships in bold is not super user friendly. It's really hard to > distinguish the bold names in a very long list of names. > When one of the processors is running and we're just showing the details of > the connection without the option to change the list of selected > relationships for that connection, I suggest that we only display the list of > the selected relationships. > For reference, the related code is here: > https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/js/nf/nf-connection-details.js#L520 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12972) UI - Improve listing of relationships for a connection
[ https://issues.apache.org/jira/browse/NIFI-12972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Gilman updated NIFI-12972: --- Fix Version/s: 2.0.0-M3 Resolution: Fixed Status: Resolved (was: Patch Available) > UI - Improve listing of relationships for a connection > -- > > Key: NIFI-12972 > URL: https://issues.apache.org/jira/browse/NIFI-12972 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > Fix For: 2.0.0-M3 > > Time Spent: 40m > Remaining Estimate: 0h > > Consider a RouteOnAttribute processor with "Route to Property name" with many > dynamic properties. This will lead to many relationships (r1, r2, r3, etc). > Now consider an UpdateAttribute processor. > Connect RouteOnAttribute to this UpdateAttribute processor and select r1 and > r3. > If the two processors are stopped, opening the configuration of the > connection will show an alphabetically sorted list of the ALL the > relationships (of the RouteOnAttribute processor) with checkboxes and r1/r3 > will show as checked. > Now, if one of the two processors is started, opening the configuration of > the connection will show an alphabetically sorted list of the ALL the > relationships (of the RouteOnAttribute processor) and the only distinction > will be that r1 and r3 will be in bold. > When having many connections (think 50+), just showing the selected > relationships in bold is not super user friendly. It's really hard to > distinguish the bold names in a very long list of names. > When one of the processors is running and we're just showing the details of > the connection without the option to change the list of selected > relationships for that connection, I suggest that we only display the list of > the selected relationships. > For reference, the related code is here: > https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/js/nf/nf-connection-details.js#L520 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12972 - Only show selected relationships in read-only connection details [nifi]
mcgilman merged PR #8582: URL: https://github.com/apache/nifi/pull/8582 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-12969) Under heavy load, nifi node unable to rejoin cluster, graph modified with temp funnel
[
https://issues.apache.org/jira/browse/NIFI-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17833763#comment-17833763
]
Peter Gyori commented on NIFI-12969:
Thank you [~markap14] ! It makes sense. I did not catch that creating the temp
funnel wasn't necessary in the first place.
> Under heavy load, nifi node unable to rejoin cluster, graph modified with
> temp funnel
> -
>
> Key: NIFI-12969
> URL: https://issues.apache.org/jira/browse/NIFI-12969
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.24.0, 2.0.0-M2
>Reporter: Nissim Shiman
>Assignee: Mark Payne
>Priority: Critical
> Fix For: 2.0.0-M3, 1.26.0
>
> Attachments: nifi-app.log, simple_flow.png,
> simple_flow_with_temp-funnel.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Under heavy load, if a node leaves the cluster (due to heartbeat time out),
> many times it is unable to rejoin the cluster.
> The nodes' graph will have been modified with a temp-funnel as well.
> Appears to be some sort of [timing
> issue|https://github.com/apache/nifi/blob/rel/nifi-2.0.0-M2/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/connectable/StandardConnection.java#L298]
> # To reproduce, on a nifi cluster of three nodes, set up:
> 2 GenerateFlowFile processors -> PG
> Inside PG:
> inputPort -> UpdateAttribute
> # Keep all defaults except for the following:
> For UpdateAttribute terminate the success relationship
> One of the GenerateFlowFile processors can be disabled,
> the other one should have Run Schedule to be 0 min (this will allow for the
> heavy load)
> # In nifi.properties (on all 3 nodes) to allow for nodes to fall out of the
> cluster, set: nifi.cluster.protocol.heartbeat.interval=2 sec (default is 5)
> nifi.cluster.protocol.heartbeat.missable.max=1 (default is 8)
> Restart nifi. Start flow. The nodes will quickly fall out and rejoin cluster.
> After a few minutes one will likely not be able to rejoin. The graph for
> that node will have the disabled GenerateFlowFile now pointing to a funnel (a
> temp-funnel) instead of the PG
> Stack trace on that nodes nifi-app.log will look like this: (this is from
> 2.0.0-M2):
> {code:java}
> 2024-03-28 13:55:19,395 INFO [Reconnect to Cluster]
> o.a.nifi.controller.StandardFlowService Node disconnected due to Failed to
> properly handle Reconnection request due to org.apache.nifi.control
> ler.serialization.FlowSynchronizationException: Failed to connect node to
> cluster because local flow controller partially updated. Administrator should
> disconnect node and review flow for corrup
> tion.
> 2024-03-28 13:55:19,395 ERROR [Reconnect to Cluster]
> o.a.nifi.controller.StandardFlowService Handling reconnection request failed
> due to: org.apache.nifi.controller.serialization.FlowSynchroniza
> tionException: Failed to connect node to cluster because local flow
> controller partially updated. Administrator should disconnect node and review
> flow for corruption.
> org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed
> to connect node to cluster because local flow controller partially updated.
> Administrator should disconnect node and
> review flow for corruption.
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:985)
> at
> org.apache.nifi.controller.StandardFlowService.handleReconnectionRequest(StandardFlowService.java:655)
> at
> org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:384)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> Caused by:
> org.apache.nifi.controller.serialization.FlowSynchronizationException:
> java.lang.IllegalStateException: Cannot change destination of Connection
> because FlowFiles from this Connection
> are currently held by LocalPort[id=99213c00-78ca-4848-112f-5454cc20656b,
> type=INPUT_PORT, name=inputPort, group=innerPG]
> at
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:472)
> at
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.sync(VersionedFlowSynchronizer.java:223)
> at
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1740)
> at
> org.apache.nifi.persistence.StandardFlowConfigurationDAO.load(StandardFlowConfigurationDAO.java:91)
> at
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:805)
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:954)
> ... 3 common frames omitted
> Caused by:
Re: [PR] WIP NIFI-12996 removing commons-compress from shared nar/bom and ensuring… [nifi]
joewitt commented on PR #8598: URL: https://github.com/apache/nifi/pull/8598#issuecomment-2035658402 paraphrasing you're basically saying "Hey this is great. How about doing it exactly the opposite way" :). Haha all good. You're probably right. Will give that a go. only means we need zstd as the other deps are direct deps and bypass commons compress anyway. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-12400) Remaining items to migrate UI to currently supported/active framework
[ https://issues.apache.org/jira/browse/NIFI-12400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Gilman updated NIFI-12400: --- Description: The purpose of this Jira is to track all remaining items following the initial commit [1] for NIFI-11481. The description will be kept up to date with remaining features, tasks, and improvements. As each items is worked, a new sub task Jira will be created and referenced in this description. * Support Parameters in Properties with Allowable Values (NIFI-12401) * Summary (NIFI-12437) ** Remaining work not addressed in initial Jira: *** input ports (NIFI-12504) *** output ports (NIFI-12504) *** remote process groups (NIFI-12504) *** process groups (NIFI-12504) *** connections (NIFI-12504) *** System Diagnostics (NIFI-12505) *** support for cluster-specific ui elements (NIFI-12537) *** Add pagination (NIFI-12552) * Counters (NIFI-12415) ** Counter table has extra unnecessary can modify check (NIFI-12948) * Bulletin Board (NIFI-12560) * Provenance (NIFI-12445) ** Event Listing (NIFI-12445) ** Search (NIFI-12445) ** Event Dialog (NIFI-12445) ** Lineage (NIFI-12485) ** Replay from context menu (NIFI-12445) ** Clustering (NIFI-12807) * Configure Reporting Task (NIFI-12563) * Flow Analysis Rules (NIFI-12588) * Registry Clients (NIFI-12486) * Import from Registry (NIFI-12734) * Parameter Providers (NIFI-12622) ** Fetch parameters from provider, map to parameter context (dialog) - (NIFI-12665) * Cluster ** Status History - node specific values (NIFI-12848) * Flow Configuration History (NIFI-12754) ** ActionEntity.action should be optional (NIFI-12948) * Node Status History (NIFI-12553) * Status history for components from canvas context menu (NIFI-12553) * Users (NIFI-12543) ** Don't show users or groups in create/edit dialog is there are none (NIFI-12948) * Policies (NIFI-12548) ** Overridden policy Empty or Copy (NIFI-12679) ** Select Empty by default (NIFI-12948) * Help (NIFI-12795) * About * Show Upstream/Downstream * Align * List Queue (NIFI-12589) ** Clustering (NIFI-12807) * Empty [all] Queue (NIFI-12604) * View Content (NIFI-12589 and NIFI-12445) * View State (NIFI-12611) ** Clustering * Change Component Version * Consider PG permissions in Toolbox (NIFI-12683) * Handle linking to components that are not on the canvas * PG Version (NIFI-12963 & NIFI-12995) ** Start (NIFI-12963) ** Commit (NIFI-12963) ** Force Commit (NIFI-12963) ** Show changes (NIFI-12995) ** Revert changes (NIFI-12995) ** Change Flow version (NIFI-12995) ** Stop (NIFI-12963) * Configure PG (NIFI-12417) * Process Group Services (NIFI-12425) ** Listing (NIFI-12425) ** Create (NIFI-12425) ** Configure (NIFI-12425) ** Delete (NIFI-12425) ** Enable (NIFI-12529) ** Disable (NIFI-12529) ** Improve layout and breadcrumbs ** Disable and Configure * Configure Processor ** Service Link (NIFI-12425) ** Create inline Service (NIFI-12425) ** Parameter Link (NIFI-12502) ** Convert to Parameter (NIFI-12502) ** Fix issue with Property Editor width (NIFI-12547) ** Stop and Configure ** Open Custom UI (NIFI-12958) ** Property History ** Unable to re-add any removed Property (NIFI-12743) ** Shift-Enter new line when editing Property (NIFI-12743) * Property Verification * More Details (Processor, Controller Service, Reporting Task) * Download Flow * Create RPG (NIFI-12758) * Configure RPG (NIFI-12774) * RPG Remote Ports (NIFI-12778) * RPG Go To (NIFI-12759) * RPG Refresh (NIFI-12761) * Color * Move to Front * Copy/Paste * Add/Update Info Icons in dialogs throughout the application * Set viewport earlier when loading a Process Group (NIFI-12737) * Canvas global menu item should navigate user back to where they were on the canvas (NIFI-12737) * Better theme support (NIFI-12655) * Set up development/production environments files * Run unit tests are part of standard build (NIFI-12941) * Update all API calls to consider disconnect node confirmation * Update API calls to use uiOnly flag (NIFI-12950) * Use polling interval from API * Load FlowConfiguration in guard (NIFI-12948) * Routing error handling * General API response error handling ** Management CS (NIFI-12663) ** Canvas CS (NIFI-12684) ** Remainder of Settings (NIFI-12723) ** Counters (NIFI-12723) ** Bulletins (NIFI-12723) ** Flow Designer ** Parameter Contexts (NIFI-12937) ** Parameter ** Provenance (NIFI-12767) ** Queue Listing (NIFI-12742) ** Summary (NIFI-12742) ** Users (NIFI-12742) ** Policies ** Status History * Introduce header in new pages to unify with canvas and offer better navigation. (NIFI-12597) * Theme docs, view flow file, and custom ui's * Prompt user to save Parameter Context when Edit form is dirty * Upgrade to Angular 17 (NIFI-12790) * Start/Stop processors, process groups, ... (NIFI-12568) * Dialog vertical resizing on smaller screens do not allow
[jira] [Resolved] (NIFI-12551) Fix dialog resizing when changing tabs
[ https://issues.apache.org/jira/browse/NIFI-12551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Gilman resolved NIFI-12551. Fix Version/s: 2.0.0-M3 Resolution: Fixed > Fix dialog resizing when changing tabs > -- > > Key: NIFI-12551 > URL: https://issues.apache.org/jira/browse/NIFI-12551 > Project: Apache NiFi > Issue Type: Sub-task >Reporter: Rob Fellows >Priority: Major > Fix For: 2.0.0-M3 > > Attachments: Kapture 2023-12-27 at 11.46.46.gif > > > The processor configuration dialog seems to adjust size when changing tabs. > See the attached screen recording. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-12603) Dialog vertical resizing on smaller screens do not allow users to access all fields
[ https://issues.apache.org/jira/browse/NIFI-12603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Gilman resolved NIFI-12603. Fix Version/s: 2.0.0-M3 Resolution: Fixed > Dialog vertical resizing on smaller screens do not allow users to access all > fields > --- > > Key: NIFI-12603 > URL: https://issues.apache.org/jira/browse/NIFI-12603 > Project: Apache NiFi > Issue Type: Sub-task >Reporter: Scott Aslan >Priority: Major > Fix For: 2.0.0-M3 > > Attachments: Kapture 2024-01-12 at 11.33.57.gif > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12969) Under heavy load, nifi node unable to rejoin cluster, graph modified with temp funnel
[
https://issues.apache.org/jira/browse/NIFI-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17833738#comment-17833738
]
Mark Payne commented on NIFI-12969:
---
[~Nissim Shiman] [~pgyori] I pushed a PR that appears to address the issue. I
believe you're on the right track, that the situation is caused by the fact
that the temp funnel was incorrectly used. But instead of trying to detect when
it's going to happen and/or rollback, the issue is that we had a bug in the
logic for when the temp funnel was created. In this case, there should never be
a temp funnel. In cases where we DO need a temp funnel, the existing logic
should handle stopping the Port, which would make this work smoothly. The issue
arose here because the Port was (rightly) left running. We just need to avoid
creating the temp funnel unnecessarily.
> Under heavy load, nifi node unable to rejoin cluster, graph modified with
> temp funnel
> -
>
> Key: NIFI-12969
> URL: https://issues.apache.org/jira/browse/NIFI-12969
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.24.0, 2.0.0-M2
>Reporter: Nissim Shiman
>Assignee: Mark Payne
>Priority: Critical
> Fix For: 2.0.0-M3, 1.26.0
>
> Attachments: nifi-app.log, simple_flow.png,
> simple_flow_with_temp-funnel.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Under heavy load, if a node leaves the cluster (due to heartbeat time out),
> many times it is unable to rejoin the cluster.
> The nodes' graph will have been modified with a temp-funnel as well.
> Appears to be some sort of [timing
> issue|https://github.com/apache/nifi/blob/rel/nifi-2.0.0-M2/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/connectable/StandardConnection.java#L298]
> # To reproduce, on a nifi cluster of three nodes, set up:
> 2 GenerateFlowFile processors -> PG
> Inside PG:
> inputPort -> UpdateAttribute
> # Keep all defaults except for the following:
> For UpdateAttribute terminate the success relationship
> One of the GenerateFlowFile processors can be disabled,
> the other one should have Run Schedule to be 0 min (this will allow for the
> heavy load)
> # In nifi.properties (on all 3 nodes) to allow for nodes to fall out of the
> cluster, set: nifi.cluster.protocol.heartbeat.interval=2 sec (default is 5)
> nifi.cluster.protocol.heartbeat.missable.max=1 (default is 8)
> Restart nifi. Start flow. The nodes will quickly fall out and rejoin cluster.
> After a few minutes one will likely not be able to rejoin. The graph for
> that node will have the disabled GenerateFlowFile now pointing to a funnel (a
> temp-funnel) instead of the PG
> Stack trace on that nodes nifi-app.log will look like this: (this is from
> 2.0.0-M2):
> {code:java}
> 2024-03-28 13:55:19,395 INFO [Reconnect to Cluster]
> o.a.nifi.controller.StandardFlowService Node disconnected due to Failed to
> properly handle Reconnection request due to org.apache.nifi.control
> ler.serialization.FlowSynchronizationException: Failed to connect node to
> cluster because local flow controller partially updated. Administrator should
> disconnect node and review flow for corrup
> tion.
> 2024-03-28 13:55:19,395 ERROR [Reconnect to Cluster]
> o.a.nifi.controller.StandardFlowService Handling reconnection request failed
> due to: org.apache.nifi.controller.serialization.FlowSynchroniza
> tionException: Failed to connect node to cluster because local flow
> controller partially updated. Administrator should disconnect node and review
> flow for corruption.
> org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed
> to connect node to cluster because local flow controller partially updated.
> Administrator should disconnect node and
> review flow for corruption.
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:985)
> at
> org.apache.nifi.controller.StandardFlowService.handleReconnectionRequest(StandardFlowService.java:655)
> at
> org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:384)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> Caused by:
> org.apache.nifi.controller.serialization.FlowSynchronizationException:
> java.lang.IllegalStateException: Cannot change destination of Connection
> because FlowFiles from this Connection
> are currently held by LocalPort[id=99213c00-78ca-4848-112f-5454cc20656b,
> type=INPUT_PORT, name=inputPort, group=innerPG]
> at
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:472)
> at
>
[jira] [Updated] (NIFI-12958) Add support to open custom UIs
[
https://issues.apache.org/jira/browse/NIFI-12958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Gilman updated NIFI-12958:
---
Status: Patch Available (was: In Progress)
> Add support to open custom UIs
> --
>
> Key: NIFI-12958
> URL: https://issues.apache.org/jira/browse/NIFI-12958
> Project: Apache NiFi
> Issue Type: Sub-task
> Components: Core UI
>Reporter: Matt Gilman
>Assignee: Matt Gilman
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Allow the user to open a custom UI.
> 1) Open Custom UI from context menu. Not from Advanced button in Configure
> dialog.
> 2) Use route like the following...
> {noformat}
> /#/process-groups/43eb7385-018e-1000-2fb6-8dbbe88212cd/Processor/7bbab5aa-018e-1000-7de7-dc4052553b02/advanced{noformat}
> 3) Load component in question and using context path from component load UI
> in iframe using appropriate values for it's query params.
> 4) New page should use the header.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[PR] NIFI-12958: Adding support for custom UIs [nifi]
mcgilman opened a new pull request, #8601: URL: https://github.com/apache/nifi/pull/8601 NIFI-12958: - Adding support for custom UIs. - Running NiFi dev server at context path /nf. - Fixing link used when clicking the logo in the header. - Updating titles and icons used for editing components in Settings for better consistency. - Fixed JOLT advanced UI height. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-12969) Under heavy load, nifi node unable to rejoin cluster, graph modified with temp funnel
[
https://issues.apache.org/jira/browse/NIFI-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIFI-12969:
--
Assignee: Mark Payne (was: Nissim Shiman)
Status: Patch Available (was: Open)
> Under heavy load, nifi node unable to rejoin cluster, graph modified with
> temp funnel
> -
>
> Key: NIFI-12969
> URL: https://issues.apache.org/jira/browse/NIFI-12969
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 2.0.0-M2, 1.24.0
>Reporter: Nissim Shiman
>Assignee: Mark Payne
>Priority: Critical
> Fix For: 2.0.0-M3, 1.26.0
>
> Attachments: nifi-app.log, simple_flow.png,
> simple_flow_with_temp-funnel.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Under heavy load, if a node leaves the cluster (due to heartbeat time out),
> many times it is unable to rejoin the cluster.
> The nodes' graph will have been modified with a temp-funnel as well.
> Appears to be some sort of [timing
> issue|https://github.com/apache/nifi/blob/rel/nifi-2.0.0-M2/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/connectable/StandardConnection.java#L298]
> # To reproduce, on a nifi cluster of three nodes, set up:
> 2 GenerateFlowFile processors -> PG
> Inside PG:
> inputPort -> UpdateAttribute
> # Keep all defaults except for the following:
> For UpdateAttribute terminate the success relationship
> One of the GenerateFlowFile processors can be disabled,
> the other one should have Run Schedule to be 0 min (this will allow for the
> heavy load)
> # In nifi.properties (on all 3 nodes) to allow for nodes to fall out of the
> cluster, set: nifi.cluster.protocol.heartbeat.interval=2 sec (default is 5)
> nifi.cluster.protocol.heartbeat.missable.max=1 (default is 8)
> Restart nifi. Start flow. The nodes will quickly fall out and rejoin cluster.
> After a few minutes one will likely not be able to rejoin. The graph for
> that node will have the disabled GenerateFlowFile now pointing to a funnel (a
> temp-funnel) instead of the PG
> Stack trace on that nodes nifi-app.log will look like this: (this is from
> 2.0.0-M2):
> {code:java}
> 2024-03-28 13:55:19,395 INFO [Reconnect to Cluster]
> o.a.nifi.controller.StandardFlowService Node disconnected due to Failed to
> properly handle Reconnection request due to org.apache.nifi.control
> ler.serialization.FlowSynchronizationException: Failed to connect node to
> cluster because local flow controller partially updated. Administrator should
> disconnect node and review flow for corrup
> tion.
> 2024-03-28 13:55:19,395 ERROR [Reconnect to Cluster]
> o.a.nifi.controller.StandardFlowService Handling reconnection request failed
> due to: org.apache.nifi.controller.serialization.FlowSynchroniza
> tionException: Failed to connect node to cluster because local flow
> controller partially updated. Administrator should disconnect node and review
> flow for corruption.
> org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed
> to connect node to cluster because local flow controller partially updated.
> Administrator should disconnect node and
> review flow for corruption.
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:985)
> at
> org.apache.nifi.controller.StandardFlowService.handleReconnectionRequest(StandardFlowService.java:655)
> at
> org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:384)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> Caused by:
> org.apache.nifi.controller.serialization.FlowSynchronizationException:
> java.lang.IllegalStateException: Cannot change destination of Connection
> because FlowFiles from this Connection
> are currently held by LocalPort[id=99213c00-78ca-4848-112f-5454cc20656b,
> type=INPUT_PORT, name=inputPort, group=innerPG]
> at
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:472)
> at
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.sync(VersionedFlowSynchronizer.java:223)
> at
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1740)
> at
> org.apache.nifi.persistence.StandardFlowConfigurationDAO.load(StandardFlowConfigurationDAO.java:91)
> at
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:805)
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:954)
> ... 3 common frames omitted
> Caused by: java.lang.IllegalStateException: Cannot change destination of
> Connection
[PR] NIFI-12969: Fixed a typo in the #isTempDestinationNecessary method, w… [nifi]
markap14 opened a new pull request, #8600: URL: https://github.com/apache/nifi/pull/8600 …here we were comparing existingConnection.getProcessGroup() to newDestination.getProcessGroup(), instead of comparing existingConnection.getDestination().getProcessGroup() to newDestination.getProcessGroup(). I.e., we were comparing the Destination's PG to the Connection's PG instead of comparing Destination's PG to Destination's PG. This resulted in using a temporary funnel when we shouldn't. And as a result it attempted to change a Connection's Destination while the destination was running. # Summary [NIFI-0](https://issues.apache.org/jira/browse/NIFI-0) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [ ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [ ] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [ ] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [ ] Pull Request based on current revision of the `main` branch - [ ] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 21 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-12997) Correct AWS STS Version 2 Location
[
https://issues.apache.org/jira/browse/NIFI-12997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann updated NIFI-12997:
Description: The AWS STS library for Version 2 is included in
{{nifi-aws-processors}}, but not in {{nifi-aws-service-api}}. The STS library
needs to be in the same ClassLoader as the AWS Auth library for runtime
credential loading. Version 1 of the STS library is already included in
{{nifi-aws-service-api}} and corresponding NAR, which allows S3 Processors to
work with STS. Other AWS Processors using SDK Version 2 need the STS library
moved in order to work with this credentials strategy. (was: The AWS STS
library for Version 2 is included in {{{}nifi-aws-processors{}}}, but not in
{{{}nifi-aws-service-api{}}}. The STS library needs to be in the same
ClassLoader as the AWS Auth library for runtime credential loading. Version 1
of the STS library is already included in {{nifi-aws-service-api}} and
corresponding NAR, which allows S3 Processors to work with STS. Other AWS
Processors using SDK Version 2 need the STS library moved in order to work with
this credentials strategy.)
> Correct AWS STS Version 2 Location
> --
>
> Key: NIFI-12997
> URL: https://issues.apache.org/jira/browse/NIFI-12997
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The AWS STS library for Version 2 is included in {{nifi-aws-processors}}, but
> not in {{nifi-aws-service-api}}. The STS library needs to be in the same
> ClassLoader as the AWS Auth library for runtime credential loading. Version 1
> of the STS library is already included in {{nifi-aws-service-api}} and
> corresponding NAR, which allows S3 Processors to work with STS. Other AWS
> Processors using SDK Version 2 need the STS library moved in order to work
> with this credentials strategy.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12997) Correct AWS STS Version 2 Location
[
https://issues.apache.org/jira/browse/NIFI-12997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann updated NIFI-12997:
Status: Patch Available (was: Open)
> Correct AWS STS Version 2 Location
> --
>
> Key: NIFI-12997
> URL: https://issues.apache.org/jira/browse/NIFI-12997
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The AWS STS library for Version 2 is included in {{nifi-aws-processors}}, but
> not in {{nifi-aws-service-api}}. The STS library needs to be in the same
> ClassLoader as the AWS Auth library for runtime credential loading. Version 1
> of the STS library is already included in {{nifi-aws-service-api}} and
> corresponding NAR, which allows S3 Processors to work with STS. Other AWS
> Processors using SDK Version 2 need the STS library moved in order to work
> with this credentials strategy.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[PR] NIFI-12997 Correct AWS STS Version 2 location [nifi]
exceptionfactory opened a new pull request, #8599: URL: https://github.com/apache/nifi/pull/8599 # Summary [NIFI-12997](https://issues.apache.org/jira/browse/NIFI-12997) Corrects the location of the AWS STS Version 2 library, moving it from `nifi-aws-processors` to `nifi-aws-service-api` where it will be available in the same ClassLoader as the AWS Auth Version 2 library for runtime credential loading. This change aligns with the current location of the AWS STS Version 1 library, already included in `nifi-aws-service-api`. Additional changes include declaring `nifi-aws-service-api` as provided at the bundle level to avoid unnecessary repetition, and marking the STS and Auth libraries as provided in `nifi-aws-nar` to avoid unnecessary inclusion. # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [X] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [X] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [X] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [X] Pull Request based on current revision of the `main` branch - [X] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [X] Build completed using `mvn clean install -P contrib-check` - [X] JDK 21 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-12997) Correct AWS STS Version 2 Location
[
https://issues.apache.org/jira/browse/NIFI-12997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann updated NIFI-12997:
Issue Type: Bug (was: Improvement)
> Correct AWS STS Version 2 Location
> --
>
> Key: NIFI-12997
> URL: https://issues.apache.org/jira/browse/NIFI-12997
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
>
> The AWS STS library for Version 2 is included in {{{}nifi-aws-processors{}}},
> but not in {{{}nifi-aws-service-api{}}}. The STS library needs to be in the
> same ClassLoader as the AWS Auth library for runtime credential loading.
> Version 1 of the STS library is already included in {{nifi-aws-service-api}}
> and corresponding NAR, which allows S3 Processors to work with STS. Other AWS
> Processors using SDK Version 2 need the STS library moved in order to work
> with this credentials strategy.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (NIFI-12997) Correct AWS STS Version 2 Location
David Handermann created NIFI-12997:
---
Summary: Correct AWS STS Version 2 Location
Key: NIFI-12997
URL: https://issues.apache.org/jira/browse/NIFI-12997
Project: Apache NiFi
Issue Type: Improvement
Components: Extensions
Reporter: David Handermann
Assignee: David Handermann
The AWS STS library for Version 2 is included in {{{}nifi-aws-processors{}}},
but not in {{{}nifi-aws-service-api{}}}. The STS library needs to be in the
same ClassLoader as the AWS Auth library for runtime credential loading.
Version 1 of the STS library is already included in {{nifi-aws-service-api}}
and corresponding NAR, which allows S3 Processors to work with STS. Other AWS
Processors using SDK Version 2 need the STS library moved in order to work with
this credentials strategy.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-12969) Under heavy load, nifi node unable to rejoin cluster, graph modified with temp funnel
[
https://issues.apache.org/jira/browse/NIFI-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17833713#comment-17833713
]
Joe Witt commented on NIFI-12969:
-
thanks [~pgyori]. Will flag this as a key issue to look into for 1.26 and
2.0m3. If triaged and found to be manageable we can relax it but otherwise it
will get some attention before we release.
> Under heavy load, nifi node unable to rejoin cluster, graph modified with
> temp funnel
> -
>
> Key: NIFI-12969
> URL: https://issues.apache.org/jira/browse/NIFI-12969
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.24.0, 2.0.0-M2
>Reporter: Nissim Shiman
>Assignee: Nissim Shiman
>Priority: Critical
> Fix For: 2.0.0-M3, 1.26.0
>
> Attachments: nifi-app.log, simple_flow.png,
> simple_flow_with_temp-funnel.png
>
>
> Under heavy load, if a node leaves the cluster (due to heartbeat time out),
> many times it is unable to rejoin the cluster.
> The nodes' graph will have been modified with a temp-funnel as well.
> Appears to be some sort of [timing
> issue|https://github.com/apache/nifi/blob/rel/nifi-2.0.0-M2/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/connectable/StandardConnection.java#L298]
> # To reproduce, on a nifi cluster of three nodes, set up:
> 2 GenerateFlowFile processors -> PG
> Inside PG:
> inputPort -> UpdateAttribute
> # Keep all defaults except for the following:
> For UpdateAttribute terminate the success relationship
> One of the GenerateFlowFile processors can be disabled,
> the other one should have Run Schedule to be 0 min (this will allow for the
> heavy load)
> # In nifi.properties (on all 3 nodes) to allow for nodes to fall out of the
> cluster, set: nifi.cluster.protocol.heartbeat.interval=2 sec (default is 5)
> nifi.cluster.protocol.heartbeat.missable.max=1 (default is 8)
> Restart nifi. Start flow. The nodes will quickly fall out and rejoin cluster.
> After a few minutes one will likely not be able to rejoin. The graph for
> that node will have the disabled GenerateFlowFile now pointing to a funnel (a
> temp-funnel) instead of the PG
> Stack trace on that nodes nifi-app.log will look like this: (this is from
> 2.0.0-M2):
> {code:java}
> 2024-03-28 13:55:19,395 INFO [Reconnect to Cluster]
> o.a.nifi.controller.StandardFlowService Node disconnected due to Failed to
> properly handle Reconnection request due to org.apache.nifi.control
> ler.serialization.FlowSynchronizationException: Failed to connect node to
> cluster because local flow controller partially updated. Administrator should
> disconnect node and review flow for corrup
> tion.
> 2024-03-28 13:55:19,395 ERROR [Reconnect to Cluster]
> o.a.nifi.controller.StandardFlowService Handling reconnection request failed
> due to: org.apache.nifi.controller.serialization.FlowSynchroniza
> tionException: Failed to connect node to cluster because local flow
> controller partially updated. Administrator should disconnect node and review
> flow for corruption.
> org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed
> to connect node to cluster because local flow controller partially updated.
> Administrator should disconnect node and
> review flow for corruption.
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:985)
> at
> org.apache.nifi.controller.StandardFlowService.handleReconnectionRequest(StandardFlowService.java:655)
> at
> org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:384)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> Caused by:
> org.apache.nifi.controller.serialization.FlowSynchronizationException:
> java.lang.IllegalStateException: Cannot change destination of Connection
> because FlowFiles from this Connection
> are currently held by LocalPort[id=99213c00-78ca-4848-112f-5454cc20656b,
> type=INPUT_PORT, name=inputPort, group=innerPG]
> at
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:472)
> at
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.sync(VersionedFlowSynchronizer.java:223)
> at
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1740)
> at
> org.apache.nifi.persistence.StandardFlowConfigurationDAO.load(StandardFlowConfigurationDAO.java:91)
> at
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:805)
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:954)
> ... 3 common frames
[jira] [Updated] (NIFI-12969) Under heavy load, nifi node unable to rejoin cluster, graph modified with temp funnel
[
https://issues.apache.org/jira/browse/NIFI-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe Witt updated NIFI-12969:
Fix Version/s: 2.0.0-M3
1.26.0
> Under heavy load, nifi node unable to rejoin cluster, graph modified with
> temp funnel
> -
>
> Key: NIFI-12969
> URL: https://issues.apache.org/jira/browse/NIFI-12969
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.24.0, 2.0.0-M2
>Reporter: Nissim Shiman
>Assignee: Nissim Shiman
>Priority: Major
> Fix For: 2.0.0-M3, 1.26.0
>
> Attachments: nifi-app.log, simple_flow.png,
> simple_flow_with_temp-funnel.png
>
>
> Under heavy load, if a node leaves the cluster (due to heartbeat time out),
> many times it is unable to rejoin the cluster.
> The nodes' graph will have been modified with a temp-funnel as well.
> Appears to be some sort of [timing
> issue|https://github.com/apache/nifi/blob/rel/nifi-2.0.0-M2/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/connectable/StandardConnection.java#L298]
> # To reproduce, on a nifi cluster of three nodes, set up:
> 2 GenerateFlowFile processors -> PG
> Inside PG:
> inputPort -> UpdateAttribute
> # Keep all defaults except for the following:
> For UpdateAttribute terminate the success relationship
> One of the GenerateFlowFile processors can be disabled,
> the other one should have Run Schedule to be 0 min (this will allow for the
> heavy load)
> # In nifi.properties (on all 3 nodes) to allow for nodes to fall out of the
> cluster, set: nifi.cluster.protocol.heartbeat.interval=2 sec (default is 5)
> nifi.cluster.protocol.heartbeat.missable.max=1 (default is 8)
> Restart nifi. Start flow. The nodes will quickly fall out and rejoin cluster.
> After a few minutes one will likely not be able to rejoin. The graph for
> that node will have the disabled GenerateFlowFile now pointing to a funnel (a
> temp-funnel) instead of the PG
> Stack trace on that nodes nifi-app.log will look like this: (this is from
> 2.0.0-M2):
> {code:java}
> 2024-03-28 13:55:19,395 INFO [Reconnect to Cluster]
> o.a.nifi.controller.StandardFlowService Node disconnected due to Failed to
> properly handle Reconnection request due to org.apache.nifi.control
> ler.serialization.FlowSynchronizationException: Failed to connect node to
> cluster because local flow controller partially updated. Administrator should
> disconnect node and review flow for corrup
> tion.
> 2024-03-28 13:55:19,395 ERROR [Reconnect to Cluster]
> o.a.nifi.controller.StandardFlowService Handling reconnection request failed
> due to: org.apache.nifi.controller.serialization.FlowSynchroniza
> tionException: Failed to connect node to cluster because local flow
> controller partially updated. Administrator should disconnect node and review
> flow for corruption.
> org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed
> to connect node to cluster because local flow controller partially updated.
> Administrator should disconnect node and
> review flow for corruption.
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:985)
> at
> org.apache.nifi.controller.StandardFlowService.handleReconnectionRequest(StandardFlowService.java:655)
> at
> org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:384)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> Caused by:
> org.apache.nifi.controller.serialization.FlowSynchronizationException:
> java.lang.IllegalStateException: Cannot change destination of Connection
> because FlowFiles from this Connection
> are currently held by LocalPort[id=99213c00-78ca-4848-112f-5454cc20656b,
> type=INPUT_PORT, name=inputPort, group=innerPG]
> at
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:472)
> at
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.sync(VersionedFlowSynchronizer.java:223)
> at
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1740)
> at
> org.apache.nifi.persistence.StandardFlowConfigurationDAO.load(StandardFlowConfigurationDAO.java:91)
> at
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:805)
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:954)
> ... 3 common frames omitted
> Caused by: java.lang.IllegalStateException: Cannot change destination of
> Connection because FlowFiles from this Connection are currently held by
>
[jira] [Updated] (NIFI-12969) Under heavy load, nifi node unable to rejoin cluster, graph modified with temp funnel
[
https://issues.apache.org/jira/browse/NIFI-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe Witt updated NIFI-12969:
Priority: Critical (was: Major)
> Under heavy load, nifi node unable to rejoin cluster, graph modified with
> temp funnel
> -
>
> Key: NIFI-12969
> URL: https://issues.apache.org/jira/browse/NIFI-12969
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.24.0, 2.0.0-M2
>Reporter: Nissim Shiman
>Assignee: Nissim Shiman
>Priority: Critical
> Fix For: 2.0.0-M3, 1.26.0
>
> Attachments: nifi-app.log, simple_flow.png,
> simple_flow_with_temp-funnel.png
>
>
> Under heavy load, if a node leaves the cluster (due to heartbeat time out),
> many times it is unable to rejoin the cluster.
> The nodes' graph will have been modified with a temp-funnel as well.
> Appears to be some sort of [timing
> issue|https://github.com/apache/nifi/blob/rel/nifi-2.0.0-M2/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/connectable/StandardConnection.java#L298]
> # To reproduce, on a nifi cluster of three nodes, set up:
> 2 GenerateFlowFile processors -> PG
> Inside PG:
> inputPort -> UpdateAttribute
> # Keep all defaults except for the following:
> For UpdateAttribute terminate the success relationship
> One of the GenerateFlowFile processors can be disabled,
> the other one should have Run Schedule to be 0 min (this will allow for the
> heavy load)
> # In nifi.properties (on all 3 nodes) to allow for nodes to fall out of the
> cluster, set: nifi.cluster.protocol.heartbeat.interval=2 sec (default is 5)
> nifi.cluster.protocol.heartbeat.missable.max=1 (default is 8)
> Restart nifi. Start flow. The nodes will quickly fall out and rejoin cluster.
> After a few minutes one will likely not be able to rejoin. The graph for
> that node will have the disabled GenerateFlowFile now pointing to a funnel (a
> temp-funnel) instead of the PG
> Stack trace on that nodes nifi-app.log will look like this: (this is from
> 2.0.0-M2):
> {code:java}
> 2024-03-28 13:55:19,395 INFO [Reconnect to Cluster]
> o.a.nifi.controller.StandardFlowService Node disconnected due to Failed to
> properly handle Reconnection request due to org.apache.nifi.control
> ler.serialization.FlowSynchronizationException: Failed to connect node to
> cluster because local flow controller partially updated. Administrator should
> disconnect node and review flow for corrup
> tion.
> 2024-03-28 13:55:19,395 ERROR [Reconnect to Cluster]
> o.a.nifi.controller.StandardFlowService Handling reconnection request failed
> due to: org.apache.nifi.controller.serialization.FlowSynchroniza
> tionException: Failed to connect node to cluster because local flow
> controller partially updated. Administrator should disconnect node and review
> flow for corruption.
> org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed
> to connect node to cluster because local flow controller partially updated.
> Administrator should disconnect node and
> review flow for corruption.
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:985)
> at
> org.apache.nifi.controller.StandardFlowService.handleReconnectionRequest(StandardFlowService.java:655)
> at
> org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:384)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> Caused by:
> org.apache.nifi.controller.serialization.FlowSynchronizationException:
> java.lang.IllegalStateException: Cannot change destination of Connection
> because FlowFiles from this Connection
> are currently held by LocalPort[id=99213c00-78ca-4848-112f-5454cc20656b,
> type=INPUT_PORT, name=inputPort, group=innerPG]
> at
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:472)
> at
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.sync(VersionedFlowSynchronizer.java:223)
> at
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1740)
> at
> org.apache.nifi.persistence.StandardFlowConfigurationDAO.load(StandardFlowConfigurationDAO.java:91)
> at
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:805)
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:954)
> ... 3 common frames omitted
> Caused by: java.lang.IllegalStateException: Cannot change destination of
> Connection because FlowFiles from this Connection are currently held by
>
[jira] [Comment Edited] (NIFI-12969) Under heavy load, nifi node unable to rejoin cluster, graph modified with temp funnel
[
https://issues.apache.org/jira/browse/NIFI-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17833705#comment-17833705
]
Nissim Shiman edited comment on NIFI-12969 at 4/3/24 6:45 PM:
--
Thank you very much [~pgyori] for the testing, diagrams, logs and bringing out
the finer details of this.
This greatly appreciated.
I am looking at a solution like you suggested (i.e. wait a little in
StandardConnection) and after a short period of time, to give up and in the
calling method undo the temp connections (and remove the temp funnel) to try to
keep the graph the way it was before the temp connection(s) were made.
There doesn't appear to be an easy way to predict this situation to avoid
making the temp connections in the first place. I've tried checking for
unacknowleged flowfiles on all a group's connections before making
temp-funnel/temp destinations (for any connection). But even if it checks
out, by the time we get to setting an individual connection's destination,
there could be unacknowledged flowfiles on the connection again.
Another possibility is to make a special case for this situation (i.e. adding
another
[DisconnectionCode
|https://github.com/apache/nifi/blob/rel/nifi-2.0.0-M2/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster-protocol/src/main/java/org/apache/nifi/cluster/coordination/node/DisconnectionCode.java]
and in general to have a special case where we try to connect again after
cluster reconnection fails for this case. But this will affect a lot of code
beyond this.
I am very much interested in input on other angles to look at with this.
was (Author: nissim shiman):
Thank you very much [~pgyori] for the testing, diagrams, logs and bringing out
the finer details of this.
This greatly appreciated.
I am looking at a solution like you suggested (i.e. wait a little in
StandardConnection) and after a short period of time, to give up and in the
calling method undo the temp connections (and remove the temp funnel) to try to
keep the graph the way it was before the temp connection(s) were made.
There doesn't appear to be an easy way to predict this situation to avoid
making the temp connections in the first place. I've tried checking for
unacknowleged flowfiles on all a group's connections before making
temp-funnel/temp destinations (for any connection). But even if it checks
out, by the time we get to setting an individual connection's destination,
there could be unacknowledged flowfiles on the connection again.
I am very much interested in input on other angles to look at with this.
> Under heavy load, nifi node unable to rejoin cluster, graph modified with
> temp funnel
> -
>
> Key: NIFI-12969
> URL: https://issues.apache.org/jira/browse/NIFI-12969
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.24.0, 2.0.0-M2
>Reporter: Nissim Shiman
>Assignee: Nissim Shiman
>Priority: Major
> Attachments: nifi-app.log, simple_flow.png,
> simple_flow_with_temp-funnel.png
>
>
> Under heavy load, if a node leaves the cluster (due to heartbeat time out),
> many times it is unable to rejoin the cluster.
> The nodes' graph will have been modified with a temp-funnel as well.
> Appears to be some sort of [timing
> issue|https://github.com/apache/nifi/blob/rel/nifi-2.0.0-M2/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/connectable/StandardConnection.java#L298]
> # To reproduce, on a nifi cluster of three nodes, set up:
> 2 GenerateFlowFile processors -> PG
> Inside PG:
> inputPort -> UpdateAttribute
> # Keep all defaults except for the following:
> For UpdateAttribute terminate the success relationship
> One of the GenerateFlowFile processors can be disabled,
> the other one should have Run Schedule to be 0 min (this will allow for the
> heavy load)
> # In nifi.properties (on all 3 nodes) to allow for nodes to fall out of the
> cluster, set: nifi.cluster.protocol.heartbeat.interval=2 sec (default is 5)
> nifi.cluster.protocol.heartbeat.missable.max=1 (default is 8)
> Restart nifi. Start flow. The nodes will quickly fall out and rejoin cluster.
> After a few minutes one will likely not be able to rejoin. The graph for
> that node will have the disabled GenerateFlowFile now pointing to a funnel (a
> temp-funnel) instead of the PG
> Stack trace on that nodes nifi-app.log will look like this: (this is from
> 2.0.0-M2):
> {code:java}
> 2024-03-28 13:55:19,395 INFO [Reconnect to Cluster]
> o.a.nifi.controller.StandardFlowService Node disconnected due to Failed to
> properly handle Reconnection request due to org.apache.nifi.control
> ler.serialization.FlowSynchronizationException: Failed
[jira] [Comment Edited] (NIFI-12969) Under heavy load, nifi node unable to rejoin cluster, graph modified with temp funnel
[
https://issues.apache.org/jira/browse/NIFI-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17833705#comment-17833705
]
Nissim Shiman edited comment on NIFI-12969 at 4/3/24 6:29 PM:
--
Thank you very much [~pgyori] for the testing, diagrams, logs and bringing out
the finer details of this.
This greatly appreciated.
I am looking at a solution like you suggested (i.e. wait a little in
StandardConnection) and after a short period of time, to give up and in the
calling method undo the temp connections (and remove the temp funnel) to try to
keep the graph the way it was before the temp connection(s) were made.
There doesn't appear to be an easy way to predict this situation to avoid
making the temp connections in the first place. I've tried checking for
unacknowleged flowfiles on all a group's connections before making
temp-funnel/temp destinations (for any connection). But even if it checks
out, by the time we get to setting an individual connection's destination,
there could be unacknowledged flowfiles on the connection again.
I am very much interested in input on other angles to look at with this.
was (Author: nissim shiman):
Thank you very much [~pgyori] for the testing, diagrams, logs and bringing out
the finer details of this.
This greatly appreciated.
I am looking at a solution like you suggested (i.e. wait a little in
StandardConnection) and after a short period of time, to give up and in the
calling method undo the temp connections (and remove the temp funnel) to try to
keep the graph the way it was before the temp connection(s) were made.
There doesn't appear to be an easy way to predict this situation to avoid
making the temp connections in the first place. I've tried checking for
unacknowleged flowfiles on all a group's connections before making
temp-funnel/temp destinations (for any connection). But even if it checks
out, by the time we get to setting an individual connection's destination,
there could be unacknowledged flowfiles on the connection again.
> Under heavy load, nifi node unable to rejoin cluster, graph modified with
> temp funnel
> -
>
> Key: NIFI-12969
> URL: https://issues.apache.org/jira/browse/NIFI-12969
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.24.0, 2.0.0-M2
>Reporter: Nissim Shiman
>Assignee: Nissim Shiman
>Priority: Major
> Attachments: nifi-app.log, simple_flow.png,
> simple_flow_with_temp-funnel.png
>
>
> Under heavy load, if a node leaves the cluster (due to heartbeat time out),
> many times it is unable to rejoin the cluster.
> The nodes' graph will have been modified with a temp-funnel as well.
> Appears to be some sort of [timing
> issue|https://github.com/apache/nifi/blob/rel/nifi-2.0.0-M2/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/connectable/StandardConnection.java#L298]
> # To reproduce, on a nifi cluster of three nodes, set up:
> 2 GenerateFlowFile processors -> PG
> Inside PG:
> inputPort -> UpdateAttribute
> # Keep all defaults except for the following:
> For UpdateAttribute terminate the success relationship
> One of the GenerateFlowFile processors can be disabled,
> the other one should have Run Schedule to be 0 min (this will allow for the
> heavy load)
> # In nifi.properties (on all 3 nodes) to allow for nodes to fall out of the
> cluster, set: nifi.cluster.protocol.heartbeat.interval=2 sec (default is 5)
> nifi.cluster.protocol.heartbeat.missable.max=1 (default is 8)
> Restart nifi. Start flow. The nodes will quickly fall out and rejoin cluster.
> After a few minutes one will likely not be able to rejoin. The graph for
> that node will have the disabled GenerateFlowFile now pointing to a funnel (a
> temp-funnel) instead of the PG
> Stack trace on that nodes nifi-app.log will look like this: (this is from
> 2.0.0-M2):
> {code:java}
> 2024-03-28 13:55:19,395 INFO [Reconnect to Cluster]
> o.a.nifi.controller.StandardFlowService Node disconnected due to Failed to
> properly handle Reconnection request due to org.apache.nifi.control
> ler.serialization.FlowSynchronizationException: Failed to connect node to
> cluster because local flow controller partially updated. Administrator should
> disconnect node and review flow for corrup
> tion.
> 2024-03-28 13:55:19,395 ERROR [Reconnect to Cluster]
> o.a.nifi.controller.StandardFlowService Handling reconnection request failed
> due to: org.apache.nifi.controller.serialization.FlowSynchroniza
> tionException: Failed to connect node to cluster because local flow
> controller partially updated. Administrator should disconnect node and review
> flow for corruption.
>
[jira] [Commented] (NIFI-12969) Under heavy load, nifi node unable to rejoin cluster, graph modified with temp funnel
[
https://issues.apache.org/jira/browse/NIFI-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17833705#comment-17833705
]
Nissim Shiman commented on NIFI-12969:
--
Thank you very much [~pgyori] for the testing, diagrams, logs and bringing out
the finer details of this.
This greatly appreciated.
I am looking at a solution like you suggested (i.e. wait a little in
StandardConnection) and after a short period of time, to give up and in the
calling method undo the temp connections (and remove the temp funnel) to try to
keep the graph the way it was before the temp connection(s) were made.
There doesn't appear to be an easy way to predict this situation to avoid
making the temp connections in the first place. I've tried checking for
unacknowleged flowfiles on all a group's connections before making
temp-funnel/temp destinations (for any connection). But even if it checks
out, by the time we get to setting an individual connection's destination,
there could be unacknowledged flowfiles on the connection again.
> Under heavy load, nifi node unable to rejoin cluster, graph modified with
> temp funnel
> -
>
> Key: NIFI-12969
> URL: https://issues.apache.org/jira/browse/NIFI-12969
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.24.0, 2.0.0-M2
>Reporter: Nissim Shiman
>Assignee: Nissim Shiman
>Priority: Major
> Attachments: nifi-app.log, simple_flow.png,
> simple_flow_with_temp-funnel.png
>
>
> Under heavy load, if a node leaves the cluster (due to heartbeat time out),
> many times it is unable to rejoin the cluster.
> The nodes' graph will have been modified with a temp-funnel as well.
> Appears to be some sort of [timing
> issue|https://github.com/apache/nifi/blob/rel/nifi-2.0.0-M2/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/connectable/StandardConnection.java#L298]
> # To reproduce, on a nifi cluster of three nodes, set up:
> 2 GenerateFlowFile processors -> PG
> Inside PG:
> inputPort -> UpdateAttribute
> # Keep all defaults except for the following:
> For UpdateAttribute terminate the success relationship
> One of the GenerateFlowFile processors can be disabled,
> the other one should have Run Schedule to be 0 min (this will allow for the
> heavy load)
> # In nifi.properties (on all 3 nodes) to allow for nodes to fall out of the
> cluster, set: nifi.cluster.protocol.heartbeat.interval=2 sec (default is 5)
> nifi.cluster.protocol.heartbeat.missable.max=1 (default is 8)
> Restart nifi. Start flow. The nodes will quickly fall out and rejoin cluster.
> After a few minutes one will likely not be able to rejoin. The graph for
> that node will have the disabled GenerateFlowFile now pointing to a funnel (a
> temp-funnel) instead of the PG
> Stack trace on that nodes nifi-app.log will look like this: (this is from
> 2.0.0-M2):
> {code:java}
> 2024-03-28 13:55:19,395 INFO [Reconnect to Cluster]
> o.a.nifi.controller.StandardFlowService Node disconnected due to Failed to
> properly handle Reconnection request due to org.apache.nifi.control
> ler.serialization.FlowSynchronizationException: Failed to connect node to
> cluster because local flow controller partially updated. Administrator should
> disconnect node and review flow for corrup
> tion.
> 2024-03-28 13:55:19,395 ERROR [Reconnect to Cluster]
> o.a.nifi.controller.StandardFlowService Handling reconnection request failed
> due to: org.apache.nifi.controller.serialization.FlowSynchroniza
> tionException: Failed to connect node to cluster because local flow
> controller partially updated. Administrator should disconnect node and review
> flow for corruption.
> org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed
> to connect node to cluster because local flow controller partially updated.
> Administrator should disconnect node and
> review flow for corruption.
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:985)
> at
> org.apache.nifi.controller.StandardFlowService.handleReconnectionRequest(StandardFlowService.java:655)
> at
> org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:384)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> Caused by:
> org.apache.nifi.controller.serialization.FlowSynchronizationException:
> java.lang.IllegalStateException: Cannot change destination of Connection
> because FlowFiles from this Connection
> are currently held by LocalPort[id=99213c00-78ca-4848-112f-5454cc20656b,
> type=INPUT_PORT, name=inputPort, group=innerPG]
> at
>
[jira] [Updated] (NIFI-12735) Execute flow analysis before committing flow into Registry
[
https://issues.apache.org/jira/browse/NIFI-12735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tamas Palfy updated NIFI-12735:
---
Fix Version/s: 2.0.0-M3
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Execute flow analysis before committing flow into Registry
> --
>
> Key: NIFI-12735
> URL: https://issues.apache.org/jira/browse/NIFI-12735
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Reporter: Simon Bence
>Assignee: Simon Bence
>Priority: Major
> Fix For: 2.0.0-M3
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Based on https://lists.apache.org/thread/hsg2ggj9p47lvj5m00okkj0ok58zdoqp
> When committing a flow into a Registry, it would be beneficial to have the
> opportunity to execute the existing flow analysis rules and potentially
> interrupt the commit.
> The change aims for:
> - Execute existing flow analysis rules out of the scheduled times
> - Gracefully resject commit when the flow does not meet the requirements
> - Providing a toggle to turn on or off the feature
> - In case of the feature is inactive or no rules are set, the Registry
> handling behaviour should not change*
> The change does not aim for:
> - RegistryClient level adjustments
> - Using different rules set or other kind of validation methods
> - Providing support for the 1.x line
> {*}
> Expect this following branch:
> https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/StandardNiFiServiceFacade.java#L5028
> As with the usage of this feature, failing snapshot commits will be more
> frequent and I consider "lingering" flow definitions without snapshot
> misleading for users.
> The following test cases describe the proposed behaviour in detail:
> TC1.1 Adding flow with rules violation
> GIVEN a `DisallowComponentType` flow analysis rule, disallowing
> `GenerateFlowFile`
> AND NiFi property `nifi.registry.check.for.rules.violation` is true
> AND process group G1 containing a `GenerateFlowFile`
> AND violation appears as validation failure
> WHEN the user tries to add G1 under version control, using registry R1
> THEN the UI displays the following error message "*Cannot store flow version
> to registry due to rules violations*"
> AND the flow will not appear in registry R1
> AND the UI will show G1 as un-versioned
> AND NiFi node will see G1 as un-versioned
> TC1.2 Adding flow with rules violation before scheduled
> GIVEN a `DisallowComponentType` flow analysis rule, disallowing
> `GenerateFlowFile`
> AND NiFi property `nifi.registry.check.for.rules.violation` is true
> AND process group G1 containing a `GenerateFlowFile`
> AND violation does not appear in UI
> WHEN the user tries to add G1 under version control, using registry R1
> THEN the UI displays the following error message "*Cannot store flow version
> to registry due to rules violations*"
> AND the flow will not appear in registry R1
> AND the UI will show G1 as un-versioned
> AND NiFi node will see G1 as un-versioned
> TC1.3 When turned off violating flows will be added
> GIVEN a `DisallowComponentType` flow analysis rule, disallowing
> `GenerateFlowFile`
> AND NiFi property `nifi.registry.check.for.rules.violation` is fals
> AND process group G1 containing a `GenerateFlowFile`
> AND violation appears as validation failure
> WHEN the user tries to add G1 under version control, using registry R1
> THEN the flow is committed normally
> TC1.4 Adding version with rules violation (when the rule is new)
> GIVEN NiFi property `nifi.registry.check.for.rules.violation` is true
> AND process group G1 containing a `GenerateFlowFile`
> AND process group G1 is under version control
> AND `LogAttribute` processor is added (not committed to the registry)
> GIVEN adding a `DisallowComponentType` flow analysis rule, disallowing
> `GenerateFlowFile`
> AND the user tries to add G1 under version control, using registry R1
> THEN the UI displays the following error message "*Cannot store flow version
> to registry due to rules violations*"
> AND the flow will not appear in registry R1
> AND the UI will show G1 as un-versioned
> AND NiFi node will see G1 as un-versioned
> TC1.5 Adding version with rules violation (when the violation is new)
> GIVEN a `DisallowComponentType` flow analysis rule, disallowing
> `GenerateFlowFile`
> AND NiFi property `nifi.registry.check.for.rules.violation` is true
> AND process group G1 is under version control
> AND `GenerateFlowFile` processor is added (not committed to the registry)
> WHEN the user tries to add new version of G1 to registry R1
> THEN the UI displays the following error message "*Cannot store flow version
> to registry due to rules
[jira] [Commented] (NIFI-12735) Execute flow analysis before committing flow into Registry
[
https://issues.apache.org/jira/browse/NIFI-12735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17833648#comment-17833648
]
ASF subversion and git services commented on NIFI-12735:
Commit 88d434f9eca41b939e2eb7213f45b8bb8ae00ed3 in nifi's branch
refs/heads/main from Simon Bence
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=88d434f9ec ]
NIFI-12735 Adding the option to execute flow analysis before committing
snapshot to registry
This closes #8377.
Signed-off-by: Tamas Palfy
> Execute flow analysis before committing flow into Registry
> --
>
> Key: NIFI-12735
> URL: https://issues.apache.org/jira/browse/NIFI-12735
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Reporter: Simon Bence
>Assignee: Simon Bence
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Based on https://lists.apache.org/thread/hsg2ggj9p47lvj5m00okkj0ok58zdoqp
> When committing a flow into a Registry, it would be beneficial to have the
> opportunity to execute the existing flow analysis rules and potentially
> interrupt the commit.
> The change aims for:
> - Execute existing flow analysis rules out of the scheduled times
> - Gracefully resject commit when the flow does not meet the requirements
> - Providing a toggle to turn on or off the feature
> - In case of the feature is inactive or no rules are set, the Registry
> handling behaviour should not change*
> The change does not aim for:
> - RegistryClient level adjustments
> - Using different rules set or other kind of validation methods
> - Providing support for the 1.x line
> {*}
> Expect this following branch:
> https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/StandardNiFiServiceFacade.java#L5028
> As with the usage of this feature, failing snapshot commits will be more
> frequent and I consider "lingering" flow definitions without snapshot
> misleading for users.
> The following test cases describe the proposed behaviour in detail:
> TC1.1 Adding flow with rules violation
> GIVEN a `DisallowComponentType` flow analysis rule, disallowing
> `GenerateFlowFile`
> AND NiFi property `nifi.registry.check.for.rules.violation` is true
> AND process group G1 containing a `GenerateFlowFile`
> AND violation appears as validation failure
> WHEN the user tries to add G1 under version control, using registry R1
> THEN the UI displays the following error message "*Cannot store flow version
> to registry due to rules violations*"
> AND the flow will not appear in registry R1
> AND the UI will show G1 as un-versioned
> AND NiFi node will see G1 as un-versioned
> TC1.2 Adding flow with rules violation before scheduled
> GIVEN a `DisallowComponentType` flow analysis rule, disallowing
> `GenerateFlowFile`
> AND NiFi property `nifi.registry.check.for.rules.violation` is true
> AND process group G1 containing a `GenerateFlowFile`
> AND violation does not appear in UI
> WHEN the user tries to add G1 under version control, using registry R1
> THEN the UI displays the following error message "*Cannot store flow version
> to registry due to rules violations*"
> AND the flow will not appear in registry R1
> AND the UI will show G1 as un-versioned
> AND NiFi node will see G1 as un-versioned
> TC1.3 When turned off violating flows will be added
> GIVEN a `DisallowComponentType` flow analysis rule, disallowing
> `GenerateFlowFile`
> AND NiFi property `nifi.registry.check.for.rules.violation` is fals
> AND process group G1 containing a `GenerateFlowFile`
> AND violation appears as validation failure
> WHEN the user tries to add G1 under version control, using registry R1
> THEN the flow is committed normally
> TC1.4 Adding version with rules violation (when the rule is new)
> GIVEN NiFi property `nifi.registry.check.for.rules.violation` is true
> AND process group G1 containing a `GenerateFlowFile`
> AND process group G1 is under version control
> AND `LogAttribute` processor is added (not committed to the registry)
> GIVEN adding a `DisallowComponentType` flow analysis rule, disallowing
> `GenerateFlowFile`
> AND the user tries to add G1 under version control, using registry R1
> THEN the UI displays the following error message "*Cannot store flow version
> to registry due to rules violations*"
> AND the flow will not appear in registry R1
> AND the UI will show G1 as un-versioned
> AND NiFi node will see G1 as un-versioned
> TC1.5 Adding version with rules violation (when the violation is new)
> GIVEN a `DisallowComponentType` flow analysis rule, disallowing
> `GenerateFlowFile`
> AND NiFi property `nifi.registry.check.for.rules.violation` is true
> AND process group G1 is under version
Re: [PR] NIFI-12735 Adding the option to execute flow analysis before committing snapshot to registry [nifi]
tpalfy commented on PR #8377: URL: https://github.com/apache/nifi/pull/8377#issuecomment-2034921876 LGTM Thank you for your work @simonbence ! Merged into main. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12735 Adding the option to execute flow analysis before committing snapshot to registry [nifi]
asfgit closed pull request #8377: NIFI-12735 Adding the option to execute flow analysis before committing snapshot to registry URL: https://github.com/apache/nifi/pull/8377 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-12969) Under heavy load, nifi node unable to rejoin cluster, graph modified with temp funnel
[
https://issues.apache.org/jira/browse/NIFI-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17833636#comment-17833636
]
Peter Gyori commented on NIFI-12969:
FYI [~markap14] , [~joewitt] . This concerns NiFi-2.0.0-M3.
> Under heavy load, nifi node unable to rejoin cluster, graph modified with
> temp funnel
> -
>
> Key: NIFI-12969
> URL: https://issues.apache.org/jira/browse/NIFI-12969
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.24.0, 2.0.0-M2
>Reporter: Nissim Shiman
>Assignee: Nissim Shiman
>Priority: Major
> Attachments: nifi-app.log, simple_flow.png,
> simple_flow_with_temp-funnel.png
>
>
> Under heavy load, if a node leaves the cluster (due to heartbeat time out),
> many times it is unable to rejoin the cluster.
> The nodes' graph will have been modified with a temp-funnel as well.
> Appears to be some sort of [timing
> issue|https://github.com/apache/nifi/blob/rel/nifi-2.0.0-M2/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/connectable/StandardConnection.java#L298]
> # To reproduce, on a nifi cluster of three nodes, set up:
> 2 GenerateFlowFile processors -> PG
> Inside PG:
> inputPort -> UpdateAttribute
> # Keep all defaults except for the following:
> For UpdateAttribute terminate the success relationship
> One of the GenerateFlowFile processors can be disabled,
> the other one should have Run Schedule to be 0 min (this will allow for the
> heavy load)
> # In nifi.properties (on all 3 nodes) to allow for nodes to fall out of the
> cluster, set: nifi.cluster.protocol.heartbeat.interval=2 sec (default is 5)
> nifi.cluster.protocol.heartbeat.missable.max=1 (default is 8)
> Restart nifi. Start flow. The nodes will quickly fall out and rejoin cluster.
> After a few minutes one will likely not be able to rejoin. The graph for
> that node will have the disabled GenerateFlowFile now pointing to a funnel (a
> temp-funnel) instead of the PG
> Stack trace on that nodes nifi-app.log will look like this: (this is from
> 2.0.0-M2):
> {code:java}
> 2024-03-28 13:55:19,395 INFO [Reconnect to Cluster]
> o.a.nifi.controller.StandardFlowService Node disconnected due to Failed to
> properly handle Reconnection request due to org.apache.nifi.control
> ler.serialization.FlowSynchronizationException: Failed to connect node to
> cluster because local flow controller partially updated. Administrator should
> disconnect node and review flow for corrup
> tion.
> 2024-03-28 13:55:19,395 ERROR [Reconnect to Cluster]
> o.a.nifi.controller.StandardFlowService Handling reconnection request failed
> due to: org.apache.nifi.controller.serialization.FlowSynchroniza
> tionException: Failed to connect node to cluster because local flow
> controller partially updated. Administrator should disconnect node and review
> flow for corruption.
> org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed
> to connect node to cluster because local flow controller partially updated.
> Administrator should disconnect node and
> review flow for corruption.
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:985)
> at
> org.apache.nifi.controller.StandardFlowService.handleReconnectionRequest(StandardFlowService.java:655)
> at
> org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:384)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> Caused by:
> org.apache.nifi.controller.serialization.FlowSynchronizationException:
> java.lang.IllegalStateException: Cannot change destination of Connection
> because FlowFiles from this Connection
> are currently held by LocalPort[id=99213c00-78ca-4848-112f-5454cc20656b,
> type=INPUT_PORT, name=inputPort, group=innerPG]
> at
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:472)
> at
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.sync(VersionedFlowSynchronizer.java:223)
> at
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1740)
> at
> org.apache.nifi.persistence.StandardFlowConfigurationDAO.load(StandardFlowConfigurationDAO.java:91)
> at
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:805)
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:954)
> ... 3 common frames omitted
> Caused by: java.lang.IllegalStateException: Cannot change destination of
> Connection because FlowFiles from this Connection are currently held by
>
[jira] [Commented] (NIFI-12969) Under heavy load, nifi node unable to rejoin cluster, graph modified with temp funnel
[
https://issues.apache.org/jira/browse/NIFI-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17833586#comment-17833586
]
Peter Gyori commented on NIFI-12969:
Thank you [~Nissim Shiman] for the detailed steps to reproduce the issue.
I managed to reproduce it with your steps even on a very recent build of NiFi,
where the latest commit on the main branch is
{noformat}
commit b574a7e4: NIFI-12987 allow controller service type to be
searchable{noformat}
I used a 3-node cluster. On Apple Silicon one might need a little stricter
settings for heartbeats:
{code:java}
nifi.cluster.protocol.heartbeat.interval=1
secnifi.cluster.protocol.heartbeat.missable.max=1 {code}
I needed two GenerateFlowFile processors, both with 2 simultaneous threads (and
Run Schedule=0 min as you mentioned).
Screenshot of the test flow.
!simple_flow.png|width=645,height=323!
With the following log settings, I captured the attached [^nifi-app.log]
{code:java}
{code}
I paste here the log entries that I find the most important. All of these
happen within 2 milliseconds:
{code:java}
2024-04-03 13:48:53,896 TRACE [Timer-Driven Process Thread-2]
org.apache.nifi.connectable.LocalPort
LocalPort[id=a3c770bd-018e-1000--0d6b782a, type=INPUT_PORT, name=input,
group=pg] obtained claim for FlowFileGate
2024-04-03 13:48:53,896 DEBUG [Reconnect to Cluster]
o.a.n.f.s.StandardVersionedComponentSynchronizer Changing destination of
Connection Connection[ID=a3c84edc-018e-1000--13bbf65e, Source
ID=a6393e3a-3f79-1d9a-b4eb-2a25ca912ab9, Dest
ID=a3c770bd-018e-1000--0d6b782a] from
LocalPort[id=a3c770bd-018e-1000--0d6b782a, type=INPUT_PORT, name=input,
group=pg] to StandardFunnel[id=a34c81f0-018e-1000-7faa-412adc4de192-temp-funnel]
2024-04-03 13:48:53,896 DEBUG [Timer-Driven Process Thread-2]
org.apache.nifi.connectable.LocalPort
LocalPort[id=a3c770bd-018e-1000--0d6b782a, type=INPUT_PORT, name=input,
group=pg] Transferred 1 FlowFiles
2024-04-03 13:48:53,896 TRACE [Timer-Driven Process Thread-2]
org.apache.nifi.connectable.LocalPort
LocalPort[id=a3c770bd-018e-1000--0d6b782a, type=INPUT_PORT, name=input,
group=pg] released claim for FlowFileGate
2024-04-03 13:48:53,896 INFO [Reconnect to Cluster]
o.a.n.c.s.AffectedComponentSet Starting the following components:
AffectedComponentSet[inputPorts=[], outputPorts=[], remoteInputPorts=[],
remoteOutputPorts=[], processors=[], parameterProviders=[],
flowRegistryClients=[], controllerServices=[], reportingTasks=[],
flowAnalysisRules=[], statelessProcessGroups=[]]
2024-04-03 13:48:53,897 INFO [Reconnect to Cluster]
o.a.nifi.controller.StandardFlowService Disconnecting node due to Failed to
properly handle Reconnection request due to
org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed
to connect node to cluster because local flow controller partially updated.
Administrator should disconnect node and review flow for corruption.
2024-04-03 13:48:53,897 INFO [Reconnect to Cluster]
o.apache.nifi.controller.FlowController Will no longer send heartbeats
2024-04-03 13:48:53,897 INFO [Reconnect to Cluster]
o.apache.nifi.controller.FlowController FlowController will stop sending
heartbeats to Cluster Coordinator
2024-04-03 13:48:53,897 INFO [Reconnect to Cluster]
o.apache.nifi.controller.FlowController Cluster State changed from Clustered to
Not Clustered
2024-04-03 13:48:53,897 ERROR [Reconnect to Cluster] [...]
Caused by: java.lang.IllegalStateException: Cannot change destination of
Connection because FlowFiles from this Connection are currently held by
LocalPort[id=a3c770bd-018e-1000--0d6b782a, type=INPUT_PORT, name=input,
group=pg]
at
org.apache.nifi.connectable.StandardConnection.setDestination(StandardConnection.java:299)
at
org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.updateConnectionDestinations(StandardVersionedComponentSynchronizer.java:706)
at
org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.synchronize(StandardVersionedComponentSynchronizer.java:423)
at
org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.lambda$synchronize$0(StandardVersionedComponentSynchronizer.java:248)
at
org.apache.nifi.controller.flow.AbstractFlowManager.withParameterContextResolution(AbstractFlowManager.java:638)
at
org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.synchronize(StandardVersionedComponentSynchronizer.java:243)
at
org.apache.nifi.groups.StandardProcessGroup.synchronizeFlow(StandardProcessGroup.java:3863)
at
org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:464){code}
That is: we have an input port with id a3c770bd-018e-1000--0d6b782a and
the following events happen:
# obtained claim for FlowFileGate
[jira] [Commented] (NIFI-12917) Content repository claim is being zero'd out
[ https://issues.apache.org/jira/browse/NIFI-12917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17833588#comment-17833588 ] Evan F. commented on NIFI-12917: Absolutely. I'm working to get a copy of 1.25 into our environment to test, hopefully by the end of this week. > Content repository claim is being zero'd out > > > Key: NIFI-12917 > URL: https://issues.apache.org/jira/browse/NIFI-12917 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.19.1 > Environment: Red Hat 7 running Apache NiFi 1.19.1 with no custom > NARs. >Reporter: Evan F. >Priority: Critical > Attachments: example_MergeContent.txt > > > I've had rare but consistent instances of MergeContent processors merging > files and losing the content claim/payload while merging. I've pasted an > example log entry below where there's a size to the file pre-merge and the > post-merge content claim becomes zero. I've attached a text file with the > configuration of my MergeContent processor as well. This happens a fraction > of a percent of the time but it happens daily, somewhere around 10 times a > day. I scoured forums and previous tickets but I wasn't able to find a > relevant issue. I'm hoping someone can tell me if this has already been > addressed. > > INFO [Timer-Driven Process Thread-7] o.a.n.processors.standard.MergeContent > Merge Content[id=e6f460da-018d-1000-1c97-a6bd946d4f61] Merged > [StandardFlowFileRecord[uuid=0e5b7c30-021d-4bd9-9edb-1681c9c14d4e,claim=StandardContentClaim > [resourceClaim=StandardRe sourceClaim[id=1710769372188-3700202, > container=default, section=490], offset=0, > length=52430102],offset=0,name=testFile,size=52430102]] into > StandardFlowFileRecord[uuid=536bf8e4-915b-4dad-9e16-063b35e834c1,claim=,offset=0, > name=testFile.pkg,size=0]. Reason for merging: Maximum number of > entries reached -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12969) Under heavy load, nifi node unable to rejoin cluster, graph modified with temp funnel
[
https://issues.apache.org/jira/browse/NIFI-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Gyori updated NIFI-12969:
---
Attachment: simple_flow_with_temp-funnel.png
> Under heavy load, nifi node unable to rejoin cluster, graph modified with
> temp funnel
> -
>
> Key: NIFI-12969
> URL: https://issues.apache.org/jira/browse/NIFI-12969
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.24.0, 2.0.0-M2
>Reporter: Nissim Shiman
>Assignee: Nissim Shiman
>Priority: Major
> Attachments: nifi-app.log, simple_flow.png,
> simple_flow_with_temp-funnel.png
>
>
> Under heavy load, if a node leaves the cluster (due to heartbeat time out),
> many times it is unable to rejoin the cluster.
> The nodes' graph will have been modified with a temp-funnel as well.
> Appears to be some sort of [timing
> issue|https://github.com/apache/nifi/blob/rel/nifi-2.0.0-M2/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/connectable/StandardConnection.java#L298]
> # To reproduce, on a nifi cluster of three nodes, set up:
> 2 GenerateFlowFile processors -> PG
> Inside PG:
> inputPort -> UpdateAttribute
> # Keep all defaults except for the following:
> For UpdateAttribute terminate the success relationship
> One of the GenerateFlowFile processors can be disabled,
> the other one should have Run Schedule to be 0 min (this will allow for the
> heavy load)
> # In nifi.properties (on all 3 nodes) to allow for nodes to fall out of the
> cluster, set: nifi.cluster.protocol.heartbeat.interval=2 sec (default is 5)
> nifi.cluster.protocol.heartbeat.missable.max=1 (default is 8)
> Restart nifi. Start flow. The nodes will quickly fall out and rejoin cluster.
> After a few minutes one will likely not be able to rejoin. The graph for
> that node will have the disabled GenerateFlowFile now pointing to a funnel (a
> temp-funnel) instead of the PG
> Stack trace on that nodes nifi-app.log will look like this: (this is from
> 2.0.0-M2):
> {code:java}
> 2024-03-28 13:55:19,395 INFO [Reconnect to Cluster]
> o.a.nifi.controller.StandardFlowService Node disconnected due to Failed to
> properly handle Reconnection request due to org.apache.nifi.control
> ler.serialization.FlowSynchronizationException: Failed to connect node to
> cluster because local flow controller partially updated. Administrator should
> disconnect node and review flow for corrup
> tion.
> 2024-03-28 13:55:19,395 ERROR [Reconnect to Cluster]
> o.a.nifi.controller.StandardFlowService Handling reconnection request failed
> due to: org.apache.nifi.controller.serialization.FlowSynchroniza
> tionException: Failed to connect node to cluster because local flow
> controller partially updated. Administrator should disconnect node and review
> flow for corruption.
> org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed
> to connect node to cluster because local flow controller partially updated.
> Administrator should disconnect node and
> review flow for corruption.
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:985)
> at
> org.apache.nifi.controller.StandardFlowService.handleReconnectionRequest(StandardFlowService.java:655)
> at
> org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:384)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> Caused by:
> org.apache.nifi.controller.serialization.FlowSynchronizationException:
> java.lang.IllegalStateException: Cannot change destination of Connection
> because FlowFiles from this Connection
> are currently held by LocalPort[id=99213c00-78ca-4848-112f-5454cc20656b,
> type=INPUT_PORT, name=inputPort, group=innerPG]
> at
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:472)
> at
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.sync(VersionedFlowSynchronizer.java:223)
> at
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1740)
> at
> org.apache.nifi.persistence.StandardFlowConfigurationDAO.load(StandardFlowConfigurationDAO.java:91)
> at
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:805)
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:954)
> ... 3 common frames omitted
> Caused by: java.lang.IllegalStateException: Cannot change destination of
> Connection because FlowFiles from this Connection are currently held by
> LocalPort[id=99213c00-78ca-4848-112f-5454cc20656b
> , type=INPUT_PORT,
[jira] [Updated] (NIFI-12969) Under heavy load, nifi node unable to rejoin cluster, graph modified with temp funnel
[
https://issues.apache.org/jira/browse/NIFI-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Gyori updated NIFI-12969:
---
Attachment: simple_flow.png
> Under heavy load, nifi node unable to rejoin cluster, graph modified with
> temp funnel
> -
>
> Key: NIFI-12969
> URL: https://issues.apache.org/jira/browse/NIFI-12969
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.24.0, 2.0.0-M2
>Reporter: Nissim Shiman
>Assignee: Nissim Shiman
>Priority: Major
> Attachments: simple_flow.png
>
>
> Under heavy load, if a node leaves the cluster (due to heartbeat time out),
> many times it is unable to rejoin the cluster.
> The nodes' graph will have been modified with a temp-funnel as well.
> Appears to be some sort of [timing
> issue|https://github.com/apache/nifi/blob/rel/nifi-2.0.0-M2/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/connectable/StandardConnection.java#L298]
> # To reproduce, on a nifi cluster of three nodes, set up:
> 2 GenerateFlowFile processors -> PG
> Inside PG:
> inputPort -> UpdateAttribute
> # Keep all defaults except for the following:
> For UpdateAttribute terminate the success relationship
> One of the GenerateFlowFile processors can be disabled,
> the other one should have Run Schedule to be 0 min (this will allow for the
> heavy load)
> # In nifi.properties (on all 3 nodes) to allow for nodes to fall out of the
> cluster, set: nifi.cluster.protocol.heartbeat.interval=2 sec (default is 5)
> nifi.cluster.protocol.heartbeat.missable.max=1 (default is 8)
> Restart nifi. Start flow. The nodes will quickly fall out and rejoin cluster.
> After a few minutes one will likely not be able to rejoin. The graph for
> that node will have the disabled GenerateFlowFile now pointing to a funnel (a
> temp-funnel) instead of the PG
> Stack trace on that nodes nifi-app.log will look like this: (this is from
> 2.0.0-M2):
> {code:java}
> 2024-03-28 13:55:19,395 INFO [Reconnect to Cluster]
> o.a.nifi.controller.StandardFlowService Node disconnected due to Failed to
> properly handle Reconnection request due to org.apache.nifi.control
> ler.serialization.FlowSynchronizationException: Failed to connect node to
> cluster because local flow controller partially updated. Administrator should
> disconnect node and review flow for corrup
> tion.
> 2024-03-28 13:55:19,395 ERROR [Reconnect to Cluster]
> o.a.nifi.controller.StandardFlowService Handling reconnection request failed
> due to: org.apache.nifi.controller.serialization.FlowSynchroniza
> tionException: Failed to connect node to cluster because local flow
> controller partially updated. Administrator should disconnect node and review
> flow for corruption.
> org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed
> to connect node to cluster because local flow controller partially updated.
> Administrator should disconnect node and
> review flow for corruption.
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:985)
> at
> org.apache.nifi.controller.StandardFlowService.handleReconnectionRequest(StandardFlowService.java:655)
> at
> org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:384)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> Caused by:
> org.apache.nifi.controller.serialization.FlowSynchronizationException:
> java.lang.IllegalStateException: Cannot change destination of Connection
> because FlowFiles from this Connection
> are currently held by LocalPort[id=99213c00-78ca-4848-112f-5454cc20656b,
> type=INPUT_PORT, name=inputPort, group=innerPG]
> at
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:472)
> at
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.sync(VersionedFlowSynchronizer.java:223)
> at
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1740)
> at
> org.apache.nifi.persistence.StandardFlowConfigurationDAO.load(StandardFlowConfigurationDAO.java:91)
> at
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:805)
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:954)
> ... 3 common frames omitted
> Caused by: java.lang.IllegalStateException: Cannot change destination of
> Connection because FlowFiles from this Connection are currently held by
> LocalPort[id=99213c00-78ca-4848-112f-5454cc20656b
> , type=INPUT_PORT, name=inputPort, group=innerPG]
> at
>
[jira] [Updated] (NIFI-12969) Under heavy load, nifi node unable to rejoin cluster, graph modified with temp funnel
[
https://issues.apache.org/jira/browse/NIFI-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Gyori updated NIFI-12969:
---
Attachment: nifi-app.log
> Under heavy load, nifi node unable to rejoin cluster, graph modified with
> temp funnel
> -
>
> Key: NIFI-12969
> URL: https://issues.apache.org/jira/browse/NIFI-12969
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.24.0, 2.0.0-M2
>Reporter: Nissim Shiman
>Assignee: Nissim Shiman
>Priority: Major
> Attachments: nifi-app.log, simple_flow.png
>
>
> Under heavy load, if a node leaves the cluster (due to heartbeat time out),
> many times it is unable to rejoin the cluster.
> The nodes' graph will have been modified with a temp-funnel as well.
> Appears to be some sort of [timing
> issue|https://github.com/apache/nifi/blob/rel/nifi-2.0.0-M2/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/connectable/StandardConnection.java#L298]
> # To reproduce, on a nifi cluster of three nodes, set up:
> 2 GenerateFlowFile processors -> PG
> Inside PG:
> inputPort -> UpdateAttribute
> # Keep all defaults except for the following:
> For UpdateAttribute terminate the success relationship
> One of the GenerateFlowFile processors can be disabled,
> the other one should have Run Schedule to be 0 min (this will allow for the
> heavy load)
> # In nifi.properties (on all 3 nodes) to allow for nodes to fall out of the
> cluster, set: nifi.cluster.protocol.heartbeat.interval=2 sec (default is 5)
> nifi.cluster.protocol.heartbeat.missable.max=1 (default is 8)
> Restart nifi. Start flow. The nodes will quickly fall out and rejoin cluster.
> After a few minutes one will likely not be able to rejoin. The graph for
> that node will have the disabled GenerateFlowFile now pointing to a funnel (a
> temp-funnel) instead of the PG
> Stack trace on that nodes nifi-app.log will look like this: (this is from
> 2.0.0-M2):
> {code:java}
> 2024-03-28 13:55:19,395 INFO [Reconnect to Cluster]
> o.a.nifi.controller.StandardFlowService Node disconnected due to Failed to
> properly handle Reconnection request due to org.apache.nifi.control
> ler.serialization.FlowSynchronizationException: Failed to connect node to
> cluster because local flow controller partially updated. Administrator should
> disconnect node and review flow for corrup
> tion.
> 2024-03-28 13:55:19,395 ERROR [Reconnect to Cluster]
> o.a.nifi.controller.StandardFlowService Handling reconnection request failed
> due to: org.apache.nifi.controller.serialization.FlowSynchroniza
> tionException: Failed to connect node to cluster because local flow
> controller partially updated. Administrator should disconnect node and review
> flow for corruption.
> org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed
> to connect node to cluster because local flow controller partially updated.
> Administrator should disconnect node and
> review flow for corruption.
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:985)
> at
> org.apache.nifi.controller.StandardFlowService.handleReconnectionRequest(StandardFlowService.java:655)
> at
> org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:384)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> Caused by:
> org.apache.nifi.controller.serialization.FlowSynchronizationException:
> java.lang.IllegalStateException: Cannot change destination of Connection
> because FlowFiles from this Connection
> are currently held by LocalPort[id=99213c00-78ca-4848-112f-5454cc20656b,
> type=INPUT_PORT, name=inputPort, group=innerPG]
> at
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:472)
> at
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.sync(VersionedFlowSynchronizer.java:223)
> at
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1740)
> at
> org.apache.nifi.persistence.StandardFlowConfigurationDAO.load(StandardFlowConfigurationDAO.java:91)
> at
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:805)
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:954)
> ... 3 common frames omitted
> Caused by: java.lang.IllegalStateException: Cannot change destination of
> Connection because FlowFiles from this Connection are currently held by
> LocalPort[id=99213c00-78ca-4848-112f-5454cc20656b
> , type=INPUT_PORT, name=inputPort, group=innerPG]
> at
>
[jira] [Updated] (NIFI-12963) Process Group Versioning
[ https://issues.apache.org/jira/browse/NIFI-12963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rob Fellows updated NIFI-12963: --- Status: Patch Available (was: In Progress) > Process Group Versioning > > > Key: NIFI-12963 > URL: https://issues.apache.org/jira/browse/NIFI-12963 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Rob Fellows >Assignee: Rob Fellows >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > * Start > * Commit > * Force Commit > * Show changes > * Revert changes > * Change Flow version > * Stop -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12963) Process Group Versioning
[ https://issues.apache.org/jira/browse/NIFI-12963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rob Fellows updated NIFI-12963: --- Description: * Start * Commit * Force Commit * -Show changes- (NIFI-12995) * -Revert changes- (NIFI-12995) * -Change Flow version- (NIFI-12995) * Stop was: * Start * Commit * Force Commit * Show changes * Revert changes * Change Flow version * Stop > Process Group Versioning > > > Key: NIFI-12963 > URL: https://issues.apache.org/jira/browse/NIFI-12963 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Rob Fellows >Assignee: Rob Fellows >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > * Start > * Commit > * Force Commit > * -Show changes- (NIFI-12995) > * -Revert changes- (NIFI-12995) > * -Change Flow version- (NIFI-12995) > * Stop -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] MINIFICPP-2322 MINIFI_INCLUDE_UCRT_DLLS and MINIFI_INCLUDE_VC_REDIST_… [nifi-minifi-cpp]
fgerlits closed pull request #1752: MINIFICPP-2322 MINIFI_INCLUDE_UCRT_DLLS and MINIFI_INCLUDE_VC_REDIST_… URL: https://github.com/apache/nifi-minifi-cpp/pull/1752 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12918 Fix Stateless NullPointerException on versioned sub-process groups - main branch [nifi]
slambrose commented on PR #8536: URL: https://github.com/apache/nifi/pull/8536#issuecomment-2034320750 > Thanks for working on the unit tests @slambrose. The test for the synchronizer class looks like it exercises the method changes, but the `TestRegistryUtil` appears to be running against a mock of `RegistryUtil`, as opposed to the actual class, so it doesn't exercise the real method. Ready for re-review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] MINIFICPP-2322 MINIFI_INCLUDE_UCRT_DLLS and MINIFI_INCLUDE_VC_REDIST_… [nifi-minifi-cpp]
martinzink opened a new pull request, #1752: URL: https://github.com/apache/nifi-minifi-cpp/pull/1752 …DLLS shouldnt be mutually exclusive Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFICPP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically main)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (MINIFICPP-2322) MINIFI_INCLUDE_UCRT_DLLS and MINIFI_INCLUDE_VC_REDIST_DLLS shouldnt be mutually exclusive
Martin Zink created MINIFICPP-2322: -- Summary: MINIFI_INCLUDE_UCRT_DLLS and MINIFI_INCLUDE_VC_REDIST_DLLS shouldnt be mutually exclusive Key: MINIFICPP-2322 URL: https://issues.apache.org/jira/browse/MINIFICPP-2322 Project: Apache NiFi MiNiFi C++ Issue Type: Bug Reporter: Martin Zink Assignee: Martin Zink -- This message was sent by Atlassian Jira (v8.20.10#820010)
