Re: [PR] NIFI-12918 Fix Stateless NullPointerException on versioned sub-process groups - main branch [nifi]

2024-04-03 Thread via GitHub


exceptionfactory commented on code in PR #8536:
URL: https://github.com/apache/nifi/pull/8536#discussion_r1550854635


##
nifi-stateless/nifi-stateless-bundle/nifi-stateless-engine/src/test/java/org/apache/nifi/stateless/core/TestRegistryUtil.java:
##
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.stateless.core;
+
+import org.apache.nifi.flow.VersionedFlowCoordinates;
+import org.apache.nifi.flow.VersionedProcessGroup;
+import org.apache.nifi.registry.client.FlowSnapshotClient;
+import org.apache.nifi.registry.client.NiFiRegistryClient;
+import org.apache.nifi.registry.client.NiFiRegistryException;
+import org.apache.nifi.registry.flow.VersionedFlowSnapshot;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.UUID;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestRegistryUtil {
+private static final String registryUrl = "http://localhost:18080";;
+private static final String rootBucketId = UUID.randomUUID().toString();
+private static final String rootFlowId = UUID.randomUUID().toString();
+private static final int rootVersion = 1;
+
+private static final String childBucketId = UUID.randomUUID().toString();
+private static final String childFlowId = UUID.randomUUID().toString();
+private static final int childVersion = 1;
+
+@Test
+public void testRegistryUrlCreation() throws NiFiRegistryException, 
IOException {
+NiFiRegistryClient registryClient = mock(NiFiRegistryClient.class);
+FlowSnapshotClient flowSnapshotClient = mock(FlowSnapshotClient.class);
+
+RegistryUtil registryUtil = new RegistryUtil(registryClient, 
registryUrl, null);
+
+
when(registryClient.getFlowSnapshotClient()).thenReturn(flowSnapshotClient);
+when(flowSnapshotClient.get(rootBucketId, rootFlowId, 
rootVersion)).thenAnswer(
+invocation -> buildVfs());
+when(flowSnapshotClient.get(childBucketId, childFlowId, 
childVersion)).thenAnswer(
+invocation -> buildVfs());
+
+VersionedFlowSnapshot snapshot = 
registryUtil.getFlowContents(rootBucketId, rootFlowId, rootVersion, true, null);
+VersionedFlowCoordinates coordinates = 
snapshot.getFlowContents().getVersionedFlowCoordinates();
+
+String formattedUrl = 
registryUtil.getBaseRegistryUrl(coordinates.getStorageLocation());
+assertEquals(formattedUrl, registryUrl);
+}
+
+private VersionedFlowSnapshot buildVfs(){
+VersionedFlowSnapshot vfs = new VersionedFlowSnapshot();
+VersionedProcessGroup vpg1 = new VersionedProcessGroup();
+VersionedProcessGroup vpg2 = new VersionedProcessGroup();
+VersionedFlowCoordinates vfc1 = new VersionedFlowCoordinates();
+VersionedFlowCoordinates vfc2 = new VersionedFlowCoordinates();
+
+String storageLocation1 = 
String.format(registryUrl+"/nifi-registry-api/buckets/%s/flows/%s/versions/%s", 
rootBucketId, rootFlowId, rootVersion);

Review Comment:
   The `registryUrl` parameter should be moved to a format element:
   ```suggestion
   String storageLocation1 = 
String.format("%s/nifi-registry-api/buckets/%s/flows/%s/versions/%s", 
registryUrl, rootBucketId, rootFlowId, rootVersion);
   ```



##
nifi-stateless/nifi-stateless-bundle/nifi-stateless-engine/src/main/java/org/apache/nifi/stateless/core/RegistryUtil.java:
##
@@ -119,6 +126,10 @@ public VersionedFlowSnapshot getFlowContents(final String 
bucketId, final String
 return flowSnapshot;
 }
 
+public String getBaseRegistryUrl(String storageLocation) throws 
MalformedURLException {
+URL url = new URL(storageLocation);

Review Comment:
   Recommend using `URI.create()` instead of `new URL()` to avoid the checked 
`MalformedURLException`



##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/flow/synchronization/StandardVersionedComponentSynchronizer.java:
##
@@ -509,6 +509,8 @@ private String d

Re: [PR] NIFI-12982 Extend test suite of MockProcessSession [nifi]

2024-04-03 Thread via GitHub


EndzeitBegins commented on PR #8589:
URL: https://github.com/apache/nifi/pull/8589#issuecomment-2036148323

   You're welcome. Thank you for the review and merge, @exceptionfactory. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] NIFI-12970: Generate documentation for Python processors [nifi]

2024-04-03 Thread via GitHub


exceptionfactory commented on code in PR #8579:
URL: https://github.com/apache/nifi/pull/8579#discussion_r1550834981


##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-documentation/src/main/java/org/apache/nifi/documentation/html/AbstractHtmlDocumentationWriter.java:
##
@@ -0,0 +1,356 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.documentation.html;
+
+import org.apache.nifi.documentation.DocumentationWriter;
+import org.apache.nifi.util.StringUtils;
+
+import javax.xml.stream.FactoryConfigurationError;
+import javax.xml.stream.XMLOutputFactory;
+import javax.xml.stream.XMLStreamException;
+import javax.xml.stream.XMLStreamWriter;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+abstract class AbstractHtmlDocumentationWriter implements 
DocumentationWriter {
+
+/**
+ * The filename where additional user specified information may be stored.
+ */
+public static final String ADDITIONAL_DETAILS_HTML = 
"additionalDetails.html";
+
+static final String NO_DESCRIPTION = "No description provided.";
+static final String NO_TAGS = "No tags provided.";
+static final String NO_PROPERTIES = "This component has no required or 
optional properties.";
+
+static final String H2 = "h2";
+static final String H3 = "h3";
+static final String H4 = "h4";
+static final String P = "p";
+static final String BR = "br";
+static final String SPAN = "span";
+static final String STRONG = "strong";
+static final String TABLE = "table";
+static final String TH = "th";
+static final String TR = "tr";
+static final String TD = "td";
+static final String UL = "ul";
+static final String LI = "li";
+static final String ID = "id";
+
+@Override
+public void write(final T component, final OutputStream streamToWriteTo, 
final boolean includesAdditionalDocumentation) throws IOException {
+try {
+XMLStreamWriter xmlStreamWriter = 
XMLOutputFactory.newInstance().createXMLStreamWriter(streamToWriteTo, "UTF-8");
+xmlStreamWriter.writeDTD("");
+xmlStreamWriter.writeStartElement("html");
+xmlStreamWriter.writeAttribute("lang", "en");
+writeHead(component, xmlStreamWriter);
+writeBody(component, xmlStreamWriter, 
includesAdditionalDocumentation);
+xmlStreamWriter.writeEndElement();
+xmlStreamWriter.close();
+} catch (XMLStreamException | FactoryConfigurationError e) {
+throw new IOException("Unable to create XMLOutputStream", e);
+}
+}
+
+/**
+ * Writes the head portion of the HTML documentation.
+ *
+ * @param component   the component to describe
+ * @param xmlStreamWriter the stream to write to
+ * @throws XMLStreamException thrown if there was a problem writing to the 
stream
+ */
+protected void writeHead(final T component, final XMLStreamWriter 
xmlStreamWriter) throws XMLStreamException {
+xmlStreamWriter.writeStartElement("head");
+xmlStreamWriter.writeStartElement("meta");
+xmlStreamWriter.writeAttribute("charset", "utf-8");
+xmlStreamWriter.writeEndElement();
+writeSimpleElement(xmlStreamWriter, "title", getTitle(component));
+
+xmlStreamWriter.writeStartElement("link");
+xmlStreamWriter.writeAttribute("rel", "stylesheet");
+xmlStreamWriter.writeAttribute("href", 
"../../../../../css/component-usage.css");
+xmlStreamWriter.writeAttribute("type", "text/css");
+xmlStreamWriter.writeEndElement();
+xmlStreamWriter.writeEndElement();
+
+xmlStreamWriter.writeStartElement("script");
+xmlStreamWriter.writeAttribute("type", "text/javascript");
+xmlStreamWriter.writeCharacters("window.onload = 
function(){if(self==top) { " +
+"document.getElementById('nameHeader').style.display = 
\"inherit\"; } }");
+xmlStreamWriter.writeEndElement();
+}
+
+/**
+ * Gets the class name of the component.
+ *
+ * @param component the component to describe
+ * @retu

[PR] Revert "Bump webpack-dev-middleware and karma-webpack (#8547)" [nifi]

2024-04-03 Thread via GitHub


sardell opened a new pull request, #8603:
URL: https://github.com/apache/nifi/pull/8603

   This PR reverts commit 69f1a3be4dd0ede7912ee8cfd67d39bdc3d6eca0 (PR #8547). 
This commit on `main` is currently causing local builds of NiFi to fail due to 
a dependency in nifi-registry's package-lock.json requiring a version of 
Node.js higher than the one we use to build nifi-registry.
   
   https://github.com/apache/nifi/assets/7304869/712a8326-4751-4bd1-b383-81cfa968f3f3";>
   
   https://github.com/apache/nifi/assets/7304869/83eb590a-fab1-400e-829b-a92462d08867";>
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   # Summary
   
   [NIFI-0](https://issues.apache.org/jira/browse/NIFI-0)
   
   # Tracking
   
   Please complete the following tracking steps prior to pull request creation.
   
   ### Issue Tracking
   
   - [ ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue 
created
   
   ### Pull Request Tracking
   
   - [ ] Pull Request title starts with Apache NiFi Jira issue number, such as 
`NIFI-0`
   - [ ] Pull Request commit message starts with Apache NiFi Jira issue number, 
as such `NIFI-0`
   
   ### Pull Request Formatting
   
   - [ ] Pull Request based on current revision of the `main` branch
   - [ ] Pull Request refers to a feature branch with one commit containing 
changes
   
   # Verification
   
   Please indicate the verification steps performed prior to pull request 
creation.
   
   ### Build
   
   - [ ] Build completed using `mvn clean install -P contrib-check`
 - [ ] JDK 21
   
   ### Licensing
   
   - [ ] New dependencies are compatible with the [Apache License 
2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License 
Policy](https://www.apache.org/legal/resolved.html)
   - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` 
files
   
   ### Documentation
   
   - [ ] Documentation formatting appears as expected in rendered files
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Updated] (NIFI-12875) Add Response Handling Strategy to RestLookupService

2024-04-03 Thread David Handermann (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Handermann updated NIFI-12875:

Fix Version/s: 2.0.0-M3
 Assignee: Gregory M. Foreman
   Resolution: Fixed
   Status: Resolved  (was: Patch Available)

> Add Response Handling Strategy to RestLookupService
> ---
>
> Key: NIFI-12875
> URL: https://issues.apache.org/jira/browse/NIFI-12875
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 2.0.0-M2
>Reporter: Gregory M. Foreman
>Assignee: Gregory M. Foreman
>Priority: Major
> Fix For: 2.0.0-M3
>
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> The RestLookupService service does not handle HTTP error codes correctly.  If 
> the web service being consumed returns a 4xx or 5xx error code, the 
> RestLookupService service ignores the error and passes the response content 
> to the record reader, causing unexpected exception reporting.
> Example: a web service that communicates with clients via JSON is behind a 
> proxy.  The proxy returns a 502 error and responds with HTML content 
> describing the error.  The RestLookupService uses a JsonTreeReader reader and 
> attempts to parse the HTML characters.  This leads to it throwing an 
> exception that it was unable to parse the '<' character.  The real reason is 
> the proxy is sending back a 502 error, but this detail is hidden.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Resolved] (NIFI-7851) RestLookupService does not throw a LookupFailureException when the rest endpoint response-code is not successful.

2024-04-03 Thread David Handermann (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-7851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Handermann resolved NIFI-7851.

Fix Version/s: 2.0.0-M3
   Resolution: Fixed

> RestLookupService does not throw a LookupFailureException when the rest 
> endpoint response-code is not successful.
> -
>
> Key: NIFI-7851
> URL: https://issues.apache.org/jira/browse/NIFI-7851
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Esteban Capoccetti
>Assignee: Nadeem
>Priority: Major
> Fix For: 2.0.0-M3
>
>
> RestLookupService does not throw a LookupFailureException when the rest 
> endpoint response-code is not successful.
> Lookup Processor using a "Lookup Service" of type "RestLookupService" do not 
> send the flowFile into a Failure Connections when the rest service 
> http-response-code is not a successful one; it sends those flowfiles to the 
> "success/matched" connections instead.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-7851) RestLookupService does not throw a LookupFailureException when the rest endpoint response-code is not successful.

2024-04-03 Thread David Handermann (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-7851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Handermann updated NIFI-7851:
---
Issue Type: Improvement  (was: Bug)

> RestLookupService does not throw a LookupFailureException when the rest 
> endpoint response-code is not successful.
> -
>
> Key: NIFI-7851
> URL: https://issues.apache.org/jira/browse/NIFI-7851
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Esteban Capoccetti
>Assignee: Nadeem
>Priority: Major
>
> RestLookupService does not throw a LookupFailureException when the rest 
> endpoint response-code is not successful.
> Lookup Processor using a "Lookup Service" of type "RestLookupService" do not 
> send the flowFile into a Failure Connections when the rest service 
> http-response-code is not a successful one; it sends those flowfiles to the 
> "success/matched" connections instead.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12875) Add Response Handling Strategy to RestLookupService

2024-04-03 Thread David Handermann (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Handermann updated NIFI-12875:

Summary: Add Response Handling Strategy to RestLookupService  (was: 
RestLookupService not handling HTTP 4xx/5xx error codes)

> Add Response Handling Strategy to RestLookupService
> ---
>
> Key: NIFI-12875
> URL: https://issues.apache.org/jira/browse/NIFI-12875
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 2.0.0-M2
>Reporter: Gregory M. Foreman
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> The RestLookupService service does not handle HTTP error codes correctly.  If 
> the web service being consumed returns a 4xx or 5xx error code, the 
> RestLookupService service ignores the error and passes the response content 
> to the record reader, causing unexpected exception reporting.
> Example: a web service that communicates with clients via JSON is behind a 
> proxy.  The proxy returns a 502 error and responds with HTML content 
> describing the error.  The RestLookupService uses a JsonTreeReader reader and 
> attempts to parse the HTML characters.  This leads to it throwing an 
> exception that it was unable to parse the '<' character.  The real reason is 
> the proxy is sending back a 502 error, but this detail is hidden.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12875) RestLookupService not handling HTTP 4xx/5xx error codes

2024-04-03 Thread David Handermann (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Handermann updated NIFI-12875:

Issue Type: Improvement  (was: Bug)

> RestLookupService not handling HTTP 4xx/5xx error codes
> ---
>
> Key: NIFI-12875
> URL: https://issues.apache.org/jira/browse/NIFI-12875
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 2.0.0-M2
>Reporter: Gregory M. Foreman
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> The RestLookupService service does not handle HTTP error codes correctly.  If 
> the web service being consumed returns a 4xx or 5xx error code, the 
> RestLookupService service ignores the error and passes the response content 
> to the record reader, causing unexpected exception reporting.
> Example: a web service that communicates with clients via JSON is behind a 
> proxy.  The proxy returns a 502 error and responds with HTML content 
> describing the error.  The RestLookupService uses a JsonTreeReader reader and 
> attempts to parse the HTML characters.  This leads to it throwing an 
> exception that it was unable to parse the '<' character.  The real reason is 
> the proxy is sending back a 502 error, but this detail is hidden.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (NIFI-12875) RestLookupService not handling HTTP 4xx/5xx error codes

2024-04-03 Thread ASF subversion and git services (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17833805#comment-17833805
 ] 

ASF subversion and git services commented on NIFI-12875:


Commit 84df025ccfe68bf9eefc0a43ad14fdc727e3723d in nifi's branch 
refs/heads/main from Gregory M. Foreman
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=84df025ccf ]

NIFI-12875 Added Response Handling Strategy to RestLookupService

This closes #8484

Signed-off-by: David Handermann 


> RestLookupService not handling HTTP 4xx/5xx error codes
> ---
>
> Key: NIFI-12875
> URL: https://issues.apache.org/jira/browse/NIFI-12875
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 2.0.0-M2
>Reporter: Gregory M. Foreman
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> The RestLookupService service does not handle HTTP error codes correctly.  If 
> the web service being consumed returns a 4xx or 5xx error code, the 
> RestLookupService service ignores the error and passes the response content 
> to the record reader, causing unexpected exception reporting.
> Example: a web service that communicates with clients via JSON is behind a 
> proxy.  The proxy returns a 502 error and responds with HTML content 
> describing the error.  The RestLookupService uses a JsonTreeReader reader and 
> attempts to parse the HTML characters.  This leads to it throwing an 
> exception that it was unable to parse the '<' character.  The real reason is 
> the proxy is sending back a 502 error, but this detail is hidden.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Re: [PR] NIFI-12875: Add unsuccessful HTTP status code handling logic [nifi]

2024-04-03 Thread via GitHub


exceptionfactory closed pull request #8484: NIFI-12875: Add unsuccessful HTTP 
status code handling logic
URL: https://github.com/apache/nifi/pull/8484


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Commented] (NIFI-12982) Extend test suite of MockProcessSession

2024-04-03 Thread ASF subversion and git services (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17833801#comment-17833801
 ] 

ASF subversion and git services commented on NIFI-12982:


Commit 6939ffc08d6a6d6ce6a932637fde8523fb2811cd in nifi's branch 
refs/heads/main from EndzeitBegins
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=6939ffc08d ]

NIFI-12982 Extend test suite of MockProcessSession

This closes #8589

Signed-off-by: David Handermann 


> Extend test suite of MockProcessSession
> ---
>
> Key: NIFI-12982
> URL: https://issues.apache.org/jira/browse/NIFI-12982
> Project: Apache NiFi
>  Issue Type: Sub-task
>Reporter: endzeit
>Assignee: endzeit
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> As part of NIFI-12971 (or a subtask thereof) most likely changes will be 
> introduced to both {{StandardProcessorTestRunner}} and {{MockProcessSession}}.
>  
> To reduce the risk of regressions  introduced by those changes, the test 
> suite of {{MockProcessSession}} should be extended to account for more 
> scenarios, regarding both expected successful and failing interactions.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Resolved] (NIFI-12982) Extend test suite of MockProcessSession

2024-04-03 Thread David Handermann (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Handermann resolved NIFI-12982.
-
Fix Version/s: 2.0.0-M3
   Resolution: Fixed

> Extend test suite of MockProcessSession
> ---
>
> Key: NIFI-12982
> URL: https://issues.apache.org/jira/browse/NIFI-12982
> Project: Apache NiFi
>  Issue Type: Sub-task
>Reporter: endzeit
>Assignee: endzeit
>Priority: Major
> Fix For: 2.0.0-M3
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> As part of NIFI-12971 (or a subtask thereof) most likely changes will be 
> introduced to both {{StandardProcessorTestRunner}} and {{MockProcessSession}}.
>  
> To reduce the risk of regressions  introduced by those changes, the test 
> suite of {{MockProcessSession}} should be extended to account for more 
> scenarios, regarding both expected successful and failing interactions.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Re: [PR] NIFI-12982 Extend test suite of MockProcessSession [nifi]

2024-04-03 Thread via GitHub


exceptionfactory closed pull request #8589: NIFI-12982 Extend test suite of 
MockProcessSession
URL: https://github.com/apache/nifi/pull/8589


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] NIFI-12982 Extend test suite of MockProcessSession [nifi]

2024-04-03 Thread via GitHub


exceptionfactory commented on code in PR #8589:
URL: https://github.com/apache/nifi/pull/8589#discussion_r1550787557


##
nifi-mock/src/test/java/org/apache/nifi/util/TestMockProcessSession.java:
##
@@ -45,189 +52,588 @@
 
 public class TestMockProcessSession {
 
-@Test
-public void testReadWithoutCloseThrowsExceptionOnCommit() throws 
IOException {
-final Processor processor = new PoorlyBehavedProcessor();
-final MockProcessSession session = new MockProcessSession(new 
SharedSessionState(processor, new AtomicLong(0L)), processor, true, new 
MockStateManager(processor), true);
-FlowFile flowFile = session.createFlowFile("hello, world".getBytes());
-final InputStream in = session.read(flowFile);
-final byte[] buffer = new byte[12];
-fillBuffer(in, buffer);
+private final Processor processor = new TestProcessor();
+private final SharedSessionState sharedState = new 
SharedSessionState(processor, new AtomicLong(0L));
+private final MockStateManager stateManager = new 
MockStateManager(processor);
+private final MockProcessSession session = new 
MockProcessSession(sharedState, processor, stateManager);
+
+private final Processor statefulProcessor = new StatefulTestProcessor();
+private final SharedSessionState sharedStateOfStatefulProcessor = new 
SharedSessionState(statefulProcessor, new AtomicLong(0L));
+private final MockStateManager stateManagerOfStatefulProcessor = new 
MockStateManager(statefulProcessor);
+private final MockProcessSession sessionOfStatefulProcessor = new 
MockProcessSession(sharedStateOfStatefulProcessor, statefulProcessor, 
stateManagerOfStatefulProcessor);

Review Comment:
   Thanks for the helpful reply @EndzeitBegins, very good point regarding the 
standard per-method behavior. In light of this, and the fact that there are no 
Mockito-based mock objects, the current approach makes sense.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] NIFI-12959: Support loading python processors from NARs [nifi]

2024-04-03 Thread via GitHub


exceptionfactory commented on code in PR #8573:
URL: https://github.com/apache/nifi/pull/8573#discussion_r1550777551


##
nifi-nar-bundles/nifi-py4j-bundle/nifi-py4j-bridge/src/main/java/org/apache/nifi/py4j/PythonProcess.java:
##
@@ -224,6 +226,10 @@ private String generateAuthToken() {
 return Base64.getEncoder().encodeToString(bytes);
 }
 
+private boolean isUseVirtualEnv() {

Review Comment:
   It looks like the only use of this method is a negated reference. Can this 
be removed and replaced with `if (packagedWithDependencies)`?



##
nifi-nar-bundles/nifi-py4j-bundle/nifi-py4j-bridge/src/main/java/org/apache/nifi/py4j/PythonProcess.java:
##
@@ -234,8 +240,18 @@ private Process launchPythonProcess(final int 
listeningPort, final String authTo
 
 final List commands = new ArrayList<>();
 commands.add(pythonCommand);
+if (!isUseVirtualEnv()) {
+// If not using venv, we will not launch a separate virtual 
environment, so we need to use the -S
+// flag in order to prevent the Python process from using the 
installation's site-packages. This provides
+// proper dependency isolation to the Python process.
+commands.add("-S");
+}
 
 String pythonPath = pythonApiDirectory.getAbsolutePath();
+final String absolutePath = virtualEnvHome.getAbsolutePath();
+final File dependenciesDir = new File(new File(absolutePath), 
"NAR-INF/bundled-dependencies");

Review Comment:
   Is the intent to add `NAR-INF/bundled-dependencies` in all cases? It seems 
like it should be conditional on whether the Processor is packaged with 
dependencies.



##
nifi-nar-bundles/nifi-py4j-bundle/nifi-py4j-bridge/src/main/java/org/apache/nifi/py4j/PythonProcess.java:
##
@@ -287,6 +309,13 @@ String resolvePythonCommand() throws IOException {
 
 
 private void setupEnvironment() throws IOException {
+// Environment creation is only necessary if using PIP. Otherwise, the 
Process requires no outside dependencies, other than those
+// provided in the package and thus we can simply include those 
packages in the PYTHON_PATH.
+if (!isUseVirtualEnv()) {
+logger.debug("Will not create Python Virtual Environment because 
PIP is disabled in nifi.properties");

Review Comment:
   Very minor, but general usage of `pip` is all lowercase and opposed to all 
uppercase. More importantly though, the message does not reflect the behavior, 
as it is only based on the the presence of packaged dependencies.
   ```suggestion
   logger.debug("Python Virtual Environment not created: Python 
Processor packaged with dependencies");
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Resolved] (NIFI-12996) nifi-shared-bom construct broke classloading for optional dependencies of common-compress including zstd

2024-04-03 Thread David Handermann (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Handermann resolved NIFI-12996.
-
Resolution: Fixed

> nifi-shared-bom construct broke classloading for optional dependencies of 
> common-compress including zstd
> 
>
> Key: NIFI-12996
> URL: https://issues.apache.org/jira/browse/NIFI-12996
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.24.0, 1.25.0, 2.0.0-M2
>Reporter: Joe Witt
>Assignee: Joe Witt
>Priority: Critical
> Fix For: 2.0.0-M3, 1.26.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> In Apache Slack a user reported this was broken for them in NiFi 1.25.0 but 
> used to work in 1.19.1.
> On apache 2.0 m2+latest I can reproduce this by simply trying to use it at 
> all.  First question is do we not have unit tests but anyway this needs to be 
> evaluated as to why it isn't working.
> {noformat}
> 2024-04-02 13:30:12,514 ERROR [Timer-Driven Process Thread-3] 
> o.a.n.p.standard.CompressContent 
> CompressContent[id=a07f7ae5-018e-1000-c82c-2e2143f9c320] Processing halted: 
> yielding [1 sec]
> java.lang.NoClassDefFoundError: com/github/luben/zstd/ZstdOutputStream
> at 
> org.apache.commons.compress.compressors.zstandard.ZstdCompressorOutputStream.(ZstdCompressorOutputStream.java:57)
> at 
> org.apache.nifi.processors.standard.CompressContent$1.process(CompressContent.java:353)
> at 
> org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:3426)
> at 
> org.apache.nifi.processors.standard.CompressContent.onTrigger(CompressContent.java:300)
> at 
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at 
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1274)
> at 
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:244)
> at 
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:102)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at 
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
> at 
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358)
> at 
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> at 
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
> at 
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> Caused by: java.lang.ClassNotFoundException: 
> com.github.luben.zstd.ZstdOutputStream
> at 
> java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:593)
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)
> ... 15 common frames omitted
> {noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (NIFI-12996) nifi-shared-bom construct broke classloading for optional dependencies of common-compress including zstd

2024-04-03 Thread ASF subversion and git services (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17833794#comment-17833794
 ] 

ASF subversion and git services commented on NIFI-12996:


Commit d3ea429d25db6f339ddd0701d2bb2616f054eed6 in nifi's branch 
refs/heads/support/nifi-1.x from Joe Witt
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=d3ea429d25 ]

NIFI-12996 Moved zstd-jni to standard-shared-nar

- zstd-jni must be present in the same NAR as commons-compress

This closes #8598

Signed-off-by: David Handermann 

(cherry picked from commit 98c4061cfee78590a7d391e53a835c5b87ca4bed)


> nifi-shared-bom construct broke classloading for optional dependencies of 
> common-compress including zstd
> 
>
> Key: NIFI-12996
> URL: https://issues.apache.org/jira/browse/NIFI-12996
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.24.0, 1.25.0, 2.0.0-M2
>Reporter: Joe Witt
>Assignee: Joe Witt
>Priority: Critical
> Fix For: 2.0.0-M3, 1.26.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> In Apache Slack a user reported this was broken for them in NiFi 1.25.0 but 
> used to work in 1.19.1.
> On apache 2.0 m2+latest I can reproduce this by simply trying to use it at 
> all.  First question is do we not have unit tests but anyway this needs to be 
> evaluated as to why it isn't working.
> {noformat}
> 2024-04-02 13:30:12,514 ERROR [Timer-Driven Process Thread-3] 
> o.a.n.p.standard.CompressContent 
> CompressContent[id=a07f7ae5-018e-1000-c82c-2e2143f9c320] Processing halted: 
> yielding [1 sec]
> java.lang.NoClassDefFoundError: com/github/luben/zstd/ZstdOutputStream
> at 
> org.apache.commons.compress.compressors.zstandard.ZstdCompressorOutputStream.(ZstdCompressorOutputStream.java:57)
> at 
> org.apache.nifi.processors.standard.CompressContent$1.process(CompressContent.java:353)
> at 
> org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:3426)
> at 
> org.apache.nifi.processors.standard.CompressContent.onTrigger(CompressContent.java:300)
> at 
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at 
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1274)
> at 
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:244)
> at 
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:102)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at 
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
> at 
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358)
> at 
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> at 
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
> at 
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> Caused by: java.lang.ClassNotFoundException: 
> com.github.luben.zstd.ZstdOutputStream
> at 
> java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:593)
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)
> ... 15 common frames omitted
> {noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Re: [PR] WIP NIFI-12996 removing commons-compress from shared nar/bom and ensuring… [nifi]

2024-04-03 Thread via GitHub


exceptionfactory closed pull request #8598: WIP NIFI-12996 removing 
commons-compress from shared nar/bom and ensuring…
URL: https://github.com/apache/nifi/pull/8598


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Commented] (NIFI-12996) nifi-shared-bom construct broke classloading for optional dependencies of common-compress including zstd

2024-04-03 Thread ASF subversion and git services (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17833793#comment-17833793
 ] 

ASF subversion and git services commented on NIFI-12996:


Commit 98c4061cfee78590a7d391e53a835c5b87ca4bed in nifi's branch 
refs/heads/main from Joe Witt
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=98c4061cfe ]

NIFI-12996 Moved zstd-jni to standard-shared-nar

- zstd-jni must be present in the same NAR as commons-compress

This closes #8598

Signed-off-by: David Handermann 


> nifi-shared-bom construct broke classloading for optional dependencies of 
> common-compress including zstd
> 
>
> Key: NIFI-12996
> URL: https://issues.apache.org/jira/browse/NIFI-12996
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.24.0, 1.25.0, 2.0.0-M2
>Reporter: Joe Witt
>Assignee: Joe Witt
>Priority: Critical
> Fix For: 2.0.0-M3, 1.26.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> In Apache Slack a user reported this was broken for them in NiFi 1.25.0 but 
> used to work in 1.19.1.
> On apache 2.0 m2+latest I can reproduce this by simply trying to use it at 
> all.  First question is do we not have unit tests but anyway this needs to be 
> evaluated as to why it isn't working.
> {noformat}
> 2024-04-02 13:30:12,514 ERROR [Timer-Driven Process Thread-3] 
> o.a.n.p.standard.CompressContent 
> CompressContent[id=a07f7ae5-018e-1000-c82c-2e2143f9c320] Processing halted: 
> yielding [1 sec]
> java.lang.NoClassDefFoundError: com/github/luben/zstd/ZstdOutputStream
> at 
> org.apache.commons.compress.compressors.zstandard.ZstdCompressorOutputStream.(ZstdCompressorOutputStream.java:57)
> at 
> org.apache.nifi.processors.standard.CompressContent$1.process(CompressContent.java:353)
> at 
> org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:3426)
> at 
> org.apache.nifi.processors.standard.CompressContent.onTrigger(CompressContent.java:300)
> at 
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at 
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1274)
> at 
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:244)
> at 
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:102)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at 
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
> at 
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358)
> at 
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> at 
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
> at 
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> Caused by: java.lang.ClassNotFoundException: 
> com.github.luben.zstd.ZstdOutputStream
> at 
> java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:593)
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)
> ... 15 common frames omitted
> {noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12998) maxmind geoip2 does not belong in nifi-nar-bundle pom

2024-04-03 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-12998:

Fix Version/s: (was: 2.0.0-M3)

> maxmind geoip2 does not belong in nifi-nar-bundle pom
> -
>
> Key: NIFI-12998
> URL: https://issues.apache.org/jira/browse/NIFI-12998
> Project: Apache NiFi
>  Issue Type: Task
>Reporter: Joe Witt
>Assignee: Joe Witt
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> found in nifi-nar-bundles pom
>   
> com.maxmind.geoip2
> geoip2
> 4.2.0
> 
> This should not be here.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12998) maxmind geoip2 does not belong in nifi-nar-bundle pom

2024-04-03 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-12998:

Affects Version/s: 2.0.0-M2

> maxmind geoip2 does not belong in nifi-nar-bundle pom
> -
>
> Key: NIFI-12998
> URL: https://issues.apache.org/jira/browse/NIFI-12998
> Project: Apache NiFi
>  Issue Type: Task
>Affects Versions: 2.0.0-M2
>Reporter: Joe Witt
>Assignee: Joe Witt
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> found in nifi-nar-bundles pom
>   
> com.maxmind.geoip2
> geoip2
> 4.2.0
> 
> This should not be here.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12998) maxmind geoip2 does not belong in nifi-nar-bundle pom

2024-04-03 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-12998:

Status: Patch Available  (was: Open)

> maxmind geoip2 does not belong in nifi-nar-bundle pom
> -
>
> Key: NIFI-12998
> URL: https://issues.apache.org/jira/browse/NIFI-12998
> Project: Apache NiFi
>  Issue Type: Task
>Reporter: Joe Witt
>Assignee: Joe Witt
>Priority: Major
> Fix For: 2.0.0-M3
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> found in nifi-nar-bundles pom
>   
> com.maxmind.geoip2
> geoip2
> 4.2.0
> 
> This should not be here.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[PR] NIFI-12998 clean up version reference [nifi]

2024-04-03 Thread via GitHub


joewitt opened a new pull request, #8602:
URL: https://github.com/apache/nifi/pull/8602

   
   
   
   
   
   
   
   
   
   
   
   
   
   # Summary
   
   [NIFI-0](https://issues.apache.org/jira/browse/NIFI-0)
   
   # Tracking
   
   Please complete the following tracking steps prior to pull request creation.
   
   ### Issue Tracking
   
   - [ ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue 
created
   
   ### Pull Request Tracking
   
   - [ ] Pull Request title starts with Apache NiFi Jira issue number, such as 
`NIFI-0`
   - [ ] Pull Request commit message starts with Apache NiFi Jira issue number, 
as such `NIFI-0`
   
   ### Pull Request Formatting
   
   - [ ] Pull Request based on current revision of the `main` branch
   - [ ] Pull Request refers to a feature branch with one commit containing 
changes
   
   # Verification
   
   Please indicate the verification steps performed prior to pull request 
creation.
   
   ### Build
   
   - [ ] Build completed using `mvn clean install -P contrib-check`
 - [ ] JDK 21
   
   ### Licensing
   
   - [ ] New dependencies are compatible with the [Apache License 
2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License 
Policy](https://www.apache.org/legal/resolved.html)
   - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` 
files
   
   ### Documentation
   
   - [ ] Documentation formatting appears as expected in rendered files
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Updated] (NIFI-12969) Under heavy load, nifi node unable to rejoin cluster, graph modified with temp funnel

2024-04-03 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-12969:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Under heavy load, nifi node unable to rejoin cluster, graph modified with 
> temp funnel
> -
>
> Key: NIFI-12969
> URL: https://issues.apache.org/jira/browse/NIFI-12969
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.24.0, 2.0.0-M2
>Reporter: Nissim Shiman
>Assignee: Mark Payne
>Priority: Critical
> Fix For: 2.0.0-M3, 1.26.0
>
> Attachments: nifi-app.log, simple_flow.png, 
> simple_flow_with_temp-funnel.png
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Under heavy load, if a node leaves the cluster (due to heartbeat time out), 
> many times it is unable to rejoin the cluster.
> The nodes' graph will have been modified with a temp-funnel as well.
> Appears to be some sort of [timing 
> issue|https://github.com/apache/nifi/blob/rel/nifi-2.0.0-M2/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/connectable/StandardConnection.java#L298]
>  # To reproduce, on a nifi cluster of three nodes, set up:
> 2 GenerateFlowFile processors -> PG
> Inside PG:
> inputPort -> UpdateAttribute
>  # Keep all defaults except for the following:
> For UpdateAttribute terminate the success relationship
> One of the GenerateFlowFile processors can be disabled,
> the other one should have Run Schedule to be 0 min (this will allow for the 
> heavy load)
>  # In nifi.properties (on all 3 nodes) to allow for nodes to fall out of the 
> cluster, set: nifi.cluster.protocol.heartbeat.interval=2 sec  (default is 5) 
> nifi.cluster.protocol.heartbeat.missable.max=1   (default is 8)
> Restart nifi. Start flow. The nodes will quickly fall out and rejoin cluster. 
> After a few minutes one will likely not be able to rejoin.  The graph for 
> that node will have the disabled GenerateFlowFile now pointing to a funnel (a 
> temp-funnel) instead of the PG
> Stack trace on that nodes nifi-app.log will look like this: (this is from 
> 2.0.0-M2):
> {code:java}
> 2024-03-28 13:55:19,395 INFO [Reconnect to Cluster] 
> o.a.nifi.controller.StandardFlowService Node disconnected due to Failed to 
> properly handle Reconnection request due to org.apache.nifi.control
> ler.serialization.FlowSynchronizationException: Failed to connect node to 
> cluster because local flow controller partially updated. Administrator should 
> disconnect node and review flow for corrup
> tion.
> 2024-03-28 13:55:19,395 ERROR [Reconnect to Cluster] 
> o.a.nifi.controller.StandardFlowService Handling reconnection request failed 
> due to: org.apache.nifi.controller.serialization.FlowSynchroniza
> tionException: Failed to connect node to cluster because local flow 
> controller partially updated. Administrator should disconnect node and review 
> flow for corruption.
> org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed 
> to connect node to cluster because local flow controller partially updated. 
> Administrator should disconnect node and
>  review flow for corruption.
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:985)
> at 
> org.apache.nifi.controller.StandardFlowService.handleReconnectionRequest(StandardFlowService.java:655)
> at 
> org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:384)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> Caused by: 
> org.apache.nifi.controller.serialization.FlowSynchronizationException: 
> java.lang.IllegalStateException: Cannot change destination of Connection 
> because FlowFiles from this Connection
> are currently held by LocalPort[id=99213c00-78ca-4848-112f-5454cc20656b, 
> type=INPUT_PORT, name=inputPort, group=innerPG]
> at 
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:472)
> at 
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.sync(VersionedFlowSynchronizer.java:223)
> at 
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1740)
> at 
> org.apache.nifi.persistence.StandardFlowConfigurationDAO.load(StandardFlowConfigurationDAO.java:91)
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:805)
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:954)
> ... 3 common frames omitted
> Caused by: java.lang.IllegalStateException: Cannot change destination of 
> Connection because FlowFiles from t

[jira] [Commented] (NIFI-12969) Under heavy load, nifi node unable to rejoin cluster, graph modified with temp funnel

2024-04-03 Thread ASF subversion and git services (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17833780#comment-17833780
 ] 

ASF subversion and git services commented on NIFI-12969:


Commit 212763dd79b3652c9870ff6a59db28c232113c55 in nifi's branch 
refs/heads/support/nifi-1.x from Mark Payne
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=212763dd79 ]

NIFI-12969: Fixed a typo in the #isTempDestinationNecessary method, where we 
were comparing existingConnection.getProcessGroup() to 
newDestination.getProcessGroup(), instead of comparing 
existingConnection.getDestination().getProcessGroup() to 
newDestination.getProcessGroup(). I.e., we were comparing the Destination's PG 
to the Connection's PG instead of comparing Destination's PG to Destination's 
PG. This resulted in using a temporary funnel when we shouldn't. And as a 
result it attempted to change a Connection's Destination while the destination 
was running.

Signed-off-by: Joseph Witt 


> Under heavy load, nifi node unable to rejoin cluster, graph modified with 
> temp funnel
> -
>
> Key: NIFI-12969
> URL: https://issues.apache.org/jira/browse/NIFI-12969
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.24.0, 2.0.0-M2
>Reporter: Nissim Shiman
>Assignee: Mark Payne
>Priority: Critical
> Fix For: 2.0.0-M3, 1.26.0
>
> Attachments: nifi-app.log, simple_flow.png, 
> simple_flow_with_temp-funnel.png
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Under heavy load, if a node leaves the cluster (due to heartbeat time out), 
> many times it is unable to rejoin the cluster.
> The nodes' graph will have been modified with a temp-funnel as well.
> Appears to be some sort of [timing 
> issue|https://github.com/apache/nifi/blob/rel/nifi-2.0.0-M2/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/connectable/StandardConnection.java#L298]
>  # To reproduce, on a nifi cluster of three nodes, set up:
> 2 GenerateFlowFile processors -> PG
> Inside PG:
> inputPort -> UpdateAttribute
>  # Keep all defaults except for the following:
> For UpdateAttribute terminate the success relationship
> One of the GenerateFlowFile processors can be disabled,
> the other one should have Run Schedule to be 0 min (this will allow for the 
> heavy load)
>  # In nifi.properties (on all 3 nodes) to allow for nodes to fall out of the 
> cluster, set: nifi.cluster.protocol.heartbeat.interval=2 sec  (default is 5) 
> nifi.cluster.protocol.heartbeat.missable.max=1   (default is 8)
> Restart nifi. Start flow. The nodes will quickly fall out and rejoin cluster. 
> After a few minutes one will likely not be able to rejoin.  The graph for 
> that node will have the disabled GenerateFlowFile now pointing to a funnel (a 
> temp-funnel) instead of the PG
> Stack trace on that nodes nifi-app.log will look like this: (this is from 
> 2.0.0-M2):
> {code:java}
> 2024-03-28 13:55:19,395 INFO [Reconnect to Cluster] 
> o.a.nifi.controller.StandardFlowService Node disconnected due to Failed to 
> properly handle Reconnection request due to org.apache.nifi.control
> ler.serialization.FlowSynchronizationException: Failed to connect node to 
> cluster because local flow controller partially updated. Administrator should 
> disconnect node and review flow for corrup
> tion.
> 2024-03-28 13:55:19,395 ERROR [Reconnect to Cluster] 
> o.a.nifi.controller.StandardFlowService Handling reconnection request failed 
> due to: org.apache.nifi.controller.serialization.FlowSynchroniza
> tionException: Failed to connect node to cluster because local flow 
> controller partially updated. Administrator should disconnect node and review 
> flow for corruption.
> org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed 
> to connect node to cluster because local flow controller partially updated. 
> Administrator should disconnect node and
>  review flow for corruption.
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:985)
> at 
> org.apache.nifi.controller.StandardFlowService.handleReconnectionRequest(StandardFlowService.java:655)
> at 
> org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:384)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> Caused by: 
> org.apache.nifi.controller.serialization.FlowSynchronizationException: 
> java.lang.IllegalStateException: Cannot change destination of Connection 
> because FlowFiles from this Connection
> are currently held by LocalPort[id=99213c00-78ca-4848-112f-5454cc20656b, 
> type=INPUT_PORT, name=inputPort, group=innerPG]
> at 
> org.apache.nifi.cont

Re: [PR] NIFI-12969: Fixed a typo in the #isTempDestinationNecessary method, w… [nifi]

2024-04-03 Thread via GitHub


asfgit closed pull request #8600: NIFI-12969: Fixed a typo in the 
#isTempDestinationNecessary method, w…
URL: https://github.com/apache/nifi/pull/8600


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Commented] (NIFI-12969) Under heavy load, nifi node unable to rejoin cluster, graph modified with temp funnel

2024-04-03 Thread ASF subversion and git services (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17833773#comment-17833773
 ] 

ASF subversion and git services commented on NIFI-12969:


Commit 3a03caa14968d0c9a2e6790373871e886b632f09 in nifi's branch 
refs/heads/main from Mark Payne
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=3a03caa149 ]

NIFI-12969: Fixed a typo in the #isTempDestinationNecessary method, where we 
were comparing existingConnection.getProcessGroup() to 
newDestination.getProcessGroup(), instead of comparing 
existingConnection.getDestination().getProcessGroup() to 
newDestination.getProcessGroup(). I.e., we were comparing the Destination's PG 
to the Connection's PG instead of comparing Destination's PG to Destination's 
PG. This resulted in using a temporary funnel when we shouldn't. And as a 
result it attempted to change a Connection's Destination while the destination 
was running.

This closes #8600.

Signed-off-by: Joseph Witt 


> Under heavy load, nifi node unable to rejoin cluster, graph modified with 
> temp funnel
> -
>
> Key: NIFI-12969
> URL: https://issues.apache.org/jira/browse/NIFI-12969
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.24.0, 2.0.0-M2
>Reporter: Nissim Shiman
>Assignee: Mark Payne
>Priority: Critical
> Fix For: 2.0.0-M3, 1.26.0
>
> Attachments: nifi-app.log, simple_flow.png, 
> simple_flow_with_temp-funnel.png
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Under heavy load, if a node leaves the cluster (due to heartbeat time out), 
> many times it is unable to rejoin the cluster.
> The nodes' graph will have been modified with a temp-funnel as well.
> Appears to be some sort of [timing 
> issue|https://github.com/apache/nifi/blob/rel/nifi-2.0.0-M2/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/connectable/StandardConnection.java#L298]
>  # To reproduce, on a nifi cluster of three nodes, set up:
> 2 GenerateFlowFile processors -> PG
> Inside PG:
> inputPort -> UpdateAttribute
>  # Keep all defaults except for the following:
> For UpdateAttribute terminate the success relationship
> One of the GenerateFlowFile processors can be disabled,
> the other one should have Run Schedule to be 0 min (this will allow for the 
> heavy load)
>  # In nifi.properties (on all 3 nodes) to allow for nodes to fall out of the 
> cluster, set: nifi.cluster.protocol.heartbeat.interval=2 sec  (default is 5) 
> nifi.cluster.protocol.heartbeat.missable.max=1   (default is 8)
> Restart nifi. Start flow. The nodes will quickly fall out and rejoin cluster. 
> After a few minutes one will likely not be able to rejoin.  The graph for 
> that node will have the disabled GenerateFlowFile now pointing to a funnel (a 
> temp-funnel) instead of the PG
> Stack trace on that nodes nifi-app.log will look like this: (this is from 
> 2.0.0-M2):
> {code:java}
> 2024-03-28 13:55:19,395 INFO [Reconnect to Cluster] 
> o.a.nifi.controller.StandardFlowService Node disconnected due to Failed to 
> properly handle Reconnection request due to org.apache.nifi.control
> ler.serialization.FlowSynchronizationException: Failed to connect node to 
> cluster because local flow controller partially updated. Administrator should 
> disconnect node and review flow for corrup
> tion.
> 2024-03-28 13:55:19,395 ERROR [Reconnect to Cluster] 
> o.a.nifi.controller.StandardFlowService Handling reconnection request failed 
> due to: org.apache.nifi.controller.serialization.FlowSynchroniza
> tionException: Failed to connect node to cluster because local flow 
> controller partially updated. Administrator should disconnect node and review 
> flow for corruption.
> org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed 
> to connect node to cluster because local flow controller partially updated. 
> Administrator should disconnect node and
>  review flow for corruption.
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:985)
> at 
> org.apache.nifi.controller.StandardFlowService.handleReconnectionRequest(StandardFlowService.java:655)
> at 
> org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:384)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> Caused by: 
> org.apache.nifi.controller.serialization.FlowSynchronizationException: 
> java.lang.IllegalStateException: Cannot change destination of Connection 
> because FlowFiles from this Connection
> are currently held by LocalPort[id=99213c00-78ca-4848-112f-5454cc20656b, 
> type=INPUT_PORT, name=inputPort, group=innerPG]
> at 
> org.apache.n

Re: [PR] WIP NIFI-12996 removing commons-compress from shared nar/bom and ensuring… [nifi]

2024-04-03 Thread via GitHub


joewitt commented on PR #8598:
URL: https://github.com/apache/nifi/pull/8598#issuecomment-2035807614

   @exceptionfactory Ok I believe I've updated to what you've described.  
Tested on a live install and it works as well.  Checked where commons-compress 
and zstd libs are now as well and checks out
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Created] (NIFI-12998) maxmind geoip2 does not belong in nifi-nar-bundle pom

2024-04-03 Thread Joe Witt (Jira)
Joe Witt created NIFI-12998:
---

 Summary: maxmind geoip2 does not belong in nifi-nar-bundle pom
 Key: NIFI-12998
 URL: https://issues.apache.org/jira/browse/NIFI-12998
 Project: Apache NiFi
  Issue Type: Task
Reporter: Joe Witt
Assignee: Joe Witt
 Fix For: 2.0.0-M3


found in nifi-nar-bundles pom

  
com.maxmind.geoip2
geoip2
4.2.0


This should not be here.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (NIFI-12972) UI - Improve listing of relationships for a connection

2024-04-03 Thread ASF subversion and git services (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17833764#comment-17833764
 ] 

ASF subversion and git services commented on NIFI-12972:


Commit 644d086ff2d8d98bb8c31e714e1a254493f9daf3 in nifi's branch 
refs/heads/main from Pierre Villard
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=644d086ff2 ]

NIFI-12972 - Only show selected relationships in read-only connection details 
(#8582)

* NIFI-12972 - Only show selected relationships in read-only connection details

* review

This closes #8582 

> UI - Improve listing of relationships for a connection
> --
>
> Key: NIFI-12972
> URL: https://issues.apache.org/jira/browse/NIFI-12972
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core UI
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Consider a RouteOnAttribute processor with "Route to Property name" with many 
> dynamic properties. This will lead to many relationships (r1, r2, r3, etc). 
> Now consider an UpdateAttribute processor.
> Connect RouteOnAttribute to this UpdateAttribute processor and select r1 and 
> r3.
> If the two processors are stopped, opening the configuration of the 
> connection will show an alphabetically sorted list of the ALL the 
> relationships (of the RouteOnAttribute processor) with checkboxes and r1/r3 
> will show as checked.
> Now, if one of the two processors is started, opening the configuration of 
> the connection will show an alphabetically sorted list of the ALL the 
> relationships (of the RouteOnAttribute processor) and the only distinction 
> will be that r1 and r3 will be in bold.
> When having many connections (think 50+), just showing the selected 
> relationships in bold is not super user friendly. It's really hard to 
> distinguish the bold names in a very long list of names.
> When one of the processors is running and we're just showing the details of 
> the connection without the option to change the list of selected 
> relationships for that connection, I suggest that we only display the list of 
> the selected relationships.
> For reference, the related code is here:
> https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/js/nf/nf-connection-details.js#L520



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12972) UI - Improve listing of relationships for a connection

2024-04-03 Thread Matt Gilman (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Gilman updated NIFI-12972:
---
Fix Version/s: 2.0.0-M3
   Resolution: Fixed
   Status: Resolved  (was: Patch Available)

> UI - Improve listing of relationships for a connection
> --
>
> Key: NIFI-12972
> URL: https://issues.apache.org/jira/browse/NIFI-12972
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core UI
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
> Fix For: 2.0.0-M3
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Consider a RouteOnAttribute processor with "Route to Property name" with many 
> dynamic properties. This will lead to many relationships (r1, r2, r3, etc). 
> Now consider an UpdateAttribute processor.
> Connect RouteOnAttribute to this UpdateAttribute processor and select r1 and 
> r3.
> If the two processors are stopped, opening the configuration of the 
> connection will show an alphabetically sorted list of the ALL the 
> relationships (of the RouteOnAttribute processor) with checkboxes and r1/r3 
> will show as checked.
> Now, if one of the two processors is started, opening the configuration of 
> the connection will show an alphabetically sorted list of the ALL the 
> relationships (of the RouteOnAttribute processor) and the only distinction 
> will be that r1 and r3 will be in bold.
> When having many connections (think 50+), just showing the selected 
> relationships in bold is not super user friendly. It's really hard to 
> distinguish the bold names in a very long list of names.
> When one of the processors is running and we're just showing the details of 
> the connection without the option to change the list of selected 
> relationships for that connection, I suggest that we only display the list of 
> the selected relationships.
> For reference, the related code is here:
> https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/js/nf/nf-connection-details.js#L520



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Re: [PR] NIFI-12972 - Only show selected relationships in read-only connection details [nifi]

2024-04-03 Thread via GitHub


mcgilman merged PR #8582:
URL: https://github.com/apache/nifi/pull/8582


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Commented] (NIFI-12969) Under heavy load, nifi node unable to rejoin cluster, graph modified with temp funnel

2024-04-03 Thread Peter Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17833763#comment-17833763
 ] 

Peter Gyori commented on NIFI-12969:


Thank you [~markap14] ! It makes sense. I did not catch that creating the temp 
funnel wasn't necessary in the first place.

> Under heavy load, nifi node unable to rejoin cluster, graph modified with 
> temp funnel
> -
>
> Key: NIFI-12969
> URL: https://issues.apache.org/jira/browse/NIFI-12969
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.24.0, 2.0.0-M2
>Reporter: Nissim Shiman
>Assignee: Mark Payne
>Priority: Critical
> Fix For: 2.0.0-M3, 1.26.0
>
> Attachments: nifi-app.log, simple_flow.png, 
> simple_flow_with_temp-funnel.png
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Under heavy load, if a node leaves the cluster (due to heartbeat time out), 
> many times it is unable to rejoin the cluster.
> The nodes' graph will have been modified with a temp-funnel as well.
> Appears to be some sort of [timing 
> issue|https://github.com/apache/nifi/blob/rel/nifi-2.0.0-M2/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/connectable/StandardConnection.java#L298]
>  # To reproduce, on a nifi cluster of three nodes, set up:
> 2 GenerateFlowFile processors -> PG
> Inside PG:
> inputPort -> UpdateAttribute
>  # Keep all defaults except for the following:
> For UpdateAttribute terminate the success relationship
> One of the GenerateFlowFile processors can be disabled,
> the other one should have Run Schedule to be 0 min (this will allow for the 
> heavy load)
>  # In nifi.properties (on all 3 nodes) to allow for nodes to fall out of the 
> cluster, set: nifi.cluster.protocol.heartbeat.interval=2 sec  (default is 5) 
> nifi.cluster.protocol.heartbeat.missable.max=1   (default is 8)
> Restart nifi. Start flow. The nodes will quickly fall out and rejoin cluster. 
> After a few minutes one will likely not be able to rejoin.  The graph for 
> that node will have the disabled GenerateFlowFile now pointing to a funnel (a 
> temp-funnel) instead of the PG
> Stack trace on that nodes nifi-app.log will look like this: (this is from 
> 2.0.0-M2):
> {code:java}
> 2024-03-28 13:55:19,395 INFO [Reconnect to Cluster] 
> o.a.nifi.controller.StandardFlowService Node disconnected due to Failed to 
> properly handle Reconnection request due to org.apache.nifi.control
> ler.serialization.FlowSynchronizationException: Failed to connect node to 
> cluster because local flow controller partially updated. Administrator should 
> disconnect node and review flow for corrup
> tion.
> 2024-03-28 13:55:19,395 ERROR [Reconnect to Cluster] 
> o.a.nifi.controller.StandardFlowService Handling reconnection request failed 
> due to: org.apache.nifi.controller.serialization.FlowSynchroniza
> tionException: Failed to connect node to cluster because local flow 
> controller partially updated. Administrator should disconnect node and review 
> flow for corruption.
> org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed 
> to connect node to cluster because local flow controller partially updated. 
> Administrator should disconnect node and
>  review flow for corruption.
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:985)
> at 
> org.apache.nifi.controller.StandardFlowService.handleReconnectionRequest(StandardFlowService.java:655)
> at 
> org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:384)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> Caused by: 
> org.apache.nifi.controller.serialization.FlowSynchronizationException: 
> java.lang.IllegalStateException: Cannot change destination of Connection 
> because FlowFiles from this Connection
> are currently held by LocalPort[id=99213c00-78ca-4848-112f-5454cc20656b, 
> type=INPUT_PORT, name=inputPort, group=innerPG]
> at 
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:472)
> at 
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.sync(VersionedFlowSynchronizer.java:223)
> at 
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1740)
> at 
> org.apache.nifi.persistence.StandardFlowConfigurationDAO.load(StandardFlowConfigurationDAO.java:91)
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:805)
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:954)
> ... 3 common frames omit

Re: [PR] WIP NIFI-12996 removing commons-compress from shared nar/bom and ensuring… [nifi]

2024-04-03 Thread via GitHub


joewitt commented on PR #8598:
URL: https://github.com/apache/nifi/pull/8598#issuecomment-2035658402

   paraphrasing you're basically saying "Hey this is great.  How about doing it 
exactly the opposite way" :).  Haha all good.  You're probably right.  Will 
give that a go.  only means we need zstd as the other deps are direct deps and 
bypass commons compress anyway.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Updated] (NIFI-12400) Remaining items to migrate UI to currently supported/active framework

2024-04-03 Thread Matt Gilman (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Gilman updated NIFI-12400:
---
Description: 
The purpose of this Jira is to track all remaining items following the initial 
commit [1] for NIFI-11481. The description will be kept up to date with 
remaining features, tasks, and improvements. As each items is worked, a new sub 
task Jira will be created and referenced in this description.
 * Support Parameters in Properties with Allowable Values (NIFI-12401)
 * Summary (NIFI-12437)
 ** Remaining work not addressed in initial Jira:
 *** input ports (NIFI-12504)
 *** output ports (NIFI-12504)
 *** remote process groups (NIFI-12504)
 *** process groups (NIFI-12504)
 *** connections (NIFI-12504)
 *** System Diagnostics (NIFI-12505)
 *** support for cluster-specific ui elements (NIFI-12537)
 *** Add pagination (NIFI-12552)
 * Counters (NIFI-12415)
 ** Counter table has extra unnecessary can modify check (NIFI-12948)
 * Bulletin Board (NIFI-12560)
 * Provenance (NIFI-12445)
 ** Event Listing (NIFI-12445)
 ** Search (NIFI-12445)
 ** Event Dialog (NIFI-12445)
 ** Lineage (NIFI-12485)
 ** Replay from context menu (NIFI-12445)
 ** Clustering (NIFI-12807)

 * Configure Reporting Task (NIFI-12563)
 * Flow Analysis Rules (NIFI-12588)
 * Registry Clients (NIFI-12486)
 * Import from Registry (NIFI-12734)
 * Parameter Providers (NIFI-12622)
 ** Fetch parameters from provider, map to parameter context (dialog) - 
(NIFI-12665)
 * Cluster
 ** Status History - node specific values (NIFI-12848)
 * Flow Configuration History (NIFI-12754)
 ** ActionEntity.action should be optional (NIFI-12948)
 * Node Status History (NIFI-12553)
 * Status history for components from canvas context menu (NIFI-12553)
 * Users (NIFI-12543)
 ** Don't show users or groups in create/edit dialog is there are none 
(NIFI-12948)
 * Policies (NIFI-12548)
 ** Overridden policy Empty or Copy (NIFI-12679)
 ** Select Empty by default (NIFI-12948)
 * Help (NIFI-12795)
 * About
 * Show Upstream/Downstream
 * Align
 * List Queue (NIFI-12589)
 ** Clustering (NIFI-12807)
 * Empty [all] Queue (NIFI-12604)
 * View Content (NIFI-12589 and NIFI-12445)
 * View State (NIFI-12611)
 ** Clustering
 * Change Component Version
 * Consider PG permissions in Toolbox (NIFI-12683)
 * Handle linking to components that are not on the canvas
 * PG Version (NIFI-12963 & NIFI-12995)
 ** Start (NIFI-12963)
 ** Commit (NIFI-12963)
 ** Force Commit (NIFI-12963)
 ** Show changes (NIFI-12995)
 ** Revert changes (NIFI-12995)
 ** Change Flow version (NIFI-12995)
 ** Stop (NIFI-12963)

 * Configure PG (NIFI-12417)
 * Process Group Services (NIFI-12425)
 ** Listing (NIFI-12425)
 ** Create (NIFI-12425)
 ** Configure (NIFI-12425)
 ** Delete (NIFI-12425)
 ** Enable (NIFI-12529)
 ** Disable (NIFI-12529)
 ** Improve layout and breadcrumbs
 ** Disable and Configure
 * Configure Processor
 ** Service Link (NIFI-12425)
 ** Create inline Service (NIFI-12425)
 ** Parameter Link (NIFI-12502)
 ** Convert to Parameter (NIFI-12502)
 ** Fix issue with Property Editor width (NIFI-12547)
 ** Stop and Configure
 ** Open Custom UI (NIFI-12958)
 ** Property History
 ** Unable to re-add any removed Property (NIFI-12743)
 ** Shift-Enter new line when editing Property (NIFI-12743)
 * Property Verification
 * More Details (Processor, Controller Service, Reporting Task)

 * Download Flow
 * Create RPG (NIFI-12758)
 * Configure RPG (NIFI-12774)
 * RPG Remote Ports (NIFI-12778)
 * RPG Go To (NIFI-12759)
 * RPG Refresh (NIFI-12761)
 * Color
 * Move to Front
 * Copy/Paste
 * Add/Update Info Icons in dialogs throughout the application
 * Set viewport earlier when loading a Process Group (NIFI-12737)
 * Canvas global menu item should navigate user back to where they were on the 
canvas (NIFI-12737)
 * Better theme support (NIFI-12655)
 * Set up development/production environments files
 * Run unit tests are part of standard build (NIFI-12941)
 * Update all API calls to consider disconnect node confirmation
 * Update API calls to use uiOnly flag (NIFI-12950)
 * Use polling interval from API
 * Load FlowConfiguration in guard (NIFI-12948)
 * Routing error handling
 * General API response error handling
 ** Management CS (NIFI-12663)
 ** Canvas CS (NIFI-12684)
 ** Remainder of Settings (NIFI-12723)
 ** Counters (NIFI-12723)
 ** Bulletins (NIFI-12723)
 ** Flow Designer
 ** Parameter Contexts (NIFI-12937)
 ** Parameter
 ** Provenance (NIFI-12767)
 ** Queue Listing (NIFI-12742)
 ** Summary (NIFI-12742)
 ** Users (NIFI-12742)
 ** Policies
 ** Status History
 * Introduce header in new pages to unify with canvas and offer better 
navigation. (NIFI-12597)
 * Theme docs, view flow file, and custom ui's
 * Prompt user to save Parameter Context when Edit form is dirty
 * Upgrade to Angular 17 (NIFI-12790)
 * Start/Stop processors, process groups, ... (NIFI-12568)
 * Dialog vertical resizing on smaller screens do not allow 

[jira] [Resolved] (NIFI-12551) Fix dialog resizing when changing tabs

2024-04-03 Thread Matt Gilman (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Gilman resolved NIFI-12551.

Fix Version/s: 2.0.0-M3
   Resolution: Fixed

> Fix dialog resizing when changing tabs
> --
>
> Key: NIFI-12551
> URL: https://issues.apache.org/jira/browse/NIFI-12551
> Project: Apache NiFi
>  Issue Type: Sub-task
>Reporter: Rob Fellows
>Priority: Major
> Fix For: 2.0.0-M3
>
> Attachments: Kapture 2023-12-27 at 11.46.46.gif
>
>
> The processor configuration dialog seems to adjust size when changing tabs. 
> See the attached screen recording.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Resolved] (NIFI-12603) Dialog vertical resizing on smaller screens do not allow users to access all fields

2024-04-03 Thread Matt Gilman (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Gilman resolved NIFI-12603.

Fix Version/s: 2.0.0-M3
   Resolution: Fixed

> Dialog vertical resizing on smaller screens do not allow users to access all 
> fields
> ---
>
> Key: NIFI-12603
> URL: https://issues.apache.org/jira/browse/NIFI-12603
> Project: Apache NiFi
>  Issue Type: Sub-task
>Reporter: Scott Aslan
>Priority: Major
> Fix For: 2.0.0-M3
>
> Attachments: Kapture 2024-01-12 at 11.33.57.gif
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (NIFI-12969) Under heavy load, nifi node unable to rejoin cluster, graph modified with temp funnel

2024-04-03 Thread Mark Payne (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17833738#comment-17833738
 ] 

Mark Payne commented on NIFI-12969:
---

[~Nissim Shiman] [~pgyori] I pushed a PR that appears to address the issue. I 
believe you're on the right track, that the situation is caused by the fact 
that the temp funnel was incorrectly used. But instead of trying to detect when 
it's going to happen and/or rollback, the issue is that we had a bug in the 
logic for when the temp funnel was created. In this case, there should never be 
a temp funnel. In cases where we DO need a temp funnel, the existing logic 
should handle stopping the Port, which would make this work smoothly. The issue 
arose here because the Port was (rightly) left running. We just need to avoid 
creating the temp funnel unnecessarily.

> Under heavy load, nifi node unable to rejoin cluster, graph modified with 
> temp funnel
> -
>
> Key: NIFI-12969
> URL: https://issues.apache.org/jira/browse/NIFI-12969
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.24.0, 2.0.0-M2
>Reporter: Nissim Shiman
>Assignee: Mark Payne
>Priority: Critical
> Fix For: 2.0.0-M3, 1.26.0
>
> Attachments: nifi-app.log, simple_flow.png, 
> simple_flow_with_temp-funnel.png
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Under heavy load, if a node leaves the cluster (due to heartbeat time out), 
> many times it is unable to rejoin the cluster.
> The nodes' graph will have been modified with a temp-funnel as well.
> Appears to be some sort of [timing 
> issue|https://github.com/apache/nifi/blob/rel/nifi-2.0.0-M2/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/connectable/StandardConnection.java#L298]
>  # To reproduce, on a nifi cluster of three nodes, set up:
> 2 GenerateFlowFile processors -> PG
> Inside PG:
> inputPort -> UpdateAttribute
>  # Keep all defaults except for the following:
> For UpdateAttribute terminate the success relationship
> One of the GenerateFlowFile processors can be disabled,
> the other one should have Run Schedule to be 0 min (this will allow for the 
> heavy load)
>  # In nifi.properties (on all 3 nodes) to allow for nodes to fall out of the 
> cluster, set: nifi.cluster.protocol.heartbeat.interval=2 sec  (default is 5) 
> nifi.cluster.protocol.heartbeat.missable.max=1   (default is 8)
> Restart nifi. Start flow. The nodes will quickly fall out and rejoin cluster. 
> After a few minutes one will likely not be able to rejoin.  The graph for 
> that node will have the disabled GenerateFlowFile now pointing to a funnel (a 
> temp-funnel) instead of the PG
> Stack trace on that nodes nifi-app.log will look like this: (this is from 
> 2.0.0-M2):
> {code:java}
> 2024-03-28 13:55:19,395 INFO [Reconnect to Cluster] 
> o.a.nifi.controller.StandardFlowService Node disconnected due to Failed to 
> properly handle Reconnection request due to org.apache.nifi.control
> ler.serialization.FlowSynchronizationException: Failed to connect node to 
> cluster because local flow controller partially updated. Administrator should 
> disconnect node and review flow for corrup
> tion.
> 2024-03-28 13:55:19,395 ERROR [Reconnect to Cluster] 
> o.a.nifi.controller.StandardFlowService Handling reconnection request failed 
> due to: org.apache.nifi.controller.serialization.FlowSynchroniza
> tionException: Failed to connect node to cluster because local flow 
> controller partially updated. Administrator should disconnect node and review 
> flow for corruption.
> org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed 
> to connect node to cluster because local flow controller partially updated. 
> Administrator should disconnect node and
>  review flow for corruption.
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:985)
> at 
> org.apache.nifi.controller.StandardFlowService.handleReconnectionRequest(StandardFlowService.java:655)
> at 
> org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:384)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> Caused by: 
> org.apache.nifi.controller.serialization.FlowSynchronizationException: 
> java.lang.IllegalStateException: Cannot change destination of Connection 
> because FlowFiles from this Connection
> are currently held by LocalPort[id=99213c00-78ca-4848-112f-5454cc20656b, 
> type=INPUT_PORT, name=inputPort, group=innerPG]
> at 
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:472)
> at 
> org.apache.nifi.controller.serializa

[jira] [Updated] (NIFI-12958) Add support to open custom UIs

2024-04-03 Thread Matt Gilman (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Gilman updated NIFI-12958:
---
Status: Patch Available  (was: In Progress)

> Add support to open custom UIs
> --
>
> Key: NIFI-12958
> URL: https://issues.apache.org/jira/browse/NIFI-12958
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Core UI
>Reporter: Matt Gilman
>Assignee: Matt Gilman
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Allow the user to open a custom UI.
> 1) Open Custom UI from context menu. Not from Advanced button in Configure 
> dialog.
> 2) Use route like the following...
> {noformat}
> /#/process-groups/43eb7385-018e-1000-2fb6-8dbbe88212cd/Processor/7bbab5aa-018e-1000-7de7-dc4052553b02/advanced{noformat}
> 3) Load component in question and using context path from component load UI 
> in iframe using appropriate values for it's query params.
> 4) New page should use the header.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[PR] NIFI-12958: Adding support for custom UIs [nifi]

2024-04-03 Thread via GitHub


mcgilman opened a new pull request, #8601:
URL: https://github.com/apache/nifi/pull/8601

   NIFI-12958:
   - Adding support for custom UIs.
   - Running NiFi dev server at context path /nf.
   - Fixing link used when clicking the logo in the header.
   - Updating titles and icons used for editing components in Settings for 
better consistency.
   - Fixed JOLT advanced UI height.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Updated] (NIFI-12969) Under heavy load, nifi node unable to rejoin cluster, graph modified with temp funnel

2024-04-03 Thread Mark Payne (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Payne updated NIFI-12969:
--
Assignee: Mark Payne  (was: Nissim Shiman)
  Status: Patch Available  (was: Open)

> Under heavy load, nifi node unable to rejoin cluster, graph modified with 
> temp funnel
> -
>
> Key: NIFI-12969
> URL: https://issues.apache.org/jira/browse/NIFI-12969
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 2.0.0-M2, 1.24.0
>Reporter: Nissim Shiman
>Assignee: Mark Payne
>Priority: Critical
> Fix For: 2.0.0-M3, 1.26.0
>
> Attachments: nifi-app.log, simple_flow.png, 
> simple_flow_with_temp-funnel.png
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Under heavy load, if a node leaves the cluster (due to heartbeat time out), 
> many times it is unable to rejoin the cluster.
> The nodes' graph will have been modified with a temp-funnel as well.
> Appears to be some sort of [timing 
> issue|https://github.com/apache/nifi/blob/rel/nifi-2.0.0-M2/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/connectable/StandardConnection.java#L298]
>  # To reproduce, on a nifi cluster of three nodes, set up:
> 2 GenerateFlowFile processors -> PG
> Inside PG:
> inputPort -> UpdateAttribute
>  # Keep all defaults except for the following:
> For UpdateAttribute terminate the success relationship
> One of the GenerateFlowFile processors can be disabled,
> the other one should have Run Schedule to be 0 min (this will allow for the 
> heavy load)
>  # In nifi.properties (on all 3 nodes) to allow for nodes to fall out of the 
> cluster, set: nifi.cluster.protocol.heartbeat.interval=2 sec  (default is 5) 
> nifi.cluster.protocol.heartbeat.missable.max=1   (default is 8)
> Restart nifi. Start flow. The nodes will quickly fall out and rejoin cluster. 
> After a few minutes one will likely not be able to rejoin.  The graph for 
> that node will have the disabled GenerateFlowFile now pointing to a funnel (a 
> temp-funnel) instead of the PG
> Stack trace on that nodes nifi-app.log will look like this: (this is from 
> 2.0.0-M2):
> {code:java}
> 2024-03-28 13:55:19,395 INFO [Reconnect to Cluster] 
> o.a.nifi.controller.StandardFlowService Node disconnected due to Failed to 
> properly handle Reconnection request due to org.apache.nifi.control
> ler.serialization.FlowSynchronizationException: Failed to connect node to 
> cluster because local flow controller partially updated. Administrator should 
> disconnect node and review flow for corrup
> tion.
> 2024-03-28 13:55:19,395 ERROR [Reconnect to Cluster] 
> o.a.nifi.controller.StandardFlowService Handling reconnection request failed 
> due to: org.apache.nifi.controller.serialization.FlowSynchroniza
> tionException: Failed to connect node to cluster because local flow 
> controller partially updated. Administrator should disconnect node and review 
> flow for corruption.
> org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed 
> to connect node to cluster because local flow controller partially updated. 
> Administrator should disconnect node and
>  review flow for corruption.
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:985)
> at 
> org.apache.nifi.controller.StandardFlowService.handleReconnectionRequest(StandardFlowService.java:655)
> at 
> org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:384)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> Caused by: 
> org.apache.nifi.controller.serialization.FlowSynchronizationException: 
> java.lang.IllegalStateException: Cannot change destination of Connection 
> because FlowFiles from this Connection
> are currently held by LocalPort[id=99213c00-78ca-4848-112f-5454cc20656b, 
> type=INPUT_PORT, name=inputPort, group=innerPG]
> at 
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:472)
> at 
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.sync(VersionedFlowSynchronizer.java:223)
> at 
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1740)
> at 
> org.apache.nifi.persistence.StandardFlowConfigurationDAO.load(StandardFlowConfigurationDAO.java:91)
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:805)
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:954)
> ... 3 common frames omitted
> Caused by: java.lang.IllegalStateException: Cannot change destination of 
> Connection b

[PR] NIFI-12969: Fixed a typo in the #isTempDestinationNecessary method, w… [nifi]

2024-04-03 Thread via GitHub


markap14 opened a new pull request, #8600:
URL: https://github.com/apache/nifi/pull/8600

   …here we were comparing existingConnection.getProcessGroup() to 
newDestination.getProcessGroup(), instead of comparing 
existingConnection.getDestination().getProcessGroup() to 
newDestination.getProcessGroup(). I.e., we were comparing the Destination's PG 
to the Connection's PG instead of comparing Destination's PG to Destination's 
PG. This resulted in using a temporary funnel when we shouldn't. And as a 
result it attempted to change a Connection's Destination while the destination 
was running.
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   # Summary
   
   [NIFI-0](https://issues.apache.org/jira/browse/NIFI-0)
   
   # Tracking
   
   Please complete the following tracking steps prior to pull request creation.
   
   ### Issue Tracking
   
   - [ ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue 
created
   
   ### Pull Request Tracking
   
   - [ ] Pull Request title starts with Apache NiFi Jira issue number, such as 
`NIFI-0`
   - [ ] Pull Request commit message starts with Apache NiFi Jira issue number, 
as such `NIFI-0`
   
   ### Pull Request Formatting
   
   - [ ] Pull Request based on current revision of the `main` branch
   - [ ] Pull Request refers to a feature branch with one commit containing 
changes
   
   # Verification
   
   Please indicate the verification steps performed prior to pull request 
creation.
   
   ### Build
   
   - [ ] Build completed using `mvn clean install -P contrib-check`
 - [ ] JDK 21
   
   ### Licensing
   
   - [ ] New dependencies are compatible with the [Apache License 
2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License 
Policy](https://www.apache.org/legal/resolved.html)
   - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` 
files
   
   ### Documentation
   
   - [ ] Documentation formatting appears as expected in rendered files
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Updated] (NIFI-12997) Correct AWS STS Version 2 Location

2024-04-03 Thread David Handermann (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Handermann updated NIFI-12997:

Description: The AWS STS library for Version 2 is included in 
{{nifi-aws-processors}}, but not in {{nifi-aws-service-api}}. The STS library 
needs to be in the same ClassLoader as the AWS Auth library for runtime 
credential loading. Version 1 of the STS library is already included in 
{{nifi-aws-service-api}} and corresponding NAR, which allows S3 Processors to 
work with STS. Other AWS Processors using SDK Version 2 need the STS library 
moved in order to work with this credentials strategy.  (was: The AWS STS 
library for Version 2 is included in {{{}nifi-aws-processors{}}}, but not in 
{{{}nifi-aws-service-api{}}}. The STS library needs to be in the same 
ClassLoader as the AWS Auth library for runtime credential loading. Version 1 
of the STS library is already included in {{nifi-aws-service-api}} and 
corresponding NAR, which allows S3 Processors to work with STS. Other AWS 
Processors using SDK Version 2 need the STS library moved in order to work with 
this credentials strategy.)

> Correct AWS STS Version 2 Location
> --
>
> Key: NIFI-12997
> URL: https://issues.apache.org/jira/browse/NIFI-12997
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The AWS STS library for Version 2 is included in {{nifi-aws-processors}}, but 
> not in {{nifi-aws-service-api}}. The STS library needs to be in the same 
> ClassLoader as the AWS Auth library for runtime credential loading. Version 1 
> of the STS library is already included in {{nifi-aws-service-api}} and 
> corresponding NAR, which allows S3 Processors to work with STS. Other AWS 
> Processors using SDK Version 2 need the STS library moved in order to work 
> with this credentials strategy.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12997) Correct AWS STS Version 2 Location

2024-04-03 Thread David Handermann (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Handermann updated NIFI-12997:

Status: Patch Available  (was: Open)

> Correct AWS STS Version 2 Location
> --
>
> Key: NIFI-12997
> URL: https://issues.apache.org/jira/browse/NIFI-12997
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The AWS STS library for Version 2 is included in {{nifi-aws-processors}}, but 
> not in {{nifi-aws-service-api}}. The STS library needs to be in the same 
> ClassLoader as the AWS Auth library for runtime credential loading. Version 1 
> of the STS library is already included in {{nifi-aws-service-api}} and 
> corresponding NAR, which allows S3 Processors to work with STS. Other AWS 
> Processors using SDK Version 2 need the STS library moved in order to work 
> with this credentials strategy.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[PR] NIFI-12997 Correct AWS STS Version 2 location [nifi]

2024-04-03 Thread via GitHub


exceptionfactory opened a new pull request, #8599:
URL: https://github.com/apache/nifi/pull/8599

   # Summary
   
   [NIFI-12997](https://issues.apache.org/jira/browse/NIFI-12997) Corrects the 
location of the AWS STS Version 2 library, moving it from `nifi-aws-processors` 
to `nifi-aws-service-api` where it will be available in the same ClassLoader as 
the AWS Auth Version 2 library for runtime credential loading. This change 
aligns with the current location of the AWS STS Version 1 library, already 
included in `nifi-aws-service-api`.
   
   Additional changes include declaring `nifi-aws-service-api` as provided at 
the bundle level to avoid unnecessary repetition, and marking the STS and Auth 
libraries as provided in `nifi-aws-nar` to avoid unnecessary inclusion.
   
   # Tracking
   
   Please complete the following tracking steps prior to pull request creation.
   
   ### Issue Tracking
   
   - [X] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue 
created
   
   ### Pull Request Tracking
   
   - [X] Pull Request title starts with Apache NiFi Jira issue number, such as 
`NIFI-0`
   - [X] Pull Request commit message starts with Apache NiFi Jira issue number, 
as such `NIFI-0`
   
   ### Pull Request Formatting
   
   - [X] Pull Request based on current revision of the `main` branch
   - [X] Pull Request refers to a feature branch with one commit containing 
changes
   
   # Verification
   
   Please indicate the verification steps performed prior to pull request 
creation.
   
   ### Build
   
   - [X] Build completed using `mvn clean install -P contrib-check`
 - [X] JDK 21
   
   ### Licensing
   
   - [ ] New dependencies are compatible with the [Apache License 
2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License 
Policy](https://www.apache.org/legal/resolved.html)
   - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` 
files
   
   ### Documentation
   
   - [ ] Documentation formatting appears as expected in rendered files
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Updated] (NIFI-12997) Correct AWS STS Version 2 Location

2024-04-03 Thread David Handermann (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Handermann updated NIFI-12997:

Issue Type: Bug  (was: Improvement)

> Correct AWS STS Version 2 Location
> --
>
> Key: NIFI-12997
> URL: https://issues.apache.org/jira/browse/NIFI-12997
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
>
> The AWS STS library for Version 2 is included in {{{}nifi-aws-processors{}}}, 
> but not in {{{}nifi-aws-service-api{}}}. The STS library needs to be in the 
> same ClassLoader as the AWS Auth library for runtime credential loading. 
> Version 1 of the STS library is already included in {{nifi-aws-service-api}} 
> and corresponding NAR, which allows S3 Processors to work with STS. Other AWS 
> Processors using SDK Version 2 need the STS library moved in order to work 
> with this credentials strategy.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (NIFI-12997) Correct AWS STS Version 2 Location

2024-04-03 Thread David Handermann (Jira)
David Handermann created NIFI-12997:
---

 Summary: Correct AWS STS Version 2 Location
 Key: NIFI-12997
 URL: https://issues.apache.org/jira/browse/NIFI-12997
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Extensions
Reporter: David Handermann
Assignee: David Handermann


The AWS STS library for Version 2 is included in {{{}nifi-aws-processors{}}}, 
but not in {{{}nifi-aws-service-api{}}}. The STS library needs to be in the 
same ClassLoader as the AWS Auth library for runtime credential loading. 
Version 1 of the STS library is already included in {{nifi-aws-service-api}} 
and corresponding NAR, which allows S3 Processors to work with STS. Other AWS 
Processors using SDK Version 2 need the STS library moved in order to work with 
this credentials strategy.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (NIFI-12969) Under heavy load, nifi node unable to rejoin cluster, graph modified with temp funnel

2024-04-03 Thread Joe Witt (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17833713#comment-17833713
 ] 

Joe Witt commented on NIFI-12969:
-

thanks [~pgyori].  Will flag this as a key issue to look into for 1.26 and 
2.0m3.  If triaged and found to be manageable we can relax it but otherwise it 
will get some attention before we release.

> Under heavy load, nifi node unable to rejoin cluster, graph modified with 
> temp funnel
> -
>
> Key: NIFI-12969
> URL: https://issues.apache.org/jira/browse/NIFI-12969
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.24.0, 2.0.0-M2
>Reporter: Nissim Shiman
>Assignee: Nissim Shiman
>Priority: Critical
> Fix For: 2.0.0-M3, 1.26.0
>
> Attachments: nifi-app.log, simple_flow.png, 
> simple_flow_with_temp-funnel.png
>
>
> Under heavy load, if a node leaves the cluster (due to heartbeat time out), 
> many times it is unable to rejoin the cluster.
> The nodes' graph will have been modified with a temp-funnel as well.
> Appears to be some sort of [timing 
> issue|https://github.com/apache/nifi/blob/rel/nifi-2.0.0-M2/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/connectable/StandardConnection.java#L298]
>  # To reproduce, on a nifi cluster of three nodes, set up:
> 2 GenerateFlowFile processors -> PG
> Inside PG:
> inputPort -> UpdateAttribute
>  # Keep all defaults except for the following:
> For UpdateAttribute terminate the success relationship
> One of the GenerateFlowFile processors can be disabled,
> the other one should have Run Schedule to be 0 min (this will allow for the 
> heavy load)
>  # In nifi.properties (on all 3 nodes) to allow for nodes to fall out of the 
> cluster, set: nifi.cluster.protocol.heartbeat.interval=2 sec  (default is 5) 
> nifi.cluster.protocol.heartbeat.missable.max=1   (default is 8)
> Restart nifi. Start flow. The nodes will quickly fall out and rejoin cluster. 
> After a few minutes one will likely not be able to rejoin.  The graph for 
> that node will have the disabled GenerateFlowFile now pointing to a funnel (a 
> temp-funnel) instead of the PG
> Stack trace on that nodes nifi-app.log will look like this: (this is from 
> 2.0.0-M2):
> {code:java}
> 2024-03-28 13:55:19,395 INFO [Reconnect to Cluster] 
> o.a.nifi.controller.StandardFlowService Node disconnected due to Failed to 
> properly handle Reconnection request due to org.apache.nifi.control
> ler.serialization.FlowSynchronizationException: Failed to connect node to 
> cluster because local flow controller partially updated. Administrator should 
> disconnect node and review flow for corrup
> tion.
> 2024-03-28 13:55:19,395 ERROR [Reconnect to Cluster] 
> o.a.nifi.controller.StandardFlowService Handling reconnection request failed 
> due to: org.apache.nifi.controller.serialization.FlowSynchroniza
> tionException: Failed to connect node to cluster because local flow 
> controller partially updated. Administrator should disconnect node and review 
> flow for corruption.
> org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed 
> to connect node to cluster because local flow controller partially updated. 
> Administrator should disconnect node and
>  review flow for corruption.
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:985)
> at 
> org.apache.nifi.controller.StandardFlowService.handleReconnectionRequest(StandardFlowService.java:655)
> at 
> org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:384)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> Caused by: 
> org.apache.nifi.controller.serialization.FlowSynchronizationException: 
> java.lang.IllegalStateException: Cannot change destination of Connection 
> because FlowFiles from this Connection
> are currently held by LocalPort[id=99213c00-78ca-4848-112f-5454cc20656b, 
> type=INPUT_PORT, name=inputPort, group=innerPG]
> at 
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:472)
> at 
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.sync(VersionedFlowSynchronizer.java:223)
> at 
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1740)
> at 
> org.apache.nifi.persistence.StandardFlowConfigurationDAO.load(StandardFlowConfigurationDAO.java:91)
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:805)
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:954)
> ... 3

[jira] [Updated] (NIFI-12969) Under heavy load, nifi node unable to rejoin cluster, graph modified with temp funnel

2024-04-03 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-12969:

Fix Version/s: 2.0.0-M3
   1.26.0

> Under heavy load, nifi node unable to rejoin cluster, graph modified with 
> temp funnel
> -
>
> Key: NIFI-12969
> URL: https://issues.apache.org/jira/browse/NIFI-12969
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.24.0, 2.0.0-M2
>Reporter: Nissim Shiman
>Assignee: Nissim Shiman
>Priority: Major
> Fix For: 2.0.0-M3, 1.26.0
>
> Attachments: nifi-app.log, simple_flow.png, 
> simple_flow_with_temp-funnel.png
>
>
> Under heavy load, if a node leaves the cluster (due to heartbeat time out), 
> many times it is unable to rejoin the cluster.
> The nodes' graph will have been modified with a temp-funnel as well.
> Appears to be some sort of [timing 
> issue|https://github.com/apache/nifi/blob/rel/nifi-2.0.0-M2/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/connectable/StandardConnection.java#L298]
>  # To reproduce, on a nifi cluster of three nodes, set up:
> 2 GenerateFlowFile processors -> PG
> Inside PG:
> inputPort -> UpdateAttribute
>  # Keep all defaults except for the following:
> For UpdateAttribute terminate the success relationship
> One of the GenerateFlowFile processors can be disabled,
> the other one should have Run Schedule to be 0 min (this will allow for the 
> heavy load)
>  # In nifi.properties (on all 3 nodes) to allow for nodes to fall out of the 
> cluster, set: nifi.cluster.protocol.heartbeat.interval=2 sec  (default is 5) 
> nifi.cluster.protocol.heartbeat.missable.max=1   (default is 8)
> Restart nifi. Start flow. The nodes will quickly fall out and rejoin cluster. 
> After a few minutes one will likely not be able to rejoin.  The graph for 
> that node will have the disabled GenerateFlowFile now pointing to a funnel (a 
> temp-funnel) instead of the PG
> Stack trace on that nodes nifi-app.log will look like this: (this is from 
> 2.0.0-M2):
> {code:java}
> 2024-03-28 13:55:19,395 INFO [Reconnect to Cluster] 
> o.a.nifi.controller.StandardFlowService Node disconnected due to Failed to 
> properly handle Reconnection request due to org.apache.nifi.control
> ler.serialization.FlowSynchronizationException: Failed to connect node to 
> cluster because local flow controller partially updated. Administrator should 
> disconnect node and review flow for corrup
> tion.
> 2024-03-28 13:55:19,395 ERROR [Reconnect to Cluster] 
> o.a.nifi.controller.StandardFlowService Handling reconnection request failed 
> due to: org.apache.nifi.controller.serialization.FlowSynchroniza
> tionException: Failed to connect node to cluster because local flow 
> controller partially updated. Administrator should disconnect node and review 
> flow for corruption.
> org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed 
> to connect node to cluster because local flow controller partially updated. 
> Administrator should disconnect node and
>  review flow for corruption.
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:985)
> at 
> org.apache.nifi.controller.StandardFlowService.handleReconnectionRequest(StandardFlowService.java:655)
> at 
> org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:384)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> Caused by: 
> org.apache.nifi.controller.serialization.FlowSynchronizationException: 
> java.lang.IllegalStateException: Cannot change destination of Connection 
> because FlowFiles from this Connection
> are currently held by LocalPort[id=99213c00-78ca-4848-112f-5454cc20656b, 
> type=INPUT_PORT, name=inputPort, group=innerPG]
> at 
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:472)
> at 
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.sync(VersionedFlowSynchronizer.java:223)
> at 
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1740)
> at 
> org.apache.nifi.persistence.StandardFlowConfigurationDAO.load(StandardFlowConfigurationDAO.java:91)
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:805)
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:954)
> ... 3 common frames omitted
> Caused by: java.lang.IllegalStateException: Cannot change destination of 
> Connection because FlowFiles from this Connection are currently held by 
> LocalPort[id=99213c00-78ca-484

[jira] [Updated] (NIFI-12969) Under heavy load, nifi node unable to rejoin cluster, graph modified with temp funnel

2024-04-03 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-12969:

Priority: Critical  (was: Major)

> Under heavy load, nifi node unable to rejoin cluster, graph modified with 
> temp funnel
> -
>
> Key: NIFI-12969
> URL: https://issues.apache.org/jira/browse/NIFI-12969
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.24.0, 2.0.0-M2
>Reporter: Nissim Shiman
>Assignee: Nissim Shiman
>Priority: Critical
> Fix For: 2.0.0-M3, 1.26.0
>
> Attachments: nifi-app.log, simple_flow.png, 
> simple_flow_with_temp-funnel.png
>
>
> Under heavy load, if a node leaves the cluster (due to heartbeat time out), 
> many times it is unable to rejoin the cluster.
> The nodes' graph will have been modified with a temp-funnel as well.
> Appears to be some sort of [timing 
> issue|https://github.com/apache/nifi/blob/rel/nifi-2.0.0-M2/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/connectable/StandardConnection.java#L298]
>  # To reproduce, on a nifi cluster of three nodes, set up:
> 2 GenerateFlowFile processors -> PG
> Inside PG:
> inputPort -> UpdateAttribute
>  # Keep all defaults except for the following:
> For UpdateAttribute terminate the success relationship
> One of the GenerateFlowFile processors can be disabled,
> the other one should have Run Schedule to be 0 min (this will allow for the 
> heavy load)
>  # In nifi.properties (on all 3 nodes) to allow for nodes to fall out of the 
> cluster, set: nifi.cluster.protocol.heartbeat.interval=2 sec  (default is 5) 
> nifi.cluster.protocol.heartbeat.missable.max=1   (default is 8)
> Restart nifi. Start flow. The nodes will quickly fall out and rejoin cluster. 
> After a few minutes one will likely not be able to rejoin.  The graph for 
> that node will have the disabled GenerateFlowFile now pointing to a funnel (a 
> temp-funnel) instead of the PG
> Stack trace on that nodes nifi-app.log will look like this: (this is from 
> 2.0.0-M2):
> {code:java}
> 2024-03-28 13:55:19,395 INFO [Reconnect to Cluster] 
> o.a.nifi.controller.StandardFlowService Node disconnected due to Failed to 
> properly handle Reconnection request due to org.apache.nifi.control
> ler.serialization.FlowSynchronizationException: Failed to connect node to 
> cluster because local flow controller partially updated. Administrator should 
> disconnect node and review flow for corrup
> tion.
> 2024-03-28 13:55:19,395 ERROR [Reconnect to Cluster] 
> o.a.nifi.controller.StandardFlowService Handling reconnection request failed 
> due to: org.apache.nifi.controller.serialization.FlowSynchroniza
> tionException: Failed to connect node to cluster because local flow 
> controller partially updated. Administrator should disconnect node and review 
> flow for corruption.
> org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed 
> to connect node to cluster because local flow controller partially updated. 
> Administrator should disconnect node and
>  review flow for corruption.
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:985)
> at 
> org.apache.nifi.controller.StandardFlowService.handleReconnectionRequest(StandardFlowService.java:655)
> at 
> org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:384)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> Caused by: 
> org.apache.nifi.controller.serialization.FlowSynchronizationException: 
> java.lang.IllegalStateException: Cannot change destination of Connection 
> because FlowFiles from this Connection
> are currently held by LocalPort[id=99213c00-78ca-4848-112f-5454cc20656b, 
> type=INPUT_PORT, name=inputPort, group=innerPG]
> at 
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:472)
> at 
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.sync(VersionedFlowSynchronizer.java:223)
> at 
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1740)
> at 
> org.apache.nifi.persistence.StandardFlowConfigurationDAO.load(StandardFlowConfigurationDAO.java:91)
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:805)
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:954)
> ... 3 common frames omitted
> Caused by: java.lang.IllegalStateException: Cannot change destination of 
> Connection because FlowFiles from this Connection are currently held by 
> LocalPort[id=99213c00-78ca-4848-112f-5454cc2

[jira] [Comment Edited] (NIFI-12969) Under heavy load, nifi node unable to rejoin cluster, graph modified with temp funnel

2024-04-03 Thread Nissim Shiman (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17833705#comment-17833705
 ] 

Nissim Shiman edited comment on NIFI-12969 at 4/3/24 6:45 PM:
--

Thank you very much [~pgyori] for the testing, diagrams, logs and bringing out 
the finer details of this.
This greatly appreciated.

I am looking at a solution like you suggested (i.e. wait a little in 
StandardConnection) and after a short period of time, to give up and in the 
calling method undo the temp connections (and remove the temp funnel) to try to 
keep the graph the way it was before the temp connection(s) were made.

There doesn't appear to be an easy way to predict this situation to avoid 
making the temp connections in the first place. I've tried checking for 
unacknowleged flowfiles on all a group's connections before making 
temp-funnel/temp destinations (for any connection).   But even if it checks 
out, by the time we get to setting an individual connection's destination, 
there could be unacknowledged flowfiles on the connection again.

Another possibility is to make a special case for this situation (i.e. adding 
another
[DisconnectionCode 
|https://github.com/apache/nifi/blob/rel/nifi-2.0.0-M2/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster-protocol/src/main/java/org/apache/nifi/cluster/coordination/node/DisconnectionCode.java]
and in general to have a special case where we try to connect again after 
cluster reconnection fails for this case.  But this will affect a lot of code 
beyond this.

I am very much interested in input on other angles to look at with this. 


was (Author: nissim shiman):
Thank you very much [~pgyori] for the testing, diagrams, logs and bringing out 
the finer details of this.
This greatly appreciated.

I am looking at a solution like you suggested (i.e. wait a little in 
StandardConnection) and after a short period of time, to give up and in the 
calling method undo the temp connections (and remove the temp funnel) to try to 
keep the graph the way it was before the temp connection(s) were made.

There doesn't appear to be an easy way to predict this situation to avoid 
making the temp connections in the first place. I've tried checking for 
unacknowleged flowfiles on all a group's connections before making 
temp-funnel/temp destinations (for any connection).   But even if it checks 
out, by the time we get to setting an individual connection's destination, 
there could be unacknowledged flowfiles on the connection again.

I am very much interested in input on other angles to look at with this. 

> Under heavy load, nifi node unable to rejoin cluster, graph modified with 
> temp funnel
> -
>
> Key: NIFI-12969
> URL: https://issues.apache.org/jira/browse/NIFI-12969
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.24.0, 2.0.0-M2
>Reporter: Nissim Shiman
>Assignee: Nissim Shiman
>Priority: Major
> Attachments: nifi-app.log, simple_flow.png, 
> simple_flow_with_temp-funnel.png
>
>
> Under heavy load, if a node leaves the cluster (due to heartbeat time out), 
> many times it is unable to rejoin the cluster.
> The nodes' graph will have been modified with a temp-funnel as well.
> Appears to be some sort of [timing 
> issue|https://github.com/apache/nifi/blob/rel/nifi-2.0.0-M2/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/connectable/StandardConnection.java#L298]
>  # To reproduce, on a nifi cluster of three nodes, set up:
> 2 GenerateFlowFile processors -> PG
> Inside PG:
> inputPort -> UpdateAttribute
>  # Keep all defaults except for the following:
> For UpdateAttribute terminate the success relationship
> One of the GenerateFlowFile processors can be disabled,
> the other one should have Run Schedule to be 0 min (this will allow for the 
> heavy load)
>  # In nifi.properties (on all 3 nodes) to allow for nodes to fall out of the 
> cluster, set: nifi.cluster.protocol.heartbeat.interval=2 sec  (default is 5) 
> nifi.cluster.protocol.heartbeat.missable.max=1   (default is 8)
> Restart nifi. Start flow. The nodes will quickly fall out and rejoin cluster. 
> After a few minutes one will likely not be able to rejoin.  The graph for 
> that node will have the disabled GenerateFlowFile now pointing to a funnel (a 
> temp-funnel) instead of the PG
> Stack trace on that nodes nifi-app.log will look like this: (this is from 
> 2.0.0-M2):
> {code:java}
> 2024-03-28 13:55:19,395 INFO [Reconnect to Cluster] 
> o.a.nifi.controller.StandardFlowService Node disconnected due to Failed to 
> properly handle Reconnection request due to org.apache.nifi.control
> ler.serialization.FlowSynchronizationEx

[jira] [Comment Edited] (NIFI-12969) Under heavy load, nifi node unable to rejoin cluster, graph modified with temp funnel

2024-04-03 Thread Nissim Shiman (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17833705#comment-17833705
 ] 

Nissim Shiman edited comment on NIFI-12969 at 4/3/24 6:29 PM:
--

Thank you very much [~pgyori] for the testing, diagrams, logs and bringing out 
the finer details of this.
This greatly appreciated.

I am looking at a solution like you suggested (i.e. wait a little in 
StandardConnection) and after a short period of time, to give up and in the 
calling method undo the temp connections (and remove the temp funnel) to try to 
keep the graph the way it was before the temp connection(s) were made.

There doesn't appear to be an easy way to predict this situation to avoid 
making the temp connections in the first place. I've tried checking for 
unacknowleged flowfiles on all a group's connections before making 
temp-funnel/temp destinations (for any connection).   But even if it checks 
out, by the time we get to setting an individual connection's destination, 
there could be unacknowledged flowfiles on the connection again.

I am very much interested in input on other angles to look at with this. 


was (Author: nissim shiman):
Thank you very much [~pgyori] for the testing, diagrams, logs and bringing out 
the finer details of this.
This greatly appreciated.

I am looking at a solution like you suggested (i.e. wait a little in 
StandardConnection) and after a short period of time, to give up and in the 
calling method undo the temp connections (and remove the temp funnel) to try to 
keep the graph the way it was before the temp connection(s) were made.

There doesn't appear to be an easy way to predict this situation to avoid 
making the temp connections in the first place. I've tried checking for 
unacknowleged flowfiles on all a group's connections before making 
temp-funnel/temp destinations (for any connection).   But even if it checks 
out, by the time we get to setting an individual connection's destination, 
there could be unacknowledged flowfiles on the connection again.

> Under heavy load, nifi node unable to rejoin cluster, graph modified with 
> temp funnel
> -
>
> Key: NIFI-12969
> URL: https://issues.apache.org/jira/browse/NIFI-12969
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.24.0, 2.0.0-M2
>Reporter: Nissim Shiman
>Assignee: Nissim Shiman
>Priority: Major
> Attachments: nifi-app.log, simple_flow.png, 
> simple_flow_with_temp-funnel.png
>
>
> Under heavy load, if a node leaves the cluster (due to heartbeat time out), 
> many times it is unable to rejoin the cluster.
> The nodes' graph will have been modified with a temp-funnel as well.
> Appears to be some sort of [timing 
> issue|https://github.com/apache/nifi/blob/rel/nifi-2.0.0-M2/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/connectable/StandardConnection.java#L298]
>  # To reproduce, on a nifi cluster of three nodes, set up:
> 2 GenerateFlowFile processors -> PG
> Inside PG:
> inputPort -> UpdateAttribute
>  # Keep all defaults except for the following:
> For UpdateAttribute terminate the success relationship
> One of the GenerateFlowFile processors can be disabled,
> the other one should have Run Schedule to be 0 min (this will allow for the 
> heavy load)
>  # In nifi.properties (on all 3 nodes) to allow for nodes to fall out of the 
> cluster, set: nifi.cluster.protocol.heartbeat.interval=2 sec  (default is 5) 
> nifi.cluster.protocol.heartbeat.missable.max=1   (default is 8)
> Restart nifi. Start flow. The nodes will quickly fall out and rejoin cluster. 
> After a few minutes one will likely not be able to rejoin.  The graph for 
> that node will have the disabled GenerateFlowFile now pointing to a funnel (a 
> temp-funnel) instead of the PG
> Stack trace on that nodes nifi-app.log will look like this: (this is from 
> 2.0.0-M2):
> {code:java}
> 2024-03-28 13:55:19,395 INFO [Reconnect to Cluster] 
> o.a.nifi.controller.StandardFlowService Node disconnected due to Failed to 
> properly handle Reconnection request due to org.apache.nifi.control
> ler.serialization.FlowSynchronizationException: Failed to connect node to 
> cluster because local flow controller partially updated. Administrator should 
> disconnect node and review flow for corrup
> tion.
> 2024-03-28 13:55:19,395 ERROR [Reconnect to Cluster] 
> o.a.nifi.controller.StandardFlowService Handling reconnection request failed 
> due to: org.apache.nifi.controller.serialization.FlowSynchroniza
> tionException: Failed to connect node to cluster because local flow 
> controller partially updated. Administrator should disconnect node and review 
> flow for corruption.
> org.apache.nifi.contr

[jira] [Commented] (NIFI-12969) Under heavy load, nifi node unable to rejoin cluster, graph modified with temp funnel

2024-04-03 Thread Nissim Shiman (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17833705#comment-17833705
 ] 

Nissim Shiman commented on NIFI-12969:
--

Thank you very much [~pgyori] for the testing, diagrams, logs and bringing out 
the finer details of this.
This greatly appreciated.

I am looking at a solution like you suggested (i.e. wait a little in 
StandardConnection) and after a short period of time, to give up and in the 
calling method undo the temp connections (and remove the temp funnel) to try to 
keep the graph the way it was before the temp connection(s) were made.

There doesn't appear to be an easy way to predict this situation to avoid 
making the temp connections in the first place. I've tried checking for 
unacknowleged flowfiles on all a group's connections before making 
temp-funnel/temp destinations (for any connection).   But even if it checks 
out, by the time we get to setting an individual connection's destination, 
there could be unacknowledged flowfiles on the connection again.

> Under heavy load, nifi node unable to rejoin cluster, graph modified with 
> temp funnel
> -
>
> Key: NIFI-12969
> URL: https://issues.apache.org/jira/browse/NIFI-12969
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.24.0, 2.0.0-M2
>Reporter: Nissim Shiman
>Assignee: Nissim Shiman
>Priority: Major
> Attachments: nifi-app.log, simple_flow.png, 
> simple_flow_with_temp-funnel.png
>
>
> Under heavy load, if a node leaves the cluster (due to heartbeat time out), 
> many times it is unable to rejoin the cluster.
> The nodes' graph will have been modified with a temp-funnel as well.
> Appears to be some sort of [timing 
> issue|https://github.com/apache/nifi/blob/rel/nifi-2.0.0-M2/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/connectable/StandardConnection.java#L298]
>  # To reproduce, on a nifi cluster of three nodes, set up:
> 2 GenerateFlowFile processors -> PG
> Inside PG:
> inputPort -> UpdateAttribute
>  # Keep all defaults except for the following:
> For UpdateAttribute terminate the success relationship
> One of the GenerateFlowFile processors can be disabled,
> the other one should have Run Schedule to be 0 min (this will allow for the 
> heavy load)
>  # In nifi.properties (on all 3 nodes) to allow for nodes to fall out of the 
> cluster, set: nifi.cluster.protocol.heartbeat.interval=2 sec  (default is 5) 
> nifi.cluster.protocol.heartbeat.missable.max=1   (default is 8)
> Restart nifi. Start flow. The nodes will quickly fall out and rejoin cluster. 
> After a few minutes one will likely not be able to rejoin.  The graph for 
> that node will have the disabled GenerateFlowFile now pointing to a funnel (a 
> temp-funnel) instead of the PG
> Stack trace on that nodes nifi-app.log will look like this: (this is from 
> 2.0.0-M2):
> {code:java}
> 2024-03-28 13:55:19,395 INFO [Reconnect to Cluster] 
> o.a.nifi.controller.StandardFlowService Node disconnected due to Failed to 
> properly handle Reconnection request due to org.apache.nifi.control
> ler.serialization.FlowSynchronizationException: Failed to connect node to 
> cluster because local flow controller partially updated. Administrator should 
> disconnect node and review flow for corrup
> tion.
> 2024-03-28 13:55:19,395 ERROR [Reconnect to Cluster] 
> o.a.nifi.controller.StandardFlowService Handling reconnection request failed 
> due to: org.apache.nifi.controller.serialization.FlowSynchroniza
> tionException: Failed to connect node to cluster because local flow 
> controller partially updated. Administrator should disconnect node and review 
> flow for corruption.
> org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed 
> to connect node to cluster because local flow controller partially updated. 
> Administrator should disconnect node and
>  review flow for corruption.
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:985)
> at 
> org.apache.nifi.controller.StandardFlowService.handleReconnectionRequest(StandardFlowService.java:655)
> at 
> org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:384)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> Caused by: 
> org.apache.nifi.controller.serialization.FlowSynchronizationException: 
> java.lang.IllegalStateException: Cannot change destination of Connection 
> because FlowFiles from this Connection
> are currently held by LocalPort[id=99213c00-78ca-4848-112f-5454cc20656b, 
> type=INPUT_PORT, name=inputPort, group=innerPG]
> at 
> org.apache.nifi.controller.serialization.VersionedFlowSynchroni

[jira] [Updated] (NIFI-12735) Execute flow analysis before committing flow into Registry

2024-04-03 Thread Tamas Palfy (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tamas Palfy updated NIFI-12735:
---
Fix Version/s: 2.0.0-M3
   Resolution: Fixed
   Status: Resolved  (was: Patch Available)

> Execute flow analysis before committing flow into Registry
> --
>
> Key: NIFI-12735
> URL: https://issues.apache.org/jira/browse/NIFI-12735
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Simon Bence
>Assignee: Simon Bence
>Priority: Major
> Fix For: 2.0.0-M3
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Based on https://lists.apache.org/thread/hsg2ggj9p47lvj5m00okkj0ok58zdoqp
> When committing a flow into a Registry, it would be beneficial to have the 
> opportunity to execute the existing flow analysis rules and potentially 
> interrupt the commit.
> The change aims for:
> - Execute existing flow analysis rules out of the scheduled times
> - Gracefully resject commit when the flow does not meet the requirements
> - Providing a toggle to turn on or off the feature
> - In case of the feature is inactive or no rules are set, the Registry 
> handling behaviour should not change*
> The change does not aim for:
> - RegistryClient level adjustments
> - Using different rules set or other kind of validation methods
> - Providing support for the 1.x line
> {*}
> Expect this following branch: 
> https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/StandardNiFiServiceFacade.java#L5028
> As with the usage of this feature, failing snapshot commits will be more 
> frequent and I consider "lingering" flow definitions without snapshot 
> misleading for users.
> The following test cases describe the proposed behaviour in detail:
> TC1.1 Adding flow with rules violation
> GIVEN a `DisallowComponentType` flow analysis rule, disallowing 
> `GenerateFlowFile`
> AND NiFi property `nifi.registry.check.for.rules.violation` is true
> AND process group G1 containing a `GenerateFlowFile`
> AND violation appears as validation failure
> WHEN the user tries to add G1 under version control, using registry R1
> THEN the UI displays the following error message "*Cannot store flow version 
> to registry due to rules violations*"
> AND the flow will not appear in registry R1
> AND the UI will show G1 as un-versioned
> AND NiFi node will see G1 as un-versioned
> TC1.2 Adding flow with rules violation before scheduled
> GIVEN a `DisallowComponentType` flow analysis rule, disallowing 
> `GenerateFlowFile`
> AND NiFi property `nifi.registry.check.for.rules.violation` is true
> AND process group G1 containing a `GenerateFlowFile`
> AND violation does not appear in UI
> WHEN the user tries to add G1 under version control, using registry R1
> THEN the UI displays the following error message "*Cannot store flow version 
> to registry due to rules violations*"
> AND the flow will not appear in registry R1
> AND the UI will show G1 as un-versioned
> AND NiFi node will see G1 as un-versioned
> TC1.3 When turned off violating flows will be added
> GIVEN a `DisallowComponentType` flow analysis rule, disallowing 
> `GenerateFlowFile`
> AND NiFi property `nifi.registry.check.for.rules.violation` is fals
> AND process group G1 containing a `GenerateFlowFile`
> AND violation appears as validation failure
> WHEN the user tries to add G1 under version control, using registry R1
> THEN the flow is committed normally
> TC1.4 Adding version with rules violation (when the rule is new)
> GIVEN NiFi property `nifi.registry.check.for.rules.violation` is true
> AND process group G1 containing a `GenerateFlowFile`
> AND process group G1 is under version control
> AND `LogAttribute` processor is added (not committed to the registry)
> GIVEN adding a `DisallowComponentType` flow analysis rule, disallowing 
> `GenerateFlowFile`
> AND  the user tries to add G1 under version control, using registry R1
> THEN the UI displays the following error message "*Cannot store flow version 
> to registry due to rules violations*"
> AND the flow will not appear in registry R1
> AND the UI will show G1 as un-versioned
> AND NiFi node will see G1 as un-versioned
> TC1.5 Adding version with rules violation (when the violation is new)
> GIVEN a `DisallowComponentType` flow analysis rule, disallowing 
> `GenerateFlowFile`
> AND NiFi property `nifi.registry.check.for.rules.violation` is true
> AND process group G1 is under version control
> AND `GenerateFlowFile` processor is added (not committed to the registry)
> WHEN the user tries to add new version of G1 to registry R1
> THEN the UI displays the following error message "*Cannot store flow version 
> to registry due to rules 

[jira] [Commented] (NIFI-12735) Execute flow analysis before committing flow into Registry

2024-04-03 Thread ASF subversion and git services (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17833648#comment-17833648
 ] 

ASF subversion and git services commented on NIFI-12735:


Commit 88d434f9eca41b939e2eb7213f45b8bb8ae00ed3 in nifi's branch 
refs/heads/main from Simon Bence
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=88d434f9ec ]

NIFI-12735 Adding the option to execute flow analysis before committing 
snapshot to registry

This closes #8377.

Signed-off-by: Tamas Palfy 


> Execute flow analysis before committing flow into Registry
> --
>
> Key: NIFI-12735
> URL: https://issues.apache.org/jira/browse/NIFI-12735
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Simon Bence
>Assignee: Simon Bence
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Based on https://lists.apache.org/thread/hsg2ggj9p47lvj5m00okkj0ok58zdoqp
> When committing a flow into a Registry, it would be beneficial to have the 
> opportunity to execute the existing flow analysis rules and potentially 
> interrupt the commit.
> The change aims for:
> - Execute existing flow analysis rules out of the scheduled times
> - Gracefully resject commit when the flow does not meet the requirements
> - Providing a toggle to turn on or off the feature
> - In case of the feature is inactive or no rules are set, the Registry 
> handling behaviour should not change*
> The change does not aim for:
> - RegistryClient level adjustments
> - Using different rules set or other kind of validation methods
> - Providing support for the 1.x line
> {*}
> Expect this following branch: 
> https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/StandardNiFiServiceFacade.java#L5028
> As with the usage of this feature, failing snapshot commits will be more 
> frequent and I consider "lingering" flow definitions without snapshot 
> misleading for users.
> The following test cases describe the proposed behaviour in detail:
> TC1.1 Adding flow with rules violation
> GIVEN a `DisallowComponentType` flow analysis rule, disallowing 
> `GenerateFlowFile`
> AND NiFi property `nifi.registry.check.for.rules.violation` is true
> AND process group G1 containing a `GenerateFlowFile`
> AND violation appears as validation failure
> WHEN the user tries to add G1 under version control, using registry R1
> THEN the UI displays the following error message "*Cannot store flow version 
> to registry due to rules violations*"
> AND the flow will not appear in registry R1
> AND the UI will show G1 as un-versioned
> AND NiFi node will see G1 as un-versioned
> TC1.2 Adding flow with rules violation before scheduled
> GIVEN a `DisallowComponentType` flow analysis rule, disallowing 
> `GenerateFlowFile`
> AND NiFi property `nifi.registry.check.for.rules.violation` is true
> AND process group G1 containing a `GenerateFlowFile`
> AND violation does not appear in UI
> WHEN the user tries to add G1 under version control, using registry R1
> THEN the UI displays the following error message "*Cannot store flow version 
> to registry due to rules violations*"
> AND the flow will not appear in registry R1
> AND the UI will show G1 as un-versioned
> AND NiFi node will see G1 as un-versioned
> TC1.3 When turned off violating flows will be added
> GIVEN a `DisallowComponentType` flow analysis rule, disallowing 
> `GenerateFlowFile`
> AND NiFi property `nifi.registry.check.for.rules.violation` is fals
> AND process group G1 containing a `GenerateFlowFile`
> AND violation appears as validation failure
> WHEN the user tries to add G1 under version control, using registry R1
> THEN the flow is committed normally
> TC1.4 Adding version with rules violation (when the rule is new)
> GIVEN NiFi property `nifi.registry.check.for.rules.violation` is true
> AND process group G1 containing a `GenerateFlowFile`
> AND process group G1 is under version control
> AND `LogAttribute` processor is added (not committed to the registry)
> GIVEN adding a `DisallowComponentType` flow analysis rule, disallowing 
> `GenerateFlowFile`
> AND  the user tries to add G1 under version control, using registry R1
> THEN the UI displays the following error message "*Cannot store flow version 
> to registry due to rules violations*"
> AND the flow will not appear in registry R1
> AND the UI will show G1 as un-versioned
> AND NiFi node will see G1 as un-versioned
> TC1.5 Adding version with rules violation (when the violation is new)
> GIVEN a `DisallowComponentType` flow analysis rule, disallowing 
> `GenerateFlowFile`
> AND NiFi property `nifi.registry.check.for.rules.violation` is true
> AND process group G1 is und

Re: [PR] NIFI-12735 Adding the option to execute flow analysis before committing snapshot to registry [nifi]

2024-04-03 Thread via GitHub


tpalfy commented on PR #8377:
URL: https://github.com/apache/nifi/pull/8377#issuecomment-2034921876

   LGTM
   Thank you for your work @simonbence !
   Merged into main.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] NIFI-12735 Adding the option to execute flow analysis before committing snapshot to registry [nifi]

2024-04-03 Thread via GitHub


asfgit closed pull request #8377: NIFI-12735 Adding the option to execute flow 
analysis before committing snapshot to registry
URL: https://github.com/apache/nifi/pull/8377


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Commented] (NIFI-12969) Under heavy load, nifi node unable to rejoin cluster, graph modified with temp funnel

2024-04-03 Thread Peter Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17833636#comment-17833636
 ] 

Peter Gyori commented on NIFI-12969:


FYI [~markap14] , [~joewitt] . This concerns NiFi-2.0.0-M3.

> Under heavy load, nifi node unable to rejoin cluster, graph modified with 
> temp funnel
> -
>
> Key: NIFI-12969
> URL: https://issues.apache.org/jira/browse/NIFI-12969
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.24.0, 2.0.0-M2
>Reporter: Nissim Shiman
>Assignee: Nissim Shiman
>Priority: Major
> Attachments: nifi-app.log, simple_flow.png, 
> simple_flow_with_temp-funnel.png
>
>
> Under heavy load, if a node leaves the cluster (due to heartbeat time out), 
> many times it is unable to rejoin the cluster.
> The nodes' graph will have been modified with a temp-funnel as well.
> Appears to be some sort of [timing 
> issue|https://github.com/apache/nifi/blob/rel/nifi-2.0.0-M2/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/connectable/StandardConnection.java#L298]
>  # To reproduce, on a nifi cluster of three nodes, set up:
> 2 GenerateFlowFile processors -> PG
> Inside PG:
> inputPort -> UpdateAttribute
>  # Keep all defaults except for the following:
> For UpdateAttribute terminate the success relationship
> One of the GenerateFlowFile processors can be disabled,
> the other one should have Run Schedule to be 0 min (this will allow for the 
> heavy load)
>  # In nifi.properties (on all 3 nodes) to allow for nodes to fall out of the 
> cluster, set: nifi.cluster.protocol.heartbeat.interval=2 sec  (default is 5) 
> nifi.cluster.protocol.heartbeat.missable.max=1   (default is 8)
> Restart nifi. Start flow. The nodes will quickly fall out and rejoin cluster. 
> After a few minutes one will likely not be able to rejoin.  The graph for 
> that node will have the disabled GenerateFlowFile now pointing to a funnel (a 
> temp-funnel) instead of the PG
> Stack trace on that nodes nifi-app.log will look like this: (this is from 
> 2.0.0-M2):
> {code:java}
> 2024-03-28 13:55:19,395 INFO [Reconnect to Cluster] 
> o.a.nifi.controller.StandardFlowService Node disconnected due to Failed to 
> properly handle Reconnection request due to org.apache.nifi.control
> ler.serialization.FlowSynchronizationException: Failed to connect node to 
> cluster because local flow controller partially updated. Administrator should 
> disconnect node and review flow for corrup
> tion.
> 2024-03-28 13:55:19,395 ERROR [Reconnect to Cluster] 
> o.a.nifi.controller.StandardFlowService Handling reconnection request failed 
> due to: org.apache.nifi.controller.serialization.FlowSynchroniza
> tionException: Failed to connect node to cluster because local flow 
> controller partially updated. Administrator should disconnect node and review 
> flow for corruption.
> org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed 
> to connect node to cluster because local flow controller partially updated. 
> Administrator should disconnect node and
>  review flow for corruption.
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:985)
> at 
> org.apache.nifi.controller.StandardFlowService.handleReconnectionRequest(StandardFlowService.java:655)
> at 
> org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:384)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> Caused by: 
> org.apache.nifi.controller.serialization.FlowSynchronizationException: 
> java.lang.IllegalStateException: Cannot change destination of Connection 
> because FlowFiles from this Connection
> are currently held by LocalPort[id=99213c00-78ca-4848-112f-5454cc20656b, 
> type=INPUT_PORT, name=inputPort, group=innerPG]
> at 
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:472)
> at 
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.sync(VersionedFlowSynchronizer.java:223)
> at 
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1740)
> at 
> org.apache.nifi.persistence.StandardFlowConfigurationDAO.load(StandardFlowConfigurationDAO.java:91)
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:805)
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:954)
> ... 3 common frames omitted
> Caused by: java.lang.IllegalStateException: Cannot change destination of 
> Connection because FlowFiles from this Connection are currently held by 
> Lo

[jira] [Commented] (NIFI-12969) Under heavy load, nifi node unable to rejoin cluster, graph modified with temp funnel

2024-04-03 Thread Peter Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17833586#comment-17833586
 ] 

Peter Gyori commented on NIFI-12969:


Thank you [~Nissim Shiman] for the detailed steps to reproduce the issue.

I managed to reproduce it with your steps even on a very recent build of NiFi, 
where the latest commit on the main branch is 
{noformat}
commit b574a7e4: NIFI-12987 allow controller service type to be 
searchable{noformat}
I used a 3-node cluster. On Apple Silicon one might need a little stricter 
settings for heartbeats:

 

 
{code:java}
nifi.cluster.protocol.heartbeat.interval=1
secnifi.cluster.protocol.heartbeat.missable.max=1 {code}
I needed two GenerateFlowFile processors, both with 2 simultaneous threads (and 
Run Schedule=0 min as you mentioned).

 

Screenshot of the test flow.

!simple_flow.png|width=645,height=323!

With the following log settings, I captured the attached [^nifi-app.log]

 
{code:java}


 {code}
I paste here the log entries that I find the most important. All of these 
happen within 2 milliseconds:

 

 
{code:java}
2024-04-03 13:48:53,896 TRACE [Timer-Driven Process Thread-2] 
org.apache.nifi.connectable.LocalPort 
LocalPort[id=a3c770bd-018e-1000--0d6b782a, type=INPUT_PORT, name=input, 
group=pg] obtained claim for FlowFileGate
2024-04-03 13:48:53,896 DEBUG [Reconnect to Cluster] 
o.a.n.f.s.StandardVersionedComponentSynchronizer Changing destination of 
Connection Connection[ID=a3c84edc-018e-1000--13bbf65e, Source 
ID=a6393e3a-3f79-1d9a-b4eb-2a25ca912ab9, Dest 
ID=a3c770bd-018e-1000--0d6b782a] from 
LocalPort[id=a3c770bd-018e-1000--0d6b782a, type=INPUT_PORT, name=input, 
group=pg] to StandardFunnel[id=a34c81f0-018e-1000-7faa-412adc4de192-temp-funnel]
2024-04-03 13:48:53,896 DEBUG [Timer-Driven Process Thread-2] 
org.apache.nifi.connectable.LocalPort 
LocalPort[id=a3c770bd-018e-1000--0d6b782a, type=INPUT_PORT, name=input, 
group=pg] Transferred 1 FlowFiles
2024-04-03 13:48:53,896 TRACE [Timer-Driven Process Thread-2] 
org.apache.nifi.connectable.LocalPort 
LocalPort[id=a3c770bd-018e-1000--0d6b782a, type=INPUT_PORT, name=input, 
group=pg] released claim for FlowFileGate
2024-04-03 13:48:53,896 INFO [Reconnect to Cluster] 
o.a.n.c.s.AffectedComponentSet Starting the following components: 
AffectedComponentSet[inputPorts=[], outputPorts=[], remoteInputPorts=[], 
remoteOutputPorts=[], processors=[], parameterProviders=[], 
flowRegistryClients=[], controllerServices=[], reportingTasks=[], 
flowAnalysisRules=[], statelessProcessGroups=[]]
2024-04-03 13:48:53,897 INFO [Reconnect to Cluster] 
o.a.nifi.controller.StandardFlowService Disconnecting node due to Failed to 
properly handle Reconnection request due to 
org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed 
to connect node to cluster because local flow controller partially updated. 
Administrator should disconnect node and review flow for corruption.
2024-04-03 13:48:53,897 INFO [Reconnect to Cluster] 
o.apache.nifi.controller.FlowController Will no longer send heartbeats
2024-04-03 13:48:53,897 INFO [Reconnect to Cluster] 
o.apache.nifi.controller.FlowController FlowController will stop sending 
heartbeats to Cluster Coordinator
2024-04-03 13:48:53,897 INFO [Reconnect to Cluster] 
o.apache.nifi.controller.FlowController Cluster State changed from Clustered to 
Not Clustered
2024-04-03 13:48:53,897 ERROR [Reconnect to Cluster] [...] 
    Caused by: java.lang.IllegalStateException: Cannot change destination of 
Connection because FlowFiles from this Connection are currently held by 
LocalPort[id=a3c770bd-018e-1000--0d6b782a, type=INPUT_PORT, name=input, 
group=pg]
    at 
org.apache.nifi.connectable.StandardConnection.setDestination(StandardConnection.java:299)
    at 
org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.updateConnectionDestinations(StandardVersionedComponentSynchronizer.java:706)
    at 
org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.synchronize(StandardVersionedComponentSynchronizer.java:423)
    at 
org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.lambda$synchronize$0(StandardVersionedComponentSynchronizer.java:248)
    at 
org.apache.nifi.controller.flow.AbstractFlowManager.withParameterContextResolution(AbstractFlowManager.java:638)
    at 
org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.synchronize(StandardVersionedComponentSynchronizer.java:243)
    at 
org.apache.nifi.groups.StandardProcessGroup.synchronizeFlow(StandardProcessGroup.java:3863)
    at 
org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:464){code}
That is: we have an input port with id a3c770bd-018e-1000--0d6b782a and 
the following events happen:
 # obtained claim f

[jira] [Commented] (NIFI-12917) Content repository claim is being zero'd out

2024-04-03 Thread Evan F. (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17833588#comment-17833588
 ] 

Evan F. commented on NIFI-12917:


Absolutely. I'm working to get a copy of 1.25 into our environment to test, 
hopefully by the end of this week. 

> Content repository claim is being zero'd out
> 
>
> Key: NIFI-12917
> URL: https://issues.apache.org/jira/browse/NIFI-12917
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.19.1
> Environment: Red Hat 7 running Apache NiFi 1.19.1 with no custom 
> NARs. 
>Reporter: Evan F.
>Priority: Critical
> Attachments: example_MergeContent.txt
>
>
> I've had rare but consistent instances of MergeContent processors merging 
> files and losing the content claim/payload while merging. I've pasted an 
> example log entry below where there's a size to the file pre-merge and the 
> post-merge content claim becomes zero. I've attached a text file with the 
> configuration of my MergeContent processor as well. This happens a fraction 
> of a percent of the time but it happens daily, somewhere around 10 times a 
> day. I scoured forums and previous tickets but I wasn't able to find a 
> relevant issue. I'm hoping someone can tell me if this has already been 
> addressed. 
>  
> INFO [Timer-Driven Process Thread-7] o.a.n.processors.standard.MergeContent 
> Merge Content[id=e6f460da-018d-1000-1c97-a6bd946d4f61] Merged 
> [StandardFlowFileRecord[uuid=0e5b7c30-021d-4bd9-9edb-1681c9c14d4e,claim=StandardContentClaim
>  [resourceClaim=StandardRe         sourceClaim[id=1710769372188-3700202, 
> container=default, section=490], offset=0, 
> length=52430102],offset=0,name=testFile,size=52430102]] into 
> StandardFlowFileRecord[uuid=536bf8e4-915b-4dad-9e16-063b35e834c1,claim=,offset=0,
>          name=testFile.pkg,size=0]. Reason for merging: Maximum number of 
> entries reached



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12969) Under heavy load, nifi node unable to rejoin cluster, graph modified with temp funnel

2024-04-03 Thread Peter Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Gyori updated NIFI-12969:
---
Attachment: simple_flow_with_temp-funnel.png

> Under heavy load, nifi node unable to rejoin cluster, graph modified with 
> temp funnel
> -
>
> Key: NIFI-12969
> URL: https://issues.apache.org/jira/browse/NIFI-12969
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.24.0, 2.0.0-M2
>Reporter: Nissim Shiman
>Assignee: Nissim Shiman
>Priority: Major
> Attachments: nifi-app.log, simple_flow.png, 
> simple_flow_with_temp-funnel.png
>
>
> Under heavy load, if a node leaves the cluster (due to heartbeat time out), 
> many times it is unable to rejoin the cluster.
> The nodes' graph will have been modified with a temp-funnel as well.
> Appears to be some sort of [timing 
> issue|https://github.com/apache/nifi/blob/rel/nifi-2.0.0-M2/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/connectable/StandardConnection.java#L298]
>  # To reproduce, on a nifi cluster of three nodes, set up:
> 2 GenerateFlowFile processors -> PG
> Inside PG:
> inputPort -> UpdateAttribute
>  # Keep all defaults except for the following:
> For UpdateAttribute terminate the success relationship
> One of the GenerateFlowFile processors can be disabled,
> the other one should have Run Schedule to be 0 min (this will allow for the 
> heavy load)
>  # In nifi.properties (on all 3 nodes) to allow for nodes to fall out of the 
> cluster, set: nifi.cluster.protocol.heartbeat.interval=2 sec  (default is 5) 
> nifi.cluster.protocol.heartbeat.missable.max=1   (default is 8)
> Restart nifi. Start flow. The nodes will quickly fall out and rejoin cluster. 
> After a few minutes one will likely not be able to rejoin.  The graph for 
> that node will have the disabled GenerateFlowFile now pointing to a funnel (a 
> temp-funnel) instead of the PG
> Stack trace on that nodes nifi-app.log will look like this: (this is from 
> 2.0.0-M2):
> {code:java}
> 2024-03-28 13:55:19,395 INFO [Reconnect to Cluster] 
> o.a.nifi.controller.StandardFlowService Node disconnected due to Failed to 
> properly handle Reconnection request due to org.apache.nifi.control
> ler.serialization.FlowSynchronizationException: Failed to connect node to 
> cluster because local flow controller partially updated. Administrator should 
> disconnect node and review flow for corrup
> tion.
> 2024-03-28 13:55:19,395 ERROR [Reconnect to Cluster] 
> o.a.nifi.controller.StandardFlowService Handling reconnection request failed 
> due to: org.apache.nifi.controller.serialization.FlowSynchroniza
> tionException: Failed to connect node to cluster because local flow 
> controller partially updated. Administrator should disconnect node and review 
> flow for corruption.
> org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed 
> to connect node to cluster because local flow controller partially updated. 
> Administrator should disconnect node and
>  review flow for corruption.
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:985)
> at 
> org.apache.nifi.controller.StandardFlowService.handleReconnectionRequest(StandardFlowService.java:655)
> at 
> org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:384)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> Caused by: 
> org.apache.nifi.controller.serialization.FlowSynchronizationException: 
> java.lang.IllegalStateException: Cannot change destination of Connection 
> because FlowFiles from this Connection
> are currently held by LocalPort[id=99213c00-78ca-4848-112f-5454cc20656b, 
> type=INPUT_PORT, name=inputPort, group=innerPG]
> at 
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:472)
> at 
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.sync(VersionedFlowSynchronizer.java:223)
> at 
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1740)
> at 
> org.apache.nifi.persistence.StandardFlowConfigurationDAO.load(StandardFlowConfigurationDAO.java:91)
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:805)
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:954)
> ... 3 common frames omitted
> Caused by: java.lang.IllegalStateException: Cannot change destination of 
> Connection because FlowFiles from this Connection are currently held by 
> LocalPort[id=99213c00-78ca-4848-112f-5454cc20656b
> , type=INPUT_PORT, 

[jira] [Updated] (NIFI-12969) Under heavy load, nifi node unable to rejoin cluster, graph modified with temp funnel

2024-04-03 Thread Peter Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Gyori updated NIFI-12969:
---
Attachment: simple_flow.png

> Under heavy load, nifi node unable to rejoin cluster, graph modified with 
> temp funnel
> -
>
> Key: NIFI-12969
> URL: https://issues.apache.org/jira/browse/NIFI-12969
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.24.0, 2.0.0-M2
>Reporter: Nissim Shiman
>Assignee: Nissim Shiman
>Priority: Major
> Attachments: simple_flow.png
>
>
> Under heavy load, if a node leaves the cluster (due to heartbeat time out), 
> many times it is unable to rejoin the cluster.
> The nodes' graph will have been modified with a temp-funnel as well.
> Appears to be some sort of [timing 
> issue|https://github.com/apache/nifi/blob/rel/nifi-2.0.0-M2/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/connectable/StandardConnection.java#L298]
>  # To reproduce, on a nifi cluster of three nodes, set up:
> 2 GenerateFlowFile processors -> PG
> Inside PG:
> inputPort -> UpdateAttribute
>  # Keep all defaults except for the following:
> For UpdateAttribute terminate the success relationship
> One of the GenerateFlowFile processors can be disabled,
> the other one should have Run Schedule to be 0 min (this will allow for the 
> heavy load)
>  # In nifi.properties (on all 3 nodes) to allow for nodes to fall out of the 
> cluster, set: nifi.cluster.protocol.heartbeat.interval=2 sec  (default is 5) 
> nifi.cluster.protocol.heartbeat.missable.max=1   (default is 8)
> Restart nifi. Start flow. The nodes will quickly fall out and rejoin cluster. 
> After a few minutes one will likely not be able to rejoin.  The graph for 
> that node will have the disabled GenerateFlowFile now pointing to a funnel (a 
> temp-funnel) instead of the PG
> Stack trace on that nodes nifi-app.log will look like this: (this is from 
> 2.0.0-M2):
> {code:java}
> 2024-03-28 13:55:19,395 INFO [Reconnect to Cluster] 
> o.a.nifi.controller.StandardFlowService Node disconnected due to Failed to 
> properly handle Reconnection request due to org.apache.nifi.control
> ler.serialization.FlowSynchronizationException: Failed to connect node to 
> cluster because local flow controller partially updated. Administrator should 
> disconnect node and review flow for corrup
> tion.
> 2024-03-28 13:55:19,395 ERROR [Reconnect to Cluster] 
> o.a.nifi.controller.StandardFlowService Handling reconnection request failed 
> due to: org.apache.nifi.controller.serialization.FlowSynchroniza
> tionException: Failed to connect node to cluster because local flow 
> controller partially updated. Administrator should disconnect node and review 
> flow for corruption.
> org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed 
> to connect node to cluster because local flow controller partially updated. 
> Administrator should disconnect node and
>  review flow for corruption.
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:985)
> at 
> org.apache.nifi.controller.StandardFlowService.handleReconnectionRequest(StandardFlowService.java:655)
> at 
> org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:384)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> Caused by: 
> org.apache.nifi.controller.serialization.FlowSynchronizationException: 
> java.lang.IllegalStateException: Cannot change destination of Connection 
> because FlowFiles from this Connection
> are currently held by LocalPort[id=99213c00-78ca-4848-112f-5454cc20656b, 
> type=INPUT_PORT, name=inputPort, group=innerPG]
> at 
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:472)
> at 
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.sync(VersionedFlowSynchronizer.java:223)
> at 
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1740)
> at 
> org.apache.nifi.persistence.StandardFlowConfigurationDAO.load(StandardFlowConfigurationDAO.java:91)
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:805)
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:954)
> ... 3 common frames omitted
> Caused by: java.lang.IllegalStateException: Cannot change destination of 
> Connection because FlowFiles from this Connection are currently held by 
> LocalPort[id=99213c00-78ca-4848-112f-5454cc20656b
> , type=INPUT_PORT, name=inputPort, group=innerPG]
> at 
> org.apache.nifi.conne

[jira] [Updated] (NIFI-12969) Under heavy load, nifi node unable to rejoin cluster, graph modified with temp funnel

2024-04-03 Thread Peter Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Gyori updated NIFI-12969:
---
Attachment: nifi-app.log

> Under heavy load, nifi node unable to rejoin cluster, graph modified with 
> temp funnel
> -
>
> Key: NIFI-12969
> URL: https://issues.apache.org/jira/browse/NIFI-12969
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.24.0, 2.0.0-M2
>Reporter: Nissim Shiman
>Assignee: Nissim Shiman
>Priority: Major
> Attachments: nifi-app.log, simple_flow.png
>
>
> Under heavy load, if a node leaves the cluster (due to heartbeat time out), 
> many times it is unable to rejoin the cluster.
> The nodes' graph will have been modified with a temp-funnel as well.
> Appears to be some sort of [timing 
> issue|https://github.com/apache/nifi/blob/rel/nifi-2.0.0-M2/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/connectable/StandardConnection.java#L298]
>  # To reproduce, on a nifi cluster of three nodes, set up:
> 2 GenerateFlowFile processors -> PG
> Inside PG:
> inputPort -> UpdateAttribute
>  # Keep all defaults except for the following:
> For UpdateAttribute terminate the success relationship
> One of the GenerateFlowFile processors can be disabled,
> the other one should have Run Schedule to be 0 min (this will allow for the 
> heavy load)
>  # In nifi.properties (on all 3 nodes) to allow for nodes to fall out of the 
> cluster, set: nifi.cluster.protocol.heartbeat.interval=2 sec  (default is 5) 
> nifi.cluster.protocol.heartbeat.missable.max=1   (default is 8)
> Restart nifi. Start flow. The nodes will quickly fall out and rejoin cluster. 
> After a few minutes one will likely not be able to rejoin.  The graph for 
> that node will have the disabled GenerateFlowFile now pointing to a funnel (a 
> temp-funnel) instead of the PG
> Stack trace on that nodes nifi-app.log will look like this: (this is from 
> 2.0.0-M2):
> {code:java}
> 2024-03-28 13:55:19,395 INFO [Reconnect to Cluster] 
> o.a.nifi.controller.StandardFlowService Node disconnected due to Failed to 
> properly handle Reconnection request due to org.apache.nifi.control
> ler.serialization.FlowSynchronizationException: Failed to connect node to 
> cluster because local flow controller partially updated. Administrator should 
> disconnect node and review flow for corrup
> tion.
> 2024-03-28 13:55:19,395 ERROR [Reconnect to Cluster] 
> o.a.nifi.controller.StandardFlowService Handling reconnection request failed 
> due to: org.apache.nifi.controller.serialization.FlowSynchroniza
> tionException: Failed to connect node to cluster because local flow 
> controller partially updated. Administrator should disconnect node and review 
> flow for corruption.
> org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed 
> to connect node to cluster because local flow controller partially updated. 
> Administrator should disconnect node and
>  review flow for corruption.
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:985)
> at 
> org.apache.nifi.controller.StandardFlowService.handleReconnectionRequest(StandardFlowService.java:655)
> at 
> org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:384)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> Caused by: 
> org.apache.nifi.controller.serialization.FlowSynchronizationException: 
> java.lang.IllegalStateException: Cannot change destination of Connection 
> because FlowFiles from this Connection
> are currently held by LocalPort[id=99213c00-78ca-4848-112f-5454cc20656b, 
> type=INPUT_PORT, name=inputPort, group=innerPG]
> at 
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:472)
> at 
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.sync(VersionedFlowSynchronizer.java:223)
> at 
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1740)
> at 
> org.apache.nifi.persistence.StandardFlowConfigurationDAO.load(StandardFlowConfigurationDAO.java:91)
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:805)
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:954)
> ... 3 common frames omitted
> Caused by: java.lang.IllegalStateException: Cannot change destination of 
> Connection because FlowFiles from this Connection are currently held by 
> LocalPort[id=99213c00-78ca-4848-112f-5454cc20656b
> , type=INPUT_PORT, name=inputPort, group=innerPG]
> at 
> org.apache

[jira] [Updated] (NIFI-12963) Process Group Versioning

2024-04-03 Thread Rob Fellows (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rob Fellows updated NIFI-12963:
---
Status: Patch Available  (was: In Progress)

> Process Group Versioning
> 
>
> Key: NIFI-12963
> URL: https://issues.apache.org/jira/browse/NIFI-12963
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Core UI
>Reporter: Rob Fellows
>Assignee: Rob Fellows
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> * Start
>  * Commit
>  * Force Commit
>  * Show changes
>  * Revert changes
>  * Change Flow version
>  * Stop



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12963) Process Group Versioning

2024-04-03 Thread Rob Fellows (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rob Fellows updated NIFI-12963:
---
Description: 
* Start
 * Commit
 * Force Commit
 * -Show changes- (NIFI-12995)
 * -Revert changes- (NIFI-12995)
 * -Change Flow version- (NIFI-12995)
 * Stop

  was:
* Start
 * Commit
 * Force Commit
 * Show changes
 * Revert changes
 * Change Flow version
 * Stop


> Process Group Versioning
> 
>
> Key: NIFI-12963
> URL: https://issues.apache.org/jira/browse/NIFI-12963
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Core UI
>Reporter: Rob Fellows
>Assignee: Rob Fellows
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> * Start
>  * Commit
>  * Force Commit
>  * -Show changes- (NIFI-12995)
>  * -Revert changes- (NIFI-12995)
>  * -Change Flow version- (NIFI-12995)
>  * Stop



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Re: [PR] MINIFICPP-2322 MINIFI_INCLUDE_UCRT_DLLS and MINIFI_INCLUDE_VC_REDIST_… [nifi-minifi-cpp]

2024-04-03 Thread via GitHub


fgerlits closed pull request #1752: MINIFICPP-2322 MINIFI_INCLUDE_UCRT_DLLS and 
MINIFI_INCLUDE_VC_REDIST_…
URL: https://github.com/apache/nifi-minifi-cpp/pull/1752


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] NIFI-12918 Fix Stateless NullPointerException on versioned sub-process groups - main branch [nifi]

2024-04-03 Thread via GitHub


slambrose commented on PR #8536:
URL: https://github.com/apache/nifi/pull/8536#issuecomment-2034320750

   > Thanks for working on the unit tests @slambrose. The test for the 
synchronizer class looks like it exercises the method changes, but the 
`TestRegistryUtil` appears to be running against a mock of `RegistryUtil`, as 
opposed to the actual class, so it doesn't exercise the real method.
   
   Ready for re-review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[PR] MINIFICPP-2322 MINIFI_INCLUDE_UCRT_DLLS and MINIFI_INCLUDE_VC_REDIST_… [nifi-minifi-cpp]

2024-04-03 Thread via GitHub


martinzink opened a new pull request, #1752:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1752

   …DLLS shouldnt be mutually exclusive
   
   Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
   
   In order to streamline the review of the contribution we ask you
   to ensure the following steps have been taken:
   
   ### For all changes:
   - [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
   
   - [ ] Does your PR title start with MINIFICPP- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
   
   - [ ] Has your PR been rebased against the latest commit within the target 
branch (typically main)?
   
   - [ ] Is your initial contribution a single, squashed commit?
   
   ### For code changes:
   - [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
   - [ ] If applicable, have you updated the LICENSE file?
   - [ ] If applicable, have you updated the NOTICE file?
   
   ### For documentation related changes:
   - [ ] Have you ensured that format looks appropriate for the output in which 
it is rendered?
   
   ### Note:
   Please ensure that once the PR is submitted, you check GitHub Actions CI 
results for build issues and submit an update to your PR as soon as possible.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Created] (MINIFICPP-2322) MINIFI_INCLUDE_UCRT_DLLS and MINIFI_INCLUDE_VC_REDIST_DLLS shouldnt be mutually exclusive

2024-04-03 Thread Martin Zink (Jira)
Martin Zink created MINIFICPP-2322:
--

 Summary: MINIFI_INCLUDE_UCRT_DLLS and 
MINIFI_INCLUDE_VC_REDIST_DLLS shouldnt be mutually exclusive
 Key: MINIFICPP-2322
 URL: https://issues.apache.org/jira/browse/MINIFICPP-2322
 Project: Apache NiFi MiNiFi C++
  Issue Type: Bug
Reporter: Martin Zink
Assignee: Martin Zink






--
This message was sent by Atlassian Jira
(v8.20.10#820010)