(nifi) branch support/nifi-1.x updated: NIFI-12925 Updated ListenHTTP to return 405 for TRACE and OPTIONS
This is an automated email from the ASF dual-hosted git repository. exceptionfactory pushed a commit to branch support/nifi-1.x in repository https://gitbox.apache.org/repos/asf/nifi.git The following commit(s) were added to refs/heads/support/nifi-1.x by this push: new 6a8cc9720d NIFI-12925 Updated ListenHTTP to return 405 for TRACE and OPTIONS 6a8cc9720d is described below commit 6a8cc9720d089821ae2eb8a631f26364f08f5ca0 Author: Mark Bean AuthorDate: Fri Mar 22 13:57:55 2024 -0400 NIFI-12925 Updated ListenHTTP to return 405 for TRACE and OPTIONS This closes #8548 Signed-off-by: David Handermann d (cherry picked from commit 5c150ffd786ec37be4821791654c8a65d4a49f6e) --- .../nifi/processors/standard/ListenHTTP.java | 3 +- .../standard/servlets/ListenHTTPServlet.java | 16 +++ .../nifi/processors/standard/TestListenHTTP.java | 52 +++--- 3 files changed, 55 insertions(+), 16 deletions(-) diff --git a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListenHTTP.java b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListenHTTP.java index 1a7b485f60..df3dfc6a8c 100644 --- a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListenHTTP.java +++ b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListenHTTP.java @@ -83,7 +83,8 @@ import java.util.stream.Collectors; @Tags({"ingest", "http", "https", "rest", "listen"}) @CapabilityDescription("Starts an HTTP Server and listens on a given base path to transform incoming requests into FlowFiles. " + "The default URI of the Service will be http://{hostname}:{port}/contentListener. Only HEAD and POST requests are " -+ "supported. GET, PUT, and DELETE will result in an error and the HTTP response status code 405. " ++ "supported. GET, PUT, DELETE, OPTIONS and TRACE will result in an error and the HTTP response status code 405; " ++ "CONNECT will also result in an error and the HTTP response status code 400. " + "GET is supported on /healthcheck. If the service is available, it returns \"200 OK\" with the content \"OK\". " + "The health check functionality can be configured to be accessible via a different port. " + "For details see the documentation of the \"Listening Port for health check requests\" property." diff --git a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/servlets/ListenHTTPServlet.java b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/servlets/ListenHTTPServlet.java index a321ece9f1..83c0a8bf48 100644 --- a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/servlets/ListenHTTPServlet.java +++ b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/servlets/ListenHTTPServlet.java @@ -156,6 +156,22 @@ public class ListenHTTPServlet extends HttpServlet { } } +private void notAllowed(final HttpServletRequest request, final HttpServletResponse response) throws IOException { +response.sendError(HttpServletResponse.SC_METHOD_NOT_ALLOWED, "Method Not Allowed"); +} + +@Override +protected void doTrace(final HttpServletRequest request, final HttpServletResponse response) throws ServletException, IOException { +notAllowed(request, response); +logger.debug("Denying TRACE request; method not allowed."); +} + +@Override +protected void doOptions(final HttpServletRequest request, final HttpServletResponse response) throws ServletException, IOException { +notAllowed(request, response); +logger.debug("Denying OPTIONS request; method not allowed."); +} + @Override protected void doPost(final HttpServletRequest request, final HttpServletResponse response) throws ServletException, IOException { diff --git a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestListenHTTP.java b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestListenHTTP.java index 643f50a626..53bbb6bc84 100644 --- a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestListenHTTP.java +++ b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestListenHTTP.java @@ -103,8 +103,13 @@ public class TestListenHTTP { private static final int SOCKET_CONNECT_TIMEOUT = 100; private static final long
(nifi) branch main updated: NIFI-12925 Updated ListenHTTP to return 405 for TRACE and OPTIONS
This is an automated email from the ASF dual-hosted git repository. exceptionfactory pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi.git The following commit(s) were added to refs/heads/main by this push: new 5c150ffd78 NIFI-12925 Updated ListenHTTP to return 405 for TRACE and OPTIONS 5c150ffd78 is described below commit 5c150ffd786ec37be4821791654c8a65d4a49f6e Author: Mark Bean AuthorDate: Fri Mar 22 13:57:55 2024 -0400 NIFI-12925 Updated ListenHTTP to return 405 for TRACE and OPTIONS This closes #8548 Signed-off-by: David Handermann d --- .../nifi/processors/standard/ListenHTTP.java | 3 +- .../standard/servlets/ListenHTTPServlet.java | 16 +++ .../nifi/processors/standard/TestListenHTTP.java | 52 +++--- 3 files changed, 55 insertions(+), 16 deletions(-) diff --git a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListenHTTP.java b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListenHTTP.java index 6d8fd2485d..ef07b182f7 100644 --- a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListenHTTP.java +++ b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListenHTTP.java @@ -82,7 +82,8 @@ import java.util.regex.Pattern; @Tags({"ingest", "http", "https", "rest", "listen"}) @CapabilityDescription("Starts an HTTP Server and listens on a given base path to transform incoming requests into FlowFiles. " + "The default URI of the Service will be http://{hostname}:{port}/contentListener. Only HEAD and POST requests are " -+ "supported. GET, PUT, and DELETE will result in an error and the HTTP response status code 405. " ++ "supported. GET, PUT, DELETE, OPTIONS and TRACE will result in an error and the HTTP response status code 405; " ++ "CONNECT will also result in an error and the HTTP response status code 400. " + "GET is supported on /healthcheck. If the service is available, it returns \"200 OK\" with the content \"OK\". " + "The health check functionality can be configured to be accessible via a different port. " + "For details see the documentation of the \"Listening Port for health check requests\" property." diff --git a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/servlets/ListenHTTPServlet.java b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/servlets/ListenHTTPServlet.java index 73f846e8d5..4a4bc49e32 100644 --- a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/servlets/ListenHTTPServlet.java +++ b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/servlets/ListenHTTPServlet.java @@ -157,6 +157,22 @@ public class ListenHTTPServlet extends HttpServlet { } } +private void notAllowed(final HttpServletRequest request, final HttpServletResponse response) throws IOException { +response.sendError(HttpServletResponse.SC_METHOD_NOT_ALLOWED, "Method Not Allowed"); +} + +@Override +protected void doTrace(final HttpServletRequest request, final HttpServletResponse response) throws ServletException, IOException { +notAllowed(request, response); +logger.debug("Denying TRACE request; method not allowed."); +} + +@Override +protected void doOptions(final HttpServletRequest request, final HttpServletResponse response) throws ServletException, IOException { +notAllowed(request, response); +logger.debug("Denying OPTIONS request; method not allowed."); +} + @Override protected void doPost(final HttpServletRequest request, final HttpServletResponse response) throws ServletException, IOException { diff --git a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestListenHTTP.java b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestListenHTTP.java index f0dbba3119..83c686edb5 100644 --- a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestListenHTTP.java +++ b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestListenHTTP.java @@ -103,8 +103,13 @@ public class TestListenHTTP { private static final int SOCKET_CONNECT_TIMEOUT = 100; private static final long SERVER_START_TIMEOUT = 120; + +private static final String HTTP_POST = "POST"; +private
(nifi) branch support/nifi-1.x updated: NIFI-12930 Catch FlowFileAccessException in FetchFile
This is an automated email from the ASF dual-hosted git repository. exceptionfactory pushed a commit to branch support/nifi-1.x in repository https://gitbox.apache.org/repos/asf/nifi.git The following commit(s) were added to refs/heads/support/nifi-1.x by this push: new 1e78a66a44 NIFI-12930 Catch FlowFileAccessException in FetchFile 1e78a66a44 is described below commit 1e78a66a44135191c73ac726c5e424113d9fbfaa Author: Joseph Witt AuthorDate: Thu Mar 21 13:08:09 2024 -0700 NIFI-12930 Catch FlowFileAccessException in FetchFile when importing a fetched file we cannot be sure if the problem is reading the source file or writing to the content repository. So we need to route to failure to allow flow designers to choose how to handle this. This closes #8542 Signed-off-by: David Handermann (cherry picked from commit d5ff51b6e4ee6907057b0838f627542f64b9f686) --- .../src/main/java/org/apache/nifi/processors/standard/FetchFile.java | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/FetchFile.java b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/FetchFile.java index e16eec7189..f506a3891d 100644 --- a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/FetchFile.java +++ b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/FetchFile.java @@ -37,6 +37,7 @@ import org.apache.nifi.processor.AbstractProcessor; import org.apache.nifi.processor.ProcessContext; import org.apache.nifi.processor.ProcessSession; import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.FlowFileAccessException; import org.apache.nifi.processor.exception.ProcessException; import org.apache.nifi.processor.util.StandardValidators; import org.apache.nifi.util.StopWatch; @@ -266,7 +267,7 @@ public class FetchFile extends AbstractProcessor { // import content from file system try (final FileInputStream fis = new FileInputStream(file)) { flowFile = session.importFrom(fis, flowFile); -} catch (final IOException ioe) { +} catch (final IOException | FlowFileAccessException ioe) { getLogger().error("Could not fetch file {} from file system for {} due to {}; routing to failure", file, flowFile, ioe.toString(), ioe); session.transfer(session.penalize(flowFile), REL_FAILURE); return;
(nifi) branch main updated: NIFI-12930 Catch FlowFileAccessException in FetchFile
This is an automated email from the ASF dual-hosted git repository. exceptionfactory pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi.git The following commit(s) were added to refs/heads/main by this push: new d5ff51b6e4 NIFI-12930 Catch FlowFileAccessException in FetchFile d5ff51b6e4 is described below commit d5ff51b6e4ee6907057b0838f627542f64b9f686 Author: Joseph Witt AuthorDate: Thu Mar 21 13:08:09 2024 -0700 NIFI-12930 Catch FlowFileAccessException in FetchFile when importing a fetched file we cannot be sure if the problem is reading the source file or writing to the content repository. So we need to route to failure to allow flow designers to choose how to handle this. This closes #8542 Signed-off-by: David Handermann --- .../src/main/java/org/apache/nifi/processors/standard/FetchFile.java | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/FetchFile.java b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/FetchFile.java index 8e90fe9815..cdf03e0fa0 100644 --- a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/FetchFile.java +++ b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/FetchFile.java @@ -39,6 +39,7 @@ import org.apache.nifi.processor.AbstractProcessor; import org.apache.nifi.processor.ProcessContext; import org.apache.nifi.processor.ProcessSession; import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.FlowFileAccessException; import org.apache.nifi.processor.exception.ProcessException; import org.apache.nifi.processor.util.StandardValidators; import org.apache.nifi.util.StopWatch; @@ -312,7 +313,7 @@ public class FetchFile extends AbstractProcessor { // import content from file system try (final FileInputStream fis = new FileInputStream(file)) { flowFile = session.importFrom(fis, flowFile); -} catch (final IOException ioe) { +} catch (final IOException | FlowFileAccessException ioe) { getLogger().error("Could not fetch file {} from file system for {} due to {}; routing to failure", file, flowFile, ioe.toString(), ioe); session.transfer(session.penalize(flowFile), REL_FAILURE); return;
(nifi) branch support/nifi-1.x updated (4009310f04 -> 75cc8c6a3e)
This is an automated email from the ASF dual-hosted git repository. exceptionfactory pushed a change to branch support/nifi-1.x in repository https://gitbox.apache.org/repos/asf/nifi.git from 4009310f04 NIFI-12887 Added Binary String Format property to PutDatabaseRecord add 75cc8c6a3e NIFI-6379 Added SSL Context to PutSNS, DeleteSQS, GetSQS, and PutSQS No new revisions were added by this update. Summary of changes: .../src/main/java/org/apache/nifi/processors/aws/sns/PutSNS.java| 6 +++--- .../src/main/java/org/apache/nifi/processors/aws/sqs/DeleteSQS.java | 2 +- .../src/main/java/org/apache/nifi/processors/aws/sqs/GetSQS.java| 2 +- .../src/main/java/org/apache/nifi/processors/aws/sqs/PutSQS.java| 2 +- 4 files changed, 6 insertions(+), 6 deletions(-)
(nifi) branch support/nifi-1.x updated: NIFI-12887 Added Binary String Format property to PutDatabaseRecord
This is an automated email from the ASF dual-hosted git repository. exceptionfactory pushed a commit to branch support/nifi-1.x in repository https://gitbox.apache.org/repos/asf/nifi.git The following commit(s) were added to refs/heads/support/nifi-1.x by this push: new 4009310f04 NIFI-12887 Added Binary String Format property to PutDatabaseRecord 4009310f04 is described below commit 4009310f0414719cd9c5f922b5b6330b8247aa3d Author: tpalfy AuthorDate: Tue Mar 12 13:33:01 2024 +0100 NIFI-12887 Added Binary String Format property to PutDatabaseRecord Add option to treat Strings as hexadecimal character sequences or base64-encoded binary data when inserting into a binary type column. This closes #8558 Signed-off-by: David Handermann --- .../processors/standard/PutDatabaseRecord.java | 42 +- .../processors/standard/PutDatabaseRecordTest.java | 89 +- 2 files changed, 129 insertions(+), 2 deletions(-) diff --git a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutDatabaseRecord.java b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutDatabaseRecord.java index fdad3fe56f..ef39491f4a 100644 --- a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutDatabaseRecord.java +++ b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutDatabaseRecord.java @@ -18,6 +18,7 @@ package org.apache.nifi.processors.standard; import com.github.benmanes.caffeine.cache.Cache; import com.github.benmanes.caffeine.cache.Caffeine; +import com.google.common.io.BaseEncoding; import org.apache.commons.lang3.StringUtils; import org.apache.nifi.annotation.behavior.EventDriven; import org.apache.nifi.annotation.behavior.InputRequirement; @@ -238,6 +239,34 @@ public class PutDatabaseRecord extends AbstractProcessor { .addValidator(StandardValidators.NON_EMPTY_VALIDATOR) .build(); +static final AllowableValue BINARY_STRING_FORMAT_UTF8 = new AllowableValue( +"UTF-8", +"UTF-8", +"String values for binary columns contain the original value as text via UTF-8 character encoding" +); + +static final AllowableValue BINARY_STRING_FORMAT_HEX_STRING = new AllowableValue( +"Hexadecimal", +"Hexadecimal", +"String values for binary columns contain the original value in hexadecimal format" +); + +static final AllowableValue BINARY_STRING_FORMAT_BASE64 = new AllowableValue( +"Base64", +"Base64", +"String values for binary columns contain the original value in Base64 encoded format" +); + +static final PropertyDescriptor BINARY_STRING_FORMAT = new Builder() +.name("put-db-record-binary-format") +.displayName("Binary String Format") +.description("The format to be applied when decoding string values to binary.") +.required(false) +.expressionLanguageSupported(FLOWFILE_ATTRIBUTES) +.allowableValues(BINARY_STRING_FORMAT_UTF8, BINARY_STRING_FORMAT_HEX_STRING, BINARY_STRING_FORMAT_BASE64) +.defaultValue(BINARY_STRING_FORMAT_UTF8.getValue()) +.build(); + static final PropertyDescriptor TRANSLATE_FIELD_NAMES = new Builder() .name("put-db-record-translate-field-names") .displayName("Translate Field Names") @@ -388,6 +417,7 @@ public class PutDatabaseRecord extends AbstractProcessor { pds.add(CATALOG_NAME); pds.add(SCHEMA_NAME); pds.add(TABLE_NAME); +pds.add(BINARY_STRING_FORMAT); pds.add(TRANSLATE_FIELD_NAMES); pds.add(UNMATCHED_FIELD_BEHAVIOR); pds.add(UNMATCHED_COLUMN_BEHAVIOR); @@ -619,6 +649,8 @@ public class PutDatabaseRecord extends AbstractProcessor { final int maxBatchSize = context.getProperty(MAX_BATCH_SIZE).evaluateAttributeExpressions(flowFile).asInteger(); final int timeoutMillis = context.getProperty(QUERY_TIMEOUT).evaluateAttributeExpressions().asTimePeriod(TimeUnit.MILLISECONDS).intValue(); +final String binaryStringFormat = context.getProperty(BINARY_STRING_FORMAT).evaluateAttributeExpressions(flowFile).getValue(); + // Ensure the table name has been set, the generated SQL statements (and TableSchema cache) will need it if (StringUtils.isEmpty(tableName)) { throw new IllegalArgumentException(format("Cannot process %s because Table Name is null or empty", flowFile)); @@ -777,7 +809,15 @@ public class PutDatabaseRecord extends AbstractProcessor { } currentValue = dest;
(nifi) branch support/nifi-1.x updated: NIFI-12943 Upgraded Hadoop from 3.3.6 to 3.4.0
This is an automated email from the ASF dual-hosted git repository. exceptionfactory pushed a commit to branch support/nifi-1.x in repository https://gitbox.apache.org/repos/asf/nifi.git The following commit(s) were added to refs/heads/support/nifi-1.x by this push: new e269304e23 NIFI-12943 Upgraded Hadoop from 3.3.6 to 3.4.0 e269304e23 is described below commit e269304e23abd9710f6a73ea670a318b3e305f73 Author: exceptionfactory AuthorDate: Mon Mar 25 09:06:27 2024 -0500 NIFI-12943 Upgraded Hadoop from 3.3.6 to 3.4.0 Signed-off-by: Joseph Witt --- nifi-nar-bundles/nifi-atlas-bundle/pom.xml | 4 nifi-nar-bundles/nifi-extension-utils/nifi-hadoop-utils/pom.xml | 9 + .../nifi-record-utils/nifi-hadoop-record-utils/pom.xml | 9 + nifi-nar-bundles/nifi-flume-bundle/nifi-flume-processors/pom.xml | 8 nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/pom.xml | 9 + .../nifi-hadoop-libraries-nar/pom.xml| 9 + nifi-nar-bundles/nifi-hive-bundle/nifi-hive-test-utils/pom.xml | 8 nifi-nar-bundles/nifi-hive-bundle/nifi-hive3-processors/pom.xml | 4 nifi-nar-bundles/nifi-iceberg-bundle/nifi-iceberg-common/pom.xml | 8 nifi-nar-bundles/nifi-iceberg-bundle/pom.xml | 5 + .../nifi-parquet-bundle/nifi-parquet-processors/pom.xml | 9 + .../nifi-hadoop-dbcp-service/pom.xml | 5 + pom.xml | 2 +- 13 files changed, 88 insertions(+), 1 deletion(-) diff --git a/nifi-nar-bundles/nifi-atlas-bundle/pom.xml b/nifi-nar-bundles/nifi-atlas-bundle/pom.xml index 7591140976..b94591de74 100644 --- a/nifi-nar-bundles/nifi-atlas-bundle/pom.xml +++ b/nifi-nar-bundles/nifi-atlas-bundle/pom.xml @@ -97,6 +97,10 @@ org.slf4j slf4j-reload4j + +org.bouncycastle +bcprov-jdk15on + diff --git a/nifi-nar-bundles/nifi-extension-utils/nifi-hadoop-utils/pom.xml b/nifi-nar-bundles/nifi-extension-utils/nifi-hadoop-utils/pom.xml index 1d96a417ec..94e9746c44 100644 --- a/nifi-nar-bundles/nifi-extension-utils/nifi-hadoop-utils/pom.xml +++ b/nifi-nar-bundles/nifi-extension-utils/nifi-hadoop-utils/pom.xml @@ -77,8 +77,17 @@ commons-logging commons-logging + + +org.bouncycastle +bcprov-jdk15on + + +org.bouncycastle +bcprov-jdk18on + org.apache.hadoop hadoop-auth diff --git a/nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-hadoop-record-utils/pom.xml b/nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-hadoop-record-utils/pom.xml index 97b1319ec2..2b02872246 100644 --- a/nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-hadoop-record-utils/pom.xml +++ b/nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-hadoop-record-utils/pom.xml @@ -56,8 +56,17 @@ commons-logging commons-logging + + +org.bouncycastle +bcprov-jdk15on + + +org.bouncycastle +bcprov-jdk18on + org.slf4j jcl-over-slf4j diff --git a/nifi-nar-bundles/nifi-flume-bundle/nifi-flume-processors/pom.xml b/nifi-nar-bundles/nifi-flume-bundle/nifi-flume-processors/pom.xml index b26b41cbba..b313679e91 100644 --- a/nifi-nar-bundles/nifi-flume-bundle/nifi-flume-processors/pom.xml +++ b/nifi-nar-bundles/nifi-flume-bundle/nifi-flume-processors/pom.xml @@ -140,6 +140,10 @@ commons-logging commons-logging + +org.bouncycastle +bcprov-jdk15on + @@ -158,6 +162,10 @@ + +org.bouncycastle +bcprov-jdk18on + org.apache.flume.flume-ng-sinks diff --git a/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/pom.xml b/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/pom.xml index ac4c059eb3..771b7f0fe7 100644 --- a/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/pom.xml +++ b/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/pom.xml @@ -68,8 +68,17 @@ commons-logging commons-logging + + +
(nifi) branch dependabot/npm_and_yarn/nifi-registry/nifi-registry-core/nifi-registry-web-ui/src/main/webpack-dev-middleware-and-karma-webpack-5.3.4 deleted (was d08ab61a4a)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/npm_and_yarn/nifi-registry/nifi-registry-core/nifi-registry-web-ui/src/main/webpack-dev-middleware-and-karma-webpack-5.3.4 in repository https://gitbox.apache.org/repos/asf/nifi.git was d08ab61a4a Bump webpack-dev-middleware and karma-webpack The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(nifi) branch main updated: Bump webpack-dev-middleware and karma-webpack (#8547)
This is an automated email from the ASF dual-hosted git repository. scottyaslan pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi.git The following commit(s) were added to refs/heads/main by this push: new 69f1a3be4d Bump webpack-dev-middleware and karma-webpack (#8547) 69f1a3be4d is described below commit 69f1a3be4dd0ede7912ee8cfd67d39bdc3d6eca0 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Mon Mar 25 16:18:04 2024 -0400 Bump webpack-dev-middleware and karma-webpack (#8547) Bumps [webpack-dev-middleware](https://github.com/webpack/webpack-dev-middleware) to 5.3.4 and updates ancestor dependency [karma-webpack](https://github.com/webpack-contrib/karma-webpack). These dependencies need to be updated together. Updates `webpack-dev-middleware` from 5.3.3 to 5.3.4 - [Release notes](https://github.com/webpack/webpack-dev-middleware/releases) - [Changelog](https://github.com/webpack/webpack-dev-middleware/blob/v5.3.4/CHANGELOG.md) - [Commits](https://github.com/webpack/webpack-dev-middleware/compare/v5.3.3...v5.3.4) Updates `karma-webpack` from 4.0.2 to 5.0.1 - [Release notes](https://github.com/webpack-contrib/karma-webpack/releases) - [Changelog](https://github.com/codymikol/karma-webpack/blob/master/CHANGELOG.md) - [Commits](https://github.com/webpack-contrib/karma-webpack/compare/v4.0.2...v5.0.1) --- updated-dependencies: - dependency-name: webpack-dev-middleware dependency-type: indirect - dependency-name: karma-webpack dependency-type: direct:development ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .../src/main/package-lock.json | 325 + .../nifi-registry-web-ui/src/main/package.json | 2 +- 2 files changed, 143 insertions(+), 184 deletions(-) diff --git a/nifi-registry/nifi-registry-core/nifi-registry-web-ui/src/main/package-lock.json b/nifi-registry/nifi-registry-core/nifi-registry-web-ui/src/main/package-lock.json index 5cf1680f38..1808e4f95e 100644 --- a/nifi-registry/nifi-registry-core/nifi-registry-web-ui/src/main/package-lock.json +++ b/nifi-registry/nifi-registry-core/nifi-registry-web-ui/src/main/package-lock.json @@ -66,7 +66,7 @@ "karma-jasmine": "2.0.1", "karma-jasmine-html-reporter": "1.4.0", "karma-spec-reporter": "0.0.32", -"karma-webpack": "4.0.2", +"karma-webpack": "5.0.1", "license-webpack-plugin": "2.1.1", "lint-staged": "12.2.2", "mini-css-extract-plugin": "0.12.0", @@ -2710,15 +2710,6 @@ "@angular/core": ">=2.0.0 <6.0.0" } }, -"node_modules/ansi-colors": { -"version": "3.2.4", -"resolved": "https://registry.npmjs.org/ansi-colors/-/ansi-colors-3.2.4.tgz;, -"integrity": "sha512-hHUXGagefjN2iRrID63xckIvotOXOojhQKWIPUZ4mNUZ9nLZW+7FMNoE1lOkEhNWYsx/7ysGIuJYCiMAA9FnrA==", -"dev": true, -"engines": { -"node": ">=6" -} -}, "node_modules/ansi-escapes": { "version": "4.3.2", "resolved": "https://registry.npmjs.org/ansi-escapes/-/ansi-escapes-4.3.2.tgz;, @@ -9306,58 +9297,44 @@ } }, "node_modules/karma-webpack": { -"version": "4.0.2", -"resolved": "https://registry.npmjs.org/karma-webpack/-/karma-webpack-4.0.2.tgz;, -"integrity": "sha512-970/okAsdUOmiMOCY8sb17A2I8neS25Ad9uhyK3GHgmRSIFJbDcNEFE8dqqUhNe9OHiCC9k3DMrSmtd/0ymP1A==", +"version": "5.0.1", +"resolved": "https://registry.npmjs.org/karma-webpack/-/karma-webpack-5.0.1.tgz;, +"integrity": "sha512-oo38O+P3W2mSPCSUrQdySSPv1LvPpXP+f+bBimNomS5sW+1V4SuhCuW8TfJzV+rDv921w2fDSDw0xJbPe6U+kQ==", "dev": true, "dependencies": { -"clone-deep": "^4.0.1", -"loader-utils": "^1.1.0", -"neo-async": "^2.6.1", -"schema-utils": "^1.0.0", -"source-map": "^0.7.3", -"webpack-dev-middleware": "^3.7.0" +"glob": "^7.1.3", +"minimatch": "^9.0.3", +"webpack-merge": "^4.1.5" }, "engines": { -"node": ">= 8.9.0" +"node": ">= 18" }, "peerDependencies": { -"webpack": "^4.0.0" +"webpack": "^5.0.0" } }, -"node_modules/karma-webpack/node_modules/json5": { -"version": "1.0.2", -"resolved": "https://registry.npmjs.org/json5/-/json5-1.0.2.tgz;, -"integrity":
(nifi) branch support/nifi-1.x updated: NIFI-12947 Upgraded MIME4J to 0.8.11
This is an automated email from the ASF dual-hosted git repository. pvillard pushed a commit to branch support/nifi-1.x in repository https://gitbox.apache.org/repos/asf/nifi.git The following commit(s) were added to refs/heads/support/nifi-1.x by this push: new 65ac492da0 NIFI-12947 Upgraded MIME4J to 0.8.11 65ac492da0 is described below commit 65ac492da077abce1add9639ebc60e203bbd1448 Author: exceptionfactory AuthorDate: Mon Mar 25 12:31:24 2024 -0500 NIFI-12947 Upgraded MIME4J to 0.8.11 Signed-off-by: Pierre Villard This closes #8561. --- nifi-nar-bundles/nifi-media-bundle/pom.xml | 6 ++ 1 file changed, 6 insertions(+) diff --git a/nifi-nar-bundles/nifi-media-bundle/pom.xml b/nifi-nar-bundles/nifi-media-bundle/pom.xml index 07cfa91846..d6f4c4305c 100644 --- a/nifi-nar-bundles/nifi-media-bundle/pom.xml +++ b/nifi-nar-bundles/nifi-media-bundle/pom.xml @@ -27,6 +27,7 @@ 5.2.5 +0.8.11 @@ -58,6 +59,11 @@ poi-ooxml ${poi.version} + +org.apache.james +apache-mime4j-core +${mime4j.version} +
(nifi) branch main updated: NIFI-12947 Upgraded MIME4J to 0.8.11
This is an automated email from the ASF dual-hosted git repository. pvillard pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi.git The following commit(s) were added to refs/heads/main by this push: new 407dd4d4bc NIFI-12947 Upgraded MIME4J to 0.8.11 407dd4d4bc is described below commit 407dd4d4bcb0cbf6fec037ebe2ad99551b4ce067 Author: exceptionfactory AuthorDate: Mon Mar 25 12:31:24 2024 -0500 NIFI-12947 Upgraded MIME4J to 0.8.11 Signed-off-by: Pierre Villard This closes #8561. --- nifi-code-coverage/pom.xml | 7 +++ nifi-nar-bundles/nifi-media-bundle/pom.xml | 6 ++ 2 files changed, 13 insertions(+) diff --git a/nifi-code-coverage/pom.xml b/nifi-code-coverage/pom.xml index f9cec5b9d1..a74b90f3be 100644 --- a/nifi-code-coverage/pom.xml +++ b/nifi-code-coverage/pom.xml @@ -31,6 +31,7 @@ 1.6.0 1.24.0 2.12.0 +0.8.11 @@ -125,6 +126,12 @@ sshd-osgi ${org.apache.sshd.version} + + +org.apache.james +apache-mime4j-core +${mime4j.version} + diff --git a/nifi-nar-bundles/nifi-media-bundle/pom.xml b/nifi-nar-bundles/nifi-media-bundle/pom.xml index 6555b16cc4..1c0c6a3cb4 100644 --- a/nifi-nar-bundles/nifi-media-bundle/pom.xml +++ b/nifi-nar-bundles/nifi-media-bundle/pom.xml @@ -27,6 +27,7 @@ 5.2.5 +0.8.11 @@ -58,6 +59,11 @@ poi-ooxml ${poi.version} + +org.apache.james +apache-mime4j-core +${mime4j.version} +
(nifi) branch main updated: Add auto commit property to QueryDatabaseTable and QueryDatabaseTable processors to allow disabling auto commit so PostgreSQL Fetch Size will work
This is an automated email from the ASF dual-hosted git repository. mattyb149 pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi.git The following commit(s) were added to refs/heads/main by this push: new 08ff54f5fb Add auto commit property to QueryDatabaseTable and QueryDatabaseTable processors to allow disabling auto commit so PostgreSQL Fetch Size will work 08ff54f5fb is described below commit 08ff54f5fb712d01f372fd25a2d904d605e271ba Author: Jim Steinebrey AuthorDate: Tue Mar 19 11:54:38 2024 -0400 Add auto commit property to QueryDatabaseTable and QueryDatabaseTable processors to allow disabling auto commit so PostgreSQL Fetch Size will work NIFI-1931 Add proper default value for auto commit (false) to PostgreSQLDatabaseAdapter to allow FETCH_SIZE to be honored on reads. NIFI-1931 Added customValidate code to check the auto commit property setting against the db adapter's required auto commit setting and give validation error message if they do not match. NIFI-1931 Added automated test to check the Auto Commit customValidate error message. NIFI-1931 remove clearDefaultValue() because it is not needed since required = false a;ready defaults it to null. This closes #8534 Signed-off-by: Matt Burgess --- .../standard/AbstractQueryDatabaseTable.java | 71 ++- .../processors/standard/QueryDatabaseTable.java| 1 + .../standard/QueryDatabaseTableRecord.java | 1 + .../processors/standard/db/DatabaseAdapter.java| 13 +++ .../db/impl/PostgreSQLDatabaseAdapter.java | 18 .../processors/standard/QueryDatabaseTableIT.java | 78 .../standard/QueryDatabaseTableRecordIT.java | 78 .../standard/QueryDatabaseTableRecordTest.java | 97 ++-- .../standard/QueryDatabaseTableTest.java | 101 +++-- .../db/impl/TestPostgreSQLDatabaseAdapter.java | 16 10 files changed, 456 insertions(+), 18 deletions(-) diff --git a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractQueryDatabaseTable.java b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractQueryDatabaseTable.java index e5fc6745d6..7f7a870fb7 100644 --- a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractQueryDatabaseTable.java +++ b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractQueryDatabaseTable.java @@ -90,16 +90,34 @@ public abstract class AbstractQueryDatabaseTable extends AbstractDatabaseFetchPr "TRANSACTION_SERIALIZABLE" ); +private static final String FETCH_SIZE_NAME = "Fetch Size"; +private static final String AUTO_COMMIT_NAME = "Set Auto Commit"; + public static final PropertyDescriptor FETCH_SIZE = new PropertyDescriptor.Builder() -.name("Fetch Size") +.name(FETCH_SIZE_NAME) .description("The number of result rows to be fetched from the result set at a time. This is a hint to the database driver and may not be " -+ "honored and/or exact. If the value specified is zero, then the hint is ignored.") ++ "honored and/or exact. If the value specified is zero, then the hint is ignored. " ++ "If using PostgreSQL, then '" + AUTO_COMMIT_NAME + "' must be equal to 'false' to cause '" + FETCH_SIZE_NAME + "' to take effect.") .defaultValue("0") .required(true) .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR) .expressionLanguageSupported(ExpressionLanguageScope.ENVIRONMENT) .build(); +public static final PropertyDescriptor AUTO_COMMIT = new PropertyDescriptor.Builder() +.name(AUTO_COMMIT_NAME) +.description("Allows enabling or disabling the auto commit functionality of the DB connection. Default value is 'No value set'. " + +"'No value set' will leave the db connection's auto commit mode unchanged. " + +"For some JDBC drivers such as PostgreSQL driver, it is required to disable the auto commit functionality " + +"to get the '" + FETCH_SIZE_NAME + "' setting to take effect. " + +"When auto commit is enabled, PostgreSQL driver ignores '" + FETCH_SIZE_NAME + "' setting and loads all rows of the result set to memory at once. " + +"This could lead for a large amount of memory usage when executing queries which fetch large data sets. " + +"More Details of this behaviour in PostgreSQL driver can be found in
(nifi) branch support/nifi-1.x updated (e97431f060 -> 15eeb1d5bc)
This is an automated email from the ASF dual-hosted git repository. mattyb149 pushed a change to branch support/nifi-1.x in repository https://gitbox.apache.org/repos/asf/nifi.git from e97431f060 NIFI-12936 ListGCSBucket resets its tracking state after configuration change add 15eeb1d5bc NIFI-1931: Add auto commit property to QueryDatabaseTable and QueryDatabaseTable processors to allow disabling auto commit so PostgreSQL Fetch Size will work No new revisions were added by this update. Summary of changes: .../standard/AbstractQueryDatabaseTable.java | 71 ++- .../processors/standard/QueryDatabaseTable.java| 1 + .../standard/QueryDatabaseTableRecord.java | 1 + .../processors/standard/db/DatabaseAdapter.java| 13 +++ .../db/impl/PostgreSQLDatabaseAdapter.java | 18 .../standard/QueryDatabaseTableRecordTest.java | 97 ++-- .../standard/QueryDatabaseTableTest.java | 101 +++-- .../db/impl/TestPostgreSQLDatabaseAdapter.java | 16 8 files changed, 300 insertions(+), 18 deletions(-)
(nifi) branch support/nifi-1.x updated: NIFI-12936 ListGCSBucket resets its tracking state after configuration change
This is an automated email from the ASF dual-hosted git repository. pvillard pushed a commit to branch support/nifi-1.x in repository https://gitbox.apache.org/repos/asf/nifi.git The following commit(s) were added to refs/heads/support/nifi-1.x by this push: new e97431f060 NIFI-12936 ListGCSBucket resets its tracking state after configuration change e97431f060 is described below commit e97431f0604fccad84c79048758eed567db117e1 Author: Peter Turcsanyi AuthorDate: Mon Mar 25 19:19:49 2024 +0100 NIFI-12936 ListGCSBucket resets its tracking state after configuration change Signed-off-by: Pierre Villard This closes #8563. --- .../nifi/processors/gcp/storage/ListGCSBucket.java | 63 +++- .../processors/gcp/storage/ListGCSBucketTest.java | 409 - 2 files changed, 281 insertions(+), 191 deletions(-) diff --git a/nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/storage/ListGCSBucket.java b/nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/storage/ListGCSBucket.java index 627f66654d..be18311356 100644 --- a/nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/storage/ListGCSBucket.java +++ b/nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/storage/ListGCSBucket.java @@ -69,6 +69,7 @@ import java.io.IOException; import java.io.OutputStream; import java.sql.Timestamp; import java.util.ArrayList; +import java.util.Arrays; import java.util.Collections; import java.util.HashMap; import java.util.HashSet; @@ -274,40 +275,64 @@ public class ListGCSBucket extends AbstractGCSProcessor { return relationships; } -// State tracking +private static final Set TRACKING_RESET_PROPERTIES = Collections.unmodifiableSet( +new HashSet<>(Arrays.asList( +BUCKET, +PREFIX, +LISTING_STRATEGY +)) +); + +// used by Tracking Timestamps tracking strategy public static final String CURRENT_TIMESTAMP = "currentTimestamp"; public static final String CURRENT_KEY_PREFIX = "key-"; private volatile long currentTimestamp = 0L; private final Set currentKeys = Collections.synchronizedSet(new HashSet<>()); -private volatile boolean justElectedPrimaryNode = false; -private volatile boolean resetEntityTrackingState = false; +// used by Tracking Entities tracking strategy private volatile ListedEntityTracker listedEntityTracker; +private volatile boolean justElectedPrimaryNode = false; +private volatile boolean resetTracking = false; + @OnPrimaryNodeStateChange public void onPrimaryNodeChange(final PrimaryNodeState newState) { justElectedPrimaryNode = (newState == PrimaryNodeState.ELECTED_PRIMARY_NODE); } +@Override +public void onPropertyModified(final PropertyDescriptor descriptor, final String oldValue, final String newValue) { +if (isConfigurationRestored() && TRACKING_RESET_PROPERTIES.contains(descriptor)) { +resetTracking = true; +} +} + @OnScheduled -public void initListedEntityTracker(ProcessContext context) { -final boolean isTrackingEntityStrategy = BY_ENTITIES.getValue().equals(context.getProperty(LISTING_STRATEGY).getValue()); -if (listedEntityTracker != null && (resetEntityTrackingState || !isTrackingEntityStrategy)) { +public void initTrackingStrategy(ProcessContext context) throws IOException { +final String listingStrategy = context.getProperty(LISTING_STRATEGY).getValue(); +final boolean isTrackingTimestampsStrategy = BY_TIMESTAMPS.getValue().equals(listingStrategy); +final boolean isTrackingEntityStrategy = BY_ENTITIES.getValue().equals(listingStrategy); + +if (resetTracking || !isTrackingTimestampsStrategy) { +context.getStateManager().clear(Scope.CLUSTER); +currentTimestamp = 0L; +currentKeys.clear(); +} + +if (listedEntityTracker != null && (resetTracking || !isTrackingEntityStrategy)) { try { listedEntityTracker.clearListedEntities(); +listedEntityTracker = null; } catch (IOException e) { throw new RuntimeException("Failed to reset previously listed entities", e); } } -resetEntityTrackingState = false; -if (isTrackingEntityStrategy) { -if (listedEntityTracker == null) { -listedEntityTracker = createListedEntityTracker(); -} -} else { -listedEntityTracker = null; +if (isTrackingEntityStrategy && listedEntityTracker == null) { +listedEntityTracker = createListedEntityTracker(); } + +resetTracking = false; }
(nifi) branch support/nifi-1.x updated: NIFI-12944 - Add PeerAddress as Attribute into the flowfile
This is an automated email from the ASF dual-hosted git repository. pvillard pushed a commit to branch support/nifi-1.x in repository https://gitbox.apache.org/repos/asf/nifi.git The following commit(s) were added to refs/heads/support/nifi-1.x by this push: new 3ce2deb706 NIFI-12944 - Add PeerAddress as Attribute into the flowfile 3ce2deb706 is described below commit 3ce2deb706247bbc17bc65304f0b41f4c2fabb5c Author: Ricardo Ferreira AuthorDate: Thu Mar 21 16:34:21 2024 + NIFI-12944 - Add PeerAddress as Attribute into the flowfile Signed-off-by: Pierre Villard This closes #8557. --- .../nifi/snmp/operations/SNMPTrapReceiver.java | 10 -- .../nifi/snmp/operations/SNMPTrapReceiverTest.java | 23 ++ 2 files changed, 31 insertions(+), 2 deletions(-) diff --git a/nifi-nar-bundles/nifi-snmp-bundle/nifi-snmp-processors/src/main/java/org/apache/nifi/snmp/operations/SNMPTrapReceiver.java b/nifi-nar-bundles/nifi-snmp-bundle/nifi-snmp-processors/src/main/java/org/apache/nifi/snmp/operations/SNMPTrapReceiver.java index d2eaf19528..c93bb83e46 100644 --- a/nifi-nar-bundles/nifi-snmp-bundle/nifi-snmp-processors/src/main/java/org/apache/nifi/snmp/operations/SNMPTrapReceiver.java +++ b/nifi-nar-bundles/nifi-snmp-bundle/nifi-snmp-processors/src/main/java/org/apache/nifi/snmp/operations/SNMPTrapReceiver.java @@ -25,6 +25,7 @@ import org.snmp4j.CommandResponder; import org.snmp4j.CommandResponderEvent; import org.snmp4j.PDU; import org.snmp4j.PDUv1; +import org.snmp4j.smi.Address; import java.util.Map; @@ -46,7 +47,7 @@ public class SNMPTrapReceiver implements CommandResponder { final PDU pdu = event.getPDU(); if (isValidTrapPdu(pdu)) { final ProcessSession processSession = processSessionFactory.createSession(); -final FlowFile flowFile = createFlowFile(processSession, pdu); +final FlowFile flowFile = createFlowFile(processSession,event); processSession.getProvenanceReporter().create(flowFile, event.getPeerAddress() + "/" + pdu.getRequestID()); if (pdu.getErrorStatus() == PDU.noError) { processSession.transfer(flowFile, REL_SUCCESS); @@ -59,14 +60,19 @@ public class SNMPTrapReceiver implements CommandResponder { } } -private FlowFile createFlowFile(final ProcessSession processSession, final PDU pdu) { +private FlowFile createFlowFile(final ProcessSession processSession, final CommandResponderEvent event) { FlowFile flowFile = processSession.create(); final Map attributes; +final PDU pdu = event.getPDU(); +final Address peerAddress = event.getPeerAddress(); if (pdu instanceof PDUv1) { attributes = SNMPUtils.getV1TrapPduAttributeMap((PDUv1) pdu); } else { attributes = SNMPUtils.getPduAttributeMap(pdu); } +if (peerAddress.isValid()) { +processSession.putAttribute(flowFile, SNMPUtils.SNMP_PROP_PREFIX + "peerAddress", peerAddress.toString()); +} flowFile = processSession.putAllAttributes(flowFile, attributes); return flowFile; } diff --git a/nifi-nar-bundles/nifi-snmp-bundle/nifi-snmp-processors/src/test/java/org/apache/nifi/snmp/operations/SNMPTrapReceiverTest.java b/nifi-nar-bundles/nifi-snmp-bundle/nifi-snmp-processors/src/test/java/org/apache/nifi/snmp/operations/SNMPTrapReceiverTest.java index eeca40eefe..0d273c3921 100644 --- a/nifi-nar-bundles/nifi-snmp-bundle/nifi-snmp-processors/src/test/java/org/apache/nifi/snmp/operations/SNMPTrapReceiverTest.java +++ b/nifi-nar-bundles/nifi-snmp-bundle/nifi-snmp-processors/src/test/java/org/apache/nifi/snmp/operations/SNMPTrapReceiverTest.java @@ -29,6 +29,7 @@ import org.junit.jupiter.api.Test; import org.snmp4j.CommandResponderEvent; import org.snmp4j.PDU; import org.snmp4j.PDUv1; +import org.snmp4j.smi.Address; import org.snmp4j.smi.OID; import org.snmp4j.smi.VariableBinding; @@ -95,9 +96,15 @@ class SNMPTrapReceiverTest { when(mockV1Pdu.getType()).thenReturn(PDU.V1TRAP); when(mockV1Pdu.getEnterprise()).thenReturn(new OID("1.3.6.1.2.1.1.1.0")); when(mockV1Pdu.getSpecificTrap()).thenReturn(4); + +final Address mockAddress = mock(Address.class); +when(mockAddress.toString()).thenReturn("127.0.0.1/62"); +when(mockAddress.isValid()).thenReturn(true); + final Vector vbs = new Vector<>(); doReturn(vbs).when(mockV1Pdu).getVariableBindings(); when(mockEvent.getPDU()).thenReturn(mockV1Pdu); +when(mockEvent.getPeerAddress()).thenReturn(mockAddress); when(mockProcessSessionFactory.createSession()).thenReturn(mockProcessSession); snmpTrapReceiver.processPdu(mockEvent); @@ -107,6 +114,8 @@ class SNMPTrapReceiverTest { assertEquals("1.3.6.1.2.1.1.1.0", flowFile.getAttribute("snmp$enterprise"));
(nifi) branch main updated: NIFI-12944 - Add PeerAddress as Attribute into the flowfile
This is an automated email from the ASF dual-hosted git repository. pvillard pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi.git The following commit(s) were added to refs/heads/main by this push: new 258715539e NIFI-12944 - Add PeerAddress as Attribute into the flowfile 258715539e is described below commit 258715539e7cf136152625aeafee5a2f8673eacd Author: Ricardo Ferreira AuthorDate: Thu Mar 21 16:34:21 2024 + NIFI-12944 - Add PeerAddress as Attribute into the flowfile Signed-off-by: Pierre Villard This closes #8557. --- .../nifi/snmp/operations/SNMPTrapReceiver.java | 10 -- .../nifi/snmp/operations/SNMPTrapReceiverTest.java | 23 ++ 2 files changed, 31 insertions(+), 2 deletions(-) diff --git a/nifi-nar-bundles/nifi-snmp-bundle/nifi-snmp-processors/src/main/java/org/apache/nifi/snmp/operations/SNMPTrapReceiver.java b/nifi-nar-bundles/nifi-snmp-bundle/nifi-snmp-processors/src/main/java/org/apache/nifi/snmp/operations/SNMPTrapReceiver.java index d2eaf19528..c93bb83e46 100644 --- a/nifi-nar-bundles/nifi-snmp-bundle/nifi-snmp-processors/src/main/java/org/apache/nifi/snmp/operations/SNMPTrapReceiver.java +++ b/nifi-nar-bundles/nifi-snmp-bundle/nifi-snmp-processors/src/main/java/org/apache/nifi/snmp/operations/SNMPTrapReceiver.java @@ -25,6 +25,7 @@ import org.snmp4j.CommandResponder; import org.snmp4j.CommandResponderEvent; import org.snmp4j.PDU; import org.snmp4j.PDUv1; +import org.snmp4j.smi.Address; import java.util.Map; @@ -46,7 +47,7 @@ public class SNMPTrapReceiver implements CommandResponder { final PDU pdu = event.getPDU(); if (isValidTrapPdu(pdu)) { final ProcessSession processSession = processSessionFactory.createSession(); -final FlowFile flowFile = createFlowFile(processSession, pdu); +final FlowFile flowFile = createFlowFile(processSession,event); processSession.getProvenanceReporter().create(flowFile, event.getPeerAddress() + "/" + pdu.getRequestID()); if (pdu.getErrorStatus() == PDU.noError) { processSession.transfer(flowFile, REL_SUCCESS); @@ -59,14 +60,19 @@ public class SNMPTrapReceiver implements CommandResponder { } } -private FlowFile createFlowFile(final ProcessSession processSession, final PDU pdu) { +private FlowFile createFlowFile(final ProcessSession processSession, final CommandResponderEvent event) { FlowFile flowFile = processSession.create(); final Map attributes; +final PDU pdu = event.getPDU(); +final Address peerAddress = event.getPeerAddress(); if (pdu instanceof PDUv1) { attributes = SNMPUtils.getV1TrapPduAttributeMap((PDUv1) pdu); } else { attributes = SNMPUtils.getPduAttributeMap(pdu); } +if (peerAddress.isValid()) { +processSession.putAttribute(flowFile, SNMPUtils.SNMP_PROP_PREFIX + "peerAddress", peerAddress.toString()); +} flowFile = processSession.putAllAttributes(flowFile, attributes); return flowFile; } diff --git a/nifi-nar-bundles/nifi-snmp-bundle/nifi-snmp-processors/src/test/java/org/apache/nifi/snmp/operations/SNMPTrapReceiverTest.java b/nifi-nar-bundles/nifi-snmp-bundle/nifi-snmp-processors/src/test/java/org/apache/nifi/snmp/operations/SNMPTrapReceiverTest.java index eeca40eefe..0d273c3921 100644 --- a/nifi-nar-bundles/nifi-snmp-bundle/nifi-snmp-processors/src/test/java/org/apache/nifi/snmp/operations/SNMPTrapReceiverTest.java +++ b/nifi-nar-bundles/nifi-snmp-bundle/nifi-snmp-processors/src/test/java/org/apache/nifi/snmp/operations/SNMPTrapReceiverTest.java @@ -29,6 +29,7 @@ import org.junit.jupiter.api.Test; import org.snmp4j.CommandResponderEvent; import org.snmp4j.PDU; import org.snmp4j.PDUv1; +import org.snmp4j.smi.Address; import org.snmp4j.smi.OID; import org.snmp4j.smi.VariableBinding; @@ -95,9 +96,15 @@ class SNMPTrapReceiverTest { when(mockV1Pdu.getType()).thenReturn(PDU.V1TRAP); when(mockV1Pdu.getEnterprise()).thenReturn(new OID("1.3.6.1.2.1.1.1.0")); when(mockV1Pdu.getSpecificTrap()).thenReturn(4); + +final Address mockAddress = mock(Address.class); +when(mockAddress.toString()).thenReturn("127.0.0.1/62"); +when(mockAddress.isValid()).thenReturn(true); + final Vector vbs = new Vector<>(); doReturn(vbs).when(mockV1Pdu).getVariableBindings(); when(mockEvent.getPDU()).thenReturn(mockV1Pdu); +when(mockEvent.getPeerAddress()).thenReturn(mockAddress); when(mockProcessSessionFactory.createSession()).thenReturn(mockProcessSession); snmpTrapReceiver.processPdu(mockEvent); @@ -107,6 +114,8 @@ class SNMPTrapReceiverTest { assertEquals("1.3.6.1.2.1.1.1.0", flowFile.getAttribute("snmp$enterprise")); assertEquals(String.valueOf(4),
(nifi) branch support/nifi-1.x updated: NIFI-12928 Added Simple Write strategy in PutAzureDataLakeStorage
This is an automated email from the ASF dual-hosted git repository. pvillard pushed a commit to branch support/nifi-1.x in repository https://gitbox.apache.org/repos/asf/nifi.git The following commit(s) were added to refs/heads/support/nifi-1.x by this push: new 87196e23d6 NIFI-12928 Added Simple Write strategy in PutAzureDataLakeStorage 87196e23d6 is described below commit 87196e23d651fa6edeae364c2189a7779e39060e Author: Peter Turcsanyi AuthorDate: Mon Mar 25 17:34:11 2024 +0100 NIFI-12928 Added Simple Write strategy in PutAzureDataLakeStorage Signed-off-by: Pierre Villard This closes #8559. --- .../azure/storage/PutAzureDataLakeStorage.java | 148 ++-- .../azure/storage/utils/WritingStrategy.java | 49 +++ .../additionalDetails.html | 44 -- .../azure/storage/ITPutAzureDataLakeStorage.java | 155 +++-- 4 files changed, 296 insertions(+), 100 deletions(-) diff --git a/nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/storage/PutAzureDataLakeStorage.java b/nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/storage/PutAzureDataLakeStorage.java index cfd660c289..c15c737f58 100644 --- a/nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/storage/PutAzureDataLakeStorage.java +++ b/nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/storage/PutAzureDataLakeStorage.java @@ -39,6 +39,7 @@ import org.apache.nifi.processor.ProcessSession; import org.apache.nifi.processor.exception.ProcessException; import org.apache.nifi.processors.azure.AbstractAzureDataLakeStorageProcessor; import org.apache.nifi.processors.azure.storage.utils.AzureStorageUtils; +import org.apache.nifi.processors.azure.storage.utils.WritingStrategy; import org.apache.nifi.processors.transfer.ResourceTransferSource; import org.apache.nifi.util.StringUtils; @@ -93,6 +94,15 @@ public class PutAzureDataLakeStorage extends AbstractAzureDataLakeStorageProcess .allowableValues(FAIL_RESOLUTION, REPLACE_RESOLUTION, IGNORE_RESOLUTION) .build(); +protected static final PropertyDescriptor WRITING_STRATEGY = new PropertyDescriptor.Builder() +.name("writing-strategy") +.displayName("Writing Strategy") +.description("Defines the approach for writing the Azure file.") +.required(true) +.allowableValues(WritingStrategy.class) +.defaultValue(WritingStrategy.WRITE_AND_RENAME.getValue()) +.build(); + public static final PropertyDescriptor BASE_TEMPORARY_PATH = new PropertyDescriptor.Builder() .name("base-temporary-path") .displayName("Base Temporary Path") @@ -102,6 +112,7 @@ public class PutAzureDataLakeStorage extends AbstractAzureDataLakeStorageProcess .defaultValue("") .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES) .addValidator(new DirectoryValidator("Base Temporary Path")) +.dependsOn(WRITING_STRATEGY, WritingStrategy.WRITE_AND_RENAME) .build(); private static final List PROPERTIES = Collections.unmodifiableList(Arrays.asList( @@ -109,6 +120,7 @@ public class PutAzureDataLakeStorage extends AbstractAzureDataLakeStorageProcess FILESYSTEM, DIRECTORY, FILE, +WRITING_STRATEGY, BASE_TEMPORARY_PATH, CONFLICT_RESOLUTION, RESOURCE_TRANSFER_SOURCE, @@ -131,41 +143,44 @@ public class PutAzureDataLakeStorage extends AbstractAzureDataLakeStorageProcess final long startNanos = System.nanoTime(); try { final String fileSystem = evaluateFileSystemProperty(context, flowFile); -final String originalDirectory = evaluateDirectoryProperty(context, flowFile); -final String tempPath = evaluateDirectoryProperty(context, flowFile, BASE_TEMPORARY_PATH); -final String tempDirectory = createPath(tempPath, TEMP_FILE_DIRECTORY); +final String directory = evaluateDirectoryProperty(context, flowFile); final String fileName = evaluateFileNameProperty(context, flowFile); final DataLakeFileSystemClient fileSystemClient = getFileSystemClient(context, flowFile, fileSystem); -final DataLakeDirectoryClient directoryClient = fileSystemClient.getDirectoryClient(originalDirectory); +final DataLakeDirectoryClient directoryClient = fileSystemClient.getDirectoryClient(directory); -final String tempFilePrefix = UUID.randomUUID().toString(); -final DataLakeDirectoryClient tempDirectoryClient = fileSystemClient.getDirectoryClient(tempDirectory); +final
(nifi) branch main updated: NIFI-12943 Upgraded Hadoop from 3.3.6 to 3.4.0 This closes #8556
This is an automated email from the ASF dual-hosted git repository. joewitt pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi.git The following commit(s) were added to refs/heads/main by this push: new dd9d1c978f NIFI-12943 Upgraded Hadoop from 3.3.6 to 3.4.0 This closes #8556 dd9d1c978f is described below commit dd9d1c978f7d4854ac0d53140f72f7c96c4e1d71 Author: exceptionfactory AuthorDate: Mon Mar 25 09:06:27 2024 -0500 NIFI-12943 Upgraded Hadoop from 3.3.6 to 3.4.0 This closes #8556 Signed-off-by: Joseph Witt --- nifi-nar-bundles/nifi-extension-utils/nifi-hadoop-utils/pom.xml | 9 + .../nifi-record-utils/nifi-hadoop-record-utils/pom.xml | 9 + nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/pom.xml | 9 + .../nifi-hadoop-libraries-nar/pom.xml| 9 + nifi-nar-bundles/nifi-hive-bundle/pom.xml| 5 + nifi-nar-bundles/nifi-iceberg-bundle/pom.xml | 5 + .../nifi-parquet-bundle/nifi-parquet-processors/pom.xml | 9 + .../nifi-hadoop-dbcp-service/pom.xml | 5 + pom.xml | 2 +- 9 files changed, 61 insertions(+), 1 deletion(-) diff --git a/nifi-nar-bundles/nifi-extension-utils/nifi-hadoop-utils/pom.xml b/nifi-nar-bundles/nifi-extension-utils/nifi-hadoop-utils/pom.xml index cf1d9242a3..d1e4968f83 100644 --- a/nifi-nar-bundles/nifi-extension-utils/nifi-hadoop-utils/pom.xml +++ b/nifi-nar-bundles/nifi-extension-utils/nifi-hadoop-utils/pom.xml @@ -76,8 +76,17 @@ commons-logging commons-logging + + +org.bouncycastle +bcprov-jdk15on + + +org.bouncycastle +bcprov-jdk18on + org.apache.hadoop hadoop-auth diff --git a/nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-hadoop-record-utils/pom.xml b/nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-hadoop-record-utils/pom.xml index 978495d3b3..e01ed8aa80 100644 --- a/nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-hadoop-record-utils/pom.xml +++ b/nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-hadoop-record-utils/pom.xml @@ -64,8 +64,17 @@ commons-logging commons-logging + + +org.bouncycastle +bcprov-jdk15on + + +org.bouncycastle +bcprov-jdk18on + org.slf4j jcl-over-slf4j diff --git a/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/pom.xml b/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/pom.xml index 357f057dbc..420086edde 100644 --- a/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/pom.xml +++ b/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/pom.xml @@ -67,8 +67,17 @@ commons-logging commons-logging + + +org.bouncycastle +bcprov-jdk15on + + +org.bouncycastle +bcprov-jdk18on + org.apache.hadoop hadoop-hdfs diff --git a/nifi-nar-bundles/nifi-hadoop-libraries-bundle/nifi-hadoop-libraries-nar/pom.xml b/nifi-nar-bundles/nifi-hadoop-libraries-bundle/nifi-hadoop-libraries-nar/pom.xml index 2d2a8c1f49..a0d2c466a4 100644 --- a/nifi-nar-bundles/nifi-hadoop-libraries-bundle/nifi-hadoop-libraries-nar/pom.xml +++ b/nifi-nar-bundles/nifi-hadoop-libraries-bundle/nifi-hadoop-libraries-nar/pom.xml @@ -73,8 +73,17 @@ org.eclipse.jetty jetty-rewrite + + +org.bouncycastle +bcprov-jdk15on + + +org.bouncycastle +bcprov-jdk18on + org.slf4j log4j-over-slf4j diff --git a/nifi-nar-bundles/nifi-hive-bundle/pom.xml b/nifi-nar-bundles/nifi-hive-bundle/pom.xml index 48c41e0089..1c7544f072 100644 --- a/nifi-nar-bundles/nifi-hive-bundle/pom.xml +++ b/nifi-nar-bundles/nifi-hive-bundle/pom.xml @@ -245,6 +245,11 @@ org.eclipse.jetty jetty-rewrite + + +org.bouncycastle +bcprov-jdk15on + diff --git
(nifi) branch main updated: NIFI-12936 ListGCSBucket resets its tracking state after configuration change
This is an automated email from the ASF dual-hosted git repository. pvillard pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi.git The following commit(s) were added to refs/heads/main by this push: new a9e246956c NIFI-12936 ListGCSBucket resets its tracking state after configuration change a9e246956c is described below commit a9e246956cee822e407b20e45930e05c3fd72de6 Author: Peter Turcsanyi AuthorDate: Fri Mar 22 22:32:27 2024 +0100 NIFI-12936 ListGCSBucket resets its tracking state after configuration change Signed-off-by: Pierre Villard This closes #8550. --- .../nifi/processors/gcp/storage/ListGCSBucket.java | 60 ++- .../processors/gcp/storage/ListGCSBucketTest.java | 451 - 2 files changed, 298 insertions(+), 213 deletions(-) diff --git a/nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/storage/ListGCSBucket.java b/nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/storage/ListGCSBucket.java index aed69ba1ee..97c6783a07 100644 --- a/nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/storage/ListGCSBucket.java +++ b/nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/storage/ListGCSBucket.java @@ -278,40 +278,62 @@ public class ListGCSBucket extends AbstractGCSProcessor { return RELATIONSHIPS; } -// State tracking +private static final Set TRACKING_RESET_PROPERTIES = Set.of( +BUCKET, +PREFIX, +LISTING_STRATEGY +); + +// used by Tracking Timestamps tracking strategy public static final String CURRENT_TIMESTAMP = "currentTimestamp"; public static final String CURRENT_KEY_PREFIX = "key-"; private volatile long currentTimestamp = 0L; private final Set currentKeys = Collections.synchronizedSet(new HashSet<>()); -private volatile boolean justElectedPrimaryNode = false; -private volatile boolean resetEntityTrackingState = false; +// used by Tracking Entities tracking strategy private volatile ListedEntityTracker listedEntityTracker; +private volatile boolean justElectedPrimaryNode = false; +private volatile boolean resetTracking = false; + @OnPrimaryNodeStateChange public void onPrimaryNodeChange(final PrimaryNodeState newState) { justElectedPrimaryNode = (newState == PrimaryNodeState.ELECTED_PRIMARY_NODE); } +@Override +public void onPropertyModified(final PropertyDescriptor descriptor, final String oldValue, final String newValue) { +if (isConfigurationRestored() && TRACKING_RESET_PROPERTIES.contains(descriptor)) { +resetTracking = true; +} +} + @OnScheduled -public void initListedEntityTracker(ProcessContext context) { -final boolean isTrackingEntityStrategy = BY_ENTITIES.getValue().equals(context.getProperty(LISTING_STRATEGY).getValue()); -if (listedEntityTracker != null && (resetEntityTrackingState || !isTrackingEntityStrategy)) { +public void initTrackingStrategy(ProcessContext context) throws IOException { +final String listingStrategy = context.getProperty(LISTING_STRATEGY).getValue(); +final boolean isTrackingTimestampsStrategy = BY_TIMESTAMPS.getValue().equals(listingStrategy); +final boolean isTrackingEntityStrategy = BY_ENTITIES.getValue().equals(listingStrategy); + +if (resetTracking || !isTrackingTimestampsStrategy) { +context.getStateManager().clear(Scope.CLUSTER); +currentTimestamp = 0L; +currentKeys.clear(); +} + +if (listedEntityTracker != null && (resetTracking || !isTrackingEntityStrategy)) { try { listedEntityTracker.clearListedEntities(); +listedEntityTracker = null; } catch (IOException e) { throw new RuntimeException("Failed to reset previously listed entities", e); } } -resetEntityTrackingState = false; -if (isTrackingEntityStrategy) { -if (listedEntityTracker == null) { -listedEntityTracker = createListedEntityTracker(); -} -} else { -listedEntityTracker = null; +if (isTrackingEntityStrategy && listedEntityTracker == null) { +listedEntityTracker = createListedEntityTracker(); } + +resetTracking = false; } protected ListedEntityTracker createListedEntityTracker() { @@ -1027,4 +1049,16 @@ public class ListGCSBucket extends AbstractGCSProcessor { return count; } } + +long getCurrentTimestamp() { +return currentTimestamp; +} + +ListedEntityTracker getListedEntityTracker() { +return listedEntityTracker; +} + +boolean
(nifi) branch main updated: NIFI-12942 Upgraded Jetty from 12.0.6 to 12.0.7
This is an automated email from the ASF dual-hosted git repository. pvillard pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi.git The following commit(s) were added to refs/heads/main by this push: new 15d2b49e77 NIFI-12942 Upgraded Jetty from 12.0.6 to 12.0.7 15d2b49e77 is described below commit 15d2b49e774bce5ffeb0d976e9101e3f4170504f Author: exceptionfactory AuthorDate: Mon Mar 25 08:39:55 2024 -0500 NIFI-12942 Upgraded Jetty from 12.0.6 to 12.0.7 Signed-off-by: Pierre Villard This closes #8554. --- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pom.xml b/pom.xml index c91980d907..9004ee27d3 100644 --- a/pom.xml +++ b/pom.xml @@ -129,7 +129,7 @@ 2.0.12 2.9.0 10.17.1.0 -12.0.6 +12.0.7 2.17.0 1.11.3 4.0.4
(nifi) branch main updated: NIFI-12928 Added Simple Write strategy in PutAzureDataLakeStorage
This is an automated email from the ASF dual-hosted git repository. pvillard pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi.git The following commit(s) were added to refs/heads/main by this push: new bffc342521 NIFI-12928 Added Simple Write strategy in PutAzureDataLakeStorage bffc342521 is described below commit bffc34252158103d0de727783c85c1cac727e5f4 Author: Peter Turcsanyi AuthorDate: Thu Mar 21 08:47:28 2024 +0100 NIFI-12928 Added Simple Write strategy in PutAzureDataLakeStorage Signed-off-by: Pierre Villard This closes #8540. --- .../azure/storage/PutAzureDataLakeStorage.java | 148 ++-- .../azure/storage/utils/WritingStrategy.java | 49 +++ .../additionalDetails.html | 44 -- .../storage/AbstractAzureDataLakeStorageIT.java| 2 + .../azure/storage/ITPutAzureDataLakeStorage.java | 155 +++-- 5 files changed, 298 insertions(+), 100 deletions(-) diff --git a/nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/storage/PutAzureDataLakeStorage.java b/nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/storage/PutAzureDataLakeStorage.java index 7651ad6a99..d385eeb27d 100644 --- a/nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/storage/PutAzureDataLakeStorage.java +++ b/nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/storage/PutAzureDataLakeStorage.java @@ -40,6 +40,7 @@ import org.apache.nifi.processor.exception.ProcessException; import org.apache.nifi.processors.azure.AbstractAzureDataLakeStorageProcessor; import org.apache.nifi.processors.azure.storage.utils.AzureStorageUtils; import org.apache.nifi.processors.azure.storage.utils.AzureStorageUtils.DirectoryValidator; +import org.apache.nifi.processors.azure.storage.utils.WritingStrategy; import org.apache.nifi.processors.transfer.ResourceTransferSource; import org.apache.nifi.util.StringUtils; @@ -99,6 +100,15 @@ public class PutAzureDataLakeStorage extends AbstractAzureDataLakeStorageProcess .allowableValues(FAIL_RESOLUTION, REPLACE_RESOLUTION, IGNORE_RESOLUTION) .build(); +protected static final PropertyDescriptor WRITING_STRATEGY = new PropertyDescriptor.Builder() +.name("writing-strategy") +.displayName("Writing Strategy") +.description("Defines the approach for writing the Azure file.") +.required(true) +.allowableValues(WritingStrategy.class) +.defaultValue(WritingStrategy.WRITE_AND_RENAME) +.build(); + public static final PropertyDescriptor BASE_TEMPORARY_PATH = new PropertyDescriptor.Builder() .name("base-temporary-path") .displayName("Base Temporary Path") @@ -108,6 +118,7 @@ public class PutAzureDataLakeStorage extends AbstractAzureDataLakeStorageProcess .defaultValue("") .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES) .addValidator(new DirectoryValidator("Base Temporary Path")) +.dependsOn(WRITING_STRATEGY, WritingStrategy.WRITE_AND_RENAME) .build(); private static final List PROPERTIES = List.of( @@ -115,6 +126,7 @@ public class PutAzureDataLakeStorage extends AbstractAzureDataLakeStorageProcess FILESYSTEM, DIRECTORY, FILE, +WRITING_STRATEGY, BASE_TEMPORARY_PATH, CONFLICT_RESOLUTION, RESOURCE_TRANSFER_SOURCE, @@ -137,41 +149,44 @@ public class PutAzureDataLakeStorage extends AbstractAzureDataLakeStorageProcess final long startNanos = System.nanoTime(); try { final String fileSystem = evaluateFileSystemProperty(FILESYSTEM, context, flowFile); -final String originalDirectory = evaluateDirectoryProperty(DIRECTORY, context, flowFile); -final String tempPath = evaluateDirectoryProperty(BASE_TEMPORARY_PATH, context, flowFile); -final String tempDirectory = createPath(tempPath, TEMP_FILE_DIRECTORY); +final String directory = evaluateDirectoryProperty(DIRECTORY, context, flowFile); final String fileName = evaluateFileProperty(context, flowFile); final DataLakeFileSystemClient fileSystemClient = getFileSystemClient(context, flowFile, fileSystem); -final DataLakeDirectoryClient directoryClient = fileSystemClient.getDirectoryClient(originalDirectory); +final DataLakeDirectoryClient directoryClient = fileSystemClient.getDirectoryClient(directory); -final String tempFilePrefix = UUID.randomUUID().toString(); -final DataLakeDirectoryClient tempDirectoryClient =
(nifi) branch support/nifi-1.x updated: NIFI-12929: Fix logout infinite redirect loop in case of knox
This is an automated email from the ASF dual-hosted git repository. pvillard pushed a commit to branch support/nifi-1.x in repository https://gitbox.apache.org/repos/asf/nifi.git The following commit(s) were added to refs/heads/support/nifi-1.x by this push: new 711a646197 NIFI-12929: Fix logout infinite redirect loop in case of knox 711a646197 is described below commit 711a64619798a1d79ed5e3629d2b2d0196a98417 Author: Zoltan Kornel Torok AuthorDate: Fri Mar 22 13:02:35 2024 +0100 NIFI-12929: Fix logout infinite redirect loop in case of knox Signed-off-by: Pierre Villard This closes #8546. --- .../src/main/java/org/apache/nifi/web/filter/LogoutFilter.java | 10 ++ 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/java/org/apache/nifi/web/filter/LogoutFilter.java b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/java/org/apache/nifi/web/filter/LogoutFilter.java index 832c2566df..c2f8905dbd 100644 --- a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/java/org/apache/nifi/web/filter/LogoutFilter.java +++ b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/java/org/apache/nifi/web/filter/LogoutFilter.java @@ -41,6 +41,10 @@ public class LogoutFilter implements Filter { private static final String SAML_SINGLE_LOGOUT_URL = "/nifi-api/access/saml/single-logout/request"; +private static final String KNOX_LOGOUT_URL = "/nifi-api/access/knox/logout"; + +private static final String LOGOUT_COMPLETE_URL = "/nifi-api/access/logout/complete"; + private ServletContext servletContext; @Override @@ -65,14 +69,12 @@ public class LogoutFilter implements Filter { if (supportsOidc) { sendRedirect(OIDC_LOGOUT_URL, request, response); } else if (supportsKnoxSso) { -final ServletContext apiContext = servletContext.getContext("/nifi-api"); - apiContext.getRequestDispatcher("/access/knox/logout").forward(request, response); +sendRedirect(KNOX_LOGOUT_URL, request, response); } else if (supportsSaml) { final String logoutUrl = supportsSamlSingleLogout ? SAML_SINGLE_LOGOUT_URL : SAML_LOCAL_LOGOUT_URL; sendRedirect(logoutUrl, request, response); } else { -final ServletContext apiContext = servletContext.getContext("/nifi-api"); - apiContext.getRequestDispatcher("/access/logout/complete").forward(request, response); +sendRedirect(LOGOUT_COMPLETE_URL, request, response); } }
(nifi) branch main updated: NIFI-12933 Rearranged properties on GCP processors
This is an automated email from the ASF dual-hosted git repository. pvillard pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi.git The following commit(s) were added to refs/heads/main by this push: new 8eb013a813 NIFI-12933 Rearranged properties on GCP processors 8eb013a813 is described below commit 8eb013a813ad5e8452a3eacd9774068a7bb4d3b3 Author: Peter Turcsanyi AuthorDate: Fri Mar 22 13:13:29 2024 +0100 NIFI-12933 Rearranged properties on GCP processors Also used current API methods for relationships/properties collections Signed-off-by: Pierre Villard This closes #8545. --- .../nifi/processors/gcp/AbstractGCPProcessor.java | 9 -- .../gcp/bigquery/AbstractBigQueryProcessor.java| 17 ++ .../nifi/processors/gcp/bigquery/PutBigQuery.java | 12 ++- .../pubsub/AbstractGCPubSubWithProxyProcessor.java | 10 -- .../processors/gcp/pubsub/ConsumeGCPubSub.java | 35 .../processors/gcp/pubsub/PublishGCPubSub.java | 37 -- .../gcp/pubsub/lite/ConsumeGCPubSubLite.java | 34 +++- .../gcp/pubsub/lite/PublishGCPubSubLite.java | 25 --- .../gcp/storage/AbstractGCSProcessor.java | 15 ++--- .../processors/gcp/storage/DeleteGCSObject.java| 19 +++ .../processors/gcp/storage/FetchGCSObject.java | 26 +-- .../nifi/processors/gcp/storage/ListGCSBucket.java | 31 ++ .../nifi/processors/gcp/storage/PutGCSObject.java | 33 +++ 13 files changed, 146 insertions(+), 157 deletions(-) diff --git a/nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/AbstractGCPProcessor.java b/nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/AbstractGCPProcessor.java index 0650451a5b..db6fc1e722 100644 --- a/nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/AbstractGCPProcessor.java +++ b/nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/AbstractGCPProcessor.java @@ -85,15 +85,6 @@ public abstract class AbstractGCPProcessor< return cloudService; } -@Override -public List getSupportedPropertyDescriptors() { -return List.of(PROJECT_ID, -GCP_CREDENTIALS_PROVIDER_SERVICE, -RETRY_COUNT, -PROXY_CONFIGURATION_SERVICE -); -} - @Override public void migrateProperties(final PropertyConfiguration config) { diff --git a/nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/bigquery/AbstractBigQueryProcessor.java b/nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/bigquery/AbstractBigQueryProcessor.java index c2652b29f2..2012c4c474 100644 --- a/nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/bigquery/AbstractBigQueryProcessor.java +++ b/nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/bigquery/AbstractBigQueryProcessor.java @@ -39,10 +39,8 @@ import org.apache.nifi.proxy.ProxyConfiguration; import org.apache.nifi.util.StringUtils; import java.util.ArrayList; -import java.util.Arrays; import java.util.Collection; import java.util.Collections; -import java.util.HashSet; import java.util.List; import java.util.Map; import java.util.Set; @@ -63,8 +61,7 @@ public abstract class AbstractBigQueryProcessor extends AbstractGCPProcessor relationships = Collections.unmodifiableSet( -new HashSet<>(Arrays.asList(REL_SUCCESS, REL_FAILURE))); +public static final Set RELATIONSHIPS = Set.of(REL_SUCCESS, REL_FAILURE); public static final PropertyDescriptor DATASET = new PropertyDescriptor.Builder() .name(BigQueryAttributes.DATASET_ATTR) @@ -98,17 +95,7 @@ public abstract class AbstractBigQueryProcessor extends AbstractGCPProcessor getRelationships() { -return relationships; -} - -@Override -public List getSupportedPropertyDescriptors() { -final List descriptors = new ArrayList<>(); -descriptors.addAll(super.getSupportedPropertyDescriptors()); -descriptors.add(DATASET); -descriptors.add(TABLE_NAME); -descriptors.add(IGNORE_UNKNOWN); -return Collections.unmodifiableList(descriptors); +return RELATIONSHIPS; } @Override diff --git a/nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/bigquery/PutBigQuery.java b/nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/bigquery/PutBigQuery.java index 8536f232c7..cb51362a08 100644 ---
(nifi) branch support/nifi-1.x updated: NIFI-12895 Added Timeout property to GetSmbFile and PutSmbFile
This is an automated email from the ASF dual-hosted git repository. pvillard pushed a commit to branch support/nifi-1.x in repository https://gitbox.apache.org/repos/asf/nifi.git The following commit(s) were added to refs/heads/support/nifi-1.x by this push: new cc81eeffb2 NIFI-12895 Added Timeout property to GetSmbFile and PutSmbFile cc81eeffb2 is described below commit cc81eeffb21604059b32a0cb80992d3b5d6fe941 Author: Peter Turcsanyi AuthorDate: Sat Mar 23 18:58:52 2024 +0100 NIFI-12895 Added Timeout property to GetSmbFile and PutSmbFile Signed-off-by: Pierre Villard This closes #8551. --- .../src/main/java/org/apache/nifi/processors/smb/GetSmbFile.java| 2 ++ .../src/main/java/org/apache/nifi/processors/smb/PutSmbFile.java| 2 ++ 2 files changed, 4 insertions(+) diff --git a/nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/GetSmbFile.java b/nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/GetSmbFile.java index 244ae4639f..ed843f9bff 100644 --- a/nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/GetSmbFile.java +++ b/nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/GetSmbFile.java @@ -81,6 +81,7 @@ import java.util.concurrent.locks.ReentrantLock; import java.util.regex.Pattern; import static org.apache.nifi.smb.common.SmbProperties.SMB_DIALECT; +import static org.apache.nifi.smb.common.SmbProperties.TIMEOUT; import static org.apache.nifi.smb.common.SmbProperties.USE_ENCRYPTION; import static org.apache.nifi.smb.common.SmbUtils.buildSmbClient; @@ -256,6 +257,7 @@ public class GetSmbFile extends AbstractProcessor { descriptors.add(IGNORE_HIDDEN_FILES); descriptors.add(SMB_DIALECT); descriptors.add(USE_ENCRYPTION); +descriptors.add(TIMEOUT); this.descriptors = Collections.unmodifiableList(descriptors); final Set relationships = new HashSet(); diff --git a/nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/PutSmbFile.java b/nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/PutSmbFile.java index b7ee444ab9..468828f819 100644 --- a/nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/PutSmbFile.java +++ b/nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/PutSmbFile.java @@ -65,6 +65,7 @@ import java.util.Set; import java.util.concurrent.TimeUnit; import static org.apache.nifi.smb.common.SmbProperties.SMB_DIALECT; +import static org.apache.nifi.smb.common.SmbProperties.TIMEOUT; import static org.apache.nifi.smb.common.SmbProperties.USE_ENCRYPTION; import static org.apache.nifi.smb.common.SmbUtils.buildSmbClient; @@ -193,6 +194,7 @@ public class PutSmbFile extends AbstractProcessor { descriptors.add(RENAME_SUFFIX); descriptors.add(SMB_DIALECT); descriptors.add(USE_ENCRYPTION); +descriptors.add(TIMEOUT); this.descriptors = Collections.unmodifiableList(descriptors); final Set relationships = new HashSet();
(nifi) branch main updated: NIFI-12895 Added Timeout property to GetSmbFile and PutSmbFile
This is an automated email from the ASF dual-hosted git repository. pvillard pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi.git The following commit(s) were added to refs/heads/main by this push: new 635eb9ed2a NIFI-12895 Added Timeout property to GetSmbFile and PutSmbFile 635eb9ed2a is described below commit 635eb9ed2a7f5b74e3b0914eda75c0d3ae302563 Author: Peter Turcsanyi AuthorDate: Sat Mar 23 18:58:52 2024 +0100 NIFI-12895 Added Timeout property to GetSmbFile and PutSmbFile Signed-off-by: Pierre Villard This closes #8551. --- .../src/main/java/org/apache/nifi/processors/smb/GetSmbFile.java| 2 ++ .../src/main/java/org/apache/nifi/processors/smb/PutSmbFile.java| 2 ++ 2 files changed, 4 insertions(+) diff --git a/nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/GetSmbFile.java b/nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/GetSmbFile.java index c938ce83f3..b4089f5287 100644 --- a/nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/GetSmbFile.java +++ b/nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/GetSmbFile.java @@ -81,6 +81,7 @@ import java.util.concurrent.locks.ReentrantLock; import java.util.regex.Pattern; import static org.apache.nifi.smb.common.SmbProperties.SMB_DIALECT; +import static org.apache.nifi.smb.common.SmbProperties.TIMEOUT; import static org.apache.nifi.smb.common.SmbProperties.USE_ENCRYPTION; import static org.apache.nifi.smb.common.SmbUtils.buildSmbClient; @@ -256,6 +257,7 @@ public class GetSmbFile extends AbstractProcessor { descriptors.add(IGNORE_HIDDEN_FILES); descriptors.add(SMB_DIALECT); descriptors.add(USE_ENCRYPTION); +descriptors.add(TIMEOUT); this.descriptors = Collections.unmodifiableList(descriptors); final Set relationships = new HashSet(); diff --git a/nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/PutSmbFile.java b/nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/PutSmbFile.java index b7ee444ab9..468828f819 100644 --- a/nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/PutSmbFile.java +++ b/nifi-nar-bundles/nifi-smb-bundle/nifi-smb-processors/src/main/java/org/apache/nifi/processors/smb/PutSmbFile.java @@ -65,6 +65,7 @@ import java.util.Set; import java.util.concurrent.TimeUnit; import static org.apache.nifi.smb.common.SmbProperties.SMB_DIALECT; +import static org.apache.nifi.smb.common.SmbProperties.TIMEOUT; import static org.apache.nifi.smb.common.SmbProperties.USE_ENCRYPTION; import static org.apache.nifi.smb.common.SmbUtils.buildSmbClient; @@ -193,6 +194,7 @@ public class PutSmbFile extends AbstractProcessor { descriptors.add(RENAME_SUFFIX); descriptors.add(SMB_DIALECT); descriptors.add(USE_ENCRYPTION); +descriptors.add(TIMEOUT); this.descriptors = Collections.unmodifiableList(descriptors); final Set relationships = new HashSet();
(nifi) branch main updated: NIFI-12938: Upgrade Iceberg from 1.4.3 to 1.5.0
This is an automated email from the ASF dual-hosted git repository. pvillard pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi.git The following commit(s) were added to refs/heads/main by this push: new 8c0829d711 NIFI-12938: Upgrade Iceberg from 1.4.3 to 1.5.0 8c0829d711 is described below commit 8c0829d711da9d38ef87302c4d2f12ced9e7167f Author: Mark Bathori AuthorDate: Sun Mar 24 20:31:13 2024 +0100 NIFI-12938: Upgrade Iceberg from 1.4.3 to 1.5.0 Signed-off-by: Pierre Villard This closes #8552. --- nifi-nar-bundles/nifi-iceberg-bundle/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nifi-nar-bundles/nifi-iceberg-bundle/pom.xml b/nifi-nar-bundles/nifi-iceberg-bundle/pom.xml index 313daffe3c..f6c96fb45b 100644 --- a/nifi-nar-bundles/nifi-iceberg-bundle/pom.xml +++ b/nifi-nar-bundles/nifi-iceberg-bundle/pom.xml @@ -25,7 +25,7 @@ pom -1.4.3 +1.5.0 3.1.3
(nifi) branch support/nifi-1.x updated: NIFI-12938: Upgrade Iceberg from 1.4.3 to 1.5.0
This is an automated email from the ASF dual-hosted git repository. pvillard pushed a commit to branch support/nifi-1.x in repository https://gitbox.apache.org/repos/asf/nifi.git The following commit(s) were added to refs/heads/support/nifi-1.x by this push: new c4a2738606 NIFI-12938: Upgrade Iceberg from 1.4.3 to 1.5.0 c4a2738606 is described below commit c4a27386069de8c6174f63c6f61532c092725f97 Author: Mark Bathori AuthorDate: Sun Mar 24 20:31:13 2024 +0100 NIFI-12938: Upgrade Iceberg from 1.4.3 to 1.5.0 Signed-off-by: Pierre Villard This closes #8552. --- nifi-nar-bundles/nifi-iceberg-bundle/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nifi-nar-bundles/nifi-iceberg-bundle/pom.xml b/nifi-nar-bundles/nifi-iceberg-bundle/pom.xml index 13c9096f17..44c3e85dba 100644 --- a/nifi-nar-bundles/nifi-iceberg-bundle/pom.xml +++ b/nifi-nar-bundles/nifi-iceberg-bundle/pom.xml @@ -25,7 +25,7 @@ pom -1.4.3 +1.5.0 3.1.3