[jira] [Created] (NIFI-9257) ExecuteSQLRecord processor doesn't honor the max wait time config when set to 5mins
sneha sriram created NIFI-9257: -- Summary: ExecuteSQLRecord processor doesn't honor the max wait time config when set to 5mins Key: NIFI-9257 URL: https://issues.apache.org/jira/browse/NIFI-9257 Project: Apache NiFi Issue Type: Bug Components: Extensions Affects Versions: 1.12.1 Environment: software platform Reporter: sneha sriram For postgres function/SQL which take longer to run(~more than 5 min), Assume SQL takes 8 min to complete, so to avoid query getting killed by ExecuteSQLRecord processor ( with Max Wait time = 5 min), need to increase Max Wait time > 8 min. So updated Max Wait time = 10 mins * ExecuteSQLRecord processor receives flow files from incoming queue * ExecuteSQLRecord processor triggers/submit postgres functions/SQL to aurora postgres server * If aurora postgres function/SQL is taking longer to process(~more than 5 min), *ExecuteSQLRecord processor is getting stuck in running state,* even after postgres function/SQL completes its execution on aurora postgres server * It stays in running stuck state indefinitely * All downstream processors are not getting any flow files to run * This issue was observed with different postgres functions and SQLs -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-9253) Build broken by nifi-security-socket-ssl test issue
[ https://issues.apache.org/jira/browse/NIFI-9253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-9253: --- Affects Version/s: 1.14.0 Status: Patch Available (was: In Progress) > Build broken by nifi-security-socket-ssl test issue > --- > > Key: NIFI-9253 > URL: https://issues.apache.org/jira/browse/NIFI-9253 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.14.0 > Environment: Apache Maven 3.8.1 > (05c21c65bdfed0f71a2f2ada8b84da59348c4c5d) > Maven home: /development/tools/apache-maven-3.8.1 > Java version: 11.0.12, vendor: Azul Systems, Inc., runtime: > /usr/lib/jvm/zulu11.50.19-ca-jdk11.0.12-linux_x64 > Default locale: en_US, platform encoding: UTF-8 > OS name: "linux", version: "5.13.15-200.fc34.x86_64", arch: "amd64", family: > "unix" >Reporter: Joe Witt >Assignee: David Handermann >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > [ERROR] WARNING: An illegal reflective access operation has occurred > [ERROR] WARNING: Illegal reflective access by > com.esotericsoftware.kryo.util.UnsafeUtil > (file:/home/apachenifi/.m2/repository/com/esotericsoftware/kryo/kryo/2.24.0/kryo-2.24.0.jar) > to constructor java.nio.DirectByteBuffer(long,int,java.lang.Object) > [ERROR] WARNING: Please consider reporting this to the maintainers of > com.esotericsoftware.kryo.util.UnsafeUtil > [ERROR] WARNING: Use --illegal-access=warn to enable warnings of further > illegal reflective access operations > [ERROR] WARNING: All illegal access operations will be denied in a future > release > [ERROR] WARNING: An illegal reflective access operation has occurred > [ERROR] WARNING: Illegal reflective access by > org.codehaus.groovy.reflection.CachedClass > (file:/home/apachenifi/.m2/repository/org/codehaus/groovy/groovy/2.5.14/groovy-2.5.14.jar) > to method java.lang.Object.finalize() > [ERROR] WARNING: Please consider reporting this to the maintainers of > org.codehaus.groovy.reflection.CachedClass > [ERROR] WARNING: Use --illegal-access=warn to enable warnings of further > illegal reflective access operations > [ERROR] WARNING: All illegal access operations will be denied in a future > release > [INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.424 > s - in > org.apache.nifi.authorization.CompositeConfigurableUserGroupProviderTest > [INFO] Running org.apache.nifi.authorization.StandardManagedAuthorizerTest > [INFO] Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.17 > s - in org.apache.nifi.authorization.StandardManagedAuthorizerTest > [INFO] Running org.apache.nifi.authorization.CompositeUserGroupProviderTest > [INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.014 > s - in org.apache.nifi.authorization.CompositeUserGroupProviderTest > [INFO] Running org.apache.nifi.authorization.user.NiFiUserUtilsTest > [INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.485 > s - in org.apache.nifi.properties.AESSensitivePropertyProviderTest > [ERROR] WARNING: An illegal reflective access operation has occurred > [ERROR] WARNING: Illegal reflective access by > org.codehaus.groovy.reflection.CachedClass > (file:/home/apachenifi/.m2/repository/org/codehaus/groovy/groovy/2.5.14/groovy-2.5.14.jar) > to method java.lang.Object.finalize() > [ERROR] WARNING: Please consider reporting this to the maintainers of > org.codehaus.groovy.reflection.CachedClass > [ERROR] WARNING: Use --illegal-access=warn to enable warnings of further > illegal reflective access operations > [ERROR] WARNING: All illegal access operations will be denied in a future > release > [ERROR] Errors: > [ERROR] > SSLSocketChannelTest.testClientConnectWriteReadTls13:147->assertChannelConnectedWriteReadClosed:208->processClientSslSocketChannel:255->lambda$assertChannelConnectedWriteReadClosed$1:218 > UncheckedIO > [INFO] > [ERROR] Tests run: 6, Failures: 0, Errors: 1, Skipped: 0 > [INFO] Skipping because packaging 'nar' is not pom. > [ERROR] Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 12.95 > s <<< FAILURE! - in org.apache.nifi.remote.io.socket.ssl.SSLSocketChannelTest > [ERROR] > org.apache.nifi.remote.io.socket.ssl.SSLSocketChannelTest.testClientConnectWriteReadTls13 > Time elapsed: 2.116 s <<< ERROR! > java.io.UncheckedIOException: Channel Failed for TLSv1.3 > at > org.apache.nifi.remote.io.socket.ssl.SSLSocketChannelTest.lambda$assertChannelConnectedWriteReadClosed$1(SSLSocketChannelTest.java:218) > at > org.apache.nifi.remote.io.socket.ssl.SSLSocketChannelTest.processClientSslSocketChannel(SSLSocketChannelTest.java:255) > at >
[GitHub] [nifi] exceptionfactory opened a new pull request #5421: NIFI-9253 Corrected SSLSocketChannel.available() for TLSv1.3
exceptionfactory opened a new pull request #5421:
URL: https://github.com/apache/nifi/pull/5421
Description of PR
NIFI-9253 Corrects `SSLSocketChannel` to avoid intermittent unit test and
runtime failures related to TLSv1.3. The `available()` method implementation
resulted in sporadic unit test failures for TLSv1.3 due to partial reads from
the connected channel while attempting to determine available bytes. Partial
reads produced incomplete TLS messages, causing AES-GCM decryption failures
reported as `Tag mismatch` messages with `SSLException`.
The previous unit test used a loop for checking the `available()` bytes,
causing intermittent failures. The updated unit test class includes separate
methods with and without calls to `available()` in order to reproduce the
potential problem. The corrected `available()` method in `SSLSocketChannel`
avoids direct reads from the socket channel and instead returns the size of the
application data buffer. This approach aligns with the documentation for
`InputStream.available()`, which states that the method returns an estimate of
the number of bytes that may be read, without blocking. Additional changes
include refactoring and isolating the `readChannel()` and `unwrap()` methods in
`SSLSocketChannel` for more streamlined handling of buffer underflow status.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [X] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [X] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [X] Has your PR been rebased against the latest commit within the target
branch (typically `main`)?
- [X] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [X] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [X] Have you written or updated unit tests to verify your changes?
- [X] Have you verified that the full build is successful on JDK 8?
- [X] Have you verified that the full build is successful on JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for
build issues and submit an update to your PR as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (NIFI-9256) QueryDatabaseTable Processor Default Decimal Scale not working
GorkemG created NIFI-9256:
-
Summary: QueryDatabaseTable Processor Default Decimal Scale not
working
Key: NIFI-9256
URL: https://issues.apache.org/jira/browse/NIFI-9256
Project: Apache NiFi
Issue Type: Bug
Components: Core Framework
Affects Versions: 1.14.0
Environment: Linux HOSTNAME 4.18.0-305.19.1.el8_4.x86_64 #1 SMP Tue
Sep 7 07:07:31 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux
openjdk version "1.8.0_302"
OpenJDK Runtime Environment (build 1.8.0_302-b08)
OpenJDK 64-Bit Server VM (build 25.302-b08, mixed mode)
Reporter: GorkemG
When using QueryDatabaseTable processor , and to infer schemea from table set
Use Avro Logical Types = True, whether you change Default Decimal Precision
value to other than default 10 or not, and set Default Decimal Scale to desired
number (other than default 0) , in the output avro schema scale is "always" set
to whatever Default Decimal Precision is.
E.g. if Default Decimal Precision = 38 -> Default Decimal Scale is always 38
(irrelevant to value)
On the other hand ExecuteSQL Processor is working as intended, changing scale
and precision affects avro schema (using same OracleSQL connection).
Note that QueryDatabaseTable is handy for incremental fetching, and if Avro
Logical Types is false (default) columns are always string type and when you
convert to for example Parquet, you miss the schema types and there is no handy
way to pass out that information.
Usually when reading Parquet files Precision=38 and Scale=18 is required (due
to other software conversion missing features).
Using QueryDatabaseTable
"fields":[{"name":"ID","type":["null",{"type":"bytes","logicalType":"decimal","precision":38,*"scale":38*}]}
Using ExecuteSQL
{"name":"ID","type":["null",\{"type":"bytes","logicalType":"decimal","precision":38,"scale":18}]}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [nifi] exceptionfactory commented on a change in pull request #5409: NIFI-9235 - Improve PutHDFS documentation; conflict detection
exceptionfactory commented on a change in pull request #5409:
URL: https://github.com/apache/nifi/pull/5409#discussion_r718052930
##
File path:
nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/PutHDFS.java
##
@@ -228,6 +240,14 @@ protected void preProcessConfiguration(final Configuration
config, final Process
FsPermission.setUMask(config, new FsPermission(dfsUmask));
}
+@OnScheduled
+public void onScheduled(final ProcessContext context) {
Review comment:
Should there a corresponding `onStopped` method that clears the
`aclCache`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a change in pull request #5409: NIFI-9235 - Improve PutHDFS documentation; conflict detection
exceptionfactory commented on a change in pull request #5409: URL: https://github.com/apache/nifi/pull/5409#discussion_r718052046 ## File path: nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/pom.xml ## @@ -78,6 +78,11 @@ 1.15.0-SNAPSHOT test + +com.github.ben-manes.caffeine +caffeine +2.8.1 Review comment: The latest version is `3.0.4`, can you use the latest version? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a change in pull request #5420: NIFI-9207 - limit read size, Distributed Cache Servers
exceptionfactory commented on a change in pull request #5420:
URL: https://github.com/apache/nifi/pull/5420#discussion_r718040282
##
File path:
nifi-nar-bundles/nifi-standard-services/nifi-distributed-cache-services-bundle/nifi-distributed-cache-client-service/src/main/java/org/apache/nifi/distributed/cache/client/CacheClientRequestHandler.java
##
@@ -64,6 +64,18 @@ public void channelReadComplete(ChannelHandlerContext ctx)
throws IOException {
}
}
+@Override
+public void channelUnregistered(ChannelHandlerContext ctx) {
+if (!inboundAdapter.isComplete()) {
+channelPromise.setFailure(new IOException("channelUnregistered
during request - " + ctx.channel().toString()));
Review comment:
What do you think about changing the message it indicate incomplete
processing status?
```suggestion
channelPromise.setFailure(new IOException("Channel unregistered
before processing completed: " + ctx.channel().toString()));
```
##
File path:
nifi-nar-bundles/nifi-standard-services/nifi-distributed-cache-services-bundle/nifi-distributed-cache-client-service/src/main/java/org/apache/nifi/distributed/cache/client/CacheClientRequestHandler.java
##
@@ -86,5 +98,8 @@ public void invoke(final Channel channel, final
OutboundAdapter outboundAdapter,
channel.writeAndFlush(Unpooled.wrappedBuffer(outboundAdapter.toBytes()));
channelPromise.awaitUninterruptibly();
this.inboundAdapter = new NullInboundAdapter();
+if (channelPromise.cause() != null) {
+throw new IOException(channelPromise.cause());
Review comment:
Recommend include a message with the exception:
```suggestion
throw new IOException("Request invocation failed",
channelPromise.cause());
```
##
File path:
nifi-nar-bundles/nifi-standard-services/nifi-distributed-cache-services-bundle/nifi-distributed-cache-server/src/main/java/org/apache/nifi/distributed/cache/server/SetCacheServer.java
##
@@ -102,4 +100,18 @@ protected void finalize() throws Throwable {
}
}
+private byte[] readValue(final DataInputStream dis) throws IOException {
+final int numBytes = validateInt(dis.readInt(), getMaxReadSize(),
"readValue():");
+final byte[] buffer = new byte[numBytes];
+dis.readFully(buffer);
+return buffer;
+}
+
+private int validateInt(final int value, final int max, final String
identifier) throws IOException {
+if (value <= max) {
+return value;
+} else {
+throw new IOException(new IllegalArgumentException(identifier +
value));
+}
+}
Review comment:
As mentioned above, recommend removing the `identifier` parameter and
adjusting the exception message. The wrapping `IOException` also seems
unnecessary.
```suggestion
private int validateSize(final int size) {
if (size <= getMaxReadSize()) {
return size;
} else {
throw new IllegalStateException(String.format("Size [%d] exceeds
maximum configured read [%d]", size, getMaxReadSize()));
}
}
```
##
File path:
nifi-nar-bundles/nifi-standard-services/nifi-distributed-cache-services-bundle/nifi-distributed-cache-server/src/main/java/org/apache/nifi/distributed/cache/server/DistributedCacheServer.java
##
@@ -68,6 +68,13 @@
.required(false)
.addValidator(StandardValidators.createDirectoryExistsValidator(true,
true))
.build();
+public static final PropertyDescriptor MAX_READ_SIZE = new
PropertyDescriptor.Builder()
+.name("Maximum Read Size")
Review comment:
A `displayName` should also be included:
```suggestion
.name("maximum-read-size")
.displayName("Maximum Read Size")
```
##
File path:
nifi-nar-bundles/nifi-standard-services/nifi-distributed-cache-services-bundle/nifi-distributed-cache-server/src/main/java/org/apache/nifi/distributed/cache/server/DistributedCacheServer.java
##
@@ -68,6 +68,13 @@
.required(false)
.addValidator(StandardValidators.createDirectoryExistsValidator(true,
true))
.build();
+public static final PropertyDescriptor MAX_READ_SIZE = new
PropertyDescriptor.Builder()
+.name("Maximum Read Size")
+.description("The maximum number of network bytes to read for a single
cache item")
+.required(false)
+.addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+.defaultValue("100")
Review comment:
Instead of using a positive integer, the `DATA_SIZE_VALIDATOR` supports
standard formatting for data sizes:
```suggestion
.addValidator(StandardValidators.DATA_SIZE_VALIDATOR)
.defaultValue("1 MB")
```
##
File path:
[jira] [Updated] (NIFI-9207) Add max read size property to Distributed Cache Servers
[ https://issues.apache.org/jira/browse/NIFI-9207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Grey updated NIFI-9207: Status: Patch Available (was: In Progress) > Add max read size property to Distributed Cache Servers > --- > > Key: NIFI-9207 > URL: https://issues.apache.org/jira/browse/NIFI-9207 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Paul Grey >Assignee: Paul Grey >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > NiFi services DistributedMapCacheServer and DistributedSetCacheServer include > a socket-based server, allowing service clients to collaboratively manage a > cache of data values. IPC is performed via a defined set of APIs, which are > implemented as a series of raw socket reads and writes. > Service processing of incoming socket data should validate and limit its > size, to ensure proper functioning of the server. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] greyp9 opened a new pull request #5420: NIFI-9207 - limit read size, Distributed Cache Servers
greyp9 opened a new pull request #5420:
URL: https://github.com/apache/nifi/pull/5420
Description of PR
Add max read size property to distributed cache servers (Set, Map) to
validate incoming cache client calls. Add unit tests to exercise default
limits. Explicit client handling of socket exceptions (handle server channel
disconnect).
In order to streamline the review of the contribution we ask you to ensure
the following steps have been taken:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically `main`)?
- [x] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [x] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [x] Have you written or updated unit tests to verify your changes?
- [x] Have you verified that the full build is successful on JDK 8?
- [x] Have you verified that the full build is successful on JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for
build issues and submit an update to your PR as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a change in pull request #5419: NIFI-9255: Support asynchronous session commits in ListenHTTP
exceptionfactory commented on a change in pull request #5419:
URL: https://github.com/apache/nifi/pull/5419#discussion_r718028300
##
File path:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/servlets/ListenHTTPServlet.java
##
@@ -413,12 +414,20 @@ protected void proceedFlow(final HttpServletRequest
request, final HttpServletRe
new Object[]{flowFileSet, request.getRemoteHost(),
request.getRemotePort(), foundSubject, foundIssuer, flowFileSet.size(), uuid});
}
} else {
-response.setStatus(this.returnCode);
logger.info("Received from Remote Host: [{}] Port [{}] SubjectDN
[{}] IssuerDN [{}]; transferring to 'success'",
new Object[]{request.getRemoteHost(), request.getRemotePort(),
foundSubject, foundIssuer});
session.transfer(flowFileSet, ListenHTTP.RELATIONSHIP_SUCCESS);
-session.commitAsync();
+
+final AsyncContext asyncContext = request.startAsync();
+session.commitAsync(() -> {
+response.setStatus(this.returnCode);
+asyncContext.complete();
+}, t -> {
+
response.setStatus(HttpServletResponse.SC_INTERNAL_SERVER_ERROR);
Review comment:
@turcsanyip What do you think about logging the exception in this
scenario?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (NIFI-9255) Support asynchronous session commits in ListenHTTP
[ https://issues.apache.org/jira/browse/NIFI-9255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Turcsanyi updated NIFI-9255: -- Status: Patch Available (was: In Progress) > Support asynchronous session commits in ListenHTTP > -- > > Key: NIFI-9255 > URL: https://issues.apache.org/jira/browse/NIFI-9255 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Peter Turcsanyi >Assignee: Peter Turcsanyi >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > ListenHTTP does not support async commits properly because it sends the > response back when the servlet's service method exits and it can happen > before the async commit actually gets executed. It does not provide at least > once guarantee, because the commit may fail after the successful response > sent out. > It does not affect stateful NiFi where the commit is executed synchronously > (so before the servlet's service method terminated) but can cause issues in > Stateless NiFi. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] turcsanyip opened a new pull request #5419: NIFI-9255: Support asynchronous session commits in ListenHTTP
turcsanyip opened a new pull request #5419: URL: https://github.com/apache/nifi/pull/5419 https://issues.apache.org/jira/browse/NIFI-9255 In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with **NIFI-** where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically `main`)? - [ ] Is your initial contribution a single, squashed commit? _Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not `squash` or use `--force` when pushing to allow for clean monitoring of changes._ ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via `mvn -Pcontrib-check clean install` at the root `nifi` folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] Have you verified that the full build is successful on JDK 8? - [ ] Have you verified that the full build is successful on JDK 11? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE` file, including the main `LICENSE` file under `nifi-assembly`? - [ ] If applicable, have you updated the `NOTICE` file, including the main `NOTICE` file found under `nifi-assembly`? - [ ] If adding new Properties, have you added `.displayName` in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check GitHub Actions CI for build issues and submit an update to your PR as soon as possible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni closed pull request #1177: MINIFICPP-1644: ProcessSession::read ignores the size/offset of the
adamdebreceni closed pull request #1177: URL: https://github.com/apache/nifi-minifi-cpp/pull/1177 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] mattyb149 closed pull request #5415: NIFI-7012: Refactored OnConfigurationRestored to support sensitive property validation
mattyb149 closed pull request #5415: URL: https://github.com/apache/nifi/pull/5415 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFI-9255) Support asynchronous session commits in ListenHTTP
Peter Turcsanyi created NIFI-9255: - Summary: Support asynchronous session commits in ListenHTTP Key: NIFI-9255 URL: https://issues.apache.org/jira/browse/NIFI-9255 Project: Apache NiFi Issue Type: Bug Components: Extensions Reporter: Peter Turcsanyi Assignee: Peter Turcsanyi ListenHTTP does not support async commits properly because it sends the response back when the servlet's service method exits and it can happen before the async commit actually gets executed. It does not provide at least once guarantee, because the commit may fail after the successful response sent out. It does not affect stateful NiFi where the commit is executed synchronously (so before the servlet's service method terminated) but can cause issues in Stateless NiFi. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1170: MINIFICPP-1618 Create the ReplaceText processor
szaszm commented on a change in pull request #1170:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1170#discussion_r717382168
##
File path: extensions/standard-processors/processors/ReplaceText.cpp
##
@@ -0,0 +1,339 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "ReplaceText.h"
+
+#include
+#include
+
+#include "core/Resource.h"
+#include "core/TypedValues.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "utils/LineByLineInputOutputStreamCallback.h"
+
+namespace org::apache::nifi::minifi::processors {
+
+const core::Property ReplaceText::EvaluationMode =
core::PropertyBuilder::createProperty("Evaluation Mode")
+->withDescription("Run the 'Replacement Strategy' against each line
separately (Line-by-Line) or "
+ "against the whole input treated as a single string
(Entire Text).")
+->isRequired(true)
+->withDefaultValue(toString(EvaluationModeType::LINE_BY_LINE))
+->withAllowableValues(EvaluationModeType::values())
+->build();
+
+const core::Property ReplaceText::LineByLineEvaluationMode =
core::PropertyBuilder::createProperty("Line-by-Line Evaluation Mode")
+->withDescription("Run the 'Replacement Strategy' against each line
separately (Line-by-Line) for All lines in the FlowFile, "
+ "First Line (Header) only, Last Line (Footer) only, all
Except the First Line (Header) or all Except the Last Line (Footer).")
+->isRequired(false)
+
->withDefaultValue(toString(LineByLineEvaluationModeType::ALL))
+->withAllowableValues(LineByLineEvaluationModeType::values())
+->build();
+
+const core::Property ReplaceText::ReplacementStrategy =
core::PropertyBuilder::createProperty("Replacement Strategy")
+->withDescription("The strategy for how and what to replace within the
FlowFile's text content. "
+ "Substitute Variables replaces ${attribute_name}
placeholders with the corresponding attribute's value "
+ "(if an attribute is not found, the placeholder is kept
as it was).")
+->isRequired(true)
+->withDefaultValue(toString(ReplacementStrategyType::REGEX_REPLACE))
+->withAllowableValues(ReplacementStrategyType::values())
+->build();
+
+const core::Property ReplaceText::SearchValue =
core::PropertyBuilder::createProperty("Search Value")
+->withDescription("The Search Value to search for in the FlowFile content.
"
+ "Only used for 'Literal Replace' and 'Regex Replace'
matching strategies. "
+ "Supports expression language except in Regex Replace
mode.")
+->isRequired(false)
+->supportsExpressionLanguage(true)
+->build();
+
+const core::Property ReplaceText::ReplacementValue =
core::PropertyBuilder::createProperty("Replacement Value")
+->withDescription("The value to insert using the 'Replacement Strategy'. "
+ "Using 'Regex Replace' back-references to Regular
Expression capturing groups are supported: "
+ "$& is the entire matched substring, $1, $2, ... are the
matched capturing groups. Use $$1 for a literal $1. "
+ "Back-references to non-existent capturing groups will
be replaced by empty strings. "
+ "Supports expression language except in Regex Replace
mode.")
+->isRequired(true)
+->supportsExpressionLanguage(true)
+->build();
+
+const core::Relationship ReplaceText::Success("success", "FlowFiles that have
been successfully processed are routed to this relationship. "
+ "This includes both
FlowFiles that had text replaced and those that did not.");
+const core::Relationship ReplaceText::Failure("failure", "FlowFiles that could
not be updated are routed to this relationship.");
+
+ReplaceText::ReplaceText(const std::string& name, const utils::Identifier&
uuid)
+ : core::Processor(name, uuid),
+logger_(logging::LoggerFactory::getLogger()) {
+}
+
+void ReplaceText::initialize() {
+ setSupportedProperties({
+ EvaluationMode,
+ LineByLineEvaluationMode,
+ ReplacementStrategy,
+ SearchValue,
+ ReplacementValue
+
[GitHub] [nifi] adenes edited a comment on pull request #5414: NIFI-9248 Use hive-exec:core instead of the "regular" hive-exec dependency
adenes edited a comment on pull request #5414: URL: https://github.com/apache/nifi/pull/5414#issuecomment-929212599 Thanks for the comment, @exceptionfactory > For reference, what factors determined which explicit dependencies to include after excluding `hive-exec` from `hive-streaming`? The Maven configuration for `hive-exec` appears to include a handful of additional dependencies, but are all of those available through currently declared dependencies? It would be helpful to have some process to track this down in the event of future version updates to `hive-exec`. I ran a `mvn dependency:tree` on the `nifi-nar-bundles/nifi-hive-bundle/nifi-hive3-processors` module and checked which dependencies are missing from the list [1] and added the missing ones as direct dependencies. I had an ugly oneliner: ``(for a in `cat ~/tmp/hive-shadedlibs`; do ver=$(cat deptree | grep $a | grep -E ":compile|:runtime" | head -1 | grep -Eo ":\d+(\.\d+)*"); echo $a$ver; done) >result`` where - `~/tmp/hive-shadedlibs` was manually created from the below linked hive `pom.xml` snippet in the format of `groupId:artifactId` - https://gist.github.com/adenes/7711eb7bd7873002ae46022c84d7cab1 - `deptree` is the output of `mvn dependency:tree -pl nifi-nar-bundles/nifi-hive-bundle/nifi-hive3-processors` - the output of this (`result`) should contain all the libraries from the `hive-shadedlibs` with version number - https://gist.github.com/adenes/f5a734be1fb78af9fbf3a49edf05e1fe [1] https://github.com/apache/hive/blob/rel/release-3.1.2/ql/pom.xml#L925-L961 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] mattyb149 closed pull request #5388: NIFI-9192: ResultSetRecordSet considers value of useLogicalTypes flag…
mattyb149 closed pull request #5388: URL: https://github.com/apache/nifi/pull/5388 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #1170: MINIFICPP-1618 Create the ReplaceText processor
fgerlits commented on a change in pull request #1170:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1170#discussion_r717317315
##
File path: extensions/standard-processors/processors/ReplaceText.cpp
##
@@ -0,0 +1,339 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "ReplaceText.h"
+
+#include
+#include
+
+#include "core/Resource.h"
+#include "core/TypedValues.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "utils/LineByLineInputOutputStreamCallback.h"
+
+namespace org::apache::nifi::minifi::processors {
+
+const core::Property ReplaceText::EvaluationMode =
core::PropertyBuilder::createProperty("Evaluation Mode")
+->withDescription("Run the 'Replacement Strategy' against each line
separately (Line-by-Line) or "
+ "against the whole input treated as a single string
(Entire Text).")
+->isRequired(true)
+->withDefaultValue(toString(EvaluationModeType::LINE_BY_LINE))
+->withAllowableValues(EvaluationModeType::values())
+->build();
+
+const core::Property ReplaceText::LineByLineEvaluationMode =
core::PropertyBuilder::createProperty("Line-by-Line Evaluation Mode")
+->withDescription("Run the 'Replacement Strategy' against each line
separately (Line-by-Line) for All lines in the FlowFile, "
+ "First Line (Header) only, Last Line (Footer) only, all
Except the First Line (Header) or all Except the Last Line (Footer).")
+->isRequired(false)
+
->withDefaultValue(toString(LineByLineEvaluationModeType::ALL))
+->withAllowableValues(LineByLineEvaluationModeType::values())
+->build();
+
+const core::Property ReplaceText::ReplacementStrategy =
core::PropertyBuilder::createProperty("Replacement Strategy")
+->withDescription("The strategy for how and what to replace within the
FlowFile's text content. "
+ "Substitute Variables replaces ${attribute_name}
placeholders with the corresponding attribute's value "
+ "(if an attribute is not found, the placeholder is kept
as it was).")
+->isRequired(true)
+->withDefaultValue(toString(ReplacementStrategyType::REGEX_REPLACE))
+->withAllowableValues(ReplacementStrategyType::values())
+->build();
+
+const core::Property ReplaceText::SearchValue =
core::PropertyBuilder::createProperty("Search Value")
+->withDescription("The Search Value to search for in the FlowFile content.
"
+ "Only used for 'Literal Replace' and 'Regex Replace'
matching strategies. "
+ "Supports expression language except in Regex Replace
mode.")
+->isRequired(false)
+->supportsExpressionLanguage(true)
+->build();
+
+const core::Property ReplaceText::ReplacementValue =
core::PropertyBuilder::createProperty("Replacement Value")
+->withDescription("The value to insert using the 'Replacement Strategy'. "
+ "Using 'Regex Replace' back-references to Regular
Expression capturing groups are supported: "
+ "$& is the entire matched substring, $1, $2, ... are the
matched capturing groups. Use $$1 for a literal $1. "
+ "Back-references to non-existent capturing groups will
be replaced by empty strings. "
+ "Supports expression language except in Regex Replace
mode.")
+->isRequired(true)
+->supportsExpressionLanguage(true)
+->build();
+
+const core::Relationship ReplaceText::Success("success", "FlowFiles that have
been successfully processed are routed to this relationship. "
+ "This includes both
FlowFiles that had text replaced and those that did not.");
+const core::Relationship ReplaceText::Failure("failure", "FlowFiles that could
not be updated are routed to this relationship.");
+
+ReplaceText::ReplaceText(const std::string& name, const utils::Identifier&
uuid)
+ : core::Processor(name, uuid),
+logger_(logging::LoggerFactory::getLogger()) {
+}
+
+void ReplaceText::initialize() {
+ setSupportedProperties({
+ EvaluationMode,
+ LineByLineEvaluationMode,
+ ReplacementStrategy,
+ SearchValue,
+ ReplacementValue
+
[GitHub] [nifi] adenes commented on pull request #5414: NIFI-9248 Use hive-exec:core instead of the "regular" hive-exec dependency
adenes commented on pull request #5414: URL: https://github.com/apache/nifi/pull/5414#issuecomment-929212599 Thanks for the comment, @exceptionfactory > For reference, what factors determined which explicit dependencies to include after excluding `hive-exec` from `hive-streaming`? The Maven configuration for `hive-exec` appears to include a handful of additional dependencies, but are all of those available through currently declared dependencies? It would be helpful to have some process to track this down in the event of future version updates to `hive-exec`. I ran a `mvn dependency:tree` on the `nifi-nar-bundles/nifi-hive-bundle/nifi-hive3-processors` module and checked which dependencies are missing from the list [1] and added the missing ones as direct dependencies. I had an ugly oneliner: `(for a in `cat ~/tmp/hive-shadedlibs`; do ver=$(cat deptree | grep $a | grep -E ":compile|:runtime" | head -1 | grep -Eo ":\d+(\.\d+)*"); echo $a$ver; done) >result` where - `~/tmp/hive-shadedlibs` was manually created from the below linked hive `pom.xml` snippet in the format of `groupId:artifactId` - https://gist.github.com/adenes/7711eb7bd7873002ae46022c84d7cab1 - `deptree` is the output of `mvn dependency:tree -pl nifi-nar-bundles/nifi-hive-bundle/nifi-hive3-processors` - the output of this (`result`) should contain all the libraries from the `hive-shadedlibs` with version number - https://gist.github.com/adenes/f5a734be1fb78af9fbf3a49edf05e1fe [1] https://github.com/apache/hive/blob/rel/release-3.1.2/ql/pom.xml#L925-L961 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] greyp9 commented on pull request #5311: NIFI-8919 - Refactor Distributed Cache Clients using Netty
greyp9 commented on pull request #5311: URL: https://github.com/apache/nifi/pull/5311#issuecomment-928486380 > This works at runtime, but I noticed an issue with the PropertyDescriptor references in `createChannelPool`. The references point to the `Set` service, and it works given that the property names match the `Map` service, however, this should be changed to avoid unexpected differences. Other than that, this should be good to go. Yep, thanks. Missed the significance of that. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] github-actions[bot] closed pull request #3511: NIFI-6175 Spark Livy - Improving Livy
github-actions[bot] closed pull request #3511: URL: https://github.com/apache/nifi/pull/3511 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni closed pull request #1179: MINIFICPP-1649 Replace magic fallthrough strings with attribute
adamdebreceni closed pull request #1179: URL: https://github.com/apache/nifi-minifi-cpp/pull/1179 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] greyp9 commented on a change in pull request #5409: NIFI-9235 - Improve PutHDFS documentation; conflict detection
greyp9 commented on a change in pull request #5409:
URL: https://github.com/apache/nifi/pull/5409#discussion_r716947901
##
File path:
nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/PutHDFS.java
##
@@ -228,6 +232,20 @@ protected void preProcessConfiguration(final Configuration
config, final Process
FsPermission.setUMask(config, new FsPermission(dfsUmask));
}
+@Override
+protected void preProcessFileSystem(final FileSystem fileSystem, final
ProcessContext context) throws IOException {
+if (fileSystem instanceof DistributedFileSystem) {
+final Path dirPath = new
Path(context.getProperty(DIRECTORY).getValue());
Review comment:
Changed the logic to throw on detection of this situation, as otherwise
the file is written to HDFS with less restrictive permissions than are intended.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] mattyb149 commented on pull request #5388: NIFI-9192: ResultSetRecordSet considers value of useLogicalTypes flag…
mattyb149 commented on pull request #5388: URL: https://github.com/apache/nifi/pull/5388#issuecomment-928251967 +1 LGTM, tried various scenarios with Oracle and Avro logical types on and off. Thanks for the fix! Merging to main -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] mattyb149 commented on a change in pull request #5415: NIFI-7012: Refactored OnConfigurationRestored to support sensitive property validation
mattyb149 commented on a change in pull request #5415: URL: https://github.com/apache/nifi/pull/5415#discussion_r717175855 ## File path: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/StandardReloadComponent.java ## @@ -97,6 +97,11 @@ public void reload(final ProcessorNode existingNode, final String newType, final // need to refresh the properties in case we are changing from ghost component to real component existingNode.refreshProperties(); +// Notify the processor node that the configuration (properties, e.g.) has been restored +final StandardProcessContext processContext = new StandardProcessContext(existingNode, flowController.getControllerServiceProvider(), flowController.getEncryptor(), + flowController.getStateManagerProvider().getStateManager(existingNode.getProcessor().getIdentifier()), () -> false, flowController); +existingNode.onConfigurationRestored(processContext); Review comment: Good catch, I think so, will look into it ## File path: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/ProcessorNode.java ## @@ -290,4 +290,11 @@ public abstract void runOnce(ScheduledExecutorService scheduler, long administra * @return the desired state for this Processor */ public abstract ScheduledState getDesiredState(); + +/** + * This method will be called once the processor's configuration has been restored (on startup, reload, e.g.) + * + * @param context The ProcessContext associated with the Processor configuration + */ +public abstract void onConfigurationRestored(ProcessContext context); Review comment: They will eventually need one, but I think not right now because of [NIFI-5547](https://issues.apache.org/jira/browse/NIFI-5547) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] markap14 commented on pull request #5412: NIFI-9239: Updated Consume/Publish Kafka processors to support Exactl…
markap14 commented on pull request #5412: URL: https://github.com/apache/nifi/pull/5412#issuecomment-929358954 @gresockj @exceptionfactory huge thanks for all of the very thoughtful and thorough feedback! I will push a new commit momentarily that should address all concerns. And I definitely agree @exceptionfactory that there are a lot more things that should be unit tested. I intended to write a bunch more tests and then it totally slipped my mind. So the next commit will include new tests, mostly around error conditions, such as timeouts, processors that throw exceptions, failure ports, etc. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] asfgit closed pull request #5416: NIFI-9251 Add AuthenticationManager Bean to avoid password generation
asfgit closed pull request #5416: URL: https://github.com/apache/nifi/pull/5416 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] gresockj commented on a change in pull request #5415: NIFI-7012: Refactored OnConfigurationRestored to support sensitive property validation
gresockj commented on a change in pull request #5415:
URL: https://github.com/apache/nifi/pull/5415#discussion_r716972076
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/StandardProcessorNode.java
##
@@ -1915,4 +1916,12 @@ public void setVersionedComponentId(final String
versionedComponentId) {
}
}
}
+
+@Override
+public void onConfigurationRestored(ProcessContext context) {
Review comment:
Could be `final`
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/AbstractComponentNode.java
##
@@ -1239,6 +1241,40 @@ protected void setAdditionalResourcesFingerprint(String
additionalResourcesFinge
this.additionalResourcesFingerprint = additionalResourcesFingerprint;
}
+// Determine whether the property value should be evaluated in terms of
the parameter context or not.
+// If the sensitivity of the property does not match the sensitivity of
the parameter, the literal value will be returned
+//
+// Examples when SensitiveParam value = 'abc' and MY_PROP is non-sensitive:
+// SensitiveProp--> 'abc'
+// NonSensitiveProp --> '#{SensitiveParam}'
+// context.getProperty(MY_PROP).getValue(); '#{SensitiveParam}'
+private boolean isResolveParameter(PropertyDescriptor descriptor,
PropertyConfiguration config) {
Review comment:
`final`
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/ProcessorNode.java
##
@@ -290,4 +290,11 @@ public abstract void runOnce(ScheduledExecutorService
scheduler, long administra
* @return the desired state for this Processor
*/
public abstract ScheduledState getDesiredState();
+
+/**
+ * This method will be called once the processor's configuration has been
restored (on startup, reload, e.g.)
+ *
+ * @param context The ProcessContext associated with the Processor
configuration
+ */
+public abstract void onConfigurationRestored(ProcessContext context);
Review comment:
Curious, why does `ProcessorNode` get an `onConfigurationRestored`
method, but not `ControllerServiceNode` or `ReportingTaskNode`?
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/StandardReloadComponent.java
##
@@ -97,6 +97,11 @@ public void reload(final ProcessorNode existingNode, final
String newType, final
// need to refresh the properties in case we are changing from ghost
component to real component
existingNode.refreshProperties();
+// Notify the processor node that the configuration (properties, e.g.)
has been restored
+final StandardProcessContext processContext = new
StandardProcessContext(existingNode,
flowController.getControllerServiceProvider(), flowController.getEncryptor(),
+
flowController.getStateManagerProvider().getStateManager(existingNode.getProcessor().getIdentifier()),
() -> false, flowController);
+existingNode.onConfigurationRestored(processContext);
Review comment:
Should this also be added to `StatelessReloadComponent`?
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FlowSnippet.java
##
@@ -42,6 +42,6 @@
* @throws ProcessorInstantiationException if unable to instantiate any of
the Processors within the snippet
* @throws
org.apache.nifi.controller.exception.ControllerServiceInstantiationException if
unable to instantiate any of the Controller Services within the snippet
*/
-void instantiate(FlowManager flowManager, ProcessGroup group) throws
ProcessorInstantiationException;
+void instantiate(FlowManager flowManager, FlowController flowController,
ProcessGroup group) throws ProcessorInstantiationException;
Review comment:
Javadoc param needed for `flowController`
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/AbstractComponentNode.java
##
@@ -1239,6 +1241,40 @@ protected void setAdditionalResourcesFingerprint(String
additionalResourcesFinge
this.additionalResourcesFingerprint = additionalResourcesFingerprint;
}
+// Determine whether the property value should be evaluated in terms of
the parameter context or not.
+// If the sensitivity of the property does not match the sensitivity of
the parameter, the literal value will be returned
+//
+// Examples when SensitiveParam value = 'abc' and MY_PROP is non-sensitive:
+// SensitiveProp--> 'abc'
+// NonSensitiveProp -->
[GitHub] [nifi] greyp9 commented on a change in pull request #5261: NIFI-7588: Handling dynamic properties on termination
greyp9 commented on a change in pull request #5261: URL: https://github.com/apache/nifi/pull/5261#discussion_r717047306 ## File path: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/StandardProcessorNodeIT.java ## @@ -59,12 +61,14 @@ import org.apache.nifi.registry.flow.FlowRegistryClient; import org.apache.nifi.registry.variable.FileBasedVariableRegistry; import org.apache.nifi.registry.variable.StandardComponentVariableRegistry; +import org.apache.nifi.test.processors.DynamicPropertiesTestProcessor; import org.apache.nifi.test.processors.ModifiesClasspathNoAnnotationProcessor; import org.apache.nifi.test.processors.ModifiesClasspathProcessor; import org.apache.nifi.util.MockPropertyValue; import org.apache.nifi.util.MockVariableRegistry; import org.apache.nifi.util.NiFiProperties; import org.apache.nifi.util.SynchronousValidationTrigger; +import org.jetbrains.annotations.NotNull; Review comment: Wanted to make sure this was the intended package, rather than `javax.validation.constraints`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on pull request #5414: NIFI-9248 Use hive-exec:core instead of the "regular" hive-exec dependency
exceptionfactory commented on pull request #5414: URL: https://github.com/apache/nifi/pull/5414#issuecomment-928573539 Thanks for putting together this update @adenes, it looks like it should reduce the overall size with the removal of unnecessary dependencies, while also avoiding potential runtime conflicts. For reference, what factors determined which explicit dependencies to include after excluding `hive-exec` from `hive-streaming`? The Maven configuration for `hive-exec` appears to include a handful of additional dependencies, but are all of those available through currently declared dependencies? It would be helpful to have some process to track this down in the event of future version updates to `hive-exec`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a change in pull request #5311: NIFI-8919 - Refactor Distributed Cache Clients using Netty
exceptionfactory commented on a change in pull request #5311:
URL: https://github.com/apache/nifi/pull/5311#discussion_r717085052
##
File path:
nifi-nar-bundles/nifi-standard-services/nifi-distributed-cache-services-bundle/nifi-distributed-cache-client-service/src/main/java/org/apache/nifi/distributed/cache/client/DistributedCacheClient.java
##
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.distributed.cache.client;
+
+import io.netty.channel.Channel;
+import io.netty.channel.pool.ChannelPool;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.distributed.cache.client.adapter.InboundAdapter;
+import org.apache.nifi.distributed.cache.client.adapter.OutboundAdapter;
+import org.apache.nifi.remote.VersionNegotiatorFactory;
+
+import java.io.IOException;
+
+/**
+ * Encapsulate operations which may be performed using a {@link
DistributedSetCacheClientService} or a
+ * {@link DistributedMapCacheClientService}.
+ */
+public class DistributedCacheClient {
+
+/**
+ * The pool of network connections used to service client requests.
+ */
+private final ChannelPool channelPool;
+
+/**
+ * Constructor.
+ *
+ * @param context the NiFi configuration to be applied to the channel pool
+ * @param factory creator of object used to broker the version of the
distributed cache protocol with the service
+ */
+protected DistributedCacheClient(final ConfigurationContext context, final
VersionNegotiatorFactory factory) {
Review comment:
This constructor will need to change to pass individual property values.
##
File path:
nifi-nar-bundles/nifi-standard-services/nifi-distributed-cache-services-bundle/nifi-distributed-cache-client-service/src/main/java/org/apache/nifi/distributed/cache/client/NettyDistributedMapCacheClient.java
##
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.distributed.cache.client;
+
+import org.apache.nifi.controller.ConfigurationContext;
+import
org.apache.nifi.distributed.cache.client.adapter.AtomicCacheEntryInboundAdapter;
+import org.apache.nifi.distributed.cache.client.adapter.BooleanInboundAdapter;
+import org.apache.nifi.distributed.cache.client.adapter.LongInboundAdapter;
+import org.apache.nifi.distributed.cache.client.adapter.MapInboundAdapter;
+import
org.apache.nifi.distributed.cache.client.adapter.MapValuesInboundAdapter;
+import org.apache.nifi.distributed.cache.client.adapter.OutboundAdapter;
+import org.apache.nifi.distributed.cache.client.adapter.SetInboundAdapter;
+import org.apache.nifi.distributed.cache.client.adapter.ValueInboundAdapter;
+import org.apache.nifi.distributed.cache.client.adapter.VoidInboundAdapter;
+import org.apache.nifi.distributed.cache.operations.MapOperation;
+import org.apache.nifi.distributed.cache.protocol.ProtocolVersion;
+import org.apache.nifi.remote.VersionNegotiatorFactory;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * The implementation of the {@link DistributedMapCacheClient} using the netty
library to provide the remote
+ * communication services.
+ */
+public class NettyDistributedMapCacheClient extends DistributedCacheClient {
+
+/**
+ * Constructor.
+ *
+ * @param context the NiFi configuration to be applied to the channel pool
+ * @param factory
[GitHub] [nifi] mattyb149 commented on pull request #5318: NIFI-9064:Support Oracle timestamp when `Use Avro Logical Types` is t…
mattyb149 commented on pull request #5318: URL: https://github.com/apache/nifi/pull/5318#issuecomment-929283337 There's a merge conflict now, and since you have a couple of merge commits in this PR, I recommend cleaning it up by rebasing or squashing merge commits or whatever, so your only commits are your changes to the code (and there are no merge conflicts), thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] gresockj commented on a change in pull request #5261: NIFI-7588: Handling dynamic properties on termination
gresockj commented on a change in pull request #5261: URL: https://github.com/apache/nifi/pull/5261#discussion_r717062535 ## File path: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/StandardProcessorNodeIT.java ## @@ -59,12 +61,14 @@ import org.apache.nifi.registry.flow.FlowRegistryClient; import org.apache.nifi.registry.variable.FileBasedVariableRegistry; import org.apache.nifi.registry.variable.StandardComponentVariableRegistry; +import org.apache.nifi.test.processors.DynamicPropertiesTestProcessor; import org.apache.nifi.test.processors.ModifiesClasspathNoAnnotationProcessor; import org.apache.nifi.test.processors.ModifiesClasspathProcessor; import org.apache.nifi.util.MockPropertyValue; import org.apache.nifi.util.MockVariableRegistry; import org.apache.nifi.util.NiFiProperties; import org.apache.nifi.util.SynchronousValidationTrigger; +import org.jetbrains.annotations.NotNull; Review comment: Ooh, nice catch @greyp9. Certainly not the intended package! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] asfgit closed pull request #5311: NIFI-8919 - Refactor Distributed Cache Clients using Netty
asfgit closed pull request #5311: URL: https://github.com/apache/nifi/pull/5311 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] pgyori commented on pull request #5388: NIFI-9192: ResultSetRecordSet considers value of useLogicalTypes flag…
pgyori commented on pull request #5388: URL: https://github.com/apache/nifi/pull/5388#issuecomment-929047305 Thank you, @mattyb149 ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-8727) claimantCount will never decrement to zero
[
https://issues.apache.org/jira/browse/NIFI-8727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17421589#comment-17421589
]
Mark Payne commented on NIFI-8727:
--
Thanks [~wildq] for all of your analysis and describing exactly what you're
seeing. We don't want to change the constructor that gets used for
StandardRepositoryRecord, as that changes the record type from CREATE to
UPDATE. Depending on the Serializer/Deserializer that gets used in the FlowFile
Repository, that could cause problems because it's basically going to read an
UPDATE for a FlowFile that doesn't exist.
I'll also note that {{ProcessSession.clone(originalFlowFile)}} shouldn't really
be used there - it should be {{ProcessSession.create(originalFlowFile)}} -
clone() means you want the newly created child to have the exact same content
as the original, whereas create() indicates that you want a new child with no
content. Since you plan to overwrite the content anyway, {{create}} makes more
sense, and it's slightly more efficient. That being said, as you noted, it
should still work the same!
The underlying issue appears to be in
{{StandardProcessSession.removeTemporaryClaim}}. Because the clone() event
results in a RepositoryRecord where the content has not been modified, the call
to {{ProcessSession.write}} doesn't decrement that claim as it needs to. So an
easy fix is to simply update the logic there to have it perform that logic if
either the content is modified OR if the Repository Record is CREATE (in which
case the content claim will be non-existent or will be a temporary claim that
is to be overwritten).
I've created a Pull Request that does just that, with supporting unit tests.
Please give it a try.
Thanks
> claimantCount will never decrement to zero
> --
>
> Key: NIFI-8727
> URL: https://issues.apache.org/jira/browse/NIFI-8727
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.12.0, 1.13.0, 1.12.1, 1.13.1, 1.13.2
> Environment: linux
>Reporter: wangliqiang
>Assignee: Mark Bean
>Priority: Major
> Original Estimate: 96h
> Time Spent: 10m
> Remaining Estimate: 95h 50m
>
> When running my processor code below :
> {code:java}
> //originalFlowFile has content , so ClaimantCount=1
> FlowFile multiFlowFile = session.clone(originalFlowFile); // claim count
> +1，so ClaimantCount=2
> multiFlowFile = session.write(multiFlowFile, new OutputStreamCallback() {
> @Override
> public void process(OutputStream out) throws IOException {
> IOUtils.write(tvMultiAlbumJson, out, Charset.forName("UTF-8"));
> }
> });//the new content will resuse the same resource claim , so ClaimantCount=3
> //At this point we have two flowfile and two contentClaim ，and ClaimCount=3.
> //When this two flowfiles dropped，the claimantCount should decrement to
> 0，however the result is ClaimantCount=1！
> //If we use "sh nifi.sh diagnostics --verbose dump.log" to get a dump log，we
> will find some info like this “default/465/1623853427574-10705, Claimant
> Count = 1, In Use = true, Awaiting Destruction = false, References (0) = []”
> //And the file “default/465/1623853427574-10705” will never be archived，and
> will never be destroyed，and the content_repository will use more storage than
> it configs.{code}
> The above is a sort of phenomenon. The reason is the code below：
> {code:java}
> //session.clone
> public FlowFile clone(FlowFile example, final long offset, final long size) {
> .
> final StandardRepositoryRecord record = new
> StandardRepositoryRecord(null); //here the originalFlowFileRecord of record
> is null
> .
> return clone;
> }
> //session.commit
> private void updateClaimCounts(final RepositoryRecord record) {
> ..
> if (record.isContentModified()) {
> decrementClaimCount(originalClaim); //here the originalClaim is null
> }
> }{code}
> Perhaps we should not use session.clone like that，but without official note
> it will sometimes happen to be used.
> So i change "final StandardRepositoryRecord record = new
> StandardRepositoryRecord(null)" to "final StandardRepositoryRecord record =
> new StandardRepositoryRecord(null, currRec);"
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (MINIFICPP-1650) ProcessSession::append sets the flowfile size incorrectly
[ https://issues.apache.org/jira/browse/MINIFICPP-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Zink updated MINIFICPP-1650: --- Status: Patch Available (was: In Progress) https://github.com/apache/nifi-minifi-cpp/pull/1181 > ProcessSession::append sets the flowfile size incorrectly > - > > Key: MINIFICPP-1650 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1650 > Project: Apache NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Martin Zink >Assignee: Martin Zink >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > If ProcessSession::append uses extendedResources_ to append to the flowfie > then the size of flowfile will be invalid. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] markap14 opened a new pull request #5418: NIFI-8727: Addressed bug in which ProcessSession doesn't properly dec…
markap14 opened a new pull request #5418:
URL: https://github.com/apache/nifi/pull/5418
…rement claimant count when a FlowFile is cloned and then the clone written
to. Added automated tests to ensure that we are properly handling cases where a
FlowFile is clone and then the contents modified
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
_Enables X functionality; fixes bug NIFI-._
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically `main`)?
- [ ] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] Have you verified that the full build is successful on JDK 8?
- [ ] Have you verified that the full build is successful on JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for
build issues and submit an update to your PR as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (NIFI-9251) Disable unused password generation in Registry
[ https://issues.apache.org/jira/browse/NIFI-9251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-9251: - Fix Version/s: 1.15.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Disable unused password generation in Registry > -- > > Key: NIFI-9251 > URL: https://issues.apache.org/jira/browse/NIFI-9251 > Project: Apache NiFi > Issue Type: Improvement > Components: NiFi Registry >Affects Versions: 1.14.0 >Reporter: David Handermann >Assignee: David Handermann >Priority: Minor > Labels: security > Fix For: 1.15.0 > > Time Spent: 10m > Remaining Estimate: 0h > > During initialization, NiFi Registry logs a generated UUID password as a > result of Spring UserDetailsServiceAutoConfiguration. This password is not > used, but the application configuration should be adjusted to avoid > initializing the class and logging the generated password. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-9251) Disable unused password generation in Registry
[ https://issues.apache.org/jira/browse/NIFI-9251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17421551#comment-17421551 ] ASF subversion and git services commented on NIFI-9251: --- Commit 6ef638d13d430e02b368d3566f47ae4e8fd7db08 in nifi's branch refs/heads/main from David Handermann [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=6ef638d ] NIFI-9251 Added AuthenticationManager Bean to avoid unnecessary password generation Signed-off-by: Pierre Villard This closes #5416. > Disable unused password generation in Registry > -- > > Key: NIFI-9251 > URL: https://issues.apache.org/jira/browse/NIFI-9251 > Project: Apache NiFi > Issue Type: Improvement > Components: NiFi Registry >Affects Versions: 1.14.0 >Reporter: David Handermann >Assignee: David Handermann >Priority: Minor > Labels: security > Time Spent: 10m > Remaining Estimate: 0h > > During initialization, NiFi Registry logs a generated UUID password as a > result of Spring UserDetailsServiceAutoConfiguration. This password is not > used, but the application configuration should be adjusted to avoid > initializing the class and logging the generated password. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] asfgit closed pull request #5416: NIFI-9251 Add AuthenticationManager Bean to avoid password generation
asfgit closed pull request #5416: URL: https://github.com/apache/nifi/pull/5416 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (NIFI-3929) Allow external key management for EncryptContent processor
[
https://issues.apache.org/jira/browse/NIFI-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann reassigned NIFI-3929:
--
Assignee: David Handermann (was: Andy LoPresto)
> Allow external key management for EncryptContent processor
> --
>
> Key: NIFI-3929
> URL: https://issues.apache.org/jira/browse/NIFI-3929
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Extensions
>Reporter: Andy LoPresto
>Assignee: David Handermann
>Priority: Major
> Labels: encryption, key-management, security
>
> Currently the {{EncryptContent}} processor (and a future {{EncryptAttribute}}
> processor) can only use a static key (or password-derived key) to
> encrypt/decrypt data. Similar to the {{KeyProvider}} interface used in the
> {{EncryptedWriteAheadProvenanceRepository}} that exposes multiple back-end
> implementations ({{StaticKeyProvider}}, {{FileBasedKeyProvider}}, etc.), a
> generic key management controller service is proposed in
> [NIFI-3890|https://issues.apache.org/jira/browse/NIFI-3890], and this
> processor should add support for consuming the CS and using it to dynamically
> access provided keys for encryption/decryption operations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (NIFI-3890) Create Key Management Controller Service
[
https://issues.apache.org/jira/browse/NIFI-3890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann reassigned NIFI-3890:
--
Assignee: David Handermann (was: Andy LoPresto)
> Create Key Management Controller Service
>
>
> Key: NIFI-3890
> URL: https://issues.apache.org/jira/browse/NIFI-3890
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Extensions
>Affects Versions: 1.2.0
>Reporter: Andy LoPresto
>Assignee: David Handermann
>Priority: Major
> Labels: encryption, key-management, metadata, security
>
> Similar to the {{KeyProvider}} interface and various backing implementations
> for the {{EncryptedWriteAheadProvenanceRepository}}, a generic
> {{KeyManagementControllerService}} should be made available so that other
> controller services, reporting tasks, and processors can reference it and
> retrieve the necessary keys to perform their cryptographic operations.
> It can be backed by a variety of implementations (file-based, static,
> database, HSM, etc.).
> It should have extensive auditing and granular access controls to restrict
> both user interaction and component interaction.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (NIFI-1525) Audit use of private keys throughout application
[ https://issues.apache.org/jira/browse/NIFI-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann resolved NIFI-1525. Resolution: Information Provided Closing based on history and scope, further analysis could be performed on a case-by-case basis. > Audit use of private keys throughout application > > > Key: NIFI-1525 > URL: https://issues.apache.org/jira/browse/NIFI-1525 > Project: Apache NiFi > Issue Type: Task > Components: Core Framework >Affects Versions: 0.5.0 >Reporter: Andy LoPresto >Assignee: Andy LoPresto >Priority: Major > Labels: keys, security > > Throughout the application, private keys are used for authentication and > encryption in a number of scenarios. These uses are handled independently and > sometimes inconsistently. > Identify any outstanding issues with the use and protection of private keys > (lack of enforcement of passphrases, etc.) and possibly provide a consumable > framework service to handle this task to deduplicate effort/code and provide > a consistent experience. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (NIFI-8728) archiveCount below zero, all processors with content modified will blocked in "waiting for archive cleanup"
[
https://issues.apache.org/jira/browse/NIFI-8728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne resolved NIFI-8728.
--
Resolution: Not A Problem
[~wildq] this is by design. The archiveCount here is not really relevant. When
the disk that the content repo is on reaches the configured threshold (50%),
the content repo will exert backpressure, preventing processors from writing to
it, until archival has run. This is done because on my operating systems
(including linux) creating and writing to a file can be more efficient/faster
than deleting the file. As a result, we can end up in a situation in which we
create data faster than we can delete it, which causes the content repo to fill
up all disk space. This forces the processors to stop and give the repo a
chance to cleanup.
It is not a perfect solution, but it it's working as designed. You can
certainly increase the threshold to something much higher than 50% if necessary.
> archiveCount below zero, all processors with content modified will blocked in
> "waiting for archive cleanup"
>
>
> Key: NIFI-8728
> URL: https://issues.apache.org/jira/browse/NIFI-8728
> Project: Apache NiFi
> Issue Type: Sub-task
> Components: Core Framework
>Affects Versions: 1.12.0, 1.13.0, 1.12.1, 1.13.1, 1.13.2
> Environment: linux
>Reporter: wangliqiang
>Priority: Major
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> When I found the issue “claimantCount will never decrement to zero”，I restart
> the nifi instance and a new issue happened；
> Here is the log：
> 2021-06-23 10:33:02,788 INFO [main] o.a.n.c.repository.FileSystemRepository
> Found unknown file
> /data/nifi/nifi-1.13.2/content_repository/1/1624415033830-1 (286096 bytes) in
> File System Repository; archiving file
> 2021-06-23 10:33:57,786 DEBUG [Cleanup Archive for default]
> o.a.n.c.repository.FileSystemRepository Deleted archived ContentClaim with ID
> 1624415033830-1 from Container default because the archival size was
> exceeding the max configured size, {color:#FF}archiveCount -1{color}
>
> If i config "nifi.content.repository.archive.max.usage.percentage=50%",When
> the archiveCount is “-1” and disk storage is over 50% , all processors with
> content modified will blocked in the method "isWaitRequired()" !
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (NIFI-1466) Add password strength indicator to password properties
[ https://issues.apache.org/jira/browse/NIFI-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann resolved NIFI-1466. Resolution: Won't Fix Although it might be interesting for NiFi to indicate password strength on configuration, the security of most password values is dependent on the associated service or system. > Add password strength indicator to password properties > -- > > Key: NIFI-1466 > URL: https://issues.apache.org/jira/browse/NIFI-1466 > Project: Apache NiFi > Issue Type: New Feature > Components: Core Framework >Affects Versions: 0.5.0 >Reporter: Andy LoPresto >Assignee: Andy LoPresto >Priority: Major > Labels: encryption, security > Original Estimate: 336h > Remaining Estimate: 336h > > In processor properties which accept a password, enforce minimum entropy > limits and provide real-time feedback as to the entropy estimate of the > password. This will have to be overridable (either locally or globally) for > backward compatibility, but we should require an explicit administrator > decision to do so. > Password "strength meters" and other such indicators are not perfect, but > they do provide an estimate of valuable feedback to users to encourage > stronger passwords. > Resources: > * [NIST & CMU Paper on observed password entropy and recommendations for > user-friendly > restrictions|https://www.cylab.cmu.edu/research/techreports/2011/tr_cylab11008.html] > * [J. Bonneau - Statistical metrics for individual password strength > (PDF)|http://www.jbonneau.com/doc/B12-SPW-statistical_password_strength_metrics.pdf] > * [Sophos - Why you can't trust password strength > meters|https://nakedsecurity.sophos.com/2015/03/02/why-you-cant-trust-password-strength-meters/] > * [zxcvbn - Dropbox Password Strength > Estimator|https://blogs.dropbox.com/tech/2012/04/zxcvbn-realistic-password-strength-estimation/] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8727) claimantCount will never decrement to zero
[
https://issues.apache.org/jira/browse/NIFI-8727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17421495#comment-17421495
]
Mark Bean commented on NIFI-8727:
-
Correction: After running for about 24 hours, NiFi was restarted and there were
no "Found unknown file" messages. The ones noted previously might be
attributable to claims that just recently became empty, but before the cleanup
process executed.
Either way, the symptom of a constantly growing content repository - resulting
from claims not reaching a claimant count of 0 - is no longer observed.
Note: the proposed solution will result in a StandardRepositoryRecord having a
type of UPDATE rather than the current logic having a value of CREATE.
> claimantCount will never decrement to zero
> --
>
> Key: NIFI-8727
> URL: https://issues.apache.org/jira/browse/NIFI-8727
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.12.0, 1.13.0, 1.12.1, 1.13.1, 1.13.2
> Environment: linux
>Reporter: wangliqiang
>Priority: Major
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> When running my processor code below :
> {code:java}
> //originalFlowFile has content , so ClaimantCount=1
> FlowFile multiFlowFile = session.clone(originalFlowFile); // claim count
> +1，so ClaimantCount=2
> multiFlowFile = session.write(multiFlowFile, new OutputStreamCallback() {
> @Override
> public void process(OutputStream out) throws IOException {
> IOUtils.write(tvMultiAlbumJson, out, Charset.forName("UTF-8"));
> }
> });//the new content will resuse the same resource claim , so ClaimantCount=3
> //At this point we have two flowfile and two contentClaim ，and ClaimCount=3.
> //When this two flowfiles dropped，the claimantCount should decrement to
> 0，however the result is ClaimantCount=1！
> //If we use "sh nifi.sh diagnostics --verbose dump.log" to get a dump log，we
> will find some info like this “default/465/1623853427574-10705, Claimant
> Count = 1, In Use = true, Awaiting Destruction = false, References (0) = []”
> //And the file “default/465/1623853427574-10705” will never be archived，and
> will never be destroyed，and the content_repository will use more storage than
> it configs.{code}
> The above is a sort of phenomenon. The reason is the code below：
> {code:java}
> //session.clone
> public FlowFile clone(FlowFile example, final long offset, final long size) {
> .
> final StandardRepositoryRecord record = new
> StandardRepositoryRecord(null); //here the originalFlowFileRecord of record
> is null
> .
> return clone;
> }
> //session.commit
> private void updateClaimCounts(final RepositoryRecord record) {
> ..
> if (record.isContentModified()) {
> decrementClaimCount(originalClaim); //here the originalClaim is null
> }
> }{code}
> Perhaps we should not use session.clone like that，but without official note
> it will sometimes happen to be used.
> So i change "final StandardRepositoryRecord record = new
> StandardRepositoryRecord(null)" to "final StandardRepositoryRecord record =
> new StandardRepositoryRecord(null, currRec);"
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (NIFI-8727) claimantCount will never decrement to zero
[
https://issues.apache.org/jira/browse/NIFI-8727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Bean reassigned NIFI-8727:
---
Assignee: Mark Bean
> claimantCount will never decrement to zero
> --
>
> Key: NIFI-8727
> URL: https://issues.apache.org/jira/browse/NIFI-8727
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.12.0, 1.13.0, 1.12.1, 1.13.1, 1.13.2
> Environment: linux
>Reporter: wangliqiang
>Assignee: Mark Bean
>Priority: Major
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> When running my processor code below :
> {code:java}
> //originalFlowFile has content , so ClaimantCount=1
> FlowFile multiFlowFile = session.clone(originalFlowFile); // claim count
> +1，so ClaimantCount=2
> multiFlowFile = session.write(multiFlowFile, new OutputStreamCallback() {
> @Override
> public void process(OutputStream out) throws IOException {
> IOUtils.write(tvMultiAlbumJson, out, Charset.forName("UTF-8"));
> }
> });//the new content will resuse the same resource claim , so ClaimantCount=3
> //At this point we have two flowfile and two contentClaim ，and ClaimCount=3.
> //When this two flowfiles dropped，the claimantCount should decrement to
> 0，however the result is ClaimantCount=1！
> //If we use "sh nifi.sh diagnostics --verbose dump.log" to get a dump log，we
> will find some info like this “default/465/1623853427574-10705, Claimant
> Count = 1, In Use = true, Awaiting Destruction = false, References (0) = []”
> //And the file “default/465/1623853427574-10705” will never be archived，and
> will never be destroyed，and the content_repository will use more storage than
> it configs.{code}
> The above is a sort of phenomenon. The reason is the code below：
> {code:java}
> //session.clone
> public FlowFile clone(FlowFile example, final long offset, final long size) {
> .
> final StandardRepositoryRecord record = new
> StandardRepositoryRecord(null); //here the originalFlowFileRecord of record
> is null
> .
> return clone;
> }
> //session.commit
> private void updateClaimCounts(final RepositoryRecord record) {
> ..
> if (record.isContentModified()) {
> decrementClaimCount(originalClaim); //here the originalClaim is null
> }
> }{code}
> Perhaps we should not use session.clone like that，but without official note
> it will sometimes happen to be used.
> So i change "final StandardRepositoryRecord record = new
> StandardRepositoryRecord(null)" to "final StandardRepositoryRecord record =
> new StandardRepositoryRecord(null, currRec);"
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (NIFI-7000) Document EC toolkit memory usage and best practices
[ https://issues.apache.org/jira/browse/NIFI-7000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann resolved NIFI-7000. Fix Version/s: 1.13.0 Resolution: Fixed > Document EC toolkit memory usage and best practices > --- > > Key: NIFI-7000 > URL: https://issues.apache.org/jira/browse/NIFI-7000 > Project: Apache NiFi > Issue Type: Sub-task > Components: Documentation Website >Affects Versions: 1.10.0 >Reporter: Andy LoPresto >Priority: Major > Labels: best-practice, documentation, security, toolkit > Fix For: 1.13.0 > > > As noted in the parent ticket, the documentation should be updated to explain > how the toolkit uses memory when loading/operating on/writing the files. It > should also provide a warning around potential issues, best practices to > avoid this situation, and steps to resolve. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (NIFI-7002) Handle large files internally in EC toolkit
[
https://issues.apache.org/jira/browse/NIFI-7002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann resolved NIFI-7002.
Fix Version/s: 1.13.0
Resolution: Fixed
> Handle large files internally in EC toolkit
> ---
>
> Key: NIFI-7002
> URL: https://issues.apache.org/jira/browse/NIFI-7002
> Project: Apache NiFi
> Issue Type: Sub-task
> Components: Tools and Build
>Affects Versions: 1.10.0
>Reporter: Andy LoPresto
>Priority: Major
> Labels: encryption, heap, security, streaming, toolkit
> Fix For: 1.13.0
>
>
> The EC toolkit cannot handle large flow definition files (>100 MB
> uncompressed) due to its use of {{String}} to hold the XML content internally
> through frequent manipulations. As Strings are immutable in Java, this causes
> a lot of heap usage and duplication. The read/write to the file system also
> operates on the String directly rather than perform a streaming read/write.
> We should change the internal representation of the file content (for all
> files) to be a {{BufferedInputStream}} or other similar stream representation
> rather than a String. We should test with very large files (1 GB) and clearly
> provide instructions and warnings if the file size exceeds that.
> The logic of value encryption/replacement will have to change to handle this
> approach.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (NIFI-7001) Guard against loading/operating on/serializing large files in EC toolkit
[
https://issues.apache.org/jira/browse/NIFI-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann resolved NIFI-7001.
Fix Version/s: 1.13.0
Resolution: Fixed
> Guard against loading/operating on/serializing large files in EC toolkit
>
>
> Key: NIFI-7001
> URL: https://issues.apache.org/jira/browse/NIFI-7001
> Project: Apache NiFi
> Issue Type: Sub-task
> Components: Tools and Build
>Affects Versions: 1.10.0
>Reporter: Andy LoPresto
>Priority: Major
> Labels: documentation, security, toolkit, validation
> Fix For: 1.13.0
>
>
> The EC toolkit did not anticipate the possibility of a {{flow.xml.gz}} on the
> order of magnitude of 700 MB. Many serialization/deserialization and string
> manipulation operations occur assuming that the content is of manageable
> size. Users have demonstrated that this is not a safe assumption.
> We should introduce the following protections:
> # log debug/info statements at file load, encrypt replacement and file write
> for the size of the content for visibility
> # simple conditional checks at file load to ensure the heap is properly sized
> and the file content size is reasonable
> # if the size is too large, print a helpful message and direct users to the
> Toolkit Guide for further information
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (NIFI-9254) Update Default Sensitive Properties Configuration for Stateless
David Handermann created NIFI-9254: -- Summary: Update Default Sensitive Properties Configuration for Stateless Key: NIFI-9254 URL: https://issues.apache.org/jira/browse/NIFI-9254 Project: Apache NiFi Issue Type: Improvement Components: NiFi Stateless Reporter: David Handermann Assignee: David Handermann Recent improvements to NiFi Component Property Encryption included updating the default Sensitive Properties Algorithm and Sensitive Properties Key. Improving the default settings for NiFi Stateless would provide better security for the initial configuration and reduce reliance on weak key derivation functions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-9241) Review CORS Security Configuration
[
https://issues.apache.org/jira/browse/NIFI-9241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann updated NIFI-9241:
---
Status: Patch Available (was: In Progress)
> Review CORS Security Configuration
> --
>
> Key: NIFI-9241
> URL: https://issues.apache.org/jira/browse/NIFI-9241
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core UI, Security
>Affects Versions: 1.14.0, 1.8.0
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The NiFi Web Security Configuration includes a custom CORS Configuration
> Source that disallows HTTP POST requests for Template Uploads. The works as
> expected with direct access to the NiFi UI, but causes issues when attempting
> to upload a template to NiFi through a reverse proxy.
> When a web browser sends a template upload request that includes an
> unexpected {{Origin}} header, the Spring CORS Filter returns HTTP 403
> Forbidden with a response body containing the message {{Invalid CORS
> Request}}. NIFI-6080 describes a workaround that involves setting a
> different {{Origin}} header. The current approach as implemented in
> NIFI-5595 should be evaluated for potential improvements to avoid this
> behavior when running NiFi with a reverse proxy.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [nifi] exceptionfactory opened a new pull request #5417: NIFI-9241 Refactor CSRF mitigation using random Request-Token
exceptionfactory opened a new pull request #5417:
URL: https://github.com/apache/nifi/pull/5417
Description of PR
NIFI-9241 Refactors the [Cross Site Request
Forgery](https://owasp.org/www-community/attacks/csrf) mitigation strategy to
use a random token cookie with a custom HTTP request header to implement the
[Double Submit
Cookie](https://cheatsheetseries.owasp.org/cheatsheets/Cross-Site_Request_Forgery_Prevention_Cheat_Sheet.html#double-submit-cookie)
pattern. The implementation leverages standard Spring Security CSRF filtering
with a custom CSRF Token Repository.
This approach replaces the current strategy based on comparing the
`Authorization` header with the JSON Web Token provided in a cookie header. The
new strategy sets a random token in a cookie named `__Secure-Request-Token` and
expects browser clients to send the value in a custom HTTP Header named
`Request-Token`. This strategy applies to both Token-based and
Certificate-based authentication, removing the need for a custom Cross Origin
Request Filter configuration implemented for NIFI-5595 in #3024. As a result of
removing the CORS Filter, the HTTP `OPTIONS` method is now unnecessary and
disallowed.
These changes also remove the need for storing the NiFi JSON Web Token in
sessionStorage. Instead of storing the token, the browser client stores the
session expiration time in seconds, parsed from the `exp` claim of the JSON Web
Token. This approach maintains support for indicating login status based on the
presence of a token in sessionStorage, while eliminating the need to store the
entire token. With Token-based authentication supported through a cookie,
multiple browser tabs can be opened for an authentication session.
Additional changes include refactoring Request URI construction using a new
class named `RequestUriBuilder`. This class supports building a URI based on
request information and optional proxy headers. `RequestUriBuilder` provides
consistent URI construction and path resolution for both REST Resource classes
as well as cookie paths.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [X] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [X] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [X] Has your PR been rebased against the latest commit within the target
branch (typically `main`)?
- [X] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [X] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [X] Have you written or updated unit tests to verify your changes?
- [X] Have you verified that the full build is successful on JDK 8?
- [X] Have you verified that the full build is successful on JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for
build issues and submit an update to your PR as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (MINIFICPP-1652) Linux docker image builds fail with MQTT enabled
Gábor Gyimesi created MINIFICPP-1652: Summary: Linux docker image builds fail with MQTT enabled Key: MINIFICPP-1652 URL: https://issues.apache.org/jira/browse/MINIFICPP-1652 Project: Apache NiFi MiNiFi C++ Issue Type: Bug Reporter: Gábor Gyimesi Assignee: Gábor Gyimesi Fedora, centos and ubuntu builds fail built with corresponding dockerfiles. MQTT requires OpenSSL lib on the system to be present instead of using the bundled OpenSSL library, which should be fixed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] markap14 commented on pull request #5412: NIFI-9239: Updated Consume/Publish Kafka processors to support Exactl…
markap14 commented on pull request #5412: URL: https://github.com/apache/nifi/pull/5412#issuecomment-929358954 @gresockj @exceptionfactory huge thanks for all of the very thoughtful and thorough feedback! I will push a new commit momentarily that should address all concerns. And I definitely agree @exceptionfactory that there are a lot more things that should be unit tested. I intended to write a bunch more tests and then it totally slipped my mind. So the next commit will include new tests, mostly around error conditions, such as timeouts, processors that throw exceptions, failure ports, etc. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] markap14 commented on a change in pull request #5412: NIFI-9239: Updated Consume/Publish Kafka processors to support Exactl…
markap14 commented on a change in pull request #5412: URL: https://github.com/apache/nifi/pull/5412#discussion_r717719817 ## File path: nifi-nar-bundles/nifi-stateless-processor-bundle/nifi-stateless-processor/src/main/java/org/apache/nifi/processors/stateless/ExecuteStateless.java ## @@ -0,0 +1,1202 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.processors.stateless; + +import com.fasterxml.jackson.databind.DeserializationFeature; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.fasterxml.jackson.module.jaxb.JaxbAnnotationIntrospector; +import org.apache.nifi.annotation.behavior.DynamicProperty; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.InputRequirement.Requirement; +import org.apache.nifi.annotation.behavior.Restricted; +import org.apache.nifi.annotation.behavior.SupportsBatching; +import org.apache.nifi.annotation.behavior.SystemResource; +import org.apache.nifi.annotation.behavior.SystemResourceConsideration; +import org.apache.nifi.annotation.behavior.SystemResourceConsiderations; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.annotation.lifecycle.OnStopped; +import org.apache.nifi.annotation.lifecycle.OnUnscheduled; +import org.apache.nifi.components.AllowableValue; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.PropertyDescriptor.Builder; +import org.apache.nifi.components.Validator; +import org.apache.nifi.components.resource.ResourceCardinality; +import org.apache.nifi.components.resource.ResourceReference; +import org.apache.nifi.components.resource.ResourceType; +import org.apache.nifi.flow.Bundle; +import org.apache.nifi.flow.VersionedConnection; +import org.apache.nifi.flow.VersionedControllerService; +import org.apache.nifi.flow.VersionedLabel; +import org.apache.nifi.flow.VersionedPort; +import org.apache.nifi.flow.VersionedProcessGroup; +import org.apache.nifi.flow.VersionedProcessor; +import org.apache.nifi.flow.VersionedRemoteGroupPort; +import org.apache.nifi.flow.VersionedRemoteProcessGroup; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.logging.LogLevel; +import org.apache.nifi.processor.AbstractProcessor; +import org.apache.nifi.processor.DataUnit; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.registry.bucket.Bucket; +import org.apache.nifi.registry.client.NiFiRegistryClient; +import org.apache.nifi.registry.client.NiFiRegistryClientConfig; +import org.apache.nifi.registry.client.NiFiRegistryException; +import org.apache.nifi.registry.client.impl.JerseyNiFiRegistryClient; +import org.apache.nifi.registry.flow.VersionedFlow; +import org.apache.nifi.registry.flow.VersionedFlowSnapshot; +import org.apache.nifi.reporting.Bulletin; +import org.apache.nifi.reporting.BulletinQuery; +import org.apache.nifi.reporting.BulletinRepository; +import org.apache.nifi.search.SearchContext; +import org.apache.nifi.search.SearchResult; +import org.apache.nifi.search.Searchable; +import org.apache.nifi.ssl.SSLContextService; +import org.apache.nifi.stateless.bootstrap.StatelessBootstrap; +import org.apache.nifi.stateless.config.ExtensionClientDefinition; +import org.apache.nifi.stateless.config.ParameterContextDefinition; +import org.apache.nifi.stateless.config.ParameterValueProviderDefinition; +import org.apache.nifi.stateless.config.ReportingTaskDefinition; +import org.apache.nifi.stateless.config.SslContextDefinition; +import org.apache.nifi.stateless.config.StatelessConfigurationException; +import org.apache.nifi.stateless.engine.StatelessEngineConfiguration; +import org.apache.nifi.stateless.flow.DataflowDefinition; +import org.apache.nifi.stateless.flow.DataflowTrigger; +import
[GitHub] [nifi] markap14 commented on a change in pull request #5412: NIFI-9239: Updated Consume/Publish Kafka processors to support Exactl…
markap14 commented on a change in pull request #5412: URL: https://github.com/apache/nifi/pull/5412#discussion_r717719235 ## File path: nifi-nar-bundles/nifi-stateless-processor-bundle/nifi-stateless-processor/src/main/java/org/apache/nifi/processors/stateless/ExecuteStateless.java ## @@ -0,0 +1,1202 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.processors.stateless; + +import com.fasterxml.jackson.databind.DeserializationFeature; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.fasterxml.jackson.module.jaxb.JaxbAnnotationIntrospector; +import org.apache.nifi.annotation.behavior.DynamicProperty; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.InputRequirement.Requirement; +import org.apache.nifi.annotation.behavior.Restricted; +import org.apache.nifi.annotation.behavior.SupportsBatching; +import org.apache.nifi.annotation.behavior.SystemResource; +import org.apache.nifi.annotation.behavior.SystemResourceConsideration; +import org.apache.nifi.annotation.behavior.SystemResourceConsiderations; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.annotation.lifecycle.OnStopped; +import org.apache.nifi.annotation.lifecycle.OnUnscheduled; +import org.apache.nifi.components.AllowableValue; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.PropertyDescriptor.Builder; +import org.apache.nifi.components.Validator; +import org.apache.nifi.components.resource.ResourceCardinality; +import org.apache.nifi.components.resource.ResourceReference; +import org.apache.nifi.components.resource.ResourceType; +import org.apache.nifi.flow.Bundle; +import org.apache.nifi.flow.VersionedConnection; +import org.apache.nifi.flow.VersionedControllerService; +import org.apache.nifi.flow.VersionedLabel; +import org.apache.nifi.flow.VersionedPort; +import org.apache.nifi.flow.VersionedProcessGroup; +import org.apache.nifi.flow.VersionedProcessor; +import org.apache.nifi.flow.VersionedRemoteGroupPort; +import org.apache.nifi.flow.VersionedRemoteProcessGroup; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.logging.LogLevel; +import org.apache.nifi.processor.AbstractProcessor; +import org.apache.nifi.processor.DataUnit; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.registry.bucket.Bucket; +import org.apache.nifi.registry.client.NiFiRegistryClient; +import org.apache.nifi.registry.client.NiFiRegistryClientConfig; +import org.apache.nifi.registry.client.NiFiRegistryException; +import org.apache.nifi.registry.client.impl.JerseyNiFiRegistryClient; +import org.apache.nifi.registry.flow.VersionedFlow; +import org.apache.nifi.registry.flow.VersionedFlowSnapshot; +import org.apache.nifi.reporting.Bulletin; +import org.apache.nifi.reporting.BulletinQuery; +import org.apache.nifi.reporting.BulletinRepository; +import org.apache.nifi.search.SearchContext; +import org.apache.nifi.search.SearchResult; +import org.apache.nifi.search.Searchable; +import org.apache.nifi.ssl.SSLContextService; +import org.apache.nifi.stateless.bootstrap.StatelessBootstrap; +import org.apache.nifi.stateless.config.ExtensionClientDefinition; +import org.apache.nifi.stateless.config.ParameterContextDefinition; +import org.apache.nifi.stateless.config.ParameterValueProviderDefinition; +import org.apache.nifi.stateless.config.ReportingTaskDefinition; +import org.apache.nifi.stateless.config.SslContextDefinition; +import org.apache.nifi.stateless.config.StatelessConfigurationException; +import org.apache.nifi.stateless.engine.StatelessEngineConfiguration; +import org.apache.nifi.stateless.flow.DataflowDefinition; +import org.apache.nifi.stateless.flow.DataflowTrigger; +import
[GitHub] [nifi] markap14 commented on a change in pull request #5412: NIFI-9239: Updated Consume/Publish Kafka processors to support Exactl…
markap14 commented on a change in pull request #5412: URL: https://github.com/apache/nifi/pull/5412#discussion_r717718403 ## File path: nifi-nar-bundles/nifi-stateless-processor-bundle/nifi-stateless-processor/src/main/java/org/apache/nifi/processors/stateless/ExecuteStateless.java ## @@ -0,0 +1,1202 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.processors.stateless; + +import com.fasterxml.jackson.databind.DeserializationFeature; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.fasterxml.jackson.module.jaxb.JaxbAnnotationIntrospector; +import org.apache.nifi.annotation.behavior.DynamicProperty; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.InputRequirement.Requirement; +import org.apache.nifi.annotation.behavior.Restricted; +import org.apache.nifi.annotation.behavior.SupportsBatching; +import org.apache.nifi.annotation.behavior.SystemResource; +import org.apache.nifi.annotation.behavior.SystemResourceConsideration; +import org.apache.nifi.annotation.behavior.SystemResourceConsiderations; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.annotation.lifecycle.OnStopped; +import org.apache.nifi.annotation.lifecycle.OnUnscheduled; +import org.apache.nifi.components.AllowableValue; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.PropertyDescriptor.Builder; +import org.apache.nifi.components.Validator; +import org.apache.nifi.components.resource.ResourceCardinality; +import org.apache.nifi.components.resource.ResourceReference; +import org.apache.nifi.components.resource.ResourceType; +import org.apache.nifi.flow.Bundle; +import org.apache.nifi.flow.VersionedConnection; +import org.apache.nifi.flow.VersionedControllerService; +import org.apache.nifi.flow.VersionedLabel; +import org.apache.nifi.flow.VersionedPort; +import org.apache.nifi.flow.VersionedProcessGroup; +import org.apache.nifi.flow.VersionedProcessor; +import org.apache.nifi.flow.VersionedRemoteGroupPort; +import org.apache.nifi.flow.VersionedRemoteProcessGroup; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.logging.LogLevel; +import org.apache.nifi.processor.AbstractProcessor; +import org.apache.nifi.processor.DataUnit; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.registry.bucket.Bucket; +import org.apache.nifi.registry.client.NiFiRegistryClient; +import org.apache.nifi.registry.client.NiFiRegistryClientConfig; +import org.apache.nifi.registry.client.NiFiRegistryException; +import org.apache.nifi.registry.client.impl.JerseyNiFiRegistryClient; +import org.apache.nifi.registry.flow.VersionedFlow; +import org.apache.nifi.registry.flow.VersionedFlowSnapshot; +import org.apache.nifi.reporting.Bulletin; +import org.apache.nifi.reporting.BulletinQuery; +import org.apache.nifi.reporting.BulletinRepository; +import org.apache.nifi.search.SearchContext; +import org.apache.nifi.search.SearchResult; +import org.apache.nifi.search.Searchable; +import org.apache.nifi.ssl.SSLContextService; +import org.apache.nifi.stateless.bootstrap.StatelessBootstrap; +import org.apache.nifi.stateless.config.ExtensionClientDefinition; +import org.apache.nifi.stateless.config.ParameterContextDefinition; +import org.apache.nifi.stateless.config.ParameterValueProviderDefinition; +import org.apache.nifi.stateless.config.ReportingTaskDefinition; +import org.apache.nifi.stateless.config.SslContextDefinition; +import org.apache.nifi.stateless.config.StatelessConfigurationException; +import org.apache.nifi.stateless.engine.StatelessEngineConfiguration; +import org.apache.nifi.stateless.flow.DataflowDefinition; +import org.apache.nifi.stateless.flow.DataflowTrigger; +import
[GitHub] [nifi] markap14 commented on a change in pull request #5412: NIFI-9239: Updated Consume/Publish Kafka processors to support Exactl…
markap14 commented on a change in pull request #5412:
URL: https://github.com/apache/nifi/pull/5412#discussion_r717715814
##
File path:
nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-2-6-processors/src/main/java/org/apache/nifi/processors/kafka/pubsub/PublishKafka_2_6.java
##
@@ -493,6 +517,82 @@ public void process(final InputStream rawIn) throws
IOException {
}
}
+private List pollFlowFiles(final ProcessContext context, final
ProcessSession session) {
Review comment:
Yes, good call. Can pull a few methods into a Publisher util class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (NIFI-9253) Build broken by nifi-security-socket-ssl test issue
[ https://issues.apache.org/jira/browse/NIFI-9253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17421463#comment-17421463 ] Joe Witt commented on NIFI-9253: The issue is intermittent. I have run two builds since and the issue has not happened. So brittle build problem. > Build broken by nifi-security-socket-ssl test issue > --- > > Key: NIFI-9253 > URL: https://issues.apache.org/jira/browse/NIFI-9253 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework > Environment: Apache Maven 3.8.1 > (05c21c65bdfed0f71a2f2ada8b84da59348c4c5d) > Maven home: /development/tools/apache-maven-3.8.1 > Java version: 11.0.12, vendor: Azul Systems, Inc., runtime: > /usr/lib/jvm/zulu11.50.19-ca-jdk11.0.12-linux_x64 > Default locale: en_US, platform encoding: UTF-8 > OS name: "linux", version: "5.13.15-200.fc34.x86_64", arch: "amd64", family: > "unix" >Reporter: Joe Witt >Assignee: David Handermann >Priority: Major > > [ERROR] WARNING: An illegal reflective access operation has occurred > [ERROR] WARNING: Illegal reflective access by > com.esotericsoftware.kryo.util.UnsafeUtil > (file:/home/apachenifi/.m2/repository/com/esotericsoftware/kryo/kryo/2.24.0/kryo-2.24.0.jar) > to constructor java.nio.DirectByteBuffer(long,int,java.lang.Object) > [ERROR] WARNING: Please consider reporting this to the maintainers of > com.esotericsoftware.kryo.util.UnsafeUtil > [ERROR] WARNING: Use --illegal-access=warn to enable warnings of further > illegal reflective access operations > [ERROR] WARNING: All illegal access operations will be denied in a future > release > [ERROR] WARNING: An illegal reflective access operation has occurred > [ERROR] WARNING: Illegal reflective access by > org.codehaus.groovy.reflection.CachedClass > (file:/home/apachenifi/.m2/repository/org/codehaus/groovy/groovy/2.5.14/groovy-2.5.14.jar) > to method java.lang.Object.finalize() > [ERROR] WARNING: Please consider reporting this to the maintainers of > org.codehaus.groovy.reflection.CachedClass > [ERROR] WARNING: Use --illegal-access=warn to enable warnings of further > illegal reflective access operations > [ERROR] WARNING: All illegal access operations will be denied in a future > release > [INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.424 > s - in > org.apache.nifi.authorization.CompositeConfigurableUserGroupProviderTest > [INFO] Running org.apache.nifi.authorization.StandardManagedAuthorizerTest > [INFO] Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.17 > s - in org.apache.nifi.authorization.StandardManagedAuthorizerTest > [INFO] Running org.apache.nifi.authorization.CompositeUserGroupProviderTest > [INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.014 > s - in org.apache.nifi.authorization.CompositeUserGroupProviderTest > [INFO] Running org.apache.nifi.authorization.user.NiFiUserUtilsTest > [INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.485 > s - in org.apache.nifi.properties.AESSensitivePropertyProviderTest > [ERROR] WARNING: An illegal reflective access operation has occurred > [ERROR] WARNING: Illegal reflective access by > org.codehaus.groovy.reflection.CachedClass > (file:/home/apachenifi/.m2/repository/org/codehaus/groovy/groovy/2.5.14/groovy-2.5.14.jar) > to method java.lang.Object.finalize() > [ERROR] WARNING: Please consider reporting this to the maintainers of > org.codehaus.groovy.reflection.CachedClass > [ERROR] WARNING: Use --illegal-access=warn to enable warnings of further > illegal reflective access operations > [ERROR] WARNING: All illegal access operations will be denied in a future > release > [ERROR] Errors: > [ERROR] > SSLSocketChannelTest.testClientConnectWriteReadTls13:147->assertChannelConnectedWriteReadClosed:208->processClientSslSocketChannel:255->lambda$assertChannelConnectedWriteReadClosed$1:218 > UncheckedIO > [INFO] > [ERROR] Tests run: 6, Failures: 0, Errors: 1, Skipped: 0 > [INFO] Skipping because packaging 'nar' is not pom. > [ERROR] Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 12.95 > s <<< FAILURE! - in org.apache.nifi.remote.io.socket.ssl.SSLSocketChannelTest > [ERROR] > org.apache.nifi.remote.io.socket.ssl.SSLSocketChannelTest.testClientConnectWriteReadTls13 > Time elapsed: 2.116 s <<< ERROR! > java.io.UncheckedIOException: Channel Failed for TLSv1.3 > at > org.apache.nifi.remote.io.socket.ssl.SSLSocketChannelTest.lambda$assertChannelConnectedWriteReadClosed$1(SSLSocketChannelTest.java:218) > at > org.apache.nifi.remote.io.socket.ssl.SSLSocketChannelTest.processClientSslSocketChannel(SSLSocketChannelTest.java:255) > at >
[jira] [Assigned] (NIFI-9253) Build broken by nifi-security-socket-ssl test issue
[ https://issues.apache.org/jira/browse/NIFI-9253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann reassigned NIFI-9253: -- Assignee: David Handermann > Build broken by nifi-security-socket-ssl test issue > --- > > Key: NIFI-9253 > URL: https://issues.apache.org/jira/browse/NIFI-9253 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework > Environment: Apache Maven 3.8.1 > (05c21c65bdfed0f71a2f2ada8b84da59348c4c5d) > Maven home: /development/tools/apache-maven-3.8.1 > Java version: 11.0.12, vendor: Azul Systems, Inc., runtime: > /usr/lib/jvm/zulu11.50.19-ca-jdk11.0.12-linux_x64 > Default locale: en_US, platform encoding: UTF-8 > OS name: "linux", version: "5.13.15-200.fc34.x86_64", arch: "amd64", family: > "unix" >Reporter: Joe Witt >Assignee: David Handermann >Priority: Major > > [ERROR] WARNING: An illegal reflective access operation has occurred > [ERROR] WARNING: Illegal reflective access by > com.esotericsoftware.kryo.util.UnsafeUtil > (file:/home/apachenifi/.m2/repository/com/esotericsoftware/kryo/kryo/2.24.0/kryo-2.24.0.jar) > to constructor java.nio.DirectByteBuffer(long,int,java.lang.Object) > [ERROR] WARNING: Please consider reporting this to the maintainers of > com.esotericsoftware.kryo.util.UnsafeUtil > [ERROR] WARNING: Use --illegal-access=warn to enable warnings of further > illegal reflective access operations > [ERROR] WARNING: All illegal access operations will be denied in a future > release > [ERROR] WARNING: An illegal reflective access operation has occurred > [ERROR] WARNING: Illegal reflective access by > org.codehaus.groovy.reflection.CachedClass > (file:/home/apachenifi/.m2/repository/org/codehaus/groovy/groovy/2.5.14/groovy-2.5.14.jar) > to method java.lang.Object.finalize() > [ERROR] WARNING: Please consider reporting this to the maintainers of > org.codehaus.groovy.reflection.CachedClass > [ERROR] WARNING: Use --illegal-access=warn to enable warnings of further > illegal reflective access operations > [ERROR] WARNING: All illegal access operations will be denied in a future > release > [INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.424 > s - in > org.apache.nifi.authorization.CompositeConfigurableUserGroupProviderTest > [INFO] Running org.apache.nifi.authorization.StandardManagedAuthorizerTest > [INFO] Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.17 > s - in org.apache.nifi.authorization.StandardManagedAuthorizerTest > [INFO] Running org.apache.nifi.authorization.CompositeUserGroupProviderTest > [INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.014 > s - in org.apache.nifi.authorization.CompositeUserGroupProviderTest > [INFO] Running org.apache.nifi.authorization.user.NiFiUserUtilsTest > [INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.485 > s - in org.apache.nifi.properties.AESSensitivePropertyProviderTest > [ERROR] WARNING: An illegal reflective access operation has occurred > [ERROR] WARNING: Illegal reflective access by > org.codehaus.groovy.reflection.CachedClass > (file:/home/apachenifi/.m2/repository/org/codehaus/groovy/groovy/2.5.14/groovy-2.5.14.jar) > to method java.lang.Object.finalize() > [ERROR] WARNING: Please consider reporting this to the maintainers of > org.codehaus.groovy.reflection.CachedClass > [ERROR] WARNING: Use --illegal-access=warn to enable warnings of further > illegal reflective access operations > [ERROR] WARNING: All illegal access operations will be denied in a future > release > [ERROR] Errors: > [ERROR] > SSLSocketChannelTest.testClientConnectWriteReadTls13:147->assertChannelConnectedWriteReadClosed:208->processClientSslSocketChannel:255->lambda$assertChannelConnectedWriteReadClosed$1:218 > UncheckedIO > [INFO] > [ERROR] Tests run: 6, Failures: 0, Errors: 1, Skipped: 0 > [INFO] Skipping because packaging 'nar' is not pom. > [ERROR] Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 12.95 > s <<< FAILURE! - in org.apache.nifi.remote.io.socket.ssl.SSLSocketChannelTest > [ERROR] > org.apache.nifi.remote.io.socket.ssl.SSLSocketChannelTest.testClientConnectWriteReadTls13 > Time elapsed: 2.116 s <<< ERROR! > java.io.UncheckedIOException: Channel Failed for TLSv1.3 > at > org.apache.nifi.remote.io.socket.ssl.SSLSocketChannelTest.lambda$assertChannelConnectedWriteReadClosed$1(SSLSocketChannelTest.java:218) > at > org.apache.nifi.remote.io.socket.ssl.SSLSocketChannelTest.processClientSslSocketChannel(SSLSocketChannelTest.java:255) > at > org.apache.nifi.remote.io.socket.ssl.SSLSocketChannelTest.assertChannelConnectedWriteReadClosed(SSLSocketChannelTest.java:208) > at >
[jira] [Updated] (NIFI-9253) Build broken by nifi-security-socket-ssl test issue
[ https://issues.apache.org/jira/browse/NIFI-9253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Witt updated NIFI-9253: --- Description: [ERROR] WARNING: An illegal reflective access operation has occurred [ERROR] WARNING: Illegal reflective access by com.esotericsoftware.kryo.util.UnsafeUtil (file:/home/apachenifi/.m2/repository/com/esotericsoftware/kryo/kryo/2.24.0/kryo-2.24.0.jar) to constructor java.nio.DirectByteBuffer(long,int,java.lang.Object) [ERROR] WARNING: Please consider reporting this to the maintainers of com.esotericsoftware.kryo.util.UnsafeUtil [ERROR] WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations [ERROR] WARNING: All illegal access operations will be denied in a future release [ERROR] WARNING: An illegal reflective access operation has occurred [ERROR] WARNING: Illegal reflective access by org.codehaus.groovy.reflection.CachedClass (file:/home/apachenifi/.m2/repository/org/codehaus/groovy/groovy/2.5.14/groovy-2.5.14.jar) to method java.lang.Object.finalize() [ERROR] WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.reflection.CachedClass [ERROR] WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations [ERROR] WARNING: All illegal access operations will be denied in a future release [INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.424 s - in org.apache.nifi.authorization.CompositeConfigurableUserGroupProviderTest [INFO] Running org.apache.nifi.authorization.StandardManagedAuthorizerTest [INFO] Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.17 s - in org.apache.nifi.authorization.StandardManagedAuthorizerTest [INFO] Running org.apache.nifi.authorization.CompositeUserGroupProviderTest [INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.014 s - in org.apache.nifi.authorization.CompositeUserGroupProviderTest [INFO] Running org.apache.nifi.authorization.user.NiFiUserUtilsTest [INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.485 s - in org.apache.nifi.properties.AESSensitivePropertyProviderTest [ERROR] WARNING: An illegal reflective access operation has occurred [ERROR] WARNING: Illegal reflective access by org.codehaus.groovy.reflection.CachedClass (file:/home/apachenifi/.m2/repository/org/codehaus/groovy/groovy/2.5.14/groovy-2.5.14.jar) to method java.lang.Object.finalize() [ERROR] WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.reflection.CachedClass [ERROR] WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations [ERROR] WARNING: All illegal access operations will be denied in a future release [ERROR] Errors: [ERROR] SSLSocketChannelTest.testClientConnectWriteReadTls13:147->assertChannelConnectedWriteReadClosed:208->processClientSslSocketChannel:255->lambda$assertChannelConnectedWriteReadClosed$1:218 UncheckedIO [INFO] [ERROR] Tests run: 6, Failures: 0, Errors: 1, Skipped: 0 [INFO] Skipping because packaging 'nar' is not pom. [ERROR] Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 12.95 s <<< FAILURE! - in org.apache.nifi.remote.io.socket.ssl.SSLSocketChannelTest [ERROR] org.apache.nifi.remote.io.socket.ssl.SSLSocketChannelTest.testClientConnectWriteReadTls13 Time elapsed: 2.116 s <<< ERROR! java.io.UncheckedIOException: Channel Failed for TLSv1.3 at org.apache.nifi.remote.io.socket.ssl.SSLSocketChannelTest.lambda$assertChannelConnectedWriteReadClosed$1(SSLSocketChannelTest.java:218) at org.apache.nifi.remote.io.socket.ssl.SSLSocketChannelTest.processClientSslSocketChannel(SSLSocketChannelTest.java:255) at org.apache.nifi.remote.io.socket.ssl.SSLSocketChannelTest.assertChannelConnectedWriteReadClosed(SSLSocketChannelTest.java:208) at org.apache.nifi.remote.io.socket.ssl.SSLSocketChannelTest.testClientConnectWriteReadTls13(SSLSocketChannelTest.java:147) Caused by: javax.net.ssl.SSLException: Tag mismatch! at org.apache.nifi.remote.io.socket.ssl.SSLSocketChannelTest.assertChannelWriteRead(SSLSocketChannelTest.java:234) at org.apache.nifi.remote.io.socket.ssl.SSLSocketChannelTest.lambda$assertChannelConnectedWriteReadClosed$1(SSLSocketChannelTest.java:213) at org.apache.nifi.remote.io.socket.ssl.SSLSocketChannelTest.processClientSslSocketChannel(SSLSocketChannelTest.java:255) at org.apache.nifi.remote.io.socket.ssl.SSLSocketChannelTest.assertChannelConnectedWriteReadClosed(SSLSocketChannelTest.java:208) at org.apache.nifi.remote.io.socket.ssl.SSLSocketChannelTest.testClientConnectWriteReadTls13(SSLSocketChannelTest.java:147) Caused by: javax.crypto.AEADBadTagException: Tag mismatch! at
[jira] [Updated] (NIFI-9253) Build broken by nifi-security-socket-ssl test issue
[ https://issues.apache.org/jira/browse/NIFI-9253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Witt updated NIFI-9253: --- Description: [ERROR] WARNING: An illegal reflective access operation has occurred [ERROR] WARNING: Illegal reflective access by com.esotericsoftware.kryo.util.UnsafeUtil (file:/home/apachenifi/.m2/repository/com/esotericsoftware/kryo/kryo/2.24.0/kryo-2.24.0.jar) to constructor java.nio.DirectByteBuffer(long,int,java.lang.Object) [ERROR] WARNING: Please consider reporting this to the maintainers of com.esotericsoftware.kryo.util.UnsafeUtil [ERROR] WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations [ERROR] WARNING: All illegal access operations will be denied in a future release [ERROR] WARNING: An illegal reflective access operation has occurred [ERROR] WARNING: Illegal reflective access by org.codehaus.groovy.reflection.CachedClass (file:/home/apachenifi/.m2/repository/org/codehaus/groovy/groovy/2.5.14/groovy-2.5.14.jar) to method java.lang.Object.finalize() [ERROR] WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.reflection.CachedClass [ERROR] WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations [ERROR] WARNING: All illegal access operations will be denied in a future release [INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.424 s - in org.apache.nifi.authorization.CompositeConfigurableUserGroupProviderTest [INFO] Running org.apache.nifi.authorization.StandardManagedAuthorizerTest [INFO] Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.17 s - in org.apache.nifi.authorization.StandardManagedAuthorizerTest [INFO] Running org.apache.nifi.authorization.CompositeUserGroupProviderTest [INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.014 s - in org.apache.nifi.authorization.CompositeUserGroupProviderTest [INFO] Running org.apache.nifi.authorization.user.NiFiUserUtilsTest [INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.485 s - in org.apache.nifi.properties.AESSensitivePropertyProviderTest [ERROR] WARNING: An illegal reflective access operation has occurred [ERROR] WARNING: Illegal reflective access by org.codehaus.groovy.reflection.CachedClass (file:/home/apachenifi/.m2/repository/org/codehaus/groovy/groovy/2.5.14/groovy-2.5.14.jar) to method java.lang.Object.finalize() [ERROR] WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.reflection.CachedClass [ERROR] WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations [ERROR] WARNING: All illegal access operations will be denied in a future release [ERROR] Errors: [ERROR] SSLSocketChannelTest.testClientConnectWriteReadTls13:147->assertChannelConnectedWriteReadClosed:208->processClientSslSocketChannel:255->lambda$assertChannelConnectedWriteReadClosed$1:218 UncheckedIO [INFO] [ERROR] Tests run: 6, Failures: 0, Errors: 1, Skipped: 0 was: [ERROR] WARNING: An illegal reflective access operation has occurred [ERROR] WARNING: Illegal reflective access by com.esotericsoftware.kryo.util.UnsafeUtil (file:/home/apachenifi/.m2/repository/com/esotericsoftware/kryo/kryo/2.24.0/kryo-2.24.0.jar) to constructor java.nio.DirectByteBuffer(long,int,java.lang.Object) [ERROR] WARNING: Please consider reporting this to the maintainers of com.esotericsoftware.kryo.util.UnsafeUtil [ERROR] WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations [ERROR] WARNING: All illegal access operations will be denied in a future release [ERROR] WARNING: An illegal reflective access operation has occurred [ERROR] WARNING: Illegal reflective access by org.codehaus.groovy.reflection.CachedClass (file:/home/apachenifi/.m2/repository/org/codehaus/groovy/groovy/2.5.14/groovy-2.5.14.jar) to method java.lang.Object.finalize() [ERROR] WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.reflection.CachedClass [ERROR] WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations [ERROR] WARNING: All illegal access operations will be denied in a future release [INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.424 s - in org.apache.nifi.authorization.CompositeConfigurableUserGroupProviderTest [INFO] Running org.apache.nifi.authorization.StandardManagedAuthorizerTest [INFO] Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.17 s - in org.apache.nifi.authorization.StandardManagedAuthorizerTest [INFO] Running org.apache.nifi.authorization.CompositeUserGroupProviderTest [INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.014 s - in org.apache.nifi.authorization.CompositeUserGroupProviderTest [INFO] Running
[jira] [Created] (NIFI-9253) Build broken by nifi-security-socket-ssl test issue
Joe Witt created NIFI-9253: -- Summary: Build broken by nifi-security-socket-ssl test issue Key: NIFI-9253 URL: https://issues.apache.org/jira/browse/NIFI-9253 Project: Apache NiFi Issue Type: Bug Components: Core Framework Environment: Apache Maven 3.8.1 (05c21c65bdfed0f71a2f2ada8b84da59348c4c5d) Maven home: /development/tools/apache-maven-3.8.1 Java version: 11.0.12, vendor: Azul Systems, Inc., runtime: /usr/lib/jvm/zulu11.50.19-ca-jdk11.0.12-linux_x64 Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "5.13.15-200.fc34.x86_64", arch: "amd64", family: "unix" Reporter: Joe Witt [ERROR] WARNING: An illegal reflective access operation has occurred [ERROR] WARNING: Illegal reflective access by com.esotericsoftware.kryo.util.UnsafeUtil (file:/home/apachenifi/.m2/repository/com/esotericsoftware/kryo/kryo/2.24.0/kryo-2.24.0.jar) to constructor java.nio.DirectByteBuffer(long,int,java.lang.Object) [ERROR] WARNING: Please consider reporting this to the maintainers of com.esotericsoftware.kryo.util.UnsafeUtil [ERROR] WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations [ERROR] WARNING: All illegal access operations will be denied in a future release [ERROR] WARNING: An illegal reflective access operation has occurred [ERROR] WARNING: Illegal reflective access by org.codehaus.groovy.reflection.CachedClass (file:/home/apachenifi/.m2/repository/org/codehaus/groovy/groovy/2.5.14/groovy-2.5.14.jar) to method java.lang.Object.finalize() [ERROR] WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.reflection.CachedClass [ERROR] WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations [ERROR] WARNING: All illegal access operations will be denied in a future release [INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.424 s - in org.apache.nifi.authorization.CompositeConfigurableUserGroupProviderTest [INFO] Running org.apache.nifi.authorization.StandardManagedAuthorizerTest [INFO] Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.17 s - in org.apache.nifi.authorization.StandardManagedAuthorizerTest [INFO] Running org.apache.nifi.authorization.CompositeUserGroupProviderTest [INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.014 s - in org.apache.nifi.authorization.CompositeUserGroupProviderTest [INFO] Running org.apache.nifi.authorization.user.NiFiUserUtilsTest [INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.485 s - in org.apache.nifi.properties.AESSensitivePropertyProviderTest [ERROR] WARNING: An illegal reflective access operation has occurred [ERROR] WARNING: Illegal reflective access by org.codehaus.groovy.reflection.CachedClass (file:/home/apachenifi/.m2/repository/org/codehaus/groovy/groovy/2.5.14/groovy-2.5.14.jar) to method java.lang.Object.finalize() [ERROR] WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.reflection.CachedClass [ERROR] WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations [ERROR] WARNING: All illegal access operations will be denied in a future release -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-9251) Disable unused password generation in Registry
[ https://issues.apache.org/jira/browse/NIFI-9251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-9251: --- Status: Patch Available (was: Open) > Disable unused password generation in Registry > -- > > Key: NIFI-9251 > URL: https://issues.apache.org/jira/browse/NIFI-9251 > Project: Apache NiFi > Issue Type: Improvement > Components: NiFi Registry >Affects Versions: 1.14.0 >Reporter: David Handermann >Assignee: David Handermann >Priority: Minor > Labels: security > Time Spent: 10m > Remaining Estimate: 0h > > During initialization, NiFi Registry logs a generated UUID password as a > result of Spring UserDetailsServiceAutoConfiguration. This password is not > used, but the application configuration should be adjusted to avoid > initializing the class and logging the generated password. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] exceptionfactory opened a new pull request #5416: NIFI-9251 Add AuthenticationManager Bean to avoid password generation
exceptionfactory opened a new pull request #5416:
URL: https://github.com/apache/nifi/pull/5416
Description of PR
NIFI-9251 Adds an `AuthenticationManager` Bean definition to the NiFi
Registry Spring Security configuration to avoid unnecessary automatic
configuration, which generates and logs a random password. NiFi Registry
authentication does not use the generated password, so this update clarifies
the expected behavior and removes unexpected log messages.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [X] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [X] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [X] Has your PR been rebased against the latest commit within the target
branch (typically `main`)?
- [X] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] Have you verified that the full build is successful on JDK 8?
- [X] Have you verified that the full build is successful on JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for
build issues and submit an update to your PR as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] mattyb149 commented on pull request #5318: NIFI-9064:Support Oracle timestamp when `Use Avro Logical Types` is t…
mattyb149 commented on pull request #5318: URL: https://github.com/apache/nifi/pull/5318#issuecomment-929283337 There's a merge conflict now, and since you have a couple of merge commits in this PR, I recommend cleaning it up by rebasing or squashing merge commits or whatever, so your only commits are your changes to the code (and there are no merge conflicts), thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFI-9252) Remove nifi.sensitive.props.provider Property
David Handermann created NIFI-9252:
--
Summary: Remove nifi.sensitive.props.provider Property
Key: NIFI-9252
URL: https://issues.apache.org/jira/browse/NIFI-9252
Project: Apache NiFi
Issue Type: Improvement
Components: Core Framework, Security
Reporter: David Handermann
Assignee: David Handermann
With recent updates to support additional Sensitive Properties Key Algorithms,
the framework no longer uses the {{nifi.sensitive.props.provider}} property
defined in {{nifi.properties}}. Property name references and example values
should be removed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [nifi] adenes edited a comment on pull request #5414: NIFI-9248 Use hive-exec:core instead of the "regular" hive-exec dependency
adenes edited a comment on pull request #5414: URL: https://github.com/apache/nifi/pull/5414#issuecomment-929212599 Thanks for the comment, @exceptionfactory > For reference, what factors determined which explicit dependencies to include after excluding `hive-exec` from `hive-streaming`? The Maven configuration for `hive-exec` appears to include a handful of additional dependencies, but are all of those available through currently declared dependencies? It would be helpful to have some process to track this down in the event of future version updates to `hive-exec`. I ran a `mvn dependency:tree` on the `nifi-nar-bundles/nifi-hive-bundle/nifi-hive3-processors` module and checked which dependencies are missing from the list [1] and added the missing ones as direct dependencies. I had an ugly oneliner: ``(for a in `cat ~/tmp/hive-shadedlibs`; do ver=$(cat deptree | grep $a | grep -E ":compile|:runtime" | head -1 | grep -Eo ":\d+(\.\d+)*"); echo $a$ver; done) >result`` where - `~/tmp/hive-shadedlibs` was manually created from the below linked hive `pom.xml` snippet in the format of `groupId:artifactId` - https://gist.github.com/adenes/7711eb7bd7873002ae46022c84d7cab1 - `deptree` is the output of `mvn dependency:tree -pl nifi-nar-bundles/nifi-hive-bundle/nifi-hive3-processors` - the output of this (`result`) should contain all the libraries from the `hive-shadedlibs` with version number - https://gist.github.com/adenes/f5a734be1fb78af9fbf3a49edf05e1fe [1] https://github.com/apache/hive/blob/rel/release-3.1.2/ql/pom.xml#L925-L961 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] adenes commented on pull request #5414: NIFI-9248 Use hive-exec:core instead of the "regular" hive-exec dependency
adenes commented on pull request #5414: URL: https://github.com/apache/nifi/pull/5414#issuecomment-929212599 Thanks for the comment, @exceptionfactory > For reference, what factors determined which explicit dependencies to include after excluding `hive-exec` from `hive-streaming`? The Maven configuration for `hive-exec` appears to include a handful of additional dependencies, but are all of those available through currently declared dependencies? It would be helpful to have some process to track this down in the event of future version updates to `hive-exec`. I ran a `mvn dependency:tree` on the `nifi-nar-bundles/nifi-hive-bundle/nifi-hive3-processors` module and checked which dependencies are missing from the list [1] and added the missing ones as direct dependencies. I had an ugly oneliner: `(for a in `cat ~/tmp/hive-shadedlibs`; do ver=$(cat deptree | grep $a | grep -E ":compile|:runtime" | head -1 | grep -Eo ":\d+(\.\d+)*"); echo $a$ver; done) >result` where - `~/tmp/hive-shadedlibs` was manually created from the below linked hive `pom.xml` snippet in the format of `groupId:artifactId` - https://gist.github.com/adenes/7711eb7bd7873002ae46022c84d7cab1 - `deptree` is the output of `mvn dependency:tree -pl nifi-nar-bundles/nifi-hive-bundle/nifi-hive3-processors` - the output of this (`result`) should contain all the libraries from the `hive-shadedlibs` with version number - https://gist.github.com/adenes/f5a734be1fb78af9fbf3a49edf05e1fe [1] https://github.com/apache/hive/blob/rel/release-3.1.2/ql/pom.xml#L925-L961 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFI-9251) Disable used password generation in Registry
David Handermann created NIFI-9251: -- Summary: Disable used password generation in Registry Key: NIFI-9251 URL: https://issues.apache.org/jira/browse/NIFI-9251 Project: Apache NiFi Issue Type: Improvement Components: NiFi Registry Affects Versions: 1.14.0 Reporter: David Handermann Assignee: David Handermann During initialization, NiFi Registry logs a generated UUID password as a result of Spring UserDetailsServiceAutoConfiguration. This password is not used, but the application configuration should be adjusted to avoid initializing the class and logging the generated password. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-9251) Disable unused password generation in Registry
[ https://issues.apache.org/jira/browse/NIFI-9251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-9251: --- Summary: Disable unused password generation in Registry (was: Disable used password generation in Registry) > Disable unused password generation in Registry > -- > > Key: NIFI-9251 > URL: https://issues.apache.org/jira/browse/NIFI-9251 > Project: Apache NiFi > Issue Type: Improvement > Components: NiFi Registry >Affects Versions: 1.14.0 >Reporter: David Handermann >Assignee: David Handermann >Priority: Minor > Labels: security > > During initialization, NiFi Registry logs a generated UUID password as a > result of Spring UserDetailsServiceAutoConfiguration. This password is not > used, but the application configuration should be adjusted to avoid > initializing the class and logging the generated password. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (NIFI-9249) OIDC with Java 11 - the trustAnchors parameter must be non-empty
[
https://issues.apache.org/jira/browse/NIFI-9249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre Villard resolved NIFI-9249.
--
Resolution: Not A Bug
> OIDC with Java 11 - the trustAnchors parameter must be non-empty
>
>
> Key: NIFI-9249
> URL: https://issues.apache.org/jira/browse/NIFI-9249
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.14.0
> Environment: openjdk version "11.0.6" 2020-01-14
> OpenJDK Runtime Environment (build 11.0.6+10-post-Debian-1bpo91)
> OpenJDK 64-Bit Server VM (build 11.0.6+10-post-Debian-1bpo91, mixed mode,
> sharing)
>Reporter: Pierre Villard
>Priority: Major
> Attachments: nifi-bootstrap (1).log, nifi-bootstrap (2).log,
> nifi-bootstrap.log, stacktrace.txt
>
>
> With the exact same configuration, when switching from Java 8 to Java 11, I
> get the following error when using OIDC for authentication:
> {code:java}
> 2021-09-27 09:52:57,733 WARN [main] org.apache.nifi.web.server.JettyServer
> Failed to start web server... shutting down.
> org.springframework.beans.factory.BeanCreationException: Error creating bean
> with name 'oidcService' defined in class path resource
> [nifi-web-security-context.xml]: Bean instantiation via constructor failed;
> nested exception is org.springframework.beans.BeanInstantiationException:
> Failed to instantiate [org.apache.nifi.web.security.oidc.OidcService]:
> Constructor threw exception; nested exception is java.lang.RuntimeException:
> Unable to retrieve OpenId Connect Provider metadata from:
> https://accounts.google.com/.well-known/openid-configuration
> at
> org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:315)
> at
> org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:296)
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1354)
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1204)
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:564)
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:524)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
> at
> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
> at
> org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:944)
> at
> org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918)
> at
> org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583)
> at
> org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:401)
> at
> org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:292)
> at
> org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:103)
> at
> org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1068)
> at
> org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)
> at
> org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:997)
> at
> org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:746)
> at
> org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
> at
> org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
> at
> org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:911)
> at
> org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
> at
>
[jira] [Commented] (NIFI-9249) OIDC with Java 11 - the trustAnchors parameter must be non-empty
[
https://issues.apache.org/jira/browse/NIFI-9249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17421329#comment-17421329
]
Pierre Villard commented on NIFI-9249:
--
When trying with
{code:java}
$ java -version
openjdk version "11.0.12" 2021-07-20 LTS
OpenJDK Runtime Environment Zulu11.50+19-CA (build 11.0.12+7-LTS)
OpenJDK 64-Bit Server VM Zulu11.50+19-CA (build 11.0.12+7-LTS, mixed mode){code}
This works as expected. Closing. Thanks [~exceptionfactory]!
> OIDC with Java 11 - the trustAnchors parameter must be non-empty
>
>
> Key: NIFI-9249
> URL: https://issues.apache.org/jira/browse/NIFI-9249
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.14.0
> Environment: openjdk version "11.0.6" 2020-01-14
> OpenJDK Runtime Environment (build 11.0.6+10-post-Debian-1bpo91)
> OpenJDK 64-Bit Server VM (build 11.0.6+10-post-Debian-1bpo91, mixed mode,
> sharing)
>Reporter: Pierre Villard
>Priority: Major
> Attachments: nifi-bootstrap (1).log, nifi-bootstrap (2).log,
> nifi-bootstrap.log, stacktrace.txt
>
>
> With the exact same configuration, when switching from Java 8 to Java 11, I
> get the following error when using OIDC for authentication:
> {code:java}
> 2021-09-27 09:52:57,733 WARN [main] org.apache.nifi.web.server.JettyServer
> Failed to start web server... shutting down.
> org.springframework.beans.factory.BeanCreationException: Error creating bean
> with name 'oidcService' defined in class path resource
> [nifi-web-security-context.xml]: Bean instantiation via constructor failed;
> nested exception is org.springframework.beans.BeanInstantiationException:
> Failed to instantiate [org.apache.nifi.web.security.oidc.OidcService]:
> Constructor threw exception; nested exception is java.lang.RuntimeException:
> Unable to retrieve OpenId Connect Provider metadata from:
> https://accounts.google.com/.well-known/openid-configuration
> at
> org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:315)
> at
> org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:296)
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1354)
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1204)
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:564)
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:524)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
> at
> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
> at
> org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:944)
> at
> org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918)
> at
> org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583)
> at
> org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:401)
> at
> org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:292)
> at
> org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:103)
> at
> org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1068)
> at
> org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)
> at
> org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:997)
> at
> org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:746)
> at
> org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
> at
> org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
> at
>
[GitHub] [nifi] pgyori commented on pull request #5388: NIFI-9192: ResultSetRecordSet considers value of useLogicalTypes flag…
pgyori commented on pull request #5388: URL: https://github.com/apache/nifi/pull/5388#issuecomment-929047305 Thank you, @mattyb149 ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] martinzink opened a new pull request #1181: MINIFICPP-1650: ProcessSession::append sets the flowfile size
martinzink opened a new pull request #1181: URL: https://github.com/apache/nifi-minifi-cpp/pull/1181 incorrectly Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFICPP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically main)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (MINIFICPP-1644) ProcessSession::read ignores the size/offset of the flowfile
[
https://issues.apache.org/jira/browse/MINIFICPP-1644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martin Zink updated MINIFICPP-1644:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
https://github.com/apache/nifi-minifi-cpp/commit/f6bbc46601d17a46aed2cea17b9aec68bfc4f629
> ProcessSession::read ignores the size/offset of the flowfile
>
>
> Key: MINIFICPP-1644
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1644
> Project: Apache NiFi MiNiFi C++
> Issue Type: Bug
>Reporter: Martin Zink
>Assignee: Martin Zink
>Priority: Minor
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
>
> The ProcessSession::read will read until the resource claims size, not the
> flowfiles size.
> If the flowfiles size/offset doesnt match the size/offset of its resource
> claim, this could cause problems. One way this could happend is using the
> _ProcessSession::clone(const std::shared_ptr , int64_t
> offset, int64_t size)_
> The behaviour is also dependant on the type of ContentRepository
> e.g.
> using FileSystemContentRepository or VolatileContentRepository
> {code:java}
> auto first_half_clone = session->clone(original_ff, 0,
> original_ff->getSize()/2);
> auto second_half_clone = session->clone(original_ff,
> original_ff->getSize()/2, original_ff->getSize()/2);
> session->read(original_ff, reader); // Reads foobar
> session->read(first_half_clone, reader); // Reads foobar expected foo
> session->read(second_half_clone, reader); // Reads bar
> {code}
> using DatabaseContentRepository
> {code:java}
> auto first_half_clone = session->clone(original_ff, 0,
> original_ff->getSize()/2);
> auto second_half_clone = session->clone(original_ff,
> original_ff->getSize()/2, original_ff->getSize()/2);
> session->read(original_ff, reader); // Reads foobar
> session->read(first_half_clone, reader); // Reads foobar expected foo
> session->read(second_half_clone, reader); // Reads foobar expected bar{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1170: MINIFICPP-1618 Create the ReplaceText processor
szaszm commented on a change in pull request #1170:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1170#discussion_r717382168
##
File path: extensions/standard-processors/processors/ReplaceText.cpp
##
@@ -0,0 +1,339 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "ReplaceText.h"
+
+#include
+#include
+
+#include "core/Resource.h"
+#include "core/TypedValues.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "utils/LineByLineInputOutputStreamCallback.h"
+
+namespace org::apache::nifi::minifi::processors {
+
+const core::Property ReplaceText::EvaluationMode =
core::PropertyBuilder::createProperty("Evaluation Mode")
+->withDescription("Run the 'Replacement Strategy' against each line
separately (Line-by-Line) or "
+ "against the whole input treated as a single string
(Entire Text).")
+->isRequired(true)
+->withDefaultValue(toString(EvaluationModeType::LINE_BY_LINE))
+->withAllowableValues(EvaluationModeType::values())
+->build();
+
+const core::Property ReplaceText::LineByLineEvaluationMode =
core::PropertyBuilder::createProperty("Line-by-Line Evaluation Mode")
+->withDescription("Run the 'Replacement Strategy' against each line
separately (Line-by-Line) for All lines in the FlowFile, "
+ "First Line (Header) only, Last Line (Footer) only, all
Except the First Line (Header) or all Except the Last Line (Footer).")
+->isRequired(false)
+
->withDefaultValue(toString(LineByLineEvaluationModeType::ALL))
+->withAllowableValues(LineByLineEvaluationModeType::values())
+->build();
+
+const core::Property ReplaceText::ReplacementStrategy =
core::PropertyBuilder::createProperty("Replacement Strategy")
+->withDescription("The strategy for how and what to replace within the
FlowFile's text content. "
+ "Substitute Variables replaces ${attribute_name}
placeholders with the corresponding attribute's value "
+ "(if an attribute is not found, the placeholder is kept
as it was).")
+->isRequired(true)
+->withDefaultValue(toString(ReplacementStrategyType::REGEX_REPLACE))
+->withAllowableValues(ReplacementStrategyType::values())
+->build();
+
+const core::Property ReplaceText::SearchValue =
core::PropertyBuilder::createProperty("Search Value")
+->withDescription("The Search Value to search for in the FlowFile content.
"
+ "Only used for 'Literal Replace' and 'Regex Replace'
matching strategies. "
+ "Supports expression language except in Regex Replace
mode.")
+->isRequired(false)
+->supportsExpressionLanguage(true)
+->build();
+
+const core::Property ReplaceText::ReplacementValue =
core::PropertyBuilder::createProperty("Replacement Value")
+->withDescription("The value to insert using the 'Replacement Strategy'. "
+ "Using 'Regex Replace' back-references to Regular
Expression capturing groups are supported: "
+ "$& is the entire matched substring, $1, $2, ... are the
matched capturing groups. Use $$1 for a literal $1. "
+ "Back-references to non-existent capturing groups will
be replaced by empty strings. "
+ "Supports expression language except in Regex Replace
mode.")
+->isRequired(true)
+->supportsExpressionLanguage(true)
+->build();
+
+const core::Relationship ReplaceText::Success("success", "FlowFiles that have
been successfully processed are routed to this relationship. "
+ "This includes both
FlowFiles that had text replaced and those that did not.");
+const core::Relationship ReplaceText::Failure("failure", "FlowFiles that could
not be updated are routed to this relationship.");
+
+ReplaceText::ReplaceText(const std::string& name, const utils::Identifier&
uuid)
+ : core::Processor(name, uuid),
+logger_(logging::LoggerFactory::getLogger()) {
+}
+
+void ReplaceText::initialize() {
+ setSupportedProperties({
+ EvaluationMode,
+ LineByLineEvaluationMode,
+ ReplacementStrategy,
+ SearchValue,
+ ReplacementValue
+
[jira] [Resolved] (MINIFICPP-1649) Replace magic fallthrough strings with attribute
[ https://issues.apache.org/jira/browse/MINIFICPP-1649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gábor Gyimesi resolved MINIFICPP-1649. -- Resolution: Fixed > Replace magic fallthrough strings with attribute > > > Key: MINIFICPP-1649 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1649 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Gábor Gyimesi >Assignee: Gábor Gyimesi >Priority: Trivial > Labels: MiNiFi-CPP-Hygiene > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] adamdebreceni closed pull request #1177: MINIFICPP-1644: ProcessSession::read ignores the size/offset of the
adamdebreceni closed pull request #1177: URL: https://github.com/apache/nifi-minifi-cpp/pull/1177 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni closed pull request #1179: MINIFICPP-1649 Replace magic fallthrough strings with attribute
adamdebreceni closed pull request #1179: URL: https://github.com/apache/nifi-minifi-cpp/pull/1179 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #1170: MINIFICPP-1618 Create the ReplaceText processor
fgerlits commented on a change in pull request #1170:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1170#discussion_r717317315
##
File path: extensions/standard-processors/processors/ReplaceText.cpp
##
@@ -0,0 +1,339 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "ReplaceText.h"
+
+#include
+#include
+
+#include "core/Resource.h"
+#include "core/TypedValues.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "utils/LineByLineInputOutputStreamCallback.h"
+
+namespace org::apache::nifi::minifi::processors {
+
+const core::Property ReplaceText::EvaluationMode =
core::PropertyBuilder::createProperty("Evaluation Mode")
+->withDescription("Run the 'Replacement Strategy' against each line
separately (Line-by-Line) or "
+ "against the whole input treated as a single string
(Entire Text).")
+->isRequired(true)
+->withDefaultValue(toString(EvaluationModeType::LINE_BY_LINE))
+->withAllowableValues(EvaluationModeType::values())
+->build();
+
+const core::Property ReplaceText::LineByLineEvaluationMode =
core::PropertyBuilder::createProperty("Line-by-Line Evaluation Mode")
+->withDescription("Run the 'Replacement Strategy' against each line
separately (Line-by-Line) for All lines in the FlowFile, "
+ "First Line (Header) only, Last Line (Footer) only, all
Except the First Line (Header) or all Except the Last Line (Footer).")
+->isRequired(false)
+
->withDefaultValue(toString(LineByLineEvaluationModeType::ALL))
+->withAllowableValues(LineByLineEvaluationModeType::values())
+->build();
+
+const core::Property ReplaceText::ReplacementStrategy =
core::PropertyBuilder::createProperty("Replacement Strategy")
+->withDescription("The strategy for how and what to replace within the
FlowFile's text content. "
+ "Substitute Variables replaces ${attribute_name}
placeholders with the corresponding attribute's value "
+ "(if an attribute is not found, the placeholder is kept
as it was).")
+->isRequired(true)
+->withDefaultValue(toString(ReplacementStrategyType::REGEX_REPLACE))
+->withAllowableValues(ReplacementStrategyType::values())
+->build();
+
+const core::Property ReplaceText::SearchValue =
core::PropertyBuilder::createProperty("Search Value")
+->withDescription("The Search Value to search for in the FlowFile content.
"
+ "Only used for 'Literal Replace' and 'Regex Replace'
matching strategies. "
+ "Supports expression language except in Regex Replace
mode.")
+->isRequired(false)
+->supportsExpressionLanguage(true)
+->build();
+
+const core::Property ReplaceText::ReplacementValue =
core::PropertyBuilder::createProperty("Replacement Value")
+->withDescription("The value to insert using the 'Replacement Strategy'. "
+ "Using 'Regex Replace' back-references to Regular
Expression capturing groups are supported: "
+ "$& is the entire matched substring, $1, $2, ... are the
matched capturing groups. Use $$1 for a literal $1. "
+ "Back-references to non-existent capturing groups will
be replaced by empty strings. "
+ "Supports expression language except in Regex Replace
mode.")
+->isRequired(true)
+->supportsExpressionLanguage(true)
+->build();
+
+const core::Relationship ReplaceText::Success("success", "FlowFiles that have
been successfully processed are routed to this relationship. "
+ "This includes both
FlowFiles that had text replaced and those that did not.");
+const core::Relationship ReplaceText::Failure("failure", "FlowFiles that could
not be updated are routed to this relationship.");
+
+ReplaceText::ReplaceText(const std::string& name, const utils::Identifier&
uuid)
+ : core::Processor(name, uuid),
+logger_(logging::LoggerFactory::getLogger()) {
+}
+
+void ReplaceText::initialize() {
+ setSupportedProperties({
+ EvaluationMode,
+ LineByLineEvaluationMode,
+ ReplacementStrategy,
+ SearchValue,
+ ReplacementValue
+
