Re: [PR] NIFI-12200 - Remove deprecated module nifi-toolkit-tls [nifi]

2023-10-17 Thread via GitHub


ChrisSamo632 commented on code in PR #7879:
URL: https://github.com/apache/nifi/pull/7879#discussion_r1363113162


##
nifi-registry/nifi-registry-core/nifi-registry-web-api/src/test/resources/keys/README.md:
##
@@ -72,20 +72,10 @@ WD="/tmp/test-keys-$(date +"%Y%m%d-%H%M%S")"
 mkdir "$WD"
 cd "$WD"
 
-# copy existing CA key/cert pair to working directory, rename to default 
tls-toolkit names
+# copy existing CA key/cert pair to working directory, rename to default names
 cp 
/path/to/nifi-registry/nifi-registry-core/nifi-registry-web-api/src/test/resources/keys/ca-key.pem
 ./nifi-key.key
 cp 
/path/to/nifi-registry/nifi-registry-core/nifi-registry-web-api/src/test/resources/keys/ca-cert.pem
 ./nifi-cert.pem
 

Review Comment:
   Sounds reasonable, I couldn't recall whether I'd seen the new although being 
added to the docs already, but such a link seems fair



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] NIFI-12226: Add flag to skip doc generation [nifi-maven]

2023-10-17 Thread via GitHub


mattyb149 commented on PR #35:
URL: https://github.com/apache/nifi-maven/pull/35#issuecomment-1767342297

   +1 on this PR from me, I can merge unless @nandorsoma wants to :)  
@nicoloboschi is this the only feature of NiFi that LangStream uses? For 
LangStream and Pulsar, is it the metadata and/or approach to classloader 
isolation that makes the NAR format useful? If so I definitely think we should 
we refactor this to be a generic "extension packager" or something and publish 
it as such (as @nandorsoma suggested), and we can add NiFi-specific stuff for 
the NiFi plugin and publish that too.
   
   If we want to get fancy we can accept extensions for doc generation (so it's 
not NiFi-specific) and even the supported annotations (to make it even more 
flexible). That's quite a bit of work though, I'd tackle it in my spare time 
but that's pretty sparse these days. Would be happy to review a "Feature 
Proposal" on the 
[wiki](https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals) 
if someone wanted to put up a requirements and/or design document.
   
   For the "different context" to which Soma is referring, stay tuned and keep 
an eye on that Feature Proposals page ;)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Created] (NIFI-12240) Add Python processors that are capable of interacting with vector stores

2023-10-17 Thread Mark Payne (Jira)
Mark Payne created NIFI-12240:
-

 Summary: Add Python processors that are capable of interacting 
with vector stores
 Key: NIFI-12240
 URL: https://issues.apache.org/jira/browse/NIFI-12240
 Project: Apache NiFi
  Issue Type: New Feature
  Components: Extensions
Reporter: Mark Payne
Assignee: Mark Payne
 Fix For: 2.latest


There are many different vector stores these days. We should build processors 
that are capable of ingesting unstructured text, chunking it, and ingesting 
into at least Pinecone and Chroma. We should also have the ability to query the 
vector stores.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Re: [PR] NIFI-12200 - Remove deprecated module nifi-toolkit-tls [nifi]

2023-10-17 Thread via GitHub


greyp9 commented on code in PR #7879:
URL: https://github.com/apache/nifi/pull/7879#discussion_r1362822241


##
nifi-registry/nifi-registry-core/nifi-registry-web-api/src/test/resources/keys/README.md:
##
@@ -72,20 +72,10 @@ WD="/tmp/test-keys-$(date +"%Y%m%d-%H%M%S")"
 mkdir "$WD"
 cd "$WD"
 
-# copy existing CA key/cert pair to working directory, rename to default 
tls-toolkit names
+# copy existing CA key/cert pair to working directory, rename to default names
 cp 
/path/to/nifi-registry/nifi-registry-core/nifi-registry-web-api/src/test/resources/keys/ca-key.pem
 ./nifi-key.key
 cp 
/path/to/nifi-registry/nifi-registry-core/nifi-registry-web-api/src/test/resources/keys/ca-cert.pem
 ./nifi-cert.pem
 

Review Comment:
   We could point them at this walkthrough (link to target documentation?):
   
https://github.com/apache/nifi/blob/main/nifi-docs/src/main/asciidoc/walkthroughs.adoc#securing-nifi-with-mtls
   
   ... which will be homed here:
   
https://nifi.apache.org/docs/nifi-docs/html/walkthroughs.html#securing-nifi-with-mtls
   
   Something like:
   ```
   place new keystores / truststores (for more info, see: 
https://github.com/apache/nifi/blob/main/nifi-docs/src/main/asciidoc/walkthroughs.adoc#securing-nifi-with-mtls)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] NIFI-12200 - Remove deprecated module nifi-toolkit-tls [nifi]

2023-10-17 Thread via GitHub


ChrisSamo632 commented on code in PR #7879:
URL: https://github.com/apache/nifi/pull/7879#discussion_r1362755726


##
nifi-registry/nifi-registry-core/nifi-registry-web-api/src/test/resources/keys/README.md:
##
@@ -72,20 +72,10 @@ WD="/tmp/test-keys-$(date +"%Y%m%d-%H%M%S")"
 mkdir "$WD"
 cd "$WD"
 
-# copy existing CA key/cert pair to working directory, rename to default 
tls-toolkit names
+# copy existing CA key/cert pair to working directory, rename to default names
 cp 
/path/to/nifi-registry/nifi-registry-core/nifi-registry-web-api/src/test/resources/keys/ca-key.pem
 ./nifi-key.key
 cp 
/path/to/nifi-registry/nifi-registry-core/nifi-registry-web-api/src/test/resources/keys/ca-cert.pem
 ./nifi-cert.pem
 

Review Comment:
   Some alternative wording here and elsewhere in the docs would seem sensible, 
e.g. based on the recent [dev mailing list 
thread](https://lists.apache.org/thread/fny5phr1mq2pqo86hqs1lmsnfrqt06sf) 
started by @exceptionfactory
   
   I think the important thing is to strike the right balance between pointing 
administrators at the need for obtaining certificates, but without detailing 
the multitude of possible ways to do so (as people's needs are often different 
for different environments)
   
   Do we note a set of minimum requirements and suggest people look to obtain a 
certificate (in the required format) from a reputable CA, or maybe generate 
self-signed certificates for development deployments?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Updated] (NIFI-12088) UI - Dependent Properties are not shown when depending on newly created Service

2023-10-17 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-12088:

Fix Version/s: 2.0.0

> UI - Dependent Properties are not shown when depending on newly created 
> Service
> ---
>
> Key: NIFI-12088
> URL: https://issues.apache.org/jira/browse/NIFI-12088
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core UI
>Reporter: Mark Bathori
>Assignee: Shane Ardell
>Priority: Major
> Fix For: 2.0.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> When properties depends on a property containing controller services, the 
> dependent fields are not shown if the controller service is newly created.
> E.g.: When using the *ListenHTTP* processor, if the *Record Reader* property 
> is being populated by choosing the ‘{_}Create new service…{_}’ option then 
> the *Record Writer* property which depends on this filed stays invisible. The 
> *Record Writer* property will be visible only after re-opening the window.
> It works properly if I choose a previously created controller service in the 
> *Record Reader* property.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12088) UI - Dependent Properties are not shown when depending on newly created Service

2023-10-17 Thread Shane Ardell (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shane Ardell updated NIFI-12088:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

> UI - Dependent Properties are not shown when depending on newly created 
> Service
> ---
>
> Key: NIFI-12088
> URL: https://issues.apache.org/jira/browse/NIFI-12088
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core UI
>Reporter: Mark Bathori
>Assignee: Shane Ardell
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> When properties depends on a property containing controller services, the 
> dependent fields are not shown if the controller service is newly created.
> E.g.: When using the *ListenHTTP* processor, if the *Record Reader* property 
> is being populated by choosing the ‘{_}Create new service…{_}’ option then 
> the *Record Writer* property which depends on this filed stays invisible. The 
> *Record Writer* property will be visible only after re-opening the window.
> It works properly if I choose a previously created controller service in the 
> *Record Reader* property.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Re: [PR] NIFI-12200 - Remove deprecated module nifi-toolkit-tls [nifi]

2023-10-17 Thread via GitHub


greyp9 commented on code in PR #7879:
URL: https://github.com/apache/nifi/pull/7879#discussion_r1362711361


##
nifi-registry/nifi-registry-core/nifi-registry-web-api/src/test/resources/keys/README.md:
##
@@ -72,20 +72,10 @@ WD="/tmp/test-keys-$(date +"%Y%m%d-%H%M%S")"
 mkdir "$WD"
 cd "$WD"
 
-# copy existing CA key/cert pair to working directory, rename to default 
tls-toolkit names
+# copy existing CA key/cert pair to working directory, rename to default names
 cp 
/path/to/nifi-registry/nifi-registry-core/nifi-registry-web-api/src/test/resources/keys/ca-key.pem
 ./nifi-key.key
 cp 
/path/to/nifi-registry/nifi-registry-core/nifi-registry-web-api/src/test/resources/keys/ca-cert.pem
 ./nifi-cert.pem
 

Review Comment:
   Is some sort of blurb needed here to replace the `tls-toolkit` invocation?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] NIFI-12200 - Remove deprecated module nifi-toolkit-tls [nifi]

2023-10-17 Thread via GitHub


greyp9 commented on code in PR #7879:
URL: https://github.com/apache/nifi/pull/7879#discussion_r1362674146


##
nifi-toolkit/pom.xml:
##
@@ -23,7 +23,6 @@
 nifi-toolkit
 pom
 
-nifi-toolkit-tls

Review Comment:
   Thanks!  Missed those...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] NIFI-12173 - Nifi Registry there's no documentation about OIDC connection [nifi]

2023-10-17 Thread via GitHub


marcelo225 commented on PR #7842:
URL: https://github.com/apache/nifi/pull/7842#issuecomment-1766985402

   @exceptionfactory I'm done with it!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] NIFI-12178 add integration-tests and docker-tests github actions [nifi]

2023-10-17 Thread via GitHub


exceptionfactory commented on PR #7858:
URL: https://github.com/apache/nifi/pull/7858#issuecomment-1766981607

   > @exceptionfactory RE [#7858 
(comment)](https://github.com/apache/nifi/pull/7858#discussion_r1362094730) 
(skipping the `rm -rf` Clean step for the `docker-tests`)
   > 
   > The last run of the Workflow in this PR that didn't include the `rm -rf` 
ran out of disk space, it would appear these removals are still required.
   > 
   > Building without the ZIP wouldn't work - the `dockermaven` builds use the 
ZIP files (I don't want to change that either as that's what the `dockerhub` 
builds do when downloading the build convenience artefacts from the Apache 
Archive servers), so the clean up step:
   > 
   > * cleans the Maven module `target` folders (except for the `assembly` 
modules as that's what we need for the Docker Image build step)
   > 
   > * deletes the un-zipped files from the assembly module `target` folders
   > 
   > 
   > The Docker Image builds each create a new Image (TAR file) containing the 
assembled ZIP, then unzips them before deleting the ZIP file - this uses a lot 
of disk space
   
   Thanks for the additional background @ChrisSamo632, that makes sense now, it 
sounds like the current approach is the best option.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] NIFI-12178 add integration-tests and docker-tests github actions [nifi]

2023-10-17 Thread via GitHub


ChrisSamo632 commented on PR #7858:
URL: https://github.com/apache/nifi/pull/7858#issuecomment-1766976777

   @exceptionfactory RE 
https://github.com/apache/nifi/pull/7858#discussion_r1362094730 (skipping the 
`rm -rf` Clean step for the `docker-tests`)
   
   The last run of the Workflow in this PR that didn't include the `rm -rf` ran 
out of disk space, it would appear these removals are still required.
   
   Building without the ZIP wouldn't work - the `dockermaven` builds use the 
ZIP files (I don't want to change that either as that's what the `dockerhub` 
builds do when downloading the build convenience artefacts from the Apache 
Archive servers), so the clean up step:
   - cleans the Maven module `target` folders (except for the `assembly` 
modules as that's what we need for the Docker Image build step)
   - deletes the un-zipped files from the assembly module `target` folders
   
   The Docker Image builds each create a new Image (TAR file) containing the 
assembled ZIP, then unzips them before deleting the ZIP file - this uses a lot 
of disk space


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] NIFI-12178 add integration-tests and docker-tests github actions [nifi]

2023-10-17 Thread via GitHub


ChrisSamo632 commented on code in PR #7858:
URL: https://github.com/apache/nifi/pull/7858#discussion_r1362608944


##
.github/workflows/docker-tests.yml:
##
@@ -0,0 +1,189 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+name: docker-tests
+
+on:
+  # Run every day at 02:00
+  schedule:
+- cron: "0 2 * * *"
+  push:
+paths:
+  - '.github/workflows/docker-tests.yml'
+  - 'nifi-assembly/**'
+  - 'nifi-docker/**'
+  - 'nifi-registry/nifi-registry-assembly/**'
+  - 'nifi-registry/nifi-registry-docker-maven/**'
+  - 'nifi-toolkit/nifi-toolkit-assembly/**'
+  - 'nifi-toolkit/nifi-toolkit-cli/**'
+  - 'nifi-toolkit/nifi-toolkit-encrypt-config/**'
+  - 'nifi-toolkit/nifi-toolkit-tls/**'
+  - 'minifi/minifi-assembly/**'
+  - 'minifi/minifi-docker/**'
+  - 'minifi/minifi-c2/minifi-c2-assembly/**'
+  - 'minifi/minifi-c2/minifi-c2-docker/**'
+  pull_request:
+paths:
+  - '.github/workflows/docker-tests.yml'
+  - 'nifi-assembly/**'
+  - 'nifi-docker/**'
+  - 'nifi-registry/nifi-registry-assembly/**'
+  - 'nifi-registry/nifi-registry-docker-maven/**'
+  - 'nifi-toolkit/nifi-toolkit-assembly/**'
+  - 'nifi-toolkit/nifi-toolkit-cli/**'
+  - 'nifi-toolkit/nifi-toolkit-encrypt-config/**'
+  - 'nifi-toolkit/nifi-toolkit-tls/**'
+  - 'minifi/minifi-assembly/**'
+  - 'minifi/minifi-docker/**'
+  - 'minifi/minifi-c2/minifi-c2-assembly/**'
+  - 'minifi/minifi-c2/minifi-c2-docker/**'
+
+env:
+  DEFAULT_MAVEN_OPTS: >-
+-Xmx4g
+-XX:ReservedCodeCacheSize=1g
+-XX:+UseG1GC
+-Dorg.slf4j.simpleLogger.defaultLogLevel=WARN
+-Daether.connector.http.retryHandler.count=5
+-Daether.connector.http.connectionMaxTtl=30
+  MAVEN_COMMAND: >-
+./mvnw
+-V
+-nsu
+-ntp
+-ff
+
+# build assemblies (including binaries) for use in Docker Image builds
+  MAVEN_BUILD_ARGUMENTS: >-
+-am
+-D skipTests
+-T 2.0C
+install
+  MAVEN_BUILD_PROJECTS: >-
+-pl nifi-assembly
+-pl nifi-registry/nifi-registry-assembly
+-pl nifi-toolkit/nifi-toolkit-assembly
+-pl minifi/minifi-assembly
+-pl minifi/minifi-c2/minifi-c2-assembly
+
+  MAVEN_CLEAN_ARGUMENTS: >-
+-T 2.0C
+clean
+  MAVEN_CLEAN_PROJECTS: >-
+-pl -nifi-assembly
+-pl -nifi-registry/nifi-registry-assembly
+-pl -nifi-toolkit/nifi-toolkit-assembly
+-pl -nifi-toolkit/nifi-toolkit-cli
+-pl -nifi-toolkit/nifi-toolkit-encrypt-config
+-pl -nifi-toolkit/nifi-toolkit-tls
+-pl -minifi/minifi-assembly
+-pl -minifi/minifi-c2/minifi-c2-assembly
+
+# build and test Docker Images
+  MAVEN_DOCKER_ARGUMENTS: >-
+verify
+-P docker
+  MAVEN_DOCKER_PROJECTS: >-
+-pl minifi/minifi-docker
+-pl minifi/minifi-c2/minifi-c2-docker
+-pl nifi-docker/dockermaven
+-pl nifi-registry/nifi-registry-docker-maven/dockermaven
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+permissions:
+  contents: read
+
+jobs:
+  build_and_test:
+strategy:
+  fail-fast: false
+  matrix:
+# TODO: macos-latest doesn't come with Docker and Colima seems to have 
problems with the nifi-docker integration_test scripts - see 
NIFI-12177/NIFI-12191

Review Comment:
   Yeah that's fine now that I've raised the Jira tickets (I've also, I think, 
got a fix for these TODOs, but don't want to extend this existing PR further), 
will remove the TODOs



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] NIFI-12178 add integration-tests and docker-tests github actions [nifi]

2023-10-17 Thread via GitHub


ChrisSamo632 commented on code in PR #7858:
URL: https://github.com/apache/nifi/pull/7858#discussion_r1362608175


##
.github/workflows/ci-workflow.yml:
##
@@ -75,7 +76,7 @@ env:
 -pl -nifi-toolkit/nifi-toolkit-assembly
 
 concurrency:
-  group: ${{ github.ref }}
+  group: ${{ github.workflow }}-${{ github.ref }}

Review Comment:
   Adding a `concurrency` entry for the new Workflows (and existing 
`system-tests`) seemed sensible to avoid running duplicate Workflows for PRs 
that trigger them (the `cron` schedule, of course, should not result in 
duplicate runs)
   
   Having a `group` of just `${{ github.ref }}` means that the Workflows cancel 
each other because they're all running in the same `group`. Adding the `${{ 
github.workflow }}` name prefix means that a Workflow can only cancel an 
existing run of itself. There's not really a *need* to add the prefix to the 
`ci-workflow` Workflow definition, as it could be left as the only Workflow 
without the prefix, meaning it would still be in a `group` of its own. But as I 
was upting this Workflow definition for other reasons, then making this 
consistent with all other NiFi Workflows seemed sensible for the sake of future 
maintenance.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] NIFI-12239 - Add OS/Java details at startup in bootstrap log [nifi]

2023-10-17 Thread via GitHub


pvillard31 commented on PR #7891:
URL: https://github.com/apache/nifi/pull/7891#issuecomment-1766798572

   Updated with your suggestion


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] NIFI-12239 - Add OS/Java details at startup in bootstrap log [nifi]

2023-10-17 Thread via GitHub


pvillard31 commented on PR #7891:
URL: https://github.com/apache/nifi/pull/7891#issuecomment-1766764306

   New format:
   
   
   2023-10-17 18:24:24,923 INFO [main] org.apache.nifi.bootstrap.RunNiFi 
{totalPhysicalMemoryMB=16384, xmx=1g, cores=16, javaVersion=21, 
maxOpenFileDescriptors=10240, xms=1g}
   
   
   Switched to ``Runtime`` for available processors. Open to suggestions for 
the two other values unless we believe we should just drop those values because 
not important enough to justify the use of ``MXBean``


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Updated] (NIFI-12238) SplitText trims text ending character when max fragment size is specified and multiple endlines are present

2023-10-17 Thread Jira


 [ 
https://issues.apache.org/jira/browse/NIFI-12238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gábor Gyimesi updated NIFI-12238:
-
Status: Patch Available  (was: In Progress)

https://github.com/apache/nifi/pull/7892

> SplitText trims text ending character when max fragment size is specified and 
> multiple endlines are present
> ---
>
> Key: NIFI-12238
> URL: https://issues.apache.org/jira/browse/NIFI-12238
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Gábor Gyimesi
>Assignee: Gábor Gyimesi
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> There seems to be an issue with SplitText processor when max fragment size is 
> specified for fragment limit and the ending of the fragment contains multiple 
> endline characters to be trimmed. In this case not only the endline 
> characters, but the last character of the text is also trimmed.
> Test case:
> {code:java}
> @Test
> public void testMaxFragmentSizeWithTrimmedEndlines() {
> final TestRunner splitRunner = TestRunners.newTestRunner(new SplitText());
> splitRunner.setProperty(SplitText.HEADER_LINE_COUNT, "2");
> splitRunner.setProperty(SplitText.LINE_SPLIT_COUNT, "0");
> splitRunner.setProperty(SplitText.FRAGMENT_MAX_SIZE, "30 B");
> splitRunner.setProperty(SplitText.REMOVE_TRAILING_NEWLINES, "true");
> splitRunner.enqueue("header1\nheader2\nline1 longer than 
> limit\nline2\nline3\n\n\n\n\n");
> splitRunner.run();
> splitRunner.assertTransferCount(SplitText.REL_SPLITS, 3);
> splitRunner.assertTransferCount(SplitText.REL_ORIGINAL, 1);
> splitRunner.assertTransferCount(SplitText.REL_FAILURE, 0);
> final List splits = 
> splitRunner.getFlowFilesForRelationship(SplitText.REL_SPLITS);
> splits.get(0).assertContentEquals("header1\nheader2\nline1 longer than 
> limit");
> splits.get(1).assertContentEquals("header1\nheader2\nline2\nline3");
> splits.get(2).assertContentEquals("header1\nheader2");
> }
> {code}
> Result:
> {code:java}
> expected:  header2
> line2
> line3> but was:  header2
> line2
> line>
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Re: [PR] NIFI-12239 - Add OS/Java details at startup in bootstrap log [nifi]

2023-10-17 Thread via GitHub


pvillard31 commented on PR #7891:
URL: https://github.com/apache/nifi/pull/7891#issuecomment-1766744739

   Yeah JSON was kind of a default choice to ease automatic parsing of this 
data but a KV approach is fine with me too, will update


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] NIFI-12239 - Add OS/Java details at startup in bootstrap log [nifi]

2023-10-17 Thread via GitHub


exceptionfactory commented on code in PR #7891:
URL: https://github.com/apache/nifi/pull/7891#discussion_r1362384318


##
nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java:
##
@@ -1165,13 +1171,22 @@ public void start(final boolean monitor) throws 
IOException {
 
 nifiPropsFilename = nifiPropsFilename.trim();
 
+String xmx = null;
+String xms = null;
+
 final List javaAdditionalArgs = new ArrayList<>();
 for (final Map.Entry entry : props.entrySet()) {
 final String key = entry.getKey();
 final String value = entry.getValue();
 
 if (key.startsWith("java.arg")) {
 javaAdditionalArgs.add(value);
+if(value.toLowerCase().startsWith("-xms")) {
+xms = StringUtils.substringAfter(value.toLowerCase(), 
"-xms");
+}
+if(value.toLowerCase().startsWith("-xmx")) {
+xmx = StringUtils.substringAfter(value.toLowerCase(), 
"-xmx");
+}

Review Comment:
   That's a good point, thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] NIFI-12239 - Add OS/Java details at startup in bootstrap log [nifi]

2023-10-17 Thread via GitHub


pvillard31 commented on code in PR #7891:
URL: https://github.com/apache/nifi/pull/7891#discussion_r1362383549


##
nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java:
##
@@ -1392,6 +1407,45 @@ public boolean accept(final File dir, final String 
filename) {
 }
 }
 
+private String printBasicOSJavaDetails(final String xms, final String xmx) 
{
+ObjectMapper mapper = new ObjectMapper();
+ObjectNode details = mapper.createObjectNode();
+
+// java version
+details.put("javaVersion", System.getProperty("java.version"));
+
+// num cores
+final OperatingSystemMXBean os = 
ManagementFactory.getOperatingSystemMXBean();
+details.put("cores", os.getAvailableProcessors());
+
+// max file descriptor count + total physical memory
+try {
+final OperatingSystemMXBean osStats = 
ManagementFactory.getOperatingSystemMXBean();
+final Class unixOsMxBeanClass = 
Class.forName("com.sun.management.UnixOperatingSystemMXBean");
+if (unixOsMxBeanClass.isAssignableFrom(osStats.getClass())) {
+final Method totalPhysicalMemory = 
unixOsMxBeanClass.getMethod("getTotalPhysicalMemorySize");

Review Comment:
   I believe that some users tend to set a very high heap, not leaving enough 
room for off-heap. Having the total memory available on the host may be useful. 
If the information is not available, we would see a null value.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[PR] NIFI-12238 Fix SplitText endline trimming with max fragment size [nifi]

2023-10-17 Thread via GitHub


lordgamez opened a new pull request, #7892:
URL: https://github.com/apache/nifi/pull/7892

   
   
   
   
   
   
   
   
   
   
   
   
   
   # Summary
   
   [NIFI-12238](https://issues.apache.org/jira/browse/NIFI-12238)
   
   # Tracking
   
   Please complete the following tracking steps prior to pull request creation.
   
   ### Issue Tracking
   
   - [ ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue 
created
   
   ### Pull Request Tracking
   
   - [ ] Pull Request title starts with Apache NiFi Jira issue number, such as 
`NIFI-0`
   - [ ] Pull Request commit message starts with Apache NiFi Jira issue number, 
as such `NIFI-0`
   
   ### Pull Request Formatting
   
   - [ ] Pull Request based on current revision of the `main` branch
   - [ ] Pull Request refers to a feature branch with one commit containing 
changes
   
   # Verification
   
   Please indicate the verification steps performed prior to pull request 
creation.
   
   ### Build
   
   - [ ] Build completed using `mvn clean install -P contrib-check`
 - [ ] JDK 21
   
   ### Licensing
   
   - [ ] New dependencies are compatible with the [Apache License 
2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License 
Policy](https://www.apache.org/legal/resolved.html)
   - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` 
files
   
   ### Documentation
   
   - [ ] Documentation formatting appears as expected in rendered files
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] NIFI-12239 - Add OS/Java details at startup in bootstrap log [nifi]

2023-10-17 Thread via GitHub


pvillard31 commented on code in PR #7891:
URL: https://github.com/apache/nifi/pull/7891#discussion_r1362380042


##
nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java:
##
@@ -1165,13 +1171,22 @@ public void start(final boolean monitor) throws 
IOException {
 
 nifiPropsFilename = nifiPropsFilename.trim();
 
+String xmx = null;
+String xms = null;
+
 final List javaAdditionalArgs = new ArrayList<>();
 for (final Map.Entry entry : props.entrySet()) {
 final String key = entry.getKey();
 final String value = entry.getValue();
 
 if (key.startsWith("java.arg")) {
 javaAdditionalArgs.add(value);
+if(value.toLowerCase().startsWith("-xms")) {
+xms = StringUtils.substringAfter(value.toLowerCase(), 
"-xms");
+}
+if(value.toLowerCase().startsWith("-xmx")) {
+xmx = StringUtils.substringAfter(value.toLowerCase(), 
"-xmx");
+}

Review Comment:
   That would be the value for the bootstrap process (48MB), not for the NiFi 
JVM



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] NIFI-12239 - Add OS/Java details at startup in bootstrap log [nifi]

2023-10-17 Thread via GitHub


exceptionfactory commented on code in PR #7891:
URL: https://github.com/apache/nifi/pull/7891#discussion_r1362369758


##
nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java:
##
@@ -1165,13 +1171,22 @@ public void start(final boolean monitor) throws 
IOException {
 
 nifiPropsFilename = nifiPropsFilename.trim();
 
+String xmx = null;
+String xms = null;
+
 final List javaAdditionalArgs = new ArrayList<>();
 for (final Map.Entry entry : props.entrySet()) {
 final String key = entry.getKey();
 final String value = entry.getValue();
 
 if (key.startsWith("java.arg")) {
 javaAdditionalArgs.add(value);
+if(value.toLowerCase().startsWith("-xms")) {
+xms = StringUtils.substringAfter(value.toLowerCase(), 
"-xms");
+}
+if(value.toLowerCase().startsWith("-xmx")) {
+xmx = StringUtils.substringAfter(value.toLowerCase(), 
"-xmx");
+}

Review Comment:
   Instead of parsing `Xmx`, what about using 
[Runtime.maxMemory()](https://docs.oracle.com/javase/8/docs/api/java/lang/Runtime.html#maxMemory--)?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] NIFI-12239 - Add OS/Java details at startup in bootstrap log [nifi]

2023-10-17 Thread via GitHub


exceptionfactory commented on code in PR #7891:
URL: https://github.com/apache/nifi/pull/7891#discussion_r136239


##
nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java:
##
@@ -1392,6 +1407,45 @@ public boolean accept(final File dir, final String 
filename) {
 }
 }
 
+private String printBasicOSJavaDetails(final String xms, final String xmx) 
{
+ObjectMapper mapper = new ObjectMapper();
+ObjectNode details = mapper.createObjectNode();
+
+// java version
+details.put("javaVersion", System.getProperty("java.version"));
+
+// num cores
+final OperatingSystemMXBean os = 
ManagementFactory.getOperatingSystemMXBean();
+details.put("cores", os.getAvailableProcessors());
+
+// max file descriptor count + total physical memory
+try {
+final OperatingSystemMXBean osStats = 
ManagementFactory.getOperatingSystemMXBean();
+final Class unixOsMxBeanClass = 
Class.forName("com.sun.management.UnixOperatingSystemMXBean");
+if (unixOsMxBeanClass.isAssignableFrom(osStats.getClass())) {
+final Method totalPhysicalMemory = 
unixOsMxBeanClass.getMethod("getTotalPhysicalMemorySize");

Review Comment:
   In this case, is the total physical memory useful? The `Runtime` memory 
methods provide heap information, which is actually addressable for the 
process, versus the physical memory, which may or may not be available to the 
JVM.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Updated] (NIFI-12239) Print basic OS/Java details in bootstrap log file

2023-10-17 Thread Pierre Villard (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard updated NIFI-12239:
--
Fix Version/s: 1.latest
   2.latest

> Print basic OS/Java details in bootstrap log file
> -
>
> Key: NIFI-12239
> URL: https://issues.apache.org/jira/browse/NIFI-12239
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
> Fix For: 1.latest, 2.latest
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The goal here is to add a log line in the bootstrap.log file when NiFi starts 
> that is printing some basic OS/Java details. On busy clusters, nifi-app.log 
> may be rolling over quickly and/or contain huge amount of data. Adding this 
> data in the bootstrap log file makes it easy to retrieve.
> The associated PR will add a log line as below:
> {code:java}
> 2023-10-17 17:26:33,501 INFO [main] org.apache.nifi.bootstrap.RunNiFi 
> {"javaVersion":"21","cores":16,"totalPhysicalMemoryMB":16384,"maxOpenFileDescriptors":10240,"xms":"1g","xmx":"1g"}
>  {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[PR] NIFI-12239 - Add OS/Java details at startup in bootstrap log [nifi]

2023-10-17 Thread via GitHub


pvillard31 opened a new pull request, #7891:
URL: https://github.com/apache/nifi/pull/7891

   # Summary
   
   [NIFI-0](https://issues.apache.org/jira/browse/NIFI-12239) - Add OS/Java 
details at startup in bootstrap log
   
   Example of the added log:
   
   
   2023-10-17 17:26:33,501 INFO [main] org.apache.nifi.bootstrap.RunNiFi 
{"javaVersion":"21","cores":16,"totalPhysicalMemoryMB":16384,"maxOpenFileDescriptors":10240,"xms":"1g","xmx":"1g"}
   
   
   # Tracking
   
   Please complete the following tracking steps prior to pull request creation.
   
   ### Issue Tracking
   
   - [ ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue 
created
   
   ### Pull Request Tracking
   
   - [ ] Pull Request title starts with Apache NiFi Jira issue number, such as 
`NIFI-0`
   - [ ] Pull Request commit message starts with Apache NiFi Jira issue number, 
as such `NIFI-0`
   
   ### Pull Request Formatting
   
   - [ ] Pull Request based on current revision of the `main` branch
   - [ ] Pull Request refers to a feature branch with one commit containing 
changes
   
   # Verification
   
   Please indicate the verification steps performed prior to pull request 
creation.
   
   ### Build
   
   - [ ] Build completed using `mvn clean install -P contrib-check`
 - [ ] JDK 21
   
   ### Licensing
   
   - [ ] New dependencies are compatible with the [Apache License 
2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License 
Policy](https://www.apache.org/legal/resolved.html)
   - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` 
files
   
   ### Documentation
   
   - [ ] Documentation formatting appears as expected in rendered files
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Updated] (NIFI-12239) Print basic OS/Java details in bootstrap log file

2023-10-17 Thread Pierre Villard (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard updated NIFI-12239:
--
Status: Patch Available  (was: Open)

> Print basic OS/Java details in bootstrap log file
> -
>
> Key: NIFI-12239
> URL: https://issues.apache.org/jira/browse/NIFI-12239
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
>
> The goal here is to add a log line in the bootstrap.log file when NiFi starts 
> that is printing some basic OS/Java details. On busy clusters, nifi-app.log 
> may be rolling over quickly and/or contain huge amount of data. Adding this 
> data in the bootstrap log file makes it easy to retrieve.
> The associated PR will add a log line as below:
> {code:java}
> 2023-10-17 17:26:33,501 INFO [main] org.apache.nifi.bootstrap.RunNiFi 
> {"javaVersion":"21","cores":16,"totalPhysicalMemoryMB":16384,"maxOpenFileDescriptors":10240,"xms":"1g","xmx":"1g"}
>  {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (NIFI-12239) Print basic OS/Java details in bootstrap log file

2023-10-17 Thread Pierre Villard (Jira)
Pierre Villard created NIFI-12239:
-

 Summary: Print basic OS/Java details in bootstrap log file
 Key: NIFI-12239
 URL: https://issues.apache.org/jira/browse/NIFI-12239
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Core Framework
Reporter: Pierre Villard
Assignee: Pierre Villard


The goal here is to add a log line in the bootstrap.log file when NiFi starts 
that is printing some basic OS/Java details. On busy clusters, nifi-app.log may 
be rolling over quickly and/or contain huge amount of data. Adding this data in 
the bootstrap log file makes it easy to retrieve.

The associated PR will add a log line as below:
{code:java}
2023-10-17 17:26:33,501 INFO [main] org.apache.nifi.bootstrap.RunNiFi 
{"javaVersion":"21","cores":16,"totalPhysicalMemoryMB":16384,"maxOpenFileDescriptors":10240,"xms":"1g","xmx":"1g"}
 {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Re: [PR] NIFI-5137 - Fixed the path to controller service grid item's state [nifi]

2023-10-17 Thread via GitHub


NissimShiman commented on code in PR #7833:
URL: https://github.com/apache/nifi/pull/7833#discussion_r1362277274


##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/js/nf/canvas/nf-controller-services.js:
##
@@ -1090,7 +1090,7 @@
 } else if (target.hasClass('delete-controller-service')) {
 nfControllerService.promptToDeleteController(serviceTable, 
controllerServiceEntity);
 } else if (target.hasClass('view-state-controller-service')) {
-nfComponentState.showState(controllerServiceEntity, 
controllerServiceEntity.state === 'DISABLED');
+nfComponentState.showState(controllerServiceEntity, 
controllerServiceEntity.component.state === 'DISABLED');

Review Comment:
   @annanys23 I tried it and it works.  Very nice!
   
   I was thinking maybe there should be a check for activeThreadCount as well.  
We do this for 
[reportingTasks](https://github.com/apache/nifi/blob/rel/nifi-1.23.2/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/js/nf/canvas/nf-settings.js#L1793-L1794)
   
   So we would have something like:
   ```
   var canClear = controllerServiceEntity.status.runStatus === 'DISABLED' && 
(nfCommon.isUndefinedOrNull(controllerServiceEntity.status.activeThreadCount) 
|| controllerServiceEntity.status.activeThreadCount === 0);
   nfComponentState.showState(controllerServiceEntity, canClear);
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] MINIFICPP-1797 Python bootstrap (part 1) [nifi-minifi-cpp]

2023-10-17 Thread via GitHub


szaszm commented on code in PR #1681:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1681#discussion_r1362202851


##
CMakeLists.txt:
##
@@ -269,13 +260,12 @@ add_library(ut INTERFACE)
 target_include_directories(ut SYSTEM INTERFACE 
"${CMAKE_CURRENT_SOURCE_DIR}/thirdparty/ut")
 
 # cURL
-if(NOT DISABLE_CURL)
+if(ENABLE_CURL)

Review Comment:
   It doesn't strictly belong here, but I wouldn't be against hardcoding these 
to true. The usability of minifi is very limited without curl and openssl.



##
.gitignore:
##
@@ -48,6 +48,7 @@ cmake-build-debug
 *flowfile_checkpoint
 build
 /*build*
+!win_build_vs.bat

Review Comment:
   Is there another rule that matches `win_build_vs.bat` that makes this 
exclusion necessary?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Created] (NIFI-12238) SplitText trims text ending character when max fragment size is specified and multiple endlines are present

2023-10-17 Thread Jira
Gábor Gyimesi created NIFI-12238:


 Summary: SplitText trims text ending character when max fragment 
size is specified and multiple endlines are present
 Key: NIFI-12238
 URL: https://issues.apache.org/jira/browse/NIFI-12238
 Project: Apache NiFi
  Issue Type: Bug
Reporter: Gábor Gyimesi
Assignee: Gábor Gyimesi


There seems to be an issue with SplitText processor when max fragment size is 
specified for fragment limit and the ending of the fragment contains multiple 
endline characters to be trimmed. In this case not only the endline characters, 
but the last character of the text is also trimmed.

Test case:
{code:java}
@Test
public void testMaxFragmentSizeWithTrimmedEndlines() {
final TestRunner splitRunner = TestRunners.newTestRunner(new SplitText());
splitRunner.setProperty(SplitText.HEADER_LINE_COUNT, "2");
splitRunner.setProperty(SplitText.LINE_SPLIT_COUNT, "0");
splitRunner.setProperty(SplitText.FRAGMENT_MAX_SIZE, "30 B");
splitRunner.setProperty(SplitText.REMOVE_TRAILING_NEWLINES, "true");

splitRunner.enqueue("header1\nheader2\nline1 longer than 
limit\nline2\nline3\n\n\n\n\n");

splitRunner.run();
splitRunner.assertTransferCount(SplitText.REL_SPLITS, 3);
splitRunner.assertTransferCount(SplitText.REL_ORIGINAL, 1);
splitRunner.assertTransferCount(SplitText.REL_FAILURE, 0);

final List splits = 
splitRunner.getFlowFilesForRelationship(SplitText.REL_SPLITS);
splits.get(0).assertContentEquals("header1\nheader2\nline1 longer than 
limit");
splits.get(1).assertContentEquals("header1\nheader2\nline2\nline3");
splits.get(2).assertContentEquals("header1\nheader2");
}
{code}

Result:

{code:java}
expected:  but was: 
{code}




--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12165) JoltTransformJSON Processor - Vanishing properties

2023-10-17 Thread Daniel Stieglitz (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Stieglitz updated NIFI-12165:

Status: Patch Available  (was: Open)

> JoltTransformJSON Processor - Vanishing properties
> --
>
> Key: NIFI-12165
> URL: https://issues.apache.org/jira/browse/NIFI-12165
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core UI
>Affects Versions: 1.21.0
> Environment: Docker
>Reporter: Benji Benning
>Assignee: Daniel Stieglitz
>Priority: Minor
> Attachments: JoltTransformJSON.mp4
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> When selecting any Property to edit (then either accepting or canceling the 
> change) the "Custom Transformation Class Name" and the "Custom Module 
> Directory" properties vanish.
> Note that the change does seem to get applied.
> See attached video



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Re: [PR] NIFI-12178 add integration-tests and docker-tests github actions [nifi]

2023-10-17 Thread via GitHub


exceptionfactory commented on code in PR #7858:
URL: https://github.com/apache/nifi/pull/7858#discussion_r1362093065


##
.github/workflows/docker-tests.yml:
##
@@ -0,0 +1,189 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+name: docker-tests
+
+on:
+  # Run every day at 02:00
+  schedule:
+- cron: "0 2 * * *"
+  push:
+paths:
+  - '.github/workflows/docker-tests.yml'
+  - 'nifi-assembly/**'
+  - 'nifi-docker/**'
+  - 'nifi-registry/nifi-registry-assembly/**'
+  - 'nifi-registry/nifi-registry-docker-maven/**'
+  - 'nifi-toolkit/nifi-toolkit-assembly/**'
+  - 'nifi-toolkit/nifi-toolkit-cli/**'
+  - 'nifi-toolkit/nifi-toolkit-encrypt-config/**'
+  - 'nifi-toolkit/nifi-toolkit-tls/**'
+  - 'minifi/minifi-assembly/**'
+  - 'minifi/minifi-docker/**'
+  - 'minifi/minifi-c2/minifi-c2-assembly/**'
+  - 'minifi/minifi-c2/minifi-c2-docker/**'
+  pull_request:
+paths:
+  - '.github/workflows/docker-tests.yml'
+  - 'nifi-assembly/**'
+  - 'nifi-docker/**'
+  - 'nifi-registry/nifi-registry-assembly/**'
+  - 'nifi-registry/nifi-registry-docker-maven/**'
+  - 'nifi-toolkit/nifi-toolkit-assembly/**'
+  - 'nifi-toolkit/nifi-toolkit-cli/**'
+  - 'nifi-toolkit/nifi-toolkit-encrypt-config/**'
+  - 'nifi-toolkit/nifi-toolkit-tls/**'
+  - 'minifi/minifi-assembly/**'
+  - 'minifi/minifi-docker/**'
+  - 'minifi/minifi-c2/minifi-c2-assembly/**'
+  - 'minifi/minifi-c2/minifi-c2-docker/**'
+
+env:
+  DEFAULT_MAVEN_OPTS: >-
+-Xmx4g
+-XX:ReservedCodeCacheSize=1g
+-XX:+UseG1GC
+-Dorg.slf4j.simpleLogger.defaultLogLevel=WARN
+-Daether.connector.http.retryHandler.count=5
+-Daether.connector.http.connectionMaxTtl=30
+  MAVEN_COMMAND: >-
+./mvnw
+-V
+-nsu
+-ntp
+-ff
+
+# build assemblies (including binaries) for use in Docker Image builds
+  MAVEN_BUILD_ARGUMENTS: >-
+-am
+-D skipTests
+-T 2.0C
+install
+  MAVEN_BUILD_PROJECTS: >-
+-pl nifi-assembly
+-pl nifi-registry/nifi-registry-assembly
+-pl nifi-toolkit/nifi-toolkit-assembly
+-pl minifi/minifi-assembly
+-pl minifi/minifi-c2/minifi-c2-assembly
+
+  MAVEN_CLEAN_ARGUMENTS: >-
+-T 2.0C
+clean
+  MAVEN_CLEAN_PROJECTS: >-
+-pl -nifi-assembly
+-pl -nifi-registry/nifi-registry-assembly
+-pl -nifi-toolkit/nifi-toolkit-assembly
+-pl -nifi-toolkit/nifi-toolkit-cli
+-pl -nifi-toolkit/nifi-toolkit-encrypt-config
+-pl -nifi-toolkit/nifi-toolkit-tls
+-pl -minifi/minifi-assembly
+-pl -minifi/minifi-c2/minifi-c2-assembly
+
+# build and test Docker Images
+  MAVEN_DOCKER_ARGUMENTS: >-
+verify
+-P docker
+  MAVEN_DOCKER_PROJECTS: >-
+-pl minifi/minifi-docker
+-pl minifi/minifi-c2/minifi-c2-docker
+-pl nifi-docker/dockermaven
+-pl nifi-registry/nifi-registry-docker-maven/dockermaven
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+permissions:
+  contents: read
+
+jobs:
+  build_and_test:
+strategy:
+  fail-fast: false
+  matrix:
+# TODO: macos-latest doesn't come with Docker and Colima seems to have 
problems with the nifi-docker integration_test scripts - see 
NIFI-12177/NIFI-12191
+os: [ ubuntu-latest ]
+version: [ 21 ]
+timeout-minutes: 120
+runs-on: ${{ matrix.os }}
+name: ${{ matrix.os }} Java ${{ matrix.version }}
+env:
+  # default to Amazon Corretto JDK
+  JAVA_DISTRIBUTION: corretto

Review Comment:
   In this case, it seems much simpler to use `Zulu` for all builds. The JDK 
version is less important for this workflow because the focus is running Docker 
containers.



##
.github/workflows/docker-tests.yml:
##
@@ -0,0 +1,189 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+# 

Re: [PR] NIFI-12226: Add flag to skip doc generation [nifi-maven]

2023-10-17 Thread via GitHub


nicoloboschi commented on PR #35:
URL: https://github.com/apache/nifi-maven/pull/35#issuecomment-1766328766

   @nandorsoma It's widely used also in Apache Pulsar for distributing 
connectors that run over Pulsar IO.  
   https://pulsar.apache.org/docs/next/functions-package-java/#package-as-nar
   
   The documentation part is useless in Pulsar too so this change improves the 
build time there too
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] NIFI-11958 Azure Data Explorer as sink [nifi]

2023-10-17 Thread via GitHub


kustonaut commented on PR #7624:
URL: https://github.com/apache/nifi/pull/7624#issuecomment-1766312186

   @joewitt & @exceptionfactory - Thank you so much for your review and 
comments. @tanmaya-panda1 has fixed the issues. Could you please take a look 
when feasible?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] MINIFICPP-2248 Refactor string::join_pack [nifi-minifi-cpp]

2023-10-17 Thread via GitHub


szaszm commented on code in PR #1680:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1680#discussion_r1362018977


##
libminifi/test/unit/StringUtilsTests.cpp:
##
@@ -469,15 +476,16 @@ TEST_CASE("test string::testJoinPackWstring", "[test 
join_pack wstring]") {
   == L"rvalue c string, c string, rval std::string, std::string, 
std::string_view, char array");
 }
 
-/* doesn't and shouldn't compile
-TEST_CASE("test string::testJoinPackNegative", "[test join_pack negative]") {
-  std::wstring stdstr = L"std::string";
-  const wchar_t* cstr = L"c string";
-  const wchar_t carr[] = L"char array";
-  REQUIRE(string::join_pack("rvalue c string, ", cstr, std::string{ ", rval 
std::string, " }, stdstr, ", ", carr)
-  == "rvalue c string, c string, rval std::string, std::string, 
char array");
+namespace detail {
+template
+concept join_pack_works_with_args = requires(Strs... strs) {
+  string::join_pack(strs...);
+};
+}  // namespace detail
+
+TEST_CASE("test string::join_pack can't combine different char types", "[test 
join_pack negative][different char types]") {
+  STATIC_REQUIRE(!detail::join_pack_works_with_args);  // 
NOLINT: testing C array

Review Comment:
   changed these in 
[e26bfc4](https://github.com/apache/nifi-minifi-cpp/pull/1680/commits/e26bfc40fc7da54375ea20168c502514a8fec259)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Updated] (MINIFICPP-62) Create SplitText processor

2023-10-17 Thread Jira


 [ 
https://issues.apache.org/jira/browse/MINIFICPP-62?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gábor Gyimesi updated MINIFICPP-62:
---
Status: Patch Available  (was: Open)

https://github.com/apache/nifi-minifi-cpp/pull/1682

> Create SplitText processor
> --
>
> Key: MINIFICPP-62
> URL: https://issues.apache.org/jira/browse/MINIFICPP-62
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Improvement
>Reporter: Jeremy Dyer
>Assignee: Gábor Gyimesi
>Priority: Minor
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Create a SplitText processor that mimics the current functionality of the 
> NiFi SplitText.java processor for MiNiFi C++. This will allow for text to be 
> split on the edge devices in a more distributed manner so that upstream nifi 
> instances will not have the burden of trying to split text from multiple edge 
> devices and help NiFi throughput in those scenarios since it will no longer 
> have to handle that.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[PR] MINIFICPP-62 Add SplitText processor [nifi-minifi-cpp]

2023-10-17 Thread via GitHub


lordgamez opened a new pull request, #1682:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1682

   https://issues.apache.org/jira/browse/MINIFICPP-62
   
   -
   Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
   
   In order to streamline the review of the contribution we ask you
   to ensure the following steps have been taken:
   
   ### For all changes:
   - [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
   
   - [ ] Does your PR title start with MINIFICPP- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
   
   - [ ] Has your PR been rebased against the latest commit within the target 
branch (typically main)?
   
   - [ ] Is your initial contribution a single, squashed commit?
   
   ### For code changes:
   - [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
   - [ ] If applicable, have you updated the LICENSE file?
   - [ ] If applicable, have you updated the NOTICE file?
   
   ### For documentation related changes:
   - [ ] Have you ensured that format looks appropriate for the output in which 
it is rendered?
   
   ### Note:
   Please ensure that once the PR is submitted, you check GitHub Actions CI 
results for build issues and submit an update to your PR as soon as possible.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Updated] (NIFI-12232) Frequent "failed to connect node to cluster because local flow controller partially updated. Administrator should disconnect node and review flow for corruption"

2023-10-17 Thread John Joseph (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Joseph updated NIFI-12232:
---
Description: 
This is an issue that we have been observing in the 1.23.2 version of NiFi when 
we try upgrade,

Since Rolling upgrade is not supported in NiFi, we scale out the revision that 
is running and {_}run a helm upgrade{_}.

We have NIFI running in k8s cluster mode, there is a post job that call the 
Tenants and policies API. On a successful run it would run like this
{code:java}
set_policies() Action: 'read' Resource: '/flow' entity_id: 
'ad2d3ad6-5d69-3e0f-95e9-c7feb36e2de5' entity_name: 'CN=nifi-api-admin' 
entity_type: 'USER'
set_policies() status: '200'
'read' '/flow' policy already exists. It will be updated...
set_policies() fetching policy inside -eq 200 status: '200'
set_policies() after update PUT: '200'
set_policies() Action: 'read' Resource: '/tenants' entity_id: 
'ad2d3ad6-5d69-3e0f-95e9-c7feb36e2de5' entity_name: 'CN=nifi-api-admin' 
entity_type: 'USER'
set_policies() status: '200'{code}
*_This job was running fine in 1.23.0, 1.22 and other previous versions._* In 
{*}{{1.23.2}}{*}, we are noticing that the job is failing very frequently with 
the error logs;
{code:java}
set_policies() Action: 'read' Resource: '/flow' entity_id: 
'ad2d3ad6-5d69-3e0f-95e9-c7feb36e2de5' entity_name: 'CN=nifi-api-admin' 
entity_type: 'USER'
set_policies() status: '200'
'read' '/flow' policy already exists. It will be updated...
set_policies() fetching policy inside -eq 200 status: '200'
set_policies() after update PUT: '400'
An error occurred getting 'read' '/flow' policy: 'This node is disconnected 
from its configured cluster. The requested change will only be allowed if the 
flag to acknowledge the disconnected node is set.'{code}
{{_*'This node is disconnected from its configured cluster. The requested 
change will only be allowed if the flag to acknowledge the disconnected node is 
set.'*_}}

The job is configured to run only after all the pods are up and running. Though 
the pods are up we see exception is the inside pods
{code:java}
org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed 
to connect node to cluster because local flow controller partially updated. 
Administrator should disconnect node and review flow for corruption.
at 
org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:1059)
at 
org.apache.nifi.controller.StandardFlowService.handleReconnectionRequest(StandardFlowService.java:667)
at 
org.apache.nifi.controller.StandardFlowService.access$200(StandardFlowService.java:107)
at 
org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:396)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: 
org.apache.nifi.controller.serialization.FlowSynchronizationException: 
java.lang.IllegalStateException: Cannot change destination of Connection 
because the current destination is running
at 
org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:448)
at 
org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.sync(VersionedFlowSynchronizer.java:206)
at 
org.apache.nifi.controller.serialization.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:42)
at 
org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1530)
at 
org.apache.nifi.persistence.StandardFlowConfigurationDAO.load(StandardFlowConfigurationDAO.java:104)
at 
org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:817)
at 
org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:1028)
... 4 common frames omitted
Caused by: java.lang.IllegalStateException: Cannot change destination of 
Connection because the current destination is running
at 
org.apache.nifi.connectable.StandardConnection.setDestination(StandardConnection.java:310)
at 
org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.updateConnectionDestinations(StandardVersionedComponentSynchronizer.java:700)
at 
org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.synchronize(StandardVersionedComponentSynchronizer.java:405)
at 
org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.synchronizeChildGroups(StandardVersionedComponentSynchronizer.java:543)
at 
org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.synchronize(StandardVersionedComponentSynchronizer.java:427)
at 
org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.lambda$synchronize$0(StandardVersionedComponentSynchronizer.java:266)
at 
org.apache.nifi.controller.flow.AbstractFlowManager.withParameterContextResolution(AbstractFlowManager.java:550)
at 

[jira] [Assigned] (MINIFICPP-1415) Pass references to onTrigger and onSchedule instead of shared_ptr

2023-10-17 Thread Martin Zink (Jira)


 [ 
https://issues.apache.org/jira/browse/MINIFICPP-1415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Martin Zink reassigned MINIFICPP-1415:
--

Assignee: Martin Zink

> Pass references to onTrigger and onSchedule instead of shared_ptr
> -
>
> Key: MINIFICPP-1415
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1415
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Improvement
>Reporter: Marton Szasz
>Assignee: Martin Zink
>Priority: Minor
>
> To avoid mandatory shared ownership of ProcessSessionFactory, ProcessSession 
> and ProcessContext, possibly enabling future refactoring to simplify 
> processor code by removing these overloads.
> This would also simplify processor implementation which is important for 
> third-party extensibility. I would love to eventually see an ecosystem where 
> people start creating their own independent processors for minifi c++ to 
> interface with other software.
>  
> original idea: 
> https://github.com/apache/nifi-minifi-cpp/pull/943#discussion_r532575479



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Re: [PR] NIFI-12201 - Deprecation marking for nifi-toolkit-tls in NiFi 1.x [nifi]

2023-10-17 Thread via GitHub


ChrisSamo632 closed pull request #7880: NIFI-12201 - Deprecation marking for 
nifi-toolkit-tls in NiFi 1.x
URL: https://github.com/apache/nifi/pull/7880


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] NIFI-12201 - Deprecation marking for nifi-toolkit-tls in NiFi 1.x [nifi]

2023-10-17 Thread via GitHub


ChrisSamo632 commented on PR #7880:
URL: https://github.com/apache/nifi/pull/7880#issuecomment-1765947991

   This has been merged to `support/nifi-1.x`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] MINIFICPP-2248 Refactor string::join_pack [nifi-minifi-cpp]

2023-10-17 Thread via GitHub


fgerlits commented on code in PR #1680:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1680#discussion_r1361673320


##
libminifi/test/unit/StringUtilsTests.cpp:
##
@@ -469,15 +476,16 @@ TEST_CASE("test string::testJoinPackWstring", "[test 
join_pack wstring]") {
   == L"rvalue c string, c string, rval std::string, std::string, 
std::string_view, char array");
 }
 
-/* doesn't and shouldn't compile
-TEST_CASE("test string::testJoinPackNegative", "[test join_pack negative]") {
-  std::wstring stdstr = L"std::string";
-  const wchar_t* cstr = L"c string";
-  const wchar_t carr[] = L"char array";
-  REQUIRE(string::join_pack("rvalue c string, ", cstr, std::string{ ", rval 
std::string, " }, stdstr, ", ", carr)
-  == "rvalue c string, c string, rval std::string, std::string, 
char array");
+namespace detail {
+template
+concept join_pack_works_with_args = requires(Strs... strs) {
+  string::join_pack(strs...);
+};
+}  // namespace detail
+
+TEST_CASE("test string::join_pack can't combine different char types", "[test 
join_pack negative][different char types]") {
+  STATIC_REQUIRE(!detail::join_pack_works_with_args);  // 
NOLINT: testing C array

Review Comment:
   I would change this to `STATIC_CHECK` and add a bunch more, eg. 
``, ``, `` -- the main thing being different types for 
the first parameter.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] NIFI-8135 allow CHOICE data types in conversion of Records to Java Maps [nifi]

2023-10-17 Thread via GitHub


ChrisSamo632 commented on PR #7746:
URL: https://github.com/apache/nifi/pull/7746#issuecomment-1765802460

   @mattyb149 certainly 2, 1 as well if it doesn't break things (I think it 
should be OK). Jira ticket updated


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Updated] (NIFI-8135) DataTypeUtils conversion methods for Records to java Maps should respect Choice types

2023-10-17 Thread Chris Sampson (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-8135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Sampson updated NIFI-8135:

Fix Version/s: 2.0.0
   1.latest

> DataTypeUtils conversion methods for Records to java Maps should respect 
> Choice types
> -
>
> Key: NIFI-8135
> URL: https://issues.apache.org/jira/browse/NIFI-8135
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.11.4
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Major
> Fix For: 2.0.0, 1.latest
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Where a schema field is a Map of Choice, with options of Array (of Strings),  
> Map (of Strings) or String, Record Readers incorrectly deserialise the data 
> of the nested Map/Array as Records containing no schema. When converting the 
> overall Record into a java Map using DataTypeUtils, the nested MapRecord is 
> not converted into a Map.
> This can lead to data loss between readers & writers.
> This appears to be related to one/more of:
> * missing schema in the nested MapRecord object (even if it's presented with 
> the data to the Record Reader) - the conversion methods could infer a schema 
> for such MapRecords rather than effectively skipping them
> * Choice types don't appear to be respected for nested fields (e.g. checking 
> whether the options within a Choice are for Map or Array,  not just whether 
> the field dataType passed to the conversion method is for Map or Array, etc. 
> ... it's a Choice that should allow for these)
> * Array/ Map conversion should not assume all elements within the collection 
> are of the same type, especially if the field's type is Choice



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-8135) DataTypeUtils conversion methods for Records to java Maps should respect Choice types

2023-10-17 Thread Chris Sampson (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-8135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Sampson updated NIFI-8135:

Fix Version/s: 2.latest
   (was: 2.0.0)

> DataTypeUtils conversion methods for Records to java Maps should respect 
> Choice types
> -
>
> Key: NIFI-8135
> URL: https://issues.apache.org/jira/browse/NIFI-8135
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.11.4
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Major
> Fix For: 1.latest, 2.latest
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Where a schema field is a Map of Choice, with options of Array (of Strings),  
> Map (of Strings) or String, Record Readers incorrectly deserialise the data 
> of the nested Map/Array as Records containing no schema. When converting the 
> overall Record into a java Map using DataTypeUtils, the nested MapRecord is 
> not converted into a Map.
> This can lead to data loss between readers & writers.
> This appears to be related to one/more of:
> * missing schema in the nested MapRecord object (even if it's presented with 
> the data to the Record Reader) - the conversion methods could infer a schema 
> for such MapRecords rather than effectively skipping them
> * Choice types don't appear to be respected for nested fields (e.g. checking 
> whether the options within a Choice are for Map or Array,  not just whether 
> the field dataType passed to the conversion method is for Map or Array, etc. 
> ... it's a Choice that should allow for these)
> * Array/ Map conversion should not assume all elements within the collection 
> are of the same type, especially if the field's type is Choice



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Re: [PR] NIFI-8135 allow CHOICE data types in conversion of Records to Java Maps [nifi]

2023-10-17 Thread via GitHub


mattyb149 commented on PR #7746:
URL: https://github.com/apache/nifi/pull/7746#issuecomment-1765772261

   Is this meant for both 1.x and 2.x? Can you update the Jira with `1.latest` 
and/or `2.latest` to indicate where you'd like to see this merged? Please and 
thanks! I'll give it a try on main in the meantime :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Updated] (NIFI-12234) Upgrade SSHJ to 0.37.0

2023-10-17 Thread Matt Burgess (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess updated NIFI-12234:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Upgrade SSHJ to 0.37.0
> --
>
> Key: NIFI-12234
> URL: https://issues.apache.org/jira/browse/NIFI-12234
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Minor
>  Labels: dependency-upgrade
> Fix For: 2.0.0, 1.24.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> SSHJ [0.37|https://github.com/hierynomus/sshj/#release-history] includes 
> several bug fixes, including a resolution for a potential race condition 
> during key exchange, and updates to support additional encryption algorithms 
> for OpenSSH Private Keys.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (NIFI-12234) Upgrade SSHJ to 0.37.0

2023-10-17 Thread ASF subversion and git services (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17776036#comment-17776036
 ] 

ASF subversion and git services commented on NIFI-12234:


Commit 574c2b21682c78eb0b38bdeec6752b1688ce95a5 in nifi's branch 
refs/heads/main from David Handermann
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=574c2b2168 ]

NIFI-12234 Upgraded SSHJ from 0.35.0 to 0.37.0

Signed-off-by: Matt Burgess 

This closes #7886


> Upgrade SSHJ to 0.37.0
> --
>
> Key: NIFI-12234
> URL: https://issues.apache.org/jira/browse/NIFI-12234
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Minor
>  Labels: dependency-upgrade
> Fix For: 2.0.0, 1.24.0
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> SSHJ [0.37|https://github.com/hierynomus/sshj/#release-history] includes 
> several bug fixes, including a resolution for a potential race condition 
> during key exchange, and updates to support additional encryption algorithms 
> for OpenSSH Private Keys.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Re: [PR] NIFI-12234 Upgrade SSHJ from 0.35.0 to 0.37.0 [nifi]

2023-10-17 Thread via GitHub


mattyb149 closed pull request #7886: NIFI-12234 Upgrade SSHJ from 0.35.0 to 
0.37.0
URL: https://github.com/apache/nifi/pull/7886


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] NIFI-12234 Upgrade SSHJ from 0.35.0 to 0.37.0 [nifi]

2023-10-17 Thread via GitHub


mattyb149 commented on PR #7886:
URL: https://github.com/apache/nifi/pull/7886#issuecomment-1765770146

   +1 LGTM, thanks for the upgrade! Merged to main and support/nifi-1.x


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Commented] (NIFI-12234) Upgrade SSHJ to 0.37.0

2023-10-17 Thread ASF subversion and git services (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17776028#comment-17776028
 ] 

ASF subversion and git services commented on NIFI-12234:


Commit d4ca334813054c49cc91065a7f12140154bfe8ec in nifi's branch 
refs/heads/support/nifi-1.x from David Handermann
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=d4ca334813 ]

NIFI-12234 Upgraded SSHJ from 0.35.0 to 0.37.0

Signed-off-by: Matt Burgess 


> Upgrade SSHJ to 0.37.0
> --
>
> Key: NIFI-12234
> URL: https://issues.apache.org/jira/browse/NIFI-12234
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Minor
>  Labels: dependency-upgrade
> Fix For: 2.0.0, 1.24.0
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> SSHJ [0.37|https://github.com/hierynomus/sshj/#release-history] includes 
> several bug fixes, including a resolution for a potential race condition 
> during key exchange, and updates to support additional encryption algorithms 
> for OpenSSH Private Keys.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)