[jira] [Commented] (NIFI-8930) Add ability for ScanAttribute processor to match on attribute values containing a delimited list of values.

2024-03-27 Thread Joe Witt (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-8930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17831439#comment-17831439
 ] 

Joe Witt commented on NIFI-8930:


[~tlsmith]Hey sorry this one must have gotten lost in the shuffle.  The 
processor works on the basis of an extremely fast [search 
algorithm|https://en.wikipedia.org/wiki/Aho%E2%80%93Corasick_algorithm] which 
is a very different animal than regex so that might be part of the delta.  What 
you want to do makes sense for sure just not sure if 'this' is the right 
component or not.  

> Add ability for ScanAttribute processor to match on attribute values 
> containing a delimited list of values.
> ---
>
> Key: NIFI-8930
> URL: https://issues.apache.org/jira/browse/NIFI-8930
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.14.0
>Reporter: Tim Smith
>Assignee: Tim Smith
>Priority: Minor
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> The ScanAttribute processor is limited in that an exact match is required 
> when applied for attribute value. There are many times an attribute contains 
> a delimited list of values. It would be useful to specify a delimiter and  
> have the match criteria applied to each delimited value in the attribute. Two 
> additional property descriptors need created, 'Delimiter' and 'Delimited 
> Match Criteria'. 'Delimiter' sets the delimiter to apply to the attribute(s). 
> 'Delimited Match Criteria' specifies how the delimited attributes should 
> match the dictionary. If 'All Must Match' is selected, all delimited values 
> must match a dictionary term/pattern for attribute to be matched. For 'At 
> Least 1 Must Match' , if any one delimited value matches,then the attribute 
> matches.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (NIFI-12930) FetchFile on failure during StandardProcessSession.importFrom should route to failure instead of rollback

2024-03-25 Thread Joe Witt (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17830716#comment-17830716
 ] 

Joe Witt commented on NIFI-12930:
-

[~jrsteinebrey]I have it assigned to me as there is already a PR that addresses 
it.  Thanks

> FetchFile on failure during StandardProcessSession.importFrom should route to 
> failure instead of rollback
> -
>
> Key: NIFI-12930
> URL: https://issues.apache.org/jira/browse/NIFI-12930
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joe Witt
>Assignee: Joe Witt
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Consider a scenario involving corrupt files on a disk.
> FetchFile during the importFrom call can fail in such cases as would any 
> process.  But the current handling calls rollback instead of routing to 
> failure.   As a result the flow could be stuck in an endless loop trying the 
> same objects over and over and not giving the flow designer a chance to 
> reasonably handle such cases.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Assigned] (NIFI-12930) FetchFile on failure during StandardProcessSession.importFrom should route to failure instead of rollback

2024-03-25 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt reassigned NIFI-12930:
---

Assignee: Joe Witt  (was: Jim Steinebrey)

> FetchFile on failure during StandardProcessSession.importFrom should route to 
> failure instead of rollback
> -
>
> Key: NIFI-12930
> URL: https://issues.apache.org/jira/browse/NIFI-12930
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joe Witt
>Assignee: Joe Witt
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Consider a scenario involving corrupt files on a disk.
> FetchFile during the importFrom call can fail in such cases as would any 
> process.  But the current handling calls rollback instead of routing to 
> failure.   As a result the flow could be stuck in an endless loop trying the 
> same objects over and over and not giving the flow designer a chance to 
> reasonably handle such cases.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12943) Upgrade Hadoop to 3.4.0

2024-03-25 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-12943:

Fix Version/s: 2.0.0-M3
   Resolution: Fixed
   Status: Resolved  (was: Patch Available)

> Upgrade Hadoop to 3.4.0
> ---
>
> Key: NIFI-12943
> URL: https://issues.apache.org/jira/browse/NIFI-12943
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
> Fix For: 2.0.0-M3
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Apache Hadoop dependencies should be upgraded to 
> [3.4.0|https://hadoop.apache.org/docs/r3.4.0/index.html] to incorporate bug 
> fixes and improvements, including a number of transitive dependency upgrades.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (NIFI-12930) FetchFile on failure during StandardProcessSession.importFrom should route to failure instead of rollback

2024-03-21 Thread Joe Witt (Jira)
Joe Witt created NIFI-12930:
---

 Summary: FetchFile on failure during 
StandardProcessSession.importFrom should route to failure instead of rollback
 Key: NIFI-12930
 URL: https://issues.apache.org/jira/browse/NIFI-12930
 Project: Apache NiFi
  Issue Type: Improvement
Reporter: Joe Witt


Consider a scenario involving corrupt files on a disk.

FetchFile during the importFrom call can fail in such cases as would any 
process.  But the current handling calls rollback instead of routing to 
failure.   As a result the flow could be stuck in an endless loop trying the 
same objects over and over and not giving the flow designer a chance to 
reasonably handle such cases.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12911) Upgrade Jagged to 0.3.2

2024-03-15 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-12911:

Fix Version/s: 2.0.0
   1.26.0
   Resolution: Fixed
   Status: Resolved  (was: Patch Available)

> Upgrade Jagged to 0.3.2
> ---
>
> Key: NIFI-12911
> URL: https://issues.apache.org/jira/browse/NIFI-12911
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Minor
>  Labels: backport-needed
> Fix For: 2.0.0, 1.26.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Jagged [0.3.2|https://github.com/exceptionfactory/jagged/releases/tag/0.3.2] 
> for the EncryptContentAge and DecryptContentAge Processors includes minor bug 
> fixes related to stream closure handling.
> This upgrade should be applied to both the main and support branches.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12910) Upgrade Spring Framework to 6.0.18

2024-03-15 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-12910:

Fix Version/s: 2.0.0
   Resolution: Fixed
   Status: Resolved  (was: Patch Available)

> Upgrade Spring Framework to 6.0.18
> --
>
> Key: NIFI-12910
> URL: https://issues.apache.org/jira/browse/NIFI-12910
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework, NiFi Registry
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Minor
> Fix For: 2.0.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Spring Framework dependencies should be upgraded to 
> [6.0.18|https://github.com/spring-projects/spring-framework/releases/tag/v6.0.18]
>  for NiFi.
> Spring Framework dependencies for NiFi Registry should be upgraded to 
> [6.1.5.|https://github.com/spring-projects/spring-framework/releases/tag/v6.1.5]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (NIFI-12907) Update to Groovy 4.0.20

2024-03-14 Thread Joe Witt (Jira)
Joe Witt created NIFI-12907:
---

 Summary: Update to Groovy 4.0.20
 Key: NIFI-12907
 URL: https://issues.apache.org/jira/browse/NIFI-12907
 Project: Apache NiFi
  Issue Type: Task
Reporter: Joe Witt
Assignee: Joe Witt
 Fix For: 2.0.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12906) Upgrade zookeeper to move past server side issues in CVE-2024-23944

2024-03-14 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-12906:

Fix Version/s: 2.0.0
   1.26.0

> Upgrade zookeeper to move past server side issues in CVE-2024-23944
> ---
>
> Key: NIFI-12906
> URL: https://issues.apache.org/jira/browse/NIFI-12906
> Project: Apache NiFi
>  Issue Type: Task
>Reporter: Joe Witt
>Assignee: Joe Witt
>Priority: Major
> Fix For: 2.0.0, 1.26.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (NIFI-12906) Upgrade zookeeper to move past server side issues in CVE-2024-23944

2024-03-14 Thread Joe Witt (Jira)
Joe Witt created NIFI-12906:
---

 Summary: Upgrade zookeeper to move past server side issues in 
CVE-2024-23944
 Key: NIFI-12906
 URL: https://issues.apache.org/jira/browse/NIFI-12906
 Project: Apache NiFi
  Issue Type: Task
Reporter: Joe Witt
Assignee: Joe Witt






--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12826) Unstable test TestFTP testListFtpHostPortVariablesFileFound

2024-02-20 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-12826:

Summary: Unstable test TestFTP testListFtpHostPortVariablesFileFound  (was: 
Unstable test)

> Unstable test TestFTP testListFtpHostPortVariablesFileFound
> ---
>
> Key: NIFI-12826
> URL: https://issues.apache.org/jira/browse/NIFI-12826
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joe Witt
>Assignee: Joe Witt
>Priority: Major
>
> {noformat}
> Error:  
> org.apache.nifi.processors.standard.TestFTP.testListFtpHostPortVariablesFileFound
>  -- Time elapsed: 0.232 s <<< FAILURE!
> org.opentest4j.AssertionFailedError: expected: <1> but was: <0>
>   at 
> org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
>   at 
> org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
>   at 
> org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
>   at 
> org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:150)
>   at 
> org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:145)
>   at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:527)
>   at 
> org.apache.nifi.util.StandardProcessorTestRunner.assertTransferCount(StandardProcessorTestRunner.java:404)
>   at 
> org.apache.nifi.processors.standard.TestFTP.testListFtpHostPortVariablesFileFound(TestFTP.java:319)
>   at java.base/java.lang.reflect.Method.invoke(Method.java:580)
>   at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
>   at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
> {noformat}
> Failed in a random build 
> https://github.com/apache/nifi/actions/runs/7981892840/job/2179960?pr=8438



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (NIFI-12826) Unstable test

2024-02-20 Thread Joe Witt (Jira)
Joe Witt created NIFI-12826:
---

 Summary: Unstable test
 Key: NIFI-12826
 URL: https://issues.apache.org/jira/browse/NIFI-12826
 Project: Apache NiFi
  Issue Type: Improvement
Reporter: Joe Witt
Assignee: Joe Witt



{noformat}
Error:  
org.apache.nifi.processors.standard.TestFTP.testListFtpHostPortVariablesFileFound
 -- Time elapsed: 0.232 s <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <1> but was: <0>
at 
org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
at 
org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
at 
org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
at 
org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:150)
at 
org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:145)
at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:527)
at 
org.apache.nifi.util.StandardProcessorTestRunner.assertTransferCount(StandardProcessorTestRunner.java:404)
at 
org.apache.nifi.processors.standard.TestFTP.testListFtpHostPortVariablesFileFound(TestFTP.java:319)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
{noformat}

Failed in a random build 
https://github.com/apache/nifi/actions/runs/7981892840/job/2179960?pr=8438




--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (NIFI-12768) Intermittent Failures in TestListFile.testFilterAge

2024-02-20 Thread Joe Witt (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17819019#comment-17819019
 ] 

Joe Witt commented on NIFI-12768:
-

New PR will remove the same pattern of checks in the same test but preceding 
the change already made in this JIRA.  It presumes ordering that isn't 
guaranteed which makes these already timing dependent tests worse.

> Intermittent Failures in TestListFile.testFilterAge
> ---
>
> Key: NIFI-12768
> URL: https://issues.apache.org/jira/browse/NIFI-12768
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 2.0.0-M2
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
> Fix For: 2.0.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The TestListFIle class has not changed substantively in quite some time, but 
> it has begun to fail more recently across multiple platforms on GitHub Action 
> runners.
> The {{testFilterAge}} method often fails with the same stack trace:
> {noformat}
> Error:  org.apache.nifi.processors.standard.TestListFile.testFilterAge -- 
> Time elapsed: 6.436 s <<< FAILURE!
> org.opentest4j.AssertionFailedError: expected:  but was: 
>   at 
> org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
>   at 
> org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
>   at 
> org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
>   at 
> org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
>   at 
> org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:177)
>   at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1141)
>   at 
> org.apache.nifi.processors.standard.TestListFile.testFilterAge(TestListFile.java:331)
>   at java.base/java.lang.reflect.Method.invoke(Method.java:580)
>   at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
>   at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
> {noformat}
> The test method use recalculated timestamps to set file modification time, so 
> the problem appears to be related to these timing calculations.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Resolved] (NIFI-12824) Remove ExecuteStateless Processor

2024-02-20 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt resolved NIFI-12824.
-
Resolution: Fixed

> Remove ExecuteStateless Processor
> -
>
> Key: NIFI-12824
> URL: https://issues.apache.org/jira/browse/NIFI-12824
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
> Fix For: 2.0.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> The {{ExecuteStateless}} Processor and the {{nifi-stateless-processor-nar}} 
> should be removed from the main branch as part of preparation for NiFi 2.0.0. 
> The {{ExecuteStateless}} Processor provided a transitional solution prior to 
> Stateless Execution of Process Groups in standard NiFi deployments. Now that 
> Stateless Execution is implemented, the Processor should be removed. As the 
> NiFi 2.0.0-M1 and 2.0.0-M2 releases include both the {{ExecuteStateless}} 
> Processor and Stateless Execution options, those releases can provide a 
> transitional step for deployments that need to have both options available 
> for an interim migration approach.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (NIFI-12768) Intermittent Failures in TestListFile.testFilterAge

2024-02-20 Thread Joe Witt (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17819016#comment-17819016
 ] 

Joe Witt commented on NIFI-12768:
-

Reopened since this JIRA/fix is still on this unreleased line.  But a very 
similar problem remains in a similar test.
{code:java}
Error:  org.apache.nifi.processors.standard.TestListFile.testFilterAge -- Time 
elapsed: 6.302 s <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <2> but was: <3>
at 
org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
at 
org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
at 
org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
at 
org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:150)
at 
org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:145)
at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:527)
at 
org.apache.nifi.processors.standard.TestListFile.testFilterAge(TestListFile.java:319)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
{code}


> Intermittent Failures in TestListFile.testFilterAge
> ---
>
> Key: NIFI-12768
> URL: https://issues.apache.org/jira/browse/NIFI-12768
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 2.0.0-M2
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
> Fix For: 2.0.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The TestListFIle class has not changed substantively in quite some time, but 
> it has begun to fail more recently across multiple platforms on GitHub Action 
> runners.
> The {{testFilterAge}} method often fails with the same stack trace:
> {noformat}
> Error:  org.apache.nifi.processors.standard.TestListFile.testFilterAge -- 
> Time elapsed: 6.436 s <<< FAILURE!
> org.opentest4j.AssertionFailedError: expected:  but was: 
>   at 
> org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
>   at 
> org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
>   at 
> org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
>   at 
> org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
>   at 
> org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:177)
>   at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1141)
>   at 
> org.apache.nifi.processors.standard.TestListFile.testFilterAge(TestListFile.java:331)
>   at java.base/java.lang.reflect.Method.invoke(Method.java:580)
>   at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
>   at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
> {noformat}
> The test method use recalculated timestamps to set file modification time, so 
> the problem appears to be related to these timing calculations.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Reopened] (NIFI-12768) Intermittent Failures in TestListFile.testFilterAge

2024-02-20 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt reopened NIFI-12768:
-

> Intermittent Failures in TestListFile.testFilterAge
> ---
>
> Key: NIFI-12768
> URL: https://issues.apache.org/jira/browse/NIFI-12768
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 2.0.0-M2
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
> Fix For: 2.0.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The TestListFIle class has not changed substantively in quite some time, but 
> it has begun to fail more recently across multiple platforms on GitHub Action 
> runners.
> The {{testFilterAge}} method often fails with the same stack trace:
> {noformat}
> Error:  org.apache.nifi.processors.standard.TestListFile.testFilterAge -- 
> Time elapsed: 6.436 s <<< FAILURE!
> org.opentest4j.AssertionFailedError: expected:  but was: 
>   at 
> org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
>   at 
> org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
>   at 
> org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
>   at 
> org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
>   at 
> org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:177)
>   at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1141)
>   at 
> org.apache.nifi.processors.standard.TestListFile.testFilterAge(TestListFile.java:331)
>   at java.base/java.lang.reflect.Method.invoke(Method.java:580)
>   at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
>   at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
> {noformat}
> The test method use recalculated timestamps to set file modification time, so 
> the problem appears to be related to these timing calculations.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Resolved] (NIFI-12772) Remote poll batch size not exposed for ListSFTP

2024-02-20 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt resolved NIFI-12772.
-
Fix Version/s: 2.0.0
   Resolution: Fixed

> Remote poll batch size not exposed for ListSFTP
> ---
>
> Key: NIFI-12772
> URL: https://issues.apache.org/jira/browse/NIFI-12772
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.25.0
>Reporter: Tom Brisland
>Priority: Minor
> Fix For: 2.0.0
>
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> I was planning on adding support for a batch size in ListSFTP due to some 
> issues we're seeing with a large number of files on an SFTP server.
> Thankfully, it seems that REMOTE_POLL_BATCH_SIZE is already supported in all 
> the FTP processor variations, just not exposed in ListSFTP.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12796) PutDatabaseRecord operation should support u/c/d for Debezium

2024-02-20 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-12796:

Fix Version/s: 2.0.0
   Resolution: Fixed
   Status: Resolved  (was: Patch Available)

> PutDatabaseRecord operation should support u/c/d for Debezium
> -
>
> Key: NIFI-12796
> URL: https://issues.apache.org/jira/browse/NIFI-12796
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
> Fix For: 2.0.0
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Similar to NIFI-12344, PutDatabaseRecord operation path property should 
> support the values u/c/d for out of the box integration with Debezium events.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12818) Deprecate Apache Atlas Reporting Task for Removal

2024-02-20 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-12818:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Deprecate Apache Atlas Reporting Task for Removal
> -
>
> Key: NIFI-12818
> URL: https://issues.apache.org/jira/browse/NIFI-12818
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Minor
> Fix For: 1.26.0
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The {{ReportLineageToAtlas}} Reporting Task should be deprecated on the 
> support branch for removal from the main branch.
> The Apache Atlas Reporting Task includes a number of transitive dependencies 
> that do not align with current Apache NiFi project versions, including 
> Servlet API, Java XML Binding, and Jersey. These differences are in addition 
> existing differences for outdated version references such as Guava, Gson, 
> Jackson, and Netty. The current Reporting Task implementation also includes a 
> number of classes that are specific to certain NiFi components, which is not 
> a maintainable implementation. The current {{ReportLineageToAtlas}} Task 
> should be removed from the main branch for NiFi 2.0 to provide clear baseline 
> for any potential future implementation.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12817) Move Hadoop DBCP NAR to Hadoop Build Profile

2024-02-20 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-12817:

Fix Version/s: 2.0.0
   Resolution: Fixed
   Status: Resolved  (was: Patch Available)

> Move Hadoop DBCP NAR to Hadoop Build Profile
> 
>
> Key: NIFI-12817
> URL: https://issues.apache.org/jira/browse/NIFI-12817
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Minor
> Fix For: 2.0.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Recent updates moved Hadoop components to an optional build profile named 
> {{{}include-hadoop{}}}. The {{nifi-hadoop-dbcp-service-nar}} should be moved 
> to the same optional profile for common grouping of related components.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12821) Set docker-maven-plugin Version

2024-02-20 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-12821:

Fix Version/s: 2.0.0
   Resolution: Fixed
   Status: Resolved  (was: Patch Available)

> Set docker-maven-plugin Version
> ---
>
> Key: NIFI-12821
> URL: https://issues.apache.org/jira/browse/NIFI-12821
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework, Tools and Build
>Affects Versions: 2.0.0-M2
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
> Fix For: 2.0.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The docker-maven-plugin supports building Docker images for multiple modules. 
> The parent Maven configuration should have an explicit version listed to 
> avoid unexpected upgrades resulting in build failures.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12820) Upgrade Email Processors to Jakarta Mail 2

2024-02-20 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-12820:

Fix Version/s: 2.0.0

> Upgrade Email Processors to Jakarta Mail 2
> --
>
> Key: NIFI-12820
> URL: https://issues.apache.org/jira/browse/NIFI-12820
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
> Fix For: 2.0.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The {{nifi-email-bundle}} includes multiple Processors for sending and 
> receiving email using standard protocols including SMTP, POP3, and IMAP. 
> Processors for POP3 and IMAP use Spring Integration Mail libraries while 
> ListenSMTP uses the SubEtha SMTP library. These libraries have shared 
> dependencies on Java Mail 1, which has been superseded by Jakarta Mail 2. All 
> libraries need to be upgraded together to avoid runtime conflicts.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12820) Upgrade Email Processors to Jakarta Mail 2

2024-02-20 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-12820:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Upgrade Email Processors to Jakarta Mail 2
> --
>
> Key: NIFI-12820
> URL: https://issues.apache.org/jira/browse/NIFI-12820
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The {{nifi-email-bundle}} includes multiple Processors for sending and 
> receiving email using standard protocols including SMTP, POP3, and IMAP. 
> Processors for POP3 and IMAP use Spring Integration Mail libraries while 
> ListenSMTP uses the SubEtha SMTP library. These libraries have shared 
> dependencies on Java Mail 1, which has been superseded by Jakarta Mail 2. All 
> libraries need to be upgraded together to avoid runtime conflicts.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12819) Remove Apache Atlas Reporting Task

2024-02-20 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-12819:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Remove Apache Atlas Reporting Task
> --
>
> Key: NIFI-12819
> URL: https://issues.apache.org/jira/browse/NIFI-12819
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Minor
> Fix For: 2.0.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The {{ReportLineageToAtlas}} Task and associated {{nifi-atlas-bundle}} should 
> be removed from the main branch for NiFi 2.0 based on legacy dependencies as 
> described in NIFI-12818.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (NIFI-12232) Frequent "failed to connect node to cluster because local flow controller partially updated. Administrator should disconnect node and review flow for corruption"

2024-02-16 Thread Joe Witt (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17818091#comment-17818091
 ] 

Joe Witt commented on NIFI-12232:
-

Also hit by

https://apachenifi.slack.com/archives/C0L9VCD47/p1708113098305609

Roman Wesołowski
  29 minutes ago
Hi all,
I have 3 nodes Nifi cluster with 2.0.0-M1 version. Till today everything was 
working correctly, during my development something strange happeded. For some 
reason 2 nodes disconnected from cluster, and I am not able to reconnect them 
to the cluster. I have resterted nodes but without successes. All machines are 
up but can not connect each other.  Any help would be appreciated.
2024-02-16 15:14:36,663 ERROR [Reconnect to Cluster] 
o.a.n.c.c.node.NodeClusterCoordinator Event Reported for 10.120.8.252:8080 -- 
Node disconnected from cluster due to 
org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed 
to connect node to cluster because local flow controller partially updated. 
Administrator should disconnect node andreview flow for corruption.

> Frequent "failed to connect node to cluster because local flow controller 
> partially updated. Administrator should disconnect node and review flow for 
> corruption"
> -
>
> Key: NIFI-12232
> URL: https://issues.apache.org/jira/browse/NIFI-12232
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Configuration Management
>Affects Versions: 1.23.2
>Reporter: John Joseph
>Assignee: Mark Payne
>Priority: Major
> Attachments: image-2023-10-16-16-12-31-027.png, 
> image-2024-02-14-13-33-44-354.png
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> This is an issue that we have been observing in the 1.23.2 version of NiFi 
> when we try upgrade,
> Since Rolling upgrade is not supported in NiFi, we scale out the revision 
> that is running and {_}run a helm upgrade{_}.
> We have NIFI running in k8s cluster mode, there is a post job that call the 
> Tenants and policies API. On a successful run it would run like this
> {code:java}
> set_policies() Action: 'read' Resource: '/flow' entity_id: 
> 'ad2d3ad6-5d69-3e0f-95e9-c7feb36e2de5' entity_name: 'CN=nifi-api-admin' 
> entity_type: 'USER'
> set_policies() status: '200'
> 'read' '/flow' policy already exists. It will be updated...
> set_policies() fetching policy inside -eq 200 status: '200'
> set_policies() after update PUT: '200'
> set_policies() Action: 'read' Resource: '/tenants' entity_id: 
> 'ad2d3ad6-5d69-3e0f-95e9-c7feb36e2de5' entity_name: 'CN=nifi-api-admin' 
> entity_type: 'USER'
> set_policies() status: '200'{code}
> *_This job was running fine in 1.23.0, 1.22 and other previous versions._* In 
> {*}{{1.23.2}}{*}, we are noticing that the job is failing very frequently 
> with the error logs;
> {code:java}
> set_policies() Action: 'read' Resource: '/flow' entity_id: 
> 'ad2d3ad6-5d69-3e0f-95e9-c7feb36e2de5' entity_name: 'CN=nifi-api-admin' 
> entity_type: 'USER'
> set_policies() status: '200'
> 'read' '/flow' policy already exists. It will be updated...
> set_policies() fetching policy inside -eq 200 status: '200'
> set_policies() after update PUT: '400'
> An error occurred getting 'read' '/flow' policy: 'This node is disconnected 
> from its configured cluster. The requested change will only be allowed if the 
> flag to acknowledge the disconnected node is set.'{code}
> {{_*'This node is disconnected from its configured cluster. The requested 
> change will only be allowed if the flag to acknowledge the disconnected node 
> is set.'*_}}
> The job is configured to run only after all the pods are up and running. 
> Though the pods are up we see exception is the inside pods
> {code:java}
> org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed 
> to connect node to cluster because local flow controller partially updated. 
> Administrator should disconnect node and review flow for corruption.
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:1059)
> at 
> org.apache.nifi.controller.StandardFlowService.handleReconnectionRequest(StandardFlowService.java:667)
> at 
> org.apache.nifi.controller.StandardFlowService.access$200(StandardFlowService.java:107)
> at 
> org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:396)
> at java.base/java.lang.Thread.run(Thread.java:833)
> Caused by: 
> org.apache.nifi.controller.serialization.FlowSynchronizationException: 
> java.lang.IllegalStateException: Cannot change destination of Connection 
> because the current destination is running
> at 
> 

[jira] [Comment Edited] (NIFI-12809) PublishKafkaRecord_2_6 - RoundRobin partitioner skipping every other partition

2024-02-16 Thread Joe Witt (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17818086#comment-17818086
 ] 

Joe Witt edited comment on NIFI-12809 at 2/16/24 7:56 PM:
--

[~slyouts] Did you review any Kafka bugs for the noted scenario?  I see 
https://issues.apache.org/jira/browse/KAFKA-13180 that sounds very closely 
related.

If there is a reasonable workaround for this issue that we can control on the 
NiFi side we should ensure we remove any such change in versions it is resolved 
in (such as Kafka 3 components/etc..) 



was (Author: joewitt):
[~slyouts] Did you review any Kafka bugs for the noted scenario?  I see 
https://issues.apache.org/jira/browse/KAFKA-13180 that sounds very closely 
related.


> PublishKafkaRecord_2_6 - RoundRobin partitioner skipping every other partition
> --
>
> Key: NIFI-12809
> URL: https://issues.apache.org/jira/browse/NIFI-12809
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.25.0
>Reporter: Steven Youtsey
>Priority: Major
>  Labels: kafka, partitioners, publish
>
> When configured to use the NiFi RoundRobin partitioner, the processor 
> publishes to every other partition. If the number of partitions in the topic 
> and the number of records being published are the right combination, this 
> problem is masked. We see this issue when we set the partitions to 26, but 
> not when set to 25. 
> I took a code-dive into the o.a.k.c.producer.KafkaProducer and discovered 
> that it is invoking the Partitioner twice when a "new batch" is created. 
> Thus, the RoundRobin partitioner bumps the index by 2. If the RoundRobin 
> partitioner overwrote the onNewBatch method, this problem could be solved.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (NIFI-12809) PublishKafkaRecord_2_6 - RoundRobin partitioner skipping every other partition

2024-02-16 Thread Joe Witt (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17818086#comment-17818086
 ] 

Joe Witt commented on NIFI-12809:
-

[~slyouts] Did you review any Kafka bugs for the noted scenario?  I see 
https://issues.apache.org/jira/browse/KAFKA-13180 that sounds very closely 
related.


> PublishKafkaRecord_2_6 - RoundRobin partitioner skipping every other partition
> --
>
> Key: NIFI-12809
> URL: https://issues.apache.org/jira/browse/NIFI-12809
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.25.0
>Reporter: Steven Youtsey
>Priority: Major
>  Labels: kafka, partitioners, publish
>
> When configured to use the NiFi RoundRobin partitioner, the processor 
> publishes to every other partition. If the number of partitions in the topic 
> and the number of records being published are the right combination, this 
> problem is masked. We see this issue when we set the partitions to 26, but 
> not when set to 25. 
> I took a code-dive into the o.a.k.c.producer.KafkaProducer and discovered 
> that it is invoking the Partitioner twice when a "new batch" is created. 
> Thus, the RoundRobin partitioner bumps the index by 2. If the RoundRobin 
> partitioner overwrote the onNewBatch method, this problem could be solved.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12792) Deprecate Spark Livy Components for Removal

2024-02-14 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-12792:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Deprecate Spark Livy Components for Removal
> ---
>
> Key: NIFI-12792
> URL: https://issues.apache.org/jira/browse/NIFI-12792
> Project: Apache NiFi
>  Issue Type: Task
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
> Fix For: 1.26.0
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The {{nifi-spark-bundle}} contains several components for interacting with 
> [Apache Livy|https://livy.apache.org/], which supports executing Spark jobs 
> over HTTP. The Apache Livy project has graduated from incubation after 
> several years and the NiFi Spark Livy components are not published as part of 
> standard binary builds. For these reasons, the components should be 
> deprecated on the support branch for subsequent removal from the main branch.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12770) Deprecate Apache Ranger Integration for Removal

2024-02-14 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-12770:

Fix Version/s: 1.26.0
   Resolution: Fixed
   Status: Resolved  (was: Patch Available)

> Deprecate Apache Ranger Integration for Removal
> ---
>
> Key: NIFI-12770
> URL: https://issues.apache.org/jira/browse/NIFI-12770
> Project: Apache NiFi
>  Issue Type: Task
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Minor
> Fix For: 1.26.0
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Apache Ranger integration should be deprecated on the support branch for 
> subsequent removal on the main branch due to incompatibilities with Jetty 12 
> and related libraries. The Ranger plugins require Jetty 9, which was marked 
> as [End of Community 
> Support|https://github.com/jetty/jetty.project/issues/7958] in June 2022.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12793) Remove Spark Livy Components

2024-02-14 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-12793:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Remove Spark Livy Components
> 
>
> Key: NIFI-12793
> URL: https://issues.apache.org/jira/browse/NIFI-12793
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
> Fix For: 2.0.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The {{nifi-spark-bundle}} with associated components for interacting with 
> Spark using Apache Livy should be removed from the main branch as part of 
> maintenance efforts for NiFi 2.0.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Resolved] (NIFI-12765) Nifi and nifi registry ranger audit is broken

2024-02-14 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt resolved NIFI-12765.
-
Fix Version/s: 2.0.0
   Resolution: Fixed

> Nifi and nifi registry ranger audit is broken
> -
>
> Key: NIFI-12765
> URL: https://issues.apache.org/jira/browse/NIFI-12765
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 2.0.0-M2, 2.0.0
>Reporter: Zoltán Kornél Török
>Assignee: Zoltán Kornél Török
>Priority: Major
> Fix For: 2.0.0
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> h3. Bug description
> Currently ranger plugins are not reporting audit events into ranger.
> h2. Investigation
> In the nifi log I found the following ("classic") NoClassDefFoundError:
> {code:java}
> ERROR org.apache.ranger.audit.destination.SolrAuditDestination: Can't connect 
> to Solr server. 
> ZooKeepers=cfm-oudjal-dd-master0.cfm-5pax.svbr-nqvp.int.cldr.work:2181/solr-infrajava.lang.NoClassDefFoundError:
>  org/eclipse/jetty/client/util/SPNEGOAuthentication
>   at 
> org.apache.ranger.audit.destination.SolrAuditDestination.connect(SolrAuditDestination.java:168)
>   at 
> org.apache.ranger.audit.destination.SolrAuditDestination.log(SolrAuditDestination.java:227)
>   at 
> org.apache.ranger.audit.queue.AuditBatchQueue.runLogAudit(AuditBatchQueue.java:309)
>   at 
> org.apache.ranger.audit.queue.AuditBatchQueue.run(AuditBatchQueue.java:215)
>   at java.base/java.lang.Thread.run(Thread.java:1583)
> Caused by: java.lang.ClassNotFoundException: 
> org.eclipse.jetty.client.util.SPNEGOAuthentication
>   at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
>   at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:593)
>   at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)
>   ... 5 common frames omitted {code}
> As you can see ranger-audit depends on solr client which depends on jetty 
> client.
> The problem is that solr client class use 
> org.eclipse.jetty.client.util.SPNEGOAuthentication - 
> [https://github.infra.cloudera.com/CDH/solr/blob/solr9-master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/Krb5HttpClientBuilder.java#L46]
> However in case jetty 12.x line, this class is moved to another package: 
> [https://github.com/jetty/jetty.project/commit/a1c5cefd0d5657df04e5364cca9315aa4e2a1aef]
>  
> So the problem exist, since jetty version upgraded to 12
> h2. Proposed solution
> Sadly there is no available solr client (or ranger client), which haven't had 
> this dependency. The only solution what I found (and propose in my pr) is to 
> override jetty version in case of ranger plugins to jetty line 11, where this 
> class is not moved. I tested it on my environment and the audit logging to 
> ranger worked well with that version.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (NIFI-12770) Deprecate Apache Ranger Integration for Removal

2024-02-12 Thread Joe Witt (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17816719#comment-17816719
 ] 

Joe Witt commented on NIFI-12770:
-

[~dstiegli1]We set fix versions on things generally when we're merging them.

> Deprecate Apache Ranger Integration for Removal
> ---
>
> Key: NIFI-12770
> URL: https://issues.apache.org/jira/browse/NIFI-12770
> Project: Apache NiFi
>  Issue Type: Task
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Minor
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Apache Ranger integration should be deprecated on the support branch for 
> subsequent removal on the main branch due to incompatibilities with Jetty 12 
> and related libraries. The Ranger plugins require Jetty 9, which was marked 
> as [End of Community 
> Support|https://github.com/jetty/jetty.project/issues/7958] in June 2022.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (NIFI-12781) Remove file size from UPLOAD provenance event

2024-02-12 Thread Joe Witt (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17816717#comment-17816717
 ] 

Joe Witt commented on NIFI-12781:
-

Lehel can you elaborate more on what the purpose of this JIRA is?

The subject says you want to remove the file size.  The body says the filesize 
is the primary concern.

The point of provenance data is to give an accurate accounting of what was done 
to/for data and where data came from or went to. What is the proposed outcome 
of this change?

Thanks

> Remove file size from UPLOAD provenance event
> -
>
> Key: NIFI-12781
> URL: https://issues.apache.org/jira/browse/NIFI-12781
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Lehel Boér
>Assignee: Lehel Boér
>Priority: Major
>
> Introducing a file location in the UPLOAD provenance event adds complexity, 
> particularly in regard to modifications in the FileResourceService. Given 
> that the primary concern is displaying file size in the Status History, for 
> now it's advisable to remove the ProvenanceFileResource class. Since the 
> UPLOAD event hasn't been utilized, there are no backward compatibility issues.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (NIFI-12772) Remote poll batch size not exposed for ListSFTP

2024-02-09 Thread Joe Witt (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17816245#comment-17816245
 ] 

Joe Witt commented on NIFI-12772:
-

The challenge here is whether the underlying library makes it possible to 
control that.  If it does then great.  If not then...not sure how to solve.  A 
similar difficulty existed back in the day for ListFile or equivalent cases.  
Java itself didnt expose a way to control this.  You could bring down a system 
just by making a listing call in a large directory.  New IO mechanisms resolved 
that scenario but SFTP is a different animal.

Love the idea just need to make sure the library exposes that power.

> Remote poll batch size not exposed for ListSFTP
> ---
>
> Key: NIFI-12772
> URL: https://issues.apache.org/jira/browse/NIFI-12772
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.25.0
>Reporter: Tom Brisland
>Priority: Minor
>
> I was planning on adding support for a batch size in ListSFTP due to some 
> issues we're seeing with a large number of files on an SFTP server.
> Thankfully, it seems that REMOTE_POLL_BATCH_SIZE is already supported in all 
> the FTP processor variations, just not exposed in ListSFTP.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (NIFI-6344) Add Failure Relationship to UpdateAttribute

2024-02-09 Thread Joe Witt (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-6344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17816244#comment-17816244
 ] 

Joe Witt commented on NIFI-6344:


Given the new capabilities for migrating configs in NiFi 2.0 we can fix this.

Add a property to UpdateAttribute that is 'Failure Strategy' and the options 
are 'rollback' or 'route to failure'.  If that property is set with rollback it 
behaves like it does now and I recommend that remain the default.  If that 
property is set to 'route to failure' then we add a relationship which needs to 
be set which is of course called 'failure'.  For flows being migrated from a 
version before this behavior was available to a version that has this 
capability we just set the value of this parameter to our default.

This lets existing flows migrate over just fine.  It lets us give users a 
failure path for the cases they want one.  It lets us keep the vast majority of 
flows and uses of this where failure is not relevant stay clean.  And it 
handles migration.

The processor needs to be updated to catch the exceptions and then follow this 
logic.  Today it just lets it fly to the framework which causes the processor 
to yield and penalizes the flowfile for the default time.  When now catching 
the problem we should just avoid yielding and instead penalize the specific 
offending flowfile which lets everything else operate super fast.

Thanks to Mark Payne for the chat on this.

> Add Failure Relationship to UpdateAttribute
> ---
>
> Key: NIFI-6344
> URL: https://issues.apache.org/jira/browse/NIFI-6344
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 1.9.2
>Reporter: Peter Wicks
>Assignee: Peter Wicks
>Priority: Minor
>  Labels: attribute, backwards-compatibility, expression-language, 
> routing
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> EL makes it possible for an UpdateAttribute processor to fail. When this 
> happens the FlowFile is rolled back, and there is no way to route it to 
> handle the failure automatically.
> Considerations:
> UpdateAttribute is used in probably all but the simplest of flows, thus any 
> change made to support a failure relationship must be handled delicately. The 
> goal of this change is for users to have no change in functionality unless 
> they specifically configure it.
> Proposal: 
> It was proposed on the Slack channel to create the failure relationship, but 
> default it to auto-terminate. This is a good start, but without further work 
> would result in a change in functionality. I propose that we will default to 
> auto-terminate, but also detect this behavior in the code. If the Failure 
> relationship is set to auto-terminate then we will rollback the transaction.
> The only downside I see with this is you can't actually auto-terminate 
> Failures without the addition of another property, such as Failure Behavior: 
> Route to Failure and Rollback options.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Resolved] (NIFI-5448) Failed EL date parsing live-locks processors without a failure relationship

2024-02-09 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-5448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt resolved NIFI-5448.

Resolution: Duplicate

Will address the fundamental concern in this other jira

> Failed EL date parsing live-locks processors without a failure relationship
> ---
>
> Key: NIFI-5448
> URL: https://issues.apache.org/jira/browse/NIFI-5448
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: David Koster
>Assignee: Mike Thomsen
>Priority: Major
>
> Processors that utilize the Expression Language need to always present a 
> failure relationship.
> If a processor with only a success relationship, for example UpdateAttribute, 
> utilizes the expression language to perform type coercion to a date and 
> fails, the processor will be unable to dispose of the FlowFile and remain 
> blocked indefinitely.
> Recreation flow:
> GenerateFlowFile -> Update Attribute #1 -> Update Attribute #2 -> Anything
> Update Attribute #1 - test = "Hello World"
> Update Attribute #2 - test = ${test:toDate('-MM-dd')}
>  
> Generates an IllegalAttributeException on UpdateAttribute.
>  
> The behavior should match numerical type coercion and silently skip the 
> processing or offer failure relationships on processors supporting EL



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (NIFI-12765) Nifi and nifi registry ranger audit is broken

2024-02-09 Thread Joe Witt (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17816227#comment-17816227
 ] 

Joe Witt commented on NIFI-12765:
-

This JIRA seems strongly related to NIFI-12738 and should probably be combined.

The comments on that JIRA apply here as well 
https://issues.apache.org/jira/browse/NIFI-12738

Based on my recent reading it appears Ranger depends on Jetty 9 so it is 
actually surprising Jetty 10 or 11 works for it in any capacity but notably we 
clearly seem to have no such tests/validations for this and it is not being 
maintained in our apache nifi codebase. Also notably Jetty 9, 10, 11 are all 
end of life from a community support point of view.   My recommendations on the 
other JIRA apply here as well. 

> Nifi and nifi registry ranger audit is broken
> -
>
> Key: NIFI-12765
> URL: https://issues.apache.org/jira/browse/NIFI-12765
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 2.0.0-M2, 2.0.0
>Reporter: Zoltán Kornél Török
>Assignee: Zoltán Kornél Török
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> h3. Bug description
> Currently ranger plugins are not reporting audit events into ranger.
> h2. Investigation
> In the nifi log I found the following ("classic") NoClassDefFoundError:
> {code:java}
> ERROR org.apache.ranger.audit.destination.SolrAuditDestination: Can't connect 
> to Solr server. 
> ZooKeepers=cfm-oudjal-dd-master0.cfm-5pax.svbr-nqvp.int.cldr.work:2181/solr-infrajava.lang.NoClassDefFoundError:
>  org/eclipse/jetty/client/util/SPNEGOAuthentication
>   at 
> org.apache.ranger.audit.destination.SolrAuditDestination.connect(SolrAuditDestination.java:168)
>   at 
> org.apache.ranger.audit.destination.SolrAuditDestination.log(SolrAuditDestination.java:227)
>   at 
> org.apache.ranger.audit.queue.AuditBatchQueue.runLogAudit(AuditBatchQueue.java:309)
>   at 
> org.apache.ranger.audit.queue.AuditBatchQueue.run(AuditBatchQueue.java:215)
>   at java.base/java.lang.Thread.run(Thread.java:1583)
> Caused by: java.lang.ClassNotFoundException: 
> org.eclipse.jetty.client.util.SPNEGOAuthentication
>   at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
>   at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:593)
>   at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)
>   ... 5 common frames omitted {code}
> As you can see ranger-audit depends on solr client which depends on jetty 
> client.
> The problem is that solr client class use 
> org.eclipse.jetty.client.util.SPNEGOAuthentication - 
> [https://github.infra.cloudera.com/CDH/solr/blob/solr9-master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/Krb5HttpClientBuilder.java#L46]
> However in case jetty 12.x line, this class is moved to another package: 
> [https://github.com/jetty/jetty.project/commit/a1c5cefd0d5657df04e5364cca9315aa4e2a1aef]
>  
> So the problem exist, since jetty version upgraded to 12
> h2. Proposed solution
> Sadly there is no available solr client (or ranger client), which haven't had 
> this dependency. The only solution what I found (and propose in my pr) is to 
> override jetty version in case of ranger plugins to jetty line 11, where this 
> class is not moved. I tested it on my environment and the audit logging to 
> ranger worked well with that version.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12769) Update Copyright for 2024

2024-02-09 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-12769:

Fix Version/s: 2.0.0
   1.26.0
   Resolution: Fixed
   Status: Resolved  (was: Patch Available)

> Update Copyright for 2024
> -
>
> Key: NIFI-12769
> URL: https://issues.apache.org/jira/browse/NIFI-12769
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Peter Turcsanyi
>Assignee: Peter Turcsanyi
>Priority: Major
> Fix For: 2.0.0, 1.26.0
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (NIFI-12744) Solr processors do not work with Nifi 2 M2

2024-02-09 Thread Joe Witt (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17816181#comment-17816181
 ] 

Joe Witt commented on NIFI-12744:
-

The Apache SOLR community does appear to have an initial plan being formed to 
enable moving to Jetty 12 and thus Java 17 which also likely means Spring 
6/etc.. so we might be able to restore the SOLR integration quite soon in that 
case.

https://issues.apache.org/jira/browse/SOLR-17069

> Solr processors do not work with Nifi 2 M2
> --
>
> Key: NIFI-12744
> URL: https://issues.apache.org/jira/browse/NIFI-12744
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Andreas Koch
>Priority: Critical
>
> *steps to reproduce*
>  * we launched Nifi in Docker
>  * build the nar file for nifi-solr-processor
>  * deploy the nar file to nifi
>  * Configure Query Solr Processor for cloude
>  * execute query
>  
> {code:java}
> // code placeholder2024-02-06 08:47:28,886 ERROR [Timer-Driven Process 
> Thread-4] o.apache.nifi.processors.solr.QuerySolr 
> QuerySolr[id=7d932b59-018d-1000-404a-2744f73f9f8c] Failed to properly 
> initialize Processor. If still scheduled to run, NiFi will attempt to 
> initialize and run the Processor again after the 'Administrative Yield 
> Duration' has elapsed. Failure is due to 
> java.lang.IncompatibleClassChangeError: class 
> org.eclipse.jetty.io.ArrayRetainableByteBufferPool$RetainedBucket has 
> interface org.eclipse.jetty.util.Pool as super class 
> java.lang.IncompatibleClassChangeError: class 
> org.eclipse.jetty.io.ArrayRetainableByteBufferPool$RetainedBucket has 
> interface org.eclipse.jetty.util.Pool as super class         at 
> java.base/java.lang.ClassLoader.defineClass1(Native Method)         at 
> java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1027)         at 
> java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:150)
>          at 
> java.base/java.net.URLClassLoader.defineClass(URLClassLoader.java:524)        
>  at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:427)         
> at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:421)         
> at 
> java.base/java.security.AccessController.doPrivileged(AccessController.java:714)
>          at 
> java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:420)         
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:593)         at 
> java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)         at 
> org.eclipse.jetty.io.ArrayRetainableByteBufferPool.(ArrayRetainableByteBufferPool.java:148)
>          at 
> org.eclipse.jetty.io.ArrayRetainableByteBufferPool.(ArrayRetainableByteBufferPool.java:95)
>          at 
> org.eclipse.jetty.io.MappedByteBufferPool$Retained.(MappedByteBufferPool.java:317)
>          at 
> org.eclipse.jetty.io.MappedByteBufferPool.newRetainableByteBufferPool(MappedByteBufferPool.java:140)
>          at 
> org.eclipse.jetty.io.AbstractByteBufferPool.(AbstractByteBufferPool.java:68)
>          at 
> org.eclipse.jetty.io.MappedByteBufferPool.(MappedByteBufferPool.java:133)
>          at 
> org.eclipse.jetty.io.MappedByteBufferPool.(MappedByteBufferPool.java:89)
>          at 
> org.eclipse.jetty.io.MappedByteBufferPool.(MappedByteBufferPool.java:77)
>          at org.eclipse.jetty.client.HttpClient.doStart(HttpClient.java:206)  
>        at 
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)
>          at 
> org.apache.solr.client.solrj.impl.Http2SolrClient.createHttpClient(Http2SolrClient.java:288)
>          at 
> org.apache.solr.client.solrj.impl.Http2SolrClient.(Http2SolrClient.java:180)
>          at 
> org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.build(Http2SolrClient.java:1099)
>          at 
> org.apache.nifi.processors.solr.SolrUtils.createSolrClient(SolrUtils.java:201)
>          at 
> org.apache.nifi.processors.solr.SolrProcessor.createSolrClient(SolrProcessor.java:153)
>          at 
> org.apache.nifi.processors.solr.SolrProcessor.onScheduled(SolrProcessor.java:79)
>          at 
> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
>          at java.base/java.lang.reflect.Method.invoke(Method.java:580)        
>  at 
> org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:145)
>          at 
> org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:133)
>          at 
> org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:78)
>          at 
> org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:55)
>          at 
> org.apache.nifi.controller.StandardProcessorNode.lambda$initiateStart$10(StandardProcessorNode.java:1668)
>          

[jira] [Commented] (NIFI-12738) Nifi registry ranger integration is not working

2024-02-09 Thread Joe Witt (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17816179#comment-17816179
 ] 

Joe Witt commented on NIFI-12738:
-

Via JIRA [this|https://issues.apache.org/jira/browse/RANGER-4038] appears to be 
the most recent discussion on the topic and indeed it starts to touch on Spring 
versions/Jetty (via jakarta) versions etc... 

That was March 2023.  No other reflections found.  It sounds like they remain 
committed to Java 8 at this point which in turn has the same knock-on effects 
we had to consider for the NiFi community.

I recommend we pursue deprecation in 1.x/removal in 2.x and let this component 
by maintained/operated by downstream builders of NiFi.

> Nifi registry ranger integration is not working
> ---
>
> Key: NIFI-12738
> URL: https://issues.apache.org/jira/browse/NIFI-12738
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: NiFi Registry
>Affects Versions: 2.0.0-M2, 2.0.0
>Reporter: Zoltán Kornél Török
>Priority: Critical
>
> h1. Bug description
> I want to try recently released nifi 2 m2 release in an environment,
> where we use ranger. Nifi registry ui was not able to load and I got the 
> following error in the log:
> {code:java}
> Caused by: 
> org.apache.nifi.registry.security.authorization.AuthorizerFactoryException: 
> Failed to construct Authorizer.
> at 
> org.apache.nifi.registry.security.authorization.AuthorizerFactory.getAuthorizer(AuthorizerFactory.java:267)
> at 
> org.apache.nifi.registry.security.authorization.AuthorizerFactory$$SpringCGLIB$$0.CGLIB$getAuthorizer$4()
> at 
> org.apache.nifi.registry.security.authorization.AuthorizerFactory$$SpringCGLIB$$FastClass$$1.invoke()
> at 
> org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:258)
> at 
> org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331)
> at 
> org.apache.nifi.registry.security.authorization.AuthorizerFactory$$SpringCGLIB$$0.getAuthorizer()
> at 
> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
> at java.base/java.lang.reflect.Method.invoke(Method.java:580)
> at 
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:351)
> at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)
> at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> at 
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:765)
> at 
> org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123)
> at 
> org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:385)
> at 
> org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
> at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
> at 
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:765)
> at 
> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:717)
> at 
> org.apache.nifi.registry.security.authorization.AuthorizerFactory$$SpringCGLIB$$1.getAuthorizer()
> at 
> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
> at java.base/java.lang.reflect.Method.invoke(Method.java:580)
> at 
> org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:140)
> ... 89 common frames omitted
> Caused by: 
> org.apache.nifi.registry.security.exception.SecurityProviderCreationException:
>  Error creating RangerBasePlugin
> at 
> org.apache.nifi.registry.ranger.RangerAuthorizer.onConfigured(RangerAuthorizer.java:178)
> at 
> org.apache.nifi.registry.security.authorization.AuthorizerFactory$ManagedAuthorizerWrapper.onConfigured(AuthorizerFactory.java:817)
> at 
> org.apache.nifi.registry.security.authorization.AuthorizerFactory.getAuthorizer(AuthorizerFactory.java:260)
> ... 110 common frames omitted
> Caused by: java.lang.NoClassDefFoundError: javax/ws/rs/core/Cookie
> at java.base/java.lang.Class.forName0(Native Method)
> at java.base/java.lang.Class.forName(Class.java:421)
> at java.base/java.lang.Class.forName(Class.java:412)
> at 
> 

[jira] [Commented] (NIFI-12738) Nifi registry ranger integration is not working

2024-02-09 Thread Joe Witt (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17816178#comment-17816178
 ] 

Joe Witt commented on NIFI-12738:
-

Did some investigation into Ranger's mailing lists on the topic of Jetty and/or 
Spring.  They're on Jetty 9.x and Spring 5.x as of Nov which is the last 
reflection I see.  Previous attempts by dependabot to move to Jetty 10 were not 
pursued.  No sign of discussion there.

Will now look into JIRA or Git to see if there is any signs of changes there.

> Nifi registry ranger integration is not working
> ---
>
> Key: NIFI-12738
> URL: https://issues.apache.org/jira/browse/NIFI-12738
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: NiFi Registry
>Affects Versions: 2.0.0-M2, 2.0.0
>Reporter: Zoltán Kornél Török
>Priority: Critical
>
> h1. Bug description
> I want to try recently released nifi 2 m2 release in an environment,
> where we use ranger. Nifi registry ui was not able to load and I got the 
> following error in the log:
> {code:java}
> Caused by: 
> org.apache.nifi.registry.security.authorization.AuthorizerFactoryException: 
> Failed to construct Authorizer.
> at 
> org.apache.nifi.registry.security.authorization.AuthorizerFactory.getAuthorizer(AuthorizerFactory.java:267)
> at 
> org.apache.nifi.registry.security.authorization.AuthorizerFactory$$SpringCGLIB$$0.CGLIB$getAuthorizer$4()
> at 
> org.apache.nifi.registry.security.authorization.AuthorizerFactory$$SpringCGLIB$$FastClass$$1.invoke()
> at 
> org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:258)
> at 
> org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331)
> at 
> org.apache.nifi.registry.security.authorization.AuthorizerFactory$$SpringCGLIB$$0.getAuthorizer()
> at 
> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
> at java.base/java.lang.reflect.Method.invoke(Method.java:580)
> at 
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:351)
> at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)
> at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> at 
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:765)
> at 
> org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123)
> at 
> org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:385)
> at 
> org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
> at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
> at 
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:765)
> at 
> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:717)
> at 
> org.apache.nifi.registry.security.authorization.AuthorizerFactory$$SpringCGLIB$$1.getAuthorizer()
> at 
> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
> at java.base/java.lang.reflect.Method.invoke(Method.java:580)
> at 
> org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:140)
> ... 89 common frames omitted
> Caused by: 
> org.apache.nifi.registry.security.exception.SecurityProviderCreationException:
>  Error creating RangerBasePlugin
> at 
> org.apache.nifi.registry.ranger.RangerAuthorizer.onConfigured(RangerAuthorizer.java:178)
> at 
> org.apache.nifi.registry.security.authorization.AuthorizerFactory$ManagedAuthorizerWrapper.onConfigured(AuthorizerFactory.java:817)
> at 
> org.apache.nifi.registry.security.authorization.AuthorizerFactory.getAuthorizer(AuthorizerFactory.java:260)
> ... 110 common frames omitted
> Caused by: java.lang.NoClassDefFoundError: javax/ws/rs/core/Cookie
> at java.base/java.lang.Class.forName0(Native Method)
> at java.base/java.lang.Class.forName(Class.java:421)
> at java.base/java.lang.Class.forName(Class.java:412)
> at 
> org.apache.ranger.plugin.policyengine.RangerPluginContext.createAdminClient(RangerPluginContext.java:96)
> at 
> 

[jira] [Commented] (NIFI-12744) Solr processors do not work with Nifi 2 M2

2024-02-09 Thread Joe Witt (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17816177#comment-17816177
 ] 

Joe Witt commented on NIFI-12744:
-

Thanks for flagging/finding this issue.  We need to investigate the plans of 
Solr to move off these now end of life versions of Jetty (and perhaps need to 
vet other libraries as well in use).  If there is no near term plan we need to 
deprecate our Solr processors in the 1.x line and remove them in the NiFi 2.x 
line.  We could restore them later should the updates end up happening but we 
cannot/should not run support these end of life variants as the 
dependency/vulnerability management then is highly problematic.  Nothing of 
course stops a downstream user of nifi maintaining them as they wish and 
honoring them in their distro.  Needs investigation similar to what I'm looking 
into for Apache Ranger right now.

> Solr processors do not work with Nifi 2 M2
> --
>
> Key: NIFI-12744
> URL: https://issues.apache.org/jira/browse/NIFI-12744
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Andreas Koch
>Priority: Critical
>
> *steps to reproduce*
>  * we launched Nifi in Docker
>  * build the nar file for nifi-solr-processor
>  * deploy the nar file to nifi
>  * Configure Query Solr Processor for cloude
>  * execute query
>  
> {code:java}
> // code placeholder2024-02-06 08:47:28,886 ERROR [Timer-Driven Process 
> Thread-4] o.apache.nifi.processors.solr.QuerySolr 
> QuerySolr[id=7d932b59-018d-1000-404a-2744f73f9f8c] Failed to properly 
> initialize Processor. If still scheduled to run, NiFi will attempt to 
> initialize and run the Processor again after the 'Administrative Yield 
> Duration' has elapsed. Failure is due to 
> java.lang.IncompatibleClassChangeError: class 
> org.eclipse.jetty.io.ArrayRetainableByteBufferPool$RetainedBucket has 
> interface org.eclipse.jetty.util.Pool as super class 
> java.lang.IncompatibleClassChangeError: class 
> org.eclipse.jetty.io.ArrayRetainableByteBufferPool$RetainedBucket has 
> interface org.eclipse.jetty.util.Pool as super class         at 
> java.base/java.lang.ClassLoader.defineClass1(Native Method)         at 
> java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1027)         at 
> java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:150)
>          at 
> java.base/java.net.URLClassLoader.defineClass(URLClassLoader.java:524)        
>  at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:427)         
> at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:421)         
> at 
> java.base/java.security.AccessController.doPrivileged(AccessController.java:714)
>          at 
> java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:420)         
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:593)         at 
> java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)         at 
> org.eclipse.jetty.io.ArrayRetainableByteBufferPool.(ArrayRetainableByteBufferPool.java:148)
>          at 
> org.eclipse.jetty.io.ArrayRetainableByteBufferPool.(ArrayRetainableByteBufferPool.java:95)
>          at 
> org.eclipse.jetty.io.MappedByteBufferPool$Retained.(MappedByteBufferPool.java:317)
>          at 
> org.eclipse.jetty.io.MappedByteBufferPool.newRetainableByteBufferPool(MappedByteBufferPool.java:140)
>          at 
> org.eclipse.jetty.io.AbstractByteBufferPool.(AbstractByteBufferPool.java:68)
>          at 
> org.eclipse.jetty.io.MappedByteBufferPool.(MappedByteBufferPool.java:133)
>          at 
> org.eclipse.jetty.io.MappedByteBufferPool.(MappedByteBufferPool.java:89)
>          at 
> org.eclipse.jetty.io.MappedByteBufferPool.(MappedByteBufferPool.java:77)
>          at org.eclipse.jetty.client.HttpClient.doStart(HttpClient.java:206)  
>        at 
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)
>          at 
> org.apache.solr.client.solrj.impl.Http2SolrClient.createHttpClient(Http2SolrClient.java:288)
>          at 
> org.apache.solr.client.solrj.impl.Http2SolrClient.(Http2SolrClient.java:180)
>          at 
> org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.build(Http2SolrClient.java:1099)
>          at 
> org.apache.nifi.processors.solr.SolrUtils.createSolrClient(SolrUtils.java:201)
>          at 
> org.apache.nifi.processors.solr.SolrProcessor.createSolrClient(SolrProcessor.java:153)
>          at 
> org.apache.nifi.processors.solr.SolrProcessor.onScheduled(SolrProcessor.java:79)
>          at 
> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
>          at java.base/java.lang.reflect.Method.invoke(Method.java:580)        
>  at 
> org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:145)
>          at 
> 

[jira] [Commented] (NIFI-12738) Nifi registry ranger integration is not working

2024-02-09 Thread Joe Witt (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17816174#comment-17816174
 ] 

Joe Witt commented on NIFI-12738:
-

Unless we have strong indications that the Apache Ranger community plans to 
update to the latest of these key components soon then for the reasons noted 
above - I would encourage we deprecate the ranger nar entirely in the 1.x line 
and remove from the 2.x line.  Usage/maintenance of the Ranger nar can live on 
in downstream users of this component.   

We are facing strong counter currents as it relates to open source components
* Spring, Jetty, and many others moving forward to dependencies on Java 11, 17 
and newer which has led us to move forward in NiFi to Java 21 
* At the same time the vulnerability landscape for dependencies is not slowing 
down. 
* Meanwhile, there are many vendors that have to honor older Java version 
obligations such as Java 8 that make them updating their components something 
that cannot be done as quickly.

We cannot reasonably optimize for all three cases at once but we can from a 
nifi community prioritize the first two. 

> Nifi registry ranger integration is not working
> ---
>
> Key: NIFI-12738
> URL: https://issues.apache.org/jira/browse/NIFI-12738
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: NiFi Registry
>Affects Versions: 2.0.0-M2, 2.0.0
>Reporter: Zoltán Kornél Török
>Priority: Critical
>
> h1. Bug description
> I want to try recently released nifi 2 m2 release in an environment,
> where we use ranger. Nifi registry ui was not able to load and I got the 
> following error in the log:
> {code:java}
> Caused by: 
> org.apache.nifi.registry.security.authorization.AuthorizerFactoryException: 
> Failed to construct Authorizer.
> at 
> org.apache.nifi.registry.security.authorization.AuthorizerFactory.getAuthorizer(AuthorizerFactory.java:267)
> at 
> org.apache.nifi.registry.security.authorization.AuthorizerFactory$$SpringCGLIB$$0.CGLIB$getAuthorizer$4()
> at 
> org.apache.nifi.registry.security.authorization.AuthorizerFactory$$SpringCGLIB$$FastClass$$1.invoke()
> at 
> org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:258)
> at 
> org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331)
> at 
> org.apache.nifi.registry.security.authorization.AuthorizerFactory$$SpringCGLIB$$0.getAuthorizer()
> at 
> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
> at java.base/java.lang.reflect.Method.invoke(Method.java:580)
> at 
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:351)
> at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)
> at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> at 
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:765)
> at 
> org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123)
> at 
> org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:385)
> at 
> org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
> at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
> at 
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:765)
> at 
> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:717)
> at 
> org.apache.nifi.registry.security.authorization.AuthorizerFactory$$SpringCGLIB$$1.getAuthorizer()
> at 
> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
> at java.base/java.lang.reflect.Method.invoke(Method.java:580)
> at 
> org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:140)
> ... 89 common frames omitted
> Caused by: 
> org.apache.nifi.registry.security.exception.SecurityProviderCreationException:
>  Error creating RangerBasePlugin
> at 
> org.apache.nifi.registry.ranger.RangerAuthorizer.onConfigured(RangerAuthorizer.java:178)
> at 
> org.apache.nifi.registry.security.authorization.AuthorizerFactory$ManagedAuthorizerWrapper.onConfigured(AuthorizerFactory.java:817)
> 

[jira] [Commented] (NIFI-12753) Update various Github Actions versions for Github CI

2024-02-07 Thread Joe Witt (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17815375#comment-17815375
 ] 

Joe Witt commented on NIFI-12753:
-

https://github.com/apache/nifi/actions/runs/7817671166

> Update various Github Actions versions for Github CI
> 
>
> Key: NIFI-12753
> URL: https://issues.apache.org/jira/browse/NIFI-12753
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joe Witt
>Assignee: Joe Witt
>Priority: Major
>
> Shows some deprecation warnings related to github actions.  We need to update 
> our workflows to use latest action versions



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (NIFI-12753) Update various Github Actions versions for Github CI

2024-02-07 Thread Joe Witt (Jira)
Joe Witt created NIFI-12753:
---

 Summary: Update various Github Actions versions for Github CI
 Key: NIFI-12753
 URL: https://issues.apache.org/jira/browse/NIFI-12753
 Project: Apache NiFi
  Issue Type: Improvement
Reporter: Joe Witt
Assignee: Joe Witt


Shows some deprecation warnings related to github actions.  We need to update 
our workflows to use latest action versions



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Resolved] (NIFI-12708) UnpackContent should allow the user to specify a character set to apply in reading paths and filenames

2024-02-07 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt resolved NIFI-12708.
-
Resolution: Fixed

> UnpackContent should allow the user to specify a character set to apply in 
> reading paths and filenames
> --
>
> Key: NIFI-12708
> URL: https://issues.apache.org/jira/browse/NIFI-12708
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joe Witt
>Assignee: Umar Hussain
>Priority: Major
> Fix For: 2.0.0
>
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> https://apachenifi.slack.com/archives/C0L9VCD47/p1706716977280569
> Timon Faerber
>   1 hour ago
> I am currently struggling with an encoding problem for unzipped files.
> The following:
> I have a .zip in my content, which Im not aware of how it was created (dont 
> know Character Set).
> Then I use UnpackContent processor.
> The path (folder) and filename is after that for unpacked files not encoded 
> in UTF-8 and the characters are output as ?.
> I have already tried this solutions like 
> https://community.cloudera.com/t5/Support-Questions/Unable-to-write-a-file-with-Chinese-Characters-filename-in/m-p/177183,
>  for example, but it does not work for me.
> Does anyone know another solution?
> Joe Witt
>   43 minutes ago
> If you take nifi out of the equation and just unpack the zip using a command 
> line tool - does it see the paths/names correctly?
> Joe Witt
>   43 minutes ago
> is there a sample zip you can share which has this problem?
> Umar Hussain
>   9 minutes ago
> We tried it with unzip on Linux and if we give parameter -O Cp347 the German 
> characters ü ä ö in the path appear correctly in output.
> But a simple unzip command also doesn't produce correct paths in output.
> Joe Witt
>   5 minutes ago
> Interesting.  So if you tell the zip program the encoding is cp347 the output 
> appears correct.  otherwise it is incorrect yes?
> New
> Umar Hussain
>   3 minutes ago
> Yes, I think its the encoding of zip and the zip was created on a windows 
> machine and on Linux it's by default a different one.
> The processor current implementation takes the platforms default encoding
> Joe Witt
>   3 minutes ago
> Yeah this is probably a good summary of behavior we need to consider.  
> https://stackoverflow.com/questions/13261347/correctly-decoding-zip-entry-file-names-cp437-utf-8-or
> Stack OverflowStack Overflow
> Correctly decoding zip entry file names -- CP437, UTF-8 or?
> I recently wrote a zip file I/O library called zipzap, but I'm struggling 
> with correctly decoding zip entry file names from arbitrary zip files.
> Now, the PKWARE spec states:
> D.1 The ZIP format ...
> Joe Witt
>   2 minutes ago
> My guess is we need to allow the user to override the default behavior by 
> selecting the character set we'll read the filenames/paths as in some cases 
> of reading legacy app created zips



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12708) UnpackContent should allow the user to specify a character set to apply in reading paths and filenames

2024-02-07 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-12708:

Fix Version/s: 2.0.0

> UnpackContent should allow the user to specify a character set to apply in 
> reading paths and filenames
> --
>
> Key: NIFI-12708
> URL: https://issues.apache.org/jira/browse/NIFI-12708
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joe Witt
>Assignee: Umar Hussain
>Priority: Major
> Fix For: 2.0.0
>
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> https://apachenifi.slack.com/archives/C0L9VCD47/p1706716977280569
> Timon Faerber
>   1 hour ago
> I am currently struggling with an encoding problem for unzipped files.
> The following:
> I have a .zip in my content, which Im not aware of how it was created (dont 
> know Character Set).
> Then I use UnpackContent processor.
> The path (folder) and filename is after that for unpacked files not encoded 
> in UTF-8 and the characters are output as ?.
> I have already tried this solutions like 
> https://community.cloudera.com/t5/Support-Questions/Unable-to-write-a-file-with-Chinese-Characters-filename-in/m-p/177183,
>  for example, but it does not work for me.
> Does anyone know another solution?
> Joe Witt
>   43 minutes ago
> If you take nifi out of the equation and just unpack the zip using a command 
> line tool - does it see the paths/names correctly?
> Joe Witt
>   43 minutes ago
> is there a sample zip you can share which has this problem?
> Umar Hussain
>   9 minutes ago
> We tried it with unzip on Linux and if we give parameter -O Cp347 the German 
> characters ü ä ö in the path appear correctly in output.
> But a simple unzip command also doesn't produce correct paths in output.
> Joe Witt
>   5 minutes ago
> Interesting.  So if you tell the zip program the encoding is cp347 the output 
> appears correct.  otherwise it is incorrect yes?
> New
> Umar Hussain
>   3 minutes ago
> Yes, I think its the encoding of zip and the zip was created on a windows 
> machine and on Linux it's by default a different one.
> The processor current implementation takes the platforms default encoding
> Joe Witt
>   3 minutes ago
> Yeah this is probably a good summary of behavior we need to consider.  
> https://stackoverflow.com/questions/13261347/correctly-decoding-zip-entry-file-names-cp437-utf-8-or
> Stack OverflowStack Overflow
> Correctly decoding zip entry file names -- CP437, UTF-8 or?
> I recently wrote a zip file I/O library called zipzap, but I'm struggling 
> with correctly decoding zip entry file names from arbitrary zip files.
> Now, the PKWARE spec states:
> D.1 The ZIP format ...
> Joe Witt
>   2 minutes ago
> My guess is we need to allow the user to override the default behavior by 
> selecting the character set we'll read the filenames/paths as in some cases 
> of reading legacy app created zips



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (NIFI-12708) UnpackContent should allow the user to specify a character set to apply in reading paths and filenames

2024-01-31 Thread Joe Witt (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17812833#comment-17812833
 ] 

Joe Witt commented on NIFI-12708:
-

Looks like a straightforward solution here is to have a validator that ensures 
a user supplied character set is from the list of character sets the JVM 
understands.  Then pass that user supplied value into the Zip stream reader and 
it will take care of the filename/path reading from there properly.  Should be 
pretty straight forward.

> UnpackContent should allow the user to specify a character set to apply in 
> reading paths and filenames
> --
>
> Key: NIFI-12708
> URL: https://issues.apache.org/jira/browse/NIFI-12708
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joe Witt
>Priority: Major
>
> https://apachenifi.slack.com/archives/C0L9VCD47/p1706716977280569
> Timon Faerber
>   1 hour ago
> I am currently struggling with an encoding problem for unzipped files.
> The following:
> I have a .zip in my content, which Im not aware of how it was created (dont 
> know Character Set).
> Then I use UnpackContent processor.
> The path (folder) and filename is after that for unpacked files not encoded 
> in UTF-8 and the characters are output as ?.
> I have already tried this solutions like 
> https://community.cloudera.com/t5/Support-Questions/Unable-to-write-a-file-with-Chinese-Characters-filename-in/m-p/177183,
>  for example, but it does not work for me.
> Does anyone know another solution?
> Joe Witt
>   43 minutes ago
> If you take nifi out of the equation and just unpack the zip using a command 
> line tool - does it see the paths/names correctly?
> Joe Witt
>   43 minutes ago
> is there a sample zip you can share which has this problem?
> Umar Hussain
>   9 minutes ago
> We tried it with unzip on Linux and if we give parameter -O Cp347 the German 
> characters ü ä ö in the path appear correctly in output.
> But a simple unzip command also doesn't produce correct paths in output.
> Joe Witt
>   5 minutes ago
> Interesting.  So if you tell the zip program the encoding is cp347 the output 
> appears correct.  otherwise it is incorrect yes?
> New
> Umar Hussain
>   3 minutes ago
> Yes, I think its the encoding of zip and the zip was created on a windows 
> machine and on Linux it's by default a different one.
> The processor current implementation takes the platforms default encoding
> Joe Witt
>   3 minutes ago
> Yeah this is probably a good summary of behavior we need to consider.  
> https://stackoverflow.com/questions/13261347/correctly-decoding-zip-entry-file-names-cp437-utf-8-or
> Stack OverflowStack Overflow
> Correctly decoding zip entry file names -- CP437, UTF-8 or?
> I recently wrote a zip file I/O library called zipzap, but I'm struggling 
> with correctly decoding zip entry file names from arbitrary zip files.
> Now, the PKWARE spec states:
> D.1 The ZIP format ...
> Joe Witt
>   2 minutes ago
> My guess is we need to allow the user to override the default behavior by 
> selecting the character set we'll read the filenames/paths as in some cases 
> of reading legacy app created zips



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (NIFI-12708) UnpackContent should allow the user to specify a character set to apply in reading paths and filenames

2024-01-31 Thread Joe Witt (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17812823#comment-17812823
 ] 

Joe Witt commented on NIFI-12708:
-

We do tell the zip library to leverage the unicode extra fields to determine 
how to read filenames.  But for some inputs the archive tool might not indicate 
this thus leaving the scenario whereby a user would need to indicate character 
set explicitly.

> UnpackContent should allow the user to specify a character set to apply in 
> reading paths and filenames
> --
>
> Key: NIFI-12708
> URL: https://issues.apache.org/jira/browse/NIFI-12708
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joe Witt
>Priority: Major
>
> https://apachenifi.slack.com/archives/C0L9VCD47/p1706716977280569
> Timon Faerber
>   1 hour ago
> I am currently struggling with an encoding problem for unzipped files.
> The following:
> I have a .zip in my content, which Im not aware of how it was created (dont 
> know Character Set).
> Then I use UnpackContent processor.
> The path (folder) and filename is after that for unpacked files not encoded 
> in UTF-8 and the characters are output as ?.
> I have already tried this solutions like 
> https://community.cloudera.com/t5/Support-Questions/Unable-to-write-a-file-with-Chinese-Characters-filename-in/m-p/177183,
>  for example, but it does not work for me.
> Does anyone know another solution?
> Joe Witt
>   43 minutes ago
> If you take nifi out of the equation and just unpack the zip using a command 
> line tool - does it see the paths/names correctly?
> Joe Witt
>   43 minutes ago
> is there a sample zip you can share which has this problem?
> Umar Hussain
>   9 minutes ago
> We tried it with unzip on Linux and if we give parameter -O Cp347 the German 
> characters ü ä ö in the path appear correctly in output.
> But a simple unzip command also doesn't produce correct paths in output.
> Joe Witt
>   5 minutes ago
> Interesting.  So if you tell the zip program the encoding is cp347 the output 
> appears correct.  otherwise it is incorrect yes?
> New
> Umar Hussain
>   3 minutes ago
> Yes, I think its the encoding of zip and the zip was created on a windows 
> machine and on Linux it's by default a different one.
> The processor current implementation takes the platforms default encoding
> Joe Witt
>   3 minutes ago
> Yeah this is probably a good summary of behavior we need to consider.  
> https://stackoverflow.com/questions/13261347/correctly-decoding-zip-entry-file-names-cp437-utf-8-or
> Stack OverflowStack Overflow
> Correctly decoding zip entry file names -- CP437, UTF-8 or?
> I recently wrote a zip file I/O library called zipzap, but I'm struggling 
> with correctly decoding zip entry file names from arbitrary zip files.
> Now, the PKWARE spec states:
> D.1 The ZIP format ...
> Joe Witt
>   2 minutes ago
> My guess is we need to allow the user to override the default behavior by 
> selecting the character set we'll read the filenames/paths as in some cases 
> of reading legacy app created zips



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (NIFI-12709) UnpackContent should save attributes from the zip entries as flowfile attributes where possible

2024-01-31 Thread Joe Witt (Jira)
Joe Witt created NIFI-12709:
---

 Summary: UnpackContent should save attributes from the zip entries 
as flowfile attributes where possible
 Key: NIFI-12709
 URL: https://issues.apache.org/jira/browse/NIFI-12709
 Project: Apache NiFi
  Issue Type: Improvement
Reporter: Joe Witt


In an email from Jan 31st to users list titled 'ExecuteStreamCommand failing to 
unzip incoming flowfiles'

Issue is that UnpackContent doesn't capture much useful metadata.  The user 
wants last modified date which is easily available, but also creator, creation 
time, and owner which are less obviously avaialble at least not consistently.  
But there is a concept of extra fields we can extract metadata from.  We have 
those same fields available from Tar files so it is natural users would also 
want these.  Given their names aren't standard though I see why Tar is the only 
one we currently say we support pulling those for.  If we at least captured the 
metadata we can then flow builders can use it in their flows as they wish 
whereas right now we dont expose that information.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (NIFI-12708) UnpackContent should allow the user to specify a character set to apply in reading paths and filenames

2024-01-31 Thread Joe Witt (Jira)
Joe Witt created NIFI-12708:
---

 Summary: UnpackContent should allow the user to specify a 
character set to apply in reading paths and filenames
 Key: NIFI-12708
 URL: https://issues.apache.org/jira/browse/NIFI-12708
 Project: Apache NiFi
  Issue Type: Improvement
Reporter: Joe Witt


https://apachenifi.slack.com/archives/C0L9VCD47/p1706716977280569

Timon Faerber
  1 hour ago
I am currently struggling with an encoding problem for unzipped files.
The following:
I have a .zip in my content, which Im not aware of how it was created (dont 
know Character Set).
Then I use UnpackContent processor.
The path (folder) and filename is after that for unpacked files not encoded in 
UTF-8 and the characters are output as ?.
I have already tried this solutions like 
https://community.cloudera.com/t5/Support-Questions/Unable-to-write-a-file-with-Chinese-Characters-filename-in/m-p/177183,
 for example, but it does not work for me.
Does anyone know another solution?


Joe Witt
  43 minutes ago
If you take nifi out of the equation and just unpack the zip using a command 
line tool - does it see the paths/names correctly?


Joe Witt
  43 minutes ago
is there a sample zip you can share which has this problem?


Umar Hussain
  9 minutes ago
We tried it with unzip on Linux and if we give parameter -O Cp347 the German 
characters ü ä ö in the path appear correctly in output.
But a simple unzip command also doesn't produce correct paths in output.


Joe Witt
  5 minutes ago
Interesting.  So if you tell the zip program the encoding is cp347 the output 
appears correct.  otherwise it is incorrect yes?
New


Umar Hussain
  3 minutes ago
Yes, I think its the encoding of zip and the zip was created on a windows 
machine and on Linux it's by default a different one.
The processor current implementation takes the platforms default encoding


Joe Witt
  3 minutes ago
Yeah this is probably a good summary of behavior we need to consider.  
https://stackoverflow.com/questions/13261347/correctly-decoding-zip-entry-file-names-cp437-utf-8-or

Stack OverflowStack Overflow
Correctly decoding zip entry file names -- CP437, UTF-8 or?
I recently wrote a zip file I/O library called zipzap, but I'm struggling with 
correctly decoding zip entry file names from arbitrary zip files.
Now, the PKWARE spec states:
D.1 The ZIP format ...


Joe Witt
  2 minutes ago
My guess is we need to allow the user to override the default behavior by 
selecting the character set we'll read the filenames/paths as in some cases of 
reading legacy app created zips



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Resolved] (NIFI-12681) Archetype should use NiFi version instead of version for nifi-standard-services-api-nar

2024-01-27 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt resolved NIFI-12681.
-
Fix Version/s: 2.0.0
   Resolution: Fixed

Thanks Bob.  +1 merged to main.

Back in the day we always specified the version due to perceived issues with 
maven dep resolution specifically during the release process but nowadays it is 
much better.  I think we just overlooked the archetype module.

> Archetype should use NiFi version instead of version for 
> nifi-standard-services-api-nar
> ---
>
> Key: NIFI-12681
> URL: https://issues.apache.org/jira/browse/NIFI-12681
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Tools and Build
>Affects Versions: 2.0.0
>Reporter: Bob Paulin
>Priority: Minor
> Fix For: 2.0.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The current nifi-processor-bundle-archetype sets the 
> nifi-standard-services-api-nar to ${version} which matches the processor 
> version instead of the version of NiFi being used.
>  
> Suggest either removing the version entirely (it should inherit from the 
> parent pom or set to ${nifiVersion} so it matches the version of NiFi being 
> used.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12676) HandleHttpRequest Servlet Not Registered

2024-01-25 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-12676:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

> HandleHttpRequest Servlet Not Registered
> 
>
> Key: NIFI-12676
> URL: https://issues.apache.org/jira/browse/NIFI-12676
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Critical
> Fix For: 2.0.0-M2
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Following recent upgrades to Jetty 12, the HandleHttpRequest Processor is not 
> registered the standard Servlet that it uses to handle HTTP requests.
> The Jetty ServletContextHandler should be used together with a root path 
> mapping to allow HandleHttpRequest to process requests according to 
> configured properties.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12657) Remove MiNiFi C2 Server Modules

2024-01-22 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-12657:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

+1 merged to main

> Remove MiNiFi C2 Server Modules
> ---
>
> Key: NIFI-12657
> URL: https://issues.apache.org/jira/browse/NIFI-12657
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: MiNiFi
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
> Fix For: 2.0.0-M2
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The MiNiFi C2 server modules contain partial functionality migrated from the 
> historical MiNiFi Java repository. NIFI-9428 described a general effort to 
> implement C2 capabilities, but the task to implement a C2 server reference 
> implementation in NIFI-9533 was not completed. Rather than continuing to 
> maintain the partial implementation, the existing MiNiFi C2 server modules 
> should be removed from the main branch as part of general baseline efforts 
> for NiFi 2.0.0.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (NIFI-12657) Remove MiNiFi C2 Server Modules

2024-01-22 Thread Joe Witt (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17809664#comment-17809664
 ] 

Joe Witt commented on NIFI-12657:
-

[~exceptionfactory] Thanks for capturing this and I agree with the assessment.  
It does not make sense to maintain a partial implementation especially for 
something that is meant as a reference implementation anyway.  The c2 api 
allows others to build from and if a group becomes well motivated can have a 
fully maintained implementation here as well but this current model this 
doesn't tie fully into the ci/cd/build processes does not add value.  Thanks!

> Remove MiNiFi C2 Server Modules
> ---
>
> Key: NIFI-12657
> URL: https://issues.apache.org/jira/browse/NIFI-12657
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: MiNiFi
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
>
> The MiNiFi C2 server modules contain partial functionality migrated from the 
> historical MiNiFi Java repository. NIFI-9428 described a general effort to 
> implement C2 capabilities, but the task to implement a C2 server reference 
> implementation in NIFI-9533 was not completed. Rather than continuing to 
> maintain the partial implementation, the existing MiNiFi C2 server modules 
> should be removed from the main branch as part of general baseline efforts 
> for NiFi 2.0.0.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (NIFI-12604) Add support to empty queues

2024-01-16 Thread Joe Witt (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17807349#comment-17807349
 ] 

Joe Witt commented on NIFI-12604:
-

[~markbean]Take a look at the epic/parent task this is part of 
https://issues.apache.org/jira/browse/NIFI-12400.  This is the new UI 
implementation

> Add support to empty queues
> ---
>
> Key: NIFI-12604
> URL: https://issues.apache.org/jira/browse/NIFI-12604
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Core UI
>Reporter: Matt Gilman
>Assignee: Matt Gilman
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Introduce actions for emptying a queue or all queues within a Process Group.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12606) Update Apache parent POM to version 31

2024-01-12 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-12606:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Update Apache parent POM to version 31
> --
>
> Key: NIFI-12606
> URL: https://issues.apache.org/jira/browse/NIFI-12606
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Tools and Build
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Minor
> Fix For: 1.25.0, 2.0.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The Apache parent POM (used at the root pom.xml in NiFi) has a new release, 
> version 31:
> https://github.com/apache/maven-apache-parent/releases/tag/apache-31
> This includes a number of dependency upgrades.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12607) Remove TIMED_WAIT doc for kernel 2.6

2024-01-12 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-12607:

Fix Version/s: 2.0.0
   Resolution: Fixed
   Status: Resolved  (was: Patch Available)

Thanks Andrew!

> Remove TIMED_WAIT doc for kernel 2.6
> 
>
> Key: NIFI-12607
> URL: https://issues.apache.org/jira/browse/NIFI-12607
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation  Website
>Affects Versions: 2.0.0-M1
>Reporter: Andrew Atwood
>Assignee: Andrew Atwood
>Priority: Trivial
> Fix For: 2.0.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Admin guide includes how to configure TIMED_WAIT for linux kernel 2.6, but no 
> up-to-date linux distro should still be using 2.6, as its support ended in 
> 2016 per 
> https://en.wikipedia.org/wiki/Linux_kernel_version_history#Releases_2.6.x.y



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (NIFI-12513) Regression in InvokeHTTP - Not a valid URL

2023-12-22 Thread Joe Witt (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799865#comment-17799865
 ] 

Joe Witt commented on NIFI-12513:
-

Code that is released under ASLv2 we can quite literally copy as-is in whole or 
part generally.  We do have some such code in our codebase today.  Anything 
considered under a Category-A license in ASF terms is fair game for this.  We 
just need to carry forward any copyright mechanisms and I think we continue the 
good form whereby we list in our notice code we have copied.

> Regression in InvokeHTTP - Not a valid URL
> --
>
> Key: NIFI-12513
> URL: https://issues.apache.org/jira/browse/NIFI-12513
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 2.0.0-M1, 1.24.0
>Reporter: Pierre Villard
>Assignee: Daniel Stieglitz
>Priority: Major
>
> On NiFi 1.24 and NiFi 2.0-M1, the below URL
> {code:java}
> https://en.wikipedia.org/w/api.php?action=query=recentchanges=json=user|comment|parsedcomment|timestamp|title|sizes|tags{code}
> Is no longer considered as valid but it was a valid one before.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Resolved] (NIFI-12532) Possible memory leak in ConnectionLoadBalanceServer

2023-12-21 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt resolved NIFI-12532.
-
Fix Version/s: 2.0.0
   Resolution: Fixed

> Possible memory leak in ConnectionLoadBalanceServer
> ---
>
> Key: NIFI-12532
> URL: https://issues.apache.org/jira/browse/NIFI-12532
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Stijn Caerts
>Priority: Major
> Fix For: 2.0.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The ConnectionLoadBalanceServer possibly contains a memory leak. 
> When analyzing a heap dump of a NiFi node in our clustered setup with version 
> 1.16.3, a large part of the used memory was related to the 
> {{ConnectionLoadBalanceServer}}.
> It had a retained size of 1,770,787,761 B (25.1%), with 106,710 
> {{CommunicationAction}}  objects taking up most of that size 1,770,787,608 B 
> (25.1%).
> I checked the code of the {{ConnectionLoadBalanceServer}} to try to 
> understand why it uses this much memory. This is when I noticed that 
> {{CommunicationAction}} objects are only removed when the 
> {{ConnectionLoadBalanceServer}} is stopped, creating a possible memory leak.
> Although we're not running the most recent version of NiFi, this is still 
> present in the current version of the code.
> https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/server/ConnectionLoadBalanceServer.java#L110



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (NIFI-12535) Error in expression language guide page

2023-12-21 Thread Joe Witt (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799644#comment-17799644
 ] 

Joe Witt commented on NIFI-12535:
-

Congrats and thanks for your first PR [~aethier]

> Error in expression language guide page
> ---
>
> Key: NIFI-12535
> URL: https://issues.apache.org/jira/browse/NIFI-12535
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation  Website
>Affects Versions: 2.0.0-M1, 1.24.0
>Reporter: Alex Ethier
>Assignee: Alex Ethier
>Priority: Minor
>  Labels: backport-needed
> Fix For: 1.25.0, 2.0.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> On the NiFi docs: [!https://nifi.apache.org/assets/images/nifi16.ico!Apache 
> NiFi Documentation|https://nifi.apache.org/project-documentation.html] in the 
> {{Expression Language Guide}} section in the {{PadRight}} sub section of the 
> document Table 14 is incorrectly named ??PadLeft Examples?? instead of 
> ??PadRight Examples?? and the last example in that table incorrectly says 
> padLeft when it should be padRight.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12535) Error in expression language guide page

2023-12-21 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-12535:

Fix Version/s: 1.25.0
   2.0.0
   Resolution: Fixed
   Status: Resolved  (was: Patch Available)

> Error in expression language guide page
> ---
>
> Key: NIFI-12535
> URL: https://issues.apache.org/jira/browse/NIFI-12535
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation  Website
>Affects Versions: 2.0.0-M1, 1.24.0
>Reporter: Alex Ethier
>Assignee: Alex Ethier
>Priority: Minor
>  Labels: backport-needed
> Fix For: 1.25.0, 2.0.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> On the NiFi docs: [!https://nifi.apache.org/assets/images/nifi16.ico!Apache 
> NiFi Documentation|https://nifi.apache.org/project-documentation.html] in the 
> {{Expression Language Guide}} section in the {{PadRight}} sub section of the 
> document Table 14 is incorrectly named ??PadLeft Examples?? instead of 
> ??PadRight Examples?? and the last example in that table incorrectly says 
> padLeft when it should be padRight.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12528) Funnels bug

2023-12-21 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-12528:

Fix Version/s: 2.0.0

> Funnels bug
> ---
>
> Key: NIFI-12528
> URL: https://issues.apache.org/jira/browse/NIFI-12528
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.23.2
> Environment: NiFi cluster
>Reporter: Giovanni
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: image001.png
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> I've notice a bug in our working cluster of 4 nodes.
> When two funnels are connected, with no other connections as themselves, like 
> in attached image, is not possible to delete them. 
> After trying to modify one of their connections, the Node where the action is 
> performed crashes, and will not turn on unless the entire cluster gets 
> restarted.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Resolved] (NIFI-12528) Funnels bug

2023-12-21 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt resolved NIFI-12528.
-
Resolution: Fixed

> Funnels bug
> ---
>
> Key: NIFI-12528
> URL: https://issues.apache.org/jira/browse/NIFI-12528
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.23.2
> Environment: NiFi cluster
>Reporter: Giovanni
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: image001.png
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> I've notice a bug in our working cluster of 4 nodes.
> When two funnels are connected, with no other connections as themselves, like 
> in attached image, is not possible to delete them. 
> After trying to modify one of their connections, the Node where the action is 
> performed crashes, and will not turn on unless the entire cluster gets 
> restarted.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12536) ParseDocument incorrectly converts byte array to String, result in text like b'...' instead of just ...

2023-12-21 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-12536:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

> ParseDocument incorrectly converts byte array to String, result in text like 
> b'...' instead of just ...
> ---
>
> Key: NIFI-12536
> URL: https://issues.apache.org/jira/browse/NIFI-12536
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 2.0.0-M1
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Trivial
> Fix For: 2.0.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Documents that are produced from ParseDocument use 
> {{{}str(flowfile.getContentsAsBytes(){}}}) when it should use 
> {{{}flowfile.getContentsAsBytes().decode('utf-8'){}}}.
> This results in text such as {{One Two Three}} to be produces as {{b'One Two 
> Three'}}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Comment Edited] (NIFI-12236) Improving fault tolerancy of the QuestDB backed metrics repository

2023-12-07 Thread Joe Witt (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17794312#comment-17794312
 ] 

Joe Witt edited comment on NIFI-12236 at 12/7/23 3:42 PM:
--

Regarding defaults - thank you.

API Changes: 
These should get the highest degree of scrutiny by reviewers and the most 
effort and thought by authors.  At this stage I do not understand the intention 
for the change.  It appears that concern is shared. If there is a 
non-API/specific to this implementation option that should be pursued instead.

Suggestion about the nar:
I am not aware of other cases where the optional implementation of a thing is 
the lighter option in terms of dependencies path.  In this case the optional 
implementation (QuestDB backed stats) is the heavier one in terms of 
dependency/operations.  By suggesting it be in its own nar I am not of the view 
it is a lot more work to be clear.  I could be wrong.  But the more work 
consideration is also not a strong one.  The 'it is already in there...' 
argument is better since well - that is true.  I'm just asking that the effort 
to do that be strongly considered because then those who would not use this 
option don't have to carry the component into deployment nor the potential 
vulnerabilities the libraries might bring.  But those that want it are good to 
go and can use it.  If it is baked into the framework nar this is not an option.

As evidence of my concern regarding the vulnerability it does give pause that 
the version of QuestDB was not considered in this PR to this point.  Keeping 
dependencies up to date is *everyones* responsibility.  

The NiFi 2.0 line is by the way super wildly close to being entirely free of 
static coding practice and dependency vulnerabilities (excluding hadoop related 
components) which is in itself a miracle.  Please ensure the dependencies in 
this PR are up to date.  On that note do we know what QuestDB version changes 
mean in terms of breaking changes, changes that might require users to do 
something to keep their state, etc..?  I ask because we learned our lessons the 
hard way here with H2.  H2 served us extremely well over the years but 
eventually it created some very complicated compelling events.

Anyway - I dont want to come off like this thing can't happen.  The root idea 
of this I think we all agree is useful. There is some debate on whether this 
implementation approach is desirable vs another but even that does not matter.  
He who does the work for a given implementation ultimately wins.  What I am 
asking then though is please make that implementation as narrow and specific as 
possible to give others choice on whether they are exposed to it.  That is the 
heart of the replies I am making.

Thanks


was (Author: joewitt):
Regarding defaults - thank you.

API Changes: 
These should get the highest degree of scrutiny by reviewers and the most 
effort and thought by authors.  At this stage I do not understand the intention 
for the change.  It appears that concern is shared. If there is a 
non-API/specific to this implementation option that should be pursued instead.

Suggestion about the nar:
I am not aware of other cases where the optional implementation of a thing is 
the light in terms of dependencies path.  In this case the optional 
implementation (QuestDB backed stats) is the heavier one in terms of 
dependency/operations.  By suggesting it be in its own nar I am not of the view 
it is a lot more work to be clear.  I could be wrong.  But the more work 
consideration is also not a strong one.  The 'it is already in there...' 
argument is better since well - that is true.  I'm just asking that the effort 
to do that be strongly considered because then those who would not use this 
option don't have to carry the component into deployment nor the potential 
vulnerabilities the libraries might bring.  But those that want it are good to 
go and can use it.  If it is baked into the framework nar this is not an option.

As evidence of my concern regarding the vulnerability it does give pause that 
the version of QuestDB was not considered in this PR to this point.  Keeping 
dependencies up to date is *everyones* responsibility.  

The NiFi 2.0 line is by the way super wildly close to being entirely free of 
static coding practice and dependency vulnerabilities (excluding hadoop related 
components) which is in itself a miracle.  Please ensure the dependencies in 
this PR are up to date.  On that note do we know what QuestDB version changes 
mean in terms of breaking changes, changes that might require users to do 
something to keep their state, etc..?  I ask because we learned our lessons the 
hard way here with H2.  H2 served us extremely well over the years but 
eventually it created some very complicated compelling events.

Anyway - I dont want to come off like this thing can't happen.  The root idea 
of this 

[jira] [Commented] (NIFI-12236) Improving fault tolerancy of the QuestDB backed metrics repository

2023-12-07 Thread Joe Witt (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17794312#comment-17794312
 ] 

Joe Witt commented on NIFI-12236:
-

Regarding defaults - thank you.

API Changes: 
These should get the highest degree of scrutiny by reviewers and the most 
effort and thought by authors.  At this stage I do not understand the intention 
for the change.  It appears that concern is shared. If there is a 
non-API/specific to this implementation option that should be pursued instead.

Suggestion about the nar:
I am not aware of other cases where the optional implementation of a thing is 
the light in terms of dependencies path.  In this case the optional 
implementation (QuestDB backed stats) is the heavier one in terms of 
dependency/operations.  By suggesting it be in its own nar I am not of the view 
it is a lot more work to be clear.  I could be wrong.  But the more work 
consideration is also not a strong one.  The 'it is already in there...' 
argument is better since well - that is true.  I'm just asking that the effort 
to do that be strongly considered because then those who would not use this 
option don't have to carry the component into deployment nor the potential 
vulnerabilities the libraries might bring.  But those that want it are good to 
go and can use it.  If it is baked into the framework nar this is not an option.

As evidence of my concern regarding the vulnerability it does give pause that 
the version of QuestDB was not considered in this PR to this point.  Keeping 
dependencies up to date is *everyones* responsibility.  

The NiFi 2.0 line is by the way super wildly close to being entirely free of 
static coding practice and dependency vulnerabilities (excluding hadoop related 
components) which is in itself a miracle.  Please ensure the dependencies in 
this PR are up to date.  On that note do we know what QuestDB version changes 
mean in terms of breaking changes, changes that might require users to do 
something to keep their state, etc..?  I ask because we learned our lessons the 
hard way here with H2.  H2 served us extremely well over the years but 
eventually it created some very complicated compelling events.

Anyway - I dont want to come off like this thing can't happen.  The root idea 
of this I think we all agree is useful. There is some debate on whether this 
implementation approach is desirable vs another but even that does not matter.  
He who does the work for a given implementation ultimately wins.  What I am 
asking then though is please make that implementation as narrow and specific as 
possible to give others choice on whether they are exposed to it.  That is the 
heart of the replies I am making.

Thanks

> Improving fault tolerancy of the QuestDB backed metrics repository
> --
>
> Key: NIFI-12236
> URL: https://issues.apache.org/jira/browse/NIFI-12236
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Simon Bence
>Assignee: Simon Bence
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Based on the related discussion on the dev email list, the QuestDB handling 
> of the metrics repository needs to be improved to have better fault tolerance 
> in order to be possible to use as a viable option for default metrics data 
> store. This should primarily focus on handling unexpeted database events like 
> corrupted database or loss of space on the disk. Any issues should be handled 
> with an attempt to keep the database service healthy but in case of that is 
> impossible, the priority is to keep NiFi and the core services running, even 
> with the price of metrics collection / presentation outage.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (NIFI-12236) Improving fault tolerancy of the QuestDB backed metrics repository

2023-12-06 Thread Joe Witt (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793842#comment-17793842
 ] 

Joe Witt commented on NIFI-12236:
-

I think it would help to better explain what the properties/tunables are.  
Maybe that is made available elsewhere but i'm not sure what batch size means 
for this and I am also curious whether the user can control how much data is 
retained and what the default is for that.

A path forward that I think keeps things moving but doesn't create new risks 
currently not well enough explored
1. Avoid making this the default.  It can be opted into.
2. Avoid making nifi-api changes.
3. Move this into its own nar instead of the core framework nar.  It can 
certainly still be included as it is today.

Remaining concerns can be addressed later and based on needs and findings from 
actual usage.

Thanks

> Improving fault tolerancy of the QuestDB backed metrics repository
> --
>
> Key: NIFI-12236
> URL: https://issues.apache.org/jira/browse/NIFI-12236
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Simon Bence
>Assignee: Simon Bence
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Based on the related discussion on the dev email list, the QuestDB handling 
> of the metrics repository needs to be improved to have better fault tolerance 
> in order to be possible to use as a viable option for default metrics data 
> store. This should primarily focus on handling unexpeted database events like 
> corrupted database or loss of space on the disk. Any issues should be handled 
> with an attempt to keep the database service healthy but in case of that is 
> impossible, the priority is to keep NiFi and the core services running, even 
> with the price of metrics collection / presentation outage.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (NIFI-11777) Component State Identifier

2023-12-06 Thread Joe Witt (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-11777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793809#comment-17793809
 ] 

Joe Witt commented on NIFI-11777:
-

[~pvillard][~simonbence]Please before there is an implementation for this can 
we generate and discuss on a [feature 
proposal|https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals].

This way we can first be on the same page on the purpose/options for an idea 
before there is a ton of implementation work done.

> Component State Identifier
> --
>
> Key: NIFI-11777
> URL: https://issues.apache.org/jira/browse/NIFI-11777
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Core Framework
>Reporter: Pierre Villard
>Assignee: Simon Bence
>Priority: Major
>
> In the Settings tab of a component, there should a new property "State 
> Identifier" that would always default to the UUID of the component.
> Users would be able to override this value and change it to something custom. 
> We would:
>  * ensure that the state identifier is something like [a-z]+[a-z0-9\-]*
>  * ensure that the state identifier is unique across all components
> The "State Identifier" should be used when storing the state of the component 
> in the state providers.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (NIFI-12236) Improving fault tolerancy of the QuestDB backed metrics repository

2023-12-06 Thread Joe Witt (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793786#comment-17793786
 ] 

Joe Witt commented on NIFI-12236:
-

I want to keep the different threads of concern clear.

The idea:
* I think we all agree there is good value in having these data points 
persisted across restarts.  The example Pierre gives is a perfect example of 
why.
* The reality though is our user interface is not designed for data to be held 
for long.  It isnt clear to me from the properties mentioned in PR but what is 
the plan for helping the user configure how long this data is 
retained/queryable?  I see batch size and frequency but not quite sure what 
those would mean relative to retention.  Perhaps that is part of the earlier 
implementation.

The implementation options:
* Offer an embedded state storage mechanism.  QuestDB is one example. A 
database per nifi instance or a database per cluster.  This embedded/batteries 
included mode is quite convenient for the user but we then of course have to do 
it quite thoroughly and consider upgrade scenarios.  I think our pucker factor 
here is far higher given the challenges we've had to work through from H2 in 
the recent year or two.
* Offer the ability to connect to/use a database of the user choosing.  Defer 
installation/durability/security of that database to the user as part of their 
normal database operations/etc..  This I find is more in-line with deployment 
styles we see in the Cloud, or automated with Ansible, or how one would deploy 
in K8S.

This implementation:
* It is a questdb per nifi node.  Does not address the other mode should a user 
want that.  Of note when we offered the Zookeeper embedded mode we also offered 
to connect to a real zookeeper install.  That was to reflect the likely 
non-prod vs prod usage and I think that holds here as well.
* It has a change to the nifi-api.  We should strongly avoid any such API 
changes as part of this activity unless that change has value to various other 
components and the purpose/meaning of that change is very clear.
* It includes a Retry mechanism that the PR suggests might be addressed in a 
later commit.  I don't quite follow what that really means but I recommend 
making this implementation as simple/straight forward as possible.

The default selection:
* We should not be changing the default until this model is proven to be stable 
and recoverable in the same manner to the current in memory implementation.



> Improving fault tolerancy of the QuestDB backed metrics repository
> --
>
> Key: NIFI-12236
> URL: https://issues.apache.org/jira/browse/NIFI-12236
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Simon Bence
>Assignee: Simon Bence
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Based on the related discussion on the dev email list, the QuestDB handling 
> of the metrics repository needs to be improved to have better fault tolerance 
> in order to be possible to use as a viable option for default metrics data 
> store. This should primarily focus on handling unexpeted database events like 
> corrupted database or loss of space on the disk. Any issues should be handled 
> with an attempt to keep the database service healthy but in case of that is 
> impossible, the priority is to keep NiFi and the core services running, even 
> with the price of metrics collection / presentation outage.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (NIFI-11777) Component State Identifier

2023-12-06 Thread Joe Witt (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-11777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793771#comment-17793771
 ] 

Joe Witt commented on NIFI-11777:
-

[~pvillard] Can you add more to this in terms of what the purpose/function is?  
As written this is more like an implementation/what.  How does this differ from 
the component UUID?  What happens to whatever is currently used to associate a 
given component to its state?  

> Component State Identifier
> --
>
> Key: NIFI-11777
> URL: https://issues.apache.org/jira/browse/NIFI-11777
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Core Framework
>Reporter: Pierre Villard
>Assignee: Simon Bence
>Priority: Major
>
> In the Settings tab of a component, there should a new property "State 
> Identifier" that would always default to the UUID of the component.
> Users would be able to override this value and change it to something custom. 
> We would:
>  * ensure that the state identifier is something like [a-z]+[a-z0-9\-]*
>  * ensure that the state identifier is unique across all components
> The "State Identifier" should be used when storing the state of the component 
> in the state providers.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (NIFI-12236) Improving fault tolerancy of the QuestDB backed metrics repository

2023-12-05 Thread Joe Witt (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793412#comment-17793412
 ] 

Joe Witt commented on NIFI-12236:
-

I should clarify as part of that I think was confusing...

Ultimately we should just allow them to have a database instance that they run 
as per normal.  All such desired DBs have installers/tools/to have a healthy 
instance setup.  Then the nifi side would be configured to talk to it for 
sending/querying.  

> Improving fault tolerancy of the QuestDB backed metrics repository
> --
>
> Key: NIFI-12236
> URL: https://issues.apache.org/jira/browse/NIFI-12236
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Simon Bence
>Assignee: Simon Bence
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Based on the related discussion on the dev email list, the QuestDB handling 
> of the metrics repository needs to be improved to have better fault tolerance 
> in order to be possible to use as a viable option for default metrics data 
> store. This should primarily focus on handling unexpeted database events like 
> corrupted database or loss of space on the disk. Any issues should be handled 
> with an attempt to keep the database service healthy but in case of that is 
> impossible, the priority is to keep NiFi and the core services running, even 
> with the price of metrics collection / presentation outage.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (NIFI-12236) Improving fault tolerancy of the QuestDB backed metrics repository

2023-12-05 Thread Joe Witt (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793409#comment-17793409
 ] 

Joe Witt commented on NIFI-12236:
-

I went back to quickly review the presumably referenced mailing list thread 
titled 'NiFi 2.0 - QuestDB'.

The concerns I noted then remain for me.  It would certainly be nice for the 
users to have access to persistent and longer duration metric data.  That said 
I'm not sure our user interface for this is very good when it comes to holding 
this information and making interesting visualizations of it for longer 
durations.  For the type of usage the in memory durations we can reasonably 
sustain today seem about right.

What I am seeing more and more is that users of NiFi want this type of 
information available in their favorite monitoring or observability tool 
whether that is Datadog or Prometheus or something else.  

If our goal is a short term but durable store then perhaps we ought to give 
them a simple QuestDB based service/process they can run on some node.  Then 
their NiFi nodes are configured to send/query metrics from that service rather 
than it having to live on every node.  This also means it would be better 
externalized such that maybe they dont even use Quest but instead MySQL or 
Postgre or whatever they prefer.  In a k8s based deployment I can certainly see 
such a model working well.

For users that are looking for more robust data retention and query and 
analysis we're better off focusing on getting the data to their preferred tools.


> Improving fault tolerancy of the QuestDB backed metrics repository
> --
>
> Key: NIFI-12236
> URL: https://issues.apache.org/jira/browse/NIFI-12236
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Simon Bence
>Assignee: Simon Bence
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Based on the related discussion on the dev email list, the QuestDB handling 
> of the metrics repository needs to be improved to have better fault tolerance 
> in order to be possible to use as a viable option for default metrics data 
> store. This should primarily focus on handling unexpeted database events like 
> corrupted database or loss of space on the disk. Any issues should be handled 
> with an attempt to keep the database service healthy but in case of that is 
> impossible, the priority is to keep NiFi and the core services running, even 
> with the price of metrics collection / presentation outage.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12451) Remove Bootstrap Notification Services

2023-12-04 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-12451:

Fix Version/s: 2.0.0
   Resolution: Fixed
   Status: Resolved  (was: Patch Available)

> Remove Bootstrap Notification Services
> --
>
> Key: NIFI-12451
> URL: https://issues.apache.org/jira/browse/NIFI-12451
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 2.0.0
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
> Fix For: 2.0.0
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Following the deprecation of email and HTTP notification services on the 
> version 1 support branch, the Notification Service components should be 
> removed, and configuration elements should be removed from the bootstrap.conf 
> configuration. Additional bootstrap.conf settings for optional components 
> should also be reviewed and removed to streamline the default bootstrap.conf 
> settings.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12450) Deprecate Bootstrap Notification Services for Removal

2023-12-04 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-12450:

Fix Version/s: 1.25.0
   Resolution: Fixed
   Status: Resolved  (was: Patch Available)

> Deprecate Bootstrap Notification Services for Removal
> -
>
> Key: NIFI-12450
> URL: https://issues.apache.org/jira/browse/NIFI-12450
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 1.25.0
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
> Fix For: 1.25.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The NiFi bootstrap process includes optional notification services to send 
> email messages or call HTTP resources to indicate when NiFi is stopped or 
> started.
> Although this level of notification is useful in some circumstances, it does 
> not provide sufficient capabilities to warrant continued maintenance. Various 
> service monitoring tools provide more robust health and status checking 
> capabilities, and much more flexible notification options. For these reasons, 
> the Notification Services should be deprecated on the version 1 support 
> branch for subsequent removal in the next major release version.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12384) Upgrade Spring Framework to 6.0.13 for Registry

2023-11-21 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-12384:

Fix Version/s: 2.0.0-M1
   (was: 2.latest)
   Resolution: Fixed
   Status: Resolved  (was: Patch Available)

> Upgrade Spring Framework to 6.0.13 for Registry
> ---
>
> Key: NIFI-12384
> URL: https://issues.apache.org/jira/browse/NIFI-12384
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: NiFi Registry
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
> Fix For: 2.0.0-M1
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Spring Framework 6 represents a major upgrade from version 5 and requires a 
> number of supporting libraries to be upgraded. These supporting libraries 
> include the following:
> - Spring Security 6
> - Spring Boot 3
> - Jakarta Servlet API 5
> - Jetty 11
> - JAX RS 3
> - Jersey 3
> NiFi Registry should be upgraded independently of NiFi itself to provide an 
> incremental path forward.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12378) Remove Jython support in NiFi 2.x given real Python support in 2.x

2023-11-16 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-12378:

Fix Version/s: 2.latest

> Remove Jython support in NiFi 2.x given real Python support in 2.x
> --
>
> Key: NIFI-12378
> URL: https://issues.apache.org/jira/browse/NIFI-12378
> Project: Apache NiFi
>  Issue Type: Task
>Reporter: Joe Witt
>Assignee: Joe Witt
>Priority: Major
> Fix For: 2.latest
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (NIFI-12378) Remove Jython support in NiFi 2.x given real Python support in 2.x

2023-11-15 Thread Joe Witt (Jira)
Joe Witt created NIFI-12378:
---

 Summary: Remove Jython support in NiFi 2.x given real Python 
support in 2.x
 Key: NIFI-12378
 URL: https://issues.apache.org/jira/browse/NIFI-12378
 Project: Apache NiFi
  Issue Type: Task
Reporter: Joe Witt
Assignee: Joe Witt






--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Resolved] (NIFI-4947) Bind dynamic properties in ScriptedLookupService as variables in script

2023-11-15 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-4947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt resolved NIFI-4947.

Resolution: Won't Fix

> Bind dynamic properties in ScriptedLookupService as variables in script
> ---
>
> Key: NIFI-4947
> URL: https://issues.apache.org/jira/browse/NIFI-4947
> Project: Apache NiFi
>  Issue Type: Wish
>  Components: Extensions
>Affects Versions: 1.5.0
>Reporter: Mohd Izhar Firdaus Bin Ismail
>Priority: Minor
>
> It would be good to have dynamic properties declared on the 
> ScriptedLookupService to be made available as variables in the script, like 
> in the ExecuteScript processor. It would help with use-cases where user want 
> to build a configurable LookupService using Jython etc. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Resolved] (NIFI-7370) Update scripted component documentation about Jython issues

2023-11-15 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-7370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt resolved NIFI-7370.

Resolution: Won't Fix

> Update scripted component documentation about Jython issues
> ---
>
> Key: NIFI-7370
> URL: https://issues.apache.org/jira/browse/NIFI-7370
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Matt Burgess
>Priority: Minor
>
> NIFI-4004 introduced a new default interface method for RecordReaderFactory, 
> in order
> to avoid breaking API compatibility for those with custom
> record readers. This exposes the bug in [1] for the Jython script
> engine. At a major release point (NiFi 2.0) we could refactor the NiFi
> codebase to remove the default interface method, update all internal
> implementations, and announce that the LookupService API has changed
> and thus custom implementations would have to be updated. Not sure if
> we can get away with that for minor releases or not, usually breaking
> API compatibility is a no-no except for major releases.
> As was done in NIFI-5995 for ScriptedLookupService, this Jira proposes to 
> update the documentation for any other scripted component for which its 
> underlying interface has default methods, and also to remove the Jython 
> script engine from the allowable values for the Script Engine property. One 
> such component is ScriptedReader, but there may be others as well.
> [1] [https://bugs.jython.org/issue2403|https://bugs.jython.org/issue2403]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Resolved] (NIFI-5434) ExecuteScript/Jython imports fail randomly

2023-11-15 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-5434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt resolved NIFI-5434.

Resolution: Won't Fix

> ExecuteScript/Jython imports fail randomly
> --
>
> Key: NIFI-5434
> URL: https://issues.apache.org/jira/browse/NIFI-5434
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.6.0, 1.7.0, 1.7.1
> Environment: Debian Stretch (9.5)
>Reporter: Wietze B
>Priority: Major
> Attachments: ibm.png, java_thread.png, log_extract.log
>
>
> I have a number of ExecuteScript processors running Jython that import 
> libraries such as {{threading}}, {{urllib3}}, {{requests}}, etc.
> When I boot up my machine or initially start the NiFi service, the 
> ExecuteScript processors relying on external libraries usually to don't work 
> straight away, throwing {{AttributeError}} s or {{ImportError}} s on 
> libraries that definitely exist and are accessible. Restarting the machine or 
> the NiFi service again usually fixes the problem, but it is unclear to me why 
> the imports sometimes fail, and sometimes work.
> See attached two screenshots and an extract from the logs.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Resolved] (NIFI-8615) ExecuteScript with python when use module directory

2023-11-15 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-8615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt resolved NIFI-8615.

Resolution: Won't Fix

> ExecuteScript with python when use module directory
> ---
>
> Key: NIFI-8615
> URL: https://issues.apache.org/jira/browse/NIFI-8615
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.13.2
> Environment: debian linux
>Reporter: Aldric DUPONT
>Priority: Major
>  Labels: Debian
> Attachments: Capture2.PNG, Capture3.PNG, test_bug.xml
>
>
>  
> *I use additional module "pytz" in python, this module directory is locate in 
> "/usr/local/lib/python3.9/site-packages"*
> *I use  sample1 in "Script Body" : (no comment)*
> _from org.python.core.util.FileUtil import wrap_
> _from org.apache.nifi.processors.script import ExecuteScript_
> _from datetime import datetime, tzinfo_
> _flow_file = session.get()_
> _import pytz_
> _from pytz import timezone_
> _utc = pytz.utc_
> _eastern = timezone('US/Eastern')_
> _flow_file = session.putAttribute(flow_file, 'timezone', eastern.zone)_
> _flow_file = session.putAttribute(flow_file, 'utc', utc.zone)_
> _flow_file = session.putAttribute(flow_file, 'test', 'salut')_
> _session.transfer(flow_file, ExecuteScript.REL_SUCCESS)_
>  
> *and sample2 with comment :* __ 
> _from org.python.core.util.FileUtil import wrap_
> _from org.apache.nifi.processors.script import ExecuteScript_
> _from datetime import datetime, tzinfo_
> _flow_file = session.get()_
> _"""_
> _import pytz_
> _from pytz import timezone_
> _utc = pytz.utc_
> _eastern = timezone('US/Eastern')_
> _flow_file = session.putAttribute(flow_file, 'timezone', eastern.zone)_
> _flow_file = session.putAttribute(flow_file, 'utc', utc.zone)_
> _"""_
> _flow_file = session.putAttribute(flow_file, 'test', 'salut')_
> _session.transfer(flow_file, ExecuteScript.REL_SUCCESS)_
>  
> +*Try 1*+
> *When use sample1 in version 1.13.2 ExecuteScript make error " :* 
> {color:#ff8b00}_ERROR [Timer-Driven Process Thread-4] 
> o.a.nifi.processors.script.ExecuteScript 
> ExecuteScript[id=790ea9a2-0179-1000-a662-30042349b329] Failed to process 
> session due to org.apache.nifi.processor.exception.ProcessException: 
> javax.script.ScriptException: ImportError: No module named pytz in 

[jira] [Created] (NIFI-12377) Deprecate support for Jython in NiFi 1.x line

2023-11-15 Thread Joe Witt (Jira)
Joe Witt created NIFI-12377:
---

 Summary: Deprecate support for Jython in NiFi 1.x line
 Key: NIFI-12377
 URL: https://issues.apache.org/jira/browse/NIFI-12377
 Project: Apache NiFi
  Issue Type: Task
Reporter: Joe Witt
Assignee: Joe Witt
 Fix For: 1.latest


Jython is not being actively maintained.  There are a number of vulnerability 
reports tied to it.  Python 2 is EOL.  The NiFi 2.x line has real integration 
with real Python and is a better go forward approach.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Resolved] (NIFI-11788) Make adjustments to logback.xml to nudge logback into more consistent cleanup behavior

2023-11-13 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-11788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt resolved NIFI-11788.
-
Fix Version/s: 2.0.0
   Resolution: Fixed

> Make adjustments to logback.xml to nudge logback into more consistent cleanup 
> behavior
> --
>
> Key: NIFI-11788
> URL: https://issues.apache.org/jira/browse/NIFI-11788
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.latest, 2.latest
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Major
> Fix For: 2.0.0
>
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> A lot of users have been reporting problems with enforcing log rollover with 
> logback. Disk overflow appears to be a common problem with this.
> Based on what I have found, it appears that there are two flags that need to 
> be set on the appenders as appropriate: totalSizeCap and cleanHistoryOnStart. 
> After experimenting with these settings, I found that with them enabled on 
> the APP appender, they enforce log cleanup.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (NIFI-12342) Remove extraneous 'relativePath' entry in pom.xml for Apache parent

2023-11-09 Thread Joe Witt (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17784572#comment-17784572
 ] 

Joe Witt commented on NIFI-12342:
-

This explains what that means: 
https://robintegg.com/2019/01/20/why-does-spring-initializr-set-the-parent-pom-relativepath-to-empty.html

As does this: https://www.baeldung.com/maven-relativepath

All happened in this commit during initial nifi release: 
https://github.com/apache/nifi/commit/a2f63c7c886495ccee52a448e7a9625ad65ad1f9#diff-d9daf395e85220dd6e54928b5d2cec7c7a2dbcd00c231b4128b92e57b52a314a

Seems like it is fine to be there. Mostly just an optimization to skip checking 
locally first.  But should not break anything having it though seeing some 
build issues for this in another env.

> Remove extraneous 'relativePath' entry in pom.xml for Apache parent
> ---
>
> Key: NIFI-12342
> URL: https://issues.apache.org/jira/browse/NIFI-12342
> Project: Apache NiFi
>  Issue Type: Task
>Reporter: Joe Witt
>Assignee: Joe Witt
>Priority: Major
> Fix For: 2.latest
>
>
> This relativePath entry dates back to 2015.  Dont recall why we needed it but 
> it is problematic now.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12342) Remove extraneous 'relativePath' entry in pom.xml for Apache parent

2023-11-09 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-12342:

Fix Version/s: 2.latest
   (was: 2.0.0)

> Remove extraneous 'relativePath' entry in pom.xml for Apache parent
> ---
>
> Key: NIFI-12342
> URL: https://issues.apache.org/jira/browse/NIFI-12342
> Project: Apache NiFi
>  Issue Type: Task
>Reporter: Joe Witt
>Assignee: Joe Witt
>Priority: Major
> Fix For: 2.latest
>
>
> This relativePath entry dates back to 2015.  Dont recall why we needed it but 
> it is problematic now.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (NIFI-12342) Remove extraneous 'relativePath' entry in pom.xml for Apache parent

2023-11-09 Thread Joe Witt (Jira)
Joe Witt created NIFI-12342:
---

 Summary: Remove extraneous 'relativePath' entry in pom.xml for 
Apache parent
 Key: NIFI-12342
 URL: https://issues.apache.org/jira/browse/NIFI-12342
 Project: Apache NiFi
  Issue Type: Task
Reporter: Joe Witt
Assignee: Joe Witt
 Fix For: 2.0.0


This relativePath entry dates back to 2015.  Dont recall why we needed it but 
it is problematic now.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12288) Improve Number Handling in Utilities

2023-10-28 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-12288:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Improve Number Handling in Utilities
> 
>
> Key: NIFI-12288
> URL: https://issues.apache.org/jira/browse/NIFI-12288
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework, Extensions
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
> Fix For: 2.0.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Several utilities and components, such as FormatUtils, as well as the Wait 
> and Notify Processors, include imprecise conversion between long and integer 
> numbers. These references should be updated to maintain the original typing 
> and use explicit methods when converting to integers.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (NIFI-12288) Improve Number Handling in Utilities

2023-10-28 Thread Joe Witt (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17780634#comment-17780634
 ] 

Joe Witt commented on NIFI-12288:
-

[~exceptionfactory]thanks!  if this should also go on the 1.x line as implied 
can you please confirm that build works and self-merge off this +1.  THanks

> Improve Number Handling in Utilities
> 
>
> Key: NIFI-12288
> URL: https://issues.apache.org/jira/browse/NIFI-12288
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework, Extensions
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
> Fix For: 2.0.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Several utilities and components, such as FormatUtils, as well as the Wait 
> and Notify Processors, include imprecise conversion between long and integer 
> numbers. These references should be updated to maintain the original typing 
> and use explicit methods when converting to integers.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12288) Improve Number Handling in Utilities

2023-10-28 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-12288:

Fix Version/s: 2.0.0
   (was: 1.latest)
   (was: 2.latest)

> Improve Number Handling in Utilities
> 
>
> Key: NIFI-12288
> URL: https://issues.apache.org/jira/browse/NIFI-12288
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework, Extensions
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
> Fix For: 2.0.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Several utilities and components, such as FormatUtils, as well as the Wait 
> and Notify Processors, include imprecise conversion between long and integer 
> numbers. These references should be updated to maintain the original typing 
> and use explicit methods when converting to integers.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (NIFI-12194) Nifi fails when ConsumeKafka_2_6 processor is started with PLAINTEXT securityProtocol

2023-10-25 Thread Joe Witt (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17779665#comment-17779665
 ] 

Joe Witt commented on NIFI-12194:
-

slack thread: 
https://apachenifi.slack.com/archives/C0L9VCD47/p1698242200810429. 

Guillaume
  6 hours ago
Hello,
I just experienced something bad in NiFi 1.20.0
Let me know if that souds normal to you and if that may have been fixed in 
newer versions.
We're working on switching all our kafka consumers to latest version (2.0 to 
2.6 mainly) and also add SSL authentication to the brokers.
On one processors, I made a mistake :
Used SSL brokers (so with port 9093) but forgot to configure security protocol 
(left @ plaintext) and SSL context service (left @ No value )
What we observed :
The memory usage on all our cluster nodes increased from around 48% to 68% 
percent (and never went down, even after fix)
We reproduced that on a nonprod cluster : the cluster went OOM in less than 1 
minute.
Question :
Is that normal that the connections attempts lead to global outage ? Is it 
possible to fix a max number of attempts for connection ?
Is it normal the memory usage didn't go down after fix (our cluster is 
something like very stable in term of memory usage, for months now)
Has the behaviour changed in latest versions ?

> Nifi fails when ConsumeKafka_2_6 processor is started with PLAINTEXT 
> securityProtocol
> -
>
> Key: NIFI-12194
> URL: https://issues.apache.org/jira/browse/NIFI-12194
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.21.0, 1.23.0
>Reporter: Peter Schmitzer
>Priority: Major
> Attachments: image-2023-09-27-15-56-02-438.png
>
>
> When starting ConsumeKafka_2_6 processor with sasl mechanism GSSAPI and the 
> securityProtocol PLAINTEXT (although SSL would be correct) the UI crashed and 
> nifi was no longer accessible. Not only the frontend was not accessible 
> anymore, also the other processors in our flow stopped performing well 
> according to our dashboards.
> We were able to reproduce this by using the config as described above.
> Our nifi in preprod (where this was detected) runs in a kubernetes cluster.
>  * version 1.21.0
>  * 3 nodes
>  * jvmMemory: 1536m
>  * 3G memory (limit)
>  * 400m cpu (request)
>  * zookeeper
> The logs do not offer any unusual entries when the issue is triggered. 
> Inspecting the pod metrics we found a spike in memory.
> The issue is a bit scary for us because a rather innocent config parameter in 
> one single processor is able to let our whole cluster break down.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12088) UI - Dependent Properties are not shown when depending on newly created Service

2023-10-17 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-12088:

Fix Version/s: 2.0.0

> UI - Dependent Properties are not shown when depending on newly created 
> Service
> ---
>
> Key: NIFI-12088
> URL: https://issues.apache.org/jira/browse/NIFI-12088
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core UI
>Reporter: Mark Bathori
>Assignee: Shane Ardell
>Priority: Major
> Fix For: 2.0.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> When properties depends on a property containing controller services, the 
> dependent fields are not shown if the controller service is newly created.
> E.g.: When using the *ListenHTTP* processor, if the *Record Reader* property 
> is being populated by choosing the ‘{_}Create new service…{_}’ option then 
> the *Record Writer* property which depends on this filed stays invisible. The 
> *Record Writer* property will be visible only after re-opening the window.
> It works properly if I choose a previously created controller service in the 
> *Record Reader* property.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12124) Add a RenameRecordField processor

2023-10-13 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-12124:

Fix Version/s: 2.0.0
   (was: 2.latest)
   Resolution: Fixed
   Status: Resolved  (was: Patch Available)

> Add a RenameRecordField processor
> -
>
> Key: NIFI-12124
> URL: https://issues.apache.org/jira/browse/NIFI-12124
> Project: Apache NiFi
>  Issue Type: Task
>  Components: Extensions
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 2.0.0
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> The RemoveRecordField processor that was recently added has made it much 
> simpler to remove specific fields from a Record. However, it's still not 
> trivial to rename a field. There are multiple ways to achieve this, but none 
> of them are straight-forward.
> It should take as its configuration user-defined properties such that the key 
> is a RecordPath and the value is a String literal or a Expression Language 
> Expression - much like UpdateRecord does when configured with "Replacement 
> Value Strategy" = "Literal Value." When evaluating EL, it should expose 
> FlowFile attributes as well as {{field.name}} and {{{}field.value{}}}. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12221) Make heartbeat responses more lenient in some cases

2023-10-13 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-12221:

Fix Version/s: 2.0.0
   (was: 2.latest)
   Resolution: Fixed
   Status: Resolved  (was: Patch Available)

> Make heartbeat responses more lenient in some cases
> ---
>
> Key: NIFI-12221
> URL: https://issues.apache.org/jira/browse/NIFI-12221
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 2.0.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> When a heartbeat is received by the Cluster Coordinator, it responds based on 
> the node's current connection state. In the case of a disconnected node, it 
> either notifies the node that it is disconnected so that it will stop 
> hearting, or it requests the node to reconnect to the cluster.
> Due to changes that were made in 1.16, as well as a few additional changes 
> that have been made since, we can be much more lenient about when we ask the 
> node to reconnect vs. disconnect. For example, if a node was disconnected due 
> to not handling an update request, we previously needed to request that the 
> node disconnect again. However, now we can ask the node to reconnect, as it 
> may well be able to reconcile any differences and rejoin.
> We even currently request that a node disconnect if receiving a heartbeat 
> from a node whose last state was "Disconnected because Node was Shutdown". We 
> should definitely be more lenient in this case, as it's occasionally causing 
> System Test failures (e.g., 
> [https://github.com/apache/nifi/actions/runs/6498488206).|https://github.com/apache/nifi/actions/runs/6498488206)]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Resolved] (NIFI-12228) Concurrency bug can occasionally lead to allowing a group with Single FlowFile per Node input pulling in multiple FlowFiles

2023-10-13 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt resolved NIFI-12228.
-
Fix Version/s: 2.0.0
   1.24.0
   (was: 1.latest)
   (was: 2.latest)
   Resolution: Fixed

> Concurrency bug can occasionally lead to allowing a group with Single 
> FlowFile per Node input pulling in multiple FlowFiles
> ---
>
> Key: NIFI-12228
> URL: https://issues.apache.org/jira/browse/NIFI-12228
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 2.0.0, 1.24.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Every now and then we see a failure in the system tests:
> {code:java}
> BatchFlowBetweenGroupsIT.testSingleConcurrencyAndBatchOutputToBatchInputOutput
>  » Timeout testSingleConcurrencyAndBatchOutputToBatchInputOutput() timed out 
> after 5 minutes {code}
> Looking at the logs shows that this is happening because data enters Group A, 
> and then before the Output Port has a chance to push the data out of Group A, 
> a second FlowFile enters. As a result, the test fails because the queue 
> between Groups A and B, or the connection inside of Group B, never reach the 
> expected size.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12214) ConsumeElasticsearch doc generation broken

2023-10-11 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-12214:

Fix Version/s: 1.24.0
   (was: 1.latest)

> ConsumeElasticsearch doc generation broken
> --
>
> Key: NIFI-12214
> URL: https://issues.apache.org/jira/browse/NIFI-12214
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.24.0
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Minor
> Fix For: 2.0.0, 1.24.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The NiFi DocGenerator is throwing a WARNing during startup when processing 
> the {{ConsumeElasticsearch}} properties.
> {code:java}
> java.io.IOException: Unable to create XMLOutputStream
>   at 
> org.apache.nifi.documentation.html.HtmlDocumentationWriter.write(HtmlDocumentationWriter.java:102)
>   at 
> org.apache.nifi.documentation.DocGenerator.document(DocGenerator.java:142)
>   at 
> org.apache.nifi.documentation.DocGenerator.documentConfigurableComponent(DocGenerator.java:107)
>   at 
> org.apache.nifi.documentation.DocGenerator.generate(DocGenerator.java:64)
>   at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:763)
>   at org.apache.nifi.NiFi.(NiFi.java:172)
>   at org.apache.nifi.NiFi.(NiFi.java:83)
>   at org.apache.nifi.NiFi.main(NiFi.java:332)
> Caused by: javax.xml.stream.XMLStreamException: No property was found 
> matching the name 'el-rest-query-definition-style'
>   at 
> org.apache.nifi.documentation.html.HtmlDocumentationWriter.writeProperties(HtmlDocumentationWriter.java:743)
>   at 
> org.apache.nifi.documentation.html.HtmlDocumentationWriter.writeBody(HtmlDocumentationWriter.java:166)
>   at 
> org.apache.nifi.documentation.html.HtmlDocumentationWriter.write(HtmlDocumentationWriter.java:98)
>   ... 7 common frames omitted
> {code}
> This is due to the Query Builder properties inherrited from 
> {{ElasticsearchRestProcessor}} that {{dependOn}} the 
> {{QUERY_DEFINITION_STYLE}} property, which is omitted from 
> {{ConsumeElasticsearch}} because it doesn't make sense to use (one must build 
> the Elasticsearch {{query}} from the Query Builder properties rather than 
> defining the whole {{query}}).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12214) ConsumeElasticsearch doc generation broken

2023-10-11 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-12214:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

> ConsumeElasticsearch doc generation broken
> --
>
> Key: NIFI-12214
> URL: https://issues.apache.org/jira/browse/NIFI-12214
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.24.0
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Minor
> Fix For: 2.0.0, 1.24.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The NiFi DocGenerator is throwing a WARNing during startup when processing 
> the {{ConsumeElasticsearch}} properties.
> {code:java}
> java.io.IOException: Unable to create XMLOutputStream
>   at 
> org.apache.nifi.documentation.html.HtmlDocumentationWriter.write(HtmlDocumentationWriter.java:102)
>   at 
> org.apache.nifi.documentation.DocGenerator.document(DocGenerator.java:142)
>   at 
> org.apache.nifi.documentation.DocGenerator.documentConfigurableComponent(DocGenerator.java:107)
>   at 
> org.apache.nifi.documentation.DocGenerator.generate(DocGenerator.java:64)
>   at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:763)
>   at org.apache.nifi.NiFi.(NiFi.java:172)
>   at org.apache.nifi.NiFi.(NiFi.java:83)
>   at org.apache.nifi.NiFi.main(NiFi.java:332)
> Caused by: javax.xml.stream.XMLStreamException: No property was found 
> matching the name 'el-rest-query-definition-style'
>   at 
> org.apache.nifi.documentation.html.HtmlDocumentationWriter.writeProperties(HtmlDocumentationWriter.java:743)
>   at 
> org.apache.nifi.documentation.html.HtmlDocumentationWriter.writeBody(HtmlDocumentationWriter.java:166)
>   at 
> org.apache.nifi.documentation.html.HtmlDocumentationWriter.write(HtmlDocumentationWriter.java:98)
>   ... 7 common frames omitted
> {code}
> This is due to the Query Builder properties inherrited from 
> {{ElasticsearchRestProcessor}} that {{dependOn}} the 
> {{QUERY_DEFINITION_STYLE}} property, which is omitted from 
> {{ConsumeElasticsearch}} because it doesn't make sense to use (one must build 
> the Elasticsearch {{query}} from the Query Builder properties rather than 
> defining the whole {{query}}).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12213) Since nifi-bom was introduced, using LdapUserGroupProvider causes NiFi to fail to start

2023-10-11 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-12213:

Fix Version/s: 2.0.0
   (was: 2.latest)
   Resolution: Fixed
   Status: Resolved  (was: Patch Available)

> Since nifi-bom was introduced, using LdapUserGroupProvider causes NiFi to 
> fail to start
> ---
>
> Key: NIFI-12213
> URL: https://issues.apache.org/jira/browse/NIFI-12213
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Michael W Moser
>Assignee: Michael W Moser
>Priority: Major
> Fix For: 2.0.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> I built the main branch and tried to use the LdapUserGroupProvider in 
> authorizers.xml.  I get a ClassNotFoundException looking for FormatUtils in 
> nifi-utils.jar.  The nifi-bom marks nifi-utils as provided by 
> nifi-standard-services-api-nar but nifi-ldap-iaa-providers-nar doesn't depend 
> on nifi-standard-services-api-nar.
> {noformat}
> Caused by: java.lang.NoClassDefFoundError: org/apache/nifi/util/FormatUtils
>     at 
> org.apache.nifi.ldap.tenants.LdapUserGroupProvider.setTimeout(LdapUserGroupProvider.java:824)
>     at 
> org.apache.nifi.ldap.tenants.LdapUserGroupProvider.onConfigured(LdapUserGroupProvider.java:166)
>     at 
> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
>     at java.base/java.lang.reflect.Method.invoke(Method.java:580)
>     at 
> org.apache.nifi.authorization.UserGroupProviderInvocationHandler.invoke(UserGroupProviderInvocationHandler.java:38)
>     at jdk.proxy5/jdk.proxy5.$Proxy59.onConfigured(Unknown Source)
>     at 
> org.apache.nifi.authorization.AuthorizerFactoryBean.loadProviderProperties(AuthorizerFactoryBean.java:198)
>     at 
> org.apache.nifi.authorization.AuthorizerFactoryBean.getObject(AuthorizerFactoryBean.java:167)
>     at 
> org.apache.nifi.authorization.AuthorizerFactoryBean.getObject(AuthorizerFactoryBean.java:71)
>     at 
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:169)
>     ... 106 common frames omitted
> Caused by: java.lang.ClassNotFoundException: org.apache.nifi.util.FormatUtils
>     at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
>     at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:593)
>     at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)
>     ... 116 common frames omitted
> {noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-12214) ConsumeElasticsearch doc generation broken

2023-10-11 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-12214:

Fix Version/s: 2.0.0
   (was: 2.latest)

> ConsumeElasticsearch doc generation broken
> --
>
> Key: NIFI-12214
> URL: https://issues.apache.org/jira/browse/NIFI-12214
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.24.0
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Minor
> Fix For: 2.0.0, 1.latest
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The NiFi DocGenerator is throwing a WARNing during startup when processing 
> the {{ConsumeElasticsearch}} properties.
> {code:java}
> java.io.IOException: Unable to create XMLOutputStream
>   at 
> org.apache.nifi.documentation.html.HtmlDocumentationWriter.write(HtmlDocumentationWriter.java:102)
>   at 
> org.apache.nifi.documentation.DocGenerator.document(DocGenerator.java:142)
>   at 
> org.apache.nifi.documentation.DocGenerator.documentConfigurableComponent(DocGenerator.java:107)
>   at 
> org.apache.nifi.documentation.DocGenerator.generate(DocGenerator.java:64)
>   at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:763)
>   at org.apache.nifi.NiFi.(NiFi.java:172)
>   at org.apache.nifi.NiFi.(NiFi.java:83)
>   at org.apache.nifi.NiFi.main(NiFi.java:332)
> Caused by: javax.xml.stream.XMLStreamException: No property was found 
> matching the name 'el-rest-query-definition-style'
>   at 
> org.apache.nifi.documentation.html.HtmlDocumentationWriter.writeProperties(HtmlDocumentationWriter.java:743)
>   at 
> org.apache.nifi.documentation.html.HtmlDocumentationWriter.writeBody(HtmlDocumentationWriter.java:166)
>   at 
> org.apache.nifi.documentation.html.HtmlDocumentationWriter.write(HtmlDocumentationWriter.java:98)
>   ... 7 common frames omitted
> {code}
> This is due to the Query Builder properties inherrited from 
> {{ElasticsearchRestProcessor}} that {{dependOn}} the 
> {{QUERY_DEFINITION_STYLE}} property, which is omitted from 
> {{ConsumeElasticsearch}} because it doesn't make sense to use (one must build 
> the Elasticsearch {{query}} from the Query Builder properties rather than 
> defining the whole {{query}}).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Resolved] (NIFI-12211) TinkerpopClientService is enable-able when setting the username or password in the service

2023-10-11 Thread Joe Witt (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt resolved NIFI-12211.
-
Fix Version/s: 2.0.0
   Resolution: Fixed

> TinkerpopClientService is enable-able when setting the username or password 
> in the service
> --
>
> Key: NIFI-12211
> URL: https://issues.apache.org/jira/browse/NIFI-12211
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Levi Lentz
>Assignee: Levi Lentz
>Priority: Minor
> Fix For: 2.0.0
>
>   Original Estimate: 24h
>  Time Spent: 0.5h
>  Remaining Estimate: 23.5h
>
> In the TinkperpopClientService we are missing validators on the username and 
> password as introduced by this PR: 
> [https://github.com/apache/nifi/pull/7677]
> Example stacktrace when setting the username/password: 
> {code:java}
> java.lang.IllegalStateException: Cannot enable Controller Service 
> TestableGremlinClientService[id=gremlinService] because it is in an invalid 
> state: 'user-name' validated against 'test' is invalid because 'user-name' is 
> not a supported property or has no Validator associated with it
>  {code}
> The fix:
> Add Validators.VALID to the username/password fields.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


  1   2   3   4   5   6   7   8   9   10   >