[jira] [Updated] (NIFI-12936) ListGCSBucket should reset its tracking state after configuration change

2024-03-22 Thread Peter Turcsanyi (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Turcsanyi updated NIFI-12936:
---
Status: Patch Available  (was: Open)

> ListGCSBucket should reset its tracking state after configuration change
> 
>
> Key: NIFI-12936
> URL: https://issues.apache.org/jira/browse/NIFI-12936
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Peter Turcsanyi
>Assignee: Peter Turcsanyi
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Same as NIFI-12732 but for ListGCSBucket.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[PR] NIFI-12936 ListGCSBucket resets its tracking state after configuratio… [nifi]

2024-03-22 Thread via GitHub


turcsanyip opened a new pull request, #8550:
URL: https://github.com/apache/nifi/pull/8550

   …n change
   
   # Summary
   
   [NIFI-12936](https://issues.apache.org/jira/browse/NIFI-12936)
   
   # Tracking
   
   Please complete the following tracking steps prior to pull request creation.
   
   ### Issue Tracking
   
   - [ ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue 
created
   
   ### Pull Request Tracking
   
   - [ ] Pull Request title starts with Apache NiFi Jira issue number, such as 
`NIFI-0`
   - [ ] Pull Request commit message starts with Apache NiFi Jira issue number, 
as such `NIFI-0`
   
   ### Pull Request Formatting
   
   - [ ] Pull Request based on current revision of the `main` branch
   - [ ] Pull Request refers to a feature branch with one commit containing 
changes
   
   # Verification
   
   Please indicate the verification steps performed prior to pull request 
creation.
   
   ### Build
   
   - [ ] Build completed using `mvn clean install -P contrib-check`
 - [ ] JDK 21
   
   ### Licensing
   
   - [ ] New dependencies are compatible with the [Apache License 
2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License 
Policy](https://www.apache.org/legal/resolved.html)
   - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` 
files
   
   ### Documentation
   
   - [ ] Documentation formatting appears as expected in rendered files
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Created] (NIFI-12936) ListGCSBucket should reset its tracking state after configuration change

2024-03-22 Thread Peter Turcsanyi (Jira)
Peter Turcsanyi created NIFI-12936:
--

 Summary: ListGCSBucket should reset its tracking state after 
configuration change
 Key: NIFI-12936
 URL: https://issues.apache.org/jira/browse/NIFI-12936
 Project: Apache NiFi
  Issue Type: Bug
Reporter: Peter Turcsanyi
Assignee: Peter Turcsanyi


Same as NIFI-12732 but for ListGCSBucket.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Re: [PR] NIFI-12925 disallow TRACE and OPTIONS methods in ListenHTTP [nifi]

2024-03-22 Thread via GitHub


markobean commented on PR #8548:
URL: https://github.com/apache/nifi/pull/8548#issuecomment-2015974533

   > @markobean Changes should also be made to `ListenHttp` documentation on 
lines 84-85 to indicate what TRACE, OPTIONS and CONNECT will return.
   
   Documentation updated.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[PR] NIFI-1931 Add auto commit property to QueryDatabaseTable and QueryDatabaseTable [nifi]

2024-03-22 Thread via GitHub


jrsteinebrey opened a new pull request, #8549:
URL: https://github.com/apache/nifi/pull/8549

   NIFI-1931 Add auto commit property to QueryDatabaseTable and 
QueryDatabaseTable
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   # Summary
   
   [NIFI-1931](https://issues.apache.org/jira/browse/NIFI-1931)
   
   # Tracking
   
   Please complete the following tracking steps prior to pull request creation.
   
   ### Issue Tracking
   
   - [X] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI-1931) 
issue created
   
   ### Pull Request Tracking
   
   - [X] Pull Request title starts with Apache NiFi Jira issue number, such as 
`NIFI-1931`
   - [X] Pull Request commit message starts with Apache NiFi Jira issue number, 
as such `NIFI-1931`
   
   ### Pull Request Formatting
   
   - [X] Pull Request based on current revision of the `main` branch
   - [X] Pull Request refers to a feature branch with one commit containing 
changes
   
   # Verification
   
   Automated tests all passed 
   
   ### Build
   
   - [X] Build completed using `mvn clean install -P contrib-check`
   
   ### Licensing
   
   No new dependencies
   
   ### Documentation
   
   - [X] Documentation formatting appears as expected in rendered files
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] NIFI-12925 disallow TRACE and OPTIONS methods in ListenHTTP [nifi]

2024-03-22 Thread via GitHub


dan-s1 commented on PR #8548:
URL: https://github.com/apache/nifi/pull/8548#issuecomment-2015871331

   @markobean Changes should also be made to `ListenHttp` documentation on 
lines 84-85 to indicate what TRACE and OPTIONS will return.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Created] (NIFI-12935) Upgrade Karma Webpack to 5.0.1

2024-03-22 Thread David Handermann (Jira)
David Handermann created NIFI-12935:
---

 Summary: Upgrade Karma Webpack to 5.0.1
 Key: NIFI-12935
 URL: https://issues.apache.org/jira/browse/NIFI-12935
 Project: Apache NiFi
  Issue Type: Improvement
  Components: NiFi Registry
Reporter: David Handermann
Assignee: Matt Gilman


NiFi Registry UI depends on Karma Webpack as part of the build process. Karma 
Webpack 5.0.1 depends on Webpack 5, so further evaluation is necessary to 
determine the impact of upgrading.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Re: [PR] NIFI-12925 disallow TRACE and OPTIONS methods in ListenHTTP [nifi]

2024-03-22 Thread via GitHub


markobean commented on PR #8548:
URL: https://github.com/apache/nifi/pull/8548#issuecomment-2015850430

   Please apply this commit to the nifi-1.x branch as well.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Commented] (NIFI-12925) ListenHTTP should disable unused HTTP methods

2024-03-22 Thread Mark Bean (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17829975#comment-17829975
 ] 

Mark Bean commented on NIFI-12925:
--

No, CONNECT already receives 400 response.

> ListenHTTP should disable unused HTTP methods
> -
>
> Key: NIFI-12925
> URL: https://issues.apache.org/jira/browse/NIFI-12925
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Michael W Moser
>Assignee: Mark Bean
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> For [security 
> reasons|https://owasp.org/www-project-web-security-testing-guide/stable/4-Web_Application_Security_Testing/02-Configuration_and_Deployment_Management_Testing/06-Test_HTTP_Methods],
>  ListenHTTP should reply with 405 Method Not Allowed for HTTP methods OPTIONS 
> and TRACE.
> PUT already returns 405.
> GET, POST, DELETE, HEAD are used by the processor.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (NIFI-12925) ListenHTTP should disable unused HTTP methods

2024-03-22 Thread Daniel Stieglitz (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17829973#comment-17829973
 ] 

Daniel Stieglitz commented on NIFI-12925:
-

[~mosermw] Based on the listed link do we have to be concerned about CONNECT?

> ListenHTTP should disable unused HTTP methods
> -
>
> Key: NIFI-12925
> URL: https://issues.apache.org/jira/browse/NIFI-12925
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Michael W Moser
>Assignee: Mark Bean
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> For [security 
> reasons|https://owasp.org/www-project-web-security-testing-guide/stable/4-Web_Application_Security_Testing/02-Configuration_and_Deployment_Management_Testing/06-Test_HTTP_Methods],
>  ListenHTTP should reply with 405 Method Not Allowed for HTTP methods OPTIONS 
> and TRACE.
> PUT already returns 405.
> GET, POST, DELETE, HEAD are used by the processor.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Re: [PR] NIFI-12925 disallow TRACE and OPTIONS methods in ListenHTTP [nifi]

2024-03-22 Thread via GitHub


dan-s1 commented on code in PR #8548:
URL: https://github.com/apache/nifi/pull/8548#discussion_r1536061304


##
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/servlets/ListenHTTPServlet.java:
##
@@ -157,6 +157,21 @@ protected void doHead(final HttpServletRequest request, 
final HttpServletRespons
 }
 }
 
+private void notAllowed(final HttpServletRequest request, final 
HttpServletResponse response) throws IOException {
+response.sendError(HttpServletResponse.SC_METHOD_NOT_ALLOWED, "Method 
Not Allowed");
+}

Review Comment:
   ```suggestion
   }
   
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Updated] (NIFI-12925) ListenHTTP should disable unused HTTP methods

2024-03-22 Thread Mark Bean (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Bean updated NIFI-12925:
-
Status: Patch Available  (was: In Progress)

> ListenHTTP should disable unused HTTP methods
> -
>
> Key: NIFI-12925
> URL: https://issues.apache.org/jira/browse/NIFI-12925
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Michael W Moser
>Assignee: Mark Bean
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> For [security 
> reasons|https://owasp.org/www-project-web-security-testing-guide/stable/4-Web_Application_Security_Testing/02-Configuration_and_Deployment_Management_Testing/06-Test_HTTP_Methods],
>  ListenHTTP should reply with 405 Method Not Allowed for HTTP methods OPTIONS 
> and TRACE.
> PUT already returns 405.
> GET, POST, DELETE, HEAD are used by the processor.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[PR] NIFI-12925 disallow TRACE and OPTIONS methods in ListenHTTP [nifi]

2024-03-22 Thread via GitHub


markobean opened a new pull request, #8548:
URL: https://github.com/apache/nifi/pull/8548

   
   
   
   
   
   
   
   
   
   
   
   
   
   # Summary
   
   Updates ListenHTTP to disallow TRACE and OPTIONS requests. Any such requests 
will be sent a 405 response.
   
   [NIFI-12925](https://issues.apache.org/jira/browse/NIFI-12925)
   
   # Tracking
   
   Please complete the following tracking steps prior to pull request creation.
   
   ### Issue Tracking
   
   - [X] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue 
created
   
   ### Pull Request Tracking
   
   - [X] Pull Request title starts with Apache NiFi Jira issue number, such as 
`NIFI-0`
   - [X] Pull Request commit message starts with Apache NiFi Jira issue number, 
as such `NIFI-0`
   
   ### Pull Request Formatting
   
   - [X] Pull Request based on current revision of the `main` branch
   - [X] Pull Request refers to a feature branch with one commit containing 
changes
   
   # Verification
   
   Please indicate the verification steps performed prior to pull request 
creation.
   
   Verified via new unit tests and within a running NiFi. Used InvokeHTTP to 
send OPTIONS and TRACE requests to ListenHTTP. Both requests received "405" 
responses with the FlowFile going to No Retry relationship.
   
   ### Build
   
   - [X] Build completed using `mvn clean install -P contrib-check`
 - [X] JDK 21
   
   ### Licensing
   
   - [ ] New dependencies are compatible with the [Apache License 
2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License 
Policy](https://www.apache.org/legal/resolved.html)
   - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` 
files
   
   ### Documentation
   
   - [ ] Documentation formatting appears as expected in rendered files
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Commented] (NIFI-12918) Stateless NiFi NullPointerException on versioned PGs

2024-03-22 Thread Stephanie Ambrose (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-12918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17829955#comment-17829955
 ] 

Stephanie Ambrose commented on NIFI-12918:
--

[~dstiegli1] - I confirmed the problem exists on 2.0 as well as 1.x. I also 
confirmed that the PR submitted fixes the problem on both versions. Apparently, 
there is an integration test that is now failing, so I'm troubleshooting that 
now. 

> Stateless NiFi NullPointerException on versioned PGs
> 
>
> Key: NIFI-12918
> URL: https://issues.apache.org/jira/browse/NIFI-12918
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Stephanie Ambrose
>Assignee: Stephanie Ambrose
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> If you create a process group and commit that to version control, and then 
> add a sub-process group under that one and commit that to its own version 
> control, it will throw a NullPointerException:
> *NOTE: This error only occurs for stateless nifi processing.*
> {code:java}
> 2024-03-15T12:25:38.565156967Z 2024-03-15 12:25:38,564 INFO [main] 
> o.a.nifi.groups.StandardProcessGroup 
> StandardProcessGroup[identifier=9688e64b-3f4a-3668-81b9-e25d9ea61454,name=Sub 
> Group] added to StandardProcessGroup[identifier=stateless-flow,name=Test Flow]
> 2024-03-15T12:25:38.570693223Z Exception in thread "main" 
> java.lang.NullPointerException: Registry ID must be specified
>   at java.base/java.util.Objects.requireNonNull(Unknown Source)
> 2024-03-15T12:25:38.570722016Zat 
> org.apache.nifi.registry.flow.StandardVersionControlInformation$Builder.build(StandardVersionControlInformation.java:134)
> 2024-03-15T12:25:38.570725111Zat 
> org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.synchronize(StandardVersionedComponentSynchronizer.java:354)
> 2024-03-15T12:25:38.570729455Zat 
> org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.addProcessGroup(StandardVersionedComponentSynchronizer.java:1185)
>   at 
> org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.synchronizeChildGroups(StandardVersionedComponentSynchronizer.java:528)
> 2024-03-15T12:25:38.573083029Zat 
> org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.synchronize(StandardVersionedComponentSynchronizer.java:426)
> 2024-03-15T12:25:38.573086447Zat 
> org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.lambda$synchronize$0(StandardVersionedComponentSynchronizer.java:265)
> 2024-03-15T12:25:38.573092747Zat 
> org.apache.nifi.controller.flow.AbstractFlowManager.withParameterContextResolution(AbstractFlowManager.java:551)
> 2024-03-15T12:25:38.573095677Zat 
> org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.synchronize(StandardVersionedComponentSynchronizer.java:260)
> 2024-03-15T12:25:38.573098999Zat 
> org.apache.nifi.groups.StandardProcessGroup.synchronizeFlow(StandardProcessGroup.java:3977)
>   at 
> org.apache.nifi.groups.StandardProcessGroup.updateFlow(StandardProcessGroup.java:3957)
>   at 
> org.apache.nifi.stateless.engine.StandardStatelessEngine.createFlow(StandardStatelessEngine.java:180)
> 2024-03-15T12:25:38.573108177Zat 
> org.apache.nifi.stateless.flow.StandardStatelessDataflowFactory.createDataflow(StandardStatelessDataflowFactory.java:243)
>   at 
> org.apache.nifi.stateless.bootstrap.StatelessBootstrap.createDataflow(StatelessBootstrap.java:73)
>  {code}
> When stepping through the debugger, it appears the "registryId" property is 
> null. The JerseyClient call is supposed to map the response from registry to 
> the proper class, but registryId is never set:
> {code:java}
> java.lang.Thread.State: RUNNABLE
> at 
> org.apache.nifi.flow.VersionedProcessGroup.setVersionedFlowCoordinates(VersionedProcessGroup.java:150)
> at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Unknown 
> Source:-1)
> at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown 
> Source:-1)
> at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown 
> Source:-1)
> at java.lang.reflect.Method.invoke(Unknown Source:-1)
> at 
> com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:141)
> at 
> com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:314)
> at 
> com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177)
> at 
> com.fasterxml.jackson.databind.deser.std.CollectionDeserializer._deserializeFromArray(CollectionDeserializer.java:359)
> at 
> 

[jira] [Created] (NIFI-12934) RenameRecordField does not clear serialized form of records

2024-03-22 Thread Mark Payne (Jira)
Mark Payne created NIFI-12934:
-

 Summary: RenameRecordField does not clear serialized form of 
records
 Key: NIFI-12934
 URL: https://issues.apache.org/jira/browse/NIFI-12934
 Project: Apache NiFi
  Issue Type: Bug
  Components: Extensions
Reporter: Mark Payne


When RenameRecordField runs, it updates the parent record for any matches. 
However, when this happens, the Record's "serialized form" does not get 
cleared. As a result, the Record Writer may (depending on its configuration) 
write out the 'cached' / serialized form of the Record, resulting in no change 
to the records.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[PR] Bump webpack-dev-middleware and karma-webpack in /nifi-registry/nifi-registry-core/nifi-registry-web-ui/src/main [nifi]

2024-03-22 Thread via GitHub


dependabot[bot] opened a new pull request, #8547:
URL: https://github.com/apache/nifi/pull/8547

   Bumps 
[webpack-dev-middleware](https://github.com/webpack/webpack-dev-middleware) to 
5.3.4 and updates ancestor dependency 
[karma-webpack](https://github.com/webpack-contrib/karma-webpack). These 
dependencies need to be updated together.
   
   Updates `webpack-dev-middleware` from 5.3.3 to 5.3.4
   
   Release notes
   Sourced from https://github.com/webpack/webpack-dev-middleware/releases;>webpack-dev-middleware's
 releases.
   
   v5.3.4
   https://github.com/webpack/webpack-dev-middleware/compare/v5.3.3...v5.3.4;>5.3.4
 (2024-03-20)
   Bug Fixes
   
   security: do not allow to read files above (https://redirect.github.com/webpack/webpack-dev-middleware/issues/1779;>#1779)
 (https://github.com/webpack/webpack-dev-middleware/commit/189c4ac7d2344ec132a4689e74dc837ec5be0132;>189c4ac)
   
   
   
   
   Changelog
   Sourced from https://github.com/webpack/webpack-dev-middleware/blob/v5.3.4/CHANGELOG.md;>webpack-dev-middleware's
 changelog.
   
   https://github.com/webpack/webpack-dev-middleware/compare/v5.3.3...v5.3.4;>5.3.4
 (2024-03-20)
   Bug Fixes
   
   security: do not allow to read files above (https://redirect.github.com/webpack/webpack-dev-middleware/issues/1779;>#1779)
 (https://github.com/webpack/webpack-dev-middleware/commit/189c4ac7d2344ec132a4689e74dc837ec5be0132;>189c4ac)
   
   
   
   
   Commits
   
   https://github.com/webpack/webpack-dev-middleware/commit/86071ead69e946ada25497d3e281923e885229a4;>86071ea
 chore(release): 5.3.4
   https://github.com/webpack/webpack-dev-middleware/commit/189c4ac7d2344ec132a4689e74dc837ec5be0132;>189c4ac
 fix(security): do not allow to read files above (https://redirect.github.com/webpack/webpack-dev-middleware/issues/1779;>#1779)
   See full diff in https://github.com/webpack/webpack-dev-middleware/compare/v5.3.3...v5.3.4;>compare
 view
   
   
   
   
   Updates `karma-webpack` from 4.0.2 to 5.0.1
   
   Release notes
   Sourced from https://github.com/webpack-contrib/karma-webpack/releases;>karma-webpack's
 releases.
   
   v5.0.1
   https://github.com/webpack-contrib/karma-webpack/compare/v5.0.0...v5.0.1;>5.0.1
 (2024-02-01)
   Fixes
   
   security fixes
   
   v5.0.0
   No release notes provided.
   v5.0.0-alpha.6
   Bug Fixes
   
   automatically fix missing webpack framework and report a warning (https://github.com/ryanclark/karma-webpack/commit/ea5dc8ec56ac2752f76e1b1858bf1320ff2ee127;>ea5dc8e)
   fix an issue where multiple karma-webpack processes could not run in 
parallel (https://github.com/ryanclark/karma-webpack/commit/ea3dabe7f7cd8910b61d2bc462c40fa2a37d5feb;>ea3dabe)
   bump hotfix dependencies (https://github.com/ryanclark/karma-webpack/commit/98b3ec9c83c336e73465459b7f02d5d1d987ba2c;>98b3ec9)
   
   v5.0.0-alpha.5
   Bug Fixes
   
   change the webpack peer dependency to webpack v5 (https://github.com/ryanclark/karma-webpack/commit/2e0ca74c75b173abd5af9f2766b62463252a83e5;>2e0ca74)
   
   v5.0.0-alpha.4
   Bug Fixes
   
   fix compatibility issues for webpack v5 (https://github.com/ryanclark/karma-webpack/commit/8d7366f3cb747a9493c9f5542b9cb0317ac32d9e;>8d7366f),
 closes https://redirect.github.com/ryanclark/karma-webpack/issues/452;>#452
   remove deprecation warning for .watch() (https://github.com/ryanclark/karma-webpack/commit/4fe1f6093fc6ff25503f1422123b2b3944a4145a;>4fe1f60)
   
   
   
   
   Changelog
   Sourced from https://github.com/codymikol/karma-webpack/blob/master/CHANGELOG.md;>karma-webpack's
 changelog.
   
   https://github.com/webpack-contrib/karma-webpack/compare/v5.0.0...v5.0.1;>5.0.1
 (2024-02-01)
   Fixes
   
   security fixes
   
   https://github.com/webpack-contrib/karma-webpack/compare/v5.0.0-alpha.6...v5.0.0;>5.0.0
 (2021-02-02)
   No changes, just a new stable release.
   https://github.com/webpack-contrib/karma-webpack/compare/v5.0.0-alpha.5...v5.0.0-alpha.6;>5.0.0-alpha.6
 (2021-01-30)
   Bug Fixes
   
   automatically fix missing webpack framework and report a warning (https://github.com/ryanclark/karma-webpack/commit/ea5dc8ec56ac2752f76e1b1858bf1320ff2ee127;>ea5dc8e)
   fix an issue where multiple karma-webpack processes could not run in 
parallel (https://github.com/ryanclark/karma-webpack/commit/ea3dabe7f7cd8910b61d2bc462c40fa2a37d5feb;>ea3dabe)
   bump hotfix dependencies (https://github.com/ryanclark/karma-webpack/commit/98b3ec9c83c336e73465459b7f02d5d1d987ba2c;>98b3ec9)
   
   https://github.com/webpack-contrib/karma-webpack/compare/v5.0.0-alpha.4...v5.0.0-alpha.5;>5.0.0-alpha.5
 (2020-12-06)
   Bug Fixes
   
   change the webpack peer dependency to webpack v5 (https://github.com/ryanclark/karma-webpack/commit/2e0ca74c75b173abd5af9f2766b62463252a83e5;>2e0ca74)
   
   https://github.com/webpack-contrib/karma-webpack/compare/v5.0.0-alpha.3.0...v5.0.0-alpha.4;>5.0.0-alpha.4
 (2020-12-06)
   Bug Fixes
   
   fix compatibility issues for webpack v5 

[PR] MINIFICPP-2319 Set the home directory on the Configure object in the unit test [nifi-minifi-cpp]

2024-03-22 Thread via GitHub


fgerlits opened a new pull request, #1747:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1747

   https://issues.apache.org/jira/browse/MINIFICPP-2319
   
   --
   
   Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
   
   In order to streamline the review of the contribution we ask you
   to ensure the following steps have been taken:
   
   ### For all changes:
   - [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
   
   - [x] Does your PR title start with MINIFICPP- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
   
   - [x] Has your PR been rebased against the latest commit within the target 
branch (typically main)?
   
   - [x] Is your initial contribution a single, squashed commit?
   
   ### For code changes:
   - [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
   - [ ] If applicable, have you updated the LICENSE file?
   - [ ] If applicable, have you updated the NOTICE file?
   
   ### For documentation related changes:
   - [ ] Have you ensured that format looks appropriate for the output in which 
it is rendered?
   
   ### Note:
   Please ensure that once the PR is submitted, you check GitHub Actions CI 
results for build issues and submit an update to your PR as soon as possible.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Created] (MINIFICPP-2319) Prevent RepoTests from creating file in the source tree

2024-03-22 Thread Ferenc Gerlits (Jira)
Ferenc Gerlits created MINIFICPP-2319:
-

 Summary: Prevent RepoTests from creating file in the source tree
 Key: MINIFICPP-2319
 URL: https://issues.apache.org/jira/browse/MINIFICPP-2319
 Project: Apache NiFi MiNiFi C++
  Issue Type: Improvement
Reporter: Ferenc Gerlits
Assignee: Ferenc Gerlits


A new test was added to RepoTests, which does not set MINIFI_HOME to a temp 
directory, so it creates the bootstrap.conf file in the source tree.  It would 
be good to figure out how to prevent this from happening in the first place, 
but for now, I'm just going to fix this one test.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Assigned] (NIFI-12925) ListenHTTP should disable unused HTTP methods

2024-03-22 Thread Mark Bean (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Bean reassigned NIFI-12925:


Assignee: Mark Bean

> ListenHTTP should disable unused HTTP methods
> -
>
> Key: NIFI-12925
> URL: https://issues.apache.org/jira/browse/NIFI-12925
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Michael W Moser
>Assignee: Mark Bean
>Priority: Major
>
> For [security 
> reasons|https://owasp.org/www-project-web-security-testing-guide/stable/4-Web_Application_Security_Testing/02-Configuration_and_Deployment_Management_Testing/06-Test_HTTP_Methods],
>  ListenHTTP should reply with 405 Method Not Allowed for HTTP methods OPTIONS 
> and TRACE.
> PUT already returns 405.
> GET, POST, DELETE, HEAD are used by the processor.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-1931) QueryDatabaseTable processor; setFetchSize not working for postgres driver causing out of memory

2024-03-22 Thread Matt Burgess (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-1931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess updated NIFI-1931:
---
Affects Version/s: (was: 0.6.1)
   Status: Patch Available  (was: Open)

> QueryDatabaseTable processor; setFetchSize not working for postgres driver 
> causing out of memory
> 
>
> Key: NIFI-1931
> URL: https://issues.apache.org/jira/browse/NIFI-1931
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Paul Bormans
>Assignee: Jim Steinebrey
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> With NIFI-1691 the ability to specify the fetch size is added. However for 
> postgres driver (or at least for postgresql-9.4.1208.jre6.jar) this seems not 
> to be working since after some time out of memory is reported in the logs.
> See https://jdbc.postgresql.org/documentation/head/query.html also for some 
> constraints; like auto commit needs to be set to False.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[PR] NIFI-12929: Fix logout infinite redirect loop in case of knox [nifi]

2024-03-22 Thread via GitHub


taz1988 opened a new pull request, #8546:
URL: https://github.com/apache/nifi/pull/8546

   
   
   
   
   
   
   
   
   
   
   
   
   
   # Summary
   Fix logout infinite redirect loop in case of knox - happens only on the 1.x 
line
   
   [NIFI-12929](https://issues.apache.org/jira/browse/NIFI-12929)
   
   # Tracking
   
   Please complete the following tracking steps prior to pull request creation.
   
   ### Issue Tracking
   
   - [x] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI-12929) 
issue created
   
   ### Pull Request Tracking
   
   - [x] Pull Request title starts with Apache NiFi Jira issue number, such as 
`NIFI-0`
   - [x] Pull Request commit message starts with Apache NiFi Jira issue number, 
as such `NIFI-0`
   
   ### Pull Request Formatting
   
   - [x] Pull Request based on current revision of the `support-1.x` branch
   - [ ] Pull Request refers to a feature branch with one commit containing 
changes
   
   # Verification
   
   Please indicate the verification steps performed prior to pull request 
creation.
   
   ### Build
   
   - [x] Build completed using `mvn clean install -P contrib-check`
 - [x] JDK 8
   
   ### Licensing
   
   - [ ] New dependencies are compatible with the [Apache License 
2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License 
Policy](https://www.apache.org/legal/resolved.html)
   - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` 
files
   
   ### Documentation
   
   - [ ] Documentation formatting appears as expected in rendered files
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] MINIFICPP-2309 Add JSON flow config examples [nifi-minifi-cpp]

2024-03-22 Thread via GitHub


fgerlits commented on code in PR #1740:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1740#discussion_r1535688898


##
examples/README.md:
##
@@ -37,122 +43,158 @@ The following examples show simple flow configurations 
for several common use ca
   - [Upload Blob to Azure Storage](#upload-blob-to-azure-storage)
   - [Put Object in AWS S3 Bucket](#put-object-in-aws-s3-bucket)
   - [List and Fetch Content from AWS S3 
Bucket](#list-and-fetch-content-from-aws-s3-bucket)
+  - [Merge, compress and upload files to Google Cloud 
Storage](#merge-compress-and-upload-files-to-google-cloud-storage)
 - [SQL Operations](#sql-operations)
   - [Query Database Table](#query-database-table)
-- [ExecuteScript](#ExecuteScript)
-  - [Reverse flowfile's content with lua](#reverse-content-with-lua)
-  - [Reverse flowfile's content with python](#reverse-content-with-python)
+- [ExecuteScript](#executescript)
+  - [Reverse Content with Scripts](#reverse-content-with-scripts)
+- [Grafana Loki](#grafana-loki)
+  - [Send logs to Grafana Loki](#send-logs-to-grafana-loki)
+- [MQTT Operations](#mqtt-operations)
+  - [Publish Message to MQTT Broker](#publish-message-to-mqtt-broker)
+- [Network Operations](#network-operations)
+  - [Send Data to Remote Host through 
TCP](#send-data-to-remote-host-through-tcp)
 
 ## Filesystem Operations
 
 ### Getting File Data and Putting It in an Output Directory
 
-Using the [getfile_putfile_config.yml](getfile_putfile_config.yml) flow 
configuration MiNiFi gets all files of minimum 1MB size from the 
`/tmp/getfile_dir` directory and puts them in the `/tmp/out_dir` output 
directory.
+Using the 
[getfile_putfile_config.yml](getfile_putfile_config.yml)/[getfile_putfile_config.json](getfile_putfile_config.json)
 flow configuration MiNiFi gets all files of minimum 1MB size from the 
`/tmp/getfile_dir` directory and puts them in the `/tmp/out_dir` output 
directory.
 
-The flow: GetFile  success  PutFile
+The flow: [GetFile](../PROCESSORS.md#getfile)  
[PutFile](../PROCESSORS.md#putfile)
 
 ### Tailing a Single File
 
-Using the [tailfile_config.yml](tailfile_config.yml) flow configuration MiNiFi 
tails a single file `/tmp/test_file.log` and creates flowfiles from every 
single line, then logs the flowfile attributes.
+Using the 
[tailfile_config.yml](tailfile_config.yml)/[tailfile_config.json](tailfile_config.json)
 flow configuration MiNiFi tails a single file `/tmp/test_file.log` and creates 
flowfiles from every single line, then logs the flowfile attributes.
 
-The flow: TailFile  success  LogAttribute
+The flow: [TailFile](../PROCESSORS.md#tailfile)  
[LogAttribute](../PROCESSORS.md#logattribute)
 
 ## Windows Specific Processors
 
 ### Consuming Windows Event Logs
 
-Using the [cwel_config.yml](cwel_config.yml) flow configuration MiNiFi queries 
all Windows system events and puts them to the `C:\temp\` directory in 
flattened JSON format.
+Using the 
[cwel_config.yml](cwel_config.yml)/[cwel_config.json](cwel_config.json) flow 
configuration MiNiFi queries all Windows system events and puts them to the 
`C:\temp\` directory in flattened JSON format.
 
-The flow: ConsumeWindowsEventLog  PutFile
+The flow: [ConsumeWindowsEventLog](../PROCESSORS.md#consumewindowseventlog) 
 [PutFile](../PROCESSORS.md#putfile)
 
 ### Reading System Performance Data
 
-Using the [pdh_config.yml](pdh_config.yml) flow configuration MiNiFi reads CPU 
and Disk performance data through Windows' Performance Data Helper (PDH) 
component and puts the data to the `C:\temp\` directory in a compact JSON 
format.
+Using the [pdh_config.yml](pdh_config.yml)/[pdh_config.json](pdh_config.json) 
flow configuration MiNiFi reads CPU and Disk performance data through Windows' 
Performance Data Helper (PDH) component and puts the data to the `C:\temp\` 
directory in a compact JSON format.
 
-The flow: PerformanceDataMonitor  PutFile
+The flow: [PerformanceDataMonitor](../PROCESSORS.md#performancedatamonitor) 
 [PutFile](../PROCESSORS.md#putfile)
 
 ## Linux Specific Processors
 
 ### Consume Systemd-Journald System Journal Messages
 
-Using the [consumejournald_config.yml](consumejournald_config.yml) flow 
configuration MiNiFi reads systemd-journald journal messages and logs them on 
`info` level.
+Using the 
[consumejournald_config.yml](consumejournald_config.yml)/[consumejournald_config.json](consumejournald_config.json)/[consumejournald_config.nifi.schema.json](consumejournald_config.nifi.schema.json)
 flow configuration MiNiFi reads systemd-journald journal messages and logs 
them on `info` level.
 
-The flow: ConsumeJournald  LogAttribute
+The flow: [ConsumeJournald](../PROCESSORS.md#consumejournald)  
[LogAttribute](../PROCESSORS.md#logattribute)
 
 ## HTTP Operations
 
 ### HTTP POST Invocation
 
-Using the [http_post_config.yml](http_post_config.yml) flow configuration 
MiNiFi transfers flowfile data received from the GetFile processor by invoking 
an HTTP endpoint with POST method.
+Using the 

[PR] NIFI-12933 Rearranged properties on GCP processors [nifi]

2024-03-22 Thread via GitHub


turcsanyip opened a new pull request, #8545:
URL: https://github.com/apache/nifi/pull/8545

   Also used current API methods for relationships/properties collections
   
   # Summary
   
   [NIFI-12933](https://issues.apache.org/jira/browse/NIFI-12933)
   
   # Tracking
   
   Please complete the following tracking steps prior to pull request creation.
   
   ### Issue Tracking
   
   - [ ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue 
created
   
   ### Pull Request Tracking
   
   - [ ] Pull Request title starts with Apache NiFi Jira issue number, such as 
`NIFI-0`
   - [ ] Pull Request commit message starts with Apache NiFi Jira issue number, 
as such `NIFI-0`
   
   ### Pull Request Formatting
   
   - [ ] Pull Request based on current revision of the `main` branch
   - [ ] Pull Request refers to a feature branch with one commit containing 
changes
   
   # Verification
   
   Please indicate the verification steps performed prior to pull request 
creation.
   
   ### Build
   
   - [ ] Build completed using `mvn clean install -P contrib-check`
 - [ ] JDK 21
   
   ### Licensing
   
   - [ ] New dependencies are compatible with the [Apache License 
2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License 
Policy](https://www.apache.org/legal/resolved.html)
   - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` 
files
   
   ### Documentation
   
   - [ ] Documentation formatting appears as expected in rendered files
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Updated] (NIFI-12933) Rearrange properties on GCP processors

2024-03-22 Thread Peter Turcsanyi (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-12933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Turcsanyi updated NIFI-12933:
---
Status: Patch Available  (was: Open)

> Rearrange properties on GCP processors
> --
>
> Key: NIFI-12933
> URL: https://issues.apache.org/jira/browse/NIFI-12933
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Peter Turcsanyi
>Assignee: Peter Turcsanyi
>Priority: Minor
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Parent classes of the GCP processors define some common properties and they 
> are added to the processors via 
> {{{}descriptors.addAll(super.getSupportedPropertyDescriptors()){}}}.
> This practice leads to weird property ordering in most cases and it is not 
> worth using it. E.g. "Number of retries" precedes "Bucket" and "Key" 
> properties in PutGCSObject.
> Get rid of {{super.getSupportedPropertyDescriptors()}} and reorder the 
> properties in a more reasonable way.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (NIFI-12933) Rearrange properties on GCP processors

2024-03-22 Thread Peter Turcsanyi (Jira)
Peter Turcsanyi created NIFI-12933:
--

 Summary: Rearrange properties on GCP processors
 Key: NIFI-12933
 URL: https://issues.apache.org/jira/browse/NIFI-12933
 Project: Apache NiFi
  Issue Type: Improvement
Reporter: Peter Turcsanyi
Assignee: Peter Turcsanyi


Parent classes of the GCP processors define some common properties and they are 
added to the processors via 
{{{}descriptors.addAll(super.getSupportedPropertyDescriptors()){}}}.
This practice leads to weird property ordering in most cases and it is not 
worth using it. E.g. "Number of retries" precedes "Bucket" and "Key" properties 
in PutGCSObject.
Get rid of {{super.getSupportedPropertyDescriptors()}} and reorder the 
properties in a more reasonable way.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Re: [PR] NIFI-12928 Added Simple Write strategy in PutAzureDataLakeStorage [nifi]

2024-03-22 Thread via GitHub


turcsanyip commented on PR #8540:
URL: https://github.com/apache/nifi/pull/8540#issuecomment-2014868697

   @takraj @Lehel44 Thanks for your review and comments! Updated the PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[PR] NIFI-12923 Added append avro mode to PutHDFS [nifi]

2024-03-22 Thread via GitHub


balazsgerner opened a new pull request, #8544:
URL: https://github.com/apache/nifi/pull/8544

   
   
   
   
   
   
   
   
   
   
   
   
   
   # Summary
   
   [NIFI-12923](https://issues.apache.org/jira/browse/NIFI-12923)
   
   # Tracking
   
   Please complete the following tracking steps prior to pull request creation.
   
   ### Issue Tracking
   
   - [x] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue 
created
   
   ### Pull Request Tracking
   
   - [x] Pull Request title starts with Apache NiFi Jira issue number, such as 
`NIFI-0`
   - [x] Pull Request commit message starts with Apache NiFi Jira issue number, 
as such `NIFI-0`
   
   ### Pull Request Formatting
   
   - [x] Pull Request based on current revision of the `main` branch
   - [x] Pull Request refers to a feature branch with one commit containing 
changes
   
   # Verification
   
   Please indicate the verification steps performed prior to pull request 
creation.
   
   ### Build
   
   - [x] Build completed using `mvn clean install -P contrib-check`
 - [x] JDK 21
   
   ### Licensing
   
   - [ ] New dependencies are compatible with the [Apache License 
2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License 
Policy](https://www.apache.org/legal/resolved.html)
   - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` 
files
   
   ### Documentation
   
   - [x] Documentation formatting appears as expected in rendered files
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Updated] (NIFI-12923) PutHDFS to support appending avro data

2024-03-22 Thread Jira


 [ 
https://issues.apache.org/jira/browse/NIFI-12923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Balázs Gerner updated NIFI-12923:
-
Description: 
The goal of this ticket is to extend the PutHDFS processor with the ability to 
append avro records. The processor already provides an option to set 'append' 
as conflict resolution strategy, but that does not work correctly in case of 
avro files, because the serialized avro file cannot be deserialized again 
(because the binary content is invalid).

Some notes about the implementation:
 * The user needs to explicitly select avro as file format and append as 
conflict resolution mode to enable 'avro append' mode, otherwise regular append 
mode will work just as before. There is no auto detection of mimetype for the 
incoming flowfile.
 * The records of the incoming flowfile and the ones in the existing avro file 
need to conform to the same avro schema, otherwise the append operation fails 
with incompatible schema.
 * The 'avro append' mode should only work when compression type is set to 
'none', if any other compression type is selected in 'avro append' mode the 
user should get a validation error.

The changes will have to be added to *support/nifi-1.x* branch also.

  was:
The goal of this ticket is to extend the PutHDFS processor with the ability to 
append avro records. The processor already provides an option to set 'append' 
as conflict resolution strategy, but that does not work correctly in case of 
avro files, because the serialized avro file cannot be deserialized again 
(because the binary content is invalid).

Some notes about the implementation:
 * The user needs to explicitly select avro as file format and append as 
conflict resolution mode to enable 'avro append' mode, otherwise regular append 
mode will work just as before. There is no auto detection of mimetype for the 
incoming flowfile.
 * The records of the incoming flowfile and the ones in the existing avro file 
need to conform to the same avro schema, otherwise the append operation fails 
with incompatible schema.
 * The 'avro append' mode should only work when compression type is set to 
'none', if any other compression type is selected in 'avro append' mode the 
user should get a validation error.


> PutHDFS to support appending avro data
> --
>
> Key: NIFI-12923
> URL: https://issues.apache.org/jira/browse/NIFI-12923
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Balázs Gerner
>Assignee: Balázs Gerner
>Priority: Major
>
> The goal of this ticket is to extend the PutHDFS processor with the ability 
> to append avro records. The processor already provides an option to set 
> 'append' as conflict resolution strategy, but that does not work correctly in 
> case of avro files, because the serialized avro file cannot be deserialized 
> again (because the binary content is invalid).
> Some notes about the implementation:
>  * The user needs to explicitly select avro as file format and append as 
> conflict resolution mode to enable 'avro append' mode, otherwise regular 
> append mode will work just as before. There is no auto detection of mimetype 
> for the incoming flowfile.
>  * The records of the incoming flowfile and the ones in the existing avro 
> file need to conform to the same avro schema, otherwise the append operation 
> fails with incompatible schema.
>  * The 'avro append' mode should only work when compression type is set to 
> 'none', if any other compression type is selected in 'avro append' mode the 
> user should get a validation error.
> The changes will have to be added to *support/nifi-1.x* branch also.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Re: [PR] NIFI-12928 Added Simple Write strategy in PutAzureDataLakeStorage [nifi]

2024-03-22 Thread via GitHub


turcsanyip commented on code in PR #8540:
URL: https://github.com/apache/nifi/pull/8540#discussion_r1535144271


##
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/storage/PutAzureDataLakeStorage.java:
##
@@ -229,38 +259,65 @@ static void uploadContent(DataLakeFileClient fileClient, 
InputStream in, long le
 }
 
 /**
- * This method serves as a "commit" for the upload process. Upon upload, a 
0-byte file is created, then the payload is appended to it.
- * Because of that, a work-in-progress file is available for readers 
before the upload is complete. It is not an efficient approach in
- * case of conflicts because FlowFiles are uploaded unnecessarily, but it 
is a calculated risk because consistency is more important.
+ * Creates the file on Azure for 'Simple Write' strategy. Upon upload, a 
0-byte file is created, then the payload is appended to it.
+ * Because of that, a work-in-progress file is available for readers 
before the upload is complete.
+ *
+ * @param directoryClient directory client of the uploaded file's parent 
directory
+ * @param fileName name of the uploaded file
+ * @param conflictResolution conflict resolution strategy
+ * @return the file client of the uploaded file or {@code null} if the 
file already exists and conflict resolution strategy is 'ignore'
+ * @throws ProcessException if the file already exists and the conflict 
resolution strategy is 'fail'; also in case of other errors
+ */
+DataLakeFileClient createFile(DataLakeDirectoryClient directoryClient, 
final String fileName, final String conflictResolution) {
+final String destinationPath = 
createPath(directoryClient.getDirectoryPath(), fileName);
+
+try {
+final boolean overwrite = 
conflictResolution.equals(REPLACE_RESOLUTION);
+return directoryClient.createFile(fileName, overwrite);
+} catch (DataLakeStorageException dataLakeStorageException) {
+return handleDataLakeStorageException(dataLakeStorageException, 
destinationPath, conflictResolution);
+}
+}
+
+/**
+ * This method serves as a "commit" for the upload process in case of 
'Write and Rename' strategy. In order to prevent work-in-progress files from 
being available for readers,
+ * a temporary file is written first, and then renamed/moved to its final 
destination. It is not an efficient approach in case of conflicts because 
FlowFiles are uploaded unnecessarily,
+ * but it is a calculated risk because consistency is more important for 
'Write and Rename' strategy.
  * 
  * Visible for testing
  *
- * @param sourceFileClient client of the temporary file
+ * @param sourceFileClient file client of the temporary file
  * @param destinationDirectory final location of the uploaded file
  * @param destinationFileName final name of the uploaded file
  * @param conflictResolution conflict resolution strategy
- * @return URL of the uploaded file
+ * @return the file client of the uploaded file or {@code null} if the 
file already exists and conflict resolution strategy is 'ignore'
+ * @throws ProcessException if the file already exists and the conflict 
resolution strategy is 'fail'; also in case of other errors
  */
-String renameFile(final DataLakeFileClient sourceFileClient, final String 
destinationDirectory, final String destinationFileName, final String 
conflictResolution) {
+DataLakeFileClient renameFile(final DataLakeFileClient sourceFileClient, 
final String destinationDirectory, final String destinationFileName, final 
String conflictResolution) {
 final String destinationPath = createPath(destinationDirectory, 
destinationFileName);
 
 try {
 final DataLakeRequestConditions destinationCondition = new 
DataLakeRequestConditions();
 if (!conflictResolution.equals(REPLACE_RESOLUTION)) {
 destinationCondition.setIfNoneMatch("*");
 }
-return sourceFileClient.renameWithResponse(null, destinationPath, 
null, destinationCondition, null, null).getValue().getFileUrl();
+return sourceFileClient.renameWithResponse(null, destinationPath, 
null, destinationCondition, null, null).getValue();
 } catch (DataLakeStorageException dataLakeStorageException) {
-removeTempFile(sourceFileClient);
-if (dataLakeStorageException.getStatusCode() == 409 && 
conflictResolution.equals(IGNORE_RESOLUTION)) {
-getLogger().info("File [{}] already exists. Remote file not 
modified due to {} being set to '{}'.",
-destinationPath, CONFLICT_RESOLUTION.getDisplayName(), 
conflictResolution);
-return null;
-} else if (dataLakeStorageException.getStatusCode() == 409 && 
conflictResolution.equals(FAIL_RESOLUTION)) {
-throw 

Re: [PR] NIFI-12928 Added Simple Write strategy in PutAzureDataLakeStorage [nifi]

2024-03-22 Thread via GitHub


turcsanyip commented on code in PR #8540:
URL: https://github.com/apache/nifi/pull/8540#discussion_r1535123254


##
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/AbstractAzureDataLakeStorageIT.java:
##
@@ -53,6 +54,7 @@ protected String getDefaultEndpointSuffix() {
 protected void setUpCredentials() throws Exception {
 ADLSCredentialsService service = new 
ADLSCredentialsControllerService();
 runner.addControllerService("ADLSCredentials", service);
+runner.setProperty(service, AzureStorageUtils.CREDENTIALS_TYPE, 
AzureStorageCredentialsType.ACCOUNT_KEY);

Review Comment:
   It was a bug in the ADLS integration tests due to 
[NIFI-8278](https://issues.apache.org/jira/browse/NIFI-8278) but I just spotted 
it now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



Re: [PR] NIFI-12928 Added Simple Write strategy in PutAzureDataLakeStorage [nifi]

2024-03-22 Thread via GitHub


turcsanyip commented on code in PR #8540:
URL: https://github.com/apache/nifi/pull/8540#discussion_r1535120414


##
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/storage/PutAzureDataLakeStorage.java:
##
@@ -99,6 +100,15 @@ public class PutAzureDataLakeStorage extends 
AbstractAzureDataLakeStorageProcess
 .allowableValues(FAIL_RESOLUTION, REPLACE_RESOLUTION, 
IGNORE_RESOLUTION)
 .build();
 
+protected static final PropertyDescriptor WRITING_STRATEGY = new 
PropertyDescriptor.Builder()
+.name("writing-strategy")
+.displayName("Writing Strategy")

Review Comment:
   I would rather follow the existing one in the processor.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org