[jira] [Updated] (NIFI-12479) Add pg-export to toolkit CLI
[ https://issues.apache.org/jira/browse/NIFI-12479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timea Barna updated NIFI-12479: --- Status: Patch Available (was: In Progress) > Add pg-export to toolkit CLI > > > Key: NIFI-12479 > URL: https://issues.apache.org/jira/browse/NIFI-12479 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Timea Barna >Assignee: Timea Barna >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Add a CLI command allowing to save a process group in NiFi as a JSON file. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] NIFI-12479 Add pg-export to toolkit CLI [nifi]
timeabarna opened a new pull request, #8137: URL: https://github.com/apache/nifi/pull/8137 # Summary Add a CLI command allowing to save a process group in NiFi as a JSON file. [NIFI-12479](https://issues.apache.org/jira/browse/NIFI-12479) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [ ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [ ] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [ ] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [ ] Pull Request based on current revision of the `main` branch - [ ] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 21 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] NIFI-12487 Add CSRF Protection to Registry [nifi]
exceptionfactory opened a new pull request, #8136: URL: https://github.com/apache/nifi/pull/8136 # Summary [NIFI-12487](https://issues.apache.org/jira/browse/NIFI-12487) Adds Cross-Site Request Forgery protection to NiFi Registry using the Spring Security CSRF Filter and several components that follow the same approach currently implemented for NiFi CSRF protection. NiFi Registry does not use cookies for passing Application Bearer Tokens, and instead relies on the HTTP Authorization header to be populated using a custom JavaScript request interceptor. This approach mitigates a number of potential threats. Introducing the CSRF Filter provides an additional layer of protection. Following a strategy similar to NiFi, the CSRF Request Matcher is based on the default Spring Security HTTP Method matching plus the presence of the Request Token cookie. This enables programmatic clients to continue working without reconfiguration, while requiring browser-based clients to pass the `Request-Token` header, as implemented through updates to the JavaScript request interceptor. # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [X] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [X] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [X] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [X] Pull Request based on current revision of the `main` branch - [X] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [X] Build completed using `mvn clean install -P contrib-check` - [X] JDK 21 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-12487) Add CSRF Filter to Registry Configuration
[ https://issues.apache.org/jira/browse/NIFI-12487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-12487: Status: Patch Available (was: In Progress) > Add CSRF Filter to Registry Configuration > - > > Key: NIFI-12487 > URL: https://issues.apache.org/jira/browse/NIFI-12487 > Project: Apache NiFi > Issue Type: Improvement > Components: NiFi Registry, Security >Reporter: David Handermann >Assignee: David Handermann >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > NiFi Registry supports several authentication strategies including username > and password, X.509 certificates, and OpenID Connect. Strategies that involve > exchanging temporary credentials produce an Application Bearer Token, which > the Registry interface stores and sends on subsequent requests for the > duration of the session. The Registry interface passes the Bearer Token using > the standard HTTP Authorization header, which requires custom JavaScript > request processing. This approach mitigates general concerns related to > Cross-Site Request Forgery as external requests from a web browser cannot > send the Authorization header. > Despite general protection based on the current implementation, adding > standard Cross-Site Request Forgery checking using Spring Security would > provide additional defenses. Enabling CSRF protection also aligns with > existing capabilities in NiFi, and would provide a basis for future align of > Bearer Token handling strategies. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-12487) Add CSRF Filter to Registry Configuration
David Handermann created NIFI-12487: --- Summary: Add CSRF Filter to Registry Configuration Key: NIFI-12487 URL: https://issues.apache.org/jira/browse/NIFI-12487 Project: Apache NiFi Issue Type: Improvement Components: NiFi Registry, Security Reporter: David Handermann Assignee: David Handermann NiFi Registry supports several authentication strategies including username and password, X.509 certificates, and OpenID Connect. Strategies that involve exchanging temporary credentials produce an Application Bearer Token, which the Registry interface stores and sends on subsequent requests for the duration of the session. The Registry interface passes the Bearer Token using the standard HTTP Authorization header, which requires custom JavaScript request processing. This approach mitigates general concerns related to Cross-Site Request Forgery as external requests from a web browser cannot send the Authorization header. Despite general protection based on the current implementation, adding standard Cross-Site Request Forgery checking using Spring Security would provide additional defenses. Enabling CSRF protection also aligns with existing capabilities in NiFi, and would provide a basis for future align of Bearer Token handling strategies. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-11308 Adding expression language function to return nifi version [nifi]
exceptionfactory commented on code in PR #8101:
URL: https://github.com/apache/nifi/pull/8101#discussion_r1418145271
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-nar-utils/src/main/java/org/apache/nifi/nar/NarClassLoaders.java:
##
@@ -74,6 +74,7 @@ private InitContext(
this.jettyBundle = jettyBundle;
this.serverInstance = serverInstance;
this.bundles = bundles;
+System.setProperty("nifi.version",
frameworkBundle.getBundleDetails().getCoordinate().getVersion());
Review Comment:
Good point @dan-s1. The [Expression Language
Guide](https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#usage)
notes that System properties can be referenced, but does not list any of them.
This probably warrants a new short sub-section along the lines of `Framework
System Properties`, above the `Functions` section.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (NIFI-12486) Registry Clients
Matt Gilman created NIFI-12486: -- Summary: Registry Clients Key: NIFI-12486 URL: https://issues.apache.org/jira/browse/NIFI-12486 Project: Apache NiFi Issue Type: Sub-task Components: Core UI Reporter: Matt Gilman Assignee: Matt Gilman Add support for registry clients. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-12485) Lineage Graph
Matt Gilman created NIFI-12485: -- Summary: Lineage Graph Key: NIFI-12485 URL: https://issues.apache.org/jira/browse/NIFI-12485 Project: Apache NiFi Issue Type: Sub-task Components: Core UI Reporter: Matt Gilman Assignee: Matt Gilman Implement the lineage graph that is opened from a Provenance event. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] NIFI-12484: Bumping dependency to latest minor/incremental release [nifi]
mcgilman opened a new pull request, #8135: URL: https://github.com/apache/nifi/pull/8135 NIFI-12484: - Bumping dependency to latest minor/incremental release. - Adding an explicit override for vite which is needed until Angular can be upgraded to version 17. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-11308 Adding expression language function to return nifi version [nifi]
dan-s1 commented on code in PR #8101:
URL: https://github.com/apache/nifi/pull/8101#discussion_r1418103963
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-nar-utils/src/main/java/org/apache/nifi/nar/NarClassLoaders.java:
##
@@ -74,6 +74,7 @@ private InitContext(
this.jettyBundle = jettyBundle;
this.serverInstance = serverInstance;
this.bundles = bundles;
+System.setProperty("nifi.version",
frameworkBundle.getBundleDetails().getCoordinate().getVersion());
Review Comment:
I see there is a section for System properties in the [user
guide](https://nifi.apache.org/docs/nifi-docs/html/user-guide.html) but I am
not sure that would be the place for this particular system property.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-11308 Adding expression language function to return nifi version [nifi]
dan-s1 commented on code in PR #8101:
URL: https://github.com/apache/nifi/pull/8101#discussion_r1418103963
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-nar-utils/src/main/java/org/apache/nifi/nar/NarClassLoaders.java:
##
@@ -74,6 +74,7 @@ private InitContext(
this.jettyBundle = jettyBundle;
this.serverInstance = serverInstance;
this.bundles = bundles;
+System.setProperty("nifi.version",
frameworkBundle.getBundleDetails().getCoordinate().getVersion());
Review Comment:
I am not sure. I see there is a section for System properties in the [user
guide](https://nifi.apache.org/docs/nifi-docs/html/user-guide.html) but I am
not sure that would be the place for this particular system property.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-11308 Adding expression language function to return nifi version [nifi]
annanys23 commented on code in PR #8101:
URL: https://github.com/apache/nifi/pull/8101#discussion_r1418090897
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-nar-utils/src/main/java/org/apache/nifi/nar/NarClassLoaders.java:
##
@@ -74,6 +74,7 @@ private InitContext(
this.jettyBundle = jettyBundle;
this.serverInstance = serverInstance;
this.bundles = bundles;
+System.setProperty("nifi.version",
frameworkBundle.getBundleDetails().getCoordinate().getVersion());
Review Comment:
agreed. where are the properties documented?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-11308 Adding expression language function to return nifi version [nifi]
dan-s1 commented on code in PR #8101:
URL: https://github.com/apache/nifi/pull/8101#discussion_r1418067510
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-nar-utils/src/main/java/org/apache/nifi/nar/NarClassLoaders.java:
##
@@ -74,6 +74,7 @@ private InitContext(
this.jettyBundle = jettyBundle;
this.serverInstance = serverInstance;
this.bundles = bundles;
+System.setProperty("nifi.version",
frameworkBundle.getBundleDetails().getCoordinate().getVersion());
Review Comment:
@exceptionfactory A concern I have is how will users know this property
exists in order to take advantage of it? Shouldn't this PR include documenting
this system property is now available?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (NIFI-12445) Provenance Event Table
[ https://issues.apache.org/jira/browse/NIFI-12445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Gilman updated NIFI-12445: --- Status: Patch Available (was: In Progress) > Provenance Event Table > -- > > Key: NIFI-12445 > URL: https://issues.apache.org/jira/browse/NIFI-12445 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Matt Gilman >Assignee: Matt Gilman >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Introduce capabilities to search provenance and view event listing. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-12484) Bump dependency versions
Matt Gilman created NIFI-12484: -- Summary: Bump dependency versions Key: NIFI-12484 URL: https://issues.apache.org/jira/browse/NIFI-12484 Project: Apache NiFi Issue Type: Sub-task Components: Core UI Reporter: Matt Gilman Assignee: Matt Gilman -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12483) Streamline JMX Metrics Filtering Parameters
[ https://issues.apache.org/jira/browse/NIFI-12483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-12483: Status: Patch Available (was: In Progress) > Streamline JMX Metrics Filtering Parameters > --- > > Key: NIFI-12483 > URL: https://issues.apache.org/jira/browse/NIFI-12483 > Project: Apache NiFi > Issue Type: Improvement >Reporter: David Handermann >Assignee: David Handermann >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > The JMX Metrics REST Resource provides selected access to registered bean > statistics based on application configuration properties. The default > configuration in application properties should be sufficient in most > scenarios to return only those beans necessary for targeted monitoring. In > other scenarios, allowing limited filtering based on HTTP request parameters > is supported. The HTTP request parameters should be processed prior to > filtering to provide optimal matching. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] NIFI-12483 Streamline JMX Metrics Filtering Parameters [nifi]
exceptionfactory opened a new pull request, #8134: URL: https://github.com/apache/nifi/pull/8134 # Summary [NIFI-12483](https://issues.apache.org/jira/browse/NIFI-12483) Streamlines JMX Metrics filtering parameter handling to run simpler string comparison when the request bean name filter parameter is provided. Application properties contain a default filter pattern that continues to apply before any request filter parameters, which is the optimal approach for providing a scoped set of JMX Metrics. The streamlined parameter parsing supports most existing use cases and also includes additional tests for a pattern with trailing wildcards. # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [X] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [X] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [X] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [X] Pull Request based on current revision of the `main` branch - [X] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [X] Build completed using `mvn clean install -P contrib-check` - [X] JDK 21 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-12400) Remaining items to migrate UI to currently supported/active framework
[ https://issues.apache.org/jira/browse/NIFI-12400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Gilman updated NIFI-12400: --- Description: The purpose of this Jira is to track all remaining items following the initial commit [1] for NIFI-11481. The description will be kept up to date with remaining features, tasks, and improvements. As each items is worked, a new sub task Jira will be created and referenced in this description. * Support Parameters in Properties with Allowable Values (NIFI-12401) * Summary (NIFI-12437) * Counters (NIFI-12415) * Bulletin Board * Provenance (NIFI-12445) ** Event Listing (NIFI-12445) ** Search (NIFI-12445) ** Event Dialog (NIFI-12445) ** Lineage ** Replay from context menu (NIFI-12445) * Flow Analysis Rules * Registry Clients * Import from Registry * Parameter Providers * Cluster * Flow Configuration History * Node Status History * Users * Policies * Help * About * Show Upstream/Downstream * Align * List Queue * Empty [all] Queue * View Content * View State * Change Version * PG Version ** Start ** Commit ** Force Commit ** Show changes ** Revert changes ** Change Flow version ** Stop * Configure PG (NIFI-12417) * Process Group Services (NIFI-12425) ** Listing (NIFI-12425) ** Create (NIFI-12425) ** Configure (NIFI-12425) ** Delete (NIFI-12425) ** Enable ** Disable ** Improve layout and breadcrumbs * Configure Processor ** Service Link (NIFI-12425) ** Create inline Service (NIFI-12425) ** Parameter Link ** Convert to Parameter * Download Flow * Create RPG * Configure RPG * RPG Remote Ports * RPG Go To * Color * Move to Front * Copy/Paste * Run unit tests are part of standard build * Update all API calls to consider disconnect node confirmation * Update API calls to use uiOnly flag * Routing error handling * Prevent navigate to Service when configuration form is dirty * Introduce header in new pages to unify with canvas and offer better navigation. [1] [https://github.com/apache/nifi/pull/8053] was: The purpose of this Jira is to track all remaining items following the initial commit [1] for NIFI-11481. The description will be kept up to date with remaining features, tasks, and improvements. As each items is worked, a new sub task Jira will be created and referenced in this description. * Support Parameters in Properties with Allowable Values (NIFI-12401) * Summary (NIFI-12437) * Counters (NIFI-12415) * Bulletin Board * Provenance (NIFI-12445) ** Event Listing (NIFI-12445) ** Search (NIFI-12445) ** Event Dialog (NIFI-12445) ** Lineage ** Replay from context menu (NIFI-12445) * Flow Analysis Rules * Registry Clients * Parameter Providers * Cluster * Flow Configuration History * Node Status History * Users * Policies * Help * About * Show Upstream/Downstream * Align * List Queue * Empty [all] Queue * View Content * View State * Change Version * PG Version ** Start ** Commit ** Force Commit ** Show changes ** Revert changes ** Change Flow version ** Stop * Configure PG (NIFI-12417) * Process Group Services (NIFI-12425) ** Listing (NIFI-12425) ** Create (NIFI-12425) ** Configure (NIFI-12425) ** Delete (NIFI-12425) ** Enable ** Disable ** Improve layout and breadcrumbs * Configure Processor ** Service Link (NIFI-12425) ** Create inline Service (NIFI-12425) ** Parameter Link ** Convert to Parameter * Download Flow * Create RPG * Configure RPG * RPG Remote Ports * RPG Go To * Color * Move to Front * Copy/Paste * Run unit tests are part of standard build * Update all API calls to consider disconnect node confirmation * Update API calls to use uiOnly flag * Routing error handling * Prevent navigate to Service when configuration form is dirty * Introduce header in new pages to unify with canvas and offer better navigation. [1] [https://github.com/apache/nifi/pull/8053] > Remaining items to migrate UI to currently supported/active framework > - > > Key: NIFI-12400 > URL: https://issues.apache.org/jira/browse/NIFI-12400 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI >Reporter: Matt Gilman >Assignee: Matt Gilman >Priority: Major > > The purpose of this Jira is to track all remaining items following the > initial commit [1] for NIFI-11481. The description will be kept up to date > with remaining features, tasks, and improvements. As each items is worked, a > new sub task Jira will be created and referenced in this description. > * Support Parameters in Properties with Allowable Values (NIFI-12401) > * Summary (NIFI-12437) > * Counters (NIFI-12415) > * Bulletin Board > * Provenance (NIFI-12445) > ** Event Listing (NIFI-12445) > ** Search (NIFI-12445) > ** Event Dialog
Re: [PR] NIFI-11308 Adding expression language function to return nifi version [nifi]
exceptionfactory commented on code in PR #8101:
URL: https://github.com/apache/nifi/pull/8101#discussion_r1418010127
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-nar-utils/src/main/java/org/apache/nifi/nar/NarClassLoaders.java:
##
@@ -74,6 +74,7 @@ private InitContext(
this.jettyBundle = jettyBundle;
this.serverInstance = serverInstance;
this.bundles = bundles;
+System.setProperty("nifi.version",
frameworkBundle.getBundleDetails().getCoordinate().getVersion());
Review Comment:
Recommend adding a comment indicating the reason for this addition. I would
recommend naming this `framework.version` to avoid potential confusion that
could result from different NAR extension versions, which are also part of
NiFi, but could be different in particular installations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-11308 Adding expression language function to return nifi version [nifi]
annanys23 commented on PR #8101: URL: https://github.com/apache/nifi/pull/8101#issuecomment-1843727016 Thanks for the review and suggestions! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-12445) Provenance Event Table
[ https://issues.apache.org/jira/browse/NIFI-12445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Gilman updated NIFI-12445: --- Summary: Provenance Event Table (was: Provenance Table and Lineage) > Provenance Event Table > -- > > Key: NIFI-12445 > URL: https://issues.apache.org/jira/browse/NIFI-12445 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Matt Gilman >Assignee: Matt Gilman >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Introduce capabilities to search provenance and view events and lineage. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12445) Provenance Event Table
[ https://issues.apache.org/jira/browse/NIFI-12445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Gilman updated NIFI-12445: --- Description: Introduce capabilities to search provenance and view event listing. (was: Introduce capabilities to search provenance and view events and lineage.) > Provenance Event Table > -- > > Key: NIFI-12445 > URL: https://issues.apache.org/jira/browse/NIFI-12445 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Matt Gilman >Assignee: Matt Gilman >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Introduce capabilities to search provenance and view event listing. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12400) Remaining items to migrate UI to currently supported/active framework
[ https://issues.apache.org/jira/browse/NIFI-12400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Gilman updated NIFI-12400: --- Description: The purpose of this Jira is to track all remaining items following the initial commit [1] for NIFI-11481. The description will be kept up to date with remaining features, tasks, and improvements. As each items is worked, a new sub task Jira will be created and referenced in this description. * Support Parameters in Properties with Allowable Values (NIFI-12401) * Summary (NIFI-12437) * Counters (NIFI-12415) * Bulletin Board * Provenance (NIFI-12445) ** Event Listing (NIFI-12445) ** Search (NIFI-12445) ** Event Dialog (NIFI-12445) ** Lineage ** Replay from context menu (NIFI-12445) * Flow Analysis Rules * Registry Clients * Parameter Providers * Cluster * Flow Configuration History * Node Status History * Users * Policies * Help * About * Show Upstream/Downstream * Align * List Queue * Empty [all] Queue * View Content * View State * Change Version * PG Version ** Start ** Commit ** Force Commit ** Show changes ** Revert changes ** Change Flow version ** Stop * Configure PG (NIFI-12417) * Process Group Services (NIFI-12425) ** Listing (NIFI-12425) ** Create (NIFI-12425) ** Configure (NIFI-12425) ** Delete (NIFI-12425) ** Enable ** Disable ** Improve layout and breadcrumbs * Configure Processor ** Service Link (NIFI-12425) ** Create inline Service (NIFI-12425) ** Parameter Link ** Convert to Parameter * Download Flow * Create RPG * Configure RPG * RPG Remote Ports * RPG Go To * Color * Move to Front * Copy/Paste * Run unit tests are part of standard build * Update all API calls to consider disconnect node confirmation * Update API calls to use uiOnly flag * Routing error handling * Prevent navigate to Service when configuration form is dirty * Introduce header in new pages to unify with canvas and offer better navigation. [1] [https://github.com/apache/nifi/pull/8053] was: The purpose of this Jira is to track all remaining items following the initial commit [1] for NIFI-11481. The description will be kept up to date with remaining features, tasks, and improvements. As each items is worked, a new sub task Jira will be created and referenced in this description. * Support Parameters in Properties with Allowable Values (NIFI-12401) * Summary (NIFI-12437) * Counters (NIFI-12415) * Bulletin Board * Provenance (NIFI-12445) ** Replay from context menu (NIFI-12445) * Flow Analysis Rules * Registry Clients * Parameter Providers * Cluster * Flow Configuration History * Node Status History * Users * Policies * Help * About * Show Upstream/Downstream * Align * List Queue * Empty [all] Queue * View Content * View State * Change Version * PG Version ** Start ** Commit ** Force Commit ** Show changes ** Revert changes ** Change Flow version ** Stop * Configure PG (NIFI-12417) * Process Group Services (NIFI-12425) ** Listing (NIFI-12425) ** Create (NIFI-12425) ** Configure (NIFI-12425) ** Delete (NIFI-12425) ** Enable ** Disable ** Improve layout and breadcrumbs * Configure Processor ** Service Link (NIFI-12425) ** Create inline Service (NIFI-12425) ** Parameter Link ** Convert to Parameter * Download Flow * Create RPG * Configure RPG * RPG Remote Ports * RPG Go To * Color * Move to Front * Copy/Paste * Run unit tests are part of standard build * Update all API calls to consider disconnect node confirmation * Update API calls to use uiOnly flag * Routing error handling * Prevent navigate to Service when configuration form is dirty * Introduce header in new pages to unify with canvas and offer better navigation. [1] [https://github.com/apache/nifi/pull/8053] > Remaining items to migrate UI to currently supported/active framework > - > > Key: NIFI-12400 > URL: https://issues.apache.org/jira/browse/NIFI-12400 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI >Reporter: Matt Gilman >Assignee: Matt Gilman >Priority: Major > > The purpose of this Jira is to track all remaining items following the > initial commit [1] for NIFI-11481. The description will be kept up to date > with remaining features, tasks, and improvements. As each items is worked, a > new sub task Jira will be created and referenced in this description. > * Support Parameters in Properties with Allowable Values (NIFI-12401) > * Summary (NIFI-12437) > * Counters (NIFI-12415) > * Bulletin Board > * Provenance (NIFI-12445) > ** Event Listing (NIFI-12445) > ** Search (NIFI-12445) > ** Event Dialog (NIFI-12445) > ** Lineage > ** Replay from context menu (NIFI-12445) > * Flow Analysis Rules > * Registry Clients > *
[PR] NIFI-12445: Provenance Event Listing [nifi]
mcgilman opened a new pull request, #8133: URL: https://github.com/apache/nifi/pull/8133 NIFI-12445: - Provenance Event Listing. - View Provenance Event dialog. - Provenance routing. - Provenance search. - Replay. - Download content. - View content. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12452 Improve support for DescribedValue [nifi]
EndzeitBegins commented on PR #8102: URL: https://github.com/apache/nifi/pull/8102#issuecomment-1843684252 I just rebased & squashed the commits, @exceptionfactory. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFI-12483) Streamline JMX Metrics Filtering Parameters
David Handermann created NIFI-12483: --- Summary: Streamline JMX Metrics Filtering Parameters Key: NIFI-12483 URL: https://issues.apache.org/jira/browse/NIFI-12483 Project: Apache NiFi Issue Type: Improvement Reporter: David Handermann Assignee: David Handermann The JMX Metrics REST Resource provides selected access to registered bean statistics based on application configuration properties. The default configuration in application properties should be sufficient in most scenarios to return only those beans necessary for targeted monitoring. In other scenarios, allowing limited filtering based on HTTP request parameters is supported. The HTTP request parameters should be processed prior to filtering to provide optimal matching. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-9269) ExtractEmailHeaders & ExtractEmailAttachments -- NoClassDefFoundError
[ https://issues.apache.org/jira/browse/NIFI-9269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793886#comment-17793886 ] David Handermann commented on NIFI-9269: Thanks for updating this issue and providing the stack trace [~sergserg]. For additional reference, the problem appears to be related to the use of Apache Commons Email, which has not been updated in several years. There are several Java Mail library versions in the nifi-email-module, so it seems like evaluating the removal of Apache Commons Email and making direct use of the Java Mail libraries could be a way forward. > ExtractEmailHeaders & ExtractEmailAttachments -- NoClassDefFoundError > - > > Key: NIFI-9269 > URL: https://issues.apache.org/jira/browse/NIFI-9269 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.14.0 > Environment: Nifi: > 1.14.0 > 07/10/2021 12:25:36 SAST > Tagged nifi-1.14.0-RC2 > From fcbf1d5 on branch UNKNOWN > Java: > java --version > openjdk 11.0.12 2021-07-20 > OpenJDK Runtime Environment 18.9 (build 11.0.12+7) > OpenJDK 64-Bit Server VM 18.9 (build 11.0.12+7, mixed mode, sharing) > OS: > uname -a > Linux blue.centilliard.io 5.13.16-200.fc34.x86_64 #1 SMP Mon Sep 13 > 12:39:36 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux >Reporter: Stefan >Assignee: Pierre Villard >Priority: Major > > The above two processors fail with the below errors: > /*** *ExtractEmailAttachments* ***/ > ExtractEmailAttachments[id=3738d7b2-017c-1000-c120-29f7760d3084] Failed > to process session due to com/sun/activation/registries/LogSupport; > Processor Administratively Yielded for 1 sec: > java.lang.NoClassDefFoundError: > com/sun/activation/registries/LogSupport > /*** *ExtractEmailHeaders* ***/ > ExtractEmailHeaders[id=36a7997a-017c-1000-0405-0197e1cb8792] Failed to > process session due to com/sun/activation/registries/LogSupport; > Processor Administratively Yielded for 1 sec: > java.lang.NoClassDefFoundError: > com/sun/activation/registries/LogSupport > *P.S* > Adding javax.activation-1.2.0.jar and leaving javax.activation-1.2.0.jar in > */lib/java11* also seems to resolve the problem. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-9269) ExtractEmailHeaders & ExtractEmailAttachments -- NoClassDefFoundError
[
https://issues.apache.org/jira/browse/NIFI-9269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793880#comment-17793880
]
Serg commented on NIFI-9269:
This (or a very similar) issue is presenting in
openjdk version "21.0.1" 2023-10-17 LTS
{code:java}
nifi-properties-2.0.0-M1{code}
{color:#1d1c1d}{color}which gives the following when using
either{color:#1d1c1d}
{color:#1d1c1d}ExtractEmailHeaders{color}{color} or
{color:#1d1c1d}{color:#1d1c1d}{color:#1d1c1d}ExtractEmailAttachments
{color}{color}
{color}
2023-12-06 17:21:10,367 ERROR [Timer-Driven Process Thread-67]
o.a.n.p.email.ExtractEmailHeaders
ExtractEmailHeaders[id=3fc27178-018c-1000-3413-6cc9f843e114] Processing halted:
yielding [1 sec]
java.lang.NoClassDefFoundError: com/sun/activation/registries/LogSupport
at javax.activation.MailcapCommandMap.(MailcapCommandMap.java:149)
at javax.activation.CommandMap.getDefaultCommandMap(CommandMap.java:55)
at javax.activation.DataHandler.getCommandMap(DataHandler.java:137)
at
javax.activation.DataHandler.getDataContentHandler(DataHandler.java:596)
at javax.activation.DataHandler.getContent(DataHandler.java:524)
at javax.mail.internet.MimeMessage.getContent(MimeMessage.java:1508)
at
org.apache.commons.mail.util.MimeMessageParser.parse(MimeMessageParser.java:180)
at
org.apache.commons.mail.util.MimeMessageParser.parse(MimeMessageParser.java:95)
at
org.apache.nifi.processors.email.ExtractEmailHeaders$1.process(ExtractEmailHeaders.java:171)
at
org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2687)
at
org.apache.nifi.processors.email.ExtractEmailHeaders.onTrigger(ExtractEmailHeaders.java:163)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1274)
at
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:244)
at
org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:59)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.ClassNotFoundException:
com.sun.activation.registries.LogSupport
... 22 common frames omitted
2023-12-06 17:21:10,367 WARN [Timer-Driven Process Thread-67]
o.a.n.controller.tasks.ConnectableTask Processing halted: uncaught exception in
Component [ExtractEmailHeaders[id=3fc27178-018c-1000-3413-6cc9f843e114]]
java.lang.NoClassDefFoundError: com/sun/activation/registries/LogSupport
at javax.activation.MailcapCommandMap.(MailcapCommandMap.java:149)
at javax.activation.CommandMap.getDefaultCommandMap(CommandMap.java:55)
at javax.activation.DataHandler.getCommandMap(DataHandler.java:137)
at
javax.activation.DataHandler.getDataContentHandler(DataHandler.java:596)
at javax.activation.DataHandler.getContent(DataHandler.java:524)
at javax.mail.internet.MimeMessage.getContent(MimeMessage.java:1508)
at
org.apache.commons.mail.util.MimeMessageParser.parse(MimeMessageParser.java:180)
at
org.apache.commons.mail.util.MimeMessageParser.parse(MimeMessageParser.java:95)
at
org.apache.nifi.processors.email.ExtractEmailHeaders$1.process(ExtractEmailHeaders.java:171)
at
org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2687)
at
org.apache.nifi.processors.email.ExtractEmailHeaders.onTrigger(ExtractEmailHeaders.java:163)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1274)
at
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:244)
at
org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:59)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at
[jira] [Commented] (NIFI-5291) Add stats for number object to CalculateRecordStats
[ https://issues.apache.org/jira/browse/NIFI-5291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793857#comment-17793857 ] Matt Burgess commented on NIFI-5291: Do we need this if we have QueryRecord? I suppose if there are memory issues for large files it would still be helpful to be able to calculate a few simple stats incrementally while reading records > Add stats for number object to CalculateRecordStats > --- > > Key: NIFI-5291 > URL: https://issues.apache.org/jira/browse/NIFI-5291 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Koji Kawamura >Priority: Minor > > Currently, CalculateRecordStats processor counts each string value > occurrence. In addition to that, we can add more stats for numeric values, > such as min, max and sum. Time, Timestamp and Date can have min and max > similarly. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-2735) Add processor to perform simple aggregations
[ https://issues.apache.org/jira/browse/NIFI-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-2735: --- Resolution: Won't Fix Status: Resolved (was: Patch Available) Resolving as QueryRecord and/or NIFI-5291 cover this > Add processor to perform simple aggregations > > > Key: NIFI-2735 > URL: https://issues.apache.org/jira/browse/NIFI-2735 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Matt Burgess >Priority: Major > > This is a proposal for a new processor (AggregateValues, for example) that > can perform simple aggregation operations such as count, sum, average, min, > max, and concatenate, over a set of "related" flow files. For example, when a > JSON file is split on an array (using the SplitJson processor), the total > count of the splits, the index of each split, and the unique identifier > (shared by each split) are stored as attributes in each flow file sent to the > "splits" relationship: > https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.SplitJson/index.html > These attributes are the "fragment.*" attributes in the documentation for > SplitText, SplitXml, and SplitJson, for example. > Such a processor could perform these operations for each flow file split from > the original document, and when all documents from a split have been > processed, a flow file could be transferred to an "aggregate" relationship > containing attributes for the operation, aggregate value, etc. > An interesting application of this (besides the actual aggregation > operations) is that you can use the "aggregate" relationship as an event > trigger. For example if you need to wait until all files from a group are > processed, you can use AggregateValues and the "aggregate" relationship to > indicate downstream that the entire group has been processed. If there is not > a Split processor upstream, then the attributes (fragment.*) would have to be > manipulated by the data flow designer, but this can be accomplished with > other processors (including the scripting processors if necessary). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-2735) Add processor to perform simple aggregations
[ https://issues.apache.org/jira/browse/NIFI-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess reassigned NIFI-2735: -- Assignee: (was: Matt Burgess) > Add processor to perform simple aggregations > > > Key: NIFI-2735 > URL: https://issues.apache.org/jira/browse/NIFI-2735 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Matt Burgess >Priority: Major > > This is a proposal for a new processor (AggregateValues, for example) that > can perform simple aggregation operations such as count, sum, average, min, > max, and concatenate, over a set of "related" flow files. For example, when a > JSON file is split on an array (using the SplitJson processor), the total > count of the splits, the index of each split, and the unique identifier > (shared by each split) are stored as attributes in each flow file sent to the > "splits" relationship: > https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.SplitJson/index.html > These attributes are the "fragment.*" attributes in the documentation for > SplitText, SplitXml, and SplitJson, for example. > Such a processor could perform these operations for each flow file split from > the original document, and when all documents from a split have been > processed, a flow file could be transferred to an "aggregate" relationship > containing attributes for the operation, aggregate value, etc. > An interesting application of this (besides the actual aggregation > operations) is that you can use the "aggregate" relationship as an event > trigger. For example if you need to wait until all files from a group are > processed, you can use AggregateValues and the "aggregate" relationship to > indicate downstream that the entire group has been processed. If there is not > a Split processor upstream, then the attributes (fragment.*) would have to be > manipulated by the data flow designer, but this can be accomplished with > other processors (including the scripting processors if necessary). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-5604) Allow GenerateTableFetch to send empty flow files when no rows would be fetched
[ https://issues.apache.org/jira/browse/NIFI-5604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-5604: --- Fix Version/s: 1.9.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Allow GenerateTableFetch to send empty flow files when no rows would be > fetched > --- > > Key: NIFI-5604 > URL: https://issues.apache.org/jira/browse/NIFI-5604 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Fix For: 1.9.0 > > > Currently, GenerateTableFetch will not output a flow file if there are no > rows to be fetched. However, it may be desired (especially with incoming flow > files) that a flow file be sent out even if GTF does not generate any SQL. > This capability, along with the fragment attributes from NIFI-5601, would > allow the user to handle this downstream. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-7054) Add RecordSinkServiceLookup
[ https://issues.apache.org/jira/browse/NIFI-7054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-7054: --- Fix Version/s: 1.12.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Add RecordSinkServiceLookup > --- > > Key: NIFI-7054 > URL: https://issues.apache.org/jira/browse/NIFI-7054 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Fix For: 1.12.0 > > Time Spent: 50m > Remaining Estimate: 0h > > The RecordSinkService controller service interface was added in NiFi 1.10 > (via NIFI-6780) to decouple the destination for records in a FlowFile from > the format of those records. Since then there have been various > implementations (NIFI-6799, NIFI-6819). Other controller services have been > augmented with a "lookup" pattern where the actual CS can be swapped out > during the flow based on an attribute/variable (such as > DBCPConnectionPoolLookup). > RecordSinkService could be improved to have such a lookup as well, especially > with the advent of the PutRecord processor (NIFI-6947). This would allow some > flow files to be routed to Kafka while others are sent Site-to-Site for > example, all with a single configured controller service. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-6670) Create a RecordReader that reads lines of text into single-field records
[ https://issues.apache.org/jira/browse/NIFI-6670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess reassigned NIFI-6670: -- Assignee: (was: Matt Burgess) > Create a RecordReader that reads lines of text into single-field records > > > Key: NIFI-6670 > URL: https://issues.apache.org/jira/browse/NIFI-6670 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Matt Burgess >Priority: Major > Time Spent: 2h 10m > Remaining Estimate: 0h > > It would be nice to have a reader that can take any textual input and treat > each "line" as a single-field record. This is like CSVReader but there > wouldn't be a field delimiter; rather, a property to specify the name of the > field, and each line becomes a value for that field in the record. > Additional capabilities could be added as well, such as skipping header > lines, grouping lines together as a single field value, ignoring empty lines, > etc. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-7516) Predictions model throws intermittent SingularMatrixExceptions
[ https://issues.apache.org/jira/browse/NIFI-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-7516: --- Resolution: Fixed Status: Resolved (was: Patch Available) > Predictions model throws intermittent SingularMatrixExceptions > -- > > Key: NIFI-7516 > URL: https://issues.apache.org/jira/browse/NIFI-7516 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Fix For: 1.12.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Under some circumstances, the Connection Status Analytics model (specifically > the Ordinary Least Squares model) throws a SingularMatrix exception: > org.apache.commons.math3.linear.SingularMatrixException: matrix is singular > This can happen (usually intermittently) when the data points used to update > the model form a matrix that has no inverse (i.e. singular). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-4946) nifi-spark-bundle : Adding support for pyfiles, file, jars options
[ https://issues.apache.org/jira/browse/NIFI-4946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess reassigned NIFI-4946: -- Assignee: (was: Matt Burgess) > nifi-spark-bundle : Adding support for pyfiles, file, jars options > -- > > Key: NIFI-4946 > URL: https://issues.apache.org/jira/browse/NIFI-4946 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions > Environment: Ubuntu 16.04, IntelliJ >Reporter: Mageswaran >Priority: Major > Attachments: nifi-spark-options.png, nifi-spark.png > > Time Spent: 20m > Remaining Estimate: 0h > > Adding support for submitting PySpark based Sparks jobs (which is normally > structured as modules) over Livy on existing "ExecuteSparkInteractive" > processor. > This is done by reading file paths for pyfiles and file and an option from > user whether the processor should trigger a batch job or not. > [https://livy.incubator.apache.org/docs/latest/rest-api.html] > *Current Work flow Logic ( [https://github.com/apache/nifi/pull/2521 > )|https://github.com/apache/nifi/pull/2521]* > * Check whether the processor has to handle code or submit a Spark job > * Read incoming flow file > ** If batch == true > *** If flow file matches Livy `batches` JSON response through `wait` loop > Wait for Status Check Interval > Read the state > If state is `running` route it to `wait` or if it is `success` or > `dead` route it accordingly > *** Else > Ignore the flow file > Trigger the Spark job over Livy `batches` endpoint > Read the state of the submitted job > If state is `running` route it to `wait` or if it is `success` or > `dead` route it accordingly > ** Else: > *** Existing Logic to handle `Code` > > !nifi-spark-options.png! > !nifi-spark.png! > > Thanks. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-9814) Add range sampling to SampleRecord
[ https://issues.apache.org/jira/browse/NIFI-9814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-9814: --- Fix Version/s: 1.19.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Add range sampling to SampleRecord > -- > > Key: NIFI-9814 > URL: https://issues.apache.org/jira/browse/NIFI-9814 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Fix For: 1.19.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > It would be nice to be able to specify exactly which records or ranges of > records are sampled from a FlowFile. For example if the first 5 lines of a > comma-delimited file are free-text (meaning it's not technically a CSV file > from NiFi's perspective as the first 5 lines do not constitute a header in > this example), it would be handy to be able to exclude them by specifying a > range filter of "6-" to say the 6th and every following record should be > output. > In that vein SampleRecord could have a "range sampling" strategy where the > user could specify something like "2, 5-7, 25-" where the second, fifth, > sixth, seventh, and every record from the twenty-fifth record on would be > included in the outgoing flow file. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-4491) Add a CaptureChangeMySQLRecord processor
[ https://issues.apache.org/jira/browse/NIFI-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess reassigned NIFI-4491: -- Assignee: (was: Matt Burgess) > Add a CaptureChangeMySQLRecord processor > > > Key: NIFI-4491 > URL: https://issues.apache.org/jira/browse/NIFI-4491 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Priority: Major > > The main reason CaptureChangeMySQL doesn't leverage the RecordSetWriter API > is that those capabilities were being developed in parallel with that > processor. Whether a new record-aware processor is better than an improvement > to the original is up for discussion; however, it would be a good idea to > support the RecordSetWriter API for any CDC (CaptureChangeXYZ) processor. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12236 Improving fault tolerancy of the QuestDB backed metrics repository [nifi]
exceptionfactory commented on code in PR #8123:
URL: https://github.com/apache/nifi/pull/8123#discussion_r1417672045
##
nifi-api/src/main/java/org/apache/nifi/controller/status/NodeStatus.java:
##
@@ -35,7 +35,6 @@ public class NodeStatus implements Cloneable {
private double processorLoadAverage;
private long totalThreads;
-private long eventDrivenThreads;
Review Comment:
This change is a helpful cleanup, which would be useful to handle
separately. As this particular pull request may take some time to review, would
you consider separating out this removal to its own pull request?
##
nifi-commons/nifi-questdb/pom.xml:
##
@@ -0,0 +1,71 @@
+
+
+http://maven.apache.org/POM/4.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
https://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+
+org.apache.nifi
+nifi-commons
+2.0.0-SNAPSHOT
+
+nifi-questdb
+
+
+
+org.apache.nifi
+nifi-utils
+2.0.0-SNAPSHOT
+
+
+
+org.apache.commons
+commons-lang3
+3.13.0
+
+
+commons-io
+commons-io
+2.13.0
+
+
+org.slf4j
+slf4j-api
+2.0.9
+
+
+org.questdb
+questdb
+7.2
+
+
+
+org.junit.jupiter
+junit-jupiter-engine
+5.9.2
+test
+
+
+org.mockito
+mockito-core
+5.6.0
+test
+
+
+org.mockito
+mockito-junit-jupiter
+5.6.0
+
Review Comment:
With the exception of `questdb`, all of these dependency versions are
managed at the root Maven configuration, so the specific version numbers should
be removed.
##
nifi-api/src/main/java/org/apache/nifi/controller/status/ConnectionStatus.java:
##
@@ -22,6 +22,7 @@
/**
*/
public class ConnectionStatus implements Cloneable {
+private long createdAtInMs;
Review Comment:
Although it might require some additional replication in the QuestDB
implementation itself, it is important to avoid introducing specific concepts
to the NiFi API that do not have general applicability. I would have to take a
closer look at how this is used, but in general, I believe any changes to the
nifi-api should be avoided for the purpose of this implementation.
##
nifi-commons/nifi-questdb/src/test/java/org/apache/nifi/questdb/QuestDbTestUtil.java:
##
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.questdb;
+
+import java.io.File;
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.UUID;
+
+public class QuestDbTestUtil {
+static final File TEST_DB_PATH = new File("src/test/resources/testdb");
Review Comment:
The JUnit 5 `@TempDir` annotation is preferable to hard-coded paths that
reference source directories, since that annotation provides lifecycle
management for temporary directories.
##
nifi-commons/nifi-questdb/src/test/java/org/apache/nifi/questdb/CompositeClientTest.java:
##
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the
[jira] [Assigned] (NIFI-11789) ExecuteSQL doesn't set fragment.count attribute when Output Batch Size is set
[ https://issues.apache.org/jira/browse/NIFI-11789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess reassigned NIFI-11789: --- Assignee: (was: Matt Burgess) > ExecuteSQL doesn't set fragment.count attribute when Output Batch Size is set > - > > Key: NIFI-11789 > URL: https://issues.apache.org/jira/browse/NIFI-11789 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Tamas Neumer >Priority: Minor > Time Spent: 0.5h > Remaining Estimate: 0h > > Hi, > I am working with the ExecuteSQL processor and discovered an unexpected > behavior. If I specify the attribute "Output Batch Size", I get the > fragment.index on the outflowing flowing Flowfiles, but the fragment.count > attribute is not set (according to the documentation). > The behavior I would expect (in line with how merge processors work) is that > the attribute fragment.count is just set at the last Flowfile for the batch. > This would make it possible to merge all the batches together afterward. > So my proposal, in short, is that the fragment.count should be set in the > last Flowfile of a batch. > BR Florian -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-12467) HadoopDBCPConnectionPool doesn't use KerberosUserService
[ https://issues.apache.org/jira/browse/NIFI-12467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess reassigned NIFI-12467: --- Assignee: Matt Burgess > HadoopDBCPConnectionPool doesn't use KerberosUserService > > > Key: NIFI-12467 > URL: https://issues.apache.org/jira/browse/NIFI-12467 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Fix For: 1.25.0, 2.0.0 > > > NIFI-8978 added some KerberosUserService integration with the > DBCPConnectionPool implementations, but for HadoopDBCPConnectionPool its > values are only used during customValidate(). They are not read in > onEnabled() and will fail when authenticating. > The workaround is to use the Principal and Keytab/Password properties or a > Keytab Credentials Service instead of a KerberosUserService. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12236) Improving fault tolerancy of the QuestDB backed metrics repository
[ https://issues.apache.org/jira/browse/NIFI-12236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793842#comment-17793842 ] Joe Witt commented on NIFI-12236: - I think it would help to better explain what the properties/tunables are. Maybe that is made available elsewhere but i'm not sure what batch size means for this and I am also curious whether the user can control how much data is retained and what the default is for that. A path forward that I think keeps things moving but doesn't create new risks currently not well enough explored 1. Avoid making this the default. It can be opted into. 2. Avoid making nifi-api changes. 3. Move this into its own nar instead of the core framework nar. It can certainly still be included as it is today. Remaining concerns can be addressed later and based on needs and findings from actual usage. Thanks > Improving fault tolerancy of the QuestDB backed metrics repository > -- > > Key: NIFI-12236 > URL: https://issues.apache.org/jira/browse/NIFI-12236 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Simon Bence >Assignee: Simon Bence >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > Based on the related discussion on the dev email list, the QuestDB handling > of the metrics repository needs to be improved to have better fault tolerance > in order to be possible to use as a viable option for default metrics data > store. This should primarily focus on handling unexpeted database events like > corrupted database or loss of space on the disk. Any issues should be handled > with an attempt to keep the database service healthy but in case of that is > impossible, the priority is to keep NiFi and the core services running, even > with the price of metrics collection / presentation outage. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12452 Improve support for DescribedValue [nifi]
EndzeitBegins commented on PR #8102: URL: https://github.com/apache/nifi/pull/8102#issuecomment-1843250321 Thank you for adjusting the migration guide @exceptionfactory. I'll try to land the rebased PR in a few hours time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12452 Improve support for DescribedValue [nifi]
EndzeitBegins commented on PR #8102: URL: https://github.com/apache/nifi/pull/8102#issuecomment-1843243596 Sounds reasonable to me. Thank you both @exceptionfactory & @markap14 for your valuable feedback. From my understanding the current PR proposal with the overload of `defaultValue` corresponds to the desired approach then. Could one of you take on the task to enhance the [migration guide](https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=57905503#MigrationGuidance-Migratingto2.0.0-M1) then? As far as I'm aware sole contributors cannot propose or make changes to Confluence? I've seen that there are conflicts with `main` now. I assume this is due to `HttpNotificationService` being removed by #8104. Should I rebase and force-push this PR to be compatible again or should I let the commiter resolve the issue? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (MINIFICPP-2267) Add openssl dylib to package on intel macos
[ https://issues.apache.org/jira/browse/MINIFICPP-2267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Zink resolved MINIFICPP-2267. Resolution: Fixed https://github.com/apache/nifi-minifi-cpp/commit/513cb5f5324d2ba8535daf4c9b73286efa8823a0 > Add openssl dylib to package on intel macos > --- > > Key: MINIFICPP-2267 > URL: https://issues.apache.org/jira/browse/MINIFICPP-2267 > Project: Apache NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Adam Debreceni >Assignee: Adam Debreceni >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (MINIFICPP-2268) Remove fetchcontent workaround for CMake versions under 3.18
[ https://issues.apache.org/jira/browse/MINIFICPP-2268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Zink resolved MINIFICPP-2268. Resolution: Fixed https://github.com/apache/nifi-minifi-cpp/commit/7ce91add169ded37334336d75432be9ed8adbd67 > Remove fetchcontent workaround for CMake versions under 3.18 > > > Key: MINIFICPP-2268 > URL: https://issues.apache.org/jira/browse/MINIFICPP-2268 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Gábor Gyimesi >Assignee: Gábor Gyimesi >Priority: Trivial > Time Spent: 20m > Remaining Estimate: 0h > > FetchContent_MakeAvailable cannot be used under CMake 3.18 with > SOURCE_SUBDIR in FetchContent_Declare, so a workaround is used in a couple of > libraries. This can be removed, as our current CMake version requirement is > 3.24. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12236) Improving fault tolerancy of the QuestDB backed metrics repository
[ https://issues.apache.org/jira/browse/NIFI-12236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793816#comment-17793816 ] Simon Bence commented on NIFI-12236: Thanks for the clarification! Let me give some context for the different questions. The idea: - The funcionality this provides almost identical to the original implementation not talking about error management. This is truth to the possible parametrization which however was not fully exposed to properties before. Now some of these settings (like the mentioned batch size) are exposed to provide the possibility to tune them in the case of need. The implementation options: - I completely agree with you on the part of in the long run it would be beneficial to have a variery of possible storage mechanism. Even so I think opening up the Status History Registry as a pluggable service would be a good step forward. It was not however the aim of this PR. The PR focused mainly on "bulletproofing" the existing implementation not extending the capabilities. The implementation: - I think this is the same case as the previous. The PR did not aim for extending the feature but make it more safe for production use. Personally I think the changes were added can be used as a good basis for future improvements but those should be the subject of further PRs. - In the PR I provided a short reasoning for the API changes to [~exceptionfactory] and I consider it as an ongoing discussion. The same result might be achieved without API changes but please find my thinking process there. I do not inist for this approach however I see some merits of it. - The Retry mechanism is actively used by the QuestDB implementation as part of error management. It is possible that I was not clear of that in the PR summary, but it is not something for a later commit. I separated it from the QuestDB related code in order to 1. have a more focused code structure 2. it is a possible tool for later usage within the project. The default selection: - This topic is already touched on the PR and I am completely okay with moving that part out from the commit. In the long term I would see benefits of making it a default (the mail thread started with this and I still have the same standing) but of course lets be sure about its safeness before moving on I hope I could give some useful information and clear some fog around the PR. Please if you still have conflicted feeling about the approach (and I deliberately do not use the phrase "final result", as I see some possible furute steps), share it. > Improving fault tolerancy of the QuestDB backed metrics repository > -- > > Key: NIFI-12236 > URL: https://issues.apache.org/jira/browse/NIFI-12236 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Simon Bence >Assignee: Simon Bence >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > Based on the related discussion on the dev email list, the QuestDB handling > of the metrics repository needs to be improved to have better fault tolerance > in order to be possible to use as a viable option for default metrics data > store. This should primarily focus on handling unexpeted database events like > corrupted database or loss of space on the disk. Any issues should be handled > with an attempt to keep the database service healthy but in case of that is > impossible, the priority is to keep NiFi and the core services running, even > with the price of metrics collection / presentation outage. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-11777) Component State Identifier
[ https://issues.apache.org/jira/browse/NIFI-11777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793809#comment-17793809 ] Joe Witt commented on NIFI-11777: - [~pvillard][~simonbence]Please before there is an implementation for this can we generate and discuss on a [feature proposal|https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals]. This way we can first be on the same page on the purpose/options for an idea before there is a ton of implementation work done. > Component State Identifier > -- > > Key: NIFI-11777 > URL: https://issues.apache.org/jira/browse/NIFI-11777 > Project: Apache NiFi > Issue Type: New Feature > Components: Core Framework >Reporter: Pierre Villard >Assignee: Simon Bence >Priority: Major > > In the Settings tab of a component, there should a new property "State > Identifier" that would always default to the UUID of the component. > Users would be able to override this value and change it to something custom. > We would: > * ensure that the state identifier is something like [a-z]+[a-z0-9\-]* > * ensure that the state identifier is unique across all components > The "State Identifier" should be used when storing the state of the component > in the state providers. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-12482) Improve authorization of registry clients
Bryan Bende created NIFI-12482:
--
Summary: Improve authorization of registry clients
Key: NIFI-12482
URL: https://issues.apache.org/jira/browse/NIFI-12482
Project: Apache NiFi
Issue Type: Improvement
Affects Versions: 1.24.0, 2.0.0-M1
Reporter: Bryan Bende
Currently the authorization for accessing a registry client is based on
permissions to /controller.
* Create registry client
**
authorizeController({color:#00}RequestAction{color}.{color:#871094}READ{color});
** Seems like this should be checking WRITE
* Update registry client
**
authorizeController({color:#00}RequestAction{color}.{color:#871094}WRITE{color});
* Delete registry client
**
authorizeController({color:#00}RequestAction{color}.{color:#871094}WRITE{color});
* List registry clients
** Creates permissions from
{color:#871094}authorizableLookup{color}.getController()
** Nulls out entity if !permissions.canRead()
It seems too restrictive that a user must have READ to /controller in order to
perform any action like import/start/stop version control. There should be
ability to have more specific policies on a registry client, similar to
parameter contexts.
It also creates a weird situation with parameter contexts since their parent is
/controller, so now if a user needs access to registry clients, they also
inherently get access to all parameter contexts that don't have a specific
policy on them.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] NIFI-8134 allow unescapeJson Record Path function to recursively convert Maps to Records [nifi]
markap14 commented on code in PR #7745:
URL: https://github.com/apache/nifi/pull/7745#discussion_r1417519143
##
nifi-commons/nifi-record-path/src/test/java/org/apache/nifi/record/path/TestRecordPath.java:
##
@@ -1837,7 +1838,41 @@ public void testUnescapeJson() {
put("firstName", "John");
put("age", 30);
}}, "json_str"),
-RecordPath.compile("unescapeJson(/json_str,
'true')").evaluate(recordFromMap).getSelectedFields().findFirst().orElseThrow(IllegalStateException::new).getValue()
+RecordPath.compile("unescapeJson(/json_str,
'true')").evaluate(recordFromMap).getSelectedFields().findFirst().orElseThrow(AssertionError::new).getValue()
+);
+
+// test nested Record converted from Map Object
+final Record nestedRecordFromMap = new MapRecord(schema,
+Collections.singletonMap(
+"json_str",
+
"{\"firstName\":\"John\",\"age\":30,\"addresses\":[{\"address_1\":\"123 Fake
Street\"}]}")
+);
+// recursively convert Maps to Records (addresses becomes and ARRAY or
RECORDs)
+assertEquals(
+DataTypeUtils.toRecord(new LinkedHashMap(){{
+put("firstName", "John");
+put("age", 30);
+put("addresses", new Object[]
{DataTypeUtils.toRecord(Collections.singletonMap("address_1", "123 Fake
Street"), "addresses")});
+}}, "json_str"),
Review Comment:
We should be creating subclasses of LinkedHashMap for initialization
purposes. Should just declare a LinkedHashMap and add elements to it:
```
Map values = new LinkedHashMap<>();
values.put("firstName", "John");
values.put("age", 30);
values.put("addresses", new Object[]
{DataTypeUtils.toRecord(Collections.singletonMap("address_1", "123 Fake
Street"), "addresses")});
```
##
nifi-commons/nifi-record-path/src/test/java/org/apache/nifi/record/path/TestRecordPath.java:
##
@@ -1837,7 +1838,41 @@ public void testUnescapeJson() {
put("firstName", "John");
put("age", 30);
}}, "json_str"),
-RecordPath.compile("unescapeJson(/json_str,
'true')").evaluate(recordFromMap).getSelectedFields().findFirst().orElseThrow(IllegalStateException::new).getValue()
+RecordPath.compile("unescapeJson(/json_str,
'true')").evaluate(recordFromMap).getSelectedFields().findFirst().orElseThrow(AssertionError::new).getValue()
+);
+
+// test nested Record converted from Map Object
+final Record nestedRecordFromMap = new MapRecord(schema,
+Collections.singletonMap(
+"json_str",
+
"{\"firstName\":\"John\",\"age\":30,\"addresses\":[{\"address_1\":\"123 Fake
Street\"}]}")
Review Comment:
Would probably be cleaner here to use a text block here:
```
"""
{"firstName":"John","age":30,"address"...}"""
```
##
nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/util/DataTypeUtils.java:
##
@@ -469,21 +473,51 @@ public static RecordSchema inferSchema(final Map values, final S
final RecordField recordField = new RecordField(key,
inferredDataType, true);
inferredFieldTypes.add(recordField);
-final Object coercedValue = convertType(rawValue,
inferredDataType, fieldName, charset);
-coercedValues.put(key, coercedValue);
+convertType(rawValue, inferredDataType, fieldName, charset);
}
-final RecordSchema inferredSchema = new
SimpleRecordSchema(inferredFieldTypes);
-return inferredSchema;
+return new SimpleRecordSchema(inferredFieldTypes);
}
public static Record toRecord(final Object value, final String fieldName,
final Charset charset) {
+return toRecord(value, fieldName, charset, false);
+}
+
+private static Object covertObjectToRecord(final Object rawValue, final
String key, final Charset charset) {
Review Comment:
typo - name is `covert...` instead of `convert...`
But we should not have a method named `convertObjectToRecord` that returns
an Object that may be an array, may be a Record, or may be any other object
that was provided. Need to either always return a Record or create a method
name that reflects what is truly done.
##
nifi-commons/nifi-record-path/src/test/java/org/apache/nifi/record/path/TestRecordPath.java:
##
@@ -1837,7 +1838,41 @@ public void testUnescapeJson() {
put("firstName", "John");
put("age", 30);
}}, "json_str"),
-RecordPath.compile("unescapeJson(/json_str,
'true')").evaluate(recordFromMap).getSelectedFields().findFirst().orElseThrow(IllegalStateException::new).getValue()
+
Re: [PR] MINIFICPP-1951 - Prevent append overwriting resource [nifi-minifi-cpp]
szaszm commented on code in PR #1640:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1640#discussion_r1416646565
##
libminifi/include/core/ContentSession.h:
##
@@ -39,6 +52,12 @@ class ContentSession {
virtual void rollback() = 0;
virtual ~ContentSession() = default;
+
+ protected:
+ virtual std::shared_ptr append(const
std::shared_ptr& resource_id) = 0;
+
+ std::map, ExtensionData> extensions_;
Review Comment:
What do you think about append_state_ or append_info_?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (NIFI-12481) UI error when listing registry clients and not authorized
Bryan Bende created NIFI-12481:
--
Summary: UI error when listing registry clients and not authorized
Key: NIFI-12481
URL: https://issues.apache.org/jira/browse/NIFI-12481
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 1.24.0, 2.0.0-M1
Reporter: Bryan Bende
Currently the authorization for registry clients is based on READ to
/controller (this is a separate issue that should be addressed).
Steps:
* Run a secure instance locally
* Use initial admin to create a registry client
* Remove initial admin from controller policies
* Create a new PG and choose Import from Registry
* Notice nothing happens
UI Error in dev tools:
{code:java}
nf-canvas-all.js?2.0.0-SNAPSHOT:47 Uncaught TypeError: Cannot read properties
of undefined (reading 'name')
at Object. (nf-canvas-all.js?2.0.0-SNAPSHOT:47:3678)
at Function.each (jquery.min.js:2:3003)
at b.each (jquery.each.js:1:96)
at Object. (nf-canvas-all.js?2.0.0-SNAPSHOT:47:3610)
at c (jquery.min.js:2:28447)
at Object.fireWith [as resolveWith] (jquery.min.js:2:29192)
at l (jquery.min.js:2:80176)
at XMLHttpRequest. (jquery.min.js:2:82630) {code}
The issue is that the listing of registry clients will optionally fill in the
DTO in the entity based on the user's permissions for the entity, but the
permissions are always based on /controller, so if they don't have /controller
the DTO will be null.
The UI should still be able to load a screen with an empty list of registry
clients.
cc [~mcgilman]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-12236) Improving fault tolerancy of the QuestDB backed metrics repository
[ https://issues.apache.org/jira/browse/NIFI-12236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793786#comment-17793786 ] Joe Witt commented on NIFI-12236: - I want to keep the different threads of concern clear. The idea: * I think we all agree there is good value in having these data points persisted across restarts. The example Pierre gives is a perfect example of why. * The reality though is our user interface is not designed for data to be held for long. It isnt clear to me from the properties mentioned in PR but what is the plan for helping the user configure how long this data is retained/queryable? I see batch size and frequency but not quite sure what those would mean relative to retention. Perhaps that is part of the earlier implementation. The implementation options: * Offer an embedded state storage mechanism. QuestDB is one example. A database per nifi instance or a database per cluster. This embedded/batteries included mode is quite convenient for the user but we then of course have to do it quite thoroughly and consider upgrade scenarios. I think our pucker factor here is far higher given the challenges we've had to work through from H2 in the recent year or two. * Offer the ability to connect to/use a database of the user choosing. Defer installation/durability/security of that database to the user as part of their normal database operations/etc.. This I find is more in-line with deployment styles we see in the Cloud, or automated with Ansible, or how one would deploy in K8S. This implementation: * It is a questdb per nifi node. Does not address the other mode should a user want that. Of note when we offered the Zookeeper embedded mode we also offered to connect to a real zookeeper install. That was to reflect the likely non-prod vs prod usage and I think that holds here as well. * It has a change to the nifi-api. We should strongly avoid any such API changes as part of this activity unless that change has value to various other components and the purpose/meaning of that change is very clear. * It includes a Retry mechanism that the PR suggests might be addressed in a later commit. I don't quite follow what that really means but I recommend making this implementation as simple/straight forward as possible. The default selection: * We should not be changing the default until this model is proven to be stable and recoverable in the same manner to the current in memory implementation. > Improving fault tolerancy of the QuestDB backed metrics repository > -- > > Key: NIFI-12236 > URL: https://issues.apache.org/jira/browse/NIFI-12236 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Simon Bence >Assignee: Simon Bence >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > Based on the related discussion on the dev email list, the QuestDB handling > of the metrics repository needs to be improved to have better fault tolerance > in order to be possible to use as a viable option for default metrics data > store. This should primarily focus on handling unexpeted database events like > corrupted database or loss of space on the disk. Any issues should be handled > with an attempt to keep the database service healthy but in case of that is > impossible, the priority is to keep NiFi and the core services running, even > with the price of metrics collection / presentation outage. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] MINIFICPP-2268 Remove FetchContent workaround for CMake<3.18 [nifi-minifi-cpp]
martinzink closed pull request #1699: MINIFICPP-2268 Remove FetchContent workaround for CMake<3.18 URL: https://github.com/apache/nifi-minifi-cpp/pull/1699 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINIFICPP-2267 - Add dynamic library to package on intel macos [nifi-minifi-cpp]
martinzink closed pull request #1700: MINIFICPP-2267 - Add dynamic library to package on intel macos URL: https://github.com/apache/nifi-minifi-cpp/pull/1700 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (MINIFICPP-2264) GenerateFlowFile: 'Custom Text' is not reevaluated
[ https://issues.apache.org/jira/browse/MINIFICPP-2264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793778#comment-17793778 ] Martin Zink commented on MINIFICPP-2264: I have a patch for this, but based it on almost merge ready [https://github.com/apache/nifi-minifi-cpp/pull/1693] so we don't have to resolve conflicts once that merges. > GenerateFlowFile: 'Custom Text' is not reevaluated > --- > > Key: MINIFICPP-2264 > URL: https://issues.apache.org/jira/browse/MINIFICPP-2264 > Project: Apache NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Marton Szasz >Assignee: Martin Zink >Priority: Major > Labels: beginner, beginner-friendly > > 'Unique flow files' must be false for the custom text to be evaluated, but in > that case, it should be reevaluated on every trigger, like in NiFi. This > allows EL expressions like now() or nextInt() to return different values each > time. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (MINIFICPP-2260) OpenCV extension build failure on macOS
[
https://issues.apache.org/jira/browse/MINIFICPP-2260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martin Zink updated MINIFICPP-2260:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
https://github.com/apache/nifi-minifi-cpp/commit/470c266fd8f3f5971defed1d2b42e654c16e8d38
> OpenCV extension build failure on macOS
> ---
>
> Key: MINIFICPP-2260
> URL: https://issues.apache.org/jira/browse/MINIFICPP-2260
> Project: Apache NiFi MiNiFi C++
> Issue Type: Bug
>Reporter: Martin Zink
>Assignee: Martin Zink
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> When trying to compile the minifi-opencv extension I encountered this (among
> other errors)
> {code:java}
> Undefined symbols for architecture arm64:
> "_AVCaptureSessionPresetMedium", referenced from:
> CvCaptureCAM::startCaptureDevice(int) in
> libopencv_videoio.a(cap_avfoundation_mac.mm.o) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12480) Improve handling of embedded JSON records
[
https://issues.apache.org/jira/browse/NIFI-12480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIFI-12480:
--
Fix Version/s: 2.0.0
Status: Patch Available (was: Open)
> Improve handling of embedded JSON records
> -
>
> Key: NIFI-12480
> URL: https://issues.apache.org/jira/browse/NIFI-12480
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework, Extensions
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 2.0.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> It proves to be difficult to treat embedded JSON elements as Strings. There
> are times when this is necessary, such as when pushing to a database that
> declares a field of type JSON or interacting with a web service that expects
> incoming JSON as a String. However, there's no easy way to do this in NiFi
> today.
> Instead, what typically happens is that the Record gets converted to a String
> via a call to {{toString()}} and that produces something like
> {{{}MapRecord[name=John Doe, color=blue]{}}}, which is not helpful.
> However, when a JSON Reader is used, we already have a JSON representation of
> the Record in the record's SerializedForm. When {{toString()}} is called, we
> should always use the SerializedForm of a Record, if it is available and only
> fall back to the given version.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[PR] NIFI-12480: Updated MapRecord's toString() method to use the Serializ… [nifi]
markap14 opened a new pull request, #8132: URL: https://github.com/apache/nifi/pull/8132 …edForm of the record when available and fixed bugs around ensuring that the serialized form is properly set # Summary [NIFI-0](https://issues.apache.org/jira/browse/NIFI-0) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [ ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [ ] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [ ] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [ ] Pull Request based on current revision of the `main` branch - [ ] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 21 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFI-12480) Improve handling of embedded JSON records
Mark Payne created NIFI-12480:
-
Summary: Improve handling of embedded JSON records
Key: NIFI-12480
URL: https://issues.apache.org/jira/browse/NIFI-12480
Project: Apache NiFi
Issue Type: Improvement
Components: Core Framework, Extensions
Reporter: Mark Payne
Assignee: Mark Payne
It proves to be difficult to treat embedded JSON elements as Strings. There are
times when this is necessary, such as when pushing to a database that declares
a field of type JSON or interacting with a web service that expects incoming
JSON as a String. However, there's no easy way to do this in NiFi today.
Instead, what typically happens is that the Record gets converted to a String
via a call to {{toString()}} and that produces something like
{{{}MapRecord[name=John Doe, color=blue]{}}}, which is not helpful.
However, when a JSON Reader is used, we already have a JSON representation of
the Record in the record's SerializedForm. When {{toString()}} is called, we
should always use the SerializedForm of a Record, if it is available and only
fall back to the given version.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-11777) Component State Identifier
[ https://issues.apache.org/jira/browse/NIFI-11777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793771#comment-17793771 ] Joe Witt commented on NIFI-11777: - [~pvillard] Can you add more to this in terms of what the purpose/function is? As written this is more like an implementation/what. How does this differ from the component UUID? What happens to whatever is currently used to associate a given component to its state? > Component State Identifier > -- > > Key: NIFI-11777 > URL: https://issues.apache.org/jira/browse/NIFI-11777 > Project: Apache NiFi > Issue Type: New Feature > Components: Core Framework >Reporter: Pierre Villard >Assignee: Simon Bence >Priority: Major > > In the Settings tab of a component, there should a new property "State > Identifier" that would always default to the UUID of the component. > Users would be able to override this value and change it to something custom. > We would: > * ensure that the state identifier is something like [a-z]+[a-z0-9\-]* > * ensure that the state identifier is unique across all components > The "State Identifier" should be used when storing the state of the component > in the state providers. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-12479) Add pg-export to toolkit CLI
Timea Barna created NIFI-12479: -- Summary: Add pg-export to toolkit CLI Key: NIFI-12479 URL: https://issues.apache.org/jira/browse/NIFI-12479 Project: Apache NiFi Issue Type: Improvement Reporter: Timea Barna Assignee: Timea Barna Add a CLI command allowing to save a process group in NiFi as a JSON file. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-12471) Update bolt-socket-mode to 1.36.1
[ https://issues.apache.org/jira/browse/NIFI-12471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann resolved NIFI-12471. - Fix Version/s: 1.25.0 2.0.0 Assignee: David Handermann Resolution: Fixed Merging the new PublishSlack Processor incorporated the update to bolt-socket-mode 1.36.1. > Update bolt-socket-mode to 1.36.1 > - > > Key: NIFI-12471 > URL: https://issues.apache.org/jira/browse/NIFI-12471 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 2.0.0-M1 >Reporter: Mike R >Assignee: David Handermann >Priority: Minor > Fix For: 1.25.0, 2.0.0 > > > Update bolt-socket-mode to 1.36.1 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12478) Return Message Type as body for JMS Object Messages
[
https://issues.apache.org/jira/browse/NIFI-12478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann updated NIFI-12478:
Status: Patch Available (was: Open)
> Return Message Type as body for JMS Object Messages
> ---
>
> Key: NIFI-12478
> URL: https://issues.apache.org/jira/browse/NIFI-12478
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The ConsumeJMS Processor supports receiving multiple types of JMS messages
> and implements different serialization strategies for each type of message.
> The JMS ObjectMessage Type provides a generic wrapper around an opaque Java
> Object without any further information. The ConsumeJMS Processor currently
> writes the bytes of an Object using Java Object serialization, which presents
> several issues. Java Object serialization is not compatible with services
> outside of Java, it writes the exact version of the Java Object, and it can
> reference classes that may not be present on the receiving system. This can
> lead to unexpected errors when receiving JMS messages in the context of a
> NiFi Processor. Instead of reporting the message as an error, the message
> metadata could still be useful in some flows. Using the Message Type of
> {{ObjectMessage}} as the output bytes enables this edge case scenario,
> although any system designed to interoperate with NiFi should use other types
> of JMS messages to enable subsequent handling in other Processors.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12478) Return Message Type as body for JMS Object Messages
[
https://issues.apache.org/jira/browse/NIFI-12478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann updated NIFI-12478:
Priority: Minor (was: Major)
> Return Message Type as body for JMS Object Messages
> ---
>
> Key: NIFI-12478
> URL: https://issues.apache.org/jira/browse/NIFI-12478
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The ConsumeJMS Processor supports receiving multiple types of JMS messages
> and implements different serialization strategies for each type of message.
> The JMS ObjectMessage Type provides a generic wrapper around an opaque Java
> Object without any further information. The ConsumeJMS Processor currently
> writes the bytes of an Object using Java Object serialization, which presents
> several issues. Java Object serialization is not compatible with services
> outside of Java, it writes the exact version of the Java Object, and it can
> reference classes that may not be present on the receiving system. This can
> lead to unexpected errors when receiving JMS messages in the context of a
> NiFi Processor. Instead of reporting the message as an error, the message
> metadata could still be useful in some flows. Using the Message Type of
> {{ObjectMessage}} as the output bytes enables this edge case scenario,
> although any system designed to interoperate with NiFi should use other types
> of JMS messages to enable subsequent handling in other Processors.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[PR] NIFI-12478 Return Message Type as body for JMS Object Messages [nifi]
exceptionfactory opened a new pull request, #8131: URL: https://github.com/apache/nifi/pull/8131 # Summary [NIFI-12478](https://issues.apache.org/jira/browse/NIFI-12478) Updates supporting utilities for the `ConsumeJMS` Processor to return the Message Type as the body when receiving JMS [ObjectMessage](https://docs.oracle.com/javaee/7/api/javax/jms/ObjectMessage.html) types. This approach avoids various problems with attempting to serialize a wrapped opaque `Object` from a JMS `ObjectMessage`. Java Object serialization is not suitable for inter-operation with subsequent Processors, and the wrapped Object may reference classes that are not available on a given Java Virtual Machine. Rather than writing the message as an error, this approach allows NiFi flows to continue receiving `ObjectMessage` types and handling the associated metadata, while using a standard value of `ObjectMessage` as the bytes for the message body. Additional changes include correcting expected values in a JMS integration test that previously resulted in failures. # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [X] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [X] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [X] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [X] Pull Request based on current revision of the `main` branch - [X] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [X] Build completed using `mvn clean install -P contrib-check` - [X] JDK 21 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (NIFI-12476) [MiNiFi] Remove nifi-framework-core dependencies from bootstrap
[ https://issues.apache.org/jira/browse/NIFI-12476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Erdei resolved NIFI-12476. - Resolution: Fixed > [MiNiFi] Remove nifi-framework-core dependencies from bootstrap > --- > > Key: NIFI-12476 > URL: https://issues.apache.org/jira/browse/NIFI-12476 > Project: Apache NiFi > Issue Type: Improvement > Components: MiNiFi >Reporter: Ferenc Kis >Assignee: Ferenc Kis >Priority: Major > Labels: minifi-java > Fix For: 2.0.0 > > Time Spent: 40m > Remaining Estimate: 0h > > With NIFI-12372 several classes were moved to minifi-commons. This module > both used by minifi-framework and minifi-bootstrap. However minifi-commons > already pulls in nifi-framework-core, so those dependencies got into > minifi-bootstrap, resulting in a 50 MBs larger bootstrap. > The goal is to remove nifi-framework-core from minifi-bootstrap and reduce > it's size. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-12478) Return Message Type as body for JMS Object Messages
David Handermann created NIFI-12478:
---
Summary: Return Message Type as body for JMS Object Messages
Key: NIFI-12478
URL: https://issues.apache.org/jira/browse/NIFI-12478
Project: Apache NiFi
Issue Type: Improvement
Components: Extensions
Reporter: David Handermann
Assignee: David Handermann
The ConsumeJMS Processor supports receiving multiple types of JMS messages and
implements different serialization strategies for each type of message. The JMS
ObjectMessage Type provides a generic wrapper around an opaque Java Object
without any further information. The ConsumeJMS Processor currently writes the
bytes of an Object using Java Object serialization, which presents several
issues. Java Object serialization is not compatible with services outside of
Java, it writes the exact version of the Java Object, and it can reference
classes that may not be present on the receiving system. This can lead to
unexpected errors when receiving JMS messages in the context of a NiFi
Processor. Instead of reporting the message as an error, the message metadata
could still be useful in some flows. Using the Message Type of
{{ObjectMessage}} as the output bytes enables this edge case scenario, although
any system designed to interoperate with NiFi should use other types of JMS
messages to enable subsequent handling in other Processors.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12476) [MiNiFi] Remove nifi-framework-core dependencies from bootstrap
[ https://issues.apache.org/jira/browse/NIFI-12476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Erdei updated NIFI-12476: Fix Version/s: 2.0.0 > [MiNiFi] Remove nifi-framework-core dependencies from bootstrap > --- > > Key: NIFI-12476 > URL: https://issues.apache.org/jira/browse/NIFI-12476 > Project: Apache NiFi > Issue Type: Improvement > Components: MiNiFi >Reporter: Ferenc Kis >Assignee: Ferenc Kis >Priority: Major > Labels: minifi-java > Fix For: 2.0.0 > > Time Spent: 40m > Remaining Estimate: 0h > > With NIFI-12372 several classes were moved to minifi-commons. This module > both used by minifi-framework and minifi-bootstrap. However minifi-commons > already pulls in nifi-framework-core, so those dependencies got into > minifi-bootstrap, resulting in a 50 MBs larger bootstrap. > The goal is to remove nifi-framework-core from minifi-bootstrap and reduce > it's size. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12476 Exclude nifi-framework-core from minifi-bootstrap [nifi]
ferencerdei commented on PR #8128: URL: https://github.com/apache/nifi/pull/8128#issuecomment-1842957687 Thanks for the change, LGTM, the size of the bootstrap dir is now went back to normal -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] NIFI-12477 Update org.apache.commons.io.version to 2.15.1 [nifi]
mr1716 opened a new pull request, #8130: URL: https://github.com/apache/nifi/pull/8130 # Summary [NIFI-12477](https://issues.apache.org/jira/browse/NIFI-12477) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [X] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI-12477) issue created ### Pull Request Tracking - [X] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [X] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [X] Pull Request based on current revision of the `main` branch - [X] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 21 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-12476) [MiNiFi] Remove nifi-framework-core dependencies from bootstrap
[ https://issues.apache.org/jira/browse/NIFI-12476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793738#comment-17793738 ] ASF subversion and git services commented on NIFI-12476: Commit effbb76baa180b244d8072ccca1bc479cf850934 in nifi's branch refs/heads/main from Ferenc Kis [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=effbb76baa ] NIFI-12476 Exclude nifi-framework-core from minifi-bootstrap Signed-off-by: Ferenc Erdei This closes #8128. > [MiNiFi] Remove nifi-framework-core dependencies from bootstrap > --- > > Key: NIFI-12476 > URL: https://issues.apache.org/jira/browse/NIFI-12476 > Project: Apache NiFi > Issue Type: Improvement > Components: MiNiFi >Reporter: Ferenc Kis >Assignee: Ferenc Kis >Priority: Major > Labels: minifi-java > Time Spent: 0.5h > Remaining Estimate: 0h > > With NIFI-12372 several classes were moved to minifi-commons. This module > both used by minifi-framework and minifi-bootstrap. However minifi-commons > already pulls in nifi-framework-core, so those dependencies got into > minifi-bootstrap, resulting in a 50 MBs larger bootstrap. > The goal is to remove nifi-framework-core from minifi-bootstrap and reduce > it's size. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-12471) Update bolt-socket-mode to 1.36.1
[ https://issues.apache.org/jira/browse/NIFI-12471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike R reassigned NIFI-12471: - Assignee: (was: Mike R) > Update bolt-socket-mode to 1.36.1 > - > > Key: NIFI-12471 > URL: https://issues.apache.org/jira/browse/NIFI-12471 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 2.0.0-M1 >Reporter: Mike R >Priority: Minor > > Update bolt-socket-mode to 1.36.1 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12476 Exclude nifi-framework-core from minifi-bootstrap [nifi]
ferencerdei closed pull request #8128: NIFI-12476 Exclude nifi-framework-core from minifi-bootstrap URL: https://github.com/apache/nifi/pull/8128 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFI-12477) Update org.apache.commons.io.version to 2.15.1
Mike R created NIFI-12477: - Summary: Update org.apache.commons.io.version to 2.15.1 Key: NIFI-12477 URL: https://issues.apache.org/jira/browse/NIFI-12477 Project: Apache NiFi Issue Type: Improvement Affects Versions: 2.0.0-M1 Reporter: Mike R Update org.apache.commons.io.version to 2.15.1 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12236) Improving fault tolerancy of the QuestDB backed metrics repository
[ https://issues.apache.org/jira/browse/NIFI-12236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793687#comment-17793687 ] Pierre Villard commented on NIFI-12236: --- I'd like to add my 2ct here: This feature is already used by some of the NiFi users and has proved to be extremely useful when it comes to debug problematic situations. It's really not about providing a long term monitoring solution but more about being able to troubleshoot things in case something bad happened. In particular, when something bad happens and a node is restarted, all of the data, with the current default implementation is lost, and this is making things really complicated for debugging/troubleshooting. Regarding Datadog/Prometheus, as someone passionate about NiFi's monitoring, I definitely agree that anyone using NiFi in production should be deploying NiFi and have those solutions for the monitoring of the NiFi service. However, what we're talking about here is very different (in my opinion). We all know that NiFi users are most of the time using NiFi in a very "multi tenant" approach (ie. many different use cases / process groups running in the same NiFi environment), while the Prometheus endpoint and the reporting tasks are great to report high level monitoring metrics to tools where you'll have very advanced dashboarding capabilities, it's a completely different story to have "per use case" monitoring. And even if you're sending everything into something like Prometheus, doing the monitoring dashboards per use case will be quite some work. And I think this would be the same if we say: please provide your database and we'll push data there and then it's up to you to build dashboards on top of that. As far as I'm concerned and just to give a single example, when trying to look at performance optimizations in a flow, I may want to look at the graph for a specific processor showing the metrics average task duration / average lineage duration. This level of detail is something that would be quite hard to get if not offered by NiFi. Having a capability persisting all of this data across restarts is really useful. While I can definitely accept the concerns around making it the default for NiFi 2.0, I think the work in the corresponding PR is a tremendous amount of work that would be valuable for the NiFi users. > Improving fault tolerancy of the QuestDB backed metrics repository > -- > > Key: NIFI-12236 > URL: https://issues.apache.org/jira/browse/NIFI-12236 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Simon Bence >Assignee: Simon Bence >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > Based on the related discussion on the dev email list, the QuestDB handling > of the metrics repository needs to be improved to have better fault tolerance > in order to be possible to use as a viable option for default metrics data > store. This should primarily focus on handling unexpeted database events like > corrupted database or loss of space on the disk. Any issues should be handled > with an attempt to keep the database service healthy but in case of that is > impossible, the priority is to keep NiFi and the core services running, even > with the price of metrics collection / presentation outage. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-11777) Component State Identifier
[ https://issues.apache.org/jira/browse/NIFI-11777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simon Bence reassigned NIFI-11777: -- Assignee: Simon Bence > Component State Identifier > -- > > Key: NIFI-11777 > URL: https://issues.apache.org/jira/browse/NIFI-11777 > Project: Apache NiFi > Issue Type: New Feature > Components: Core Framework >Reporter: Pierre Villard >Assignee: Simon Bence >Priority: Major > > In the Settings tab of a component, there should a new property "State > Identifier" that would always default to the UUID of the component. > Users would be able to override this value and change it to something custom. > We would: > * ensure that the state identifier is something like [a-z]+[a-z0-9\-]* > * ensure that the state identifier is unique across all components > The "State Identifier" should be used when storing the state of the component > in the state providers. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12105) ExecuteStateless should support multiple directories for NAR Directory property
[ https://issues.apache.org/jira/browse/NIFI-12105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Gyori updated NIFI-12105: --- Status: Patch Available (was: In Progress) https://github.com/apache/nifi/pull/8129 > ExecuteStateless should support multiple directories for NAR Directory > property > --- > > Key: NIFI-12105 > URL: https://issues.apache.org/jira/browse/NIFI-12105 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions, NiFi Stateless >Reporter: Pierre Villard >Assignee: Peter Gyori >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > ExecuteStateless has a property for the NAR Directory which currently accepts > a single directory. When deploying custom NARs, it's usually done in a > dedicated directory that is not the lib directory. It'd be nice to support a > comma separated list of directories when configuring ExecuteStateless so that > the execute flow can reference custom NARs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] MINIFICPP-1415 Pass references to onTrigger and onSchedule instead of… [nifi-minifi-cpp]
martinzink commented on code in PR #1693: URL: https://github.com/apache/nifi-minifi-cpp/pull/1693#discussion_r1417112439 ## extensions/splunk/PutSplunkHTTP.cpp: ## Review Comment: I've checked we have a bunch of processors (mainly source processors, e.g. ConsumeWindowsEventLog, ListS3 etc...) that evaluate without flowfile context and they document their EL support the same way. So I am not sure if we want to manually overwrite the documentation. Maybe we could include this information inside the Property description? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] NIFI-12105: ExecuteStateless processor accepts additional NAR directories [nifi]
pgyori opened a new pull request, #8129: URL: https://github.com/apache/nifi/pull/8129 # Summary [NIFI-12105](https://issues.apache.org/jira/browse/NIFI-12105) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [x] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [x] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [x] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [x] Pull Request based on current revision of the `main` branch - [x] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [x] Build completed using `mvn clean install -P contrib-check` - [x] JDK 21 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [x] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12236 Improving fault tolerancy of the QuestDB backed metrics repository [nifi]
simonbence commented on PR #8123: URL: https://github.com/apache/nifi/pull/8123#issuecomment-1842633320 Thansk @joewitt ! I answered on the JIRA as well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12236 Improving fault tolerancy of the QuestDB backed metrics repository [nifi]
simonbence commented on code in PR #8123:
URL: https://github.com/apache/nifi/pull/8123#discussion_r1417093557
##
nifi-api/src/main/java/org/apache/nifi/controller/status/ConnectionStatus.java:
##
@@ -22,6 +22,7 @@
/**
*/
public class ConnectionStatus implements Cloneable {
+private long createdAtInMs;
Review Comment:
Based on most usage I would think a time related information would make
sense within status (at what time the status snapshot is relevant). I am not
sure this is true for all usages but the current setup on main (these two
coupled information travels together) does not seem straightforward to me. I
cannot rule out however a possible third solution which complies with both
things.
Instant is a good idea. I think I went down this road because some already
contained creation time in this flavor but the `Instant` really makes more
sense (especially that we convert it to instant many places already)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Assigned] (MINIFICPP-2264) GenerateFlowFile: 'Custom Text' is not reevaluated
[ https://issues.apache.org/jira/browse/MINIFICPP-2264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Zink reassigned MINIFICPP-2264: -- Assignee: Martin Zink > GenerateFlowFile: 'Custom Text' is not reevaluated > --- > > Key: MINIFICPP-2264 > URL: https://issues.apache.org/jira/browse/MINIFICPP-2264 > Project: Apache NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Marton Szasz >Assignee: Martin Zink >Priority: Major > Labels: beginner, beginner-friendly > > 'Unique flow files' must be false for the custom text to be evaluated, but in > that case, it should be reevaluated on every trigger, like in NiFi. This > allows EL expressions like now() or nextInt() to return different values each > time. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12236 Improving fault tolerancy of the QuestDB backed metrics repository [nifi]
simonbence commented on PR #8123: URL: https://github.com/apache/nifi/pull/8123#issuecomment-1842625796 Hi @exceptionfactory ! Thanks for taking care! The separation of the default setting sounds a great idea, I will apply it with other requested changes. As for the API changes, I looked around and the usage of the *Status classes looks to be limited to some dedicated things. It did not seem harmful, however in the other hand in many times it travels together with the capture time of the status making it somewhat coincidental. An other idea I had (to keep the API intact) is to provide a wrapper class extending the original ones with this information but as I see it would come with a the creation of a huge amount of unnecessary object creation. I am open for ideas but this looked the most fitting solution so far -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12476 Exclude nifi-framework-core from minifi-bootstrap [nifi]
briansolo1985 commented on PR #8128: URL: https://github.com/apache/nifi/pull/8128#issuecomment-1842625048 Tested with FileChangeIngestor, flow ser/de and enrichment was successful -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] NIFI-12476 Exclude nifi-framework-core from minifi-bootstrap [nifi]
briansolo1985 opened a new pull request, #8128: URL: https://github.com/apache/nifi/pull/8128 # Summary [NIFI-12476](https://issues.apache.org/jira/browse/NIFI-12476) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [x] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [x] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [x] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [ ] Pull Request based on current revision of the `main` branch - [ ] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [x] Build completed using `mvn clean install -P contrib-check` - [x] JDK 21 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFI-12476) [MiNiFi] Remove nifi-framework-core dependencies from bootstrap
Ferenc Kis created NIFI-12476: - Summary: [MiNiFi] Remove nifi-framework-core dependencies from bootstrap Key: NIFI-12476 URL: https://issues.apache.org/jira/browse/NIFI-12476 Project: Apache NiFi Issue Type: Improvement Components: MiNiFi Reporter: Ferenc Kis Assignee: Ferenc Kis With NIFI-12372 several classes were moved to minifi-commons. This module both used by minifi-framework and minifi-bootstrap. However minifi-commons already pulls in nifi-framework-core, so those dependencies got into minifi-bootstrap, resulting in a 50 MBs larger bootstrap. The goal is to remove nifi-framework-core from minifi-bootstrap and reduce it's size. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12236) Improving fault tolerancy of the QuestDB backed metrics repository
[ https://issues.apache.org/jira/browse/NIFI-12236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793614#comment-17793614 ] Simon Bence commented on NIFI-12236: Hi [~joewitt]! Thanks for your attention on the topic! My understanding at the time was that your main concerns were about the implementation's possible implications on the core functionality of NiFi such as handlink different kinds of errors like corrupted database or insufficient disk space. So the main focus of the change was to rule out possible situations where an unexpected situation with QuestDb could prevent core functionality from working properly. I think this part is aimed reasonably and are covered with tests. As for visualisation, I did not aim for replacing that part however it sound a good idea to follow the changes on the backend. From my perspective the main benefit is to have some persistent storage behind is to be able to access statistics after NiFi has stopped (or shutdown abruptly). I think in the long run the opportunity to "plug in" different kind of statistics repositories just like with processors could (and maybe should) be opened, but I did not aim for that this time. I consider this as a step forward which might be used as a stepping stone for future developments, including non-embedded solutions for persisting and other things. In many environments having Datatod or even MySQL backend might be more satifsfying (again: with this story I did not aim for that which does not closes out the possibility of furute steps towards that either from me or other member of the community) but I think, having an embedded solution comes with the merit of adding the capability of persistent statistical information out of the box. > Improving fault tolerancy of the QuestDB backed metrics repository > -- > > Key: NIFI-12236 > URL: https://issues.apache.org/jira/browse/NIFI-12236 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Simon Bence >Assignee: Simon Bence >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > Based on the related discussion on the dev email list, the QuestDB handling > of the metrics repository needs to be improved to have better fault tolerance > in order to be possible to use as a viable option for default metrics data > store. This should primarily focus on handling unexpeted database events like > corrupted database or loss of space on the disk. Any issues should be handled > with an attempt to keep the database service healthy but in case of that is > impossible, the priority is to keep NiFi and the core services running, even > with the price of metrics collection / presentation outage. -- This message was sent by Atlassian Jira (v8.20.10#820010)
