[jira] [Commented] (NIFI-12236) Improving fault tolerancy of the QuestDB backed metrics repository
[ https://issues.apache.org/jira/browse/NIFI-12236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17815864#comment-17815864 ] ASF subversion and git services commented on NIFI-12236: Commit a519585b02f37c654638ece4681928fd3abbe355 in nifi's branch refs/heads/main from Simon Bence [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=a519585b02 ] NIFI-12236 Improved Fault Tolerance in QuestDB Status Repository - Moved QuestDB components to nifi-questdb-bundle This closes #8152 Signed-off-by: David Handermann > Improving fault tolerancy of the QuestDB backed metrics repository > -- > > Key: NIFI-12236 > URL: https://issues.apache.org/jira/browse/NIFI-12236 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Simon Bence >Assignee: Simon Bence >Priority: Major > Time Spent: 9h > Remaining Estimate: 0h > > Based on the related discussion on the dev email list, the QuestDB handling > of the metrics repository needs to be improved to have better fault tolerance > in order to be possible to use as a viable option for default metrics data > store. This should primarily focus on handling unexpeted database events like > corrupted database or loss of space on the disk. Any issues should be handled > with an attempt to keep the database service healthy but in case of that is > impossible, the priority is to keep NiFi and the core services running, even > with the price of metrics collection / presentation outage. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-12236) Improving fault tolerancy of the QuestDB backed metrics repository
[ https://issues.apache.org/jira/browse/NIFI-12236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann resolved NIFI-12236. - Fix Version/s: 2.0.0 Resolution: Fixed > Improving fault tolerancy of the QuestDB backed metrics repository > -- > > Key: NIFI-12236 > URL: https://issues.apache.org/jira/browse/NIFI-12236 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Simon Bence >Assignee: Simon Bence >Priority: Major > Fix For: 2.0.0 > > Time Spent: 9h > Remaining Estimate: 0h > > Based on the related discussion on the dev email list, the QuestDB handling > of the metrics repository needs to be improved to have better fault tolerance > in order to be possible to use as a viable option for default metrics data > store. This should primarily focus on handling unexpeted database events like > corrupted database or loss of space on the disk. Any issues should be handled > with an attempt to keep the database service healthy but in case of that is > impossible, the priority is to keep NiFi and the core services running, even > with the price of metrics collection / presentation outage. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12236 Improving fault tolerancy of the QuestDB backed metrics repository [nifi]
exceptionfactory closed pull request #8152: NIFI-12236 Improving fault tolerancy of the QuestDB backed metrics repository URL: https://github.com/apache/nifi/pull/8152 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (NIFI-12753) Update various Github Actions versions for Github CI
[ https://issues.apache.org/jira/browse/NIFI-12753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann resolved NIFI-12753. - Fix Version/s: 2.0.0 Resolution: Fixed > Update various Github Actions versions for Github CI > > > Key: NIFI-12753 > URL: https://issues.apache.org/jira/browse/NIFI-12753 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joe Witt >Assignee: Joe Witt >Priority: Major > Fix For: 2.0.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Shows some deprecation warnings related to github actions. We need to update > our workflows to use latest action versions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12753) Update various Github Actions versions for Github CI
[ https://issues.apache.org/jira/browse/NIFI-12753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17815847#comment-17815847 ] ASF subversion and git services commented on NIFI-12753: Commit 4dfbe0564139dc41b71a7f180a415371bf180a40 in nifi's branch refs/heads/main from Joe Witt [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=4dfbe05641 ] NIFI-12753 Updated GitHub Actions to current versions - Upgraded cache from 3 to 4 - Upgraded CodeQL from 2 to 3 - Upgraded Codecov from 3 to 4 - Upgraded upload-artifact from 3 to 4 - Upgraded stale from 6 to 9 This closes #8373 Signed-off-by: David Handermann > Update various Github Actions versions for Github CI > > > Key: NIFI-12753 > URL: https://issues.apache.org/jira/browse/NIFI-12753 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joe Witt >Assignee: Joe Witt >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > Shows some deprecation warnings related to github actions. We need to update > our workflows to use latest action versions -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12753 updating various github actions versions [nifi]
exceptionfactory closed pull request #8373: NIFI-12753 updating various github actions versions URL: https://github.com/apache/nifi/pull/8373 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12697: Allow disabling scientific notation when writing JSON; us… [nifi]
markap14 commented on code in PR #8319: URL: https://github.com/apache/nifi/pull/8319#discussion_r1483615155 ## nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-json-record-utils/src/main/java/org/apache/nifi/json/WriteJsonResult.java: ## @@ -19,6 +19,7 @@ Review Comment: Got it. Thanks @Lehel44 that makes sense. Great catch! Will update shortly, with new test cases. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12697: Allow disabling scientific notation when writing JSON; us… [nifi]
Lehel44 commented on code in PR #8319: URL: https://github.com/apache/nifi/pull/8319#discussion_r1483611259 ## nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-json-record-utils/src/main/java/org/apache/nifi/json/WriteJsonResult.java: ## @@ -19,6 +19,7 @@ Review Comment: I think in the section of WriteJsonResult from line 176 to 189, the scientific notation logic needs to be added for cases where the Record is already serialized. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-12636) Upgrade dependencies for Pinecone, ChromaDB and OpenAI processors
[ https://issues.apache.org/jira/browse/NIFI-12636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Gresock updated NIFI-12636: --- Fix Version/s: 2.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Upgrade dependencies for Pinecone, ChromaDB and OpenAI processors > - > > Key: NIFI-12636 > URL: https://issues.apache.org/jira/browse/NIFI-12636 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 2.0.0-M1 > Environment: Mac, JDK 21, python 3.10 >Reporter: Timothy J Spann >Assignee: Pierre Villard >Priority: Minor > Fix For: 2.0.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Upgrade langchain to 0.1.2 (from 0.0.331) > Upgrade openai to 1.9.0 (from 0.28.1) > Upgrade chromadb to 0.4.22 (from 0.4.14) > Upgrade pinecone-client to 3.0.1 > Removed some unused imports -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12636) Upgrade dependencies for Pinecone, ChromaDB and OpenAI processors
[ https://issues.apache.org/jira/browse/NIFI-12636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17815832#comment-17815832 ] ASF subversion and git services commented on NIFI-12636: Commit 3afeac634140a284f661f640c4c63f1d8a687de8 in nifi's branch refs/heads/main from Pierre Villard [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=3afeac6341 ] NIFI-12636 - Upgrade dependencies for Pinecone, ChromaDB and OpenAI processors Signed-off-by: Joe Gresock This closes #8293. > Upgrade dependencies for Pinecone, ChromaDB and OpenAI processors > - > > Key: NIFI-12636 > URL: https://issues.apache.org/jira/browse/NIFI-12636 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 2.0.0-M1 > Environment: Mac, JDK 21, python 3.10 >Reporter: Timothy J Spann >Assignee: Pierre Villard >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > Upgrade langchain to 0.1.2 (from 0.0.331) > Upgrade openai to 1.9.0 (from 0.28.1) > Upgrade chromadb to 0.4.22 (from 0.4.14) > Upgrade pinecone-client to 3.0.1 > Removed some unused imports -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12636 - Upgrade dependencies for Pinecone, ChromaDB and OpenAI processors [nifi]
asfgit closed pull request #8293: NIFI-12636 - Upgrade dependencies for Pinecone, ChromaDB and OpenAI processors URL: https://github.com/apache/nifi/pull/8293 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFI-12762) ListenHTTP request headers limited to 8192 bytes
Michael W Moser created NIFI-12762:
--
Summary: ListenHTTP request headers limited to 8192 bytes
Key: NIFI-12762
URL: https://issues.apache.org/jira/browse/NIFI-12762
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 2.0.0-M2, 1.25.0
Reporter: Michael W Moser
ListenHTTP will parse HTTP request headers if the property "HTTP Headers to
receive as Attributes" is set. An HTTP client can sent request headers that
are larger than the default allowed 8192 bytes, and we get the log message
{noformat}
2024-02-08 20:22:16,548 WARN [ListenHTTP (8a290f74-018d-1000-a9d2-c06fdb10e3f3)
Web Server-188] org.eclipse.jetty.http.HttpParser Header is too large
8193>8192{noformat}
and ListenHTTP responds with HTTP ERROR 431 Request Header Fields Too Large
The NiFi UI and REST API sets the max header size to 16384 and it is
configurable in nifi.properties "nifi.web.max.header.size", in the file
FrameworkServerConnectorFactory.java.
We should probably allow ListenHTTP to use the same max header size setting.
Should we also look into this for HandleHttpRequest?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12745) AvroReader silently drops record if it's malformed
[
https://issues.apache.org/jira/browse/NIFI-12745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tamas Palfy updated NIFI-12745:
---
Fix Version/s: 2.0.0
1.26.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> AvroReader silently drops record if it's malformed
> --
>
> Key: NIFI-12745
> URL: https://issues.apache.org/jira/browse/NIFI-12745
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 2.0.0-M1, 1.18.0, 1.19.0, 1.20.0, 1.19.1, 1.21.0,
> 1.22.0, 1.23.0, 1.24.0, 1.23.1, 1.23.2, 1.25.0, 2.0.0-M2
>Reporter: Rajmund Takacs
>Assignee: Rajmund Takacs
>Priority: Major
> Fix For: 2.0.0, 1.26.0
>
> Attachments: ValidateRecord.json
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> See the attached example flow. It reproduces the issue very reliably.
> {{GenerateFlowFile}} is set to generate the following Json:
> {code:json}
> [{
> "field_1" : 123456789,
> "field_2" : "44",
> "field_3" : 5
> }]
> {code}
> This input is converted to Avro format, using the {{ConvertRecord}}
> processor. The 'Schema Write Strategy' of {{AvroRecordSetWriter}} is set to
> anything different than 'Embed Avro Schema'.
> Then, the resulting FF is routed to a processor that uses an {{AvroReader}}
> to work on the records. The reader is set to use a predefined, fixed schema,
> which does not match with the input avro file, contains at least an extra
> field. It does not matter if that field has a default value or not.
> {code:json}
> {
> "type":"record",
> "name":"message_name",
> "namespace":"message_namespace",
> "fields":[
> {
> "name":"field_1",
> "type":["long"]
> },
> {
> "name":"field_2",
> "type":["string"]
> },
> {
> "name":"field_3",
> "type":["int"]
> },
> {
> "name":"extra_field",
> "type":["string"],
> "default":"empty"
> }
> ]
> }
> {code}
> When this processor consumes the input, the reader silently drops the record,
> without even making an error log message. At the processor level, this is
> equivalent to having no records to process, so nothing happens. The user
> won't notice that there is a misconfiguration somewhere until they start
> noticing the missing the flow files.
> The expected behavior from the processors would be to route the malformed
> input FF to their failure relationship, and shout an error on its bulletin.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] NIFI-12745: Fix AvroReader silently dropping malformed records [nifi]
tpalfy commented on PR #8361: URL: https://github.com/apache/nifi/pull/8361#issuecomment-1934744186 LGTM Thank you for your work @takraj ! Merged to main and support/nifi-1.x. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-12745) AvroReader silently drops record if it's malformed
[
https://issues.apache.org/jira/browse/NIFI-12745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17815794#comment-17815794
]
ASF subversion and git services commented on NIFI-12745:
Commit ba2e24b68f036363f333e216ee968d999a59e268 in nifi's branch
refs/heads/support/nifi-1.x from Rajmund Takacs
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=ba2e24b68f ]
NIFI-12745: Fix AvroReader silently dropping malformed records
This closes #8361.
Signed-off-by: Tamas Palfy
> AvroReader silently drops record if it's malformed
> --
>
> Key: NIFI-12745
> URL: https://issues.apache.org/jira/browse/NIFI-12745
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 2.0.0-M1, 1.18.0, 1.19.0, 1.20.0, 1.19.1, 1.21.0,
> 1.22.0, 1.23.0, 1.24.0, 1.23.1, 1.23.2, 1.25.0, 2.0.0-M2
>Reporter: Rajmund Takacs
>Assignee: Rajmund Takacs
>Priority: Major
> Attachments: ValidateRecord.json
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> See the attached example flow. It reproduces the issue very reliably.
> {{GenerateFlowFile}} is set to generate the following Json:
> {code:json}
> [{
> "field_1" : 123456789,
> "field_2" : "44",
> "field_3" : 5
> }]
> {code}
> This input is converted to Avro format, using the {{ConvertRecord}}
> processor. The 'Schema Write Strategy' of {{AvroRecordSetWriter}} is set to
> anything different than 'Embed Avro Schema'.
> Then, the resulting FF is routed to a processor that uses an {{AvroReader}}
> to work on the records. The reader is set to use a predefined, fixed schema,
> which does not match with the input avro file, contains at least an extra
> field. It does not matter if that field has a default value or not.
> {code:json}
> {
> "type":"record",
> "name":"message_name",
> "namespace":"message_namespace",
> "fields":[
> {
> "name":"field_1",
> "type":["long"]
> },
> {
> "name":"field_2",
> "type":["string"]
> },
> {
> "name":"field_3",
> "type":["int"]
> },
> {
> "name":"extra_field",
> "type":["string"],
> "default":"empty"
> }
> ]
> }
> {code}
> When this processor consumes the input, the reader silently drops the record,
> without even making an error log message. At the processor level, this is
> equivalent to having no records to process, so nothing happens. The user
> won't notice that there is a misconfiguration somewhere until they start
> noticing the missing the flow files.
> The expected behavior from the processors would be to route the malformed
> input FF to their failure relationship, and shout an error on its bulletin.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-12745) AvroReader silently drops record if it's malformed
[
https://issues.apache.org/jira/browse/NIFI-12745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17815790#comment-17815790
]
ASF subversion and git services commented on NIFI-12745:
Commit 85dc637a9601cf584374f8ea8c07214ac9ece576 in nifi's branch
refs/heads/main from Rajmund Takacs
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=85dc637a96 ]
NIFI-12745: Fix AvroReader silently dropping malformed records
This closes #8361.
Signed-off-by: Tamas Palfy
> AvroReader silently drops record if it's malformed
> --
>
> Key: NIFI-12745
> URL: https://issues.apache.org/jira/browse/NIFI-12745
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 2.0.0-M1, 1.18.0, 1.19.0, 1.20.0, 1.19.1, 1.21.0,
> 1.22.0, 1.23.0, 1.24.0, 1.23.1, 1.23.2, 1.25.0, 2.0.0-M2
>Reporter: Rajmund Takacs
>Assignee: Rajmund Takacs
>Priority: Major
> Attachments: ValidateRecord.json
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> See the attached example flow. It reproduces the issue very reliably.
> {{GenerateFlowFile}} is set to generate the following Json:
> {code:json}
> [{
> "field_1" : 123456789,
> "field_2" : "44",
> "field_3" : 5
> }]
> {code}
> This input is converted to Avro format, using the {{ConvertRecord}}
> processor. The 'Schema Write Strategy' of {{AvroRecordSetWriter}} is set to
> anything different than 'Embed Avro Schema'.
> Then, the resulting FF is routed to a processor that uses an {{AvroReader}}
> to work on the records. The reader is set to use a predefined, fixed schema,
> which does not match with the input avro file, contains at least an extra
> field. It does not matter if that field has a default value or not.
> {code:json}
> {
> "type":"record",
> "name":"message_name",
> "namespace":"message_namespace",
> "fields":[
> {
> "name":"field_1",
> "type":["long"]
> },
> {
> "name":"field_2",
> "type":["string"]
> },
> {
> "name":"field_3",
> "type":["int"]
> },
> {
> "name":"extra_field",
> "type":["string"],
> "default":"empty"
> }
> ]
> }
> {code}
> When this processor consumes the input, the reader silently drops the record,
> without even making an error log message. At the processor level, this is
> equivalent to having no records to process, so nothing happens. The user
> won't notice that there is a misconfiguration somewhere until they start
> noticing the missing the flow files.
> The expected behavior from the processors would be to route the malformed
> input FF to their failure relationship, and shout an error on its bulletin.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] NIFI-12745: Fix AvroReader silently dropping malformed records [nifi]
asfgit closed pull request #8361: NIFI-12745: Fix AvroReader silently dropping malformed records URL: https://github.com/apache/nifi/pull/8361 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-12400) Remaining items to migrate UI to currently supported/active framework
[ https://issues.apache.org/jira/browse/NIFI-12400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Aslan updated NIFI-12400: --- Description: The purpose of this Jira is to track all remaining items following the initial commit [1] for NIFI-11481. The description will be kept up to date with remaining features, tasks, and improvements. As each items is worked, a new sub task Jira will be created and referenced in this description. * Support Parameters in Properties with Allowable Values (NIFI-12401) * Summary (NIFI-12437) ** Remaining work not addressed in initial Jira: *** input ports (NIFI-12504) *** output ports (NIFI-12504) *** remote process groups (NIFI-12504) *** process groups (NIFI-12504) *** connections (NIFI-12504) *** System Diagnostics (NIFI-12505) *** support for cluster-specific ui elements (NIFI-12537) *** Add pagination (NIFI-12552) * Counters (NIFI-12415) * Bulletin Board (NIFI-12560) * Provenance (NIFI-12445) ** Event Listing (NIFI-12445) ** Search (NIFI-12445) ** Event Dialog (NIFI-12445) ** Lineage (NIFI-12485) ** Replay from context menu (NIFI-12445) * Configure Reporting Task (NIFI-12563) * Flow Analysis Rules (NIFI-12588) * Registry Clients (NIFI-12486) * Import from Registry (NIFI-12734) * Parameter Providers (NIFI-12622) ** Fetch parameters from provider, map to parameter context (dialog) - (NIFI-12665) * Cluster ** Status History - node specific values * Flow Configuration History (NIFI-12754) * Node Status History (NIFI-12553) * Status history for components from canvas context menu (NIFI-12553) * Users (NIFI-12543) * Policies (NIFI-12548) ** Overridden policy Empty or Copy (NIFI-12679) * Help * About * Show Upstream/Downstream * Align * List Queue (NIFI-12589) * Empty [all] Queue (NIFI-12604) * View Content (NIFI-12589 and NIFI-12445) * View State (NIFI-12611) * Change Component Version * Consider PG permissions in Toolbox (NIFI-12683) * PG Version ** Start ** Commit ** Force Commit ** Show changes ** Revert changes ** Change Flow version ** Stop * Configure PG (NIFI-12417) * Process Group Services (NIFI-12425) ** Listing (NIFI-12425) ** Create (NIFI-12425) ** Configure (NIFI-12425) ** Delete (NIFI-12425) ** Enable (NIFI-12529) ** Disable (NIFI-12529) ** Improve layout and breadcrumbs ** Disable and Configure * Configure Processor ** Service Link (NIFI-12425) ** Create inline Service (NIFI-12425) ** Parameter Link (NIFI-12502) ** Convert to Parameter (NIFI-12502) ** Fix issue with Property Editor width (NIFI-12547) ** Stop and Configure ** Open Custom UI ** Property History ** Unable to re-add any removed Property (NIFI-12743) ** Shift-Enter new line when editing Property (NIFI-12743) * Property Verification * More Details (Processor, Controller Service, Reporting Task) * Download Flow * Create RPG (NIFI-12758) * Configure RPG * RPG Remote Ports * RPG Go To (NIFI-12759) * RPG Refresh (NIFI-12761) * Color * Move to Front * Copy/Paste * Add/Update Info Icons in dialogs throughout the application * Set viewport earlier when loading a Process Group (NIFI-12737) * Canvas global menu item should navigate user back to where they were on the canvas (NIFI-12737) * Better theme support (NIFI-12655) * Set up development/production environments files * Run unit tests are part of standard build * Update all API calls to consider disconnect node confirmation * Update API calls to use uiOnly flag * Use polling interval from API * Routing error handling * General API response error handling ** Management CS (NIFI-12663) ** Canvas CS (NIFI-12684) ** Remainder of Settings (NIFI-12723) ** Counters (NIFI-12723) ** Bulletins (NIFI-12723) ** Flow Designer ** Parameter Contexts ** Provenance ** Queue Listing (NIFI-12742) ** Summary (NIFI-12742) ** Users (NIFI-12742) ** Policies ** Status History * Introduce header in new pages to unify with canvas and offer better navigation. (NIFI-12597) * Prompt user to save Parameter Context when Edit form is dirty * Start/Stop processors, process groups, ... (NIFI-12568) * Dialog vertical resizing on smaller screens do not allow users to access all fields (NIFI-12603) * Flow Analysis report menu [1] [https://github.com/apache/nifi/pull/8053] was: The purpose of this Jira is to track all remaining items following the initial commit [1] for NIFI-11481. The description will be kept up to date with remaining features, tasks, and improvements. As each items is worked, a new sub task Jira will be created and referenced in this description. * Support Parameters in Properties with Allowable Values (NIFI-12401) * Summary (NIFI-12437) ** Remaining work not addressed in initial Jira: *** input ports (NIFI-12504) *** output ports (NIFI-12504) *** remote process groups (NIFI-12504) *** process groups (NIFI-12504) *** connections (NIFI-12504) *** System Diagnostics
[PR] Update minifi-java-agent-quick-start.md [nifi]
wrender opened a new pull request, #8379: URL: https://github.com/apache/nifi/pull/8379 - Add clarity around configuration with json vs yaml # Summary [NIFI-0](https://issues.apache.org/jira/browse/NIFI-0) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [ ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [ ] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [ ] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [ ] Pull Request based on current revision of the `main` branch - [ ] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 21 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFI-12761) RPG refresh
Scott Aslan created NIFI-12761: -- Summary: RPG refresh Key: NIFI-12761 URL: https://issues.apache.org/jira/browse/NIFI-12761 Project: Apache NiFi Issue Type: Sub-task Reporter: Scott Aslan Assignee: Scott Aslan -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12759) RPG GoTo
[ https://issues.apache.org/jira/browse/NIFI-12759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Aslan updated NIFI-12759: --- Status: Patch Available (was: In Progress) > RPG GoTo > > > Key: NIFI-12759 > URL: https://issues.apache.org/jira/browse/NIFI-12759 > Project: Apache NiFi > Issue Type: Sub-task >Reporter: Scott Aslan >Assignee: Scott Aslan >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] NIFI-12735 Adding the option to execute flow analysis before committing snapshot to registry [nifi]
simonbence opened a new pull request, #8377: URL: https://github.com/apache/nifi/pull/8377 # Summary [NIFI-12735](https://issues.apache.org/jira/browse/NIFI-12735) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [ ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [ ] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [ ] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [ ] Pull Request based on current revision of the `main` branch - [ ] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 21 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12745: Fix AvroReader silently dropping malformed records [nifi]
tpalfy commented on code in PR #8361:
URL: https://github.com/apache/nifi/pull/8361#discussion_r1483208617
##
nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/test/java/org/apache/nifi/avro/TestAvroReaderWithExplicitSchema.java:
##
@@ -105,6 +105,34 @@ public void
testAvroExplicitReaderWithEmbeddedSchemaFileDifferentFromExplicitSch
assertThrows(IOException.class, () -> new
AvroReaderWithExplicitSchema(fileInputStream, recordSchema, dataSchema));
}
+@Test
+public void testAvroExplicitReaderWithSchemalessFileAndExplicitSchema()
throws Exception {
+AvroReaderWithExplicitSchema avroReader =
createAvroReaderWithExplicitSchema(
+"src/test/resources/avro/schemaless_simple_record.avro",
+"src/test/resources/avro/schemaless_simple_record.avsc"
+);
+
+GenericData.Record expected = new GenericData.Record(new
Schema.Parser().parse(new
File("src/test/resources/avro/schemaless_simple_record.avsc")));
+expected.put("field_1", 123);
+expected.put("field_2", "44");
+expected.put("field_3", 5);
+
+GenericRecord actual1 = avroReader.nextAvroRecord();
+assertNotNull(actual1);
Review Comment:
```suggestion
assertEquals(expected, actual1);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-12710: Allow for microsecond precision timestamps when pushing t… [nifi]
exceptionfactory commented on code in PR #8332:
URL: https://github.com/apache/nifi/pull/8332#discussion_r1483154296
##
nifi-commons/nifi-record/src/test/java/org/apache/nifi/serialization/record/field/TestObjectLocalDateTimeFieldConverter.java:
##
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.serialization.record.field;
+
+import org.junit.jupiter.api.Test;
+
+import java.time.Instant;
+import java.time.LocalDateTime;
+import java.time.ZoneId;
+import java.util.Optional;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+public class TestObjectLocalDateTimeFieldConverter {
+private final long MILLIS_TIMESTAMP_LONG = 1707238288351L;
+private final long MICROS_TIMESTAMP_LONG = 1707238288351567L;
+private final String MICROS_TIMESTAMP_STRING =
Long.toString(MICROS_TIMESTAMP_LONG);
+private final double MICROS_TIMESTAMP_DOUBLE = ((double)
MICROS_TIMESTAMP_LONG) / 100D;
+private final long NANOS_AFTER_SECOND = 351567000L;
+private final Instant INSTANT_MILLIS_PRECISION =
Instant.ofEpochMilli(MILLIS_TIMESTAMP_LONG);
+// Create an instant to represent the same time as the microsecond
precision timestamp. We add nanoseconds after second but then have to subtract
the milliseconds after the second that are already
+// present in the MILLIS_TIMESTAMP_LONG value.
+private final Instant INSTANT_MICROS_PRECISION =
Instant.ofEpochMilli(MILLIS_TIMESTAMP_LONG).plusNanos(NANOS_AFTER_SECOND).minusMillis(MILLIS_TIMESTAMP_LONG
% 1000);
+private final LocalDateTime LOCAL_DATE_TIME_MILLIS_PRECISION =
LocalDateTime.ofInstant(INSTANT_MILLIS_PRECISION, ZoneId.systemDefault());
+private final LocalDateTime LOCAL_DATE_TIME_MICROS_PRECISION =
LocalDateTime.ofInstant(INSTANT_MICROS_PRECISION, ZoneId.systemDefault());
+
+private final ObjectLocalDateTimeFieldConverter converter = new
ObjectLocalDateTimeFieldConverter();
+
+
+@Test
+public void testConvertTimestampMillis() {
+final LocalDateTime result =
converter.convertField(MILLIS_TIMESTAMP_LONG, Optional.empty(), "test");
Review Comment:
Just a minor bit, but recommend declaring a static `FIELD_NAME` value and
reusing across all test methods.
##
nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/field/ObjectLocalDateTimeFieldConverter.java:
##
@@ -67,22 +70,59 @@ public LocalDateTime convertField(final Object field, final
Optional pat
try {
return LocalDateTime.parse(string, formatter);
} catch (final DateTimeParseException e) {
-throw new FieldConversionException(LocalDateTime.class,
field, name, e);
+return tryParseAsNumber(string, name);
}
} else {
-try {
-final long number = Long.parseLong(string);
-final Instant instant = Instant.ofEpochMilli(number);
-return ofInstant(instant);
-} catch (final NumberFormatException e) {
-throw new FieldConversionException(LocalDateTime.class,
field, name, e);
-}
+return tryParseAsNumber(string, name);
}
}
throw new FieldConversionException(LocalDateTime.class, field, name);
}
+private LocalDateTime tryParseAsNumber(final String value, final String
fieldName) {
+// If decimal, treat as a double and convert to seconds and
nanoseconds.
+if (value.contains(".")) {
+final double number = Double.parseDouble(value);
Review Comment:
The `Double.parseDouble()` method could throw a `NumberFormatException`,
similar to below with `Long.parseLong()`, so it would be helpful to have the
same try-catch that throws a `FieldConversionException` for troubleshooting.
For example, if the string contains two period characters, instead of one,
which would be a good test case.
##
nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/field/ObjectLocalDateTimeFieldConverter.java:
##
@@ -67,22 +70,59 @@ public
Re: [PR] MINIFICPP-2276 Support FlowFileTransform NiFi Python processors [nifi-minifi-cpp]
lordgamez commented on code in PR #1712:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1712#discussion_r1483114670
##
extensions/python/types/PyProcessContext.cpp:
##
@@ -65,12 +66,31 @@ PyObject* PyProcessContext::getProperty(PyProcessContext*
self, PyObject* args)
return nullptr;
}
- const char* property;
- if (!PyArg_ParseTuple(args, "s", )) {
+ const char* property_name = nullptr;
+ PyObject* script_flow_file = nullptr;
+ if (!PyArg_ParseTuple(args, "s|O", _name, _flow_file)) {
throw PyException();
}
+
std::string value;
- context->getProperty(property, value);
+ if (!script_flow_file) {
+if (!context->getProperty(property_name, value)) {
+ Py_RETURN_NONE;
+}
+ } else {
+auto py_flow = reinterpret_cast(script_flow_file);
+const auto flow_file = py_flow->script_flow_file_.lock();
+if (!flow_file) {
+ PyErr_SetString(PyExc_AttributeError, "tried reading FlowFile outside
'on_trigger'");
+ return nullptr;
+}
Review Comment:
Currently the held type of is `std::weak_ptr` in both
`PyScriptFlowFile` and `PyScriptFlowFile`, where this is the way the held
objects are handled in both files. We can refactor this in the future, but for
the time being I think we should stick with using the same handling here as
well, or is there anything else you meant by leaving the door open?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] MINIFICPP-2276 Support FlowFileTransform NiFi Python processors [nifi-minifi-cpp]
lordgamez commented on code in PR #1712:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1712#discussion_r1483114670
##
extensions/python/types/PyProcessContext.cpp:
##
@@ -65,12 +66,31 @@ PyObject* PyProcessContext::getProperty(PyProcessContext*
self, PyObject* args)
return nullptr;
}
- const char* property;
- if (!PyArg_ParseTuple(args, "s", )) {
+ const char* property_name = nullptr;
+ PyObject* script_flow_file = nullptr;
+ if (!PyArg_ParseTuple(args, "s|O", _name, _flow_file)) {
throw PyException();
}
+
std::string value;
- context->getProperty(property, value);
+ if (!script_flow_file) {
+if (!context->getProperty(property_name, value)) {
+ Py_RETURN_NONE;
+}
+ } else {
+auto py_flow = reinterpret_cast(script_flow_file);
+const auto flow_file = py_flow->script_flow_file_.lock();
+if (!flow_file) {
+ PyErr_SetString(PyExc_AttributeError, "tried reading FlowFile outside
'on_trigger'");
+ return nullptr;
+}
Review Comment:
Currently the held type of is `std::weak_ptr` in both
`PyScriptFlowFile` and `PyScriptFlowFile`, where this is the way the held
objects are handled in both files. We can refactor this in the future, but for
the time being I think we should stick with using the same handling here as
well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] MINIFICPP-2276 Support FlowFileTransform NiFi Python processors [nifi-minifi-cpp]
lordgamez commented on code in PR #1712: URL: https://github.com/apache/nifi-minifi-cpp/pull/1712#discussion_r1483106461 ## docker/test/integration/resources/minifi-c2-server/config-ssl.yml: ## @@ -0,0 +1,43 @@ +MiNiFi Config Version: 3 +Flow Controller: + name: MiNiFi Flow +Processors: +- name: Get files from /tmp/input + id: 2f2a3b47-f5ba-49f6-82b5-bc1c86b96e27 + class: org.apache.nifi.minifi.processors.GetFile + scheduling strategy: TIMER_DRIVEN + scheduling period: 1000 ms + Properties: +Input Directory: /tmp/input +- name: Put files to /tmp/output + id: e143601d-de4f-44ba-a6ec-d1f97d77ec94 + class: org.apache.nifi.minifi.processors.PutFile + scheduling strategy: EVENT_DRIVEN + auto-terminated relationships list: + - failure + - success + Properties: +Conflict Resolution Strategy: fail +Create Missing Directories: 'true' +Directory: /tmp/output +Connections: +- name: GetFile/success/PutFile + id: 098a56ba-f4bf-4323-a3f3-6f8a5e3586bf + source id: 2f2a3b47-f5ba-49f6-82b5-bc1c86b96e27 + source relationship names: + - success + destination id: e143601d-de4f-44ba-a6ec-d1f97d77ec94 +Controller Services: + - name: SSLContextService +id: 2438e3c8-015a-1000-79ca-83af40ec1994 +class: SSLContextService Review Comment: This is only used for C2, and this is only the config that is sent from the C2 server after the update. I actually just reverted the json files to the previously used yamls files for these tests and didn't check them. There are separate tests for using the controller service and using the ssl properties in the minifi.properties file. But we can actually remove this file, as we only check if the flow update succeeds and we don't care about the C2 communication afterwards so we can use the same config without the SSL context. Updated in 3d207dca7c2cdf1f86f4b6308a8c02addb2a852d -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [NIFI-12758] create remote process group [nifi]
scottyaslan commented on code in PR #8376: URL: https://github.com/apache/nifi/pull/8376#discussion_r1483087073 ## nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-frontend/src/main/nifi/src/app/_app.component-theme.scss: ## @@ -26,14 +26,12 @@ // Get the color palette from the color-config. $primary-palette: map.get($color-config, 'primary'); $accent-palette: map.get($color-config, 'accent'); -$warn-palette: map.get($color-config, 'warn'); Review Comment: This should have been removed in the snackbar theming PR. I cleaned them up here. ## nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-frontend/src/main/nifi/src/app/_app.component-theme.scss: ## @@ -26,14 +26,12 @@ // Get the color palette from the color-config. $primary-palette: map.get($color-config, 'primary'); $accent-palette: map.get($color-config, 'accent'); -$warn-palette: map.get($color-config, 'warn'); $canvas-primary-palette: map.get($canvas-color-config, 'primary'); // Get hues from palette $primary-palette-300: mat.get-color-from-palette($primary-palette, 300); $primary-palette-500: mat.get-color-from-palette($primary-palette, 500); $accent-palette-A400: mat.get-color-from-palette($accent-palette, 'A400'); -$warn-palette-500: mat.get-color-from-palette($warn-palette, 500); Review Comment: Same thing. This should have been removed in the snackbar theming PR. I cleaned them up here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12671 Added AwsFileResourceService [nifi]
balazsgerner commented on PR #8368: URL: https://github.com/apache/nifi/pull/8368#issuecomment-1934239454 > Thanks @balazsgerner for implementing this new feature. I've tested the `ControllerService` and it works as expected, I was able to move my data from S3 to Azure. I've left a couple of comments please see them below. Thanks for the review @mark-bathori, I pushed a commit with the review item fixes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-12749) NiFi Toolkit export-all-flows fails when flow name contains '/'
[ https://issues.apache.org/jira/browse/NIFI-12749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17815696#comment-17815696 ] ASF subversion and git services commented on NIFI-12749: Commit 99912c559c574885cb41eb7e8e028de28a995d1f in nifi's branch refs/heads/support/nifi-1.x from Timea Barna [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=99912c559c ] NIFI-12749 Handled Forward Slash in Flow Name for nifi-toolkit-cli - The export-all-flows command fails when the Flow Name contains a forward slash This closes #8369 Signed-off-by: David Handermann (cherry picked from commit 1781e166196d2bb04cef3b4e661f191303b3b40c) > NiFi Toolkit export-all-flows fails when flow name contains '/' > --- > > Key: NIFI-12749 > URL: https://issues.apache.org/jira/browse/NIFI-12749 > Project: Apache NiFi > Issue Type: Bug >Reporter: Timea Barna >Assignee: Timea Barna >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > The command fails if there are any flows in the registry that has a / > character because when exporting the flow, the toolkit assumes / is a path > separator as the toolkit will use the versioned flow name and version when > creating the exported files. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12749) NiFi Toolkit export-all-flows fails when flow name contains '/'
[ https://issues.apache.org/jira/browse/NIFI-12749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-12749: Fix Version/s: 2.0.0 1.26.0 Resolution: Fixed Status: Resolved (was: Patch Available) > NiFi Toolkit export-all-flows fails when flow name contains '/' > --- > > Key: NIFI-12749 > URL: https://issues.apache.org/jira/browse/NIFI-12749 > Project: Apache NiFi > Issue Type: Bug >Reporter: Timea Barna >Assignee: Timea Barna >Priority: Major > Fix For: 2.0.0, 1.26.0 > > Time Spent: 20m > Remaining Estimate: 0h > > The command fails if there are any flows in the registry that has a / > character because when exporting the flow, the toolkit assumes / is a path > separator as the toolkit will use the versioned flow name and version when > creating the exported files. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12749) NiFi Toolkit export-all-flows fails when flow name contains '/'
[ https://issues.apache.org/jira/browse/NIFI-12749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17815694#comment-17815694 ] ASF subversion and git services commented on NIFI-12749: Commit 1781e166196d2bb04cef3b4e661f191303b3b40c in nifi's branch refs/heads/main from Timea Barna [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=1781e16619 ] NIFI-12749 Handled Forward Slash in Flow Name for nifi-toolkit-cli - The export-all-flows command fails when the Flow Name contains a forward slash This closes #8369 Signed-off-by: David Handermann > NiFi Toolkit export-all-flows fails when flow name contains '/' > --- > > Key: NIFI-12749 > URL: https://issues.apache.org/jira/browse/NIFI-12749 > Project: Apache NiFi > Issue Type: Bug >Reporter: Timea Barna >Assignee: Timea Barna >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > The command fails if there are any flows in the registry that has a / > character because when exporting the flow, the toolkit assumes / is a path > separator as the toolkit will use the versioned flow name and version when > creating the exported files. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12749 NiFi Toolkit export-all-flows fails when flow name contain… [nifi]
exceptionfactory closed pull request #8369: NIFI-12749 NiFi Toolkit export-all-flows fails when flow name contain… URL: https://github.com/apache/nifi/pull/8369 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-12270) PuAzureBlobStorage with FileResourceService sends 0 byte PROVENANCE event
[ https://issues.apache.org/jira/browse/NIFI-12270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-12270: Issue Type: Improvement (was: Bug) > PuAzureBlobStorage with FileResourceService sends 0 byte PROVENANCE event > - > > Key: NIFI-12270 > URL: https://issues.apache.org/jira/browse/NIFI-12270 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Lehel Boér >Assignee: Lehel Boér >Priority: Major > Time Spent: 3h 10m > Remaining Estimate: 0h > > There'a a need for a new provanace event type which can be emitted in case of > uploading an external resource to an external system without storing it in a > FlowFile. There's an example for the need in PutAzureBlobStorage when > transferring files from the local file system. > See discussion: > [https://lists.apache.org/thread/hwt87hz4679vbmtv9yg78dw38jrwc861] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12270) Add UPLOAD Provenance Event Type for External Resources
[ https://issues.apache.org/jira/browse/NIFI-12270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-12270: Fix Version/s: 2.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Add UPLOAD Provenance Event Type for External Resources > --- > > Key: NIFI-12270 > URL: https://issues.apache.org/jira/browse/NIFI-12270 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Lehel Boér >Assignee: Lehel Boér >Priority: Major > Fix For: 2.0.0 > > Time Spent: 3h 10m > Remaining Estimate: 0h > > There'a a need for a new provanace event type which can be emitted in case of > uploading an external resource to an external system without storing it in a > FlowFile. There's an example for the need in PutAzureBlobStorage when > transferring files from the local file system. > See discussion: > [https://lists.apache.org/thread/hwt87hz4679vbmtv9yg78dw38jrwc861] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12270) Add UPLOAD Provenance Event Type for External Resources
[ https://issues.apache.org/jira/browse/NIFI-12270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-12270: Summary: Add UPLOAD Provenance Event Type for External Resources (was: PuAzureBlobStorage with FileResourceService sends 0 byte PROVENANCE event) > Add UPLOAD Provenance Event Type for External Resources > --- > > Key: NIFI-12270 > URL: https://issues.apache.org/jira/browse/NIFI-12270 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Lehel Boér >Assignee: Lehel Boér >Priority: Major > Time Spent: 3h 10m > Remaining Estimate: 0h > > There'a a need for a new provanace event type which can be emitted in case of > uploading an external resource to an external system without storing it in a > FlowFile. There's an example for the need in PutAzureBlobStorage when > transferring files from the local file system. > See discussion: > [https://lists.apache.org/thread/hwt87hz4679vbmtv9yg78dw38jrwc861] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12270) PuAzureBlobStorage with FileResourceService sends 0 byte PROVENANCE event
[ https://issues.apache.org/jira/browse/NIFI-12270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17815692#comment-17815692 ] ASF subversion and git services commented on NIFI-12270: Commit 6a007f9159f4b2cc71a01cbff650136a2fa62bbf in nifi's branch refs/heads/main from lehelb [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=6a007f9159 ] NIFI-12270 Added UPLOAD Provenance Event Type to nifi-api This closes #8094 Signed-off-by: David Handermann > PuAzureBlobStorage with FileResourceService sends 0 byte PROVENANCE event > - > > Key: NIFI-12270 > URL: https://issues.apache.org/jira/browse/NIFI-12270 > Project: Apache NiFi > Issue Type: Bug >Reporter: Lehel Boér >Assignee: Lehel Boér >Priority: Major > Time Spent: 3h 10m > Remaining Estimate: 0h > > There'a a need for a new provanace event type which can be emitted in case of > uploading an external resource to an external system without storing it in a > FlowFile. There's an example for the need in PutAzureBlobStorage when > transferring files from the local file system. > See discussion: > [https://lists.apache.org/thread/hwt87hz4679vbmtv9yg78dw38jrwc861] -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12270: Add new provenance event: UPLOAD [nifi]
exceptionfactory closed pull request #8094: NIFI-12270: Add new provenance event: UPLOAD URL: https://github.com/apache/nifi/pull/8094 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12745: Fix AvroReader silently dropping malformed records [nifi]
tpalfy commented on code in PR #8361:
URL: https://github.com/apache/nifi/pull/8361#discussion_r1483024295
##
nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/test/java/org/apache/nifi/avro/TestAvroReaderWithExplicitSchema.java:
##
@@ -105,6 +105,36 @@ public void
testAvroExplicitReaderWithEmbeddedSchemaFileDifferentFromExplicitSch
assertThrows(IOException.class, () -> new
AvroReaderWithExplicitSchema(fileInputStream, recordSchema, dataSchema));
}
+@Test
+public void testAvroExplicitReaderWithSchemalessFileAndExplicitSchema()
throws Exception {
+File avroFileWithoutSchema = new
File("src/test/resources/avro/schemaless_simple_record.avro");
+FileInputStream fileInputStream = new
FileInputStream(avroFileWithoutSchema);
+Schema dataSchema = new Schema.Parser().parse(new
File("src/test/resources/avro/schemaless_simple_record.avsc"));
+RecordSchema recordSchema = new
SimpleRecordSchema(dataSchema.toString(), AvroTypeUtil.AVRO_SCHEMA_FORMAT,
null);
+
+AvroReaderWithExplicitSchema avroReader = new
AvroReaderWithExplicitSchema(fileInputStream, recordSchema, dataSchema);
+GenericRecord record = avroReader.nextAvroRecord();
+assertNotNull(record);
+assertEquals(123, record.get("field_1"));
+assertNotNull(record.get("field_2"));
+assertEquals("44", record.get("field_2").toString());
+assertEquals(5, record.get("field_3"));
+
+record = avroReader.nextAvroRecord();
+assertNull(record);
Review Comment:
```suggestion
AvroReaderWithExplicitSchema avroReader =
createAvroReaderWithExplicitSchema(
"src/test/resources/avro/schemaless_simple_record.avro",
"src/test/resources/avro/schemaless_simple_record.avsc"
);
GenericData.Record expected = new GenericData.Record(new
Schema.Parser().parse(new
File("src/test/resources/avro/schemaless_simple_record.avsc")));
expected.put("field_1", 123);
expected.put("field_2", "44");
expected.put("field_3", 5);
GenericRecord actual1 = avroReader.nextAvroRecord();
assertNotNull(actual1);
GenericRecord actual2 = avroReader.nextAvroRecord();
assertNull(actual2);
```
##
nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/test/java/org/apache/nifi/avro/TestAvroReaderWithExplicitSchema.java:
##
@@ -105,6 +105,36 @@ public void
testAvroExplicitReaderWithEmbeddedSchemaFileDifferentFromExplicitSch
assertThrows(IOException.class, () -> new
AvroReaderWithExplicitSchema(fileInputStream, recordSchema, dataSchema));
}
+@Test
+public void testAvroExplicitReaderWithSchemalessFileAndExplicitSchema()
throws Exception {
+File avroFileWithoutSchema = new
File("src/test/resources/avro/schemaless_simple_record.avro");
+FileInputStream fileInputStream = new
FileInputStream(avroFileWithoutSchema);
+Schema dataSchema = new Schema.Parser().parse(new
File("src/test/resources/avro/schemaless_simple_record.avsc"));
+RecordSchema recordSchema = new
SimpleRecordSchema(dataSchema.toString(), AvroTypeUtil.AVRO_SCHEMA_FORMAT,
null);
+
+AvroReaderWithExplicitSchema avroReader = new
AvroReaderWithExplicitSchema(fileInputStream, recordSchema, dataSchema);
+GenericRecord record = avroReader.nextAvroRecord();
+assertNotNull(record);
+assertEquals(123, record.get("field_1"));
+assertNotNull(record.get("field_2"));
+assertEquals("44", record.get("field_2").toString());
+assertEquals(5, record.get("field_3"));
+
+record = avroReader.nextAvroRecord();
+assertNull(record);
+}
+
+@Test
+public void
testAvroExplicitReaderWithSchemalessFileAndWrongExplicitSchema() throws
Exception {
+File avroFileWithoutSchema = new
File("src/test/resources/avro/schemaless_simple_record.avro");
+FileInputStream fileInputStream = new
FileInputStream(avroFileWithoutSchema);
+Schema dataSchema = new Schema.Parser().parse(new
File("src/test/resources/avro/schemaless_simple_record_extra_field.avsc"));
+RecordSchema recordSchema = new
SimpleRecordSchema(dataSchema.toString(), AvroTypeUtil.AVRO_SCHEMA_FORMAT,
null);
+
+AvroReaderWithExplicitSchema avroReader = new
AvroReaderWithExplicitSchema(fileInputStream, recordSchema, dataSchema);
Review Comment:
```suggestion
AvroReaderWithExplicitSchema avroReader =
createAvroReaderWithExplicitSchema(
"src/test/resources/avro/schemaless_simple_record.avro",
"src/test/resources/avro/schemaless_simple_record_extra_field.avsc"
);
```
--
This is an automated message from the Apache
Re: [PR] MINIFICPP-2276 Support FlowFileTransform NiFi Python processors [nifi-minifi-cpp]
szaszm commented on code in PR #1712:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1712#discussion_r1479706920
##
docker/test/integration/resources/minifi-c2-server/config-ssl.yml:
##
@@ -0,0 +1,43 @@
+MiNiFi Config Version: 3
+Flow Controller:
+ name: MiNiFi Flow
+Processors:
+- name: Get files from /tmp/input
+ id: 2f2a3b47-f5ba-49f6-82b5-bc1c86b96e27
+ class: org.apache.nifi.minifi.processors.GetFile
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 1000 ms
+ Properties:
+Input Directory: /tmp/input
+- name: Put files to /tmp/output
+ id: e143601d-de4f-44ba-a6ec-d1f97d77ec94
+ class: org.apache.nifi.minifi.processors.PutFile
+ scheduling strategy: EVENT_DRIVEN
+ auto-terminated relationships list:
+ - failure
+ - success
+ Properties:
+Conflict Resolution Strategy: fail
+Create Missing Directories: 'true'
+Directory: /tmp/output
+Connections:
+- name: GetFile/success/PutFile
+ id: 098a56ba-f4bf-4323-a3f3-6f8a5e3586bf
+ source id: 2f2a3b47-f5ba-49f6-82b5-bc1c86b96e27
+ source relationship names:
+ - success
+ destination id: e143601d-de4f-44ba-a6ec-d1f97d77ec94
+Controller Services:
+ - name: SSLContextService
+id: 2438e3c8-015a-1000-79ca-83af40ec1994
+class: SSLContextService
Review Comment:
Is this used for C2? Because the flow doesn't use it. If it's C2, I think it
would be nicer to include a reference to C2 in its name, or configure SSL in
the config properties in minifi.properties.
##
extensions/python/types/PyProcessContext.cpp:
##
@@ -65,12 +66,31 @@ PyObject* PyProcessContext::getProperty(PyProcessContext*
self, PyObject* args)
return nullptr;
}
- const char* property;
- if (!PyArg_ParseTuple(args, "s", )) {
+ const char* property_name = nullptr;
+ PyObject* script_flow_file = nullptr;
+ if (!PyArg_ParseTuple(args, "s|O", _name, _flow_file)) {
throw PyException();
}
+
std::string value;
- context->getProperty(property, value);
+ if (!script_flow_file) {
+if (!context->getProperty(property_name, value)) {
+ Py_RETURN_NONE;
+}
+ } else {
+auto py_flow = reinterpret_cast(script_flow_file);
+const auto flow_file = py_flow->script_flow_file_.lock();
+if (!flow_file) {
+ PyErr_SetString(PyExc_AttributeError, "tried reading FlowFile outside
'on_trigger'");
+ return nullptr;
+}
Review Comment:
We should leave the door open for a future refactoring of flow file
ownership, where onTrigger calls have non-owning references to flow files,
without shared_ptr usage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-12697: Allow disabling scientific notation when writing JSON; us… [nifi]
markap14 commented on PR #8319: URL: https://github.com/apache/nifi/pull/8319#issuecomment-1934185083 @Lehel44 I'm not sure that I understand the concern. Can you provide a concrete use case? The JSON Reader should be able to read the JSON regardless of whether the incoming data is in scientific notation or not. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12674 Modified ValidateCSV to make the schema optional if a head… [nifi]
Freedom9339 commented on code in PR #8362: URL: https://github.com/apache/nifi/pull/8362#discussion_r1482982203 ## nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ValidateCsv.java: ## @@ -458,156 +476,148 @@ public void onTrigger(final ProcessContext context, final ProcessSession session final CsvPreference csvPref = getPreference(context, flowFile); final boolean header = context.getProperty(HEADER).asBoolean(); final ComponentLog logger = getLogger(); -final String schema = context.getProperty(SCHEMA).evaluateAttributeExpressions(flowFile).getValue(); -final CellProcessor[] cellProcs = this.parseSchema(schema); -final boolean isWholeFFValidation = context.getProperty(VALIDATION_STRATEGY).getValue().equals(VALIDATE_WHOLE_FLOWFILE.getValue()); +String schema = context.getProperty(SCHEMA).evaluateAttributeExpressions(flowFile).getValue(); Review Comment: I could, I just didn't want to make it confusing if down the line the schema variable needs to be used. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12671 Added AwsFileResourceService [nifi]
balazsgerner commented on code in PR #8368:
URL: https://github.com/apache/nifi/pull/8368#discussion_r1482961413
##
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/AwsFileResourceService.java:
##
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.aws.s3;
+
+import com.amazonaws.SdkClientException;
+import com.amazonaws.auth.AWSCredentialsProvider;
+import com.amazonaws.regions.Region;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.AmazonS3Client;
+import com.amazonaws.services.s3.model.S3Object;
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.documentation.UseCase;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.fileresource.service.api.FileResource;
+import org.apache.nifi.fileresource.service.api.FileResourceService;
+import org.apache.nifi.processor.exception.ProcessException;
+import
org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderService;
+
+import java.util.List;
+import java.util.Map;
+
+import static
org.apache.nifi.processors.aws.AbstractAWSCredentialsProviderProcessor.AWS_CREDENTIALS_PROVIDER_SERVICE;
+import static org.apache.nifi.processors.aws.util.RegionUtilV1.resolveRegion;
+import static org.apache.nifi.util.StringUtils.isBlank;
+
+@Tags({"Amazon", "S3", "AWS", "file", "resource"})
+@SeeAlso({FetchS3Object.class})
+@CapabilityDescription("Provides an Amazon Web Services (AWS) S3 file resource
for other components.")
+@UseCase(
+description = "Fetch a specific file from S3. " +
+"The service provides higher performance compared to fetch
processors when the data should be moved between different storages without any
transformation.",
+configuration = """
+"Bucket" = "${s3.bucket}"
+"Name" = "${filename}"
+
+The "Region" property must be set to denote the S3 region that
the Bucket resides in.
+If the flow being built is to be reused elsewhere, it's a good
idea to parameterize this property by setting it to something like #{S3_REGION}.
+
+The "AWS Credentials Provider Service" property should specify
an instance of the AWSCredentialsProviderService in order to provide
credentials for accessing the bucket.
+"""
+)
+public class AwsFileResourceService extends AbstractControllerService
implements FileResourceService {
Review Comment:
Yes, I think it is a better name too, I became too focused on the cloud
provider name, but the only service accessed here is S3 so it makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-12671 Added AwsFileResourceService [nifi]
mark-bathori commented on code in PR #8368:
URL: https://github.com/apache/nifi/pull/8368#discussion_r1482928857
##
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/AwsFileResourceService.java:
##
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.aws.s3;
+
+import com.amazonaws.SdkClientException;
+import com.amazonaws.auth.AWSCredentialsProvider;
+import com.amazonaws.regions.Region;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.AmazonS3Client;
+import com.amazonaws.services.s3.model.S3Object;
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.documentation.UseCase;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.fileresource.service.api.FileResource;
+import org.apache.nifi.fileresource.service.api.FileResourceService;
+import org.apache.nifi.processor.exception.ProcessException;
+import
org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderService;
+
+import java.util.List;
+import java.util.Map;
+
+import static
org.apache.nifi.processors.aws.AbstractAWSCredentialsProviderProcessor.AWS_CREDENTIALS_PROVIDER_SERVICE;
+import static org.apache.nifi.processors.aws.util.RegionUtilV1.resolveRegion;
+import static org.apache.nifi.util.StringUtils.isBlank;
+
+@Tags({"Amazon", "S3", "AWS", "file", "resource"})
+@SeeAlso({FetchS3Object.class})
+@CapabilityDescription("Provides an Amazon Web Services (AWS) S3 file resource
for other components.")
+@UseCase(
+description = "Fetch a specific file from S3. " +
+"The service provides higher performance compared to fetch
processors when the data should be moved between different storages without any
transformation.",
+configuration = """
+"Bucket" = "${s3.bucket}"
+"Name" = "${filename}"
+
+The "Region" property must be set to denote the S3 region that
the Bucket resides in.
+If the flow being built is to be reused elsewhere, it's a good
idea to parameterize this property by setting it to something like #{S3_REGION}.
+
+The "AWS Credentials Provider Service" property should specify
an instance of the AWSCredentialsProviderService in order to provide
credentials for accessing the bucket.
+"""
+)
+public class AwsFileResourceService extends AbstractControllerService
implements FileResourceService {
Review Comment:
I was wondering that maybe `S3FileResourceService` could be a better name
since the Controller Service is fetching data from S3 and also other processors
are using the 'S3' naming eg.: FethcS3, ListS3, PutS3 etc. What is your opinion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-12671 Added AwsFileResourceService [nifi]
mark-bathori commented on code in PR #8368:
URL: https://github.com/apache/nifi/pull/8368#discussion_r1482928857
##
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/AwsFileResourceService.java:
##
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.aws.s3;
+
+import com.amazonaws.SdkClientException;
+import com.amazonaws.auth.AWSCredentialsProvider;
+import com.amazonaws.regions.Region;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.AmazonS3Client;
+import com.amazonaws.services.s3.model.S3Object;
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.documentation.UseCase;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.fileresource.service.api.FileResource;
+import org.apache.nifi.fileresource.service.api.FileResourceService;
+import org.apache.nifi.processor.exception.ProcessException;
+import
org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderService;
+
+import java.util.List;
+import java.util.Map;
+
+import static
org.apache.nifi.processors.aws.AbstractAWSCredentialsProviderProcessor.AWS_CREDENTIALS_PROVIDER_SERVICE;
+import static org.apache.nifi.processors.aws.util.RegionUtilV1.resolveRegion;
+import static org.apache.nifi.util.StringUtils.isBlank;
+
+@Tags({"Amazon", "S3", "AWS", "file", "resource"})
+@SeeAlso({FetchS3Object.class})
+@CapabilityDescription("Provides an Amazon Web Services (AWS) S3 file resource
for other components.")
+@UseCase(
+description = "Fetch a specific file from S3. " +
+"The service provides higher performance compared to fetch
processors when the data should be moved between different storages without any
transformation.",
+configuration = """
+"Bucket" = "${s3.bucket}"
+"Name" = "${filename}"
+
+The "Region" property must be set to denote the S3 region that
the Bucket resides in.
+If the flow being built is to be reused elsewhere, it's a good
idea to parameterize this property by setting it to something like #{S3_REGION}.
+
+The "AWS Credentials Provider Service" property should specify
an instance of the AWSCredentialsProviderService in order to provide
credentials for accessing the bucket.
+"""
+)
+public class AwsFileResourceService extends AbstractControllerService
implements FileResourceService {
Review Comment:
I was wondering that maybe S3FileResourceService could be a better name
since the Controller Service is fetching data from S3 and also other processors
are using the 'S3' naming eg.: FethcS3, ListS3, PutS3 etc. What is your opinion?
##
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/util/RegionUtilV1.java:
##
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the
[PR] MINIFICPP-2231 Replace global CXX flags with target specific ones [nifi-minifi-cpp]
lordgamez opened a new pull request, #1724: URL: https://github.com/apache/nifi-minifi-cpp/pull/1724 https://issues.apache.org/jira/browse/MINIFICPP-2231 Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFICPP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically main)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINIFICPP-2297 Remove unmaintained iOS build support [nifi-minifi-cpp]
lordgamez commented on code in PR #1723:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1723#discussion_r1482852424
##
libminifi/CMakeLists.txt:
##
@@ -26,13 +26,8 @@ set(PROJECT_NAME "nifi-libcore-minifi")
if (WIN32)
add_definitions(-DWIN32_LEAN_AND_MEAN)
endif()
-if (IOS)
-set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fobjc-abi-version=2 -fobjc-arc
-std=gnu++2a -stdlib=libc++ -isysroot ${CMAKE_OSX_SYSROOT} -DIOS")
-set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fobjc-abi-version=2 -fobjc-arc
-isysroot ${CMAKE_OSX_SYSROOT} -DIOS")
-else ()
-set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -DYAML_SUPPORT")
-set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -DYAML_SUPPORT")
-endif()
+set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -DYAML_SUPPORT")
+set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -DYAML_SUPPORT")
Review Comment:
Removed in 7126436f488d0b733dbec018fd61afa32638805a
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] MINIFICPP-2276 Support FlowFileTransform NiFi Python processors [nifi-minifi-cpp]
lordgamez commented on code in PR #1712:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1712#discussion_r1482851080
##
extensions/python/PYTHON.md:
##
@@ -106,20 +127,39 @@ class VaderSentiment(object):
To enable python Processor capabilities, the following options need to be
provided in minifi.properties. The directory specified
can contain processors. Note that the processor name will be the reference in
your flow. Directories are treated like package names.
Therefore if the nifi.python.processor.dir is /tmp/ and you have a
subdirectory named packagedir with the file name file.py, it will
-produce a processor with the name
org.apache.nifi.minifi.processors.packagedir.file. Note that each subdirectory
will append a package
-to the reference class name.
+produce a processor with the name
org.apache.nifi.minifi.processors.packagedir.file. Note that each subdirectory
will append a package
+to the reference class name.
in minifi.properties
#directory where processors exist
nifi.python.processor.dir=
-
-
+
+
## Processors
The python directory (extensions/pythonprocessors) contains implementations
that will be available for flows if the required dependencies
exist.
-
-## Sentiment Analysis
+
+### Sentiment Analysis
The SentimentAnalysis processor will perform a Vader Sentiment Analysis. This
requires that you install nltk and VaderSentiment
pip install nltk
pip install VaderSentiment
+
+## Using NiFi Python Processors
+
+MiNiFi C++ supports the use of NiFi Python processors, that are inherited from
the FlowFileTransform base class. To use these processors, you must copy the
Python processor module under the nifi_python_processors directory located
under the python directory (by default it can be found at
${minifi_root}/minifi-python/nifi_python_processors). To see how to to write
NiFi Python processors, please refer to the Python Developer Guide under the
[Apache NiFi documentation](https://nifi.apache.org/documentation/v2/).
Review Comment:
Fixed in fc069e05f898ca1b70414d26b325b3e6af1456c0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (NIFI-12751) [MiNiFi] Prefer IPv4 address for creating NetworkInfo
[
https://issues.apache.org/jira/browse/NIFI-12751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferenc Erdei updated NIFI-12751:
Fix Version/s: 2.0.0
> [MiNiFi] Prefer IPv4 address for creating NetworkInfo
> -
>
> Key: NIFI-12751
> URL: https://issues.apache.org/jira/browse/NIFI-12751
> Project: Apache NiFi
> Issue Type: Improvement
> Components: MiNiFi
>Reporter: Ferenc Kis
>Assignee: Ferenc Kis
>Priority: Major
> Fix For: 2.0.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> On Windows Server 2019 the agent sends the following
> {code}
> "networkInfo": \{
> "deviceId": "ethernet_32777",
> "hostname": "fe80:0:0:0:6f46:95b8:1b4d:9ab5%ethernet_32777",
> "ipAddress": "fe80:0:0:0:6f46:95b8:1b4d:9ab5%ethernet_32777"
> },
> {code}
> The same network interface also has an IPv4 address where the hostname
> matches what the hostname command would give back in the shell.
> In such cases when the same network interface has IPv4 and IPv6 addresses,
> prefer the the IPv4 as it provides more information for the user.
> Note: this does not affect anything in terms of network connectivity, only
> the information which is sent by the agent to the C2 server
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (NIFI-12751) [MiNiFi] Prefer IPv4 address for creating NetworkInfo
[
https://issues.apache.org/jira/browse/NIFI-12751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferenc Erdei resolved NIFI-12751.
-
Resolution: Fixed
> [MiNiFi] Prefer IPv4 address for creating NetworkInfo
> -
>
> Key: NIFI-12751
> URL: https://issues.apache.org/jira/browse/NIFI-12751
> Project: Apache NiFi
> Issue Type: Improvement
> Components: MiNiFi
>Reporter: Ferenc Kis
>Assignee: Ferenc Kis
>Priority: Major
> Fix For: 2.0.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> On Windows Server 2019 the agent sends the following
> {code}
> "networkInfo": \{
> "deviceId": "ethernet_32777",
> "hostname": "fe80:0:0:0:6f46:95b8:1b4d:9ab5%ethernet_32777",
> "ipAddress": "fe80:0:0:0:6f46:95b8:1b4d:9ab5%ethernet_32777"
> },
> {code}
> The same network interface also has an IPv4 address where the hostname
> matches what the hostname command would give back in the shell.
> In such cases when the same network interface has IPv4 and IPv6 addresses,
> prefer the the IPv4 as it provides more information for the user.
> Note: this does not affect anything in terms of network connectivity, only
> the information which is sent by the agent to the C2 server
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-12751) [MiNiFi] Prefer IPv4 address for creating NetworkInfo
[
https://issues.apache.org/jira/browse/NIFI-12751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17815645#comment-17815645
]
ASF subversion and git services commented on NIFI-12751:
Commit 84878df61ab7340bd7340922f651e862afa74b9a in nifi's branch
refs/heads/main from Ferenc Kis
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=84878df61a ]
NIFI-12751 [MiNiFi] Prefer IPv4 address when sending NetworkInfo to C2
Signed-off-by: Ferenc Erdei
This closes #8370
> [MiNiFi] Prefer IPv4 address for creating NetworkInfo
> -
>
> Key: NIFI-12751
> URL: https://issues.apache.org/jira/browse/NIFI-12751
> Project: Apache NiFi
> Issue Type: Improvement
> Components: MiNiFi
>Reporter: Ferenc Kis
>Assignee: Ferenc Kis
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> On Windows Server 2019 the agent sends the following
> {code}
> "networkInfo": \{
> "deviceId": "ethernet_32777",
> "hostname": "fe80:0:0:0:6f46:95b8:1b4d:9ab5%ethernet_32777",
> "ipAddress": "fe80:0:0:0:6f46:95b8:1b4d:9ab5%ethernet_32777"
> },
> {code}
> The same network interface also has an IPv4 address where the hostname
> matches what the hostname command would give back in the shell.
> In such cases when the same network interface has IPv4 and IPv6 addresses,
> prefer the the IPv4 as it provides more information for the user.
> Note: this does not affect anything in terms of network connectivity, only
> the information which is sent by the agent to the C2 server
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] NIFI-12751 [MiNiFi] Prefer IPv4 address when sending NetworkInfo to C2 [nifi]
ferencerdei closed pull request #8370: NIFI-12751 [MiNiFi] Prefer IPv4 address when sending NetworkInfo to C2 URL: https://github.com/apache/nifi/pull/8370 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12751 [MiNiFi] Prefer IPv4 address when sending NetworkInfo to C2 [nifi]
ferencerdei commented on PR #8370: URL: https://github.com/apache/nifi/pull/8370#issuecomment-1933918832 LGTM, merging -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINIFICPP-2297 Remove unmaintained iOS build support [nifi-minifi-cpp]
szaszm commented on code in PR #1723:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1723#discussion_r1482815102
##
libminifi/CMakeLists.txt:
##
@@ -26,13 +26,8 @@ set(PROJECT_NAME "nifi-libcore-minifi")
if (WIN32)
add_definitions(-DWIN32_LEAN_AND_MEAN)
endif()
-if (IOS)
-set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fobjc-abi-version=2 -fobjc-arc
-std=gnu++2a -stdlib=libc++ -isysroot ${CMAKE_OSX_SYSROOT} -DIOS")
-set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fobjc-abi-version=2 -fobjc-arc
-isysroot ${CMAKE_OSX_SYSROOT} -DIOS")
-else ()
-set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -DYAML_SUPPORT")
-set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -DYAML_SUPPORT")
-endif()
+set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -DYAML_SUPPORT")
+set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -DYAML_SUPPORT")
Review Comment:
I think these are not used anywhere, and can be removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[PR] MINIFICPP-2297 Remove unmaintained iOS build support [nifi-minifi-cpp]
lordgamez opened a new pull request, #1723: URL: https://github.com/apache/nifi-minifi-cpp/pull/1723 https://issues.apache.org/jira/browse/MINIFICPP-2297 --- Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFICPP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically main)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
