[jira] [Created] (NIFI-11552) Support FlowFile attributes in PutIceberg's Table Name property
Matt Burgess created NIFI-11552: --- Summary: Support FlowFile attributes in PutIceberg's Table Name property Key: NIFI-11552 URL: https://issues.apache.org/jira/browse/NIFI-11552 Project: Apache NiFi Issue Type: Improvement Components: Extensions Reporter: Matt Burgess Fix For: 1.latest, 2.latest The documentation for PutIceberg's Table Name property says it doesn’t support any Expression Language but the code calls the evaluate method on the property without passing in a FlowFile, so at the very least it supports Variable Registry. this Jira proposes to add EL support including the FlowFile attributes for the Table Name property and update the documentation to reflect the new behavior. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11551) Default Use Avro Logical Types to 'true'
Matt Burgess created NIFI-11551: --- Summary: Default Use Avro Logical Types to 'true' Key: NIFI-11551 URL: https://issues.apache.org/jira/browse/NIFI-11551 Project: Apache NiFi Issue Type: Improvement Components: Extensions Reporter: Matt Burgess Fix For: 2.latest Starting with Apache NiFi 2.0, we should default any Use Avro Logical Types properties to 'true'. NiFi supports most/all Avro logical types as native NiFi record field types, so to get better datatype resolution downstream, we should be using Avro logical types wherever possible. Historically, the reason it defaulted to false was because NiFi originally used Avro before logical types were implemented, so when they were added to Avro, NiFi kept the existing behavior of not using logical types by default. At this point we should use logical types wherever possible (by default). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11538) PutIceberg does not correctly convert primitive source objects into target objects
[ https://issues.apache.org/jira/browse/NIFI-11538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-11538: Summary: PutIceberg does not correctly convert primitive source objects into target objects (was: PutIceberg does not correctly convert source objects into target objects) > PutIceberg does not correctly convert primitive source objects into target > objects > -- > > Key: NIFI-11538 > URL: https://issues.apache.org/jira/browse/NIFI-11538 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > > When inserting data into Iceberg table using the PutIceberg processor, if the > incoming record field(s) being inserted have a different datatype than the > target column type, the data is not automatically converted but throws a > ClassCastException. This happens for primitive types such as long <-> int, > int <-> string, etc. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-11538) PutIceberg does not correctly convert source objects into target objects
[ https://issues.apache.org/jira/browse/NIFI-11538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess reassigned NIFI-11538: --- Assignee: Matt Burgess > PutIceberg does not correctly convert source objects into target objects > > > Key: NIFI-11538 > URL: https://issues.apache.org/jira/browse/NIFI-11538 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > > When inserting data into Iceberg table using the PutIceberg processor, if the > incoming record field(s) being inserted have a different datatype than the > target column type, the data is not automatically converted but throws a > ClassCastException. This happens for primitive types such as long <-> int, > int <-> string, etc. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11538) PutIceberg does not correctly convert source objects into target objects
Matt Burgess created NIFI-11538: --- Summary: PutIceberg does not correctly convert source objects into target objects Key: NIFI-11538 URL: https://issues.apache.org/jira/browse/NIFI-11538 Project: Apache NiFi Issue Type: Bug Components: Extensions Reporter: Matt Burgess When inserting data into Iceberg table using the PutIceberg processor, if the incoming record field(s) being inserted have a different datatype than the target column type, the data is not automatically converted but throws a ClassCastException. This happens for primitive types such as long <-> int, int <-> string, etc. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11537) Add support for Iceberg tables to UpdateHive3Table
[ https://issues.apache.org/jira/browse/NIFI-11537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-11537: Fix Version/s: 1.latest 2.latest Status: Patch Available (was: In Progress) > Add support for Iceberg tables to UpdateHive3Table > --- > > Key: NIFI-11537 > URL: https://issues.apache.org/jira/browse/NIFI-11537 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Fix For: 1.latest, 2.latest > > Time Spent: 10m > Remaining Estimate: 0h > > UpdateHive3Table currently adds columns to Iceberg-backed tables > successfully, but Iceberg needs a special CREATE TABLE command to specify the > Iceberg Storage Handler and table properties. > This Jira proposes to add a Create Table Storage Handler property with > Default and Iceberg as the initial choices. Default does not generate a > STORED BY clause, and Iceberg will generate the appropriate STORED BY clause > and set the necessary table properties. This approach can be used in the > future to add support for HBase- and Kudu-backed Hive tables. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-11537) Add support for Iceberg tables to UpdateHive3Table
[ https://issues.apache.org/jira/browse/NIFI-11537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess reassigned NIFI-11537: --- Assignee: Matt Burgess > Add support for Iceberg tables to UpdateHive3Table > --- > > Key: NIFI-11537 > URL: https://issues.apache.org/jira/browse/NIFI-11537 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > > UpdateHive3Table currently adds columns to Iceberg-backed tables > successfully, but Iceberg needs a special CREATE TABLE command to specify the > Iceberg Storage Handler and table properties. > This Jira proposes to add a Create Table Storage Handler property with > Default and Iceberg as the initial choices. Default does not generate a > STORED BY clause, and Iceberg will generate the appropriate STORED BY clause > and set the necessary table properties. This approach can be used in the > future to add support for HBase- and Kudu-backed Hive tables. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11537) Add support for Iceberg tables to UpdateHive3Table
Matt Burgess created NIFI-11537: --- Summary: Add support for Iceberg tables to UpdateHive3Table Key: NIFI-11537 URL: https://issues.apache.org/jira/browse/NIFI-11537 Project: Apache NiFi Issue Type: Improvement Components: Extensions Reporter: Matt Burgess UpdateHive3Table currently adds columns to Iceberg-backed tables successfully, but Iceberg needs a special CREATE TABLE command to specify the Iceberg Storage Handler and table properties. This Jira proposes to add a Create Table Storage Handler property with Default and Iceberg as the initial choices. Default does not generate a STORED BY clause, and Iceberg will generate the appropriate STORED BY clause and set the necessary table properties. This approach can be used in the future to add support for HBase- and Kudu-backed Hive tables. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-11449) add autocommit property to PutDatabaseRecord processor
[
https://issues.apache.org/jira/browse/NIFI-11449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess reassigned NIFI-11449:
---
Assignee: (was: Matt Burgess)
> add autocommit property to PutDatabaseRecord processor
> --
>
> Key: NIFI-11449
> URL: https://issues.apache.org/jira/browse/NIFI-11449
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Extensions
>Affects Versions: 1.21.0
> Environment: Any Nifi Deployment
>Reporter: Abdelrahim Ahmad
>Priority: Blocker
> Labels: Trino, autocommit, database, iceberg, putdatabaserecord
>
> The issue is with the {{PutDatabaseRecord}} processor in Apache NiFi. When
> using the processor with the Trino-JDBC-Driver or Dremio-JDBC-Driver to write
> to an Iceberg catalog, it disables the autocommit feature. This leads to
> errors such as "{*}Catalog only supports writes using autocommit: iceberg{*}".
> the autocommit feature needs to be added in the processor to be
> enabled/disabled.
> enabling auto-commit in the Nifi PutDatabaseRecord processor is important for
> Deltalake, Iceberg, and Hudi as it ensures data consistency and integrity by
> allowing atomic writes to be performed in the underlying database. This will
> allow the process to be widely used with bigger range of databases.
> _Improving this processor will allow Nifi to be the main tool to ingest data
> into these new Technologies. So we don't have to deal with another tool to do
> so._
> +*_{color:#de350b}BUT:{color}_*+
> I have reviewed The {{PutDatabaseRecord}} processor in NiFi. It inserts
> records one by one into the database using a prepared statement, and commits
> the transaction at the end of the loop that processes each record. This
> approach can be inefficient and slow when inserting large volumes of data
> into tables that are optimized for bulk ingestion, such as Delta Lake,
> Iceberg, and Hudi tables.
> These tables use various techniques to optimize the performance of bulk
> ingestion, such as partitioning, clustering, and indexing. Inserting records
> one by one using a prepared statement can bypass these optimizations, leading
> to poor performance and potentially causing issues such as excessive disk
> usage, increased memory consumption, and decreased query performance.
> To avoid these issues, it is recommended to have a new processor, or add
> feature to the current one, to bulk insert method with AutoCommit feature
> when inserting large volumes of data into Delta Lake, Iceberg, and Hudi
> tables.
>
> P.S.: using PutSQL is not a have autoCommit but have the same performance
> problem described above..
> Thanks and best regards :)
> Abdelrahim Ahmad
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (NIFI-11449) add autocommit property to PutDatabaseRecord processor
[
https://issues.apache.org/jira/browse/NIFI-11449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess reassigned NIFI-11449:
---
Assignee: Matt Burgess
> add autocommit property to PutDatabaseRecord processor
> --
>
> Key: NIFI-11449
> URL: https://issues.apache.org/jira/browse/NIFI-11449
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Extensions
>Affects Versions: 1.21.0
> Environment: Any Nifi Deployment
>Reporter: Abdelrahim Ahmad
>Assignee: Matt Burgess
>Priority: Blocker
> Labels: Trino, autocommit, database, iceberg, putdatabaserecord
>
> The issue is with the {{PutDatabaseRecord}} processor in Apache NiFi. When
> using the processor with the Trino-JDBC-Driver or Dremio-JDBC-Driver to write
> to an Iceberg catalog, it disables the autocommit feature. This leads to
> errors such as "{*}Catalog only supports writes using autocommit: iceberg{*}".
> the autocommit feature needs to be added in the processor to be
> enabled/disabled.
> enabling auto-commit in the Nifi PutDatabaseRecord processor is important for
> Deltalake, Iceberg, and Hudi as it ensures data consistency and integrity by
> allowing atomic writes to be performed in the underlying database. This will
> allow the process to be widely used with bigger range of databases.
> _Improving this processor will allow Nifi to be the main tool to ingest data
> into these new Technologies. So we don't have to deal with another tool to do
> so._
> +*_{color:#de350b}BUT:{color}_*+
> I have reviewed The {{PutDatabaseRecord}} processor in NiFi. It inserts
> records one by one into the database using a prepared statement, and commits
> the transaction at the end of the loop that processes each record. This
> approach can be inefficient and slow when inserting large volumes of data
> into tables that are optimized for bulk ingestion, such as Delta Lake,
> Iceberg, and Hudi tables.
> These tables use various techniques to optimize the performance of bulk
> ingestion, such as partitioning, clustering, and indexing. Inserting records
> one by one using a prepared statement can bypass these optimizations, leading
> to poor performance and potentially causing issues such as excessive disk
> usage, increased memory consumption, and decreased query performance.
> To avoid these issues, it is recommended to have a new processor, or add
> feature to the current one, to bulk insert method with AutoCommit feature
> when inserting large volumes of data into Delta Lake, Iceberg, and Hudi
> tables.
>
> P.S.: using PutSQL is not a have autoCommit but have the same performance
> problem described above..
> Thanks and best regards :)
> Abdelrahim Ahmad
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-11221) Remove support for processor-level connection configuration in the MongoDB package
[ https://issues.apache.org/jira/browse/NIFI-11221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-11221: Fix Version/s: 2.latest (was: 2.0.0) > Remove support for processor-level connection configuration in the MongoDB > package > -- > > Key: NIFI-11221 > URL: https://issues.apache.org/jira/browse/NIFI-11221 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Assignee: Mike Thomsen >Priority: Major > Fix For: 2.latest > > Time Spent: 0.5h > Remaining Estimate: 0h > > The configuration should be done entirely through the controller service that > manages the connections. That class has been around for a few years now, so > it shouldn't be a surprise to anyone. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11221) Remove support for processor-level connection configuration in the MongoDB package
[ https://issues.apache.org/jira/browse/NIFI-11221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-11221: Status: Patch Available (was: Open) > Remove support for processor-level connection configuration in the MongoDB > package > -- > > Key: NIFI-11221 > URL: https://issues.apache.org/jira/browse/NIFI-11221 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Assignee: Mike Thomsen >Priority: Major > Fix For: 2.0.0 > > Time Spent: 10m > Remaining Estimate: 0h > > The configuration should be done entirely through the controller service that > manages the connections. That class has been around for a few years now, so > it shouldn't be a surprise to anyone. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11034) Image Viewer not available in Apache NiFi release
[ https://issues.apache.org/jira/browse/NIFI-11034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-11034: Status: Patch Available (was: In Progress) > Image Viewer not available in Apache NiFi release > - > > Key: NIFI-11034 > URL: https://issues.apache.org/jira/browse/NIFI-11034 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Fix For: 1.latest, 2.latest > > Time Spent: 10m > Remaining Estimate: 0h > > The image viewer in the UI (used to display JPEG, GIF, WEBP, etc.) is in the > nifi-media-nar, which is no longer included in the Apache NiFi binary release > due to size constraints. However, the image viewer itself should be available > in the binary release (unless it is too large by itself). > Recommend breaking the image viewer out into its own module and include it in > the NiFi assembly by default. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-11034) Image Viewer not available in Apache NiFi release
[ https://issues.apache.org/jira/browse/NIFI-11034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess reassigned NIFI-11034: --- Assignee: Matt Burgess > Image Viewer not available in Apache NiFi release > - > > Key: NIFI-11034 > URL: https://issues.apache.org/jira/browse/NIFI-11034 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > > The image viewer in the UI (used to display JPEG, GIF, WEBP, etc.) is in the > nifi-media-nar, which is no longer included in the Apache NiFi binary release > due to size constraints. However, the image viewer itself should be available > in the binary release (unless it is too large by itself). > Recommend breaking the image viewer out into its own module and include it in > the NiFi assembly by default. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11034) Image Viewer not available in Apache NiFi release
[ https://issues.apache.org/jira/browse/NIFI-11034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-11034: Fix Version/s: 1.latest 2.latest > Image Viewer not available in Apache NiFi release > - > > Key: NIFI-11034 > URL: https://issues.apache.org/jira/browse/NIFI-11034 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Fix For: 1.latest, 2.latest > > > The image viewer in the UI (used to display JPEG, GIF, WEBP, etc.) is in the > nifi-media-nar, which is no longer included in the Apache NiFi binary release > due to size constraints. However, the image viewer itself should be available > in the binary release (unless it is too large by itself). > Recommend breaking the image viewer out into its own module and include it in > the NiFi assembly by default. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-10946) ScriptedRecord processors script error handling
[ https://issues.apache.org/jira/browse/NIFI-10946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17717931#comment-17717931 ] Matt Burgess commented on NIFI-10946: - Is there a reason to have the processor itself set the attribute for the error, or should we just leave it to the script writer to handle errors and set the attribute in the script and route the FlowFile to failure? > ScriptedRecord processors script error handling > --- > > Key: NIFI-10946 > URL: https://issues.apache.org/jira/browse/NIFI-10946 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Julien G. >Priority: Major > > An attribute is written if an error occurred in the ScriptedProcessor's > reader. > But if an error occurs in the script execution, the FlowFile will be routed > to failure but the error will not be written to an attribute in the FlowFile. > Can an improvment be made in this direction? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11473) Flow version change in NiFi should not stop a component when only position is changed
[ https://issues.apache.org/jira/browse/NIFI-11473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-11473: Fix Version/s: 2.0.0 1.22.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Flow version change in NiFi should not stop a component when only position is > changed > - > > Key: NIFI-11473 > URL: https://issues.apache.org/jira/browse/NIFI-11473 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Timea Barna >Assignee: Timea Barna >Priority: Major > Fix For: 2.0.0, 1.22.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > When going from one flow version to another and the position of a component > is changing, but not its configuration, the component should not be stopped. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11466) Add a ModifyCompression processor
[ https://issues.apache.org/jira/browse/NIFI-11466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-11466: Status: Patch Available (was: In Progress) > Add a ModifyCompression processor > - > > Key: NIFI-11466 > URL: https://issues.apache.org/jira/browse/NIFI-11466 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Fix For: 2.latest > > Time Spent: 10m > Remaining Estimate: 0h > > If a user would like to convert from one compression format to another, they > currently have to use CompressContent to decompress, then another > CompressContent to compress into a different format. Two processors plus disk > I/O for the FlowFiles and their underlying content claims can be I/O > intensive in that case. > Instead, a new ModifyCompression processor is proposed, to allow for both > decompression of the incoming FlowFile and compression for the outgoing > FlowFile, using appropriate memory buffers for the > decompression/recompression. Adding "no decompression" and "no compression" > options for the respective properties could allow this property to function > like CompressContent does now, plus the ability to convert from one > compression format (gzip, e.g.) to another (snappy-hadoop, e.g.). One example > of a use case where this would be helpful is an I/O bound flow to get > compressed data from a legacy source system into HDFS for faster (and > larger-volume / distributed) processing of the data. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-11466) Add a ModifyCompression processor
[ https://issues.apache.org/jira/browse/NIFI-11466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess reassigned NIFI-11466: --- Assignee: Matt Burgess > Add a ModifyCompression processor > - > > Key: NIFI-11466 > URL: https://issues.apache.org/jira/browse/NIFI-11466 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Fix For: 2.latest > > > If a user would like to convert from one compression format to another, they > currently have to use CompressContent to decompress, then another > CompressContent to compress into a different format. Two processors plus disk > I/O for the FlowFiles and their underlying content claims can be I/O > intensive in that case. > Instead, a new ModifyCompression processor is proposed, to allow for both > decompression of the incoming FlowFile and compression for the outgoing > FlowFile, using appropriate memory buffers for the > decompression/recompression. Adding "no decompression" and "no compression" > options for the respective properties could allow this property to function > like CompressContent does now, plus the ability to convert from one > compression format (gzip, e.g.) to another (snappy-hadoop, e.g.). One example > of a use case where this would be helpful is an I/O bound flow to get > compressed data from a legacy source system into HDFS for faster (and > larger-volume / distributed) processing of the data. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11466) Add a ModifyCompression processor
Matt Burgess created NIFI-11466: --- Summary: Add a ModifyCompression processor Key: NIFI-11466 URL: https://issues.apache.org/jira/browse/NIFI-11466 Project: Apache NiFi Issue Type: New Feature Components: Extensions Reporter: Matt Burgess Fix For: 2.latest If a user would like to convert from one compression format to another, they currently have to use CompressContent to decompress, then another CompressContent to compress into a different format. Two processors plus disk I/O for the FlowFiles and their underlying content claims can be I/O intensive in that case. Instead, a new ModifyCompression processor is proposed, to allow for both decompression of the incoming FlowFile and compression for the outgoing FlowFile, using appropriate memory buffers for the decompression/recompression. Adding "no decompression" and "no compression" options for the respective properties could allow this property to function like CompressContent does now, plus the ability to convert from one compression format (gzip, e.g.) to another (snappy-hadoop, e.g.). One example of a use case where this would be helpful is an I/O bound flow to get compressed data from a legacy source system into HDFS for faster (and larger-volume / distributed) processing of the data. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11380) Refactor CaptureChangeMySQL with improvements
[ https://issues.apache.org/jira/browse/NIFI-11380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-11380: Status: Patch Available (was: In Progress) > Refactor CaptureChangeMySQL with improvements > - > > Key: NIFI-11380 > URL: https://issues.apache.org/jira/browse/NIFI-11380 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Fix For: 1.latest, 2.latest > > Time Spent: 10m > Remaining Estimate: 0h > > The CaptureChangeMySQL processor can be improved in many ways: > - Eliminate use of DistributedCacheClient > - MySQLCDCUtils - delete this class. It’s unnecessary. Just put the single > method that exists in the parent class of the writers > - Eliminate mutable member variables. Gather all state together into an > object and store that as a single volatile member variable. > - The outputEvents method is a huge block of switch/case and if/then/else > blocks. Kill all of this. Create an interface that’s capable of handling a > given event type and have multiple implementations. Determine appropriate > impl and call the method. > - Do not keep a bunch of member variables to “rollback local state”. Keep > this in variables. If we fail, no harm, no foul. If we succeed, then update > member variable. > - Remove onStopped method, just annotate stop() method with @OnStopped. No > need for @OnShutdown > - Change name of “hostname” property to "node", and don’t require the port! > Default to 3306. > - Remove unused hasRun member variable -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11380) Refactor CaptureChangeMySQL with improvements
[ https://issues.apache.org/jira/browse/NIFI-11380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-11380: Fix Version/s: 1.latest 2.latest > Refactor CaptureChangeMySQL with improvements > - > > Key: NIFI-11380 > URL: https://issues.apache.org/jira/browse/NIFI-11380 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Fix For: 1.latest, 2.latest > > > The CaptureChangeMySQL processor can be improved in many ways: > - Eliminate use of DistributedCacheClient > - MySQLCDCUtils - delete this class. It’s unnecessary. Just put the single > method that exists in the parent class of the writers > - Eliminate mutable member variables. Gather all state together into an > object and store that as a single volatile member variable. > - The outputEvents method is a huge block of switch/case and if/then/else > blocks. Kill all of this. Create an interface that’s capable of handling a > given event type and have multiple implementations. Determine appropriate > impl and call the method. > - Do not keep a bunch of member variables to “rollback local state”. Keep > this in variables. If we fail, no harm, no foul. If we succeed, then update > member variable. > - Remove onStopped method, just annotate stop() method with @OnStopped. No > need for @OnShutdown > - Change name of “hostname” property to "node", and don’t require the port! > Default to 3306. > - Remove unused hasRun member variable -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-11380) Refactor CaptureChangeMySQL with improvements
[ https://issues.apache.org/jira/browse/NIFI-11380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess reassigned NIFI-11380: --- Assignee: Matt Burgess > Refactor CaptureChangeMySQL with improvements > - > > Key: NIFI-11380 > URL: https://issues.apache.org/jira/browse/NIFI-11380 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > > The CaptureChangeMySQL processor can be improved in many ways: > - Eliminate use of DistributedCacheClient > - MySQLCDCUtils - delete this class. It’s unnecessary. Just put the single > method that exists in the parent class of the writers > - Eliminate mutable member variables. Gather all state together into an > object and store that as a single volatile member variable. > - The outputEvents method is a huge block of switch/case and if/then/else > blocks. Kill all of this. Create an interface that’s capable of handling a > given event type and have multiple implementations. Determine appropriate > impl and call the method. > - Do not keep a bunch of member variables to “rollback local state”. Keep > this in variables. If we fail, no harm, no foul. If we succeed, then update > member variable. > - Remove onStopped method, just annotate stop() method with @OnStopped. No > need for @OnShutdown > - Change name of “hostname” property to "node", and don’t require the port! > Default to 3306. > - Remove unused hasRun member variable -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11380) Refactor CaptureChangeMySQL with improvements
Matt Burgess created NIFI-11380: --- Summary: Refactor CaptureChangeMySQL with improvements Key: NIFI-11380 URL: https://issues.apache.org/jira/browse/NIFI-11380 Project: Apache NiFi Issue Type: Improvement Reporter: Matt Burgess The CaptureChangeMySQL processor can be improved in many ways: - Eliminate use of DistributedCacheClient - MySQLCDCUtils - delete this class. It’s unnecessary. Just put the single method that exists in the parent class of the writers - Eliminate mutable member variables. Gather all state together into an object and store that as a single volatile member variable. - The outputEvents method is a huge block of switch/case and if/then/else blocks. Kill all of this. Create an interface that’s capable of handling a given event type and have multiple implementations. Determine appropriate impl and call the method. - Do not keep a bunch of member variables to “rollback local state”. Keep this in variables. If we fail, no harm, no foul. If we succeed, then update member variable. - Remove onStopped method, just annotate stop() method with @OnStopped. No need for @OnShutdown - Change name of “hostname” property to "node", and don’t require the port! Default to 3306. - Remove unused hasRun member variable -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-8710) StandardProvenanceReporter - receive() incorrectly overiding details parameter
[ https://issues.apache.org/jira/browse/NIFI-8710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-8710: --- Fix Version/s: 2.latest > StandardProvenanceReporter - receive() incorrectly overiding details parameter > -- > > Key: NIFI-8710 > URL: https://issues.apache.org/jira/browse/NIFI-8710 > Project: Apache NiFi > Issue Type: Bug >Reporter: Nissim Shiman >Assignee: Matt Burgess >Priority: Major > Fix For: 2.latest > > Time Spent: 1h > Remaining Estimate: 0h > > The ProvenanceReporter interface has receive() with details as the third > parameter [1]: > void receive(FlowFile flowFile, String transitUri, String details, long > transmissionMillis); > StandardProvenanceReporter.java implements this with > sourceSystemFlowFileIdentifier as the third parameter [2]: > public void receive(final FlowFile flowFile, final String transitUri, final > String sourceSystemFlowFileIdentifier, final long transmissionMillis) > This implementation in StandardProvenanceReporter should be modified to > reflect the interface. > > [1] > [https://github.com/apache/nifi/blob/main/nifi-api/src/main/java/org/apache/nifi/provenance/ProvenanceReporter.java#L100] > [2] > [https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/repository/StandardProvenanceReporter.java#L159] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11337) PutDatabaseRecord docs don't reflect actual behavior and Max Batch Size shouldn't be a dependent property
[ https://issues.apache.org/jira/browse/NIFI-11337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-11337: Fix Version/s: 1.latest 2.latest Status: Patch Available (was: In Progress) > PutDatabaseRecord docs don't reflect actual behavior and Max Batch Size > shouldn't be a dependent property > - > > Key: NIFI-11337 > URL: https://issues.apache.org/jira/browse/NIFI-11337 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Fix For: 1.latest, 2.latest > > Time Spent: 10m > Remaining Estimate: 0h > > The current documentation for PutDatabaseRecord's Maximum Batch Size property > says: "Specifies maximum batch size for INSERT and UPDATE statements. This > parameter has no effect for other statements specified in 'Statement Type'. > Zero means the batch size is not limited." This is not accurate as the > Maximum Batch Size is used for all batches regardless of statement type. This > also makes this property not dependent so that should be removed as well as > the documentation updated to reflect the the behavior. The default value of > Maximum Batch size should also be changed from 0 to something like 1000 to > avoid possible memory usage issues for large numbers of incoming records. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11337) PutDatabaseRecord docs don't reflect actual behavior and Max Batch Size shouldn't be a dependent property
Matt Burgess created NIFI-11337: --- Summary: PutDatabaseRecord docs don't reflect actual behavior and Max Batch Size shouldn't be a dependent property Key: NIFI-11337 URL: https://issues.apache.org/jira/browse/NIFI-11337 Project: Apache NiFi Issue Type: Bug Components: Extensions Reporter: Matt Burgess The current documentation for PutDatabaseRecord's Maximum Batch Size property says: "Specifies maximum batch size for INSERT and UPDATE statements. This parameter has no effect for other statements specified in 'Statement Type'. Zero means the batch size is not limited." This is not accurate as the Maximum Batch Size is used for all batches regardless of statement type. This also makes this property not dependent so that should be removed as well as the documentation updated to reflect the the behavior. The default value of Maximum Batch size should also be changed from 0 to something like 1000 to avoid possible memory usage issues for large numbers of incoming records. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-11337) PutDatabaseRecord docs don't reflect actual behavior and Max Batch Size shouldn't be a dependent property
[ https://issues.apache.org/jira/browse/NIFI-11337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess reassigned NIFI-11337: --- Assignee: Matt Burgess > PutDatabaseRecord docs don't reflect actual behavior and Max Batch Size > shouldn't be a dependent property > - > > Key: NIFI-11337 > URL: https://issues.apache.org/jira/browse/NIFI-11337 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > > The current documentation for PutDatabaseRecord's Maximum Batch Size property > says: "Specifies maximum batch size for INSERT and UPDATE statements. This > parameter has no effect for other statements specified in 'Statement Type'. > Zero means the batch size is not limited." This is not accurate as the > Maximum Batch Size is used for all batches regardless of statement type. This > also makes this property not dependent so that should be removed as well as > the documentation updated to reflect the the behavior. The default value of > Maximum Batch size should also be changed from 0 to something like 1000 to > avoid possible memory usage issues for large numbers of incoming records. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (NIFI-11305) CaptureChangeMySQL does not stop if the queue is not empty
[ https://issues.apache.org/jira/browse/NIFI-11305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess reopened NIFI-11305: - Reopening as there is an order-of-events bug where the binlog client waits on the event listener thread which no longer has anything pulling events off the queue. The event listener should be unregistered before the binlog client disconnects. > CaptureChangeMySQL does not stop if the queue is not empty > -- > > Key: NIFI-11305 > URL: https://issues.apache.org/jira/browse/NIFI-11305 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Fix For: 2.0.0, 1.21.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > There is a logic bug in the handling of stopping the CaptureChangeMySQL > processor. This causes it to not stop while there are events in the queue. If > the processor isn't running fast enough to drain the queue, stopping the > processor will have no effect. > The logic was being handled in OnStopped, but that won't get called until the > onTrigger has finished. Instead the loop should be checking to see if the > processor is still scheduled using isScheduled(), and if not should break out > of the loop and finish the onTrigger processing, thereby allowing the > OnStopped logic to be executed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11310) When Processor terminated, any resources that modify the classpath are not reloaded
[
https://issues.apache.org/jira/browse/NIFI-11310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess updated NIFI-11310:
Fix Version/s: 2.0.0
1.21.0
(was: 1.latest)
(was: 2.latest)
Resolution: Fixed
Status: Resolved (was: Patch Available)
> When Processor terminated, any resources that modify the classpath are not
> reloaded
> ---
>
> Key: NIFI-11310
> URL: https://issues.apache.org/jira/browse/NIFI-11310
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 2.0.0, 1.21.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Processors often provide Property Descriptors that can be used to modify the
> classpath. For example, processors may expose a property for specifying .jar
> files to load, such as JDBC drivers.
> When a Processor is terminated, though, and restarted, the Processor no
> longer has access to those additional classpath resources. The processor must
> have its properties modified in some way in order to re-establish those
> classpath resources. Otherwise, we see errors such as
> {{ClassNotFoundException}} being thrown
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-4651) PutSQL should return error messages and error codes in an attribute
[ https://issues.apache.org/jira/browse/NIFI-4651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-4651: --- Fix Version/s: 2.0.0 1.21.0 (was: 1.latest) (was: 2.latest) Resolution: Fixed Status: Resolved (was: Patch Available) > PutSQL should return error messages and error codes in an attribute > --- > > Key: NIFI-4651 > URL: https://issues.apache.org/jira/browse/NIFI-4651 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.20.0 >Reporter: Kay-Uwe Moosheimer >Assignee: Zsihovszki Krisztina >Priority: Minor > Fix For: 2.0.0, 1.21.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > In case of an error PutSQL should return the error messages and the error > code in attributes to be able to edit each FlowFile manually or automatically > according to the error. > A selection "Return errors" -> "Yes/No" and the possibility to specify two > attribute names (one for error message and one for error code) would be > helpful for postprocessing. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11305) CaptureChangeMySQL does not stop if the queue is not empty
[ https://issues.apache.org/jira/browse/NIFI-11305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-11305: Status: Patch Available (was: In Progress) > CaptureChangeMySQL does not stop if the queue is not empty > -- > > Key: NIFI-11305 > URL: https://issues.apache.org/jira/browse/NIFI-11305 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Fix For: 1.latest, 2.latest > > Time Spent: 10m > Remaining Estimate: 0h > > There is a logic bug in the handling of stopping the CaptureChangeMySQL > processor. This causes it to not stop while there are events in the queue. If > the processor isn't running fast enough to drain the queue, stopping the > processor will have no effect. > The logic was being handled in OnStopped, but that won't get called until the > onTrigger has finished. Instead the loop should be checking to see if the > processor is still scheduled using isScheduled(), and if not should break out > of the loop and finish the onTrigger processing, thereby allowing the > OnStopped logic to be executed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11305) CaptureChangeMySQL does not stop if the queue is not empty
[ https://issues.apache.org/jira/browse/NIFI-11305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-11305: Fix Version/s: 1.latest 2.latest > CaptureChangeMySQL does not stop if the queue is not empty > -- > > Key: NIFI-11305 > URL: https://issues.apache.org/jira/browse/NIFI-11305 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Fix For: 1.latest, 2.latest > > > There is a logic bug in the handling of stopping the CaptureChangeMySQL > processor. This causes it to not stop while there are events in the queue. If > the processor isn't running fast enough to drain the queue, stopping the > processor will have no effect. > The logic was being handled in OnStopped, but that won't get called until the > onTrigger has finished. Instead the loop should be checking to see if the > processor is still scheduled using isScheduled(), and if not should break out > of the loop and finish the onTrigger processing, thereby allowing the > OnStopped logic to be executed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11305) CaptureChangeMySQL does not stop if the queue is not empty
Matt Burgess created NIFI-11305: --- Summary: CaptureChangeMySQL does not stop if the queue is not empty Key: NIFI-11305 URL: https://issues.apache.org/jira/browse/NIFI-11305 Project: Apache NiFi Issue Type: Bug Components: Extensions Reporter: Matt Burgess There is a logic bug in the handling of stopping the CaptureChangeMySQL processor. This causes it to not stop while there are events in the queue. If the processor isn't running fast enough to drain the queue, stopping the processor will have no effect. The logic was being handled in OnStopped, but that won't get called until the onTrigger has finished. Instead the loop should be checking to see if the processor is still scheduled using isScheduled(), and if not should break out of the loop and finish the onTrigger processing, thereby allowing the OnStopped logic to be executed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-11305) CaptureChangeMySQL does not stop if the queue is not empty
[ https://issues.apache.org/jira/browse/NIFI-11305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess reassigned NIFI-11305: --- Assignee: Matt Burgess > CaptureChangeMySQL does not stop if the queue is not empty > -- > > Key: NIFI-11305 > URL: https://issues.apache.org/jira/browse/NIFI-11305 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > > There is a logic bug in the handling of stopping the CaptureChangeMySQL > processor. This causes it to not stop while there are events in the queue. If > the processor isn't running fast enough to drain the queue, stopping the > processor will have no effect. > The logic was being handled in OnStopped, but that won't get called until the > onTrigger has finished. Instead the loop should be checking to see if the > processor is still scheduled using isScheduled(), and if not should break out > of the loop and finish the onTrigger processing, thereby allowing the > OnStopped logic to be executed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11285) Unable to build module created by maven archetype plugin
[ https://issues.apache.org/jira/browse/NIFI-11285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-11285: Fix Version/s: 2.0.0 1.21.0 (was: 1.latest) (was: 2.latest) Resolution: Fixed Status: Resolved (was: Patch Available) > Unable to build module created by maven archetype plugin > > > Key: NIFI-11285 > URL: https://issues.apache.org/jira/browse/NIFI-11285 > Project: Apache NiFi > Issue Type: Bug >Reporter: Nandor Soma Abonyi >Assignee: Nandor Soma Abonyi >Priority: Major > Fix For: 2.0.0, 1.21.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Modules created by the archetype plugin use dependencies that are not > imported. Also, they have check-style issues. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11279) CaptureChangeMySQL should not stop processing if event stream is out of sync
[ https://issues.apache.org/jira/browse/NIFI-11279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-11279: Status: Patch Available (was: In Progress) > CaptureChangeMySQL should not stop processing if event stream is out of sync > > > Key: NIFI-11279 > URL: https://issues.apache.org/jira/browse/NIFI-11279 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Currently if CaptureChangeMySQL gets a BEGIN event while already in a > transaction or a COMMIT event without being in a transaction, processing of > the event queue stalls and no data will be emitted. Instead a warning should > be issued to the user that the event stream may be out of sync, but should > continue processing binlog events. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11279) CaptureChangeMySQL should not stop processing if event stream is out of sync
[ https://issues.apache.org/jira/browse/NIFI-11279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-11279: Fix Version/s: 1.latest 2.latest > CaptureChangeMySQL should not stop processing if event stream is out of sync > > > Key: NIFI-11279 > URL: https://issues.apache.org/jira/browse/NIFI-11279 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Fix For: 1.latest, 2.latest > > Time Spent: 10m > Remaining Estimate: 0h > > Currently if CaptureChangeMySQL gets a BEGIN event while already in a > transaction or a COMMIT event without being in a transaction, processing of > the event queue stalls and no data will be emitted. Instead a warning should > be issued to the user that the event stream may be out of sync, but should > continue processing binlog events. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-11279) CaptureChangeMySQL should not stop processing if event stream is out of sync
[ https://issues.apache.org/jira/browse/NIFI-11279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess reassigned NIFI-11279: --- Assignee: Matt Burgess > CaptureChangeMySQL should not stop processing if event stream is out of sync > > > Key: NIFI-11279 > URL: https://issues.apache.org/jira/browse/NIFI-11279 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > > Currently if CaptureChangeMySQL gets a BEGIN event while already in a > transaction or a COMMIT event without being in a transaction, processing of > the event queue stalls and no data will be emitted. Instead a warning should > be issued to the user that the event stream may be out of sync, but should > continue processing binlog events. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11279) CaptureChangeMySQL should not stop processing if event stream is out of sync
Matt Burgess created NIFI-11279: --- Summary: CaptureChangeMySQL should not stop processing if event stream is out of sync Key: NIFI-11279 URL: https://issues.apache.org/jira/browse/NIFI-11279 Project: Apache NiFi Issue Type: Bug Components: Extensions Reporter: Matt Burgess Currently if CaptureChangeMySQL gets a BEGIN event while already in a transaction or a COMMIT event without being in a transaction, processing of the event queue stalls and no data will be emitted. Instead a warning should be issued to the user that the event stream may be out of sync, but should continue processing binlog events. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-5501) MySQL connection cleanup thread leak
[
https://issues.apache.org/jira/browse/NIFI-5501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess updated NIFI-5501:
---
Fix Version/s: 1.21.0
> MySQL connection cleanup thread leak
>
>
> Key: NIFI-5501
> URL: https://issues.apache.org/jira/browse/NIFI-5501
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Environment: - Ubuntu 18.04 LTS
> - mysql-connector-java-5.1.46
> - Java 1.8.0_171
>Reporter: Tanapol Nearunchorn
>Assignee: Matt Burgess
>Priority: Major
> Fix For: 2.0.0, 1.21.0
>
> Attachments: nifi-threaddump.txt
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> There are thousands of "Abandoned connection cleanup thread" left running
> that cause memory leak in NiFi.
> I got a thread dump example here (full thread dump also attached):
> {code:java}
> "Abandoned connection cleanup thread" #18371 daemon prio=5 os_prio=0
> tid=0x7f3b840e7800 nid=0x76a3 in Object.wait() [0x7f3b24ebb000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x000348a12628> (a java.lang.ref.ReferenceQueue$Lock)
> at
> com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:64)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - <0x000348a12648> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> {code}
> I'm not sure where these threads come from which component because in my
> flow, I used CaptureChangeMySQL processor and DBCPConnectionPool controller
> service.
> As I also found related problem here: [mysql Bug
> #69526|https://bugs.mysql.com/bug.php?id=69526] but it quite a bit old.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-5501) MySQL connection cleanup thread leak
[
https://issues.apache.org/jira/browse/NIFI-5501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess updated NIFI-5501:
---
Affects Version/s: (was: 1.7.1)
Status: Patch Available (was: In Progress)
> MySQL connection cleanup thread leak
>
>
> Key: NIFI-5501
> URL: https://issues.apache.org/jira/browse/NIFI-5501
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Environment: - Ubuntu 18.04 LTS
> - mysql-connector-java-5.1.46
> - Java 1.8.0_171
>Reporter: Tanapol Nearunchorn
>Assignee: Matt Burgess
>Priority: Major
> Attachments: nifi-threaddump.txt
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> There are thousands of "Abandoned connection cleanup thread" left running
> that cause memory leak in NiFi.
> I got a thread dump example here (full thread dump also attached):
> {code:java}
> "Abandoned connection cleanup thread" #18371 daemon prio=5 os_prio=0
> tid=0x7f3b840e7800 nid=0x76a3 in Object.wait() [0x7f3b24ebb000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x000348a12628> (a java.lang.ref.ReferenceQueue$Lock)
> at
> com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:64)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - <0x000348a12648> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> {code}
> I'm not sure where these threads come from which component because in my
> flow, I used CaptureChangeMySQL processor and DBCPConnectionPool controller
> service.
> As I also found related problem here: [mysql Bug
> #69526|https://bugs.mysql.com/bug.php?id=69526] but it quite a bit old.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Reopened] (NIFI-5501) MySQL connection cleanup thread leak
[
https://issues.apache.org/jira/browse/NIFI-5501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess reopened NIFI-5501:
Assignee: Matt Burgess
Reopening as this processor should handle the driver and connections more
cleanly to avoid creating multiple threads
> MySQL connection cleanup thread leak
>
>
> Key: NIFI-5501
> URL: https://issues.apache.org/jira/browse/NIFI-5501
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.7.1
> Environment: - Ubuntu 18.04 LTS
> - mysql-connector-java-5.1.46
> - Java 1.8.0_171
>Reporter: Tanapol Nearunchorn
>Assignee: Matt Burgess
>Priority: Major
> Attachments: nifi-threaddump.txt
>
>
> There are thousands of "Abandoned connection cleanup thread" left running
> that cause memory leak in NiFi.
> I got a thread dump example here (full thread dump also attached):
> {code:java}
> "Abandoned connection cleanup thread" #18371 daemon prio=5 os_prio=0
> tid=0x7f3b840e7800 nid=0x76a3 in Object.wait() [0x7f3b24ebb000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x000348a12628> (a java.lang.ref.ReferenceQueue$Lock)
> at
> com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:64)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - <0x000348a12648> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> {code}
> I'm not sure where these threads come from which component because in my
> flow, I used CaptureChangeMySQL processor and DBCPConnectionPool controller
> service.
> As I also found related problem here: [mysql Bug
> #69526|https://bugs.mysql.com/bug.php?id=69526] but it quite a bit old.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-11253) Remove H2 Database Migrator
[ https://issues.apache.org/jira/browse/NIFI-11253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-11253: Resolution: Fixed Status: Resolved (was: Patch Available) > Remove H2 Database Migrator > --- > > Key: NIFI-11253 > URL: https://issues.apache.org/jira/browse/NIFI-11253 > Project: Apache NiFi > Issue Type: Task > Components: Core Framework >Reporter: David Handermann >Assignee: David Handermann >Priority: Major > Fix For: 2.0.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > NiFi 1.16.0 and following included a shaded version of the H2 1.4 library to > support automated migration from H2 1.4 to 2.1. All installations upgrading > from NiFi 1.16.0 or later to a newer version do not need the migration. The > custom migration should be removed for NiFi 2.0.0 since H2 2.1 is the > baseline version. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-11147) Allow QuerySalesforceObject to query all existing fields
[ https://issues.apache.org/jira/browse/NIFI-11147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess resolved NIFI-11147. - Fix Version/s: 2.0.0 Resolution: Fixed > Allow QuerySalesforceObject to query all existing fields > > > Key: NIFI-11147 > URL: https://issues.apache.org/jira/browse/NIFI-11147 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Matt Burgess >Assignee: Lehel Boér >Priority: Major > Fix For: 2.0.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Currently the Field Names property of QuerySalesforceObject is required and > must contain the names of the fields the user wants to return. However in a > schema drift use case, the user may want to add a field to a Salesforce > object and have the NiFi flow continue without needing alteration. > This Jira is to make it possible for QuerySalesforceObject to return all > fields from an object. A suggestion is to make Field Names optional and if it > is not set, all fields are queried. The documentation should be updated to > match the behavior. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11209) UpdateHive3Table and UpdateDatabaseTable erase newly created columns in output FlowFile
[ https://issues.apache.org/jira/browse/NIFI-11209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-11209: Status: Patch Available (was: In Progress) > UpdateHive3Table and UpdateDatabaseTable erase newly created columns in > output FlowFile > --- > > Key: NIFI-11209 > URL: https://issues.apache.org/jira/browse/NIFI-11209 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > When using UpdateHive3Table or PutDatabaseTable with the Update Field Names > property set to "true", the output will include only those fields that > already had existing columns in the target table. This means if an incoming > FlowFile has a "new" field, the column will be created and populated in the > target database, but the output FlowFile will no longer have the "new" fields > in the outgoing records. > This is because the original database columns are used to populate the > outgoing record schema when the Update Field Names property is set to true. > Instead the database columns should be refreshed/maintained after any DDL is > executed in order to get the updated set of database column names. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11209) UpdateHive3Table and UpdateDatabaseTable erase newly created columns in output FlowFile
[ https://issues.apache.org/jira/browse/NIFI-11209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-11209: Description: When using UpdateHive3Table or PutDatabaseTable with the Update Field Names property set to "true", the output will include only those fields that already had existing columns in the target table. This means if an incoming FlowFile has a "new" field, the column will be created and populated in the target database, but the output FlowFile will no longer have the "new" fields in the outgoing records. This is because the original database columns are used to populate the outgoing record schema when the Update Field Names property is set to true. Instead the database columns should be refreshed/maintained after any DDL is executed in order to get the updated set of database column names. was: When using UpdateHive3Table or PutDatabaseTable with the Update Field Names property set to "true", the output will include only those fields that already had existing columns in the target table. This means if an incoming FlowFile has a "new" field, the column will be created and populated in the target database, but the output FlowFile will no longer have the "new" fields in the outgoing records. This is because the original database columns are used to populate the outgoing record schema when the Update Field Names property is set to true. Instead the database columns should be refreshed after any DDL is executed in order to get the updated set of database column names. > UpdateHive3Table and UpdateDatabaseTable erase newly created columns in > output FlowFile > --- > > Key: NIFI-11209 > URL: https://issues.apache.org/jira/browse/NIFI-11209 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > > When using UpdateHive3Table or PutDatabaseTable with the Update Field Names > property set to "true", the output will include only those fields that > already had existing columns in the target table. This means if an incoming > FlowFile has a "new" field, the column will be created and populated in the > target database, but the output FlowFile will no longer have the "new" fields > in the outgoing records. > This is because the original database columns are used to populate the > outgoing record schema when the Update Field Names property is set to true. > Instead the database columns should be refreshed/maintained after any DDL is > executed in order to get the updated set of database column names. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-11209) UpdateHive3Table and UpdateDatabaseTable erase newly created columns in output FlowFile
[ https://issues.apache.org/jira/browse/NIFI-11209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess reassigned NIFI-11209: --- Assignee: Matt Burgess > UpdateHive3Table and UpdateDatabaseTable erase newly created columns in > output FlowFile > --- > > Key: NIFI-11209 > URL: https://issues.apache.org/jira/browse/NIFI-11209 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > > When using UpdateHive3Table or PutDatabaseTable with the Update Field Names > property set to "true", the output will include only those fields that > already had existing columns in the target table. This means if an incoming > FlowFile has a "new" field, the column will be created and populated in the > target database, but the output FlowFile will no longer have the "new" fields > in the outgoing records. > This is because the original database columns are used to populate the > outgoing record schema when the Update Field Names property is set to true. > Instead the database columns should be refreshed after any DDL is executed in > order to get the updated set of database column names. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11192) If Port moved from parent to child group or vice versa between flow versions, version change can leave nifi in bad state
[
https://issues.apache.org/jira/browse/NIFI-11192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess updated NIFI-11192:
Resolution: Fixed
Status: Resolved (was: Patch Available)
> If Port moved from parent to child group or vice versa between flow versions,
> version change can leave nifi in bad state
>
>
> Key: NIFI-11192
> URL: https://issues.apache.org/jira/browse/NIFI-11192
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Blocker
> Fix For: 2.0.0, 1.21.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> To reproduce:
> Create a Process Group, Parent.
> Inside of Parent, create:
> * A processor, for example, ReplaceText
> * A Process Group, Child.
> Inside of Child, create:
> * An Input Port, In
> * A Processor, for example, UpdateAttribute
> * A connection between the two components
> Then connect ReplaceText to the Input Port, In.
> Save the Parent PG flow as Version 1 of a flow.
> Now, create a new Processor, say RouteOnAttribute, within the Parent PG.
> Move the destination of the connection from the Input Port to
> RouteOnAttribute.
> Step into the Child PG. Select all components, right-click, and choose "Move
> to Parent"
> Save Parent PG as Version 2 of the flow.
> Now, attempt to Change Version on the Parent Group. Change the version to
> Version 1.
> The version change will fail with an error: "Failed to perform update flow
> request due to 42fb2904-c774-359b-5368-2e48b60ac02d is the destination of
> another component" and the logs will have a stack trace:
> {code:java}
> 2023-02-16 15:18:28,830 ERROR [Process Group Update Thread-1]
> o.apache.nifi.web.api.FlowUpdateResource Failed to perform update flow request
> java.lang.IllegalStateException: 42fb2904-c774-359b-5368-2e48b60ac02d is the
> destination of another component
> at
> org.apache.nifi.controller.AbstractPort.verifyCanDelete(AbstractPort.java:562)
> at
> org.apache.nifi.controller.AbstractPort.verifyCanDelete(AbstractPort.java:542)
> at
> org.apache.nifi.groups.StandardProcessGroup.removeInputPort(StandardProcessGroup.java:637)
> at
> org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.removeMissingComponents(StandardVersionedComponentSynchronizer.java:948)
> at
> org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.removeMissingInputPorts(StandardVersionedComponentSynchronizer.java:873)
> at
> org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.synchronize(StandardVersionedComponentSynchronizer.java:410)
> at
> org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.lambda$synchronize$0(StandardVersionedComponentSynchronizer.java:260)
> at
> org.apache.nifi.controller.flow.AbstractFlowManager.withParameterContextResolution(AbstractFlowManager.java:556)
> at
> org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.synchronize(StandardVersionedComponentSynchronizer.java:255)
> at
> org.apache.nifi.groups.StandardProcessGroup.synchronizeFlow(StandardProcessGroup.java:3972)
> at
> org.apache.nifi.groups.StandardProcessGroup.updateFlow(StandardProcessGroup.java:3952)
> at
> org.apache.nifi.web.dao.impl.StandardProcessGroupDAO.updateProcessGroupFlow(StandardProcessGroupDAO.java:435)
> at
> org.apache.nifi.web.dao.impl.StandardProcessGroupDAO$$FastClassBySpringCGLIB$$10a99b47.invoke()
> at
> org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
> at
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> at
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
> at
> org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89)
> at
> org.apache.nifi.audit.ProcessGroupAuditor.updateProcessGroupFlowAdvice(ProcessGroupAuditor.java:308)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java
[jira] [Updated] (NIFI-11192) If Port moved from parent to child group or vice versa between flow versions, version change can leave nifi in bad state
[
https://issues.apache.org/jira/browse/NIFI-11192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess updated NIFI-11192:
Status: Patch Available (was: Open)
> If Port moved from parent to child group or vice versa between flow versions,
> version change can leave nifi in bad state
>
>
> Key: NIFI-11192
> URL: https://issues.apache.org/jira/browse/NIFI-11192
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Blocker
> Fix For: 2.0.0, 1.21.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> To reproduce:
> Create a Process Group, Parent.
> Inside of Parent, create:
> * A processor, for example, ReplaceText
> * A Process Group, Child.
> Inside of Child, create:
> * An Input Port, In
> * A Processor, for example, UpdateAttribute
> * A connection between the two components
> Then connect ReplaceText to the Input Port, In.
> Save the Parent PG flow as Version 1 of a flow.
> Now, create a new Processor, say RouteOnAttribute, within the Parent PG.
> Move the destination of the connection from the Input Port to
> RouteOnAttribute.
> Step into the Child PG. Select all components, right-click, and choose "Move
> to Parent"
> Save Parent PG as Version 2 of the flow.
> Now, attempt to Change Version on the Parent Group. Change the version to
> Version 1.
> The version change will fail with an error: "Failed to perform update flow
> request due to 42fb2904-c774-359b-5368-2e48b60ac02d is the destination of
> another component" and the logs will have a stack trace:
> {code:java}
> 2023-02-16 15:18:28,830 ERROR [Process Group Update Thread-1]
> o.apache.nifi.web.api.FlowUpdateResource Failed to perform update flow request
> java.lang.IllegalStateException: 42fb2904-c774-359b-5368-2e48b60ac02d is the
> destination of another component
> at
> org.apache.nifi.controller.AbstractPort.verifyCanDelete(AbstractPort.java:562)
> at
> org.apache.nifi.controller.AbstractPort.verifyCanDelete(AbstractPort.java:542)
> at
> org.apache.nifi.groups.StandardProcessGroup.removeInputPort(StandardProcessGroup.java:637)
> at
> org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.removeMissingComponents(StandardVersionedComponentSynchronizer.java:948)
> at
> org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.removeMissingInputPorts(StandardVersionedComponentSynchronizer.java:873)
> at
> org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.synchronize(StandardVersionedComponentSynchronizer.java:410)
> at
> org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.lambda$synchronize$0(StandardVersionedComponentSynchronizer.java:260)
> at
> org.apache.nifi.controller.flow.AbstractFlowManager.withParameterContextResolution(AbstractFlowManager.java:556)
> at
> org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.synchronize(StandardVersionedComponentSynchronizer.java:255)
> at
> org.apache.nifi.groups.StandardProcessGroup.synchronizeFlow(StandardProcessGroup.java:3972)
> at
> org.apache.nifi.groups.StandardProcessGroup.updateFlow(StandardProcessGroup.java:3952)
> at
> org.apache.nifi.web.dao.impl.StandardProcessGroupDAO.updateProcessGroupFlow(StandardProcessGroupDAO.java:435)
> at
> org.apache.nifi.web.dao.impl.StandardProcessGroupDAO$$FastClassBySpringCGLIB$$10a99b47.invoke()
> at
> org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
> at
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> at
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
> at
> org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89)
> at
> org.apache.nifi.audit.ProcessGroupAuditor.updateProcessGroupFlowAdvice(ProcessGroupAuditor.java:308)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:634)
> at
> org.springfr
[jira] [Assigned] (NIFI-11192) If Port moved from parent to child group or vice versa between flow versions, version change can leave nifi in bad state
[
https://issues.apache.org/jira/browse/NIFI-11192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess reassigned NIFI-11192:
---
Assignee: Mark Payne
> If Port moved from parent to child group or vice versa between flow versions,
> version change can leave nifi in bad state
>
>
> Key: NIFI-11192
> URL: https://issues.apache.org/jira/browse/NIFI-11192
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Blocker
> Fix For: 2.0.0, 1.21.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> To reproduce:
> Create a Process Group, Parent.
> Inside of Parent, create:
> * A processor, for example, ReplaceText
> * A Process Group, Child.
> Inside of Child, create:
> * An Input Port, In
> * A Processor, for example, UpdateAttribute
> * A connection between the two components
> Then connect ReplaceText to the Input Port, In.
> Save the Parent PG flow as Version 1 of a flow.
> Now, create a new Processor, say RouteOnAttribute, within the Parent PG.
> Move the destination of the connection from the Input Port to
> RouteOnAttribute.
> Step into the Child PG. Select all components, right-click, and choose "Move
> to Parent"
> Save Parent PG as Version 2 of the flow.
> Now, attempt to Change Version on the Parent Group. Change the version to
> Version 1.
> The version change will fail with an error: "Failed to perform update flow
> request due to 42fb2904-c774-359b-5368-2e48b60ac02d is the destination of
> another component" and the logs will have a stack trace:
> {code:java}
> 2023-02-16 15:18:28,830 ERROR [Process Group Update Thread-1]
> o.apache.nifi.web.api.FlowUpdateResource Failed to perform update flow request
> java.lang.IllegalStateException: 42fb2904-c774-359b-5368-2e48b60ac02d is the
> destination of another component
> at
> org.apache.nifi.controller.AbstractPort.verifyCanDelete(AbstractPort.java:562)
> at
> org.apache.nifi.controller.AbstractPort.verifyCanDelete(AbstractPort.java:542)
> at
> org.apache.nifi.groups.StandardProcessGroup.removeInputPort(StandardProcessGroup.java:637)
> at
> org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.removeMissingComponents(StandardVersionedComponentSynchronizer.java:948)
> at
> org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.removeMissingInputPorts(StandardVersionedComponentSynchronizer.java:873)
> at
> org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.synchronize(StandardVersionedComponentSynchronizer.java:410)
> at
> org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.lambda$synchronize$0(StandardVersionedComponentSynchronizer.java:260)
> at
> org.apache.nifi.controller.flow.AbstractFlowManager.withParameterContextResolution(AbstractFlowManager.java:556)
> at
> org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.synchronize(StandardVersionedComponentSynchronizer.java:255)
> at
> org.apache.nifi.groups.StandardProcessGroup.synchronizeFlow(StandardProcessGroup.java:3972)
> at
> org.apache.nifi.groups.StandardProcessGroup.updateFlow(StandardProcessGroup.java:3952)
> at
> org.apache.nifi.web.dao.impl.StandardProcessGroupDAO.updateProcessGroupFlow(StandardProcessGroupDAO.java:435)
> at
> org.apache.nifi.web.dao.impl.StandardProcessGroupDAO$$FastClassBySpringCGLIB$$10a99b47.invoke()
> at
> org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
> at
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> at
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
> at
> org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89)
> at
> org.apache.nifi.audit.ProcessGroupAuditor.updateProcessGroupFlowAdvice(ProcessGroupAuditor.java:308)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:634)
> at
> org.springframework.a
[jira] [Updated] (NIFI-11187) Remove ActiveMQ from Standard Processors
[ https://issues.apache.org/jira/browse/NIFI-11187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-11187: Fix Version/s: 2.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Remove ActiveMQ from Standard Processors > > > Key: NIFI-11187 > URL: https://issues.apache.org/jira/browse/NIFI-11187 > Project: Apache NiFi > Issue Type: Task > Components: Extensions >Reporter: David Handermann >Assignee: David Handermann >Priority: Major > Fix For: 2.0.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > With the removal of GetJMS and PutJMS Processors from the standard processors > module, the ActiveMQ dependency and related classes should be removed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11208) Remove Hortonworks Schema Registry
[
https://issues.apache.org/jira/browse/NIFI-11208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess updated NIFI-11208:
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Remove Hortonworks Schema Registry
> --
>
> Key: NIFI-11208
> URL: https://issues.apache.org/jira/browse/NIFI-11208
> Project: Apache NiFi
> Issue Type: Task
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
> Fix For: 2.0.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> The {{HortonworksSchemaRegistry}} was deprecated for removal in NiFi 1.20.0
> and should be removed along with the {{nifi-hwx-schema-registry-bundle}}
> modules.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (NIFI-11209) UpdateHive3Table and UpdateDatabaseTable erase newly created columns in output FlowFile
Matt Burgess created NIFI-11209: --- Summary: UpdateHive3Table and UpdateDatabaseTable erase newly created columns in output FlowFile Key: NIFI-11209 URL: https://issues.apache.org/jira/browse/NIFI-11209 Project: Apache NiFi Issue Type: Bug Components: Extensions Reporter: Matt Burgess When using UpdateHive3Table or PutDatabaseTable with the Update Field Names property set to "true", the output will include only those fields that already had existing columns in the target table. This means if an incoming FlowFile has a "new" field, the column will be created and populated in the target database, but the output FlowFile will no longer have the "new" fields in the outgoing records. This is because the original database columns are used to populate the outgoing record schema when the Update Field Names property is set to true. Instead the database columns should be refreshed after any DDL is executed in order to get the updated set of database column names. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11149) Add PutRedisHashRecord processor
[ https://issues.apache.org/jira/browse/NIFI-11149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-11149: Summary: Add PutRedisHashRecord processor (was: Add PutRedisRecord processor) > Add PutRedisHashRecord processor > > > Key: NIFI-11149 > URL: https://issues.apache.org/jira/browse/NIFI-11149 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > This case is to add a record enabled processor to send data into Redis. The > hash should be chosen such that the field values can be retrieved later. It > should use the existing RedisConnectionPool controller service -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11149) Add PutRedisRecord processor
[ https://issues.apache.org/jira/browse/NIFI-11149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-11149: Status: Patch Available (was: In Progress) > Add PutRedisRecord processor > > > Key: NIFI-11149 > URL: https://issues.apache.org/jira/browse/NIFI-11149 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > This case is to add a record enabled processor to send data into Redis. The > hash should be chosen such that the field values can be retrieved later. It > should use the existing RedisConnectionPool controller service -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-11149) Add PutRedisRecord processor
[ https://issues.apache.org/jira/browse/NIFI-11149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess reassigned NIFI-11149: --- Assignee: Matt Burgess > Add PutRedisRecord processor > > > Key: NIFI-11149 > URL: https://issues.apache.org/jira/browse/NIFI-11149 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > > This case is to add a record enabled processor to send data into Redis. The > hash should be chosen such that the field values can be retrieved later. It > should use the existing RedisConnectionPool controller service -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11149) Add PutRedisRecord processor
Matt Burgess created NIFI-11149: --- Summary: Add PutRedisRecord processor Key: NIFI-11149 URL: https://issues.apache.org/jira/browse/NIFI-11149 Project: Apache NiFi Issue Type: New Feature Components: Extensions Reporter: Matt Burgess This case is to add a record enabled processor to send data into Redis. The hash should be chosen such that the field values can be retrieved later. It should use the existing RedisConnectionPool controller service -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11148) Race condition loading script engines in InvokeScriptedProcessor
Matt Burgess created NIFI-11148: --- Summary: Race condition loading script engines in InvokeScriptedProcessor Key: NIFI-11148 URL: https://issues.apache.org/jira/browse/NIFI-11148 Project: Apache NiFi Issue Type: Bug Reporter: Matt Burgess In the 1.20.0 release candidate I noticed this error when loading a flow with an InvokeScriptedProcessor: 2023-02-07 16:26:39,825 ERROR [NiFi Web Server-32] o.a.n.p.script.InvokeScriptedProcessor InvokeScriptedProcessor[id=2bbc053d-8b08-3206-7c47-cc00a08beb64] Error adding script engine Groovy 2023-02-07 16:26:39,826 ERROR [NiFi Web Server-32] o.a.n.p.script.InvokeScriptedProcessor InvokeScriptedProcessor[id=2bbc053d-8b08-3206-7c47-cc00a08beb64] Unable to load script: No script runner available org.apache.nifi.processor.exception.ProcessException: No script runner available at org.apache.nifi.processors.script.InvokeScriptedProcessor.reloadScript(InvokeScriptedProcessor.java:371) at org.apache.nifi.processors.script.InvokeScriptedProcessor.reloadScriptBody(InvokeScriptedProcessor.java:326) at org.apache.nifi.processors.script.InvokeScriptedProcessor.setup(InvokeScriptedProcessor.java:230) at org.apache.nifi.processors.script.InvokeScriptedProcessor.onConfigurationRestored(InvokeScriptedProcessor.java:222) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:145) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:133) at org.apache.nifi.util.ReflectionUtils.quietlyInvokeMethodsWithAnnotations(ReflectionUtils.java:316) at org.apache.nifi.util.ReflectionUtils.quietlyInvokeMethodsWithAnnotation(ReflectionUtils.java:93) at org.apache.nifi.controller.StandardProcessorNode.onConfigurationRestored(StandardProcessorNode.java:2115) at org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.addProcessor(StandardVersionedComponentSynchronizer.java:2417) at org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.synchronizeProcessors(StandardVersionedComponentSynchronizer.java:932) at org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.synchronize(StandardVersionedComponentSynchronizer.java:422) at org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.lambda$synchronize$0(StandardVersionedComponentSynchronizer.java:260) at org.apache.nifi.controller.flow.AbstractFlowManager.withParameterContextResolution(AbstractFlowManager.java:550) at org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.synchronize(StandardVersionedComponentSynchronizer.java:255) at org.apache.nifi.groups.StandardProcessGroup.synchronizeFlow(StandardProcessGroup.java:3972) at org.apache.nifi.groups.StandardProcessGroup.updateFlow(StandardProcessGroup.java:3952) at org.apache.nifi.web.dao.impl.StandardProcessGroupDAO.updateProcessGroupFlow(StandardProcessGroupDAO.java:435) at org.apache.nifi.web.dao.impl.StandardProcessGroupDAO$$FastClassBySpringCGLIB$$10a99b47.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89) at org.apache.nifi.audit.ProcessGroupAuditor.updateProcessGroupFlowAdvice(ProcessGroupAuditor.java:308) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:634) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:624)
[jira] [Assigned] (NIFI-11147) Allow QuerySalesforceObject to query all existing fields
[ https://issues.apache.org/jira/browse/NIFI-11147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess reassigned NIFI-11147: --- Assignee: Matt Burgess > Allow QuerySalesforceObject to query all existing fields > > > Key: NIFI-11147 > URL: https://issues.apache.org/jira/browse/NIFI-11147 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > > Currently the Field Names property of QuerySalesforceObject is required and > must contain the names of the fields the user wants to return. However in a > schema drift use case, the user may want to add a field to a Salesforce > object and have the NiFi flow continue without needing alteration. > This Jira is to make it possible for QuerySalesforceObject to return all > fields from an object. A suggestion is to make Field Names optional and if it > is not set, all fields are queried. The documentation should be updated to > match the behavior. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-11147) Allow QuerySalesforceObject to query all existing fields
[ https://issues.apache.org/jira/browse/NIFI-11147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess reassigned NIFI-11147: --- Assignee: Lehel Boér (was: Matt Burgess) > Allow QuerySalesforceObject to query all existing fields > > > Key: NIFI-11147 > URL: https://issues.apache.org/jira/browse/NIFI-11147 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Matt Burgess >Assignee: Lehel Boér >Priority: Major > > Currently the Field Names property of QuerySalesforceObject is required and > must contain the names of the fields the user wants to return. However in a > schema drift use case, the user may want to add a field to a Salesforce > object and have the NiFi flow continue without needing alteration. > This Jira is to make it possible for QuerySalesforceObject to return all > fields from an object. A suggestion is to make Field Names optional and if it > is not set, all fields are queried. The documentation should be updated to > match the behavior. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11147) Allow QuerySalesforceObject to query all existing fields
Matt Burgess created NIFI-11147: --- Summary: Allow QuerySalesforceObject to query all existing fields Key: NIFI-11147 URL: https://issues.apache.org/jira/browse/NIFI-11147 Project: Apache NiFi Issue Type: Improvement Reporter: Matt Burgess Currently the Field Names property of QuerySalesforceObject is required and must contain the names of the fields the user wants to return. However in a schema drift use case, the user may want to add a field to a Salesforce object and have the NiFi flow continue without needing alteration. This Jira is to make it possible for QuerySalesforceObject to return all fields from an object. A suggestion is to make Field Names optional and if it is not set, all fields are queried. The documentation should be updated to match the behavior. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11110) Create processor for triggering HMS events
[ https://issues.apache.org/jira/browse/NIFI-0?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-0: Status: Patch Available (was: Open) > Create processor for triggering HMS events > -- > > Key: NIFI-0 > URL: https://issues.apache.org/jira/browse/NIFI-0 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Mark Bathori >Assignee: Mark Bathori >Priority: Major > Time Spent: 2h 10m > Remaining Estimate: 0h > > Create a processor which is capable to trigger HiveMetaStore actions and > generate notifications for them. The main goal is to be able to register file > and directory insertions and removals in the HiveMetaStore done by other > processors. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11110) Create processor for triggering HMS events
[ https://issues.apache.org/jira/browse/NIFI-0?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-0: Fix Version/s: 1.20.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Create processor for triggering HMS events > -- > > Key: NIFI-0 > URL: https://issues.apache.org/jira/browse/NIFI-0 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Mark Bathori >Assignee: Mark Bathori >Priority: Major > Fix For: 1.20.0 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > Create a processor which is capable to trigger HiveMetaStore actions and > generate notifications for them. The main goal is to be able to register file > and directory insertions and removals in the HiveMetaStore done by other > processors. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11094) Allow CaptureChangeMySQL to send multiple events per FlowFile
[ https://issues.apache.org/jira/browse/NIFI-11094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-11094: Status: Patch Available (was: In Progress) > Allow CaptureChangeMySQL to send multiple events per FlowFile > - > > Key: NIFI-11094 > URL: https://issues.apache.org/jira/browse/NIFI-11094 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > It would be nice if there could be a Events Per FlowFile Strategy property > for CaptureChangeMySQL that could allow for things like N events per FlowFile > or one full transaction per FlowFile. It can help lower overhead downstream > and increase the overall performance of the flow. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11047) Issue when upgrading version of dataflow with external service
[
https://issues.apache.org/jira/browse/NIFI-11047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess updated NIFI-11047:
Fix Version/s: 1.20.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Issue when upgrading version of dataflow with external service
> --
>
> Key: NIFI-11047
> URL: https://issues.apache.org/jira/browse/NIFI-11047
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.19.1
>Reporter: Julien G.
>Assignee: Paul Grey
>Priority: Major
> Fix For: 1.20.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> There is a bug/issue when upgrading a dataflow that was referencing an
> external service. In the dataflow, I'm referencing an external service (a
> service in higher scope, at the root).
> But if in a new version I stop referencing it, when upgrading the dataflow,
> the service will still be referenced. So the dataflow will be marked as
> modified.
> Scenario to reproduce (that can be reproduced in 1.16.3 and 1.19.1):
> # Create a {{*StandardProxyConfigurationService*}} at the root level
> # Create a process group
> # In the process group, create a {{*PutGCSObject*}} processor
> # In the processor attach the {{*StandardProxyConfigurationService*}}
> # Create a {{*GCPCredentialsControllerService*}} at the process group level
> # Attach the {{*GCPCredentialsControllerService*}} to the processor
> # Attach the {{*StandardProxyConfigurationService*}} to the
> {{*GCPCredentialsControllerService*}}
> # Version the dataflow
> # Remove the referenced {{*StandardProxyConfigurationService*}} in the
> processor and the service
> # Commit a new version
> # Rollback to version 1
> # Upgrade to version 2
> When in this state, you can't rollback the change. To unstuck the versionning
> you need to remove the referenced *{{StandardProxyConfigurationService}}* in
> the processor and the service manually to be up to date.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-10853) UpdateAttribute - expression language syntax error not caught until processor runtime
[
https://issues.apache.org/jira/browse/NIFI-10853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess updated NIFI-10853:
Fix Version/s: 1.20.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> UpdateAttribute - expression language syntax error not caught until processor
> runtime
> -
>
> Key: NIFI-10853
> URL: https://issues.apache.org/jira/browse/NIFI-10853
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Nissim Shiman
>Assignee: Nissim Shiman
>Priority: Major
> Fix For: 1.20.0
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> The UpdateAttribute will not show any alerts when the following expression
> language value is used for a property value:
> {code:java}
> ${filename:equals('testFile'}
> {code}
> This is also true when using the 'Verify Properties' button. (A green arrow
> is returned saying "Component Validation Passed").
> The processor will start, but when a flowfile passed through on runtime,
> though, it will fail
> with error
> {code}
> org.apache.nifi.attribute.expression.language.exception.AttributeExpressionLanguageException:
> Invalid Expression: ${filename:equals('testFile'} due to Unexpected token
> '}' at line 1, column 28. Query: ${filename:equals(testFile}
> {code}
> (This is not the case with RouteOnAttribute, which will leave the processor
> in an invalid state, flagging this as an alert, until the expression is
> corrected.)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (NIFI-11094) Allow CaptureChangeMySQL to send multiple events per FlowFile
[ https://issues.apache.org/jira/browse/NIFI-11094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess reassigned NIFI-11094: --- Assignee: Matt Burgess > Allow CaptureChangeMySQL to send multiple events per FlowFile > - > > Key: NIFI-11094 > URL: https://issues.apache.org/jira/browse/NIFI-11094 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > > It would be nice if there could be a Events Per FlowFile Strategy property > for CaptureChangeMySQL that could allow for things like N events per FlowFile > or one full transaction per FlowFile. It can help lower overhead downstream > and increase the overall performance of the flow. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11094) Allow CaptureChangeMySQL to send multiple events per FlowFile
Matt Burgess created NIFI-11094: --- Summary: Allow CaptureChangeMySQL to send multiple events per FlowFile Key: NIFI-11094 URL: https://issues.apache.org/jira/browse/NIFI-11094 Project: Apache NiFi Issue Type: Improvement Reporter: Matt Burgess It would be nice if there could be a Events Per FlowFile Strategy property for CaptureChangeMySQL that could allow for things like N events per FlowFile or one full transaction per FlowFile. It can help lower overhead downstream and increase the overall performance of the flow. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11079) Extend QueryNifiReportingTask and ProcessGroup Summary Table
[ https://issues.apache.org/jira/browse/NIFI-11079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-11079: Fix Version/s: 1.20.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Extend QueryNifiReportingTask and ProcessGroup Summary Table > > > Key: NIFI-11079 > URL: https://issues.apache.org/jira/browse/NIFI-11079 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Timea Barna >Assignee: Timea Barna >Priority: Major > Fix For: 1.20.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Add Total Task Duration and Total Active Thread Count to > QueryNiFiReportingTask and ProcessGroup Summary Table. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11075) Add additional fields to component definition
[ https://issues.apache.org/jira/browse/NIFI-11075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-11075: Fix Version/s: 1.20.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Add additional fields to component definition > - > > Key: NIFI-11075 > URL: https://issues.apache.org/jira/browse/NIFI-11075 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.19.1 >Reporter: Bryan Bende >Assignee: Bryan Bende >Priority: Minor > Fix For: 1.20.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > There are some fields available in the Extension definition that are not > provided in the C2 component definition. These fields need to be added... > * reads attributes > * writes attributes > * dynamic relationship description > * system resource considerations > * deprecation notice > * see also -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11033) Update QueryNiFiReportingTask additionalDetails page to include Flow Config History table
[ https://issues.apache.org/jira/browse/NIFI-11033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-11033: Status: Patch Available (was: In Progress) > Update QueryNiFiReportingTask additionalDetails page to include Flow Config > History table > - > > Key: NIFI-11033 > URL: https://issues.apache.org/jira/browse/NIFI-11033 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation & Website, Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > The additionalDetails.html page for QueryNiFiReportingTask should have been > updated under NIFI-10699 to include the new table and its schema. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-11033) Update QueryNiFiReportingTask additionalDetails page to include Flow Config History table
[ https://issues.apache.org/jira/browse/NIFI-11033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess reassigned NIFI-11033: --- Assignee: Matt Burgess > Update QueryNiFiReportingTask additionalDetails page to include Flow Config > History table > - > > Key: NIFI-11033 > URL: https://issues.apache.org/jira/browse/NIFI-11033 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation & Website, Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > > The additionalDetails.html page for QueryNiFiReportingTask should have been > updated under NIFI-10699 to include the new table and its schema. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11033) Update QueryNiFiReportingTask additionalDetails page to include Flow Config History table
[ https://issues.apache.org/jira/browse/NIFI-11033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-11033: Summary: Update QueryNiFiReportingTask additionalDetails page to include Flow Config History table (was: Update QueryRecord additionalDetails page to include Flow Config History table) > Update QueryNiFiReportingTask additionalDetails page to include Flow Config > History table > - > > Key: NIFI-11033 > URL: https://issues.apache.org/jira/browse/NIFI-11033 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation & Website, Extensions >Reporter: Matt Burgess >Priority: Major > > The additionalDetails.html page for QueryRecord should have been updated > under NIFI-10699 to include the new table and its schema. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11033) Update QueryNiFiReportingTask additionalDetails page to include Flow Config History table
[ https://issues.apache.org/jira/browse/NIFI-11033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-11033: Description: The additionalDetails.html page for QueryNiFiReportingTask should have been updated under NIFI-10699 to include the new table and its schema. (was: The additionalDetails.html page for QueryRecord should have been updated under NIFI-10699 to include the new table and its schema.) > Update QueryNiFiReportingTask additionalDetails page to include Flow Config > History table > - > > Key: NIFI-11033 > URL: https://issues.apache.org/jira/browse/NIFI-11033 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation & Website, Extensions >Reporter: Matt Burgess >Priority: Major > > The additionalDetails.html page for QueryNiFiReportingTask should have been > updated under NIFI-10699 to include the new table and its schema. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-1954) Add Additional Details page for ExecuteScript processor
[ https://issues.apache.org/jira/browse/NIFI-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-1954: --- Fix Version/s: 1.18.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Add Additional Details page for ExecuteScript processor > --- > > Key: NIFI-1954 > URL: https://issues.apache.org/jira/browse/NIFI-1954 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Fix For: 1.18.0 > > Time Spent: 2h 50m > Remaining Estimate: 0h > > There are some currently undocumented features of ExecuteScript, such as > variables being bound to the scripting context (for ExecuteScript). These and > other details should be provided to the user via the standard documentation, > such as an Additional Details page. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-10241) Add comments tooltip for Controller Services and Reporting Tasks
[ https://issues.apache.org/jira/browse/NIFI-10241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-10241: Fix Version/s: 1.20.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Add comments tooltip for Controller Services and Reporting Tasks > > > Key: NIFI-10241 > URL: https://issues.apache.org/jira/browse/NIFI-10241 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Nissim Shiman >Assignee: Nissim Shiman >Priority: Major > Fix For: 1.20.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Add tooltip enabled Comments icon for Controller Services and Reporting Tasks. > This will alert nifi administrators that the controller service/reporting > task has a special comment relating to it. > This icon should be near the Usage icon that is visible when the list of > available controller services/reporting tasks are displayed > (i.e. Global Menu -> Controller Settings -> Management Controller Services > and > Global Menu -> Controller Settings -> Reporting Tasks) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-10887) Improve Performance of ReplaceText processor
[
https://issues.apache.org/jira/browse/NIFI-10887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess updated NIFI-10887:
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Improve Performance of ReplaceText processor
>
>
> Key: NIFI-10887
> URL: https://issues.apache.org/jira/browse/NIFI-10887
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Labels: performance
> Fix For: 1.20.0
>
> Attachments: ReplaceText-LiteralReplace-0msRunDuration.png,
> ReplaceText-LiteralReplace-25msRunDuration.png,
> ReplaceText-LiteralReplace-AfterChanges.png,
> ReplaceText-LiteralReplace-BeforeChanges.png,
> ReplaceText-RegexReplace-AfterChanges.png,
> ReplaceText-RegexReplace-BeforeChanges.png
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> When performing some tests with the ReplaceText processor, I found that it
> seemed to be quite a bit slower than I expected, especially when using a
> Replacement Strategy of "Literal Replace" and when using a lot of small
> FlowFiles.
> As a result, I performed some profiling and identified a few areas that could
> use some improvement:
> * When using the Literal Replace strategy, we find matches using
> {{Pattern.compile(Pattern.quote(...));}} and then using
> {{{}Pattern.matcher(...).find(){}}}. This is quite inefficient compared to
> just using {{String.indexOf(...)}} and accounted for approximately 30% of the
> time spent in the processor.
> * A significant amount of time was spent flushing the write buffer, as it
> flushes to disk when finished writing to each individual FlowFile. Even when
> we set a Run Duration > 0 ms, we flush for each FlowFile. This flush() gets
> delegated all the way down to the FileOutputStream. However, when using
> ProcessSession.append(), we intercept this with a NonFlushableOutputStream.
> We should do this when calling ProcessSession.write() as well. While it makes
> sense to flush data from the Processor layer's buffer, there's no need to
> flush past the session layer until the session is committed.
> * A decent bit of time was spent in the session's get() method calling
> {{{}final Set set =
> unacknowledgedFlowFiles.computeIfAbsent(connection.getFlowFileQueue(), k ->
> new HashSet<>());{}}}. The time here was spent in StandardFlowFileQueue's
> hashCode() method, which is the JVM default. We can easily implement
> hashCode() to just return the hashCode of the identifier, which is a String.
> This is a pre-computed hashcode so provides constant time of 0 ms (with the
> exception of the method call itself) so eliminates the expense here.
> * When using a Run Duration > 0 ms, we can hold InputStreams open by
> processing multiple FlowFiles in a given Session. This can also significantly
> improve performance. As such, we should make the default run duration 25 ms
> instead of 0 ms.
> * A common pattern with ReplaceText is to prepend text to the beginning of a
> FlowFile, or line. And then use another ReplaceText to append text to the end
> of a FlowFile, or line. We should have a strategy for "Surround" that allow
> us to both Prepend text and Append text. This will result in double the
> performance for this use case.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-10772) Unattributable error on nifi shutdown when controller service was unable to be started
[
https://issues.apache.org/jira/browse/NIFI-10772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess updated NIFI-10772:
Fix Version/s: 1.20.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Unattributable error on nifi shutdown when controller service was unable to
> be started
> --
>
> Key: NIFI-10772
> URL: https://issues.apache.org/jira/browse/NIFI-10772
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.18.0, 1.20.0
>Reporter: Nissim Shiman
>Assignee: Nissim Shiman
>Priority: Major
> Fix For: 1.20.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> This error occurs when nifi is unable to start a controller service that is
> supposed to be in an enabled state. On shutdown, nifi will give an error
> (stacktrace below)
> To reproduce, for example using, StandardRestrictedSSLContextService:
> Enable StandardRestrictedSSLContextService
> Shutdown nifi
> remove keystore StandardRestrictedSSLContextService relied on (or move it to
> different location on filesystem)
> start nifi
> stop nifi
> When nifi is shutdown the following uncaught/non-attributable error is in
> nifi-app.log:
> {code:java}
> 2023-01-06 15:46:41,085 ERROR [Timer-Driven Process Thread-5]
> org.apache.nifi.engine.FlowEngine Uncaught Exception in Runnable task
> java.util.concurrent.RejectedExecutionException: Task
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@2867c735
> rejected from org.apache.nifi.e
> ngine.FlowEngine@a814d7d[Shutting down, pool size = 10, active threads = 3,
> queued tasks = 0, completed tasks = 257823]
> at
> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
> at
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533)
> at org.apache.nifi.engine.FlowEngine.schedule(FlowEngine.java:87)
> at
> org.apache.nifi.controller.service.StandardControllerServiceNode$2.run(StandardControllerServiceNode.java:591)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750)
> {code}
> It is unclear from the current log output as to what the underlying cause of
> it was (i.e. which controller service StandardControllerServiceNode is having
> trouble with)
> A similar non-attributable error is also seen on nifi shutdown for a
> processor that relies on this controller service.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-9167) Refactor nifi-framework-bundle to use JUnit 5
[ https://issues.apache.org/jira/browse/NIFI-9167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-9167: --- Fix Version/s: 1.20.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Refactor nifi-framework-bundle to use JUnit 5 > - > > Key: NIFI-9167 > URL: https://issues.apache.org/jira/browse/NIFI-9167 > Project: Apache NiFi > Issue Type: Sub-task >Reporter: Mike Thomsen >Assignee: David Handermann >Priority: Minor > Fix For: 1.20.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11036) Add Cluster Summary metrics to Prometheus components/endpoint
[ https://issues.apache.org/jira/browse/NIFI-11036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-11036: Description: It would be nice to include Cluster Summary metrics such as "connected node count" and "total node count" to the NiFi REST API Prometheus endpoint. (was: It would be nice to include Cluster Summary metrics such as "connected node count" and "total node count". This could be added to both the NiFi REST API Prometheus endpoint as well as the ReportingTask and RecordSink.) > Add Cluster Summary metrics to Prometheus components/endpoint > - > > Key: NIFI-11036 > URL: https://issues.apache.org/jira/browse/NIFI-11036 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Minor > > It would be nice to include Cluster Summary metrics such as "connected node > count" and "total node count" to the NiFi REST API Prometheus endpoint. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-11036) Add Cluster Summary metrics to Prometheus components/endpoint
[ https://issues.apache.org/jira/browse/NIFI-11036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess reassigned NIFI-11036: --- Assignee: Matt Burgess > Add Cluster Summary metrics to Prometheus components/endpoint > - > > Key: NIFI-11036 > URL: https://issues.apache.org/jira/browse/NIFI-11036 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Minor > > It would be nice to include Cluster Summary metrics such as "connected node > count" and "total node count". This could be added to both the NiFi REST API > Prometheus endpoint as well as the ReportingTask and RecordSink. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11036) Add Cluster Summary metrics to Prometheus components/endpoint
Matt Burgess created NIFI-11036: --- Summary: Add Cluster Summary metrics to Prometheus components/endpoint Key: NIFI-11036 URL: https://issues.apache.org/jira/browse/NIFI-11036 Project: Apache NiFi Issue Type: Improvement Components: Extensions Reporter: Matt Burgess It would be nice to include Cluster Summary metrics such as "connected node count" and "total node count". This could be added to both the NiFi REST API Prometheus endpoint as well as the ReportingTask and RecordSink. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11034) Image Viewer not available in Apache NiFi release
Matt Burgess created NIFI-11034: --- Summary: Image Viewer not available in Apache NiFi release Key: NIFI-11034 URL: https://issues.apache.org/jira/browse/NIFI-11034 Project: Apache NiFi Issue Type: Bug Components: Extensions Reporter: Matt Burgess The image viewer in the UI (used to display JPEG, GIF, WEBP, etc.) is in the nifi-media-nar, which is no longer included in the Apache NiFi binary release due to size constraints. However, the image viewer itself should be available in the binary release (unless it is too large by itself). Recommend breaking the image viewer out into its own module and include it in the NiFi assembly by default. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11033) Update QueryRecord additionalDetails page to include Flow Config History table
Matt Burgess created NIFI-11033: --- Summary: Update QueryRecord additionalDetails page to include Flow Config History table Key: NIFI-11033 URL: https://issues.apache.org/jira/browse/NIFI-11033 Project: Apache NiFi Issue Type: Improvement Components: Documentation & Website, Extensions Reporter: Matt Burgess The additionalDetails.html page for QueryRecord should have been updated under NIFI-10699 to include the new table and its schema. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11031) Intermittent Failures in TestGenerateRecord
[
https://issues.apache.org/jira/browse/NIFI-11031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess updated NIFI-11031:
Status: Patch Available (was: In Progress)
> Intermittent Failures in TestGenerateRecord
> ---
>
> Key: NIFI-11031
> URL: https://issues.apache.org/jira/browse/NIFI-11031
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: David Handermann
>Assignee: Matt Burgess
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The {{TestGenerateRecord}} fails intermittently in automated builds with the
> following error:
> {noformat}
> Error:
> org.apache.nifi.processors.standard.TestGenerateRecord.testGenerateNullableFieldsZeroNullPercentageSchemaText
> Time elapsed: 1.035 s <<< FAILURE!
> org.opentest4j.AssertionFailedError: expected: but was:
> at
> org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
> at
> org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
> at org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
> at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
> at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:31)
> at org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:180)
> at
> org.apache.nifi.processors.standard.TestGenerateRecord.testGenerateNullableFieldsZeroNullPercentageSchemaText(TestGenerateRecord.java:237)
> {noformat}
> The problem appears to be related to the supporting Faker service returning
> {{null}} when the code is not expecting any {{null}} values according to the
> {{Null Percentage}} property setting.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (NIFI-11031) Intermittent Failures in TestGenerateRecord
[
https://issues.apache.org/jira/browse/NIFI-11031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess reassigned NIFI-11031:
---
Assignee: Matt Burgess
> Intermittent Failures in TestGenerateRecord
> ---
>
> Key: NIFI-11031
> URL: https://issues.apache.org/jira/browse/NIFI-11031
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: David Handermann
>Assignee: Matt Burgess
>Priority: Major
>
> The {{TestGenerateRecord}} fails intermittently in automated builds with the
> following error:
> {noformat}
> Error:
> org.apache.nifi.processors.standard.TestGenerateRecord.testGenerateNullableFieldsZeroNullPercentageSchemaText
> Time elapsed: 1.035 s <<< FAILURE!
> org.opentest4j.AssertionFailedError: expected: but was:
> at
> org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
> at
> org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
> at org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
> at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
> at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:31)
> at org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:180)
> at
> org.apache.nifi.processors.standard.TestGenerateRecord.testGenerateNullableFieldsZeroNullPercentageSchemaText(TestGenerateRecord.java:237)
> {noformat}
> The problem appears to be related to the supporting Faker service returning
> {{null}} when the code is not expecting any {{null}} values according to the
> {{Null Percentage}} property setting.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-11029) Add Standard XML Parsing to ExtractCCDAAttributes
[
https://issues.apache.org/jira/browse/NIFI-11029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess updated NIFI-11029:
Fix Version/s: 1.20.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Add Standard XML Parsing to ExtractCCDAAttributes
> -
>
> Key: NIFI-11029
> URL: https://issues.apache.org/jira/browse/NIFI-11029
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Minor
> Fix For: 1.20.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The {{ExtractCCDAAttributes}} Processor uses a custom {{CDAUtil}} class to
> load and parse the FlowFile {{InputStream}}. The {{CDAUtil}} class also
> includes a {{load}} method that takes a standard DOM {{Document}}. The
> Processor should be updated to use the standard {{nifi-xml-processing}}
> library for parsing the XML prior to calling {{CDAUtil.load}}.
> In addition to implementing standard XML parsing, the
> {{ExtractCCDAAttributes}} Processor should be deprecated for removal because
> the implementation relies on outdated libraries, and the extensive use of
> FlowFile attributes does not align with best practices for record-oriented
> data handling.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-5515) [CaptureChangeMysql] Cannot get real value of primitive type when column type is unsigned.
[ https://issues.apache.org/jira/browse/NIFI-5515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17655140#comment-17655140 ] Matt Burgess commented on NIFI-5515: Turns out this will cause problems as NiFi Record types are all signed (so is Avro which we currently use for the schema). No solution would really be complete and I don't think we want to "half-support" unsigned values across NiFi. I looked at Debezium as well and they don't appear to support unsigned types either (except TINYINT(1) UNSIGNED which can be converted to a boolean). > [CaptureChangeMysql] Cannot get real value of primitive type when column type > is unsigned. > -- > > Key: NIFI-5515 > URL: https://issues.apache.org/jira/browse/NIFI-5515 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.6.0 >Reporter: ashwin konale >Priority: Minor > > *Processor: CaptureChangeMysql* > When mysql column type is of unsigned primitive type, CaptureChangeMysql is > not able to read the data properly when the value overflows. I investigated a > bit and came across this > ([https://github.com/shyiko/mysql-binlog-connector-java/issues/104|https://github.com/shyiko/mysql-binlog-connector-java/issues/104]). > The library which is internally being used in CaptureChangeMysql Processor > ([https://github.com/shyiko/mysql-binlog-connector-java|https://github.com/shyiko/mysql-binlog-connector-java]) > does not handle this case and its upto consumer to handle it. More > description can be found in the link. > *Issue summary Eg.* > Column type : unsigned tinyint > Actual value : 128 > Value from CaptureChangeMysql: -128 (Because its still 8bits storage and > mostsignificant bit is identified as sign from java.) > *Possible solution:* > Column types are read and stored Distributed Map Cache Client so we should be > able to do these conversions. > Nifi version : 1.6.0 > Mysql version : 5.7.22-log > Binlog type. : Row-based logging -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11021) Upgrade Groovy to 3.0.14
[jira] [Updated] (NIFI-10608) Copied Processor Group no longer contains non-referenced Controller Services
[ https://issues.apache.org/jira/browse/NIFI-10608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-10608: Fix Version/s: 1.20.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Copied Processor Group no longer contains non-referenced Controller Services > > > Key: NIFI-10608 > URL: https://issues.apache.org/jira/browse/NIFI-10608 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.16.3, 1.18.0 >Reporter: Nissim Shiman >Assignee: Nissim Shiman >Priority: Major > Fix For: 1.20.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > A copied processor group does not contain controller service that original > processor group had if controller service had not been referenced to a > processor. > To reproduce: > create new PG -> right click -> Configure -> Controller Sevices -> Add any > controller service (AvroReader for example) > copy PG > new PG -> right click -> Configure -> Controller Services -> (new controller > service will not be listed/ i.e. AvroReader does not show up here) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-5515) [CaptureChangeMysql] Cannot get real value of primitive type when column type is unsigned.
[ https://issues.apache.org/jira/browse/NIFI-5515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17654083#comment-17654083 ] Matt Burgess commented on NIFI-5515: This information would have to be stored in the cache which would break compatibility. I recommend we ensure this capability works correctly in the CaptureChangeMySQLRecord processor under NIFI-4491. > [CaptureChangeMysql] Cannot get real value of primitive type when column type > is unsigned. > -- > > Key: NIFI-5515 > URL: https://issues.apache.org/jira/browse/NIFI-5515 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.6.0 >Reporter: ashwin konale >Priority: Minor > > *Processor: CaptureChangeMysql* > When mysql column type is of unsigned primitive type, CaptureChangeMysql is > not able to read the data properly when the value overflows. I investigated a > bit and came across this > ([https://github.com/shyiko/mysql-binlog-connector-java/issues/104|https://github.com/shyiko/mysql-binlog-connector-java/issues/104]). > The library which is internally being used in CaptureChangeMysql Processor > ([https://github.com/shyiko/mysql-binlog-connector-java|https://github.com/shyiko/mysql-binlog-connector-java]) > does not handle this case and its upto consumer to handle it. More > description can be found in the link. > *Issue summary Eg.* > Column type : unsigned tinyint > Actual value : 128 > Value from CaptureChangeMysql: -128 (Because its still 8bits storage and > mostsignificant bit is identified as sign from java.) > *Possible solution:* > Column types are read and stored Distributed Map Cache Client so we should be > able to do these conversions. > Nifi version : 1.6.0 > Mysql version : 5.7.22-log > Binlog type. : Row-based logging -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-10996) CSV output with header - but always, even for 0 record flowfiles
[
https://issues.apache.org/jira/browse/NIFI-10996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess updated NIFI-10996:
Affects Version/s: (was: 1.19.1)
Status: Patch Available (was: In Progress)
> CSV output with header - but always, even for 0 record flowfiles
>
>
> Key: NIFI-10996
> URL: https://issues.apache.org/jira/browse/NIFI-10996
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Reporter: Josef Zahner
>Assignee: Matt Burgess
>Priority: Minor
> Labels: csv, writer
> Attachments: NiFi_CSV_header_true.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We use a “ConvertRecord” processor where we convert an AVRO to a CSV. For
> that *CSV* output we would like to have the {*}header enabled{*}, so we tried
> to set “{{{}Include Header Line – true{}}}” for the Controller Service of the
> CSVRecordSetWriter. The issue is, *if we have zero records, the header
> doesn’t show up* (but it was there of course in the AVRO file). We need to
> have it as the columns are important for us, even if we have 0 records.
> At the moment we solve it with an extra ExecuteScript processor just before
> the ConvertRecord, there we add always an extra record with the header lines
> as string. But it feels a bit hacky as the record.count attribute is 1 record
> too high (due to the fake header record).
> !NiFi_CSV_header_true.png!
> Comment from [~joewitt] from users mailinglist: _"Makes sense what you're
> looking for. Just not sure where this 'concern' would live whether it is in
> the processors themselves or the controller services for the writers."_
> It seems that I'm not alone with that requirement, at least one other person
> (Jens M. Kofoed) uses a similar workaround.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-10996) CSV output with header - but always, even for 0 record flowfiles
[
https://issues.apache.org/jira/browse/NIFI-10996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17651444#comment-17651444
]
Matt Burgess commented on NIFI-10996:
-
I see the issue but am still looking into it for "raw records" which do not use
the schema to write the header.
> CSV output with header - but always, even for 0 record flowfiles
>
>
> Key: NIFI-10996
> URL: https://issues.apache.org/jira/browse/NIFI-10996
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.19.1
>Reporter: Josef Zahner
>Priority: Minor
> Labels: csv, writer
> Attachments: NiFi_CSV_header_true.png
>
>
> We use a “ConvertRecord” processor where we convert an AVRO to a CSV. For
> that *CSV* output we would like to have the {*}header enabled{*}, so we tried
> to set “{{{}Include Header Line – true{}}}” for the Controller Service of the
> CSVRecordSetWriter. The issue is, *if we have zero records, the header
> doesn’t show up* (but it was there of course in the AVRO file). We need to
> have it as the columns are important for us, even if we have 0 records.
> At the moment we solve it with an extra ExecuteScript processor just before
> the ConvertRecord, there we add always an extra record with the header lines
> as string. But it feels a bit hacky as the record.count attribute is 1 record
> too high (due to the fake header record).
> !NiFi_CSV_header_true.png!
> Comment from [~joewitt] from users mailinglist: _"Makes sense what you're
> looking for. Just not sure where this 'concern' would live whether it is in
> the processors themselves or the controller services for the writers."_
> It seems that I'm not alone with that requirement, at least one other person
> (Jens M. Kofoed) uses a similar workaround.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (NIFI-10996) CSV output with header - but always, even for 0 record flowfiles
[
https://issues.apache.org/jira/browse/NIFI-10996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess reassigned NIFI-10996:
---
Assignee: Matt Burgess
> CSV output with header - but always, even for 0 record flowfiles
>
>
> Key: NIFI-10996
> URL: https://issues.apache.org/jira/browse/NIFI-10996
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.19.1
>Reporter: Josef Zahner
>Assignee: Matt Burgess
>Priority: Minor
> Labels: csv, writer
> Attachments: NiFi_CSV_header_true.png
>
>
> We use a “ConvertRecord” processor where we convert an AVRO to a CSV. For
> that *CSV* output we would like to have the {*}header enabled{*}, so we tried
> to set “{{{}Include Header Line – true{}}}” for the Controller Service of the
> CSVRecordSetWriter. The issue is, *if we have zero records, the header
> doesn’t show up* (but it was there of course in the AVRO file). We need to
> have it as the columns are important for us, even if we have 0 records.
> At the moment we solve it with an extra ExecuteScript processor just before
> the ConvertRecord, there we add always an extra record with the header lines
> as string. But it feels a bit hacky as the record.count attribute is 1 record
> too high (due to the fake header record).
> !NiFi_CSV_header_true.png!
> Comment from [~joewitt] from users mailinglist: _"Makes sense what you're
> looking for. Just not sure where this 'concern' would live whether it is in
> the processors themselves or the controller services for the writers."_
> It seems that I'm not alone with that requirement, at least one other person
> (Jens M. Kofoed) uses a similar workaround.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-10998) SplitJson does not honor latest setting for JsonPath Expression
[ https://issues.apache.org/jira/browse/NIFI-10998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-10998: Status: Patch Available (was: In Progress) > SplitJson does not honor latest setting for JsonPath Expression > --- > > Key: NIFI-10998 > URL: https://issues.apache.org/jira/browse/NIFI-10998 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Due to a behavior change in NIFI-10851 (being investigated separately as a > possible regression), SplitJson will no longer recompile the latest setting > for the JsonPath Expression property. This is due to a check in > onPropertyModified() that oldValue not be null, but the new behavior is that > oldValue is being set to null, preventing the cache from being cleared and > the new expression compiled and cached. > As this appears to be a very minor performance improvement (to not bother > clearing the cache if it's the first time this property is set), I propose to > clear the JsonPath Expression cache whenever the property is changed > (regardless of its previous value). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-4572) CaptureChangeMySQL does not return database and table info without DistributedMapCache
[ https://issues.apache.org/jira/browse/NIFI-4572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-4572: --- Fix Version/s: 1.20.0 Resolution: Fixed Status: Resolved (was: Patch Available) > CaptureChangeMySQL does not return database and table info without > DistributedMapCache > -- > > Key: NIFI-4572 > URL: https://issues.apache.org/jira/browse/NIFI-4572 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Fix For: 1.20.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > CaptureChangeMySQL will leverage a DistributedMapCache, when enabled, to > store database, table, and column information for the configured MySQL > instance. This is due to the possibly large amount of information needed to > be stored for large tables and large numbers of tables. > However the database and table information (not columns) is available in the > TABLE_MAP event, associated with each MySQL transaction, and it applies to > the entire transaction. So this information can be used to populate the > database and table fields in the outgoing CDC events, even if the > DistributedMapCache is not enabled/configured. This is slightly related to > NIFI-3902, although the code locations/paths are likely different. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-10998) SplitJson does not honor latest setting for JsonPath Expression
[ https://issues.apache.org/jira/browse/NIFI-10998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess reassigned NIFI-10998: --- Assignee: Matt Burgess > SplitJson does not honor latest setting for JsonPath Expression > --- > > Key: NIFI-10998 > URL: https://issues.apache.org/jira/browse/NIFI-10998 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > > Due to a behavior change in NIFI-10851 (being investigated separately as a > possible regression), SplitJson will no longer recompile the latest setting > for the JsonPath Expression property. This is due to a check in > onPropertyModified() that oldValue not be null, but the new behavior is that > oldValue is being set to null, preventing the cache from being cleared and > the new expression compiled and cached. > As this appears to be a very minor performance improvement (to not bother > clearing the cache if it's the first time this property is set), I propose to > clear the JsonPath Expression cache whenever the property is changed > (regardless of its previous value). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-10998) SplitJson does not honor latest setting for JsonPath Expression
Matt Burgess created NIFI-10998: --- Summary: SplitJson does not honor latest setting for JsonPath Expression Key: NIFI-10998 URL: https://issues.apache.org/jira/browse/NIFI-10998 Project: Apache NiFi Issue Type: Bug Components: Extensions Reporter: Matt Burgess Due to a behavior change in NIFI-10851 (being investigated separately as a possible regression), SplitJson will no longer recompile the latest setting for the JsonPath Expression property. This is due to a check in onPropertyModified() that oldValue not be null, but the new behavior is that oldValue is being set to null, preventing the cache from being cleared and the new expression compiled and cached. As this appears to be a very minor performance improvement (to not bother clearing the cache if it's the first time this property is set), I propose to clear the JsonPath Expression cache whenever the property is changed (regardless of its previous value). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-10800) CaptureChangeMySQL Indicates an error after the restart
[ https://issues.apache.org/jira/browse/NIFI-10800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess resolved NIFI-10800. - Resolution: Duplicate I believe the cause of this to be a duplicate of NIFI-6428, which has been fixed in NiFi 1.19.1 and will be 1.20. If you find this error remains in either of those releases, please feel free to reopen the case with comments. > CaptureChangeMySQL Indicates an error after the restart > --- > > Key: NIFI-10800 > URL: https://issues.apache.org/jira/browse/NIFI-10800 > Project: Apache NiFi > Issue Type: Bug > Environment: Nifi: 1.17 > mysql: 5.7.39 >Reporter: YangHai >Priority: Major > Attachments: nifi-bug2.1.PNG, nifi-bug2.2.PNG > > > If the CaptureChangeMySQL status is not cleared when CaptureChangeMySQL is > restarted after one or more data is transferred, an error message is > displayed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
