[GitHub] [nifi] rrjanbiah commented on pull request #4065: NIFI-4239 - Adding CaptureChangePostgreSQL processor to capture data changes (INSERT/UPDATE/DELETE) in PostgreSQL tables via Logical Replicat
rrjanbiah commented on pull request #4065: URL: https://github.com/apache/nifi/pull/4065#issuecomment-1002862247 Bump! Holidays and the contributors might be away -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (NIFI-9485) BouncyCastle conflicted when compiled and running nifi-1.16.0 on windows11
[ https://issues.apache.org/jira/browse/NIFI-9485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sunyd1224 resolved NIFI-9485. - > BouncyCastle conflicted when compiled and running nifi-1.16.0 on windows11 > --- > > Key: NIFI-9485 > URL: https://issues.apache.org/jira/browse/NIFI-9485 > Project: Apache NiFi > Issue Type: Improvement > Components: Tools and Build >Affects Versions: 1.16.0 > Environment: os: windows 11 > java version: 1.8.0_171 > maven version: 3.8.4 >Reporter: sunyd1224 >Priority: Minor > Labels: beginner > Fix For: 1.16.0 > > Attachments: error.png > > Time Spent: 40m > Remaining Estimate: 0h > > I built nifi-1.16.0 from the source code for a long time. When finished > successfully,I run the run-nifi.bat. Hower,the error occurs: > !error.png! > The reason is that there is another lower version of BouncyCastle than needed > in the java classpath on my computer and the BouncyCastle is not specified > with the precise version in some of the pom.xml of nifi. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [nifi] github-actions[bot] commented on pull request #4065: NIFI-4239 - Adding CaptureChangePostgreSQL processor to capture data changes (INSERT/UPDATE/DELETE) in PostgreSQL tables via Logica
github-actions[bot] commented on pull request #4065: URL: https://github.com/apache/nifi/pull/4065#issuecomment-1002820121 We're marking this PR as stale due to lack of updates in the past few months. If after another couple of weeks the stale label has not been removed this PR will be closed. This stale marker and eventual auto close does not indicate a judgement of the PR just lack of reviewer bandwidth and helps us keep the PR queue more manageable. If you would like this PR re-opened you can do so and a committer can remove the stale tag. Or you can open a new PR. Try to help review other PRs to increase PR review bandwidth which in turn helps yours. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-9523) Streamline Maven Enforcer Plugin Executions
[ https://issues.apache.org/jira/browse/NIFI-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-9523: --- Status: Patch Available (was: Open) > Streamline Maven Enforcer Plugin Executions > --- > > Key: NIFI-9523 > URL: https://issues.apache.org/jira/browse/NIFI-9523 > Project: Apache NiFi > Issue Type: Improvement > Components: Tools and Build >Reporter: David Handermann >Assignee: David Handermann >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > The Maven Enforcer Plugin configuration includes multiple execution > definitions to standardize project configuration and ban several > dependencies. These execution configurations can be consolidated into a > single execution to avoid multiple invocations of the Enforcer Plugin. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [nifi] exceptionfactory opened a new pull request #5623: NIFI-9523 Consolidate Maven Enforcer Plugin executions
exceptionfactory opened a new pull request #5623:
URL: https://github.com/apache/nifi/pull/5623
Description of PR
NIFI-9523 Streamlines the Maven Enforcer Plugin configuration by combining
multiple execution definitions into a single set of rules. This change reduces
the number of Enforcer Plugin executions during the build process. Additional
changes include removing the duplicative Enforcer Plugin definition from the
`nifi-registry` Maven configuration.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [X] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [X] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [X] Has your PR been rebased against the latest commit within the target
branch (typically `main`)?
- [X] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [X] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [X] Have you verified that the full build is successful on JDK 8?
- [ ] Have you verified that the full build is successful on JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for
build issues and submit an update to your PR as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (NIFI-9462) JsonTreeReader schema inference only examines first record for parts of structure, causing data loss for subsequent records
[
https://issues.apache.org/jira/browse/NIFI-9462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17466546#comment-17466546
]
Otto Fowler commented on NIFI-9462:
---
Sorry [~Josiah.Johnston], I was greatly mistaken
> JsonTreeReader schema inference only examines first record for parts of
> structure, causing data loss for subsequent records
> ---
>
> Key: NIFI-9462
> URL: https://issues.apache.org/jira/browse/NIFI-9462
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.13.0
>Reporter: Josiah Johnston
>Priority: Major
> Attachments: JSON record set writer.png, JSON tree reader.png,
> NIFI-9462_example.xml, flow.png, updateRecord.png
>
>
> I use GenerateFlowFile with this JSON line content, and send it through an
> UpdateRecord processor that uses a JsonTreeReader with schema inference. The
> UpdateRecord processor adds a top level `s3_key` element with the filename.
> {"_source": {"name": "battery-voltage-changed", "metadata":
> {"voltage": 2.8}
> }}
> {"_source": {"name": "temperature-changed", "metadata":
> {"temperature": 19.54}
> }}
> {"other_L1_keys": "are_preserved", "_source": {"other_L2_keys":
> "are_preserved", "metadata":
> {"voltage": 6.3, "other_L3_keys": "are_lost"}
> }}
> In the output, the structure of `_source.metadata.*` is always strictly based
> on the first record, causing data loss for subsequent records that have
> different fields.
> {"_source":{"name":"battery-voltage-changed","metadata":
> {"voltage":2.8}
> ,"other_L2_keys":null},"other_L1_keys":null,"s3_key":"9830423c-c8b6-4a03-a4a1-427750e94d26"}
> {"_source":{"name":"temperature-changed","metadata":
> {"voltage":null}
> ,"other_L2_keys":null},"other_L1_keys":null,"s3_key":"9830423c-c8b6-4a03-a4a1-427750e94d26"}
> {"_source":{"name":null,"metadata":
> {"voltage":6.3}
> ,"other_L2_keys":"are_preserved"},"other_L1_keys":"are_preserved","s3_key":"9830423c-c8b6-4a03-a4a1-427750e94d26"}
> In general it drops all 3rd level keys weren't seen in the first record
> (_source.metadata.temperature in record 2, _source.metadata.other_L3_keys in
> record 3). This behavior only applies to keys in the 3rd level; schema
> inference works as documented (scanning through all records) for alternative
> keys in the 1st & 2nd level.
> This behavior persists whether I specify the input as JSON lines (shown in
> this example), or if I rearrange it to be a JSON array.
> I've attached screenshots of a minimal example and settings of JSON reader &
> writer. I've also attached a template of a minimal example. If you import it,
> you'll need to create & enable controller services of JsonTreeReader &
> JsonRecordSetWriter (default values for each).
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (NIFI-9523) Streamline Maven Enforcer Plugin Executions
David Handermann created NIFI-9523: -- Summary: Streamline Maven Enforcer Plugin Executions Key: NIFI-9523 URL: https://issues.apache.org/jira/browse/NIFI-9523 Project: Apache NiFi Issue Type: Improvement Components: Tools and Build Reporter: David Handermann Assignee: David Handermann The Maven Enforcer Plugin configuration includes multiple execution definitions to standardize project configuration and ban several dependencies. These execution configurations can be consolidated into a single execution to avoid multiple invocations of the Enforcer Plugin. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (NIFI-8927) Add option to enable/disable all Controller Services in a group
[ https://issues.apache.org/jira/browse/NIFI-8927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-8927: - Summary: Add option to enable/disable all Controller Services in a group (was: Add option to start/stop all controllers) > Add option to enable/disable all Controller Services in a group > --- > > Key: NIFI-8927 > URL: https://issues.apache.org/jira/browse/NIFI-8927 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI >Affects Versions: 1.14.0 >Reporter: Eduardo Mota Fontes >Assignee: Eduardo Mota Fontes >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > Add options in context menu to start and stop all controllers from a Process > Group like start/stop processors. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (NIFI-6866) Create a persistent or stateful stats repository
[ https://issues.apache.org/jira/browse/NIFI-6866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne resolved NIFI-6866. -- Resolution: Fixed > Create a persistent or stateful stats repository > > > Key: NIFI-6866 > URL: https://issues.apache.org/jira/browse/NIFI-6866 > Project: Apache NiFi > Issue Type: New Feature > Components: Core Framework >Reporter: Jon Kessler >Assignee: Jon Kessler >Priority: Minor > Time Spent: 3h > Remaining Estimate: 0h > > Create a stats repository that will hold stats across a restart. This could > be useful in diagnosing issues that required or caused a restart by giving a > better picture of what was happening leading up to the restart. I believe > this can be accomplished using NiFi's state management. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (NIFI-9462) JsonTreeReader schema inference only examines first record for parts of structure, causing data loss for subsequent records
[
https://issues.apache.org/jira/browse/NIFI-9462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17466517#comment-17466517
]
Mark Payne commented on NIFI-9462:
--
[~Josiah.Johnston] can you try running this flow in a newer version of NiFi? I
just tried running it in 1.15.1, and I got the output that I was expecting.
Namely:
{noformat}
{"_source":{"name":"battery-voltage-changed","metadata":{"voltage":2.8},"other_L2_keys":null},"other_L1_keys":null,"s3_key":"d8670196-f302-4860-8753-213cdcdf9c24"}
{"_source":{"name":"temperature-changed","metadata":{"temperature":19.54},"other_L2_keys":null},"other_L1_keys":null,"s3_key":"d8670196-f302-4860-8753-213cdcdf9c24"}
{"_source":{"name":null,"metadata":{"voltage":6.3,"other_L3_keys":"are_lost"},"other_L2_keys":"are_preserved"},"other_L1_keys":"are_preserved","s3_key":"d8670196-f302-4860-8753-213cdcdf9c24"}
{noformat}
Is this the output that you expected? If so please try with the latest, 1.15.2
If this is NOT the expected output then please help me to understand what I
overlooked.
Thanks
> JsonTreeReader schema inference only examines first record for parts of
> structure, causing data loss for subsequent records
> ---
>
> Key: NIFI-9462
> URL: https://issues.apache.org/jira/browse/NIFI-9462
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.13.0
>Reporter: Josiah Johnston
>Priority: Major
> Attachments: JSON record set writer.png, JSON tree reader.png,
> NIFI-9462_example.xml, flow.png, updateRecord.png
>
>
> I use GenerateFlowFile with this JSON line content, and send it through an
> UpdateRecord processor that uses a JsonTreeReader with schema inference. The
> UpdateRecord processor adds a top level `s3_key` element with the filename.
> {"_source": {"name": "battery-voltage-changed", "metadata":
> {"voltage": 2.8}
> }}
> {"_source": {"name": "temperature-changed", "metadata":
> {"temperature": 19.54}
> }}
> {"other_L1_keys": "are_preserved", "_source": {"other_L2_keys":
> "are_preserved", "metadata":
> {"voltage": 6.3, "other_L3_keys": "are_lost"}
> }}
> In the output, the structure of `_source.metadata.*` is always strictly based
> on the first record, causing data loss for subsequent records that have
> different fields.
> {"_source":{"name":"battery-voltage-changed","metadata":
> {"voltage":2.8}
> ,"other_L2_keys":null},"other_L1_keys":null,"s3_key":"9830423c-c8b6-4a03-a4a1-427750e94d26"}
> {"_source":{"name":"temperature-changed","metadata":
> {"voltage":null}
> ,"other_L2_keys":null},"other_L1_keys":null,"s3_key":"9830423c-c8b6-4a03-a4a1-427750e94d26"}
> {"_source":{"name":null,"metadata":
> {"voltage":6.3}
> ,"other_L2_keys":"are_preserved"},"other_L1_keys":"are_preserved","s3_key":"9830423c-c8b6-4a03-a4a1-427750e94d26"}
> In general it drops all 3rd level keys weren't seen in the first record
> (_source.metadata.temperature in record 2, _source.metadata.other_L3_keys in
> record 3). This behavior only applies to keys in the 3rd level; schema
> inference works as documented (scanning through all records) for alternative
> keys in the 1st & 2nd level.
> This behavior persists whether I specify the input as JSON lines (shown in
> this example), or if I rearrange it to be a JSON array.
> I've attached screenshots of a minimal example and settings of JSON reader &
> writer. I've also attached a template of a minimal example. If you import it,
> you'll need to create & enable controller services of JsonTreeReader &
> JsonRecordSetWriter (default values for each).
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (NIFI-9462) JsonTreeReader schema inference only examines first record for parts of structure, causing data loss for subsequent records
[
https://issues.apache.org/jira/browse/NIFI-9462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17466516#comment-17466516
]
Mark Payne commented on NIFI-9462:
--
[~otto] the schema inference should not use the first record only. Rather, it
should be reading ALL records and creating an "uber schema" that encapsulates
all fields. In this way, all records will be homogeneous in that they all
implement the same schema. But it's very common to have some records with
missing fields, especially with JSON where null fields are often left unwritten
in order to improve efficiency, etc.
> JsonTreeReader schema inference only examines first record for parts of
> structure, causing data loss for subsequent records
> ---
>
> Key: NIFI-9462
> URL: https://issues.apache.org/jira/browse/NIFI-9462
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.13.0
>Reporter: Josiah Johnston
>Priority: Major
> Attachments: JSON record set writer.png, JSON tree reader.png,
> NIFI-9462_example.xml, flow.png, updateRecord.png
>
>
> I use GenerateFlowFile with this JSON line content, and send it through an
> UpdateRecord processor that uses a JsonTreeReader with schema inference. The
> UpdateRecord processor adds a top level `s3_key` element with the filename.
> {"_source": {"name": "battery-voltage-changed", "metadata":
> {"voltage": 2.8}
> }}
> {"_source": {"name": "temperature-changed", "metadata":
> {"temperature": 19.54}
> }}
> {"other_L1_keys": "are_preserved", "_source": {"other_L2_keys":
> "are_preserved", "metadata":
> {"voltage": 6.3, "other_L3_keys": "are_lost"}
> }}
> In the output, the structure of `_source.metadata.*` is always strictly based
> on the first record, causing data loss for subsequent records that have
> different fields.
> {"_source":{"name":"battery-voltage-changed","metadata":
> {"voltage":2.8}
> ,"other_L2_keys":null},"other_L1_keys":null,"s3_key":"9830423c-c8b6-4a03-a4a1-427750e94d26"}
> {"_source":{"name":"temperature-changed","metadata":
> {"voltage":null}
> ,"other_L2_keys":null},"other_L1_keys":null,"s3_key":"9830423c-c8b6-4a03-a4a1-427750e94d26"}
> {"_source":{"name":null,"metadata":
> {"voltage":6.3}
> ,"other_L2_keys":"are_preserved"},"other_L1_keys":"are_preserved","s3_key":"9830423c-c8b6-4a03-a4a1-427750e94d26"}
> In general it drops all 3rd level keys weren't seen in the first record
> (_source.metadata.temperature in record 2, _source.metadata.other_L3_keys in
> record 3). This behavior only applies to keys in the 3rd level; schema
> inference works as documented (scanning through all records) for alternative
> keys in the 1st & 2nd level.
> This behavior persists whether I specify the input as JSON lines (shown in
> this example), or if I rearrange it to be a JSON array.
> I've attached screenshots of a minimal example and settings of JSON reader &
> writer. I've also attached a template of a minimal example. If you import it,
> you'll need to create & enable controller services of JsonTreeReader &
> JsonRecordSetWriter (default values for each).
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (NIFI-9462) JsonTreeReader schema inference only examines first record for parts of structure, causing data loss for subsequent records
[
https://issues.apache.org/jira/browse/NIFI-9462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17466504#comment-17466504
]
Otto Fowler commented on NIFI-9462:
---
I believe this is working correctly. When you are passing multiple json
structures to the schema inference it is going to assume that they are
homogeneous and use the first one.
> JsonTreeReader schema inference only examines first record for parts of
> structure, causing data loss for subsequent records
> ---
>
> Key: NIFI-9462
> URL: https://issues.apache.org/jira/browse/NIFI-9462
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.13.0
>Reporter: Josiah Johnston
>Priority: Major
> Attachments: JSON record set writer.png, JSON tree reader.png,
> NIFI-9462_example.xml, flow.png, updateRecord.png
>
>
> I use GenerateFlowFile with this JSON line content, and send it through an
> UpdateRecord processor that uses a JsonTreeReader with schema inference. The
> UpdateRecord processor adds a top level `s3_key` element with the filename.
> {"_source": {"name": "battery-voltage-changed", "metadata":
> {"voltage": 2.8}
> }}
> {"_source": {"name": "temperature-changed", "metadata":
> {"temperature": 19.54}
> }}
> {"other_L1_keys": "are_preserved", "_source": {"other_L2_keys":
> "are_preserved", "metadata":
> {"voltage": 6.3, "other_L3_keys": "are_lost"}
> }}
> In the output, the structure of `_source.metadata.*` is always strictly based
> on the first record, causing data loss for subsequent records that have
> different fields.
> {"_source":{"name":"battery-voltage-changed","metadata":
> {"voltage":2.8}
> ,"other_L2_keys":null},"other_L1_keys":null,"s3_key":"9830423c-c8b6-4a03-a4a1-427750e94d26"}
> {"_source":{"name":"temperature-changed","metadata":
> {"voltage":null}
> ,"other_L2_keys":null},"other_L1_keys":null,"s3_key":"9830423c-c8b6-4a03-a4a1-427750e94d26"}
> {"_source":{"name":null,"metadata":
> {"voltage":6.3}
> ,"other_L2_keys":"are_preserved"},"other_L1_keys":"are_preserved","s3_key":"9830423c-c8b6-4a03-a4a1-427750e94d26"}
> In general it drops all 3rd level keys weren't seen in the first record
> (_source.metadata.temperature in record 2, _source.metadata.other_L3_keys in
> record 3). This behavior only applies to keys in the 3rd level; schema
> inference works as documented (scanning through all records) for alternative
> keys in the 1st & 2nd level.
> This behavior persists whether I specify the input as JSON lines (shown in
> this example), or if I rearrange it to be a JSON array.
> I've attached screenshots of a minimal example and settings of JSON reader &
> writer. I've also attached a template of a minimal example. If you import it,
> you'll need to create & enable controller services of JsonTreeReader &
> JsonRecordSetWriter (default values for each).
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
