[jira] [Commented] (NIFI-11766) ElasticSearchClientServiceImpl malformed (nonstandard?) expires attribute when connecting through AWS LoadBalancer
[
https://issues.apache.org/jira/browse/NIFI-11766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849791#comment-17849791
]
Chris Sampson commented on NIFI-11766:
--
This appears to be a warning logged by the difference between what the
Elasticsearch REST client's use of Apache HttpClient expects with Cookie
Headers vs. what the AWS ALB is sending. See
https://github.com/elastic/support-diagnostics/issues/233, for example.
This appears to be ignorable and could be excluded from the NiFi logs by
changing one's {{logback.xml}} configuration, but that comes at the expense of
also removing logs for these classes/packages that may well be relevant for
other issues.
A suggested approach is to change the unerlying Cookie Handler, e.g.
{code:java}
builder.setRequestConfigCallback(
requestConfigBuilder ->
requestConfigBuilder.setCookieSpec(CookieSpecs.STANDARD));
{code}
which could be done in the {{ElasticSearchClientServiceImpl#setupClient}},
**but** that's likely to cause problems with some of the available
Elasticsearch authentication strategies/functionality, so is probably not the
right answer here.
Instead, this is maybe something that would be handled without warning by the
AWS OpenSearch Client library equivalent, if NiFi implements such an
alternative in the future.
> ElasticSearchClientServiceImpl malformed (nonstandard?) expires attribute
> when connecting through AWS LoadBalancer
> --
>
> Key: NIFI-11766
> URL: https://issues.apache.org/jira/browse/NIFI-11766
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.19.1
> Environment: Ubuntu 22.04, EC2 machine in AWS,
> Java: 11.0.18, AWS
>Reporter: Marek Radwański
>Priority: Minor
>
> Each request to elasticsearch from a processor that utilizes
> ElasticSearchClientServiceImpl to connect to the cluster results in a logged
> Warning entry of:
> 2023-06-30 11:11:42,194 WARN [I/O dispatcher 2]
> o.a.h.c.protocol.ResponseProcessCookies Invalid cookie header: "Set-Cookie:
> AWSALBCORS=m0NDYBGa0v2NsPURJUVvxtso3AQ4Zatbj8EQDO2qLpAYRcHoG4oLoOQkQNhzXAuSxoJhTUD944olr/wNhfszNBS5ENVkivErGMFyys2EfL;
> Expires=Fri, 07 Jul 2023 11:11:42 GMT; Path=/; SameSite=None". Invalid
> 'expires' attribute: Fri, 07 Jul 2023 11:11:42 GMT
> Resulting in millions of those entries accumulating over time on a large
> cluster.
> This seems like a mismatch of cookie standards the underlying HTTP client
> uses and what AWS ALB expects.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (NIFI-12343) PutElasticsearchJson exception with JSON strings over 20 MB
[
https://issues.apache.org/jira/browse/NIFI-12343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson reassigned NIFI-12343:
Assignee: Chris Sampson
> PutElasticsearchJson exception with JSON strings over 20 MB
> ---
>
> Key: NIFI-12343
> URL: https://issues.apache.org/jira/browse/NIFI-12343
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.23.2, 2.0.0-M2
>Reporter: Gregory M. Foreman
>Assignee: Chris Sampson
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
>
> PutElasticsearchJson throws an exception when reading JSON documents that
> contain string fields over 20 MB:
> {code:java}
> PutElasticsearchJson[id=] No FlowFiles successfully parsed for sending to
> Elasticsearch
> PutElasticsearchJson[id=] Could not read FlowFile content valid JSON.:
> com.fasterxml.jackson.core.exc.StreamConstraintsException: String length
> (20050553) exceeds the maximum length (2000)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12343) PutElasticsearchJson exception with JSON strings over 20 MB
[
https://issues.apache.org/jira/browse/NIFI-12343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson updated NIFI-12343:
-
Status: Patch Available (was: In Progress)
> PutElasticsearchJson exception with JSON strings over 20 MB
> ---
>
> Key: NIFI-12343
> URL: https://issues.apache.org/jira/browse/NIFI-12343
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 2.0.0-M2, 1.23.2
>Reporter: Gregory M. Foreman
>Assignee: Chris Sampson
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
>
> PutElasticsearchJson throws an exception when reading JSON documents that
> contain string fields over 20 MB:
> {code:java}
> PutElasticsearchJson[id=] No FlowFiles successfully parsed for sending to
> Elasticsearch
> PutElasticsearchJson[id=] Could not read FlowFile content valid JSON.:
> com.fasterxml.jackson.core.exc.StreamConstraintsException: String length
> (20050553) exceeds the maximum length (2000)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-12343) PutElasticsearchJson exception with JSON strings over 20 MB
[
https://issues.apache.org/jira/browse/NIFI-12343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849691#comment-17849691
]
Chris Sampson commented on NIFI-12343:
--
This is caused by the fact that the default Jackson {{ObjectMapper}} maximum
string read length is limited to 20MB.
As with the {{JsonTreeReader}} and other components in NiFi, we could allow the
override of this setting when creating the {{ObjectMapper}} in the
{{PutElasticsearchJson}} processor (and other Elasticsearch related processors
where they're using an {{ObjectMapper}} to parse thngs such as the {{query}}
attribute, etc. although they're much less likely to container such large
String values).
A likely workaround for this issue is to use the {{PutElasticsearchRecord}}
processor instead, with a {{JsonTreeReader}} using a larger {{Max String
Length}} setting.
> PutElasticsearchJson exception with JSON strings over 20 MB
> ---
>
> Key: NIFI-12343
> URL: https://issues.apache.org/jira/browse/NIFI-12343
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.23.2, 2.0.0-M2
>Reporter: Gregory M. Foreman
>Priority: Major
>
>
> PutElasticsearchJson throws an exception when reading JSON documents that
> contain string fields over 20 MB:
> {code:java}
> PutElasticsearchJson[id=] No FlowFiles successfully parsed for sending to
> Elasticsearch
> PutElasticsearchJson[id=] Could not read FlowFile content valid JSON.:
> com.fasterxml.jackson.core.exc.StreamConstraintsException: String length
> (20050553) exceeds the maximum length (2000)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-10948) QueryElasticsearchHTTP doesnt work on Opensearch (and possibly newer Elasticsearch versions)
[ https://issues.apache.org/jira/browse/NIFI-10948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844461#comment-17844461 ] Chris Sampson commented on NIFI-10948: -- I tested the REST API based Elasticsearch components of NiFi against AWS OpenSearch after upgrading the Elasticsearch (low-level) REST Client library to the latest version (8.13.3) as part of NIFI-11686. Aside from the lack of Point in Time queries (an Elastic XPack feature that provides an improved pagination approach over the older Scroll functionality), things seemed to work without issue. Document types have been deprecated in Elasticsearch since long before AWS forked the libraries to create OpenSearch. Use of OS without Types works fine if seems, and I imagine would be the recommended approach for OS use as it is in Elasticsearch (ES 8 no longer supports Types at all, NiFi allows for Types but discourages their use). So I'd suggest this isn't something that will be fixed in the NiFi 1 branch, and works fine in the NiFi 2 branch for both OS & ES. Work to create OS specific implementations could be raised as separate tickets to include: * use of the AWS OpenSearch client libraries (rather than Elastic libraries) * Allowing AWS-style auth, e.g. assume IAM roles (hopefully via the existing NiFi AWS controller services), rather than simple BASIC auth (username & password) > QueryElasticsearchHTTP doesnt work on Opensearch (and possibly newer > Elasticsearch versions) > > > Key: NIFI-10948 > URL: https://issues.apache.org/jira/browse/NIFI-10948 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.15.3 > Environment: Nifi 1.15.3 on centos machine >Reporter: David Vassallo >Priority: Minor > Attachments: image-2022-12-05-16-34-20-513.png > > > The QueryElasticsearchHTTP processor will return a "Null Pointer Exception" > when running agasint Opensearch and probably newer versions of Elasticsearch. > I tracked down the null pointer to this line: > > [https://github.com/apache/nifi/blob/0ebc6d31489e975dcbbe078fa572332ef8ffa9e2/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/QueryElasticsearchHttp.java#L418] > > The processor is looking for the "_type" field which is no longer in the > right place. In opensearch v 1.x and above (possibly Elasticsearch too, but I > dont have the means to test this), the field has moved to be a child of the > "@metadata" field: > !image-2022-12-05-16-34-20-513.png! > > If you try to circumvent this issue by populating the "Type" property of the > processor, you will get an HTTP 400 Bad Message response since the processor > builds a URL with the "size" and "from" arguments as part of the URL which > are not supported by Opensearch/Elasticsearch in more recent versions, so you > are forced to leave this property empty but then run into the null problem > described above -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-10948) QueryElasticsearchHTTP doesnt work on Opensearch (and possibly newer Elasticsearch versions)
[ https://issues.apache.org/jira/browse/NIFI-10948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Sampson resolved NIFI-10948. -- Resolution: Won't Fix > QueryElasticsearchHTTP doesnt work on Opensearch (and possibly newer > Elasticsearch versions) > > > Key: NIFI-10948 > URL: https://issues.apache.org/jira/browse/NIFI-10948 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.15.3 > Environment: Nifi 1.15.3 on centos machine >Reporter: David Vassallo >Priority: Minor > Attachments: image-2022-12-05-16-34-20-513.png > > > The QueryElasticsearchHTTP processor will return a "Null Pointer Exception" > when running agasint Opensearch and probably newer versions of Elasticsearch. > I tracked down the null pointer to this line: > > [https://github.com/apache/nifi/blob/0ebc6d31489e975dcbbe078fa572332ef8ffa9e2/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/QueryElasticsearchHttp.java#L418] > > The processor is looking for the "_type" field which is no longer in the > right place. In opensearch v 1.x and above (possibly Elasticsearch too, but I > dont have the means to test this), the field has moved to be a child of the > "@metadata" field: > !image-2022-12-05-16-34-20-513.png! > > If you try to circumvent this issue by populating the "Type" property of the > processor, you will get an HTTP 400 Bad Message response since the processor > builds a URL with the "size" and "from" arguments as part of the URL which > are not supported by Opensearch/Elasticsearch in more recent versions, so you > are forced to leave this property empty but then run into the null problem > described above -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-11686) Update elasticsearch.client.version to 7.17.10
[
https://issues.apache.org/jira/browse/NIFI-11686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844450#comment-17844450
]
Chris Sampson commented on NIFI-11686:
--
I upgraded the Eliastcsearch (low-level) REST Client library to {{8.13.3}} and
confirmed that various NiFi Elasticsearch components work against a local
Elasticsearch instance (hosted in Docker) - [^NiFi_Flow-2.json].
Confirmed the NiFi Elasticsearch {{integration-tests}} work with the upgraded
dependency version (against Elasticsearch {{8.13.3}} and {{7.17.21}} - Elastic
continue to support the latest minor/patch version of the last major release).
Created an AWS OpenSearch Domain (Free-Tier single instance}) version {{2.11}
with a {{master user}} (username & password for {{BASIC}} auth) and selected
the {{Only use fine-grained access control}} security setting with {{Public
access}} - i.e. the Domain was available to access over the internet (so I
could connect from my laptop) and the BASIC-Auth'd user was allowed to perform
all actions within the Domain (N.B. it appears the AWS OpenSearch defaults are
to only allow access from within a VPC and Deny all permissions by defualt -
which makes a lot of sense, but isn't what I wanted here). I was then able to
use the same Flow to {{Verify}} the {{ElasticsearchClientSerice}} controller
service and run the same set of processors as for Elasticsearch {{8.13.1}} -
[^NiFi_Flow-2-OpenSearch.json].
So it would appear that the previous move to using the Elasticsearch low-level
REST Client (rather than the high-level REST client that was originally part of
the NiFi components) means that the upgrade to the most recent client libraries
*doesn't* break use of non-Elastic services!
The only thing I couldn't do was use a {{Point in Time}} ({{_pit}}) search type
for pagination (as that's an Elastic XPack feature), so I've updated the NiFi
docs to indicate that PiT is not available for AWS OpenSearch.
> Update elasticsearch.client.version to 7.17.10
> --
>
> Key: NIFI-11686
> URL: https://issues.apache.org/jira/browse/NIFI-11686
> Project: Apache NiFi
> Issue Type: Improvement
>Affects Versions: 1.22.0
>Reporter: Mike R
>Assignee: Chris Sampson
>Priority: Major
> Attachments: NiFi_Flow-2-OpenSearch.json, NiFi_Flow-2.json
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Update elasticsearch.client.version to 7.17.10
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-11686) Update elasticsearch.client.version to 7.17.10
[ https://issues.apache.org/jira/browse/NIFI-11686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Sampson updated NIFI-11686: - Attachment: NiFi_Flow-2-OpenSearch.json > Update elasticsearch.client.version to 7.17.10 > -- > > Key: NIFI-11686 > URL: https://issues.apache.org/jira/browse/NIFI-11686 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.22.0 >Reporter: Mike R >Assignee: Chris Sampson >Priority: Major > Attachments: NiFi_Flow-2-OpenSearch.json, NiFi_Flow-2.json > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Update elasticsearch.client.version to 7.17.10 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11686) Update elasticsearch.client.version to 7.17.10
[ https://issues.apache.org/jira/browse/NIFI-11686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Sampson updated NIFI-11686: - Attachment: NiFi_Flow-2.json > Update elasticsearch.client.version to 7.17.10 > -- > > Key: NIFI-11686 > URL: https://issues.apache.org/jira/browse/NIFI-11686 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.22.0 >Reporter: Mike R >Assignee: Chris Sampson >Priority: Major > Attachments: NiFi_Flow-2.json > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Update elasticsearch.client.version to 7.17.10 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11686) Update elasticsearch.client.version to 7.17.10
[ https://issues.apache.org/jira/browse/NIFI-11686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Sampson updated NIFI-11686: - Status: Patch Available (was: In Progress) > Update elasticsearch.client.version to 7.17.10 > -- > > Key: NIFI-11686 > URL: https://issues.apache.org/jira/browse/NIFI-11686 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.22.0 >Reporter: Mike R >Assignee: Chris Sampson >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > > Update elasticsearch.client.version to 7.17.10 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-11686) Update elasticsearch.client.version to 7.17.10
[ https://issues.apache.org/jira/browse/NIFI-11686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Sampson reassigned NIFI-11686: Assignee: Chris Sampson (was: Jeyassri Balachandran) > Update elasticsearch.client.version to 7.17.10 > -- > > Key: NIFI-11686 > URL: https://issues.apache.org/jira/browse/NIFI-11686 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.22.0 >Reporter: Mike R >Assignee: Chris Sampson >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > Update elasticsearch.client.version to 7.17.10 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-11686) Update elasticsearch.client.version to 7.17.10
[ https://issues.apache.org/jira/browse/NIFI-11686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844389#comment-17844389 ] Chris Sampson commented on NIFI-11686: -- [~joewitt] yes, exactly - I'm certainly in favour of upgrading Elasticsearch dependencies! Knowing whether such an upgrade would break anything (e.g. OpenSearch access) would be useful **but** not a blocker to doing the upgrade. I keep hoping to find the time to try some tests, but haven't so far had the opportunity. I have no problem with someone simply performing the dependency upgrade (provided the Integration/System Tests continue to work, of course), and the community can deal with any breakages that occur. Doing this as part of the NiFi 2.x move would probably make sense too - a potential breaking change for the community (but there's no reason this couldn't be done in the 1.x line if needed too). > Update elasticsearch.client.version to 7.17.10 > -- > > Key: NIFI-11686 > URL: https://issues.apache.org/jira/browse/NIFI-11686 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.22.0 >Reporter: Mike R >Assignee: Jeyassri Balachandran >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > Update elasticsearch.client.version to 7.17.10 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (NIFI-11686) Update elasticsearch.client.version to 7.17.10
[
https://issues.apache.org/jira/browse/NIFI-11686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844380#comment-17844380
]
Chris Sampson edited comment on NIFI-11686 at 5/7/24 4:46 PM:
--
[~joewitt] I agree completely.
There was an email from some AWS engineers a long time ago about contributing
an OpenSearch capability, but that hasn't materialised unfortuantely.
I think a sensible thing would be to:
* test the existing {{ElasticsearchClientServiceImpl}} against OpenSearch to
see whether it works
* upgrade the Elasticsearch low-level REST client library to the latest
(currently {{8.13.3}}) and re-test the OpenSearch connectivity (it is assumed
from some reading/online articles that this may not work, but I for one have
never confirmed it as I've never had occasion to use OpenSearch)
** if this works, great, just upgrade the library
** if this highlights connectivity problems, raise a ticket/change for
implementing an OpenSearch-based implementation of the
{{ElasticsearchClientService}} interface in NiFi (which will likely result in
some refactoring of common functionality between the two {{Impl}})
was (Author: chris s):
[~joewitt] I agree completely.
There was an email from some AWS engineers a long time ago about contributing
an OpenSearch capability, but that hasn't materialised unfortuantely.
I think a sensible thing would be to:
* test the existing {{ElasticsearchClientServiceImpl}} against OpenSearch to
see whether it works
* upgrade the Elasticsearch low-level REST client library to the latest
(currently {{8.13.3}}) and re-test the OpenSearch connectivity (it is assumed
from some reading/online articles that this may not work, but I for one have
never confirmed it as I've never had occasion to use OpenSearch)
** if this works, great, just upgrade the library
** if this highlights connectivity problems, raise a ticket/change for
implementing an OpenSearch-based implementation of the
{{ElasticsearchClientService}} interface in NiFi (which will likely result in
some refactoring of common functionality between the two {{Impl}}s)
> Update elasticsearch.client.version to 7.17.10
> --
>
> Key: NIFI-11686
> URL: https://issues.apache.org/jira/browse/NIFI-11686
> Project: Apache NiFi
> Issue Type: Improvement
>Affects Versions: 1.22.0
>Reporter: Mike R
>Assignee: Jeyassri Balachandran
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Update elasticsearch.client.version to 7.17.10
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-11686) Update elasticsearch.client.version to 7.17.10
[
https://issues.apache.org/jira/browse/NIFI-11686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844380#comment-17844380
]
Chris Sampson commented on NIFI-11686:
--
[~joewitt] I agree completely.
There was an email from some AWS engineers a long time ago about contributing
an OpenSearch capability, but that hasn't materialised unfortuantely.
I think a sensible thing would be to:
* test the existing {{ElasticsearchClientServiceImpl}} against OpenSearch to
see whether it works
* upgrade the Elasticsearch low-level REST client library to the latest
(currently {{8.13.3}}) and re-test the OpenSearch connectivity (it is assumed
from some reading/online articles that this may not work, but I for one have
never confirmed it as I've never had occasion to use OpenSearch)
** if this works, great, just upgrade the library
** if this highlights connectivity problems, raise a ticket/change for
implementing an OpenSearch-based implementation of the
{{ElasticsearchClientService}} interface in NiFi (which will likely result in
some refactoring of common functionality between the two {{Impl}}s)
> Update elasticsearch.client.version to 7.17.10
> --
>
> Key: NIFI-11686
> URL: https://issues.apache.org/jira/browse/NIFI-11686
> Project: Apache NiFi
> Issue Type: Improvement
>Affects Versions: 1.22.0
>Reporter: Mike R
>Assignee: Jeyassri Balachandran
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Update elasticsearch.client.version to 7.17.10
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-13144) Improve dependency-update detection configuration and bump various detected dependency versions
[
https://issues.apache.org/jira/browse/NIFI-13144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson updated NIFI-13144:
-
Fix Version/s: 2.0.0-M3
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Improve dependency-update detection configuration and bump various detected
> dependency versions
> ---
>
> Key: NIFI-13144
> URL: https://issues.apache.org/jira/browse/NIFI-13144
> Project: Apache NiFi
> Issue Type: Task
>Reporter: Joe Witt
>Assignee: Joe Witt
>Priority: Major
> Fix For: 2.0.0-M3
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> {noformat}
> com.puppycrawl.tools:checkstyle ... 10.15.0 -> 10.16.0
> software.amazon.awssdk:xray ... 2.25.40 -> 2.25.45
> com.amazonaws:aws-java-sdk-keyspaces 1.12.710 -> 1.12.715
> com.azure:azure-sdk-bom 1.2.21 -> 1.2.23
> com.google.cloud:libraries-bom ... 26.37.0 -> 26.38.0
> net.bytebuddy:byte-buddy .. 1.14.12 -> 1.14.14
> org.checkerframework:checker-qual ... 3.37.0 -> 3.43.0
> org.eclipse.jdt:ecj . 3.33.0 -> 3.37.0
> com.microsoft.azure:msal4j .. 1.14.3 -> 1.15.0
> com.google.guava:guava .. 33.1.0-jre -> 33.2.0-jre
> com.ibm.icu:icu4j ... 74.2 -> 75.1
> org.opensaml:opensaml-* 4.3.1 -> 4.3.2
> org.apache.sshd:sshd-* ... 2.12.0 -> 2.12.1
> com.nimbusds:oauth2-oidc-sdk ... 9.43.3 -> 9.43.4
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-13139) PythonNarIT tests failing, missing StandardPythonBridge
[ https://issues.apache.org/jira/browse/NIFI-13139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Sampson updated NIFI-13139: - Fix Version/s: 2.0.0-M3 Resolution: Fixed Status: Resolved (was: Patch Available) > PythonNarIT tests failing, missing StandardPythonBridge > --- > > Key: NIFI-13139 > URL: https://issues.apache.org/jira/browse/NIFI-13139 > Project: Apache NiFi > Issue Type: Bug >Reporter: Tamas Palfy >Assignee: Joe Witt >Priority: Major > Fix For: 2.0.0-M3 > > Time Spent: 0.5h > Remaining Estimate: 0h > > The issue seems to be due to > nifi-system-tests/nifi-system-test-suite/target/pythonic-instance/lib not > prepared to have the nifi-py4j-nar-2.0.0-SNAPSHOT.nar. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-12891) ElasticSearchClientService fails config verification with Nullpointer before first enable
[ https://issues.apache.org/jira/browse/NIFI-12891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Sampson resolved NIFI-12891. -- Fix Version/s: 2.0.0 Resolution: Fixed > ElasticSearchClientService fails config verification with Nullpointer before > first enable > - > > Key: NIFI-12891 > URL: https://issues.apache.org/jira/browse/NIFI-12891 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Bob Paulin >Priority: Minor > Fix For: 2.0.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > If a user clicks the "Verify Properties" button before the > ElasticSearchClientService Controller Service is first enabled it fails to > verify the connection. This is due to a null pointer that occurs because > both the responseEncoding and objectmapper is not set in the verify method > before the test. This gives the user the incorrect error that the connection > failed even though the connection would have succeed. > > Recommend managing these variables in the verify method. > > ``` > 2024-03-12 15:36:39,437 WARN [Verify Controller Service Config Thread-1] > o.a.n.e.ElasticSearchClientServiceImpl > ElasticSearchClientServiceImpl[id=46f88c3f-53ce-3fb6-6014-995ad23028de] > Unable to connect to Elasticsearch > org.apache.nifi.elasticsearch.ElasticsearchException: > java.lang.NullPointerException > at > org.apache.nifi.elasticsearch.ElasticSearchClientServiceImpl.parseResponse(ElasticSearchClientServiceImpl.java:566) > at > org.apache.nifi.elasticsearch.ElasticSearchClientServiceImpl.getElasticsearchRoot(ElasticSearchClientServiceImpl.java:329) > at > org.apache.nifi.elasticsearch.ElasticSearchClientServiceImpl.verifyRootConnection(ElasticSearchClientServiceImpl.java:336) > at > org.apache.nifi.elasticsearch.ElasticSearchClientServiceImpl.verify(ElasticSearchClientServiceImpl.java:243) > at > org.apache.nifi.controller.service.StandardControllerServiceNode.verifyConfiguration(StandardControllerServiceNode.java:531) > at > org.apache.nifi.web.dao.impl.StandardControllerServiceDAO.verifyConfiguration(StandardControllerServiceDAO.java:433) > at > java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) > at java.base/java.lang.reflect.Method.invoke(Method.java:580) > at > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) > at > org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:751) > at > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) > at > org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:751) > at > org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) > at > org.apache.nifi.web.dao.impl.StandardControllerServiceDAO$$SpringCGLIB$$0.verifyConfiguration() > at > org.apache.nifi.web.StandardNiFiServiceFacade.performControllerServiceConfigVerification(StandardNiFiServiceFacade.java:2945) > at > java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) > at java.base/java.lang.reflect.Method.invoke(Method.java:580) > at > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) > at > org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:751) > at > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) > at > org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:751) > at > org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) > at > org.apache.nifi.web.StandardNiFiServiceFacade$$SpringCGLIB$$0.performControllerServiceConfigVerification() >
[jira] [Updated] (NIFI-12891) ElasticSearchClientService fails config verification with Nullpointer before first enable
[ https://issues.apache.org/jira/browse/NIFI-12891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Sampson updated NIFI-12891: - Affects Version/s: 2.0.0-M2 (was: 2.0.0) > ElasticSearchClientService fails config verification with Nullpointer before > first enable > - > > Key: NIFI-12891 > URL: https://issues.apache.org/jira/browse/NIFI-12891 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 2.0.0-M2 >Reporter: Bob Paulin >Priority: Minor > Fix For: 2.0.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > If a user clicks the "Verify Properties" button before the > ElasticSearchClientService Controller Service is first enabled it fails to > verify the connection. This is due to a null pointer that occurs because > both the responseEncoding and objectmapper is not set in the verify method > before the test. This gives the user the incorrect error that the connection > failed even though the connection would have succeed. > > Recommend managing these variables in the verify method. > > ``` > 2024-03-12 15:36:39,437 WARN [Verify Controller Service Config Thread-1] > o.a.n.e.ElasticSearchClientServiceImpl > ElasticSearchClientServiceImpl[id=46f88c3f-53ce-3fb6-6014-995ad23028de] > Unable to connect to Elasticsearch > org.apache.nifi.elasticsearch.ElasticsearchException: > java.lang.NullPointerException > at > org.apache.nifi.elasticsearch.ElasticSearchClientServiceImpl.parseResponse(ElasticSearchClientServiceImpl.java:566) > at > org.apache.nifi.elasticsearch.ElasticSearchClientServiceImpl.getElasticsearchRoot(ElasticSearchClientServiceImpl.java:329) > at > org.apache.nifi.elasticsearch.ElasticSearchClientServiceImpl.verifyRootConnection(ElasticSearchClientServiceImpl.java:336) > at > org.apache.nifi.elasticsearch.ElasticSearchClientServiceImpl.verify(ElasticSearchClientServiceImpl.java:243) > at > org.apache.nifi.controller.service.StandardControllerServiceNode.verifyConfiguration(StandardControllerServiceNode.java:531) > at > org.apache.nifi.web.dao.impl.StandardControllerServiceDAO.verifyConfiguration(StandardControllerServiceDAO.java:433) > at > java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) > at java.base/java.lang.reflect.Method.invoke(Method.java:580) > at > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) > at > org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:751) > at > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) > at > org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:751) > at > org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) > at > org.apache.nifi.web.dao.impl.StandardControllerServiceDAO$$SpringCGLIB$$0.verifyConfiguration() > at > org.apache.nifi.web.StandardNiFiServiceFacade.performControllerServiceConfigVerification(StandardNiFiServiceFacade.java:2945) > at > java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) > at java.base/java.lang.reflect.Method.invoke(Method.java:580) > at > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) > at > org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:751) > at > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) > at > org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:751) > at > org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) > at >
[jira] [Updated] (NIFI-12773) Add 'join' and 'anchored' RecordPath functions
[
https://issues.apache.org/jira/browse/NIFI-12773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson updated NIFI-12773:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Add 'join' and 'anchored' RecordPath functions
> --
>
> Key: NIFI-12773
> URL: https://issues.apache.org/jira/browse/NIFI-12773
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 2.0.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> I've come across two functions that would make flow design much simpler in
> RecordPath.
> The first one, 'join' would be similar to the 'concat' method but provides a
> delimiter between each element instead of just smashing the values together.
> The other provides the ability to anchor the context node while evaluating a
> RecordPath. For example, given the following record:
> {code:java}
> {
> "id": "1234",
> "elements": [{
> "name": "book",
> "color": "red"
> }, {
> "name": "computer",
> "color": "black"
> }]
> } {code}
> We should be able to use:
> {code:java}
> anchored(/elements, concat(/name, ': ', /color)) {code}
> In order to obtain an array of 2 elements:
> {code:java}
> book: red {code}
> and
> {code:java}
> computer: black {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (NIFI-12337) GitHub Action Workflows could better report test results and summaries
Chris Sampson created NIFI-12337:
Summary: GitHub Action Workflows could better report test results
and summaries
Key: NIFI-12337
URL: https://issues.apache.org/jira/browse/NIFI-12337
Project: Apache NiFi
Issue Type: Improvement
Reporter: Chris Sampson
Fix For: 2.latest
Several of the GitHub Action Workflows produce {{JUnit}} XML result output
files through {{surefire}} and {{failsafe}} plugins. These are currently
uploaded as raw logs/XML files for people to download from GitHub, but it
should be possible to upload these results as {{checks}} within GitHub instead,
which would be easier for people to consume and understand what tests are
failing in their PRs.
One possible action to use would be
[test-reporter|https://github.com/marketplace/actions/test-reporter], although
an attempt to use this in NIFI-12177 as part of the new {{integration-tests}}
Workflow failed because it seems we're unable to {{write}} to the GitHub repo
(even with {code:yaml}permissions.checks: write{code} set within the Workflow
definition). This might be an intentional setting, and could require discussion
with the ASF Infrastructure team to understand whether this is/could be
permitted.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12192) GitHub Action Workflows should report Code Coverage for CI Unit Tests
[ https://issues.apache.org/jira/browse/NIFI-12192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Sampson updated NIFI-12192: - Fix Version/s: 2.0.0 (was: 2.latest) Resolution: Fixed Status: Resolved (was: Patch Available) > GitHub Action Workflows should report Code Coverage for CI Unit Tests > - > > Key: NIFI-12192 > URL: https://issues.apache.org/jira/browse/NIFI-12192 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Chris Sampson >Assignee: David Handermann >Priority: Minor > Fix For: 2.0.0 > > Time Spent: 2h > Remaining Estimate: 0h > > Consider adding test coverage reporting, at least for one of the > ci-integration jobs, see tools such as: > * https://github.com/marketplace/actions/jacoco-report > * https://github.com/codecov/codecov-action (see > https://github.com/exceptionfactory/jagged for example implementation) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12192) GitHub Action Workflows should report Code Coverage for CI Unit Tests
[
https://issues.apache.org/jira/browse/NIFI-12192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson updated NIFI-12192:
-
Summary: GitHub Action Workflows should report Code Coverage for CI Unit
Tests (was: GitHub Action Workflows could better report Tests Results and
Summaries)
> GitHub Action Workflows should report Code Coverage for CI Unit Tests
> -
>
> Key: NIFI-12192
> URL: https://issues.apache.org/jira/browse/NIFI-12192
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Chris Sampson
>Assignee: David Handermann
>Priority: Minor
> Fix For: 2.latest
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Several of the GitHub Action Workflows produce {{JUnit}} XML result output
> files through {{surefire}} and {{failsafe}} plugins. These are currently
> uploaded as raw logs/XML files for people to download from GitHub, but it
> should be possible to upload these results as {{checks}} within GitHub
> instead, which would be easier for people to consume and understand what
> tests are failing in their PRs.
> One possible action to use would be
> [test-reporter|https://github.com/marketplace/actions/test-reporter],
> although an attempt to use this in NIFI-12177 as part of the new
> {{integration-tests}} Workflow failed because it seems we're unable to
> {{write}} to the GitHub repo (even with {code:yaml}permissions.checks:
> write{code} set within the Workflow definition). This might be an intentional
> setting, and could require discussion with the ASF Infrastructure team to
> understand whether this is/could be permitted.
> Also consider adding test coverage reporting, at least for one of the
> ci-integration jobs, see tools such as:
> * https://github.com/marketplace/actions/jacoco-report
> * https://github.com/codecov/codecov-action (see
> https://github.com/exceptionfactory/jagged for example implementation)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12192) GitHub Action Workflows should report Code Coverage for CI Unit Tests
[
https://issues.apache.org/jira/browse/NIFI-12192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson updated NIFI-12192:
-
Description:
Consider adding test coverage reporting, at least for one of the ci-integration
jobs, see tools such as:
* https://github.com/marketplace/actions/jacoco-report
* https://github.com/codecov/codecov-action (see
https://github.com/exceptionfactory/jagged for example implementation)
was:
Several of the GitHub Action Workflows produce {{JUnit}} XML result output
files through {{surefire}} and {{failsafe}} plugins. These are currently
uploaded as raw logs/XML files for people to download from GitHub, but it
should be possible to upload these results as {{checks}} within GitHub instead,
which would be easier for people to consume and understand what tests are
failing in their PRs.
One possible action to use would be
[test-reporter|https://github.com/marketplace/actions/test-reporter], although
an attempt to use this in NIFI-12177 as part of the new {{integration-tests}}
Workflow failed because it seems we're unable to {{write}} to the GitHub repo
(even with {code:yaml}permissions.checks: write{code} set within the Workflow
definition). This might be an intentional setting, and could require discussion
with the ASF Infrastructure team to understand whether this is/could be
permitted.
Also consider adding test coverage reporting, at least for one of the
ci-integration jobs, see tools such as:
* https://github.com/marketplace/actions/jacoco-report
* https://github.com/codecov/codecov-action (see
https://github.com/exceptionfactory/jagged for example implementation)
> GitHub Action Workflows should report Code Coverage for CI Unit Tests
> -
>
> Key: NIFI-12192
> URL: https://issues.apache.org/jira/browse/NIFI-12192
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Chris Sampson
>Assignee: David Handermann
>Priority: Minor
> Fix For: 2.latest
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Consider adding test coverage reporting, at least for one of the
> ci-integration jobs, see tools such as:
> * https://github.com/marketplace/actions/jacoco-report
> * https://github.com/codecov/codecov-action (see
> https://github.com/exceptionfactory/jagged for example implementation)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12192) GitHub Action Workflows could better report Tests Results and Summaries
[
https://issues.apache.org/jira/browse/NIFI-12192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson updated NIFI-12192:
-
Description:
Several of the GitHub Action Workflows produce {{JUnit}} XML result output
files through {{surefire}} and {{failsafe}} plugins. These are currently
uploaded as raw logs/XML files for people to download from GitHub, but it
should be possible to upload these results as {{checks}} within GitHub instead,
which would be easier for people to consume and understand what tests are
failing in their PRs.
One possible action to use would be
[test-reporter|https://github.com/marketplace/actions/test-reporter], although
an attempt to use this in NIFI-12177 as part of the new {{integration-tests}}
Workflow failed because it seems we're unable to {{write}} to the GitHub repo
(even with {code:yaml}permissions.checks: write{code} set within the Workflow
definition). This might be an intentional setting, and could require discussion
with the ASF Infrastructure team to understand whether this is/could be
permitted.
Also consider adding test coverage reporting, at least for one of the
ci-integration jobs, see tools such as:
* https://github.com/marketplace/actions/jacoco-report
* https://github.com/codecov/codecov-action (see
https://github.com/exceptionfactory/jagged for example implementation)
was:
Several of the GitHub Action Workflows produce {{JUnit}} XML result output
files through {{surefire}} and {{failsafe}} plugins. These are currently
uploaded as raw logs/XML files for people to download from GitHub, but it
should be possible to upload these results as {{checks}} within GitHub instead,
which would be easier for people to consume and understand what tests are
failing in their PRs.
One possible action to use would be
[test-reporter|https://github.com/marketplace/actions/test-reporter], although
an attempt to use this in NIFI-12177 as part of the new {{integration-tests}}
Workflow failed because it seems we're unable to {{write}} to the GitHub repo
(even with {code:yaml}permissions.checks: write{code} set within the Workflow
definition). This might be an intentional setting, and could require discussion
with the ASF Infrastructure team to understand whether this is/could be
permitted.
Also consider adding test coverage reporting, at least for one of the
ci-integration jobs - https://github.com/marketplace/actions/jacoco-report
> GitHub Action Workflows could better report Tests Results and Summaries
> ---
>
> Key: NIFI-12192
> URL: https://issues.apache.org/jira/browse/NIFI-12192
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Minor
> Fix For: 2.latest
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Several of the GitHub Action Workflows produce {{JUnit}} XML result output
> files through {{surefire}} and {{failsafe}} plugins. These are currently
> uploaded as raw logs/XML files for people to download from GitHub, but it
> should be possible to upload these results as {{checks}} within GitHub
> instead, which would be easier for people to consume and understand what
> tests are failing in their PRs.
> One possible action to use would be
> [test-reporter|https://github.com/marketplace/actions/test-reporter],
> although an attempt to use this in NIFI-12177 as part of the new
> {{integration-tests}} Workflow failed because it seems we're unable to
> {{write}} to the GitHub repo (even with {code:yaml}permissions.checks:
> write{code} set within the Workflow definition). This might be an intentional
> setting, and could require discussion with the ASF Infrastructure team to
> understand whether this is/could be permitted.
> Also consider adding test coverage reporting, at least for one of the
> ci-integration jobs, see tools such as:
> * https://github.com/marketplace/actions/jacoco-report
> * https://github.com/codecov/codecov-action (see
> https://github.com/exceptionfactory/jagged for example implementation)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (NIFI-12192) GitHub Action Workflows could better report Tests Results and Summaries
[
https://issues.apache.org/jira/browse/NIFI-12192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson reassigned NIFI-12192:
Assignee: Chris Sampson
> GitHub Action Workflows could better report Tests Results and Summaries
> ---
>
> Key: NIFI-12192
> URL: https://issues.apache.org/jira/browse/NIFI-12192
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Minor
> Fix For: 2.latest
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Several of the GitHub Action Workflows produce {{JUnit}} XML result output
> files through {{surefire}} and {{failsafe}} plugins. These are currently
> uploaded as raw logs/XML files for people to download from GitHub, but it
> should be possible to upload these results as {{checks}} within GitHub
> instead, which would be easier for people to consume and understand what
> tests are failing in their PRs.
> One possible action to use would be
> [test-reporter|https://github.com/marketplace/actions/test-reporter],
> although an attempt to use this in NIFI-12177 as part of the new
> {{integration-tests}} Workflow failed because it seems we're unable to
> {{write}} to the GitHub repo (even with {code:yaml}permissions.checks:
> write{code} set within the Workflow definition). This might be an intentional
> setting, and could require discussion with the ASF Infrastructure team to
> understand whether this is/could be permitted.
> Also consider adding test coverage reporting, at least for one of the
> ci-integration jobs - https://github.com/marketplace/actions/jacoco-report
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12299) Integration Tests should be triggered if pom.xml is updated
[
https://issues.apache.org/jira/browse/NIFI-12299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson updated NIFI-12299:
-
Status: Patch Available (was: In Progress)
> Integration Tests should be triggered if pom.xml is updated
> ---
>
> Key: NIFI-12299
> URL: https://issues.apache.org/jira/browse/NIFI-12299
> Project: Apache NiFi
> Issue Type: Task
>Affects Versions: 2.0.0
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Minor
> Fix For: 2.latest
>
>
> The {{integration-tests}} GitHub Actions workflow should be triggered if the
> top-level {{pom.xml}} is updated as part of a PR.
> This ensures the ITs are exercised if wide-ranging dependencies are updated
> and/or the {{integration-tests-ci}} list is modified.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (NIFI-12299) Integration Tests should be triggered if pom.xml is updated
Chris Sampson created NIFI-12299:
Summary: Integration Tests should be triggered if pom.xml is
updated
Key: NIFI-12299
URL: https://issues.apache.org/jira/browse/NIFI-12299
Project: Apache NiFi
Issue Type: Task
Affects Versions: 2.0.0
Reporter: Chris Sampson
Fix For: 2.latest
The {{integration-tests}} GitHub Actions workflow should be triggered if the
top-level {{pom.xml}} is updated as part of a PR.
This ensures the ITs are exercised if wide-ranging dependencies are updated
and/or the {{integration-tests-ci}} list is modified.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (NIFI-12299) Integration Tests should be triggered if pom.xml is updated
[
https://issues.apache.org/jira/browse/NIFI-12299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson reassigned NIFI-12299:
Assignee: Chris Sampson
> Integration Tests should be triggered if pom.xml is updated
> ---
>
> Key: NIFI-12299
> URL: https://issues.apache.org/jira/browse/NIFI-12299
> Project: Apache NiFi
> Issue Type: Task
>Affects Versions: 2.0.0
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Minor
> Fix For: 2.latest
>
>
> The {{integration-tests}} GitHub Actions workflow should be triggered if the
> top-level {{pom.xml}} is updated as part of a PR.
> This ensures the ITs are exercised if wide-ranging dependencies are updated
> and/or the {{integration-tests-ci}} list is modified.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12190) Fix failing Integration Tests
[
https://issues.apache.org/jira/browse/NIFI-12190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson updated NIFI-12190:
-
Description:
NIFI-12177 adds an {{integration-tests}} GitHub Actions Workflow to regularly
run the {{integration-tests}} defined throughout the NiFi repo. However, many
tests have had to be skipped from the {{maven-failsafe plugin}} execution due
to failures within the GitHub runner.
Typically these are caused by:
* the need for external resources to be setup
* incorrect assertions that might have worked during the original IT creation
but have not been updated since
* or missing/incompatible OS resources within the GitHub runner
* missing/misconfigured dependencies needed to execute the integration tests
This Epic is to review the failing ITs within NiFi codebase and either:
* fix them, e.g. correct assertions, and/or by using
[Testcontainers|https://testcontainers.com/] for hosting external resources,
such as for the Elasticsearch Processors
* remove them if no longer useful/cannot be reliably executed on
developer's/GitHub runners
* annotate them to be skipped in CI environments if the tests are useful and
work on a developer machine but won't be able to work in a CI runner, e.g. a
[DisabledIfEnvironmentVariable|https://junit.org/junit5/docs/5.7.0/api/org.junit.jupiter.api/org/junit/jupiter/api/condition/DisabledIfEnvironmentVariable.html]
to check whether the {{CI}} env var is present and set to {{true}}
Once fixed, the appropriate entry from the top-level {{pom.xml}}'s
{{integration-tests-ci}} profile configuration for the
{{maven-failsafe-plugin}} **must** be removed.
At the time of writing, the ITs known to be failing (either in part or as a
whole) are:
{quote}
!AwsSecretsManagerSensitivePropertyProviderIT,
!AwsKmsSensitivePropertyProviderIT,
!AzureKeyVaultKeySensitivePropertyProviderIT,
!GcpKmsSensitivePropertyProviderIT,
!StandardHashiCorpVaultCommunicationServiceIT,
!ConversionWithSchemaInferenceIT#testXmlToAvro,
!ConversionWithSchemaInferenceIT#testXmlToJson,
!ConversionWithSchemaInferenceIT#testJsonToCsv,
!ConversionWithSchemaInferenceIT#testCsvToXml,
!ConversionWithSchemaInferenceIT#testXmlToAvroToXml,
!ConversionWithSchemaInferenceIT#testCsvToAvro,
!ConversionWithSchemaInferenceIT#testCsvToJson,
!ConversionWithSchemaInferenceIT#testJsonToAvroToJson,
!ConversionWithSchemaInferenceIT#testCsvToAvroToCsv,
!ConversionWithExplicitSchemaIT#testJsonToCsv,
!ConversionWithExplicitSchemaIT#testJsonToAvro,
!ConversionWithExplicitSchemaIT#testJsonToAvroToJson,
!ITConsumeKinesisStreamEndpointOverride,
!ITConsumeKinesisStreamConnectAWS,
!ITPutKinesisStream,
!ITPutKinesisStreamWithEndpointOverride#testIntegrationSuccess,
!ITPutKinesisFirehoseWithEndpointOverride#testIntegrationSuccess,
!ITPutKinesisFirehose,
!ITTagS3Object,
!ITFetchS3Object,
!ITDeleteS3Object,
!ITPutS3Object,
!ITListS3,
!ITPutSNS#testPublish,
!ITPutSNS#testPublishWithCredentialsProviderService,
!ITDeleteSQS,
!ITGetSQS,
!ITPutSQS,
!ITPutLambda,
!GetMongoIT#testDatabaseEL,
!PutGridFSIT#testFileNameAndHashUniqueness,
!PutGridFSIT#testHashUniqueness,
!ITPutAzureCosmosDBRecord,
!ITListAzureBlobStorage_v12,
!ITPutAzureDataLakeStorage,
!ITFetchAzureDataLakeStorage,
!ITListAzureDataLakeStorage,
!ITFetchAzureBlobStorage_v12,
!ITPutAzureBlobStorage_v12,
!ITDeleteAzureDataLakeStorage,
!ITDeleteAzureBlobStorage_v12,
!ITMoveAzureDataLakeStorage,
[jira] [Updated] (NIFI-12192) GitHub Action Workflows could better report Tests Results and Summaries
[
https://issues.apache.org/jira/browse/NIFI-12192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson updated NIFI-12192:
-
Description:
Several of the GitHub Action Workflows produce {{JUnit}} XML result output
files through {{surefire}} and {{failsafe}} plugins. These are currently
uploaded as raw logs/XML files for people to download from GitHub, but it
should be possible to upload these results as {{checks}} within GitHub instead,
which would be easier for people to consume and understand what tests are
failing in their PRs.
One possible action to use would be
[test-reporter|https://github.com/marketplace/actions/test-reporter], although
an attempt to use this in NIFI-12177 as part of the new {{integration-tests}}
Workflow failed because it seems we're unable to {{write}} to the GitHub repo
(even with {code:yaml}permissions.checks: write{code} set within the Workflow
definition). This might be an intentional setting, and could require discussion
with the ASF Infrastructure team to understand whether this is/could be
permitted.
Also consider adding test coverage reporting, at least for one of the
ci-integration jobs - https://github.com/marketplace/actions/jacoco-report
was:
Several of the GitHub Action Workflows produce {{JUnit}} XML result output
files through {{surefire}} and {{failsafe}} plugins. These are currently
uploaded as raw logs/XML files for people to download from GitHub, but it
should be possible to upload these results as {{checks}} within GitHub instead,
which would be easier for people to consume and understand what tests are
failing in their PRs.
One possible action to use would be
[test-reporter|https://github.com/marketplace/actions/test-reporter], although
an attempt to use this in NIFI-12177 as part of the new {{integration-tests}}
Workflow failed because it seems we're unable to {{write}} to the GitHub repo
(even with {code:yaml}permissions.checks: write{code} set within the Workflow
definition). This might be an intentional setting, and could require discussion
with the ASF Infrastructure team to understand whether this is/could be
permitted.
> GitHub Action Workflows could better report Tests Results and Summaries
> ---
>
> Key: NIFI-12192
> URL: https://issues.apache.org/jira/browse/NIFI-12192
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Chris Sampson
>Priority: Minor
> Fix For: 2.latest
>
>
> Several of the GitHub Action Workflows produce {{JUnit}} XML result output
> files through {{surefire}} and {{failsafe}} plugins. These are currently
> uploaded as raw logs/XML files for people to download from GitHub, but it
> should be possible to upload these results as {{checks}} within GitHub
> instead, which would be easier for people to consume and understand what
> tests are failing in their PRs.
> One possible action to use would be
> [test-reporter|https://github.com/marketplace/actions/test-reporter],
> although an attempt to use this in NIFI-12177 as part of the new
> {{integration-tests}} Workflow failed because it seems we're unable to
> {{write}} to the GitHub repo (even with {code:yaml}permissions.checks:
> write{code} set within the Workflow definition). This might be an intentional
> setting, and could require discussion with the ASF Infrastructure team to
> understand whether this is/could be permitted.
> Also consider adding test coverage reporting, at least for one of the
> ci-integration jobs - https://github.com/marketplace/actions/jacoco-report
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (NIFI-12255) PutElasticsearch* processor relationships should be renamed
[
https://issues.apache.org/jira/browse/NIFI-12255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson reassigned NIFI-12255:
Assignee: Chris Sampson
> PutElasticsearch* processor relationships should be renamed
> ---
>
> Key: NIFI-12255
> URL: https://issues.apache.org/jira/browse/NIFI-12255
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Minor
> Fix For: 2.latest
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The {{PutElasticsearchRecord}} (and possibly {{PutElasticsearchJson}}) should
> have their relationships renamed to make them clearer for users, for example:
> * failure
> * -original (was success)- - I'm unsure on this
> * successful (i.e. Records that processed in Elasticsearch without error)
> * errors (i.e. Records that resulted in an error within Elasticsearch)
> * error_responses (new for 1.24; the error responses received from
> Elasticsearch, if any)
> The {{Result Record Writer}} should also be made *mandatory* so that
> {{error}} responses from Elasticsearch are always checked and processed to
> send Records to {{successful}} or {{errors}} relationships as appropriate
> (other properties may also want updating/removing to make the UX of these
> processors simpler in this area)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12279) Add Standard Shared NAR to System Test Modules
[
https://issues.apache.org/jira/browse/NIFI-12279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson updated NIFI-12279:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Add Standard Shared NAR to System Test Modules
> --
>
> Key: NIFI-12279
> URL: https://issues.apache.org/jira/browse/NIFI-12279
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework, Tools and Build
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Blocker
> Fix For: 2.0.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Recent changes introducing the {{nifi-standard-shared-nar}} did not include
> updating several system test modules, resulting in failures when running the
> PythonProcessorIT and Stateless CreateFlowFileIT. The Shared NAR needs to be
> added to the list of scoped dependencies for system tests to correct these
> test failures.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12278) Add GitHub CodeQL Scanning to Static Analysis Job
[
https://issues.apache.org/jira/browse/NIFI-12278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson updated NIFI-12278:
-
Fix Version/s: 2.0.0
(was: 2.latest)
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Add GitHub CodeQL Scanning to Static Analysis Job
> -
>
> Key: NIFI-12278
> URL: https://issues.apache.org/jira/browse/NIFI-12278
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Tools and Build
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
> Fix For: 2.0.0
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> [GitHub
> CodeQL|https://docs.github.com/en/code-security/code-scanning/introduction-to-code-scanning/about-code-scanning-with-codeql]
> scanning is available for public repositories and provides a default set of
> rules for common coding security vulnerabilities.
> The {{ci-workflow}} Static Analysis Job should be updated to include CodeQL
> scanning actions to detect potential concerns during the build process.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12275) Disable SFTP Keepalive in Test Classes
[ https://issues.apache.org/jira/browse/NIFI-12275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Sampson updated NIFI-12275: - Fix Version/s: 2.0.0 1.24.0 (was: 1.latest) (was: 2.latest) Resolution: Fixed Status: Resolved (was: Patch Available) > Disable SFTP Keepalive in Test Classes > -- > > Key: NIFI-12275 > URL: https://issues.apache.org/jira/browse/NIFI-12275 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: David Handermann >Assignee: David Handermann >Priority: Major > Fix For: 2.0.0, 1.24.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Standard SFTP Processors have a Connection Keepalive property that is enabled > in the default configuration. This property should be disabled for unit tests > which have inherently short connection lifespans. Disabling the Keepalive > setting simplifies the initial client configuration handshake process and > could avoid some potential failures. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12272) Add Standard Certificate Principal Formatting
[
https://issues.apache.org/jira/browse/NIFI-12272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson updated NIFI-12272:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Add Standard Certificate Principal Formatting
> -
>
> Key: NIFI-12272
> URL: https://issues.apache.org/jira/browse/NIFI-12272
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework, Extensions
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
> Fix For: 2.0.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> The {{X509Certificate.getSubjectDN()}} is denigrated in favor of
> {{getSubjectX500Principal()}} and recent improvements to the main branch
> replaced legacy usage.
> The getSubjectDN method returns Distinguished Names formatted according to
> RFC 1779, which includes separating Relative Distinguished Name elements
> using spaces. This impacts configuration properties such as Identity Mapping,
> and also impacts flow designs that evaluate Certificate Subject and Issuer
> attributes.
> The default behavior of {{X500Principal.getName()}} formats Distinguished
> Names according to RFC 2253, which does not use space separators. Passing
> {{X500Principal.RFC1779}} to the {{getName()}} method follows the same
> formatting approach as {{{}getSubjectDN().getName(){}}}.
> Existing behavior should be maintained to avoid unexpected changes when
> upgrading NiFi versions, and a new utility should be introduced to provide
> standardized formatting.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12268) Upgrade OkHttp to 4.12.0
[ https://issues.apache.org/jira/browse/NIFI-12268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Sampson updated NIFI-12268: - Resolution: Fixed Status: Resolved (was: Patch Available) > Upgrade OkHttp to 4.12.0 > > > Key: NIFI-12268 > URL: https://issues.apache.org/jira/browse/NIFI-12268 > Project: Apache NiFi > Issue Type: Improvement >Reporter: David Handermann >Assignee: David Handermann >Priority: Major > Labels: dependency-upgrade > Fix For: 2.0.0, 1.24.0 > > Time Spent: 20m > Remaining Estimate: 0h > > OkHttp > [4.12.0|https://square.github.io/okhttp/changelogs/changelog_4x/#version-4120] > includes several minor bug fixes for connection handling and other issues. > Transitive dependencies on Okio and Kotlin should also be upgraded. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12262) Remove Unused Groovy Dependency Definitions
[ https://issues.apache.org/jira/browse/NIFI-12262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Sampson updated NIFI-12262: - Resolution: Fixed Status: Resolved (was: Patch Available) > Remove Unused Groovy Dependency Definitions > --- > > Key: NIFI-12262 > URL: https://issues.apache.org/jira/browse/NIFI-12262 > Project: Apache NiFi > Issue Type: Improvement > Components: Tools and Build >Reporter: David Handermann >Assignee: David Handermann >Priority: Minor > Fix For: 2.0.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Following the removal of Groovy-based tests and components, several Maven > configuration elements reference Groovy repositories and dependency versions > should be removed from the build. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12191) Simplify Integraton Test & Docker Test action workflows - MacOS GitHub Runner does not correctly run Docker (Colima)
[
https://issues.apache.org/jira/browse/NIFI-12191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson updated NIFI-12191:
-
Description:
NIFI-12177 introduced the {{integration-tests}} and {{docker-tests}} GitHub
Action Workflows, which runs fine on {{Ubuntu}}, but cannot run as successfully
on {{MacOS}}.
This is because the MacOS runner **does not** contain Docker by default, and
installing it via {{brew install docker}} actually installs
[Colima|https://github.com/abiosoft/colima]. While this allows Docker Images to
be built and (to a degree) run as Containers, it has been noted that
Testcontainers **do not** work correctly with such a setup, and some of the
Docker Image Tests (e.g. for NiFi - see NIFI-12177) fail because containers
don't run in a consistent manner compared to Ubuntu with `docker` installed.
-This ticket is to investigate and fix the use of Docker within the MacOS based
GitHub runner for these workflows.- After some investigation and
[discussion|https://github.com/apache/nifi/pull/7920], it has been decided that
the potential benefits of running the tests on MacOS as well as Ubuntu are not
sufficient to justify the extra resource cost of the GitHub Runners. Instead,
it has been decided to simplify the Workflows added in NIFI-12177 to only run
{{integration-tests}} and {{docker-tests}} on Ubuntu-based GitHub Action
Runners.
was:
NIFI-12177 introduced the {{integration-tests}} and {{docker-tests}} GitHub
Action Workflows, which runs fine on {{Ubuntu}}, but cannot run as successfully
on {{MacOS}}.
This is because the MacOS runner **does not** contain Docker by default, and
installing it via {{brew install docker}} actually installs
[Colima|https://github.com/abiosoft/colima]. While this allows Docker Images to
be built and (to a degree) run as Containers, it has been noted that
Testcontainers **do not** work correctly with such a setup, and some of the
Docker Image Tests (e.g. for NiFi - see NIFI-12177) fail because containers
don't run in a consistent manner compared to Ubuntu with `docker` installed.
~~This ticket is to investigate and fix the use of Docker within the MacOS
based GitHub runner for these workflows.~~ After some investigation and
[discussion|https://github.com/apache/nifi/pull/7920], it has been decided that
the potential benefits of running the tests on MacOS as well as Ubuntu are not
sufficient to justify the extra resource cost of the GitHub Runners. Instead,
it has been decided to simplify the Workflows added in NIFI-12177 to only run
{{integration-tests}} and {{docker-tests}} on Ubuntu-based GitHub Action
Runners.
> Simplify Integraton Test & Docker Test action workflows - MacOS GitHub Runner
> does not correctly run Docker (Colima)
>
>
> Key: NIFI-12191
> URL: https://issues.apache.org/jira/browse/NIFI-12191
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Major
> Fix For: 2.latest
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> NIFI-12177 introduced the {{integration-tests}} and {{docker-tests}} GitHub
> Action Workflows, which runs fine on {{Ubuntu}}, but cannot run as
> successfully on {{MacOS}}.
> This is because the MacOS runner **does not** contain Docker by default, and
> installing it via {{brew install docker}} actually installs
> [Colima|https://github.com/abiosoft/colima]. While this allows Docker Images
> to be built and (to a degree) run as Containers, it has been noted that
> Testcontainers **do not** work correctly with such a setup, and some of the
> Docker Image Tests (e.g. for NiFi - see NIFI-12177) fail because containers
> don't run in a consistent manner compared to Ubuntu with `docker` installed.
> -This ticket is to investigate and fix the use of Docker within the MacOS
> based GitHub runner for these workflows.- After some investigation and
> [discussion|https://github.com/apache/nifi/pull/7920], it has been decided
> that the potential benefits of running the tests on MacOS as well as Ubuntu
> are not sufficient to justify the extra resource cost of the GitHub Runners.
> Instead, it has been decided to simplify the Workflows added in NIFI-12177 to
> only run {{integration-tests}} and {{docker-tests}} on Ubuntu-based GitHub
> Action Runners.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12191) Simplify Integraton Test & Docker Test action workflows - MacOS GitHub Runner does not correctly run Docker (Colima)
[
https://issues.apache.org/jira/browse/NIFI-12191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson updated NIFI-12191:
-
Description:
NIFI-12177 introduced the {{integration-tests}} and {{docker-tests}} GitHub
Action Workflows, which runs fine on {{Ubuntu}}, but cannot run as successfully
on {{MacOS}}.
This is because the MacOS runner **does not** contain Docker by default, and
installing it via {{brew install docker}} actually installs
[Colima|https://github.com/abiosoft/colima]. While this allows Docker Images to
be built and (to a degree) run as Containers, it has been noted that
Testcontainers **do not** work correctly with such a setup, and some of the
Docker Image Tests (e.g. for NiFi - see NIFI-12177) fail because containers
don't run in a consistent manner compared to Ubuntu with `docker` installed.
~~This ticket is to investigate and fix the use of Docker within the MacOS
based GitHub runner for these workflows.~~ After some investigation and
[discussion|https://github.com/apache/nifi/pull/7920], it has been decided that
the potential benefits of running the tests on MacOS as well as Ubuntu are not
sufficient to justify the extra resource cost of the GitHub Runners. Instead,
it has been decided to simplify the Workflows added in NIFI-12177 to only run
{{integration-tests}} and {{docker-tests}} on Ubuntu-based GitHub Action
Runners.
was:
NIFI-12177 introduced the {{integration-tests}} and {{docker-tests}} GitHub
Action Workflows, which runs fine on {{Ubuntu}}, but cannot run as successfully
on {{MacOS}}.
This is because the MacOS runner **does not** contain Docker by default, and
installing it via {{brew install docker}} actually installs
[Colima|https://github.com/abiosoft/colima]. While this allows Docker Images to
be built and (to a degree) run as Containers, it has been noted that
Testcontainers **do not** work correctly with such a setup, and some of the
Docker Image Tests (e.g. for NiFi - see NIFI-12177) fail because containers
don't run in a consistent manner compared to Ubuntu with `docker` installed.
This ticket is to investigate and fix the use of Docker within the MacOS based
GitHub runner for these workflows.
> Simplify Integraton Test & Docker Test action workflows - MacOS GitHub Runner
> does not correctly run Docker (Colima)
>
>
> Key: NIFI-12191
> URL: https://issues.apache.org/jira/browse/NIFI-12191
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Major
> Fix For: 2.latest
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> NIFI-12177 introduced the {{integration-tests}} and {{docker-tests}} GitHub
> Action Workflows, which runs fine on {{Ubuntu}}, but cannot run as
> successfully on {{MacOS}}.
> This is because the MacOS runner **does not** contain Docker by default, and
> installing it via {{brew install docker}} actually installs
> [Colima|https://github.com/abiosoft/colima]. While this allows Docker Images
> to be built and (to a degree) run as Containers, it has been noted that
> Testcontainers **do not** work correctly with such a setup, and some of the
> Docker Image Tests (e.g. for NiFi - see NIFI-12177) fail because containers
> don't run in a consistent manner compared to Ubuntu with `docker` installed.
> ~~This ticket is to investigate and fix the use of Docker within the MacOS
> based GitHub runner for these workflows.~~ After some investigation and
> [discussion|https://github.com/apache/nifi/pull/7920], it has been decided
> that the potential benefits of running the tests on MacOS as well as Ubuntu
> are not sufficient to justify the extra resource cost of the GitHub Runners.
> Instead, it has been decided to simplify the Workflows added in NIFI-12177 to
> only run {{integration-tests}} and {{docker-tests}} on Ubuntu-based GitHub
> Action Runners.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12191) Simplify Integraton Test & Docker Test action workflows - MacOS GitHub Runner does not correctly run Docker (Colima)
[
https://issues.apache.org/jira/browse/NIFI-12191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson updated NIFI-12191:
-
Summary: Simplify Integraton Test & Docker Test action workflows - MacOS
GitHub Runner does not correctly run Docker (Colima) (was: MacOS GitHub Runner
does not correctly run Docker (Colima))
> Simplify Integraton Test & Docker Test action workflows - MacOS GitHub Runner
> does not correctly run Docker (Colima)
>
>
> Key: NIFI-12191
> URL: https://issues.apache.org/jira/browse/NIFI-12191
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Major
> Fix For: 2.latest
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> NIFI-12177 introduced the {{integration-tests}} and {{docker-tests}} GitHub
> Action Workflows, which runs fine on {{Ubuntu}}, but cannot run as
> successfully on {{MacOS}}.
> This is because the MacOS runner **does not** contain Docker by default, and
> installing it via {{brew install docker}} actually installs
> [Colima|https://github.com/abiosoft/colima]. While this allows Docker Images
> to be built and (to a degree) run as Containers, it has been noted that
> Testcontainers **do not** work correctly with such a setup, and some of the
> Docker Image Tests (e.g. for NiFi - see NIFI-12177) fail because containers
> don't run in a consistent manner compared to Ubuntu with `docker` installed.
> This ticket is to investigate and fix the use of Docker within the MacOS
> based GitHub runner for these workflows.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12191) MacOS GitHub Runner does not correctly run Docker (Colima)
[
https://issues.apache.org/jira/browse/NIFI-12191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson updated NIFI-12191:
-
Status: Patch Available (was: In Progress)
> MacOS GitHub Runner does not correctly run Docker (Colima)
> --
>
> Key: NIFI-12191
> URL: https://issues.apache.org/jira/browse/NIFI-12191
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Major
> Fix For: 2.latest
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> NIFI-12177 introduced the {{integration-tests}} and {{docker-tests}} GitHub
> Action Workflows, which runs fine on {{Ubuntu}}, but cannot run as
> successfully on {{MacOS}}.
> This is because the MacOS runner **does not** contain Docker by default, and
> installing it via {{brew install docker}} actually installs
> [Colima|https://github.com/abiosoft/colima]. While this allows Docker Images
> to be built and (to a degree) run as Containers, it has been noted that
> Testcontainers **do not** work correctly with such a setup, and some of the
> Docker Image Tests (e.g. for NiFi - see NIFI-12177) fail because containers
> don't run in a consistent manner compared to Ubuntu with `docker` installed.
> This ticket is to investigate and fix the use of Docker within the MacOS
> based GitHub runner for these workflows.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12177) Docker builds (dockermaven) should include testing of the built images
[
https://issues.apache.org/jira/browse/NIFI-12177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson updated NIFI-12177:
-
Status: Patch Available (was: In Progress)
> Docker builds (dockermaven) should include testing of the built images
> --
>
> Key: NIFI-12177
> URL: https://issues.apache.org/jira/browse/NIFI-12177
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Major
> Fix For: 1.latest, 2.latest
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The NiFi, NiFi Toolkit & NiFi Registry {{dockermaven}} builds currently run
> an {{integration_test}} script once the image has been built in order to
> confirm a container can be started and the component can be contacted.
> The same (or similar) approaches should be considered for:
> * MiNiFi
> * MiNiFi C2
> This would help avoid situations such as NIFI-12175
> Also, the NiFi `dockermaven` test script fails if run with `macos-latest` in
> GitHub Actions (NIFI-12178), likely an incompatibility with the use of Colima
> to get Docker working within the runner, an alternative should be found (to
> the test assertions or the GitHub workflow - see NIFI-12191 as this is
> possibly an issue with the use of Colima)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12254) PutElasticsearchRecord documentation for Dynamic Properties isn't clear
[
https://issues.apache.org/jira/browse/NIFI-12254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson updated NIFI-12254:
-
Status: Patch Available (was: Open)
> PutElasticsearchRecord documentation for Dynamic Properties isn't clear
> ---
>
> Key: NIFI-12254
> URL: https://issues.apache.org/jira/browse/NIFI-12254
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.23.2
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Minor
> Fix For: 1.latest, 2.latest
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The Dynamic Property documentation for {{PutElasticsearchRecord}} doesn't
> indicate that the values of the properties are expected to be Record Path
> expressions in order to obtain values from within the Records processed for
> sending to Elasticsearch.
> This is very confusing for users.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (NIFI-12254) PutElasticsearchRecord documentation for Dynamic Properties isn't clear
[
https://issues.apache.org/jira/browse/NIFI-12254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson reassigned NIFI-12254:
Assignee: Chris Sampson
> PutElasticsearchRecord documentation for Dynamic Properties isn't clear
> ---
>
> Key: NIFI-12254
> URL: https://issues.apache.org/jira/browse/NIFI-12254
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.23.2
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Minor
> Fix For: 1.latest, 2.latest
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The Dynamic Property documentation for {{PutElasticsearchRecord}} doesn't
> indicate that the values of the properties are expected to be Record Path
> expressions in order to obtain values from within the Records processed for
> sending to Elasticsearch.
> This is very confusing for users.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (NIFI-12255) PutElasticsearch* processor relationships should be renamed
Chris Sampson created NIFI-12255:
Summary: PutElasticsearch* processor relationships should be
renamed
Key: NIFI-12255
URL: https://issues.apache.org/jira/browse/NIFI-12255
Project: Apache NiFi
Issue Type: Improvement
Reporter: Chris Sampson
Fix For: 2.latest
The {{PutElasticsearchRecord}} (and possibly {{PutElasticsearchJson}}) should
have their relationships renamed to make them clearer for users, for example:
* failure
* -original (was success)- - I'm unsure on this
* successful (i.e. Records that processed in Elasticsearch without error)
* errors (i.e. Records that resulted in an error within Elasticsearch)
* error_responses (new for 1.24; the error responses received from
Elasticsearch, if any)
The {{Result Record Writer}} should also be made *mandatory* so that {{error}}
responses from Elasticsearch are always checked and processed to send Records
to {{successful}} or {{errors}} relationships as appropriate (other properties
may also want updating/removing to make the UX of these processors simpler in
this area)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (NIFI-12254) PutElasticsearchRecord documentation for Dynamic Properties isn't clear
Chris Sampson created NIFI-12254:
Summary: PutElasticsearchRecord documentation for Dynamic
Properties isn't clear
Key: NIFI-12254
URL: https://issues.apache.org/jira/browse/NIFI-12254
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 1.23.2
Reporter: Chris Sampson
Fix For: 1.latest, 2.latest
The Dynamic Property documentation for {{PutElasticsearchRecord}} doesn't
indicate that the values of the properties are expected to be Record Path
expressions in order to obtain values from within the Records processed for
sending to Elasticsearch.
This is very confusing for users.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-8135) DataTypeUtils conversion methods for Records to java Maps should respect Choice types
[ https://issues.apache.org/jira/browse/NIFI-8135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Sampson updated NIFI-8135: Fix Version/s: 2.0.0 1.latest > DataTypeUtils conversion methods for Records to java Maps should respect > Choice types > - > > Key: NIFI-8135 > URL: https://issues.apache.org/jira/browse/NIFI-8135 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.11.4 >Reporter: Chris Sampson >Assignee: Chris Sampson >Priority: Major > Fix For: 2.0.0, 1.latest > > Time Spent: 40m > Remaining Estimate: 0h > > Where a schema field is a Map of Choice, with options of Array (of Strings), > Map (of Strings) or String, Record Readers incorrectly deserialise the data > of the nested Map/Array as Records containing no schema. When converting the > overall Record into a java Map using DataTypeUtils, the nested MapRecord is > not converted into a Map. > This can lead to data loss between readers & writers. > This appears to be related to one/more of: > * missing schema in the nested MapRecord object (even if it's presented with > the data to the Record Reader) - the conversion methods could infer a schema > for such MapRecords rather than effectively skipping them > * Choice types don't appear to be respected for nested fields (e.g. checking > whether the options within a Choice are for Map or Array, not just whether > the field dataType passed to the conversion method is for Map or Array, etc. > ... it's a Choice that should allow for these) > * Array/ Map conversion should not assume all elements within the collection > are of the same type, especially if the field's type is Choice -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-8135) DataTypeUtils conversion methods for Records to java Maps should respect Choice types
[ https://issues.apache.org/jira/browse/NIFI-8135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Sampson updated NIFI-8135: Fix Version/s: 2.latest (was: 2.0.0) > DataTypeUtils conversion methods for Records to java Maps should respect > Choice types > - > > Key: NIFI-8135 > URL: https://issues.apache.org/jira/browse/NIFI-8135 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.11.4 >Reporter: Chris Sampson >Assignee: Chris Sampson >Priority: Major > Fix For: 1.latest, 2.latest > > Time Spent: 40m > Remaining Estimate: 0h > > Where a schema field is a Map of Choice, with options of Array (of Strings), > Map (of Strings) or String, Record Readers incorrectly deserialise the data > of the nested Map/Array as Records containing no schema. When converting the > overall Record into a java Map using DataTypeUtils, the nested MapRecord is > not converted into a Map. > This can lead to data loss between readers & writers. > This appears to be related to one/more of: > * missing schema in the nested MapRecord object (even if it's presented with > the data to the Record Reader) - the conversion methods could infer a schema > for such MapRecords rather than effectively skipping them > * Choice types don't appear to be respected for nested fields (e.g. checking > whether the options within a Choice are for Map or Array, not just whether > the field dataType passed to the conversion method is for Map or Array, etc. > ... it's a Choice that should allow for these) > * Array/ Map conversion should not assume all elements within the collection > are of the same type, especially if the field's type is Choice -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-12177) Docker builds (dockermaven) should include testing of the built images
[
https://issues.apache.org/jira/browse/NIFI-12177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson reassigned NIFI-12177:
Assignee: Chris Sampson
> Docker builds (dockermaven) should include testing of the built images
> --
>
> Key: NIFI-12177
> URL: https://issues.apache.org/jira/browse/NIFI-12177
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Major
> Fix For: 1.latest, 2.latest
>
>
> The NiFi, NiFi Toolkit & NiFi Registry {{dockermaven}} builds currently run
> an {{integration_test}} script once the image has been built in order to
> confirm a container can be started and the component can be contacted.
> The same (or similar) approaches should be considered for:
> * MiNiFi
> * MiNiFi C2
> This would help avoid situations such as NIFI-12175
> Also, the NiFi `dockermaven` test script fails if run with `macos-latest` in
> GitHub Actions (NIFI-12178), likely an incompatibility with the use of Colima
> to get Docker working within the runner, an alternative should be found (to
> the test assertions or the GitHub workflow - see NIFI-12191 as this is
> possibly an issue with the use of Colima)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (NIFI-12191) MacOS GitHub Runner does not correctly run Docker (Colima)
[
https://issues.apache.org/jira/browse/NIFI-12191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson reassigned NIFI-12191:
Assignee: Chris Sampson
> MacOS GitHub Runner does not correctly run Docker (Colima)
> --
>
> Key: NIFI-12191
> URL: https://issues.apache.org/jira/browse/NIFI-12191
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Major
> Fix For: 2.latest
>
>
> NIFI-12177 introduced the {{integration-tests}} and {{docker-tests}} GitHub
> Action Workflows, which runs fine on {{Ubuntu}}, but cannot run as
> successfully on {{MacOS}}.
> This is because the MacOS runner **does not** contain Docker by default, and
> installing it via {{brew install docker}} actually installs
> [Colima|https://github.com/abiosoft/colima]. While this allows Docker Images
> to be built and (to a degree) run as Containers, it has been noted that
> Testcontainers **do not** work correctly with such a setup, and some of the
> Docker Image Tests (e.g. for NiFi - see NIFI-12177) fail because containers
> don't run in a consistent manner compared to Ubuntu with `docker` installed.
> This ticket is to investigate and fix the use of Docker within the MacOS
> based GitHub runner for these workflows.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (NIFI-7194) Record Path (and Readers) should be able to select/filter on fields within arrays
[
https://issues.apache.org/jira/browse/NIFI-7194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17775068#comment-17775068
]
Chris Sampson edited comment on NIFI-7194 at 10/13/23 9:19 PM:
---
Partially resolved by NIFI-12124
was (Author: chris s):
Partially resolved by NIFI-12194
> Record Path (and Readers) should be able to select/filter on fields within
> arrays
> -
>
> Key: NIFI-7194
> URL: https://issues.apache.org/jira/browse/NIFI-7194
> Project: Apache NiFi
> Issue Type: Improvement
>Affects Versions: 1.11.1
>Reporter: Chris Sampson
>Priority: Minor
>
> Given a sample input like
> {code:json}
> {
> "field": "value",
> "a_date": "2020-02-01",
> "ids": [
> {
> "id": "1",
> "id_date": "2020-02-24"
> },
> {
> "id": "2",
> "id_date": "2020-02-23"
> }
> ],
> "another": {
> "some_date": "2020-02-02"
> }
> }
> {code}
> It would be useful if one could use NiFi's Record Path to generically
> reference fields nested within arrays as well as top-level/nested objects in
> order to make updates to their values, e.g. as part of a call to the
> UpdateRecord processor.
> For example, to reference all non-array fields whose name contains `date`,
> one could currently use: ` //*[matchesRegex(fieldName(.),
> '(^|.+_)date(_.+|$)')]`; however there is no equivalent for referencing the
> date fields within the arrays.
> Such a RecordPath could look like `//[0..-1]/*[matchesRegex(fieldName(.),
> '(^|.+_)date(_.+|$)')]`, however this is currently marked as invalid by the
> UpdateRecord processor validation (and presumably therefore doesn't work).
> One tricky addition to this would be how to handle multiple levels of nesting
> of arrays/objects and how much descending through the Record tree should be
> possible.
> An addition to this would be to allow for matching of Map fields too, e.g.
> with something like `//[]/*[matchesRegex(fieldName(.),
> '(^|.+_)date(_.+|$)')]`.
> An example use-case for this is where incoming data structure is not
> set/known and may contain arrays (or even maps) with date fields where the
> flow is trying to support multiple date formats (e.g. -MM-dd and
> dd/MM/) but wants to harmonise all dates to a single format for further
> processing (e.g. dd/MM/).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-7194) Record Path (and Readers) should be able to select/filter on fields within arrays
[
https://issues.apache.org/jira/browse/NIFI-7194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17775068#comment-17775068
]
Chris Sampson commented on NIFI-7194:
-
Partially resolved by NIFI-12194
> Record Path (and Readers) should be able to select/filter on fields within
> arrays
> -
>
> Key: NIFI-7194
> URL: https://issues.apache.org/jira/browse/NIFI-7194
> Project: Apache NiFi
> Issue Type: Improvement
>Affects Versions: 1.11.1
>Reporter: Chris Sampson
>Priority: Minor
>
> Given a sample input like
> {code:json}
> {
> "field": "value",
> "a_date": "2020-02-01",
> "ids": [
> {
> "id": "1",
> "id_date": "2020-02-24"
> },
> {
> "id": "2",
> "id_date": "2020-02-23"
> }
> ],
> "another": {
> "some_date": "2020-02-02"
> }
> }
> {code}
> It would be useful if one could use NiFi's Record Path to generically
> reference fields nested within arrays as well as top-level/nested objects in
> order to make updates to their values, e.g. as part of a call to the
> UpdateRecord processor.
> For example, to reference all non-array fields whose name contains `date`,
> one could currently use: ` //*[matchesRegex(fieldName(.),
> '(^|.+_)date(_.+|$)')]`; however there is no equivalent for referencing the
> date fields within the arrays.
> Such a RecordPath could look like `//[0..-1]/*[matchesRegex(fieldName(.),
> '(^|.+_)date(_.+|$)')]`, however this is currently marked as invalid by the
> UpdateRecord processor validation (and presumably therefore doesn't work).
> One tricky addition to this would be how to handle multiple levels of nesting
> of arrays/objects and how much descending through the Record tree should be
> possible.
> An addition to this would be to allow for matching of Map fields too, e.g.
> with something like `//[]/*[matchesRegex(fieldName(.),
> '(^|.+_)date(_.+|$)')]`.
> An example use-case for this is where incoming data structure is not
> set/known and may contain arrays (or even maps) with date fields where the
> flow is trying to support multiple date formats (e.g. -MM-dd and
> dd/MM/) but wants to harmonise all dates to a single format for further
> processing (e.g. dd/MM/).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-12191) MacOS GitHub Runner does not correctly run Docker (Colima)
[
https://issues.apache.org/jira/browse/NIFI-12191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17774687#comment-17774687
]
Chris Sampson commented on NIFI-12191:
--
It might simply be a docker/colima config problem, see
https://golang.testcontainers.org/system_requirements/using_colima/ for some
suggestions.
Other things to consider are whether colima needs to be configured to use more
resources within the runner -
https://github.com/abiosoft/colima#customizing-the-vm, or use containerd
runtime - https://github.com/abiosoft/colima#containerd
> MacOS GitHub Runner does not correctly run Docker (Colima)
> --
>
> Key: NIFI-12191
> URL: https://issues.apache.org/jira/browse/NIFI-12191
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Chris Sampson
>Priority: Major
> Fix For: 2.latest
>
>
> NIFI-12177 introduced the {{integration-tests}} and {{docker-tests}} GitHub
> Action Workflows, which runs fine on {{Ubuntu}}, but cannot run as
> successfully on {{MacOS}}.
> This is because the MacOS runner **does not** contain Docker by default, and
> installing it via {{brew install docker}} actually installs
> [Colima|https://github.com/abiosoft/colima]. While this allows Docker Images
> to be built and (to a degree) run as Containers, it has been noted that
> Testcontainers **do not** work correctly with such a setup, and some of the
> Docker Image Tests (e.g. for NiFi - see NIFI-12177) fail because containers
> don't run in a consistent manner compared to Ubuntu with `docker` installed.
> This ticket is to investigate and fix the use of Docker within the MacOS
> based GitHub runner for these workflows.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12177) Docker builds (dockermaven) should include testing of the built images
[
https://issues.apache.org/jira/browse/NIFI-12177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson updated NIFI-12177:
-
Description:
The NiFi, NiFi Toolkit & NiFi Registry {{dockermaven}} builds currently run an
{{integration_test}} script once the image has been built in order to confirm a
container can be started and the component can be contacted.
The same (or similar) approaches should be considered for:
* MiNiFi
* MiNiFi C2
This would help avoid situations such as NIFI-12175
Also, the NiFi `dockermaven` test script fails if run with `macos-latest` in
GitHub Actions (NIFI-12178), likely an incompatibility with the use of Colima
to get Docker working within the runner, an alternative should be found (to the
test assertions or the GitHub workflow - see NIFI-12191 as this is possibly an
issue with the use of Colima)
was:
The NiFi & NiFi Registry {{dockermaven}} builds currently run an
{{integration_test}} script once the image has been built in order to confirm a
container can be started and the component can be contacted.
The same (or similar) approaches should be considered for:
* NiFi Toolkit
* MiNiFi
* MiNiFi C2
This would help avoid situations such as NIFI-12175
Also, the NiFi `dockermaven` test script fails if run with `macos-latest` in
GitHub Actions (NIFI-12178), likely an incompatibility with the use of Colima
to get Docker working within the runner, an alternative should be found (to the
test assertions or the GitHub workflow - see NIFI-12191 as this is possibly an
issue with the use of Colima)
> Docker builds (dockermaven) should include testing of the built images
> --
>
> Key: NIFI-12177
> URL: https://issues.apache.org/jira/browse/NIFI-12177
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Chris Sampson
>Priority: Major
> Fix For: 1.latest, 2.latest
>
>
> The NiFi, NiFi Toolkit & NiFi Registry {{dockermaven}} builds currently run
> an {{integration_test}} script once the image has been built in order to
> confirm a container can be started and the component can be contacted.
> The same (or similar) approaches should be considered for:
> * MiNiFi
> * MiNiFi C2
> This would help avoid situations such as NIFI-12175
> Also, the NiFi `dockermaven` test script fails if run with `macos-latest` in
> GitHub Actions (NIFI-12178), likely an incompatibility with the use of Colima
> to get Docker working within the runner, an alternative should be found (to
> the test assertions or the GitHub workflow - see NIFI-12191 as this is
> possibly an issue with the use of Colima)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-10841) Enable dockermaven builds for arm64 architectures
[
https://issues.apache.org/jira/browse/NIFI-10841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17774679#comment-17774679
]
Chris Sampson commented on NIFI-10841:
--
[~kdoran] I think this is fixed now, do you agree? I've recently been
successfully building images on arm64 & amd64 based machine as part of
NIFI-12175 & NIFI-12178
The only issue I've seen has been in running the {{apache/nifi}} image as part
of a {{macos}} runner in github, but the build seems to work fine (NIFI-12191)
> Enable dockermaven builds for arm64 architectures
> -
>
> Key: NIFI-10841
> URL: https://issues.apache.org/jira/browse/NIFI-10841
> Project: Apache NiFi
> Issue Type: Improvement
>Affects Versions: 1.18.0
>Reporter: Chris Sampson
>Priority: Major
>
> The dockermaven builds (e.g. nifi, nifi-toolkit, nifi-registry, minifi,
> minifi-c2) currently do not work on arm64 architecture machines.
> This is due to a known limitation of the Spotify docker build plugin used in
> the maven profiles.
> An alternative (e.g. fabric8) should allow builds on both amd64 and arm64
> architectures (and possibly others).
> NIFI-9177 allowed the docker image (e.g. produced using the dockerhub
> modules) to run on arm64 machines, so that's not a problem to fix, just the
> actual dockermaven module builds (and tests).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12177) Docker builds (dockermaven) should include testing of the built images
[
https://issues.apache.org/jira/browse/NIFI-12177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson updated NIFI-12177:
-
Description:
The NiFi & NiFi Registry {{dockermaven}} builds currently run an
{{integration_test}} script once the image has been built in order to confirm a
container can be started and the component can be contacted.
The same (or similar) approaches should be considered for:
* NiFi Toolkit
* MiNiFi
* MiNiFi C2
This would help avoid situations such as NIFI-12175
Also, the NiFi `dockermaven` test script fails if run with `macos-latest` in
GitHub Actions (NIFI-12178), likely an incompatibility with the use of Colima
to get Docker working within the runner, an alternative should be found (to the
test assertions or the GitHub workflow - see NIFI-12191 as this is possibly an
issue with the use of Colima)
was:
The NiFi & NiFi Registry {{dockermaven}} builds currently run an
{{integration_test}} script once the image has been built in order to confirm a
container can be started and the component can be contacted.
The same (or similar) approaches should be considered for:
* NiFi Toolkit
* MiNiFi
* MiNiFi C2
This would help avoid situations such as NIFI-12175
Also, the NiFi `dockermaven` test script fails if run with `macos-latest` in
GitHub Actions (NIFI-12178), likely an incompatibility with the use of Colima
to get Docker working within the runner, an alternative should be found (to the
test assertions or the GitHub workflow).
> Docker builds (dockermaven) should include testing of the built images
> --
>
> Key: NIFI-12177
> URL: https://issues.apache.org/jira/browse/NIFI-12177
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Chris Sampson
>Priority: Major
> Fix For: 1.latest, 2.latest
>
>
> The NiFi & NiFi Registry {{dockermaven}} builds currently run an
> {{integration_test}} script once the image has been built in order to confirm
> a container can be started and the component can be contacted.
> The same (or similar) approaches should be considered for:
> * NiFi Toolkit
> * MiNiFi
> * MiNiFi C2
> This would help avoid situations such as NIFI-12175
> Also, the NiFi `dockermaven` test script fails if run with `macos-latest` in
> GitHub Actions (NIFI-12178), likely an incompatibility with the use of Colima
> to get Docker working within the runner, an alternative should be found (to
> the test assertions or the GitHub workflow - see NIFI-12191 as this is
> possibly an issue with the use of Colima)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12175) Update all component Docker Image builds to java 21
[
https://issues.apache.org/jira/browse/NIFI-12175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson updated NIFI-12175:
-
Status: Patch Available (was: In Progress)
> Update all component Docker Image builds to java 21
> ---
>
> Key: NIFI-12175
> URL: https://issues.apache.org/jira/browse/NIFI-12175
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Major
> Fix For: 2.latest
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> All nifi components (nifi, nifi-registry, nifi-toolkit, minifi, minifi-c2)
> come with dockerhub and dockermaven modules to build Docker Images. The
> former builds an image using the source code released to the Apache servers,
> the latter uses the binary files from the component's assembly module one it
> has been built locally.
> Nifi has already been updated. Nifi Registry's dockermaven has been updated.
> Other image builds are outstanding:
> * Nifi registry (dockerhub, found in nifi-registry-core/nifi-registry-docker)
> * Nifi toolkit (both)
> * minifi (both)
> * minifi-c2 (both)
> Note that the {{docker}} maven profile must be enabled to trigger the build
> of the Docker Images during {{install}} (although some builds only need
> {{package}} instead, confusingly).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (NIFI-12175) Update all component Docker Image builds to java 21
[
https://issues.apache.org/jira/browse/NIFI-12175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson reassigned NIFI-12175:
Assignee: Chris Sampson
> Update all component Docker Image builds to java 21
> ---
>
> Key: NIFI-12175
> URL: https://issues.apache.org/jira/browse/NIFI-12175
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Major
> Fix For: 2.latest
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> All nifi components (nifi, nifi-registry, nifi-toolkit, minifi, minifi-c2)
> come with dockerhub and dockermaven modules to build Docker Images. The
> former builds an image using the source code released to the Apache servers,
> the latter uses the binary files from the component's assembly module one it
> has been built locally.
> Nifi has already been updated. Nifi Registry's dockermaven has been updated.
> Other image builds are outstanding:
> * Nifi registry (dockerhub, found in nifi-registry-core/nifi-registry-docker)
> * Nifi toolkit (both)
> * minifi (both)
> * minifi-c2 (both)
> Note that the {{docker}} maven profile must be enabled to trigger the build
> of the Docker Images during {{install}} (although some builds only need
> {{package}} instead, confusingly).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12214) ConsumeElasticsearch doc generation broken
[
https://issues.apache.org/jira/browse/NIFI-12214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson updated NIFI-12214:
-
Status: Patch Available (was: In Progress)
> ConsumeElasticsearch doc generation broken
> --
>
> Key: NIFI-12214
> URL: https://issues.apache.org/jira/browse/NIFI-12214
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.24.0
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Minor
> Fix For: 1.latest, 2.latest
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The NiFi DocGenerator is throwing a WARNing during startup when processing
> the {{ConsumeElasticsearch}} properties.
> {code:java}
> java.io.IOException: Unable to create XMLOutputStream
> at
> org.apache.nifi.documentation.html.HtmlDocumentationWriter.write(HtmlDocumentationWriter.java:102)
> at
> org.apache.nifi.documentation.DocGenerator.document(DocGenerator.java:142)
> at
> org.apache.nifi.documentation.DocGenerator.documentConfigurableComponent(DocGenerator.java:107)
> at
> org.apache.nifi.documentation.DocGenerator.generate(DocGenerator.java:64)
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:763)
> at org.apache.nifi.NiFi.(NiFi.java:172)
> at org.apache.nifi.NiFi.(NiFi.java:83)
> at org.apache.nifi.NiFi.main(NiFi.java:332)
> Caused by: javax.xml.stream.XMLStreamException: No property was found
> matching the name 'el-rest-query-definition-style'
> at
> org.apache.nifi.documentation.html.HtmlDocumentationWriter.writeProperties(HtmlDocumentationWriter.java:743)
> at
> org.apache.nifi.documentation.html.HtmlDocumentationWriter.writeBody(HtmlDocumentationWriter.java:166)
> at
> org.apache.nifi.documentation.html.HtmlDocumentationWriter.write(HtmlDocumentationWriter.java:98)
> ... 7 common frames omitted
> {code}
> This is due to the Query Builder properties inherrited from
> {{ElasticsearchRestProcessor}} that {{dependOn}} the
> {{QUERY_DEFINITION_STYLE}} property, which is omitted from
> {{ConsumeElasticsearch}} because it doesn't make sense to use (one must build
> the Elasticsearch {{query}} from the Query Builder properties rather than
> defining the whole {{query}}).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (NIFI-12214) ConsumeElasticsearch doc generation broken
[
https://issues.apache.org/jira/browse/NIFI-12214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson reassigned NIFI-12214:
Assignee: Chris Sampson
> ConsumeElasticsearch doc generation broken
> --
>
> Key: NIFI-12214
> URL: https://issues.apache.org/jira/browse/NIFI-12214
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.24.0
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Minor
> Fix For: 1.latest, 2.latest
>
>
> The NiFi DocGenerator is throwing a WARNing during startup when processing
> the {{ConsumeElasticsearch}} properties.
> {code:java}
> java.io.IOException: Unable to create XMLOutputStream
> at
> org.apache.nifi.documentation.html.HtmlDocumentationWriter.write(HtmlDocumentationWriter.java:102)
> at
> org.apache.nifi.documentation.DocGenerator.document(DocGenerator.java:142)
> at
> org.apache.nifi.documentation.DocGenerator.documentConfigurableComponent(DocGenerator.java:107)
> at
> org.apache.nifi.documentation.DocGenerator.generate(DocGenerator.java:64)
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:763)
> at org.apache.nifi.NiFi.(NiFi.java:172)
> at org.apache.nifi.NiFi.(NiFi.java:83)
> at org.apache.nifi.NiFi.main(NiFi.java:332)
> Caused by: javax.xml.stream.XMLStreamException: No property was found
> matching the name 'el-rest-query-definition-style'
> at
> org.apache.nifi.documentation.html.HtmlDocumentationWriter.writeProperties(HtmlDocumentationWriter.java:743)
> at
> org.apache.nifi.documentation.html.HtmlDocumentationWriter.writeBody(HtmlDocumentationWriter.java:166)
> at
> org.apache.nifi.documentation.html.HtmlDocumentationWriter.write(HtmlDocumentationWriter.java:98)
> ... 7 common frames omitted
> {code}
> This is due to the Query Builder properties inherrited from
> {{ElasticsearchRestProcessor}} that {{dependOn}} the
> {{QUERY_DEFINITION_STYLE}} property, which is omitted from
> {{ConsumeElasticsearch}} because it doesn't make sense to use (one must build
> the Elasticsearch {{query}} from the Query Builder properties rather than
> defining the whole {{query}}).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (NIFI-12214) ConsumeElasticsearch doc generation broken
Chris Sampson created NIFI-12214:
Summary: ConsumeElasticsearch doc generation broken
Key: NIFI-12214
URL: https://issues.apache.org/jira/browse/NIFI-12214
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 2.0.0, 1.24.0
Reporter: Chris Sampson
Fix For: 1.latest, 2.latest
The NiFi DocGenerator is throwing a WARNing during startup when processing the
{{ConsumeElasticsearch}} properties.
{code:java}
java.io.IOException: Unable to create XMLOutputStream
at
org.apache.nifi.documentation.html.HtmlDocumentationWriter.write(HtmlDocumentationWriter.java:102)
at
org.apache.nifi.documentation.DocGenerator.document(DocGenerator.java:142)
at
org.apache.nifi.documentation.DocGenerator.documentConfigurableComponent(DocGenerator.java:107)
at
org.apache.nifi.documentation.DocGenerator.generate(DocGenerator.java:64)
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:763)
at org.apache.nifi.NiFi.(NiFi.java:172)
at org.apache.nifi.NiFi.(NiFi.java:83)
at org.apache.nifi.NiFi.main(NiFi.java:332)
Caused by: javax.xml.stream.XMLStreamException: No property was found matching
the name 'el-rest-query-definition-style'
at
org.apache.nifi.documentation.html.HtmlDocumentationWriter.writeProperties(HtmlDocumentationWriter.java:743)
at
org.apache.nifi.documentation.html.HtmlDocumentationWriter.writeBody(HtmlDocumentationWriter.java:166)
at
org.apache.nifi.documentation.html.HtmlDocumentationWriter.write(HtmlDocumentationWriter.java:98)
... 7 common frames omitted
{code}
This is due to the Query Builder properties inherrited from
{{ElasticsearchRestProcessor}} that {{dependOn}} the {{QUERY_DEFINITION_STYLE}}
property, which is omitted from {{ConsumeElasticsearch}} because it doesn't
make sense to use (one must build the Elasticsearch {{query}} from the Query
Builder properties rather than defining the whole {{query}}).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-12178) Add integration-tests and docker-tests GithHub Action Workflows
[
https://issues.apache.org/jira/browse/NIFI-12178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17773145#comment-17773145
]
Chris Sampson commented on NIFI-12178:
--
I've set the new tests to run daily, like system tests, and on update to
specific files. But the frequency can certainly be adjusted.
I've been trying to run tests during development in my personal account (hoping
not to reach any limits), so as to reduce the overall apache load there too.
We could set the frequency to weekly, then increase later if that would be a
more sensible cadence?
> Add integration-tests and docker-tests GithHub Action Workflows
> ---
>
> Key: NIFI-12178
> URL: https://issues.apache.org/jira/browse/NIFI-12178
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Major
> Fix For: 2.latest
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> There are a lot of {{integration-tests}} and {{docker}} builds in the NiFi
> repository that aren't being regularly executed and verified.
> GitHub Action Workflows should be added (similar to the existing
> {{system-tests}} Workflow) that regularly exercise these tests, allowing the
> community to better monitor the health of NiFi components.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (NIFI-12192) GitHub Action Workflows could better report Tests Results and Summaries
Chris Sampson created NIFI-12192:
Summary: GitHub Action Workflows could better report Tests Results
and Summaries
Key: NIFI-12192
URL: https://issues.apache.org/jira/browse/NIFI-12192
Project: Apache NiFi
Issue Type: Improvement
Reporter: Chris Sampson
Fix For: 2.latest
Several of the GitHub Action Workflows produce {{JUnit}} XML result output
files through {{surefire}} and {{failsafe}} plugins. These are currently
uploaded as raw logs/XML files for people to download from GitHub, but it
should be possible to upload these results as {{checks}} within GitHub instead,
which would be easier for people to consume and understand what tests are
failing in their PRs.
One possible action to use would be
[test-reporter|https://github.com/marketplace/actions/test-reporter], although
an attempt to use this in NIFI-12177 as part of the new {{integration-tests}}
Workflow failed because it seems we're unable to {{write}} to the GitHub repo
(even with {code:yaml}permissions.checks: write{code} set within the Workflow
definition). This might be an intentional setting, and could require discussion
with the ASF Infrastructure team to understand whether this is/could be
permitted.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (NIFI-12189) Upgrade AWS Cloudwatch Processors to use AWS 2.x libraries
[ https://issues.apache.org/jira/browse/NIFI-12189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Sampson resolved NIFI-12189. -- Fix Version/s: 2.0.0 (was: 2.latest) Resolution: Fixed > Upgrade AWS Cloudwatch Processors to use AWS 2.x libraries > -- > > Key: NIFI-12189 > URL: https://issues.apache.org/jira/browse/NIFI-12189 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Joe Gresock >Assignee: Joe Gresock >Priority: Minor > Fix For: 2.0.0 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12190) Fix failing Integration Tests
[
https://issues.apache.org/jira/browse/NIFI-12190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17773022#comment-17773022
]
Chris Sampson commented on NIFI-12190:
--
NIFI-12144 will hopefully address a lot of the AWS-related ITs (although the
list in the {{integration-tests}} Workflow will need updating if [~markap14]'s
PR gets merged before NIFI-12178)
> Fix failing Integration Tests
> -
>
> Key: NIFI-12190
> URL: https://issues.apache.org/jira/browse/NIFI-12190
> Project: Apache NiFi
> Issue Type: Epic
>Reporter: Chris Sampson
>Priority: Major
> Fix For: 2.latest
>
>
> NIFI-12177 adds an {{integraiton-tests}} GitHub Actions Workflow to regularly
> run the {{integration-tests}} defined throughout the NiFi repo. However, many
> tests have had to be skipped from the {{failsafe plugin}} execution due to
> failures within the GitHub runner.
> Typically these are caused by:
> * the need for external resources to be setup
> * incorrect assertions that might have worked during the original IT creation
> but have not been updated since
> * or missing/incompatible OS resources within the GitHub runner
> This Epic is to review the failing ITs within NiFi codebase and either:
> * fix them, e.g. correct assertions, and/or by using Testcontainers for
> hosting external resources, such as for the Elasticsearch Processors
> * remove them if no longer useful/cannot be reliably executed on
> developer's/GitHub runners
> * annotate them to be skipped in CI environments if the tests are useful and
> work on a developer machine but won't be able to work in a CI runner, e.g. a
> [DisabledIfEnvironmentVariable|https://junit.org/junit5/docs/5.7.0/api/org.junit.jupiter.api/org/junit/jupiter/api/condition/DisabledIfEnvironmentVariable.html]
> to check whether the `CI` env var is present and set to `true`
> Once fixed, the appropriate entry from the {{integration-tests}} Workflow's
> {{MAVEN_FAILSAFE_SKIP_ITS}} env var **must** be removed
> At the time of writing, the ITs known to be failing (either in part or as a
> whole) are:
> {quote}!AwsSecretsManagerSensitivePropertyProviderIT,
> !AwsKmsSensitivePropertyProviderIT,
> !AzureKeyVaultKeySensitivePropertyProviderIT,
> !GcpKmsSensitivePropertyProviderIT,
> !StandardHashiCorpVaultCommunicationServiceIT,
> !ConversionWithSchemaInferenceIT#testXmlToAvro,
> !ConversionWithSchemaInferenceIT#testXmlToJson,
> !ConversionWithSchemaInferenceIT#testJsonToCsv,
> !ConversionWithSchemaInferenceIT#testCsvToXml,
> !ConversionWithSchemaInferenceIT#testXmlToAvroToXml,
> !ConversionWithSchemaInferenceIT#testCsvToAvro,
> !ConversionWithSchemaInferenceIT#testCsvToJson,
> !ConversionWithSchemaInferenceIT#testJsonToAvroToJson,
> !ConversionWithSchemaInferenceIT#testCsvToAvroToCsv,
> !ConversionWithExplicitSchemaIT#testJsonToCsv,
> !ConversionWithExplicitSchemaIT#testJsonToAvro,
> !ConversionWithExplicitSchemaIT#testJsonToAvroToJson,
> !ITConsumeKinesisStreamEndpointOverride,
> !ITConsumeKinesisStreamConnectAWS,
> !ITPutKinesisStream,
> !ITPutKinesisStreamWithEndpointOverride#testIntegrationSuccess,
> !ITPutKinesisFirehoseWithEndpointOverride#testIntegrationSuccess,
> !ITPutKinesisFirehose,
> !ITTagS3Object,
> !ITFetchS3Object,
> !ITDeleteS3Object,
> !ITPutS3Object,
> !ITListS3,
> !ITPutSNS#testPublish,
> !ITPutSNS#testPublishWithCredentialsProviderService,
> !ITDeleteSQS,
> !ITGetSQS,
> !ITPutSQS,
> !ITPutLambda,
> !GetMongoIT#testDatabaseEL,
> !PutGridFSIT#testFileNameAndHashUniqueness,
> !PutGridFSIT#testHashUniqueness,
> !ITPutAzureCosmosDBRecord,
> !ITListAzureBlobStorage_v12,
> !ITPutAzureDataLakeStorage,
> !ITFetchAzureDataLakeStorage,
> !ITListAzureDataLakeStorage,
> !ITFetchAzureBlobStorage_v12,
> !ITPutAzureBlobStorage_v12,
> !ITDeleteAzureDataLakeStorage,
> !ITDeleteAzureBlobStorage_v12,
> !ITMoveAzureDataLakeStorage,
> !AzureGraphUserGroupProviderIT,
> !JMSPublisherConsumerIT#validateMessageRedeliveryWhenNotAcked,
> !ITPutInfluxDB,
> !ITExecuteInfluxDBQuery,
> !GremlinClientServiceYamlSettingsAndBytecodeIT,
> !GremlinClientServiceControllerSettingsIT,
> !ITestConsumeEmail#validateUrl,
> !PrometheusReportingTaskIT#testNullLabel,
> !SnowflakeConnectionPoolIT,
> !SalesforceRestServiceIT,
> !QuerySalesforceObjectIT,
> !PutSalesforceObjectIT,
> !FetchDropboxIT,
> !PutDropboxIT,
> !ListDropboxIT,
> !FetchBoxFileIT,
> !PutBoxFileIT,
> !ListBoxFileIT,
> !JsonConfigBasedBoxClientServiceIT,
> !QueryIoTDBIT,
> !PutIoTDBRecordIT,
> !StandardProcessSessionIT#testCloneThenWriteCountsClaimReferencesProperly,
>
[jira] [Updated] (NIFI-12178) Add integration-tests and docker-tests GithHub Action Workflows
[
https://issues.apache.org/jira/browse/NIFI-12178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson updated NIFI-12178:
-
Status: Patch Available (was: In Progress)
> Add integration-tests and docker-tests GithHub Action Workflows
> ---
>
> Key: NIFI-12178
> URL: https://issues.apache.org/jira/browse/NIFI-12178
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Major
> Fix For: 2.latest
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> There are a lot of {{integration-tests}} and {{docker}} builds in the NiFi
> repository that aren't being regularly executed and verified.
> GitHub Action Workflows should be added (similar to the existing
> {{system-tests}} Workflow) that regularly exercise these tests, allowing the
> community to better monitor the health of NiFi components.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12190) Fix failing Integration Tests
[
https://issues.apache.org/jira/browse/NIFI-12190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson updated NIFI-12190:
-
Description:
NIFI-12177 adds an {{integraiton-tests}} GitHub Actions Workflow to regularly
run the {{integration-tests}} defined throughout the NiFi repo. However, many
tests have had to be skipped from the {{failsafe plugin}} execution due to
failures within the GitHub runner.
Typically these are caused by:
* the need for external resources to be setup
* incorrect assertions that might have worked during the original IT creation
but have not been updated since
* or missing/incompatible OS resources within the GitHub runner
This Epic is to review the failing ITs within NiFi codebase and either:
* fix them, e.g. correct assertions, and/or by using Testcontainers for hosting
external resources, such as for the Elasticsearch Processors
* remove them if no longer useful/cannot be reliably executed on
developer's/GitHub runners
* annotate them to be skipped in CI environments if the tests are useful and
work on a developer machine but won't be able to work in a CI runner, e.g. a
[DisabledIfEnvironmentVariable|https://junit.org/junit5/docs/5.7.0/api/org.junit.jupiter.api/org/junit/jupiter/api/condition/DisabledIfEnvironmentVariable.html]
to check whether the `CI` env var is present and set to `true`
Once fixed, the appropriate entry from the {{integration-tests}} Workflow's
{{MAVEN_FAILSAFE_SKIP_ITS}} env var **must** be removed
At the time of writing, the ITs known to be failing (either in part or as a
whole) are:
{quote}!AwsSecretsManagerSensitivePropertyProviderIT,
!AwsKmsSensitivePropertyProviderIT,
!AzureKeyVaultKeySensitivePropertyProviderIT,
!GcpKmsSensitivePropertyProviderIT,
!StandardHashiCorpVaultCommunicationServiceIT,
!ConversionWithSchemaInferenceIT#testXmlToAvro,
!ConversionWithSchemaInferenceIT#testXmlToJson,
!ConversionWithSchemaInferenceIT#testJsonToCsv,
!ConversionWithSchemaInferenceIT#testCsvToXml,
!ConversionWithSchemaInferenceIT#testXmlToAvroToXml,
!ConversionWithSchemaInferenceIT#testCsvToAvro,
!ConversionWithSchemaInferenceIT#testCsvToJson,
!ConversionWithSchemaInferenceIT#testJsonToAvroToJson,
!ConversionWithSchemaInferenceIT#testCsvToAvroToCsv,
!ConversionWithExplicitSchemaIT#testJsonToCsv,
!ConversionWithExplicitSchemaIT#testJsonToAvro,
!ConversionWithExplicitSchemaIT#testJsonToAvroToJson,
!ITConsumeKinesisStreamEndpointOverride,
!ITConsumeKinesisStreamConnectAWS,
!ITPutKinesisStream,
!ITPutKinesisStreamWithEndpointOverride#testIntegrationSuccess,
!ITPutKinesisFirehoseWithEndpointOverride#testIntegrationSuccess,
!ITPutKinesisFirehose,
!ITTagS3Object,
!ITFetchS3Object,
!ITDeleteS3Object,
!ITPutS3Object,
!ITListS3,
!ITPutSNS#testPublish,
!ITPutSNS#testPublishWithCredentialsProviderService,
!ITDeleteSQS,
!ITGetSQS,
!ITPutSQS,
!ITPutLambda,
!GetMongoIT#testDatabaseEL,
!PutGridFSIT#testFileNameAndHashUniqueness,
!PutGridFSIT#testHashUniqueness,
!ITPutAzureCosmosDBRecord,
!ITListAzureBlobStorage_v12,
!ITPutAzureDataLakeStorage,
!ITFetchAzureDataLakeStorage,
!ITListAzureDataLakeStorage,
!ITFetchAzureBlobStorage_v12,
!ITPutAzureBlobStorage_v12,
!ITDeleteAzureDataLakeStorage,
!ITDeleteAzureBlobStorage_v12,
!ITMoveAzureDataLakeStorage,
!AzureGraphUserGroupProviderIT,
!JMSPublisherConsumerIT#validateMessageRedeliveryWhenNotAcked,
!ITPutInfluxDB,
!ITExecuteInfluxDBQuery,
!GremlinClientServiceYamlSettingsAndBytecodeIT,
!GremlinClientServiceControllerSettingsIT,
!ITestConsumeEmail#validateUrl,
!PrometheusReportingTaskIT#testNullLabel,
!SnowflakeConnectionPoolIT,
!SalesforceRestServiceIT,
!QuerySalesforceObjectIT,
!PutSalesforceObjectIT,
!FetchDropboxIT,
!PutDropboxIT,
!ListDropboxIT,
!FetchBoxFileIT,
!PutBoxFileIT,
!ListBoxFileIT,
!JsonConfigBasedBoxClientServiceIT,
!QueryIoTDBIT,
!PutIoTDBRecordIT,
!StandardProcessSessionIT#testCloneThenWriteCountsClaimReferencesProperly,
!StandardProcessSessionIT#testUpdateFlowFileRepoFailsOnSessionCommit,
!JMSPublisherConsumerIT#testMultipleThreads,
!ITRedisDistributedMapCacheClientService#testSentinelRedisWithAuthentication,
!ITRedisDistributedMapCacheClientService#testSentinelRedis,
!SnowflakePipeIT,
!SecureFileIT,
!SecureProxyIT#testAccessStatus,
!SecureProxyIT#testAccessStatusAsProxiedAdmin,
!SecureProxyIT#testAccessStatusUsingRegistryClient,
!SecureNiFiRegistryClientIT#testGetAccessStatus,
!SecureNiFiRegistryClientIT#testTenantsClientGroups,
!SecureNiFiRegistryClientIT#testTenantsClientUsers,
!SecureNiFiRegistryClientIT#testCrudOperations,
!SecureDatabaseIT
!ITConnectionAccessControl,
!ITLabelAccessControl,
[jira] [Created] (NIFI-12191) MacOS GitHub Runner does not correctly run Docker (Colima)
Chris Sampson created NIFI-12191:
Summary: MacOS GitHub Runner does not correctly run Docker (Colima)
Key: NIFI-12191
URL: https://issues.apache.org/jira/browse/NIFI-12191
Project: Apache NiFi
Issue Type: Bug
Reporter: Chris Sampson
Fix For: 2.latest
NIFI-12177 introduced the {{integration-tests}} and {{docker-tests}} GitHub
Action Workflows, which runs fine on {{Ubuntu}}, but cannot run as successfully
on {{MacOS}}.
This is because the MacOS runner **does not** contain Docker by default, and
installing it via {{brew install docker}} actually installs
[Colima|https://github.com/abiosoft/colima]. While this allows Docker Images to
be built and (to a degree) run as Containers, it has been noted that
Testcontainers **do not** work correctly with such a setup, and some of the
Docker Image Tests (e.g. for NiFi - see NIFI-12177) fail because containers
don't run in a consistent manner compared to Ubuntu with `docker` installed.
This ticket is to investigate and fix the use of Docker within the MacOS based
GitHub runner for these workflows.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (NIFI-12190) Fix failing Integration Tests
Chris Sampson created NIFI-12190:
Summary: Fix failing Integration Tests
Key: NIFI-12190
URL: https://issues.apache.org/jira/browse/NIFI-12190
Project: Apache NiFi
Issue Type: Epic
Reporter: Chris Sampson
Fix For: 2.latest
NIFI-12177 adds an {{integraiton-tests}} GitHub Actions Workflow to regularly
run the {{integration-tests}} defined throughout the NiFi repo. However, many
tests have had to be skipped from the {{failsafe plugin}} execution due to
failures within the GitHub runner.
Typically these are caused by:
* the need for external resources to be setup
* incorrect assertions that might have worked during the original IT creation
but have not been updated since
* or missing/incompatible OS resources within the GitHub runner
This Epic is to review the failing ITs within NiFi codebase and either:
* fix them, e.g. correct assertions, and/or by using Testcontainers for hosting
external resources, such as for the Elasticsearch Processors
* remove them if no longer useful/cannot be reliably executed on
developer's/GitHub runners
* annotate them to be skipped in CI environments if the tests are useful and
work on a developer machine but won't be able to work in a CI runner, e.g. a
[DisabledIfEnvironmentVariable|https://junit.org/junit5/docs/5.7.0/api/org.junit.jupiter.api/org/junit/jupiter/api/condition/DisabledIfEnvironmentVariable.html]
to check whether the `CI` env var is present and set to `true`
Once fixed, the appropriate entry from the {{integration-tests}} Workflow's
{{MAVEN_FAILSAFE_SKIP_ITS}} env var **must** be removed
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-12178) Add integration-tests and docker-tests GithHub Action Workflows
[
https://issues.apache.org/jira/browse/NIFI-12178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17772792#comment-17772792
]
Chris Sampson commented on NIFI-12178:
--
Thanks [~markap14], useful to know as the AWS bundle ITs are definitely seeing
a lot of failures. I've discussed with [~jgresock] before about a possible move
to using Localstack via Testcontainers or similar, maybe how they've been
integrated into processors such as Elasticsearch and Mongo.
> Add integration-tests and docker-tests GithHub Action Workflows
> ---
>
> Key: NIFI-12178
> URL: https://issues.apache.org/jira/browse/NIFI-12178
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Major
> Fix For: 2.latest
>
>
> There are a lot of {{integration-tests}} and {{docker}} builds in the NiFi
> repository that aren't being regularly executed and verified.
> GitHub Action Workflows should be added (similar to the existing
> {{system-tests}} Workflow) that regularly exercise these tests, allowing the
> community to better monitor the health of NiFi components.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-12178) Add integration-tests and docker-tests GithHub Action Workflows
[
https://issues.apache.org/jira/browse/NIFI-12178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17772587#comment-17772587
]
Chris Sampson commented on NIFI-12178:
--
[~exceptionfactory] I started off by trying to run all of the tests/builds
together, but have separated them into their own workflows
({{integration-tests}} and {{docker-tests}}) now due to disk space issues more
than anything else.
My approach currently is:
* build and test Docker Images (which is working for an Ubuntu runner, but not
MacOS as that doesn't come with Docker installed - we can install Colima but
the {{nifi-docker}} tests fail, which I propose to be handled by NIFI-12177 in
future)
* build and run all {{integration-tests}} that aren't already covered by the
existing {{system-tests}} workflow
** I've identified a lot of failures/errors already, but rather than trying to
fix them (other than general issues like not having Docker running), I'm
filtering out the {{IT}} classes from the [failsafe
plugin|https://maven.apache.org/surefire/maven-failsafe-plugin/examples/single-test.html#multiple-formats-in-one]
execution
** this will give us a list of failing test classes and/or methods (some
classes have tests that pass but others are failing), which we can look to
diagnose and fix as part of a follow-on ticket in the future
This feels like a sensible/iterative approach to getting something running in
GitHub to prevent deterioration of existing ITs, capture any new ITs added in
future, and allow us to find and remove/fix existing failing ITs already in the
repository.
Open to suggestions on a different approach if you/anyone has other ideas
(ongoing trials can be seen in my fork at
https://github.com/ChrisSamo632/nifi/actions)
> Add integration-tests and docker-tests GithHub Action Workflows
> ---
>
> Key: NIFI-12178
> URL: https://issues.apache.org/jira/browse/NIFI-12178
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Major
> Fix For: 2.latest
>
>
> There are a lot of {{integration-tests}} and {{docker}} builds in the NiFi
> repository that aren't being regularly executed and verified.
> GitHub Action Workflows should be added (similar to the existing
> {{system-tests}} Workflow) that regularly exercise these tests, allowing the
> community to better monitor the health of NiFi components.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12177) Docker builds (dockermaven) should include testing of the built images
[
https://issues.apache.org/jira/browse/NIFI-12177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson updated NIFI-12177:
-
Description:
The NiFi & NiFi Registry {{dockermaven}} builds currently run an
{{integration_test}} script once the image has been built in order to confirm a
container can be started and the component can be contacted.
The same (or similar) approaches should be considered for:
* NiFi Toolkit
* MiNiFi
* MiNiFi C2
This would help avoid situations such as NIFI-12175
Also, the NiFi `dockermaven` test script fails if run with `macos-latest` in
GitHub Actions (NIFI-12178), likely an incompatibility with the use of Colima
to get Docker working within the runner, an alternative should be found (to the
test assertions or the GitHub workflow).
was:
The NiFi & NiFi Registry {{dockermaven}} builds currently run an
{{integration_test}} script once the image has been built in order to confirm a
container can be started and the component can be contacted.
The same (or similar) approaches should be considered for:
* NiFi Toolkit
* MiNiFi
* MiNiFi C2
This would help avoid situations such as NIFI-12175
> Docker builds (dockermaven) should include testing of the built images
> --
>
> Key: NIFI-12177
> URL: https://issues.apache.org/jira/browse/NIFI-12177
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Chris Sampson
>Priority: Major
> Fix For: 1.latest, 2.latest
>
>
> The NiFi & NiFi Registry {{dockermaven}} builds currently run an
> {{integration_test}} script once the image has been built in order to confirm
> a container can be started and the component can be contacted.
> The same (or similar) approaches should be considered for:
> * NiFi Toolkit
> * MiNiFi
> * MiNiFi C2
> This would help avoid situations such as NIFI-12175
> Also, the NiFi `dockermaven` test script fails if run with `macos-latest` in
> GitHub Actions (NIFI-12178), likely an incompatibility with the use of Colima
> to get Docker working within the runner, an alternative should be found (to
> the test assertions or the GitHub workflow).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (NIFI-12178) Add integration-tests and docker-tests GithHub Action Workflows
Chris Sampson created NIFI-12178:
Summary: Add integration-tests and docker-tests GithHub Action
Workflows
Key: NIFI-12178
URL: https://issues.apache.org/jira/browse/NIFI-12178
Project: Apache NiFi
Issue Type: Improvement
Reporter: Chris Sampson
Fix For: 2.latest
There are a lot of {{integration-tests}} and {{docker}} builds in the NiFi
repository that aren't being regularly executed and verified.
GitHub Action Workflows should be added (similar to the existing
{{system-tests}} Workflow) that regularly exercise these tests, allowing the
community to better monitor the health of NiFi components.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (NIFI-12178) Add integration-tests and docker-tests GithHub Action Workflows
[
https://issues.apache.org/jira/browse/NIFI-12178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson reassigned NIFI-12178:
Assignee: Chris Sampson
> Add integration-tests and docker-tests GithHub Action Workflows
> ---
>
> Key: NIFI-12178
> URL: https://issues.apache.org/jira/browse/NIFI-12178
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Major
> Fix For: 2.latest
>
>
> There are a lot of {{integration-tests}} and {{docker}} builds in the NiFi
> repository that aren't being regularly executed and verified.
> GitHub Action Workflows should be added (similar to the existing
> {{system-tests}} Workflow) that regularly exercise these tests, allowing the
> community to better monitor the health of NiFi components.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (NIFI-12177) Docker builds (dockermaven) should include testing of the built images
Chris Sampson created NIFI-12177:
Summary: Docker builds (dockermaven) should include testing of the
built images
Key: NIFI-12177
URL: https://issues.apache.org/jira/browse/NIFI-12177
Project: Apache NiFi
Issue Type: Improvement
Reporter: Chris Sampson
Fix For: 1.latest, 2.latest
The NiFi & NiFi Registry {{dockermaven}} builds currently run an
{{integration_test}} script once the image has been built in order to confirm a
container can be started and the component can be contacted.
The same (or similar) approaches should be considered for:
* NiFi Toolkit
* MiNiFi
* MiNiFi C2
This would help avoid situations such as NIFI-12175
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (NIFI-12175) Update all component Docker Image builds to java 21
Chris Sampson created NIFI-12175:
Summary: Update all component Docker Image builds to java 21
Key: NIFI-12175
URL: https://issues.apache.org/jira/browse/NIFI-12175
Project: Apache NiFi
Issue Type: Improvement
Reporter: Chris Sampson
Fix For: 2.latest
All nifi components (nifi, nifi-registry, nifi-toolkit, minifi, minifi-c2) come
with dockerhub and dockermaven modules to build Docker Images. The former
builds an image using the source code released to the Apache servers, the
latter uses the binary files from the component's assembly module one it has
been built locally.
Nifi has already been updated. Nifi Registry's dockermaven has been updated.
Other image builds are outstanding:
* Nifi registry (dockerhub, found in nifi-registry-core/nifi-registry-docker)
* Nifi toolkit (both)
* minifi (both)
* minifi-c2 (both)
Note that the {{docker}} maven profile must be enabled to trigger the build of
the Docker Images during {{install}} (although some builds only need
{{package}} instead, confusingly).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12153) Allow Max String Length and Comments to be configurable in Json Record Readers
[ https://issues.apache.org/jira/browse/NIFI-12153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Sampson updated NIFI-12153: - Fix Version/s: 1.latest 2.latest > Allow Max String Length and Comments to be configurable in Json Record Readers > -- > > Key: NIFI-12153 > URL: https://issues.apache.org/jira/browse/NIFI-12153 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Chris Sampson >Assignee: Chris Sampson >Priority: Minor > Fix For: 1.latest, 2.latest > > Time Spent: 10m > Remaining Estimate: 0h > > Recent versions of Jackson have introduced a smaller maximum string length > when parsing JSON. We should allow this to be configurable for the JSON-based > Record Readers (e.g. like for some processors in NIFI-12127 & NIFI-12117). > It is also possible for Jackson to allow comments within JSON (this is > allowed in `ValidateJson`, for example). We should allow this to be > configurable for the JSON Record Readers too. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12153) Allow Max String Length and Comments to be configurable in Json Record Readers
[ https://issues.apache.org/jira/browse/NIFI-12153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Sampson updated NIFI-12153: - Status: Patch Available (was: In Progress) > Allow Max String Length and Comments to be configurable in Json Record Readers > -- > > Key: NIFI-12153 > URL: https://issues.apache.org/jira/browse/NIFI-12153 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Chris Sampson >Assignee: Chris Sampson >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > Recent versions of Jackson have introduced a smaller maximum string length > when parsing JSON. We should allow this to be configurable for the JSON-based > Record Readers (e.g. like for some processors in NIFI-12127 & NIFI-12117). > It is also possible for Jackson to allow comments within JSON (this is > allowed in `ValidateJson`, for example). We should allow this to be > configurable for the JSON Record Readers too. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-12153) Allow Max String Length and Comments to be configurable in Json Record Readers
Chris Sampson created NIFI-12153: Summary: Allow Max String Length and Comments to be configurable in Json Record Readers Key: NIFI-12153 URL: https://issues.apache.org/jira/browse/NIFI-12153 Project: Apache NiFi Issue Type: Improvement Reporter: Chris Sampson Recent versions of Jackson have introduced a smaller maximum string length when parsing JSON. We should allow this to be configurable for the JSON-based Record Readers (e.g. like for some processors in NIFI-12127 & NIFI-12117). It is also possible for Jackson to allow comments within JSON (this is allowed in `ValidateJson`, for example). We should allow this to be configurable for the JSON Record Readers too. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-12153) Allow Max String Length and Comments to be configurable in Json Record Readers
[ https://issues.apache.org/jira/browse/NIFI-12153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Sampson reassigned NIFI-12153: Assignee: Chris Sampson > Allow Max String Length and Comments to be configurable in Json Record Readers > -- > > Key: NIFI-12153 > URL: https://issues.apache.org/jira/browse/NIFI-12153 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Chris Sampson >Assignee: Chris Sampson >Priority: Minor > > Recent versions of Jackson have introduced a smaller maximum string length > when parsing JSON. We should allow this to be configurable for the JSON-based > Record Readers (e.g. like for some processors in NIFI-12127 & NIFI-12117). > It is also possible for Jackson to allow comments within JSON (this is > allowed in `ValidateJson`, for example). We should allow this to be > configurable for the JSON Record Readers too. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12028) Add s3.region attribute to ListS3 generated flow files
[ https://issues.apache.org/jira/browse/NIFI-12028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Sampson updated NIFI-12028: - Fix Version/s: 2.0.0 1.24.0 (was: 1.latest) (was: 2.latest) > Add s3.region attribute to ListS3 generated flow files > -- > > Key: NIFI-12028 > URL: https://issues.apache.org/jira/browse/NIFI-12028 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > Fix For: 2.0.0, 1.24.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Add s3.region attribute to ListS3 generated flow files so that it can be > referenced in FetchS3 processor. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12118) RemoveRecordField does not work if specifying FlowFile attributes
[
https://issues.apache.org/jira/browse/NIFI-12118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson updated NIFI-12118:
-
Fix Version/s: 2.0.0
1.24.0
(was: 1.latest)
(was: 2.latest)
Resolution: Fixed
Status: Resolved (was: Patch Available)
> RemoveRecordField does not work if specifying FlowFile attributes
> -
>
> Key: NIFI-12118
> URL: https://issues.apache.org/jira/browse/NIFI-12118
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 2.0.0, 1.24.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The Processor compiles all Record Paths on the first call to onTrigger, in
> the {{process}} method. To do this, it evaluates all dynamic properties
> against expression language. It then caches these compiled RecordPaths, and
> it never evaluates FlowFile attributes again. So if I configure the processor
> to remove /fields/${fieldName} and I then pass in a FlowFile where attribute
> {{fieldName}} is {{hello}} it will remove {{{}/fields/hello{}}}. But if the
> next FlowFile has the attribute value {{greeting}} it will still remove
> {{/fields/hello}} instead of removing {{/fields/greeting}}
> Additionally, the processor is not thread-safe. It uses a {{volatile List}}
> to store these RecordPaths. But if more than one Concurrent Task is set, any
> of the threads could win the race and set the value.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-8135) DataTypeUtils conversion methods for Records to java Maps should respect Choice types
[ https://issues.apache.org/jira/browse/NIFI-8135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Sampson updated NIFI-8135: Status: Patch Available (was: In Progress) > DataTypeUtils conversion methods for Records to java Maps should respect > Choice types > - > > Key: NIFI-8135 > URL: https://issues.apache.org/jira/browse/NIFI-8135 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.11.4 >Reporter: Chris Sampson >Assignee: Chris Sampson >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Where a schema field is a Map of Choice, with options of Array (of Strings), > Map (of Strings) or String, Record Readers incorrectly deserialise the data > of the nested Map/Array as Records containing no schema. When converting the > overall Record into a java Map using DataTypeUtils, the nested MapRecord is > not converted into a Map. > This can lead to data loss between readers & writers. > This appears to be related to one/more of: > * missing schema in the nested MapRecord object (even if it's presented with > the data to the Record Reader) - the conversion methods could infer a schema > for such MapRecords rather than effectively skipping them > * Choice types don't appear to be respected for nested fields (e.g. checking > whether the options within a Choice are for Map or Array, not just whether > the field dataType passed to the conversion method is for Map or Array, etc. > ... it's a Choice that should allow for these) > * Array/ Map conversion should not assume all elements within the collection > are of the same type, especially if the field's type is Choice -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-8135) DataTypeUtils conversion methods for Records to java Maps should respect Choice types
[ https://issues.apache.org/jira/browse/NIFI-8135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Sampson reassigned NIFI-8135: --- Assignee: Chris Sampson > DataTypeUtils conversion methods for Records to java Maps should respect > Choice types > - > > Key: NIFI-8135 > URL: https://issues.apache.org/jira/browse/NIFI-8135 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.11.4 >Reporter: Chris Sampson >Assignee: Chris Sampson >Priority: Major > > Where a schema field is a Map of Choice, with options of Array (of Strings), > Map (of Strings) or String, Record Readers incorrectly deserialise the data > of the nested Map/Array as Records containing no schema. When converting the > overall Record into a java Map using DataTypeUtils, the nested MapRecord is > not converted into a Map. > This can lead to data loss between readers & writers. > This appears to be related to one/more of: > * missing schema in the nested MapRecord object (even if it's presented with > the data to the Record Reader) - the conversion methods could infer a schema > for such MapRecords rather than effectively skipping them > * Choice types don't appear to be respected for nested fields (e.g. checking > whether the options within a Choice are for Map or Array, not just whether > the field dataType passed to the conversion method is for Map or Array, etc. > ... it's a Choice that should allow for these) > * Array/ Map conversion should not assume all elements within the collection > are of the same type, especially if the field's type is Choice -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-8134) DataTypeUtils.toRecord methods do not recursively convert Maps into Records
[ https://issues.apache.org/jira/browse/NIFI-8134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Sampson updated NIFI-8134: Fix Version/s: 1.latest 2.latest > DataTypeUtils.toRecord methods do not recursively convert Maps into Records > --- > > Key: NIFI-8134 > URL: https://issues.apache.org/jira/browse/NIFI-8134 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.11.4 >Reporter: Chris Sampson >Assignee: Chris Sampson >Priority: Major > Fix For: 1.latest, 2.latest > > Time Spent: 10m > Remaining Estimate: 0h > > Given a java Map that contains one or more Maps as values (optionally nested > within arrays), the DataTypeUtils.toRecord method should convert the child > Maps to Records before converting the to level Map. > This assumes the associated schema for the data represents these objects as > Records (including as part of a Choice of Array type). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-8134) DataTypeUtils.toRecord methods do not recursively convert Maps into Records
[ https://issues.apache.org/jira/browse/NIFI-8134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Sampson updated NIFI-8134: Status: Patch Available (was: In Progress) > DataTypeUtils.toRecord methods do not recursively convert Maps into Records > --- > > Key: NIFI-8134 > URL: https://issues.apache.org/jira/browse/NIFI-8134 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.11.4 >Reporter: Chris Sampson >Assignee: Chris Sampson >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Given a java Map that contains one or more Maps as values (optionally nested > within arrays), the DataTypeUtils.toRecord method should convert the child > Maps to Records before converting the to level Map. > This assumes the associated schema for the data represents these objects as > Records (including as part of a Choice of Array type). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12063) Paginated Elasticsearch processors do not pass Dynamic Properties as Request Parameters for SCROLL pagination
[ https://issues.apache.org/jira/browse/NIFI-12063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Sampson updated NIFI-12063: - Status: Patch Available (was: In Progress) > Paginated Elasticsearch processors do not pass Dynamic Properties as Request > Parameters for SCROLL pagination > - > > Key: NIFI-12063 > URL: https://issues.apache.org/jira/browse/NIFI-12063 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.23.2 >Reporter: Chris Sampson >Assignee: Chris Sampson >Priority: Trivial > Fix For: 1.latest, 2.latest > > Time Spent: 10m > Remaining Estimate: 0h > > Dynamic Properties added to the Paginated Elasticsearch processors are meant > to be used as Request Parameters when searching Elasticsearch. > This *does* happen, *except* for subsequent pages of a *SCROLL* type query. > This is because the [Elasticsearch Scroll > API|https://www.elastic.co/guide/en/elasticsearch/reference/current/scroll-api.html] > does not accept such request parameters. > NiFi should clarify this in its documentation for users. NiFi users can > easily change to using SEARCH_AFTER or POINT_IN_TIME (requires Elasticsearch > 7.10+ and X-Pack) instead of SCROLL, where the Request Parameters are used > for every call to the [Elasticsearch Search > API|https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-12063) Paginated Elasticsearch processors do not pass Dynamic Properties as Request Parameters for SCROLL pagination
[ https://issues.apache.org/jira/browse/NIFI-12063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Sampson reassigned NIFI-12063: Assignee: Chris Sampson > Paginated Elasticsearch processors do not pass Dynamic Properties as Request > Parameters for SCROLL pagination > - > > Key: NIFI-12063 > URL: https://issues.apache.org/jira/browse/NIFI-12063 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.23.2 >Reporter: Chris Sampson >Assignee: Chris Sampson >Priority: Trivial > Fix For: 1.latest, 2.latest > > Time Spent: 10m > Remaining Estimate: 0h > > Dynamic Properties added to the Paginated Elasticsearch processors are meant > to be used as Request Parameters when searching Elasticsearch. > This *does* happen, *except* for subsequent pages of a *SCROLL* type query. > This is because the [Elasticsearch Scroll > API|https://www.elastic.co/guide/en/elasticsearch/reference/current/scroll-api.html] > does not accept such request parameters. > NiFi should clarify this in its documentation for users. NiFi users can > easily change to using SEARCH_AFTER or POINT_IN_TIME (requires Elasticsearch > 7.10+ and X-Pack) instead of SCROLL, where the Request Parameters are used > for every call to the [Elasticsearch Search > API|https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-12063) Paginated Elasticsearch processors do not pass Dynamic Properties as Request Parameters for SCROLL pagination
Chris Sampson created NIFI-12063: Summary: Paginated Elasticsearch processors do not pass Dynamic Properties as Request Parameters for SCROLL pagination Key: NIFI-12063 URL: https://issues.apache.org/jira/browse/NIFI-12063 Project: Apache NiFi Issue Type: Bug Affects Versions: 1.23.2 Reporter: Chris Sampson Fix For: 1.latest, 2.latest Dynamic Properties added to the Paginated Elasticsearch processors are meant to be used as Request Parameters when searching Elasticsearch. This *does* happen, *except* for subsequent pages of a *SCROLL* type query. This is because the [Elasticsearch Scroll API|https://www.elastic.co/guide/en/elasticsearch/reference/current/scroll-api.html] does not accept such request parameters. NiFi should clarify this in its documentation for users. NiFi users can easily change to using SEARCH_AFTER or POINT_IN_TIME (requires Elasticsearch 7.10+ and X-Pack) instead of SCROLL, where the Request Parameters are used for every call to the [Elasticsearch Search API|https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-12032) Nifi Toolkit doesn't allow you to specify what type of truststore it creates
[ https://issues.apache.org/jira/browse/NIFI-12032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Sampson resolved NIFI-12032. -- Resolution: Won't Fix The community has decided to remove the NiFi TLS Toolkit in NiFi 2.0 (see the [dev mailing list discussion](https://lists.apache.org/thread/vn1nzobtz4fh7fs461sgg8jj9zygrk0f)), so users will need to generate valid certificates using available alternatives. Given this decision, it wouldn't make sense to make this change now for NiFi 1.x. > Nifi Toolkit doesn't allow you to specify what type of truststore it creates > > > Key: NIFI-12032 > URL: https://issues.apache.org/jira/browse/NIFI-12032 > Project: Apache NiFi > Issue Type: Bug > Components: Tools and Build >Affects Versions: 1.23.2 >Reporter: Ryan >Priority: Minor > Labels: nifi-toolkit > > Nifi toolkit's command `./bin/tls-toolkit.sh` allows you to specify a > keystore type but not a truststore type. > > As JKS is no longer the recommended format since JDK9 this command should > allow you to specify a type. > > https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#usage-9 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-11985) Implement a processor to consume documents from Elasticsearch indices
[
https://issues.apache.org/jira/browse/NIFI-11985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17763452#comment-17763452
]
Chris Sampson commented on NIFI-11985:
--
[^NIFI-11985_Flow.json] provides a Flow Definition exhibiting the
{{ConsumeElasticsearch}} processor (along with all of the other Elasticsearch
processors and the changes added for NIFI-11016)
> Implement a processor to consume documents from Elasticsearch indices
> -
>
> Key: NIFI-11985
> URL: https://issues.apache.org/jira/browse/NIFI-11985
> Project: Apache NiFi
> Issue Type: New Feature
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Minor
> Fix For: 1.latest, 2.latest
>
> Attachments: NIFI-11985_Flow.json
>
>
> It is possible to use Elasticsearch to store series data, i.e. data is
> continually added to an Elasticsearch index over time, with a {{date}} or a
> 1-up numeric {{long}} field.
> This is more likely with the advent of [Data
> Streams|https://www.elastic.co/guide/en/elasticsearch/reference/current/data-streams.html]
> or the recent [Time Series Data
> Streams|https://www.elastic.co/guide/en/elasticsearch/reference/current/tsds.html],
> both of which use a {{@timestamp}} field to indicate when a document was
> added to the stream.
> There are use cases where NiFi users may want to consume new data from the
> Elasticsearch index/data stream after it's arrived, then pass it to another
> service.
> NiFi would need to:
> * know which field to use as the "series field" (e.g. {{@timestamp}})
> * track the last read "series field" value via State so that the same
> documents are not retrieved from Elasticsearch multiple times
> * allow for the optional specification of the "last read" field value, e.g.
> if a user wants to offset the start of the documents to be read (this value
> should only be used if a value doesn't also exist within the processor's
> State)
> * allow for the fact that the "last read" vlaue will be blank when the
> processor is first run (and the value is not otherwise specified), meaning we
> want to retrieve all existing data
> * allow for users to specify an optional Query Filter to apply to the search
> within Elasticsearch when finding documents to retrieve
> Possible implementations should consider using the {{SearchElasticsearch}}
> processor as a basis, which already uses State tracking between processor
> executions and allows for the retrieval of Elasticsearch documents in a
> paginated manner (thus avoiding pulling too much data in a single request).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-11985) Implement a processor to consume documents from Elasticsearch indices
[
https://issues.apache.org/jira/browse/NIFI-11985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson updated NIFI-11985:
-
Attachment: NIFI-11985_Flow.json
> Implement a processor to consume documents from Elasticsearch indices
> -
>
> Key: NIFI-11985
> URL: https://issues.apache.org/jira/browse/NIFI-11985
> Project: Apache NiFi
> Issue Type: New Feature
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Minor
> Fix For: 1.latest, 2.latest
>
> Attachments: NIFI-11985_Flow.json
>
>
> It is possible to use Elasticsearch to store series data, i.e. data is
> continually added to an Elasticsearch index over time, with a {{date}} or a
> 1-up numeric {{long}} field.
> This is more likely with the advent of [Data
> Streams|https://www.elastic.co/guide/en/elasticsearch/reference/current/data-streams.html]
> or the recent [Time Series Data
> Streams|https://www.elastic.co/guide/en/elasticsearch/reference/current/tsds.html],
> both of which use a {{@timestamp}} field to indicate when a document was
> added to the stream.
> There are use cases where NiFi users may want to consume new data from the
> Elasticsearch index/data stream after it's arrived, then pass it to another
> service.
> NiFi would need to:
> * know which field to use as the "series field" (e.g. {{@timestamp}})
> * track the last read "series field" value via State so that the same
> documents are not retrieved from Elasticsearch multiple times
> * allow for the optional specification of the "last read" field value, e.g.
> if a user wants to offset the start of the documents to be read (this value
> should only be used if a value doesn't also exist within the processor's
> State)
> * allow for the fact that the "last read" vlaue will be blank when the
> processor is first run (and the value is not otherwise specified), meaning we
> want to retrieve all existing data
> * allow for users to specify an optional Query Filter to apply to the search
> within Elasticsearch when finding documents to retrieve
> Possible implementations should consider using the {{SearchElasticsearch}}
> processor as a basis, which already uses State tracking between processor
> executions and allows for the retrieval of Elasticsearch documents in a
> paginated manner (thus avoiding pulling too much data in a single request).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-11985) Implement a processor to consume documents from Elasticsearch indices
[
https://issues.apache.org/jira/browse/NIFI-11985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762973#comment-17762973
]
Chris Sampson commented on NIFI-11985:
--
Query Builder needed for constructing a full Query within new
ConsumeElasticsearch processor
> Implement a processor to consume documents from Elasticsearch indices
> -
>
> Key: NIFI-11985
> URL: https://issues.apache.org/jira/browse/NIFI-11985
> Project: Apache NiFi
> Issue Type: New Feature
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Minor
> Fix For: 1.latest, 2.latest
>
>
> It is possible to use Elasticsearch to store series data, i.e. data is
> continually added to an Elasticsearch index over time, with a {{date}} or a
> 1-up numeric {{long}} field.
> This is more likely with the advent of [Data
> Streams|https://www.elastic.co/guide/en/elasticsearch/reference/current/data-streams.html]
> or the recent [Time Series Data
> Streams|https://www.elastic.co/guide/en/elasticsearch/reference/current/tsds.html],
> both of which use a {{@timestamp}} field to indicate when a document was
> added to the stream.
> There are use cases where NiFi users may want to consume new data from the
> Elasticsearch index/data stream after it's arrived, then pass it to another
> service.
> NiFi would need to:
> * know which field to use as the "series field" (e.g. {{@timestamp}})
> * track the last read "series field" value via State so that the same
> documents are not retrieved from Elasticsearch multiple times
> * allow for the optional specification of the "last read" field value, e.g.
> if a user wants to offset the start of the documents to be read (this value
> should only be used if a value doesn't also exist within the processor's
> State)
> * allow for the fact that the "last read" vlaue will be blank when the
> processor is first run (and the value is not otherwise specified), meaning we
> want to retrieve all existing data
> * allow for users to specify an optional Query Filter to apply to the search
> within Elasticsearch when finding documents to retrieve
> Possible implementations should consider using the {{SearchElasticsearch}}
> processor as a basis, which already uses State tracking between processor
> executions and allows for the retrieval of Elasticsearch documents in a
> paginated manner (thus avoiding pulling too much data in a single request).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-11985) Implement a processor to consume documents from Elasticsearch indices
[
https://issues.apache.org/jira/browse/NIFI-11985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson updated NIFI-11985:
-
Description:
It is possible to use Elasticsearch to store series data, i.e. data is
continually added to an Elasticsearch index over time, with a {{date}} or a
1-up numeric {{long}} field.
This is more likely with the advent of [Data
Streams|https://www.elastic.co/guide/en/elasticsearch/reference/current/data-streams.html]
or the recent [Time Series Data
Streams|https://www.elastic.co/guide/en/elasticsearch/reference/current/tsds.html],
both of which use a {{@timestamp}} field to indicate when a document was added
to the stream.
There are use cases where NiFi users may want to consume new data from the
Elasticsearch index/data stream after it's arrived, then pass it to another
service.
NiFi would need to:
* know which field to use as the "series field" (e.g. {{@timestamp}})
* track the last read "series field" value via State so that the same documents
are not retrieved from Elasticsearch multiple times
* allow for the optional specification of the "last read" field value, e.g. if
a user wants to offset the start of the documents to be read (this value should
only be used if a value doesn't also exist within the processor's State)
* allow for the fact that the "last read" vlaue will be blank when the
processor is first run (and the value is not otherwise specified), meaning we
want to retrieve all existing data
* allow for users to specify an optional Query Filter to apply to the search
within Elasticsearch when finding documents to retrieve
Possible implementations should consider using the {{SearchElasticsearch}}
processor as a basis, which already uses State tracking between processor
executions and allows for the retrieval of Elasticsearch documents in a
paginated manner (thus avoiding pulling too much data in a single request).
was:
It is possible to use Elasticsearch to store series data, i.e. data is
continually added to an Elasticsearch index over time, with a {{date}} or a
1-up numeric {{long}} field.
This is more likely with the advent of [Data
Streams](https://www.elastic.co/guide/en/elasticsearch/reference/current/data-streams.html)
or the recent [Time Series Data
Streams](https://www.elastic.co/guide/en/elasticsearch/reference/current/tsds.html),
both of which use a {{@timestamp}} field to indicate when a document was added
to the stream.
There are use cases where NiFi users may want to consume new data from the
Elasticsearch index/data stream after it's arrived, then pass it to another
service.
NiFi would need to:
* know which field to use as the "series field" (e.g. {{@timestamp}})
* track the last read "series field" value via State so that the same documents
are not retrieved from Elasticsearch multiple times
* allow for the optional specification of the "last read" field value, e.g. if
a user wants to offset the start of the documents to be read (this value should
only be used if a value doesn't also exist within the processor's State)
* allow for the fact that the "last read" vlaue will be blank when the
processor is first run (and the value is not otherwise specified), meaning we
want to retrieve all existing data
* allow for users to specify an optional Query Filter to apply to the search
within Elasticsearch when finding documents to retrieve
Possible implementations should consider using the {{SearchElasticsearch}}
processor as a basis, which already uses State tracking between processor
executions and allows for the retrieval of Elasticsearch documents in a
paginated manner (thus avoiding pulling too much data in a single request).
> Implement a processor to consume documents from Elasticsearch indices
> -
>
> Key: NIFI-11985
> URL: https://issues.apache.org/jira/browse/NIFI-11985
> Project: Apache NiFi
> Issue Type: New Feature
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Minor
> Fix For: 1.latest, 2.latest
>
>
> It is possible to use Elasticsearch to store series data, i.e. data is
> continually added to an Elasticsearch index over time, with a {{date}} or a
> 1-up numeric {{long}} field.
> This is more likely with the advent of [Data
> Streams|https://www.elastic.co/guide/en/elasticsearch/reference/current/data-streams.html]
> or the recent [Time Series Data
> Streams|https://www.elastic.co/guide/en/elasticsearch/reference/current/tsds.html],
> both of which use a {{@timestamp}} field to indicate when a document was
> added to the stream.
> There are use cases where NiFi users may want to consume new data from the
> Elasticsearch index/data stream after it's arrived, then pass it to another
> service.
> NiFi would need to:
> * know which
[jira] [Updated] (NIFI-11985) Implement a processor to consume documents from Elasticsearch indices
[
https://issues.apache.org/jira/browse/NIFI-11985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson updated NIFI-11985:
-
Fix Version/s: 1.latest
2.latest
> Implement a processor to consume documents from Elasticsearch indices
> -
>
> Key: NIFI-11985
> URL: https://issues.apache.org/jira/browse/NIFI-11985
> Project: Apache NiFi
> Issue Type: New Feature
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Minor
> Fix For: 1.latest, 2.latest
>
>
> It is possible to use Elasticsearch to store series data, i.e. data is
> continually added to an Elasticsearch index over time, with a {{date}} or a
> 1-up numeric {{long}} field.
> This is more likely with the advent of [Data
> Streams](https://www.elastic.co/guide/en/elasticsearch/reference/current/data-streams.html)
> or the recent [Time Series Data
> Streams](https://www.elastic.co/guide/en/elasticsearch/reference/current/tsds.html),
> both of which use a {{@timestamp}} field to indicate when a document was
> added to the stream.
> There are use cases where NiFi users may want to consume new data from the
> Elasticsearch index/data stream after it's arrived, then pass it to another
> service.
> NiFi would need to:
> * know which field to use as the "series field" (e.g. {{@timestamp}})
> * track the last read "series field" value via State so that the same
> documents are not retrieved from Elasticsearch multiple times
> * allow for the optional specification of the "last read" field value, e.g.
> if a user wants to offset the start of the documents to be read (this value
> should only be used if a value doesn't also exist within the processor's
> State)
> * allow for the fact that the "last read" vlaue will be blank when the
> processor is first run (and the value is not otherwise specified), meaning we
> want to retrieve all existing data
> * allow for users to specify an optional Query Filter to apply to the search
> within Elasticsearch when finding documents to retrieve
> Possible implementations should consider using the {{SearchElasticsearch}}
> processor as a basis, which already uses State tracking between processor
> executions and allows for the retrieval of Elasticsearch documents in a
> paginated manner (thus avoiding pulling too much data in a single request).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-11016) Consider providing a query-builder set of properties for the UpdateByQueryElasticsearch processor
[
https://issues.apache.org/jira/browse/NIFI-11016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson updated NIFI-11016:
-
Status: Patch Available (was: In Progress)
> Consider providing a query-builder set of properties for the
> UpdateByQueryElasticsearch processor
> -
>
> Key: NIFI-11016
> URL: https://issues.apache.org/jira/browse/NIFI-11016
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Minor
> Fix For: 1.latest, 2.latest
>
>
> While it is already possible to send a {{script}} field as part of the
> {{Query}} parameter in the {{UpdateByQueryElasticsearch}} processor, [there
> is question|https://github.com/apache/nifi/pull/6628#issuecomment-1308972782]
> about whether this is sufficiently clear to NiFi users or a "query builder"
> kind of experience would be preferred, e.g. if the {{query}} field were
> specified via one processor property and {{script}} via another
> One thing to be careful of with any changes is to retain the [current
> (default)
> behaviour|https://github.com/apache/nifi/pull/6628#issuecomment-1308978863]
> of the {{Query}} parameter of the processor so as to be backwards
> compatible/for anyone already specifying the whole Elasticsearch Update/Query
> DSL via the existing parameters
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-11016) Consider providing a query-builder set of properties for the UpdateByQueryElasticsearch processor
[
https://issues.apache.org/jira/browse/NIFI-11016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson updated NIFI-11016:
-
Fix Version/s: 1.latest
2.latest
> Consider providing a query-builder set of properties for the
> UpdateByQueryElasticsearch processor
> -
>
> Key: NIFI-11016
> URL: https://issues.apache.org/jira/browse/NIFI-11016
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Minor
> Fix For: 1.latest, 2.latest
>
>
> While it is already possible to send a {{script}} field as part of the
> {{Query}} parameter in the {{UpdateByQueryElasticsearch}} processor, [there
> is question|https://github.com/apache/nifi/pull/6628#issuecomment-1308972782]
> about whether this is sufficiently clear to NiFi users or a "query builder"
> kind of experience would be preferred, e.g. if the {{query}} field were
> specified via one processor property and {{script}} via another
> One thing to be careful of with any changes is to retain the [current
> (default)
> behaviour|https://github.com/apache/nifi/pull/6628#issuecomment-1308978863]
> of the {{Query}} parameter of the processor so as to be backwards
> compatible/for anyone already specifying the whole Elasticsearch Update/Query
> DSL via the existing parameters
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-11985) Implement a processor to consume documents from Elasticsearch indices
[
https://issues.apache.org/jira/browse/NIFI-11985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson updated NIFI-11985:
-
Status: Patch Available (was: In Progress)
> Implement a processor to consume documents from Elasticsearch indices
> -
>
> Key: NIFI-11985
> URL: https://issues.apache.org/jira/browse/NIFI-11985
> Project: Apache NiFi
> Issue Type: New Feature
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Minor
>
> It is possible to use Elasticsearch to store series data, i.e. data is
> continually added to an Elasticsearch index over time, with a {{date}} or a
> 1-up numeric {{long}} field.
> This is more likely with the advent of [Data
> Streams](https://www.elastic.co/guide/en/elasticsearch/reference/current/data-streams.html)
> or the recent [Time Series Data
> Streams](https://www.elastic.co/guide/en/elasticsearch/reference/current/tsds.html),
> both of which use a {{@timestamp}} field to indicate when a document was
> added to the stream.
> There are use cases where NiFi users may want to consume new data from the
> Elasticsearch index/data stream after it's arrived, then pass it to another
> service.
> NiFi would need to:
> * know which field to use as the "series field" (e.g. {{@timestamp}})
> * track the last read "series field" value via State so that the same
> documents are not retrieved from Elasticsearch multiple times
> * allow for the optional specification of the "last read" field value, e.g.
> if a user wants to offset the start of the documents to be read (this value
> should only be used if a value doesn't also exist within the processor's
> State)
> * allow for the fact that the "last read" vlaue will be blank when the
> processor is first run (and the value is not otherwise specified), meaning we
> want to retrieve all existing data
> * allow for users to specify an optional Query Filter to apply to the search
> within Elasticsearch when finding documents to retrieve
> Possible implementations should consider using the {{SearchElasticsearch}}
> processor as a basis, which already uses State tracking between processor
> executions and allows for the retrieval of Elasticsearch documents in a
> paginated manner (thus avoiding pulling too much data in a single request).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (NIFI-12020) Enable templated searches for Elasticsearch
Chris Sampson created NIFI-12020:
Summary: Enable templated searches for Elasticsearch
Key: NIFI-12020
URL: https://issues.apache.org/jira/browse/NIFI-12020
Project: Apache NiFi
Issue Type: New Feature
Reporter: Chris Sampson
See
https://www.elastic.co/guide/en/elasticsearch/reference/current/search-template.html
This could be an extension to {{JsonQueryElasticsearch}} with an optional
"templated" property setting that sends the {{Query}} to the
{{_search/template}} endpoint instead of just {{_search}}. Paginated queries
should also be considered (if they are possible with templated searches in
Elasticsearch).
Separate processor(s) may be preferred.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (NIFI-11016) Consider providing a query-builder set of properties for the UpdateByQueryElasticsearch processor
[
https://issues.apache.org/jira/browse/NIFI-11016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson reassigned NIFI-11016:
Assignee: Chris Sampson
> Consider providing a query-builder set of properties for the
> UpdateByQueryElasticsearch processor
> -
>
> Key: NIFI-11016
> URL: https://issues.apache.org/jira/browse/NIFI-11016
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Minor
>
> While it is already possible to send a {{script}} field as part of the
> {{Query}} parameter in the {{UpdateByQueryElasticsearch}} processor, [there
> is question|https://github.com/apache/nifi/pull/6628#issuecomment-1308972782]
> about whether this is sufficiently clear to NiFi users or a "query builder"
> kind of experience would be preferred, e.g. if the {{query}} field were
> specified via one processor property and {{script}} via another
> One thing to be careful of with any changes is to retain the [current
> (default)
> behaviour|https://github.com/apache/nifi/pull/6628#issuecomment-1308978863]
> of the {{Query}} parameter of the processor so as to be backwards
> compatible/for anyone already specifying the whole Elasticsearch Update/Query
> DSL via the existing parameters
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
