[jira] [Commented] (NIFI-12178) Add integration-tests and docker-tests GithHub Action Workflows
[
https://issues.apache.org/jira/browse/NIFI-12178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17778234#comment-17778234
]
ASF subversion and git services commented on NIFI-12178:
Commit ed3a70c9859f7cccdf03c6133f05d42ab108cd25 in nifi's branch
refs/heads/main from Chris Sampson
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=ed3a70c985 ]
NIFI-12178 Add integration-tests and docker-tests GitHub Workflows
Use Java Zulu distribution for all docker-tests and integration-tests Github
Workflows
This closes #7858
Signed-off-by: David Handermann
> Add integration-tests and docker-tests GithHub Action Workflows
> ---
>
> Key: NIFI-12178
> URL: https://issues.apache.org/jira/browse/NIFI-12178
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Major
> Fix For: 2.latest
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> There are a lot of {{integration-tests}} and {{docker}} builds in the NiFi
> repository that aren't being regularly executed and verified.
> GitHub Action Workflows should be added (similar to the existing
> {{system-tests}} Workflow) that regularly exercise these tests, allowing the
> community to better monitor the health of NiFi components.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-12178) Add integration-tests and docker-tests GithHub Action Workflows
[
https://issues.apache.org/jira/browse/NIFI-12178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17773145#comment-17773145
]
Chris Sampson commented on NIFI-12178:
--
I've set the new tests to run daily, like system tests, and on update to
specific files. But the frequency can certainly be adjusted.
I've been trying to run tests during development in my personal account (hoping
not to reach any limits), so as to reduce the overall apache load there too.
We could set the frequency to weekly, then increase later if that would be a
more sensible cadence?
> Add integration-tests and docker-tests GithHub Action Workflows
> ---
>
> Key: NIFI-12178
> URL: https://issues.apache.org/jira/browse/NIFI-12178
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Major
> Fix For: 2.latest
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> There are a lot of {{integration-tests}} and {{docker}} builds in the NiFi
> repository that aren't being regularly executed and verified.
> GitHub Action Workflows should be added (similar to the existing
> {{system-tests}} Workflow) that regularly exercise these tests, allowing the
> community to better monitor the health of NiFi components.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-12178) Add integration-tests and docker-tests GithHub Action Workflows
[
https://issues.apache.org/jira/browse/NIFI-12178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17773068#comment-17773068
]
Otto Fowler commented on NIFI-12178:
I think this stuff is great. The only thing I will add for consideration is
that Apache's GitHub resources are not infinite, ( join the build list for the
repetitive discussions on these limits as some projects exhaust them for all
the others ) and we should be cognizant of that.
> Add integration-tests and docker-tests GithHub Action Workflows
> ---
>
> Key: NIFI-12178
> URL: https://issues.apache.org/jira/browse/NIFI-12178
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Major
> Fix For: 2.latest
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> There are a lot of {{integration-tests}} and {{docker}} builds in the NiFi
> repository that aren't being regularly executed and verified.
> GitHub Action Workflows should be added (similar to the existing
> {{system-tests}} Workflow) that regularly exercise these tests, allowing the
> community to better monitor the health of NiFi components.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-12178) Add integration-tests and docker-tests GithHub Action Workflows
[
https://issues.apache.org/jira/browse/NIFI-12178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17772792#comment-17772792
]
Chris Sampson commented on NIFI-12178:
--
Thanks [~markap14], useful to know as the AWS bundle ITs are definitely seeing
a lot of failures. I've discussed with [~jgresock] before about a possible move
to using Localstack via Testcontainers or similar, maybe how they've been
integrated into processors such as Elasticsearch and Mongo.
> Add integration-tests and docker-tests GithHub Action Workflows
> ---
>
> Key: NIFI-12178
> URL: https://issues.apache.org/jira/browse/NIFI-12178
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Major
> Fix For: 2.latest
>
>
> There are a lot of {{integration-tests}} and {{docker}} builds in the NiFi
> repository that aren't being regularly executed and verified.
> GitHub Action Workflows should be added (similar to the existing
> {{system-tests}} Workflow) that regularly exercise these tests, allowing the
> community to better monitor the health of NiFi components.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-12178) Add integration-tests and docker-tests GithHub Action Workflows
[
https://issues.apache.org/jira/browse/NIFI-12178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17772642#comment-17772642
]
Mark Payne commented on NIFI-12178:
---
I've been working on some of this as well, [~Chris S] , as I'm currently
working on eliminating all of the deprecation warnings in AWS Processors. As
part of that, I need to verify that all of the components still work, so I'm
replacing all of the Integration Tests with the Localstack Test Container. It's
been extremely helpful. So just wanted to mention this, to avoid duplicating
any effort.
> Add integration-tests and docker-tests GithHub Action Workflows
> ---
>
> Key: NIFI-12178
> URL: https://issues.apache.org/jira/browse/NIFI-12178
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Major
> Fix For: 2.latest
>
>
> There are a lot of {{integration-tests}} and {{docker}} builds in the NiFi
> repository that aren't being regularly executed and verified.
> GitHub Action Workflows should be added (similar to the existing
> {{system-tests}} Workflow) that regularly exercise these tests, allowing the
> community to better monitor the health of NiFi components.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-12178) Add integration-tests and docker-tests GithHub Action Workflows
[
https://issues.apache.org/jira/browse/NIFI-12178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17772591#comment-17772591
]
David Handermann commented on NIFI-12178:
-
Thanks for the summary [~Chris S], sounds like you have made a lot of progress
thus far!
Filtering out the failing test classes sounds like a good place to start. As
you noted, it provides a useful list for follow-on efforts.
I agree this will be very helpful in maintaining meaningful integration tests.
> Add integration-tests and docker-tests GithHub Action Workflows
> ---
>
> Key: NIFI-12178
> URL: https://issues.apache.org/jira/browse/NIFI-12178
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Major
> Fix For: 2.latest
>
>
> There are a lot of {{integration-tests}} and {{docker}} builds in the NiFi
> repository that aren't being regularly executed and verified.
> GitHub Action Workflows should be added (similar to the existing
> {{system-tests}} Workflow) that regularly exercise these tests, allowing the
> community to better monitor the health of NiFi components.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-12178) Add integration-tests and docker-tests GithHub Action Workflows
[
https://issues.apache.org/jira/browse/NIFI-12178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17772587#comment-17772587
]
Chris Sampson commented on NIFI-12178:
--
[~exceptionfactory] I started off by trying to run all of the tests/builds
together, but have separated them into their own workflows
({{integration-tests}} and {{docker-tests}}) now due to disk space issues more
than anything else.
My approach currently is:
* build and test Docker Images (which is working for an Ubuntu runner, but not
MacOS as that doesn't come with Docker installed - we can install Colima but
the {{nifi-docker}} tests fail, which I propose to be handled by NIFI-12177 in
future)
* build and run all {{integration-tests}} that aren't already covered by the
existing {{system-tests}} workflow
** I've identified a lot of failures/errors already, but rather than trying to
fix them (other than general issues like not having Docker running), I'm
filtering out the {{IT}} classes from the [failsafe
plugin|https://maven.apache.org/surefire/maven-failsafe-plugin/examples/single-test.html#multiple-formats-in-one]
execution
** this will give us a list of failing test classes and/or methods (some
classes have tests that pass but others are failing), which we can look to
diagnose and fix as part of a follow-on ticket in the future
This feels like a sensible/iterative approach to getting something running in
GitHub to prevent deterioration of existing ITs, capture any new ITs added in
future, and allow us to find and remove/fix existing failing ITs already in the
repository.
Open to suggestions on a different approach if you/anyone has other ideas
(ongoing trials can be seen in my fork at
https://github.com/ChrisSamo632/nifi/actions)
> Add integration-tests and docker-tests GithHub Action Workflows
> ---
>
> Key: NIFI-12178
> URL: https://issues.apache.org/jira/browse/NIFI-12178
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Major
> Fix For: 2.latest
>
>
> There are a lot of {{integration-tests}} and {{docker}} builds in the NiFi
> repository that aren't being regularly executed and verified.
> GitHub Action Workflows should be added (similar to the existing
> {{system-tests}} Workflow) that regularly exercise these tests, allowing the
> community to better monitor the health of NiFi components.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-12178) Add integration-tests and docker-tests GithHub Action Workflows
[
https://issues.apache.org/jira/browse/NIFI-12178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17772579#comment-17772579
]
David Handermann commented on NIFI-12178:
-
Thanks for taking on this effort [~Chris S], I think this would be a very
helpful improvement.
Handling integration tests may be somewhat challenging, as some work on their
own with or without Testcontainers, but others require manual adjust for
credentials. This could be a good opportunity to remove some stale integration
tests. It may be worth separating the effort into Docker workflow builds and
integration test builds depending on the types of changes required. JUnit 5
also provides some new annotations for conditional enabling based on
environment variables or system properties. This could be used to filter and
select integration tests that do not need any additional configuration or
external properties. This could also support adjusting some integration tests
to use system properties instead of relying on manual changes to set properties
like credentials and URLs.
> Add integration-tests and docker-tests GithHub Action Workflows
> ---
>
> Key: NIFI-12178
> URL: https://issues.apache.org/jira/browse/NIFI-12178
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Major
> Fix For: 2.latest
>
>
> There are a lot of {{integration-tests}} and {{docker}} builds in the NiFi
> repository that aren't being regularly executed and verified.
> GitHub Action Workflows should be added (similar to the existing
> {{system-tests}} Workflow) that regularly exercise these tests, allowing the
> community to better monitor the health of NiFi components.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
