Hi all, As you may have noticed, we have been running Azure Pipelines for the tests for some time and have recently retired Travis CI in this PR <https://github.com/apache/hudi/pull/3556>.
Background It was a pain for the CI process in the past with Travis, which from time to time queued up CI jobs forever. This severely affects the developer experience for making contributions, and also the release process. The New Setup Thanks to the Flink community, who pioneered the CI setup, and MS Azure, who provided the free resources, we are able to mirror the repo and PRs to a separate GitHub organization <https://github.com/apachehudi-ci/> and run the tests in Azure Pipelines. Hudi's ci-bot <https://github.com/apachehudi-ci/ci-bot> (forked from Flink's ci-bot <https://github.com/flink-ci/ci-bot>) runs on a GCP server and periodically scans recently changed PRs for CI submission. CI results are commented back to the PR by hudi-bot <https://github.com/hudi-bot>. Full details about the setup are documented in this <https://cwiki.apache.org/confluence/display/HUDI/Guide+on+CI+infrastructure> page <https://cwiki.apache.org/confluence/display/HUDI/Guide+on+CI+infrastructure> . Azure Pipelines provides 10 free managed parallel jobs. CI tests are split into 5 jobs. We have dedicated resources to test 2 PRs in parallel. - master builds: https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build?definitionId=3 - branch builds: https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build?definitionId=5 Note: PRs against asf-site (website updates) will be ignored by this setup. Additionally, we make use of GitHub Actions to build against different Spark and Scala versions. GitHub Actions jobs also provide fast feedback for compliance like checkstyle and apache-rat. For PR Owners and Reviewers With these changes, PR owners and reviewers should pay attention to the following: - CI results are indicated in hudi-bot's comment - A new commit in the same PR will trigger a new build and cancel any existing build - Comment `@hudi-bot run azure` to manually trigger a new build - GitHub Actions jobs will show as checks in the PR - Minimum conditions to merge: - Azure CI report shows success, and - GitHub Actions jobs passed - For website update PRs (for asf-site branch), owners post screenshots to show the changes in lieu of CI tests. Hope this contributes towards a more seamless developer experience. Please reach out to the community for CI issues or further questions. Best, Raymond
