[
https://issues.apache.org/jira/browse/HADOOP-19857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18080070#comment-18080070
]
ASF GitHub Bot commented on HADOOP-19857:
-----------------------------------------
ajfabbri commented on code in PR #8452:
URL: https://github.com/apache/hadoop/pull/8452#discussion_r3220553530
##########
.github/workflows/tmpl_build_and_test.yml:
##########
@@ -177,3 +178,146 @@ jobs:
- name: Build
shell: bash
run: ./mvnw $MAVEN_ARGS clean install -DskipTests
+ build-and-test:
+ if: (!cancelled()) && contains(fromJSON(inputs.jobs), 'build-and-test')
+ name: Test ${{ matrix.comment }} (Java ${{ inputs.java }}) ${{ inputs.os
}}-${{ inputs.branch }}
+ runs-on: ubuntu-24.04
+ needs: [ precondition, build-image ]
+ container:
+ image: ${{ needs.precondition.outputs.build_image_url }}
+ options: --user ${{ needs.build-image.outputs.uid }}
+ strategy:
+ fail-fast: false
+ max-parallel: 8
+ matrix:
+ include:
+ - comment: hdfs - slow
+ modules:
+ -pl :hadoop-hdfs
+ -Dgroups=slow
+ - comment: hdfs - other
+ modules:
+ -pl :hadoop-hdfs
+ -DexcludedGroups=slow
+ - comment: yarn-server-rm
+ modules:
+ -pl :hadoop-yarn-server-resourcemanager
+ - comment: mr
+ modules:
+ -pl :hadoop-mapreduce-client-core
+ -pl :hadoop-mapreduce-client-common
+ -pl :hadoop-mapreduce-client-shuffle
+ -pl :hadoop-mapreduce-client-jobclient
+ -pl :hadoop-mapreduce-client-app
+ -pl :hadoop-mapreduce-client-hs
+ -pl :hadoop-mapreduce-client-hs-plugins
+ -pl :hadoop-mapreduce-client-nativetask
+ -pl :hadoop-mapreduce-client-uploader
+ -pl :hadoop-mapreduce-examples
+ - comment: common
+ modules:
+ -pl :hadoop-common
+ -pl :hadoop-yarn-server-nodemanager
+ -pl :hadoop-yarn-client
+ -pl :hadoop-distcp
+ - comment: hdfs-rbf
+ modules:
+ -pl :hadoop-hdfs-rbf
+ -pl :hadoop-yarn-applications-distributedshell
+ -pl :hadoop-yarn-services-core
+ - comment: other
+ modules:
+ -pl :hadoop-minikdc
+ -pl :hadoop-auth
+ -pl :hadoop-auth-examples
+ -pl :hadoop-nfs
+ -pl :hadoop-kms
+ -pl :hadoop-registry
+ -pl :hadoop-hdfs-client
+ -pl :hadoop-hdfs-native-client
+ -pl :hadoop-hdfs-httpfs
+ -pl :hadoop-hdfs-nfs
+ -pl :hadoop-yarn-api
+ -pl :hadoop-yarn-common
+ -pl :hadoop-yarn-server-common
+ -pl :hadoop-yarn-server-applicationhistoryservice
+ -pl :hadoop-yarn-server-timelineservice
+ -pl :hadoop-yarn-server-web-proxy
+ -pl :hadoop-yarn-server-tests
+ -pl :hadoop-yarn-server-sharedcachemanager
+ -pl :hadoop-yarn-server-timeline-pluginstorage
+ -pl :hadoop-yarn-server-timelineservice-hbase-common
+ -pl :hadoop-yarn-server-timelineservice-hbase-client
+ -pl :hadoop-yarn-server-timelineservice-hbase-server-2
+ -pl :hadoop-yarn-server-timelineservice-hbase-tests
+ -pl :hadoop-yarn-server-router
+ -pl :hadoop-yarn-server-timelineservice-documentstore
+ -pl :hadoop-yarn-server-globalpolicygenerator
+ -pl :hadoop-yarn-applications-unmanaged-am-launcher
+ -pl :hadoop-yarn-services-api
+ -pl :hadoop-yarn-registry
+ -pl
org.apache.hadoop.applications.mawo:hadoop-yarn-applications-mawo-core
+ -pl :hadoop-yarn-csi
+ -pl :hadoop-minicluster
+ -pl :hadoop-federation-balance
+ -pl :hadoop-streaming
+ -pl :hadoop-client
+ -pl :hadoop-dynamometer-workload
+ -pl :hadoop-dynamometer-infra
+ -pl :hadoop-dynamometer-blockgen
+ -pl :hadoop-dynamometer-dist
+ -pl :hadoop-archives
+ -pl :hadoop-archive-logs
+ -pl :hadoop-rumen
+ -pl :hadoop-gridmix
+ -pl :hadoop-datajoin
+ -pl :hadoop-extras
+ -pl :hadoop-aws
+ -pl :hadoop-kafka
+ -pl :hadoop-aliyun
+ -pl :hadoop-sls
+ -pl :hadoop-resourceestimator
+ -pl :hadoop-azure
+ -pl :hadoop-azure-datalake
+ -pl :hadoop-fs2img
+ -pl :hadoop-tools-dist
+ -pl :hadoop-benchmark
+ -pl :hadoop-compat-bench
+ -pl :hadoop-client-api
+ -pl :hadoop-client-runtime
+ -pl :hadoop-client-minicluster
+ -pl :hadoop-client-integration-tests
+ -pl :hadoop-cos
+ -pl :hadoop-tos
+ -pl :hadoop-huaweicloud
+ -pl :hadoop-cloud-storage
+ -pl :hadoop-dist
+ # skipped modules
+ # -pl :hadoop-yarn-applications-catalog-webapp
+ steps:
+ - name: Checkout Hadoop repository
+ uses: actions/checkout@v6
+ # In order to fetch changed files
+ with:
+ fetch-depth: 0
+ - name: Setup JDK ${{ inputs.java }}
+ uses: actions/setup-java@v5
+ with:
+ distribution: zulu
+ java-version: ${{ inputs.java }}
+ cache: 'maven'
+ check-latest: false
+ - name: Build (Java ${{ inputs.java }}) ${{ inputs.os }}-${{
inputs.branch }}
+ shell: bash
+ run: ./mvnw $MAVEN_ARGS ${{ matrix.modules }} clean install -am
-DskipTests
+ - name: Test (Java ${{ inputs.java }}) ${{ inputs.os }}-${{
inputs.branch }}
+ shell: bash
+ run: ./mvnw $MAVEN_ARGS ${{ matrix.modules }} test
-Dsurefire.excludesFile=$PWD/.github/gha-tests/exclude-tests.txt
Review Comment:
Can we elide these `$MAVEN_ARGS` in previous two steps? (Also search for
other existing uses).
##########
.github/gha-tests/README.md:
##########
@@ -0,0 +1,63 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+### Excluded tests for GitHub Actions (GHA)
+
+Initial excluded tests: run the GHA workflow, if a test fails or aborts, add it
+to `exclude-tests.txt`. Repeat until 5 consecutive successes.
+
+Contributors are encouraged to diagnose and improve the excluded tests, and
remove
+them from the excluded list once they are stable. Stability assessment: when
deleting
+tests from `exclude-tests.txt`, the GHA workflow was successfully executed 5
times
+consecutively.
+
+### Slow tests
+
+Test classes takes more than 60s to complete in module
`hadoop-hdfs-project/hadoop-hdfs`
+are marked as slow tests, by adding JUnit5 annotation `@Tag("slow")` to the
test class.
+
+Slow tests are executed in a dedicated GHA job and roughly take 2.5 hours to
complete.
+Contributors are encouraged to diagnose and improve the slow tests to speed up
the CI.
+
+### Run test locally
+
+Create a standard build environment using Docker.
+```
+$ cd <hadoop source code directory>
+$ ./start-build-env.sh
+... (wait for the container to start)
+```
+
+Run single test suite inside container
+```
+$ export MAVEN_ARGS="-Pnative -Drequire.fuse -Drequire.openssl
-Drequire.snappy -Drequire.valgrind -Drequire.test.libhadoop"
+$ ./mvnw $MAVEN_ARGS -pl :hadoop-common -am clean install -DskipTests
Review Comment:
Is it necessary to add `$MAVEN_ARGS` explicitly? IIUC this should
automatically be picked up [since maven
3.9.0](https://maven.apache.org/configure.html), and we use `maven-enforcer` to
ensure we are on >=3.9.11.
> Set up CI on GitHub Actions
> ---------------------------
>
> Key: HADOOP-19857
> URL: https://issues.apache.org/jira/browse/HADOOP-19857
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
> Reporter: Cheng Pan
> Priority: Major
> Labels: pull-request-available
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
