areusch commented on code in PR #13300:
URL: https://github.com/apache/tvm/pull/13300#discussion_r1037580584
##########
ci/jenkins/templates/utils/Build.groovy.j2:
##########
@@ -0,0 +1,57 @@
+def ci_setup(image) {
+ sh (
+ script: "${docker_run} ${image} ./tests/scripts/task_clear_pytest.sh",
+ label: 'Clean up old workspace',
+ )
+}
+
+def python_unittest(image) {
Review Comment:
same question here--maybe we just call these task_python_unittest?
##########
docker/dev_common.sh:
##########
@@ -31,9 +31,9 @@ GIT_TOPLEVEL=$(cd $(dirname ${BASH_SOURCE[0]}) && git
rev-parse --show-toplevel)
function filter_jenkinsfile() {
local echo_on=0;
while read line; do
- if [ "${line}" == "// NOTE: these lines are scanned by
docker/dev_common.sh. Please update the regex as needed. -->" ]; then
+ if [ "${line}" == "# NOTE: these lines are scanned by
docker/dev_common.sh. Please update the regex as needed. -->" ]; then
Review Comment:
now that this is in Python, we could just run like:
```
$(python3 -c "import data; print(getattr(data, '"{ci_name}'))"
```
##########
ci/jenkins/templates/utils/Build.groovy.j2:
##########
@@ -0,0 +1,57 @@
+def ci_setup(image) {
+ sh (
+ script: "${docker_run} ${image} ./tests/scripts/task_clear_pytest.sh",
+ label: 'Clean up old workspace',
+ )
+}
+
+def python_unittest(image) {
+ sh (
+ script: "${docker_run} ${image} ./tests/scripts/task_python_unittest.sh",
+ label: 'Run Python unit tests',
+ )
+}
+
+def fsim_test(image) {
+ sh (
+ script: "${docker_run} ${image} ./tests/scripts/task_python_vta_fsim.sh",
+ label: 'Run VTA tests in FSIM',
+ )
+}
+
+def make_standalone_crt(image, build_dir) {
Review Comment:
the target is crttest in addtion to standalone_crt; should we consider
reflecting that in the function name and/or just writing a generic task_build()
function?
##########
ci/jenkins/generated/arm_jenkinsfile.groovy:
##########
@@ -0,0 +1,933 @@
+#!groovy
+// -*- mode: groovy -*-
+
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Jenkins pipeline
+// See documents at https://jenkins.io/doc/book/pipeline/jenkinsfile/
+
+// Docker env used for testing
+// Different image may have different version tag
+// because some of them are more stable than anoter.
+//
+// Docker images are maintained by PMC, cached in dockerhub
+// and remains relatively stable over the time.
+// Flow for upgrading docker env(need commiter)
+//
+// - Send PR to upgrade build script in the repo
+// - Build the new docker image
+// - Tag the docker image with a new version and push to a binary cache.
+// - Update the version in the Jenkinsfile, send a PR
+// - Fix any issues wrt to the new image version in the PR
+// - Merge the PR and now we are in new version
+// - Tag the new version as the lates
+// - Periodically cleanup the old versions on local workers
+//
+
+// ============================= IMPORTANT NOTE =============================
+// This file is generated by 'jenkins/generate.py'. Do not edit this file
directly!
+// Make edits to 'jenkins/Jenkinsfile.j2' and regenerate this with
+// 'python3 jenkins/generate.py'
+// Note: This timestamp is here to ensure that updates to the Jenkinsfile are
+// always rebased on main before merging:
+// Generated at 2022-11-18T13:53:23.418898
+
+import org.jenkinsci.plugins.pipeline.modeldefinition.Utils
+// NOTE: these lines are scanned by docker/dev_common.sh. Please update the
regex as needed. -->
+ci_lint = 'tlcpack/ci-lint:20221013-060115-61c9742ea'
+ci_gpu = 'tlcpack/ci-gpu:20221019-060125-0b4836739'
+ci_cpu = 'tlcpack/ci-cpu:20221013-060115-61c9742ea'
+ci_minimal = 'tlcpack/ci-minimal:20221013-060115-61c9742ea'
+ci_wasm = 'tlcpack/ci-wasm:20221013-060115-61c9742ea'
+ci_i386 = 'tlcpack/ci-i386:20221013-060115-61c9742ea'
+ci_cortexm = 'tlcpack/ci-cortexm:20221013-060115-61c9742ea'
+ci_arm = 'tlcpack/ci-arm:20221013-060115-61c9742ea'
+ci_hexagon = 'tlcpack/ci-hexagon:20221013-060115-61c9742ea'
+ci_riscv = 'tlcpack/ci-riscv:20221013-060115-61c9742ea'
+// <--- End of regex-scanned config.
+
+// Parameters to allow overriding (in Jenkins UI), the images
+// to be used by a given build. When provided, they take precedence
+// over default values above.
+properties([
+ parameters([
+ string(name: 'ci_arm_param', defaultValue: ''),
+ string(name: 'ci_cortexm_param', defaultValue: ''),
+ string(name: 'ci_cpu_param', defaultValue: ''),
+ string(name: 'ci_gpu_param', defaultValue: ''),
+ string(name: 'ci_hexagon_param', defaultValue: ''),
+ string(name: 'ci_i386_param', defaultValue: ''),
+ string(name: 'ci_lint_param', defaultValue: ''),
+ string(name: 'ci_minimal_param', defaultValue: ''),
+ string(name: 'ci_riscv_param', defaultValue: ''),
+ string(name: 'ci_wasm_param', defaultValue: ''),
+ ])
+])
+
+// Placeholders for newly built Docker image names (if rebuild_docker_images
+// is used)
+ built_ci_arm = null;
+ built_ci_cortexm = null;
+ built_ci_cpu = null;
+ built_ci_gpu = null;
+ built_ci_hexagon = null;
+ built_ci_i386 = null;
+ built_ci_lint = null;
+ built_ci_minimal = null;
+ built_ci_riscv = null;
+ built_ci_wasm = null;
+
+// Global variable assigned during Sanity Check that holds the sha1 which
should be
+// merged into the PR in all branches.
+upstream_revision = null
+
+// command to start a docker container
+docker_run = 'docker/bash.sh --env CI --env TVM_SHARD_INDEX --env
TVM_NUM_SHARDS --env RUN_DISPLAY_URL --env PLATFORM --env SKIP_SLOW_TESTS --env
TEST_STEP_NAME'
+docker_build = 'docker/build.sh'
+// timeout in minutes
+max_time = 180
+rebuild_docker_images = false
+
+s3_bucket = 'tvm-jenkins-artifacts-prod'
+s3_prefix = "tvm/${env.BRANCH_NAME}/${env.BUILD_NUMBER}"
+
+// Jenkins script root directory
+jenkins_scripts_root = "ci/scripts/jenkins"
+
+// General note: Jenkins has limits on the size of a method (or top level code)
+// that are pretty strict, so most usage of groovy methods in these templates
+// are purely to satisfy the JVM
+def per_exec_ws(folder) {
+ return "workspace/exec_${env.EXECUTOR_NUMBER}/" + folder
+}
+
+// initialize source codes
+def init_git() {
+ retry(5) {
+ checkout scm
+ }
+
+ // Add more info about job node
+ sh (
+ script: './tests/scripts/task_show_node_info.sh',
+ label: 'Show executor node info',
+ )
+
+ // Determine merge commit to use for all stages
+ if (env.BRANCH_NAME == 'main') {
+ // Only set upstream_revision to HEAD and skip merging to avoid a race
with another commit merged to main.
+ update_upstream_revision("HEAD")
+ } else {
+ // This is PR branch so merge with latest main.
+ merge_with_main()
+ }
+
+ sh(
+ script: """
+ set -eux
+ . ${jenkins_scripts_root}/retry.sh
+ retry 3 timeout 5m git submodule update --init -f --jobs 0
+ """,
+ label: 'Update git submodules',
+ )
+ checkout_trusted_files()
+}
+
+def update_upstream_revision(git_ref) {
+ if (upstream_revision == null) {
+ upstream_revision = sh(
+ script: "git log -1 ${git_ref} --format=\'%H\'",
+ label: 'Determine upstream revision',
+ returnStdout: true,
+ ).trim()
+ }
+}
+
+def merge_with_main() {
+ sh (
+ script: 'git fetch origin main',
+ label: 'Fetch upstream',
+ )
+ update_upstream_revision("FETCH_HEAD")
+ sh (
+ script: "git -c user.name=TVM-Jenkins -c [email protected]
merge ${upstream_revision}",
+ label: 'Merge to origin/main'
+ )
+}
+
+def docker_init(image) {
+ // Clear out all Docker images that aren't going to be used
+ sh(
+ script: """
+ set -eux
+ docker image ls --all
+ IMAGES=\$(docker image ls --all --format '{{.Repository}}:{{.Tag}}
{{.ID}}')
+
+ echo -e "Found images:\\n\$IMAGES"
+ echo "\$IMAGES" | { grep -vE '${image}' || test \$? = 1; } | { xargs
docker rmi || test \$? = 123; }
+
+ docker image ls --all
+ """,
+ label: 'Clean old Docker images',
+ )
+
+ if (image.contains("amazonaws.com")) {
+ // If this string is in the image name it's from ECR and needs to be pulled
+ // with the right credentials
+ // ecr_pull(image)
Review Comment:
do we need to re-enable for CI rebuild?
##########
ci/scripts/jenkins/open_docker_update_pr.py:
##########
@@ -125,64 +125,53 @@ def latest_tlcpackstaging_image(source: str) ->
Optional[str]:
user, repo = parse_remote(remote)
# Read the existing images from the Jenkinsfile
- logging.info(f"Reading {JENKINSFILE}")
- with open(JENKINSFILE) as f:
+ logging.info(f"Reading {IMAGES_FILE}")
+ with open(IMAGES_FILE) as f:
content = f.readlines()
# Build a new Jenkinsfile with the latest images from tlcpack or
tlcpackstaging
- new_content = []
replacements = {}
+
for line in content:
- m = re.match(r"^(ci_[a-zA-Z0-9]+) = \'(.*)\'", line.strip())
+ m = re.match(r"^(ci_[a-zA-Z0-9]+) = \"(.*)\"", line.strip())
if m is not None:
logging.info(f"Found match on line {line.strip()}")
groups = m.groups()
new_image = latest_tlcpackstaging_image(groups[1])
if new_image is None:
logging.info(f"No new image found")
- new_content.append(line)
else:
logging.info(f"Using new image {new_image}")
new_line = f"{groups[0]} = '{new_image}'\n"
- new_content.append(new_line)
replacements[line] = new_line
- else:
- new_content.append(line)
- # Write out the new content
- if args.dry_run:
- logging.info(f"Dry run, would have written new content to
{JENKINSFILE}")
- else:
- logging.info(f"Writing new content to {JENKINSFILE}")
- with open(JENKINSFILE, "w") as f:
- f.write("".join(new_content))
+ # Re-generate the Jenkinsfiles
+ command = f"python3 {GENERATE_SCRIPT}"
Review Comment:
suggest making this a list if possible in case REPO_ROOT has spaces
##########
ci/jenkins/data.py:
##########
@@ -0,0 +1,123 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Files to stash between builds
+files = {
Review Comment:
could fix the name a bit e.g. STASH_FILES. can you document the keys and
values?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
