Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
XComp merged PR #23970: URL: https://github.com/apache/flink/pull/23970 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
XComp opened a new pull request, #24233: URL: https://github.com/apache/flink/pull/24233 ## What is the purpose of the change Backport of FLINK-33914 to enable GHA workflows also in `release-1.18` ## Brief change log * Cherry-picked FLINK-33914 changes * Conflict resolution for JDK21 related changes ## Verifying this change * Azure Pipelines should succeed. * GHA workflow should succeed. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
XComp commented on PR #23970: URL: https://github.com/apache/flink/pull/23970#issuecomment-1912346703 Deprecation warning started to appear which forced me to update the version for `actions/checkout` and `actions/cache` to `v4`. ![Uploading Screenshot from 2024-01-26 17-32-23.png…]() -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
XComp commented on PR #23970: URL: https://github.com/apache/flink/pull/23970#issuecomment-1908054230 * [This force push](https://github.com/apache/flink/compare/2d8452c5abe034a48df6b08eabd1e7088f54c1d4..1557192d0ed323a1265d11512384f720ae3bc268) updates the workflow name to include `(beta)` * [The subsequent force push](https://github.com/apache/flink/compare/1557192d0ed323a1265d11512384f720ae3bc268..a04a4c6127a9edeffa391b1801acd013fd56cd8f) adds functionality to resetting `DEBUG_FILES_OUTPUT_DIR`. This disables the warning in the `AdaptiveScheduler` stage (see [example run](https://github.com/XComp/flink/actions/runs/7634986155) w/o the fix and [another run w/ the fix](https://github.com/XComp/flink/actions/runs/7640007894)) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
XComp commented on PR #23970: URL: https://github.com/apache/flink/pull/23970#issuecomment-1904151874 The past two force pushes add the following changes: * `--privileged` parameter was added * container image was changed to the result of FLINK-34194 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
XComp commented on PR #23970: URL: https://github.com/apache/flink/pull/23970#issuecomment-1901781107 The [past force push](https://github.com/apache/flink/compare/947e77f8535285ebe45d4300e0a6e088aafeae79..081dcd6fa0aebc00eb9a8cb6277695ef431b2e49) squashes the following changes into the ready made PR: * Fixing the ubuntu version to 22.04 instead of relying on the latest * Changing CI image to a new version that is based on Ubuntu 22.04 with OpenSSL installed * Added older OpenSSL version to e2e test step (that wasn't causing any issues in the past because of FLINK-34150) * Update of the `*-artifact` actions from v3 to v4 (more details in FLINK-33923) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
XComp commented on PR #23970: URL: https://github.com/apache/flink/pull/23970#issuecomment-1901775011 The force push above is a plain rebase to most recent master to include FLINK-33907 and FLINK-34510 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
XComp commented on PR #23970: URL: https://github.com/apache/flink/pull/23970#issuecomment-1898009721 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
zentol commented on code in PR #23970:
URL: https://github.com/apache/flink/pull/23970#discussion_r1455049190
##
tools/ci/maven-utils.sh:
##
@@ -17,6 +17,7 @@
# Utility for invoking Maven in CI
function run_mvn {
if [[ "$MVN_RUN_VERBOSE" != "false" ]]; then
+ $MAVEN_WRAPPER --version
Review Comment:
indentation is off
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
XComp commented on code in PR #23970:
URL: https://github.com/apache/flink/pull/23970#discussion_r1453642234
##
pom.xml:
##
@@ -1110,7 +1122,11 @@ under the License.
org.apache.maven.plugins
maven-surefire-plugin
-
org.apache.flink.testutils.junit.FailsOnJava17
+
+
${github-actions.surefire.excludedGroups},
+
org.apache.flink.testutils.junit.FailsOnJava11,
+
org.apache.flink.testutils.junit.FailsOnJava17,
Review Comment:
Fair point. Right now, there's only a single test that's JDK-specific.
That's why I didn't bother that much. But I added a commit now that removes
this dependency between the different JDK version supports. I also fixed the
scenario where we want to run the AdaptiveScheduler with a JDK other than 8.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
zentol commented on code in PR #23970:
URL: https://github.com/apache/flink/pull/23970#discussion_r1453451950
##
pom.xml:
##
@@ -1110,7 +1122,11 @@ under the License.
org.apache.maven.plugins
maven-surefire-plugin
-
org.apache.flink.testutils.junit.FailsOnJava17
+
+
${github-actions.surefire.excludedGroups},
+
org.apache.flink.testutils.junit.FailsOnJava11,
+
org.apache.flink.testutils.junit.FailsOnJava17,
Review Comment:
Wondering if this is correct; isn't it possible for a test to fail on Java
11 but work on Java 17? E.g., tests for JDK 17 exclusive features.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
zentol commented on code in PR #23970: URL: https://github.com/apache/flink/pull/23970#discussion_r1453456239 ## pom.xml: ## @@ -998,6 +1000,28 @@ under the License. + + github-actions Review Comment: nit: move this down to to reduce the diff? ## .github/workflows/template.pre-compile-checks.yml: ## @@ -0,0 +1,76 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This workflow collects all checks that do not require compilation and are, therefore, +# JDK independent. + +name: "Pre-compile Checks" + +on: + workflow_dispatch: +inputs: + jdk_version: +description: "The JDK version that shall be used as a default within the Flink CI Docker container." +default: "8" +type: choice +options: ["8", "11", "17"] Review Comment: add 21? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
zentol commented on code in PR #23970:
URL: https://github.com/apache/flink/pull/23970#discussion_r1453451950
##
pom.xml:
##
@@ -1110,7 +1122,11 @@ under the License.
org.apache.maven.plugins
maven-surefire-plugin
-
org.apache.flink.testutils.junit.FailsOnJava17
+
+
${github-actions.surefire.excludedGroups},
+
org.apache.flink.testutils.junit.FailsOnJava11,
+
org.apache.flink.testutils.junit.FailsOnJava17,
Review Comment:
Wondering if this is correct; isn't it possible for a test to fail on Java
11 but work on Java 17? Specifically cases for JDK 17 exclusive features.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
XComp commented on PR #23970: URL: https://github.com/apache/flink/pull/23970#issuecomment-1893362185 Force push removed the `inputs.branch` parameter which is not needed anymore when you use branch-specific configuration ([related GHA workflow run](https://github.com/XComp/flink/actions/runs/7539522828)) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
XComp commented on PR #23970: URL: https://github.com/apache/flink/pull/23970#issuecomment-1893339680 [Related GHA workflow run](https://github.com/XComp/flink/actions/runs/7538904416) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
XComp commented on code in PR #23970:
URL: https://github.com/apache/flink/pull/23970#discussion_r1448735098
##
.github/workflows/template.flink-ci.yml:
##
@@ -0,0 +1,438 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Workflow template for triggering the Flink's test suite.
+
+name: "Apache Flink Test Workflow Template"
+
+on:
+ workflow_call:
+inputs:
+ workflow-caller-id:
+description: "The calling job's ID that can be used for build artifact
names (that need to be different between different jobs of the same workflow)."
+default: ""
+type: string
+ environment:
+description: "Defines environment variables for downstream scripts."
+required: true
+type: string
+ jdk_version:
+description: "The Java version to use."
+default: 8
+type: number
+ branch:
+description: "The branch the test suite should run on."
+default: "master"
+type: string
+secrets:
+ s3_bucket:
+required: false
+ s3_access_key:
+required: false
+ s3_secret_key:
+required: false
+
+permissions: read-all
+
+# Running logic within a container comes with challenges around file
permissions (e.g. when trying
+# to generate the hash for a certain set of files; see
https://github.com/actions/cache/issues/753):
+#
+# The code is cloned into the runner's workspace /home/runner/work/ but
outside the container.
+# The runner's workspace folder is then mounted to /__w within the container.
Files that were generated
+# as part of the checkout will have the runner's user as the owner. Any files
that are generated during
+# the workflow execution will have the container's user as the owner (i.e.
root). That can cause issues
+# with actions that need to access the files of the checkout.
+#
+# We can work around this issue by copying the source code to a separate
location (i.e. the container
+# user's home folder). It's important to notice that any file that is subject
to caching should still
+# live in the mounted folder to ensure accessibility by the cache action.
+env:
+ MOUNTED_WORKING_DIR: /__w/flink/flink
+ CONTAINER_LOCAL_WORKING_DIR: /root/flink
+ # The following two variables are used by the utility scripts for creating
+ # and unpacking the build artifacts.
+ FLINK_ARTIFACT_DIR: /root/artifact-directory
+ FLINK_ARTIFACT_FILENAME: flink_artifacts.tar.gz
+
+ MAVEN_REPO_FOLDER: /root/.m2/repository
+ MAVEN_ARGS: -Dmaven.repo.local=/root/.m2/repository
+ # required by tools/azure-pipelines/cache_docker_images.sh
+ DOCKER_IMAGES_CACHE_FOLDER: /root/.docker-cache
+
+jobs:
+ compile:
+name: "Compile"
+runs-on: ubuntu-latest
+container:
+ image: chesnay/flink-ci:java_8_11_17_21_maven_386
+ options: --init
+timeout-minutes: 240
+outputs:
+ stringified-workflow-name: ${{
steps.workflow-prep-step.outputs.stringified-workflow-name }}
+steps:
+ - name: "Flink Checkout"
+uses: actions/checkout@v3
+with:
+ ref: ${{ inputs.branch }}
+ persist-credentials: false
+
+ - name: "Stringify workflow name"
+id: workflow-prep-step
+run: |
+ stringified_workflow_name=$(echo "${{ github.workflow }}-${{
inputs.workflow-caller-id }}" | tr -C '[:alnum:]._' '-' | tr '[:upper:]'
'[:lower:]' | sed -e 's/--*/-/g' -e 's/^-*//g' -e 's/-*$//g')
+ echo "stringified-workflow-name=${stringified_workflow_name}" >>
$GITHUB_OUTPUT
+
+ - name: "Set JDK version to Java ${{ inputs.jdk_version }}"
+uses: "./.github/actions/set_java_in_container"
+with:
+ jdk_version: ${{ inputs.jdk_version }}
+
+ - name: "Setup Maven package cache"
+uses: actions/cache@v3
+with:
+ path: ${{ env.MAVEN_REPO_FOLDER }}
+ key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
+ restore-keys: ${{ runner.os }}-maven-
+
+ - name: "Moves relevant checkout content to container-local working
directory"
+uses: "./.github/actions/move_checkout"
+with:
+ source_directory: ${{ env.MOUNTED_WORKING_DIR }}
+ target_directory: ${{ env.CONTAINER_LO
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
XComp commented on code in PR #23970: URL: https://github.com/apache/flink/pull/23970#discussion_r1448732521 ## pom.xml: ## @@ -1104,17 +1103,19 @@ under the License. - - - - org.apache.maven.plugins - maven-surefire-plugin - - org.apache.flink.testutils.junit.FailsOnJava17 - - - - + Review Comment: Yikes, I did a mistake when initializing the default variable [in this PR](https://github.com/apache/flink/pull/23970/files#diff-9c5fb3d1b7e3b0f54bc5c4182965c4fe1f9023d449017cece3005d3f90e8e4d8R1789). This change required for each profile to have the plugin enabled. I fixed that now :+1: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
zentol commented on code in PR #23970:
URL: https://github.com/apache/flink/pull/23970#discussion_r1448720407
##
.github/actions/job_init/action.yml:
##
@@ -0,0 +1,72 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+name: "Initializes the job"
+description: "Does all the necessary steps to set up the job"
+inputs:
+ jdk_version:
+description: "The JDK version that's supposed to be used."
+required: true
+ maven_repo_folder:
+description: "The location of the local Maven repository (not setting this
parameter will omit the caching of Maven artifacts)."
+required: false
+default: ""
+ source_directory:
+description: "Specifies the directory from which the code should be moved
from (needed for containerized runs; not setting this parameter will omit
moving the checkout)."
+required: false
+default: ""
+ target_directory:
+description: "Specifies the directory to which the code should be moved to
(needed for containerized runs; not setting this parameter will omit moving the
checkout)."
+required: false
+default: ""
+runs:
+ using: "composite"
+ steps:
+- name: "Initializes GHA_PIPELINE_START_TIME with the job's start time"
+ shell: bash
+ run: |
+job_start_time="$(date --rfc-3339=seconds)"
+echo "GHA_PIPELINE_START_TIME=${job_start_time}" >> "${GITHUB_ENV}"
Review Comment:
Let's not overthink this. In the end it only matters for cases where the
setup takes an exceptionally long time AND the tests would run into a timeout,
which should be quite rare.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
zentol commented on code in PR #23970:
URL: https://github.com/apache/flink/pull/23970#discussion_r1448720407
##
.github/actions/job_init/action.yml:
##
@@ -0,0 +1,72 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+name: "Initializes the job"
+description: "Does all the necessary steps to set up the job"
+inputs:
+ jdk_version:
+description: "The JDK version that's supposed to be used."
+required: true
+ maven_repo_folder:
+description: "The location of the local Maven repository (not setting this
parameter will omit the caching of Maven artifacts)."
+required: false
+default: ""
+ source_directory:
+description: "Specifies the directory from which the code should be moved
from (needed for containerized runs; not setting this parameter will omit
moving the checkout)."
+required: false
+default: ""
+ target_directory:
+description: "Specifies the directory to which the code should be moved to
(needed for containerized runs; not setting this parameter will omit moving the
checkout)."
+required: false
+default: ""
+runs:
+ using: "composite"
+ steps:
+- name: "Initializes GHA_PIPELINE_START_TIME with the job's start time"
+ shell: bash
+ run: |
+job_start_time="$(date --rfc-3339=seconds)"
+echo "GHA_PIPELINE_START_TIME=${job_start_time}" >> "${GITHUB_ENV}"
Review Comment:
Let's not overthink this. In the end it only matters for cases where the
setup takes a long time AND the tests would run into a timeout, which should be
quite rare.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
XComp commented on code in PR #23970:
URL: https://github.com/apache/flink/pull/23970#discussion_r1448653464
##
.github/actions/job_init/action.yml:
##
@@ -0,0 +1,72 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+name: "Initializes the job"
+description: "Does all the necessary steps to set up the job"
+inputs:
+ jdk_version:
+description: "The JDK version that's supposed to be used."
+required: true
+ maven_repo_folder:
+description: "The location of the local Maven repository (not setting this
parameter will omit the caching of Maven artifacts)."
+required: false
+default: ""
+ source_directory:
+description: "Specifies the directory from which the code should be moved
from (needed for containerized runs; not setting this parameter will omit
moving the checkout)."
+required: false
+default: ""
+ target_directory:
+description: "Specifies the directory to which the code should be moved to
(needed for containerized runs; not setting this parameter will omit moving the
checkout)."
+required: false
+default: ""
+runs:
+ using: "composite"
+ steps:
+- name: "Initializes GHA_PIPELINE_START_TIME with the job's start time"
+ shell: bash
+ run: |
+job_start_time="$(date --rfc-3339=seconds)"
+echo "GHA_PIPELINE_START_TIME=${job_start_time}" >> "${GITHUB_ENV}"
Review Comment:
~on another thought. We can calculate the start time outside of the template
and pass it in as a parameter. That would include all the steps of the job run,
I guess.~ :bulb:
nah, that wouldn't work because that would include the waiting time for
runners.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
XComp commented on code in PR #23970:
URL: https://github.com/apache/flink/pull/23970#discussion_r1448653464
##
.github/actions/job_init/action.yml:
##
@@ -0,0 +1,72 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+name: "Initializes the job"
+description: "Does all the necessary steps to set up the job"
+inputs:
+ jdk_version:
+description: "The JDK version that's supposed to be used."
+required: true
+ maven_repo_folder:
+description: "The location of the local Maven repository (not setting this
parameter will omit the caching of Maven artifacts)."
+required: false
+default: ""
+ source_directory:
+description: "Specifies the directory from which the code should be moved
from (needed for containerized runs; not setting this parameter will omit
moving the checkout)."
+required: false
+default: ""
+ target_directory:
+description: "Specifies the directory to which the code should be moved to
(needed for containerized runs; not setting this parameter will omit moving the
checkout)."
+required: false
+default: ""
+runs:
+ using: "composite"
+ steps:
+- name: "Initializes GHA_PIPELINE_START_TIME with the job's start time"
+ shell: bash
+ run: |
+job_start_time="$(date --rfc-3339=seconds)"
+echo "GHA_PIPELINE_START_TIME=${job_start_time}" >> "${GITHUB_ENV}"
Review Comment:
on another thought. We can calculate the start time outside of the template
and pass it in as a parameter. That would include all the steps of the job run,
I guess. :bulb:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
zentol commented on code in PR #23970: URL: https://github.com/apache/flink/pull/23970#discussion_r1448651392 ## pom.xml: ## @@ -1171,6 +1172,19 @@ under the License. + + + org.apache.maven.plugins + maven-surefire-plugin + + Review Comment: Can we have a separate commit that sets up surefire for this plugin (+link it in FLINK-33163), and then modify it for the GHA stuff in another one? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
XComp commented on code in PR #23970:
URL: https://github.com/apache/flink/pull/23970#discussion_r1448648462
##
.github/actions/job_init/action.yml:
##
@@ -0,0 +1,72 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+name: "Initializes the job"
+description: "Does all the necessary steps to set up the job"
+inputs:
+ jdk_version:
+description: "The JDK version that's supposed to be used."
+required: true
+ maven_repo_folder:
+description: "The location of the local Maven repository (not setting this
parameter will omit the caching of Maven artifacts)."
+required: false
+default: ""
+ source_directory:
+description: "Specifies the directory from which the code should be moved
from (needed for containerized runs; not setting this parameter will omit
moving the checkout)."
+required: false
+default: ""
+ target_directory:
+description: "Specifies the directory to which the code should be moved to
(needed for containerized runs; not setting this parameter will omit moving the
checkout)."
+required: false
+default: ""
+runs:
+ using: "composite"
+ steps:
+- name: "Initializes GHA_PIPELINE_START_TIME with the job's start time"
+ shell: bash
+ run: |
+job_start_time="$(date --rfc-3339=seconds)"
+echo "GHA_PIPELINE_START_TIME=${job_start_time}" >> "${GITHUB_ENV}"
Review Comment:
It wouldn't. Docker image pulling is happening before anything we can
configure kicks in as far as I can see.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
zentol commented on code in PR #23970:
URL: https://github.com/apache/flink/pull/23970#discussion_r1448648335
##
pom.xml:
##
@@ -1104,17 +1103,19 @@ under the License.
-
-
-
-
org.apache.maven.plugins
-
maven-surefire-plugin
-
-
org.apache.flink.testutils.junit.FailsOnJava17
-
-
-
-
+
Review Comment:
can we keep it in the plugin management? A profile like this shouldn't be
_enabling_ a plugin.
##
pom.xml:
##
@@ -1067,6 +1054,18 @@ under the License.
+
+
+
org.apache.maven.plugins
+
maven-surefire-plugin
+
+
+
${github-actions.surefire.excludedGroups},
Review Comment:
Interesting that this doesn't fail if the property is empty (i.e., theres a
leading comma).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
zentol commented on code in PR #23970:
URL: https://github.com/apache/flink/pull/23970#discussion_r1448611651
##
.github/actions/job_init/action.yml:
##
@@ -0,0 +1,72 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+name: "Initializes the job"
+description: "Does all the necessary steps to set up the job"
+inputs:
+ jdk_version:
+description: "The JDK version that's supposed to be used."
+required: true
+ maven_repo_folder:
+description: "The location of the local Maven repository (not setting this
parameter will omit the caching of Maven artifacts)."
+required: false
+default: ""
+ source_directory:
+description: "Specifies the directory from which the code should be moved
from (needed for containerized runs; not setting this parameter will omit
moving the checkout)."
+required: false
+default: ""
+ target_directory:
+description: "Specifies the directory to which the code should be moved to
(needed for containerized runs; not setting this parameter will omit moving the
checkout)."
+required: false
+default: ""
+runs:
+ using: "composite"
+ steps:
+- name: "Initializes GHA_PIPELINE_START_TIME with the job's start time"
+ shell: bash
+ run: |
+job_start_time="$(date --rfc-3339=seconds)"
+echo "GHA_PIPELINE_START_TIME=${job_start_time}" >> "${GITHUB_ENV}"
Review Comment:
This should be good enough I think.
> One thing we could do here is to do a partial initial checkout that only
checks out the .github subfolder and do a full checkout in the job_init
composite action. But I opted against it for now. We could do this as a
follow-up if we run into issues due to the not so precise calculation of the
job's "real timeout time".
How would this cover the docker image pulling?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
XComp commented on code in PR #23970:
URL: https://github.com/apache/flink/pull/23970#discussion_r1448478938
##
.github/workflows/template.flink-ci.yml:
##
@@ -0,0 +1,438 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Workflow template for triggering the Flink's test suite.
+
+name: "Apache Flink Test Workflow Template"
+
+on:
+ workflow_call:
+inputs:
+ workflow-caller-id:
+description: "The calling job's ID that can be used for build artifact
names (that need to be different between different jobs of the same workflow)."
+default: ""
+type: string
+ environment:
+description: "Defines environment variables for downstream scripts."
+required: true
+type: string
+ jdk_version:
+description: "The Java version to use."
+default: 8
+type: number
+ branch:
+description: "The branch the test suite should run on."
+default: "master"
+type: string
+secrets:
+ s3_bucket:
+required: false
+ s3_access_key:
+required: false
+ s3_secret_key:
+required: false
+
+permissions: read-all
+
+# Running logic within a container comes with challenges around file
permissions (e.g. when trying
+# to generate the hash for a certain set of files; see
https://github.com/actions/cache/issues/753):
+#
+# The code is cloned into the runner's workspace /home/runner/work/ but
outside the container.
+# The runner's workspace folder is then mounted to /__w within the container.
Files that were generated
+# as part of the checkout will have the runner's user as the owner. Any files
that are generated during
+# the workflow execution will have the container's user as the owner (i.e.
root). That can cause issues
+# with actions that need to access the files of the checkout.
+#
+# We can work around this issue by copying the source code to a separate
location (i.e. the container
+# user's home folder). It's important to notice that any file that is subject
to caching should still
+# live in the mounted folder to ensure accessibility by the cache action.
+env:
+ MOUNTED_WORKING_DIR: /__w/flink/flink
+ CONTAINER_LOCAL_WORKING_DIR: /root/flink
+ # The following two variables are used by the utility scripts for creating
+ # and unpacking the build artifacts.
+ FLINK_ARTIFACT_DIR: /root/artifact-directory
+ FLINK_ARTIFACT_FILENAME: flink_artifacts.tar.gz
+
+ MAVEN_REPO_FOLDER: /root/.m2/repository
+ MAVEN_ARGS: -Dmaven.repo.local=/root/.m2/repository
+ # required by tools/azure-pipelines/cache_docker_images.sh
+ DOCKER_IMAGES_CACHE_FOLDER: /root/.docker-cache
+
+jobs:
+ compile:
+name: "Compile"
+runs-on: ubuntu-latest
+container:
+ image: chesnay/flink-ci:java_8_11_17_21_maven_386
+ options: --init
+timeout-minutes: 240
+outputs:
+ stringified-workflow-name: ${{
steps.workflow-prep-step.outputs.stringified-workflow-name }}
+steps:
+ - name: "Flink Checkout"
+uses: actions/checkout@v3
+with:
+ ref: ${{ inputs.branch }}
+ persist-credentials: false
+
+ - name: "Stringify workflow name"
+id: workflow-prep-step
+run: |
+ stringified_workflow_name=$(echo "${{ github.workflow }}-${{
inputs.workflow-caller-id }}" | tr -C '[:alnum:]._' '-' | tr '[:upper:]'
'[:lower:]' | sed -e 's/--*/-/g' -e 's/^-*//g' -e 's/-*$//g')
+ echo "stringified-workflow-name=${stringified_workflow_name}" >>
$GITHUB_OUTPUT
+
+ - name: "Set JDK version to Java ${{ inputs.jdk_version }}"
+uses: "./.github/actions/set_java_in_container"
+with:
+ jdk_version: ${{ inputs.jdk_version }}
+
+ - name: "Setup Maven package cache"
+uses: actions/cache@v3
+with:
+ path: ${{ env.MAVEN_REPO_FOLDER }}
+ key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
+ restore-keys: ${{ runner.os }}-maven-
+
+ - name: "Moves relevant checkout content to container-local working
directory"
+uses: "./.github/actions/move_checkout"
+with:
+ source_directory: ${{ env.MOUNTED_WORKING_DIR }}
+ target_directory: ${{ env.CONTAINER_LO
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
XComp commented on code in PR #23970:
URL: https://github.com/apache/flink/pull/23970#discussion_r1448473230
##
.github/actions/job_init/action.yml:
##
@@ -0,0 +1,72 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+name: "Initializes the job"
+description: "Does all the necessary steps to set up the job"
+inputs:
+ jdk_version:
+description: "The JDK version that's supposed to be used."
+required: true
+ maven_repo_folder:
+description: "The location of the local Maven repository (not setting this
parameter will omit the caching of Maven artifacts)."
+required: false
+default: ""
+ source_directory:
+description: "Specifies the directory from which the code should be moved
from (needed for containerized runs; not setting this parameter will omit
moving the checkout)."
+required: false
+default: ""
+ target_directory:
+description: "Specifies the directory to which the code should be moved to
(needed for containerized runs; not setting this parameter will omit moving the
checkout)."
+required: false
+default: ""
+runs:
+ using: "composite"
+ steps:
+- name: "Initializes GHA_PIPELINE_START_TIME with the job's start time"
+ shell: bash
+ run: |
+job_start_time="$(date --rfc-3339=seconds)"
+echo "GHA_PIPELINE_START_TIME=${job_start_time}" >> "${GITHUB_ENV}"
Review Comment:
GitHub Actions doesn't provide this environment variable (and I didn't find
anything similar) to calculate the actual timeout in `uploading_watchdog.sh`.
The problem is as well that this start up doesn't include the Docker image pull
and the git checkout which take some time.
One thing we could do here is to do a partial initial checkout that only
checks out the `.github` subfolder and do a full checkout in the `job_init`
composite action. But I opted against it for now. We could do this as a
follow-up if we run into issues due to the not so precise calculation of the
job's "real timeout time".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
XComp commented on PR #23970: URL: https://github.com/apache/flink/pull/23970#issuecomment-1885215058 I triggered another workflow with lower timeouts: https://github.com/XComp/flink/actions/runs/7478072394 ...to double-check that running into time outs works as expected. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
XComp commented on PR #23970: URL: https://github.com/apache/flink/pull/23970#issuecomment-1885016323 Ok, looks like the CI pipeline is working again: https://github.com/XComp/flink/actions/runs/7475753305/job/20344970380 (the failure in packaging/licensing happened 32019a6f6d7371c0fce23978086324f9583f9534 seems to have been lost in one of the rebasings. I cherry-picked it into the PR again ([GHA workflow](https://github.com/XComp/flink/actions/runs/7476550564)). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
XComp commented on PR #23970: URL: https://github.com/apache/flink/pull/23970#issuecomment-1884622423 I reorganized the PRs. This one is now including FLINK-34047 and the updated FLINK-33906. [This workflow run](https://github.com/XComp/flink/actions/runs/7473830290) includes all the changes and should fail because I haven't set the parameters that are need for running `tools/azure-pipelines/uploading_watchdog.sh` (see [corresponding code snippet](https://github.com/apache/flink/pull/24061/files#diff-4f7402a3476386ef15dc89110bc5b401a957e7d139a26871ee7d6ca78b73e755R41) in PR #24061) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
XComp commented on code in PR #23970:
URL: https://github.com/apache/flink/pull/23970#discussion_r1447181164
##
.github/workflows/template.flink-ci.yml:
##
@@ -0,0 +1,438 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Workflow template for triggering the Flink's test suite.
+
+name: "Apache Flink Test Workflow Template"
+
+on:
+ workflow_call:
+inputs:
+ workflow-caller-id:
+description: "The calling job's ID that can be used for build artifact
names (that need to be different between different jobs of the same workflow)."
+default: ""
+type: string
+ environment:
+description: "Defines environment variables for downstream scripts."
+required: true
+type: string
+ jdk_version:
+description: "The Java version to use."
+default: 8
+type: number
+ branch:
+description: "The branch the test suite should run on."
+default: "master"
+type: string
+secrets:
+ s3_bucket:
+required: false
+ s3_access_key:
+required: false
+ s3_secret_key:
+required: false
+
+permissions: read-all
+
+# Running logic within a container comes with challenges around file
permissions (e.g. when trying
+# to generate the hash for a certain set of files; see
https://github.com/actions/cache/issues/753):
+#
+# The code is cloned into the runner's workspace /home/runner/work/ but
outside the container.
+# The runner's workspace folder is then mounted to /__w within the container.
Files that were generated
+# as part of the checkout will have the runner's user as the owner. Any files
that are generated during
+# the workflow execution will have the container's user as the owner (i.e.
root). That can cause issues
+# with actions that need to access the files of the checkout.
+#
+# We can work around this issue by copying the source code to a separate
location (i.e. the container
+# user's home folder). It's important to notice that any file that is subject
to caching should still
+# live in the mounted folder to ensure accessibility by the cache action.
+env:
+ MOUNTED_WORKING_DIR: /__w/flink/flink
+ CONTAINER_LOCAL_WORKING_DIR: /root/flink
+ # The following two variables are used by the utility scripts for creating
+ # and unpacking the build artifacts.
+ FLINK_ARTIFACT_DIR: /root/artifact-directory
+ FLINK_ARTIFACT_FILENAME: flink_artifacts.tar.gz
+
+ MAVEN_REPO_FOLDER: /root/.m2/repository
+ MAVEN_ARGS: -Dmaven.repo.local=/root/.m2/repository
+ # required by tools/azure-pipelines/cache_docker_images.sh
+ DOCKER_IMAGES_CACHE_FOLDER: /root/.docker-cache
+
+jobs:
+ compile:
+name: "Compile"
+runs-on: ubuntu-latest
+container:
+ image: chesnay/flink-ci:java_8_11_17_21_maven_386
+ options: --init
+timeout-minutes: 240
+outputs:
+ stringified-workflow-name: ${{
steps.workflow-prep-step.outputs.stringified-workflow-name }}
+steps:
+ - name: "Flink Checkout"
+uses: actions/checkout@v3
+with:
+ ref: ${{ inputs.branch }}
+ persist-credentials: false
+
+ - name: "Stringify workflow name"
+id: workflow-prep-step
+run: |
+ stringified_workflow_name=$(echo "${{ github.workflow }}-${{
inputs.workflow-caller-id }}" | tr -C '[:alnum:]._' '-' | tr '[:upper:]'
'[:lower:]' | sed -e 's/--*/-/g' -e 's/^-*//g' -e 's/-*$//g')
+ echo "stringified-workflow-name=${stringified_workflow_name}" >>
$GITHUB_OUTPUT
+
+ - name: "Set JDK version to Java ${{ inputs.jdk_version }}"
+uses: "./.github/actions/set_java_in_container"
+with:
+ jdk_version: ${{ inputs.jdk_version }}
+
+ - name: "Setup Maven package cache"
+uses: actions/cache@v3
+with:
+ path: ${{ env.MAVEN_REPO_FOLDER }}
+ key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
+ restore-keys: ${{ runner.os }}-maven-
+
+ - name: "Moves relevant checkout content to container-local working
directory"
+uses: "./.github/actions/move_checkout"
+with:
+ source_directory: ${{ env.MOUNTED_WORKING_DIR }}
+ target_directory: ${{ env.CONTAINER_LO
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
zentol commented on code in PR #23970:
URL: https://github.com/apache/flink/pull/23970#discussion_r1447122636
##
.github/workflows/template.pre-compile-checks.yml:
##
@@ -0,0 +1,109 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This workflow collects all checks that do not require compilation and are,
therefore,
+# JDK independent.
+
+name: "Pre-compile Checks"
+
+on:
+ workflow_dispatch:
+inputs:
+ jdk_version:
+description: "The JDK version that shall be used as a default within
the Flink CI Docker container."
+default: "8"
+type: choice
+options: ["8", "11", "17"]
+ branch:
+description: "The branch the source code analysis should run on."
+default: "master"
+type: string
+
+ workflow_call:
+inputs:
+ jdk_version:
+description: "The JDK version that shall be used as a default within
the Flink CI Docker container."
+default: 8
+type: number
+ branch:
+description: "The branch the source code analysis should run on."
+default: "master"
+type: string
+
+permissions: read-all
+
+# This workflow should only contain steps that do not require the compilation
of Flink (and therefore, are
+# independent of the used JDK)
+jobs:
+ qa:
+name: "Basic QA"
+runs-on: ubuntu-latest
+container:
+ image: chesnay/flink-ci:java_8_11_17_21_maven_386
+ options: --init
+
+steps:
+ - name: "Flink Checkout"
+uses: actions/checkout@v3
+with:
+ ref: ${{ inputs.branch }}
+ persist-credentials: false
+
+ - name: "Set JDK version to Java ${{ inputs.jdk_version }}"
+uses: "./.github/actions/set_java_in_container"
+with:
+ jdk_version: ${{ inputs.jdk_version }}
+
+ - name: "Checkstyle"
+uses: "./.github/actions/run_mvn"
+with:
+ maven-parameters: "checkstyle:check -T1C"
+
+ - name: "Spotless"
+if: (success() || failure())
+uses: "./.github/actions/run_mvn"
+with:
+ maven-parameters: "spotless:check -T1C"
+
+ - name: "License Headers"
+if: (success() || failure())
+uses: "./.github/actions/run_mvn"
+with:
+ maven-parameters: "org.apache.rat:apache-rat-plugin:check -N"
+
+ docs-404-check:
+name: "Docs 404 Check"
+runs-on: ubuntu-latest
+container:
+ image: chesnay/flink-ci:java_8_11_17_21_maven_386
+steps:
+ - name: "Checks out Flink"
+uses: actions/checkout@v3
+with:
+ ref: ${{ inputs.branch }}
+ persist-credentials: false
+
+ - name: "Mark GHA checkout as a safe directory (workaround for
https://github.com/actions/checkout/issues/1169)"
+run: git config --system --add safe.directory $GITHUB_WORKSPACE
+shell: bash
+
+ - name: "Check if PR contains docs change"
+run: |
+ source ./tools/azure-pipelines/build_properties.sh
Review Comment:
sounds good
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
XComp commented on code in PR #23970:
URL: https://github.com/apache/flink/pull/23970#discussion_r1447078872
##
tools/azure-pipelines/uploading_watchdog.sh:
##
@@ -19,6 +19,17 @@
# b) It prints a warning if the test has reached 80% of it's execution time
# c) N minutes before the end of the execution time, it will start uploading
the current output as azure artifacts
+while getopts 'a:d:t:' flag; do
+ case "${flag}" in
+a) export AGENT_JOBNAME="$OPTARG";;
+d) export AGENT_TEMPDIRECTORY="$OPTARG";;
+t) export SYSTEM_JOBTIMEOUT="$OPTARG";;
+*) echo "Wrong parameter passed. $OPTARG";;
Review Comment:
You're right - I missed that one. That was a not-so-nice workaround for
making the pipeline work in GHA. I created a separate issue to fix that
properly and to check for other environment variables in FLINK-34047 to have a
clean PR for that issue. I will base this PR on
https://github.com/apache/flink/pull/24061 and resolve this change here based
on the FLINK-34047 changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
XComp commented on code in PR #23970:
URL: https://github.com/apache/flink/pull/23970#discussion_r1446999796
##
tools/azure-pipelines/create_build_artifact.sh:
##
@@ -28,15 +28,15 @@ echo "Minimizing artifact files"
# by removing files not required for subsequent stages
# jars are re-built in subsequent stages, so no need to cache them (cannot be
avoided)
-find "$FLINK_ARTIFACT_DIR" -maxdepth 8 -type f -name '*.jar' | xargs rm -rf
+find "$FLINK_ARTIFACT_DIR" -maxdepth 8 -type f -name '*.jar' -exec rm -rf {} \;
Review Comment:
I reorganized this change and moved it into the hotfix commit :+1:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
XComp commented on code in PR #23970:
URL: https://github.com/apache/flink/pull/23970#discussion_r1446291478
##
.github/workflows/template.pre-compile-checks.yml:
##
@@ -0,0 +1,109 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This workflow collects all checks that do not require compilation and are,
therefore,
+# JDK independent.
+
+name: "Pre-compile Checks"
+
+on:
+ workflow_dispatch:
+inputs:
+ jdk_version:
+description: "The JDK version that shall be used as a default within
the Flink CI Docker container."
+default: "8"
+type: choice
+options: ["8", "11", "17"]
+ branch:
+description: "The branch the source code analysis should run on."
+default: "master"
+type: string
+
+ workflow_call:
+inputs:
+ jdk_version:
+description: "The JDK version that shall be used as a default within
the Flink CI Docker container."
+default: 8
+type: number
+ branch:
+description: "The branch the source code analysis should run on."
+default: "master"
+type: string
+
+permissions: read-all
+
+# This workflow should only contain steps that do not require the compilation
of Flink (and therefore, are
+# independent of the used JDK)
+jobs:
+ qa:
+name: "Basic QA"
+runs-on: ubuntu-latest
+container:
+ image: chesnay/flink-ci:java_8_11_17_21_maven_386
+ options: --init
+
+steps:
+ - name: "Flink Checkout"
+uses: actions/checkout@v3
+with:
+ ref: ${{ inputs.branch }}
+ persist-credentials: false
+
+ - name: "Set JDK version to Java ${{ inputs.jdk_version }}"
+uses: "./.github/actions/set_java_in_container"
+with:
+ jdk_version: ${{ inputs.jdk_version }}
+
+ - name: "Checkstyle"
+uses: "./.github/actions/run_mvn"
+with:
+ maven-parameters: "checkstyle:check -T1C"
+
+ - name: "Spotless"
+if: (success() || failure())
+uses: "./.github/actions/run_mvn"
+with:
+ maven-parameters: "spotless:check -T1C"
+
+ - name: "License Headers"
+if: (success() || failure())
+uses: "./.github/actions/run_mvn"
+with:
+ maven-parameters: "org.apache.rat:apache-rat-plugin:check -N"
+
+ docs-404-check:
+name: "Docs 404 Check"
+runs-on: ubuntu-latest
+container:
+ image: chesnay/flink-ci:java_8_11_17_21_maven_386
+steps:
+ - name: "Checks out Flink"
+uses: actions/checkout@v3
+with:
+ ref: ${{ inputs.branch }}
+ persist-credentials: false
+
+ - name: "Mark GHA checkout as a safe directory (workaround for
https://github.com/actions/checkout/issues/1169)"
+run: git config --system --add safe.directory $GITHUB_WORKSPACE
+shell: bash
+
+ - name: "Check if PR contains docs change"
+run: |
+ source ./tools/azure-pipelines/build_properties.sh
Review Comment:
Created FLINK-34045 as a follow-up. I'm gonna remove the doc creation in
this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
XComp commented on code in PR #23970: URL: https://github.com/apache/flink/pull/23970#discussion_r1446358342 ## .github/actions/run_mvn/action.yml: ## @@ -0,0 +1,42 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +--- +name: "Runs Maven Command" Review Comment: You're right, I used the wrong environment variable in my comment above. The example should be `GITHUB_ENV`. The JDK setting is added to `GITHUB_ENV` in the `set_java_in_container` custom action. It's not "automagically" added to the environment. I did test runs for bash functions and exporting env variables (see [workflow](https://github.com/XComp/github-actions-playground/actions/runs/7463171863/job/20307209534#step:4:5)). Neither the function nor the variable are passed down to subsequent steps if you don't use `GITHUB_ENV` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
XComp commented on code in PR #23970:
URL: https://github.com/apache/flink/pull/23970#discussion_r1446291478
##
.github/workflows/template.pre-compile-checks.yml:
##
@@ -0,0 +1,109 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This workflow collects all checks that do not require compilation and are,
therefore,
+# JDK independent.
+
+name: "Pre-compile Checks"
+
+on:
+ workflow_dispatch:
+inputs:
+ jdk_version:
+description: "The JDK version that shall be used as a default within
the Flink CI Docker container."
+default: "8"
+type: choice
+options: ["8", "11", "17"]
+ branch:
+description: "The branch the source code analysis should run on."
+default: "master"
+type: string
+
+ workflow_call:
+inputs:
+ jdk_version:
+description: "The JDK version that shall be used as a default within
the Flink CI Docker container."
+default: 8
+type: number
+ branch:
+description: "The branch the source code analysis should run on."
+default: "master"
+type: string
+
+permissions: read-all
+
+# This workflow should only contain steps that do not require the compilation
of Flink (and therefore, are
+# independent of the used JDK)
+jobs:
+ qa:
+name: "Basic QA"
+runs-on: ubuntu-latest
+container:
+ image: chesnay/flink-ci:java_8_11_17_21_maven_386
+ options: --init
+
+steps:
+ - name: "Flink Checkout"
+uses: actions/checkout@v3
+with:
+ ref: ${{ inputs.branch }}
+ persist-credentials: false
+
+ - name: "Set JDK version to Java ${{ inputs.jdk_version }}"
+uses: "./.github/actions/set_java_in_container"
+with:
+ jdk_version: ${{ inputs.jdk_version }}
+
+ - name: "Checkstyle"
+uses: "./.github/actions/run_mvn"
+with:
+ maven-parameters: "checkstyle:check -T1C"
+
+ - name: "Spotless"
+if: (success() || failure())
+uses: "./.github/actions/run_mvn"
+with:
+ maven-parameters: "spotless:check -T1C"
+
+ - name: "License Headers"
+if: (success() || failure())
+uses: "./.github/actions/run_mvn"
+with:
+ maven-parameters: "org.apache.rat:apache-rat-plugin:check -N"
+
+ docs-404-check:
+name: "Docs 404 Check"
+runs-on: ubuntu-latest
+container:
+ image: chesnay/flink-ci:java_8_11_17_21_maven_386
+steps:
+ - name: "Checks out Flink"
+uses: actions/checkout@v3
+with:
+ ref: ${{ inputs.branch }}
+ persist-credentials: false
+
+ - name: "Mark GHA checkout as a safe directory (workaround for
https://github.com/actions/checkout/issues/1169)"
+run: git config --system --add safe.directory $GITHUB_WORKSPACE
+shell: bash
+
+ - name: "Check if PR contains docs change"
+run: |
+ source ./tools/azure-pipelines/build_properties.sh
Review Comment:
Created FLINK-34045 as a follow-up
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
XComp commented on code in PR #23970:
URL: https://github.com/apache/flink/pull/23970#discussion_r1446284364
##
.github/workflows/template.pre-compile-checks.yml:
##
@@ -0,0 +1,109 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This workflow collects all checks that do not require compilation and are,
therefore,
+# JDK independent.
+
+name: "Pre-compile Checks"
+
+on:
+ workflow_dispatch:
+inputs:
+ jdk_version:
+description: "The JDK version that shall be used as a default within
the Flink CI Docker container."
+default: "8"
+type: choice
+options: ["8", "11", "17"]
+ branch:
+description: "The branch the source code analysis should run on."
+default: "master"
+type: string
+
+ workflow_call:
+inputs:
+ jdk_version:
+description: "The JDK version that shall be used as a default within
the Flink CI Docker container."
+default: 8
+type: number
+ branch:
+description: "The branch the source code analysis should run on."
+default: "master"
+type: string
+
+permissions: read-all
+
+# This workflow should only contain steps that do not require the compilation
of Flink (and therefore, are
+# independent of the used JDK)
+jobs:
+ qa:
+name: "Basic QA"
+runs-on: ubuntu-latest
+container:
+ image: chesnay/flink-ci:java_8_11_17_21_maven_386
+ options: --init
+
+steps:
+ - name: "Flink Checkout"
+uses: actions/checkout@v3
+with:
+ ref: ${{ inputs.branch }}
+ persist-credentials: false
+
+ - name: "Set JDK version to Java ${{ inputs.jdk_version }}"
+uses: "./.github/actions/set_java_in_container"
+with:
+ jdk_version: ${{ inputs.jdk_version }}
+
+ - name: "Checkstyle"
+uses: "./.github/actions/run_mvn"
+with:
+ maven-parameters: "checkstyle:check -T1C"
+
+ - name: "Spotless"
+if: (success() || failure())
+uses: "./.github/actions/run_mvn"
+with:
+ maven-parameters: "spotless:check -T1C"
+
+ - name: "License Headers"
+if: (success() || failure())
+uses: "./.github/actions/run_mvn"
+with:
+ maven-parameters: "org.apache.rat:apache-rat-plugin:check -N"
+
+ docs-404-check:
+name: "Docs 404 Check"
+runs-on: ubuntu-latest
+container:
+ image: chesnay/flink-ci:java_8_11_17_21_maven_386
+steps:
+ - name: "Checks out Flink"
+uses: actions/checkout@v3
+with:
+ ref: ${{ inputs.branch }}
+ persist-credentials: false
+
+ - name: "Mark GHA checkout as a safe directory (workaround for
https://github.com/actions/checkout/issues/1169)"
+run: git config --system --add safe.directory $GITHUB_WORKSPACE
+shell: bash
+
+ - name: "Check if PR contains docs change"
+run: |
+ source ./tools/azure-pipelines/build_properties.sh
Review Comment:
I'm wondering whether we should remove this part of the workflow for now and
create a separate issue. Just to not increase the PR even more. :thinking:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
XComp commented on code in PR #23970:
URL: https://github.com/apache/flink/pull/23970#discussion_r1446280701
##
.github/workflows/template.pre-compile-checks.yml:
##
@@ -0,0 +1,109 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This workflow collects all checks that do not require compilation and are,
therefore,
+# JDK independent.
+
+name: "Pre-compile Checks"
+
+on:
+ workflow_dispatch:
+inputs:
+ jdk_version:
+description: "The JDK version that shall be used as a default within
the Flink CI Docker container."
+default: "8"
+type: choice
+options: ["8", "11", "17"]
+ branch:
+description: "The branch the source code analysis should run on."
+default: "master"
+type: string
+
+ workflow_call:
+inputs:
+ jdk_version:
+description: "The JDK version that shall be used as a default within
the Flink CI Docker container."
+default: 8
+type: number
+ branch:
+description: "The branch the source code analysis should run on."
+default: "master"
+type: string
+
+permissions: read-all
+
+# This workflow should only contain steps that do not require the compilation
of Flink (and therefore, are
+# independent of the used JDK)
+jobs:
+ qa:
+name: "Basic QA"
+runs-on: ubuntu-latest
+container:
+ image: chesnay/flink-ci:java_8_11_17_21_maven_386
+ options: --init
+
+steps:
+ - name: "Flink Checkout"
+uses: actions/checkout@v3
+with:
+ ref: ${{ inputs.branch }}
+ persist-credentials: false
+
+ - name: "Set JDK version to Java ${{ inputs.jdk_version }}"
+uses: "./.github/actions/set_java_in_container"
+with:
+ jdk_version: ${{ inputs.jdk_version }}
+
+ - name: "Checkstyle"
+uses: "./.github/actions/run_mvn"
+with:
+ maven-parameters: "checkstyle:check -T1C"
+
+ - name: "Spotless"
+if: (success() || failure())
+uses: "./.github/actions/run_mvn"
+with:
+ maven-parameters: "spotless:check -T1C"
+
+ - name: "License Headers"
+if: (success() || failure())
+uses: "./.github/actions/run_mvn"
+with:
+ maven-parameters: "org.apache.rat:apache-rat-plugin:check -N"
+
+ docs-404-check:
+name: "Docs 404 Check"
+runs-on: ubuntu-latest
+container:
+ image: chesnay/flink-ci:java_8_11_17_21_maven_386
+steps:
+ - name: "Checks out Flink"
+uses: actions/checkout@v3
+with:
+ ref: ${{ inputs.branch }}
+ persist-credentials: false
+
+ - name: "Mark GHA checkout as a safe directory (workaround for
https://github.com/actions/checkout/issues/1169)"
+run: git config --system --add safe.directory $GITHUB_WORKSPACE
+shell: bash
+
+ - name: "Check if PR contains docs change"
+run: |
+ source ./tools/azure-pipelines/build_properties.sh
Review Comment:
It's generally buggy when comparing this part of the code with how we do it
in the azure pipelines
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
XComp commented on code in PR #23970:
URL: https://github.com/apache/flink/pull/23970#discussion_r1446269230
##
.github/workflows/template.pre-compile-checks.yml:
##
@@ -0,0 +1,109 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This workflow collects all checks that do not require compilation and are,
therefore,
+# JDK independent.
+
+name: "Pre-compile Checks"
+
+on:
+ workflow_dispatch:
+inputs:
+ jdk_version:
+description: "The JDK version that shall be used as a default within
the Flink CI Docker container."
+default: "8"
+type: choice
+options: ["8", "11", "17"]
+ branch:
+description: "The branch the source code analysis should run on."
+default: "master"
+type: string
+
+ workflow_call:
+inputs:
+ jdk_version:
+description: "The JDK version that shall be used as a default within
the Flink CI Docker container."
+default: 8
+type: number
+ branch:
+description: "The branch the source code analysis should run on."
+default: "master"
+type: string
+
+permissions: read-all
+
+# This workflow should only contain steps that do not require the compilation
of Flink (and therefore, are
+# independent of the used JDK)
+jobs:
+ qa:
+name: "Basic QA"
+runs-on: ubuntu-latest
+container:
+ image: chesnay/flink-ci:java_8_11_17_21_maven_386
+ options: --init
+
+steps:
+ - name: "Flink Checkout"
+uses: actions/checkout@v3
+with:
+ ref: ${{ inputs.branch }}
+ persist-credentials: false
+
+ - name: "Set JDK version to Java ${{ inputs.jdk_version }}"
+uses: "./.github/actions/set_java_in_container"
+with:
+ jdk_version: ${{ inputs.jdk_version }}
+
+ - name: "Checkstyle"
+uses: "./.github/actions/run_mvn"
+with:
+ maven-parameters: "checkstyle:check -T1C"
+
+ - name: "Spotless"
+if: (success() || failure())
+uses: "./.github/actions/run_mvn"
+with:
+ maven-parameters: "spotless:check -T1C"
+
+ - name: "License Headers"
+if: (success() || failure())
+uses: "./.github/actions/run_mvn"
+with:
+ maven-parameters: "org.apache.rat:apache-rat-plugin:check -N"
+
+ docs-404-check:
+name: "Docs 404 Check"
+runs-on: ubuntu-latest
+container:
+ image: chesnay/flink-ci:java_8_11_17_21_maven_386
+steps:
+ - name: "Checks out Flink"
+uses: actions/checkout@v3
+with:
+ ref: ${{ inputs.branch }}
+ persist-credentials: false
+
+ - name: "Mark GHA checkout as a safe directory (workaround for
https://github.com/actions/checkout/issues/1169)"
+run: git config --system --add safe.directory $GITHUB_WORKSPACE
+shell: bash
+
+ - name: "Check if PR contains docs change"
+run: |
+ source ./tools/azure-pipelines/build_properties.sh
Review Comment:
tbh that was running right from the start when I picked up the work from you
:innocent: I haven't looked into it, yet. But skimming over it, you seem to be
right: The function `github_num_commits` seems to rely on Azure Pipeline env
variables which makes the check always succeed (see [sample
run](https://github.com/XComp/flink/actions/runs/7461657259/job/20303196275#step:5:15)).
This results in always building the docs. Good catch! :+1:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
zentol commented on code in PR #23970: URL: https://github.com/apache/flink/pull/23970#discussion_r1446257865 ## .github/actions/run_mvn/action.yml: ## @@ -0,0 +1,42 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +--- +name: "Runs Maven Command" Review Comment: Wouldn't you export into GITHUB_ENV? I don't know if these are automatically added to subsequent steps, but I'd think they have to be or else the JDK setting wouldn't work🤔 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
zentol commented on code in PR #23970: URL: https://github.com/apache/flink/pull/23970#discussion_r1446254396 ## pom.xml: ## @@ -998,6 +998,21 @@ under the License. + + github-actions + + + + org.apache.maven.plugins + maven-surefire-plugin + + org.apache.flink.testutils.junit.FailsInGHAContainerWithRootUser Review Comment: I'll double check this as well because I may have tried that in the past 🤔 There's also more esoteric options but let's not get into those just yet... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
XComp commented on code in PR #23970: URL: https://github.com/apache/flink/pull/23970#discussion_r1446249763 ## .github/actions/run_mvn/action.yml: ## @@ -0,0 +1,42 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +--- +name: "Runs Maven Command" Review Comment: The environment is not shared between steps. For environment variables, we would have to use the `echo "=" >> $GITHUB_OUTPUT` to make the variables available. I'm hesitent to try exporting functions into `GITHUB_OUTPUT`, though. The custom action for Maven commands gives us the ability to call the `run_mvn` also from within the GHA workflow. It seems like the steps are still executed in some kind of sandbox. That's why I feel like source'ing `maven-utils.sh` each time should be ok. :thinking: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
XComp commented on code in PR #23970: URL: https://github.com/apache/flink/pull/23970#discussion_r1446249763 ## .github/actions/run_mvn/action.yml: ## @@ -0,0 +1,42 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +--- +name: "Runs Maven Command" Review Comment: The environment is not shared between steps. For environment variables, we would have to use the `echo "=" >> $GITHUB_OUTPUT` to make the variables available. I'm hesitent to try exporting functions into `GITHUB_OUTPUT`, though. The custom action for Maven commands gives us the ability to call the `run_mvn` also from within the GHA workflow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
XComp commented on code in PR #23970: URL: https://github.com/apache/flink/pull/23970#discussion_r1446151938 ## pom.xml: ## @@ -998,6 +998,21 @@ under the License. + + github-actions + + + + org.apache.maven.plugins + maven-surefire-plugin + + org.apache.flink.testutils.junit.FailsInGHAContainerWithRootUser Review Comment: I actually tried this and failed. But trying it once more was successful. :+1: thanks for the hint. That's definitely a better more straight-forward approach. I wasn't happy with the one I came up with in the first place. :innocent: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
zentol commented on code in PR #23970:
URL: https://github.com/apache/flink/pull/23970#discussion_r1445866846
##
.github/actions/run_mvn/action.yml:
##
@@ -0,0 +1,42 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+name: "Runs Maven Command"
Review Comment:
Don't quite understand yet why we have this. bit concerned about repeatedly
sourcing `maven-utils.sh`.
##
pom.xml:
##
@@ -998,6 +998,21 @@ under the License.
+
+ github-actions
+
+
+
+
org.apache.maven.plugins
+
maven-surefire-plugin
+
+
org.apache.flink.testutils.junit.FailsInGHAContainerWithRootUser
Review Comment:
What you can do alternatively is set a variable `ghaExcludeFails...` to
`,org.apache.flink...` and reference said variable in the other profiles.
`${ghaExcludeFails...}MyOtherFancyExclusion`
I think the profiles could set such a variable and the base surefire config
has the references.
##
.github/actions/run_mvn/action.yml:
##
@@ -0,0 +1,42 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+name: "Runs Maven Command"
+description: "Sets the Java version within Flink' CI Docker container"
Review Comment:
outdated
##
.github/workflows/template.flink-ci.yml:
##
@@ -0,0 +1,438 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Workflow template for triggering the Flink's test suite.
+
+name: "Apache Flink Test Workflow Template"
+
+on:
+ workflow_call:
+inputs:
+ workflow-caller-id:
+description: "The calling job's ID that can be used for build artifact
names (that need to be different between different jobs of the same workflow)."
+default: ""
+type: string
+ environment:
+description: "Defines environment variables for downstream scripts."
+required: true
+type: string
+ jdk_version:
+description: "The Java version to use."
+default: 8
+type: number
+ branch:
+description: "The branch the test suite should run on."
+default: "master"
+type: string
+secrets:
+ s3_bucket:
+required: false
+ s3_access_key:
+required: false
+ s3_secret_key:
+required: false
+
+permissions: read-all
+
+# Running logic within a container comes with challenges around file
permissions (e.g. when trying
+# to ge
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
zentol commented on code in PR #23970:
URL: https://github.com/apache/flink/pull/23970#discussion_r1445855300
##
tools/azure-pipelines/unpack_build_artifact.sh:
##
@@ -28,15 +28,14 @@ cp -RT "$FLINK_ARTIFACT_DIR" "."
echo "Adjusting timestamps"
# adjust timestamps of proto file to avoid re-generation
-find . -type f -name '*.proto' | xargs touch
+find . -type f -name '*.proto' -exec touch {} \;
# wait a bit for better odds of different timestamps
sleep 5
# adjust timestamps to prevent recompilation
-find . -type f -name '*.java' | xargs touch
-find . -type f -name '*.scala' | xargs touch
+find . -type f -name '*.java' -exec touch {} \;
+find . -type f -name '*.scala' -exec touch {} \;
Review Comment:
ah nice, I ran into this so many times when experimenting with CI...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
snuyanzin commented on code in PR #23970: URL: https://github.com/apache/flink/pull/23970#discussion_r1433730346 ## .github/actions/set_java_in_container/action.yml: ## @@ -0,0 +1,32 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +--- +name: "Sets Java version" +description: "Sets the Java version within Flink' CI Docker container" +inputs: + jdk-version: Review Comment: nit ```suggestion jdk_version: ``` I would suggest to use underscore to have same names as for connector's gha https://github.com/apache/flink-connector-shared-utils/blob/969edc5814d021d636b5fed1877f7932214be7ea/.github/workflows/ci.yml#L30 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
flinkbot commented on PR #23970: URL: https://github.com/apache/flink/pull/23970#issuecomment-1864549461 ## CI report: * ff8cb87a2880efbf3c210b31fab2d79a6c57ddee UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
XComp opened a new pull request, #23970: URL: https://github.com/apache/flink/pull/23970 Based on the following PR(s): * https://github.com/apache/flink/pull/23961 * https://github.com/apache/flink/pull/23962 * https://github.com/apache/flink/pull/23963 * https://github.com/apache/flink/pull/23964 * https://github.com/apache/flink/pull/23965 ## What is the purpose of the change Creates CI workflow that runs on Flink's default setup in branches. ## Brief change log * Adds custom actions for selecting the Java version * Adds custom action for moving data within the container * Adds template for pre-compile checks * Adds template for executing Flink's test suite * Adds basic CI workflow ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
