This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 1fd6d51e4f Remove submodules specification from checkout steps in GA
(#35454)
1fd6d51e4f is described below
commit 1fd6d51e4f70932ff23d1a7038b4fb298044cf44
Author: Jarek Potiuk <[email protected]>
AuthorDate: Sun Nov 5 15:32:35 2023 +0100
Remove submodules specification from checkout steps in GA (#35454)
We do not have submodules any more in our repository, so we
can safely remove them fro Github Actions and from all the other
places where we referred to them.
---
.github/workflows/build-images.yml | 7 --
.github/workflows/ci.yml | 9 ---
.github/workflows/release_dockerhub_image.yml | 1 -
.gitmodules | 0
...-using-contributed-code-when-building-images.md | 3 +
.../doc/adr/0010-use-pipx-to-install-breeze.md | 2 +-
dev/breeze/doc/adr/0013-get-rid-of-submodules.md | 90 ++++++++++++++++++++++
7 files changed, 94 insertions(+), 18 deletions(-)
diff --git a/.github/workflows/build-images.yml
b/.github/workflows/build-images.yml
index f95b0c4d8b..749aa58312 100644
--- a/.github/workflows/build-images.yml
+++ b/.github/workflows/build-images.yml
@@ -153,7 +153,6 @@ jobs:
path: "target-airflow"
ref: ${{ github.base_ref }}
persist-credentials: false
- submodules: recursive
- name: >
Override "scripts/ci", "dev" and ".github/actions" with the target
branch
so that the PR does not override it
@@ -209,14 +208,12 @@ jobs:
with:
ref: ${{ needs.build-info.outputs.target-commit-sha }}
persist-credentials: false
- submodules: recursive
- name: Checkout target branch to 'target-airflow' folder to use
ci/scripts and breeze from there.
uses: actions/checkout@v4
with:
path: "target-airflow"
ref: ${{ github.base_ref }}
persist-credentials: false
- submodules: recursive
- name: >
Override "scripts/ci", "dev" and ".github/actions" with the target
branch
so that the PR does not override it
@@ -282,14 +279,12 @@ jobs:
with:
ref: ${{ needs.build-info.outputs.target-commit-sha }}
persist-credentials: false
- submodules: recursive
- name: Checkout target branch to 'target-airflow' folder to use
ci/scripts and breeze from there.
uses: actions/checkout@v4
with:
path: "target-airflow"
ref: ${{ github.base_ref }}
persist-credentials: false
- submodules: recursive
- name: >
Override "scripts/ci", "dev" and ".github/actions" with the target
branch
so that the PR does not override it
@@ -343,14 +338,12 @@ jobs:
with:
ref: ${{ needs.build-info.outputs.target-commit-sha }}
persist-credentials: false
- submodules: recursive
- name: Checkout target branch to 'target-airflow' folder to use
ci/scripts and breeze from there.
uses: actions/checkout@v4
with:
path: "target-airflow"
ref: ${{ github.base_ref }}
persist-credentials: false
- submodules: recursive
- name: >
Override "scripts/ci", "dev" and ".github/actions" with the target
branch
so that the PR does not override it
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 5bf2e916e2..20bb983886 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -135,7 +135,6 @@ jobs:
uses: actions/checkout@v4
with:
persist-credentials: false
- submodules: recursive
- name: Fetch incoming commit ${{ github.sha }} with its parent
uses: actions/checkout@v4
with:
@@ -318,7 +317,6 @@ jobs:
with:
ref: ${{ needs.build-info.outputs.targetCommitSha }}
persist-credentials: false
- submodules: recursive
if: needs.build-info.outputs.in-workflow-build == 'true'
- name: >
Build CI Images
@@ -353,7 +351,6 @@ jobs:
uses: actions/checkout@v4
with:
persist-credentials: false
- submodules: recursive
- name: "Install Breeze"
uses: ./.github/actions/breeze
- name: Pull CI images ${{ env.PYTHON_VERSIONS }}:${{ env.IMAGE_TAG }}
@@ -417,7 +414,6 @@ jobs:
with:
ref: ${{ needs.build-info.outputs.targetCommitSha }}
persist-credentials: false
- submodules: recursive
if: needs.build-info.outputs.in-workflow-build == 'true'
- name: >
Build PROD Images
@@ -538,7 +534,6 @@ jobs:
with:
fetch-depth: 2
persist-credentials: false
-
wait-for-ci-images:
timeout-minutes: 120
name: "Wait for CI images"
@@ -684,7 +679,6 @@ jobs:
uses: actions/checkout@v4
with:
persist-credentials: false
- submodules: recursive
- name: >
Prepare breeze & CI image:
${{needs.build-info.outputs.default-python-version}}:${{env.IMAGE_TAG}}
uses: ./.github/actions/prepare_breeze_and_image
@@ -1596,7 +1590,6 @@ jobs:
uses: actions/checkout@v4
with:
persist-credentials: false
- submodules: recursive
- name: "Download all artifacts from the current build"
uses: actions/download-artifact@v3
with:
@@ -1783,7 +1776,6 @@ jobs:
with:
# Needed to perform push action
persist-credentials: false
- submodules: recursive
- name: "Set constraints branch name"
id: constraints-branch
run: ./scripts/ci/constraints/ci_branch_constraints.sh >>
${GITHUB_OUTPUT}
@@ -1944,7 +1936,6 @@ jobs:
with:
ref: ${{ needs.build-info.outputs.targetCommitSha }}
persist-credentials: false
- submodules: recursive
- name: "Install Breeze"
uses: ./.github/actions/breeze
- name: "Start ARM instance"
diff --git a/.github/workflows/release_dockerhub_image.yml
b/.github/workflows/release_dockerhub_image.yml
index 2053dc5b9d..9f898eb584 100644
--- a/.github/workflows/release_dockerhub_image.yml
+++ b/.github/workflows/release_dockerhub_image.yml
@@ -58,7 +58,6 @@ jobs:
uses: actions/checkout@v4
with:
persist-credentials: false
- submodules: recursive
- name: "Install Breeze"
uses: ./.github/actions/breeze
- name: Selective checks
diff --git a/.gitmodules b/.gitmodules
deleted file mode 100644
index e69de29bb2..0000000000
diff --git
a/dev/breeze/doc/adr/0005-preventing-using-contributed-code-when-building-images.md
b/dev/breeze/doc/adr/0005-preventing-using-contributed-code-when-building-images.md
index 9e051fdc00..1c334b03ab 100644
---
a/dev/breeze/doc/adr/0005-preventing-using-contributed-code-when-building-images.md
+++
b/dev/breeze/doc/adr/0005-preventing-using-contributed-code-when-building-images.md
@@ -120,6 +120,9 @@ but to make sure that the following rules are in-place:
SHA (not tag) and integration with Pull Request Review process when someone
creates a PR to upgrade the
action, which makes it ideal to securely and seamlessly keep the action
updated if needed.
+ UPDATE: The decision for using submodules have been changed in subsequent
ADR
+ [0013-get-rid-of-submodules.md](0013-get-rid-of-submodules.md) to use
full-length SHA references.
+
3) No user code coming from the PR can be executed directly in the "Build
image" workflow. For example, the
build scripts should not import `setup.py` or execute bash scripts coming
from other places than:
* scripts/ci
diff --git a/dev/breeze/doc/adr/0010-use-pipx-to-install-breeze.md
b/dev/breeze/doc/adr/0010-use-pipx-to-install-breeze.md
index 6f357b1c9c..c309440a40 100644
--- a/dev/breeze/doc/adr/0010-use-pipx-to-install-breeze.md
+++ b/dev/breeze/doc/adr/0010-use-pipx-to-install-breeze.md
@@ -37,7 +37,7 @@ Date: 2022-04-04
Accepted
-Supersedes [--](0003-bootstrapping-virtual-environment.md)
+Supersedes [3. Bootstrapping virtual
environment](0003-bootstrapping-virtual-environment.md)
## Context
diff --git a/dev/breeze/doc/adr/0013-get-rid-of-submodules.md
b/dev/breeze/doc/adr/0013-get-rid-of-submodules.md
new file mode 100644
index 0000000000..3ace81ad37
--- /dev/null
+++ b/dev/breeze/doc/adr/0013-get-rid-of-submodules.md
@@ -0,0 +1,90 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+<!-- START doctoc generated TOC please keep comment here to allow auto update
-->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+**Table of Contents** *generated with
[DocToc](https://github.com/thlorenz/doctoc)*
+
+- [13. Get rid of submodules](#13-get-rid-of-submodules)
+ - [Status](#status)
+ - [Context](#context)
+ - [Decision](#decision)
+ - [Consequences](#consequences)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+
+# 13. Get rid of submodules
+
+Date: 2023-11-04
+
+## Status
+
+Accepted
+
+Builds on [5. Prevents using contributed code when building
images](0005-preventing-using-contributed-code-when-building-images.md)
+
+## Context
+
+Airflow Build system has to be secure and follow Apache Software Foundation
guidelines in order to prevent
+Supply chain attacks, we want to avoid possibility where 3rd-party contributed
code is executed during the
+build system in order to inject code into the artifacts we release to our
users.
+
+While we have a number of safeguards in place (protected branches, releasing
only source code, reviews etc.)
+we are also using 3rd-party GitHub Actions in our build and we need to make
sure unknown/potentially
+malicious code is not injected by people who manage the actions without our
knowledge.
+
+Previously in
[0005-preventing-using-contributed-code-when-building-images.md](0005-preventing-using-contributed-code-when-building-images.md)
+we decided to use ``submodules`` to keep references to those actions as
submodule links
+the repository including a specific commit and when you review the code when
adding submodule the code cannot
+change without running "submodule update" and committing resulting change.
+
+However submodules are rarely used in general, are very arcane knowledge and
if you use them very rarely,
+it is not well known or intuitive how to keep such repository updated. And
over time it is also important
+to have an easy way to keep such 3rd-party actions updated, because of
security updates of their dependencies
+and retention policies of GitHub. Some of our actions raised deprecation
warnings about old Nodejs version
+for example and the only way was to update the actions.
+
+Updating the submodules then becomes a bit of "arcane chore" - which is quite
a bit of contradiction. Chores
+by definitions should be easy to do, otherwise they are not getting done
regularly.
+
+## Decision
+
+The decision is to use the approach suggested by
+[Github Actions Security
Hardening](https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions#using-third-party-actions)
+by simply pinning the potentially insecure GitHub Actions via full-length SHA
of the commit
+
+The point 2) about submodules should take the following form
+
+2) For security reason for non-GitHub owned actions (which are heavily
protected asset of Github Actions
+ and compromising them would mean compromising the whole GitHub Actions
infrastructure) we use full-length
+ SHA of the action to refer to the action code. We do it after careful
review of the action code - with the
+ comment explaining version of the action the SHA refers to. When we upgrade
the action we update the SHA
+ and the version after careful review of the new action code - ideally
comparing the code to the SHA we had
+ in the past, to make sure that no new unwanted code has been added.
+
+Example:
+
+```
+uses:
aws-actions/configure-aws-credentials@010d0da01d0b5a38af31e9c3470dbfdabdecca3a
# v4.0.1
+```
+
+## Consequences
+
+It should be easier to keep our actions updated, we are also going to get rid
of the submodules from Airflow
+repository and can remove any usage of submodules when checking out the code.
