kou commented on code in PR #47600:
URL: https://github.com/apache/arrow/pull/47600#discussion_r2397529260
##########
dev/release/binary-task.rb:
##########
@@ -1876,7 +1851,6 @@ def available_yum_targets
["almalinux", "8"],
["amazon-linux", "2023"],
["centos", "9-stream"],
- ["centos", "8-stream"],
Review Comment:
We already dropped support for CentOS Stream 8:
https://github.com/apache/arrow/pull/46953
##########
.github/workflows/release_candidate.yml:
##########
@@ -23,9 +23,7 @@ on:
- '**'
- '!dependabot/**'
tags:
- # Trigger workflow when a tag whose name matches the pattern
- # "apache-arrow-{MAJOR}.{MINOR}.{PATCH}-rc{RC_NUM}" is pushed.
- - "apache-arrow-[0-9]+.[0-9]+.[0-9]+-rc[0-9]+"
+ - "apache-arrow-*-rc*"
Review Comment:
Simplify and unify tag patterns.
##########
dev/release/binary-task.rb:
##########
@@ -1649,41 +1649,16 @@ def define_apt_rc_tasks
pool_dir = "#{distribution_dir}/pool/#{code_name}"
rm_rf(pool_dir, verbose: verbose?)
mkdir_p(pool_dir, verbose: verbose?)
+
source_dir_prefix = "#{artifacts_dir}/#{distribution}-#{code_name}"
- # apache/arrow uses debian-bookworm-{amd64,arm64} but
- # apache/arrow-adbc uses debian-bookworm. So the following
+ # apache/arrow uses debian-bookworm-{amd64,arm64}.tar.gz but
+ # apache/arrow-adbc uses debian-bookworm.tar.gz So the following
# glob must much both of them.
- Dir.glob("#{source_dir_prefix}*/*") do |path|
- base_name = File.basename(path)
- package_name = ENV["DEB_PACKAGE_NAME"]
- if package_name.nil? or package_name.empty?
- if base_name.start_with?("apache-arrow-apt-source")
- package_name = "apache-arrow-apt-source"
- else
- package_name = "apache-arrow"
- end
- end
- destination_path = [
- pool_dir,
- component,
- package_name[0],
- package_name,
- base_name,
- ].join("/")
- copy_artifact(path,
- destination_path,
- progress_reporter)
- case base_name
- when /\A[^_]+-apt-source_.*\.deb\z/
- latest_apt_source_package_path = [
- distribution_dir,
- "#{package_name}-latest-#{code_name}.deb"
- ].join("/")
- copy_artifact(path,
- latest_apt_source_package_path,
- progress_reporter)
- end
+ Dir.glob("#{source_dir_prefix}*.tar.gz") do |tar_gz|
+ sh("tar", "xf", tar_gz, "-C", incoming_dir)
+ progress_reporter.advance
end
Review Comment:
We can use this for ADBC too but we need some changes in ADBC:
https://github.com/apache/arrow-adbc/issues/3510
##########
dev/tasks/linux-packages/helper.rb:
##########
@@ -72,13 +72,12 @@ def detect_env(name)
raise "Failed to detect #{name} environment variable"
end
- def detect_repo
- detect_env("REPO")
+ def github_repository
+ ENV["GITHUB_REPOSITORY"] || "apache/arrow"
Review Comment:
GitHub Actions provides `GITHUB_REPOSITORY` environment variable by default.
So this uses fork's GitHub Container registry on fork automatically.
##########
.github/workflows/package_linux.yml:
##########
@@ -0,0 +1,301 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: Package Linux
+
+on:
+ push:
+ branches:
+ - '**'
+ - '!dependabot/**'
+ paths:
+ - '.dockerignore'
+ - '.github/workflows/check_labels.yml'
+ - '.github/workflows/package_linux.yml'
+ - '.github/workflows/report_ci.yml'
+ - 'cpp/**'
+ - 'c_glib/**'
+ - 'dev/tasks/linux-packages/**'
+ - 'format/Flight.proto'
+ - 'testing'
+ tags:
+ - "apache-arrow-*-rc*"
+ pull_request:
+ paths:
+ - '.dockerignore'
+ - '.github/workflows/check_labels.yml'
+ - '.github/workflows/package_linux.yml'
+ - '.github/workflows/report_ci.yml'
+ - 'cpp/**'
+ - 'c_glib/**'
+ - 'dev/tasks/linux-packages/**'
+ - 'format/Flight.proto'
+ - 'testing'
+ types:
+ - labeled
+ - opened
+ - reopened
+ - synchronize
+ schedule:
+ - cron: "0 2 * * *"
+
+concurrency:
+ group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{
github.workflow }}
+ cancel-in-progress: true
+
+permissions:
+ # Upload to GitHub Release
+ contents: write
+
+jobs:
+ check-labels:
+ uses: ./.github/workflows/check_labels.yml
+ secrets: inherit
+ with:
+ parent-workflow: package_linux
+
+ package:
+ name: ${{ matrix.id }}
+ runs-on: ${{ contains(matrix.id, 'amd64') && 'ubuntu-latest' ||
'ubuntu-24.04-arm' }}
+ needs: check-labels
+ if: >-
+ needs.check-labels.outputs.force == 'true' ||
+ contains(fromJSON(needs.check-labels.outputs.ci-extra-labels || '[]'),
'CI: Extra') ||
+ contains(fromJSON(needs.check-labels.outputs.ci-extra-labels || '[]'),
'CI: Extra: Package: Linux')
+ timeout-minutes: 75
+ strategy:
+ fail-fast: false
+ matrix:
+ id:
+ - almalinux-8-amd64
+ - almalinux-8-arm64
+ - almalinux-9-amd64
+ - almalinux-9-arm64
+ - almalinux-10-amd64
+ - almalinux-10-arm64
+ - amazon-linux-2023-amd64
+ - amazon-linux-2023-arm64
+ - centos-9-stream-amd64
+ - centos-9-stream-arm64
+ - centos-7-amd64
+ - debian-bookworm-amd64
+ - debian-bookworm-arm64
+ - debian-trixie-amd64
+ - debian-trixie-arm64
+ - debian-forky-amd64
+ - debian-forky-arm64
+ - ubuntu-jammy-amd64
+ - ubuntu-jammy-arm64
+ - ubuntu-noble-amd64
+ - ubuntu-noble-arm64
+ env:
+ DOCKER_VOLUME_PREFIX: ".docker/"
+ steps:
+ - name: Checkout Arrow
+ uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 #
v5.0.0
+ with:
+ fetch-depth: 0
+ submodules: recursive
+ - name: Prepare environment variables
+ env:
+ ID: ${{ matrix.id }}
+ run: |
+ set -x
+ # Example: almalinux-8-amd64 -> almalinux
+ distribution="${ID%-*-*}"
+ echo "DISTRIBUTION=${distribution}" >> "${GITHUB_ENV}"
+ # Example: almalinux-8-amd64 -> amd64
+ architecture="${ID##*-}"
+ echo "ARCHITECTURE=${architecture}" >> "${GITHUB_ENV}"
+ # Example: almalinux-8-amd64 -> almalinux-8
+ target="${ID%-*}"
+ case "${target}" in
+ almalinux-*|amazon-linux-*|centos-*)
+ echo "TASK_NAMESPACE=yum" >> "${GITHUB_ENV}"
+ if [ "${architecture}" = "arm64" ]; then
+ # Example: almalinux-8 -> almalinux-8-aarch64
+ target="${target}-aarch64"
+ fi
+ echo "YUM_TARGETS=${target}" >> "${GITHUB_ENV}"
+ ;;
+ *)
+ echo "TASK_NAMESPACE=apt" >> "${GITHUB_ENV}"
+ if [ "${architecture}" = "arm64" ]; then
+ # Example: ubuntu-noble -> ubuntu-noble-arm64
+ target="${target}-arm64"
+ fi
+ echo "APT_TARGETS=${target}" >> "${GITHUB_ENV}"
+ ;;
+ esac
+ echo "TARGET=${target}" >> "${GITHUB_ENV}"
+
+ if [ "${GITHUB_REF_TYPE}" = "tag" ]; then
+ # Example: apache-arrow-21.0.0-rc0 -> 21.0.0-rc0
+ version="${GITHUB_REF_NAME#apache-arrow-}"
+ echo "ARROW_VERSION=${version}" >> "${GITHUB_ENV}"
+ fi
+ - name: Cache Docker Volumes
+ uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
+ with:
+ path: .docker
+ key: package-linux-${{ matrix.id }}-${{ hashFiles('cpp/**',
'c_glib/**') }}
+ restore-keys: package-linux-${{ matrix.id }}-
+ - name: Set up Ruby
+ run: |
+ sudo apt update
+ sudo apt install -y \
+ rake \
+ ruby \
+ ruby-dev
+ - name: Prepare apache-arrow-apt-source for arm64
+ if: env.ARCHITECTURE == 'arm64'
+ run: |
+ pushd dev/tasks/linux-packages/apache-arrow-apt-source/apt
+ for target in *-*; do
+ cp -a ${target} ${target}-arm64
+ done
+ popd
+ - name: Prepare apache-arrow-release for arm64
+ if: env.ARCHITECTURE == 'arm64'
+ run: |
+ pushd dev/tasks/linux-packages/apache-arrow-release/yum
+ for target in *-*; do
+ cp -a ${target} ${target}-aarch64
+ done
+ popd
+ - name: Update version
+ if: github.ref_type != 'tag'
+ run: |
+ pushd dev/tasks/linux-packages
+ rake version:update
+ popd
+ - name: Login to GitHub Container registry
+ uses: docker/login-action@184bdaa0721073962dff0199f1fb9940f07167d1 #
v3.5.0
+ with:
+ registry: ghcr.io
+ username: ${{ github.actor }}
+ password: ${{ secrets.GITHUB_TOKEN }}
+ - name: Wait for creating GitHub Release
+ if: github.ref_type == 'tag'
+ run: |
+ dev/release/utils-watch-gh-workflow.sh \
+ ${GITHUB_REF_NAME} \
+ release_candidate.yml
+ - name: Build
+ run: |
+ pushd dev/tasks/linux-packages
+ rake docker:pull || :
+ rake --trace ${TASK_NAMESPACE}:build BUILD_DIR=build
+ popd
+ - name: Docker Push
+ continue-on-error: true
+ if: >-
+ success() &&
+ github.event_name == 'push' &&
+ github.ref_name == 'main'
+ run: |
+ pushd dev/tasks/linux-packages
+ rake docker:push
+ popd
+ - name: Build artifact tarball
+ run: |
+ mkdir -p "${DISTRIBUTION}"
+ cp -a \
+
dev/tasks/linux-packages/*/${TASK_NAMESPACE}/repositories/${DISTRIBUTION}/* \
+ "${DISTRIBUTION}/"
+ # We use latest .deb/.rpm of
+ # apache-arrow-apt-source/apache-arrow-release built for
+ # amd64 because they are architecture independent.
+ if [ "${ARCHITECTURE}" = "amd64" ]; then
+ if [ "${APT_TASK_NAMESPACE}" = "apt" ]; then
+ # Example: debian-bookworm-amd64 -> debian-bookworm
+ code_name="${ID%-*}"
+ # Example: debian-bookworm -> bookworm
+ code_name="${code_name#*-}"
+ # Create
+ #
https://packages.apache.org/artifactory/arrow/${DISTRIBUTION}/apache-arrow-apt-source-latest-${code_name}.deb
+ # for easy to install.
+ cp -a \
+
"${DISTRIBUTION}/pool/${code_name}/a/apache-arrow-apt-source/*.deb \
+
"${DISTRIBUTION}/apache-arrow-apt-source-latest-${code_name}.deb"
+ else
+ # Example: amazon-linux-2023-amd64 -> amazon-linux-2023
+ version="${ID%-*}"
+ # Example: amazon-linux-2023 -> 2023
+ version="${version##*-}"
+ # Create
+ #
https://packages.apache.org/artifactory/arrow/${DISTRIBUTION}/${version}/apache-arrow-release-latest.rpm
+ # for easy to install.
+ cp -a \
+
"${DISTRIBUTION}/${version}/x86_64/Packages/apache-arrow-release-*.rpm \
+ "${DISTRIBUTION}/${version}/apache-arrow-release-latest.rpm"
+ fi
+ fi
Review Comment:
This is moved from `binary-task.rb`.
We'll create latest `.deb`/`.rpm` here to simplify `binary-task.rb`.
##########
.github/workflows/package_linux.yml:
##########
@@ -0,0 +1,301 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: Package Linux
+
+on:
+ push:
+ branches:
+ - '**'
+ - '!dependabot/**'
+ paths:
+ - '.dockerignore'
+ - '.github/workflows/check_labels.yml'
+ - '.github/workflows/package_linux.yml'
+ - '.github/workflows/report_ci.yml'
+ - 'cpp/**'
+ - 'c_glib/**'
+ - 'dev/tasks/linux-packages/**'
+ - 'format/Flight.proto'
+ - 'testing'
+ tags:
+ - "apache-arrow-*-rc*"
+ pull_request:
+ paths:
+ - '.dockerignore'
+ - '.github/workflows/check_labels.yml'
+ - '.github/workflows/package_linux.yml'
+ - '.github/workflows/report_ci.yml'
+ - 'cpp/**'
+ - 'c_glib/**'
+ - 'dev/tasks/linux-packages/**'
+ - 'format/Flight.proto'
+ - 'testing'
+ types:
+ - labeled
+ - opened
+ - reopened
+ - synchronize
+ schedule:
+ - cron: "0 2 * * *"
+
+concurrency:
+ group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{
github.workflow }}
+ cancel-in-progress: true
+
+permissions:
+ # Upload to GitHub Release
+ contents: write
+
+jobs:
+ check-labels:
+ uses: ./.github/workflows/check_labels.yml
+ secrets: inherit
+ with:
+ parent-workflow: package_linux
+
+ package:
+ name: ${{ matrix.id }}
+ runs-on: ${{ contains(matrix.id, 'amd64') && 'ubuntu-latest' ||
'ubuntu-24.04-arm' }}
+ needs: check-labels
+ if: >-
+ needs.check-labels.outputs.force == 'true' ||
+ contains(fromJSON(needs.check-labels.outputs.ci-extra-labels || '[]'),
'CI: Extra') ||
+ contains(fromJSON(needs.check-labels.outputs.ci-extra-labels || '[]'),
'CI: Extra: Package: Linux')
+ timeout-minutes: 75
+ strategy:
+ fail-fast: false
+ matrix:
+ id:
+ - almalinux-8-amd64
+ - almalinux-8-arm64
+ - almalinux-9-amd64
+ - almalinux-9-arm64
+ - almalinux-10-amd64
+ - almalinux-10-arm64
+ - amazon-linux-2023-amd64
+ - amazon-linux-2023-arm64
+ - centos-9-stream-amd64
+ - centos-9-stream-arm64
+ - centos-7-amd64
+ - debian-bookworm-amd64
+ - debian-bookworm-arm64
+ - debian-trixie-amd64
+ - debian-trixie-arm64
+ - debian-forky-amd64
+ - debian-forky-arm64
+ - ubuntu-jammy-amd64
+ - ubuntu-jammy-arm64
+ - ubuntu-noble-amd64
+ - ubuntu-noble-arm64
+ env:
+ DOCKER_VOLUME_PREFIX: ".docker/"
+ steps:
+ - name: Checkout Arrow
+ uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 #
v5.0.0
+ with:
+ fetch-depth: 0
+ submodules: recursive
+ - name: Prepare environment variables
+ env:
+ ID: ${{ matrix.id }}
+ run: |
+ set -x
+ # Example: almalinux-8-amd64 -> almalinux
+ distribution="${ID%-*-*}"
+ echo "DISTRIBUTION=${distribution}" >> "${GITHUB_ENV}"
+ # Example: almalinux-8-amd64 -> amd64
+ architecture="${ID##*-}"
+ echo "ARCHITECTURE=${architecture}" >> "${GITHUB_ENV}"
+ # Example: almalinux-8-amd64 -> almalinux-8
+ target="${ID%-*}"
+ case "${target}" in
+ almalinux-*|amazon-linux-*|centos-*)
+ echo "TASK_NAMESPACE=yum" >> "${GITHUB_ENV}"
+ if [ "${architecture}" = "arm64" ]; then
+ # Example: almalinux-8 -> almalinux-8-aarch64
+ target="${target}-aarch64"
+ fi
+ echo "YUM_TARGETS=${target}" >> "${GITHUB_ENV}"
+ ;;
+ *)
+ echo "TASK_NAMESPACE=apt" >> "${GITHUB_ENV}"
+ if [ "${architecture}" = "arm64" ]; then
+ # Example: ubuntu-noble -> ubuntu-noble-arm64
+ target="${target}-arm64"
+ fi
+ echo "APT_TARGETS=${target}" >> "${GITHUB_ENV}"
+ ;;
+ esac
+ echo "TARGET=${target}" >> "${GITHUB_ENV}"
+
+ if [ "${GITHUB_REF_TYPE}" = "tag" ]; then
+ # Example: apache-arrow-21.0.0-rc0 -> 21.0.0-rc0
+ version="${GITHUB_REF_NAME#apache-arrow-}"
+ echo "ARROW_VERSION=${version}" >> "${GITHUB_ENV}"
Review Comment:
We can use `21.0.0-rc0` not `21.0.0` here. The `-rc0` part will be removed
by
https://github.com/apache/arrow/blob/16ceade8f8b6a2720acf028ca3b6352000c60cdb/dev/tasks/linux-packages/package-task.rb#L37-L45
.
##########
dev/tasks/linux-packages/yum/build.sh:
##########
@@ -93,7 +93,7 @@ fi
repositories="/host/repositories"
repository="${repositories}/${distribution}/${distribution_version}"
rpm_dir="${repository}/${architecture}/Packages"
-srpm_dir="${repository}/source/SRPMS"
+srpm_dir="${repository}/Source/Packages"
Review Comment:
AlmaLinux use `Source/Packages`:
https://vault.almalinux.org/10.0/BaseOS/Source/Packages/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
