paleolimbot commented on code in PR #2718: URL: https://github.com/apache/arrow-adbc/pull/2718#discussion_r2054434056
########## ci/scripts/verify_ubuntu.sh: ########## @@ -0,0 +1,104 @@ +#!/bin/bash +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +# Test the validation script using system dependencies instead of Conda. This +# script is meant for automation (e.g. Docker), not a local/developer machine +# (except via Docker). + +set -euo pipefail + +main() { + local -r source_dir="${1}" + + # Install all the dependencies we need for all the subprojects + + # When installing tzdata, don't block and wait for user input + export DEBIAN_FRONTEND=noninteractive + export TZ=Etc/UTC + + apt update + apt install -y \ + apt-transport-https \ + build-essential \ + ca-certificates \ + cmake \ + curl \ + dirmngr \ + git \ + gobject-introspection \ + gpg \ + libgirepository1.0-dev \ + libglib2.0-dev \ + libgmock-dev \ + libgtest-dev \ + libpq-dev \ + libsqlite3-dev \ + lsb-release \ + ninja-build \ + pkg-config \ + python3 \ + python3-dev \ + python3-pip \ + python3-venv \ + ruby-full \ + software-properties-common \ + wget + + # Install Java + + wget -qO - https://packages.adoptium.net/artifactory/api/gpg/key/public | \ + gpg --dearmor | \ + tee /etc/apt/trusted.gpg.d/adoptium.gpg > /dev/null + + echo "deb https://packages.adoptium.net/artifactory/deb $(awk -F= '/^VERSION_CODENAME/{print$2}' /etc/os-release) main" | \ + tee /etc/apt/sources.list.d/adoptium.list + + # Install R + # https://cloud.r-project.org/bin/linux/ubuntu/ + + wget -qO- https://cloud.r-project.org/bin/linux/ubuntu/marutter_pubkey.asc | \ + tee -a /etc/apt/trusted.gpg.d/cran_ubuntu_key.asc + + add-apt-repository "deb https://cloud.r-project.org/bin/linux/ubuntu $(lsb_release -cs)-cran40/" + + # Install Arrow GLib + wget https://repo1.maven.org/maven2/org/apache/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb + apt install -y -V ./apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb + + apt update + apt install -y \ + libarrow-dev \ + libarrow-glib-dev \ + r-base \ + temurin-21-jdk + + # Install Maven + wget https://dlcdn.apache.org/maven/maven-3/3.9.9/binaries/apache-maven-3.9.9-bin.tar.gz + mkdir -p /opt/maven + tar -C /opt/maven -xzvf apache-maven-3.9.9-bin.tar.gz --strip-components=1 + export PATH=/opt/maven/bin:$PATH + + # We run under Docker and this is necessary since the source dir is mounted in Review Comment: ```suggestion # We run under Docker and this is necessary since the source dir is typically mounted as a volume ``` Maybe? ########## dev/release/verify-release-candidate.sh: ########## @@ -246,38 +246,42 @@ install_dotnet() { return 0 fi - show_info "Ensuring that .NET is installed..." - - if dotnet --version | grep 8\.0 > /dev/null 2>&1; then - local csharp_bin=$(dirname $(which dotnet)) - show_info "Found C# at $(which csharp) (.NET $(dotnet --version))" - else - if which dotnet > /dev/null 2>&1; then - show_info "dotnet found but it is the wrong version and will be ignored." + if command -v dotnet; then + show_info "Found $(dotnet --version) at $(which dotnet)" + + if dotnet --version | grep 8\.0 > /dev/null 2>&1; then + local csharp_bin=$(dirname $(which dotnet)) + show_info "Found C# at $(which csharp) (.NET $(dotnet --version))" + DOTNET_ALREADY_INSTALLED=1 + return 0 fi - local csharp_bin=${ARROW_TMPDIR}/csharp/bin - local dotnet_version=8.0.204 - local dotnet_platform= - case "$(uname)" in + fi + + show_info "dotnet found but it is the wrong version or dotnet not found" + + local csharp_bin=${ARROW_TMPDIR}/csharp/bin + local dotnet_version=8.0.204 + local dotnet_platform= + case "$(uname)" in Linux) - dotnet_platform=linux - ;; + dotnet_platform=linux + ;; Darwin) - dotnet_platform=macos - ;; - esac - local dotnet_download_thank_you_url=https://dotnet.microsoft.com/download/thank-you/dotnet-sdk-${dotnet_version}-${dotnet_platform}-x64-binaries - local dotnet_download_url=$( \ - curl -sL ${dotnet_download_thank_you_url} | \ - grep 'directLink' | \ - grep -E -o 'https://download[^"]+' | \ - sed -n 2p) - mkdir -p ${csharp_bin} - curl -sL ${dotnet_download_url} | \ + dotnet_platform=macos + ;; + esac + local dotnet_download_thank_you_url=https://dotnet.microsoft.com/download/thank-you/dotnet-sdk-${dotnet_version}-${dotnet_platform}-x64-binaries + show_info "Getting .NET download URL from ${dotnet_download_thank_you_url}" Review Comment: I appreciate the politeness exuded by dotnet's download URL 🙂 ########## .github/workflows/nightly-verify.yml: ########## @@ -201,3 +205,31 @@ jobs: run: | pushd arrow-adbc docker compose run -e PYTHON=3.12 --rm python-debug + + source-verify-docker: + name: "Verify Source (OS)/${{ matrix.os }} ${{ matrix.version }}" + runs-on: ubuntu-latest + strategy: + max-parallel: 2 + matrix: + include: + - os: ubuntu + version: "22.04" + - os: ubuntu + version: "24.04" + steps: + - uses: actions/checkout@v4 + with: + fetch-depth: 0 + path: arrow-adbc + persist-credentials: false + submodules: recursive + + - name: Verify + env: + OS: ${{ matrix.os }} + OS_VERSION: ${{ matrix.version }} + run: | + # Hmm, -e doesn't work? Review Comment: Does this work now? ########## ci/scripts/verify_ubuntu.sh: ########## @@ -0,0 +1,104 @@ +#!/bin/bash +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +# Test the validation script using system dependencies instead of Conda. This +# script is meant for automation (e.g. Docker), not a local/developer machine +# (except via Docker). + +set -euo pipefail + +main() { + local -r source_dir="${1}" + + # Install all the dependencies we need for all the subprojects + + # When installing tzdata, don't block and wait for user input + export DEBIAN_FRONTEND=noninteractive + export TZ=Etc/UTC + + apt update + apt install -y \ + apt-transport-https \ + build-essential \ + ca-certificates \ + cmake \ + curl \ + dirmngr \ + git \ + gobject-introspection \ + gpg \ + libgirepository1.0-dev \ + libglib2.0-dev \ + libgmock-dev \ + libgtest-dev \ + libpq-dev \ + libsqlite3-dev \ + lsb-release \ + ninja-build \ + pkg-config \ + python3 \ + python3-dev \ + python3-pip \ + python3-venv \ + ruby-full \ + software-properties-common \ + wget + + # Install Java + + wget -qO - https://packages.adoptium.net/artifactory/api/gpg/key/public | \ + gpg --dearmor | \ + tee /etc/apt/trusted.gpg.d/adoptium.gpg > /dev/null + + echo "deb https://packages.adoptium.net/artifactory/deb $(awk -F= '/^VERSION_CODENAME/{print$2}' /etc/os-release) main" | \ + tee /etc/apt/sources.list.d/adoptium.list + + # Install R + # https://cloud.r-project.org/bin/linux/ubuntu/ + + wget -qO- https://cloud.r-project.org/bin/linux/ubuntu/marutter_pubkey.asc | \ + tee -a /etc/apt/trusted.gpg.d/cran_ubuntu_key.asc + + add-apt-repository "deb https://cloud.r-project.org/bin/linux/ubuntu $(lsb_release -cs)-cran40/" Review Comment: I believe you can also just `apt-get install r-base`: https://github.com/apache/arrow-nanoarrow/blob/4241f5f07a642a615912f339a5c089c1fe597ed0/ci/docker/ubuntu.dockerfile#L23 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
