kou commented on code in PR #14472:
URL: https://github.com/apache/arrow/pull/14472#discussion_r1007505218
##########
dev/tasks/java-jars/github.yml:
##########
@@ -28,13 +28,21 @@ jobs:
{{ macros.github_checkout_arrow()|indent }}
{{ macros.github_install_archery()|indent }}
- name: Build C++ libraries
+ env:
+ {{ macros.github_set_sccache_envvars()|indent(8) }}
run: |
archery docker run \
-e ARROW_JAVA_BUILD=OFF \
-e ARROW_JAVA_TEST=OFF \
java-jni-manylinux-2014
- name: Compress into single artifact to keep directory structure
- run: tar -cvzf arrow-shared-libs-linux.tar.gz arrow/java-dist/
+ run: |
+ mkdir -p /tmp/arrow/java-dist/x86_64
+ cp arrow/java-dist/*.* /tmp/arrow/java-dist/x86_64/
+ pushd /tmp/
+ tar -cvzf arrow-shared-libs-linux.tar.gz arrow/java-dist/x86_64/
+ popd
+ mv /tmp/arrow-shared-libs-linux.tar.gz .
Review Comment:
How about changing `ci/script/java-jni_manylinux_build.sh` instead?
```diff
diff --git a/ci/scripts/java_jni_manylinux_build.sh
b/ci/scripts/java_jni_manylinux_build.sh
index 884b76bc05..051ebe41d0 100755
--- a/ci/scripts/java_jni_manylinux_build.sh
+++ b/ci/scripts/java_jni_manylinux_build.sh
@@ -22,7 +22,7 @@ set -ex
arrow_dir=${1}
build_dir=${2}
# The directory where the final binaries will be stored when scripts finish
-dist_dir=${3}
+dist_dir=${3}/$(arch)
echo "=== Clear output directories and leftovers ==="
# Clear output directories and leftovers
```
##########
dev/tasks/java-jars/github.yml:
##########
@@ -48,47 +56,62 @@ jobs:
{% endif %}
build-cpp-macos:
- name: Build C++ libraries macOS
- runs-on: macos-latest
+ {% set arch = '${{ matrix.platform.arch }}' %}
+ name: Build C++ libraries macOS {{ arch }}
+ runs-on: {{ '${{ matrix.platform.runs_on }}' }}
+ strategy:
+ fail-fast: false
+ matrix:
+ platform:
+ - { runs_on: 'macos-latest', arch: "x86_64"}
+ - { runs_on: ["self-hosted", "macOS", "arm64", "devops-managed"],
arch: "aarch_64" }
env:
MACOSX_DEPLOYMENT_TARGET: "10.13"
steps:
{{ macros.github_checkout_arrow()|indent }}
- {{ macros.github_install_archery()|indent }}
+ - name: Install Archery
+ shell: bash
+ run: python3 -m pip install -e arrow/dev/archery[all]
- name: Install dependencies
run: |
- brew install --overwrite git
brew bundle --file=arrow/cpp/Brewfile
# We want to link aws-sdk-cpp statically but Homebrew's
# aws-sdk-cpp provides only shared library. If we have
# Homebrew's aws-sdk-cpp, our build mix Homebrew's
# aws-sdk-cpp and bundled aws-sdk-cpp. We uninstall Homebrew's
# aws-sdk-cpp to ensure using only bundled aws-sdk-cpp.
brew uninstall aws-sdk-cpp
- - name: Setup ccache
- run: |
- arrow/ci/scripts/ccache_setup.sh
+ brew install openjdk@11
+ brew install sccache
- name: Build C++ libraries
+ env:
+ {{ macros.github_set_sccache_envvars()|indent(8) }}
run: |
set -e
+ # make brew Java available to CMake
+ if [ {{ '"${{ matrix.platform.arch }}"' }} = "aarch_64" ]; then
+ export JAVA_HOME=$(brew --prefix
openjdk@11)/libexec/openjdk.jdk/Contents/Home
+ fi
arrow/ci/scripts/java_jni_macos_build.sh \
$GITHUB_WORKSPACE/arrow \
$GITHUB_WORKSPACE/arrow/cpp-build \
$GITHUB_WORKSPACE/arrow/java-dist
- name: Compress into single artifact to keep directory structure
- run: tar -cvzf arrow-shared-libs-macos.tar.gz arrow/java-dist/
+ run: |
+ mkdir -p arrow/java-dist/{{ arch }}
+ mv arrow/java-dist/*.* arrow/java-dist/{{ arch }}/
+ tar -cvzf arrow-shared-libs-macos-{{ arch }}.tar.gz arrow/java-dist/
Review Comment:
How about specifying `{{ arch }}` to `java_jni_macos_build.sh` instead?
```diff
diff --git a/dev/tasks/java-jars/github.yml b/dev/tasks/java-jars/github.yml
index 763f5df5cd..2d054f2cd7 100644
--- a/dev/tasks/java-jars/github.yml
+++ b/dev/tasks/java-jars/github.yml
@@ -74,7 +74,7 @@ jobs:
arrow/ci/scripts/java_jni_macos_build.sh \
$GITHUB_WORKSPACE/arrow \
$GITHUB_WORKSPACE/arrow/cpp-build \
- $GITHUB_WORKSPACE/arrow/java-dist
+ $GITHUB_WORKSPACE/arrow/java-dist/{{ arch }}
- name: Compress into single artifact to keep directory structure
run: tar -cvzf arrow-shared-libs-macos.tar.gz arrow/java-dist/
- name: Upload artifacts
```
##########
dev/tasks/java-jars/github.yml:
##########
@@ -164,6 +194,9 @@ jobs:
arrow/ci/scripts/java_full_build.sh \
$GITHUB_WORKSPACE/arrow \
$GITHUB_WORKSPACE/arrow/java-dist
- {{ macros.github_upload_releases(["arrow/java-dist/*.jar",
- "arrow/java-dist/*.pom",
- "arrow/java-dist/*.zip"])|indent }}
+ {{ macros.github_upload_releases(["arrow/java-dist/x86_64/*.jar",
+ "arrow/java-dist/x86_64/*.pom",
+ "arrow/java-dist/x86_64/*.zip",
+ "arrow/java-dist/aarch_64/*.jar",
+ "arrow/java-dist/aarch_64/*.pom",
+
"arrow/java-dist/aarch_64/*.zip"])|indent }}
Review Comment:
Why do we need this change?
Artifacts should be copied to `arrow/java-dist/`:
https://github.com/apache/arrow/blob/master/ci/scripts/java_full_build.sh#L62-L70
##########
dev/tasks/java-jars/github.yml:
##########
@@ -97,16 +120,24 @@ jobs:
- name: Download Timezone Database
shell: bash
run: arrow/ci/scripts/download_tz_database.sh
+ - name: Install sccache
+ shell: bash
+ run: arrow/ci/scripts/install_sccache.sh pc-windows-msvc $(pwd)/sccache
- name: Build C++ libraries
shell: cmd
+ env:
+ {{ macros.github_set_sccache_envvars()|indent(8) }}
run: |
call "C:\Program Files (x86)\Microsoft Visual
Studio\2019\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64
REM For ORC
set TZDIR=/c/msys64/usr/share/zoneinfo
bash -c "arrow/ci/scripts/java_jni_windows_build.sh $(pwd)/arrow
$(pwd)/arrow/cpp-build $(pwd)/arrow/java-dist"
- name: Compress into single artifact to keep directory structure
shell: bash
- run: tar -cvzf arrow-shared-libs-windows.tar.gz arrow/java-dist/
+ run: |
+ mkdir -p arrow/java-dist/x86_64
+ mv arrow/java-dist/*.* arrow/java-dist/x86_64/
+ tar -cvzf arrow-shared-libs-windows.tar.gz arrow/java-dist/x86_64/
Review Comment:
How about specifying `x86_64` to `java_jni_windows_build.sh` instead?
```diff
diff --git a/dev/tasks/java-jars/github.yml b/dev/tasks/java-jars/github.yml
index 763f5df5cd..a27f668f44 100644
--- a/dev/tasks/java-jars/github.yml
+++ b/dev/tasks/java-jars/github.yml
@@ -103,7 +103,7 @@ jobs:
call "C:\Program Files (x86)\Microsoft Visual
Studio\2019\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64
REM For ORC
set TZDIR=/c/msys64/usr/share/zoneinfo
- bash -c "arrow/ci/scripts/java_jni_windows_build.sh $(pwd)/arrow
$(pwd)/arrow/cpp-build $(pwd)/arrow/java-dist"
+ bash -c "arrow/ci/scripts/java_jni_windows_build.sh $(pwd)/arrow
$(pwd)/arrow/cpp-build $(pwd)/arrow/java-dist/x86_64"
- name: Compress into single artifact to keep directory structure
shell: bash
run: tar -cvzf arrow-shared-libs-windows.tar.gz arrow/java-dist/
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
