[
https://issues.apache.org/jira/browse/HDDS-6263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-6263:
-----------------------------
Description:
This is the uber jira for arm64 support in Ozone docker development environment.
Status Quo:
- Most images used in docker dev environment supports amd64 only.
- It runs VERY slowly on Apple Silicon (M1) Macs' Docker Desktop due to qemu
x86-64 -> arm64 translation. For instance, SCM startup in the upgrade
acceptance test should take ~23 seconds on a regular Intel MacBook Pro to boot
up and exit safe mode; while it times out (after 240s~300s) starting up the SCM
alone on an M1 Pro MacBook Pro due to the translation/emulation.
Goal:
- Add arm64 support for all Docker images used for Ozone development. The
resulting images (that are pushed to Docker Hub) should be dual-architecture
(e.g. ozone-docker-runner).
Foreseeable work to be done:
- Bump Centos base image to a version that supports arm64 (Done)
- dumbinit and tools included in the image build stage should be architecture
aware. Currently the tools are x64 only.
- After that the ozone runner base image should be working, provided that ALL
the Ozone dependencies at runtime that uses native library pulled from maven
supports Linux arm64/aarch64 (afaik RocksDBJNI does, gRPC also
[does|https://repo1.maven.org/maven2/io/grpc/protoc-gen-grpc-java/1.33.0/],
newer protobuf 3 should also have it, but older protobuf 2.5.0 might not.
However, it seems gRPC and protobuf 2/3 doesn't matter at the moment as they
are only used at compile time but not runtime.)
- Have arm64 version of Hadoop 3 / KMS / KDC images as well, which are
referenced in some testing environment like ozonesecure for KDC.
- Add new CI check when someone pulls in new Docker image that warns if the new
image doesn't have the linux/arm64 architecture.
- The last point might not be strictly required as IIRC Docker itself
supports running some containers natively but others in emulation at the same
time.
- Hadoop 3 already has the arm64 Dockerfile ready. But no one has built and
pushed an arm64 image to Docker Hub yet:
- https://hub.docker.com/r/apache/hadoop/tags
- https://hub.docker.com/r/apache/hadoop-runner/tags
was:
This is the uber jira for arm64 support in Ozone docker development environment.
Status Quo:
- Most images used in docker dev environment supports amd64 only.
- It runs VERY slowly on Apple Silicon (M1) Macs' Docker Desktop due to qemu
x86-64 -> arm64 translation. For instance, SCM startup in the upgrade
acceptance test should take ~23 seconds on a regular Intel MacBook Pro to boot
up and exit safe mode; while it times out (after 240s~300s) starting up the SCM
alone on an M1 Pro MacBook Pro due to the translation/emulation.
Goal:
- Add arm64 support for all Docker images used for Ozone development. The
resulting images (that are pushed to Docker Hub) should be dual-architecture
(e.g. ozone-docker-runner).
Foreseeable work to be done:
- Bump Centos base image to a version that (Almost done, pending review)
- dumbinit and tools included in the image build stage should be architecture
aware. Currently the tools are x64 only. After this the ozone runner base image
should be working, provided that ALL the Ozone dependencies at runtime that
uses native library pulled from maven supports Linux arm64/aarch64 (afaik
RocksDBJNI does, gRPC also
[does|https://repo1.maven.org/maven2/io/grpc/protoc-gen-grpc-java/1.33.0/],
newer protobuf 3 should also have it, but older protobuf 2.5.0 might not.
However, it seems gRPC and protobuf 2/3 doesn't matter at the moment as they
are only used at compile time but not runtime.)
- Have arm64 version of Hadoop 3 / KMS / KDC images as well, which are
referenced in some testing environment like ozonesecure for KDC.
- Add new CI check when someone pulls in new Docker image that warns if the new
image doesn't have the linux/arm64 architecture.
- The last point might not be strictly required as IIRC Docker itself
supports running some containers natively but others in emulation at the same
time.
- Hadoop 3 already has the arm64 Dockerfile ready. But no one has built and
pushed an arm64 image to Docker Hub yet:
- https://hub.docker.com/r/apache/hadoop/tags
- https://hub.docker.com/r/apache/hadoop-runner/tags
> Full arm64 support for Docker Images
> ------------------------------------
>
> Key: HDDS-6263
> URL: https://issues.apache.org/jira/browse/HDDS-6263
> Project: Apache Ozone
> Issue Type: Task
> Components: docker
> Reporter: Siyao Meng
> Priority: Major
>
> This is the uber jira for arm64 support in Ozone docker development
> environment.
> Status Quo:
> - Most images used in docker dev environment supports amd64 only.
> - It runs VERY slowly on Apple Silicon (M1) Macs' Docker Desktop due to qemu
> x86-64 -> arm64 translation. For instance, SCM startup in the upgrade
> acceptance test should take ~23 seconds on a regular Intel MacBook Pro to
> boot up and exit safe mode; while it times out (after 240s~300s) starting up
> the SCM alone on an M1 Pro MacBook Pro due to the translation/emulation.
> Goal:
> - Add arm64 support for all Docker images used for Ozone development. The
> resulting images (that are pushed to Docker Hub) should be dual-architecture
> (e.g. ozone-docker-runner).
> Foreseeable work to be done:
> - Bump Centos base image to a version that supports arm64 (Done)
> - dumbinit and tools included in the image build stage should be architecture
> aware. Currently the tools are x64 only.
> - After that the ozone runner base image should be working, provided that ALL
> the Ozone dependencies at runtime that uses native library pulled from maven
> supports Linux arm64/aarch64 (afaik RocksDBJNI does, gRPC also
> [does|https://repo1.maven.org/maven2/io/grpc/protoc-gen-grpc-java/1.33.0/],
> newer protobuf 3 should also have it, but older protobuf 2.5.0 might not.
> However, it seems gRPC and protobuf 2/3 doesn't matter at the moment as they
> are only used at compile time but not runtime.)
> - Have arm64 version of Hadoop 3 / KMS / KDC images as well, which are
> referenced in some testing environment like ozonesecure for KDC.
> - Add new CI check when someone pulls in new Docker image that warns if the
> new image doesn't have the linux/arm64 architecture.
> - The last point might not be strictly required as IIRC Docker itself
> supports running some containers natively but others in emulation at the same
> time.
> - Hadoop 3 already has the arm64 Dockerfile ready. But no one has built and
> pushed an arm64 image to Docker Hub yet:
> - https://hub.docker.com/r/apache/hadoop/tags
> - https://hub.docker.com/r/apache/hadoop-runner/tags
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
