This is an automated email from the ASF dual-hosted git repository.
wesm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/master by this push:
new 9198f63 ARROW-3364: [Docs] Add docker-compose integration
documentation
9198f63 is described below
commit 9198f630663e7f8ddaecbd4d9f037f61e5b450d3
Author: François Saint-Jacques <[email protected]>
AuthorDate: Thu Mar 14 07:54:56 2019 -0500
ARROW-3364: [Docs] Add docker-compose integration documentation
Author: François Saint-Jacques <[email protected]>
Closes #3882 from fsaintjacques/ARROW-3364-docker-compose-documentation and
squashes the following commits:
08f91ff35 <François Saint-Jacques> Simplify makefile
595d50ac0 <François Saint-Jacques> Refactor ambiguous makefile run target
9d90bbaf8 <François Saint-Jacques> ARROW-3364: Add docker-compose
integration documentation
---
Makefile.docker | 92 ++++++++--------------
.../{building.rst => developers/documentation.rst} | 5 +-
docs/source/{ => developers}/index.rst | 31 +-------
docs/source/developers/integration.rst | 67 ++++++++++++++++
docs/source/index.rst | 4 +-
5 files changed, 109 insertions(+), 90 deletions(-)
diff --git a/Makefile.docker b/Makefile.docker
index dc1f0bf..8e7d861 100644
--- a/Makefile.docker
+++ b/Makefile.docker
@@ -16,70 +16,48 @@
# under the License.
# build docker compose images:
+# $ make -f Makefile.docker build-cpp
+# To run the test suite
# $ make -f Makefile.docker cpp
-# run the built image:
-# $ make -f Makefile.docker run cpp
-.PHONY: clean run cpp cpp-alpine go js java rust r
+LANGUAGES = cpp cpp-alpine cpp-cmake32 c_glib go java js python python-alpine
rust r
+MISC = lint iwyu clang-format docs pandas-master
+SERVERS = dask hdfs-integration spark-integration
+
+# declare images dependencies
+DEPENDS_ON_CPP = build-c_glib build-python build-r
+DEPENDS_ON_CPP_ALPINE = build-python-alpine
+DEPENDS_ON_PYTHON = build-lint build-docs build-dask build-hdfs-integration
build-spark-integration
+DEPENDS_ON_LINT = build-iwyu build-clang-format
+
+SERVICES = $(LANGUAGES) $(MISC) $(SERVERS)
+.PHONY: clean build-% run-% $(SERVICES)
DC := docker-compose
clean:
$(DC) down -v
-run:
- $(DC) run --rm $(filter-out $@,$(MAKECMDGOALS))
-
-go:
- $(DC) build go
-
-js:
- $(DC) build js
-
-java:
- $(DC) build java
-
-rust:
- $(DC) build rust
-
-cpp:
- $(DC) build cpp
-
-cpp-alpine:
- $(DC) build cpp-alpine
-
-cpp-cmake32:
- $(DC) build cpp-cmake32
-
-c_glib: cpp
- $(DC) build c_glib
-
-r: cpp
- $(DC) build r
-
-python: cpp
- $(DC) build python
-
-python-alpine: cpp-alpine
- $(DC) build python-alpine
-
-lint: python
- $(DC) build lint
-
-iwyu: lint
-
-clang-format: lint
-
-docs: python
-
-dask: python
- $(DC) build dask
-
-hdfs: python
- $(DC) build hdfs-integration
+# Default build target if no dependencies
+build-%:
+ $(DC) build $*
+
+# The following targets create the dependencies of the form `build-X: build-Y`
+$(DEPENDS_ON_CPP): build-%: build-cpp
+ $(DC) build $*
+$(DEPENDS_ON_CPP_ALPINE): build-%: build-cpp-alpine
+ $(DC) build $*
+$(DEPENDS_ON_PYTHON): build-%: build-python
+ $(DC) build $*
+# The dependents of lint image don't build anything
+$(DEPENDS_ON_LINT): build-%: build-lint
+
+# panda master is a special case due to --no-cache
+build-pandas-master: build-python
+ $(DC) build --no-cache pandas-master
-spark: python
- $(DC) build spark-integration
+run-%: build-%
+ $(DC) run --rm $*
-pandas-master: python
- $(DC) build --no-cache pandas-master
+# Trick to get `service` expand to `run-service`
+$(SERVICES): % : run-%
diff --git a/docs/source/building.rst b/docs/source/developers/documentation.rst
similarity index 96%
rename from docs/source/building.rst
rename to docs/source/developers/documentation.rst
index 2239a19..1fbab43 100644
--- a/docs/source/building.rst
+++ b/docs/source/developers/documentation.rst
@@ -90,9 +90,6 @@ You can use Docker to build the documentation:
.. code-block:: shell
- docker-compose build cpp
- docker-compose build python
- docker-compose build docs
- docker-compose run docs
+ make -f Makefile.docker docs
The final output is located under ``docs/_build/html``.
diff --git a/docs/source/index.rst b/docs/source/developers/index.rst
similarity index 56%
copy from docs/source/index.rst
copy to docs/source/developers/index.rst
index 2b367b3..e99f7c5 100644
--- a/docs/source/index.rst
+++ b/docs/source/developers/index.rst
@@ -15,34 +15,11 @@
.. specific language governing permissions and limitations
.. under the License.
-Apache Arrow
-============
-
-Apache Arrow is a cross-language development platform for in-memory data. It
-specifies a standardized language-independent columnar memory format for flat
-and hierarchical data, organized for efficient analytic operations on modern
-hardware. It also provides computational libraries and zero-copy streaming
-messaging and interprocess communication.
-
-.. toctree::
- :maxdepth: 1
- :caption: Memory Format
-
- format/README
- format/Guidelines
- format/Layout
- format/Metadata
- format/IPC
-
-.. toctree::
- :maxdepth: 2
- :caption: Languages
-
- cpp/index
- python/index
+Developing Apache Arrow
+=======================
.. toctree::
:maxdepth: 2
- :caption: Other Topics
- building
+ integration
+ documentation
diff --git a/docs/source/developers/integration.rst
b/docs/source/developers/integration.rst
new file mode 100644
index 0000000..83597fa
--- /dev/null
+++ b/docs/source/developers/integration.rst
@@ -0,0 +1,67 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+Integration Testing
+===================
+
+Prerequisites
+-------------
+
+Arrow uses `Docker <https://docs.docker.com/>`_ and
+`docker-compose <https://docs.docker.com/compose/>`_ for integration testing.
+You can follow the installation `instructions
<https://docs.docker.com/compose/install/>`_.
+
+Docker images (services)
+------------------------
+
+The docker-compose services are defined in the ``docker-compose.yml`` file.
+Each service usually correspond to a language binding or an important service
to
+test with Arrow.
+
+Services are configured with 2 local mounts, ``/arrow`` for the top-level
source
+directory and ``/build`` for caching build artifacts. The source level
+directory mount can be paired with git checkout to test a specific commit. The
+build mount is used for caching and sharing state between staged images.
+
+- *c_glib*: Builds the GLib bindings
+- *cpp*: Builds the C++ project
+- *go*: Builds the go project
+- *java*: Builds the Java project
+- *js*: Builds the Javascript project
+- *python*: Builds the python bindings
+- *r*: Builds the R bindings
+- *rust*: Builds the rust project
+- *lint*: Run various lint on the C++ sources
+- *iwyu*: Run include-what-you-use on the C++ sources
+- *clang-format*: Run clang-format on the C++ sources
+- *docs*: Builds this documentation
+
+You can build and run a service by using the `build` and `run` docker-compose
+sub-command, e.g. `docker-compose build python && docker-compose run python`.
+We do not publish the build images, you need to build them manually. This
+method requires the user to build the images in reverse dependency order. To
+simplify this, we provide a Makefile.
+
+.. code-block:: shell
+
+ # Build and run manually
+ docker-compose build cpp
+ docker-compose build python
+ docker-compose run python
+
+ # Using the makefile with proper image dependency resolution
+ make -f Makefile.docker python
diff --git a/docs/source/index.rst b/docs/source/index.rst
index 2b367b3..ea5e3bc 100644
--- a/docs/source/index.rst
+++ b/docs/source/index.rst
@@ -43,6 +43,6 @@ messaging and interprocess communication.
.. toctree::
:maxdepth: 2
- :caption: Other Topics
+ :caption: Developers
- building
+ developers/index
