Repository: arrow Updated Branches: refs/heads/master 4cb3e97e9 -> 60cb1c372
ARROW-905 [Docs] Dockerize document generation This PR has two parts. First part is to use Docker to generate the API documentation for C, C++, Python and Java. This part is now complete. The second part is to be able to run the site locally, which is still in progress. Author: Heimir Sverrisson <[email protected]> Closes #1162 from heimir-sverrisson/hs/dockerize_api_docs and squashes the following commits: f758fdc3 [Heimir Sverrisson] ARROW-905 [Docs] Add Dockerfile for reproducible documentation generation Project: http://git-wip-us.apache.org/repos/asf/arrow/repo Commit: http://git-wip-us.apache.org/repos/asf/arrow/commit/60cb1c37 Tree: http://git-wip-us.apache.org/repos/asf/arrow/tree/60cb1c37 Diff: http://git-wip-us.apache.org/repos/asf/arrow/diff/60cb1c37 Branch: refs/heads/master Commit: 60cb1c372a7c4cf98aeeb5b9deb1ab6c67a5956a Parents: 4cb3e97 Author: Heimir Sverrisson <[email protected]> Authored: Thu Oct 12 00:49:23 2017 -0400 Committer: Wes McKinney <[email protected]> Committed: Thu Oct 12 00:49:23 2017 -0400 ---------------------------------------------------------------------- c_glib/.gitignore | 1 + dev/README.md | 32 ++++++++- dev/docker-compose.yml | 30 +++++++++ dev/gen_apidocs.sh | 21 ++++++ dev/gen_apidocs/Dockerfile | 83 +++++++++++++++++++++++ dev/gen_apidocs/create_documents.sh | 110 +++++++++++++++++++++++++++++++ dev/run_docker_compose.sh | 40 +++++++++++ dev/run_site.sh | 21 ++++++ dev/run_site/Dockerfile | 34 ++++++++++ dev/run_site/run_site.sh | 24 +++++++ java/.gitignore | 1 + site/.gitignore | 2 + site/_config.yml | 3 + 13 files changed, 401 insertions(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/arrow/blob/60cb1c37/c_glib/.gitignore ---------------------------------------------------------------------- diff --git a/c_glib/.gitignore b/c_glib/.gitignore index ff3615a..373ae73 100644 --- a/c_glib/.gitignore +++ b/c_glib/.gitignore @@ -30,6 +30,7 @@ Makefile.in /doc/reference/*.stamp /doc/reference/html/ /doc/reference/xml/ +/doc/reference/tmpl/ /libtool /m4/ /stamp-h1 http://git-wip-us.apache.org/repos/asf/arrow/blob/60cb1c37/dev/README.md ---------------------------------------------------------------------- diff --git a/dev/README.md b/dev/README.md index 91d688d..05a68fc 100644 --- a/dev/README.md +++ b/dev/README.md @@ -109,4 +109,34 @@ Studio 2015): ``` dev/release/verify-release-candidate.bat apache-arrow-0.7.0.tar.gz -``` \ No newline at end of file +``` + +## Creating API documentation + +The generation of API documentation for `C++`, `C Glib`, `Python` +and `Java` has been Dockerized. To generate the API documentation +run the following command: + +```shell +bash dev/gen_apidocs.sh +``` + +This script assumes that the `parquet-cpp` Git repository +https://github.com/apache/parquet-cpp has been cloned +besides the Arrow repository and a `dist` directory can be created +at the same level by the current user. Please note that most of the +software must be built in order to create the documentation, so this +step may take some time to run, especially the first time around as the +Docker container will also have to be built. + +After successfully creating the API documentation the website can be +run locally to browse the API documentation from the top level +`Documentation` menu. To run the website issue the command: + +```shell +bash dev/run_site.sh +``` + +The local URL for the website running inside the docker container +will be shown as `Server address:` in the output of the command. +To stop the server press `Ctrl-C` in that window. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow/blob/60cb1c37/dev/docker-compose.yml ---------------------------------------------------------------------- diff --git a/dev/docker-compose.yml b/dev/docker-compose.yml new file mode 100644 index 0000000..7bd2cd4 --- /dev/null +++ b/dev/docker-compose.yml @@ -0,0 +1,30 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +version: '3' +services: + gen_apidocs: + build: + context: gen_apidocs + volumes: + - ../..:/apache-arrow + run_site: + build: + context: run_site + ports: + - "4000:4000" + volumes: + - ../..:/apache-arrow http://git-wip-us.apache.org/repos/asf/arrow/blob/60cb1c37/dev/gen_apidocs.sh ---------------------------------------------------------------------- diff --git a/dev/gen_apidocs.sh b/dev/gen_apidocs.sh new file mode 100755 index 0000000..4bba226 --- /dev/null +++ b/dev/gen_apidocs.sh @@ -0,0 +1,21 @@ +#!/usr/bin/env bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# Pass the service name to run_docker_compse.sh +# Which validates environment and runs the service +exec "$(dirname ${BASH_SOURCE})"/run_docker_compose.sh gen_apidocs http://git-wip-us.apache.org/repos/asf/arrow/blob/60cb1c37/dev/gen_apidocs/Dockerfile ---------------------------------------------------------------------- diff --git a/dev/gen_apidocs/Dockerfile b/dev/gen_apidocs/Dockerfile new file mode 100644 index 0000000..0b2844c --- /dev/null +++ b/dev/gen_apidocs/Dockerfile @@ -0,0 +1,83 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +FROM ubuntu:14.04 +ADD . /apache-arrow +WORKDIR /apache-arrow +# Prerequsites for apt-add-repository +RUN apt-get update && apt-get install -y \ + software-properties-common python-software-properties +# Basic OS dependencies +RUN apt-add-repository -y ppa:ubuntu-toolchain-r/test && \ + apt-get update && apt-get install -y \ + wget \ + rsync \ + git \ + gcc-4.9 \ + g++-4.9 \ + build-essential +# This will install conda in /home/ubuntu/miniconda +RUN wget -O /tmp/miniconda.sh \ + https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh && \ + bash /tmp/miniconda.sh -b -p /home/ubuntu/miniconda && \ + rm /tmp/miniconda.sh +# C++ dependencies +RUN /home/ubuntu/miniconda/bin/conda install -c conda-forge \ + boost-cpp \ + doxygen \ + maven \ + cmake \ + zlib \ + thrift-cpp +# C_Glib dependencies +RUN apt-get install -y \ + libgtk2.0-dev \ + gtk-doc-tools \ + gobject-introspection \ + libgirepository1.0-dev \ + autogen \ + autoconf-archive +# Python dependencies +RUN apt-get install -y \ + pkg-config +# Create Conda environment +RUN /home/ubuntu/miniconda/bin/conda create -y -q -n pyarrow-dev \ + # Python + python=3.6 \ + numpy \ + pandas \ + pytest \ + cython \ + ipython \ + matplotlib \ + numpydoc \ + sphinx \ + sphinx_bootstrap_theme \ + six \ + setuptools \ + # C++ + cmake \ + flatbuffers \ + rapidjson \ + thrift-cpp \ + snappy \ + zlib \ + brotli \ + jemalloc \ + lz4-c \ + zstd \ + -c conda-forge +CMD arrow/dev/gen_apidocs/create_documents.sh http://git-wip-us.apache.org/repos/asf/arrow/blob/60cb1c37/dev/gen_apidocs/create_documents.sh ---------------------------------------------------------------------- diff --git a/dev/gen_apidocs/create_documents.sh b/dev/gen_apidocs/create_documents.sh new file mode 100755 index 0000000..573a328 --- /dev/null +++ b/dev/gen_apidocs/create_documents.sh @@ -0,0 +1,110 @@ +#!/usr/bin/env bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# Set up environment and output directory for C++ libraries +cd /apache-arrow +rm -rf dist +mkdir dist +export ARROW_BUILD_TYPE=release +export ARROW_HOME=$(pwd)/dist +export PARQUET_HOME=$(pwd)/dist +CONDA_BASE=/home/ubuntu/miniconda +export LD_LIBRARY_PATH=$(pwd)/dist/lib:${CONDA_BASE}/lib:${LD_LIBRARY_PATH} +export THRIFT_HOME=${CONDA_BASE} +export BOOST_ROOT=${CONDA_BASE} +export PATH=${CONDA_BASE}/bin:${PATH} + +# Prepare the asf-site before copying api docs +pushd arrow/site +rm -rf asf-site +export GIT_COMMITTER_NAME="Nobody" +export GIT_COMMITTER_EMAIL="[email protected]" +git clone --branch=asf-site \ + https://git-wip-us.apache.org/repos/asf/arrow-site.git asf-site +popd + +# Make Python documentation (Depends on C++ ) +# Build Arrow C++ +source activate pyarrow-dev +rm -rf arrow/cpp/build +mkdir arrow/cpp/build +pushd arrow/cpp/build +cmake -DCMAKE_BUILD_TYPE=$ARROW_BUILD_TYPE \ + -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \ + -DARROW_PYTHON=on \ + -DARROW_PLASMA=on \ + -DARROW_BUILD_TESTS=OFF \ + .. +make -j4 +make install +popd + +# Build Parquet C++ +rm -rf parquet-cpp/build +mkdir parquet-cpp/build +pushd parquet-cpp/build +cmake -DCMAKE_BUILD_TYPE=$ARROW_BUILD_TYPE \ + -DCMAKE_INSTALL_PREFIX=$PARQUET_HOME \ + -DPARQUET_BUILD_BENCHMARKS=off \ + -DPARQUET_BUILD_EXECUTABLES=off \ + -DPARQUET_BUILD_TESTS=off \ + .. +make -j4 +make install +popd + +# Now Python documentation can be built +pushd arrow/python +rm -rf build/* +rm -rf doc/_build +python setup.py build_ext --build-type=$ARROW_BUILD_TYPE \ + --with-plasma --with-parquet --inplace +python setup.py build_sphinx -s doc/source +mkdir -p ../site/asf-site/docs/python +rsync -r doc/_build/html/ ../site/asf-site/docs/python +popd + +# Build c_glib documentation +pushd arrow/c_glib +rm -rf doc/reference/html/* +./autogen.sh +./configure \ + --with-arrow-cpp-build-dir=$(pwd)/../cpp/build \ + --with-arrow-cpp-build-type=$ARROW_BUILD_TYPE \ + --enable-gtk-doc +LD_LIBRARY_PATH=$(pwd)/../cpp/build/$ARROW_BUILD_TYPE make GTK_DOC_V_XREF=": " +mkdir -p ../site/asf-site/docs/c_glib +rsync -r doc/reference/html/ ../site/asf-site/docs/c_glib +popd + +# Make C++ documentation +pushd arrow/cpp/apidoc +rm -rf html/* +doxygen Doxyfile +mkdir -p ../../site/asf-site/docs/cpp +rsync -r html/ ../../site/asf-site/docs/cpp +popd + +# Make Java documentation +pushd arrow/java +rm -rf target/site/apidocs/* +mvn -Drat.skip=true install +mvn -Drat.skip=true site +mkdir -p ../site/asf-site/docs/java/ +rsync -r target/site/apidocs ../site/asf-site/docs/java/ +popd http://git-wip-us.apache.org/repos/asf/arrow/blob/60cb1c37/dev/run_docker_compose.sh ---------------------------------------------------------------------- diff --git a/dev/run_docker_compose.sh b/dev/run_docker_compose.sh new file mode 100755 index 0000000..f46879e --- /dev/null +++ b/dev/run_docker_compose.sh @@ -0,0 +1,40 @@ +#!/usr/bin/env bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +if [ $# -lt 1 ]; then + echo "This script takes one argument - the docker service to run" >&2 + exit 1 +fi + +# Make sure this is always run in the directory above Arrow root +cd $(dirname "${BASH_SOURCE}")/../.. + +if [ ! -d arrow ]; then + echo "Please make sure that the top level Arrow directory" >&2 + echo "is named 'arrow'" + exit 1 +fi + +if [ ! -d parquet-cpp ]; then + echo "Please clone the Parquet repo next to the Arrow repo" >&2 + exit 1 +fi + +GID=$(id -g ${USERNAME}) +docker-compose -f arrow/dev/docker-compose.yml run \ + -u "${UID}:${GID}" "${1}" http://git-wip-us.apache.org/repos/asf/arrow/blob/60cb1c37/dev/run_site.sh ---------------------------------------------------------------------- diff --git a/dev/run_site.sh b/dev/run_site.sh new file mode 100755 index 0000000..1e64dad --- /dev/null +++ b/dev/run_site.sh @@ -0,0 +1,21 @@ +#!/usr/bin/env bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# Pass the service name to run_docker_compse.sh +# Which validates environment and runs the service +exec "$(dirname ${BASH_SOURCE})"/run_docker_compose.sh run_site http://git-wip-us.apache.org/repos/asf/arrow/blob/60cb1c37/dev/run_site/Dockerfile ---------------------------------------------------------------------- diff --git a/dev/run_site/Dockerfile b/dev/run_site/Dockerfile new file mode 100644 index 0000000..cea4ceb --- /dev/null +++ b/dev/run_site/Dockerfile @@ -0,0 +1,34 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +FROM ubuntu:14.04 +ADD . /apache-arrow +WORKDIR /apache-arrow +# Prerequsites for apt-add-repository +RUN apt-get update && apt-get install -y \ + software-properties-common python-software-properties +# Set up Ruby repository +RUN apt-add-repository ppa:brightbox/ruby-ng +# The publication tools +RUN apt-get update; apt-get install -y \ + apt-transport-https \ + ruby2.2-dev \ + ruby2.2 \ + zlib1g-dev \ + make \ + gcc +RUN gem install jekyll bundler +CMD arrow/dev/run_site/run_site.sh http://git-wip-us.apache.org/repos/asf/arrow/blob/60cb1c37/dev/run_site/run_site.sh ---------------------------------------------------------------------- diff --git a/dev/run_site/run_site.sh b/dev/run_site/run_site.sh new file mode 100755 index 0000000..8527b2b --- /dev/null +++ b/dev/run_site/run_site.sh @@ -0,0 +1,24 @@ +#!/usr/bin/env bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +cd /apache-arrow/arrow/site +export GEM_HOME=$(pwd) +export BUNDLE_PATH=$(pwd) +export HOME=$(pwd) +bundle install --path . +HOST_IP=$(grep $(hostname) /etc/hosts | cut -f 1) +bundle exec jekyll serve -c _config.yml -s asf-site --host ${HOST_IP} --port 4000 http://git-wip-us.apache.org/repos/asf/arrow/blob/60cb1c37/java/.gitignore ---------------------------------------------------------------------- diff --git a/java/.gitignore b/java/.gitignore index 73c1be4..03f5bf7 100644 --- a/java/.gitignore +++ b/java/.gitignore @@ -20,3 +20,4 @@ CMakeFiles Makefile cmake_install.cmake install_manifest.txt +?/ http://git-wip-us.apache.org/repos/asf/arrow/blob/60cb1c37/site/.gitignore ---------------------------------------------------------------------- diff --git a/site/.gitignore b/site/.gitignore index 46bc466..d3c112a 100644 --- a/site/.gitignore +++ b/site/.gitignore @@ -4,3 +4,5 @@ _site Gemfile.lock asf-site build/ +.bundle/ +ruby/ http://git-wip-us.apache.org/repos/asf/arrow/blob/60cb1c37/site/_config.yml ---------------------------------------------------------------------- diff --git a/site/_config.yml b/site/_config.yml index fcb76a3..a6c5575 100644 --- a/site/_config.yml +++ b/site/_config.yml @@ -23,6 +23,9 @@ kramdown: input: GFM syntax_highlighter: rouge +include: + - _static + exclude: - Gemfile - Gemfile.lock
