This is an automated email from the ASF dual-hosted git repository.
amolina pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-cookbook.git
The following commit(s) were added to refs/heads/main by this push:
new 7fb66c9 [Java]: Setup java arrow cookbook doc and testing baseline
(#125)
7fb66c9 is described below
commit 7fb66c9d01927ee7ca0d3900ca5d1e368ab86c97
Author: david dali susanibar arce <[email protected]>
AuthorDate: Tue Jan 25 09:25:27 2022 -0500
[Java]: Setup java arrow cookbook doc and testing baseline (#125)
* Setup java arrow cookbook doc and testing baseline
* Update java/ext/javadoctest.py
Co-authored-by: David Li <[email protected]>
* Adding support for jdk9+ that does not support dash in bash/pipe
Co-authored-by: David Li <[email protected]>
---
.gitignore | 4 ++
CONTRIBUTING.md | 3 +-
Makefile | 23 ++++++++++--
build/index.html | 1 +
java/.flake8 | 1 +
java/.isort.cfg | 2 +
java/CONTRIBUTING.rst | 80 ++++++++++++++++++++++++++++++++++++++++
java/Makefile | 20 ++++++++++
java/ext/javadoctest.py | 95 ++++++++++++++++++++++++++++++++++++++++++++++++
java/make.bat | 35 ++++++++++++++++++
java/requirements.txt | 3 ++
java/source/conf.py | 53 +++++++++++++++++++++++++++
java/source/create.rst | 37 +++++++++++++++++++
java/source/demo/pom.xml | 55 ++++++++++++++++++++++++++++
java/source/index.rst | 20 ++++++++++
15 files changed, 428 insertions(+), 4 deletions(-)
diff --git a/.gitignore b/.gitignore
index 040bc4c..2381cee 100644
--- a/.gitignore
+++ b/.gitignore
@@ -16,3 +16,7 @@ r/*.Rproj
*.feather
*.pyc
+
+*target/
+.DS_Store
+.cp.tmp
\ No newline at end of file
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index d1f74a6..58cd6da 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -27,9 +27,10 @@ is run, please refer to the language specific cookbook
contribution documentatio
how to make your recipes testable.
* [Contributing to C++ Cookbook](cpp/CONTRIBUTING.md)
+ * [Contributing to Java Cookbook](java/CONTRIBUTING.md)
* [Contributing to Python Cookbook](python/CONTRIBUTING.rst)
* [Contributing to R Cookbook](r/CONTRIBUTING.md)
-
+
------------------------------------------------------------------------
All participation in the Apache Arrow project is governed by the Apache
diff --git a/Makefile b/Makefile
index 95978b5..76e7d0b 100644
--- a/Makefile
+++ b/Makefile
@@ -1,11 +1,11 @@
all: html
-html: py r
+html: py r java
@echo "\n\n>>> Cookbooks (except C++) Available in ./build <<<"
-test: pytest rtest
+test: pytest rtest javatest
help:
@@ -15,12 +15,16 @@ help:
@echo "make r Build the Cookbook for R only."
@echo "make pytest Verify the cookbook for Python only."
@echo "make rtest Verify the cookbook for R only."
-
+ @echo "make java Build the Cookbook for Java only."
+ @echo "make javatest Verify the cookbook for Java only."
pydeps:
@echo ">>> Installing Python Dependencies <<<\n"
cd python && pip install -r requirements.txt
+javadeps:
+ @echo ">>> Installing Java Dependencies <<<\n"
+ cd java && pip install -r requirements.txt
py: pydeps
@echo ">>> Building Python Cookbook <<<\n"
@@ -69,3 +73,16 @@ cpp: cpptest
cd cpp && make html
mkdir -p build/cpp
cp -r cpp/build/html/* build/cpp
+
+
+.PHONY: java
+java: javadeps
+ @echo ">>> Building Java Cookbook <<<\n"
+ cd java && make html
+ mkdir -p build/java
+ cp -r java/build/html/* build/java
+
+
+javatest: javadeps
+ @echo ">>> Testing Java Cookbook <<<\n"
+ cd java && make javadoctest
diff --git a/build/index.html b/build/index.html
index a1a2a93..ca4a15e 100644
--- a/build/index.html
+++ b/build/index.html
@@ -39,6 +39,7 @@
</p>
<ul>
<li><a href="cpp/index.html">C++ Cookbook</a></li>
+ <li><a href="java/index.html">Java Cookbook</a></li>
<li><a href="py/index.html">Python Cookbook</a></li>
<li><a href="r/index.html">R Cookbook</a></li>
</ul>
diff --git a/java/.flake8 b/java/.flake8
new file mode 100644
index 0000000..ef09bcb
--- /dev/null
+++ b/java/.flake8
@@ -0,0 +1 @@
+[flake8]
diff --git a/java/.isort.cfg b/java/.isort.cfg
new file mode 100644
index 0000000..6860bdb
--- /dev/null
+++ b/java/.isort.cfg
@@ -0,0 +1,2 @@
+[settings]
+profile = black
\ No newline at end of file
diff --git a/java/CONTRIBUTING.rst b/java/CONTRIBUTING.rst
new file mode 100644
index 0000000..2419b57
--- /dev/null
+++ b/java/CONTRIBUTING.rst
@@ -0,0 +1,80 @@
+Bulding the Java Cookbook
+=========================
+
+The Java cookbook uses the Sphinx documentation system.
+
+Running ``make java`` from the cookbook root directory (the one where
+the ``README.rst`` exists) will install all necessary dependencies
+and will compile the cookbook to HTML.
+
+For java cookbook we are running these with Java Shell tool -
+`JShell
<https://docs.oracle.com/en/java/javase/11/jshell/introduction-jshell.html>`_
+
+.. code-block:: bash
+
+ > java --version
+ java 11.0.14 2022-01-18 LTS
+ Java(TM) SE Runtime Environment 18.9 (build 11.0.14+8-LTS-263)
+
+.. code-block:: bash
+
+ > jshell --version
+ jshell 11.0.14
+
+You will see the compiled result inside the ``build/java`` directory.
+
+Testing Java Recipes
+====================
+
+All recipes in the cookbook must be tested. The cookbook uses
+``javadoctest`` to verify the recipes.
+
+Running ``make javatest`` from the cookbook root directory
+will verify that the code for all the recipes runs correctly
+and provides the expected output.
+
+Adding Java Recipes
+===================
+
+The recipes are written in **reStructuredText** format using
+the `Sphinx <https://www.sphinx-doc.org/>`_ documentation system.
+
+New recipes can be added to one of the existing ``.rst`` files if
+they suit that section or you can create new sections by adding
+additional ``.rst`` files in the ``source`` directory. You just
+need to remember to add them to the ``index.rst`` file in the
+``toctree`` for them to become visible.
+
+Java Sphinx Directive
+=====================
+
+To support testing java code documentation a sphinx directive
+was created: ext/javadoctest.py
+
+Execute validations before commit sphinx directive extension:
+
+Format code (before committing)
+
+.. code-block:: bash
+
+ > cd java/ext
+ > black javadoctest.py
+
+Sort imports (before committing)
+
+.. code-block:: bash
+
+ > cd java/ext
+ > isort javadoctest.py
+
+Lint code (before committing)
+
+.. code-block:: bash
+
+ > cd java/ext
+ > flake8
+------------------------------------------------------------------------
+
+All participation in the Apache Arrow project is governed by the Apache
+Software Foundation’s
+`code of conduct <https://www.apache.org/foundation/policies/conduct.html>`_.
diff --git a/java/Makefile b/java/Makefile
new file mode 100644
index 0000000..d0c3cbf
--- /dev/null
+++ b/java/Makefile
@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS ?=
+SPHINXBUILD ?= sphinx-build
+SOURCEDIR = source
+BUILDDIR = build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+ @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+ @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
diff --git a/java/ext/javadoctest.py b/java/ext/javadoctest.py
new file mode 100644
index 0000000..9e8836f
--- /dev/null
+++ b/java/ext/javadoctest.py
@@ -0,0 +1,95 @@
+import os
+import pathlib
+import subprocess
+
+from sphinx.ext.doctest import (Any, Dict, DocTestBuilder, TestcodeDirective,
+ TestoutputDirective, doctest, sphinx)
+from sphinx.locale import __
+
+
+class JavaDocTestBuilder(DocTestBuilder):
+ """
+ Runs java test snippets in the documentation.
+ """
+
+ name = "javadoctest"
+ epilog = __(
+ "Java testing of doctests in the sources finished, look at the "
+ "results in %(outdir)s/output.txt."
+ )
+
+ def compile(
+ self, code: str, name: str, type: str, flags: Any, dont_inherit: bool
+ ) -> Any:
+ # go to project that contains all your arrow maven dependencies
+ path_arrow_project = os.path.join(pathlib.Path.cwd(), "source", "demo")
+
+ # create list of all arrow jar dependencies
+ subprocess.check_call(
+ [
+ "mvn",
+ "-q",
+ "dependency:build-classpath",
+ "-DincludeTypes=jar",
+ "-Dmdep.outputFile=.cp.tmp",
+ ],
+ cwd=path_arrow_project,
+ text=True,
+ )
+ if not os.path.exists(path_arrow_project + "/.cp.tmp"):
+ raise RuntimeError(
+ __("invalid process to create jshell dependencies library")
+ )
+
+ # get list of all arrow jar dependencies
+ with open(os.path.join(path_arrow_project, ".cp.tmp")) as f:
+ stdout_dependency = f.read()
+ if not stdout_dependency:
+ raise RuntimeError(
+ __("invalid process to list jshell dependencies library")
+ )
+
+ # execute java testing code thru jshell and read output
+ # JDK11 support '-' This allows the pipe to work as expected without
requiring a shell
+ # Migrating to /dev/stdin to also support JDK9+
+ proc_jshell_process = subprocess.Popen(
+ ["jshell", "--class-path", stdout_dependency, "-s", "/dev/stdin"],
+ stdin=subprocess.PIPE,
+ stdout=subprocess.PIPE,
+ text=True,
+ )
+ out_java_arrow, err_java_arrow = proc_jshell_process.communicate(code)
+ if err_java_arrow:
+ raise RuntimeError(__("invalid process to run jshell"))
+
+ # continue with python logic code to do java output validation
+ output = f"print('''{self.clean_output(out_java_arrow)}''')"
+
+ # continue with sphinx default logic
+ return compile(output, name, self.type, flags, dont_inherit)
+
+ def clean_output(self, output: str):
+ if output[-3:] == '-> ':
+ output = output[:-3]
+ if output[-1:] == '\n':
+ output = output[:-1]
+ output = (4*' ').join(output.split('\t'))
+ return output
+
+def setup(app) -> Dict[str, Any]:
+ app.add_directive("testcode", TestcodeDirective)
+ app.add_directive("testoutput", TestoutputDirective)
+ app.add_builder(JavaDocTestBuilder)
+ # this config value adds to sys.path
+ app.add_config_value("doctest_path", [], False)
+ app.add_config_value("doctest_test_doctest_blocks", "default", False)
+ app.add_config_value("doctest_global_setup", "", False)
+ app.add_config_value("doctest_global_cleanup", "", False)
+ app.add_config_value(
+ "doctest_default_flags",
+ doctest.DONT_ACCEPT_TRUE_FOR_1
+ | doctest.ELLIPSIS
+ | doctest.IGNORE_EXCEPTION_DETAIL,
+ False,
+ )
+ return {"version": sphinx.__display_version__, "parallel_read_safe": True}
diff --git a/java/make.bat b/java/make.bat
new file mode 100644
index 0000000..6fcf05b
--- /dev/null
+++ b/java/make.bat
@@ -0,0 +1,35 @@
+@ECHO OFF
+
+pushd %~dp0
+
+REM Command file for Sphinx documentation
+
+if "%SPHINXBUILD%" == "" (
+ set SPHINXBUILD=sphinx-build
+)
+set SOURCEDIR=source
+set BUILDDIR=build
+
+if "%1" == "" goto help
+
+%SPHINXBUILD% >NUL 2>NUL
+if errorlevel 9009 (
+ echo.
+ echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
+ echo.installed, then set the SPHINXBUILD environment variable to point
+ echo.to the full path of the 'sphinx-build' executable. Alternatively
you
+ echo.may add the Sphinx directory to PATH.
+ echo.
+ echo.If you don't have Sphinx installed, grab it from
+ echo.https://www.sphinx-doc.org/
+ exit /b 1
+)
+
+%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+goto end
+
+:help
+%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+
+:end
+popd
diff --git a/java/requirements.txt b/java/requirements.txt
new file mode 100644
index 0000000..9e7f275
--- /dev/null
+++ b/java/requirements.txt
@@ -0,0 +1,3 @@
+black==21.7b0
+flake8==3.8.3
+isort==4.3.21
\ No newline at end of file
diff --git a/java/source/conf.py b/java/source/conf.py
new file mode 100644
index 0000000..767fab7
--- /dev/null
+++ b/java/source/conf.py
@@ -0,0 +1,53 @@
+# Configuration file for the Sphinx documentation builder.
+#
+# This file only contains a selection of the most common options. For a full
+# list see the documentation:
+# https://www.sphinx-doc.org/en/master/usage/configuration.html
+
+# -- Path setup --------------------------------------------------------------
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+import os
+import sys
+
+sys.path.append(os.path.abspath("../ext"))
+
+
+# -- Project information -----------------------------------------------------
+
+project = 'Apache Arrow Java Cookbook'
+copyright = '2022, Apache Software Foundation'
+author = 'The Apache Software Foundation'
+
+
+# -- General configuration ---------------------------------------------------
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = [
+ "javadoctest"
+]
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ['_templates']
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+# This pattern also affects html_static_path and html_extra_path.
+exclude_patterns = []
+
+
+# -- Options for HTML output -------------------------------------------------
+
+# The theme to use for HTML and HTML Help pages. See the documentation for
+# a list of builtin themes.
+#
+html_theme = 'alabaster'
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+html_static_path = ['_static']
diff --git a/java/source/create.rst b/java/source/create.rst
new file mode 100644
index 0000000..39017c4
--- /dev/null
+++ b/java/source/create.rst
@@ -0,0 +1,37 @@
+.. _arrow-create:
+
+======================
+Creating Arrow Objects
+======================
+
+| A vector is the basic unit in the Arrow Java library. Vector by definition
is intended to be mutable, a Vector can be changed it is mutable.
+
+| Vectors are provided by java arrow for the interface `FieldVector
<https://arrow.apache.org/docs/java/reference/org/apache/arrow/vector/FieldVector.html>`_
that extends `ValueVector <https://arrow.apache.org/docs/java/vector.html>`_.
+
+.. contents::
+
+Creating Vectors (arrays)
+=========================
+
+Array of Int (32-bit integer value)
+-----------------------------------
+
+.. testcode::
+
+ import org.apache.arrow.vector.IntVector;
+ import org.apache.arrow.memory.RootAllocator;
+
+ RootAllocator rootAllocator = new RootAllocator(Long.MAX_VALUE);
+
+ IntVector intVector = new IntVector("intVector", rootAllocator);
+ intVector.allocateNew(3);
+ intVector.set(0, 1);
+ intVector.set(1, 2);
+ intVector.set(2, 3);
+ intVector.setValueCount(3);
+
+ System.out.println(intVector);
+
+.. testoutput::
+
+ [1, 2, 3]
\ No newline at end of file
diff --git a/java/source/demo/pom.xml b/java/source/demo/pom.xml
new file mode 100644
index 0000000..2d1c9fd
--- /dev/null
+++ b/java/source/demo/pom.xml
@@ -0,0 +1,55 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0";
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+ <modelVersion>4.0.0</modelVersion>
+
+ <groupId>org.example</groupId>
+ <artifactId>demo</artifactId>
+ <version>1.0-SNAPSHOT</version>
+
+ <properties>
+ <maven.compiler.source>8</maven.compiler.source>
+ <maven.compiler.target>8</maven.compiler.target>
+ <arrow.version>6.0.0</arrow.version>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.arrow</groupId>
+ <artifactId>arrow-vector</artifactId>
+ <version>${arrow.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.arrow</groupId>
+ <artifactId>arrow-memory-unsafe</artifactId>
+ <version>${arrow.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.arrow</groupId>
+ <artifactId>arrow-algorithm</artifactId>
+ <version>${arrow.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.arrow</groupId>
+ <artifactId>flight-core</artifactId>
+ <version>${arrow.version}</version>
+ <exclusions>
+ <exclusion>
+ <groupId>io.netty</groupId>
+ <artifactId>netty-transport-native-unix-common</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>io.netty</groupId>
+ <artifactId>netty-transport-native-kqueue</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>junit</groupId>
+ <artifactId>junit</artifactId>
+ <version>4.13.2</version>
+ </dependency>
+ </dependencies>
+
+</project>
\ No newline at end of file
diff --git a/java/source/index.rst b/java/source/index.rst
new file mode 100644
index 0000000..d6c1ccd
--- /dev/null
+++ b/java/source/index.rst
@@ -0,0 +1,20 @@
+.. java arrow documentation master file, created by
+ sphinx-quickstart on Wed Jan 19 08:34:06 2022.
+ You can adapt this file completely to your liking, but it should at least
+ contain the root `toctree` directive.
+
+Welcome to java arrow's documentation!
+======================================
+
+.. toctree::
+ :maxdepth: 2
+ :caption: Contents:
+
+ create
+
+Indices and tables
+==================
+
+* :ref:`genindex`
+* :ref:`modindex`
+* :ref:`search`
