lidavidm commented on a change in pull request #12534: URL: https://github.com/apache/arrow/pull/12534#discussion_r817156534
########## File path: docs/source/developers/java/building.rst ########## @@ -0,0 +1,308 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. highlight:: console + +.. _building-arrow-cpp: Review comment: ```suggestion .. _building-arrow-java: ``` ########## File path: docs/source/developers/java/building.rst ########## @@ -0,0 +1,308 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. highlight:: console + +.. _building-arrow-cpp: + +=================== +Building Arrow Java +=================== + +.. contents:: + +System setup +============ + +Arrow Java use Maven as a project management. Review comment: ```suggestion Arrow Java uses the `Maven <https://maven.apache.org/>`_ build system. ``` ########## File path: docs/source/developers/java/building.rst ########## @@ -0,0 +1,308 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. highlight:: console + +.. _building-arrow-cpp: + +=================== +Building Arrow Java +=================== + +.. contents:: + +System setup +============ + +Arrow Java use Maven as a project management. + +Building requires: + +* JRE 8, 9, 10, 11 +* Maven 3+ + +Building +======== + +All the instructions below assume that you have cloned the Arrow git +repository: + +.. code-block:: + + $ git clone https://github.com/apache/arrow.git + $ cd arrow + $ git submodule update --init --recursive + +Basic Installation +------------------ + +You can also build java project for default modules configured. +Go to parent project and execute: + +.. code-block:: + + $ cd arrow/java + $ export JAVA_HOME=<absolute path to your java home> + $ java --version + $ mvn clean install + [INFO] Reactor Summary for Apache Arrow Java Root POM 8.0.0-SNAPSHOT: + [INFO] + [INFO] Apache Arrow Java Root POM ......................... SUCCESS [ 4.922 s] + [INFO] Arrow Format ....................................... SUCCESS [ 4.395 s] + [INFO] Arrow Memory ....................................... SUCCESS [ 0.938 s] + [INFO] Arrow Memory - Core ................................ SUCCESS [ 11.491 s] + [INFO] Arrow Memory - Unsafe .............................. SUCCESS [ 5.449 s] + [INFO] Arrow Memory - Netty ............................... SUCCESS [ 8.754 s] + [INFO] Arrow Vectors ...................................... SUCCESS [01:37 min] + [INFO] Arrow Compression .................................. SUCCESS [ 5.907 s] + [INFO] Arrow Tools ........................................ SUCCESS [ 11.015 s] + [INFO] Arrow JDBC Adapter ................................. SUCCESS [ 13.751 s] + [INFO] Arrow Plasma Client ................................ SUCCESS [ 3.934 s] + [INFO] Arrow Flight ....................................... SUCCESS [ 0.649 s] + [INFO] Arrow Flight Core .................................. SUCCESS [ 38.331 s] + [INFO] Arrow Flight GRPC .................................. SUCCESS [ 5.894 s] + [INFO] Arrow Flight SQL ................................... SUCCESS [ 21.862 s] + [INFO] Arrow Flight Integration Tests ..................... SUCCESS [ 7.638 s] + [INFO] Arrow AVRO Adapter ................................. SUCCESS [ 14.447 s] + [INFO] Arrow Algorithms ................................... SUCCESS [ 35.105 s] + [INFO] Arrow Performance Benchmarks ....................... SUCCESS [ 5.323 s] + [INFO] ------------------------------------------------------------------------ + [INFO] BUILD SUCCESS + [INFO] ------------------------------------------------------------------------\ + +Advanced Installation +--------------------- + +Build C / C++ Libs +++++++++++++++++++ + +Build C / C++ Libs Ubuntu +######################### + +For Build C Data Interface lib & Build C++ Libs. Consider: Next step requires: Docker & Docker Compose. + +.. code-block:: + + $ cd arrow + $ source dev/conbench_envs/hooks.sh install_archery Review comment: Let's not use the internal hooks, we can link to the Archery docs instead ########## File path: docs/source/developers/java/building.rst ########## @@ -0,0 +1,308 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. highlight:: console + +.. _building-arrow-cpp: + +=================== +Building Arrow Java +=================== + +.. contents:: + +System setup +============ + +Arrow Java use Maven as a project management. + +Building requires: + +* JRE 8, 9, 10, 11 +* Maven 3+ + +Building +======== + +All the instructions below assume that you have cloned the Arrow git +repository: + +.. code-block:: + + $ git clone https://github.com/apache/arrow.git + $ cd arrow + $ git submodule update --init --recursive + +Basic Installation +------------------ + +You can also build java project for default modules configured. +Go to parent project and execute: + +.. code-block:: + + $ cd arrow/java + $ export JAVA_HOME=<absolute path to your java home> + $ java --version + $ mvn clean install + [INFO] Reactor Summary for Apache Arrow Java Root POM 8.0.0-SNAPSHOT: + [INFO] + [INFO] Apache Arrow Java Root POM ......................... SUCCESS [ 4.922 s] + [INFO] Arrow Format ....................................... SUCCESS [ 4.395 s] + [INFO] Arrow Memory ....................................... SUCCESS [ 0.938 s] + [INFO] Arrow Memory - Core ................................ SUCCESS [ 11.491 s] + [INFO] Arrow Memory - Unsafe .............................. SUCCESS [ 5.449 s] + [INFO] Arrow Memory - Netty ............................... SUCCESS [ 8.754 s] + [INFO] Arrow Vectors ...................................... SUCCESS [01:37 min] + [INFO] Arrow Compression .................................. SUCCESS [ 5.907 s] + [INFO] Arrow Tools ........................................ SUCCESS [ 11.015 s] + [INFO] Arrow JDBC Adapter ................................. SUCCESS [ 13.751 s] + [INFO] Arrow Plasma Client ................................ SUCCESS [ 3.934 s] + [INFO] Arrow Flight ....................................... SUCCESS [ 0.649 s] + [INFO] Arrow Flight Core .................................. SUCCESS [ 38.331 s] + [INFO] Arrow Flight GRPC .................................. SUCCESS [ 5.894 s] + [INFO] Arrow Flight SQL ................................... SUCCESS [ 21.862 s] + [INFO] Arrow Flight Integration Tests ..................... SUCCESS [ 7.638 s] + [INFO] Arrow AVRO Adapter ................................. SUCCESS [ 14.447 s] + [INFO] Arrow Algorithms ................................... SUCCESS [ 35.105 s] + [INFO] Arrow Performance Benchmarks ....................... SUCCESS [ 5.323 s] + [INFO] ------------------------------------------------------------------------ + [INFO] BUILD SUCCESS + [INFO] ------------------------------------------------------------------------\ + +Advanced Installation +--------------------- + +Build C / C++ Libs +++++++++++++++++++ + +Build C / C++ Libs Ubuntu +######################### + +For Build C Data Interface lib & Build C++ Libs. Consider: Next step requires: Docker & Docker Compose. + +.. code-block:: + + $ cd arrow + $ source dev/conbench_envs/hooks.sh install_archery + Successfully installed archery + $ archery docker run java-jni-manylinux-2014 + $ ls -latr java-dist/ + |__ libarrow_cdata_jni.so + |__ libarrow_dataset_jni.so + |__ libarrow_orc_jni.so + |__ libgandiva_jni.so + +Build C / C++ Libs MacOS +######################## + +For Build C Data Interface lib: + +.. code-block:: + + $ cd arrow + $ brew bundle --file=cpp/Brewfile + Homebrew Bundle complete! 25 Brewfile dependencies now installed. + $ export GITHUB_WORKSPACE=<absolute path to your arrow project> Review comment: Let's not use CI-specific variables, we can just get rid of the env var and use `./java_dist`, etc ########## File path: docs/source/developers/java/building.rst ########## @@ -0,0 +1,308 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. highlight:: console + +.. _building-arrow-cpp: + +=================== +Building Arrow Java +=================== + +.. contents:: + +System setup +============ + +Arrow Java use Maven as a project management. + +Building requires: + +* JRE 8, 9, 10, 11 +* Maven 3+ + +Building +======== + +All the instructions below assume that you have cloned the Arrow git +repository: + +.. code-block:: + + $ git clone https://github.com/apache/arrow.git + $ cd arrow + $ git submodule update --init --recursive + +Basic Installation +------------------ + +You can also build java project for default modules configured. +Go to parent project and execute: + +.. code-block:: + + $ cd arrow/java + $ export JAVA_HOME=<absolute path to your java home> + $ java --version + $ mvn clean install + [INFO] Reactor Summary for Apache Arrow Java Root POM 8.0.0-SNAPSHOT: + [INFO] + [INFO] Apache Arrow Java Root POM ......................... SUCCESS [ 4.922 s] + [INFO] Arrow Format ....................................... SUCCESS [ 4.395 s] + [INFO] Arrow Memory ....................................... SUCCESS [ 0.938 s] + [INFO] Arrow Memory - Core ................................ SUCCESS [ 11.491 s] + [INFO] Arrow Memory - Unsafe .............................. SUCCESS [ 5.449 s] + [INFO] Arrow Memory - Netty ............................... SUCCESS [ 8.754 s] + [INFO] Arrow Vectors ...................................... SUCCESS [01:37 min] + [INFO] Arrow Compression .................................. SUCCESS [ 5.907 s] + [INFO] Arrow Tools ........................................ SUCCESS [ 11.015 s] + [INFO] Arrow JDBC Adapter ................................. SUCCESS [ 13.751 s] + [INFO] Arrow Plasma Client ................................ SUCCESS [ 3.934 s] + [INFO] Arrow Flight ....................................... SUCCESS [ 0.649 s] + [INFO] Arrow Flight Core .................................. SUCCESS [ 38.331 s] + [INFO] Arrow Flight GRPC .................................. SUCCESS [ 5.894 s] + [INFO] Arrow Flight SQL ................................... SUCCESS [ 21.862 s] + [INFO] Arrow Flight Integration Tests ..................... SUCCESS [ 7.638 s] + [INFO] Arrow AVRO Adapter ................................. SUCCESS [ 14.447 s] + [INFO] Arrow Algorithms ................................... SUCCESS [ 35.105 s] + [INFO] Arrow Performance Benchmarks ....................... SUCCESS [ 5.323 s] + [INFO] ------------------------------------------------------------------------ + [INFO] BUILD SUCCESS + [INFO] ------------------------------------------------------------------------\ + +Advanced Installation +--------------------- + +Build C / C++ Libs +++++++++++++++++++ + +Build C / C++ Libs Ubuntu +######################### + +For Build C Data Interface lib & Build C++ Libs. Consider: Next step requires: Docker & Docker Compose. + +.. code-block:: + + $ cd arrow + $ source dev/conbench_envs/hooks.sh install_archery + Successfully installed archery + $ archery docker run java-jni-manylinux-2014 + $ ls -latr java-dist/ + |__ libarrow_cdata_jni.so + |__ libarrow_dataset_jni.so + |__ libarrow_orc_jni.so + |__ libgandiva_jni.so + +Build C / C++ Libs MacOS +######################## + +For Build C Data Interface lib: + +.. code-block:: + + $ cd arrow + $ brew bundle --file=cpp/Brewfile + Homebrew Bundle complete! 25 Brewfile dependencies now installed. + $ export GITHUB_WORKSPACE=<absolute path to your arrow project> + $ export JAVA_HOME=<absolute path to your java home> + $ mkdir -p java-dist java-native-c + $ cd java-native-c + $ cmake \ + -DCMAKE_BUILD_TYPE=Release \ + -DCMAKE_INSTALL_LIBDIR=lib \ + -DCMAKE_INSTALL_PREFIX=$GITHUB_WORKSPACE/java-dist \ + $GITHUB_WORKSPACE/java/c + $ cmake --build . --target install + $ ls -latr $GITHUB_WORKSPACE/java-dist/lib + |__ libarrow_cdata_jni.dylib + +For Build C++ Interface libs: + +.. code-block:: + + $ cd arrow + $ brew bundle --file=cpp/Brewfile + Homebrew Bundle complete! 25 Brewfile dependencies now installed. + $ export GITHUB_WORKSPACE=<absolute path to your arrow project> + $ export JAVA_HOME=<absolute path to your java home> + $ mkdir -p java-dist java-native-cpp + $ cd java-native-cpp + $ cmake \ + -DARROW_BOOST_USE_SHARED=OFF \ + -DARROW_BROTLI_USE_SHARED=OFF \ + -DARROW_BZ2_USE_SHARED=OFF \ + -DARROW_GFLAGS_USE_SHARED=OFF \ + -DARROW_GRPC_USE_SHARED=OFF \ + -DARROW_LZ4_USE_SHARED=OFF \ + -DARROW_OPENSSL_USE_SHARED=OFF \ + -DARROW_PROTOBUF_USE_SHARED=OFF \ + -DARROW_SNAPPY_USE_SHARED=OFF \ + -DARROW_THRIFT_USE_SHARED=OFF \ + -DARROW_UTF8PROC_USE_SHARED=OFF \ + -DARROW_ZSTD_USE_SHARED=OFF \ + -DARROW_JNI=ON \ + -DARROW_PARQUET=ON \ + -DARROW_FILESYSTEM=ON \ + -DARROW_DATASET=ON \ + -DARROW_GANDIVA_JAVA=ON \ + -DARROW_GANDIVA_STATIC_LIBSTDCPP=ON \ + -DARROW_GANDIVA=ON \ + -DARROW_ORC=ON \ + -DARROW_PLASMA_JAVA_CLIENT=ON \ + -DARROW_PLASMA=ON \ + -DCMAKE_BUILD_TYPE=Release \ + -DCMAKE_INSTALL_LIBDIR=lib \ + -DCMAKE_INSTALL_PREFIX=$GITHUB_WORKSPACE/java-dist \ + -DCMAKE_UNITY_BUILD=ON \ + -Dre2_SOURCE=BUNDLED \ + -DBoost_SOURCE=BUNDLED \ + -Dutf8proc_SOURCE=BUNDLED \ + -DSnappy_SOURCE=BUNDLED \ + -DORC_SOURCE=BUNDLED \ + -DZLIB_SOURCE=BUNDLED \ + $GITHUB_WORKSPACE/cpp + $ cmake --build . --target install + $ ls -latr $GITHUB_WORKSPACE/java-dist/lib + |__ libarrow_dataset_jni.dylib + |__ libarrow_orc_jni.dylib + |__ libgandiva_jni.dylib + +Build Java Modules That Consume C / C++ Libs +++++++++++++++++++++++++++++++++++++++++++++ + +Build Java C Data Interface +########################### + +Java arrow-c-data maven profile: + +.. code-block:: + + $ cd arrow/java + $ mvn -Darrow.c.jni.dist.dir=$GITHUB_WORKSPACE/java-dist/lib -Parrow-c-data clean install + [INFO] Reactor Summary for Apache Arrow Java Root POM 7.0.0: + [INFO] + [INFO] Apache Arrow Java Root POM ......................... SUCCESS [ 4.385 s] + [INFO] Arrow Format ....................................... SUCCESS [ 4.832 s] + [INFO] Arrow Memory ....................................... SUCCESS [ 1.862 s] + [INFO] Arrow Memory - Core ................................ SUCCESS [ 12.162 s] + [INFO] Arrow Memory - Unsafe .............................. SUCCESS [ 6.768 s] + [INFO] Arrow Memory - Netty ............................... SUCCESS [ 6.897 s] + [INFO] Arrow Vectors ...................................... SUCCESS [01:43 min] + [INFO] Arrow Compression .................................. SUCCESS [ 6.510 s] + [INFO] Arrow Tools ........................................ SUCCESS [ 13.280 s] + [INFO] Arrow JDBC Adapter ................................. SUCCESS [ 13.726 s] + [INFO] Arrow Plasma Client ................................ SUCCESS [ 5.237 s] + [INFO] Arrow Flight ....................................... SUCCESS [ 0.809 s] + [INFO] Arrow Flight Core .................................. SUCCESS [ 47.461 s] + [INFO] Arrow Flight GRPC .................................. SUCCESS [ 6.864 s] + [INFO] Arrow Flight SQL ................................... SUCCESS [ 20.334 s] + [INFO] Arrow Flight Integration Tests ..................... SUCCESS [ 7.542 s] + [INFO] Arrow AVRO Adapter ................................. SUCCESS [ 13.826 s] + [INFO] Arrow Algorithms ................................... SUCCESS [ 32.442 s] + [INFO] Arrow Performance Benchmarks ....................... SUCCESS [ 5.418 s] + [INFO] Arrow Java C Data Interface ........................ SUCCESS [ 7.907 s] + [INFO] ------------------------------------------------------------------------ + [INFO] BUILD SUCCESS + [INFO] ------------------------------------------------------------------------ + +Build Java C++ Interface +######################## + +Java arrow-jni maven profile (build ORC / Gandiva / Dataset): + +.. code-block:: + + $ cd arrow/java + $ mvn -Darrow.cpp.build.dir=$GITHUB_WORKSPACE/java-dist/lib -Parrow-jni clean install + [INFO] Reactor Summary for Apache Arrow Java Root POM 7.0.0: + [INFO] + [INFO] Apache Arrow Java Root POM ......................... SUCCESS [ 7.342 s] + [INFO] Arrow Format ....................................... SUCCESS [ 2.417 s] + [INFO] Arrow Memory ....................................... SUCCESS [ 1.967 s] + [INFO] Arrow Memory - Core ................................ SUCCESS [ 4.714 s] + [INFO] Arrow Memory - Unsafe .............................. SUCCESS [ 3.157 s] + [INFO] Arrow Memory - Netty ............................... SUCCESS [ 3.334 s] + [INFO] Arrow Vectors ...................................... SUCCESS [ 21.791 s] + [INFO] Arrow Compression .................................. SUCCESS [ 3.854 s] + [INFO] Arrow Tools ........................................ SUCCESS [ 8.359 s] + [INFO] Arrow JDBC Adapter ................................. SUCCESS [ 8.847 s] + [INFO] Arrow Plasma Client ................................ SUCCESS [ 2.459 s] + [INFO] Arrow Flight ....................................... SUCCESS [ 2.357 s] + [INFO] Arrow Flight Core .................................. SUCCESS [ 38.837 s] + [INFO] Arrow Flight GRPC .................................. SUCCESS [ 5.955 s] + [INFO] Arrow Flight SQL ................................... SUCCESS [ 17.390 s] + [INFO] Arrow Flight Integration Tests ..................... SUCCESS [ 6.148 s] + [INFO] Arrow AVRO Adapter ................................. SUCCESS [ 9.635 s] + [INFO] Arrow Algorithms ................................... SUCCESS [ 26.949 s] + [INFO] Arrow Performance Benchmarks ....................... SUCCESS [ 4.211 s] + [INFO] Arrow Orc Adapter .................................. SUCCESS [ 6.522 s] + [INFO] Arrow Gandiva ...................................... SUCCESS [01:20 min] + [INFO] Arrow Java Dataset ................................. SUCCESS [ 12.949 s] + [INFO] ------------------------------------------------------------------------ + [INFO] BUILD SUCCESS + [INFO] ------------------------------------------------------------------------ + +IDE Configuration +================= + +Go to open java project and select java folder. + +This is the initial view for java project loaded with default profiles: + +.. image:: img/java_welcome.png + +Let's create our maven configuration: java-dataset + +.. image:: img/java_mvn_configuration.png + +Let's define our JRE runner: + +.. image:: img/java_jre_runner.png + +Let's configure additional environment properties: + +.. image:: img/java_jre_env_properties.png + +Let's run our java-dataset maven configuration: + +.. image:: img/java_run_mvn_configuration.png + +.. image:: img/java_run_mvn_configuration_result.png + +Common Errors +============= + +1.- Errors related to: + - Could NOT find Boost (missing: Boost_INCLUDE_DIR system filesystem) + - Could NOT find Lz4 (missing: LZ4_LIB) + - Could NOT find zstd (missing: ZSTD_LIB) + + Consider to configure parameters that download software at the build time with: Review comment: ```suggestion Download the dependencies at build time: ``` ########## File path: docs/source/developers/java/development.rst ########## @@ -0,0 +1,170 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. highlight:: console + +====================== +Development Guidelines +====================== + +.. contents:: + +Unit Testing +============ + +.. code-block:: + + $ cd arrow/java + $ mvn \ + -Darrow.cpp.build.dir=$GITHUB_WORKSPACE/java-dist/lib -Parrow-jni \ + -Darrow.c.jni.dist.dir=$GITHUB_WORKSPACE/java-dist/lib -Parrow-c-data \ + clean install + [INFO] Reactor Summary for Apache Arrow Java Root POM 7.0.0: + [INFO] + [INFO] Apache Arrow Java Root POM ......................... SUCCESS [ 7.308 s] + [INFO] Arrow Format ....................................... SUCCESS [ 4.899 s] + [INFO] Arrow Memory ....................................... SUCCESS [ 2.065 s] + [INFO] Arrow Memory - Core ................................ SUCCESS [ 11.417 s] + [INFO] Arrow Memory - Unsafe .............................. SUCCESS [ 8.132 s] + [INFO] Arrow Memory - Netty ............................... SUCCESS [ 10.211 s] + [INFO] Arrow Vectors ...................................... SUCCESS [01:48 min] + [INFO] Arrow Compression .................................. SUCCESS [ 7.282 s] + [INFO] Arrow Tools ........................................ SUCCESS [ 12.612 s] + [INFO] Arrow JDBC Adapter ................................. SUCCESS [ 15.558 s] + [INFO] Arrow Plasma Client ................................ SUCCESS [ 6.303 s] + [INFO] Arrow Flight ....................................... SUCCESS [ 2.326 s] + [INFO] Arrow Flight Core .................................. SUCCESS [ 50.632 s] + [INFO] Arrow Flight GRPC .................................. SUCCESS [ 9.932 s] + [INFO] Arrow Flight SQL ................................... SUCCESS [ 22.559 s] + [INFO] Arrow Flight Integration Tests ..................... SUCCESS [ 9.649 s] + [INFO] Arrow AVRO Adapter ................................. SUCCESS [ 17.145 s] + [INFO] Arrow Algorithms ................................... SUCCESS [ 44.571 s] + [INFO] Arrow Performance Benchmarks ....................... SUCCESS [ 9.202 s] + [INFO] Arrow Java C Data Interface ........................ SUCCESS [ 15.794 s] + [INFO] Arrow Orc Adapter .................................. SUCCESS [ 14.205 s] + [INFO] Arrow Gandiva ...................................... SUCCESS [02:06 min] + [INFO] Arrow Java Dataset ................................. SUCCESS [ 15.261 s] + [INFO] ------------------------------------------------------------------------ + [INFO] BUILD SUCCESS + [INFO] ------------------------------------------------------------------------ + +Performance Testing +=================== + +Java module that contains performance tests is: `performance` Review comment: Backticks don't do what you want here, you want something like ``` The ``arrow-performance`` module contains benchmarks. ``` ########## File path: docs/source/developers/java/development.rst ########## @@ -0,0 +1,170 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. highlight:: console + +====================== +Development Guidelines +====================== + +.. contents:: + +Unit Testing +============ + +.. code-block:: + + $ cd arrow/java + $ mvn \ + -Darrow.cpp.build.dir=$GITHUB_WORKSPACE/java-dist/lib -Parrow-jni \ + -Darrow.c.jni.dist.dir=$GITHUB_WORKSPACE/java-dist/lib -Parrow-c-data \ + clean install + [INFO] Reactor Summary for Apache Arrow Java Root POM 7.0.0: + [INFO] + [INFO] Apache Arrow Java Root POM ......................... SUCCESS [ 7.308 s] + [INFO] Arrow Format ....................................... SUCCESS [ 4.899 s] + [INFO] Arrow Memory ....................................... SUCCESS [ 2.065 s] + [INFO] Arrow Memory - Core ................................ SUCCESS [ 11.417 s] + [INFO] Arrow Memory - Unsafe .............................. SUCCESS [ 8.132 s] + [INFO] Arrow Memory - Netty ............................... SUCCESS [ 10.211 s] + [INFO] Arrow Vectors ...................................... SUCCESS [01:48 min] + [INFO] Arrow Compression .................................. SUCCESS [ 7.282 s] + [INFO] Arrow Tools ........................................ SUCCESS [ 12.612 s] + [INFO] Arrow JDBC Adapter ................................. SUCCESS [ 15.558 s] + [INFO] Arrow Plasma Client ................................ SUCCESS [ 6.303 s] + [INFO] Arrow Flight ....................................... SUCCESS [ 2.326 s] + [INFO] Arrow Flight Core .................................. SUCCESS [ 50.632 s] + [INFO] Arrow Flight GRPC .................................. SUCCESS [ 9.932 s] + [INFO] Arrow Flight SQL ................................... SUCCESS [ 22.559 s] + [INFO] Arrow Flight Integration Tests ..................... SUCCESS [ 9.649 s] + [INFO] Arrow AVRO Adapter ................................. SUCCESS [ 17.145 s] + [INFO] Arrow Algorithms ................................... SUCCESS [ 44.571 s] + [INFO] Arrow Performance Benchmarks ....................... SUCCESS [ 9.202 s] + [INFO] Arrow Java C Data Interface ........................ SUCCESS [ 15.794 s] + [INFO] Arrow Orc Adapter .................................. SUCCESS [ 14.205 s] + [INFO] Arrow Gandiva ...................................... SUCCESS [02:06 min] + [INFO] Arrow Java Dataset ................................. SUCCESS [ 15.261 s] + [INFO] ------------------------------------------------------------------------ + [INFO] BUILD SUCCESS + [INFO] ------------------------------------------------------------------------ + +Performance Testing +=================== + +Java module that contains performance tests is: `performance` + +Let's configure our performance environment to run performance tests: + +- Install `benchmark`_ +- Install `archery`_ +- Install `conbench`_ + +Lets execute benchmark tests: + +.. code-block:: + + $ cd benchmarks + $ conbench java-micro --help + $ conbench java-micro + --iterations=1 + --commit=e90472e35b40f58b17d408438bb8de1641bfe6ef + --java-home=<absolute path to your java home> + --src=<absolute path to your arrow project> + --benchmark-filter=org.apache.arrow.adapter.AvroAdapterBenchmarks.testAvroToArrow + Benchmark Mode Cnt Score Error Units + AvroAdapterBenchmarks.testAvroToArrow avgt 725545.783 ns/op + Time to POST http://localhost:5000/api/login/ 0.14911699295043945 + Time to POST http://localhost:5000/api/benchmarks/ 0.06116318702697754 + +Then go to: http://127.0.0.1:5000/ to see reports: + +UI Home: + +.. image:: img/conbench_ui.png + +UI Runs: + +.. image:: img/conbench_runs.png + +UI Benchmark: + +.. image:: img/conbench_benchmark.png + +Code Style +========== + +Code style are configured at `java/dev/checkstyle` + +.. code-block:: + + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-checkstyle-plugin</artifactId> + <version>3.1.0</version> + <executions> + <execution> + <id>validate</id> + <phase>validate</phase> + <goals> + <goal>check</goal> + </goals> + </execution> + </executions> + <configuration> + <configLocation>dev/checkstyle/checkstyle.xml</configLocation> + <headerLocation>dev/checkstyle/checkstyle.license</headerLocation> + <suppressionsLocation>dev/checkstyle/suppressions.xml</suppressionsLocation> + <includeTestSourceDirectory>true</includeTestSourceDirectory> + ... + </configuration> + </plugin> + +Validate check style configuration: Review comment: "The Maven build will validate code against the Checkstyle configuration. You can also just check the style without building the project:" Maybe mention how to skip this check as well. ########## File path: docs/source/developers/java/building.rst ########## @@ -0,0 +1,308 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. highlight:: console + +.. _building-arrow-cpp: + +=================== +Building Arrow Java +=================== + +.. contents:: + +System setup +============ + +Arrow Java use Maven as a project management. + +Building requires: + +* JRE 8, 9, 10, 11 +* Maven 3+ + +Building +======== + +All the instructions below assume that you have cloned the Arrow git +repository: + +.. code-block:: + + $ git clone https://github.com/apache/arrow.git + $ cd arrow + $ git submodule update --init --recursive + +Basic Installation +------------------ + +You can also build java project for default modules configured. +Go to parent project and execute: + +.. code-block:: + + $ cd arrow/java + $ export JAVA_HOME=<absolute path to your java home> + $ java --version + $ mvn clean install + [INFO] Reactor Summary for Apache Arrow Java Root POM 8.0.0-SNAPSHOT: + [INFO] + [INFO] Apache Arrow Java Root POM ......................... SUCCESS [ 4.922 s] + [INFO] Arrow Format ....................................... SUCCESS [ 4.395 s] + [INFO] Arrow Memory ....................................... SUCCESS [ 0.938 s] + [INFO] Arrow Memory - Core ................................ SUCCESS [ 11.491 s] + [INFO] Arrow Memory - Unsafe .............................. SUCCESS [ 5.449 s] + [INFO] Arrow Memory - Netty ............................... SUCCESS [ 8.754 s] + [INFO] Arrow Vectors ...................................... SUCCESS [01:37 min] + [INFO] Arrow Compression .................................. SUCCESS [ 5.907 s] + [INFO] Arrow Tools ........................................ SUCCESS [ 11.015 s] + [INFO] Arrow JDBC Adapter ................................. SUCCESS [ 13.751 s] + [INFO] Arrow Plasma Client ................................ SUCCESS [ 3.934 s] + [INFO] Arrow Flight ....................................... SUCCESS [ 0.649 s] + [INFO] Arrow Flight Core .................................. SUCCESS [ 38.331 s] + [INFO] Arrow Flight GRPC .................................. SUCCESS [ 5.894 s] + [INFO] Arrow Flight SQL ................................... SUCCESS [ 21.862 s] + [INFO] Arrow Flight Integration Tests ..................... SUCCESS [ 7.638 s] + [INFO] Arrow AVRO Adapter ................................. SUCCESS [ 14.447 s] + [INFO] Arrow Algorithms ................................... SUCCESS [ 35.105 s] + [INFO] Arrow Performance Benchmarks ....................... SUCCESS [ 5.323 s] + [INFO] ------------------------------------------------------------------------ + [INFO] BUILD SUCCESS + [INFO] ------------------------------------------------------------------------\ + +Advanced Installation Review comment: Maybe make this explicitly about "Installing JNI Modules"? ########## File path: docs/source/developers/java/building.rst ########## @@ -0,0 +1,308 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. highlight:: console + +.. _building-arrow-cpp: + +=================== +Building Arrow Java +=================== + +.. contents:: + +System setup +============ + +Arrow Java use Maven as a project management. + +Building requires: + +* JRE 8, 9, 10, 11 +* Maven 3+ + +Building +======== + +All the instructions below assume that you have cloned the Arrow git +repository: + +.. code-block:: + + $ git clone https://github.com/apache/arrow.git + $ cd arrow + $ git submodule update --init --recursive + +Basic Installation +------------------ + +You can also build java project for default modules configured. +Go to parent project and execute: Review comment: ```suggestion To build the default modules, go to the project root and execute: ``` ########## File path: docs/source/developers/java/building.rst ########## @@ -0,0 +1,308 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. highlight:: console + +.. _building-arrow-cpp: + +=================== +Building Arrow Java +=================== + +.. contents:: + +System setup +============ + +Arrow Java use Maven as a project management. + +Building requires: + +* JRE 8, 9, 10, 11 +* Maven 3+ + +Building +======== + +All the instructions below assume that you have cloned the Arrow git +repository: + +.. code-block:: + + $ git clone https://github.com/apache/arrow.git + $ cd arrow + $ git submodule update --init --recursive + +Basic Installation +------------------ + +You can also build java project for default modules configured. +Go to parent project and execute: + +.. code-block:: + + $ cd arrow/java + $ export JAVA_HOME=<absolute path to your java home> + $ java --version + $ mvn clean install + [INFO] Reactor Summary for Apache Arrow Java Root POM 8.0.0-SNAPSHOT: + [INFO] + [INFO] Apache Arrow Java Root POM ......................... SUCCESS [ 4.922 s] + [INFO] Arrow Format ....................................... SUCCESS [ 4.395 s] + [INFO] Arrow Memory ....................................... SUCCESS [ 0.938 s] + [INFO] Arrow Memory - Core ................................ SUCCESS [ 11.491 s] + [INFO] Arrow Memory - Unsafe .............................. SUCCESS [ 5.449 s] + [INFO] Arrow Memory - Netty ............................... SUCCESS [ 8.754 s] + [INFO] Arrow Vectors ...................................... SUCCESS [01:37 min] + [INFO] Arrow Compression .................................. SUCCESS [ 5.907 s] + [INFO] Arrow Tools ........................................ SUCCESS [ 11.015 s] + [INFO] Arrow JDBC Adapter ................................. SUCCESS [ 13.751 s] + [INFO] Arrow Plasma Client ................................ SUCCESS [ 3.934 s] + [INFO] Arrow Flight ....................................... SUCCESS [ 0.649 s] + [INFO] Arrow Flight Core .................................. SUCCESS [ 38.331 s] + [INFO] Arrow Flight GRPC .................................. SUCCESS [ 5.894 s] + [INFO] Arrow Flight SQL ................................... SUCCESS [ 21.862 s] + [INFO] Arrow Flight Integration Tests ..................... SUCCESS [ 7.638 s] + [INFO] Arrow AVRO Adapter ................................. SUCCESS [ 14.447 s] + [INFO] Arrow Algorithms ................................... SUCCESS [ 35.105 s] + [INFO] Arrow Performance Benchmarks ....................... SUCCESS [ 5.323 s] + [INFO] ------------------------------------------------------------------------ + [INFO] BUILD SUCCESS + [INFO] ------------------------------------------------------------------------\ + +Advanced Installation +--------------------- + +Build C / C++ Libs +++++++++++++++++++ Review comment: There's getting to be too many levels of headings here, maybe we can remove this heading and make the headings "Building C++ JNI Libraries on Linux", "Building C++ JNI Libraries on MacOS", and "Building Arrow JNI Libraries"? ########## File path: docs/source/developers/java/building.rst ########## @@ -0,0 +1,308 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. highlight:: console + +.. _building-arrow-cpp: + +=================== +Building Arrow Java +=================== + +.. contents:: + +System setup +============ + +Arrow Java use Maven as a project management. + +Building requires: + +* JRE 8, 9, 10, 11 +* Maven 3+ + +Building +======== + +All the instructions below assume that you have cloned the Arrow git +repository: + +.. code-block:: + + $ git clone https://github.com/apache/arrow.git + $ cd arrow + $ git submodule update --init --recursive + +Basic Installation +------------------ + +You can also build java project for default modules configured. +Go to parent project and execute: + +.. code-block:: + + $ cd arrow/java + $ export JAVA_HOME=<absolute path to your java home> + $ java --version + $ mvn clean install + [INFO] Reactor Summary for Apache Arrow Java Root POM 8.0.0-SNAPSHOT: + [INFO] + [INFO] Apache Arrow Java Root POM ......................... SUCCESS [ 4.922 s] + [INFO] Arrow Format ....................................... SUCCESS [ 4.395 s] + [INFO] Arrow Memory ....................................... SUCCESS [ 0.938 s] + [INFO] Arrow Memory - Core ................................ SUCCESS [ 11.491 s] + [INFO] Arrow Memory - Unsafe .............................. SUCCESS [ 5.449 s] + [INFO] Arrow Memory - Netty ............................... SUCCESS [ 8.754 s] + [INFO] Arrow Vectors ...................................... SUCCESS [01:37 min] + [INFO] Arrow Compression .................................. SUCCESS [ 5.907 s] + [INFO] Arrow Tools ........................................ SUCCESS [ 11.015 s] + [INFO] Arrow JDBC Adapter ................................. SUCCESS [ 13.751 s] + [INFO] Arrow Plasma Client ................................ SUCCESS [ 3.934 s] + [INFO] Arrow Flight ....................................... SUCCESS [ 0.649 s] + [INFO] Arrow Flight Core .................................. SUCCESS [ 38.331 s] + [INFO] Arrow Flight GRPC .................................. SUCCESS [ 5.894 s] + [INFO] Arrow Flight SQL ................................... SUCCESS [ 21.862 s] + [INFO] Arrow Flight Integration Tests ..................... SUCCESS [ 7.638 s] + [INFO] Arrow AVRO Adapter ................................. SUCCESS [ 14.447 s] + [INFO] Arrow Algorithms ................................... SUCCESS [ 35.105 s] + [INFO] Arrow Performance Benchmarks ....................... SUCCESS [ 5.323 s] + [INFO] ------------------------------------------------------------------------ + [INFO] BUILD SUCCESS + [INFO] ------------------------------------------------------------------------\ + +Advanced Installation +--------------------- + +Build C / C++ Libs +++++++++++++++++++ + +Build C / C++ Libs Ubuntu +######################### + +For Build C Data Interface lib & Build C++ Libs. Consider: Next step requires: Docker & Docker Compose. + +.. code-block:: + + $ cd arrow + $ source dev/conbench_envs/hooks.sh install_archery + Successfully installed archery + $ archery docker run java-jni-manylinux-2014 + $ ls -latr java-dist/ + |__ libarrow_cdata_jni.so + |__ libarrow_dataset_jni.so + |__ libarrow_orc_jni.so + |__ libgandiva_jni.so + +Build C / C++ Libs MacOS +######################## + +For Build C Data Interface lib: + +.. code-block:: + + $ cd arrow + $ brew bundle --file=cpp/Brewfile + Homebrew Bundle complete! 25 Brewfile dependencies now installed. + $ export GITHUB_WORKSPACE=<absolute path to your arrow project> + $ export JAVA_HOME=<absolute path to your java home> + $ mkdir -p java-dist java-native-c + $ cd java-native-c + $ cmake \ + -DCMAKE_BUILD_TYPE=Release \ + -DCMAKE_INSTALL_LIBDIR=lib \ + -DCMAKE_INSTALL_PREFIX=$GITHUB_WORKSPACE/java-dist \ + $GITHUB_WORKSPACE/java/c + $ cmake --build . --target install + $ ls -latr $GITHUB_WORKSPACE/java-dist/lib + |__ libarrow_cdata_jni.dylib + +For Build C++ Interface libs: + +.. code-block:: + + $ cd arrow + $ brew bundle --file=cpp/Brewfile + Homebrew Bundle complete! 25 Brewfile dependencies now installed. + $ export GITHUB_WORKSPACE=<absolute path to your arrow project> + $ export JAVA_HOME=<absolute path to your java home> + $ mkdir -p java-dist java-native-cpp + $ cd java-native-cpp + $ cmake \ + -DARROW_BOOST_USE_SHARED=OFF \ + -DARROW_BROTLI_USE_SHARED=OFF \ + -DARROW_BZ2_USE_SHARED=OFF \ + -DARROW_GFLAGS_USE_SHARED=OFF \ + -DARROW_GRPC_USE_SHARED=OFF \ + -DARROW_LZ4_USE_SHARED=OFF \ + -DARROW_OPENSSL_USE_SHARED=OFF \ + -DARROW_PROTOBUF_USE_SHARED=OFF \ + -DARROW_SNAPPY_USE_SHARED=OFF \ + -DARROW_THRIFT_USE_SHARED=OFF \ + -DARROW_UTF8PROC_USE_SHARED=OFF \ + -DARROW_ZSTD_USE_SHARED=OFF \ + -DARROW_JNI=ON \ + -DARROW_PARQUET=ON \ + -DARROW_FILESYSTEM=ON \ + -DARROW_DATASET=ON \ + -DARROW_GANDIVA_JAVA=ON \ + -DARROW_GANDIVA_STATIC_LIBSTDCPP=ON \ + -DARROW_GANDIVA=ON \ + -DARROW_ORC=ON \ + -DARROW_PLASMA_JAVA_CLIENT=ON \ + -DARROW_PLASMA=ON \ + -DCMAKE_BUILD_TYPE=Release \ + -DCMAKE_INSTALL_LIBDIR=lib \ + -DCMAKE_INSTALL_PREFIX=$GITHUB_WORKSPACE/java-dist \ + -DCMAKE_UNITY_BUILD=ON \ + -Dre2_SOURCE=BUNDLED \ + -DBoost_SOURCE=BUNDLED \ + -Dutf8proc_SOURCE=BUNDLED \ + -DSnappy_SOURCE=BUNDLED \ + -DORC_SOURCE=BUNDLED \ + -DZLIB_SOURCE=BUNDLED \ + $GITHUB_WORKSPACE/cpp + $ cmake --build . --target install + $ ls -latr $GITHUB_WORKSPACE/java-dist/lib + |__ libarrow_dataset_jni.dylib + |__ libarrow_orc_jni.dylib + |__ libgandiva_jni.dylib + +Build Java Modules That Consume C / C++ Libs +++++++++++++++++++++++++++++++++++++++++++++ + +Build Java C Data Interface +########################### + +Java arrow-c-data maven profile: + +.. code-block:: + + $ cd arrow/java + $ mvn -Darrow.c.jni.dist.dir=$GITHUB_WORKSPACE/java-dist/lib -Parrow-c-data clean install + [INFO] Reactor Summary for Apache Arrow Java Root POM 7.0.0: + [INFO] + [INFO] Apache Arrow Java Root POM ......................... SUCCESS [ 4.385 s] + [INFO] Arrow Format ....................................... SUCCESS [ 4.832 s] + [INFO] Arrow Memory ....................................... SUCCESS [ 1.862 s] + [INFO] Arrow Memory - Core ................................ SUCCESS [ 12.162 s] + [INFO] Arrow Memory - Unsafe .............................. SUCCESS [ 6.768 s] + [INFO] Arrow Memory - Netty ............................... SUCCESS [ 6.897 s] + [INFO] Arrow Vectors ...................................... SUCCESS [01:43 min] + [INFO] Arrow Compression .................................. SUCCESS [ 6.510 s] + [INFO] Arrow Tools ........................................ SUCCESS [ 13.280 s] + [INFO] Arrow JDBC Adapter ................................. SUCCESS [ 13.726 s] + [INFO] Arrow Plasma Client ................................ SUCCESS [ 5.237 s] + [INFO] Arrow Flight ....................................... SUCCESS [ 0.809 s] + [INFO] Arrow Flight Core .................................. SUCCESS [ 47.461 s] + [INFO] Arrow Flight GRPC .................................. SUCCESS [ 6.864 s] + [INFO] Arrow Flight SQL ................................... SUCCESS [ 20.334 s] + [INFO] Arrow Flight Integration Tests ..................... SUCCESS [ 7.542 s] + [INFO] Arrow AVRO Adapter ................................. SUCCESS [ 13.826 s] + [INFO] Arrow Algorithms ................................... SUCCESS [ 32.442 s] + [INFO] Arrow Performance Benchmarks ....................... SUCCESS [ 5.418 s] + [INFO] Arrow Java C Data Interface ........................ SUCCESS [ 7.907 s] + [INFO] ------------------------------------------------------------------------ + [INFO] BUILD SUCCESS + [INFO] ------------------------------------------------------------------------ + +Build Java C++ Interface +######################## + +Java arrow-jni maven profile (build ORC / Gandiva / Dataset): + +.. code-block:: + + $ cd arrow/java + $ mvn -Darrow.cpp.build.dir=$GITHUB_WORKSPACE/java-dist/lib -Parrow-jni clean install + [INFO] Reactor Summary for Apache Arrow Java Root POM 7.0.0: + [INFO] + [INFO] Apache Arrow Java Root POM ......................... SUCCESS [ 7.342 s] + [INFO] Arrow Format ....................................... SUCCESS [ 2.417 s] + [INFO] Arrow Memory ....................................... SUCCESS [ 1.967 s] + [INFO] Arrow Memory - Core ................................ SUCCESS [ 4.714 s] + [INFO] Arrow Memory - Unsafe .............................. SUCCESS [ 3.157 s] + [INFO] Arrow Memory - Netty ............................... SUCCESS [ 3.334 s] + [INFO] Arrow Vectors ...................................... SUCCESS [ 21.791 s] + [INFO] Arrow Compression .................................. SUCCESS [ 3.854 s] + [INFO] Arrow Tools ........................................ SUCCESS [ 8.359 s] + [INFO] Arrow JDBC Adapter ................................. SUCCESS [ 8.847 s] + [INFO] Arrow Plasma Client ................................ SUCCESS [ 2.459 s] + [INFO] Arrow Flight ....................................... SUCCESS [ 2.357 s] + [INFO] Arrow Flight Core .................................. SUCCESS [ 38.837 s] + [INFO] Arrow Flight GRPC .................................. SUCCESS [ 5.955 s] + [INFO] Arrow Flight SQL ................................... SUCCESS [ 17.390 s] + [INFO] Arrow Flight Integration Tests ..................... SUCCESS [ 6.148 s] + [INFO] Arrow AVRO Adapter ................................. SUCCESS [ 9.635 s] + [INFO] Arrow Algorithms ................................... SUCCESS [ 26.949 s] + [INFO] Arrow Performance Benchmarks ....................... SUCCESS [ 4.211 s] + [INFO] Arrow Orc Adapter .................................. SUCCESS [ 6.522 s] + [INFO] Arrow Gandiva ...................................... SUCCESS [01:20 min] + [INFO] Arrow Java Dataset ................................. SUCCESS [ 12.949 s] + [INFO] ------------------------------------------------------------------------ + [INFO] BUILD SUCCESS + [INFO] ------------------------------------------------------------------------ + +IDE Configuration +================= + +Go to open java project and select java folder. + +This is the initial view for java project loaded with default profiles: + +.. image:: img/java_welcome.png + +Let's create our maven configuration: java-dataset + +.. image:: img/java_mvn_configuration.png + +Let's define our JRE runner: + +.. image:: img/java_jre_runner.png + +Let's configure additional environment properties: + +.. image:: img/java_jre_env_properties.png + +Let's run our java-dataset maven configuration: + +.. image:: img/java_run_mvn_configuration.png + +.. image:: img/java_run_mvn_configuration_result.png + +Common Errors +============= + +1.- Errors related to: Review comment: ```suggestion 1. If the build cannot find dependencies, with errors like these: ``` ########## File path: docs/source/developers/java/development.rst ########## @@ -0,0 +1,170 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. highlight:: console + +====================== +Development Guidelines +====================== + +.. contents:: + +Unit Testing +============ + +.. code-block:: + + $ cd arrow/java + $ mvn \ + -Darrow.cpp.build.dir=$GITHUB_WORKSPACE/java-dist/lib -Parrow-jni \ + -Darrow.c.jni.dist.dir=$GITHUB_WORKSPACE/java-dist/lib -Parrow-c-data \ Review comment: Don't use CI-specific env vars. ########## File path: docs/source/developers/java/building.rst ########## @@ -0,0 +1,308 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. highlight:: console + +.. _building-arrow-cpp: + +=================== +Building Arrow Java +=================== + +.. contents:: + +System setup +============ + +Arrow Java use Maven as a project management. + +Building requires: + +* JRE 8, 9, 10, 11 +* Maven 3+ + +Building +======== + +All the instructions below assume that you have cloned the Arrow git +repository: + +.. code-block:: + + $ git clone https://github.com/apache/arrow.git + $ cd arrow + $ git submodule update --init --recursive + +Basic Installation +------------------ + +You can also build java project for default modules configured. +Go to parent project and execute: + +.. code-block:: + + $ cd arrow/java + $ export JAVA_HOME=<absolute path to your java home> + $ java --version + $ mvn clean install + [INFO] Reactor Summary for Apache Arrow Java Root POM 8.0.0-SNAPSHOT: + [INFO] + [INFO] Apache Arrow Java Root POM ......................... SUCCESS [ 4.922 s] + [INFO] Arrow Format ....................................... SUCCESS [ 4.395 s] + [INFO] Arrow Memory ....................................... SUCCESS [ 0.938 s] + [INFO] Arrow Memory - Core ................................ SUCCESS [ 11.491 s] + [INFO] Arrow Memory - Unsafe .............................. SUCCESS [ 5.449 s] + [INFO] Arrow Memory - Netty ............................... SUCCESS [ 8.754 s] + [INFO] Arrow Vectors ...................................... SUCCESS [01:37 min] + [INFO] Arrow Compression .................................. SUCCESS [ 5.907 s] + [INFO] Arrow Tools ........................................ SUCCESS [ 11.015 s] + [INFO] Arrow JDBC Adapter ................................. SUCCESS [ 13.751 s] + [INFO] Arrow Plasma Client ................................ SUCCESS [ 3.934 s] + [INFO] Arrow Flight ....................................... SUCCESS [ 0.649 s] + [INFO] Arrow Flight Core .................................. SUCCESS [ 38.331 s] + [INFO] Arrow Flight GRPC .................................. SUCCESS [ 5.894 s] + [INFO] Arrow Flight SQL ................................... SUCCESS [ 21.862 s] + [INFO] Arrow Flight Integration Tests ..................... SUCCESS [ 7.638 s] + [INFO] Arrow AVRO Adapter ................................. SUCCESS [ 14.447 s] + [INFO] Arrow Algorithms ................................... SUCCESS [ 35.105 s] + [INFO] Arrow Performance Benchmarks ....................... SUCCESS [ 5.323 s] + [INFO] ------------------------------------------------------------------------ + [INFO] BUILD SUCCESS + [INFO] ------------------------------------------------------------------------\ + +Advanced Installation +--------------------- + +Build C / C++ Libs +++++++++++++++++++ + +Build C / C++ Libs Ubuntu +######################### + +For Build C Data Interface lib & Build C++ Libs. Consider: Next step requires: Docker & Docker Compose. + +.. code-block:: + + $ cd arrow + $ source dev/conbench_envs/hooks.sh install_archery + Successfully installed archery + $ archery docker run java-jni-manylinux-2014 + $ ls -latr java-dist/ + |__ libarrow_cdata_jni.so + |__ libarrow_dataset_jni.so + |__ libarrow_orc_jni.so + |__ libgandiva_jni.so + +Build C / C++ Libs MacOS +######################## + +For Build C Data Interface lib: + +.. code-block:: + + $ cd arrow + $ brew bundle --file=cpp/Brewfile + Homebrew Bundle complete! 25 Brewfile dependencies now installed. + $ export GITHUB_WORKSPACE=<absolute path to your arrow project> + $ export JAVA_HOME=<absolute path to your java home> + $ mkdir -p java-dist java-native-c + $ cd java-native-c + $ cmake \ + -DCMAKE_BUILD_TYPE=Release \ + -DCMAKE_INSTALL_LIBDIR=lib \ + -DCMAKE_INSTALL_PREFIX=$GITHUB_WORKSPACE/java-dist \ + $GITHUB_WORKSPACE/java/c + $ cmake --build . --target install + $ ls -latr $GITHUB_WORKSPACE/java-dist/lib + |__ libarrow_cdata_jni.dylib + +For Build C++ Interface libs: + +.. code-block:: + + $ cd arrow + $ brew bundle --file=cpp/Brewfile + Homebrew Bundle complete! 25 Brewfile dependencies now installed. + $ export GITHUB_WORKSPACE=<absolute path to your arrow project> + $ export JAVA_HOME=<absolute path to your java home> + $ mkdir -p java-dist java-native-cpp + $ cd java-native-cpp + $ cmake \ + -DARROW_BOOST_USE_SHARED=OFF \ + -DARROW_BROTLI_USE_SHARED=OFF \ + -DARROW_BZ2_USE_SHARED=OFF \ + -DARROW_GFLAGS_USE_SHARED=OFF \ + -DARROW_GRPC_USE_SHARED=OFF \ + -DARROW_LZ4_USE_SHARED=OFF \ + -DARROW_OPENSSL_USE_SHARED=OFF \ + -DARROW_PROTOBUF_USE_SHARED=OFF \ + -DARROW_SNAPPY_USE_SHARED=OFF \ + -DARROW_THRIFT_USE_SHARED=OFF \ + -DARROW_UTF8PROC_USE_SHARED=OFF \ + -DARROW_ZSTD_USE_SHARED=OFF \ + -DARROW_JNI=ON \ + -DARROW_PARQUET=ON \ + -DARROW_FILESYSTEM=ON \ + -DARROW_DATASET=ON \ + -DARROW_GANDIVA_JAVA=ON \ + -DARROW_GANDIVA_STATIC_LIBSTDCPP=ON \ + -DARROW_GANDIVA=ON \ + -DARROW_ORC=ON \ + -DARROW_PLASMA_JAVA_CLIENT=ON \ + -DARROW_PLASMA=ON \ + -DCMAKE_BUILD_TYPE=Release \ + -DCMAKE_INSTALL_LIBDIR=lib \ + -DCMAKE_INSTALL_PREFIX=$GITHUB_WORKSPACE/java-dist \ + -DCMAKE_UNITY_BUILD=ON \ + -Dre2_SOURCE=BUNDLED \ + -DBoost_SOURCE=BUNDLED \ + -Dutf8proc_SOURCE=BUNDLED \ + -DSnappy_SOURCE=BUNDLED \ + -DORC_SOURCE=BUNDLED \ + -DZLIB_SOURCE=BUNDLED \ + $GITHUB_WORKSPACE/cpp + $ cmake --build . --target install + $ ls -latr $GITHUB_WORKSPACE/java-dist/lib + |__ libarrow_dataset_jni.dylib + |__ libarrow_orc_jni.dylib + |__ libgandiva_jni.dylib + +Build Java Modules That Consume C / C++ Libs +++++++++++++++++++++++++++++++++++++++++++++ + +Build Java C Data Interface +########################### + +Java arrow-c-data maven profile: + +.. code-block:: + + $ cd arrow/java + $ mvn -Darrow.c.jni.dist.dir=$GITHUB_WORKSPACE/java-dist/lib -Parrow-c-data clean install + [INFO] Reactor Summary for Apache Arrow Java Root POM 7.0.0: + [INFO] + [INFO] Apache Arrow Java Root POM ......................... SUCCESS [ 4.385 s] + [INFO] Arrow Format ....................................... SUCCESS [ 4.832 s] + [INFO] Arrow Memory ....................................... SUCCESS [ 1.862 s] + [INFO] Arrow Memory - Core ................................ SUCCESS [ 12.162 s] + [INFO] Arrow Memory - Unsafe .............................. SUCCESS [ 6.768 s] + [INFO] Arrow Memory - Netty ............................... SUCCESS [ 6.897 s] + [INFO] Arrow Vectors ...................................... SUCCESS [01:43 min] + [INFO] Arrow Compression .................................. SUCCESS [ 6.510 s] + [INFO] Arrow Tools ........................................ SUCCESS [ 13.280 s] + [INFO] Arrow JDBC Adapter ................................. SUCCESS [ 13.726 s] + [INFO] Arrow Plasma Client ................................ SUCCESS [ 5.237 s] + [INFO] Arrow Flight ....................................... SUCCESS [ 0.809 s] + [INFO] Arrow Flight Core .................................. SUCCESS [ 47.461 s] + [INFO] Arrow Flight GRPC .................................. SUCCESS [ 6.864 s] + [INFO] Arrow Flight SQL ................................... SUCCESS [ 20.334 s] + [INFO] Arrow Flight Integration Tests ..................... SUCCESS [ 7.542 s] + [INFO] Arrow AVRO Adapter ................................. SUCCESS [ 13.826 s] + [INFO] Arrow Algorithms ................................... SUCCESS [ 32.442 s] + [INFO] Arrow Performance Benchmarks ....................... SUCCESS [ 5.418 s] + [INFO] Arrow Java C Data Interface ........................ SUCCESS [ 7.907 s] + [INFO] ------------------------------------------------------------------------ + [INFO] BUILD SUCCESS + [INFO] ------------------------------------------------------------------------ + +Build Java C++ Interface +######################## + +Java arrow-jni maven profile (build ORC / Gandiva / Dataset): + +.. code-block:: + + $ cd arrow/java + $ mvn -Darrow.cpp.build.dir=$GITHUB_WORKSPACE/java-dist/lib -Parrow-jni clean install + [INFO] Reactor Summary for Apache Arrow Java Root POM 7.0.0: + [INFO] + [INFO] Apache Arrow Java Root POM ......................... SUCCESS [ 7.342 s] + [INFO] Arrow Format ....................................... SUCCESS [ 2.417 s] + [INFO] Arrow Memory ....................................... SUCCESS [ 1.967 s] + [INFO] Arrow Memory - Core ................................ SUCCESS [ 4.714 s] + [INFO] Arrow Memory - Unsafe .............................. SUCCESS [ 3.157 s] + [INFO] Arrow Memory - Netty ............................... SUCCESS [ 3.334 s] + [INFO] Arrow Vectors ...................................... SUCCESS [ 21.791 s] + [INFO] Arrow Compression .................................. SUCCESS [ 3.854 s] + [INFO] Arrow Tools ........................................ SUCCESS [ 8.359 s] + [INFO] Arrow JDBC Adapter ................................. SUCCESS [ 8.847 s] + [INFO] Arrow Plasma Client ................................ SUCCESS [ 2.459 s] + [INFO] Arrow Flight ....................................... SUCCESS [ 2.357 s] + [INFO] Arrow Flight Core .................................. SUCCESS [ 38.837 s] + [INFO] Arrow Flight GRPC .................................. SUCCESS [ 5.955 s] + [INFO] Arrow Flight SQL ................................... SUCCESS [ 17.390 s] + [INFO] Arrow Flight Integration Tests ..................... SUCCESS [ 6.148 s] + [INFO] Arrow AVRO Adapter ................................. SUCCESS [ 9.635 s] + [INFO] Arrow Algorithms ................................... SUCCESS [ 26.949 s] + [INFO] Arrow Performance Benchmarks ....................... SUCCESS [ 4.211 s] + [INFO] Arrow Orc Adapter .................................. SUCCESS [ 6.522 s] + [INFO] Arrow Gandiva ...................................... SUCCESS [01:20 min] + [INFO] Arrow Java Dataset ................................. SUCCESS [ 12.949 s] + [INFO] ------------------------------------------------------------------------ + [INFO] BUILD SUCCESS + [INFO] ------------------------------------------------------------------------ + +IDE Configuration +================= + +Go to open java project and select java folder. Review comment: Add a section so it's clear this is for IntelliJ ########## File path: docs/source/developers/java/building.rst ########## @@ -0,0 +1,308 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. highlight:: console + +.. _building-arrow-cpp: + +=================== +Building Arrow Java +=================== + +.. contents:: + +System setup +============ + +Arrow Java use Maven as a project management. + +Building requires: + +* JRE 8, 9, 10, 11 +* Maven 3+ + +Building +======== + +All the instructions below assume that you have cloned the Arrow git +repository: + +.. code-block:: + + $ git clone https://github.com/apache/arrow.git + $ cd arrow + $ git submodule update --init --recursive + +Basic Installation +------------------ + +You can also build java project for default modules configured. +Go to parent project and execute: + +.. code-block:: + + $ cd arrow/java + $ export JAVA_HOME=<absolute path to your java home> + $ java --version + $ mvn clean install + [INFO] Reactor Summary for Apache Arrow Java Root POM 8.0.0-SNAPSHOT: + [INFO] + [INFO] Apache Arrow Java Root POM ......................... SUCCESS [ 4.922 s] + [INFO] Arrow Format ....................................... SUCCESS [ 4.395 s] + [INFO] Arrow Memory ....................................... SUCCESS [ 0.938 s] + [INFO] Arrow Memory - Core ................................ SUCCESS [ 11.491 s] + [INFO] Arrow Memory - Unsafe .............................. SUCCESS [ 5.449 s] + [INFO] Arrow Memory - Netty ............................... SUCCESS [ 8.754 s] + [INFO] Arrow Vectors ...................................... SUCCESS [01:37 min] + [INFO] Arrow Compression .................................. SUCCESS [ 5.907 s] + [INFO] Arrow Tools ........................................ SUCCESS [ 11.015 s] + [INFO] Arrow JDBC Adapter ................................. SUCCESS [ 13.751 s] + [INFO] Arrow Plasma Client ................................ SUCCESS [ 3.934 s] + [INFO] Arrow Flight ....................................... SUCCESS [ 0.649 s] + [INFO] Arrow Flight Core .................................. SUCCESS [ 38.331 s] + [INFO] Arrow Flight GRPC .................................. SUCCESS [ 5.894 s] + [INFO] Arrow Flight SQL ................................... SUCCESS [ 21.862 s] + [INFO] Arrow Flight Integration Tests ..................... SUCCESS [ 7.638 s] + [INFO] Arrow AVRO Adapter ................................. SUCCESS [ 14.447 s] + [INFO] Arrow Algorithms ................................... SUCCESS [ 35.105 s] + [INFO] Arrow Performance Benchmarks ....................... SUCCESS [ 5.323 s] + [INFO] ------------------------------------------------------------------------ + [INFO] BUILD SUCCESS + [INFO] ------------------------------------------------------------------------\ + +Advanced Installation +--------------------- + +Build C / C++ Libs +++++++++++++++++++ + +Build C / C++ Libs Ubuntu +######################### + +For Build C Data Interface lib & Build C++ Libs. Consider: Next step requires: Docker & Docker Compose. + +.. code-block:: + + $ cd arrow + $ source dev/conbench_envs/hooks.sh install_archery + Successfully installed archery + $ archery docker run java-jni-manylinux-2014 + $ ls -latr java-dist/ + |__ libarrow_cdata_jni.so + |__ libarrow_dataset_jni.so + |__ libarrow_orc_jni.so + |__ libgandiva_jni.so + +Build C / C++ Libs MacOS +######################## + +For Build C Data Interface lib: + +.. code-block:: + + $ cd arrow + $ brew bundle --file=cpp/Brewfile + Homebrew Bundle complete! 25 Brewfile dependencies now installed. + $ export GITHUB_WORKSPACE=<absolute path to your arrow project> + $ export JAVA_HOME=<absolute path to your java home> + $ mkdir -p java-dist java-native-c + $ cd java-native-c + $ cmake \ + -DCMAKE_BUILD_TYPE=Release \ + -DCMAKE_INSTALL_LIBDIR=lib \ + -DCMAKE_INSTALL_PREFIX=$GITHUB_WORKSPACE/java-dist \ + $GITHUB_WORKSPACE/java/c + $ cmake --build . --target install + $ ls -latr $GITHUB_WORKSPACE/java-dist/lib + |__ libarrow_cdata_jni.dylib + +For Build C++ Interface libs: + +.. code-block:: + + $ cd arrow + $ brew bundle --file=cpp/Brewfile + Homebrew Bundle complete! 25 Brewfile dependencies now installed. + $ export GITHUB_WORKSPACE=<absolute path to your arrow project> + $ export JAVA_HOME=<absolute path to your java home> + $ mkdir -p java-dist java-native-cpp + $ cd java-native-cpp + $ cmake \ + -DARROW_BOOST_USE_SHARED=OFF \ + -DARROW_BROTLI_USE_SHARED=OFF \ + -DARROW_BZ2_USE_SHARED=OFF \ + -DARROW_GFLAGS_USE_SHARED=OFF \ + -DARROW_GRPC_USE_SHARED=OFF \ + -DARROW_LZ4_USE_SHARED=OFF \ + -DARROW_OPENSSL_USE_SHARED=OFF \ + -DARROW_PROTOBUF_USE_SHARED=OFF \ + -DARROW_SNAPPY_USE_SHARED=OFF \ + -DARROW_THRIFT_USE_SHARED=OFF \ + -DARROW_UTF8PROC_USE_SHARED=OFF \ + -DARROW_ZSTD_USE_SHARED=OFF \ + -DARROW_JNI=ON \ + -DARROW_PARQUET=ON \ + -DARROW_FILESYSTEM=ON \ + -DARROW_DATASET=ON \ + -DARROW_GANDIVA_JAVA=ON \ + -DARROW_GANDIVA_STATIC_LIBSTDCPP=ON \ + -DARROW_GANDIVA=ON \ + -DARROW_ORC=ON \ + -DARROW_PLASMA_JAVA_CLIENT=ON \ + -DARROW_PLASMA=ON \ + -DCMAKE_BUILD_TYPE=Release \ + -DCMAKE_INSTALL_LIBDIR=lib \ + -DCMAKE_INSTALL_PREFIX=$GITHUB_WORKSPACE/java-dist \ + -DCMAKE_UNITY_BUILD=ON \ + -Dre2_SOURCE=BUNDLED \ + -DBoost_SOURCE=BUNDLED \ + -Dutf8proc_SOURCE=BUNDLED \ + -DSnappy_SOURCE=BUNDLED \ + -DORC_SOURCE=BUNDLED \ + -DZLIB_SOURCE=BUNDLED \ + $GITHUB_WORKSPACE/cpp + $ cmake --build . --target install + $ ls -latr $GITHUB_WORKSPACE/java-dist/lib + |__ libarrow_dataset_jni.dylib + |__ libarrow_orc_jni.dylib + |__ libgandiva_jni.dylib + +Build Java Modules That Consume C / C++ Libs +++++++++++++++++++++++++++++++++++++++++++++ + +Build Java C Data Interface +########################### + +Java arrow-c-data maven profile: + +.. code-block:: + + $ cd arrow/java + $ mvn -Darrow.c.jni.dist.dir=$GITHUB_WORKSPACE/java-dist/lib -Parrow-c-data clean install + [INFO] Reactor Summary for Apache Arrow Java Root POM 7.0.0: + [INFO] + [INFO] Apache Arrow Java Root POM ......................... SUCCESS [ 4.385 s] + [INFO] Arrow Format ....................................... SUCCESS [ 4.832 s] + [INFO] Arrow Memory ....................................... SUCCESS [ 1.862 s] + [INFO] Arrow Memory - Core ................................ SUCCESS [ 12.162 s] + [INFO] Arrow Memory - Unsafe .............................. SUCCESS [ 6.768 s] + [INFO] Arrow Memory - Netty ............................... SUCCESS [ 6.897 s] + [INFO] Arrow Vectors ...................................... SUCCESS [01:43 min] + [INFO] Arrow Compression .................................. SUCCESS [ 6.510 s] + [INFO] Arrow Tools ........................................ SUCCESS [ 13.280 s] + [INFO] Arrow JDBC Adapter ................................. SUCCESS [ 13.726 s] + [INFO] Arrow Plasma Client ................................ SUCCESS [ 5.237 s] + [INFO] Arrow Flight ....................................... SUCCESS [ 0.809 s] + [INFO] Arrow Flight Core .................................. SUCCESS [ 47.461 s] + [INFO] Arrow Flight GRPC .................................. SUCCESS [ 6.864 s] + [INFO] Arrow Flight SQL ................................... SUCCESS [ 20.334 s] + [INFO] Arrow Flight Integration Tests ..................... SUCCESS [ 7.542 s] + [INFO] Arrow AVRO Adapter ................................. SUCCESS [ 13.826 s] + [INFO] Arrow Algorithms ................................... SUCCESS [ 32.442 s] + [INFO] Arrow Performance Benchmarks ....................... SUCCESS [ 5.418 s] + [INFO] Arrow Java C Data Interface ........................ SUCCESS [ 7.907 s] + [INFO] ------------------------------------------------------------------------ + [INFO] BUILD SUCCESS + [INFO] ------------------------------------------------------------------------ + +Build Java C++ Interface +######################## + +Java arrow-jni maven profile (build ORC / Gandiva / Dataset): + +.. code-block:: + + $ cd arrow/java + $ mvn -Darrow.cpp.build.dir=$GITHUB_WORKSPACE/java-dist/lib -Parrow-jni clean install + [INFO] Reactor Summary for Apache Arrow Java Root POM 7.0.0: + [INFO] + [INFO] Apache Arrow Java Root POM ......................... SUCCESS [ 7.342 s] + [INFO] Arrow Format ....................................... SUCCESS [ 2.417 s] + [INFO] Arrow Memory ....................................... SUCCESS [ 1.967 s] + [INFO] Arrow Memory - Core ................................ SUCCESS [ 4.714 s] + [INFO] Arrow Memory - Unsafe .............................. SUCCESS [ 3.157 s] + [INFO] Arrow Memory - Netty ............................... SUCCESS [ 3.334 s] + [INFO] Arrow Vectors ...................................... SUCCESS [ 21.791 s] + [INFO] Arrow Compression .................................. SUCCESS [ 3.854 s] + [INFO] Arrow Tools ........................................ SUCCESS [ 8.359 s] + [INFO] Arrow JDBC Adapter ................................. SUCCESS [ 8.847 s] + [INFO] Arrow Plasma Client ................................ SUCCESS [ 2.459 s] + [INFO] Arrow Flight ....................................... SUCCESS [ 2.357 s] + [INFO] Arrow Flight Core .................................. SUCCESS [ 38.837 s] + [INFO] Arrow Flight GRPC .................................. SUCCESS [ 5.955 s] + [INFO] Arrow Flight SQL ................................... SUCCESS [ 17.390 s] + [INFO] Arrow Flight Integration Tests ..................... SUCCESS [ 6.148 s] + [INFO] Arrow AVRO Adapter ................................. SUCCESS [ 9.635 s] + [INFO] Arrow Algorithms ................................... SUCCESS [ 26.949 s] + [INFO] Arrow Performance Benchmarks ....................... SUCCESS [ 4.211 s] + [INFO] Arrow Orc Adapter .................................. SUCCESS [ 6.522 s] + [INFO] Arrow Gandiva ...................................... SUCCESS [01:20 min] + [INFO] Arrow Java Dataset ................................. SUCCESS [ 12.949 s] + [INFO] ------------------------------------------------------------------------ + [INFO] BUILD SUCCESS + [INFO] ------------------------------------------------------------------------ + +IDE Configuration +================= + +Go to open java project and select java folder. + +This is the initial view for java project loaded with default profiles: + +.. image:: img/java_welcome.png + +Let's create our maven configuration: java-dataset + +.. image:: img/java_mvn_configuration.png + +Let's define our JRE runner: + +.. image:: img/java_jre_runner.png + +Let's configure additional environment properties: + +.. image:: img/java_jre_env_properties.png + +Let's run our java-dataset maven configuration: + +.. image:: img/java_run_mvn_configuration.png + +.. image:: img/java_run_mvn_configuration_result.png + +Common Errors +============= + +1.- Errors related to: + - Could NOT find Boost (missing: Boost_INCLUDE_DIR system filesystem) + - Could NOT find Lz4 (missing: LZ4_LIB) + - Could NOT find zstd (missing: ZSTD_LIB) + + Consider to configure parameters that download software at the build time with: + +.. code-block:: Review comment: This block needs to be indented more ########## File path: docs/source/developers/java/development.rst ########## @@ -0,0 +1,170 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. highlight:: console + +====================== +Development Guidelines +====================== + +.. contents:: + +Unit Testing +============ + Review comment: ```suggestion Unit tests are run by Maven during the build. To speed up the build, you can skip them by passing -DskipTests. ``` ########## File path: docs/source/developers/java/building.rst ########## @@ -0,0 +1,308 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. highlight:: console + +.. _building-arrow-cpp: + +=================== +Building Arrow Java +=================== + +.. contents:: + +System setup +============ + +Arrow Java use Maven as a project management. + +Building requires: + +* JRE 8, 9, 10, 11 +* Maven 3+ + +Building +======== + +All the instructions below assume that you have cloned the Arrow git +repository: + +.. code-block:: + + $ git clone https://github.com/apache/arrow.git + $ cd arrow + $ git submodule update --init --recursive + +Basic Installation +------------------ + +You can also build java project for default modules configured. +Go to parent project and execute: + +.. code-block:: + + $ cd arrow/java + $ export JAVA_HOME=<absolute path to your java home> + $ java --version + $ mvn clean install + [INFO] Reactor Summary for Apache Arrow Java Root POM 8.0.0-SNAPSHOT: + [INFO] + [INFO] Apache Arrow Java Root POM ......................... SUCCESS [ 4.922 s] + [INFO] Arrow Format ....................................... SUCCESS [ 4.395 s] + [INFO] Arrow Memory ....................................... SUCCESS [ 0.938 s] + [INFO] Arrow Memory - Core ................................ SUCCESS [ 11.491 s] + [INFO] Arrow Memory - Unsafe .............................. SUCCESS [ 5.449 s] + [INFO] Arrow Memory - Netty ............................... SUCCESS [ 8.754 s] + [INFO] Arrow Vectors ...................................... SUCCESS [01:37 min] + [INFO] Arrow Compression .................................. SUCCESS [ 5.907 s] + [INFO] Arrow Tools ........................................ SUCCESS [ 11.015 s] + [INFO] Arrow JDBC Adapter ................................. SUCCESS [ 13.751 s] + [INFO] Arrow Plasma Client ................................ SUCCESS [ 3.934 s] + [INFO] Arrow Flight ....................................... SUCCESS [ 0.649 s] + [INFO] Arrow Flight Core .................................. SUCCESS [ 38.331 s] + [INFO] Arrow Flight GRPC .................................. SUCCESS [ 5.894 s] + [INFO] Arrow Flight SQL ................................... SUCCESS [ 21.862 s] + [INFO] Arrow Flight Integration Tests ..................... SUCCESS [ 7.638 s] + [INFO] Arrow AVRO Adapter ................................. SUCCESS [ 14.447 s] + [INFO] Arrow Algorithms ................................... SUCCESS [ 35.105 s] + [INFO] Arrow Performance Benchmarks ....................... SUCCESS [ 5.323 s] + [INFO] ------------------------------------------------------------------------ + [INFO] BUILD SUCCESS + [INFO] ------------------------------------------------------------------------\ + +Advanced Installation +--------------------- + +Build C / C++ Libs +++++++++++++++++++ + +Build C / C++ Libs Ubuntu +######################### + +For Build C Data Interface lib & Build C++ Libs. Consider: Next step requires: Docker & Docker Compose. + +.. code-block:: + + $ cd arrow + $ source dev/conbench_envs/hooks.sh install_archery + Successfully installed archery + $ archery docker run java-jni-manylinux-2014 + $ ls -latr java-dist/ + |__ libarrow_cdata_jni.so + |__ libarrow_dataset_jni.so + |__ libarrow_orc_jni.so + |__ libgandiva_jni.so + +Build C / C++ Libs MacOS +######################## + +For Build C Data Interface lib: + +.. code-block:: + + $ cd arrow + $ brew bundle --file=cpp/Brewfile + Homebrew Bundle complete! 25 Brewfile dependencies now installed. + $ export GITHUB_WORKSPACE=<absolute path to your arrow project> + $ export JAVA_HOME=<absolute path to your java home> + $ mkdir -p java-dist java-native-c + $ cd java-native-c + $ cmake \ + -DCMAKE_BUILD_TYPE=Release \ + -DCMAKE_INSTALL_LIBDIR=lib \ + -DCMAKE_INSTALL_PREFIX=$GITHUB_WORKSPACE/java-dist \ + $GITHUB_WORKSPACE/java/c + $ cmake --build . --target install + $ ls -latr $GITHUB_WORKSPACE/java-dist/lib + |__ libarrow_cdata_jni.dylib + +For Build C++ Interface libs: + +.. code-block:: + + $ cd arrow + $ brew bundle --file=cpp/Brewfile + Homebrew Bundle complete! 25 Brewfile dependencies now installed. + $ export GITHUB_WORKSPACE=<absolute path to your arrow project> + $ export JAVA_HOME=<absolute path to your java home> + $ mkdir -p java-dist java-native-cpp + $ cd java-native-cpp + $ cmake \ + -DARROW_BOOST_USE_SHARED=OFF \ + -DARROW_BROTLI_USE_SHARED=OFF \ + -DARROW_BZ2_USE_SHARED=OFF \ + -DARROW_GFLAGS_USE_SHARED=OFF \ + -DARROW_GRPC_USE_SHARED=OFF \ + -DARROW_LZ4_USE_SHARED=OFF \ + -DARROW_OPENSSL_USE_SHARED=OFF \ + -DARROW_PROTOBUF_USE_SHARED=OFF \ + -DARROW_SNAPPY_USE_SHARED=OFF \ + -DARROW_THRIFT_USE_SHARED=OFF \ + -DARROW_UTF8PROC_USE_SHARED=OFF \ + -DARROW_ZSTD_USE_SHARED=OFF \ + -DARROW_JNI=ON \ + -DARROW_PARQUET=ON \ + -DARROW_FILESYSTEM=ON \ + -DARROW_DATASET=ON \ + -DARROW_GANDIVA_JAVA=ON \ + -DARROW_GANDIVA_STATIC_LIBSTDCPP=ON \ + -DARROW_GANDIVA=ON \ + -DARROW_ORC=ON \ + -DARROW_PLASMA_JAVA_CLIENT=ON \ + -DARROW_PLASMA=ON \ + -DCMAKE_BUILD_TYPE=Release \ + -DCMAKE_INSTALL_LIBDIR=lib \ + -DCMAKE_INSTALL_PREFIX=$GITHUB_WORKSPACE/java-dist \ + -DCMAKE_UNITY_BUILD=ON \ + -Dre2_SOURCE=BUNDLED \ + -DBoost_SOURCE=BUNDLED \ + -Dutf8proc_SOURCE=BUNDLED \ + -DSnappy_SOURCE=BUNDLED \ + -DORC_SOURCE=BUNDLED \ + -DZLIB_SOURCE=BUNDLED \ + $GITHUB_WORKSPACE/cpp + $ cmake --build . --target install + $ ls -latr $GITHUB_WORKSPACE/java-dist/lib + |__ libarrow_dataset_jni.dylib + |__ libarrow_orc_jni.dylib + |__ libgandiva_jni.dylib + +Build Java Modules That Consume C / C++ Libs +++++++++++++++++++++++++++++++++++++++++++++ + +Build Java C Data Interface +########################### + +Java arrow-c-data maven profile: + +.. code-block:: + + $ cd arrow/java + $ mvn -Darrow.c.jni.dist.dir=$GITHUB_WORKSPACE/java-dist/lib -Parrow-c-data clean install + [INFO] Reactor Summary for Apache Arrow Java Root POM 7.0.0: + [INFO] + [INFO] Apache Arrow Java Root POM ......................... SUCCESS [ 4.385 s] + [INFO] Arrow Format ....................................... SUCCESS [ 4.832 s] + [INFO] Arrow Memory ....................................... SUCCESS [ 1.862 s] + [INFO] Arrow Memory - Core ................................ SUCCESS [ 12.162 s] + [INFO] Arrow Memory - Unsafe .............................. SUCCESS [ 6.768 s] + [INFO] Arrow Memory - Netty ............................... SUCCESS [ 6.897 s] + [INFO] Arrow Vectors ...................................... SUCCESS [01:43 min] + [INFO] Arrow Compression .................................. SUCCESS [ 6.510 s] + [INFO] Arrow Tools ........................................ SUCCESS [ 13.280 s] + [INFO] Arrow JDBC Adapter ................................. SUCCESS [ 13.726 s] + [INFO] Arrow Plasma Client ................................ SUCCESS [ 5.237 s] + [INFO] Arrow Flight ....................................... SUCCESS [ 0.809 s] + [INFO] Arrow Flight Core .................................. SUCCESS [ 47.461 s] + [INFO] Arrow Flight GRPC .................................. SUCCESS [ 6.864 s] + [INFO] Arrow Flight SQL ................................... SUCCESS [ 20.334 s] + [INFO] Arrow Flight Integration Tests ..................... SUCCESS [ 7.542 s] + [INFO] Arrow AVRO Adapter ................................. SUCCESS [ 13.826 s] + [INFO] Arrow Algorithms ................................... SUCCESS [ 32.442 s] + [INFO] Arrow Performance Benchmarks ....................... SUCCESS [ 5.418 s] + [INFO] Arrow Java C Data Interface ........................ SUCCESS [ 7.907 s] + [INFO] ------------------------------------------------------------------------ + [INFO] BUILD SUCCESS + [INFO] ------------------------------------------------------------------------ + +Build Java C++ Interface +######################## + +Java arrow-jni maven profile (build ORC / Gandiva / Dataset): + +.. code-block:: + + $ cd arrow/java + $ mvn -Darrow.cpp.build.dir=$GITHUB_WORKSPACE/java-dist/lib -Parrow-jni clean install + [INFO] Reactor Summary for Apache Arrow Java Root POM 7.0.0: + [INFO] + [INFO] Apache Arrow Java Root POM ......................... SUCCESS [ 7.342 s] + [INFO] Arrow Format ....................................... SUCCESS [ 2.417 s] + [INFO] Arrow Memory ....................................... SUCCESS [ 1.967 s] + [INFO] Arrow Memory - Core ................................ SUCCESS [ 4.714 s] + [INFO] Arrow Memory - Unsafe .............................. SUCCESS [ 3.157 s] + [INFO] Arrow Memory - Netty ............................... SUCCESS [ 3.334 s] + [INFO] Arrow Vectors ...................................... SUCCESS [ 21.791 s] + [INFO] Arrow Compression .................................. SUCCESS [ 3.854 s] + [INFO] Arrow Tools ........................................ SUCCESS [ 8.359 s] + [INFO] Arrow JDBC Adapter ................................. SUCCESS [ 8.847 s] + [INFO] Arrow Plasma Client ................................ SUCCESS [ 2.459 s] + [INFO] Arrow Flight ....................................... SUCCESS [ 2.357 s] + [INFO] Arrow Flight Core .................................. SUCCESS [ 38.837 s] + [INFO] Arrow Flight GRPC .................................. SUCCESS [ 5.955 s] + [INFO] Arrow Flight SQL ................................... SUCCESS [ 17.390 s] + [INFO] Arrow Flight Integration Tests ..................... SUCCESS [ 6.148 s] + [INFO] Arrow AVRO Adapter ................................. SUCCESS [ 9.635 s] + [INFO] Arrow Algorithms ................................... SUCCESS [ 26.949 s] + [INFO] Arrow Performance Benchmarks ....................... SUCCESS [ 4.211 s] + [INFO] Arrow Orc Adapter .................................. SUCCESS [ 6.522 s] + [INFO] Arrow Gandiva ...................................... SUCCESS [01:20 min] + [INFO] Arrow Java Dataset ................................. SUCCESS [ 12.949 s] + [INFO] ------------------------------------------------------------------------ + [INFO] BUILD SUCCESS + [INFO] ------------------------------------------------------------------------ + +IDE Configuration +================= + +Go to open java project and select java folder. + +This is the initial view for java project loaded with default profiles: + +.. image:: img/java_welcome.png + +Let's create our maven configuration: java-dataset + +.. image:: img/java_mvn_configuration.png + +Let's define our JRE runner: + +.. image:: img/java_jre_runner.png + +Let's configure additional environment properties: + +.. image:: img/java_jre_env_properties.png + +Let's run our java-dataset maven configuration: + +.. image:: img/java_run_mvn_configuration.png + +.. image:: img/java_run_mvn_configuration_result.png + +Common Errors +============= + +1.- Errors related to: + - Could NOT find Boost (missing: Boost_INCLUDE_DIR system filesystem) + - Could NOT find Lz4 (missing: LZ4_LIB) + - Could NOT find zstd (missing: ZSTD_LIB) + + Consider to configure parameters that download software at the build time with: + +.. code-block:: + + -Dre2_SOURCE=BUNDLED \ + -DBoost_SOURCE=BUNDLED \ + -Dutf8proc_SOURCE=BUNDLED \ + -DSnappy_SOURCE=BUNDLED \ + -DORC_SOURCE=BUNDLED \ + -DZLIB_SOURCE=BUNDLED \ + +2.- Errors related to: + - Caused by: java.lang.NoSuchFieldException: reservedMemory. + + We could move from classBits.getDeclaredField("reservedMemory") to classBits.getDeclaredField("RESERVED_MEMORY") Review comment: Remove this section and just file a JIRA ########## File path: docs/source/developers/java/development.rst ########## @@ -0,0 +1,170 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. highlight:: console + +====================== +Development Guidelines +====================== + +.. contents:: + +Unit Testing +============ + +.. code-block:: + + $ cd arrow/java + $ mvn \ + -Darrow.cpp.build.dir=$GITHUB_WORKSPACE/java-dist/lib -Parrow-jni \ + -Darrow.c.jni.dist.dir=$GITHUB_WORKSPACE/java-dist/lib -Parrow-c-data \ + clean install + [INFO] Reactor Summary for Apache Arrow Java Root POM 7.0.0: + [INFO] + [INFO] Apache Arrow Java Root POM ......................... SUCCESS [ 7.308 s] + [INFO] Arrow Format ....................................... SUCCESS [ 4.899 s] + [INFO] Arrow Memory ....................................... SUCCESS [ 2.065 s] + [INFO] Arrow Memory - Core ................................ SUCCESS [ 11.417 s] + [INFO] Arrow Memory - Unsafe .............................. SUCCESS [ 8.132 s] + [INFO] Arrow Memory - Netty ............................... SUCCESS [ 10.211 s] + [INFO] Arrow Vectors ...................................... SUCCESS [01:48 min] + [INFO] Arrow Compression .................................. SUCCESS [ 7.282 s] + [INFO] Arrow Tools ........................................ SUCCESS [ 12.612 s] + [INFO] Arrow JDBC Adapter ................................. SUCCESS [ 15.558 s] + [INFO] Arrow Plasma Client ................................ SUCCESS [ 6.303 s] + [INFO] Arrow Flight ....................................... SUCCESS [ 2.326 s] + [INFO] Arrow Flight Core .................................. SUCCESS [ 50.632 s] + [INFO] Arrow Flight GRPC .................................. SUCCESS [ 9.932 s] + [INFO] Arrow Flight SQL ................................... SUCCESS [ 22.559 s] + [INFO] Arrow Flight Integration Tests ..................... SUCCESS [ 9.649 s] + [INFO] Arrow AVRO Adapter ................................. SUCCESS [ 17.145 s] + [INFO] Arrow Algorithms ................................... SUCCESS [ 44.571 s] + [INFO] Arrow Performance Benchmarks ....................... SUCCESS [ 9.202 s] + [INFO] Arrow Java C Data Interface ........................ SUCCESS [ 15.794 s] + [INFO] Arrow Orc Adapter .................................. SUCCESS [ 14.205 s] + [INFO] Arrow Gandiva ...................................... SUCCESS [02:06 min] + [INFO] Arrow Java Dataset ................................. SUCCESS [ 15.261 s] + [INFO] ------------------------------------------------------------------------ + [INFO] BUILD SUCCESS + [INFO] ------------------------------------------------------------------------ + +Performance Testing +=================== + +Java module that contains performance tests is: `performance` + +Let's configure our performance environment to run performance tests: Review comment: ```suggestion Let's configure our environment to run performance tests: ``` ########## File path: docs/source/developers/java/building.rst ########## @@ -0,0 +1,308 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. highlight:: console + +.. _building-arrow-cpp: + +=================== +Building Arrow Java +=================== + +.. contents:: + +System setup +============ + +Arrow Java use Maven as a project management. + +Building requires: + +* JRE 8, 9, 10, 11 +* Maven 3+ + +Building +======== + +All the instructions below assume that you have cloned the Arrow git +repository: + +.. code-block:: + + $ git clone https://github.com/apache/arrow.git + $ cd arrow + $ git submodule update --init --recursive + +Basic Installation +------------------ + +You can also build java project for default modules configured. +Go to parent project and execute: + +.. code-block:: + + $ cd arrow/java + $ export JAVA_HOME=<absolute path to your java home> + $ java --version + $ mvn clean install + [INFO] Reactor Summary for Apache Arrow Java Root POM 8.0.0-SNAPSHOT: + [INFO] + [INFO] Apache Arrow Java Root POM ......................... SUCCESS [ 4.922 s] + [INFO] Arrow Format ....................................... SUCCESS [ 4.395 s] + [INFO] Arrow Memory ....................................... SUCCESS [ 0.938 s] + [INFO] Arrow Memory - Core ................................ SUCCESS [ 11.491 s] + [INFO] Arrow Memory - Unsafe .............................. SUCCESS [ 5.449 s] + [INFO] Arrow Memory - Netty ............................... SUCCESS [ 8.754 s] + [INFO] Arrow Vectors ...................................... SUCCESS [01:37 min] + [INFO] Arrow Compression .................................. SUCCESS [ 5.907 s] + [INFO] Arrow Tools ........................................ SUCCESS [ 11.015 s] + [INFO] Arrow JDBC Adapter ................................. SUCCESS [ 13.751 s] + [INFO] Arrow Plasma Client ................................ SUCCESS [ 3.934 s] + [INFO] Arrow Flight ....................................... SUCCESS [ 0.649 s] + [INFO] Arrow Flight Core .................................. SUCCESS [ 38.331 s] + [INFO] Arrow Flight GRPC .................................. SUCCESS [ 5.894 s] + [INFO] Arrow Flight SQL ................................... SUCCESS [ 21.862 s] + [INFO] Arrow Flight Integration Tests ..................... SUCCESS [ 7.638 s] + [INFO] Arrow AVRO Adapter ................................. SUCCESS [ 14.447 s] + [INFO] Arrow Algorithms ................................... SUCCESS [ 35.105 s] + [INFO] Arrow Performance Benchmarks ....................... SUCCESS [ 5.323 s] + [INFO] ------------------------------------------------------------------------ + [INFO] BUILD SUCCESS + [INFO] ------------------------------------------------------------------------\ + +Advanced Installation +--------------------- + +Build C / C++ Libs +++++++++++++++++++ + +Build C / C++ Libs Ubuntu +######################### + +For Build C Data Interface lib & Build C++ Libs. Consider: Next step requires: Docker & Docker Compose. + +.. code-block:: + + $ cd arrow + $ source dev/conbench_envs/hooks.sh install_archery + Successfully installed archery + $ archery docker run java-jni-manylinux-2014 + $ ls -latr java-dist/ + |__ libarrow_cdata_jni.so + |__ libarrow_dataset_jni.so + |__ libarrow_orc_jni.so + |__ libgandiva_jni.so + +Build C / C++ Libs MacOS +######################## + +For Build C Data Interface lib: + +.. code-block:: + + $ cd arrow + $ brew bundle --file=cpp/Brewfile + Homebrew Bundle complete! 25 Brewfile dependencies now installed. + $ export GITHUB_WORKSPACE=<absolute path to your arrow project> + $ export JAVA_HOME=<absolute path to your java home> + $ mkdir -p java-dist java-native-c + $ cd java-native-c + $ cmake \ + -DCMAKE_BUILD_TYPE=Release \ + -DCMAKE_INSTALL_LIBDIR=lib \ + -DCMAKE_INSTALL_PREFIX=$GITHUB_WORKSPACE/java-dist \ + $GITHUB_WORKSPACE/java/c + $ cmake --build . --target install + $ ls -latr $GITHUB_WORKSPACE/java-dist/lib + |__ libarrow_cdata_jni.dylib + +For Build C++ Interface libs: + +.. code-block:: + + $ cd arrow + $ brew bundle --file=cpp/Brewfile + Homebrew Bundle complete! 25 Brewfile dependencies now installed. + $ export GITHUB_WORKSPACE=<absolute path to your arrow project> + $ export JAVA_HOME=<absolute path to your java home> + $ mkdir -p java-dist java-native-cpp + $ cd java-native-cpp + $ cmake \ + -DARROW_BOOST_USE_SHARED=OFF \ + -DARROW_BROTLI_USE_SHARED=OFF \ + -DARROW_BZ2_USE_SHARED=OFF \ + -DARROW_GFLAGS_USE_SHARED=OFF \ + -DARROW_GRPC_USE_SHARED=OFF \ + -DARROW_LZ4_USE_SHARED=OFF \ + -DARROW_OPENSSL_USE_SHARED=OFF \ + -DARROW_PROTOBUF_USE_SHARED=OFF \ + -DARROW_SNAPPY_USE_SHARED=OFF \ + -DARROW_THRIFT_USE_SHARED=OFF \ + -DARROW_UTF8PROC_USE_SHARED=OFF \ + -DARROW_ZSTD_USE_SHARED=OFF \ + -DARROW_JNI=ON \ + -DARROW_PARQUET=ON \ + -DARROW_FILESYSTEM=ON \ + -DARROW_DATASET=ON \ + -DARROW_GANDIVA_JAVA=ON \ + -DARROW_GANDIVA_STATIC_LIBSTDCPP=ON \ + -DARROW_GANDIVA=ON \ + -DARROW_ORC=ON \ + -DARROW_PLASMA_JAVA_CLIENT=ON \ + -DARROW_PLASMA=ON \ + -DCMAKE_BUILD_TYPE=Release \ + -DCMAKE_INSTALL_LIBDIR=lib \ + -DCMAKE_INSTALL_PREFIX=$GITHUB_WORKSPACE/java-dist \ + -DCMAKE_UNITY_BUILD=ON \ + -Dre2_SOURCE=BUNDLED \ + -DBoost_SOURCE=BUNDLED \ + -Dutf8proc_SOURCE=BUNDLED \ + -DSnappy_SOURCE=BUNDLED \ + -DORC_SOURCE=BUNDLED \ + -DZLIB_SOURCE=BUNDLED \ + $GITHUB_WORKSPACE/cpp + $ cmake --build . --target install + $ ls -latr $GITHUB_WORKSPACE/java-dist/lib + |__ libarrow_dataset_jni.dylib + |__ libarrow_orc_jni.dylib + |__ libgandiva_jni.dylib + +Build Java Modules That Consume C / C++ Libs +++++++++++++++++++++++++++++++++++++++++++++ + +Build Java C Data Interface +########################### + +Java arrow-c-data maven profile: + +.. code-block:: + + $ cd arrow/java + $ mvn -Darrow.c.jni.dist.dir=$GITHUB_WORKSPACE/java-dist/lib -Parrow-c-data clean install + [INFO] Reactor Summary for Apache Arrow Java Root POM 7.0.0: + [INFO] + [INFO] Apache Arrow Java Root POM ......................... SUCCESS [ 4.385 s] + [INFO] Arrow Format ....................................... SUCCESS [ 4.832 s] + [INFO] Arrow Memory ....................................... SUCCESS [ 1.862 s] + [INFO] Arrow Memory - Core ................................ SUCCESS [ 12.162 s] + [INFO] Arrow Memory - Unsafe .............................. SUCCESS [ 6.768 s] + [INFO] Arrow Memory - Netty ............................... SUCCESS [ 6.897 s] + [INFO] Arrow Vectors ...................................... SUCCESS [01:43 min] + [INFO] Arrow Compression .................................. SUCCESS [ 6.510 s] + [INFO] Arrow Tools ........................................ SUCCESS [ 13.280 s] + [INFO] Arrow JDBC Adapter ................................. SUCCESS [ 13.726 s] + [INFO] Arrow Plasma Client ................................ SUCCESS [ 5.237 s] + [INFO] Arrow Flight ....................................... SUCCESS [ 0.809 s] + [INFO] Arrow Flight Core .................................. SUCCESS [ 47.461 s] + [INFO] Arrow Flight GRPC .................................. SUCCESS [ 6.864 s] + [INFO] Arrow Flight SQL ................................... SUCCESS [ 20.334 s] + [INFO] Arrow Flight Integration Tests ..................... SUCCESS [ 7.542 s] + [INFO] Arrow AVRO Adapter ................................. SUCCESS [ 13.826 s] + [INFO] Arrow Algorithms ................................... SUCCESS [ 32.442 s] + [INFO] Arrow Performance Benchmarks ....................... SUCCESS [ 5.418 s] + [INFO] Arrow Java C Data Interface ........................ SUCCESS [ 7.907 s] + [INFO] ------------------------------------------------------------------------ + [INFO] BUILD SUCCESS + [INFO] ------------------------------------------------------------------------ + +Build Java C++ Interface +######################## + +Java arrow-jni maven profile (build ORC / Gandiva / Dataset): + +.. code-block:: + + $ cd arrow/java + $ mvn -Darrow.cpp.build.dir=$GITHUB_WORKSPACE/java-dist/lib -Parrow-jni clean install + [INFO] Reactor Summary for Apache Arrow Java Root POM 7.0.0: + [INFO] + [INFO] Apache Arrow Java Root POM ......................... SUCCESS [ 7.342 s] + [INFO] Arrow Format ....................................... SUCCESS [ 2.417 s] + [INFO] Arrow Memory ....................................... SUCCESS [ 1.967 s] + [INFO] Arrow Memory - Core ................................ SUCCESS [ 4.714 s] + [INFO] Arrow Memory - Unsafe .............................. SUCCESS [ 3.157 s] + [INFO] Arrow Memory - Netty ............................... SUCCESS [ 3.334 s] + [INFO] Arrow Vectors ...................................... SUCCESS [ 21.791 s] + [INFO] Arrow Compression .................................. SUCCESS [ 3.854 s] + [INFO] Arrow Tools ........................................ SUCCESS [ 8.359 s] + [INFO] Arrow JDBC Adapter ................................. SUCCESS [ 8.847 s] + [INFO] Arrow Plasma Client ................................ SUCCESS [ 2.459 s] + [INFO] Arrow Flight ....................................... SUCCESS [ 2.357 s] + [INFO] Arrow Flight Core .................................. SUCCESS [ 38.837 s] + [INFO] Arrow Flight GRPC .................................. SUCCESS [ 5.955 s] + [INFO] Arrow Flight SQL ................................... SUCCESS [ 17.390 s] + [INFO] Arrow Flight Integration Tests ..................... SUCCESS [ 6.148 s] + [INFO] Arrow AVRO Adapter ................................. SUCCESS [ 9.635 s] + [INFO] Arrow Algorithms ................................... SUCCESS [ 26.949 s] + [INFO] Arrow Performance Benchmarks ....................... SUCCESS [ 4.211 s] + [INFO] Arrow Orc Adapter .................................. SUCCESS [ 6.522 s] + [INFO] Arrow Gandiva ...................................... SUCCESS [01:20 min] + [INFO] Arrow Java Dataset ................................. SUCCESS [ 12.949 s] + [INFO] ------------------------------------------------------------------------ + [INFO] BUILD SUCCESS + [INFO] ------------------------------------------------------------------------ + +IDE Configuration +================= + +Go to open java project and select java folder. + +This is the initial view for java project loaded with default profiles: + +.. image:: img/java_welcome.png + +Let's create our maven configuration: java-dataset + +.. image:: img/java_mvn_configuration.png + +Let's define our JRE runner: + +.. image:: img/java_jre_runner.png + +Let's configure additional environment properties: + +.. image:: img/java_jre_env_properties.png + +Let's run our java-dataset maven configuration: + +.. image:: img/java_run_mvn_configuration.png + +.. image:: img/java_run_mvn_configuration_result.png + +Common Errors +============= + +1.- Errors related to: + - Could NOT find Boost (missing: Boost_INCLUDE_DIR system filesystem) + - Could NOT find Lz4 (missing: LZ4_LIB) + - Could NOT find zstd (missing: ZSTD_LIB) + + Consider to configure parameters that download software at the build time with: + +.. code-block:: + + -Dre2_SOURCE=BUNDLED \ + -DBoost_SOURCE=BUNDLED \ + -Dutf8proc_SOURCE=BUNDLED \ + -DSnappy_SOURCE=BUNDLED \ + -DORC_SOURCE=BUNDLED \ + -DZLIB_SOURCE=BUNDLED \ + +2.- Errors related to: Review comment: ```suggestion 2. Errors related to: ``` ########## File path: docs/source/developers/java/development.rst ########## @@ -0,0 +1,170 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. highlight:: console + +====================== +Development Guidelines +====================== + +.. contents:: + +Unit Testing +============ + +.. code-block:: + + $ cd arrow/java + $ mvn \ + -Darrow.cpp.build.dir=$GITHUB_WORKSPACE/java-dist/lib -Parrow-jni \ + -Darrow.c.jni.dist.dir=$GITHUB_WORKSPACE/java-dist/lib -Parrow-c-data \ + clean install + [INFO] Reactor Summary for Apache Arrow Java Root POM 7.0.0: + [INFO] + [INFO] Apache Arrow Java Root POM ......................... SUCCESS [ 7.308 s] + [INFO] Arrow Format ....................................... SUCCESS [ 4.899 s] + [INFO] Arrow Memory ....................................... SUCCESS [ 2.065 s] + [INFO] Arrow Memory - Core ................................ SUCCESS [ 11.417 s] + [INFO] Arrow Memory - Unsafe .............................. SUCCESS [ 8.132 s] + [INFO] Arrow Memory - Netty ............................... SUCCESS [ 10.211 s] + [INFO] Arrow Vectors ...................................... SUCCESS [01:48 min] + [INFO] Arrow Compression .................................. SUCCESS [ 7.282 s] + [INFO] Arrow Tools ........................................ SUCCESS [ 12.612 s] + [INFO] Arrow JDBC Adapter ................................. SUCCESS [ 15.558 s] + [INFO] Arrow Plasma Client ................................ SUCCESS [ 6.303 s] + [INFO] Arrow Flight ....................................... SUCCESS [ 2.326 s] + [INFO] Arrow Flight Core .................................. SUCCESS [ 50.632 s] + [INFO] Arrow Flight GRPC .................................. SUCCESS [ 9.932 s] + [INFO] Arrow Flight SQL ................................... SUCCESS [ 22.559 s] + [INFO] Arrow Flight Integration Tests ..................... SUCCESS [ 9.649 s] + [INFO] Arrow AVRO Adapter ................................. SUCCESS [ 17.145 s] + [INFO] Arrow Algorithms ................................... SUCCESS [ 44.571 s] + [INFO] Arrow Performance Benchmarks ....................... SUCCESS [ 9.202 s] + [INFO] Arrow Java C Data Interface ........................ SUCCESS [ 15.794 s] + [INFO] Arrow Orc Adapter .................................. SUCCESS [ 14.205 s] + [INFO] Arrow Gandiva ...................................... SUCCESS [02:06 min] + [INFO] Arrow Java Dataset ................................. SUCCESS [ 15.261 s] + [INFO] ------------------------------------------------------------------------ + [INFO] BUILD SUCCESS + [INFO] ------------------------------------------------------------------------ + +Performance Testing +=================== + +Java module that contains performance tests is: `performance` + +Let's configure our performance environment to run performance tests: + +- Install `benchmark`_ +- Install `archery`_ +- Install `conbench`_ + +Lets execute benchmark tests: + +.. code-block:: + + $ cd benchmarks + $ conbench java-micro --help + $ conbench java-micro + --iterations=1 + --commit=e90472e35b40f58b17d408438bb8de1641bfe6ef + --java-home=<absolute path to your java home> + --src=<absolute path to your arrow project> + --benchmark-filter=org.apache.arrow.adapter.AvroAdapterBenchmarks.testAvroToArrow + Benchmark Mode Cnt Score Error Units + AvroAdapterBenchmarks.testAvroToArrow avgt 725545.783 ns/op + Time to POST http://localhost:5000/api/login/ 0.14911699295043945 + Time to POST http://localhost:5000/api/benchmarks/ 0.06116318702697754 + +Then go to: http://127.0.0.1:5000/ to see reports: + +UI Home: + +.. image:: img/conbench_ui.png + +UI Runs: + +.. image:: img/conbench_runs.png + +UI Benchmark: + +.. image:: img/conbench_benchmark.png + +Code Style +========== + +Code style are configured at `java/dev/checkstyle` + +.. code-block:: + + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-checkstyle-plugin</artifactId> + <version>3.1.0</version> + <executions> + <execution> + <id>validate</id> + <phase>validate</phase> + <goals> + <goal>check</goal> + </goals> + </execution> + </executions> + <configuration> + <configLocation>dev/checkstyle/checkstyle.xml</configLocation> + <headerLocation>dev/checkstyle/checkstyle.license</headerLocation> + <suppressionsLocation>dev/checkstyle/suppressions.xml</suppressionsLocation> + <includeTestSourceDirectory>true</includeTestSourceDirectory> + ... + </configuration> + </plugin> Review comment: Don't show this here, just link to the checkstyle configuration. ########## File path: docs/source/developers/java/development.rst ########## @@ -0,0 +1,170 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. highlight:: console + +====================== +Development Guidelines +====================== + +.. contents:: + +Unit Testing +============ + +.. code-block:: + + $ cd arrow/java + $ mvn \ + -Darrow.cpp.build.dir=$GITHUB_WORKSPACE/java-dist/lib -Parrow-jni \ + -Darrow.c.jni.dist.dir=$GITHUB_WORKSPACE/java-dist/lib -Parrow-c-data \ + clean install + [INFO] Reactor Summary for Apache Arrow Java Root POM 7.0.0: + [INFO] + [INFO] Apache Arrow Java Root POM ......................... SUCCESS [ 7.308 s] + [INFO] Arrow Format ....................................... SUCCESS [ 4.899 s] + [INFO] Arrow Memory ....................................... SUCCESS [ 2.065 s] + [INFO] Arrow Memory - Core ................................ SUCCESS [ 11.417 s] + [INFO] Arrow Memory - Unsafe .............................. SUCCESS [ 8.132 s] + [INFO] Arrow Memory - Netty ............................... SUCCESS [ 10.211 s] + [INFO] Arrow Vectors ...................................... SUCCESS [01:48 min] + [INFO] Arrow Compression .................................. SUCCESS [ 7.282 s] + [INFO] Arrow Tools ........................................ SUCCESS [ 12.612 s] + [INFO] Arrow JDBC Adapter ................................. SUCCESS [ 15.558 s] + [INFO] Arrow Plasma Client ................................ SUCCESS [ 6.303 s] + [INFO] Arrow Flight ....................................... SUCCESS [ 2.326 s] + [INFO] Arrow Flight Core .................................. SUCCESS [ 50.632 s] + [INFO] Arrow Flight GRPC .................................. SUCCESS [ 9.932 s] + [INFO] Arrow Flight SQL ................................... SUCCESS [ 22.559 s] + [INFO] Arrow Flight Integration Tests ..................... SUCCESS [ 9.649 s] + [INFO] Arrow AVRO Adapter ................................. SUCCESS [ 17.145 s] + [INFO] Arrow Algorithms ................................... SUCCESS [ 44.571 s] + [INFO] Arrow Performance Benchmarks ....................... SUCCESS [ 9.202 s] + [INFO] Arrow Java C Data Interface ........................ SUCCESS [ 15.794 s] + [INFO] Arrow Orc Adapter .................................. SUCCESS [ 14.205 s] + [INFO] Arrow Gandiva ...................................... SUCCESS [02:06 min] + [INFO] Arrow Java Dataset ................................. SUCCESS [ 15.261 s] + [INFO] ------------------------------------------------------------------------ + [INFO] BUILD SUCCESS + [INFO] ------------------------------------------------------------------------ + +Performance Testing +=================== + +Java module that contains performance tests is: `performance` + +Let's configure our performance environment to run performance tests: + +- Install `benchmark`_ +- Install `archery`_ +- Install `conbench`_ Review comment: I believe this works without conbench, you just won't get the local UI? ########## File path: docs/source/developers/java/building.rst ########## @@ -0,0 +1,308 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. highlight:: console + +.. _building-arrow-cpp: + +=================== +Building Arrow Java +=================== + +.. contents:: + +System setup +============ + +Arrow Java use Maven as a project management. + +Building requires: + +* JRE 8, 9, 10, 11 +* Maven 3+ + +Building +======== + +All the instructions below assume that you have cloned the Arrow git +repository: + +.. code-block:: + + $ git clone https://github.com/apache/arrow.git + $ cd arrow + $ git submodule update --init --recursive + +Basic Installation +------------------ + +You can also build java project for default modules configured. +Go to parent project and execute: + +.. code-block:: + + $ cd arrow/java + $ export JAVA_HOME=<absolute path to your java home> + $ java --version + $ mvn clean install + [INFO] Reactor Summary for Apache Arrow Java Root POM 8.0.0-SNAPSHOT: + [INFO] + [INFO] Apache Arrow Java Root POM ......................... SUCCESS [ 4.922 s] + [INFO] Arrow Format ....................................... SUCCESS [ 4.395 s] + [INFO] Arrow Memory ....................................... SUCCESS [ 0.938 s] + [INFO] Arrow Memory - Core ................................ SUCCESS [ 11.491 s] + [INFO] Arrow Memory - Unsafe .............................. SUCCESS [ 5.449 s] + [INFO] Arrow Memory - Netty ............................... SUCCESS [ 8.754 s] + [INFO] Arrow Vectors ...................................... SUCCESS [01:37 min] + [INFO] Arrow Compression .................................. SUCCESS [ 5.907 s] + [INFO] Arrow Tools ........................................ SUCCESS [ 11.015 s] + [INFO] Arrow JDBC Adapter ................................. SUCCESS [ 13.751 s] + [INFO] Arrow Plasma Client ................................ SUCCESS [ 3.934 s] + [INFO] Arrow Flight ....................................... SUCCESS [ 0.649 s] + [INFO] Arrow Flight Core .................................. SUCCESS [ 38.331 s] + [INFO] Arrow Flight GRPC .................................. SUCCESS [ 5.894 s] + [INFO] Arrow Flight SQL ................................... SUCCESS [ 21.862 s] + [INFO] Arrow Flight Integration Tests ..................... SUCCESS [ 7.638 s] + [INFO] Arrow AVRO Adapter ................................. SUCCESS [ 14.447 s] + [INFO] Arrow Algorithms ................................... SUCCESS [ 35.105 s] + [INFO] Arrow Performance Benchmarks ....................... SUCCESS [ 5.323 s] + [INFO] ------------------------------------------------------------------------ + [INFO] BUILD SUCCESS + [INFO] ------------------------------------------------------------------------\ + +Advanced Installation +--------------------- + +Build C / C++ Libs +++++++++++++++++++ + +Build C / C++ Libs Ubuntu +######################### + +For Build C Data Interface lib & Build C++ Libs. Consider: Next step requires: Docker & Docker Compose. + +.. code-block:: + + $ cd arrow + $ source dev/conbench_envs/hooks.sh install_archery + Successfully installed archery + $ archery docker run java-jni-manylinux-2014 + $ ls -latr java-dist/ + |__ libarrow_cdata_jni.so + |__ libarrow_dataset_jni.so + |__ libarrow_orc_jni.so + |__ libgandiva_jni.so + +Build C / C++ Libs MacOS +######################## + +For Build C Data Interface lib: + +.. code-block:: + + $ cd arrow + $ brew bundle --file=cpp/Brewfile + Homebrew Bundle complete! 25 Brewfile dependencies now installed. + $ export GITHUB_WORKSPACE=<absolute path to your arrow project> + $ export JAVA_HOME=<absolute path to your java home> + $ mkdir -p java-dist java-native-c + $ cd java-native-c + $ cmake \ + -DCMAKE_BUILD_TYPE=Release \ + -DCMAKE_INSTALL_LIBDIR=lib \ + -DCMAKE_INSTALL_PREFIX=$GITHUB_WORKSPACE/java-dist \ + $GITHUB_WORKSPACE/java/c + $ cmake --build . --target install + $ ls -latr $GITHUB_WORKSPACE/java-dist/lib + |__ libarrow_cdata_jni.dylib + +For Build C++ Interface libs: + +.. code-block:: + + $ cd arrow + $ brew bundle --file=cpp/Brewfile + Homebrew Bundle complete! 25 Brewfile dependencies now installed. + $ export GITHUB_WORKSPACE=<absolute path to your arrow project> + $ export JAVA_HOME=<absolute path to your java home> + $ mkdir -p java-dist java-native-cpp + $ cd java-native-cpp + $ cmake \ + -DARROW_BOOST_USE_SHARED=OFF \ + -DARROW_BROTLI_USE_SHARED=OFF \ + -DARROW_BZ2_USE_SHARED=OFF \ + -DARROW_GFLAGS_USE_SHARED=OFF \ + -DARROW_GRPC_USE_SHARED=OFF \ + -DARROW_LZ4_USE_SHARED=OFF \ + -DARROW_OPENSSL_USE_SHARED=OFF \ + -DARROW_PROTOBUF_USE_SHARED=OFF \ + -DARROW_SNAPPY_USE_SHARED=OFF \ + -DARROW_THRIFT_USE_SHARED=OFF \ + -DARROW_UTF8PROC_USE_SHARED=OFF \ + -DARROW_ZSTD_USE_SHARED=OFF \ + -DARROW_JNI=ON \ + -DARROW_PARQUET=ON \ + -DARROW_FILESYSTEM=ON \ + -DARROW_DATASET=ON \ + -DARROW_GANDIVA_JAVA=ON \ + -DARROW_GANDIVA_STATIC_LIBSTDCPP=ON \ + -DARROW_GANDIVA=ON \ + -DARROW_ORC=ON \ + -DARROW_PLASMA_JAVA_CLIENT=ON \ + -DARROW_PLASMA=ON \ + -DCMAKE_BUILD_TYPE=Release \ + -DCMAKE_INSTALL_LIBDIR=lib \ + -DCMAKE_INSTALL_PREFIX=$GITHUB_WORKSPACE/java-dist \ + -DCMAKE_UNITY_BUILD=ON \ + -Dre2_SOURCE=BUNDLED \ + -DBoost_SOURCE=BUNDLED \ + -Dutf8proc_SOURCE=BUNDLED \ + -DSnappy_SOURCE=BUNDLED \ + -DORC_SOURCE=BUNDLED \ + -DZLIB_SOURCE=BUNDLED \ + $GITHUB_WORKSPACE/cpp + $ cmake --build . --target install + $ ls -latr $GITHUB_WORKSPACE/java-dist/lib + |__ libarrow_dataset_jni.dylib + |__ libarrow_orc_jni.dylib + |__ libgandiva_jni.dylib + +Build Java Modules That Consume C / C++ Libs +++++++++++++++++++++++++++++++++++++++++++++ + +Build Java C Data Interface +########################### + +Java arrow-c-data maven profile: + +.. code-block:: + + $ cd arrow/java + $ mvn -Darrow.c.jni.dist.dir=$GITHUB_WORKSPACE/java-dist/lib -Parrow-c-data clean install + [INFO] Reactor Summary for Apache Arrow Java Root POM 7.0.0: + [INFO] + [INFO] Apache Arrow Java Root POM ......................... SUCCESS [ 4.385 s] + [INFO] Arrow Format ....................................... SUCCESS [ 4.832 s] + [INFO] Arrow Memory ....................................... SUCCESS [ 1.862 s] + [INFO] Arrow Memory - Core ................................ SUCCESS [ 12.162 s] + [INFO] Arrow Memory - Unsafe .............................. SUCCESS [ 6.768 s] + [INFO] Arrow Memory - Netty ............................... SUCCESS [ 6.897 s] + [INFO] Arrow Vectors ...................................... SUCCESS [01:43 min] + [INFO] Arrow Compression .................................. SUCCESS [ 6.510 s] + [INFO] Arrow Tools ........................................ SUCCESS [ 13.280 s] + [INFO] Arrow JDBC Adapter ................................. SUCCESS [ 13.726 s] + [INFO] Arrow Plasma Client ................................ SUCCESS [ 5.237 s] + [INFO] Arrow Flight ....................................... SUCCESS [ 0.809 s] + [INFO] Arrow Flight Core .................................. SUCCESS [ 47.461 s] + [INFO] Arrow Flight GRPC .................................. SUCCESS [ 6.864 s] + [INFO] Arrow Flight SQL ................................... SUCCESS [ 20.334 s] + [INFO] Arrow Flight Integration Tests ..................... SUCCESS [ 7.542 s] + [INFO] Arrow AVRO Adapter ................................. SUCCESS [ 13.826 s] + [INFO] Arrow Algorithms ................................... SUCCESS [ 32.442 s] + [INFO] Arrow Performance Benchmarks ....................... SUCCESS [ 5.418 s] + [INFO] Arrow Java C Data Interface ........................ SUCCESS [ 7.907 s] + [INFO] ------------------------------------------------------------------------ + [INFO] BUILD SUCCESS + [INFO] ------------------------------------------------------------------------ + +Build Java C++ Interface +######################## + +Java arrow-jni maven profile (build ORC / Gandiva / Dataset): + +.. code-block:: + + $ cd arrow/java + $ mvn -Darrow.cpp.build.dir=$GITHUB_WORKSPACE/java-dist/lib -Parrow-jni clean install + [INFO] Reactor Summary for Apache Arrow Java Root POM 7.0.0: + [INFO] + [INFO] Apache Arrow Java Root POM ......................... SUCCESS [ 7.342 s] + [INFO] Arrow Format ....................................... SUCCESS [ 2.417 s] + [INFO] Arrow Memory ....................................... SUCCESS [ 1.967 s] + [INFO] Arrow Memory - Core ................................ SUCCESS [ 4.714 s] + [INFO] Arrow Memory - Unsafe .............................. SUCCESS [ 3.157 s] + [INFO] Arrow Memory - Netty ............................... SUCCESS [ 3.334 s] + [INFO] Arrow Vectors ...................................... SUCCESS [ 21.791 s] + [INFO] Arrow Compression .................................. SUCCESS [ 3.854 s] + [INFO] Arrow Tools ........................................ SUCCESS [ 8.359 s] + [INFO] Arrow JDBC Adapter ................................. SUCCESS [ 8.847 s] + [INFO] Arrow Plasma Client ................................ SUCCESS [ 2.459 s] + [INFO] Arrow Flight ....................................... SUCCESS [ 2.357 s] + [INFO] Arrow Flight Core .................................. SUCCESS [ 38.837 s] + [INFO] Arrow Flight GRPC .................................. SUCCESS [ 5.955 s] + [INFO] Arrow Flight SQL ................................... SUCCESS [ 17.390 s] + [INFO] Arrow Flight Integration Tests ..................... SUCCESS [ 6.148 s] + [INFO] Arrow AVRO Adapter ................................. SUCCESS [ 9.635 s] + [INFO] Arrow Algorithms ................................... SUCCESS [ 26.949 s] + [INFO] Arrow Performance Benchmarks ....................... SUCCESS [ 4.211 s] + [INFO] Arrow Orc Adapter .................................. SUCCESS [ 6.522 s] + [INFO] Arrow Gandiva ...................................... SUCCESS [01:20 min] + [INFO] Arrow Java Dataset ................................. SUCCESS [ 12.949 s] + [INFO] ------------------------------------------------------------------------ + [INFO] BUILD SUCCESS + [INFO] ------------------------------------------------------------------------ + +IDE Configuration +================= + +Go to open java project and select java folder. + +This is the initial view for java project loaded with default profiles: + +.. image:: img/java_welcome.png + +Let's create our maven configuration: java-dataset + +.. image:: img/java_mvn_configuration.png + +Let's define our JRE runner: + +.. image:: img/java_jre_runner.png + +Let's configure additional environment properties: + +.. image:: img/java_jre_env_properties.png + +Let's run our java-dataset maven configuration: + +.. image:: img/java_run_mvn_configuration.png + +.. image:: img/java_run_mvn_configuration_result.png + +Common Errors +============= + +1.- Errors related to: + - Could NOT find Boost (missing: Boost_INCLUDE_DIR system filesystem) + - Could NOT find Lz4 (missing: LZ4_LIB) + - Could NOT find zstd (missing: ZSTD_LIB) + + Consider to configure parameters that download software at the build time with: + +.. code-block:: + + -Dre2_SOURCE=BUNDLED \ + -DBoost_SOURCE=BUNDLED \ + -Dutf8proc_SOURCE=BUNDLED \ + -DSnappy_SOURCE=BUNDLED \ + -DORC_SOURCE=BUNDLED \ + -DZLIB_SOURCE=BUNDLED \ + Review comment: We can link to the C++ docs e.g. https://arrow.apache.org/docs/developers/cpp/building.html#individual-dependency-resolution ########## File path: docs/source/developers/java/development.rst ########## @@ -0,0 +1,170 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. highlight:: console + +====================== +Development Guidelines +====================== + +.. contents:: + +Unit Testing +============ + +.. code-block:: + + $ cd arrow/java + $ mvn \ + -Darrow.cpp.build.dir=$GITHUB_WORKSPACE/java-dist/lib -Parrow-jni \ + -Darrow.c.jni.dist.dir=$GITHUB_WORKSPACE/java-dist/lib -Parrow-c-data \ + clean install + [INFO] Reactor Summary for Apache Arrow Java Root POM 7.0.0: + [INFO] + [INFO] Apache Arrow Java Root POM ......................... SUCCESS [ 7.308 s] + [INFO] Arrow Format ....................................... SUCCESS [ 4.899 s] + [INFO] Arrow Memory ....................................... SUCCESS [ 2.065 s] + [INFO] Arrow Memory - Core ................................ SUCCESS [ 11.417 s] + [INFO] Arrow Memory - Unsafe .............................. SUCCESS [ 8.132 s] + [INFO] Arrow Memory - Netty ............................... SUCCESS [ 10.211 s] + [INFO] Arrow Vectors ...................................... SUCCESS [01:48 min] + [INFO] Arrow Compression .................................. SUCCESS [ 7.282 s] + [INFO] Arrow Tools ........................................ SUCCESS [ 12.612 s] + [INFO] Arrow JDBC Adapter ................................. SUCCESS [ 15.558 s] + [INFO] Arrow Plasma Client ................................ SUCCESS [ 6.303 s] + [INFO] Arrow Flight ....................................... SUCCESS [ 2.326 s] + [INFO] Arrow Flight Core .................................. SUCCESS [ 50.632 s] + [INFO] Arrow Flight GRPC .................................. SUCCESS [ 9.932 s] + [INFO] Arrow Flight SQL ................................... SUCCESS [ 22.559 s] + [INFO] Arrow Flight Integration Tests ..................... SUCCESS [ 9.649 s] + [INFO] Arrow AVRO Adapter ................................. SUCCESS [ 17.145 s] + [INFO] Arrow Algorithms ................................... SUCCESS [ 44.571 s] + [INFO] Arrow Performance Benchmarks ....................... SUCCESS [ 9.202 s] + [INFO] Arrow Java C Data Interface ........................ SUCCESS [ 15.794 s] + [INFO] Arrow Orc Adapter .................................. SUCCESS [ 14.205 s] + [INFO] Arrow Gandiva ...................................... SUCCESS [02:06 min] + [INFO] Arrow Java Dataset ................................. SUCCESS [ 15.261 s] + [INFO] ------------------------------------------------------------------------ + [INFO] BUILD SUCCESS + [INFO] ------------------------------------------------------------------------ + +Performance Testing +=================== + +Java module that contains performance tests is: `performance` + +Let's configure our performance environment to run performance tests: + +- Install `benchmark`_ +- Install `archery`_ +- Install `conbench`_ + +Lets execute benchmark tests: + +.. code-block:: + + $ cd benchmarks + $ conbench java-micro --help + $ conbench java-micro + --iterations=1 + --commit=e90472e35b40f58b17d408438bb8de1641bfe6ef + --java-home=<absolute path to your java home> + --src=<absolute path to your arrow project> + --benchmark-filter=org.apache.arrow.adapter.AvroAdapterBenchmarks.testAvroToArrow + Benchmark Mode Cnt Score Error Units + AvroAdapterBenchmarks.testAvroToArrow avgt 725545.783 ns/op + Time to POST http://localhost:5000/api/login/ 0.14911699295043945 + Time to POST http://localhost:5000/api/benchmarks/ 0.06116318702697754 + +Then go to: http://127.0.0.1:5000/ to see reports: + +UI Home: + +.. image:: img/conbench_ui.png + +UI Runs: + +.. image:: img/conbench_runs.png + +UI Benchmark: + +.. image:: img/conbench_benchmark.png + +Code Style +========== + +Code style are configured at `java/dev/checkstyle` Review comment: Backticks don't do what you want in reStructuredText, use double-backticks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
