Repository: parquet-cpp Updated Branches: refs/heads/master 4ae1b921b -> 17a204555
PARQUET-471: Use common build environment setup for sandbox and Travis CI. Travis CI cleanup This also adds better build instructions for OS X users using Homebrew (googletest is not available and needs to be build from `thirdparty/`). Closes #53 Author: Wes McKinney <[email protected]> Closes #54 from wesm/PARQUET-471 and squashes the following commits: 162f523 [Wes McKinney] Refactor .travis.yml and associated scripts to have a common sandbox setup script. Add a build matrix and disable unneeded OS X gcc build. Better thirdparty documentation for googletest and build setup Project: http://git-wip-us.apache.org/repos/asf/parquet-cpp/repo Commit: http://git-wip-us.apache.org/repos/asf/parquet-cpp/commit/17a20455 Tree: http://git-wip-us.apache.org/repos/asf/parquet-cpp/tree/17a20455 Diff: http://git-wip-us.apache.org/repos/asf/parquet-cpp/diff/17a20455 Branch: refs/heads/master Commit: 17a204555bede41d1f68e59a540f3bbef072c821 Parents: 4ae1b92 Author: Wes McKinney <[email protected]> Authored: Thu Feb 18 15:29:46 2016 -0800 Committer: Julien Le Dem <[email protected]> Committed: Thu Feb 18 15:29:46 2016 -0800 ---------------------------------------------------------------------- .travis.yml | 49 +++++++++++++++++++++-------------- README.md | 45 ++++++++++++++++++++++++++------ ci/before_script_travis.sh | 20 ++++---------- ci/run_tests.sh | 5 ---- setup_build_env.sh | 22 +++++++--------- thirdparty/set_thirdparty_env.sh | 18 +++++++++++++ 6 files changed, 98 insertions(+), 61 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/parquet-cpp/blob/17a20455/.travis.yml ---------------------------------------------------------------------- diff --git a/.travis.yml b/.travis.yml index 09f4705..ad71e29 100644 --- a/.travis.yml +++ b/.travis.yml @@ -1,15 +1,5 @@ sudo: required dist: trusty -language: cpp - -compiler: - - gcc - - clang - -os: - - linux - - osx - addons: apt: sources: @@ -28,17 +18,36 @@ addons: - bison #needed for thrift cpp compilation - flex #needed for thrift cpp compilation - pkg-config #needed for thrift cpp compilation +cache: + directories: + - $HOME/parquet-build +matrix: + include: + - compiler: gcc + os: linux + script: + - make lint + - make -j4 || exit 1 + - valgrind --tool=memcheck --leak-check=yes --error-exitcode=1 ctest + - compiler: clang + os: linux + script: + - make -j4 || exit 1 + - ctest + - os: osx + compiler: clang + addons: + script: + - make -j4 || exit 1 + - ctest + +language: cpp before_install: - - mkdir $HOME/build_dir - - cd $HOME/build_dir +- mkdir $HOME/parquet-build +- pushd $HOME/parquet-build before_script: - - source $TRAVIS_BUILD_DIR/ci/before_script_travis.sh - - cmake -DCMAKE_CXX_FLAGS="-Werror" $TRAVIS_BUILD_DIR - - export PARQUET_TEST_DATA=$TRAVIS_BUILD_DIR/data - -script: -- make -- source $TRAVIS_BUILD_DIR/ci/run_tests.sh -- make lint +- source $TRAVIS_BUILD_DIR/ci/before_script_travis.sh +- cmake -DCMAKE_CXX_FLAGS="-Werror" $TRAVIS_BUILD_DIR +- export PARQUET_TEST_DATA=$TRAVIS_BUILD_DIR/data http://git-wip-us.apache.org/repos/asf/parquet-cpp/blob/17a20455/README.md ---------------------------------------------------------------------- diff --git a/README.md b/README.md index 1578d4e..b6ec89e 100644 --- a/README.md +++ b/README.md @@ -1,35 +1,64 @@ -Parquet-cpp [](https://travis-ci.org/apache/parquet-cpp) -=========== -A C++ library to read parquet files. +## parquet-cpp: a C++ library to read and write the Apache Parquet columnar data format. + +[](https://travis-ci.org/apache/parquet-cpp) ## Third Party Dependencies + - snappy - lz4 +- zlib - thrift 0.7+ [install instructions](https://thrift.apache.org/docs/install/) +- googletest 1.7.0 (cannot be installed with package managers) + +You can install these dependencies using a package manager or using the +`thirdparty/` scripts in this repository. On Homebrew, you can run: -Many package managers support some or all of these dependencies. E.g.: ```shell -ubuntu$ sudo apt-get install libboost-dev libsnappy-dev liblz4-dev + brew install snappy lz4 thrift zlib ``` + +To build the thirdparty libraries in-tree, run: + ```shell -mac$ brew install snappy lz4 thrift +./thirdparty/download_thirdparty.sh +./thirdparty/build_thirdparty.sh +source thirdparty/set_thirdparty_env.sh ``` -`setup_build_env.sh` tries to automate setting up a build environment for you +The provided script `setup_build_env.sh` sets up a build environment for you with third party dependencies. You use it by running `source setup_build_env.sh`. By default, it will create a build directory `build/`. You can override the build directory by setting the BUILD_DIR env variable to another location. -Also feel free to take a look at our [.travis.yml](.travis.yml) to see how that build env is set up. +After building the thirdparty libraries, for future development iteration you +can set the dependency environment variables (detailed below) by running + +`source $BUILD_DIR/thirdparty/set_thirdparty_env.sh` + +Note, the environment variables are set automatically the first time you run +`setup_build_env.sh`. +The unit tests depend on `googletest` which cannot be installed with Homebrew +or normal package managers. If you wish to use system dependencies, we +recommend that you build googletest in-tree by running: + +``` +./thirdparty/download_thirdparty.sh +./thirdparty/build_thirdparty.sh gtest +source thirdparty/versions.sh +export GTEST_HOME=`pwd`/thirdparty/$GTEST_BASEDIR +``` ## Build + - `cmake .` + - You can customize dependent library locations through various environment variables: - THRIFT_HOME customizes the thrift installed location. - SNAPPY_HOME customizes the snappy installed location. - LZ4_HOME customizes the lz4 installed location. + - `make` The binaries will be built to ./debug which contains the libraries to link against as http://git-wip-us.apache.org/repos/asf/parquet-cpp/blob/17a20455/ci/before_script_travis.sh ---------------------------------------------------------------------- diff --git a/ci/before_script_travis.sh b/ci/before_script_travis.sh index b1f31c5..c467c67 100755 --- a/ci/before_script_travis.sh +++ b/ci/before_script_travis.sh @@ -3,25 +3,15 @@ # Build an isolated thirdparty cp -r $TRAVIS_BUILD_DIR/thirdparty . ./thirdparty/download_thirdparty.sh -source thirdparty/versions.sh if [ $TRAVIS_OS_NAME == "osx" ]; then - brew update - brew install thrift lz4 snappy - - # Only build gtest - ./thirdparty/build_thirdparty.sh gtest -fi - -if [ $TRAVIS_OS_NAME == "linux" ]; then - ./thirdparty/build_thirdparty.sh - export THRIFT_HOME=$HOME/build_dir/thirdparty/installed - export SNAPPY_HOME=$HOME/build_dir/thirdparty/installed - export LZ4_HOME=$HOME/build_dir/thirdparty/installed - + brew update -q + brew install thrift +else # Use a C++11 compiler on Linux export CC="gcc-4.9" export CXX="g++-4.9" fi -export GTEST_HOME=$HOME/build_dir/thirdparty/$GTEST_BASEDIR +export PARQUET_TEST_DATA=$TRAVIS_BUILD_DIR/data +source $TRAVIS_BUILD_DIR/setup_build_env.sh http://git-wip-us.apache.org/repos/asf/parquet-cpp/blob/17a20455/ci/run_tests.sh ---------------------------------------------------------------------- diff --git a/ci/run_tests.sh b/ci/run_tests.sh deleted file mode 100755 index aa7b2f6..0000000 --- a/ci/run_tests.sh +++ /dev/null @@ -1,5 +0,0 @@ -if [ $TRAVIS_OS_NAME == "linux" ]; then - valgrind --tool=memcheck --leak-check=yes --error-exitcode=1 ctest; -else - ctest; -fi http://git-wip-us.apache.org/repos/asf/parquet-cpp/blob/17a20455/setup_build_env.sh ---------------------------------------------------------------------- diff --git a/setup_build_env.sh b/setup_build_env.sh index 6df1f49..60ce7cf 100755 --- a/setup_build_env.sh +++ b/setup_build_env.sh @@ -3,27 +3,23 @@ SOURCE_DIR=$(cd "$(dirname "$BASH_SOURCE")"; pwd) : ${BUILD_DIR:=$SOURCE_DIR/build} +# Create an isolated thirdparty mkdir -p $BUILD_DIR cp -r $SOURCE_DIR/thirdparty $BUILD_DIR -cd $BUILD_DIR +pushd $BUILD_DIR + ./thirdparty/download_thirdparty.sh ./thirdparty/build_thirdparty.sh -source thirdparty/versions.sh - -export SNAPPY_HOME=$BUILD_DIR/thirdparty/installed -export LZ4_HOME=$BUILD_DIR/thirdparty/installed -export ZLIB_HOME=$BUILD_DIR/thirdparty/installed -# build script doesn't support building thrift on OSX -if [ "$(uname)" != "Darwin" ]; then - export THRIFT_HOME=$BUILD_DIR/thirdparty/installed -fi +source thirdparty/set_thirdparty_env.sh export PARQUET_TEST_DATA=$SOURCE_DIR/data -export GTEST_HOME=$BUILD_DIR/thirdparty/$GTEST_BASEDIR -cmake $SOURCE_DIR +# Thrift needs to be installed from brew on OS X +if [ "$(uname)" == "Darwin" ]; then + export -n THRIFT_HOME +fi -cd $SOURCE_DIR +popd echo echo "Build env initialized in $BUILD_DIR." http://git-wip-us.apache.org/repos/asf/parquet-cpp/blob/17a20455/thirdparty/set_thirdparty_env.sh ---------------------------------------------------------------------- diff --git a/thirdparty/set_thirdparty_env.sh b/thirdparty/set_thirdparty_env.sh new file mode 100644 index 0000000..50c5731 --- /dev/null +++ b/thirdparty/set_thirdparty_env.sh @@ -0,0 +1,18 @@ +#!/bin/bash + +SOURCE_DIR=$(cd "$(dirname "$BASH_SOURCE")"; pwd) +source $SOURCE_DIR/versions.sh + +if [ -z "$THIRDPARTY_DIR" ]; then + THIRDPARTY_DIR=$SOURCE_DIR +fi + +export SNAPPY_HOME=$THIRDPARTY_DIR/installed +export LZ4_HOME=$THIRDPARTY_DIR/installed +export ZLIB_HOME=$THIRDPARTY_DIR/installed +# build script doesn't support building thrift on OSX +if [ "$(uname)" != "Darwin" ]; then + export THRIFT_HOME=$THIRDPARTY_DIR/installed +fi + +export GTEST_HOME=$THIRDPARTY_DIR/$GTEST_BASEDIR
