Arrow / R - too many batches causes a crash
RAN (R 3.6.1) knitr 1.252019-09-18 [1] CRAN (R 3.6.1) lattice * 0.20-38 2018-11-04 [2] CRAN (R 3.6.1) latticeExtra 0.6-28 2016-02-09 [1] CRAN (R 3.6.1) lazyeval 0.2.2 2019-03-15 [1] CRAN (R 3.6.1) lifecycle 0.1.0 2019-08-01 [1] CRAN (R 3.6.1) listenv0.7.0 2018-01-21 [1] CRAN (R 3.6.1) lubridate* 1.7.4 2018-04-11 [1] CRAN (R 3.6.1) magrittr * 1.5 2014-11-22 [1] CRAN (R 3.6.1) Matrix 1.2-17 2019-03-22 [2] CRAN (R 3.6.1) memoise1.1.0 2017-04-21 [1] CRAN (R 3.6.1) munsell0.5.0 2018-06-12 [1] CRAN (R 3.6.1) nnet 7.3-12 2016-02-02 [2] CRAN (R 3.6.1) openssl1.4.1 2019-07-18 [1] CRAN (R 3.6.1) outliers * 0.142011-01-24 [1] CRAN (R 3.6.0) pillar 1.4.2 2019-06-29 [1] CRAN (R 3.6.1) pkgbuild 1.0.5 2019-08-26 [1] CRAN (R 3.6.1) pkgconfig 2.0.2 2018-08-16 [1] CRAN (R 3.6.1) pkgload1.0.2 2018-10-29 [1] CRAN (R 3.6.1) plyr * 1.8.4 2016-06-08 [1] CRAN (R 3.6.1) prettyunits1.0.2 2015-07-13 [1] CRAN (R 3.6.1) processx 3.4.1 2019-07-18 [1] CRAN (R 3.6.1) pryr * 0.1.4 2018-02-18 [1] CRAN (R 3.6.1) ps 1.3.0 2018-12-21 [1] CRAN (R 3.6.1) purrr* 0.3.2 2019-03-15 [1] CRAN (R 3.6.1) R6 * 2.4.1 2019-11-12 [1] CRAN (R 3.6.1) RColorBrewer 1.1-2 2014-12-07 [1] CRAN (R 3.6.0) Rcpp 1.0.3 2019-11-08 [1] CRAN (R 3.6.1) readxl * 1.3.1 2019-03-13 [1] CRAN (R 3.6.1) remotes2.1.0 2019-06-24 [1] CRAN (R 3.6.1) rlang* 0.4.2 2019-11-23 [1] CRAN (R 3.6.1) rmarkdown* 2.0.3 2019-12-19 [1] Github (rstudio/rmarkdown@26cc3b1) RODBC* 1.3-16 2019-09-03 [1] CRAN (R 3.6.1) roxygen2 * 6.1.1 2018-11-07 [1] CRAN (R 3.6.1) rpart 4.1-15 2019-04-12 [2] CRAN (R 3.6.1) rprojroot 1.3-2 2018-01-03 [1] CRAN (R 3.6.1) RSQLite * 2.1.2 2019-07-24 [1] CRAN (R 3.6.1) rstudioapi 0.102019-03-19 [1] CRAN (R 3.6.1) scales 1.0.0 2018-08-09 [1] CRAN (R 3.6.1) sessioninfo1.1.1 2018-11-05 [1] CRAN (R 3.6.1) slide* 0.0.0.9002 2019-11-27 [1] Github (DavisVaughan/slide@92e8e02) ssh0.6 2019-04-09 [1] CRAN (R 3.6.2) stringi1.4.3 2019-03-12 [1] CRAN (R 3.6.0) stringr * 1.4.0 2019-02-10 [1] CRAN (R 3.6.1) survival * 2.44-1.12019-04-01 [2] CRAN (R 3.6.1) testthat 2.2.1 2019-07-25 [1] CRAN (R 3.6.1) tibble 2.1.3 2019-06-06 [1] CRAN (R 3.6.1) tidyr* 1.0.0 2019-09-11 [1] CRAN (R 3.6.1) tidyselect 0.2.5 2018-10-11 [1] CRAN (R 3.6.1) usethis * 1.5.1 2019-07-04 [1] CRAN (R 3.6.1) varhandle* 2.0.3 2018-07-04 [1] CRAN (R 3.6.0) vctrs 0.2.0.9007 2019-11-27 [1] Github (r-lib/vctrs@945809e) withr 2.1.2 2018-03-15 [1] CRAN (R 3.6.1) xfun 0.9 2019-08-21 [1] CRAN (R 3.6.1) xml2 * 1.2.2 2019-08-09 [1] CRAN (R 3.6.1) xts * 0.11-2 2018-11-05 [1] CRAN (R 3.6.1) zoo * 1.8-6 2019-05-28 [1] CRAN (R 3.6.1) [1] C:/Users/cklar/Desktop/R packages [2] C:/Program Files/R/R-3.6.1/library P -- Loaded and on-disk path mismatch. D -- DLL MD5 mismatch, broken installation. Christian Klar TFG Asset Management Tetragon Financial Management 399 Park Avenue, 22nd Floor | New York, NY 10022 | United States Direct: +1 212 359 7369 | Main: +1 212 359 7300 | Mobile: +1 607 216 5045 ck...@tetragoninv.com<mailto:ck...@tetragoninv.com> www.tetragoninv.com<http://www.tetragoninv.com> This communication and all or some of the information contained therein may be confidential. If you have received this communication in error, please destroy all electronic and paper copies and notify the sender immediately. Unless specifically indicated, this communication is not a confirmation, an offer to sell or solicitation of any offer to buy any financial product, or an official statement of Tetragon Financial Group or its affiliates. TFG Asset Management L.P. and Tetragon Financial Management LP are registered as investment advisers under the U.S. Investment Advisers Act of 1940.
RE: Issues with installation on docker / R
Perfect this seems to have worked (did a brief test on one .arrow file). Thanks a lot for all the help! Really appreciate it. Christian -Original Message- From: Wes McKinney [mailto:wesmck...@gmail.com] Sent: Sunday, December 1, 2019 9:16 PM To: dev Cc: Anthony Abate; Jason De Biasio Subject: Re: Issues with installation on docker / R OK, if you are installing using what is on CRAN you need to use the corresponding version of Apache Arrow, so please use the released 0.15.1 version instead of master which contains API changes On Sun, Dec 1, 2019 at 7:51 PM Christian Klar wrote: > > Apologies! I missed that line. > > Okay this worked (make install finishes with some warnings), and the R part > gets further, however I'm getting a filesystem error now after around 15 > steps. > > https://gist.github.com/klarchristian/2947eaf93c0e7ef9dd0d6ed9d0be990b > > Christian > > -Original Message- > From: Wes McKinney [mailto:wesmck...@gmail.com] > Sent: Sunday, December 1, 2019 8:14 PM > To: dev > Cc: Anthony Abate; Jason De Biasio > Subject: Re: Issues with installation on docker / R > > Building Thrift from source requires the "flex" and "bison" packages, > which you need to install with apt-get, this is mentioned here > > http://arrow.apache.org/docs/developers/cpp.html#apache-parquet-development > > On Sun, Dec 1, 2019 at 7:09 PM Christian Klar wrote: > > > > Okay I ran it. With the additional tag the make install errored out. > > > > Here's the link to the log. Please let me know if you need further data. > > > > https://gist.github.com/klarchristian/272d0a9bffa03d4ff2265e4e181a26ef > > > > Also one more point: The arrow-master folder is in a shared volume. I don't > > think that should be an issue but just wanted to point it out (first line > > in the log). > > > > > > > > > > > > -Original Message- > > From: Wes McKinney [mailto:wesmck...@gmail.com] > > Sent: Sunday, December 1, 2019 7:48 PM > > To: dev > > Cc: Anthony Abate; Jason De Biasio > > Subject: Re: Issues with installation on docker / R > > > > Can you also pass -DARROW_VERBOSE_THIRDPARTY_BUILD=ON ? That will show > > us what's wrong with with the Thrift build in this environment (I > > tried locally on my Ubuntu 18.04 machine -- without the influence of > > conda or any other library toolchains -- and it works fine). You can > > upload the whole logs on https://gist.github.com/ if you want > > > > On Sun, Dec 1, 2019 at 6:02 PM Christian Klar wrote: > > > > > > No luck with cmake -DARROW_PARQUET=ON either. Getting another error. > > > Please see below the log from the docker session. > > > > > > root@873cfc2bc7b1:/hello/arrow-master/cpp/release# cmake > > > -DARROW_PARQUET=ON .. > > > -- Building using CMake version: 3.7.2 > > > -- The C compiler identification is GNU 6.3.0 > > > -- The CXX compiler identification is GNU 6.3.0 > > > -- Check for working C compiler: /usr/bin/cc > > > -- Check for working C compiler: /usr/bin/cc -- works > > > -- Detecting C compiler ABI info > > > -- Detecting C compiler ABI info - done > > > -- Detecting C compile features > > > -- Detecting C compile features - done > > > -- Check for working CXX compiler: /usr/bin/c++ > > > -- Check for working CXX compiler: /usr/bin/c++ -- works > > > -- Detecting CXX compiler ABI info > > > -- Detecting CXX compiler ABI info - done > > > -- Detecting CXX compile features > > > -- Detecting CXX compile features - done > > > -- Arrow version: 1.0.0 (full: '1.0.0-SNAPSHOT') > > > -- Arrow SO version: 100 (full: 100.0.0) > > > -- Found PkgConfig: /usr/bin/pkg-config (found version "0.29") > > > -- clang-tidy not found > > > -- clang-format not found > > > -- infer not found > > > -- Found PythonInterp: /usr/bin/python (found version "2.7.13") > > > -- Found cpplint executable at > > > /hello/arrow-master/cpp/build-support/cpplint.py > > > -- Performing Test CXX_SUPPORTS_SSE4_2 > > > -- Performing Test CXX_SUPPORTS_SSE4_2 - Success > > > -- Performing Test CXX_SUPPORTS_ALTIVEC > > > -- Performing Test CXX_SUPPORTS_ALTIVEC - Failed > > > -- Performing Test CXX_SUPPORTS_ARMCRC > > > -- Performing Test CXX_SUPPORTS_ARMCRC - Failed > > > -- Performing Test CXX_SUPPORTS_ARMV8_CRC_CRYPTO > > > -- Performing Test CXX_SUPPORTS_ARMV8_CRC_CRYPTO - Failed > > > -- Arrow build warning level: PRODUCTION > > > Using ld link
RE: Issues with installation on docker / R
Apologies! I missed that line.
Okay this worked (make install finishes with some warnings), and the R part
gets further, however I'm getting a filesystem error now after around 15 steps.
https://gist.github.com/klarchristian/2947eaf93c0e7ef9dd0d6ed9d0be990b
Christian
-Original Message-
From: Wes McKinney [mailto:wesmck...@gmail.com]
Sent: Sunday, December 1, 2019 8:14 PM
To: dev
Cc: Anthony Abate; Jason De Biasio
Subject: Re: Issues with installation on docker / R
Building Thrift from source requires the "flex" and "bison" packages,
which you need to install with apt-get, this is mentioned here
http://arrow.apache.org/docs/developers/cpp.html#apache-parquet-development
On Sun, Dec 1, 2019 at 7:09 PM Christian Klar wrote:
>
> Okay I ran it. With the additional tag the make install errored out.
>
> Here's the link to the log. Please let me know if you need further data.
>
> https://gist.github.com/klarchristian/272d0a9bffa03d4ff2265e4e181a26ef
>
> Also one more point: The arrow-master folder is in a shared volume. I don't
> think that should be an issue but just wanted to point it out (first line in
> the log).
>
>
>
>
>
> -Original Message-
> From: Wes McKinney [mailto:wesmck...@gmail.com]
> Sent: Sunday, December 1, 2019 7:48 PM
> To: dev
> Cc: Anthony Abate; Jason De Biasio
> Subject: Re: Issues with installation on docker / R
>
> Can you also pass -DARROW_VERBOSE_THIRDPARTY_BUILD=ON ? That will show
> us what's wrong with with the Thrift build in this environment (I
> tried locally on my Ubuntu 18.04 machine -- without the influence of
> conda or any other library toolchains -- and it works fine). You can
> upload the whole logs on https://gist.github.com/ if you want
>
> On Sun, Dec 1, 2019 at 6:02 PM Christian Klar wrote:
> >
> > No luck with cmake -DARROW_PARQUET=ON either. Getting another error. Please
> > see below the log from the docker session.
> >
> > root@873cfc2bc7b1:/hello/arrow-master/cpp/release# cmake -DARROW_PARQUET=ON
> > ..
> > -- Building using CMake version: 3.7.2
> > -- The C compiler identification is GNU 6.3.0
> > -- The CXX compiler identification is GNU 6.3.0
> > -- Check for working C compiler: /usr/bin/cc
> > -- Check for working C compiler: /usr/bin/cc -- works
> > -- Detecting C compiler ABI info
> > -- Detecting C compiler ABI info - done
> > -- Detecting C compile features
> > -- Detecting C compile features - done
> > -- Check for working CXX compiler: /usr/bin/c++
> > -- Check for working CXX compiler: /usr/bin/c++ -- works
> > -- Detecting CXX compiler ABI info
> > -- Detecting CXX compiler ABI info - done
> > -- Detecting CXX compile features
> > -- Detecting CXX compile features - done
> > -- Arrow version: 1.0.0 (full: '1.0.0-SNAPSHOT')
> > -- Arrow SO version: 100 (full: 100.0.0)
> > -- Found PkgConfig: /usr/bin/pkg-config (found version "0.29")
> > -- clang-tidy not found
> > -- clang-format not found
> > -- infer not found
> > -- Found PythonInterp: /usr/bin/python (found version "2.7.13")
> > -- Found cpplint executable at
> > /hello/arrow-master/cpp/build-support/cpplint.py
> > -- Performing Test CXX_SUPPORTS_SSE4_2
> > -- Performing Test CXX_SUPPORTS_SSE4_2 - Success
> > -- Performing Test CXX_SUPPORTS_ALTIVEC
> > -- Performing Test CXX_SUPPORTS_ALTIVEC - Failed
> > -- Performing Test CXX_SUPPORTS_ARMCRC
> > -- Performing Test CXX_SUPPORTS_ARMCRC - Failed
> > -- Performing Test CXX_SUPPORTS_ARMV8_CRC_CRYPTO
> > -- Performing Test CXX_SUPPORTS_ARMV8_CRC_CRYPTO - Failed
> > -- Arrow build warning level: PRODUCTION
> > Using ld linker
> > Configured for RELEASE build (set with cmake
> > -DCMAKE_BUILD_TYPE={release,debug,...})
> > -- Build Type: RELEASE
> > -- Using AUTO approach to find dependencies
> > -- AWSSDK_VERSION: 1.7.160
> > -- BOOST_VERSION: 1.67.0
> > -- BROTLI_VERSION: v1.0.7
> > -- BZIP2_VERSION: 1.0.8
> > -- CARES_VERSION: 1.15.0
> > -- GBENCHMARK_VERSION: v1.5.0
> > -- GFLAGS_VERSION: v2.2.0
> > -- GLOG_VERSION: v0.3.5
> > -- GRPC_VERSION: v1.24.3
> > -- GTEST_VERSION: 1.8.1
> > -- JEMALLOC_VERSION: 5.2.1
> > -- LZ4_VERSION: v1.9.2
> > -- MIMALLOC_VERSION: 270e765454f98e8bab9d42609b153425f749fff6
> > -- ORC_VERSION: 1.5.7
> > -- PROTOBUF_VERSION: v3.7.1
> > -- RAPIDJSON_VERSION: 2bbd33b33217ff4a73434ebf10cdac41e2ef5e34
> > -- RE2_VERSION: 2019-08-01
> > -- SNAPPY_VERSION: 1.1.7
> > -- THRIFT_VERSION: 0.12.0
> > -- THRIFT_MD5_CHECKSUM: 3deebbb4d1ca77dd9c9e009a1ea02183
> > -- ZLIB_VERSION: 1.2.11
> >
RE: Issues with installation on docker / R
Okay I ran it. With the additional tag the make install errored out.
Here's the link to the log. Please let me know if you need further data.
https://gist.github.com/klarchristian/272d0a9bffa03d4ff2265e4e181a26ef
Also one more point: The arrow-master folder is in a shared volume. I don't
think that should be an issue but just wanted to point it out (first line in
the log).
-Original Message-
From: Wes McKinney [mailto:wesmck...@gmail.com]
Sent: Sunday, December 1, 2019 7:48 PM
To: dev
Cc: Anthony Abate; Jason De Biasio
Subject: Re: Issues with installation on docker / R
Can you also pass -DARROW_VERBOSE_THIRDPARTY_BUILD=ON ? That will show
us what's wrong with with the Thrift build in this environment (I
tried locally on my Ubuntu 18.04 machine -- without the influence of
conda or any other library toolchains -- and it works fine). You can
upload the whole logs on https://gist.github.com/ if you want
On Sun, Dec 1, 2019 at 6:02 PM Christian Klar wrote:
>
> No luck with cmake -DARROW_PARQUET=ON either. Getting another error. Please
> see below the log from the docker session.
>
> root@873cfc2bc7b1:/hello/arrow-master/cpp/release# cmake -DARROW_PARQUET=ON ..
> -- Building using CMake version: 3.7.2
> -- The C compiler identification is GNU 6.3.0
> -- The CXX compiler identification is GNU 6.3.0
> -- Check for working C compiler: /usr/bin/cc
> -- Check for working C compiler: /usr/bin/cc -- works
> -- Detecting C compiler ABI info
> -- Detecting C compiler ABI info - done
> -- Detecting C compile features
> -- Detecting C compile features - done
> -- Check for working CXX compiler: /usr/bin/c++
> -- Check for working CXX compiler: /usr/bin/c++ -- works
> -- Detecting CXX compiler ABI info
> -- Detecting CXX compiler ABI info - done
> -- Detecting CXX compile features
> -- Detecting CXX compile features - done
> -- Arrow version: 1.0.0 (full: '1.0.0-SNAPSHOT')
> -- Arrow SO version: 100 (full: 100.0.0)
> -- Found PkgConfig: /usr/bin/pkg-config (found version "0.29")
> -- clang-tidy not found
> -- clang-format not found
> -- infer not found
> -- Found PythonInterp: /usr/bin/python (found version "2.7.13")
> -- Found cpplint executable at
> /hello/arrow-master/cpp/build-support/cpplint.py
> -- Performing Test CXX_SUPPORTS_SSE4_2
> -- Performing Test CXX_SUPPORTS_SSE4_2 - Success
> -- Performing Test CXX_SUPPORTS_ALTIVEC
> -- Performing Test CXX_SUPPORTS_ALTIVEC - Failed
> -- Performing Test CXX_SUPPORTS_ARMCRC
> -- Performing Test CXX_SUPPORTS_ARMCRC - Failed
> -- Performing Test CXX_SUPPORTS_ARMV8_CRC_CRYPTO
> -- Performing Test CXX_SUPPORTS_ARMV8_CRC_CRYPTO - Failed
> -- Arrow build warning level: PRODUCTION
> Using ld linker
> Configured for RELEASE build (set with cmake
> -DCMAKE_BUILD_TYPE={release,debug,...})
> -- Build Type: RELEASE
> -- Using AUTO approach to find dependencies
> -- AWSSDK_VERSION: 1.7.160
> -- BOOST_VERSION: 1.67.0
> -- BROTLI_VERSION: v1.0.7
> -- BZIP2_VERSION: 1.0.8
> -- CARES_VERSION: 1.15.0
> -- GBENCHMARK_VERSION: v1.5.0
> -- GFLAGS_VERSION: v2.2.0
> -- GLOG_VERSION: v0.3.5
> -- GRPC_VERSION: v1.24.3
> -- GTEST_VERSION: 1.8.1
> -- JEMALLOC_VERSION: 5.2.1
> -- LZ4_VERSION: v1.9.2
> -- MIMALLOC_VERSION: 270e765454f98e8bab9d42609b153425f749fff6
> -- ORC_VERSION: 1.5.7
> -- PROTOBUF_VERSION: v3.7.1
> -- RAPIDJSON_VERSION: 2bbd33b33217ff4a73434ebf10cdac41e2ef5e34
> -- RE2_VERSION: 2019-08-01
> -- SNAPPY_VERSION: 1.1.7
> -- THRIFT_VERSION: 0.12.0
> -- THRIFT_MD5_CHECKSUM: 3deebbb4d1ca77dd9c9e009a1ea02183
> -- ZLIB_VERSION: 1.2.11
> -- ZSTD_VERSION: v1.4.3
> -- Looking for pthread.h
> -- Looking for pthread.h - found
> -- Looking for pthread_create
> -- Looking for pthread_create - not found
> -- Check if compiler accepts -pthread
> -- Check if compiler accepts -pthread - yes
> -- Found Threads: TRUE
> -- Boost version: 1.62.0
> -- Found the following Boost libraries:
> -- regex
> -- system
> -- filesystem
> -- Boost include dir: /usr/include
> -- Boost libraries: Boost::system;Boost::filesystem
> -- Building without OpenSSL support. Minimum OpenSSL version 1.0.2 required.
> -- Checking for module 'thrift'
> -- No package 'thrift' found
> -- Could NOT find Thrift (missing: THRIFT_STATIC_LIB THRIFT_INCLUDE_DIR
> THRIFT_COMPILER)
> Building Apache Thrift from source
> /hello/arrow-master/cpp/build-support/get_apache_mirror.py:46:
> RuntimeWarning: Failed loading
> 'https://www.apache.org/dyn/closer.cgi?as_json=1': CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)>
> RuntimeWarning)
> Downloading Apache Thrift from
> http://apache.osuosl.org//thrift/0.12.0/thrift-0.12.0.tar.gz
> -- Building (vendored) jemalloc from source
> -- Could NOT f
RE: Issues with installation on docker / R
ift_ep-prefix/src/thrift_ep-stamp/thrift_ep-configure-RELEASE.cmake:16
(message):
Command failed: 1
'/usr/bin/cmake' '-DZLIB_LIBRARY=' '-DCMAKE_C_COMPILER=/usr/bin/cc'
'-DCMAKE_CXX_COMPILER=/usr/bin/c++' '-DCMAKE_AR=/usr/bin/ar'
'-DCMAKE_RANLIB=/usr/bin/ranlib' '-DCMAKE_BUILD_TYPE=RELEASE' '-DCMAKE_C_FLAGS=
-O3 -DNDEBUG -O3 -DNDEBUG -fPIC' '-DCMAKE_C_FLAGS_RELEASE= -O3 -DNDEBUG -O3
-DNDEBUG -fPIC' '-DCMAKE_CXX_FLAGS= -fdiagnostics-color=always -O3 -DNDEBUG -O3
-DNDEBUG -fPIC' '-DCMAKE_CXX_FLAGS_RELEASE= -fdiagnostics-color=always -O3
-DNDEBUG -O3 -DNDEBUG -fPIC'
'-DCMAKE_INSTALL_PREFIX=/hello/arrow-master/cpp/release/thrift_ep/src/thrift_ep-install'
'-DCMAKE_INSTALL_RPATH=/hello/arrow-master/cpp/release/thrift_ep/src/thrift_ep-install/lib'
'-DBUILD_SHARED_LIBS=OFF' '-DBUILD_TESTING=OFF' '-DBUILD_EXAMPLES=OFF'
'-DBUILD_TUTORIALS=OFF' '-DWITH_QT4=OFF' '-DWITH_C_GLIB=OFF' '-DWITH_JAVA=OFF'
'-DWITH_PYTHON=OFF' '-DWITH_HASKELL=OFF' '-DWITH_CPP=ON' '-DWITH_STATIC_LIB=ON'
'-DWITH_LIBEVENT=OFF' '-DBoost_NO_BOOST_CMAKE=ON' '-DBoost_NAMESPACE=boost'
'-GUnix Makefiles'
'/hello/arrow-master/cpp/release/thrift_ep-prefix/src/thrift_ep'
See also
/hello/arrow-master/cpp/release/thrift_ep-prefix/src/thrift_ep-stamp/thrift_ep-configure-*.log
CMakeFiles/thrift_ep.dir/build.make:106: recipe for target
'thrift_ep-prefix/src/thrift_ep-stamp/thrift_ep-configure' failed
make[2]: *** [thrift_ep-prefix/src/thrift_ep-stamp/thrift_ep-configure] Error 1
CMakeFiles/Makefile2:168: recipe for target 'CMakeFiles/thrift_ep.dir/all'
failed
make[1]: *** [CMakeFiles/thrift_ep.dir/all] Error 2
Makefile:138: recipe for target 'all' failed
make: *** [all] Error 2
root@873cfc2bc7b1:/hello/arrow-master/cpp/release#
-Original Message-
From: Christian Klar
Sent: Sunday, December 1, 2019 6:55 PM
To: dev@arrow.apache.org
Cc: Anthony Abate; Jason De Biasio
Subject: RE: Issues with installation on docker / R
Thanks!
When running "make install" it finishes fine too, but then the R installation
errors out. See below.
Are there any other libraries I need to install?
(Would potentially "cmake -DARROW_PARQUET=ON" solve this?)
Here the log.
> install.packages('arrow')
Installing package into ‘/usr/local/lib/R/site-library’
(as ‘lib’ is unspecified)
trying URL
'https://mran.microsoft.com/snapshot/2019-11-17/src/contrib/arrow_0.15.1.1.tar.gz'
Content type 'application/octet-stream' length 147277 bytes (143 KB)
==
downloaded 143 KB
* installing *source* package ‘arrow’ ...
** package ‘arrow’ successfully unpacked and MD5 sums checked
** using staged installation
PKG_CFLAGS= -DARROW_R_WITH_ARROW
PKG_LIBS=-larrow -lparquet
** libs
g++ -std=gnu++11 -I"/usr/local/lib/R/include" -DNDEBUG -DARROW_R_WITH_ARROW
-I"/usr/local/lib/R/site-library/Rcpp/include" -I/usr/local/include -fpic -g
-O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time
-D_FORTIFY_SOURCE=2 -g -c array.cpp -o array.o
In file included from array.cpp:18:0:
./arrow_types.h:198:34: fatal error: parquet/arrow/reader.h: No such file or
directory
#include
^
compilation terminated.
/usr/local/lib/R/etc/Makeconf:176: recipe for target 'array.o' failed
make: *** [array.o] Error 1
ERROR: compilation failed for package ‘arrow’
* removing ‘/usr/local/lib/R/site-library/arrow’
* restoring previous ‘/usr/local/lib/R/site-library/arrow’
The downloaded source packages are in
‘/tmp/Rtmp2POLjt/downloaded_packages’
Warning message:
In install.packages("arrow") :
installation of package ‘arrow’ had non-zero exit status
Here the docker image again.
FROM rocker/tidyverse
MAINTAINER Christian Klar
ENV DEBIAN_FRONTEND noninteractive
RUN apt-get install -y -V \
build-essential \
cmake \
libboost-filesystem-dev \
libboost-regex-dev \
libboost-system-dev
RUN apt update && \
apt install -y -V \
autoconf-archive \
gtk-doc-tools \
libgirepository1.0-dev \
libglib2.0-doc \
libtool \
pkg-config && \
apt clean && \
rm -rf /var/lib/apt/lists/*
RUN R -e "install.packages('remotes', repos = c(CRAN =
'http://cran.us.r-project.org'))"
-Original Message-
From: Wes McKinney [mailto:wesmck...@gmail.com]
Sent: Sunday, December 1, 2019 6:34 PM
To: dev@arrow.apache.org
Cc: Anthony Abate; Jason De Biasio
Subject: Re: Issues with installation on docker / R
That is a “minimal release build” but it does not install the results. Try
“make install” instead of just “make”. We can improve the documentation to
clarify this issue
On Sun, Dec 1, 2019 at 5:22 PM Christian Klar wrote:
>
> Thanks for the quick response!
>
> I attached the log to my first email but maybe it didn't got through
> because of the mailing list. Copying it below below the
> "##
RE: Issues with installation on docker / R
Thanks!
When running "make install" it finishes fine too, but then the R installation
errors out. See below.
Are there any other libraries I need to install?
(Would potentially "cmake -DARROW_PARQUET=ON" solve this?)
Here the log.
> install.packages('arrow')
Installing package into ‘/usr/local/lib/R/site-library’
(as ‘lib’ is unspecified)
trying URL
'https://mran.microsoft.com/snapshot/2019-11-17/src/contrib/arrow_0.15.1.1.tar.gz'
Content type 'application/octet-stream' length 147277 bytes (143 KB)
==
downloaded 143 KB
* installing *source* package ‘arrow’ ...
** package ‘arrow’ successfully unpacked and MD5 sums checked
** using staged installation
PKG_CFLAGS= -DARROW_R_WITH_ARROW
PKG_LIBS=-larrow -lparquet
** libs
g++ -std=gnu++11 -I"/usr/local/lib/R/include" -DNDEBUG -DARROW_R_WITH_ARROW
-I"/usr/local/lib/R/site-library/Rcpp/include" -I/usr/local/include -fpic -g
-O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time
-D_FORTIFY_SOURCE=2 -g -c array.cpp -o array.o
In file included from array.cpp:18:0:
./arrow_types.h:198:34: fatal error: parquet/arrow/reader.h: No such file or
directory
#include
^
compilation terminated.
/usr/local/lib/R/etc/Makeconf:176: recipe for target 'array.o' failed
make: *** [array.o] Error 1
ERROR: compilation failed for package ‘arrow’
* removing ‘/usr/local/lib/R/site-library/arrow’
* restoring previous ‘/usr/local/lib/R/site-library/arrow’
The downloaded source packages are in
‘/tmp/Rtmp2POLjt/downloaded_packages’
Warning message:
In install.packages("arrow") :
installation of package ‘arrow’ had non-zero exit status
Here the docker image again.
FROM rocker/tidyverse
MAINTAINER Christian Klar
ENV DEBIAN_FRONTEND noninteractive
RUN apt-get install -y -V \
build-essential \
cmake \
libboost-filesystem-dev \
libboost-regex-dev \
libboost-system-dev
RUN apt update && \
apt install -y -V \
autoconf-archive \
gtk-doc-tools \
libgirepository1.0-dev \
libglib2.0-doc \
libtool \
pkg-config && \
apt clean && \
rm -rf /var/lib/apt/lists/*
RUN R -e "install.packages('remotes', repos = c(CRAN =
'http://cran.us.r-project.org'))"
-Original Message-
From: Wes McKinney [mailto:wesmck...@gmail.com]
Sent: Sunday, December 1, 2019 6:34 PM
To: dev@arrow.apache.org
Cc: Anthony Abate; Jason De Biasio
Subject: Re: Issues with installation on docker / R
That is a “minimal release build” but it does not install the results. Try
“make install” instead of just “make”. We can improve the documentation to
clarify this issue
On Sun, Dec 1, 2019 at 5:22 PM Christian Klar wrote:
>
> Thanks for the quick response!
>
> I attached the log to my first email but maybe it didn't got through
> because of the mailing list. Copying it below below the
> "#"-line.
>
> I'm using the commands from the "Minimal release build" on
> https://arrow.apache.org/docs/developers/cpp.html.
>
> cd arrow/cpp
> mkdir release
> cd release
> cmake ..
> make
>
> The corresponding line below for the last make is here copied separately.
>
> "root@66311c2b2dc5:/hello/arrow-master/cpp/release# make"
>
> Please let me know if that's not the one you meant.
>
> #
>
> cklar@nyc-poly-tci-03:~/Desktop/Docker/arrow$ docker run -v
> /home/cklar/Desktop/tmpshare:/hello -it arrowtest /bin/bash
> root@66311c2b2dc5:/# cd hello
> root@66311c2b2dc5:/hello# cd hell^C
> root@66311c2b2dc5:/hello# ls
> arrow-master arrow-master.zip CLOPROCESS_TrancheHistory_Expense.arrow
> root@66311c2b2dc5:/hello# cd arrow-master
> root@66311c2b2dc5:/hello/arrow-master# ls
> appveyor.yml cmake-format.py csharp format java
> matlab README.mdtesting
> c_glibCODE_OF_CONDUCT.md dev go js
> NOTICE.txt ruby
> CHANGELOG.md CONTRIBUTING.md docker-compose.yml header
> LICENSE.txt python run-cmake-format.py
> cicpp docsintegration
> Makefile.docker r rust
> root@66311c2b2dc5:/hello/arrow-master# cd cpp
> root@66311c2b2dc5:/hello/arrow-master/cpp# mkdir release
> root@66311c2b2dc5:/hello/arrow-master/cpp# cd release
> root@66311c2b2dc5:/hello/arrow-master/cpp/release# cmake
> Usage
>
> cmake [options]
> cmake [options]
>
> Specify a source directory to (re-)generate a build system for it in the
> current working directory. Specify an existing build directory to
> re-generate its build system.
>
> Run 'cmake --help' for more information.
>
> root@66311c2b2dc
RE: Issues with installation on docker / R
from final location
** testing if installed package keeps a record of temporary installation path
* DONE (arrow)
The downloaded source packages are in
‘/tmp/RtmpFQeIkA/downloaded_packages’
> arrow::install_arrow()
See the Arrow C++ developer guide
<https://arrow.apache.org/docs/developers/cpp.html> for instructions on
building the library from source.
After you've installed the C++ library, you'll need to reinstall the R package
from source to find it.
Refer to the R package README
<https://github.com/apache/arrow/blob/master/r/README.md> for further details.
If you have other trouble, or if you think this message could be improved,
please report an issue here:
<https://issues.apache.org/jira/projects/ARROW/issues>
> options(arrow.use_threads=FALSE)
> r <- arrow::read_arrow('/hello/CLOPROCESS_TrancheHistory_Expense.arrow')
Error in io___ReadableFile__Open(clean_path_abs(path)) :
Cannot call io___ReadableFile__Open(). Please use arrow::install_arrow() to
install required runtime libraries.
> q()
root@66311c2b2dc5:/hello/arrow-master/cpp/release# ls
CMakeCache.txt cmake_install.cmake CTestTestfile.cmake Makefile release
tmp
CMakeFiles cmake_summary.json jemalloc_ep-prefix rapidjson_ep src
root@66311c2b2dc5:/hello/arrow-master/cpp/release# R
R version 3.6.1 (2019-07-05) -- "Action of the Toes"
Copyright (C) 2019 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)
R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.
R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.
Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.
> arrow::arrow_available()
FALSE
Christian Klar
TFG Asset Management
Tetragon Financial Management
399 Park Avenue, 22nd Floor | New York, NY 10022 | United States
Direct: +1 212 359 7369 | Main: +1 212 359 7300 | Mobile: +1 607 216 5045
ck...@tetragoninv.com
www.tetragoninv.com
This communication and all or some of the information contained therein may be
confidential. If you have received this communication in error, please destroy
all electronic and paper copies and notify the sender immediately. Unless
specifically indicated, this communication is not a confirmation, an offer to
sell or solicitation of any offer to buy any financial product, or an official
statement of Tetragon Financial Group or its affiliates. TFG Asset Management
L.P. and Tetragon Financial Management LP are registered as investment advisers
under the U.S. Investment Advisers Act of 1940.
-Original Message-
From: Wes McKinney [mailto:wesmck...@gmail.com]
Sent: Sunday, December 1, 2019 6:12 PM
To: dev@arrow.apache.org
Cc: Anthony Abate; Jason De Biasio
Subject: Re: Issues with installation on docker / R
Your instructions do not include “make install” when building the C++
library which will cause this problem. Can you provide a full build log if
you are actually installing the C++ library and the R build cannot find it
for some reason?
Thanks
On Sun, Dec 1, 2019 at 4:55 PM Christian Klar wrote:
> Hi –
>
>
>
> (Resending because I was not subscribed to the mailing list prior.)
>
>
>
> I’m trying to install arrow on a docker image with R on it but R gives me
> always the
>
>
>
> “error in io___ReadableFile__Open(clean_path_abs(path)) :
>
> Cannot call io___ReadableFile__Open(). Please use arrow::install_arrow()
> to install required runtime libraries.”
>
>
>
> error.
>
>
>
>
>
> Below the details:
>
>
>
>
>
> 1. The docker image.
>
>
>
> FROM rocker/tidyverse
>
> ENV DEBIAN_FRONTEND noninteractive
>
> RUN apt-get install -y -V \
>
> build-essential \
>
> cmake \
>
> libboost-filesystem-dev \
>
> libboost-regex-dev \
>
> libboost-system-dev
>
> RUN apt update && \
>
> apt install -y -V \
>
> autoconf-archive \
>
> gtk-doc-tools \
>
> libgirepository1.0-dev \
>
> libglib2.0-doc \
>
> libtool \
>
> pkg-config && \
>
> apt clean && \
>
> rm -rf /var/lib/apt/lists/*
>
>
>
>
>
> 2. Then within an interactive session (interactive so that I can
> debug easier) I run the actual cmake (I downloaded the arrow package
> manually). This is straight from
> https://arrow.apache.org/docs/developers/cpp.html.
>
>
>
> cd arrow/cpp
>
> mkdir release
>
> cd release
>
> cmake ..
>
> make
>
>
&
Issues with installation on docker / R
Hi – (Resending because I was not subscribed to the mailing list prior.) I’m trying to install arrow on a docker image with R on it but R gives me always the “error in io___ReadableFile__Open(clean_path_abs(path)) : Cannot call io___ReadableFile__Open(). Please use arrow::install_arrow() to install required runtime libraries.” error. Below the details: 1. The docker image. FROM rocker/tidyverse ENV DEBIAN_FRONTEND noninteractive RUN apt-get install -y -V \ build-essential \ cmake \ libboost-filesystem-dev \ libboost-regex-dev \ libboost-system-dev RUN apt update && \ apt install -y -V \ autoconf-archive \ gtk-doc-tools \ libgirepository1.0-dev \ libglib2.0-doc \ libtool \ pkg-config && \ apt clean && \ rm -rf /var/lib/apt/lists/* 2. Then within an interactive session (interactive so that I can debug easier) I run the actual cmake (I downloaded the arrow package manually). This is straight from https://arrow.apache.org/docs/developers/cpp.html. cd arrow/cpp mkdir release cd release cmake .. make 3. After number 2 finishing successfully I start R and try “install.packages(‘arrow’)” which succeeds (builds it from source), but then I get back to the error above. 4. install_arrow() just directs me to the link in number 2. I have the feeling the R package installation can’t find the path with the arrow installation, however I can’t find any environment variable that would handle this. Please let us know. At this point we spent many hours trying to get it to work – but this has to be an issue people have come across. I’m not using conda so this (the only thing comparable that I could find) is not applicable to me: https://github.com/apache/arrow/issues/4399 Happy to provide further details. I also attached the full console log. Christian Christian Klar TFG Asset Management Tetragon Financial Management 399 Park Avenue, 22nd Floor | New York, NY 10022 | United States Direct: +1 212 359 7369 | Main: +1 212 359 7300 | Mobile: +1 607 216 5045 ck...@tetragoninv.com<mailto:ck...@tetragoninv.com> www.tetragoninv.com<http://www.tetragoninv.com> This communication and all or some of the information contained therein may be confidential. If you have received this communication in error, please destroy all electronic and paper copies and notify the sender immediately. Unless specifically indicated, this communication is not a confirmation, an offer to sell or solicitation of any offer to buy any financial product, or an official statement of Tetragon Financial Group or its affiliates. TFG Asset Management L.P. and Tetragon Financial Management LP are registered as investment advisers under the U.S. Investment Advisers Act of 1940. cklar@nyc-poly-tci-03:~/Desktop/Docker/arrow$ docker run -v /home/cklar/Desktop/tmpshare:/hello -it arrowtest /bin/bash root@66311c2b2dc5:/# cd hello root@66311c2b2dc5:/hello# cd hell^C root@66311c2b2dc5:/hello# ls arrow-master arrow-master.zip CLOPROCESS_TrancheHistory_Expense.arrow root@66311c2b2dc5:/hello# cd arrow-master root@66311c2b2dc5:/hello/arrow-master# ls appveyor.yml cmake-format.py csharp format java matlab README.mdtesting c_glibCODE_OF_CONDUCT.md dev go js NOTICE.txt ruby CHANGELOG.md CONTRIBUTING.md docker-compose.yml header LICENSE.txt python run-cmake-format.py cicpp docsintegration Makefile.docker r rust root@66311c2b2dc5:/hello/arrow-master# cd cpp root@66311c2b2dc5:/hello/arrow-master/cpp# mkdir release root@66311c2b2dc5:/hello/arrow-master/cpp# cd release root@66311c2b2dc5:/hello/arrow-master/cpp/release# cmake Usage cmake [options] cmake [options] Specify a source directory to (re-)generate a build system for it in the current working directory. Specify an existing build directory to re-generate its build system. Run 'cmake --help' for more information. root@66311c2b2dc5:/hello/arrow-master/cpp/release# cmake .. -- Building using CMake version: 3.7.2 -- The C compiler identification is GNU 6.3.0 -- The CXX compiler identification is GNU 6.3.0 -- Check for working C compiler: /usr/bin/cc -- Check for working C compiler: /usr/bin/cc -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Detecting C compile features -- Detecting C compile features - done -- Check for working CXX compiler: /usr/bin/c++ -- Check for working CXX compiler: /usr/bin/c++ -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Detecting CXX compile features -- Detecting CXX compile features - done -- Arrow version: 1.0.0 (full: '1.0.0-SNAPSHOT') -- Arrow SO version: 100 (full: 100.0.0) -- Found PkgConfig: /usr/bin/pkg-config (found version "0.29") -- clang-tid
