thisisnic commented on a change in pull request #10930: URL: https://github.com/apache/arrow/pull/10930#discussion_r696079362
########## File path: r/vignettes/developing.Rmd ########## @@ -483,60 +434,156 @@ variables or other settings: * All tests are skipped on Linux if the package builds without the C++ libarrow. To make the build fail if libarrow is not available (as in, to test that the C++ build was successful), set `TEST_R_WITH_ARROW=true` + * Some tests are disabled unless `ARROW_R_DEV=true` + * Tests that require allocating >2GB of memory to test Large types are disabled unless `ARROW_LARGE_MEMORY_TESTS=true` + * Integration tests against a real S3 bucket are disabled unless credentials are set in `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`; these are available on request + * S3 tests using [MinIO](https://min.io/) locally are enabled if the `minio server` process is found running. If you're running MinIO with custom settings, you can set `MINIO_ACCESS_KEY`, `MINIO_SECRET_KEY`, and `MINIO_PORT` to override the defaults. -## Github workflows +## Running checks + +You can run package checks by using `devtools::check()` and check test coverage with `covr::package_coverage()`. + +```r +# All package checks +devtools::check() + +# See test coverage statistics +covr::report() +covr::package_coverage() + +``` + +For full package validation, you can run the following commands from a terminal. + +``` +R CMD build . +R CMD check arrow_*.tar.gz --as-cran +``` + + +## Running additional CI checks On a pull request, there are some actions you can trigger by commenting on the PR. We have additional CI checks that run nightly and can be requested on demand using an internal tool called [crosssbow](https://arrow.apache.org/docs/developers/crossbow.html). A few important GitHub comment commands include: -* `@github-actions crossbow submit -g r` for all extended R CI tests -* `@github-actions crossbow submit {task-name}` for running a specific task. See the `r:` group definition near the beginning of the [crossbow configuration](https://github.com/apache/arrow/blob/master/dev/tasks/tasks.yml) for a list of glob expression patterns that match names of items in the `tasks:` list below it. -* `@github-actions autotune` will run and fix lint c++ linting errors + run R documentation (among other cleanup tasks) and commit them to the branch +#### Run all extended R CI tasks +`@github-actions crossbow submit -g r` +This runs each of the R-related CI tasks. -## Useful functions for Arrow developers +#### Run a specific task +`@github-actions crossbow submit {task-name}` -Within an R session, these can help with package development: +See the `r:` group definition near the beginning of the [crossbow configuration](https://github.com/apache/arrow/blob/master/dev/tasks/tasks.yml) for a list of glob expression patterns that match names of items in the `tasks:` list below it. -``` r -# Load the dev package -devtools::load_all() +#### Run linting and documentation building tasks -# Run the test suite, optionally filtering file names -devtools::test(filter="^regexp$") -# or the Makefile alternative from the arrow/r directory in a shell: -make test file=regexp +`@github-actions autotune` -# Update roxygen documentation -devtools::document() +This will run and fix lint C++ linting errors, run R documentation (among other cleanup tasks), and commit the resulting updates to the branch. -# To preview the documentation website -pkgdown::build_site() +# Troubleshooting -# All package checks; see also below -devtools::check() +Note that after any change to the C++ library, you must reinstall it and +run `make clean` or `git clean -fdx .` to remove any cached object code +in the `r/src/` directory before reinstalling the R package. This is +only necessary if you make changes to the C++ library source; you do not +need to manually purge object files if you are only editing R or C++ +code inside `r/`. -# See test coverage statistics -covr::report() -covr::package_coverage() +## Arrow library-R package mismatches + +If the Arrow library and the R package have diverged, you will see errors like: + +``` +Error: package or namespace load failed for ‘arrow' in dyn.load(file, DLLpath = DLLpath, ...): + unable to load shared object '/Library/Frameworks/R.framework/Versions/4.0/Resources/library/00LOCK-r/00new/arrow/libs/arrow.so': + dlopen(/Library/Frameworks/R.framework/Versions/4.0/Resources/library/00LOCK-r/00new/arrow/libs/arrow.so, 6): Symbol not found: __ZN5arrow2io16RandomAccessFile9ReadAsyncERKNS0_9IOContextExx + Referenced from: /Library/Frameworks/R.framework/Versions/4.0/Resources/library/00LOCK-r/00new/arrow/libs/arrow.so + Expected in: flat namespace + in /Library/Frameworks/R.framework/Versions/4.0/Resources/library/00LOCK-r/00new/arrow/libs/arrow.so +Error: loading failed +Execution halted +ERROR: loading failed ``` -Any of those can be run from the command line by wrapping them in `R -e -'$COMMAND'`. There’s also a `Makefile` to help with some common tasks -from the command line (`make test`, `make doc`, `make clean`, etc.) +To resolve this, try rebuilding the Arrow library from [Building Arrow above](#step-3-building-arrow). -### Full package validation +## Multiple versions of Arrow library -``` shell -R CMD build . -R CMD check arrow_*.tar.gz --as-cran +If rebuilding the Arrow library doesn't work and you are [installing from a user-level directory](#configure-for-installing-to-a-user-directory) and you already have a previous installation of libarrow in a system directory or you get you may get errors like the following when you install the R package: + +``` +Error: package or namespace load failed for ‘arrow' in dyn.load(file, DLLpath = DLLpath, ...): + unable to load shared object '/Library/Frameworks/R.framework/Versions/4.0/Resources/library/00LOCK-r/00new/arrow/libs/arrow.so': + dlopen(/Library/Frameworks/R.framework/Versions/4.0/Resources/library/00LOCK-r/00new/arrow/libs/arrow.so, 6): Library not loaded: /usr/local/lib/libarrow.400.dylib + Referenced from: /usr/local/lib/libparquet.400.dylib + Reason: image not found ``` + +You need to make sure that you don't let R link to your system library when building arrow. You can do this a number of different ways: + +* Setting the `MAKEFLAGS` environment variable to `"LDFLAGS="` (see below for an example) this is the recommended way to accomplish this +* Using {withr}'s `with_makevars(list(LDFLAGS = ""), ...)` +* adding `LDFLAGS=` to your `~/.R/Makevars` file (the least recommended way, though it is a common debugging approach suggested online) + +```{bash, save=run & !sys_install & macos, hide=TRUE} +# Setup troubleshooting section +# install a system-level arrow on macOS +brew install apache-arrow +``` + + +```{bash, save=run & !sys_install & ubuntu, hide=TRUE} +# Setup troubleshooting section +# install a system-level arrow on Ubuntu +sudo apt update +sudo apt install -y -V ca-certificates lsb-release wget +wget https://apache.jfrog.io/artifactory/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb +sudo apt install -y -V ./apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb +sudo apt update +sudo apt install -y -V libarrow-dev +``` + +```{bash, save=run & !sys_install & macos} +MAKEFLAGS="LDFLAGS=" R CMD INSTALL . +``` + + +## `rpath` issues + +If the package fails to install/load with an error like this: + +``` + ** testing if installed package can be loaded from temporary location + Error: package or namespace load failed for 'arrow' in dyn.load(file, DLLpath = DLLpath, ...): + unable to load shared object '/Users/you/R/00LOCK-r/00new/arrow/libs/arrow.so': + dlopen(/Users/you/R/00LOCK-r/00new/arrow/libs/arrow.so, 6): Library not loaded: @rpath/libarrow.14.dylib +``` + +ensure that `-DARROW_INSTALL_NAME_RPATH=OFF` was passed (this is important on +macOS to prevent problems at link time and is a no-op on other platforms). +Alternatively, try setting the environment variable `R_LD_LIBRARY_PATH` to +wherever Arrow C++ was put in `make install`, e.g. `export +R_LD_LIBRARY_PATH=/usr/local/lib`, and retry installing the R package. + +When installing from source, if the R and C++ library versions do not +match, installation may fail. If you've previously installed the +libraries and want to upgrade the R package, you'll need to update the +Arrow C++ library first. + +For any other build/configuration challenges, see the [C++ developer +guide](https://arrow.apache.org/docs/developers/cpp/building.html). + +## Other installation issues + +There are a number of scripts that are triggered when the arrow R package is installed. For package users who are not interacting with the underlying code, these should all just work without configuration and pull in the most complete pieces (e.g. official binaries that we host). However, knowing about these scripts can help package developers troubleshoot if things go wrong in them or things go wrong in an install. See [the installation vignette](./install.html) for more information. Review comment: Done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
