thisisnic commented on a change in pull request #11705:
URL: https://github.com/apache/arrow/pull/11705#discussion_r751990024
##########
File path: r/vignettes/developers/setup.Rmd
##########
@@ -0,0 +1,410 @@
+# Developer environment setup
+
+```{r setup-options, include=FALSE}
+knitr::opts_chunk$set(error = TRUE, eval = FALSE)
+# Get environment variables describing what to evaluate
+run <- tolower(Sys.getenv("RUN_DEVDOCS", "false")) == "true"
+macos <- tolower(Sys.getenv("DEVDOCS_MACOS", "false")) == "true"
+ubuntu <- tolower(Sys.getenv("DEVDOCS_UBUNTU", "false")) == "true"
+sys_install <- tolower(Sys.getenv("DEVDOCS_SYSTEM_INSTALL", "false")) == "true"
+# Update the source knit_hook to save the chunk (if it is marked to be saved)
+knit_hooks_source <- knitr::knit_hooks$get("source")
+knitr::knit_hooks$set(source = function(x, options) {
+ # Extra paranoia about when this will write the chunks to the script, we will
+ # only save when:
+ # * CI is true
+ # * RUN_DEVDOCS is true
+ # * options$save is TRUE (and a check that not NULL won't crash it)
+ if (as.logical(Sys.getenv("CI", FALSE)) && run && !is.null(options$save) &&
options$save)
+ cat(x, file = "script.sh", append = TRUE, sep = "\n")
+ # but hide the blocks we want hidden:
+ if (!is.null(options$hide) && options$hide) {
+ return(NULL)
+ }
+ knit_hooks_source(x, options)
+})
+```
+
+```{bash, save=run, hide=TRUE}
+# Stop on failure, echo input as we go
+set -e
+set -x
+```
+
+This document is intended only for **developers** of Apache Arrow or
+the Arrow R package. R package users do not need to do any of this setup.
+If you're looking for how to install Arrow, see
+[the instructions in the
readme](https://arrow.apache.org/docs/r/#installation).
+
+This document is a work in progress and will grow and change as the Apache
Arrow
+project grows and changes. We have tried to make these steps as robust as
+possible (in fact, we even test exactly these instructions on our nightly CI
to
+ensure they don't become stale!), but custom configurations might conflict
with
+these instructions and there are differences of opinion across developers
about
+how to set up development environments like this.
+
+We welcome any feedback you have about things that are confusing or additions
+you would like to see here - please
+[report an issue](https://issues.apache.org/jira/projects/ARROW/issues) if you
+have any suggestions or requests.
+
+
+## R-only {.tabset}
+
+Windows and macOS users who wish to contribute to the R package and
+don't need to alter libarrow (Arrow's C++ library) may be able to obtain a
+recent version of the library without building from source.
+
+### Linux
+
+On Linux, you can download a .zip file containing libarrow from the
+nightly repository.
+
+To see what nightlies are available, you can use arrow's (or any other S3
client's) S3 listing functionality to see what is in the bucket
`s3://arrow-r-nightly/libarrow/bin`:
+
+```
+nightly <- s3_bucket("arrow-r-nightly")
+nightly$ls("libarrow/bin")
+```
+Version numbers in that repository correspond to dates.
+
+You'll need to create a `libarrow` directory inside the R package directory
and unzip the zip file containing the compiled libarrow binary files into it.
+
+### macOS
+On macOS, you can install libarrow using [Homebrew](https://brew.sh/):
+
+```bash
+# For the released version:
+brew install apache-arrow
+# Or for a development version, you can try:
+brew install apache-arrow --HEAD
+```
+
+### Windows
+
+On Windows, you can download a .zip file containing libarrow from the nightly
repository.
+
+To see what nightlies are available, you can use arrow's (or any other S3
client's) S3 listing functionality to see what is in the bucket
`s3://arrow-r-nightly/libarrow/bin`:
+
+```
+nightly <- s3_bucket("arrow-r-nightly")
+nightly$ls("libarrow/bin")
+```
+Version numbers in that repository correspond to dates.
+
+You can set the `RWINLIB_LOCAL` environment variable to point to the zip file
containing libarrow before installing the arrow R package.
+
+
+## R and C++
+
+If you need to alter both libarrow and the R package code, or if you can't get
a binary version of the latest libarrow elsewhere, you'll need to build it from
source. This section discusses how to set up a C++ libarrow build configured to
work with the R package. For more general resources, see the [Arrow C++
developer guide](https://arrow.apache.org/docs/developers/cpp/building.html).
+
+There are five major steps to the process.
+
+### Step 1 - Install dependencies {.tabset}
+
+When building libarrow, by default, system dependencies will be used if
suitable versions are found. If system dependencies are not present, libarrow
will build them during its own build process. The only dependencies that you
need to install _outside_ of the build process are [cmake](https://cmake.org/)
(for configuring the build) and [openssl](https://www.openssl.org/) if you are
building with S3 support.
+
+For a faster build, you may choose to pre-install more C++ library
dependencies (such as [lz4](http://lz4.github.io/lz4/),
[zstd](https://facebook.github.io/zstd/), etc.) on the system so that they
don't need to be built from source in the libarrow build.
+
+#### Ubuntu
+```{bash, save=run & ubuntu}
+sudo apt install -y cmake libcurl4-openssl-dev libssl-dev
+```
+
+#### macOS
+```{bash, save=run & macos}
+brew install cmake openssl
+```
+
+#### Windows
+
+Currently, the R package cannot be made to work with a local libarrow build.
This will be resolved in a future release.
+
+### Step 2 - Configure the libarrow build
+
+We recommend that you configure libarrow to be built to a user-level directory
rather than a system directory for your development work. This is so that the
development version you are using doesn't overwrite a released version of
libarrow you may already have installed, and so that you are also able work
with more than one version of libarrow (by using different `ARROW_HOME`
directories for the different versions).
+
+In the example below, libarrow is installed to a directory called `dist` that
has the same parent directory as the `arrow` checkout. Your installation of
the Arrow R package can point to any directory with any name, though we
recommend *not* placing it inside of the `arrow` git checkout directory as
unwanted changes could stop it working properly.
+
+```{bash, save=run & !sys_install}
+export ARROW_HOME=$(pwd)/dist
+mkdir $ARROW_HOME
+```
+
+_Special instructions on Linux:_ You will need to set `LD_LIBRARY_PATH` to the
`lib` directory that is under where you set `$ARROW_HOME`, before launching R
and using arrow. One way to do this is to add it to your profile (we use
`~/.bash_profile` here, but you might need to put this in a different file
depending on your setup, e.g. if you use a shell other than `bash`). On macOS
you do not need to do this because the macOS shared library paths are hardcoded
to their locations during build time.
+
+```{bash, save=run & ubuntu & !sys_install}
+export LD_LIBRARY_PATH=$ARROW_HOME/lib:$LD_LIBRARY_PATH
+echo "export LD_LIBRARY_PATH=$ARROW_HOME/lib:$LD_LIBRARY_PATH" >>
~/.bash_profile
+```
+
+Start by navigating in a terminal to the `arrow` repository. You will need to
create a directory into which the C++ build will put its contents. We recommend
that you make a `build` directory inside of the `cpp` directory of the Arrow
git repository (it is git-ignored, so you won't accidentally check it in).
Next, change directories to be inside `cpp/build`:
+
+```{bash, save=run & !sys_install}
+pushd arrow
+mkdir -p cpp/build
+pushd cpp/build
+```
+
+You'll first call `cmake` to configure the build and then `make install`. For
the R package, you'll need to enable several features in libarrow using `-D`
flags:
+
+```{bash, save=run & !sys_install}
+cmake \
+ -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \
+ -DCMAKE_INSTALL_LIBDIR=lib \
+ -DARROW_COMPUTE=ON \
+ -DARROW_CSV=ON \
+ -DARROW_DATASET=ON \
+ -DARROW_EXTRA_ERROR_CONTEXT=ON \
+ -DARROW_FILESYSTEM=ON \
+ -DARROW_INSTALL_NAME_RPATH=OFF \
+ -DARROW_JEMALLOC=ON \
+ -DARROW_JSON=ON \
+ -DARROW_PARQUET=ON \
+ -DARROW_WITH_SNAPPY=ON \
+ -DARROW_WITH_ZLIB=ON \
+ ..
+```
+
+`..` refers to the C++ source directory: you're in `cpp/build` and the source
is in `cpp`.
+
+#### Enabling more Arrow features
+
+To enable optional features including: S3 support, an alternative memory
allocator, and additional compression libraries, add some or all of these flags
to your call to `cmake` (the trailing `\` makes them easier to paste into a
bash shell on a new line):
+
+```bash
+ -DARROW_MIMALLOC=ON \
+ -DARROW_S3=ON \
+ -DARROW_WITH_BROTLI=ON \
+ -DARROW_WITH_BZ2=ON \
+ -DARROW_WITH_LZ4=ON \
+ -DARROW_WITH_SNAPPY=ON \
+ -DARROW_WITH_ZSTD=ON \
+```
+
+Other flags that may be useful:
+
+* `-DBoost_SOURCE=BUNDLED` and `-DThrift_SOURCE=BUNDLED`, for example, or any
other dependency `*_SOURCE`, if you have a system version of a C++ dependency
that doesn't work correctly with Arrow. This tells the build to compile its own
version of the dependency from source.
+
+* `-DCMAKE_BUILD_TYPE=debug` or `-DCMAKE_BUILD_TYPE=relwithdebinfo` can be
useful for debugging. You probably don't want to do this generally because a
debug build is much slower at runtime than the default `release` build.
+
+_Note_ `cmake` is particularly sensitive to whitespacing, if you see errors,
check that you don't have any errant whitespace.
+
+### Step 3 - Building libarrow
+
+You can add `-j#` between `make` and `install` here too to speed up
compilation by running in parallel (where `#` is the number of cores you have
available).
+
+```{bash, save=run & !(sys_install & ubuntu)}
+make -j8 install
+```
+
+### Step 4 - Build the Arrow R package
+
+Once you've built libarrow, you can install the R package and its
+dependencies, along with additional dev dependencies, from the git
+checkout:
+
+```{bash, save=run}
+popd # To go back to the root directory of the project, from cpp/build
+pushd r
+R -e 'install.packages("remotes"); remotes::install_deps(dependencies = TRUE)'
+R CMD INSTALL .
+```
+
+#### Compilation flags
+
+If you need to set any compilation flags while building the C++
+extensions, you can use the `ARROW_R_CXXFLAGS` environment variable. For
+example, if you are using `perf` to profile the R extensions, you may
+need to set
+
+```bash
+export ARROW_R_CXXFLAGS=-fno-omit-frame-pointer
+```
+
+#### Recompiling the C++ code
+
+With the setup described here, you should not need to rebuild the Arrow
library or even the C++ source in the R package as you iterate and work on the
R package. The only time those should need to be rebuilt is if you have changed
the C++ in the R package (and even then, `R CMD INSTALL .` should only need to
recompile the files that have changed) _or_ if the libarrow C++ has changed and
there is a mismatch between libarrow and the R package. If you find yourself
rebuilding either or both each time you install the package or run tests,
something is probably wrong with your set up.
+
+<details>
+<summary>For a full build: a `cmake` command with all of the R-relevant
optional dependencies turned on. Development with other languages might require
different flags as well. For example, to develop Python, you would need to also
add `-DARROW_PYTHON=ON` (though all of the other flags used for Python are
already included here).</summary>
+<p>
+
+```bash
+cmake \
+ -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \
+ -DCMAKE_INSTALL_LIBDIR=lib \
+ -DARROW_COMPUTE=ON \
+ -DARROW_CSV=ON \
+ -DARROW_DATASET=ON \
+ -DARROW_EXTRA_ERROR_CONTEXT=ON \
+ -DARROW_FILESYSTEM=ON \
+ -DARROW_INSTALL_NAME_RPATH=OFF \
+ -DARROW_JEMALLOC=ON \
+ -DARROW_JSON=ON \
+ -DARROW_MIMALLOC=ON \
+ -DARROW_PARQUET=ON \
+ -DARROW_S3=ON \
+ -DARROW_WITH_BROTLI=ON \
+ -DARROW_WITH_BZ2=ON \
+ -DARROW_WITH_LZ4=ON \
+ -DARROW_WITH_SNAPPY=ON \
+ -DARROW_WITH_ZLIB=ON \
+ -DARROW_WITH_ZSTD=ON \
+ ..
+```
+</p>
+</details>
+
+## Installing a version of the R package with a specific git reference
+
+If you need an arrow installation from a specific repository or git reference,
on most platforms except Windows, you can run:
+
+```{r}
+remotes::install_github("apache/arrow/r", build = FALSE)
+```
+
+The `build = FALSE` argument is important so that the installation can access
the
+C++ source in the `cpp/` directory in `apache/arrow`.
+
+As with other installation methods, setting the environment variables
`LIBARROW_MINIMAL=false` and `ARROW_R_DEV=true` will provide a more
full-featured version of Arrow and provide more verbose output, respectively.
+
+For example, to install from the (fictional) branch `bugfix` from
`apache/arrow` you could run:
+
+```r
+Sys.setenv(LIBARROW_MINIMAL="false")
+remotes::install_github("apache/arrow/r@bugfix", build = FALSE)
+```
+
+Developers may wish to use this method of installing a specific commit
+separate from another Arrow development environment or system installation
+(e.g. we use this in [arrowbench](https://github.com/ursacomputing/arrowbench)
+to install development versions of libarrow isolated from the system install).
If
+you already have libarrow installed system-wide, you may need to set
+some additional variables in order to isolate this build from your system
libraries:
+
+* Setting the environment variable `FORCE_BUNDLED_BUILD` to `true` will skip
the `pkg-config` search for libarrow and attempt to build from the same source
at the repository+ref given.
+
+* You may also need to set the Makevars `CPPFLAGS` and `LDFLAGS` to `""` in
order to prevent the installation process from attempting to link to already
installed system versions of libarrow. One way to do this temporarily is
wrapping your `remotes::install_github()` call like so:
+```{r}
+withr::with_makevars(list(CPPFLAGS = "", LDFLAGS = ""),
remotes::install_github(...))
+```
+
+
+## Summary of environment variables
+
+* See the user-facing [Install vignette](install.html) for a large number of
Review comment:
Good catch!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
