Bearloga has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/366170 )
Change subject: Move R-related code from shiny_server to separate module
......................................................................
Move R-related code from shiny_server to separate module
- Moves the code for installing R and its packages from
'shiny_server' and puts them into a new module 'r' to
facilitate working with R in Puppet without having to
deal with Shiny Server software.
- Also adds the new module to statistics::packages.
Bug: T153856
Change-Id: Ibd8b76f2ffd1cfaab6fdcc84117042eb668ed598
---
M modules/profile/manifests/discovery_dashboards/base.pp
A modules/r/README.md
R modules/r/files/update-library.R
R modules/r/manifests/cran.pp
R modules/r/manifests/git.pp
R modules/r/manifests/github.pp
A modules/r/manifests/init.pp
M modules/role/manifests/discovery/beta_dashboards.pp
M modules/role/manifests/discovery/dashboards.pp
M modules/shiny_server/README.md
M modules/shiny_server/manifests/init.pp
M modules/statistics/manifests/packages.pp
12 files changed, 186 insertions(+), 107 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/70/366170/1
diff --git a/modules/profile/manifests/discovery_dashboards/base.pp
b/modules/profile/manifests/discovery_dashboards/base.pp
index f148c59..e222477 100644
--- a/modules/profile/manifests/discovery_dashboards/base.pp
+++ b/modules/profile/manifests/discovery_dashboards/base.pp
@@ -5,7 +5,7 @@
#
# filtertags: labs-project-search labs-project-shiny-r
class profile::discovery_dashboards::base {
- include shiny_server
+ include ::shiny_server
$cran_packages = [
# Needed by Search metrics dashboard:
@@ -15,17 +15,17 @@
'highcharter',
'countrycode'
]
- shiny_server::cran_pkg { $cran_packages:
+ r::cran { $cran_packages:
mirror => 'https://cran.cnr.berkeley.edu',
}
# 'polloi' contains common functions & data used by all the dashboards
- shiny_server::git_pkg { 'polloi':
+ r::git { 'polloi':
url => 'https://gerrit.wikimedia.org/r/wikimedia/discovery/polloi',
}
# 'googleCharts' is used on the Wikipedia.org portal metrics dashboard
- shiny_server::github_pkg { 'googleCharts':
+ r::github { 'googleCharts':
repo => 'jcheng5/googleCharts',
}
diff --git a/modules/r/README.md b/modules/r/README.md
new file mode 100644
index 0000000..121bbe8
--- /dev/null
+++ b/modules/r/README.md
@@ -0,0 +1,53 @@
+# R: Statistical Software and Programming Language
+
+[R](https://www.r-project.org/) is a free software environment for statistical
+computing and graphics. This module facilitates setting up R in the computing
+evironment.
+
+Installs `r-base`, `r-base-dev`, `r-recommended` and Optimized BLAS (linear
+algebra) library, and makes the following resources available for installing
+R packages from various sources:
+
+- **r::cran** for installing from Comprehensive R Archive Network
+ - the resource ID should be name of the package to be installed
+ - *timeout*: default 300 (seconds)
+ - *ensure*: default 'present', but also supports 'absent'
+ - *mirror*: default 'https://cloud.r-project.org' which provides automatic
+ redirection to servers worldwide, sponsored by Rstudio. In practice, the
+ module uses [UC Berkeley mirror](https://cran.cnr.berkeley.edu/). For a
+ list of CRAN mirrors, see https://cran.r-project.org/mirrors.html
+- **r::git** for installing from any Git repository
+ - the resource ID should be name of the package to be installed
+ - *url* is forwarded to `devtools::install_git()`
+ e.g. 'https://gerrit.wikimedia.org/r/wikimedia/discovery/polloi'
+ - *ensure*: default 'present', but also supports 'absent'
+- **r::github** for installing from a GitHub-hosted repository
+ - the resource ID should be name of the package to be installed
+ - *repo* is forwarded to `devtools::install_github()`
+ e.g. 'wikimedia/wikimedia-discovery-polloi'
+ - *ensure*: default 'present', but also supports 'absent'
+
+The `notify` metaparameter is used to trigger a restart of the Shiny Server
+service.
+
+## Updating installed R packages
+
+There is a utility script - [update-library.R](files/update-library.R) - that
is
+saved to /etc/R/update-library.R and has the following options:
+
+- `-p PACKAGE, --package=PACKAGE` for updating a specific package. If missing,
+ all packages installed from CRAN will be updated.
+- `--mirror=MIRROR` for specifying the CRAN mirror URL. The default is
+ 'https://cloud.r-project.org'. For a list of CRAN mirrors, see
+ https://cran.r-project.org/mirrors.html
+- `-l LIBRARY, --library=LIBRARY` for updating packages in a specific library
location.
+ If missing, uses `.libPaths()` just like `update.packages()` does.
+
+Non-CRAN packages such as
[polloi](https://phabricator.wikimedia.org/diffusion/WDPL/)
+are updated only if specified as an option. For example:
+
+```bash
+Rscript /etc/R/update-library.R -p polloi
+```
+
+or if the user runs `devtools::update_packages()` interactively in R (as sudo).
diff --git a/modules/shiny_server/files/update-library.R
b/modules/r/files/update-library.R
similarity index 89%
rename from modules/shiny_server/files/update-library.R
rename to modules/r/files/update-library.R
index bde064a..effd922 100644
--- a/modules/shiny_server/files/update-library.R
+++ b/modules/r/files/update-library.R
@@ -9,7 +9,12 @@
make_option("--mirror", default = "https://cloud.r-project.org";,
action = "store", type = "character",
help = "The CRAN mirror to use [default %default].
- See https://cran.r-project.org/mirrors.html for more")
+ See https://cran.r-project.org/mirrors.html for more"),
+ make_option(c("-l", "--library"), default = NULL,
+ help = paste(
+ "Default (%default) uses .libPaths():\n -",
+ paste0(.libPaths(), collapse = "\n - ")
+ ))
)
# Get command line options, if help option encountered print help and exit,
@@ -17,7 +22,7 @@
opt <- parse_args(OptionParser(option_list = option_list))
if (is.na(opt$package)) {
- update.packages(ask = FALSE, checkBuilt = TRUE, repos = c(CRAN = opt$mirror))
+ update.packages(ask = FALSE, checkBuilt = TRUE, repos = c(CRAN =
opt$mirror), lib.loc = opt$library)
message("If any CRAN-installed packages were updated, restart shiny-server
via `sudo service shiny-server restart`")
message("To update all git-installed package, run
devtools::update_packages()")
message("Unfortunately, it must be run in interactive mode")
@@ -64,7 +69,7 @@
update_pkg <- FALSE
if (pkg_source %in% c("github", "git")) {
if (pkg_sha != remote_sha) {
- message("Installed version's SHA differets from remote version's SHA")
+ message("Installed version's SHA is different from remote version's SHA")
update_pkg <- TRUE
}
}
diff --git a/modules/shiny_server/manifests/cran_pkg.pp
b/modules/r/manifests/cran.pp
similarity index 73%
rename from modules/shiny_server/manifests/cran_pkg.pp
rename to modules/r/manifests/cran.pp
index e7280bc..bdab70e 100644
--- a/modules/shiny_server/manifests/cran_pkg.pp
+++ b/modules/r/manifests/cran.pp
@@ -1,4 +1,4 @@
-# = Define: shiny_server::cran_pkg
+# = Define: r::cran
#
# Facilitates installation of R packages from Comprehensive R Archive Network.
#
@@ -17,16 +17,22 @@
# worldwide (https://cloud.r-project.org) but can be overridden. Refer to
# https://cran.r-project.org/mirrors.html for the full list of mirrors.
#
-define shiny_server::cran_pkg (
+# [*library*]
+# default '/usr/local/lib/R/site-library', used
+# for specifying the path of the library for
+# installing the R package
+#
+define r::cran (
$timeout = 300,
- $ensure = 'present',
- $mirror = 'https://cloud.r-project.org'
+ $ensure = 'present',
+ $mirror = 'https://cloud.r-project.org',
+ $library = '/usr/local/lib/R/site-library'
) {
- $pkg_path = "/usr/local/lib/R/site-library/${title}"
+ $pkg_path = "${library}/${title}"
case $ensure {
'absent': {
exec { "remove-${title}":
- command => "/usr/bin/R -e \"remove.packages('${title}', lib =
'/usr/local/lib/R/site-library')\"",
+ command => "/usr/bin/R -e \"remove.packages('${title}', lib =
'${library}')\"",
notify => Service['shiny-server'],
}
}
@@ -37,7 +43,7 @@
Package['r-base-dev']
],
timeout => $timeout,
- command => "/usr/bin/R -e \"install.packages('${title}', repos
= c(CRAN = '${mirror}'), lib = '/usr/local/lib/R/site-library')\"",
+ command => "/usr/bin/R -e \"install.packages('${title}', repos
= c(CRAN = '${mirror}'), lib = '${library}')\"",
creates => $pkg_path,
}
}
diff --git a/modules/shiny_server/manifests/git_pkg.pp
b/modules/r/manifests/git.pp
similarity index 63%
rename from modules/shiny_server/manifests/git_pkg.pp
rename to modules/r/manifests/git.pp
index aa95cf1..d2e9c77 100644
--- a/modules/shiny_server/manifests/git_pkg.pp
+++ b/modules/r/manifests/git.pp
@@ -1,4 +1,4 @@
-# = Define: shiny_server::git_pkg
+# = Define: r::git
#
# Facilitates installation of R packages from a remote Git repository.
#
@@ -13,22 +13,28 @@
# default 'present' but also accepts 'absent'
# hoping to support 'latest' eventually
#
-define shiny_server::git_pkg (
+# [*library*]
+# default '/usr/local/lib/R/site-library', used
+# for specifying the path of the library for
+# installing the R package
+#
+define r::git (
$url,
- $ensure = 'present'
+ $ensure = 'present',
+ $library = '/usr/local/lib/R/site-library'
) {
- $pkg_path = "/usr/local/lib/R/site-library/${title}"
+ $pkg_path = "${library}/${title}"
case $ensure {
'absent': {
exec { "remove-${title}":
- command => "/usr/bin/R -e \"remove.packages('${title}', lib =
'/usr/local/lib/R/site-library')\"",
+ command => "/usr/bin/R -e \"remove.packages('${title}', lib =
'${library}')\"",
notify => Service['shiny-server'],
}
}
default: {
exec { "package-${title}":
- require => Shiny_server::Cran_pkg['devtools'],
- command => "/usr/bin/R -e \"devtools::install_git('${url}',
lib = '/usr/local/lib/R/site-library')\"",
+ require => R::Cran['devtools'],
+ command => "/usr/bin/R -e \"devtools::install_git('${url}',
lib = '${library}')\"",
creates => $pkg_path,
}
}
diff --git a/modules/shiny_server/manifests/github_pkg.pp
b/modules/r/manifests/github.pp
similarity index 62%
rename from modules/shiny_server/manifests/github_pkg.pp
rename to modules/r/manifests/github.pp
index c5022ed..51eb4d5 100644
--- a/modules/shiny_server/manifests/github_pkg.pp
+++ b/modules/r/manifests/github.pp
@@ -1,4 +1,4 @@
-# = Define: shiny_server::github_pkg
+# = Define: r::github
#
# Facilitates installation of R packages from GitHub.
#
@@ -12,23 +12,28 @@
# [*ensure*]
# default 'present' but also accepts 'absent'
# hoping to support 'latest' eventually
+# [*library*]
+# default '/usr/local/lib/R/site-library', used
+# for specifying the path of the library for
+# installing the R package
#
-define shiny_server::github_pkg (
+define r::github (
$repo,
- $ensure = 'present'
+ $ensure = 'present',
+ $library = '/usr/local/lib/R/site-library'
) {
- $pkg_path = "/usr/local/lib/R/site-library/${title}"
+ $pkg_path = "${library}/${title}"
case $ensure {
'absent': {
exec { "remove-${title}":
- command => "/usr/bin/R -e \"remove.packages('${title}', lib =
'/usr/local/lib/R/site-library')\"",
+ command => "/usr/bin/R -e \"remove.packages('${title}', lib =
'${library}')\"",
notify => Service['shiny-server'],
}
}
default: {
exec { "package-${title}":
- require => Shiny_server::Cran_pkg['devtools'],
- command => "/usr/bin/R -e
\"devtools::install_github('${repo}', lib = '/usr/local/lib/R/site-library')\"",
+ require => R::Cran['devtools'],
+ command => "/usr/bin/R -e
\"devtools::install_github('${repo}', lib = '${library}')\"",
creates => $pkg_path,
}
}
diff --git a/modules/r/manifests/init.pp b/modules/r/manifests/init.pp
new file mode 100644
index 0000000..7a1b653
--- /dev/null
+++ b/modules/r/manifests/init.pp
@@ -0,0 +1,59 @@
+# = Class: r
+#
+# Class containing stuff for installing R and its packages from different
sources:
+# - r::cran for installing from Comprehensive R Archive Network (CRAN)
+# - r::git for installing from any Git repository (e.g. Gerrit)
+# - r::github for installing from a GitHub-hosted repository
+#
+# Also provides a utility script for updating library of installed R packages.
+#
+class r {
+
+ $cran_mirror = 'https://cran.cnr.berkeley.edu'
+
+ $essentials = [
+ 'r-base', 'r-base-dev', 'r-recommended',
+ # To get higher performance for linear algebra operations
+ 'libopenblas-dev',
+ # For devtools:
+ 'libssl-dev', 'libcurl4-openssl-dev', 'libicu-dev', 'libssh2-1-dev'
+ ]
+ require_package($essentials)
+
+ file { '/usr/local/lib/R/site-library':
+ ensure => 'directory',
+ owner => 'root',
+ group => 'staff',
+ mode => '0770',
+ }
+
+ r::cran { 'openssl':
+ require => Package['libssl-dev'],
+ mirror => $cran_mirror
+ }
+
+ $r_packages = [
+ 'xml2',
+ 'testthat',
+ 'devtools'
+ ]
+ r::cran { $r_packages:
+ require => [
+ Package['libxml2'],
+ Package['libxml2-dev'],
+ R::Cran['openssl'],
+ Package['libcurl4-openssl-dev']
+ ],
+ mirror => $cran_mirror,
+ }
+
+ # R script for updating any particular installed R package:
+ file { '/etc/R/update-library.R':
+ ensure => 'present',
+ owner => 'root',
+ group => 'root',
+ mode => '0644',
+ source => 'puppet:///modules/r/update-library.R'
+ }
+
+}
diff --git a/modules/role/manifests/discovery/beta_dashboards.pp
b/modules/role/manifests/discovery/beta_dashboards.pp
index e379888..1adb1e9 100644
--- a/modules/role/manifests/discovery/beta_dashboards.pp
+++ b/modules/role/manifests/discovery/beta_dashboards.pp
@@ -4,6 +4,7 @@
# for tracking Search, Wikipedia.org portal, Wikidata
# Query Service, and Maps usage metrics and other KPIs.
#
+# filtertags: labs-project-search labs-project-shiny-r
class role::discovery::beta_dashboards {
include ::profile::discovery_dashboards::development
diff --git a/modules/role/manifests/discovery/dashboards.pp
b/modules/role/manifests/discovery/dashboards.pp
index fcaf261..f8d219c 100644
--- a/modules/role/manifests/discovery/dashboards.pp
+++ b/modules/role/manifests/discovery/dashboards.pp
@@ -4,6 +4,7 @@
# for tracking Search, Wikipedia.org portal, Wikidata
# Query Service, and Maps usage metrics and other KPIs.
#
+# filtertags: labs-project-search labs-project-shiny-r
class role::discovery::dashboards {
# include ::standard
# include ::base::firewall
diff --git a/modules/shiny_server/README.md b/modules/shiny_server/README.md
index 2b089f9..5b2cfe7 100644
--- a/modules/shiny_server/README.md
+++ b/modules/shiny_server/README.md
@@ -10,48 +10,5 @@
The module creates a service "shiny-server" that serves Shiny applications from
/srv/shiny-server through port 3838.
-## Resources
-
-The following resources and parameters are available:
-
-- **shiny_server::cran_pkg**
- - the resource ID should be name of the package to be installed
- - *timeout*: default 300 (seconds)
- - *ensure*: default 'present', but also supports 'absent'
- - *mirror*: default 'https://cloud.r-project.org' which provides automatic
- redirection to servers worldwide, sponsored by Rstudio. In practice, the
- module uses [UC Berkeley mirror](https://cran.cnr.berkeley.edu/). For a
- list of CRAN mirrors, see https://cran.r-project.org/mirrors.html
-- **shiny_server::git_pkg**
- - the resource ID should be name of the package to be installed
- - *url* is forwarded to `devtools::install_git()`
- e.g. 'https://gerrit.wikimedia.org/r/wikimedia/discovery/polloi'
- - *ensure*: default 'present', but also supports 'absent'
-- **shiny_server::github_pkg**
- - the resource ID should be name of the package to be installed
- - *repo* is forwarded to `devtools::install_github()`
- e.g. 'wikimedia/wikimedia-discovery-polloi'
- - *ensure*: default 'present', but also supports 'absent'
-
The `notify` metaparameter is used to trigger a restart of the Shiny Server
-service if the dashboard's source code has been updated.
-
-## Updating installed R packages
-
-There is a utility script - [update-library.R](files/update-library.R) - that
is saved
-to /etc/R/update-library.R and has the following options:
-
-- `-p PACKAGE, --package=PACKAGE` for updating a specific package. If missing,
- all packages installed from CRAN will be updated.
-- `--mirror=MIRROR` for specifying the CRAN mirror URL. The default is
- 'https://cloud.r-project.org'. For a list of CRAN mirrors, see
- https://cran.r-project.org/mirrors.html
-
-Non-CRAN packages such as
[polloi](https://phabricator.wikimedia.org/diffusion/WDPL/)
-are updated only if specified as an option:
-
-```bash
-Rscript /etc/R/update-library.R -p polloi
-```
-
-or if the user runs `devtools::update_packages()` interactively in R (as sudo).
+service.
diff --git a/modules/shiny_server/manifests/init.pp
b/modules/shiny_server/manifests/init.pp
index 485984b..160b1b2 100644
--- a/modules/shiny_server/manifests/init.pp
+++ b/modules/shiny_server/manifests/init.pp
@@ -17,56 +17,41 @@
# For a list of CRAN mirrors, see https://cran.r-project.org/mirrors.html
#
class shiny_server {
+ include ::r
$essentials = [
'gfortran', 'g++-4.8', 'gfortran-4.8',
'libssl-dev', 'libcurl4-openssl-dev', 'libxml2-dev', 'libssh2-1-dev',
- 'libcairo2-dev', 'git-core', 'gdebi', 'pandoc',
- 'r-base', 'r-base-dev', 'r-recommended'
+ 'libcairo2-dev', 'git-core', 'gdebi', 'pandoc'
]
require_package($essentials)
- file { '/usr/local/lib/R/site-library':
- ensure => 'directory',
- owner => 'root',
- group => 'staff',
- mode => '0770',
- }
-
- # R script for updating any particular installed R package:
- file { '/etc/R/update-library.R':
- ensure => 'present',
- owner => 'root',
- group => 'root',
- mode => '0644',
- source => 'puppet:///modules/shiny_server/update-library.R'
- }
-
# Install R packages from CRAN, Gerrit, and GitHub:
$cran_mirror = 'https://cran.cnr.berkeley.edu'
- shiny_server::cran_pkg { 'curl':
+ r::cran { 'curl':
require => Package['libcurl4-openssl-dev'],
mirror => $cran_mirror,
}
- shiny_server::cran_pkg { 'xml2':
+ r::cran { 'xml2':
require => Package['libxml2-dev'],
mirror => $cran_mirror,
}
- shiny_server::cran_pkg { 'devtools':
+ r::cran { 'devtools':
require => [
Package['git-core'],
- Shiny_server::Cran_pkg['curl']
+ R::Cran['curl']
],
mirror => $cran_mirror,
}
- shiny_server::cran_pkg { 'rmarkdown':
+ r::cran { 'rmarkdown':
require => Package['pandoc'],
mirror => $cran_mirror,
}
# tidyverse includes packages such as dplyr, tidyr, magrittr, readr,
# ggplot2, broom, purrr, rvest, forcats, lubridate, and jsonlite
- shiny_server::cran_pkg { 'tidyverse':
- timeout => 2700,
+ # It's a lot of packages so we *really* need to extend the timeout.
+ r::cran { 'tidyverse':
+ timeout => 6000,
mirror => $cran_mirror
}
$cran_packages = [
@@ -86,7 +71,7 @@
'knitr', 'markdown',
'optparse' # needed for /etc/update-pkg.R
]
- shiny_server::cran_pkg { $cran_packages: mirror => $cran_mirror }
+ r::cran { $cran_packages: mirror => $cran_mirror }
# Set up files, directories, and users required for RStudio's Shiny Server:
user { 'shiny':
diff --git a/modules/statistics/manifests/packages.pp
b/modules/statistics/manifests/packages.pp
index 834f70f..e1d643e 100644
--- a/modules/statistics/manifests/packages.pp
+++ b/modules/statistics/manifests/packages.pp
@@ -4,7 +4,6 @@
include ::geoip
include ::imagemagick::install
-
require_package([
'time',
'emacs',
@@ -122,15 +121,16 @@
'libxt-dev',
])
- $r_packages = [
- 'r-base',
- 'r-base-dev', # Needed for R packages that have to compile C++
code; see T147682
- 'r-cran-rmysql',
- 'r-recommended' # CRAN-recommended packages (e.g. MASS, Matrix,
boot)
- ]
+
# Use R from Jessie Backports on jessie boxes.
if os_version('debian == jessie') {
+ $r_packages = [
+ 'r-base',
+ 'r-base-dev', # Needed for R packages that have to compile
C++ code; see T147682
+ 'r-cran-rmysql',
+ 'r-recommended' # CRAN-recommended packages (e.g. MASS, Matrix,
boot)
+ ]
apt::pin { $r_packages:
pin => 'release a=jessie-backports',
priority => '1001',
@@ -138,8 +138,9 @@
}
}
- package { $r_packages:
- ensure => present,
+ else {
+ include ::r
+ require_package('r-cran-rmysql') # Note: RMariaDB
(https://github.com/rstats-db/RMariaDB) will replace RMySQL, but is currently
not on CRAN
}
if os_version('ubuntu >= trusty') {
--
To view, visit https://gerrit.wikimedia.org/r/366170
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ibd8b76f2ffd1cfaab6fdcc84117042eb668ed598
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Bearloga <mpo...@wikimedia.org>
_______________________________________________
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
