[MediaWiki-commits] [Gerrit] operations/puppet[production]: Move R-related code from shiny_server to separate module
Gehel has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/366170 )
Change subject: Move R-related code from shiny_server to separate module
..
Move R-related code from shiny_server to separate module
- Moves the code for installing R and its packages from
'shiny_server' and puts them into a new module 'r' to
facilitate working with R in Puppet without having to
deal with Shiny Server software.
- Also adds the new module to statistics::packages.
Bug: T153856
Change-Id: Ibd8b76f2ffd1cfaab6fdcc84117042eb668ed598
---
M modules/profile/manifests/discovery_dashboards/base.pp
A modules/r/README.md
R modules/r/files/update-library.R
R modules/r/manifests/cran.pp
R modules/r/manifests/git.pp
R modules/r/manifests/github.pp
A modules/r/manifests/init.pp
M modules/role/manifests/discovery/beta_dashboards.pp
M modules/role/manifests/discovery/dashboards.pp
M modules/shiny_server/README.md
M modules/shiny_server/manifests/init.pp
M modules/statistics/manifests/packages.pp
12 files changed, 241 insertions(+), 108 deletions(-)
Approvals:
jenkins-bot: Verified
Gehel: Looks good to me, approved
diff --git a/modules/profile/manifests/discovery_dashboards/base.pp
b/modules/profile/manifests/discovery_dashboards/base.pp
index f148c59..e222477 100644
--- a/modules/profile/manifests/discovery_dashboards/base.pp
+++ b/modules/profile/manifests/discovery_dashboards/base.pp
@@ -5,7 +5,7 @@
#
# filtertags: labs-project-search labs-project-shiny-r
class profile::discovery_dashboards::base {
-include shiny_server
+include ::shiny_server
$cran_packages = [
# Needed by Search metrics dashboard:
@@ -15,17 +15,17 @@
'highcharter',
'countrycode'
]
-shiny_server::cran_pkg { $cran_packages:
+r::cran { $cran_packages:
mirror => 'https://cran.cnr.berkeley.edu',
}
# 'polloi' contains common functions & data used by all the dashboards
-shiny_server::git_pkg { 'polloi':
+r::git { 'polloi':
url => 'https://gerrit.wikimedia.org/r/wikimedia/discovery/polloi',
}
# 'googleCharts' is used on the Wikipedia.org portal metrics dashboard
-shiny_server::github_pkg { 'googleCharts':
+r::github { 'googleCharts':
repo => 'jcheng5/googleCharts',
}
diff --git a/modules/r/README.md b/modules/r/README.md
new file mode 100644
index 000..5aa634d
--- /dev/null
+++ b/modules/r/README.md
@@ -0,0 +1,92 @@
+# R: Statistical Software and Programming Language
+
+[R](https://www.r-project.org/) is a free software environment for statistical
+computing and graphics. This module facilitates setting up R in the computing
+evironment.
+
+Installs `r-base`, `r-base-dev`, `r-recommended` and Optimized BLAS (linear
+algebra) library, and makes the following resources available for installing
+R packages from various sources:
+
+- **r::cran** for installing from Comprehensive R Archive Network
+- the resource ID should be name of the package to be installed
+- *timeout*: default 300 (seconds)
+- *ensure*: default 'present', but also supports 'absent'
+- *mirror*: default 'https://cloud.r-project.org' which provides automatic
+ redirection to servers worldwide, sponsored by Rstudio. In practice, the
+ module uses [UC Berkeley mirror](https://cran.cnr.berkeley.edu/). For a
+ list of CRAN mirrors, see https://cran.r-project.org/mirrors.html
+- **r::git** for installing from any Git repository
+- the resource ID should be name of the package to be installed
+- *url* is forwarded to `devtools::install_git()`
+ e.g. 'https://gerrit.wikimedia.org/r/wikimedia/discovery/polloi'
+- *ensure*: default 'present', but also supports 'absent'
+- **Notice**: this is only available if the `devtools` parameter is set to
+ `true`. Refer to [Disclaimer](#disclaimer) section below for details.
+- **r::github** for installing from a GitHub-hosted repository
+- the resource ID should be name of the package to be installed
+- *repo* is forwarded to `devtools::install_github()`
+ e.g. 'wikimedia/wikimedia-discovery-polloi'
+- *ensure*: default 'present', but also supports 'absent'
+- **Notice**: this is only available if the `devtools` parameter is set to
+ `true`. Refer to [Disclaimer](#disclaimer) section below for details.
+
+The `notify` metaparameter is used to trigger a restart of the Shiny Server
+service.
+
+## Disclaimer
+
+By default, the *devtools* R package (and its dependencies) are not installed,
+which means that **r::git** and **r::github** will not work without specifying
+`devtools => true` when using this module. This is because we do not yet allow
+installing R packages via Puppet in Production until we have some kind of our
+own, trusted CRAN mirror to install from. The work and discussion for setting
+up a Wikimedia-hosted mirror of CRAN is tracked in Phabr
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Move R-related code from shiny_server to separate module
Bearloga has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/366170 )
Change subject: Move R-related code from shiny_server to separate module
..
Move R-related code from shiny_server to separate module
- Moves the code for installing R and its packages from
'shiny_server' and puts them into a new module 'r' to
facilitate working with R in Puppet without having to
deal with Shiny Server software.
- Also adds the new module to statistics::packages.
Bug: T153856
Change-Id: Ibd8b76f2ffd1cfaab6fdcc84117042eb668ed598
---
M modules/profile/manifests/discovery_dashboards/base.pp
A modules/r/README.md
R modules/r/files/update-library.R
R modules/r/manifests/cran.pp
R modules/r/manifests/git.pp
R modules/r/manifests/github.pp
A modules/r/manifests/init.pp
M modules/role/manifests/discovery/beta_dashboards.pp
M modules/role/manifests/discovery/dashboards.pp
M modules/shiny_server/README.md
M modules/shiny_server/manifests/init.pp
M modules/statistics/manifests/packages.pp
12 files changed, 186 insertions(+), 107 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/70/366170/1
diff --git a/modules/profile/manifests/discovery_dashboards/base.pp
b/modules/profile/manifests/discovery_dashboards/base.pp
index f148c59..e222477 100644
--- a/modules/profile/manifests/discovery_dashboards/base.pp
+++ b/modules/profile/manifests/discovery_dashboards/base.pp
@@ -5,7 +5,7 @@
#
# filtertags: labs-project-search labs-project-shiny-r
class profile::discovery_dashboards::base {
-include shiny_server
+include ::shiny_server
$cran_packages = [
# Needed by Search metrics dashboard:
@@ -15,17 +15,17 @@
'highcharter',
'countrycode'
]
-shiny_server::cran_pkg { $cran_packages:
+r::cran { $cran_packages:
mirror => 'https://cran.cnr.berkeley.edu',
}
# 'polloi' contains common functions & data used by all the dashboards
-shiny_server::git_pkg { 'polloi':
+r::git { 'polloi':
url => 'https://gerrit.wikimedia.org/r/wikimedia/discovery/polloi',
}
# 'googleCharts' is used on the Wikipedia.org portal metrics dashboard
-shiny_server::github_pkg { 'googleCharts':
+r::github { 'googleCharts':
repo => 'jcheng5/googleCharts',
}
diff --git a/modules/r/README.md b/modules/r/README.md
new file mode 100644
index 000..121bbe8
--- /dev/null
+++ b/modules/r/README.md
@@ -0,0 +1,53 @@
+# R: Statistical Software and Programming Language
+
+[R](https://www.r-project.org/) is a free software environment for statistical
+computing and graphics. This module facilitates setting up R in the computing
+evironment.
+
+Installs `r-base`, `r-base-dev`, `r-recommended` and Optimized BLAS (linear
+algebra) library, and makes the following resources available for installing
+R packages from various sources:
+
+- **r::cran** for installing from Comprehensive R Archive Network
+- the resource ID should be name of the package to be installed
+- *timeout*: default 300 (seconds)
+- *ensure*: default 'present', but also supports 'absent'
+- *mirror*: default 'https://cloud.r-project.org' which provides automatic
+ redirection to servers worldwide, sponsored by Rstudio. In practice, the
+ module uses [UC Berkeley mirror](https://cran.cnr.berkeley.edu/). For a
+ list of CRAN mirrors, see https://cran.r-project.org/mirrors.html
+- **r::git** for installing from any Git repository
+- the resource ID should be name of the package to be installed
+- *url* is forwarded to `devtools::install_git()`
+ e.g. 'https://gerrit.wikimedia.org/r/wikimedia/discovery/polloi'
+- *ensure*: default 'present', but also supports 'absent'
+- **r::github** for installing from a GitHub-hosted repository
+- the resource ID should be name of the package to be installed
+- *repo* is forwarded to `devtools::install_github()`
+ e.g. 'wikimedia/wikimedia-discovery-polloi'
+- *ensure*: default 'present', but also supports 'absent'
+
+The `notify` metaparameter is used to trigger a restart of the Shiny Server
+service.
+
+## Updating installed R packages
+
+There is a utility script - [update-library.R](files/update-library.R) - that
is
+saved to /etc/R/update-library.R and has the following options:
+
+- `-p PACKAGE, --package=PACKAGE` for updating a specific package. If missing,
+ all packages installed from CRAN will be updated.
+- `--mirror=MIRROR` for specifying the CRAN mirror URL. The default is
+ 'https://cloud.r-project.org'. For a list of CRAN mirrors, see
+ https://cran.r-project.org/mirrors.html
+- `-l LIBRARY, --library=LIBRARY` for updating packages in a specific library
location.
+ If missing, uses `.libPaths()` just like `update.packages()` does.
+
+Non-CRAN packages such as
[polloi](https://phabricator.wikimedia.org/diffusion/WDPL/)
+are updated only if spec
