| bd808 added a comment. |
In T153308#2881079, @zhuyifei1999 wrote:Do we have a process for setting up a pip package globally on exec nodes? I'm only aware of deb packages.
Correct, we do not install via pip for system wide libraries. We do install some language level packages system wide using deb packages.
AFAIK virtualenvs have been the de-facto way of installing them for each account.
Use of a local virtual environment is the recommended best practice. It has several advantages including the ability to install newer packages than are available via debs for system wide install and cross-project isolation to prevent an upgrade of libraries for tool X from impacting the libraries used for tool Y. There are basic instructions on wikitech for setting up a python2 or python3 virtual env. These particular instructions are tailored to use with webservice, but the basic instructions apply to any python tool. There are also many virtualenv tutorials available elsewhere on the web that can be found with a bit of searching.
One thing to be aware of is that the host operating system used to create the virtualenv should match the host operating system used to execute python scripts using the virtualenv. Generally this means that is you create the virtualenv from tools-login or tools-dev it will work on trusty grid exec nodes. For creating a precise compatible virtualenv use shell access on tools-precise-dev.tools.eqiad.wmflabs. For virtualenvs that will be used from Kubernetes based jobs, use webservice --backend python2 shell (python2) or webservice --backend python shell (python3) to get a shell that can be used to create or update the virtualenv. When making virtualenvs for a tool such as one of the irc bots I run I generally name the virtualenv to remind me which environment it is built for (e.g. ~/venv-oge-python2).
The easiest way to use a virtualenv with a grid job is to make a small shell script that activates the environment and then executes the python script:
#!/usr/bin/env bash # Virtualenv exec wrapper
# --start config section-- # Name of the virtualenv directory that you created for this job VENV_DIR=venv-oge-py2 # Path to the script to execute (relative to the location of this shell script) SCRIPT=name-of-my-script.py # --end config section-- # Exit immediately on error set -e if [[ ! -r ${VENV_DIR}/bin/activate ]]; then echo "Virtualenv $(pwd)/${VENV_DIR} not found!" >&2 exit 254 fi source ${VENV_DIR}/bin/activate if [[ ! -r ${SCRIPT} ]]; then echo "Script $(pwd)/${SCRIPT} not found!" >&2 exit 253 fi exec python ${SCRIPT} "$*"
Cc: bd808, zhuyifei1999, Zppix, yuvipanda, valhallasw, pywikibot-bugs-list, Urbanecm, Aklapper, Magul, Tbscho, MayS, Freddy2001, Mdupont, JJMC89, Luke081515, Gryllida, jayvdb, Wesalius, Dalba, Masti, scfc, Alchimista, coren, Rxy, chasemp
_______________________________________________ pywikibot-bugs mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
