[MediaWiki-commits] [Gerrit] operations/puppet[production]: Revert "Revert "Deploy MjoLniR with new deploy repository""
Gehel has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/394002 ) Change subject: Revert "Revert "Deploy MjoLniR with new deploy repository"" .. Revert "Revert "Deploy MjoLniR with new deploy repository"" The problems with mjolnir vs MjoLniR have been resolved, it required adjustments to scap.cfg in the repo. This is a standard revert with one addition, we now create an empty directory /etc/mjolnir owned by deploy-service. This gives the deploy repo a sane place to install a configuration script to that can be auto-magicaly found by mjolnir. This reverts commit 6a7753a14ac3cb66593eabbad30e8ac72e184751. Change-Id: I599341bd16ecba0a2b8d8132fde6fe3d1443d754 --- M hieradata/role/common/deployment_server.yaml A modules/mjolnir/manifests/init.pp M modules/profile/manifests/mjolnir/kafka_daemon.pp M modules/profile/templates/mjolnir/kafka-daemon.service.erb M modules/role/manifests/elasticsearch/analytics.pp 5 files changed, 28 insertions(+), 14 deletions(-) Approvals: EBernhardson: Looks good to me, but someone else must approve jenkins-bot: Verified Gehel: Looks good to me, approved diff --git a/hieradata/role/common/deployment_server.yaml b/hieradata/role/common/deployment_server.yaml index 4f899a6..5c5e154 100644 --- a/hieradata/role/common/deployment_server.yaml +++ b/hieradata/role/common/deployment_server.yaml @@ -173,8 +173,8 @@ # Netbox software netbox/deploy: repository: operations/software/netbox-deploy - relforge/mjolnir: -repository: search/MjoLniR + search/mjolnir/deploy: +repository: search/MjoLniR/deploy statsv/statsv: repository: analytics/statsv "docker-pkg/deploy": diff --git a/modules/mjolnir/manifests/init.pp b/modules/mjolnir/manifests/init.pp new file mode 100644 index 000..07e559f --- /dev/null +++ b/modules/mjolnir/manifests/init.pp @@ -0,0 +1,21 @@ +# = Class: mjolnir +# +# This class installs the MjoLniR (Machine Learned Ranking) data +# processing package. +# +class mjolnir { +require_package('virtualenv', 'zip') + +file { '/etc/mjolnir': +ensure => 'directory', +owner => 'deploy-service', +group => 'deploy-service', +mode => '0755', +} + +scap::target { 'search/mjolnir/deploy': +deploy_user => 'deploy-service', +} +} + + diff --git a/modules/profile/manifests/mjolnir/kafka_daemon.pp b/modules/profile/manifests/mjolnir/kafka_daemon.pp index c1dcaf9..ba8cc1c 100644 --- a/modules/profile/manifests/mjolnir/kafka_daemon.pp +++ b/modules/profile/manifests/mjolnir/kafka_daemon.pp @@ -9,18 +9,11 @@ # it is named just 'eqiad'. $kafka_config = kafka_config('eqiad'), ) { -scap::target { 'relforge/mjolnir': - deploy_user => 'deploy-service', -} - -# This is a limited subset of what the full mjolnir package requires because -# the daemon is a small part of the overall application. The daemon only needs -# to read/write kafka topics and send requests to localhost. -require_package('python-kafka', 'python-requests') +class { 'mjolnir': } systemd::service { 'mjolnir-kafka-daemon': content => template('profile/mjolnir/kafka-daemon.service.erb'), -require => Scap::Target['relforge/mjolnir'], +require => Class['mjolnir'], } } diff --git a/modules/profile/templates/mjolnir/kafka-daemon.service.erb b/modules/profile/templates/mjolnir/kafka-daemon.service.erb index b6947ba..7e40bd8 100644 --- a/modules/profile/templates/mjolnir/kafka-daemon.service.erb +++ b/modules/profile/templates/mjolnir/kafka-daemon.service.erb @@ -5,9 +5,7 @@ [Service] User=nobody Group=nogroup -WorkingDirectory=/srv/deployment/relforge/mjolnir -Environment=PYTHONPATH=/srv/deployment/relforge/mjolnir -ExecStart=/usr/bin/python2 /srv/deployment/relforge/mjolnir/mjolnir/cli/kafka_daemon.py --brokers <%= @kafka_config['brokers']['string'] %> +ExecStart=/srv/deployment/search/mjolnir/venv/bin/mjolnir-utilities.py kafka_daemon --brokers <%= @kafka_config['brokers']['string'] %> StandardInput=null StandardOutput=journal StandardError=journal diff --git a/modules/role/manifests/elasticsearch/analytics.pp b/modules/role/manifests/elasticsearch/analytics.pp index a46391e..66d7789 100644 --- a/modules/role/manifests/elasticsearch/analytics.pp +++ b/modules/role/manifests/elasticsearch/analytics.pp @@ -1,5 +1,7 @@ # Supports CirrusSearch usage on the analytics cluster class role::elasticsearch::analytics { +class { 'mjolnir': } + # wikimedia/discovery/analytics will be deployed to this node scap::target { 'wikimedia/discovery/analytics': deploy_user => 'deploy-service', -- To view, visit https://gerrit.wikimedia.org/r/394002 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I599341bd16ecba0a2b8d8132fde6fe3d1443d754 Gerrit-PatchSet: 5 Gerrit-Project: oper
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Revert "Revert "Deploy MjoLniR with new deploy repository""
EBernhardson has uploaded a new change for review. ( https://gerrit.wikimedia.org/r/394002 ) Change subject: Revert "Revert "Deploy MjoLniR with new deploy repository"" .. Revert "Revert "Deploy MjoLniR with new deploy repository"" The problems with mjolnir vs MjoLniR have been resolved, it required adjustments to scap.cfg in the repo. This is a standard revert with one addition, we now create an empty directory /etc/mjolnir owned by deploy-service. This gives the deploy repo a sane place to install a configuration script to that can be auto-magicaly found by mjolnir. This reverts commit 6a7753a14ac3cb66593eabbad30e8ac72e184751. Change-Id: I599341bd16ecba0a2b8d8132fde6fe3d1443d754 --- M hieradata/role/common/deployment_server.yaml A modules/mjolnir/manifests/init.pp M modules/profile/manifests/mjolnir/kafka_daemon.pp M modules/profile/templates/mjolnir/kafka-daemon.service.erb M modules/role/manifests/elasticsearch/analytics.pp 5 files changed, 28 insertions(+), 14 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/02/394002/1 diff --git a/hieradata/role/common/deployment_server.yaml b/hieradata/role/common/deployment_server.yaml index 7c55e05..7444810 100644 --- a/hieradata/role/common/deployment_server.yaml +++ b/hieradata/role/common/deployment_server.yaml @@ -173,8 +173,8 @@ # Netbox software netbox/deploy: repository: operations/software/netbox-deploy - relforge/mjolnir: -repository: search/MjoLniR + search/mjolnir/deploy: +repository: search/MjoLniR/deploy statsv/statsv: repository: analytics/statsv "docker-pkg/deploy": diff --git a/modules/mjolnir/manifests/init.pp b/modules/mjolnir/manifests/init.pp new file mode 100644 index 000..e4bb89f --- /dev/null +++ b/modules/mjolnir/manifests/init.pp @@ -0,0 +1,21 @@ +# = Class: mjolnir +# +# This class installs the MjoLniR (Machine Learned Ranking) data +# processing package. +# +class mjolnir { +require_package('virtualenv', 'zip') + +file { '/etc/mjolnir': +ensure => 'directory', +user => 'deploy-service', +group => 'deploy-service', +mode => 0755 +} + +scap::target { 'search/mjolnir/deploy': +deploy_user => 'deploy-service', +} +} + + diff --git a/modules/profile/manifests/mjolnir/kafka_daemon.pp b/modules/profile/manifests/mjolnir/kafka_daemon.pp index c1dcaf9..479187c 100644 --- a/modules/profile/manifests/mjolnir/kafka_daemon.pp +++ b/modules/profile/manifests/mjolnir/kafka_daemon.pp @@ -9,18 +9,11 @@ # it is named just 'eqiad'. $kafka_config = kafka_config('eqiad'), ) { -scap::target { 'relforge/mjolnir': - deploy_user => 'deploy-service', -} - -# This is a limited subset of what the full mjolnir package requires because -# the daemon is a small part of the overall application. The daemon only needs -# to read/write kafka topics and send requests to localhost. -require_package('python-kafka', 'python-requests') +class { 'mjolnir': } systemd::service { 'mjolnir-kafka-daemon': content => template('profile/mjolnir/kafka-daemon.service.erb'), -require => Scap::Target['relforge/mjolnir'], +require => Scap::Target['search/mjolnir/deploy'], } } diff --git a/modules/profile/templates/mjolnir/kafka-daemon.service.erb b/modules/profile/templates/mjolnir/kafka-daemon.service.erb index b6947ba..7e40bd8 100644 --- a/modules/profile/templates/mjolnir/kafka-daemon.service.erb +++ b/modules/profile/templates/mjolnir/kafka-daemon.service.erb @@ -5,9 +5,7 @@ [Service] User=nobody Group=nogroup -WorkingDirectory=/srv/deployment/relforge/mjolnir -Environment=PYTHONPATH=/srv/deployment/relforge/mjolnir -ExecStart=/usr/bin/python2 /srv/deployment/relforge/mjolnir/mjolnir/cli/kafka_daemon.py --brokers <%= @kafka_config['brokers']['string'] %> +ExecStart=/srv/deployment/search/mjolnir/venv/bin/mjolnir-utilities.py kafka_daemon --brokers <%= @kafka_config['brokers']['string'] %> StandardInput=null StandardOutput=journal StandardError=journal diff --git a/modules/role/manifests/elasticsearch/analytics.pp b/modules/role/manifests/elasticsearch/analytics.pp index a46391e..66d7789 100644 --- a/modules/role/manifests/elasticsearch/analytics.pp +++ b/modules/role/manifests/elasticsearch/analytics.pp @@ -1,5 +1,7 @@ # Supports CirrusSearch usage on the analytics cluster class role::elasticsearch::analytics { +class { 'mjolnir': } + # wikimedia/discovery/analytics will be deployed to this node scap::target { 'wikimedia/discovery/analytics': deploy_user => 'deploy-service', -- To view, visit https://gerrit.wikimedia.org/r/394002 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I599341bd16ecba0a2b8d8132fde6fe3d1443d754 Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: