[MediaWiki-commits] [Gerrit] operations/puppet[production]: package_builder: add dh-make-golang to the list of req. pack...
Elukey has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/405892 ) Change subject: package_builder: add dh-make-golang to the list of req. packages .. package_builder: add dh-make-golang to the list of req. packages Bug: T180442 Change-Id: Ia6729e739bfa9c643bccbbbf8e805469f720e0bf --- M modules/package_builder/manifests/init.pp 1 file changed, 1 insertion(+), 0 deletions(-) Approvals: Muehlenhoff: Looks good to me, but someone else must approve Alexandros Kosiaris: Looks good to me, but someone else must approve Elukey: Looks good to me, approved jenkins-bot: Verified diff --git a/modules/package_builder/manifests/init.pp b/modules/package_builder/manifests/init.pp index 2dc64c7..f5b5ee9 100644 --- a/modules/package_builder/manifests/init.pp +++ b/modules/package_builder/manifests/init.pp @@ -37,6 +37,7 @@ 'dh-make', 'dh-autoreconf', 'dh-golang', +'dh-make-golang', 'dh-systemd', 'openstack-pkg-tools', 'git-buildpackage', -- To view, visit https://gerrit.wikimedia.org/r/405892 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ia6729e739bfa9c643bccbbbf8e805469f720e0bf Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey Gerrit-Reviewer: Alexandros Kosiaris Gerrit-Reviewer: Elukey Gerrit-Reviewer: Muehlenhoff Gerrit-Reviewer: jenkins-bot <> ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: package_builder: add dh-make-golang to the list of req. pack...
Elukey has uploaded a new change for review. ( https://gerrit.wikimedia.org/r/405892 ) Change subject: package_builder: add dh-make-golang to the list of req. packages .. package_builder: add dh-make-golang to the list of req. packages Bug: T180442 Change-Id: Ia6729e739bfa9c643bccbbbf8e805469f720e0bf --- M modules/package_builder/manifests/init.pp 1 file changed, 1 insertion(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/92/405892/1 diff --git a/modules/package_builder/manifests/init.pp b/modules/package_builder/manifests/init.pp index 2dc64c7..f5b5ee9 100644 --- a/modules/package_builder/manifests/init.pp +++ b/modules/package_builder/manifests/init.pp @@ -37,6 +37,7 @@ 'dh-make', 'dh-autoreconf', 'dh-golang', +'dh-make-golang', 'dh-systemd', 'openstack-pkg-tools', 'git-buildpackage', -- To view, visit https://gerrit.wikimedia.org/r/405892 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ia6729e739bfa9c643bccbbbf8e805469f720e0bf Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] eventlogging[master]: [WIP] kafka_confluent_writer: handle BufferError exception w...
Elukey has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/405687 )
Change subject: [WIP] kafka_confluent_writer: handle BufferError exception when
producing
..
[WIP] kafka_confluent_writer: handle BufferError exception when producing
Bug: T185291
Change-Id: I0056739fcd97e4520ce44640681c0a4c24247b3d
---
M eventlogging/handlers.py
1 file changed, 13 insertions(+), 2 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/eventlogging
refs/changes/87/405687/1
diff --git a/eventlogging/handlers.py b/eventlogging/handlers.py
index 5e00fdd..2032cea 100644
--- a/eventlogging/handlers.py
+++ b/eventlogging/handlers.py
@@ -422,8 +422,19 @@
message_value = event.encode('utf-8') if raw else \
json.dumps(event, sort_keys=True).encode('utf-8')
-# Produce the message.
-kafka_producer.produce(message_topic, message_value, message_key)
+event_delivered = False
+while not event_delivered:
+try:
+# Produce the message.
+kafka_producer.produce(message_topic, message_value,
message_key)
+event_delivered = True
+except BufferError as e:
+logging.error(
+'Local queue full error when trying to produce, poll '
+'and wait for deliveries..' % e
+)
+kafka_producer.poll(1)
+continue
# If not async, the flush Kafka produce buffer now and block
# until we are done.
--
To view, visit https://gerrit.wikimedia.org/r/405687
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I0056739fcd97e4520ce44640681c0a4c24247b3d
Gerrit-PatchSet: 1
Gerrit-Project: eventlogging
Gerrit-Branch: master
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Update mw2140 MAC address after mainboard replacement
Elukey has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/405291 )
Change subject: Update mw2140 MAC address after mainboard replacement
..
Update mw2140 MAC address after mainboard replacement
/admin1-> racadm getsysinfo
[..]
Embedded NIC MAC Addresses:
NIC.Embedded.1-1-1 Ethernet= D0:94:66:0B:6F:DE
NIC.Embedded.2-1-1 Ethernet= D0:94:66:0B:6F:DF
Bug: T184788
Change-Id: Iaa38e693170b4eebcc9749fb2496f70b8cbb8e1f
---
M modules/install_server/files/dhcpd/linux-host-entries.ttyS1-115200
1 file changed, 1 insertion(+), 1 deletion(-)
Approvals:
Elukey: Verified; Looks good to me, approved
diff --git a/modules/install_server/files/dhcpd/linux-host-entries.ttyS1-115200
b/modules/install_server/files/dhcpd/linux-host-entries.ttyS1-115200
index 668841f..dd9424f 100644
--- a/modules/install_server/files/dhcpd/linux-host-entries.ttyS1-115200
+++ b/modules/install_server/files/dhcpd/linux-host-entries.ttyS1-115200
@@ -6170,7 +6170,7 @@
}
host mw2140 {
-hardware ethernet B0:83:FE:DE:FD:BA;
+hardware ethernet D0:94:66:0B:6F:DE;
fixed-address mw2140.codfw.wmnet;
option pxelinux.pathprefix
"http://apt.wikimedia.org/tftpboot/jessie-installer/";;
filename "lpxelinux.0";
--
To view, visit https://gerrit.wikimedia.org/r/405291
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Iaa38e693170b4eebcc9749fb2496f70b8cbb8e1f
Gerrit-PatchSet: 4
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
Gerrit-Reviewer: Elukey
Gerrit-Reviewer: Muehlenhoff
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Update mw2140 MAC address after mainboard replacement
Elukey has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/405291 )
Change subject: Update mw2140 MAC address after mainboard replacement
..
Update mw2140 MAC address after mainboard replacement
/admin1-> racadm getsysinfo
[..]
Embedded NIC MAC Addresses:
NIC.Embedded.1-1-1 Ethernet= D0:94:66:0B:6F:DE
NIC.Embedded.2-1-1 Ethernet= D0:94:66:0B:6F:DF
Bug: T184788
Change-Id: Iaa38e693170b4eebcc9749fb2496f70b8cbb8e1f
---
M modules/install_server/files/dhcpd/linux-host-entries.ttyS1-115200
1 file changed, 1 insertion(+), 1 deletion(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/91/405291/1
diff --git a/modules/install_server/files/dhcpd/linux-host-entries.ttyS1-115200
b/modules/install_server/files/dhcpd/linux-host-entries.ttyS1-115200
index 668841f..dd9424f 100644
--- a/modules/install_server/files/dhcpd/linux-host-entries.ttyS1-115200
+++ b/modules/install_server/files/dhcpd/linux-host-entries.ttyS1-115200
@@ -6170,7 +6170,7 @@
}
host mw2140 {
-hardware ethernet B0:83:FE:DE:FD:BA;
+hardware ethernet D0:94:66:0B:6F:DE;
fixed-address mw2140.codfw.wmnet;
option pxelinux.pathprefix
"http://apt.wikimedia.org/tftpboot/jessie-installer/";;
filename "lpxelinux.0";
--
To view, visit https://gerrit.wikimedia.org/r/405291
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Iaa38e693170b4eebcc9749fb2496f70b8cbb8e1f
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: profile::hadoop::worker: install spark2 package after Hive c...
Elukey has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/405263 )
Change subject: profile::hadoop::worker: install spark2 package after Hive
config
..
profile::hadoop::worker: install spark2 package after Hive config
While testing a new hadoop node in labs we noticed that spark2-shell
wasn't working due to the absence of the following symlink:
/etc/spark/conf/hive-site.xml ->
/etc/hive/conf.analytics-hadoop-labs/hive-site.xml
The symlink is done by the spark2 package, that needs to be installed
after include ::profile::hive::client, not before.
Bug: T166248
Change-Id: I4a4dcf072ef5224f3b1b506c8d562e40de1cc540
---
M modules/profile/manifests/hadoop/worker.pp
1 file changed, 7 insertions(+), 1 deletion(-)
Approvals:
Elukey: Looks good to me, approved
jenkins-bot: Verified
diff --git a/modules/profile/manifests/hadoop/worker.pp
b/modules/profile/manifests/hadoop/worker.pp
index 7f32ff1..d0f2a6b 100644
--- a/modules/profile/manifests/hadoop/worker.pp
+++ b/modules/profile/manifests/hadoop/worker.pp
@@ -48,7 +48,13 @@
class { '::cdh::spark': }
# Spark 2 is manually packaged by us, it is not part of CDH.
-require_package('spark2')
+# The deb package creates as post-install step a symlink like
+# /etc/spark/conf/hive-site.xml -> /etc/hive/conf.analytics/hive-site.xml
+# This package needs to be installed after the deploy of the Hive
configuration.
+# (should be guaranteed by the puppet evaluation order).
+package { 'spark2':
+ensure => present,
+}
# sqoop needs to be on worker nodes if Oozie is to
# launch sqoop jobs.
--
To view, visit https://gerrit.wikimedia.org/r/405263
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I4a4dcf072ef5224f3b1b506c8d562e40de1cc540
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
Gerrit-Reviewer: Elukey
Gerrit-Reviewer: Ottomata
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: profile::hadoop::worker: install spark2 package after Hive c...
Elukey has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/405263 )
Change subject: profile::hadoop::worker: install spark2 package after Hive
config
..
profile::hadoop::worker: install spark2 package after Hive config
While testing a new hadoop node in labs we noticed that spark2-shell
wasn't working due to the absence of the following symlink:
/etc/spark/conf/hive-site.xml ->
/etc/hive/conf.analytics-hadoop-labs/hive-site.xml
The symlink is done by the spark2 package, that needs to be installed
after include ::profile::hive::client, not before.
Bug: T166248
Change-Id: I4a4dcf072ef5224f3b1b506c8d562e40de1cc540
---
M modules/profile/manifests/hadoop/worker.pp
1 file changed, 7 insertions(+), 1 deletion(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/63/405263/1
diff --git a/modules/profile/manifests/hadoop/worker.pp
b/modules/profile/manifests/hadoop/worker.pp
index 7f32ff1..d0f2a6b 100644
--- a/modules/profile/manifests/hadoop/worker.pp
+++ b/modules/profile/manifests/hadoop/worker.pp
@@ -48,7 +48,13 @@
class { '::cdh::spark': }
# Spark 2 is manually packaged by us, it is not part of CDH.
-require_package('spark2')
+# The deb package creates as post-install step a symlink like
+# /etc/spark/conf/hive-site.xml -> /etc/hive/conf.analytics/hive-site.xml
+# This package needs to be installed after the deploy of the Hive
configuration.
+# (should be guaranteed by the puppet evaluation order).
+package { 'spark2':
+ensure => present,
+}
# sqoop needs to be on worker nodes if Oozie is to
# launch sqoop jobs.
--
To view, visit https://gerrit.wikimedia.org/r/405263
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I4a4dcf072ef5224f3b1b506c8d562e40de1cc540
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Allow to set the JAVA_HOME env variable in hadoop/hive/oozie
Elukey has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/404954 )
Change subject: Allow to set the JAVA_HOME env variable in hadoop/hive/oozie
..
Allow to set the JAVA_HOME env variable in hadoop/hive/oozie
Bug: T166248
Change-Id: I58a0ce3e7610e21a72493571830b2a2c105e9ada
---
M modules/profile/manifests/hadoop/common.pp
M modules/profile/manifests/hive/client.pp
M modules/profile/manifests/oozie/server.pp
3 files changed, 10 insertions(+), 0 deletions(-)
Approvals:
Elukey: Looks good to me, approved
jenkins-bot: Verified
diff --git a/modules/profile/manifests/hadoop/common.pp
b/modules/profile/manifests/hadoop/common.pp
index eb03376..afe5e07 100644
--- a/modules/profile/manifests/hadoop/common.pp
+++ b/modules/profile/manifests/hadoop/common.pp
@@ -119,6 +119,9 @@
#Yarn scheduler specific setting.
#Default: undef
#
+# [*java_home*]
+#Sets the JAVA_HOME env. variable in hadoop-env.sh
+#
class profile::hadoop::common (
$zookeeper_clusters = hiera('zookeeper_clusters'),
$zookeeper_cluster_name =
hiera('profile::hadoop::common::zookeeper_cluster_name'),
@@ -148,6 +151,7 @@
$yarn_scheduler_maximum_allocation_mb =
hiera('profile::hadoop::common::yarn_scheduler_maximum_allocation_mb', undef),
$yarn_scheduler_minimum_allocation_vcores =
hiera('profile::hadoop::common::yarn_scheduler_minimum_allocation_vcores',
undef),
$yarn_scheduler_maximum_allocation_vcores =
hiera('profile::hadoop::common::yarn_scheduler_maximum_allocation_vcores',
undef),
+$java_home=
hiera('profile::hadoop::common::java_home', undef),
) {
# Include Wikimedia's thirdparty/cloudera apt component
# as an apt source on all Hadoop hosts. This is needed
@@ -255,6 +259,8 @@
# > 200G free.
'yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage'
=> '99.0',
},
+
+java_home => $java_home,
}
class { '::ores::base': }
diff --git a/modules/profile/manifests/hive/client.pp
b/modules/profile/manifests/hive/client.pp
index b8edb60..4e676a5 100644
--- a/modules/profile/manifests/hive/client.pp
+++ b/modules/profile/manifests/hive/client.pp
@@ -10,6 +10,7 @@
$metastore_host =
hiera('profile::hive::client::hive_metastore_host'),
$hive_server_opts = hiera('profile::hive::client::hive_server_opts',
undef),
$hive_metastore_opts=
hiera('profile::hive::client::hive_metastore_opts', undef),
+$java_home = hiera('profile::hive::client::java_home', undef),
) {
require ::profile::hadoop::common
@@ -38,6 +39,7 @@
hive_server_opts => $hive_server_opts,
hive_metastore_opts => $hive_metastore_opts,
metastore_host=> $metastore_host,
+java_home => $java_home,
}
# Set up a wrapper script for beeline, the command line
diff --git a/modules/profile/manifests/oozie/server.pp
b/modules/profile/manifests/oozie/server.pp
index 161543b..5738e3d 100644
--- a/modules/profile/manifests/oozie/server.pp
+++ b/modules/profile/manifests/oozie/server.pp
@@ -6,6 +6,7 @@
$monitoring_enabled = hiera('profile::oozie::server::monitoring_enabled',
false),
$ferm_srange= hiera('profile::oozie::server::ferm_srange',
'$DOMAIN_NETWORKS'),
$jvm_opts = hiera('profile::oozie::server::jvm_opts',
'-Xmx2048m'),
+$java_home = hiera('profile::oozie::server::java_home', undef)
) {
require ::profile::oozie::client
@@ -29,6 +30,7 @@
# settings look like they are properly configured.
authorization_service_authorization_enabled => false,
jvm_opts=> $jvm_opts,
+java_home => $java_home,
}
# Oozie is creating event logs in /var/log/oozie.
--
To view, visit https://gerrit.wikimedia.org/r/404954
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I58a0ce3e7610e21a72493571830b2a2c105e9ada
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
Gerrit-Reviewer: Elukey
Gerrit-Reviewer: Ottomata
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Allow to set the JAVA_HOME env variable in hadoop/hive/oozie
Elukey has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/404954 )
Change subject: Allow to set the JAVA_HOME env variable in hadoop/hive/oozie
..
Allow to set the JAVA_HOME env variable in hadoop/hive/oozie
Bug: T166248
Change-Id: I58a0ce3e7610e21a72493571830b2a2c105e9ada
---
M modules/profile/manifests/hadoop/common.pp
M modules/profile/manifests/hive/client.pp
M modules/profile/manifests/oozie/client.pp
3 files changed, 10 insertions(+), 0 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/54/404954/1
diff --git a/modules/profile/manifests/hadoop/common.pp
b/modules/profile/manifests/hadoop/common.pp
index eb03376..afe5e07 100644
--- a/modules/profile/manifests/hadoop/common.pp
+++ b/modules/profile/manifests/hadoop/common.pp
@@ -119,6 +119,9 @@
#Yarn scheduler specific setting.
#Default: undef
#
+# [*java_home*]
+#Sets the JAVA_HOME env. variable in hadoop-env.sh
+#
class profile::hadoop::common (
$zookeeper_clusters = hiera('zookeeper_clusters'),
$zookeeper_cluster_name =
hiera('profile::hadoop::common::zookeeper_cluster_name'),
@@ -148,6 +151,7 @@
$yarn_scheduler_maximum_allocation_mb =
hiera('profile::hadoop::common::yarn_scheduler_maximum_allocation_mb', undef),
$yarn_scheduler_minimum_allocation_vcores =
hiera('profile::hadoop::common::yarn_scheduler_minimum_allocation_vcores',
undef),
$yarn_scheduler_maximum_allocation_vcores =
hiera('profile::hadoop::common::yarn_scheduler_maximum_allocation_vcores',
undef),
+$java_home=
hiera('profile::hadoop::common::java_home', undef),
) {
# Include Wikimedia's thirdparty/cloudera apt component
# as an apt source on all Hadoop hosts. This is needed
@@ -255,6 +259,8 @@
# > 200G free.
'yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage'
=> '99.0',
},
+
+java_home => $java_home,
}
class { '::ores::base': }
diff --git a/modules/profile/manifests/hive/client.pp
b/modules/profile/manifests/hive/client.pp
index b8edb60..4e676a5 100644
--- a/modules/profile/manifests/hive/client.pp
+++ b/modules/profile/manifests/hive/client.pp
@@ -10,6 +10,7 @@
$metastore_host =
hiera('profile::hive::client::hive_metastore_host'),
$hive_server_opts = hiera('profile::hive::client::hive_server_opts',
undef),
$hive_metastore_opts=
hiera('profile::hive::client::hive_metastore_opts', undef),
+$java_home = hiera('profile::hive::client::java_home', undef),
) {
require ::profile::hadoop::common
@@ -38,6 +39,7 @@
hive_server_opts => $hive_server_opts,
hive_metastore_opts => $hive_metastore_opts,
metastore_host=> $metastore_host,
+java_home => $java_home,
}
# Set up a wrapper script for beeline, the command line
diff --git a/modules/profile/manifests/oozie/client.pp
b/modules/profile/manifests/oozie/client.pp
index f63c81a..c2fe377 100644
--- a/modules/profile/manifests/oozie/client.pp
+++ b/modules/profile/manifests/oozie/client.pp
@@ -2,8 +2,10 @@
#
class profile::oozie::client(
$oozie_host = hiera('profile::oozie::client::oozie_host'),
+$java_home = hiera('profile::oozie::java_home', undef)
) {
class { '::cdh::oozie':
oozie_host => $oozie_host,
+java_home => $java_home,
}
}
\ No newline at end of file
--
To view, visit https://gerrit.wikimedia.org/r/404954
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I58a0ce3e7610e21a72493571830b2a2c105e9ada
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Update the cdh module to the latest sha
Elukey has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/404685 ) Change subject: Update the cdh module to the latest sha .. Update the cdh module to the latest sha Related to https://gerrit.wikimedia.org/r/#/c/403701/ Bug: T166248 Change-Id: I8f6569d7721f09f3dad07605d0ecb6c77987dfbf --- M modules/cdh 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Elukey: Looks good to me, approved jenkins-bot: Verified diff --git a/modules/cdh b/modules/cdh index 0f137db..4fff7f3 16 --- a/modules/cdh +++ b/modules/cdh @@ -1 +1 @@ -Subproject commit 0f137dbf35996fd1a48a4984345f21398e869c4e +Subproject commit 4fff7f3ba01740786399a8afc1b88e97b7b0d480 -- To view, visit https://gerrit.wikimedia.org/r/404685 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I8f6569d7721f09f3dad07605d0ecb6c77987dfbf Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey Gerrit-Reviewer: Elukey Gerrit-Reviewer: jenkins-bot <> ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Update the cdh module to the latest sha
Elukey has uploaded a new change for review. ( https://gerrit.wikimedia.org/r/404685 ) Change subject: Update the cdh module to the latest sha .. Update the cdh module to the latest sha Related to https://gerrit.wikimedia.org/r/#/c/403701/ Bug: T166248 Change-Id: I8f6569d7721f09f3dad07605d0ecb6c77987dfbf --- M modules/cdh 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/85/404685/1 diff --git a/modules/cdh b/modules/cdh index 0f137db..4fff7f3 16 --- a/modules/cdh +++ b/modules/cdh @@ -1 +1 @@ -Subproject commit 0f137dbf35996fd1a48a4984345f21398e869c4e +Subproject commit 4fff7f3ba01740786399a8afc1b88e97b7b0d480 -- To view, visit https://gerrit.wikimedia.org/r/404685 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I8f6569d7721f09f3dad07605d0ecb6c77987dfbf Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations...cdh[master]: Allow to explicitly set the JAVA_HOME environment variable
Elukey has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/403701 ) Change subject: Allow to explicitly set the JAVA_HOME environment variable .. Allow to explicitly set the JAVA_HOME environment variable Bug: T166248 Change-Id: Ifd3298a00cb788ea1728882c966b099398f7da76 --- M manifests/hadoop.pp M manifests/hadoop/defaults.pp M manifests/hive.pp M manifests/hive/defaults.pp M manifests/oozie/defaults.pp M manifests/oozie/server.pp M templates/hadoop/hadoop-env.sh.erb M templates/hadoop/yarn-env.sh.erb M templates/hive/hive-env.sh.erb M templates/oozie/oozie-env.sh.erb 10 files changed, 26 insertions(+), 3 deletions(-) Approvals: Ottomata: Looks good to me, but someone else must approve Elukey: Looks good to me, approved jenkins-bot: Verified diff --git a/manifests/hadoop.pp b/manifests/hadoop.pp index b6ca051..b74c9bd 100644 --- a/manifests/hadoop.pp +++ b/manifests/hadoop.pp @@ -144,6 +144,8 @@ $resourcemanager_hosts = $namenode_hosts, $zookeeper_hosts = $::cdh::hadoop::defaults::zookeeper_hosts, +$java_home = $::cdh::hadoop::defaults::java_home, + $yarn_local_path = $::cdh::hadoop::defaults::yarn_local_path, $yarn_logs_path = $::cdh::hadoop::defaults::yarn_logs_path, $dfs_block_size = $::cdh::hadoop::defaults::dfs_block_size, diff --git a/manifests/hadoop/defaults.pp b/manifests/hadoop/defaults.pp index e1807b5..32d854e 100644 --- a/manifests/hadoop/defaults.pp +++ b/manifests/hadoop/defaults.pp @@ -13,6 +13,8 @@ # provided value of $namenode_hosts in hadoop.pp. $zookeeper_hosts = undef +$java_home = undef + $yarn_local_path = 'yarn/local' $yarn_logs_path = 'yarn/logs' $dfs_block_size = 67108864 # 64MB default diff --git a/manifests/hive.pp b/manifests/hive.pp index b80ee3e..711c360 100644 --- a/manifests/hive.pp +++ b/manifests/hive.pp @@ -86,6 +86,8 @@ $hive_exec_log4j_template= $cdh::hive::defaults::hive_exec_log4j_template, $hive_env_template = $cdh::hive::defaults::hive_env_template, +$java_home = $cdh::hive::defaults::java_home, + $hive_metastore_opts = $cdh::hive::defaults::hive_metastore_opts, $hive_server_opts= $cdh::hive::defaults::hive_server_opts, $hive_metastore_jmx_port = $cdh::hive::defaults::hive_metastore_jmx_port, diff --git a/manifests/hive/defaults.pp b/manifests/hive/defaults.pp index 1513b2b..c5023fd 100644 --- a/manifests/hive/defaults.pp +++ b/manifests/hive/defaults.pp @@ -49,6 +49,8 @@ $hive_exec_log4j_template= 'cdh/hive/hive-exec-log4j.properties.erb' $hive_env_template = 'cdh/hive/hive-env.sh.erb' +$java_home = undef + # Jmx ports $hive_server_jmx_port= 9978 $hive_metastore_jmx_port = 9979 diff --git a/manifests/oozie/defaults.pp b/manifests/oozie/defaults.pp index e830b43..d705578 100644 --- a/manifests/oozie/defaults.pp +++ b/manifests/oozie/defaults.pp @@ -22,6 +22,7 @@ $authorization_service_authorization_enabled = true $admin_users = ['hdfs'] +$java_home = undef $jvm_opts= '-Xmx1024m' $purge_jobs_older_than_days = 90 diff --git a/manifests/oozie/server.pp b/manifests/oozie/server.pp index 8f5340a..cfe88ad 100644 --- a/manifests/oozie/server.pp +++ b/manifests/oozie/server.pp @@ -61,6 +61,7 @@ $authorization_service_authorization_enabled = $cdh::oozie::defaults::authorization_service_authorization_enabled, $admin_users = $cdh::oozie::defaults::admin_users, +$java_home = $cdh::oozie::defaults::java_home, $jvm_opts= $cdh::oozie::defaults::jvm_opts, $purge_jobs_older_than_days = $cdh::oozie::defaults::purge_jobs_older_than_days, $oozie_site_template = $cdh::oozie::defaults::oozie_site_template, diff --git a/templates/hadoop/hadoop-env.sh.erb b/templates/hadoop/hadoop-env.sh.erb index 2857f93..c80a38d 100644 --- a/templates/hadoop/hadoop-env.sh.erb +++ b/templates/hadoop/hadoop-env.sh.erb @@ -1,5 +1,9 @@ # Note: This file is managed by Puppet. +<% if @java_home -%> +export JAVA_HOME=<%= @java_home %> +<% end -%> + # Use YARN for all hadoop commands export HADOOP_MAPRED_HOME=/usr/lib/hadoop-mapreduce diff --git a/templates/hadoop/yarn-env.sh.erb b/templates/hadoop/yarn-env.sh.erb index 127c62d..aa3b209
[MediaWiki-commits] [Gerrit] operations/puppet[production]: profile::mariadb::misc::eventlogging: add two schemas to the...
Elukey has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/402907 ) Change subject: profile::mariadb::misc::eventlogging: add two schemas to the whitelist .. profile::mariadb::misc::eventlogging: add two schemas to the whitelist Working out details for retention with Roan and team, so this is just preliminary. Right now the change is adding all fields from Schema:ChangesListFilterGrouping and Schema:ChangesListFilters except userId. That's privacy-sensitive and it prevents data from being dropped for now, while we come up with a better long-term solution. Change-Id: I3e9cc04a32a04283d2722a5d8659e3408c9bdee9 --- M modules/profile/files/mariadb/misc/eventlogging/eventlogging_purging_whitelist.tsv 1 file changed, 33 insertions(+), 0 deletions(-) Approvals: Mforns: Looks good to me, but someone else must approve Elukey: Looks good to me, approved jenkins-bot: Verified diff --git a/modules/profile/files/mariadb/misc/eventlogging/eventlogging_purging_whitelist.tsv b/modules/profile/files/mariadb/misc/eventlogging/eventlogging_purging_whitelist.tsv index ec3dd9c..ee6e92c 100644 --- a/modules/profile/files/mariadb/misc/eventlogging/eventlogging_purging_whitelist.tsv +++ b/modules/profile/files/mariadb/misc/eventlogging/eventlogging_purging_whitelist.tsv @@ -1,3 +1,36 @@ +ChangesListFilterGrouping webHost +ChangesListFilterGrouping wiki +ChangesListFilterGrouping event_action +ChangesListFilterGrouping event_groupIdentifier +ChangesListFilterGrouping event_filter +ChangesListFilters webHost +ChangesListFilters wiki +ChangesListFilters event_pagename +ChangesListFilters event_enhancedFiltersEnabled +ChangesListFilters event_hideminor +ChangesListFilters event_hidemajor +ChangesListFilters event_hidebots +ChangesListFilters event_hidehumans +ChangesListFilters event_hideanons +ChangesListFilters event_hidepatrolled +ChangesListFilters event_hideunpatrolled +ChangesListFilters event_hidemyself +ChangesListFilters event_hidebyothers +ChangesListFilters event_hideliu +ChangesListFilters event_hidecategorization +ChangesListFilters event_hidepageedits +ChangesListFilters event_hidenewpages +ChangesListFilters event_hidelog +ChangesListFilters event_hideWikibase +ChangesListFilters event_hideReviewed +ChangesListFilters event_hidenondamaging +ChangesListFilters event_damaging +ChangesListFilters event_goodfaith +ChangesListFilters event_namespace +ChangesListFilters event_userExpLevel +ChangesListFilters event_tagfilter +ChangesListFilters event_invert +ChangesListFilters event_associated DiacriticsPoll webHost DiacriticsPoll wiki DiacriticsPoll event_country -- To view, visit https://gerrit.wikimedia.org/r/402907 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I3e9cc04a32a04283d2722a5d8659e3408c9bdee9 Gerrit-PatchSet: 5 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Milimetric Gerrit-Reviewer: Catrope Gerrit-Reviewer: Elukey Gerrit-Reviewer: Mforns Gerrit-Reviewer: jenkins-bot <> ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] analytics/kafkatee[master]: Release 0.1.7-1
Elukey has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/404465 ) Change subject: Release 0.1.7-1 .. Release 0.1.7-1 Change-Id: I31c323de012ce8ffd18a220933e4ae89fb3a8a06 --- M debian/changelog 1 file changed, 13 insertions(+), 0 deletions(-) Approvals: Elukey: Verified; Looks good to me, approved diff --git a/debian/changelog b/debian/changelog index 83328d9..e3c23ba 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,3 +1,16 @@ +kafkatee (0.1.7-1) unstable; urgency=medium + + * Set Maintainer to ops, add myself to Uploaders. + * Bump debhelper compatibility level to 11. + * Bump Standards-Versions to 4.1.3. + * Remove the kafkatee upstart script, Upstart is deprecated. + * Don't create (or remove) /var/run/kafkatee anymore, not used by the +package and wasn't set up properly anyway. + * Drop versioned Build-Depends to pre-jessie versions. + * Fix indentation of the extended description. + + -- Faidon Liambotis Tue, 16 Jan 2018 16:41:39 +0200 + kafkatee (0.1.6-1) unstable; urgency=medium * Remove logrotate and syslog configuration (will be handled by -- To view, visit https://gerrit.wikimedia.org/r/404465 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I31c323de012ce8ffd18a220933e4ae89fb3a8a06 Gerrit-PatchSet: 1 Gerrit-Project: analytics/kafkatee Gerrit-Branch: master Gerrit-Owner: Faidon Liambotis Gerrit-Reviewer: Elukey ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] analytics/kafkatee[master]: Set Maintainer to ops, add myself to Uploaders
Elukey has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/404464 ) Change subject: Set Maintainer to ops, add myself to Uploaders .. Set Maintainer to ops, add myself to Uploaders Change-Id: I45bf4095b305beb598f053e8c4979386624704d1 --- M debian/control 1 file changed, 2 insertions(+), 1 deletion(-) Approvals: Elukey: Verified; Looks good to me, approved diff --git a/debian/control b/debian/control index b382259..660c8a4 100644 --- a/debian/control +++ b/debian/control @@ -1,7 +1,8 @@ Source: kafkatee Section: web Priority: optional -Maintainer: Andrew Otto (WMF) +Maintainer: Wikimedia Technical Operations +Uploaders: Andrew Otto , Faidon Liambotis Build-Depends: debhelper (>= 11~), librdkafka-dev, libyajl-dev, zlib1g-dev Standards-Version: 4.1.3 Vcs-Git: https://gerrit.wikimedia.org/r/analytics/kafkatee -- To view, visit https://gerrit.wikimedia.org/r/404464 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I45bf4095b305beb598f053e8c4979386624704d1 Gerrit-PatchSet: 1 Gerrit-Project: analytics/kafkatee Gerrit-Branch: master Gerrit-Owner: Faidon Liambotis Gerrit-Reviewer: Elukey ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] analytics/kafkatee[master]: d/control: fix indentation of the extended description
Elukey has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/404462 ) Change subject: d/control: fix indentation of the extended description .. d/control: fix indentation of the extended description Change-Id: Ib970647f236ee4086464513b7a7caaedb950a45a --- M debian/control 1 file changed, 8 insertions(+), 9 deletions(-) Approvals: Elukey: Verified; Looks good to me, approved diff --git a/debian/control b/debian/control index 1a4d923..bdd0799 100644 --- a/debian/control +++ b/debian/control @@ -19,12 +19,11 @@ rate. . Features: - . - - Supported input types: Kafka consumer or piped command. - - Supported output types: Piped command or file. - - Configuration file syntax is backwards compatible with Wikimedia's udp2log - - Configurable output queue size - - Memory frugal: message payload is shared by all output queues - - Configured with configuration file - - Operates as a daemon (daemonization, pidfile) - - Closes/stops and reopens/restarts all inputs and outputs on SIGHUP. + * Supported input types: Kafka consumer or piped command. + * Supported output types: Piped command or file. + * Configuration file syntax is backwards compatible with Wikimedia's udp2log + * Configurable output queue size + * Memory frugal: message payload is shared by all output queues + * Configured with configuration file + * Operates as a daemon (daemonization, pidfile) + * Closes/stops and reopens/restarts all inputs and outputs on SIGHUP. -- To view, visit https://gerrit.wikimedia.org/r/404462 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ib970647f236ee4086464513b7a7caaedb950a45a Gerrit-PatchSet: 1 Gerrit-Project: analytics/kafkatee Gerrit-Branch: master Gerrit-Owner: Faidon Liambotis Gerrit-Reviewer: Elukey ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] analytics/kafkatee[master]: Drop versioned Build-Depends to pre-jessie versions
Elukey has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/404463 ) Change subject: Drop versioned Build-Depends to pre-jessie versions .. Drop versioned Build-Depends to pre-jessie versions Change-Id: I3414af898ee702eebb25d607b9758dbaa235206f --- M debian/control 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Elukey: Verified; Looks good to me, approved diff --git a/debian/control b/debian/control index bdd0799..b382259 100644 --- a/debian/control +++ b/debian/control @@ -2,7 +2,7 @@ Section: web Priority: optional Maintainer: Andrew Otto (WMF) -Build-Depends: debhelper (>= 11~), librdkafka-dev (>= 0.8.3), libyajl-dev, zlib1g-dev (>= 1:1.2.8) +Build-Depends: debhelper (>= 11~), librdkafka-dev, libyajl-dev, zlib1g-dev Standards-Version: 4.1.3 Vcs-Git: https://gerrit.wikimedia.org/r/analytics/kafkatee Vcs-Browser: https://phabricator.wikimedia.org/diffusion/ANKA/ -- To view, visit https://gerrit.wikimedia.org/r/404463 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I3414af898ee702eebb25d607b9758dbaa235206f Gerrit-PatchSet: 1 Gerrit-Project: analytics/kafkatee Gerrit-Branch: master Gerrit-Owner: Faidon Liambotis Gerrit-Reviewer: Elukey ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] analytics/kafkatee[master]: Remove +x from debian/kafkatee.post{inst, rm}
Elukey has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/404461 )
Change subject: Remove +x from debian/kafkatee.post{inst,rm}
..
Remove +x from debian/kafkatee.post{inst,rm}
Change-Id: Iab569eefcdece8150fc3cf0af3b787c1d417ccb3
---
M debian/kafkatee.postinst
M debian/kafkatee.postrm
2 files changed, 0 insertions(+), 0 deletions(-)
Approvals:
Elukey: Verified; Looks good to me, approved
diff --git a/debian/kafkatee.postinst b/debian/kafkatee.postinst
old mode 100755
new mode 100644
diff --git a/debian/kafkatee.postrm b/debian/kafkatee.postrm
old mode 100755
new mode 100644
--
To view, visit https://gerrit.wikimedia.org/r/404461
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Iab569eefcdece8150fc3cf0af3b787c1d417ccb3
Gerrit-PatchSet: 1
Gerrit-Project: analytics/kafkatee
Gerrit-Branch: master
Gerrit-Owner: Faidon Liambotis
Gerrit-Reviewer: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] analytics/kafkatee[master]: Use /run instead of /var/run in the source
Elukey has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/404460 )
Change subject: Use /run instead of /var/run in the source
..
Use /run instead of /var/run in the source
Not applicable to the Debian package, but good idea to have
nevertheless.
Change-Id: I8ebb79914f5c883c594cd7731f3fcba2de1d50a8
---
M kafkatee.conf.example
M kafkatee.h
2 files changed, 4 insertions(+), 4 deletions(-)
Approvals:
Elukey: Verified; Looks good to me, approved
diff --git a/kafkatee.conf.example b/kafkatee.conf.example
index 60e6065..8ba4a00 100644
--- a/kafkatee.conf.example
+++ b/kafkatee.conf.example
@@ -33,7 +33,7 @@
###
# Pid file location
-# Default: /var/run/kafkatee.pid
+# Default: /run/kafkatee.pid
#pid.file.path = kafkatee.pid
# Daemonize (background)
@@ -106,7 +106,7 @@
# Offset file directory.
# Each topic + partition combination has its own offset file.
# Default: current directory
-#kafka.topic.offset.store.path = /var/run/offsets/
+#kafka.topic.offset.store.path = /var/cache/kafkatee/offsets/
# If the request offset was not found on broker, or there is no
# initial offset known (no stored offset), then reset the offset according
@@ -349,4 +349,4 @@
###
# Include other config file
-#include local.conf
\ No newline at end of file
+#include local.conf
diff --git a/kafkatee.h b/kafkatee.h
index f65c9ee..235ad5c 100644
--- a/kafkatee.h
+++ b/kafkatee.h
@@ -45,7 +45,7 @@
#include
#define KAFKATEE_CONF_PATH "/etc/kafkatee.conf"
-#define KAFKATEE_PID_FILE_PATH "/var/run/kafkatee.pid"
+#define KAFKATEE_PID_FILE_PATH "/run/kafkatee.pid"
#define _DBG(fmt...) do { \
if (conf.log_level > 6) { \
--
To view, visit https://gerrit.wikimedia.org/r/404460
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I8ebb79914f5c883c594cd7731f3fcba2de1d50a8
Gerrit-PatchSet: 1
Gerrit-Project: analytics/kafkatee
Gerrit-Branch: master
Gerrit-Owner: Faidon Liambotis
Gerrit-Reviewer: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] analytics/kafkatee[master]: Don't create (or remove) /var/run/kafkatee anymore
Elukey has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/404459 ) Change subject: Don't create (or remove) /var/run/kafkatee anymore .. Don't create (or remove) /var/run/kafkatee anymore We don't use the pidfile by default anymore, so it's a no-op as far as Debian packaging goes. Also, creating /var/run directories in packages is wrong and not guaranteed to work anyway (because /(var/)run may be a tmpfs, or regularly cleaned up). Change-Id: I7b02fa161c9e9f49c9854d93bbc5c35c313cbbca --- M debian/kafkatee.dirs M debian/kafkatee.postinst M debian/kafkatee.postrm 3 files changed, 1 insertion(+), 4 deletions(-) Approvals: Elukey: Verified; Looks good to me, approved diff --git a/debian/kafkatee.dirs b/debian/kafkatee.dirs index 610d1cc..5b6e9fb 100644 --- a/debian/kafkatee.dirs +++ b/debian/kafkatee.dirs @@ -1,4 +1,3 @@ etc/kafkatee.d var/cache/kafkatee var/cache/kafkatee/offsets -var/run/kafkatee \ No newline at end of file diff --git a/debian/kafkatee.postinst b/debian/kafkatee.postinst index 1b3033d..345203c 100755 --- a/debian/kafkatee.postinst +++ b/debian/kafkatee.postinst @@ -20,9 +20,8 @@ kafkatee >/dev/null 2>/dev/null || : fi -# chown /var/cache/kafkatee and /var/run/kafkatee for writing +# chown /var/cache/kafkatee for writing chown -R kafkatee:kafkatee /var/cache/kafkatee -chown -R kafkatee:kafkatee /var/run/kafkatee echo \ "NOTE: Default kafkatee config files have been installed at: diff --git a/debian/kafkatee.postrm b/debian/kafkatee.postrm index 49c1ce9..bce1e39 100755 --- a/debian/kafkatee.postrm +++ b/debian/kafkatee.postrm @@ -10,7 +10,6 @@ purge) rm -f /var/log/kafkatee.log* rm -rf /var/cache/kafkatee -rm -rf /var/run/kafkatee deluser --quiet --system kafkatee >/dev/null || true deluser --quiet --group --system --only-if-empty --remove-home kafkatee || true -- To view, visit https://gerrit.wikimedia.org/r/404459 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I7b02fa161c9e9f49c9854d93bbc5c35c313cbbca Gerrit-PatchSet: 1 Gerrit-Project: analytics/kafkatee Gerrit-Branch: master Gerrit-Owner: Faidon Liambotis Gerrit-Reviewer: Elukey ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] analytics/kafkatee[master]: Remove kafkatee upstart script
Elukey has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/404458 ) Change subject: Remove kafkatee upstart script .. Remove kafkatee upstart script Upstart is not supported anymore. Change-Id: Ic153731d266b0df8c6af841e2f02e1ef1b9937c6 --- D debian/kafkatee.upstart 1 file changed, 0 insertions(+), 13 deletions(-) Approvals: Elukey: Verified; Looks good to me, approved diff --git a/debian/kafkatee.upstart b/debian/kafkatee.upstart deleted file mode 100644 index 4aea276..000 --- a/debian/kafkatee.upstart +++ /dev/null @@ -1,13 +0,0 @@ -description "kafkatee" - -setuid kafkatee -setgid kafkatee - -respawn -respawn limit 15 5 - -start on (local-filesystems and net-device-up IFACE!=lo) -stop on runlevel [!2345] - -exec /usr/bin/kafkatee -D -c /etc/kafkatee.conf - -- To view, visit https://gerrit.wikimedia.org/r/404458 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ic153731d266b0df8c6af841e2f02e1ef1b9937c6 Gerrit-PatchSet: 1 Gerrit-Project: analytics/kafkatee Gerrit-Branch: master Gerrit-Owner: Faidon Liambotis Gerrit-Reviewer: Elukey ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] analytics/kafkatee[master]: Bump debhelper compat to 11, Standards-Version
Elukey has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/404457 ) Change subject: Bump debhelper compat to 11, Standards-Version .. Bump debhelper compat to 11, Standards-Version Change-Id: I5e8a6939519e19200860445fa25b7026adcfd801 --- M debian/compat M debian/control 2 files changed, 3 insertions(+), 3 deletions(-) Approvals: Elukey: Verified; Looks good to me, approved diff --git a/debian/compat b/debian/compat index ec63514..b4de394 100644 --- a/debian/compat +++ b/debian/compat @@ -1 +1 @@ -9 +11 diff --git a/debian/control b/debian/control index 6f93434..1a4d923 100644 --- a/debian/control +++ b/debian/control @@ -2,8 +2,8 @@ Section: web Priority: optional Maintainer: Andrew Otto (WMF) -Build-Depends: debhelper (>= 9), dh-systemd (>= 1.5), librdkafka-dev (>= 0.8.3), libyajl-dev, zlib1g-dev (>= 1:1.2.8) -Standards-Version: 3.9.5 +Build-Depends: debhelper (>= 11~), librdkafka-dev (>= 0.8.3), libyajl-dev, zlib1g-dev (>= 1:1.2.8) +Standards-Version: 4.1.3 Vcs-Git: https://gerrit.wikimedia.org/r/analytics/kafkatee Vcs-Browser: https://phabricator.wikimedia.org/diffusion/ANKA/ -- To view, visit https://gerrit.wikimedia.org/r/404457 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I5e8a6939519e19200860445fa25b7026adcfd801 Gerrit-PatchSet: 1 Gerrit-Project: analytics/kafkatee Gerrit-Branch: master Gerrit-Owner: Faidon Liambotis Gerrit-Reviewer: Elukey ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] analytics/kafkatee[master]: Define IOV_MAX properly, using _GNU_SOURCE
Elukey has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/404456 ) Change subject: Define IOV_MAX properly, using _GNU_SOURCE .. Define IOV_MAX properly, using _GNU_SOURCE Undo the hack previously done by defining __need_IOV_MAX, that is apparently not working anymore in current distributions. Instead, include limits.h, and define _GNU_SOURCE at the beginning as those definitions are only included via some XOPEN_SOURCE variant. Change-Id: I550c612a71c6c615c54c3b6c9c8bae91db1ad99b --- M output.c 1 file changed, 2 insertions(+), 1 deletion(-) Approvals: Elukey: Verified; Looks good to me, approved diff --git a/output.c b/output.c index 32e3543..53798cc 100644 --- a/output.c +++ b/output.c @@ -26,7 +26,7 @@ * POSSIBILITY OF SUCH DAMAGE. */ -#define __need_IOV_MAX +#define _GNU_SOURCE #include "kafkatee.h" #include "queue.h" @@ -43,6 +43,7 @@ #include #include #include +#include static LIST_HEAD(, output_s) outputs; /* Sorted in sample-rate order */ -- To view, visit https://gerrit.wikimedia.org/r/404456 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I550c612a71c6c615c54c3b6c9c8bae91db1ad99b Gerrit-PatchSet: 1 Gerrit-Project: analytics/kafkatee Gerrit-Branch: master Gerrit-Owner: Faidon Liambotis Gerrit-Reviewer: Elukey ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: profile::puppetmaster::puppetdb: add jmx metrics to export
Elukey has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/404427 ) Change subject: profile::puppetmaster::puppetdb: add jmx metrics to export .. profile::puppetmaster::puppetdb: add jmx metrics to export These mbeans are the ones exposed by the puppetdb's dashboard. Bug: T184796 Change-Id: I919ab2bc5de2dd6fbf4820663afc22785c8f8458 --- M modules/profile/files/puppetmaster/puppetdb/jvm_prometheus_puppetdb_jmx_exporter.yaml 1 file changed, 6 insertions(+), 1 deletion(-) Approvals: Giuseppe Lavagetto: Looks good to me, but someone else must approve Alexandros Kosiaris: Looks good to me, but someone else must approve Elukey: Looks good to me, approved jenkins-bot: Verified Filippo Giunchedi: Looks good to me, but someone else must approve diff --git a/modules/profile/files/puppetmaster/puppetdb/jvm_prometheus_puppetdb_jmx_exporter.yaml b/modules/profile/files/puppetmaster/puppetdb/jvm_prometheus_puppetdb_jmx_exporter.yaml index 6ea2bc3..b04b537 100644 --- a/modules/profile/files/puppetmaster/puppetdb/jvm_prometheus_puppetdb_jmx_exporter.yaml +++ b/modules/profile/files/puppetmaster/puppetdb/jvm_prometheus_puppetdb_jmx_exporter.yaml @@ -5,4 +5,9 @@ - 'com.puppetlabs.puppetdb.command:type=global,name=*' - 'com.puppetlabs.puppetdb.command:type=replace facts.3,name=*' - 'com.puppetlabs.puppetdb.http.server:type=/v3/commands,name=*' - - 'com.puppetlabs.puppetdb.http.server:type=/v3/nodes,name=*' \ No newline at end of file + - 'com.puppetlabs.puppetdb.http.server:type=/v3/nodes,name=*' + - 'com.puppetlabs.puppetdb.query.population:type=default,name=num-nodes' + - 'com.puppetlabs.puppetdb.query.population:type=default,name=num-resources' + - 'com.puppetlabs.puppetdb.query.population:type=default,name=pct-resource-dupes' + - 'com.puppetlabs.puppetdb.scf.storage:type=default,name=duplicate-pct' + - 'org.apache.activemq:BrokerName=localhost,Type=Queue,Destination=com.puppetlabs.puppetdb.commands' \ No newline at end of file -- To view, visit https://gerrit.wikimedia.org/r/404427 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I919ab2bc5de2dd6fbf4820663afc22785c8f8458 Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey Gerrit-Reviewer: Alexandros Kosiaris Gerrit-Reviewer: Elukey Gerrit-Reviewer: Filippo Giunchedi Gerrit-Reviewer: Giuseppe Lavagetto Gerrit-Reviewer: jenkins-bot <> ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: profile::puppetmaster::puppetdb: add jmx metrics to export
Elukey has uploaded a new change for review. ( https://gerrit.wikimedia.org/r/404427 ) Change subject: profile::puppetmaster::puppetdb: add jmx metrics to export .. profile::puppetmaster::puppetdb: add jmx metrics to export These mbeans are the ones exposed by the puppetdb's dashboard. Bug: T184796 Change-Id: I919ab2bc5de2dd6fbf4820663afc22785c8f8458 --- M modules/profile/files/puppetmaster/puppetdb/jvm_prometheus_puppetdb_jmx_exporter.yaml 1 file changed, 6 insertions(+), 1 deletion(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/27/404427/1 diff --git a/modules/profile/files/puppetmaster/puppetdb/jvm_prometheus_puppetdb_jmx_exporter.yaml b/modules/profile/files/puppetmaster/puppetdb/jvm_prometheus_puppetdb_jmx_exporter.yaml index 6ea2bc3..b04b537 100644 --- a/modules/profile/files/puppetmaster/puppetdb/jvm_prometheus_puppetdb_jmx_exporter.yaml +++ b/modules/profile/files/puppetmaster/puppetdb/jvm_prometheus_puppetdb_jmx_exporter.yaml @@ -5,4 +5,9 @@ - 'com.puppetlabs.puppetdb.command:type=global,name=*' - 'com.puppetlabs.puppetdb.command:type=replace facts.3,name=*' - 'com.puppetlabs.puppetdb.http.server:type=/v3/commands,name=*' - - 'com.puppetlabs.puppetdb.http.server:type=/v3/nodes,name=*' \ No newline at end of file + - 'com.puppetlabs.puppetdb.http.server:type=/v3/nodes,name=*' + - 'com.puppetlabs.puppetdb.query.population:type=default,name=num-nodes' + - 'com.puppetlabs.puppetdb.query.population:type=default,name=num-resources' + - 'com.puppetlabs.puppetdb.query.population:type=default,name=pct-resource-dupes' + - 'com.puppetlabs.puppetdb.scf.storage:type=default,name=duplicate-pct' + - 'org.apache.activemq:BrokerName=localhost,Type=Queue,Destination=com.puppetlabs.puppetdb.commands' \ No newline at end of file -- To view, visit https://gerrit.wikimedia.org/r/404427 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I919ab2bc5de2dd6fbf4820663afc22785c8f8458 Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::prometheus::ops: add puppetdb metrics
Elukey has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/404269 )
Change subject: role::prometheus::ops: add puppetdb metrics
..
role::prometheus::ops: add puppetdb metrics
Bug: T184796
Change-Id: If53cdce855e1550f5b142c2556ad995c678665e6
---
M modules/role/manifests/prometheus/ops.pp
1 file changed, 12 insertions(+), 1 deletion(-)
Approvals:
jenkins-bot: Verified
Filippo Giunchedi: Looks good to me, approved
diff --git a/modules/role/manifests/prometheus/ops.pp
b/modules/role/manifests/prometheus/ops.pp
index 0976ddc..b24c3e8 100644
--- a/modules/role/manifests/prometheus/ops.pp
+++ b/modules/role/manifests/prometheus/ops.pp
@@ -578,10 +578,16 @@
$jmx_exporter_jobs = [
{
'job_name'=> 'jmx_kafka',
-'scrape_timeout' => '25s',
'scheme' => 'http',
'file_sd_configs' => [
{ 'files' => [ "${targets_path}/jmx_kafka_*.yaml" ]}
+],
+ },
+ {
+'job_name'=> 'jmx_puppetdb',
+'scheme' => 'http',
+'file_sd_configs' => [
+ { 'files' => [ "${targets_path}/jmx_puppetdb_*.yaml" ]}
],
},
]
@@ -592,6 +598,11 @@
site => $::site,
}
+prometheus::jmx_exporter_config{ "puppetdb_${::site}":
+dest => "${targets_path}/jmx_puppetdb_${::site}.yaml",
+class_name => 'role::puppetmaster::puppetdb',
+site => $::site,
+}
$etherpad_jobs = [
{
--
To view, visit https://gerrit.wikimedia.org/r/404269
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: If53cdce855e1550f5b142c2556ad995c678665e6
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
Gerrit-Reviewer: Elukey
Gerrit-Reviewer: Filippo Giunchedi
Gerrit-Reviewer: Giuseppe Lavagetto
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::prometheus::ops: add puppetdb metrics
Elukey has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/404269 )
Change subject: role::prometheus::ops: add puppetdb metrics
..
role::prometheus::ops: add puppetdb metrics
Bug: T184796
Change-Id: If53cdce855e1550f5b142c2556ad995c678665e6
---
M modules/role/manifests/prometheus/ops.pp
1 file changed, 7 insertions(+), 2 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/69/404269/1
diff --git a/modules/role/manifests/prometheus/ops.pp
b/modules/role/manifests/prometheus/ops.pp
index 0976ddc..3fab650 100644
--- a/modules/role/manifests/prometheus/ops.pp
+++ b/modules/role/manifests/prometheus/ops.pp
@@ -577,11 +577,11 @@
$jmx_exporter_jobs = [
{
-'job_name'=> 'jmx_kafka',
+'job_name'=> 'jmx_exporter',
'scrape_timeout' => '25s',
'scheme' => 'http',
'file_sd_configs' => [
- { 'files' => [ "${targets_path}/jmx_kafka_*.yaml" ]}
+ { 'files' => [ "${targets_path}/jmx_*.yaml" ]}
],
},
]
@@ -592,6 +592,11 @@
site => $::site,
}
+prometheus::jmx_exporter_config{ "puppetdb_${::site}":
+dest => "${targets_path}/jmx_puppetdb_${::site}.yaml",
+class_name => 'role::puppetmaster::puppetdb',
+site => $::site,
+}
$etherpad_jobs = [
{
--
To view, visit https://gerrit.wikimedia.org/r/404269
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: If53cdce855e1550f5b142c2556ad995c678665e6
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: site.pp: add mw1338->48
Elukey has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/403928 )
Change subject: site.pp: add mw1338->48
..
site.pp: add mw1338->48
Bug: T165519
Change-Id: I688a90b1cb671f8b964d2541908ce2ef833512f7
---
M manifests/site.pp
1 file changed, 27 insertions(+), 18 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/28/403928/1
diff --git a/manifests/site.pp b/manifests/site.pp
index 8803b67..349bcdc 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1494,21 +1494,9 @@
role(mediawiki::jobrunner)
}
-# ROW C eqiad jobrunners
-node /^mw133[4-7]\.eqiad\.wmnet$/ {
-role(mediawiki::jobrunner)
-}
-
-
-# T165519
-# ROW C eqiad appservers
-node /^mw13(19|2[0-9]|3[0-3])\.eqiad\.wmnet$/ {
-role(mediawiki::appserver)
-}
-
-# ROW B eqiad api-appservers
-node /^mw13(1[2-7])\.eqiad\.wmnet$/ {
-role(mediawiki::appserver::api)
+# ROW A videoscaler
+node 'mw1307.eqiad.wmnet' {
+role(mediawiki::videoscaler)
}
# ROW A eqiad jobrunners
@@ -1516,9 +1504,9 @@
role(mediawiki::jobrunner)
}
-# ROW A videoscaler
-node 'mw1307.eqiad.wmnet' {
-role(mediawiki::videoscaler)
+# ROW B eqiad api-appservers
+node /^mw13(1[2-7])\.eqiad\.wmnet$/ {
+role(mediawiki::appserver::api)
}
# ROW B videoscaler
@@ -1526,6 +1514,27 @@
role(mediawiki::videoscaler)
}
+# T165519
+# ROW C eqiad appservers
+node /^mw13(19|2[0-9]|3[0-3])\.eqiad\.wmnet$/ {
+role(mediawiki::appserver)
+}
+
+# ROW C eqiad jobrunners
+node /^mw133[4-7]\.eqiad\.wmnet$/ {
+role(mediawiki::jobrunner)
+}
+
+# ROW C videoscaler
+node 'mw1338.eqiad.wmnet' {
+role(mediawiki::videoscaler)
+}
+
+# ROW C api-appservers
+node /^mw13(39|4[0-8])\.eqiad\.wmnet$/ {
+role(mediawiki::appserver::api)
+}
+
# ROW A codfw appservers: mw2017, mw2075-mw2079, and mw2215-2250
--
To view, visit https://gerrit.wikimedia.org/r/403928
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I688a90b1cb671f8b964d2541908ce2ef833512f7
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations...cdh[master]: Allow to explicitly set the JAVA_HOME environment variable
Elukey has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/403701 )
Change subject: Allow to explicitly set the JAVA_HOME environment variable
..
Allow to explicitly set the JAVA_HOME environment variable
Bug: T166248
Change-Id: Ifd3298a00cb788ea1728882c966b099398f7da76
---
M manifests/hadoop.pp
M manifests/hadoop/defaults.pp
M templates/hadoop/hadoop-env.sh.erb
M templates/hadoop/yarn-env.sh.erb
4 files changed, 11 insertions(+), 2 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet/cdh
refs/changes/01/403701/1
diff --git a/manifests/hadoop.pp b/manifests/hadoop.pp
index b6ca051..b74c9bd 100644
--- a/manifests/hadoop.pp
+++ b/manifests/hadoop.pp
@@ -144,6 +144,8 @@
$resourcemanager_hosts = $namenode_hosts,
$zookeeper_hosts =
$::cdh::hadoop::defaults::zookeeper_hosts,
+$java_home =
$::cdh::hadoop::defaults::java_home,
+
$yarn_local_path =
$::cdh::hadoop::defaults::yarn_local_path,
$yarn_logs_path =
$::cdh::hadoop::defaults::yarn_logs_path,
$dfs_block_size =
$::cdh::hadoop::defaults::dfs_block_size,
diff --git a/manifests/hadoop/defaults.pp b/manifests/hadoop/defaults.pp
index e1807b5..32d854e 100644
--- a/manifests/hadoop/defaults.pp
+++ b/manifests/hadoop/defaults.pp
@@ -13,6 +13,8 @@
# provided value of $namenode_hosts in hadoop.pp.
$zookeeper_hosts = undef
+$java_home = undef
+
$yarn_local_path = 'yarn/local'
$yarn_logs_path = 'yarn/logs'
$dfs_block_size = 67108864 # 64MB default
diff --git a/templates/hadoop/hadoop-env.sh.erb
b/templates/hadoop/hadoop-env.sh.erb
index 2857f93..c80a38d 100644
--- a/templates/hadoop/hadoop-env.sh.erb
+++ b/templates/hadoop/hadoop-env.sh.erb
@@ -1,5 +1,9 @@
# Note: This file is managed by Puppet.
+<% if @java_home -%>
+export JAVA_HOME=<%= @java_home %>
+<% end -%>
+
# Use YARN for all hadoop commands
export HADOOP_MAPRED_HOME=/usr/lib/hadoop-mapreduce
diff --git a/templates/hadoop/yarn-env.sh.erb b/templates/hadoop/yarn-env.sh.erb
index 127c62d..aa3b209 100644
--- a/templates/hadoop/yarn-env.sh.erb
+++ b/templates/hadoop/yarn-env.sh.erb
@@ -6,8 +6,9 @@
# resolve links - $0 may be a softlink
export YARN_CONF_DIR="${YARN_CONF_DIR:-$HADOOP_YARN_HOME/conf}"
-# some Java parameters
-# export JAVA_HOME=/home/y/libexec/jdk1.6.0/
+<% if @java_home -%>
+export JAVA_HOME=<%= @java_home %>
+<% end -%>
if [ "$JAVA_HOME" != "" ]; then
#echo "run java in $JAVA_HOME"
JAVA_HOME=$JAVA_HOME
--
To view, visit https://gerrit.wikimedia.org/r/403701
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ifd3298a00cb788ea1728882c966b099398f7da76
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet/cdh
Gerrit-Branch: master
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: profile::analytics::refinery::job::streams_check: set correc...
Elukey has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/403655 )
Change subject: profile::analytics::refinery::job::streams_check: set correct
spark jar
..
profile::analytics::refinery::job::streams_check: set correct spark jar
The jar referenced in the original patch was not the correct one.
Bug: T176983
Change-Id: I9e87859284d69dcf04a714ebb53fe1697d43ceb3
---
M modules/profile/manifests/analytics/refinery/job/streams_check.pp
1 file changed, 1 insertion(+), 1 deletion(-)
Approvals:
Elukey: Looks good to me, approved
jenkins-bot: Verified
diff --git a/modules/profile/manifests/analytics/refinery/job/streams_check.pp
b/modules/profile/manifests/analytics/refinery/job/streams_check.pp
index f0d2f5c..b99903e 100644
--- a/modules/profile/manifests/analytics/refinery/job/streams_check.pp
+++ b/modules/profile/manifests/analytics/refinery/job/streams_check.pp
@@ -11,7 +11,7 @@
# Shortcut var to DRY up cron commands.
$refinery_path = $profile::analytics::refinery::path
-$refinery_job_jar = "${refinery_path}/artifacts/refinery-job.jar"
+$refinery_job_jar = "${refinery_path}/artifacts/refinery-job-spark-2.1.jar"
$spark_num_executors = 4
$spark_executor_cores = 3
$spark_driver_memory = '2G'
--
To view, visit https://gerrit.wikimedia.org/r/403655
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I9e87859284d69dcf04a714ebb53fe1697d43ceb3
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Joal
Gerrit-Reviewer: Elukey
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: profile::analytics::refinery::job::stream_check: fix refiner...
Elukey has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/403635 )
Change subject: profile::analytics::refinery::job::stream_check: fix refinery
path
..
profile::analytics::refinery::job::stream_check: fix refinery path
Change-Id: I20dd267d77951da1d95548562fef318c59447ed3
---
M modules/profile/manifests/analytics/refinery/job/streams_check.pp
1 file changed, 1 insertion(+), 1 deletion(-)
Approvals:
Elukey: Looks good to me, approved
jenkins-bot: Verified
diff --git a/modules/profile/manifests/analytics/refinery/job/streams_check.pp
b/modules/profile/manifests/analytics/refinery/job/streams_check.pp
index 8afea43..f0d2f5c 100644
--- a/modules/profile/manifests/analytics/refinery/job/streams_check.pp
+++ b/modules/profile/manifests/analytics/refinery/job/streams_check.pp
@@ -9,7 +9,7 @@
require ::profile::analytics::refinery
# Shortcut var to DRY up cron commands.
-$refinery_path = $role::analytics_cluster::refinery::path
+$refinery_path = $profile::analytics::refinery::path
$refinery_job_jar = "${refinery_path}/artifacts/refinery-job.jar"
$spark_num_executors = 4
--
To view, visit https://gerrit.wikimedia.org/r/403635
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I20dd267d77951da1d95548562fef318c59447ed3
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
Gerrit-Reviewer: Elukey
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: profile::analytics::refinery::job::stream_check: fix refiner...
Elukey has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/403635 )
Change subject: profile::analytics::refinery::job::stream_check: fix refinery
path
..
profile::analytics::refinery::job::stream_check: fix refinery path
Change-Id: I20dd267d77951da1d95548562fef318c59447ed3
---
M modules/profile/manifests/analytics/refinery/job/streams_check.pp
1 file changed, 1 insertion(+), 1 deletion(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/35/403635/1
diff --git a/modules/profile/manifests/analytics/refinery/job/streams_check.pp
b/modules/profile/manifests/analytics/refinery/job/streams_check.pp
index 8afea43..f0d2f5c 100644
--- a/modules/profile/manifests/analytics/refinery/job/streams_check.pp
+++ b/modules/profile/manifests/analytics/refinery/job/streams_check.pp
@@ -9,7 +9,7 @@
require ::profile::analytics::refinery
# Shortcut var to DRY up cron commands.
-$refinery_path = $role::analytics_cluster::refinery::path
+$refinery_path = $profile::analytics::refinery::path
$refinery_job_jar = "${refinery_path}/artifacts/refinery-job.jar"
$spark_num_executors = 4
--
To view, visit https://gerrit.wikimedia.org/r/403635
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I20dd267d77951da1d95548562fef318c59447ed3
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::analytics_cluster::coordinator: add a profile to resta...
Elukey has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/395504 )
Change subject: role::analytics_cluster::coordinator: add a profile to restart
streaming jobs
..
role::analytics_cluster::coordinator: add a profile to restart streaming jobs
This patch adds a cron job checking and possibly relaunching banner
spark streaming job in Yarn.
Bug: T176983
Change-Id: Icdf583cce4dd6b6b69a145f57c8355bafc62aa08
---
A modules/profile/manifests/analytics/refinery/job/streams_check.pp
M modules/role/manifests/analytics_cluster/coordinator.pp
2 files changed, 35 insertions(+), 0 deletions(-)
Approvals:
Elukey: Looks good to me, approved
jenkins-bot: Verified
diff --git a/modules/profile/manifests/analytics/refinery/job/streams_check.pp
b/modules/profile/manifests/analytics/refinery/job/streams_check.pp
new file mode 100644
index 000..8afea43
--- /dev/null
+++ b/modules/profile/manifests/analytics/refinery/job/streams_check.pp
@@ -0,0 +1,34 @@
+# == Class profile::analytics::refinery::job::streams_check
+#
+# Deploy cron scripts able to check and restart (if needed) streaming jobs
+# running on the Hadoop cluster that might have failed. This profile does not
+# take care of alarming, that needs to be done separately.
+#
+
+class profile::analytics::refinery::job::streams_check {
+require ::profile::analytics::refinery
+
+# Shortcut var to DRY up cron commands.
+$refinery_path = $role::analytics_cluster::refinery::path
+
+$refinery_job_jar = "${refinery_path}/artifacts/refinery-job.jar"
+$spark_num_executors = 4
+$spark_executor_cores = 3
+$spark_driver_memory = '2G'
+$spark_executor_memory = '4G'
+$druid_segment_gran = 'HOUR'
+$tranq_window_period = 'PT10M'
+$batch_duration_secs = '60'
+$job_name = 'BannerImpressionsStream'
+
+# No log needed as job runs in cluster mode
+$command = "PYTHONPATH=${refinery_path}/python
${refinery_path}/bin/is-yarn-app-running ${job_name} || /usr/bin/spark2-submit
--master yarn --deploy-mode cluster --queue production --conf
spark.dynamicAllocation.enabled=false --driver-memory ${spark_driver_memory}
--executor-memory ${spark_executor_memory} --executor-cores
${spark_executor_cores} --num-executors ${spark_num_executors} --class
org.wikimedia.analytics.refinery.job.druid.BannerImpressionsStream --name
${job_name} ${refinery_job_jar} --druid-indexing-segment-granularity
${druid_segment_gran} --druid-indexing-window-period ${tranq_window_period}
--batch-duration-seconds ${batch_duration_secs} > /dev/null 2>&1"
+
+# This checks for banner streaming job running in Yarn, and relaunches it
if needed.
+cron { 'refinery-relaunch-banner-streaming':
+command => $command,
+environment => 'MAILTO=analytics-ale...@wikimedia.org',
+user=> 'hdfs',
+minute => '*/5'
+}
+}
diff --git a/modules/role/manifests/analytics_cluster/coordinator.pp
b/modules/role/manifests/analytics_cluster/coordinator.pp
index a96..9f38259 100644
--- a/modules/role/manifests/analytics_cluster/coordinator.pp
+++ b/modules/role/manifests/analytics_cluster/coordinator.pp
@@ -58,6 +58,7 @@
include ::profile::analytics::refinery::job::project_namespace_map
include ::profile::analytics::refinery::job::sqoop_mediawiki
include ::profile::analytics::refinery::job::json_refine
+include ::profile::analytics::refinery::job::streams_check
include standard
include ::profile::base::firewall
--
To view, visit https://gerrit.wikimedia.org/r/395504
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Icdf583cce4dd6b6b69a145f57c8355bafc62aa08
Gerrit-PatchSet: 6
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Joal
Gerrit-Reviewer: Elukey
Gerrit-Reviewer: Joal
Gerrit-Reviewer: Ottomata
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations...druid_exporter[master]: README.md: fix virtual env suggestions
Elukey has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/403613 ) Change subject: README.md: fix virtual env suggestions .. README.md: fix virtual env suggestions Thanks Aimiyoo (GH-1) Change-Id: I83ddf0b441c90dabcd6845f9bbf9e1504bdbbaca --- M README.md 1 file changed, 3 insertions(+), 1 deletion(-) Approvals: Elukey: Verified; Looks good to me, approved diff --git a/README.md b/README.md index 1000b73..9b2c2b8 100644 --- a/README.md +++ b/README.md @@ -7,9 +7,11 @@ The easiest way to run `druid_exporter` is via a virtualenv: +``` virtualenv .venv - .venv/bin/python setup.py + .venv/bin/python setup.py install .venv/bin/druid_exporter +``` By default metrics are exposed on TCP port `8000`. Python 2 is not supported. -- To view, visit https://gerrit.wikimedia.org/r/403613 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I83ddf0b441c90dabcd6845f9bbf9e1504bdbbaca Gerrit-PatchSet: 1 Gerrit-Project: operations/software/druid_exporter Gerrit-Branch: master Gerrit-Owner: Elukey Gerrit-Reviewer: Elukey ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations...druid_exporter[master]: README.md: fix virtual env suggestions
Elukey has uploaded a new change for review. ( https://gerrit.wikimedia.org/r/403613 ) Change subject: README.md: fix virtual env suggestions .. README.md: fix virtual env suggestions Thanks Aimiyoo (GH-1) Change-Id: I83ddf0b441c90dabcd6845f9bbf9e1504bdbbaca --- M README.md 1 file changed, 3 insertions(+), 1 deletion(-) git pull ssh://gerrit.wikimedia.org:29418/operations/software/druid_exporter refs/changes/13/403613/1 diff --git a/README.md b/README.md index 1000b73..9b2c2b8 100644 --- a/README.md +++ b/README.md @@ -7,9 +7,11 @@ The easiest way to run `druid_exporter` is via a virtualenv: +``` virtualenv .venv - .venv/bin/python setup.py + .venv/bin/python setup.py install .venv/bin/druid_exporter +``` By default metrics are exposed on TCP port `8000`. Python 2 is not supported. -- To view, visit https://gerrit.wikimedia.org/r/403613 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I83ddf0b441c90dabcd6845f9bbf9e1504bdbbaca Gerrit-PatchSet: 1 Gerrit-Project: operations/software/druid_exporter Gerrit-Branch: master Gerrit-Owner: Elukey ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Standardize Analytics jmx agent's configurations
Elukey has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/403123 ) Change subject: Standardize Analytics jmx agent's configurations .. Standardize Analytics jmx agent's configurations Gehel recently discovered that a set of default JVM metrics is returned by default by the jmx exporter via DefaultExports.java. Bug: T177458 Change-Id: I7f3fb7e865d8e4b33d271e8e6132af1716f0c091 --- M modules/profile/files/druid/jvm_prometheus_jmx_exporter.yaml M modules/profile/files/hadoop/prometheus_hdfs_datanode_jmx_exporter.yaml M modules/profile/files/hadoop/prometheus_hdfs_namenode_jmx_exporter.yaml M modules/profile/files/hadoop/prometheus_mapreduce_history_jmx_exporter.yaml M modules/profile/files/hadoop/prometheus_yarn_nodemanager_jmx_exporter.yaml M modules/profile/files/hadoop/prometheus_yarn_resourcemanager_jmx_exporter.yaml M modules/profile/files/hive/prometheus_hive_metastore_jmx_exporter.yaml M modules/profile/files/hive/prometheus_hive_server_jmx_exporter.yaml M modules/profile/files/oozie/prometheus_oozie_server_jmx_exporter.yaml 9 files changed, 14 insertions(+), 54 deletions(-) Approvals: Elukey: Looks good to me, approved jenkins-bot: Verified Filippo Giunchedi: Looks good to me, but someone else must approve diff --git a/modules/profile/files/druid/jvm_prometheus_jmx_exporter.yaml b/modules/profile/files/druid/jvm_prometheus_jmx_exporter.yaml index 5532f5e..72e67cf 100644 --- a/modules/profile/files/druid/jvm_prometheus_jmx_exporter.yaml +++ b/modules/profile/files/druid/jvm_prometheus_jmx_exporter.yaml @@ -1,9 +1,5 @@ --- lowercaseOutputLabelNames: true lowercaseOutputName: false -whitelistObjectNames: - - 'java.lang:type=ClassLoading' - - 'java.lang:type=Compilation' - - 'java.lang:type=GarbageCollector*' - - 'java.lang:type=Memory' - - 'java.lang:type=MemoryPool*' \ No newline at end of file +blacklistObjectNames: + - '*:*' \ No newline at end of file diff --git a/modules/profile/files/hadoop/prometheus_hdfs_datanode_jmx_exporter.yaml b/modules/profile/files/hadoop/prometheus_hdfs_datanode_jmx_exporter.yaml index f60acbe..3816217 100644 --- a/modules/profile/files/hadoop/prometheus_hdfs_datanode_jmx_exporter.yaml +++ b/modules/profile/files/hadoop/prometheus_hdfs_datanode_jmx_exporter.yaml @@ -3,9 +3,4 @@ lowercaseOutputName: false whitelistObjectNames: - 'Hadoop:service=DataNode,name=DataNodeActivity*' - - 'Hadoop:service=DataNode,name=FSDatasetState*' - - 'java.lang:type=ClassLoading' - - 'java.lang:type=Compilation' - - 'java.lang:type=GarbageCollector*' - - 'java.lang:type=Memory' - - 'java.lang:type=MemoryPool*' \ No newline at end of file + - 'Hadoop:service=DataNode,name=FSDatasetState*' \ No newline at end of file diff --git a/modules/profile/files/hadoop/prometheus_hdfs_namenode_jmx_exporter.yaml b/modules/profile/files/hadoop/prometheus_hdfs_namenode_jmx_exporter.yaml index 1063174..b10fab6 100644 --- a/modules/profile/files/hadoop/prometheus_hdfs_namenode_jmx_exporter.yaml +++ b/modules/profile/files/hadoop/prometheus_hdfs_namenode_jmx_exporter.yaml @@ -3,9 +3,4 @@ lowercaseOutputName: false whitelistObjectNames: - 'Hadoop:service=NameNode,name=FSNamesystem' - - 'Hadoop:service=NameNode,name=NameNodeActivity' - - 'java.lang:type=ClassLoading' - - 'java.lang:type=Compilation' - - 'java.lang:type=GarbageCollector*' - - 'java.lang:type=Memory' - - 'java.lang:type=MemoryPool*' \ No newline at end of file + - 'Hadoop:service=NameNode,name=NameNodeActivity' \ No newline at end of file diff --git a/modules/profile/files/hadoop/prometheus_mapreduce_history_jmx_exporter.yaml b/modules/profile/files/hadoop/prometheus_mapreduce_history_jmx_exporter.yaml index 5532f5e..72e67cf 100644 --- a/modules/profile/files/hadoop/prometheus_mapreduce_history_jmx_exporter.yaml +++ b/modules/profile/files/hadoop/prometheus_mapreduce_history_jmx_exporter.yaml @@ -1,9 +1,5 @@ --- lowercaseOutputLabelNames: true lowercaseOutputName: false -whitelistObjectNames: - - 'java.lang:type=ClassLoading' - - 'java.lang:type=Compilation' - - 'java.lang:type=GarbageCollector*' - - 'java.lang:type=Memory' - - 'java.lang:type=MemoryPool*' \ No newline at end of file +blacklistObjectNames: + - '*:*' \ No newline at end of file diff --git a/modules/profile/files/hadoop/prometheus_yarn_nodemanager_jmx_exporter.yaml b/modules/profile/files/hadoop/prometheus_yarn_nodemanager_jmx_exporter.yaml index f1a7ace..d044eae 100644 --- a/modules/profile/files/hadoop/prometheus_yarn_nodemanager_jmx_exporter.yaml +++ b/modules/profile/files/hadoop/prometheus_yarn_nodemanager_jmx_exporter.yaml @@ -3,9 +3,4 @@ lowercaseOutputName: false whitelistObjectNames: - 'Hadoop:service=NodeManager,name=NodeManagerMetrics' - - 'Hadoop:service=NodeManager,name=ShuffleMetrics' - - 'java.lang:type=ClassLoading' - - 'java.lang:type=Compilation' - - 'java.lang:type=GarbageCollector*' - - 'java.la
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Beta: Moving all docroots to standard-docroot
Elukey has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/394203 ) Change subject: Beta: Moving all docroots to standard-docroot .. Beta: Moving all docroots to standard-docroot Now that we've unified all the docroots, let's start to actually use the unified docroots instead of symlinking everywhere Bug: T126306 Change-Id: Iba9b3cfe6017bf2a1b0472ccabf31940b3ecbcf3 --- M modules/mediawiki/files/apache/beta/sites/wikibooks.conf M modules/mediawiki/files/apache/beta/sites/wikidata.conf M modules/mediawiki/files/apache/beta/sites/wikinews.conf M modules/mediawiki/files/apache/beta/sites/wikiquote.conf M modules/mediawiki/files/apache/beta/sites/wikisource.conf M modules/mediawiki/files/apache/beta/sites/wikiversity.conf M modules/mediawiki/files/apache/beta/sites/wikivoyage.conf M modules/mediawiki/files/apache/beta/sites/wiktionary.conf 8 files changed, 122 insertions(+), 122 deletions(-) Approvals: Krinkle: Looks good to me, but someone else must approve Elukey: Looks good to me, approved jenkins-bot: Verified diff --git a/modules/mediawiki/files/apache/beta/sites/wikibooks.conf b/modules/mediawiki/files/apache/beta/sites/wikibooks.conf index 0bd7de1..38bc582 100644 --- a/modules/mediawiki/files/apache/beta/sites/wikibooks.conf +++ b/modules/mediawiki/files/apache/beta/sites/wikibooks.conf @@ -2,32 +2,32 @@ ServerName wikibooks ServerAlias *.wikibooks.beta.wmflabs.org UseCanonicalName off -DocumentRoot "/srv/mediawiki/docroot/wikibooks.org" +DocumentRoot "/srv/mediawiki/docroot/standard-docroot" AllowEncodedSlashes On RewriteEngine On -ProxyPass /wiki fcgi://127.0.0.1:9000/srv/mediawiki/docroot/wikibooks.org/w/index.php retry=0 +ProxyPass /wiki fcgi://127.0.0.1:9000/srv/mediawiki/docroot/standard-docroot/w/index.php retry=0 ProxyPass /w/extensions! -ProxyPassMatch ^/w/(.*\.(php|hh))$ fcgi://127.0.0.1:9000/srv/mediawiki/docroot/wikibooks.org/w/$1 -ProxyPassMatch ^/zh(-(hans|hant|cn|hk|sg|tw))|sr(-(ec|el)) fcgi://127.0.0.1:9000/srv/mediawiki/docroot/wikibooks.org/w/index.php retry=0 -RewriteRule ^/$ fcgi://127.0.0.1:9000/srv/mediawiki/docroot/wikibooks.org/w/index.php[P] -RewriteRule ^/robots\.txt$ fcgi://127.0.0.1:9000/srv/mediawiki/docroot/wikibooks.org/w/robots.php [P] -RewriteRule ^/favicon\.ico$ fcgi://127.0.0.1:9000/srv/mediawiki/docroot/wikibooks.org/w/favicon.php [P] +ProxyPassMatch ^/w/(.*\.(php|hh))$ fcgi://127.0.0.1:9000/srv/mediawiki/docroot/standard-docroot/w/$1 +ProxyPassMatch ^/zh(-(hans|hant|cn|hk|sg|tw))|sr(-(ec|el)) fcgi://127.0.0.1:9000/srv/mediawiki/docroot/standard-docroot/w/index.php retry=0 +RewriteRule ^/$ fcgi://127.0.0.1:9000/srv/mediawiki/docroot/standard-docroot/w/index.php[P] +RewriteRule ^/robots\.txt$ fcgi://127.0.0.1:9000/srv/mediawiki/docroot/standard-docroot/w/robots.php [P] +RewriteRule ^/favicon\.ico$ fcgi://127.0.0.1:9000/srv/mediawiki/docroot/standard-docroot/w/favicon.php [P] # Primary wiki redirector -Alias /wiki /srv/mediawiki/docroot/wikibooks.org/w/index.php +Alias /wiki /srv/mediawiki/docroot/standard-docroot/w/index.php Include "sites-enabled/public-wiki-rewrites.incl" # Variant aliases -Alias /sr /srv/mediawiki/docroot/wikibooks.org/w/index.php -Alias /sr-ec /srv/mediawiki/docroot/wikibooks.org/w/index.php -Alias /sr-el /srv/mediawiki/docroot/wikibooks.org/w/index.php -Alias /zh /srv/mediawiki/docroot/wikibooks.org/w/index.php -Alias /zh-hans /srv/mediawiki/docroot/wikibooks.org/w/index.php -Alias /zh-hant /srv/mediawiki/docroot/wikibooks.org/w/index.php -Alias /zh-cn /srv/mediawiki/docroot/wikibooks.org/w/index.php -Alias /zh-hk /srv/mediawiki/docroot/wikibooks.org/w/index.php -Alias /zh-sg /srv/mediawiki/docroot/wikibooks.org/w/index.php -Alias /zh-tw /srv/mediawiki/docroot/wikibooks.org/w/index.php +Alias /sr /srv/mediawiki/docroot/standard-docroot/w/index.php +Alias /sr-ec /srv/mediawiki/docroot/standard-docroot/w/index.php +Alias /sr-el /srv/mediawiki/docroot/standard-docroot/w/index.php +Alias /zh /srv/mediawiki/docroot/standard-docroot/w/index.php +Alias /zh-hans /srv/mediawiki/docroot/standard-docroot/w/index.php +Alias /zh-hant /srv/mediawiki/docroot/standard-docroot/w/index.php +Alias /zh-cn /srv/mediawiki/docroot/standard-docroot/w/index.php +Alias /zh-hk /srv/mediawiki/docroot/standard-docroot/w/index.php +Alias /zh-sg /srv/mediawiki/docroot/standard-docroot/w/index.php +Alias /zh-tw /srv/mediawiki/docroot/standard-docroot/w/index.php diff --git a/modules/mediawiki/files/apache/beta/sites/wikidata.conf b/modules/mediawiki/files/apache/beta/sites/wikidata.conf index 523af0d..d746bef
[MediaWiki-commits] [Gerrit] operations/puppet[production]: profile::analytics::database::meta: add ferm rules hiera par...
Elukey has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/403131 )
Change subject: profile::analytics::database::meta: add ferm rules hiera
parameter
..
profile::analytics::database::meta: add ferm rules hiera parameter
This change should allow an easier deployment of the Meta Database
in labs.
Bug: T166248
Change-Id: Ic19e2e4383c5e6280815bbb47cd1a83b3d655504
---
M hieradata/role/common/analytics_cluster/coordinator.yaml
M modules/profile/manifests/analytics/database/meta.pp
2 files changed, 4 insertions(+), 2 deletions(-)
Approvals:
Elukey: Looks good to me, approved
jenkins-bot: Verified
diff --git a/hieradata/role/common/analytics_cluster/coordinator.yaml
b/hieradata/role/common/analytics_cluster/coordinator.yaml
index 460bfaf..6fa5e57 100644
--- a/hieradata/role/common/analytics_cluster/coordinator.yaml
+++ b/hieradata/role/common/analytics_cluster/coordinator.yaml
@@ -6,10 +6,11 @@
profile::hive::metastore::monitoring_enabled: true
profile::hive::metastore::ferm_srange: '$ANALYTICS_NETWORKS'
profile::analytics::database::meta::monitoring_enabled: true
+profile::analytics::database::meta::ferm_srange: '(($DRUID_PUBLIC_HOSTS
$ANALYTICS_NETWORKS))'
profile::oozie::server::monitoring_enabled: true
profile::oozie::server::ferm_srange: '$ANALYTICS_NETWORKS'
# Following Cloudera recommandations for ~20 users
profile::hive::client::hive_metastore_opts: '-Xms4g -Xmx10g'
profile::hive::client::hive_server_opts: '-Xms4g -Xmx6g'
-profile::oozie::server::jvm_opts: '-Xms2g -Xmx2g'
\ No newline at end of file
+profile::oozie::server::jvm_opts: '-Xms2g -Xmx2g'
diff --git a/modules/profile/manifests/analytics/database/meta.pp
b/modules/profile/manifests/analytics/database/meta.pp
index a28fba9..c888f12 100644
--- a/modules/profile/manifests/analytics/database/meta.pp
+++ b/modules/profile/manifests/analytics/database/meta.pp
@@ -5,6 +5,7 @@
#
class profile::analytics::database::meta(
$monitoring_enabled =
hiera('profile::analytics::database::meta::monitoring_enabled', false),
+$ferm_srange=
hiera('profile::analytics::database::meta::ferm_srange', '$DOMAIN_NETWORKS'),
) {
# Some CDH database init scripts need Java to run.
require ::profile::java::analytics
@@ -51,7 +52,7 @@
ferm::service{ 'analytics-mysql-meta':
proto => 'tcp',
port => '3306',
-srange => '(($DRUID_PUBLIC_HOSTS $ANALYTICS_NETWORKS))',
+srange => $ferm_srange,
}
# Include icinga alerts if production realm.
--
To view, visit https://gerrit.wikimedia.org/r/403131
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Ic19e2e4383c5e6280815bbb47cd1a83b3d655504
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
Gerrit-Reviewer: Elukey
Gerrit-Reviewer: Ottomata
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: profile::analytics::database::meta: add ferm rules hiera par...
Elukey has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/403131 )
Change subject: profile::analytics::database::meta: add ferm rules hiera
parameter
..
profile::analytics::database::meta: add ferm rules hiera parameter
This change should allow an easier deployment of the Meta Database
in labs.
Bug: T166248
Change-Id: Ic19e2e4383c5e6280815bbb47cd1a83b3d655504
---
M hieradata/role/common/analytics_cluster/coordinator.yaml
M modules/profile/manifests/analytics/database/meta.pp
2 files changed, 4 insertions(+), 2 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/31/403131/1
diff --git a/hieradata/role/common/analytics_cluster/coordinator.yaml
b/hieradata/role/common/analytics_cluster/coordinator.yaml
index 460bfaf..6fa5e57 100644
--- a/hieradata/role/common/analytics_cluster/coordinator.yaml
+++ b/hieradata/role/common/analytics_cluster/coordinator.yaml
@@ -6,10 +6,11 @@
profile::hive::metastore::monitoring_enabled: true
profile::hive::metastore::ferm_srange: '$ANALYTICS_NETWORKS'
profile::analytics::database::meta::monitoring_enabled: true
+profile::analytics::database::meta::ferm_srange: '(($DRUID_PUBLIC_HOSTS
$ANALYTICS_NETWORKS))'
profile::oozie::server::monitoring_enabled: true
profile::oozie::server::ferm_srange: '$ANALYTICS_NETWORKS'
# Following Cloudera recommandations for ~20 users
profile::hive::client::hive_metastore_opts: '-Xms4g -Xmx10g'
profile::hive::client::hive_server_opts: '-Xms4g -Xmx6g'
-profile::oozie::server::jvm_opts: '-Xms2g -Xmx2g'
\ No newline at end of file
+profile::oozie::server::jvm_opts: '-Xms2g -Xmx2g'
diff --git a/modules/profile/manifests/analytics/database/meta.pp
b/modules/profile/manifests/analytics/database/meta.pp
index a28fba9..c888f12 100644
--- a/modules/profile/manifests/analytics/database/meta.pp
+++ b/modules/profile/manifests/analytics/database/meta.pp
@@ -5,6 +5,7 @@
#
class profile::analytics::database::meta(
$monitoring_enabled =
hiera('profile::analytics::database::meta::monitoring_enabled', false),
+$ferm_srange=
hiera('profile::analytics::database::meta::ferm_srange', '$DOMAIN_NETWORKS'),
) {
# Some CDH database init scripts need Java to run.
require ::profile::java::analytics
@@ -51,7 +52,7 @@
ferm::service{ 'analytics-mysql-meta':
proto => 'tcp',
port => '3306',
-srange => '(($DRUID_PUBLIC_HOSTS $ANALYTICS_NETWORKS))',
+srange => $ferm_srange,
}
# Include icinga alerts if production realm.
--
To view, visit https://gerrit.wikimedia.org/r/403131
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic19e2e4383c5e6280815bbb47cd1a83b3d655504
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: profile::hive/oozie: add a hiera parameter for the ferm srange
Elukey has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/403128 )
Change subject: profile::hive/oozie: add a hiera parameter for the ferm srange
..
profile::hive/oozie: add a hiera parameter for the ferm srange
This change should allow a easier deployment/set-up of Hive/Oozie
daemons in labs.
Bug: T166248
Change-Id: Ibe97eb5b2f50260b4c4a74ad73263a76292aeccc
---
M hieradata/role/common/analytics_cluster/coordinator.yaml
M modules/profile/manifests/hive/metastore.pp
M modules/profile/manifests/hive/server.pp
M modules/profile/manifests/oozie/server.pp
4 files changed, 9 insertions(+), 3 deletions(-)
Approvals:
Elukey: Looks good to me, approved
jenkins-bot: Verified
diff --git a/hieradata/role/common/analytics_cluster/coordinator.yaml
b/hieradata/role/common/analytics_cluster/coordinator.yaml
index 545cfd1..460bfaf 100644
--- a/hieradata/role/common/analytics_cluster/coordinator.yaml
+++ b/hieradata/role/common/analytics_cluster/coordinator.yaml
@@ -2,9 +2,12 @@
profile::analytics::database::meta::backup::rsync_dest:
'analytics1002.eqiad.wmnet::backup/mysql/analytics-meta'
profile::hive::server::monitoring_enabled: true
+profile::hive::server::ferm_srange: '$ANALYTICS_NETWORKS'
profile::hive::metastore::monitoring_enabled: true
+profile::hive::metastore::ferm_srange: '$ANALYTICS_NETWORKS'
profile::analytics::database::meta::monitoring_enabled: true
profile::oozie::server::monitoring_enabled: true
+profile::oozie::server::ferm_srange: '$ANALYTICS_NETWORKS'
# Following Cloudera recommandations for ~20 users
profile::hive::client::hive_metastore_opts: '-Xms4g -Xmx10g'
diff --git a/modules/profile/manifests/hive/metastore.pp
b/modules/profile/manifests/hive/metastore.pp
index b5cc8ca..ffd51ca 100644
--- a/modules/profile/manifests/hive/metastore.pp
+++ b/modules/profile/manifests/hive/metastore.pp
@@ -4,6 +4,7 @@
#
class profile::hive::metastore(
$monitoring_enabled =
hiera('profile::hive::metastore::monitoring_enabled', false),
+$ferm_srange= hiera('profile::hive::metastore::ferm_srange',
'$DOMAIN_NETWORKS'),
$statsd = hiera('statsd'),
) {
@@ -20,7 +21,7 @@
ferm::service{ 'hive_metastore':
proto => 'tcp',
port => '9083',
-srange => '$ANALYTICS_NETWORKS',
+srange => $ferm_srange,
}
# Include icinga alerts if production realm.
diff --git a/modules/profile/manifests/hive/server.pp
b/modules/profile/manifests/hive/server.pp
index e70afca..e1ac94c 100644
--- a/modules/profile/manifests/hive/server.pp
+++ b/modules/profile/manifests/hive/server.pp
@@ -4,6 +4,7 @@
#
class profile::hive::server(
$monitoring_enabled = hiera('profile::hive::server::monitoring_enabled',
false),
+$ferm_srange = hiera('profile::hive::server::ferm_srange',
'$DOMAIN_NETWORKS'),
$statsd = hiera('statsd')
) {
include ::profile::hive::client
@@ -19,7 +20,7 @@
ferm::service{ 'hive_server':
proto => 'tcp',
port => '1',
-srange => '$ANALYTICS_NETWORKS',
+srange => $ferm_srange,
}
# Include icinga alerts if production realm.
diff --git a/modules/profile/manifests/oozie/server.pp
b/modules/profile/manifests/oozie/server.pp
index 1a7cac9..161543b 100644
--- a/modules/profile/manifests/oozie/server.pp
+++ b/modules/profile/manifests/oozie/server.pp
@@ -4,6 +4,7 @@
#
class profile::oozie::server(
$monitoring_enabled = hiera('profile::oozie::server::monitoring_enabled',
false),
+$ferm_srange= hiera('profile::oozie::server::ferm_srange',
'$DOMAIN_NETWORKS'),
$jvm_opts = hiera('profile::oozie::server::jvm_opts',
'-Xmx2048m'),
) {
require ::profile::oozie::client
@@ -43,7 +44,7 @@
ferm::service{ 'oozie_server':
proto => 'tcp',
port => '11000',
-srange => '$ANALYTICS_NETWORKS',
+srange => $ferm_srange,
}
# Include icinga alerts if production realm.
--
To view, visit https://gerrit.wikimedia.org/r/403128
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Ibe97eb5b2f50260b4c4a74ad73263a76292aeccc
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
Gerrit-Reviewer: Elukey
Gerrit-Reviewer: Ottomata
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: profile::hive/oozie: add a hiera parameter for the ferm srange
Elukey has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/403128 )
Change subject: profile::hive/oozie: add a hiera parameter for the ferm srange
..
profile::hive/oozie: add a hiera parameter for the ferm srange
This change should allow a easier deployment/set-up of Hive/Oozie
daemons in labs.
Bug: T166248
Change-Id: Ibe97eb5b2f50260b4c4a74ad73263a76292aeccc
---
M hieradata/role/common/analytics_cluster/coordinator.yaml
M modules/profile/manifests/hive/metastore.pp
M modules/profile/manifests/hive/server.pp
M modules/profile/manifests/oozie/server.pp
4 files changed, 9 insertions(+), 3 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/28/403128/1
diff --git a/hieradata/role/common/analytics_cluster/coordinator.yaml
b/hieradata/role/common/analytics_cluster/coordinator.yaml
index 545cfd1..460bfaf 100644
--- a/hieradata/role/common/analytics_cluster/coordinator.yaml
+++ b/hieradata/role/common/analytics_cluster/coordinator.yaml
@@ -2,9 +2,12 @@
profile::analytics::database::meta::backup::rsync_dest:
'analytics1002.eqiad.wmnet::backup/mysql/analytics-meta'
profile::hive::server::monitoring_enabled: true
+profile::hive::server::ferm_srange: '$ANALYTICS_NETWORKS'
profile::hive::metastore::monitoring_enabled: true
+profile::hive::metastore::ferm_srange: '$ANALYTICS_NETWORKS'
profile::analytics::database::meta::monitoring_enabled: true
profile::oozie::server::monitoring_enabled: true
+profile::oozie::server::ferm_srange: '$ANALYTICS_NETWORKS'
# Following Cloudera recommandations for ~20 users
profile::hive::client::hive_metastore_opts: '-Xms4g -Xmx10g'
diff --git a/modules/profile/manifests/hive/metastore.pp
b/modules/profile/manifests/hive/metastore.pp
index b5cc8ca..ffd51ca 100644
--- a/modules/profile/manifests/hive/metastore.pp
+++ b/modules/profile/manifests/hive/metastore.pp
@@ -4,6 +4,7 @@
#
class profile::hive::metastore(
$monitoring_enabled =
hiera('profile::hive::metastore::monitoring_enabled', false),
+$ferm_srange= hiera('profile::hive::metastore::ferm_srange',
'$DOMAIN_NETWORKS'),
$statsd = hiera('statsd'),
) {
@@ -20,7 +21,7 @@
ferm::service{ 'hive_metastore':
proto => 'tcp',
port => '9083',
-srange => '$ANALYTICS_NETWORKS',
+srange => $ferm_srange,
}
# Include icinga alerts if production realm.
diff --git a/modules/profile/manifests/hive/server.pp
b/modules/profile/manifests/hive/server.pp
index e70afca..e1ac94c 100644
--- a/modules/profile/manifests/hive/server.pp
+++ b/modules/profile/manifests/hive/server.pp
@@ -4,6 +4,7 @@
#
class profile::hive::server(
$monitoring_enabled = hiera('profile::hive::server::monitoring_enabled',
false),
+$ferm_srange = hiera('profile::hive::server::ferm_srange',
'$DOMAIN_NETWORKS'),
$statsd = hiera('statsd')
) {
include ::profile::hive::client
@@ -19,7 +20,7 @@
ferm::service{ 'hive_server':
proto => 'tcp',
port => '1',
-srange => '$ANALYTICS_NETWORKS',
+srange => $ferm_srange,
}
# Include icinga alerts if production realm.
diff --git a/modules/profile/manifests/oozie/server.pp
b/modules/profile/manifests/oozie/server.pp
index 1a7cac9..161543b 100644
--- a/modules/profile/manifests/oozie/server.pp
+++ b/modules/profile/manifests/oozie/server.pp
@@ -4,6 +4,7 @@
#
class profile::oozie::server(
$monitoring_enabled = hiera('profile::oozie::server::monitoring_enabled',
false),
+$ferm_srange= hiera('profile::oozie::server::ferm_srange',
'$DOMAIN_NETWORKS'),
$jvm_opts = hiera('profile::oozie::server::jvm_opts',
'-Xmx2048m'),
) {
require ::profile::oozie::client
@@ -43,7 +44,7 @@
ferm::service{ 'oozie_server':
proto => 'tcp',
port => '11000',
-srange => '$ANALYTICS_NETWORKS',
+srange => $ferm_srange,
}
# Include icinga alerts if production realm.
--
To view, visit https://gerrit.wikimedia.org/r/403128
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ibe97eb5b2f50260b4c4a74ad73263a76292aeccc
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Standardize Analytics jmx agent's configurations
Elukey has uploaded a new change for review. ( https://gerrit.wikimedia.org/r/403123 ) Change subject: Standardize Analytics jmx agent's configurations .. Standardize Analytics jmx agent's configurations Gehel recently discovered that a set of default JVM metrics is returned by default by the jmx exporter via DefaultExports.java. Bug: T177458 Change-Id: I7f3fb7e865d8e4b33d271e8e6132af1716f0c091 --- M modules/profile/files/druid/jvm_prometheus_jmx_exporter.yaml M modules/profile/files/hadoop/prometheus_hdfs_datanode_jmx_exporter.yaml M modules/profile/files/hadoop/prometheus_hdfs_namenode_jmx_exporter.yaml M modules/profile/files/hadoop/prometheus_mapreduce_history_jmx_exporter.yaml M modules/profile/files/hadoop/prometheus_yarn_nodemanager_jmx_exporter.yaml M modules/profile/files/hadoop/prometheus_yarn_resourcemanager_jmx_exporter.yaml M modules/profile/files/hive/prometheus_hive_metastore_jmx_exporter.yaml M modules/profile/files/hive/prometheus_hive_server_jmx_exporter.yaml M modules/profile/files/oozie/prometheus_oozie_server_jmx_exporter.yaml 9 files changed, 14 insertions(+), 54 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/23/403123/1 diff --git a/modules/profile/files/druid/jvm_prometheus_jmx_exporter.yaml b/modules/profile/files/druid/jvm_prometheus_jmx_exporter.yaml index 5532f5e..72e67cf 100644 --- a/modules/profile/files/druid/jvm_prometheus_jmx_exporter.yaml +++ b/modules/profile/files/druid/jvm_prometheus_jmx_exporter.yaml @@ -1,9 +1,5 @@ --- lowercaseOutputLabelNames: true lowercaseOutputName: false -whitelistObjectNames: - - 'java.lang:type=ClassLoading' - - 'java.lang:type=Compilation' - - 'java.lang:type=GarbageCollector*' - - 'java.lang:type=Memory' - - 'java.lang:type=MemoryPool*' \ No newline at end of file +blacklistObjectNames: + - '*:*' \ No newline at end of file diff --git a/modules/profile/files/hadoop/prometheus_hdfs_datanode_jmx_exporter.yaml b/modules/profile/files/hadoop/prometheus_hdfs_datanode_jmx_exporter.yaml index f60acbe..3816217 100644 --- a/modules/profile/files/hadoop/prometheus_hdfs_datanode_jmx_exporter.yaml +++ b/modules/profile/files/hadoop/prometheus_hdfs_datanode_jmx_exporter.yaml @@ -3,9 +3,4 @@ lowercaseOutputName: false whitelistObjectNames: - 'Hadoop:service=DataNode,name=DataNodeActivity*' - - 'Hadoop:service=DataNode,name=FSDatasetState*' - - 'java.lang:type=ClassLoading' - - 'java.lang:type=Compilation' - - 'java.lang:type=GarbageCollector*' - - 'java.lang:type=Memory' - - 'java.lang:type=MemoryPool*' \ No newline at end of file + - 'Hadoop:service=DataNode,name=FSDatasetState*' \ No newline at end of file diff --git a/modules/profile/files/hadoop/prometheus_hdfs_namenode_jmx_exporter.yaml b/modules/profile/files/hadoop/prometheus_hdfs_namenode_jmx_exporter.yaml index 1063174..b10fab6 100644 --- a/modules/profile/files/hadoop/prometheus_hdfs_namenode_jmx_exporter.yaml +++ b/modules/profile/files/hadoop/prometheus_hdfs_namenode_jmx_exporter.yaml @@ -3,9 +3,4 @@ lowercaseOutputName: false whitelistObjectNames: - 'Hadoop:service=NameNode,name=FSNamesystem' - - 'Hadoop:service=NameNode,name=NameNodeActivity' - - 'java.lang:type=ClassLoading' - - 'java.lang:type=Compilation' - - 'java.lang:type=GarbageCollector*' - - 'java.lang:type=Memory' - - 'java.lang:type=MemoryPool*' \ No newline at end of file + - 'Hadoop:service=NameNode,name=NameNodeActivity' \ No newline at end of file diff --git a/modules/profile/files/hadoop/prometheus_mapreduce_history_jmx_exporter.yaml b/modules/profile/files/hadoop/prometheus_mapreduce_history_jmx_exporter.yaml index 5532f5e..72e67cf 100644 --- a/modules/profile/files/hadoop/prometheus_mapreduce_history_jmx_exporter.yaml +++ b/modules/profile/files/hadoop/prometheus_mapreduce_history_jmx_exporter.yaml @@ -1,9 +1,5 @@ --- lowercaseOutputLabelNames: true lowercaseOutputName: false -whitelistObjectNames: - - 'java.lang:type=ClassLoading' - - 'java.lang:type=Compilation' - - 'java.lang:type=GarbageCollector*' - - 'java.lang:type=Memory' - - 'java.lang:type=MemoryPool*' \ No newline at end of file +blacklistObjectNames: + - '*:*' \ No newline at end of file diff --git a/modules/profile/files/hadoop/prometheus_yarn_nodemanager_jmx_exporter.yaml b/modules/profile/files/hadoop/prometheus_yarn_nodemanager_jmx_exporter.yaml index f1a7ace..d044eae 100644 --- a/modules/profile/files/hadoop/prometheus_yarn_nodemanager_jmx_exporter.yaml +++ b/modules/profile/files/hadoop/prometheus_yarn_nodemanager_jmx_exporter.yaml @@ -3,9 +3,4 @@ lowercaseOutputName: false whitelistObjectNames: - 'Hadoop:service=NodeManager,name=NodeManagerMetrics' - - 'Hadoop:service=NodeManager,name=ShuffleMetrics' - - 'java.lang:type=ClassLoading' - - 'java.lang:type=Compilation' - - 'java.lang:type=GarbageCollector*' - - 'java.lang:type=Memory' - - 'java.lang:type=MemoryPool*' \ No newline
[MediaWiki-commits] [Gerrit] operations/puppet[production]: profile::analytics::database::meta: simplify labs deployment
Elukey has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/402791 )
Change subject: profile::analytics::database::meta: simplify labs deployment
..
profile::analytics::database::meta: simplify labs deployment
Bug: T166248
Change-Id: Iae7157036585e20b239ac3778352bec6a6145585
---
M modules/profile/manifests/analytics/database/meta.pp
1 file changed, 1 insertion(+), 1 deletion(-)
Approvals:
Elukey: Looks good to me, approved
jenkins-bot: Verified
diff --git a/modules/profile/manifests/analytics/database/meta.pp
b/modules/profile/manifests/analytics/database/meta.pp
index bcda89e..a28fba9 100644
--- a/modules/profile/manifests/analytics/database/meta.pp
+++ b/modules/profile/manifests/analytics/database/meta.pp
@@ -4,7 +4,7 @@
# and other Analytics Cluster services.
#
class profile::analytics::database::meta(
-$monitoring_enabled =
hiera('profile::analytics::database::meta::monitoring_enabled'),
+$monitoring_enabled =
hiera('profile::analytics::database::meta::monitoring_enabled', false),
) {
# Some CDH database init scripts need Java to run.
require ::profile::java::analytics
--
To view, visit https://gerrit.wikimedia.org/r/402791
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Iae7157036585e20b239ac3778352bec6a6145585
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
Gerrit-Reviewer: Elukey
Gerrit-Reviewer: Ottomata
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: profile::analytics::database::meta: simplify labs deployment
Elukey has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/402791 )
Change subject: profile::analytics::database::meta: simplify labs deployment
..
profile::analytics::database::meta: simplify labs deployment
Bug: T166248
Change-Id: Iae7157036585e20b239ac3778352bec6a6145585
---
M modules/profile/manifests/analytics/database/meta.pp
1 file changed, 1 insertion(+), 1 deletion(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/91/402791/1
diff --git a/modules/profile/manifests/analytics/database/meta.pp
b/modules/profile/manifests/analytics/database/meta.pp
index bcda89e..a28fba9 100644
--- a/modules/profile/manifests/analytics/database/meta.pp
+++ b/modules/profile/manifests/analytics/database/meta.pp
@@ -4,7 +4,7 @@
# and other Analytics Cluster services.
#
class profile::analytics::database::meta(
-$monitoring_enabled =
hiera('profile::analytics::database::meta::monitoring_enabled'),
+$monitoring_enabled =
hiera('profile::analytics::database::meta::monitoring_enabled', false),
) {
# Some CDH database init scripts need Java to run.
require ::profile::java::analytics
--
To view, visit https://gerrit.wikimedia.org/r/402791
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Iae7157036585e20b239ac3778352bec6a6145585
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: profile::hadoop::backup::namenode: improve labs support
Elukey has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/402783 )
Change subject: profile::hadoop::backup::namenode: improve labs support
..
profile::hadoop::backup::namenode: improve labs support
Bug: T166248
Change-Id: I9cfb7feae45e8a2460e0551800da0f68780b75b6
---
M hieradata/role/common/analytics_cluster/hadoop/standby.yaml
M modules/profile/manifests/hadoop/backup/namenode.pp
2 files changed, 22 insertions(+), 11 deletions(-)
Approvals:
Elukey: Looks good to me, approved
jenkins-bot: Verified
diff --git a/hieradata/role/common/analytics_cluster/hadoop/standby.yaml
b/hieradata/role/common/analytics_cluster/hadoop/standby.yaml
index 27029c0..5875973 100644
--- a/hieradata/role/common/analytics_cluster/hadoop/standby.yaml
+++ b/hieradata/role/common/analytics_cluster/hadoop/standby.yaml
@@ -32,4 +32,6 @@
profile::analytics::database::meta::backup_dest::oozie_host:
'analytics1003.eqiad.wmnet'
profile::hadoop::firewall::master::analytics_srange: '$ANALYTICS_NETWORKS'
-profile::hadoop::firewall::master::analytics_druid_srange:
'(($ANALYTICS_NETWORKS $DRUID_PUBLIC_HOSTS))'
\ No newline at end of file
+profile::hadoop::firewall::master::analytics_druid_srange:
'(($ANALYTICS_NETWORKS $DRUID_PUBLIC_HOSTS))'
+
+profile::hadoop::backup::namenode::monitoring_enabled: true
\ No newline at end of file
diff --git a/modules/profile/manifests/hadoop/backup/namenode.pp
b/modules/profile/manifests/hadoop/backup/namenode.pp
index bdeb567..1193ea4 100644
--- a/modules/profile/manifests/hadoop/backup/namenode.pp
+++ b/modules/profile/manifests/hadoop/backup/namenode.pp
@@ -4,14 +4,21 @@
# and ensures that bacula backs up Hadoop NameNode fsimages,
# in the case we need to recover if both Hadoop NameNodes.
#
-class profile::hadoop::backup::namenode {
+class profile::hadoop::backup::namenode(
+$monitoring_enabled =
hiera('profile::hadoop::backup::namenode::monitoring_enabled', false),
+) {
require ::profile::hadoop::common
+
+$backup_dir_group = $::realm ? {
+'production' => 'analytics-admins',
+'labs' => "project-${::labsproject}",
+}
if !defined(File['/srv/backup']) {
file { '/srv/backup':
ensure => 'directory',
owner => 'root',
-group => 'analytics-admins',
+group => $backup_dir_group,
mode => '0755',
}
}
@@ -23,7 +30,7 @@
]:
ensure => 'directory',
owner => 'hdfs',
-group => 'analytics-admins',
+group => $backup_dir_group,
mode=> '0750',
require => File['/srv/backup']
}
@@ -51,13 +58,15 @@
}
}
-# Alert if backup gets stale.
-$warning_threshold_hours = 26
-$critical_threshold_hours = 48
-nrpe::monitor_service { 'hadoop-namenode-backup-age':
-description => 'Age of most recent Hadoop NameNode backup files',
-nrpe_command => "/usr/bin/sudo
/usr/local/lib/nagios/plugins/check_newest_file_age -V -C -d ${destination} -w
${$warning_threshold_hours} -c ${critical_threshold_hours}",
-contact_group => 'analytics',
+if $monitoring_enabled {
+# Alert if backup gets stale.
+$warning_threshold_hours = 26
+$critical_threshold_hours = 48
+nrpe::monitor_service { 'hadoop-namenode-backup-age':
+description => 'Age of most recent Hadoop NameNode backup files',
+nrpe_command => "/usr/bin/sudo
/usr/local/lib/nagios/plugins/check_newest_file_age -V -C -d ${destination} -w
${$warning_threshold_hours} -c ${critical_threshold_hours}",
+contact_group => 'analytics',
+}
}
# Bacula will also back up this directory.
--
To view, visit https://gerrit.wikimedia.org/r/402783
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I9cfb7feae45e8a2460e0551800da0f68780b75b6
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
Gerrit-Reviewer: Elukey
Gerrit-Reviewer: Ottomata
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: profile::hadoop::backup::namenode: improve labs support
Elukey has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/402783 )
Change subject: profile::hadoop::backup::namenode: improve labs support
..
profile::hadoop::backup::namenode: improve labs support
Bug: T166248
Change-Id: I9cfb7feae45e8a2460e0551800da0f68780b75b6
---
M hieradata/role/common/analytics_cluster/hadoop/standby.yaml
M modules/profile/manifests/hadoop/backup/namenode.pp
2 files changed, 22 insertions(+), 11 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/83/402783/1
diff --git a/hieradata/role/common/analytics_cluster/hadoop/standby.yaml
b/hieradata/role/common/analytics_cluster/hadoop/standby.yaml
index 27029c0..5875973 100644
--- a/hieradata/role/common/analytics_cluster/hadoop/standby.yaml
+++ b/hieradata/role/common/analytics_cluster/hadoop/standby.yaml
@@ -32,4 +32,6 @@
profile::analytics::database::meta::backup_dest::oozie_host:
'analytics1003.eqiad.wmnet'
profile::hadoop::firewall::master::analytics_srange: '$ANALYTICS_NETWORKS'
-profile::hadoop::firewall::master::analytics_druid_srange:
'(($ANALYTICS_NETWORKS $DRUID_PUBLIC_HOSTS))'
\ No newline at end of file
+profile::hadoop::firewall::master::analytics_druid_srange:
'(($ANALYTICS_NETWORKS $DRUID_PUBLIC_HOSTS))'
+
+profile::hadoop::backup::namenode::monitoring_enabled: true
\ No newline at end of file
diff --git a/modules/profile/manifests/hadoop/backup/namenode.pp
b/modules/profile/manifests/hadoop/backup/namenode.pp
index bdeb567..1193ea4 100644
--- a/modules/profile/manifests/hadoop/backup/namenode.pp
+++ b/modules/profile/manifests/hadoop/backup/namenode.pp
@@ -4,14 +4,21 @@
# and ensures that bacula backs up Hadoop NameNode fsimages,
# in the case we need to recover if both Hadoop NameNodes.
#
-class profile::hadoop::backup::namenode {
+class profile::hadoop::backup::namenode(
+$monitoring_enabled =
hiera('profile::hadoop::backup::namenode::monitoring_enabled', false),
+) {
require ::profile::hadoop::common
+
+$backup_dir_group = $::realm ? {
+'production' => 'analytics-admins',
+'labs' => "project-${::labsproject}",
+}
if !defined(File['/srv/backup']) {
file { '/srv/backup':
ensure => 'directory',
owner => 'root',
-group => 'analytics-admins',
+group => $backup_dir_group,
mode => '0755',
}
}
@@ -23,7 +30,7 @@
]:
ensure => 'directory',
owner => 'hdfs',
-group => 'analytics-admins',
+group => $backup_dir_group,
mode=> '0750',
require => File['/srv/backup']
}
@@ -51,13 +58,15 @@
}
}
-# Alert if backup gets stale.
-$warning_threshold_hours = 26
-$critical_threshold_hours = 48
-nrpe::monitor_service { 'hadoop-namenode-backup-age':
-description => 'Age of most recent Hadoop NameNode backup files',
-nrpe_command => "/usr/bin/sudo
/usr/local/lib/nagios/plugins/check_newest_file_age -V -C -d ${destination} -w
${$warning_threshold_hours} -c ${critical_threshold_hours}",
-contact_group => 'analytics',
+if $monitoring_enabled {
+# Alert if backup gets stale.
+$warning_threshold_hours = 26
+$critical_threshold_hours = 48
+nrpe::monitor_service { 'hadoop-namenode-backup-age':
+description => 'Age of most recent Hadoop NameNode backup files',
+nrpe_command => "/usr/bin/sudo
/usr/local/lib/nagios/plugins/check_newest_file_age -V -C -d ${destination} -w
${$warning_threshold_hours} -c ${critical_threshold_hours}",
+contact_group => 'analytics',
+}
}
# Bacula will also back up this directory.
--
To view, visit https://gerrit.wikimedia.org/r/402783
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I9cfb7feae45e8a2460e0551800da0f68780b75b6
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: profile::analytics::database::meta::backup_dest: allow labs ...
Elukey has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/402382 )
Change subject: profile::analytics::database::meta::backup_dest: allow labs dir
perms
..
profile::analytics::database::meta::backup_dest: allow labs dir perms
Bug: T166248
Change-Id: I4f64a3e9344026cb56cc6d743092cc3e3f65ba97
---
M modules/profile/manifests/analytics/database/meta/backup_dest.pp
1 file changed, 7 insertions(+), 2 deletions(-)
Approvals:
Ottomata: Looks good to me, but someone else must approve
Elukey: Looks good to me, approved
jenkins-bot: Verified
diff --git a/modules/profile/manifests/analytics/database/meta/backup_dest.pp
b/modules/profile/manifests/analytics/database/meta/backup_dest.pp
index 5c4d589..025cb65 100644
--- a/modules/profile/manifests/analytics/database/meta/backup_dest.pp
+++ b/modules/profile/manifests/analytics/database/meta/backup_dest.pp
@@ -8,11 +8,16 @@
$oozie_host =
hiera('profile::analytics::database::meta::backup_dest::oozie_host'),
) {
+$backup_dir_group = $::realm ? {
+'production' => 'analytics-admins',
+'labs' => "project-${::labsproject}",
+}
+
if !defined(File['/srv/backup']) {
file { '/srv/backup':
ensure => 'directory',
owner => 'root',
-group => 'analytics-admins',
+group => $backup_dir_group,
mode => '0755',
}
}
@@ -23,7 +28,7 @@
]:
ensure => 'directory',
owner => 'root',
-group => 'analytics-admins',
+group => $backup_dir_group,
mode => '0750',
}
--
To view, visit https://gerrit.wikimedia.org/r/402382
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I4f64a3e9344026cb56cc6d743092cc3e3f65ba97
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
Gerrit-Reviewer: Elukey
Gerrit-Reviewer: Ottomata
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: profile::analytics::database::meta::backup_dest: allow labs ...
Elukey has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/402382 )
Change subject: profile::analytics::database::meta::backup_dest: allow labs dir
perms
..
profile::analytics::database::meta::backup_dest: allow labs dir perms
Bug: T166248
Change-Id: I4f64a3e9344026cb56cc6d743092cc3e3f65ba97
---
M modules/profile/manifests/analytics/database/meta/backup_dest.pp
1 file changed, 7 insertions(+), 2 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/82/402382/1
diff --git a/modules/profile/manifests/analytics/database/meta/backup_dest.pp
b/modules/profile/manifests/analytics/database/meta/backup_dest.pp
index 5c4d589..025cb65 100644
--- a/modules/profile/manifests/analytics/database/meta/backup_dest.pp
+++ b/modules/profile/manifests/analytics/database/meta/backup_dest.pp
@@ -8,11 +8,16 @@
$oozie_host =
hiera('profile::analytics::database::meta::backup_dest::oozie_host'),
) {
+$backup_dir_group = $::realm ? {
+'production' => 'analytics-admins',
+'labs' => "project-${::labsproject}",
+}
+
if !defined(File['/srv/backup']) {
file { '/srv/backup':
ensure => 'directory',
owner => 'root',
-group => 'analytics-admins',
+group => $backup_dir_group,
mode => '0755',
}
}
@@ -23,7 +28,7 @@
]:
ensure => 'directory',
owner => 'root',
-group => 'analytics-admins',
+group => $backup_dir_group,
mode => '0750',
}
--
To view, visit https://gerrit.wikimedia.org/r/402382
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I4f64a3e9344026cb56cc6d743092cc3e3f65ba97
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] labs/private[master]: Move varnishkafka.key.pem to varnishkafka.key.private.pem
Elukey has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/402357 ) Change subject: Move varnishkafka.key.pem to varnishkafka.key.private.pem .. Move varnishkafka.key.pem to varnishkafka.key.private.pem Change-Id: Ia0edb5773fac4d092b61147323c4bb3bff729576 --- R modules/secret/secrets/certificates/varnishkafka/varnishkafka.key.private.pem 1 file changed, 0 insertions(+), 0 deletions(-) Approvals: Elukey: Verified; Looks good to me, approved diff --git a/modules/secret/secrets/certificates/varnishkafka/varnishkafka.key.pem b/modules/secret/secrets/certificates/varnishkafka/varnishkafka.key.private.pem similarity index 100% rename from modules/secret/secrets/certificates/varnishkafka/varnishkafka.key.pem rename to modules/secret/secrets/certificates/varnishkafka/varnishkafka.key.private.pem -- To view, visit https://gerrit.wikimedia.org/r/402357 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ia0edb5773fac4d092b61147323c4bb3bff729576 Gerrit-PatchSet: 1 Gerrit-Project: labs/private Gerrit-Branch: master Gerrit-Owner: Elukey Gerrit-Reviewer: Elukey ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] labs/private[master]: Move varnishkafka.key.pem to varnishkafka.key.private.pem
Elukey has uploaded a new change for review. ( https://gerrit.wikimedia.org/r/402357 ) Change subject: Move varnishkafka.key.pem to varnishkafka.key.private.pem .. Move varnishkafka.key.pem to varnishkafka.key.private.pem Change-Id: Ia0edb5773fac4d092b61147323c4bb3bff729576 --- R modules/secret/secrets/certificates/varnishkafka/varnishkafka.key.private.pem 1 file changed, 0 insertions(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/labs/private refs/changes/57/402357/1 diff --git a/modules/secret/secrets/certificates/varnishkafka/varnishkafka.key.pem b/modules/secret/secrets/certificates/varnishkafka/varnishkafka.key.private.pem similarity index 100% rename from modules/secret/secrets/certificates/varnishkafka/varnishkafka.key.pem rename to modules/secret/secrets/certificates/varnishkafka/varnishkafka.key.private.pem -- To view, visit https://gerrit.wikimedia.org/r/402357 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ia0edb5773fac4d092b61147323c4bb3bff729576 Gerrit-PatchSet: 1 Gerrit-Project: labs/private Gerrit-Branch: master Gerrit-Owner: Elukey ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: profile::hadoop::firewall::master: fix default ferm srange
Elukey has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/402354 )
Change subject: profile::hadoop::firewall::master: fix default ferm srange
..
profile::hadoop::firewall::master: fix default ferm srange
Bug: T166248
Change-Id: Ie10d3591b29d39ec99cd6f6e2e8843c2401f2f31
---
M modules/profile/manifests/hadoop/firewall/master.pp
1 file changed, 2 insertions(+), 2 deletions(-)
Approvals:
Elukey: Looks good to me, approved
jenkins-bot: Verified
diff --git a/modules/profile/manifests/hadoop/firewall/master.pp
b/modules/profile/manifests/hadoop/firewall/master.pp
index 525bfb5..262099b 100644
--- a/modules/profile/manifests/hadoop/firewall/master.pp
+++ b/modules/profile/manifests/hadoop/firewall/master.pp
@@ -3,8 +3,8 @@
# Set of common firewall rules for Hadoop Master nodes (active and standby)
#
class profile::hadoop::firewall::master(
-$analytics_srange =
hiera('profile::hadoop::firewall::master::analytics_srange', 'DOMAIN_NETWORKS'),
-$analytics_druid_srange =
hiera('profile::hadoop::firewall::master::analytics_druid_srange',
'DOMAIN_NETWORKS'),
+$analytics_srange =
hiera('profile::hadoop::firewall::master::analytics_srange',
'$DOMAIN_NETWORKS'),
+$analytics_druid_srange =
hiera('profile::hadoop::firewall::master::analytics_druid_srange',
'$DOMAIN_NETWORKS'),
) {
ferm::service{ 'hadoop-hdfs-namenode':
--
To view, visit https://gerrit.wikimedia.org/r/402354
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Ie10d3591b29d39ec99cd6f6e2e8843c2401f2f31
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
Gerrit-Reviewer: Elukey
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: profile::hadoop::firewall::master: fix default ferm srange
Elukey has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/402354 )
Change subject: profile::hadoop::firewall::master: fix default ferm srange
..
profile::hadoop::firewall::master: fix default ferm srange
Bug: T166248
Change-Id: Ie10d3591b29d39ec99cd6f6e2e8843c2401f2f31
---
M modules/profile/manifests/hadoop/firewall/master.pp
1 file changed, 2 insertions(+), 2 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/54/402354/1
diff --git a/modules/profile/manifests/hadoop/firewall/master.pp
b/modules/profile/manifests/hadoop/firewall/master.pp
index 525bfb5..262099b 100644
--- a/modules/profile/manifests/hadoop/firewall/master.pp
+++ b/modules/profile/manifests/hadoop/firewall/master.pp
@@ -3,8 +3,8 @@
# Set of common firewall rules for Hadoop Master nodes (active and standby)
#
class profile::hadoop::firewall::master(
-$analytics_srange =
hiera('profile::hadoop::firewall::master::analytics_srange', 'DOMAIN_NETWORKS'),
-$analytics_druid_srange =
hiera('profile::hadoop::firewall::master::analytics_druid_srange',
'DOMAIN_NETWORKS'),
+$analytics_srange =
hiera('profile::hadoop::firewall::master::analytics_srange',
'$DOMAIN_NETWORKS'),
+$analytics_druid_srange =
hiera('profile::hadoop::firewall::master::analytics_druid_srange',
'$DOMAIN_NETWORKS'),
) {
ferm::service{ 'hadoop-hdfs-namenode':
--
To view, visit https://gerrit.wikimedia.org/r/402354
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ie10d3591b29d39ec99cd6f6e2e8843c2401f2f31
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: profile::hadoop:*: add ferm srange defaults to allow labs de...
Elukey has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/402324 )
Change subject: profile::hadoop:*: add ferm srange defaults to allow labs
deployments
..
profile::hadoop:*: add ferm srange defaults to allow labs deployments
Bug: T166248
Change-Id: Iffaacafa326890a44f1dfec616420af5774a7023
---
M hieradata/role/common/analytics_cluster/hadoop/master.yaml
M hieradata/role/common/analytics_cluster/hadoop/standby.yaml
M hieradata/role/common/analytics_cluster/hadoop/worker.yaml
M modules/profile/manifests/hadoop/firewall/master.pp
M modules/profile/manifests/hadoop/worker.pp
5 files changed, 30 insertions(+), 18 deletions(-)
Approvals:
Elukey: Looks good to me, approved
jenkins-bot: Verified
diff --git a/hieradata/role/common/analytics_cluster/hadoop/master.yaml
b/hieradata/role/common/analytics_cluster/hadoop/master.yaml
index 210458a..06f156a 100644
--- a/hieradata/role/common/analytics_cluster/hadoop/master.yaml
+++ b/hieradata/role/common/analytics_cluster/hadoop/master.yaml
@@ -26,4 +26,7 @@
# Prometheus JMX Exporter config templates.
profile::hadoop::common::hadoop_namenode_opts: "-Xms6144m -Xmx6144m
-javaagent:/usr/share/java/prometheus/jmx_prometheus_javaagent.jar=%{::ipaddress}:10080:/etc/hadoop/prometheus_hdfs_namenode_jmx_exporter.yaml"
profile::hadoop::common::yarn_resourcemanager_opts: "-Xms2048m
-javaagent:/usr/share/java/prometheus/jmx_prometheus_javaagent.jar=%{::ipaddress}:10083:/etc/hadoop/prometheus_yarn_resourcemanager_jmx_exporter.yaml"
-profile::hadoop::common::mapreduce_history_java_opts: "-Xms4096m -Xmx4096m
-javaagent:/usr/share/java/prometheus/jmx_prometheus_javaagent.jar=%{::ipaddress}:10086:/etc/hadoop/prometheus_mapreduce_history_jmx_exporter.yaml"
\ No newline at end of file
+profile::hadoop::common::mapreduce_history_java_opts: "-Xms4096m -Xmx4096m
-javaagent:/usr/share/java/prometheus/jmx_prometheus_javaagent.jar=%{::ipaddress}:10086:/etc/hadoop/prometheus_mapreduce_history_jmx_exporter.yaml"
+
+profile::hadoop::firewall::master::analytics_srange: '$ANALYTICS_NETWORKS'
+profile::hadoop::firewall::master::analytics_druid_srange:
'(($ANALYTICS_NETWORKS $DRUID_PUBLIC_HOSTS))'
\ No newline at end of file
diff --git a/hieradata/role/common/analytics_cluster/hadoop/standby.yaml
b/hieradata/role/common/analytics_cluster/hadoop/standby.yaml
index f622d44..27029c0 100644
--- a/hieradata/role/common/analytics_cluster/hadoop/standby.yaml
+++ b/hieradata/role/common/analytics_cluster/hadoop/standby.yaml
@@ -29,4 +29,7 @@
profile::hadoop::common::yarn_resourcemanager_opts: "-Xms2048m
-javaagent:/usr/share/java/prometheus/jmx_prometheus_javaagent.jar=%{::ipaddress}:10083:/etc/hadoop/prometheus_yarn_resourcemanager_jmx_exporter.yaml"
profile::analytics::database::meta::backup_dest::hive::metastore_host:
'analytics1003.eqiad.wmnet'
-profile::analytics::database::meta::backup_dest::oozie_host:
'analytics1003.eqiad.wmnet'
\ No newline at end of file
+profile::analytics::database::meta::backup_dest::oozie_host:
'analytics1003.eqiad.wmnet'
+
+profile::hadoop::firewall::master::analytics_srange: '$ANALYTICS_NETWORKS'
+profile::hadoop::firewall::master::analytics_druid_srange:
'(($ANALYTICS_NETWORKS $DRUID_PUBLIC_HOSTS))'
\ No newline at end of file
diff --git a/hieradata/role/common/analytics_cluster/hadoop/worker.yaml
b/hieradata/role/common/analytics_cluster/hadoop/worker.yaml
index 4b9e1d8..6507947 100644
--- a/hieradata/role/common/analytics_cluster/hadoop/worker.yaml
+++ b/hieradata/role/common/analytics_cluster/hadoop/worker.yaml
@@ -19,4 +19,6 @@
# Prometheus JMX Exporter config templates.
profile::hadoop::common::hadoop_datanode_opts: "-Xms4096m
-javaagent:/usr/share/java/prometheus/jmx_prometheus_javaagent.jar=%{::ipaddress}:51010:/etc/hadoop/prometheus_hdfs_datanode_jmx_exporter.yaml"
-profile::hadoop::common::yarn_nodemanager_opts: "-Xms4096m
-javaagent:/usr/share/java/prometheus/jmx_prometheus_javaagent.jar=%{::ipaddress}:8141:/etc/hadoop/prometheus_yarn_nodemanager_jmx_exporter.yaml"
\ No newline at end of file
+profile::hadoop::common::yarn_nodemanager_opts: "-Xms4096m
-javaagent:/usr/share/java/prometheus/jmx_prometheus_javaagent.jar=%{::ipaddress}:8141:/etc/hadoop/prometheus_yarn_nodemanager_jmx_exporter.yaml"
+
+profile::hadoop::worker::ferm_srange: '(($ANALYTICS_NETWORKS
$DRUID_PUBLIC_HOSTS))'
\ No newline at end of file
diff --git a/modules/profile/manifests/hadoop/firewall/master.pp
b/modules/profile/manifests/hadoop/firewall/master.pp
index 90501e2..525bfb5 100644
--- a/modules/profile/manifests/hadoop/firewall/master.pp
+++ b/modules/profile/manifests/hadoop/firewall/master.pp
@@ -2,84 +2,87 @@
#
# Set of common firewall rules for Hadoop Master nodes (active and standby)
#
-class profile::hadoop::firewall::master {
+class profile::hadoop::firewall::master(
+$analytics_srange =
hiera('profile::hadoop::firewall::master::analytics
[MediaWiki-commits] [Gerrit] operations/puppet[production]: network::constants: add fake analytics networks for labs
Elukey has uploaded a new change for review. ( https://gerrit.wikimedia.org/r/402324 ) Change subject: network::constants: add fake analytics networks for labs .. network::constants: add fake analytics networks for labs This will allow proper deployment and bootstrap of Hadoop clusters in labs. Bug: T166248 Change-Id: Iffaacafa326890a44f1dfec616420af5774a7023 --- M modules/network/manifests/constants.pp 1 file changed, 3 insertions(+), 1 deletion(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/24/402324/1 diff --git a/modules/network/manifests/constants.pp b/modules/network/manifests/constants.pp index 4be6e71..56d0173 100644 --- a/modules/network/manifests/constants.pp +++ b/modules/network/manifests/constants.pp @@ -217,7 +217,9 @@ '208.80.155.120', # labpuppetmaster1002.wikimedia.org '2620:0:861:4:208:80:155:120', # labpuppetmaster1002.wikimedia.org ], -} +'analytics_networks' => $labs_networks, +'druid_public_hosts' => $labs_networks, +} } -- To view, visit https://gerrit.wikimedia.org/r/402324 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Iffaacafa326890a44f1dfec616420af5774a7023 Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: profile::hadoop::*: include labs firewall use case
Elukey has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/402323 )
Change subject: profile::hadoop::*: include labs firewall use case
..
profile::hadoop::*: include labs firewall use case
This change is needed to allow to bootstrap and deploy
properly Hadoop clusters in labs.
Bug: T167790
Change-Id: I67ad4e3b2b3b8c706685cf96f845d506a61fc69c
---
M modules/profile/manifests/hadoop/firewall/master.pp
M modules/profile/manifests/hadoop/worker.pp
2 files changed, 28 insertions(+), 14 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/23/402323/1
diff --git a/modules/profile/manifests/hadoop/firewall/master.pp
b/modules/profile/manifests/hadoop/firewall/master.pp
index 90501e2..e285caa 100644
--- a/modules/profile/manifests/hadoop/firewall/master.pp
+++ b/modules/profile/manifests/hadoop/firewall/master.pp
@@ -4,82 +4,90 @@
#
class profile::hadoop::firewall::master {
+if $::realm == 'production' {
+$analytics_druid_srange = '(($ANALYTICS_NETWORKS $DRUID_PUBLIC_HOSTS))'
+$analytics_only_srange = '$ANALYTICS_NETWORKS'
+} else {
+$analytics_druid_srange = '$DOMAIN_NETWORKS'
+$analytics_only_srange = '$DOMAIN_NETWORKS'
+}
+
ferm::service{ 'hadoop-hdfs-namenode':
proto => 'tcp',
port => '8020',
-srange => '(($ANALYTICS_NETWORKS $DRUID_PUBLIC_HOSTS))',
+srange => $analytics_druid_srange,
}
ferm::service{ 'hadoop-hdfs-zkfc':
proto => 'tcp',
port => '8019',
-srange => '$ANALYTICS_NETWORKS',
+srange => $analytics_only_srange,
}
ferm::service{ 'hadoop-hdfs-namenode-http-ui':
proto => 'tcp',
port => '50070',
-srange => '$ANALYTICS_NETWORKS',
+srange => $analytics_only_srange,
}
ferm::service{ 'hadoop-hdfs-namenode-jmx':
proto => 'tcp',
port => '9980',
-srange => '$ANALYTICS_NETWORKS',
+srange => $analytics_only_srange,
}
ferm::service{ 'hadoop-yarn-resourcemanager-scheduler':
proto => 'tcp',
port => '8030',
-srange => '$ANALYTICS_NETWORKS',
+srange => $analytics_only_srange,
}
ferm::service{ 'hadoop-yarn-resourcemanager-tracker':
proto => 'tcp',
port => '8031',
-srange => '$ANALYTICS_NETWORKS',
+srange => $analytics_only_srange,
}
ferm::service{ 'hadoop-yarn-resourcemanager':
proto => 'tcp',
port => '8032',
-srange => '(($ANALYTICS_NETWORKS $DRUID_PUBLIC_HOSTS))',
+srange => $analytics_druid_srange,
}
ferm::service{ 'hadoop-yarn-resourcemanager-admin':
proto => 'tcp',
port => '8033',
-srange => '$ANALYTICS_NETWORKS',
+srange => $analytics_only_srange,
}
ferm::service{ 'hadoop-yarn-resourcemanager-http-ui':
proto => 'tcp',
port => '8088',
-srange => '$ANALYTICS_NETWORKS',
+srange => $analytics_only_srange,
}
ferm::service{ 'hadoop-mapreduce-historyserver':
proto => 'tcp',
port => '10020',
-srange => '$ANALYTICS_NETWORKS',
+srange => $analytics_only_srange,
}
ferm::service{ 'hadoop-mapreduce-historyserver-admin':
proto => 'tcp',
port => '10033',
-srange => '$ANALYTICS_NETWORKS',
+srange => $analytics_only_srange,
}
ferm::service{ 'hadoop-mapreduce-historyserver-http-ui':
proto => 'tcp',
port => '19888',
-srange => '$ANALYTICS_NETWORKS',
+srange => $analytics_only_srange,
}
ferm::service{ 'hadoop-yarn-resourcemanager-jmx':
proto => 'tcp',
port => '9983',
-srange => '$ANALYTICS_NETWORKS',
+srange => $analytics_only_srange,
}
}
diff --git a/modules/profile/manifests/hadoop/worker.pp
b/modules/profile/manifests/hadoop/worker.pp
index 6af651b..52d024b 100644
--- a/modules/profile/manifests/hadoop/worker.pp
+++ b/modules/profile/manifests/hadoop/worker.pp
@@ -89,11 +89,17 @@
require => Package['python3-numpy'],
}
+if $::realm == 'production' {
+$analytics_srange = '(($ANALYTICS_NETWORKS $DRUID_PUBLIC_HOSTS))'
+} else {
+$analytics_srange = '$DOMAIN_NETWORKS'
+}
+
# This allows Hadoop daemons to talk to each other.
ferm::service{ 'hadoop-access':
proto => 'tcp',
port => '1024:65535',
-srange => '(($ANALYTICS_NETWORKS $DRUID_PUBLIC_HOSTS))',
+srange => $analytics_srange,
}
if $monitoring_enabled {
--
To view, visit https://gerrit.wikimedia.org/r/402323
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I67ad4e3b2b3b8c706685cf96f845d506a61fc69c
Gerrit-PatchSet: 1
Ge
[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::analytics_cluster::hadoop::master: fix order in profil...
Elukey has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/402316 ) Change subject: role::analytics_cluster::hadoop::master: fix order in profile inclusion .. role::analytics_cluster::hadoop::master: fix order in profile inclusion Bug: T167790 Change-Id: I6ead5d152f550b9e798b174f157a77cf40712a2a --- M modules/role/manifests/analytics_cluster/hadoop/master.pp 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Elukey: Looks good to me, approved jenkins-bot: Verified diff --git a/modules/role/manifests/analytics_cluster/hadoop/master.pp b/modules/role/manifests/analytics_cluster/hadoop/master.pp index 28deec8..4f58d36 100644 --- a/modules/role/manifests/analytics_cluster/hadoop/master.pp +++ b/modules/role/manifests/analytics_cluster/hadoop/master.pp @@ -7,9 +7,9 @@ description => 'Hadoop Master (NameNode & ResourceManager)', } -include ::profile::hadoop::mysql_password include ::profile::hadoop::master include ::profile::hadoop::users +include ::profile::hadoop::mysql_password include ::profile::hadoop::firewall::master include ::profile::hadoop::logstash include ::profile::base::firewall -- To view, visit https://gerrit.wikimedia.org/r/402316 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I6ead5d152f550b9e798b174f157a77cf40712a2a Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey Gerrit-Reviewer: Elukey Gerrit-Reviewer: Giuseppe Lavagetto Gerrit-Reviewer: Ottomata Gerrit-Reviewer: jenkins-bot <> ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::analytics_cluster::hadoop::master: fix order in profil...
Elukey has uploaded a new change for review. ( https://gerrit.wikimedia.org/r/402316 ) Change subject: role::analytics_cluster::hadoop::master: fix order in profile inclusion .. role::analytics_cluster::hadoop::master: fix order in profile inclusion Bug: T167790 Change-Id: I6ead5d152f550b9e798b174f157a77cf40712a2a --- M modules/role/manifests/analytics_cluster/hadoop/master.pp 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/16/402316/1 diff --git a/modules/role/manifests/analytics_cluster/hadoop/master.pp b/modules/role/manifests/analytics_cluster/hadoop/master.pp index 28deec8..4f58d36 100644 --- a/modules/role/manifests/analytics_cluster/hadoop/master.pp +++ b/modules/role/manifests/analytics_cluster/hadoop/master.pp @@ -7,9 +7,9 @@ description => 'Hadoop Master (NameNode & ResourceManager)', } -include ::profile::hadoop::mysql_password include ::profile::hadoop::master include ::profile::hadoop::users +include ::profile::hadoop::mysql_password include ::profile::hadoop::firewall::master include ::profile::hadoop::logstash include ::profile::base::firewall -- To view, visit https://gerrit.wikimedia.org/r/402316 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I6ead5d152f550b9e798b174f157a77cf40712a2a Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::analytics_cluster: avoid to expicitly instance the sta...
Elukey has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/402084 )
Change subject: role::analytics_cluster: avoid to expicitly instance the
standard class
..
role::analytics_cluster: avoid to expicitly instance the standard class
Bug: T167790
Change-Id: Id1c60a0c3feb6b12ac75950702e94c17d3542038
---
M modules/role/manifests/analytics_cluster/hadoop/master.pp
M modules/role/manifests/analytics_cluster/hadoop/standby.pp
M modules/role/manifests/analytics_cluster/webserver.pp
3 files changed, 3 insertions(+), 3 deletions(-)
Approvals:
Elukey: Looks good to me, approved
jenkins-bot: Verified
diff --git a/modules/role/manifests/analytics_cluster/hadoop/master.pp
b/modules/role/manifests/analytics_cluster/hadoop/master.pp
index 8c53b8a..28deec8 100644
--- a/modules/role/manifests/analytics_cluster/hadoop/master.pp
+++ b/modules/role/manifests/analytics_cluster/hadoop/master.pp
@@ -13,5 +13,5 @@
include ::profile::hadoop::firewall::master
include ::profile::hadoop::logstash
include ::profile::base::firewall
-class { 'standard': }
+include standard
}
diff --git a/modules/role/manifests/analytics_cluster/hadoop/standby.pp
b/modules/role/manifests/analytics_cluster/hadoop/standby.pp
index 03ac4bc..465ebf6 100644
--- a/modules/role/manifests/analytics_cluster/hadoop/standby.pp
+++ b/modules/role/manifests/analytics_cluster/hadoop/standby.pp
@@ -24,6 +24,6 @@
include ::profile::analytics::database::meta::backup_dest
include ::profile::base::firewall
-class { 'standard': }
+include standard
}
diff --git a/modules/role/manifests/analytics_cluster/webserver.pp
b/modules/role/manifests/analytics_cluster/webserver.pp
index b4d9ffc..3e9f0c8 100644
--- a/modules/role/manifests/analytics_cluster/webserver.pp
+++ b/modules/role/manifests/analytics_cluster/webserver.pp
@@ -13,5 +13,5 @@
include ::profile::druid::pivot
include ::profile::base::firewall
-class { 'standard': }
+include standard
}
\ No newline at end of file
--
To view, visit https://gerrit.wikimedia.org/r/402084
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Id1c60a0c3feb6b12ac75950702e94c17d3542038
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
Gerrit-Reviewer: Elukey
Gerrit-Reviewer: Giuseppe Lavagetto
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::analytics_cluster: avoid to expicitly instance the sta...
Elukey has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/402084 )
Change subject: role::analytics_cluster: avoid to expicitly instance the
standard class
..
role::analytics_cluster: avoid to expicitly instance the standard class
Bug: T167790
Change-Id: Id1c60a0c3feb6b12ac75950702e94c17d3542038
---
M modules/role/manifests/analytics_cluster/hadoop/master.pp
M modules/role/manifests/analytics_cluster/hadoop/standby.pp
M modules/role/manifests/analytics_cluster/webserver.pp
3 files changed, 3 insertions(+), 3 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/84/402084/1
diff --git a/modules/role/manifests/analytics_cluster/hadoop/master.pp
b/modules/role/manifests/analytics_cluster/hadoop/master.pp
index 8c53b8a..28deec8 100644
--- a/modules/role/manifests/analytics_cluster/hadoop/master.pp
+++ b/modules/role/manifests/analytics_cluster/hadoop/master.pp
@@ -13,5 +13,5 @@
include ::profile::hadoop::firewall::master
include ::profile::hadoop::logstash
include ::profile::base::firewall
-class { 'standard': }
+include standard
}
diff --git a/modules/role/manifests/analytics_cluster/hadoop/standby.pp
b/modules/role/manifests/analytics_cluster/hadoop/standby.pp
index 03ac4bc..465ebf6 100644
--- a/modules/role/manifests/analytics_cluster/hadoop/standby.pp
+++ b/modules/role/manifests/analytics_cluster/hadoop/standby.pp
@@ -24,6 +24,6 @@
include ::profile::analytics::database::meta::backup_dest
include ::profile::base::firewall
-class { 'standard': }
+include standard
}
diff --git a/modules/role/manifests/analytics_cluster/webserver.pp
b/modules/role/manifests/analytics_cluster/webserver.pp
index b4d9ffc..3e9f0c8 100644
--- a/modules/role/manifests/analytics_cluster/webserver.pp
+++ b/modules/role/manifests/analytics_cluster/webserver.pp
@@ -13,5 +13,5 @@
include ::profile::druid::pivot
include ::profile::base::firewall
-class { 'standard': }
+include standard
}
\ No newline at end of file
--
To view, visit https://gerrit.wikimedia.org/r/402084
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Id1c60a0c3feb6b12ac75950702e94c17d3542038
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: profile::hadoop: set hiera defaults to ease labs deployments
Elukey has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/402050 )
Change subject: profile::hadoop: set hiera defaults to ease labs deployments
..
profile::hadoop: set hiera defaults to ease labs deployments
Bug: T167790
Change-Id: Ifb5bbce5d6919ac220f9004fc7d24f52c4ef8912
---
M modules/profile/manifests/hadoop/logstash.pp
M modules/profile/manifests/hadoop/master.pp
M modules/profile/manifests/hadoop/master/standby.pp
M modules/profile/manifests/hadoop/worker.pp
M modules/role/manifests/analytics_cluster/coordinator.pp
M modules/role/manifests/analytics_cluster/hadoop/worker.pp
6 files changed, 14 insertions(+), 8 deletions(-)
Approvals:
Elukey: Looks good to me, approved
jenkins-bot: Verified
diff --git a/modules/profile/manifests/hadoop/logstash.pp
b/modules/profile/manifests/hadoop/logstash.pp
index a454109..aca0eda 100644
--- a/modules/profile/manifests/hadoop/logstash.pp
+++ b/modules/profile/manifests/hadoop/logstash.pp
@@ -2,7 +2,7 @@
# Enables gelf logging to logstash from Hadoop.
# As of 2016-02, this is not used.
class profile::hadoop::logstash (
-$gelf_logging_enabled = hiera('profile::hadoop::logstash::enabled'),
+$gelf_logging_enabled = hiera('profile::hadoop::logstash::enabled', false),
) {
file { '/usr/local/bin/hadoop-yarn-logging-helper.sh':
content => template('profile/hadoop/hadoop-yarn-logging-helper.erb'),
diff --git a/modules/profile/manifests/hadoop/master.pp
b/modules/profile/manifests/hadoop/master.pp
index ecd11c5..b846130 100644
--- a/modules/profile/manifests/hadoop/master.pp
+++ b/modules/profile/manifests/hadoop/master.pp
@@ -4,12 +4,15 @@
#
# == Parameters
#
+# [*hadoop_namenode_heapsize*]
+#Current JVM heap size to use as threshold for monitoring.
+#
# [*monitoring_enabled*]
#If production monitoring needs to be enabled or not.
#
class profile::hadoop::master(
-$monitoring_enabled =
hiera('profile::hadoop::master::monitoring_enabled'),
-$hadoop_namenode_heapsize =
hiera('profile::hadoop::master::namenode_heapsize'),
+$monitoring_enabled =
hiera('profile::hadoop::master::monitoring_enabled', false),
+$hadoop_namenode_heapsize =
hiera('profile::hadoop::master::namenode_heapsize', 2048),
$hadoop_user_groups =
hiera('profile::hadoop::master::hadoop_user_groups'),
$statsd = hiera('statsd'),
){
diff --git a/modules/profile/manifests/hadoop/master/standby.pp
b/modules/profile/manifests/hadoop/master/standby.pp
index 6a98a36..ddbf1bb 100644
--- a/modules/profile/manifests/hadoop/master/standby.pp
+++ b/modules/profile/manifests/hadoop/master/standby.pp
@@ -2,12 +2,15 @@
#
# Sets up a standby/backup Hadoop Master node.
#
+# [*hadoop_namenode_heapsize*]
+#Current JVM heap size to use as threshold for monitoring.
+#
# [*monitoring_enabled*]
#If production monitoring needs to be enabled or not.
#
class profile::hadoop::master::standby(
-$monitoring_enabled =
hiera('profile::hadoop::standby_master::monitoring_enabled'),
-$hadoop_namenode_heapsize =
hiera('profile::hadoop::standby::namenode_heapsize'),
+$monitoring_enabled =
hiera('profile::hadoop::standby_master::monitoring_enabled', false),
+$hadoop_namenode_heapsize =
hiera('profile::hadoop::standby::namenode_heapsize', 2048),
$statsd = hiera('statsd'),
) {
diff --git a/modules/profile/manifests/hadoop/worker.pp
b/modules/profile/manifests/hadoop/worker.pp
index 6895792..6af651b 100644
--- a/modules/profile/manifests/hadoop/worker.pp
+++ b/modules/profile/manifests/hadoop/worker.pp
@@ -8,7 +8,7 @@
#If production monitoring needs to be enabled or not.
#
class profile::hadoop::worker(
-$monitoring_enabled = hiera('profile::hadoop::worker::monitoring_enabled'),
+$monitoring_enabled = hiera('profile::hadoop::worker::monitoring_enabled',
false),
$statsd = hiera('statsd'),
) {
diff --git a/modules/role/manifests/analytics_cluster/coordinator.pp
b/modules/role/manifests/analytics_cluster/coordinator.pp
index 6dbecaf..8a1af31 100644
--- a/modules/role/manifests/analytics_cluster/coordinator.pp
+++ b/modules/role/manifests/analytics_cluster/coordinator.pp
@@ -58,6 +58,6 @@
include ::role::analytics_cluster::refinery::job::sqoop_mediawiki
include ::role::analytics_cluster::refinery::job::json_refine
-class { '::standard': }
+include standard
include ::profile::base::firewall
}
\ No newline at end of file
diff --git a/modules/role/manifests/analytics_cluster/hadoop/worker.pp
b/modules/role/manifests/analytics_cluster/hadoop/worker.pp
index 49eb73b..b8186e8 100644
--- a/modules/role/manifests/analytics_cluster/hadoop/worker.pp
+++ b/modules/role/manifests/analytics_cluster/hadoop/worker.pp
@@ -8,5 +8,5 @@
include ::profile::hadoop::worker
include ::profile::hadoop::logstash
i
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Refactor thorium's roles in one
Elukey has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/401927 )
Change subject: Refactor thorium's roles in one
..
Refactor thorium's roles in one
Bug: T167790
Change-Id: I0decd3c62aab104c76915eacad1ab3339107a7f3
---
D hieradata/common/cdh/hue.yaml
R hieradata/role/common/analytics_cluster/webserver.yaml
D hieradata/role/common/druid/analytics/pivot.yaml
M manifests/site.pp
A modules/profile/manifests/druid/pivot.pp
A modules/profile/manifests/hue.pp
M modules/profile/manifests/statistics/web.pp
D modules/role/manifests/analytics_cluster/hue.pp
A modules/role/manifests/analytics_cluster/webserver.pp
D modules/role/manifests/druid/analytics/pivot.pp
D modules/role/manifests/statistics/web.pp
11 files changed, 113 insertions(+), 107 deletions(-)
Approvals:
Ottomata: Looks good to me, but someone else must approve
Elukey: Looks good to me, approved
jenkins-bot: Verified
diff --git a/hieradata/common/cdh/hue.yaml b/hieradata/common/cdh/hue.yaml
deleted file mode 100644
index f6b8a7b..000
--- a/hieradata/common/cdh/hue.yaml
+++ /dev/null
@@ -1,13 +0,0 @@
-#
-# Analytics Cluster Hue Configuration
-#
-
-# Don't auto create Hue users from LDAP in production.
-cdh::hue::ldap_create_users_on_login: false
-
-# External Database configuration
-cdh::hue::database_host: 'analytics1003.eqiad.wmnet'
-cdh::hue::database_engine: 'mysql'
-cdh::hue::database_user: 'hue'
-cdh::hue::database_port: '3306'
-cdh::hue::database_name: 'hue'
\ No newline at end of file
diff --git a/hieradata/role/common/statistics/web.yaml
b/hieradata/role/common/analytics_cluster/webserver.yaml
similarity index 66%
rename from hieradata/role/common/statistics/web.yaml
rename to hieradata/role/common/analytics_cluster/webserver.yaml
index e786679..64c5135 100644
--- a/hieradata/role/common/statistics/web.yaml
+++ b/hieradata/role/common/analytics_cluster/webserver.yaml
@@ -9,3 +9,9 @@
profile::superset::database_uri:
mysql://superset@analytics1003.eqiad.wmnet/superset
profile::superset::workers: 8
profile::superset::ldap_proxy_enabled: true
+
+profile::hue::database_host: 'analytics1003.eqiad.wmnet'
+profile::hue::hive_server_host: 'analytics1003.eqiad.wmnet'
+profile::hue::monitoring_enabled: true
+
+profile::druid::pivot::druid_broker: 'druid1001.eqiad.wmnet:8082'
\ No newline at end of file
diff --git a/hieradata/role/common/druid/analytics/pivot.yaml
b/hieradata/role/common/druid/analytics/pivot.yaml
deleted file mode 100644
index c8049db..000
--- a/hieradata/role/common/druid/analytics/pivot.yaml
+++ /dev/null
@@ -1 +0,0 @@
-pivot::druid_broker: 'druid1001.eqiad.wmnet:8082'
\ No newline at end of file
diff --git a/manifests/site.pp b/manifests/site.pp
index bf2a65a..acfad3b 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -2047,13 +2047,7 @@
# related role/module.
#
# This node is not intended for data processing.
-role(statistics::web,
-druid::analytics::pivot,
-analytics_cluster::hue)
-
-
-include ::standard
-include ::base::firewall
+role(analytics_cluster::webserver)
}
# Failoid service (Ganeti VM)
diff --git a/modules/profile/manifests/druid/pivot.pp
b/modules/profile/manifests/druid/pivot.pp
new file mode 100644
index 000..85997b9
--- /dev/null
+++ b/modules/profile/manifests/druid/pivot.pp
@@ -0,0 +1,11 @@
+# Class: profile::druid::pivot
+#
+# Install and configure the Druid's Pivot nodejs UI
+#
+class profile::druid::pivot(
+$druid_broker = hiera('profile::druid::pivot::druid_broker'),
+) {
+class { 'pivot':
+druid_broker => $druid_broker,
+}
+}
\ No newline at end of file
diff --git a/modules/profile/manifests/hue.pp b/modules/profile/manifests/hue.pp
new file mode 100644
index 000..6a12058
--- /dev/null
+++ b/modules/profile/manifests/hue.pp
@@ -0,0 +1,78 @@
+# == Class profile::hue
+#
+# Installs Hue server.
+#
+class profile::hue (
+$hive_server_host = hiera('profile::hue::hive_server_host'),
+$database_host = hiera('profile::hue::database_host'),
+$database_engine= hiera('profile::hue::database_engine',
'mysql'),
+$database_user = hiera('profile::hue::database_user', 'hue'),
+$database_port = hiera('profile::hue::database_port', 3306),
+$database_name = hiera('profile::hue::database_name', 'hue'),
+$ldap_create_users_on_login =
hiera('profile::hue::ldap_create_users_on_login', false),
+$monitoring_enabled = hiera('profile::hue::monitoring_enabled',
false),
+){
+
+# Require that all Hue applications
+# have their corresponding clients
+# and configs installed.
+# Include Hadoop ecosystem client classes.
+require ::profile::hadoop::common
+require ::profile::hive::client
+require ::profile::oozie::client
+
+# These don't require any extra configuration,
+# so no role class
[MediaWiki-commits] [Gerrit] operations/puppet[production]: profile::hadoop::master: remove the last hadoop cdh auto-lookup
Elukey has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/401904 )
Change subject: profile::hadoop::master: remove the last hadoop cdh auto-lookup
..
profile::hadoop::master: remove the last hadoop cdh auto-lookup
Bug: T167790
Change-Id: Ifde824c1213da10b68164b5f9a20416d1093c58c
---
D hieradata/common/cdh/hadoop/users.yaml
M hieradata/role/common/analytics_cluster/hadoop/master.yaml
M modules/profile/manifests/hadoop/master.pp
3 files changed, 5 insertions(+), 2 deletions(-)
Approvals:
Ottomata: Looks good to me, but someone else must approve
Elukey: Looks good to me, approved
jenkins-bot: Verified
diff --git a/hieradata/common/cdh/hadoop/users.yaml
b/hieradata/common/cdh/hadoop/users.yaml
deleted file mode 100644
index fd51433..000
--- a/hieradata/common/cdh/hadoop/users.yaml
+++ /dev/null
@@ -1,2 +0,0 @@
-# Ensure that users in these posix groups have home directories in HDFS.
-cdh::hadoop::users::groups: "analytics-users analytics-privatedata-users
analytics-admins analytics-search-users"
diff --git a/hieradata/role/common/analytics_cluster/hadoop/master.yaml
b/hieradata/role/common/analytics_cluster/hadoop/master.yaml
index f257b36..210458a 100644
--- a/hieradata/role/common/analytics_cluster/hadoop/master.yaml
+++ b/hieradata/role/common/analytics_cluster/hadoop/master.yaml
@@ -20,6 +20,9 @@
# Used to set up jvm heap size usage thresholds
profile::hadoop::master::namenode_heapsize: 6144
+# Ensure that users in these posix groups have home directories in HDFS.
+profile::hadoop::master::hadoop_user_groups: "analytics-users
analytics-privatedata-users analytics-admins analytics-search-users"
+
# Prometheus JMX Exporter config templates.
profile::hadoop::common::hadoop_namenode_opts: "-Xms6144m -Xmx6144m
-javaagent:/usr/share/java/prometheus/jmx_prometheus_javaagent.jar=%{::ipaddress}:10080:/etc/hadoop/prometheus_hdfs_namenode_jmx_exporter.yaml"
profile::hadoop::common::yarn_resourcemanager_opts: "-Xms2048m
-javaagent:/usr/share/java/prometheus/jmx_prometheus_javaagent.jar=%{::ipaddress}:10083:/etc/hadoop/prometheus_yarn_resourcemanager_jmx_exporter.yaml"
diff --git a/modules/profile/manifests/hadoop/master.pp
b/modules/profile/manifests/hadoop/master.pp
index 98ec3b6..ecd11c5 100644
--- a/modules/profile/manifests/hadoop/master.pp
+++ b/modules/profile/manifests/hadoop/master.pp
@@ -10,6 +10,7 @@
class profile::hadoop::master(
$monitoring_enabled =
hiera('profile::hadoop::master::monitoring_enabled'),
$hadoop_namenode_heapsize =
hiera('profile::hadoop::master::namenode_heapsize'),
+$hadoop_user_groups =
hiera('profile::hadoop::master::hadoop_user_groups'),
$statsd = hiera('statsd'),
){
@@ -28,6 +29,7 @@
# where all users that want to use Hadoop
# must have shell accounts anyway.
class { '::cdh::hadoop::users':
+groups => $hadoop_user_groups,
require => Class['cdh::hadoop::master'],
}
--
To view, visit https://gerrit.wikimedia.org/r/401904
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Ifde824c1213da10b68164b5f9a20416d1093c58c
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
Gerrit-Reviewer: Elukey
Gerrit-Reviewer: Ottomata
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: profile::hadoop: set hiera defaults to ease labs deployments
Elukey has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/402050 )
Change subject: profile::hadoop: set hiera defaults to ease labs deployments
..
profile::hadoop: set hiera defaults to ease labs deployments
Bug: T167790
Change-Id: Ifb5bbce5d6919ac220f9004fc7d24f52c4ef8912
---
M modules/profile/manifests/hadoop/logstash.pp
M modules/profile/manifests/hadoop/master.pp
M modules/profile/manifests/hadoop/master/standby.pp
M modules/profile/manifests/hadoop/worker.pp
4 files changed, 12 insertions(+), 6 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/50/402050/1
diff --git a/modules/profile/manifests/hadoop/logstash.pp
b/modules/profile/manifests/hadoop/logstash.pp
index a454109..aca0eda 100644
--- a/modules/profile/manifests/hadoop/logstash.pp
+++ b/modules/profile/manifests/hadoop/logstash.pp
@@ -2,7 +2,7 @@
# Enables gelf logging to logstash from Hadoop.
# As of 2016-02, this is not used.
class profile::hadoop::logstash (
-$gelf_logging_enabled = hiera('profile::hadoop::logstash::enabled'),
+$gelf_logging_enabled = hiera('profile::hadoop::logstash::enabled', false),
) {
file { '/usr/local/bin/hadoop-yarn-logging-helper.sh':
content => template('profile/hadoop/hadoop-yarn-logging-helper.erb'),
diff --git a/modules/profile/manifests/hadoop/master.pp
b/modules/profile/manifests/hadoop/master.pp
index 98ec3b6..2843d71 100644
--- a/modules/profile/manifests/hadoop/master.pp
+++ b/modules/profile/manifests/hadoop/master.pp
@@ -4,12 +4,15 @@
#
# == Parameters
#
+# [*hadoop_namenode_heapsize*]
+#Current JVM heap size to use as threshold for monitoring.
+#
# [*monitoring_enabled*]
#If production monitoring needs to be enabled or not.
#
class profile::hadoop::master(
-$monitoring_enabled =
hiera('profile::hadoop::master::monitoring_enabled'),
-$hadoop_namenode_heapsize =
hiera('profile::hadoop::master::namenode_heapsize'),
+$monitoring_enabled =
hiera('profile::hadoop::master::monitoring_enabled', false),
+$hadoop_namenode_heapsize =
hiera('profile::hadoop::master::namenode_heapsize', 2048),
$statsd = hiera('statsd'),
){
diff --git a/modules/profile/manifests/hadoop/master/standby.pp
b/modules/profile/manifests/hadoop/master/standby.pp
index 6a98a36..ddbf1bb 100644
--- a/modules/profile/manifests/hadoop/master/standby.pp
+++ b/modules/profile/manifests/hadoop/master/standby.pp
@@ -2,12 +2,15 @@
#
# Sets up a standby/backup Hadoop Master node.
#
+# [*hadoop_namenode_heapsize*]
+#Current JVM heap size to use as threshold for monitoring.
+#
# [*monitoring_enabled*]
#If production monitoring needs to be enabled or not.
#
class profile::hadoop::master::standby(
-$monitoring_enabled =
hiera('profile::hadoop::standby_master::monitoring_enabled'),
-$hadoop_namenode_heapsize =
hiera('profile::hadoop::standby::namenode_heapsize'),
+$monitoring_enabled =
hiera('profile::hadoop::standby_master::monitoring_enabled', false),
+$hadoop_namenode_heapsize =
hiera('profile::hadoop::standby::namenode_heapsize', 2048),
$statsd = hiera('statsd'),
) {
diff --git a/modules/profile/manifests/hadoop/worker.pp
b/modules/profile/manifests/hadoop/worker.pp
index 6895792..6af651b 100644
--- a/modules/profile/manifests/hadoop/worker.pp
+++ b/modules/profile/manifests/hadoop/worker.pp
@@ -8,7 +8,7 @@
#If production monitoring needs to be enabled or not.
#
class profile::hadoop::worker(
-$monitoring_enabled = hiera('profile::hadoop::worker::monitoring_enabled'),
+$monitoring_enabled = hiera('profile::hadoop::worker::monitoring_enabled',
false),
$statsd = hiera('statsd'),
) {
--
To view, visit https://gerrit.wikimedia.org/r/402050
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ifb5bbce5d6919ac220f9004fc7d24f52c4ef8912
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::prometheus::analytics: add configuration for jmx hadoo...
Elukey has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/402021 )
Change subject: role::prometheus::analytics: add configuration for jmx hadoop
agents
..
role::prometheus::analytics: add configuration for jmx hadoop agents
Bug: T177458
Change-Id: I5b701695eb20ca6c8bba74af89ece91e2c71e31e
---
M modules/role/manifests/prometheus/analytics.pp
1 file changed, 26 insertions(+), 0 deletions(-)
Approvals:
Elukey: Looks good to me, approved
jenkins-bot: Verified
Filippo Giunchedi: Looks good to me, but someone else must approve
diff --git a/modules/role/manifests/prometheus/analytics.pp
b/modules/role/manifests/prometheus/analytics.pp
index 1cb5937..21d46c6 100644
--- a/modules/role/manifests/prometheus/analytics.pp
+++ b/modules/role/manifests/prometheus/analytics.pp
@@ -32,8 +32,34 @@
{ 'files' => [ "${targets_path}/jmx_druid_*.yaml" ]}
],
},
+ {
+'job_name'=> 'jmx_hadoop',
+'scrape_timeout' => '25s',
+'scheme' => 'http',
+'file_sd_configs' => [
+ { 'files' => [ "${targets_path}/jmx_hadoop_*.yaml" ]}
+],
+ },
]
+prometheus::jmx_exporter_config{ "hadoop_worker_${::site}":
+dest => "${targets_path}/jmx_hadoop_worker_${::site}.yaml",
+class_name => 'role::analytics_cluster::hadoop::worker',
+site => $::site,
+}
+
+prometheus::jmx_exporter_config{ "hadoop_master_${::site}":
+dest => "${targets_path}/jmx_hadoop_master_${::site}.yaml",
+class_name => 'role::analytics_cluster::hadoop::master',
+site => $::site,
+}
+
+prometheus::jmx_exporter_config{ "hadoop_standby_${::site}":
+dest => "${targets_path}/jmx_hadoop_standby_${::site}.yaml",
+class_name => 'role::analytics_cluster::hadoop::standby',
+site => $::site,
+}
+
prometheus::jmx_exporter_config{ "druid_public_${::site}":
dest => "${targets_path}/jmx_druid_public_${::site}.yaml",
class_name => 'role::druid::public::worker',
--
To view, visit https://gerrit.wikimedia.org/r/402021
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I5b701695eb20ca6c8bba74af89ece91e2c71e31e
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
Gerrit-Reviewer: Elukey
Gerrit-Reviewer: Filippo Giunchedi
Gerrit-Reviewer: Giuseppe Lavagetto
Gerrit-Reviewer: Ottomata
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::prometheus::analytics: add configuration for jmx hadoo...
Elukey has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/402021 )
Change subject: role::prometheus::analytics: add configuration for jmx hadoop
agents
..
role::prometheus::analytics: add configuration for jmx hadoop agents
Bug: T177458
Change-Id: I5b701695eb20ca6c8bba74af89ece91e2c71e31e
---
M modules/role/manifests/prometheus/analytics.pp
1 file changed, 26 insertions(+), 0 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/21/402021/1
diff --git a/modules/role/manifests/prometheus/analytics.pp
b/modules/role/manifests/prometheus/analytics.pp
index 1cb5937..21d46c6 100644
--- a/modules/role/manifests/prometheus/analytics.pp
+++ b/modules/role/manifests/prometheus/analytics.pp
@@ -32,8 +32,34 @@
{ 'files' => [ "${targets_path}/jmx_druid_*.yaml" ]}
],
},
+ {
+'job_name'=> 'jmx_hadoop',
+'scrape_timeout' => '25s',
+'scheme' => 'http',
+'file_sd_configs' => [
+ { 'files' => [ "${targets_path}/jmx_hadoop_*.yaml" ]}
+],
+ },
]
+prometheus::jmx_exporter_config{ "hadoop_worker_${::site}":
+dest => "${targets_path}/jmx_hadoop_worker_${::site}.yaml",
+class_name => 'role::analytics_cluster::hadoop::worker',
+site => $::site,
+}
+
+prometheus::jmx_exporter_config{ "hadoop_master_${::site}":
+dest => "${targets_path}/jmx_hadoop_master_${::site}.yaml",
+class_name => 'role::analytics_cluster::hadoop::master',
+site => $::site,
+}
+
+prometheus::jmx_exporter_config{ "hadoop_standby_${::site}":
+dest => "${targets_path}/jmx_hadoop_standby_${::site}.yaml",
+class_name => 'role::analytics_cluster::hadoop::standby',
+site => $::site,
+}
+
prometheus::jmx_exporter_config{ "druid_public_${::site}":
dest => "${targets_path}/jmx_druid_public_${::site}.yaml",
class_name => 'role::druid::public::worker',
--
To view, visit https://gerrit.wikimedia.org/r/402021
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I5b701695eb20ca6c8bba74af89ece91e2c71e31e
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Refactor thorium's roles in one
Elukey has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/401927 )
Change subject: Refactor thorium's roles in one
..
Refactor thorium's roles in one
Bug: T167790
Change-Id: I0decd3c62aab104c76915eacad1ab3339107a7f3
---
D hieradata/common/cdh/hue.yaml
R hieradata/role/common/analytics_cluster/webserver.yaml
D hieradata/role/common/druid/analytics/pivot.yaml
M manifests/site.pp
A modules/profile/manifests/druid/pivot.pp
A modules/profile/manifests/hue.pp
D modules/role/manifests/analytics_cluster/hue.pp
A modules/role/manifests/analytics_cluster/webserver.pp
D modules/role/manifests/druid/analytics/pivot.pp
9 files changed, 113 insertions(+), 94 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/27/401927/1
diff --git a/hieradata/common/cdh/hue.yaml b/hieradata/common/cdh/hue.yaml
deleted file mode 100644
index f6b8a7b..000
--- a/hieradata/common/cdh/hue.yaml
+++ /dev/null
@@ -1,13 +0,0 @@
-#
-# Analytics Cluster Hue Configuration
-#
-
-# Don't auto create Hue users from LDAP in production.
-cdh::hue::ldap_create_users_on_login: false
-
-# External Database configuration
-cdh::hue::database_host: 'analytics1003.eqiad.wmnet'
-cdh::hue::database_engine: 'mysql'
-cdh::hue::database_user: 'hue'
-cdh::hue::database_port: '3306'
-cdh::hue::database_name: 'hue'
\ No newline at end of file
diff --git a/hieradata/role/common/statistics/web.yaml
b/hieradata/role/common/analytics_cluster/webserver.yaml
similarity index 66%
rename from hieradata/role/common/statistics/web.yaml
rename to hieradata/role/common/analytics_cluster/webserver.yaml
index e786679..64c5135 100644
--- a/hieradata/role/common/statistics/web.yaml
+++ b/hieradata/role/common/analytics_cluster/webserver.yaml
@@ -9,3 +9,9 @@
profile::superset::database_uri:
mysql://superset@analytics1003.eqiad.wmnet/superset
profile::superset::workers: 8
profile::superset::ldap_proxy_enabled: true
+
+profile::hue::database_host: 'analytics1003.eqiad.wmnet'
+profile::hue::hive_server_host: 'analytics1003.eqiad.wmnet'
+profile::hue::monitoring_enabled: true
+
+profile::druid::pivot::druid_broker: 'druid1001.eqiad.wmnet:8082'
\ No newline at end of file
diff --git a/hieradata/role/common/druid/analytics/pivot.yaml
b/hieradata/role/common/druid/analytics/pivot.yaml
deleted file mode 100644
index c8049db..000
--- a/hieradata/role/common/druid/analytics/pivot.yaml
+++ /dev/null
@@ -1 +0,0 @@
-pivot::druid_broker: 'druid1001.eqiad.wmnet:8082'
\ No newline at end of file
diff --git a/manifests/site.pp b/manifests/site.pp
index 7cd34b6..31d8db9 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -2054,13 +2054,7 @@
# related role/module.
#
# This node is not intended for data processing.
-role(statistics::web,
-druid::analytics::pivot,
-analytics_cluster::hue)
-
-
-include ::standard
-include ::base::firewall
+role(analytics_cluster::webserver)
}
# Failoid service (Ganeti VM)
diff --git a/modules/profile/manifests/druid/pivot.pp
b/modules/profile/manifests/druid/pivot.pp
new file mode 100644
index 000..85997b9
--- /dev/null
+++ b/modules/profile/manifests/druid/pivot.pp
@@ -0,0 +1,11 @@
+# Class: profile::druid::pivot
+#
+# Install and configure the Druid's Pivot nodejs UI
+#
+class profile::druid::pivot(
+$druid_broker = hiera('profile::druid::pivot::druid_broker'),
+) {
+class { 'pivot':
+druid_broker => $druid_broker,
+}
+}
\ No newline at end of file
diff --git a/modules/profile/manifests/hue.pp b/modules/profile/manifests/hue.pp
new file mode 100644
index 000..6a12058
--- /dev/null
+++ b/modules/profile/manifests/hue.pp
@@ -0,0 +1,78 @@
+# == Class profile::hue
+#
+# Installs Hue server.
+#
+class profile::hue (
+$hive_server_host = hiera('profile::hue::hive_server_host'),
+$database_host = hiera('profile::hue::database_host'),
+$database_engine= hiera('profile::hue::database_engine',
'mysql'),
+$database_user = hiera('profile::hue::database_user', 'hue'),
+$database_port = hiera('profile::hue::database_port', 3306),
+$database_name = hiera('profile::hue::database_name', 'hue'),
+$ldap_create_users_on_login =
hiera('profile::hue::ldap_create_users_on_login', false),
+$monitoring_enabled = hiera('profile::hue::monitoring_enabled',
false),
+){
+
+# Require that all Hue applications
+# have their corresponding clients
+# and configs installed.
+# Include Hadoop ecosystem client classes.
+require ::profile::hadoop::common
+require ::profile::hive::client
+require ::profile::oozie::client
+
+# These don't require any extra configuration,
+# so no role class is needed.
+class { '::cdh::pig': }
+class { '::cdh::sqoop': }
+class { '::cdh::mahout': }
+class { '::cdh::spark': }
+
+#
[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::analytics_cluster::coordinator: fix system::role
Elukey has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/401907 )
Change subject: role::analytics_cluster::coordinator: fix system::role
..
role::analytics_cluster::coordinator: fix system::role
Bug: T167790
Change-Id: I2055b24fb45f7da146ecf97702d5f1b6c6aabe5a
---
M modules/role/manifests/analytics_cluster/coordinator.pp
1 file changed, 1 insertion(+), 1 deletion(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/07/401907/1
diff --git a/modules/role/manifests/analytics_cluster/coordinator.pp
b/modules/role/manifests/analytics_cluster/coordinator.pp
index e0ff78a..6dbecaf 100644
--- a/modules/role/manifests/analytics_cluster/coordinator.pp
+++ b/modules/role/manifests/analytics_cluster/coordinator.pp
@@ -19,7 +19,7 @@
class role::analytics_cluster::coordinator {
system::role { 'analytics_cluster::coordinator':
-description => 'Analytics Cluster running various Hadoop services
(Hive, Camus, Oozie, ..)'
+description => 'Analytics Cluster host running various Hadoop services
(Hive, Camus, Oozie, ..) and maintenance scripts'
}
include ::role::analytics_cluster::client
--
To view, visit https://gerrit.wikimedia.org/r/401907
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I2055b24fb45f7da146ecf97702d5f1b6c6aabe5a
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::analytics_cluster::coordinator: fix system::role
Elukey has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/401907 )
Change subject: role::analytics_cluster::coordinator: fix system::role
..
role::analytics_cluster::coordinator: fix system::role
Bug: T167790
Change-Id: I2055b24fb45f7da146ecf97702d5f1b6c6aabe5a
---
M modules/role/manifests/analytics_cluster/coordinator.pp
1 file changed, 1 insertion(+), 1 deletion(-)
Approvals:
Elukey: Looks good to me, approved
jenkins-bot: Verified
diff --git a/modules/role/manifests/analytics_cluster/coordinator.pp
b/modules/role/manifests/analytics_cluster/coordinator.pp
index e0ff78a..6dbecaf 100644
--- a/modules/role/manifests/analytics_cluster/coordinator.pp
+++ b/modules/role/manifests/analytics_cluster/coordinator.pp
@@ -19,7 +19,7 @@
class role::analytics_cluster::coordinator {
system::role { 'analytics_cluster::coordinator':
-description => 'Analytics Cluster running various Hadoop services
(Hive, Camus, Oozie, ..)'
+description => 'Analytics Cluster host running various Hadoop services
(Hive, Camus, Oozie, ..) and maintenance scripts'
}
include ::role::analytics_cluster::client
--
To view, visit https://gerrit.wikimedia.org/r/401907
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I2055b24fb45f7da146ecf97702d5f1b6c6aabe5a
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
Gerrit-Reviewer: Elukey
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: profile::hadoop::master: remove the last hadoop cdh auto-lookup
Elukey has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/401904 )
Change subject: profile::hadoop::master: remove the last hadoop cdh auto-lookup
..
profile::hadoop::master: remove the last hadoop cdh auto-lookup
Bug: T167790
Change-Id: Ifde824c1213da10b68164b5f9a20416d1093c58c
---
D hieradata/common/cdh/hadoop/users.yaml
M hieradata/role/common/analytics_cluster/hadoop/master.yaml
M modules/profile/manifests/hadoop/master.pp
3 files changed, 5 insertions(+), 2 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/04/401904/1
diff --git a/hieradata/common/cdh/hadoop/users.yaml
b/hieradata/common/cdh/hadoop/users.yaml
deleted file mode 100644
index fd51433..000
--- a/hieradata/common/cdh/hadoop/users.yaml
+++ /dev/null
@@ -1,2 +0,0 @@
-# Ensure that users in these posix groups have home directories in HDFS.
-cdh::hadoop::users::groups: "analytics-users analytics-privatedata-users
analytics-admins analytics-search-users"
diff --git a/hieradata/role/common/analytics_cluster/hadoop/master.yaml
b/hieradata/role/common/analytics_cluster/hadoop/master.yaml
index f257b36..210458a 100644
--- a/hieradata/role/common/analytics_cluster/hadoop/master.yaml
+++ b/hieradata/role/common/analytics_cluster/hadoop/master.yaml
@@ -20,6 +20,9 @@
# Used to set up jvm heap size usage thresholds
profile::hadoop::master::namenode_heapsize: 6144
+# Ensure that users in these posix groups have home directories in HDFS.
+profile::hadoop::master::hadoop_user_groups: "analytics-users
analytics-privatedata-users analytics-admins analytics-search-users"
+
# Prometheus JMX Exporter config templates.
profile::hadoop::common::hadoop_namenode_opts: "-Xms6144m -Xmx6144m
-javaagent:/usr/share/java/prometheus/jmx_prometheus_javaagent.jar=%{::ipaddress}:10080:/etc/hadoop/prometheus_hdfs_namenode_jmx_exporter.yaml"
profile::hadoop::common::yarn_resourcemanager_opts: "-Xms2048m
-javaagent:/usr/share/java/prometheus/jmx_prometheus_javaagent.jar=%{::ipaddress}:10083:/etc/hadoop/prometheus_yarn_resourcemanager_jmx_exporter.yaml"
diff --git a/modules/profile/manifests/hadoop/master.pp
b/modules/profile/manifests/hadoop/master.pp
index 98ec3b6..ecd11c5 100644
--- a/modules/profile/manifests/hadoop/master.pp
+++ b/modules/profile/manifests/hadoop/master.pp
@@ -10,6 +10,7 @@
class profile::hadoop::master(
$monitoring_enabled =
hiera('profile::hadoop::master::monitoring_enabled'),
$hadoop_namenode_heapsize =
hiera('profile::hadoop::master::namenode_heapsize'),
+$hadoop_user_groups =
hiera('profile::hadoop::master::hadoop_user_groups'),
$statsd = hiera('statsd'),
){
@@ -28,6 +29,7 @@
# where all users that want to use Hadoop
# must have shell accounts anyway.
class { '::cdh::hadoop::users':
+groups => $hadoop_user_groups,
require => Class['cdh::hadoop::master'],
}
--
To view, visit https://gerrit.wikimedia.org/r/401904
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ifde824c1213da10b68164b5f9a20416d1093c58c
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: profile::prometheus::alerts: fix Druid ingestion Prometheus ...
Elukey has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/401773 )
Change subject: profile::prometheus::alerts: fix Druid ingestion Prometheus
query
..
profile::prometheus::alerts: fix Druid ingestion Prometheus query
Change-Id: Ide24e2b5ac2e0304e8cd0062540feaef4d8ee170
---
M modules/profile/manifests/prometheus/alerts.pp
1 file changed, 1 insertion(+), 1 deletion(-)
Approvals:
Elukey: Looks good to me, approved
jenkins-bot: Verified
diff --git a/modules/profile/manifests/prometheus/alerts.pp
b/modules/profile/manifests/prometheus/alerts.pp
index 3621a2d..80361ea 100644
--- a/modules/profile/manifests/prometheus/alerts.pp
+++ b/modules/profile/manifests/prometheus/alerts.pp
@@ -10,7 +10,7 @@
# Experimental, only alerting the Analytics alias.
monitoring::check_prometheus { 'druid_realtime_banner_activity':
description => 'Number of banner_activity realtime events received
by Druid over a 30 minutes period',
-query =>
'sum_over_time(druid_realtime_ingest_events_processed_count{cluster="druid_analytics",
instance=~"druid.*:8000", datasource=~"banner_activity_minutely"}[30m])',
+query =>
'scalar(sum(sum_over_time(druid_realtime_ingest_events_processed_count{cluster="druid_analytics",
instance=~"druid.*:8000", datasource=~"banner_activity_minutely"}[30m])))',
prometheus_url => 'http://prometheus.svc.eqiad.wmnet/analytics',
method => 'le',
warning => 10,
--
To view, visit https://gerrit.wikimedia.org/r/401773
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Ide24e2b5ac2e0304e8cd0062540feaef4d8ee170
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
Gerrit-Reviewer: Elukey
Gerrit-Reviewer: Joal
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: profile::prometheus::alerts: fix Druid ingestion Prometheus ...
Elukey has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/401773 )
Change subject: profile::prometheus::alerts: fix Druid ingestion Prometheus
query
..
profile::prometheus::alerts: fix Druid ingestion Prometheus query
Change-Id: Ide24e2b5ac2e0304e8cd0062540feaef4d8ee170
---
M modules/profile/manifests/prometheus/alerts.pp
1 file changed, 1 insertion(+), 1 deletion(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/73/401773/1
diff --git a/modules/profile/manifests/prometheus/alerts.pp
b/modules/profile/manifests/prometheus/alerts.pp
index 3621a2d..80361ea 100644
--- a/modules/profile/manifests/prometheus/alerts.pp
+++ b/modules/profile/manifests/prometheus/alerts.pp
@@ -10,7 +10,7 @@
# Experimental, only alerting the Analytics alias.
monitoring::check_prometheus { 'druid_realtime_banner_activity':
description => 'Number of banner_activity realtime events received
by Druid over a 30 minutes period',
-query =>
'sum_over_time(druid_realtime_ingest_events_processed_count{cluster="druid_analytics",
instance=~"druid.*:8000", datasource=~"banner_activity_minutely"}[30m])',
+query =>
'scalar(sum(sum_over_time(druid_realtime_ingest_events_processed_count{cluster="druid_analytics",
instance=~"druid.*:8000", datasource=~"banner_activity_minutely"}[30m])))',
prometheus_url => 'http://prometheus.svc.eqiad.wmnet/analytics',
method => 'le',
warning => 10,
--
To view, visit https://gerrit.wikimedia.org/r/401773
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ide24e2b5ac2e0304e8cd0062540feaef4d8ee170
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::icinga: add prometheus alerts profile
Elukey has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/401730 )
Change subject: role::icinga: add prometheus alerts profile
..
role::icinga: add prometheus alerts profile
Change-Id: Ic955b3295f5d10472d47dcb11ffae082232dd694
---
A modules/profile/manifests/prometheus/alerts.pp
M modules/role/manifests/icinga.pp
2 files changed, 22 insertions(+), 0 deletions(-)
Approvals:
Elukey: Looks good to me, approved
jenkins-bot: Verified
Volans: Looks good to me, but someone else must approve
diff --git a/modules/profile/manifests/prometheus/alerts.pp
b/modules/profile/manifests/prometheus/alerts.pp
new file mode 100644
index 000..3621a2d
--- /dev/null
+++ b/modules/profile/manifests/prometheus/alerts.pp
@@ -0,0 +1,21 @@
+# == Class: profile::prometheus::alerts
+#
+# Install icinga alerts based on Prometheus metrics.
+# NOTE to be included only from one host, icinga will generate different alerts
+# for all hosts that include this class.
+#
+class profile::prometheus::alerts {
+
+# Monitor Druid realtime ingestion event rate.
+# Experimental, only alerting the Analytics alias.
+monitoring::check_prometheus { 'druid_realtime_banner_activity':
+description => 'Number of banner_activity realtime events received
by Druid over a 30 minutes period',
+query =>
'sum_over_time(druid_realtime_ingest_events_processed_count{cluster="druid_analytics",
instance=~"druid.*:8000", datasource=~"banner_activity_minutely"}[30m])',
+prometheus_url => 'http://prometheus.svc.eqiad.wmnet/analytics',
+method => 'le',
+warning => 10,
+critical=> 0,
+contact_group => 'analytics',
+dashboard_links =>
['https://grafana.wikimedia.org/dashboard/db/prometheus-druid?refresh=1m&panelId=41&fullscreen&orgId=1']
+}
+}
\ No newline at end of file
diff --git a/modules/role/manifests/icinga.pp b/modules/role/manifests/icinga.pp
index 9b0b87d..3d7498e 100644
--- a/modules/role/manifests/icinga.pp
+++ b/modules/role/manifests/icinga.pp
@@ -35,6 +35,7 @@
include icinga::monitor::reading_web
include icinga::event_handlers::raid
include ::profile::bird::anycast_monitoring
+include ::profile::prometheus::alerts
$monitoring_groups = hiera('monitoring::groups')
create_resources(monitoring::group, $monitoring_groups)
--
To view, visit https://gerrit.wikimedia.org/r/401730
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Ic955b3295f5d10472d47dcb11ffae082232dd694
Gerrit-PatchSet: 6
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
Gerrit-Reviewer: Alexandros Kosiaris
Gerrit-Reviewer: Elukey
Gerrit-Reviewer: Filippo Giunchedi
Gerrit-Reviewer: Giuseppe Lavagetto
Gerrit-Reviewer: Joal
Gerrit-Reviewer: Ottomata
Gerrit-Reviewer: Volans
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::graphite::alerts: add Druid realtime ingestion monitoring
Elukey has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/401730 )
Change subject: role::graphite::alerts: add Druid realtime ingestion monitoring
..
role::graphite::alerts: add Druid realtime ingestion monitoring
Change-Id: Ic955b3295f5d10472d47dcb11ffae082232dd694
---
M modules/role/manifests/graphite/alerts.pp
1 file changed, 13 insertions(+), 0 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/30/401730/1
diff --git a/modules/role/manifests/graphite/alerts.pp
b/modules/role/manifests/graphite/alerts.pp
index 8a2a317..a717a2b 100644
--- a/modules/role/manifests/graphite/alerts.pp
+++ b/modules/role/manifests/graphite/alerts.pp
@@ -103,4 +103,17 @@
from=> '10min',
percentage => 50,
}
+
+# Monitor Druid realtime ingestion event rate.
+# Experimental, only alerting the Analytics alias.
+monitoring::check_prometheus { 'druid_realtime_banner_activity':
+description => 'Number of banner_activity realtime events received
by Druid over a 30 minutes period',
+query =>
'sum_over_time(druid_realtime_ingest_events_processed_count{cluster="druid_analytics",
instance=~"druid.*:8000", datasource=~"banner_activity_minutely"}[30m])',
+prometheus_url => "http://prometheus.svc.${::site}.wmnet/analytics";,
+method => 'le'
+warning => 10,
+critical=> 0,
+contact_group => 'analytics',
+dashboard_links =>
['https://grafana.wikimedia.org/dashboard/db/prometheus-druid?refresh=1m&panelId=41&fullscreen&orgId=1']
+}
}
--
To view, visit https://gerrit.wikimedia.org/r/401730
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic955b3295f5d10472d47dcb11ffae082232dd694
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: profile::kafka::monitoring: blacklist unwanted JMX Mbeans
Elukey has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/401481 ) Change subject: profile::kafka::monitoring: blacklist unwanted JMX Mbeans .. profile::kafka::monitoring: blacklist unwanted JMX Mbeans Change-Id: I2ffaa3d8f324232ec96282d8f6f9600d566712bb --- M modules/profile/files/kafka/broker_prometheus_jmx_exporter.yaml 1 file changed, 5 insertions(+), 0 deletions(-) Approvals: Elukey: Looks good to me, approved jenkins-bot: Verified diff --git a/modules/profile/files/kafka/broker_prometheus_jmx_exporter.yaml b/modules/profile/files/kafka/broker_prometheus_jmx_exporter.yaml index 9ef14bb..28405ae 100644 --- a/modules/profile/files/kafka/broker_prometheus_jmx_exporter.yaml +++ b/modules/profile/files/kafka/broker_prometheus_jmx_exporter.yaml @@ -6,6 +6,11 @@ # for kafka related metrics to allow a better readability, more context about # the discussion to establish a convention in T177078. lowercaseOutputName: false +blacklistObjectNames: + - "com.sun.management:type=*" + - "JMImplementation:type=MBeanServerDelegate" + - "java.util.logging:type=Logging" + - "kafka:type=kafka.Log4jController" rules: - pattern : kafka.cluster<>Value name: kafka_cluster_$1_$2 -- To view, visit https://gerrit.wikimedia.org/r/401481 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I2ffaa3d8f324232ec96282d8f6f9600d566712bb Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey Gerrit-Reviewer: Elukey Gerrit-Reviewer: jenkins-bot <> ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: profile::kafka::monitoring: blacklist unwanted JMX Mbeans
Elukey has uploaded a new change for review. ( https://gerrit.wikimedia.org/r/401481 ) Change subject: profile::kafka::monitoring: blacklist unwanted JMX Mbeans .. profile::kafka::monitoring: blacklist unwanted JMX Mbeans Change-Id: I2ffaa3d8f324232ec96282d8f6f9600d566712bb --- M modules/profile/files/kafka/broker_prometheus_jmx_exporter.yaml 1 file changed, 5 insertions(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/81/401481/1 diff --git a/modules/profile/files/kafka/broker_prometheus_jmx_exporter.yaml b/modules/profile/files/kafka/broker_prometheus_jmx_exporter.yaml index 9ef14bb..28405ae 100644 --- a/modules/profile/files/kafka/broker_prometheus_jmx_exporter.yaml +++ b/modules/profile/files/kafka/broker_prometheus_jmx_exporter.yaml @@ -6,6 +6,11 @@ # for kafka related metrics to allow a better readability, more context about # the discussion to establish a convention in T177078. lowercaseOutputName: false +blacklistObjectNames: + - "com.sun.management:type=*" + - "JMImplementation:type=MBeanServerDelegate" + - "java.util.logging:type=Logging" + - "kafka:type=kafka.Log4jController" rules: - pattern : kafka.cluster<>Value name: kafka_cluster_$1_$2 -- To view, visit https://gerrit.wikimedia.org/r/401481 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I2ffaa3d8f324232ec96282d8f6f9600d566712bb Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::druid::public::worker: review jvm settings
Elukey has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/399617 )
Change subject: role::druid::public::worker: review jvm settings
..
role::druid::public::worker: review jvm settings
These settings have been applied successfully to
the Druid Analytics cluster and brought a neat
performance improvement.
Change-Id: I2f94d20dbc60183b5f3bbdedbd155955f633f4b0
---
M hieradata/role/common/druid/public/worker.yaml
1 file changed, 12 insertions(+), 10 deletions(-)
Approvals:
Elukey: Looks good to me, approved
Joal: Looks good to me, but someone else must approve
jenkins-bot: Verified
diff --git a/hieradata/role/common/druid/public/worker.yaml
b/hieradata/role/common/druid/public/worker.yaml
index 7caa004..798e733 100644
--- a/hieradata/role/common/druid/public/worker.yaml
+++ b/hieradata/role/common/druid/public/worker.yaml
@@ -63,7 +63,7 @@
druid.emitter.composing.emitters: ["logging","http"]
druid.emitter.http.recipientBaseUrl: "http://localhost:8000/";
druid.processing.numThreads: 10
- druid.processing.buffer.sizeBytes: 2147483647
+ druid.processing.buffer.sizeBytes: 268435456 # 1024 * 1024 * 256
# Set numMergeBuffers to use v2 groupBy engine
druid.processing.numMergeBuffers: 10
druid.server.http.numThreads: 20
@@ -73,8 +73,8 @@
# TBD: Perhaps we should also try using memcached?
druid.cache.sizeInBytes: 2147483648
profile::druid::broker::env:
- DRUID_HEAP_OPTS: "-Xmx25g -Xms25g"
- DRUID_EXTRA_JVM_OPTS: "-XX:NewSize=6g -XX:MaxNewSize=6g
-XX:MaxDirectMemorySize=64g -XX:+UseConcMarkSweepGC -XX:+PrintGCDetails
-XX:+PrintGCTimeStamps"
+ DRUID_HEAP_OPTS: "-Xmx10g -Xms5g"
+ DRUID_EXTRA_JVM_OPTS: "-XX:NewSize=4g -XX:MaxNewSize=4g
-XX:MaxDirectMemorySize=12g -XX:+UseConcMarkSweepGC -XX:+PrintGCDetails
-XX:+PrintGCTimeStamps"
# --- Druid Coordinator
@@ -85,7 +85,7 @@
druid.emitter.composing.emitters: ["logging","http"]
druid.emitter.http.recipientBaseUrl: "http://localhost:8000/";
profile::druid::coordinator::env:
- DRUID_HEAP_OPTS: "-Xmx10g -Xms10g"
+ DRUID_HEAP_OPTS: "-Xmx4g -Xms2g"
DRUID_EXTRA_JVM_OPTS: "-XX:NewSize=512m -XX:MaxNewSize=512m -XX:+UseG1GC
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps"
@@ -96,18 +96,20 @@
druid.emitter.composing.emitters: ["logging","http"]
druid.emitter.http.recipientBaseUrl: "http://localhost:8000/";
druid.processing.numThreads: 10
- druid.processing.buffer.sizeBytes: 1073741824
+ druid.processing.buffer.sizeBytes: 268435456 # 1024 * 1024 * 256
# Set numMergeBuffers to use v2 groupBy engine
druid.processing.numMergeBuffers: 10
druid.server.http.numThreads: 20
druid.server.maxSize: 2748779069440 # 2.5 TB
druid.segmentCache.locations:
'[{"path":"/var/lib/druid/segment-cache","maxSize"\:2748779069440}]'
- druid.historical.cache.useCache: true
- druid.historical.cache.populateCache: true
+ # For small clusters it is reccomended to only enable caching on brokers
+ # See: http://druid.io/docs/latest/querying/caching.html
+ druid.historical.cache.useCache: false
+ druid.historical.cache.populateCache: false
druid.monitoring.monitors:
["io.druid.server.metrics.HistoricalMetricsMonitor"]
profile::druid::historical::env:
- DRUID_HEAP_OPTS: "-Xmx12g -Xms12g"
- DRUID_EXTRA_JVM_OPTS: "-XX:NewSize=6g -XX:MaxNewSize=6g
-XX:MaxDirectMemorySize=32g -XX:+UseConcMarkSweepGC -XX:+PrintGCDetails
-XX:+PrintGCTimeStamps"
+ DRUID_HEAP_OPTS: "-Xmx8g -Xms4g"
+ DRUID_EXTRA_JVM_OPTS: "-XX:NewSize=2g -XX:MaxNewSize=2g
-XX:MaxDirectMemorySize=10g -XX:+UseConcMarkSweepGC -XX:+PrintGCDetails
-XX:+PrintGCTimeStamps"
# --- Druid MiddleManager
@@ -134,5 +136,5 @@
druid.indexer.runner.type: remote
druid.indexer.storage.type: metadata
profile::druid::overlord::env:
- DRUID_HEAP_OPTS: "-Xmx4g -Xms4g"
+ DRUID_HEAP_OPTS: "-Xmx4g -Xms2g"
DRUID_EXTRA_JVM_OPTS: "-XX:NewSize=256m -XX:MaxNewSize=256m
-XX:+UseConcMarkSweepGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"
--
To view, visit https://gerrit.wikimedia.org/r/399617
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I2f94d20dbc60183b5f3bbdedbd155955f633f4b0
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
Gerrit-Reviewer: Elukey
Gerrit-Reviewer: Joal
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::druid::public::worker: review jvm settings
Elukey has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/399617 )
Change subject: role::druid::public::worker: review jvm settings
..
role::druid::public::worker: review jvm settings
These settings have been applied successfully to
the Druid Analytics cluster and brought a neat
performance improvement.
Change-Id: I2f94d20dbc60183b5f3bbdedbd155955f633f4b0
---
M hieradata/role/common/druid/public/worker.yaml
1 file changed, 12 insertions(+), 10 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/17/399617/1
diff --git a/hieradata/role/common/druid/public/worker.yaml
b/hieradata/role/common/druid/public/worker.yaml
index 7caa004..798e733 100644
--- a/hieradata/role/common/druid/public/worker.yaml
+++ b/hieradata/role/common/druid/public/worker.yaml
@@ -63,7 +63,7 @@
druid.emitter.composing.emitters: ["logging","http"]
druid.emitter.http.recipientBaseUrl: "http://localhost:8000/";
druid.processing.numThreads: 10
- druid.processing.buffer.sizeBytes: 2147483647
+ druid.processing.buffer.sizeBytes: 268435456 # 1024 * 1024 * 256
# Set numMergeBuffers to use v2 groupBy engine
druid.processing.numMergeBuffers: 10
druid.server.http.numThreads: 20
@@ -73,8 +73,8 @@
# TBD: Perhaps we should also try using memcached?
druid.cache.sizeInBytes: 2147483648
profile::druid::broker::env:
- DRUID_HEAP_OPTS: "-Xmx25g -Xms25g"
- DRUID_EXTRA_JVM_OPTS: "-XX:NewSize=6g -XX:MaxNewSize=6g
-XX:MaxDirectMemorySize=64g -XX:+UseConcMarkSweepGC -XX:+PrintGCDetails
-XX:+PrintGCTimeStamps"
+ DRUID_HEAP_OPTS: "-Xmx10g -Xms5g"
+ DRUID_EXTRA_JVM_OPTS: "-XX:NewSize=4g -XX:MaxNewSize=4g
-XX:MaxDirectMemorySize=12g -XX:+UseConcMarkSweepGC -XX:+PrintGCDetails
-XX:+PrintGCTimeStamps"
# --- Druid Coordinator
@@ -85,7 +85,7 @@
druid.emitter.composing.emitters: ["logging","http"]
druid.emitter.http.recipientBaseUrl: "http://localhost:8000/";
profile::druid::coordinator::env:
- DRUID_HEAP_OPTS: "-Xmx10g -Xms10g"
+ DRUID_HEAP_OPTS: "-Xmx4g -Xms2g"
DRUID_EXTRA_JVM_OPTS: "-XX:NewSize=512m -XX:MaxNewSize=512m -XX:+UseG1GC
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps"
@@ -96,18 +96,20 @@
druid.emitter.composing.emitters: ["logging","http"]
druid.emitter.http.recipientBaseUrl: "http://localhost:8000/";
druid.processing.numThreads: 10
- druid.processing.buffer.sizeBytes: 1073741824
+ druid.processing.buffer.sizeBytes: 268435456 # 1024 * 1024 * 256
# Set numMergeBuffers to use v2 groupBy engine
druid.processing.numMergeBuffers: 10
druid.server.http.numThreads: 20
druid.server.maxSize: 2748779069440 # 2.5 TB
druid.segmentCache.locations:
'[{"path":"/var/lib/druid/segment-cache","maxSize"\:2748779069440}]'
- druid.historical.cache.useCache: true
- druid.historical.cache.populateCache: true
+ # For small clusters it is reccomended to only enable caching on brokers
+ # See: http://druid.io/docs/latest/querying/caching.html
+ druid.historical.cache.useCache: false
+ druid.historical.cache.populateCache: false
druid.monitoring.monitors:
["io.druid.server.metrics.HistoricalMetricsMonitor"]
profile::druid::historical::env:
- DRUID_HEAP_OPTS: "-Xmx12g -Xms12g"
- DRUID_EXTRA_JVM_OPTS: "-XX:NewSize=6g -XX:MaxNewSize=6g
-XX:MaxDirectMemorySize=32g -XX:+UseConcMarkSweepGC -XX:+PrintGCDetails
-XX:+PrintGCTimeStamps"
+ DRUID_HEAP_OPTS: "-Xmx8g -Xms4g"
+ DRUID_EXTRA_JVM_OPTS: "-XX:NewSize=2g -XX:MaxNewSize=2g
-XX:MaxDirectMemorySize=10g -XX:+UseConcMarkSweepGC -XX:+PrintGCDetails
-XX:+PrintGCTimeStamps"
# --- Druid MiddleManager
@@ -134,5 +136,5 @@
druid.indexer.runner.type: remote
druid.indexer.storage.type: metadata
profile::druid::overlord::env:
- DRUID_HEAP_OPTS: "-Xmx4g -Xms4g"
+ DRUID_HEAP_OPTS: "-Xmx4g -Xms2g"
DRUID_EXTRA_JVM_OPTS: "-XX:NewSize=256m -XX:MaxNewSize=256m
-XX:+UseConcMarkSweepGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"
--
To view, visit https://gerrit.wikimedia.org/r/399617
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I2f94d20dbc60183b5f3bbdedbd155955f633f4b0
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: profile::hadoop::monitoring: add explicit jmx whitelist
Elukey has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/399608 ) Change subject: profile::hadoop::monitoring: add explicit jmx whitelist .. profile::hadoop::monitoring: add explicit jmx whitelist Bug: T177458 Change-Id: Id96c27e7d3ce582052645a14d9413591dd67ab1c --- M modules/profile/files/hadoop/prometheus_hdfs_datanode_jmx_exporter.yaml M modules/profile/files/hadoop/prometheus_hdfs_namenode_jmx_exporter.yaml M modules/profile/files/hadoop/prometheus_mapreduce_history_jmx_exporter.yaml M modules/profile/files/hadoop/prometheus_yarn_nodemanager_jmx_exporter.yaml M modules/profile/files/hadoop/prometheus_yarn_resourcemanager_jmx_exporter.yaml M modules/profile/files/hive/prometheus_hive_metastore_jmx_exporter.yaml M modules/profile/files/hive/prometheus_hive_server_jmx_exporter.yaml M modules/profile/files/oozie/prometheus_oozie_server_jmx_exporter.yaml 8 files changed, 61 insertions(+), 10 deletions(-) Approvals: Elukey: Looks good to me, approved jenkins-bot: Verified diff --git a/modules/profile/files/hadoop/prometheus_hdfs_datanode_jmx_exporter.yaml b/modules/profile/files/hadoop/prometheus_hdfs_datanode_jmx_exporter.yaml index 2408036..f60acbe 100644 --- a/modules/profile/files/hadoop/prometheus_hdfs_datanode_jmx_exporter.yaml +++ b/modules/profile/files/hadoop/prometheus_hdfs_datanode_jmx_exporter.yaml @@ -1,5 +1,11 @@ --- lowercaseOutputLabelNames: true lowercaseOutputName: false -blacklistObjectNames: - - 'Hadoop:service=DataNode,name=DataNodeInfo' \ No newline at end of file +whitelistObjectNames: + - 'Hadoop:service=DataNode,name=DataNodeActivity*' + - 'Hadoop:service=DataNode,name=FSDatasetState*' + - 'java.lang:type=ClassLoading' + - 'java.lang:type=Compilation' + - 'java.lang:type=GarbageCollector*' + - 'java.lang:type=Memory' + - 'java.lang:type=MemoryPool*' \ No newline at end of file diff --git a/modules/profile/files/hadoop/prometheus_hdfs_namenode_jmx_exporter.yaml b/modules/profile/files/hadoop/prometheus_hdfs_namenode_jmx_exporter.yaml index 73530b2..1063174 100644 --- a/modules/profile/files/hadoop/prometheus_hdfs_namenode_jmx_exporter.yaml +++ b/modules/profile/files/hadoop/prometheus_hdfs_namenode_jmx_exporter.yaml @@ -1,3 +1,11 @@ --- lowercaseOutputLabelNames: true -lowercaseOutputName: false \ No newline at end of file +lowercaseOutputName: false +whitelistObjectNames: + - 'Hadoop:service=NameNode,name=FSNamesystem' + - 'Hadoop:service=NameNode,name=NameNodeActivity' + - 'java.lang:type=ClassLoading' + - 'java.lang:type=Compilation' + - 'java.lang:type=GarbageCollector*' + - 'java.lang:type=Memory' + - 'java.lang:type=MemoryPool*' \ No newline at end of file diff --git a/modules/profile/files/hadoop/prometheus_mapreduce_history_jmx_exporter.yaml b/modules/profile/files/hadoop/prometheus_mapreduce_history_jmx_exporter.yaml index 73530b2..5532f5e 100644 --- a/modules/profile/files/hadoop/prometheus_mapreduce_history_jmx_exporter.yaml +++ b/modules/profile/files/hadoop/prometheus_mapreduce_history_jmx_exporter.yaml @@ -1,3 +1,9 @@ --- lowercaseOutputLabelNames: true -lowercaseOutputName: false \ No newline at end of file +lowercaseOutputName: false +whitelistObjectNames: + - 'java.lang:type=ClassLoading' + - 'java.lang:type=Compilation' + - 'java.lang:type=GarbageCollector*' + - 'java.lang:type=Memory' + - 'java.lang:type=MemoryPool*' \ No newline at end of file diff --git a/modules/profile/files/hadoop/prometheus_yarn_nodemanager_jmx_exporter.yaml b/modules/profile/files/hadoop/prometheus_yarn_nodemanager_jmx_exporter.yaml index 42bef1c..f1a7ace 100644 --- a/modules/profile/files/hadoop/prometheus_yarn_nodemanager_jmx_exporter.yaml +++ b/modules/profile/files/hadoop/prometheus_yarn_nodemanager_jmx_exporter.yaml @@ -1,5 +1,11 @@ --- lowercaseOutputLabelNames: true lowercaseOutputName: false -blacklistObjectNames: - - 'Hadoop:service=NodeManager,name=ContainerResource_container_*' \ No newline at end of file +whitelistObjectNames: + - 'Hadoop:service=NodeManager,name=NodeManagerMetrics' + - 'Hadoop:service=NodeManager,name=ShuffleMetrics' + - 'java.lang:type=ClassLoading' + - 'java.lang:type=Compilation' + - 'java.lang:type=GarbageCollector*' + - 'java.lang:type=Memory' + - 'java.lang:type=MemoryPool*' \ No newline at end of file diff --git a/modules/profile/files/hadoop/prometheus_yarn_resourcemanager_jmx_exporter.yaml b/modules/profile/files/hadoop/prometheus_yarn_resourcemanager_jmx_exporter.yaml index 73530b2..c7ea9f6 100644 --- a/modules/profile/files/hadoop/prometheus_yarn_resourcemanager_jmx_exporter.yaml +++ b/modules/profile/files/hadoop/prometheus_yarn_resourcemanager_jmx_exporter.yaml @@ -1,3 +1,10 @@ --- lowercaseOutputLabelNames: true -lowercaseOutputName: false \ No newline at end of file +lowercaseOutputName: false +whitelistObjectNames: + - 'Hadoop:service=ResourceManager,name=ClusterMetrics' + - 'java.lan
[MediaWiki-commits] [Gerrit] operations/puppet[production]: profile::hadoop::monitoring: add explicit jmx whitelist
Elukey has uploaded a new change for review. ( https://gerrit.wikimedia.org/r/399608 ) Change subject: profile::hadoop::monitoring: add explicit jmx whitelist .. profile::hadoop::monitoring: add explicit jmx whitelist Bug: T177458 Change-Id: Id96c27e7d3ce582052645a14d9413591dd67ab1c --- M modules/profile/files/hadoop/prometheus_hdfs_datanode_jmx_exporter.yaml M modules/profile/files/hadoop/prometheus_hdfs_namenode_jmx_exporter.yaml M modules/profile/files/hadoop/prometheus_mapreduce_history_jmx_exporter.yaml M modules/profile/files/hadoop/prometheus_yarn_nodemanager_jmx_exporter.yaml M modules/profile/files/hadoop/prometheus_yarn_resourcemanager_jmx_exporter.yaml M modules/profile/files/hive/prometheus_hive_metastore_jmx_exporter.yaml M modules/profile/files/hive/prometheus_hive_server_jmx_exporter.yaml M modules/profile/files/oozie/prometheus_oozie_server_jmx_exporter.yaml 8 files changed, 61 insertions(+), 10 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/08/399608/1 diff --git a/modules/profile/files/hadoop/prometheus_hdfs_datanode_jmx_exporter.yaml b/modules/profile/files/hadoop/prometheus_hdfs_datanode_jmx_exporter.yaml index 2408036..f60acbe 100644 --- a/modules/profile/files/hadoop/prometheus_hdfs_datanode_jmx_exporter.yaml +++ b/modules/profile/files/hadoop/prometheus_hdfs_datanode_jmx_exporter.yaml @@ -1,5 +1,11 @@ --- lowercaseOutputLabelNames: true lowercaseOutputName: false -blacklistObjectNames: - - 'Hadoop:service=DataNode,name=DataNodeInfo' \ No newline at end of file +whitelistObjectNames: + - 'Hadoop:service=DataNode,name=DataNodeActivity*' + - 'Hadoop:service=DataNode,name=FSDatasetState*' + - 'java.lang:type=ClassLoading' + - 'java.lang:type=Compilation' + - 'java.lang:type=GarbageCollector*' + - 'java.lang:type=Memory' + - 'java.lang:type=MemoryPool*' \ No newline at end of file diff --git a/modules/profile/files/hadoop/prometheus_hdfs_namenode_jmx_exporter.yaml b/modules/profile/files/hadoop/prometheus_hdfs_namenode_jmx_exporter.yaml index 73530b2..1063174 100644 --- a/modules/profile/files/hadoop/prometheus_hdfs_namenode_jmx_exporter.yaml +++ b/modules/profile/files/hadoop/prometheus_hdfs_namenode_jmx_exporter.yaml @@ -1,3 +1,11 @@ --- lowercaseOutputLabelNames: true -lowercaseOutputName: false \ No newline at end of file +lowercaseOutputName: false +whitelistObjectNames: + - 'Hadoop:service=NameNode,name=FSNamesystem' + - 'Hadoop:service=NameNode,name=NameNodeActivity' + - 'java.lang:type=ClassLoading' + - 'java.lang:type=Compilation' + - 'java.lang:type=GarbageCollector*' + - 'java.lang:type=Memory' + - 'java.lang:type=MemoryPool*' \ No newline at end of file diff --git a/modules/profile/files/hadoop/prometheus_mapreduce_history_jmx_exporter.yaml b/modules/profile/files/hadoop/prometheus_mapreduce_history_jmx_exporter.yaml index 73530b2..5532f5e 100644 --- a/modules/profile/files/hadoop/prometheus_mapreduce_history_jmx_exporter.yaml +++ b/modules/profile/files/hadoop/prometheus_mapreduce_history_jmx_exporter.yaml @@ -1,3 +1,9 @@ --- lowercaseOutputLabelNames: true -lowercaseOutputName: false \ No newline at end of file +lowercaseOutputName: false +whitelistObjectNames: + - 'java.lang:type=ClassLoading' + - 'java.lang:type=Compilation' + - 'java.lang:type=GarbageCollector*' + - 'java.lang:type=Memory' + - 'java.lang:type=MemoryPool*' \ No newline at end of file diff --git a/modules/profile/files/hadoop/prometheus_yarn_nodemanager_jmx_exporter.yaml b/modules/profile/files/hadoop/prometheus_yarn_nodemanager_jmx_exporter.yaml index 42bef1c..f1a7ace 100644 --- a/modules/profile/files/hadoop/prometheus_yarn_nodemanager_jmx_exporter.yaml +++ b/modules/profile/files/hadoop/prometheus_yarn_nodemanager_jmx_exporter.yaml @@ -1,5 +1,11 @@ --- lowercaseOutputLabelNames: true lowercaseOutputName: false -blacklistObjectNames: - - 'Hadoop:service=NodeManager,name=ContainerResource_container_*' \ No newline at end of file +whitelistObjectNames: + - 'Hadoop:service=NodeManager,name=NodeManagerMetrics' + - 'Hadoop:service=NodeManager,name=ShuffleMetrics' + - 'java.lang:type=ClassLoading' + - 'java.lang:type=Compilation' + - 'java.lang:type=GarbageCollector*' + - 'java.lang:type=Memory' + - 'java.lang:type=MemoryPool*' \ No newline at end of file diff --git a/modules/profile/files/hadoop/prometheus_yarn_resourcemanager_jmx_exporter.yaml b/modules/profile/files/hadoop/prometheus_yarn_resourcemanager_jmx_exporter.yaml index 73530b2..c7ea9f6 100644 --- a/modules/profile/files/hadoop/prometheus_yarn_resourcemanager_jmx_exporter.yaml +++ b/modules/profile/files/hadoop/prometheus_yarn_resourcemanager_jmx_exporter.yaml @@ -1,3 +1,10 @@ --- lowercaseOutputLabelNames: true -lowercaseOutputName: false \ No newline at end of file +lowercaseOutputName: false +whitelistObjectNames: + - 'Hadoop:service=ResourceManager,name=ClusterMetrics' + - 'j
[MediaWiki-commits] [Gerrit] operations/puppet[production]: profile::hadoop::monitoring: add explicit jmx whitelist
Elukey has uploaded a new change for review. ( https://gerrit.wikimedia.org/r/399606 ) Change subject: profile::hadoop::monitoring: add explicit jmx whitelist .. profile::hadoop::monitoring: add explicit jmx whitelist Bug: T177458 Change-Id: I287c17d9f5ffdb3bf264b08305f8a42138d7c90b --- M modules/profile/files/hadoop/prometheus_hdfs_datanode_jmx_exporter.yaml M modules/profile/files/hadoop/prometheus_hdfs_namenode_jmx_exporter.yaml M modules/profile/files/hadoop/prometheus_mapreduce_history_jmx_exporter.yaml M modules/profile/files/hadoop/prometheus_yarn_nodemanager_jmx_exporter.yaml M modules/profile/files/hadoop/prometheus_yarn_resourcemanager_jmx_exporter.yaml 5 files changed, 40 insertions(+), 7 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/06/399606/1 diff --git a/modules/profile/files/hadoop/prometheus_hdfs_datanode_jmx_exporter.yaml b/modules/profile/files/hadoop/prometheus_hdfs_datanode_jmx_exporter.yaml index 2408036..f60acbe 100644 --- a/modules/profile/files/hadoop/prometheus_hdfs_datanode_jmx_exporter.yaml +++ b/modules/profile/files/hadoop/prometheus_hdfs_datanode_jmx_exporter.yaml @@ -1,5 +1,11 @@ --- lowercaseOutputLabelNames: true lowercaseOutputName: false -blacklistObjectNames: - - 'Hadoop:service=DataNode,name=DataNodeInfo' \ No newline at end of file +whitelistObjectNames: + - 'Hadoop:service=DataNode,name=DataNodeActivity*' + - 'Hadoop:service=DataNode,name=FSDatasetState*' + - 'java.lang:type=ClassLoading' + - 'java.lang:type=Compilation' + - 'java.lang:type=GarbageCollector*' + - 'java.lang:type=Memory' + - 'java.lang:type=MemoryPool*' \ No newline at end of file diff --git a/modules/profile/files/hadoop/prometheus_hdfs_namenode_jmx_exporter.yaml b/modules/profile/files/hadoop/prometheus_hdfs_namenode_jmx_exporter.yaml index 73530b2..1063174 100644 --- a/modules/profile/files/hadoop/prometheus_hdfs_namenode_jmx_exporter.yaml +++ b/modules/profile/files/hadoop/prometheus_hdfs_namenode_jmx_exporter.yaml @@ -1,3 +1,11 @@ --- lowercaseOutputLabelNames: true -lowercaseOutputName: false \ No newline at end of file +lowercaseOutputName: false +whitelistObjectNames: + - 'Hadoop:service=NameNode,name=FSNamesystem' + - 'Hadoop:service=NameNode,name=NameNodeActivity' + - 'java.lang:type=ClassLoading' + - 'java.lang:type=Compilation' + - 'java.lang:type=GarbageCollector*' + - 'java.lang:type=Memory' + - 'java.lang:type=MemoryPool*' \ No newline at end of file diff --git a/modules/profile/files/hadoop/prometheus_mapreduce_history_jmx_exporter.yaml b/modules/profile/files/hadoop/prometheus_mapreduce_history_jmx_exporter.yaml index 73530b2..5532f5e 100644 --- a/modules/profile/files/hadoop/prometheus_mapreduce_history_jmx_exporter.yaml +++ b/modules/profile/files/hadoop/prometheus_mapreduce_history_jmx_exporter.yaml @@ -1,3 +1,9 @@ --- lowercaseOutputLabelNames: true -lowercaseOutputName: false \ No newline at end of file +lowercaseOutputName: false +whitelistObjectNames: + - 'java.lang:type=ClassLoading' + - 'java.lang:type=Compilation' + - 'java.lang:type=GarbageCollector*' + - 'java.lang:type=Memory' + - 'java.lang:type=MemoryPool*' \ No newline at end of file diff --git a/modules/profile/files/hadoop/prometheus_yarn_nodemanager_jmx_exporter.yaml b/modules/profile/files/hadoop/prometheus_yarn_nodemanager_jmx_exporter.yaml index 42bef1c..f1a7ace 100644 --- a/modules/profile/files/hadoop/prometheus_yarn_nodemanager_jmx_exporter.yaml +++ b/modules/profile/files/hadoop/prometheus_yarn_nodemanager_jmx_exporter.yaml @@ -1,5 +1,11 @@ --- lowercaseOutputLabelNames: true lowercaseOutputName: false -blacklistObjectNames: - - 'Hadoop:service=NodeManager,name=ContainerResource_container_*' \ No newline at end of file +whitelistObjectNames: + - 'Hadoop:service=NodeManager,name=NodeManagerMetrics' + - 'Hadoop:service=NodeManager,name=ShuffleMetrics' + - 'java.lang:type=ClassLoading' + - 'java.lang:type=Compilation' + - 'java.lang:type=GarbageCollector*' + - 'java.lang:type=Memory' + - 'java.lang:type=MemoryPool*' \ No newline at end of file diff --git a/modules/profile/files/hadoop/prometheus_yarn_resourcemanager_jmx_exporter.yaml b/modules/profile/files/hadoop/prometheus_yarn_resourcemanager_jmx_exporter.yaml index 73530b2..c7ea9f6 100644 --- a/modules/profile/files/hadoop/prometheus_yarn_resourcemanager_jmx_exporter.yaml +++ b/modules/profile/files/hadoop/prometheus_yarn_resourcemanager_jmx_exporter.yaml @@ -1,3 +1,10 @@ --- lowercaseOutputLabelNames: true -lowercaseOutputName: false \ No newline at end of file +lowercaseOutputName: false +whitelistObjectNames: + - 'Hadoop:service=ResourceManager,name=ClusterMetrics' + - 'java.lang:type=ClassLoading' + - 'java.lang:type=Compilation' + - 'java.lang:type=GarbageCollector*' + - 'java.lang:type=Memory' + - 'java.lang:type=MemoryPool*' \ No newline at end of file -- To view, visit htt
[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::druid::analytics::worker: review jvm configurations
Elukey has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/399205 )
Change subject: role::druid::analytics::worker: review jvm configurations
..
role::druid::analytics::worker: review jvm configurations
Change-Id: I29e75c080e98490e0615a7d12d6e40f668eaea4e
---
M hieradata/role/common/druid/analytics/worker.yaml
1 file changed, 12 insertions(+), 12 deletions(-)
Approvals:
Elukey: Verified; Looks good to me, approved
diff --git a/hieradata/role/common/druid/analytics/worker.yaml
b/hieradata/role/common/druid/analytics/worker.yaml
index c6e5849..82ed90f 100644
--- a/hieradata/role/common/druid/analytics/worker.yaml
+++ b/hieradata/role/common/druid/analytics/worker.yaml
@@ -60,7 +60,6 @@
# -- Druid worker service configurations
# --- Druid Broker
-
# Broker gets a special ferm_srange since it is the frontend query interface
to Druid.
profile::druid::broker::monitoring_enabled: true
profile::druid::broker::ferm_srange: '$ANALYTICS_NETWORKS'
@@ -69,7 +68,7 @@
druid.emitter.composing.emitters: ["logging","http"]
druid.emitter.http.recipientBaseUrl: "http://localhost:8000/";
druid.processing.numThreads: 10
- druid.processing.buffer.sizeBytes: 2147483647
+ druid.processing.buffer.sizeBytes: 268435456 # 1024 * 1024 * 256
# Set numMergeBuffers to use v2 groupBy engine
druid.processing.numMergeBuffers: 10
druid.server.http.numThreads: 20
@@ -79,9 +78,8 @@
# TBD: Perhaps we should also try using memcached?
druid.cache.sizeInBytes: 2147483648
profile::druid::broker::env:
- DRUID_HEAP_OPTS: "-Xmx25g -Xms25g"
- DRUID_EXTRA_JVM_OPTS: "-XX:NewSize=6g -XX:MaxNewSize=6g
-XX:MaxDirectMemorySize=64g -XX:+UseConcMarkSweepGC -XX:+PrintGCDetails
-XX:+PrintGCTimeStamps"
-
+ DRUID_HEAP_OPTS: "-Xmx10g -Xms5g"
+ DRUID_EXTRA_JVM_OPTS: "-XX:NewSize=4g -XX:MaxNewSize=4g
-XX:MaxDirectMemorySize=12g -XX:+UseConcMarkSweepGC -XX:+PrintGCDetails
-XX:+PrintGCTimeStamps"
# --- Druid Coordinator
profile::druid::coordinator::monitoring_enabled: true
@@ -91,7 +89,7 @@
druid.emitter.composing.emitters: ["logging","http"]
druid.emitter.http.recipientBaseUrl: "http://localhost:8000/";
profile::druid::coordinator::env:
- DRUID_HEAP_OPTS: "-Xmx10g -Xms10g"
+ DRUID_HEAP_OPTS: "-Xmx4g -Xms2g"
DRUID_EXTRA_JVM_OPTS: "-XX:NewSize=512m -XX:MaxNewSize=512m -XX:+UseG1GC
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps"
@@ -102,18 +100,20 @@
druid.emitter.composing.emitters: ["logging","http"]
druid.emitter.http.recipientBaseUrl: "http://localhost:8000/";
druid.processing.numThreads: 10
- druid.processing.buffer.sizeBytes: 1073741824
+ druid.processing.buffer.sizeBytes: 268435456 # 1024 * 1024 * 256
# Set numMergeBuffers to use v2 groupBy engine
druid.processing.numMergeBuffers: 10
druid.server.http.numThreads: 20
druid.server.maxSize: 2748779069440 # 2.5 TB
druid.segmentCache.locations:
'[{"path":"/var/lib/druid/segment-cache","maxSize"\:2748779069440}]'
- druid.historical.cache.useCache: true
- druid.historical.cache.populateCache: true
+ # For small clusters it is reccomended to only enable caching on brokers
+ # See: http://druid.io/docs/latest/querying/caching.html
+ druid.historical.cache.useCache: false
+ druid.historical.cache.populateCache: false
druid.monitoring.monitors:
["io.druid.server.metrics.HistoricalMetricsMonitor"]
profile::druid::historical::env:
- DRUID_HEAP_OPTS: "-Xmx12g -Xms12g"
- DRUID_EXTRA_JVM_OPTS: "-XX:NewSize=6g -XX:MaxNewSize=6g
-XX:MaxDirectMemorySize=32g -XX:+UseConcMarkSweepGC -XX:+PrintGCDetails
-XX:+PrintGCTimeStamps"
+ DRUID_HEAP_OPTS: "-Xmx8g -Xms4g"
+ DRUID_EXTRA_JVM_OPTS: "-XX:NewSize=2g -XX:MaxNewSize=2g
-XX:MaxDirectMemorySize=10g -XX:+UseConcMarkSweepGC -XX:+PrintGCDetails
-XX:+PrintGCTimeStamps"
# --- Druid MiddleManager
@@ -142,5 +142,5 @@
druid.indexer.runner.type: remote
druid.indexer.storage.type: metadata
profile::druid::overlord::env:
- DRUID_HEAP_OPTS: "-Xmx4g -Xms4g"
+ DRUID_HEAP_OPTS: "-Xmx4g -Xms2g"
DRUID_EXTRA_JVM_OPTS: "-XX:NewSize=256m -XX:MaxNewSize=256m
-XX:+UseConcMarkSweepGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"
--
To view, visit https://gerrit.wikimedia.org/r/399205
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I29e75c080e98490e0615a7d12d6e40f668eaea4e
Gerrit-PatchSet: 5
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
Gerrit-Reviewer: Elukey
Gerrit-Reviewer: Joal
Gerrit-Reviewer: Ottomata
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: profile::druid::monitoring: restrict jmx mbeans to query
Elukey has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/399360 ) Change subject: profile::druid::monitoring: restrict jmx mbeans to query .. profile::druid::monitoring: restrict jmx mbeans to query Bug: T183273 Change-Id: I30843d99467b152b0aa99f53ad0aa3c382055473 --- M modules/profile/files/druid/jvm_prometheus_jmx_exporter.yaml 1 file changed, 7 insertions(+), 1 deletion(-) Approvals: Elukey: Looks good to me, approved jenkins-bot: Verified Filippo Giunchedi: Looks good to me, but someone else must approve diff --git a/modules/profile/files/druid/jvm_prometheus_jmx_exporter.yaml b/modules/profile/files/druid/jvm_prometheus_jmx_exporter.yaml index 73530b2..5532f5e 100644 --- a/modules/profile/files/druid/jvm_prometheus_jmx_exporter.yaml +++ b/modules/profile/files/druid/jvm_prometheus_jmx_exporter.yaml @@ -1,3 +1,9 @@ --- lowercaseOutputLabelNames: true -lowercaseOutputName: false \ No newline at end of file +lowercaseOutputName: false +whitelistObjectNames: + - 'java.lang:type=ClassLoading' + - 'java.lang:type=Compilation' + - 'java.lang:type=GarbageCollector*' + - 'java.lang:type=Memory' + - 'java.lang:type=MemoryPool*' \ No newline at end of file -- To view, visit https://gerrit.wikimedia.org/r/399360 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I30843d99467b152b0aa99f53ad0aa3c382055473 Gerrit-PatchSet: 3 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey Gerrit-Reviewer: Elukey Gerrit-Reviewer: Filippo Giunchedi Gerrit-Reviewer: Joal Gerrit-Reviewer: jenkins-bot <> ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: profile::druid::monitoring: restrict jmx mbeans to query
Elukey has uploaded a new change for review. ( https://gerrit.wikimedia.org/r/399360 ) Change subject: profile::druid::monitoring: restrict jmx mbeans to query .. profile::druid::monitoring: restrict jmx mbeans to query Bug: T183273 Change-Id: I30843d99467b152b0aa99f53ad0aa3c382055473 --- M modules/profile/files/druid/jvm_prometheus_jmx_exporter.yaml 1 file changed, 7 insertions(+), 1 deletion(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/60/399360/1 diff --git a/modules/profile/files/druid/jvm_prometheus_jmx_exporter.yaml b/modules/profile/files/druid/jvm_prometheus_jmx_exporter.yaml index 73530b2..60b6cd7 100644 --- a/modules/profile/files/druid/jvm_prometheus_jmx_exporter.yaml +++ b/modules/profile/files/druid/jvm_prometheus_jmx_exporter.yaml @@ -1,3 +1,9 @@ --- lowercaseOutputLabelNames: true -lowercaseOutputName: false \ No newline at end of file +lowercaseOutputName: false +whitelistObjectNames:: + - 'java.lang:type=ClassLoading' + - 'java.lang:type=Compilation' + - 'java.lang:type=GarbageCollector*' + - 'java.lang:type=Memory' + - 'java.lang:type=MemoryPool*' \ No newline at end of file -- To view, visit https://gerrit.wikimedia.org/r/399360 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I30843d99467b152b0aa99f53ad0aa3c382055473 Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Revert "role::druid::analytics: lower down all the Xms setti...
Elukey has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/399231 ) Change subject: Revert "role::druid::analytics: lower down all the Xms settings" .. Revert "role::druid::analytics: lower down all the Xms settings" This reverts commit 3a2c512f618ec019db4e20e193da0ab49de17c33. Change-Id: I8795cf9819a76daa85cbe66bb1b1f57db3502274 --- M hieradata/role/common/druid/analytics/worker.yaml 1 file changed, 4 insertions(+), 4 deletions(-) Approvals: Elukey: Looks good to me, approved jenkins-bot: Verified diff --git a/hieradata/role/common/druid/analytics/worker.yaml b/hieradata/role/common/druid/analytics/worker.yaml index 1b06e03..c6e5849 100644 --- a/hieradata/role/common/druid/analytics/worker.yaml +++ b/hieradata/role/common/druid/analytics/worker.yaml @@ -79,7 +79,7 @@ # TBD: Perhaps we should also try using memcached? druid.cache.sizeInBytes: 2147483648 profile::druid::broker::env: - DRUID_HEAP_OPTS: "-Xmx25g -Xms5g" + DRUID_HEAP_OPTS: "-Xmx25g -Xms25g" DRUID_EXTRA_JVM_OPTS: "-XX:NewSize=6g -XX:MaxNewSize=6g -XX:MaxDirectMemorySize=64g -XX:+UseConcMarkSweepGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps" @@ -91,7 +91,7 @@ druid.emitter.composing.emitters: ["logging","http"] druid.emitter.http.recipientBaseUrl: "http://localhost:8000/"; profile::druid::coordinator::env: - DRUID_HEAP_OPTS: "-Xmx10g -Xms5g" + DRUID_HEAP_OPTS: "-Xmx10g -Xms10g" DRUID_EXTRA_JVM_OPTS: "-XX:NewSize=512m -XX:MaxNewSize=512m -XX:+UseG1GC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps" @@ -112,7 +112,7 @@ druid.historical.cache.populateCache: true druid.monitoring.monitors: ["io.druid.server.metrics.HistoricalMetricsMonitor"] profile::druid::historical::env: - DRUID_HEAP_OPTS: "-Xmx12g -Xms5g" + DRUID_HEAP_OPTS: "-Xmx12g -Xms12g" DRUID_EXTRA_JVM_OPTS: "-XX:NewSize=6g -XX:MaxNewSize=6g -XX:MaxDirectMemorySize=32g -XX:+UseConcMarkSweepGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps" @@ -142,5 +142,5 @@ druid.indexer.runner.type: remote druid.indexer.storage.type: metadata profile::druid::overlord::env: - DRUID_HEAP_OPTS: "-Xmx4g -Xms2g" + DRUID_HEAP_OPTS: "-Xmx4g -Xms4g" DRUID_EXTRA_JVM_OPTS: "-XX:NewSize=256m -XX:MaxNewSize=256m -XX:+UseConcMarkSweepGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps" -- To view, visit https://gerrit.wikimedia.org/r/399231 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I8795cf9819a76daa85cbe66bb1b1f57db3502274 Gerrit-PatchSet: 2 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey Gerrit-Reviewer: Elukey Gerrit-Reviewer: jenkins-bot <> ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Revert "role::druid::analytics: lower down all the Xms setti...
Hello jenkins-bot, I'd like you to do a code review. Please visit https://gerrit.wikimedia.org/r/399231 to review the following change. Change subject: Revert "role::druid::analytics: lower down all the Xms settings" .. Revert "role::druid::analytics: lower down all the Xms settings" This reverts commit 3a2c512f618ec019db4e20e193da0ab49de17c33. Change-Id: I8795cf9819a76daa85cbe66bb1b1f57db3502274 --- M hieradata/role/common/druid/analytics/worker.yaml 1 file changed, 4 insertions(+), 4 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/31/399231/1 diff --git a/hieradata/role/common/druid/analytics/worker.yaml b/hieradata/role/common/druid/analytics/worker.yaml index 1b06e03..c6e5849 100644 --- a/hieradata/role/common/druid/analytics/worker.yaml +++ b/hieradata/role/common/druid/analytics/worker.yaml @@ -79,7 +79,7 @@ # TBD: Perhaps we should also try using memcached? druid.cache.sizeInBytes: 2147483648 profile::druid::broker::env: - DRUID_HEAP_OPTS: "-Xmx25g -Xms5g" + DRUID_HEAP_OPTS: "-Xmx25g -Xms25g" DRUID_EXTRA_JVM_OPTS: "-XX:NewSize=6g -XX:MaxNewSize=6g -XX:MaxDirectMemorySize=64g -XX:+UseConcMarkSweepGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps" @@ -91,7 +91,7 @@ druid.emitter.composing.emitters: ["logging","http"] druid.emitter.http.recipientBaseUrl: "http://localhost:8000/"; profile::druid::coordinator::env: - DRUID_HEAP_OPTS: "-Xmx10g -Xms5g" + DRUID_HEAP_OPTS: "-Xmx10g -Xms10g" DRUID_EXTRA_JVM_OPTS: "-XX:NewSize=512m -XX:MaxNewSize=512m -XX:+UseG1GC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps" @@ -112,7 +112,7 @@ druid.historical.cache.populateCache: true druid.monitoring.monitors: ["io.druid.server.metrics.HistoricalMetricsMonitor"] profile::druid::historical::env: - DRUID_HEAP_OPTS: "-Xmx12g -Xms5g" + DRUID_HEAP_OPTS: "-Xmx12g -Xms12g" DRUID_EXTRA_JVM_OPTS: "-XX:NewSize=6g -XX:MaxNewSize=6g -XX:MaxDirectMemorySize=32g -XX:+UseConcMarkSweepGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps" @@ -142,5 +142,5 @@ druid.indexer.runner.type: remote druid.indexer.storage.type: metadata profile::druid::overlord::env: - DRUID_HEAP_OPTS: "-Xmx4g -Xms2g" + DRUID_HEAP_OPTS: "-Xmx4g -Xms4g" DRUID_EXTRA_JVM_OPTS: "-XX:NewSize=256m -XX:MaxNewSize=256m -XX:+UseConcMarkSweepGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps" -- To view, visit https://gerrit.wikimedia.org/r/399231 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I8795cf9819a76daa85cbe66bb1b1f57db3502274 Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey Gerrit-Reviewer: jenkins-bot <> ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: eventlogging_purging_whitelist.tsv: remove old table
Elukey has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/399153 ) Change subject: eventlogging_purging_whitelist.tsv: remove old table .. eventlogging_purging_whitelist.tsv: remove old table On db1107 the PageContentSaveComplete table is not there anymore, and we get the following error: ERROR: line 576: Some table prefixes in the whitelist do not match any table name retrieved from the database. Please review the following entries of the whitelist: ['PageContentSaveComplete'] Bug: T108850 Change-Id: I29735369dd24f57394a408be97467915297a9905 --- M modules/profile/files/mariadb/misc/eventlogging/eventlogging_purging_whitelist.tsv 1 file changed, 0 insertions(+), 7 deletions(-) Approvals: Mforns: Looks good to me, but someone else must approve Elukey: Verified; Looks good to me, approved diff --git a/modules/profile/files/mariadb/misc/eventlogging/eventlogging_purging_whitelist.tsv b/modules/profile/files/mariadb/misc/eventlogging/eventlogging_purging_whitelist.tsv index 0a00dc9..ec3dd9c 100644 --- a/modules/profile/files/mariadb/misc/eventlogging/eventlogging_purging_whitelist.tsv +++ b/modules/profile/files/mariadb/misc/eventlogging/eventlogging_purging_whitelist.tsv @@ -665,13 +665,6 @@ NavigationTiming isTruncated NavigationTiming webHost NavigationTiming wiki -PageContentSaveCompleteclientValidated -PageContentSaveCompleteevent_isAPI -PageContentSaveCompleteevent_isMobile -PageContentSaveCompleteevent_revisionId -PageContentSaveCompleteisTruncated -PageContentSaveCompletewebHost -PageContentSaveCompletewiki Popups event_sessionToken Popups event_version Popups event_pageIdSource -- To view, visit https://gerrit.wikimedia.org/r/399153 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I29735369dd24f57394a408be97467915297a9905 Gerrit-PatchSet: 4 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey Gerrit-Reviewer: Elukey Gerrit-Reviewer: Mforns Gerrit-Reviewer: jenkins-bot <> ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::druid::analytics::worker: review jvm configurations
Elukey has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/399205 )
Change subject: role::druid::analytics::worker: review jvm configurations
..
role::druid::analytics::worker: review jvm configurations
Change-Id: I29e75c080e98490e0615a7d12d6e40f668eaea4e
---
M hieradata/role/common/druid/analytics/worker.yaml
1 file changed, 11 insertions(+), 10 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/05/399205/1
diff --git a/hieradata/role/common/druid/analytics/worker.yaml
b/hieradata/role/common/druid/analytics/worker.yaml
index 1b06e03..214b213 100644
--- a/hieradata/role/common/druid/analytics/worker.yaml
+++ b/hieradata/role/common/druid/analytics/worker.yaml
@@ -60,7 +60,6 @@
# -- Druid worker service configurations
# --- Druid Broker
-
# Broker gets a special ferm_srange since it is the frontend query interface
to Druid.
profile::druid::broker::monitoring_enabled: true
profile::druid::broker::ferm_srange: '$ANALYTICS_NETWORKS'
@@ -69,7 +68,7 @@
druid.emitter.composing.emitters: ["logging","http"]
druid.emitter.http.recipientBaseUrl: "http://localhost:8000/";
druid.processing.numThreads: 10
- druid.processing.buffer.sizeBytes: 2147483647
+ druid.processing.buffer.sizeBytes: 268435456 # 1024 * 1024 * 256
# Set numMergeBuffers to use v2 groupBy engine
druid.processing.numMergeBuffers: 10
druid.server.http.numThreads: 20
@@ -79,8 +78,8 @@
# TBD: Perhaps we should also try using memcached?
druid.cache.sizeInBytes: 2147483648
profile::druid::broker::env:
- DRUID_HEAP_OPTS: "-Xmx25g -Xms5g"
- DRUID_EXTRA_JVM_OPTS: "-XX:NewSize=6g -XX:MaxNewSize=6g
-XX:MaxDirectMemorySize=64g -XX:+UseConcMarkSweepGC -XX:+PrintGCDetails
-XX:+PrintGCTimeStamps"
+ DRUID_HEAP_OPTS: "-Xmx10g -Xms5g"
+ DRUID_EXTRA_JVM_OPTS: "-XX:NewSize=4g -XX:MaxNewSize=4g
-XX:MaxDirectMemorySize=12g -XX:+UseConcMarkSweepGC -XX:+PrintGCDetails
-XX:+PrintGCTimeStamps"
# --- Druid Coordinator
@@ -91,7 +90,7 @@
druid.emitter.composing.emitters: ["logging","http"]
druid.emitter.http.recipientBaseUrl: "http://localhost:8000/";
profile::druid::coordinator::env:
- DRUID_HEAP_OPTS: "-Xmx10g -Xms5g"
+ DRUID_HEAP_OPTS: "-Xmx4g -Xms2g"
DRUID_EXTRA_JVM_OPTS: "-XX:NewSize=512m -XX:MaxNewSize=512m -XX:+UseG1GC
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps"
@@ -102,18 +101,20 @@
druid.emitter.composing.emitters: ["logging","http"]
druid.emitter.http.recipientBaseUrl: "http://localhost:8000/";
druid.processing.numThreads: 10
- druid.processing.buffer.sizeBytes: 1073741824
+ druid.processing.buffer.sizeBytes: 268435456 # 1024 * 1024 * 256
# Set numMergeBuffers to use v2 groupBy engine
druid.processing.numMergeBuffers: 10
druid.server.http.numThreads: 20
druid.server.maxSize: 2748779069440 # 2.5 TB
druid.segmentCache.locations:
'[{"path":"/var/lib/druid/segment-cache","maxSize"\:2748779069440}]'
- druid.historical.cache.useCache: true
- druid.historical.cache.populateCache: true
+ # For small clusters it is reccomended to only enable caching on brokers
+ # See: http://druid.io/docs/latest/querying/caching.html
+ druid.historical.cache.useCache: false
+ druid.historical.cache.populateCache: false
druid.monitoring.monitors:
["io.druid.server.metrics.HistoricalMetricsMonitor"]
profile::druid::historical::env:
- DRUID_HEAP_OPTS: "-Xmx12g -Xms5g"
- DRUID_EXTRA_JVM_OPTS: "-XX:NewSize=6g -XX:MaxNewSize=6g
-XX:MaxDirectMemorySize=32g -XX:+UseConcMarkSweepGC -XX:+PrintGCDetails
-XX:+PrintGCTimeStamps"
+ DRUID_HEAP_OPTS: "-Xmx8g -Xms4g"
+ DRUID_EXTRA_JVM_OPTS: "-XX:NewSize=2g -XX:MaxNewSize=2g
-XX:MaxDirectMemorySize=10g -XX:+UseConcMarkSweepGC -XX:+PrintGCDetails
-XX:+PrintGCTimeStamps"
# --- Druid MiddleManager
--
To view, visit https://gerrit.wikimedia.org/r/399205
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I29e75c080e98490e0615a7d12d6e40f668eaea4e
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::druid::analytics: lower down all the Xms settings
Elukey has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/399195 ) Change subject: role::druid::analytics: lower down all the Xms settings .. role::druid::analytics: lower down all the Xms settings We observed via the Prometheus Druid dashboard that the heap size of all the Druid daemons is way below the Xms (min heap size) setting. This precludes the possibility to use page cache for the OS, so lowering down those values should improve perfomances. Druid segments are in fact memory mapped. Change-Id: Iaa4032f3b4c0fc43bebdc036be5d7b46b0e646ba --- M hieradata/role/common/druid/analytics/worker.yaml 1 file changed, 4 insertions(+), 4 deletions(-) Approvals: Elukey: Looks good to me, approved jenkins-bot: Verified diff --git a/hieradata/role/common/druid/analytics/worker.yaml b/hieradata/role/common/druid/analytics/worker.yaml index c6e5849..1b06e03 100644 --- a/hieradata/role/common/druid/analytics/worker.yaml +++ b/hieradata/role/common/druid/analytics/worker.yaml @@ -79,7 +79,7 @@ # TBD: Perhaps we should also try using memcached? druid.cache.sizeInBytes: 2147483648 profile::druid::broker::env: - DRUID_HEAP_OPTS: "-Xmx25g -Xms25g" + DRUID_HEAP_OPTS: "-Xmx25g -Xms5g" DRUID_EXTRA_JVM_OPTS: "-XX:NewSize=6g -XX:MaxNewSize=6g -XX:MaxDirectMemorySize=64g -XX:+UseConcMarkSweepGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps" @@ -91,7 +91,7 @@ druid.emitter.composing.emitters: ["logging","http"] druid.emitter.http.recipientBaseUrl: "http://localhost:8000/"; profile::druid::coordinator::env: - DRUID_HEAP_OPTS: "-Xmx10g -Xms10g" + DRUID_HEAP_OPTS: "-Xmx10g -Xms5g" DRUID_EXTRA_JVM_OPTS: "-XX:NewSize=512m -XX:MaxNewSize=512m -XX:+UseG1GC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps" @@ -112,7 +112,7 @@ druid.historical.cache.populateCache: true druid.monitoring.monitors: ["io.druid.server.metrics.HistoricalMetricsMonitor"] profile::druid::historical::env: - DRUID_HEAP_OPTS: "-Xmx12g -Xms12g" + DRUID_HEAP_OPTS: "-Xmx12g -Xms5g" DRUID_EXTRA_JVM_OPTS: "-XX:NewSize=6g -XX:MaxNewSize=6g -XX:MaxDirectMemorySize=32g -XX:+UseConcMarkSweepGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps" @@ -142,5 +142,5 @@ druid.indexer.runner.type: remote druid.indexer.storage.type: metadata profile::druid::overlord::env: - DRUID_HEAP_OPTS: "-Xmx4g -Xms4g" + DRUID_HEAP_OPTS: "-Xmx4g -Xms2g" DRUID_EXTRA_JVM_OPTS: "-XX:NewSize=256m -XX:MaxNewSize=256m -XX:+UseConcMarkSweepGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps" -- To view, visit https://gerrit.wikimedia.org/r/399195 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Iaa4032f3b4c0fc43bebdc036be5d7b46b0e646ba Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey Gerrit-Reviewer: Elukey Gerrit-Reviewer: jenkins-bot <> ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::druid::analytics: lower down all the Xms settings
Elukey has uploaded a new change for review. ( https://gerrit.wikimedia.org/r/399195 ) Change subject: role::druid::analytics: lower down all the Xms settings .. role::druid::analytics: lower down all the Xms settings We observed via the Prometheus Druid dashboard that the heap size of all the Druid daemons is way below the Xms (min heap size) setting. This precludes the possibility to use page cache for the OS, so lowering down those values should improve perfomances. Druid segments are in fact memory mapped. Change-Id: Iaa4032f3b4c0fc43bebdc036be5d7b46b0e646ba --- M hieradata/role/common/druid/analytics/worker.yaml 1 file changed, 4 insertions(+), 4 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/95/399195/1 diff --git a/hieradata/role/common/druid/analytics/worker.yaml b/hieradata/role/common/druid/analytics/worker.yaml index c6e5849..1b06e03 100644 --- a/hieradata/role/common/druid/analytics/worker.yaml +++ b/hieradata/role/common/druid/analytics/worker.yaml @@ -79,7 +79,7 @@ # TBD: Perhaps we should also try using memcached? druid.cache.sizeInBytes: 2147483648 profile::druid::broker::env: - DRUID_HEAP_OPTS: "-Xmx25g -Xms25g" + DRUID_HEAP_OPTS: "-Xmx25g -Xms5g" DRUID_EXTRA_JVM_OPTS: "-XX:NewSize=6g -XX:MaxNewSize=6g -XX:MaxDirectMemorySize=64g -XX:+UseConcMarkSweepGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps" @@ -91,7 +91,7 @@ druid.emitter.composing.emitters: ["logging","http"] druid.emitter.http.recipientBaseUrl: "http://localhost:8000/"; profile::druid::coordinator::env: - DRUID_HEAP_OPTS: "-Xmx10g -Xms10g" + DRUID_HEAP_OPTS: "-Xmx10g -Xms5g" DRUID_EXTRA_JVM_OPTS: "-XX:NewSize=512m -XX:MaxNewSize=512m -XX:+UseG1GC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps" @@ -112,7 +112,7 @@ druid.historical.cache.populateCache: true druid.monitoring.monitors: ["io.druid.server.metrics.HistoricalMetricsMonitor"] profile::druid::historical::env: - DRUID_HEAP_OPTS: "-Xmx12g -Xms12g" + DRUID_HEAP_OPTS: "-Xmx12g -Xms5g" DRUID_EXTRA_JVM_OPTS: "-XX:NewSize=6g -XX:MaxNewSize=6g -XX:MaxDirectMemorySize=32g -XX:+UseConcMarkSweepGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps" @@ -142,5 +142,5 @@ druid.indexer.runner.type: remote druid.indexer.storage.type: metadata profile::druid::overlord::env: - DRUID_HEAP_OPTS: "-Xmx4g -Xms4g" + DRUID_HEAP_OPTS: "-Xmx4g -Xms2g" DRUID_EXTRA_JVM_OPTS: "-XX:NewSize=256m -XX:MaxNewSize=256m -XX:+UseConcMarkSweepGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps" -- To view, visit https://gerrit.wikimedia.org/r/399195 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Iaa4032f3b4c0fc43bebdc036be5d7b46b0e646ba Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: eventlogging_purging_whitelist.tsv: remove old table
Elukey has uploaded a new change for review. ( https://gerrit.wikimedia.org/r/399153 ) Change subject: eventlogging_purging_whitelist.tsv: remove old table .. eventlogging_purging_whitelist.tsv: remove old table Change-Id: I29735369dd24f57394a408be97467915297a9905 --- M modules/profile/files/mariadb/misc/eventlogging/eventlogging_purging_whitelist.tsv 1 file changed, 0 insertions(+), 7 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/53/399153/1 diff --git a/modules/profile/files/mariadb/misc/eventlogging/eventlogging_purging_whitelist.tsv b/modules/profile/files/mariadb/misc/eventlogging/eventlogging_purging_whitelist.tsv index 0a00dc9..ec3dd9c 100644 --- a/modules/profile/files/mariadb/misc/eventlogging/eventlogging_purging_whitelist.tsv +++ b/modules/profile/files/mariadb/misc/eventlogging/eventlogging_purging_whitelist.tsv @@ -665,13 +665,6 @@ NavigationTiming isTruncated NavigationTiming webHost NavigationTiming wiki -PageContentSaveCompleteclientValidated -PageContentSaveCompleteevent_isAPI -PageContentSaveCompleteevent_isMobile -PageContentSaveCompleteevent_revisionId -PageContentSaveCompleteisTruncated -PageContentSaveCompletewebHost -PageContentSaveCompletewiki Popups event_sessionToken Popups event_version Popups event_pageIdSource -- To view, visit https://gerrit.wikimedia.org/r/399153 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I29735369dd24f57394a408be97467915297a9905 Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: profile::mariadb::misc::eventlogging: fix group/user depende...
Elukey has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/399149 )
Change subject: profile::mariadb::misc::eventlogging: fix group/user
dependencies
..
profile::mariadb::misc::eventlogging: fix group/user dependencies
Bug: T108850
Change-Id: I4c474174520b2ed87a357354dc26d24b7735ffa8
---
M modules/profile/manifests/mariadb/misc/eventlogging/replication.pp
M modules/profile/manifests/mariadb/misc/eventlogging/sanitization.pp
2 files changed, 14 insertions(+), 5 deletions(-)
Approvals:
Elukey: Looks good to me, approved
jenkins-bot: Verified
diff --git a/modules/profile/manifests/mariadb/misc/eventlogging/replication.pp
b/modules/profile/manifests/mariadb/misc/eventlogging/replication.pp
index 31b177b..655ffd4 100644
--- a/modules/profile/manifests/mariadb/misc/eventlogging/replication.pp
+++ b/modules/profile/manifests/mariadb/misc/eventlogging/replication.pp
@@ -24,14 +24,16 @@
$slave_host = 'localhost'
$database= 'log'
-group { 'eventlog':
-ensure => 'present',
-system => true,
+if !defined(Group['eventlog']) {
+group { 'eventlog':
+ensure => 'present',
+system => true,
+}
}
require_package('python3-pymysql')
-if ! defined(File['/etc/eventlogging']) {
+if !defined(File['/etc/eventlogging']) {
file { '/etc/eventlogging':
ensure => 'directory',
owner => 'root',
@@ -40,7 +42,7 @@
}
}
-if ! defined(File['/var/log/eventlogging']) {
+if !defined(File['/var/log/eventlogging']) {
file { '/var/log/eventlogging':
ensure => 'directory',
owner => 'root',
diff --git
a/modules/profile/manifests/mariadb/misc/eventlogging/sanitization.pp
b/modules/profile/manifests/mariadb/misc/eventlogging/sanitization.pp
index 2392339..d4f1c43 100644
--- a/modules/profile/manifests/mariadb/misc/eventlogging/sanitization.pp
+++ b/modules/profile/manifests/mariadb/misc/eventlogging/sanitization.pp
@@ -5,6 +5,13 @@
#
class profile::mariadb::misc::eventlogging::sanitization {
+if !defined(Group['eventlog']) {
+group { 'eventlog':
+ensure => 'present',
+system => true,
+}
+}
+
user { 'eventlogcleaner':
gid=> 'eventlog',
shell => '/bin/false',
--
To view, visit https://gerrit.wikimedia.org/r/399149
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I4c474174520b2ed87a357354dc26d24b7735ffa8
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
Gerrit-Reviewer: Elukey
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: profile::mariadb::misc::eventlogging: fix group/user depende...
Elukey has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/399149 )
Change subject: profile::mariadb::misc::eventlogging: fix group/user
dependencies
..
profile::mariadb::misc::eventlogging: fix group/user dependencies
Bug: T108850
Change-Id: I4c474174520b2ed87a357354dc26d24b7735ffa8
---
M modules/profile/manifests/mariadb/misc/eventlogging/replication.pp
M modules/profile/manifests/mariadb/misc/eventlogging/sanitization.pp
2 files changed, 14 insertions(+), 5 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/49/399149/1
diff --git a/modules/profile/manifests/mariadb/misc/eventlogging/replication.pp
b/modules/profile/manifests/mariadb/misc/eventlogging/replication.pp
index 31b177b..655ffd4 100644
--- a/modules/profile/manifests/mariadb/misc/eventlogging/replication.pp
+++ b/modules/profile/manifests/mariadb/misc/eventlogging/replication.pp
@@ -24,14 +24,16 @@
$slave_host = 'localhost'
$database= 'log'
-group { 'eventlog':
-ensure => 'present',
-system => true,
+if !defined(Group['eventlog']) {
+group { 'eventlog':
+ensure => 'present',
+system => true,
+}
}
require_package('python3-pymysql')
-if ! defined(File['/etc/eventlogging']) {
+if !defined(File['/etc/eventlogging']) {
file { '/etc/eventlogging':
ensure => 'directory',
owner => 'root',
@@ -40,7 +42,7 @@
}
}
-if ! defined(File['/var/log/eventlogging']) {
+if !defined(File['/var/log/eventlogging']) {
file { '/var/log/eventlogging':
ensure => 'directory',
owner => 'root',
diff --git
a/modules/profile/manifests/mariadb/misc/eventlogging/sanitization.pp
b/modules/profile/manifests/mariadb/misc/eventlogging/sanitization.pp
index 2392339..d4f1c43 100644
--- a/modules/profile/manifests/mariadb/misc/eventlogging/sanitization.pp
+++ b/modules/profile/manifests/mariadb/misc/eventlogging/sanitization.pp
@@ -5,6 +5,13 @@
#
class profile::mariadb::misc::eventlogging::sanitization {
+if !defined(Group['eventlog']) {
+group { 'eventlog':
+ensure => 'present',
+system => true,
+}
+}
+
user { 'eventlogcleaner':
gid=> 'eventlog',
shell => '/bin/false',
--
To view, visit https://gerrit.wikimedia.org/r/399149
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I4c474174520b2ed87a357354dc26d24b7735ffa8
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: profile::mariadb::misc::el::master: apply data sanitization ...
Elukey has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/398869 )
Change subject: profile::mariadb::misc::el::master: apply data sanitization
policies
..
profile::mariadb::misc::el::master: apply data sanitization policies
Bug: T108850
Change-Id: I1f4d273e1e1d537fdbd978cb69ea804ee9ea8802
---
M modules/profile/manifests/mariadb/misc/eventlogging/replication.pp
A modules/profile/manifests/mariadb/misc/eventlogging/sanitization.pp
M modules/role/manifests/mariadb/misc/eventlogging/master.pp
M modules/role/manifests/mariadb/misc/eventlogging/replica.pp
4 files changed, 123 insertions(+), 92 deletions(-)
Approvals:
Elukey: Looks good to me, approved
jenkins-bot: Verified
diff --git a/modules/profile/manifests/mariadb/misc/eventlogging/replication.pp
b/modules/profile/manifests/mariadb/misc/eventlogging/replication.pp
index 76ea7cb..31b177b 100644
--- a/modules/profile/manifests/mariadb/misc/eventlogging/replication.pp
+++ b/modules/profile/manifests/mariadb/misc/eventlogging/replication.pp
@@ -29,48 +29,24 @@
system => true,
}
-user { 'eventlogcleaner':
-gid=> 'eventlog',
-shell => '/bin/false',
-home => '/nonexistent',
-comment=> 'EventLogging cleaner user',
-system => true,
-managehome => false,
-require=> Group['eventlog'],
-}
-
require_package('python3-pymysql')
-file { '/usr/local/bin/eventlogging_cleaner':
-ensure => present,
-owner => 'eventlogcleaner',
-group => 'eventlog',
-mode=> '0550',
-source =>
'puppet:///modules/profile/mariadb/misc/eventlogging/eventlogging_cleaner.py',
-require => Package['python3-pymysql'],
+if ! defined(File['/etc/eventlogging']) {
+file { '/etc/eventlogging':
+ensure => 'directory',
+owner => 'root',
+group => 'root',
+mode => '0755',
+}
}
-file { '/etc/eventlogging':
-ensure => 'directory',
-owner => 'root',
-group => 'root',
-mode => '0755',
-}
-
-file { '/var/log/eventlogging':
-ensure => 'directory',
-owner => 'root',
-group => 'eventlog',
-mode => '0775',
-}
-
-file { '/etc/eventlogging/whitelist.tsv':
-ensure => 'present',
-owner => 'root',
-group => 'root',
-mode=> '0444',
-source =>
'puppet:///modules/profile/mariadb/misc/eventlogging/eventlogging_purging_whitelist.tsv',
-require => File['/etc/eventlogging'],
+if ! defined(File['/var/log/eventlogging']) {
+file { '/var/log/eventlogging':
+ensure => 'directory',
+owner => 'root',
+group => 'eventlog',
+mode => '0775',
+}
}
file { '/usr/local/bin/eventlogging_sync.sh':
@@ -81,9 +57,9 @@
source =>
'puppet:///modules/profile/mariadb/misc/eventlogging/eventlogging_sync.sh',
}
-logrotate::rule { 'eventlogging':
+logrotate::rule { 'eventlogging-sync':
ensure=> present,
-file_glob => '/var/log/eventlogging/eventlogging_*.log',
+file_glob => '/var/log/eventlogging/eventlogging_sync.log',
frequency => 'daily',
copy_truncate => true,
compress => true,
@@ -93,59 +69,16 @@
su=> 'root eventlog',
}
-# Custom init scripts only that should be deprecated as soon as
-# the profile will run on Debian OS only.
-# The init script manages stdout/stderr to two separate files,
-# meanwhile the systemd unit used below will use a rsyslog dedicated
config.
-if os_version('ubuntu >= trusty') {
-file { '/etc/init.d/eventlogging_sync':
-owner => 'root',
-group => 'root',
-mode=> '0555',
-content =>
template('profile/initscripts/mariadb/misc/eventlogging/eventlogging_sync.sysvinit.erb'),
-require => File['/usr/local/bin/eventlogging_sync.sh'],
-notify => Service['eventlogging_sync'],
-}
+rsyslog::conf { 'eventlogging_sync':
+source =>
'puppet:///modules/profile/mariadb/misc/eventlogging/eventlogging_sync_rsyslog.conf',
+priority => 20,
+}
-service { 'eventlogging_sync':
-ensure => running,
-enable => true,
-}
-} else {
-rsyslog::conf { 'eventlogging_sync':
-source =>
'puppet:///modules/profile/mariadb/misc/eventlogging/eventlogging_sync_rsyslog.conf',
-priority => 20,
-}
-
-$eventlogging_sync_uid = 'root'
-$eventlogging_sync_gid = 'root'
-base::service_unit { 'eventlogging_sync':
-ensure => present,
-systemd =>
systemd_template('mariadb/misc/event
[MediaWiki-commits] [Gerrit] operations/puppet[production]: profile::mariadb::misc::el::master: apply data sanitization ...
Elukey has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/398869 )
Change subject: profile::mariadb::misc::el::master: apply data sanitization
policies
..
profile::mariadb::misc::el::master: apply data sanitization policies
Bug: T108850
Change-Id: I1f4d273e1e1d537fdbd978cb69ea804ee9ea8802
---
M modules/profile/manifests/mariadb/misc/eventlogging/replication.pp
A modules/profile/manifests/mariadb/misc/eventlogging/sanitization.pp
M modules/role/manifests/mariadb/misc/eventlogging/master.pp
M modules/role/manifests/mariadb/misc/eventlogging/replica.pp
4 files changed, 123 insertions(+), 73 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/69/398869/1
diff --git a/modules/profile/manifests/mariadb/misc/eventlogging/replication.pp
b/modules/profile/manifests/mariadb/misc/eventlogging/replication.pp
index 76ea7cb..fdb5531 100644
--- a/modules/profile/manifests/mariadb/misc/eventlogging/replication.pp
+++ b/modules/profile/manifests/mariadb/misc/eventlogging/replication.pp
@@ -50,27 +50,22 @@
require => Package['python3-pymysql'],
}
-file { '/etc/eventlogging':
-ensure => 'directory',
-owner => 'root',
-group => 'root',
-mode => '0755',
+if ! defined(File['/etc/eventlogging']) {
+file { '/etc/eventlogging':
+ensure => 'directory',
+owner => 'root',
+group => 'root',
+mode => '0755',
+}
}
-file { '/var/log/eventlogging':
-ensure => 'directory',
-owner => 'root',
-group => 'eventlog',
-mode => '0775',
-}
-
-file { '/etc/eventlogging/whitelist.tsv':
-ensure => 'present',
-owner => 'root',
-group => 'root',
-mode=> '0444',
-source =>
'puppet:///modules/profile/mariadb/misc/eventlogging/eventlogging_purging_whitelist.tsv',
-require => File['/etc/eventlogging'],
+if ! defined(File['/var/log/eventlogging']) {
+file { '/var/log/eventlogging':
+ensure => 'directory',
+owner => 'root',
+group => 'eventlog',
+mode => '0775',
+}
}
file { '/usr/local/bin/eventlogging_sync.sh':
@@ -81,9 +76,9 @@
source =>
'puppet:///modules/profile/mariadb/misc/eventlogging/eventlogging_sync.sh',
}
-logrotate::rule { 'eventlogging':
+logrotate::rule { 'eventlogging-sync':
ensure=> present,
-file_glob => '/var/log/eventlogging/eventlogging_*.log',
+file_glob => '/var/log/eventlogging/eventlogging_sync.log',
frequency => 'daily',
copy_truncate => true,
compress => true,
@@ -93,59 +88,16 @@
su=> 'root eventlog',
}
-# Custom init scripts only that should be deprecated as soon as
-# the profile will run on Debian OS only.
-# The init script manages stdout/stderr to two separate files,
-# meanwhile the systemd unit used below will use a rsyslog dedicated
config.
-if os_version('ubuntu >= trusty') {
-file { '/etc/init.d/eventlogging_sync':
-owner => 'root',
-group => 'root',
-mode=> '0555',
-content =>
template('profile/initscripts/mariadb/misc/eventlogging/eventlogging_sync.sysvinit.erb'),
-require => File['/usr/local/bin/eventlogging_sync.sh'],
-notify => Service['eventlogging_sync'],
-}
+rsyslog::conf { 'eventlogging_sync':
+source =>
'puppet:///modules/profile/mariadb/misc/eventlogging/eventlogging_sync_rsyslog.conf',
+priority => 20,
+}
-service { 'eventlogging_sync':
-ensure => running,
-enable => true,
-}
-} else {
-rsyslog::conf { 'eventlogging_sync':
-source =>
'puppet:///modules/profile/mariadb/misc/eventlogging/eventlogging_sync_rsyslog.conf',
-priority => 20,
-}
-
-$eventlogging_sync_uid = 'root'
-$eventlogging_sync_gid = 'root'
-base::service_unit { 'eventlogging_sync':
-ensure => present,
-systemd =>
systemd_template('mariadb/misc/eventlogging/eventlogging_sync'),
-}
-
-# Sanitization of data in the log database via a custom script
-# The eventlogging_cleaner script uses the --start-ts-file file option,
-# that forces it to look for a file containing a timestamp in the
format
-# %Y%m%d%H%M%S. If the file is not existent, the script will fail
gracefully
-# without doing any action to the db. This is useful to avoid gaps in
-# records sanitized if the script fails and does not commit a new
timestamp.
-$eventlogging_cleaner_command = '/usr/local/bin/eventlogging_cleaner
--whitelist /etc/eventlogging/whitelist.ts
[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::analytics_cluster::refinery: follow up on logrotate rules
Elukey has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/398849 ) Change subject: role::analytics_cluster::refinery: follow up on logrotate rules .. role::analytics_cluster::refinery: follow up on logrotate rules Change-Id: Id19854898ea3c33e7115879742a40e9cafa809f5 --- M modules/role/files/analytics_cluster/refinery-logrotate.conf 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Elukey: Verified; Looks good to me, approved diff --git a/modules/role/files/analytics_cluster/refinery-logrotate.conf b/modules/role/files/analytics_cluster/refinery-logrotate.conf index fd20bb1..d89d299 100644 --- a/modules/role/files/analytics_cluster/refinery-logrotate.conf +++ b/modules/role/files/analytics_cluster/refinery-logrotate.conf @@ -6,5 +6,5 @@ missingok notifempty nocreate - su hdfs <%= @log_dir_group %> + su hdfs analytics-admins } -- To view, visit https://gerrit.wikimedia.org/r/398849 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Id19854898ea3c33e7115879742a40e9cafa809f5 Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey Gerrit-Reviewer: Elukey ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::analytics_cluster::refinery: follow up on logrotate rules
Elukey has uploaded a new change for review. ( https://gerrit.wikimedia.org/r/398849 ) Change subject: role::analytics_cluster::refinery: follow up on logrotate rules .. role::analytics_cluster::refinery: follow up on logrotate rules Change-Id: Id19854898ea3c33e7115879742a40e9cafa809f5 --- M modules/role/files/analytics_cluster/refinery-logrotate.conf 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/49/398849/1 diff --git a/modules/role/files/analytics_cluster/refinery-logrotate.conf b/modules/role/files/analytics_cluster/refinery-logrotate.conf index fd20bb1..d89d299 100644 --- a/modules/role/files/analytics_cluster/refinery-logrotate.conf +++ b/modules/role/files/analytics_cluster/refinery-logrotate.conf @@ -6,5 +6,5 @@ missingok notifempty nocreate - su hdfs <%= @log_dir_group %> + su hdfs analytics-admins } -- To view, visit https://gerrit.wikimedia.org/r/398849 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Id19854898ea3c33e7115879742a40e9cafa809f5 Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::analytics_cluster::refinery: fix logrotate config
Elukey has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/398848 ) Change subject: role::analytics_cluster::refinery: fix logrotate config .. role::analytics_cluster::refinery: fix logrotate config Change-Id: I2c94fafbae8ba74a23b902d3cb745d2410dbeb4a --- M modules/role/files/analytics_cluster/refinery-logrotate.conf 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Elukey: Looks good to me, approved jenkins-bot: Verified diff --git a/modules/role/files/analytics_cluster/refinery-logrotate.conf b/modules/role/files/analytics_cluster/refinery-logrotate.conf index 8d36009..fd20bb1 100644 --- a/modules/role/files/analytics_cluster/refinery-logrotate.conf +++ b/modules/role/files/analytics_cluster/refinery-logrotate.conf @@ -6,5 +6,5 @@ missingok notifempty nocreate - su root hdfs + su hdfs <%= @log_dir_group %> } -- To view, visit https://gerrit.wikimedia.org/r/398848 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I2c94fafbae8ba74a23b902d3cb745d2410dbeb4a Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey Gerrit-Reviewer: Elukey Gerrit-Reviewer: Giuseppe Lavagetto Gerrit-Reviewer: jenkins-bot <> ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::analytics_cluster::refinery: fix logrotate config
Elukey has uploaded a new change for review. ( https://gerrit.wikimedia.org/r/398848 ) Change subject: role::analytics_cluster::refinery: fix logrotate config .. role::analytics_cluster::refinery: fix logrotate config Change-Id: I2c94fafbae8ba74a23b902d3cb745d2410dbeb4a --- M modules/role/files/analytics_cluster/refinery-logrotate.conf 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/48/398848/1 diff --git a/modules/role/files/analytics_cluster/refinery-logrotate.conf b/modules/role/files/analytics_cluster/refinery-logrotate.conf index 8d36009..fd20bb1 100644 --- a/modules/role/files/analytics_cluster/refinery-logrotate.conf +++ b/modules/role/files/analytics_cluster/refinery-logrotate.conf @@ -6,5 +6,5 @@ missingok notifempty nocreate - su root hdfs + su hdfs <%= @log_dir_group %> } -- To view, visit https://gerrit.wikimedia.org/r/398848 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I2c94fafbae8ba74a23b902d3cb745d2410dbeb4a Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Set numa=off to tftpboot jessie's ttyS1-115200 config
Elukey has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/398807 ) Change subject: Set numa=off to tftpboot jessie's ttyS1-115200 config .. Set numa=off to tftpboot jessie's ttyS1-115200 config This workaround is temporarily needed to bypass a bug in the Debian's 8.10 netboot image that triggers a kernel panic while loading d-i. Bug: T182702 Change-Id: I83b31fb8fd9b815ef2048631fa5faabdaf2efb7e --- M modules/install_server/files/tftpboot/jessie-installer/pxelinux.cfg/ttyS1-115200 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Muehlenhoff: Looks good to me, but someone else must approve Elukey: Looks good to me, approved jenkins-bot: Verified Filippo Giunchedi: Looks good to me, but someone else must approve diff --git a/modules/install_server/files/tftpboot/jessie-installer/pxelinux.cfg/ttyS1-115200 b/modules/install_server/files/tftpboot/jessie-installer/pxelinux.cfg/ttyS1-115200 index a808bc5..6f3dfee 100644 --- a/modules/install_server/files/tftpboot/jessie-installer/pxelinux.cfg/ttyS1-115200 +++ b/modules/install_server/files/tftpboot/jessie-installer/pxelinux.cfg/ttyS1-115200 @@ -9,4 +9,4 @@ LABEL server kernel debian-installer/amd64/linux - append initrd=debian-installer/amd64/initrd.gz vga=normal auto-install/enable=true preseed/url=http://apt.wikimedia.org/autoinstall/preseed.cfg DEBCONF_DEBUG=5 netcfg/choose_interface=auto netcfg/get_hostname=unassigned netcfg/get_domain=unassigned netcfg/dhcp_timeout=60 --- ixgbe.allow_unsupported_sfp=1 console=ttyS1,115200n8 + append initrd=debian-installer/amd64/initrd.gz vga=normal numa=off auto-install/enable=true preseed/url=http://apt.wikimedia.org/autoinstall/preseed.cfg DEBCONF_DEBUG=5 netcfg/choose_interface=auto netcfg/get_hostname=unassigned netcfg/get_domain=unassigned netcfg/dhcp_timeout=60 --- ixgbe.allow_unsupported_sfp=1 console=ttyS1,115200n8 -- To view, visit https://gerrit.wikimedia.org/r/398807 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I83b31fb8fd9b815ef2048631fa5faabdaf2efb7e Gerrit-PatchSet: 2 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey Gerrit-Reviewer: Elukey Gerrit-Reviewer: Filippo Giunchedi Gerrit-Reviewer: Gehel Gerrit-Reviewer: Muehlenhoff Gerrit-Reviewer: jenkins-bot <> ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Add mw13[29-37] to site.pp and conftool
Elukey has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/397749 )
Change subject: Add mw13[29-37] to site.pp and conftool
..
Add mw13[29-37] to site.pp and conftool
mw1329->1333 will become ROW C appservers.
mw1334->1337 will become ROW C jobrunners.
Bug: T165519
Change-Id: Id9539e819a46c3b7d22573384f361a8f1fea4188
---
M conftool-data/node/eqiad.yaml
M manifests/site.pp
2 files changed, 17 insertions(+), 1 deletion(-)
Approvals:
Giuseppe Lavagetto: Looks good to me, but someone else must approve
Muehlenhoff: Looks good to me, but someone else must approve
Elukey: Verified; Looks good to me, approved
diff --git a/conftool-data/node/eqiad.yaml b/conftool-data/node/eqiad.yaml
index 1fc4675..9357ded 100644
--- a/conftool-data/node/eqiad.yaml
+++ b/conftool-data/node/eqiad.yaml
@@ -20,6 +20,10 @@
mw1309.eqiad.wmnet: [apache2,nginx]
mw1310.eqiad.wmnet: [apache2,nginx]
mw1311.eqiad.wmnet: [apache2,nginx]
+mw1334.eqiad.wmnet: [apache2,nginx]
+mw1335.eqiad.wmnet: [apache2,nginx]
+mw1336.eqiad.wmnet: [apache2,nginx]
+mw1337.eqiad.wmnet: [apache2,nginx]
api_appserver:
mw1189.eqiad.wmnet: [apache2,nginx]
mw1190.eqiad.wmnet: [apache2,nginx]
@@ -143,6 +147,11 @@
mw1326.eqiad.wmnet: [apache2,nginx]
mw1327.eqiad.wmnet: [apache2,nginx]
mw1328.eqiad.wmnet: [apache2,nginx]
+mw1329.eqiad.wmnet: [apache2,nginx]
+mw1330.eqiad.wmnet: [apache2,nginx]
+mw1331.eqiad.wmnet: [apache2,nginx]
+mw1332.eqiad.wmnet: [apache2,nginx]
+mw1333.eqiad.wmnet: [apache2,nginx]
cache_text:
cp1052.eqiad.wmnet: [varnish-fe, varnish-be, nginx]
cp1053.eqiad.wmnet: [varnish-fe, varnish-be, nginx]
diff --git a/manifests/site.pp b/manifests/site.pp
index 21a6eb4..c6cb9e9 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1641,9 +1641,16 @@
include ::base::firewall
}
+# ROW C eqiad jobrunners
+node /^mw133[4-7]\.eqiad\.wmnet$/ {
+role(mediawiki::jobrunner)
+include ::base::firewall
+}
+
+
# T165519
# ROW C eqiad appservers
-node /^mw13(19|2[0-8])\.eqiad\.wmnet$/ {
+node /^mw13(19|2[0-9]|3[0-3])\.eqiad\.wmnet$/ {
role(mediawiki::appserver)
}
--
To view, visit https://gerrit.wikimedia.org/r/397749
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Id9539e819a46c3b7d22573384f361a8f1fea4188
Gerrit-PatchSet: 4
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
Gerrit-Reviewer: Elukey
Gerrit-Reviewer: Giuseppe Lavagetto
Gerrit-Reviewer: Muehlenhoff
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
