[Wikidata-bugs] [Maniphest] [Commented On] T237089: Create CQS puppet configs by applying query_service module

[gerritbot]

gerritbot added a comment.


  Change 602102 abandoned by Gehel:
  Role for SDoC WDQS
  
  Reason:
  replaced by I11763c3ebbfa21e958a5933573eef627b134e573
  
  https://gerrit.wikimedia.org/r/602102

TASK DETAIL
  https://phabricator.wikimedia.org/T237089

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: EBernhardson, gerritbot
Cc: Aklapper, Igorkim78, Gehel, Liuxinyu970226, Mathew.onipe, Blissjay007, 
Oblanco79, Alter-paule, Beast1978, CBogen, Un1tY, Hook696, Daryl-TTMG, 
RomaAmorRoma, E.S.A-Sheild, darthmon_wmde, Kent7301, Meekrab2012, joker88john, 
Legado_Shulgin, CucyNoiD, Nandana, NebulousIris, JKSTNK, Gaboe420, Versusxo, 
Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, 
Techguru.pc, Lahi, PDrouin-WMF, Gq86, Af420, E1presidente, Ramsey-WMF, Cparle, 
Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, 
Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, EBjune, Tramullas, 
Acer, LawExplorer, Salgo60, WSH1906, Lewizho99, Zppix, Maathavan, Silverfish, 
_jensen, rosalieper, Scott_WUaS, Susannaanas, Wong128hk, Jane023, 
Wikidata-bugs, Base, matthiasmullie, aude, Ricordisamoa, Wesalius, 
Lydia_Pintscher, Fabrice_Florin, Raymond, faidon, Jdforrester-WMF, 
Steinsplitter, Mbch331, Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T237089: Create CQS puppet configs by applying query_service module

[gerritbot]

gerritbot added a comment.


  Change 602102 had a related patch set uploaded (by Gehel; owner: 
EBernhardson):
  [operations/puppet@production] Role for SDoC WDQS
  
  https://gerrit.wikimedia.org/r/602102

TASK DETAIL
  https://phabricator.wikimedia.org/T237089

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: EBernhardson, gerritbot
Cc: Aklapper, Igorkim78, Gehel, Liuxinyu970226, Mathew.onipe, Blissjay007, 
Oblanco79, Alter-paule, Beast1978, CBogen, Un1tY, Hook696, Daryl-TTMG, 
RomaAmorRoma, E.S.A-Sheild, darthmon_wmde, Kent7301, Meekrab2012, joker88john, 
Legado_Shulgin, CucyNoiD, Nandana, NebulousIris, JKSTNK, Gaboe420, Versusxo, 
Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, 
Techguru.pc, Lahi, PDrouin-WMF, Gq86, Af420, E1presidente, Ramsey-WMF, Cparle, 
Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, 
Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, EBjune, Tramullas, 
Acer, LawExplorer, Salgo60, WSH1906, Lewizho99, Zppix, Maathavan, Silverfish, 
_jensen, rosalieper, Scott_WUaS, Susannaanas, Wong128hk, Jane023, 
Wikidata-bugs, Base, matthiasmullie, aude, Ricordisamoa, Wesalius, 
Lydia_Pintscher, Fabrice_Florin, Raymond, faidon, Jdforrester-WMF, 
Steinsplitter, Mbch331, Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T237089: Create CQS puppet configs by applying query_service module

[gerritbot]

gerritbot added a comment.


  Change 595041 **merged** by Gehel:
  [operations/puppet@production] Role for SDoC WDQS
  
  https://gerrit.wikimedia.org/r/595041

TASK DETAIL
  https://phabricator.wikimedia.org/T237089

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: EBernhardson, gerritbot
Cc: Aklapper, Igorkim78, Gehel, Liuxinyu970226, Mathew.onipe, Blissjay007, 
Oblanco79, Alter-paule, Beast1978, CBogen, Un1tY, Hook696, Daryl-TTMG, 
RomaAmorRoma, E.S.A-Sheild, darthmon_wmde, Kent7301, Meekrab2012, joker88john, 
Legado_Shulgin, CucyNoiD, Nandana, NebulousIris, JKSTNK, Gaboe420, Versusxo, 
Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, 
Techguru.pc, Lahi, PDrouin-WMF, Gq86, Af420, E1presidente, Ramsey-WMF, Cparle, 
Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, 
Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, EBjune, Tramullas, 
Acer, LawExplorer, Salgo60, WSH1906, Lewizho99, Zppix, Maathavan, Silverfish, 
_jensen, rosalieper, Scott_WUaS, Susannaanas, Wong128hk, Jane023, 
Wikidata-bugs, Base, matthiasmullie, aude, Ricordisamoa, Wesalius, 
Lydia_Pintscher, Fabrice_Florin, Raymond, faidon, Jdforrester-WMF, 
Steinsplitter, Mbch331, Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T237089: Create CQS puppet configs by applying query_service module

[EBernhardson]

EBernhardson added a comment.


  Instructions for booting a new instance. Currently this requires pointing the 
instance at a puppetmaster in the wikidata-query project.
  
  1. Start new instance in horizon. Use debian-stretch and m1.large
  2. Set the  puppetmaster 

 to `wsqspuppet.wikidata-query.eqiad.wmflabs`
  3. Apply hiera first, then roles
  4. Run `sudo puppet agent -tv` to apply the change now (or wait and it will 
happen eventually).
  
  hiera:
  
profile::query_service::blazegraph_heap_size: 6g
profile::query_service::blazegraph_use_deployed_config: false
profile::query_service::data_dir: /srv/wdqs-data
profile::query_service::forward_rsyslog_host: 
deployment-logstash03.deployment-prep.eqiad.wmflabs
profile::query_service::load_categories: none
profile::query_service::package_dir: /srv/wdqs-package
puppetmaster: wdqspuppet.wikidata-query.eqiad.wmflabs
  
  roles;
  
role::labs::lvm::srv
role::wdqs::sdoc

TASK DETAIL
  https://phabricator.wikimedia.org/T237089

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: EBernhardson
Cc: Aklapper, Igorkim78, Gehel, Liuxinyu970226, Mathew.onipe, Blissjay007, 
Oblanco79, Alter-paule, Beast1978, CBogen, Un1tY, Hook696, Daryl-TTMG, 
RomaAmorRoma, E.S.A-Sheild, darthmon_wmde, Kent7301, Meekrab2012, joker88john, 
Legado_Shulgin, CucyNoiD, Nandana, NebulousIris, JKSTNK, Gaboe420, Versusxo, 
Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, 
Techguru.pc, Lahi, PDrouin-WMF, Gq86, Af420, E1presidente, Ramsey-WMF, Cparle, 
Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, 
Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, EBjune, Tramullas, 
Acer, LawExplorer, Salgo60, WSH1906, Lewizho99, Zppix, Maathavan, Silverfish, 
_jensen, rosalieper, Scott_WUaS, Susannaanas, Wong128hk, Jane023, 
Wikidata-bugs, Base, matthiasmullie, aude, Ricordisamoa, Wesalius, 
Lydia_Pintscher, Fabrice_Florin, Raymond, faidon, Jdforrester-WMF, 
Steinsplitter, Mbch331, Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T237089: Create CQS puppet configs by applying query_service module

[gerritbot]

gerritbot added a comment.


  Change 595041 had a related patch set uploaded (by EBernhardson; owner: 
EBernhardson):
  [operations/puppet@production] Role for SDoC WDQS
  
  https://gerrit.wikimedia.org/r/595041

TASK DETAIL
  https://phabricator.wikimedia.org/T237089

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: EBernhardson, gerritbot
Cc: Aklapper, Igorkim78, Gehel, Liuxinyu970226, Mathew.onipe, CBogen, 
darthmon_wmde, Legado_Shulgin, Nandana, JKSTNK, Davinaclare77, Qtn1293, 
Techguru.pc, Lahi, PDrouin-WMF, Gq86, E1presidente, Ramsey-WMF, Cparle, 
Anooprao, SandraF_WMF, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, 
Tramullas, Acer, LawExplorer, Salgo60, Zppix, Silverfish, _jensen, rosalieper, 
Scott_WUaS, Susannaanas, Wong128hk, Jane023, Wikidata-bugs, Base, 
matthiasmullie, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, Fabrice_Florin, 
Raymond, faidon, Jdforrester-WMF, Steinsplitter, Mbch331, Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T237089: Create CQS puppet configs by applying query_service module

[EBernhardson]

EBernhardson added a comment.


  Matt's initial work has gotten us most of the way there. In reviewing whats 
available now, and booting a test instance to see if it can fully setup a new 
instance from scratch (hint: no).
  
  Background info
  
  - The current sdcquery instance applies the `role::wdqs::labs`. There is also 
`role::wdqs::sdoc`
  - There is a tiny amount of instance specific puppet config in horizon
  - Most puppet config is in the main puppet repo at 
`hieradata/cloud/eqiad1/wikidata-query/common.yaml`. This config is for a 
wikidata query service though, not the SDoC query service.
  
  Issues to address
  
  - Logging is configured to ship to deployment-logstash2, but 
logstash-beta.wmflabs.org doesn't report any logs from the existing instance or 
the one i recently booted.
  - The instance is also configured with the beta cluster eventgate endpoint 
(for request logging), once an instance is running we can verify correct 
operation. Even if we get the events flowing, if we want to do anything with 
them followup will be required.
  - Puppet currently includes the updater and categories, afaik we want those 
disabled for the new instance
  - Brand new instances currently don't complete the puppet run due to trying 
to clone wdqs repo into a parent directory that doesn't exist
  - Once the instance was up it only responded for a few minutes after which 
jetty reports `Service Unavailable` on /bigdata/. Unclear yet what causes this.
  - Current sdcquery sets `use_deployed_config=true` which means puppet doesn't 
control the blazegraph configuration, it's whatever happens to be on the 
machine.
  - Currently `profile::query_service::blazegraph` installs the primary 
blazegraph instance, but `profile::query_service::categories` installs the 
categories specific blazegraph instance. It's unclear if sdoc should have a new 
role and new profile, or if a new sdoc role should point at the current 
blazegraph profile. There are significant amounts of duplication between 
blazegraph and cateogies query_service profiles, but puppet doesn't make it 
super elegant to put all these things together generically and still understand 
what the differences are.
  
  Probably more, but this is an initial look through. I'm going to put together 
a patch to address some of the above and get the instance starting from a cold 
boot, but there are some open questions above that could use any insight others 
might have.

TASK DETAIL
  https://phabricator.wikimedia.org/T237089

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: EBernhardson
Cc: Aklapper, Igorkim78, Gehel, Liuxinyu970226, Mathew.onipe, CBogen, 
darthmon_wmde, Legado_Shulgin, Nandana, JKSTNK, Davinaclare77, Qtn1293, 
Techguru.pc, Lahi, PDrouin-WMF, Gq86, E1presidente, Ramsey-WMF, Cparle, 
Anooprao, SandraF_WMF, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, 
Tramullas, Acer, LawExplorer, Salgo60, Zppix, Silverfish, _jensen, rosalieper, 
Scott_WUaS, Susannaanas, Wong128hk, Jane023, Wikidata-bugs, Base, 
matthiasmullie, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, Fabrice_Florin, 
Raymond, faidon, Jdforrester-WMF, Steinsplitter, Mbch331, Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T237089: Create CQS puppet configs by applying query_service module

[Mathew.onipe]

Mathew.onipe added a comment.


  @Igorkim78 can you document the config changes here?

TASK DETAIL
  https://phabricator.wikimedia.org/T237089

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Mathew.onipe
Cc: Aklapper, Igorkim78, Gehel, Liuxinyu970226, Mathew.onipe, darthmon_wmde, 
Legado_Shulgin, Nandana, JKSTNK, Davinaclare77, Qtn1293, Techguru.pc, Lahi, 
PDrouin-WMF, Gq86, E1presidente, Ramsey-WMF, Cparle, Anooprao, SandraF_WMF, 
GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, Tramullas, Acer, 
LawExplorer, Salgo60, Zppix, Silverfish, _jensen, rosalieper, Scott_WUaS, 
Susannaanas, Wong128hk, Jane023, Wikidata-bugs, Base, matthiasmullie, aude, 
Ricordisamoa, Wesalius, Lydia_Pintscher, Fabrice_Florin, Raymond, faidon, 
Jdforrester-WMF, Steinsplitter, Mbch331, Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs