Re: [tor-bugs] #30020 [Internal Services/Tor Sysadmin Team]: switch from our custom YAML implementation to Hiera

[Tor Bug Tracker & Wiki]

#30020: switch from our custom YAML implementation to Hiera
-+-
 Reporter:  anarcat  |  Owner:  anarcat
 Type:  project  | Status:
 |  accepted
 Priority:  Medium   |  Milestone:
Component:  Internal Services/Tor Sysadmin Team  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:  #29387   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by anarcat):

 awesome. so we're at step 6:

  6. create a new YAML variable that gives us a IP range -> hoster mapping
  7. create a function that looks through those to guess the hoster for a
 given IP address (probably just fixing `whohosts`?)
  8. use that function to create a fact (through a template, but with a
 variable defined in the base class) that defines the $hoster variable that
 hiera will use to load the right YAML (DONE, right?)
  9. remove hoster.yaml

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30020 [Internal Services/Tor Sysadmin Team]: switch from our custom YAML implementation to Hiera

[Tor Bug Tracker & Wiki]

#30020: switch from our custom YAML implementation to Hiera
-+-
 Reporter:  anarcat  |  Owner:  anarcat
 Type:  project  | Status:
 |  accepted
 Priority:  Medium   |  Milestone:
Component:  Internal Services/Tor Sysadmin Team  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:  #29387   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by weasel):

 hoster.yaml now only has the networks that define which hoster a node is
 at.  It's still used for the whohosts function~~ and for ferm to make up
 what we consider tor networks~~.

 We ship the hoster name as a fact to the node, and then include hieradata
 based on that fact, defining things like the debian mirror.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30020 [Internal Services/Tor Sysadmin Team]: switch from our custom YAML implementation to Hiera

[Tor Bug Tracker & Wiki]

#30020: switch from our custom YAML implementation to Hiera
-+-
 Reporter:  anarcat  |  Owner:  anarcat
 Type:  project  | Status:
 |  accepted
 Priority:  Medium   |  Milestone:
Component:  Internal Services/Tor Sysadmin Team  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:  #29387   | Points:
 Reviewer:   |Sponsor:
-+-
Description changed by anarcat:

Old description:

> We currently use a custom-made YAML database for assigning roles to
> servers and other metadata. I started using Hiera for some hosts and it
> seems to be working well.
>
> Hiera is officially supported in Puppet and shipped by default in Puppet
> 5 and later. It's the standard way of specifying metadata and class
> parameters for hosts. I suspect it covers most of our needs in terms of
> metadata and should cover most if not all of what we're currently doing
> with the YAML stuff in Puppet.
>
> We should therefore switch to using Hiera instead of our homegrown
> solution.
>
> This involves converting:
>
>  * `if has_role('foo') { include foo }` into `classes: [ 'foo' ]` in
> hiera
>  * hardcoded macros in the ferm module's `me.conf.erb` into exported
> resources
>  * templates looping over allnodeinfo into exported resources
>  * the `$roles` array into Hiera
>  * the `$localinfo` into Hiera (assuming all the data is there)
>  * the `$nodeinfo` and `$allnodeinfo` arrays into Hiera (assuming we can
> switch from LDAP for host inventory)
>  * basically any other stuff of the kind, including those files:
>{{{
> ./modules/torproject_org/misc/hoster.yaml
> ./modules/torproject_org/misc/local.yaml
> ./modules/ipsec/misc/config.yaml
> ./modules/roles/misc/static-components.yaml
> ./modules/roles/files/spec/spec-redirects.yaml
> }}}
>
> Ideally, all YAML data should end up in the hiera/ directory somehow.
> This is the first step in making our repository public (#29387) but also
> using Hiera as a more elaborate inventory system (#30273).
>
> The idea of switching from LDAP to Hiera for host inventory will
> definitely need to be evaluated more thoroughly before going ahead with
> that part of the conversion, but YAML stuff in Puppet should definitely
> be converted.
>
> The general goal of this is both to allow for a better inventory system
> but also make it easier for people to get onboarded with Puppet. By using
> community standards like Hiera, we make it easier for new people to get
> familiar with the puppet infrastructures and do things meaningfully.

New description:

 We currently use a custom-made YAML database for assigning roles to
 servers and other metadata. I started using Hiera for some hosts and it
 seems to be working well.

 Hiera is officially supported in Puppet and shipped by default in Puppet 5
 and later. It's the standard way of specifying metadata and class
 parameters for hosts. I suspect it covers most of our needs in terms of
 metadata and should cover most if not all of what we're currently doing
 with the YAML stuff in Puppet.

 We should therefore switch to using Hiera instead of our homegrown
 solution.

 This involves converting:

  * `if has_role('foo') { include foo }` into `classes: [ 'foo' ]` in hiera
 (DONE!)
  * hardcoded macros in the ferm module's `me.conf.erb` into exported
 resources (DONE, except for HOST_TPO)
  * templates looping over allnodeinfo into exported resources
  * the `$roles` array into Hiera (DONE!)
  * the `$localinfo` into Hiera (assuming all the data is there) (DONE!)
  * the `$nodeinfo` and `$allnodeinfo` arrays into Hiera (assuming we can
 switch from LDAP for host inventory)
  * basically any other stuff of the kind, including those files:
{{{
 ./modules/torproject_org/misc/hoster.yaml
 ./modules/torproject_org/misc/local.yaml <- DONE!
 ./modules/ipsec/misc/config.yaml
 ./modules/roles/misc/static-components.yaml
 ./modules/roles/files/spec/spec-redirects.yaml
 }}}

 Ideally, all YAML data should end up in the hiera/ directory somehow. This
 is the first step in making our repository public (#29387) but also using
 Hiera as a more elaborate inventory system (#30273).

 The idea of switching from LDAP to Hiera for host inventory will
 definitely need to be evaluated more thoroughly before going ahead with
 that part of the conversion, but YAML stuff in Puppet should definitely be
 converted.

 The general goal of this is both to allow for a better inventory system
 but also make it easier for people to get onboarded with Puppet. By using
 community standards like Hiera, we make it 

Re: [tor-bugs] #30020 [Internal Services/Tor Sysadmin Team]: switch from our custom YAML implementation to Hiera

[Tor Bug Tracker & Wiki]

#30020: switch from our custom YAML implementation to Hiera
-+-
 Reporter:  anarcat  |  Owner:  anarcat
 Type:  project  | Status:
 |  accepted
 Priority:  Medium   |  Milestone:
Component:  Internal Services/Tor Sysadmin Team  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:  #29387   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by anarcat):

 grand milestone today: `local.yaml` was removed from the repository, along
 with `get_role` and `yamlinfo`, which are all now useless.

 WHOOHOO!

 == Next step: hoster.yaml

 the next chunk we need to convert would be, i think,
 `./modules/torproject_org/misc/hoster.yaml`, which specifies those things:

 * `netrange`: used to create the `TPO_NET` macro in ferm (unused?) and
 determine in which `hoster` a given host is (through `whohosts.rb`, which
 does IP range calculations from the host's IP as seen from LDAP)
 * `mirror-debian`: used in `torproject_org` class to define the APT mirror
 for this host
 * `mirror-debian-security`: unused?
 * `nameservers`: used to configured upstream forwarders in unbound on each
 host
 * `nameservers_break_dnssec` : used to disable unbound forwarding in case
 of broken upstream DNS, unused
 * `allow_dns_query`: used to tell unbound to allow other network ranges
 (ie. generally on this site) to use *this* node as recursive DNS server
 (if `misc.resolver-recursive` is true, which is the case when the LDAP ip
 of the host is listed in the hoster's `nameservers` list)

 Debian.org has hooked hosters.yaml into hiera, and the way they did it is
 to have one .yaml file per hoster, for example:

 https://salsa.debian.org/dsa-team/mirror/dsa-
 puppet/blob/master/hieradata/bytemark.yaml

 Unfortunately, the `hoster.yaml` is still present on d.o:

 https://salsa.debian.org/dsa-team/mirror/dsa-
 puppet/blob/master/modules/puppetmaster/lib/puppet/parser/functions/whohosts.rb

 there we have the same code as on tpo:

 {{{
 yamlfile =
 Puppet::Parser::Files.find_file('debian_org/misc/hoster.yaml',
 compiler.environment)
 }}}

 Here the file, which contains more than just ip ranges:

 https://salsa.debian.org/dsa-team/mirror/dsa-
 puppet/blob/master/modules/debian_org/files/misc/hoster.yaml

 So their transition isn't complete, but it matches some of the ideas I had
 (namely to have one YAML file per hoster).

 So what I would suggest we do to get rid of the hoster.yaml file is this:

  1. convert all the aforementioned variables used in `hoster.yaml` into
 class variables, defaulting to the values loaded from `hoster.yaml` (ie.
 `$nodeinfo`)
  2. test the variables by overriding them in (e.g.)
 `hiera/nodes/foo.example.com.yaml`
  3. break up the `hoster.yaml` file into multiple smaller files in
 `hiera/hoster/%{hoster}.yaml`
  4. add that path to `hiera.yaml`
  5. test that host can load its variables from the `hoster` search path by
 hardcoding a value by hand in facter
  6. create a new YAML variable that gives us a IP range -> hoster mapping
  7. create a function that looks through those to guess the hoster for a
 given IP address
  8. use that function to create a fact (through a template, but with a
 variable defined in the base class) that defines the `$hoster` variable
 that hiera will use to load the right YAML
  9. remove `hoster.yaml`

 That's a first step. At that stage, `hoster.yaml` is gone, but `$nodeinfo`
 remains and still might contain host-specific configuration. Those should
 be extract *out* of the `$nodeinfo` construct and into manifest business
 logic. And *then* the `nodeinfo.rb` code can be ripped out.

 There might be a better way to define a hoster per node than guess it with
 its IP address and drop it as a fact, but I can't think of anything right
 now.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30020 [Internal Services/Tor Sysadmin Team]: switch from our custom YAML implementation to Hiera

[Tor Bug Tracker & Wiki]

#30020: switch from our custom YAML implementation to Hiera
-+-
 Reporter:  anarcat  |  Owner:  anarcat
 Type:  project  | Status:
 |  accepted
 Priority:  Medium   |  Milestone:
Component:  Internal Services/Tor Sysadmin Team  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:  #29387   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by anarcat):

 natted, mail_processing and no_hw_clock were completed this week.

 only ONE role left! whoohoo!

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30020 [Internal Services/Tor Sysadmin Team]: switch from our custom YAML implementation to Hiera

[Tor Bug Tracker & Wiki]

#30020: switch from our custom YAML implementation to Hiera
-+-
 Reporter:  anarcat  |  Owner:  anarcat
 Type:  project  | Status:
 |  accepted
 Priority:  Medium   |  Milestone:
Component:  Internal Services/Tor Sysadmin Team  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:  #29387   | Points:
 Reviewer:   |Sponsor:
-+-
Changes (by anarcat):

 * status:  assigned => accepted


Old description:

> We currently use a custom-made YAML database for assigning roles to
> servers and other metadata. I started using Hiera for some hosts and it
> seems to be working well.
>
> Hiera is officially supported in Puppet and shipped by default in Puppet
> 5 and later. It's the standard way of specifying metadata and class
> parameters for hosts. I suspect it covers most of our needs in terms of
> metadata and should cover most if not all of what we're currently doing
> with the YAML stuff in Puppet.
>
> We should therefore switch to using Hiera instead of our homegrown
> solution.
>
> This involves converting:
>
>  * `if has_role('foo') { include foo }` into `classes: [ 'foo' ]` in
> hiera
>  * hardcoded macros in the ferm module's `me.conf.erb` into exported
> resources
>  * templates looping over allnodeinfo into exported resources
>  * the `$roles` array into Hiera
>  * the `$localinfo` into Hiera (assuming all the data is there)
>  * the `$nodeinfo` and `$allnodeinfo` arrays into Hiera (assuming we can
> switch from LDAP for host inventory)
>  * basically any other stuff of the kind
>
> Ideally, all YAML data should end up in the hiera/ directory somehow.
> This is the first step in making our repository public (#29387) but also
> using Hiera as a more elaborate inventory system (#30273).
>
> The idea of switching from LDAP to Hiera for host inventory will
> definitely need to be evaluated more thoroughly before going ahead with
> that part of the conversion, but YAML stuff in Puppet should definitely
> be converted.
>
> The general goal of this is both to allow for a better inventory system
> but also make it easier for people to get onboarded with Puppet. By using
> community standards like Hiera, we make it easier for new people to get
> familiar with the puppet infrastructures and do things meaningfully.

New description:

 We currently use a custom-made YAML database for assigning roles to
 servers and other metadata. I started using Hiera for some hosts and it
 seems to be working well.

 Hiera is officially supported in Puppet and shipped by default in Puppet 5
 and later. It's the standard way of specifying metadata and class
 parameters for hosts. I suspect it covers most of our needs in terms of
 metadata and should cover most if not all of what we're currently doing
 with the YAML stuff in Puppet.

 We should therefore switch to using Hiera instead of our homegrown
 solution.

 This involves converting:

  * `if has_role('foo') { include foo }` into `classes: [ 'foo' ]` in hiera
  * hardcoded macros in the ferm module's `me.conf.erb` into exported
 resources
  * templates looping over allnodeinfo into exported resources
  * the `$roles` array into Hiera
  * the `$localinfo` into Hiera (assuming all the data is there)
  * the `$nodeinfo` and `$allnodeinfo` arrays into Hiera (assuming we can
 switch from LDAP for host inventory)
  * basically any other stuff of the kind, including those files:
{{{
 ./modules/torproject_org/misc/hoster.yaml
 ./modules/torproject_org/misc/local.yaml
 ./modules/ipsec/misc/config.yaml
 ./modules/roles/misc/static-components.yaml
 ./modules/roles/files/spec/spec-redirects.yaml
 }}}

 Ideally, all YAML data should end up in the hiera/ directory somehow. This
 is the first step in making our repository public (#29387) but also using
 Hiera as a more elaborate inventory system (#30273).

 The idea of switching from LDAP to Hiera for host inventory will
 definitely need to be evaluated more thoroughly before going ahead with
 that part of the conversion, but YAML stuff in Puppet should definitely be
 converted.

 The general goal of this is both to allow for a better inventory system
 but also make it easier for people to get onboarded with Puppet. By using
 community standards like Hiera, we make it easier for new people to get
 familiar with the puppet infrastructures and do things meaningfully.

--

Comment:

 we now have:

 * 3 `has_role` references
 * 4 roles left (`haproxy`, `mail_processing`, `natted`, `no_hw_clock`)
 * 2 `localinfo` references (in postfix, related to 

Re: [tor-bugs] #30020 [Internal Services/Tor Sysadmin Team]: switch from our custom YAML implementation to Hiera

[Tor Bug Tracker & Wiki]

#30020: switch from our custom YAML implementation to Hiera
-+-
 Reporter:  anarcat  |  Owner:  anarcat
 Type:  project  | Status:
 |  assigned
 Priority:  Medium   |  Milestone:
Component:  Internal Services/Tor Sysadmin Team  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:  #29387   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by anarcat):

 down to 6 `has_role` (down from ~50) in the main `roles` class, thanks to
 the help of hiro who joined in the effort. there are also still 18 roles
 (down from 57) left in `local.yaml`, which i'll try to tackle next. there
 are some leftovers of the `static-*` roles there that I seem to have
 skipped over. they are bound to SSH key propagation and internal class
 parameters, so it was likely deliberate.

 but we have definitely cross the halfway point, and I'd say we're getting
 close to the finish line, at least with regards to the custom `has_role`
 stuff. there's naturally more stuff that could move to Hiera and other
 YAML files strewed around the codebase, but this is a huge chunk that will
 be done shortly.

 Those are the files I am currently aware of that would benefit to be
 transitioned into Hiera:

 {{{
 ./modules/torproject_org/misc/hoster.yaml
 ./modules/torproject_org/misc/local.yaml
 ./modules/ipsec/misc/config.yaml
 ./modules/roles/misc/static-components.yaml
 ./modules/roles/files/spec/spec-redirects.yaml
 }}}

 But I suspect many of those will be easier than the wide-ranging
 `has_role` transition, as each one of those file touches one or only a few
 module, as opposed to the `local.yaml` file which touched *everything*.

 So, good progress, even if slow.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30020 [Internal Services/Tor Sysadmin Team]: switch from our custom YAML implementation to Hiera

[Tor Bug Tracker & Wiki]

#30020: switch from our custom YAML implementation to Hiera
-+-
 Reporter:  anarcat  |  Owner:  anarcat
 Type:  project  | Status:
 |  assigned
 Priority:  Medium   |  Milestone:
Component:  Internal Services/Tor Sysadmin Team  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:  #29387   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by anarcat):

 i got a little tired of battling this, so I took a small break. I still
 migrated a few roles:

 {{{
 civicrm_ext_2018
 civicrm_int_2018
 civicrm_ext
 civicrm_int
 public_git
 rt
 svn
 metrics
 exonerator
 bridges
 trac
 mandos_server
 }}}

 many of those were easy marks: the ssl::service stuff were just a lot of
 copy-paste, which might have been better implemented by having a
 parametrized class with the node-specific parameters in hiera, something
 like:

 {{{
 class profile::ssl_web($name, $onion = false) {
ssl::service { $name: notify => Exec['service apache2 reload'], key =>
 true, onion => $onion }
 }
 }}}

 And in (say) `eugeni.torproject.org.yaml`, you would have:

 {{{
 profile::ssl_web::name: "lists.torproject.org"
 profile::ssl_web::onion: true
 classes:
   - profile::ssl_web
 }}}

 ... but I didn't want to overthink this just yet. plus we might want to
 manage those services more closely in Puppet eventually and such a class
 would just make it difficult. Besides, i suspect this would belong in the
 Apache module, not in a profile. '''And''' we should have a ''role'' in
 Hiera instead of a ''profile'', so we would end up creating the equivalent
 of the ''profile'' I ended up making anyways:

 {{{
 class profile::lists {
   ssl::service { 'lists.torproject.org':
 notify => Exec['service apache2 reload'],
 key=> true,
   }
 }
 }}}

 So I think it's the right conversion for now. I'm not converting the
 entire hierarchy to R/P/M just yet anyways, just switching to Hiera is
 enough work as it is.

 There are now 22 `has_role` calls left in the main `roles` class, down
 from around 50. Unfortunately, there is actually more roles in the
 `local.yaml` file (33) that I haven't considered or noticed, so we haven't
 crossed the magic halfway point just yet.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30020 [Internal Services/Tor Sysadmin Team]: switch from our custom YAML implementation to Hiera

[Tor Bug Tracker & Wiki]

#30020: switch from our custom YAML implementation to Hiera
-+-
 Reporter:  anarcat  |  Owner:  anarcat
 Type:  project  | Status:
 |  assigned
 Priority:  Medium   |  Milestone:
Component:  Internal Services/Tor Sysadmin Team  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:  #29387   | Points:
 Reviewer:   |Sponsor:
-+-

Old description:

> We currently use a custom-made YAML database for assigning roles to
> servers and other metadata. I started using Hiera for some hosts and it
> seems to be working well.
>
> Hiera is officially supported in Puppet and shipped by default in Puppet
> 5 and later. It's the standard way of specifying metadata and class
> parameters for hosts. I suspect it covers most of our needs in terms of
> metadata and should cover most if not all of what we're currently doing
> with the YAML stuff in Puppet.
>
> We should therefore switch to using Hiera instead of our homegrown
> solution.

New description:

 We currently use a custom-made YAML database for assigning roles to
 servers and other metadata. I started using Hiera for some hosts and it
 seems to be working well.

 Hiera is officially supported in Puppet and shipped by default in Puppet 5
 and later. It's the standard way of specifying metadata and class
 parameters for hosts. I suspect it covers most of our needs in terms of
 metadata and should cover most if not all of what we're currently doing
 with the YAML stuff in Puppet.

 We should therefore switch to using Hiera instead of our homegrown
 solution.

 This involves converting:

  * `if has_role('foo') { include foo }` into `classes: [ 'foo' ]` in hiera
  * hardcoded macros in the ferm module's `me.conf.erb` into exported
 resources
  * templates looping over allnodeinfo into exported resources
  * the `$roles` array into Hiera
  * the `$localinfo` into Hiera (assuming all the data is there)
  * the `$nodeinfo` and `$allnodeinfo` arrays into Hiera (assuming we can
 switch from LDAP for host inventory)
  * basically any other stuff of the kind

 Ideally, all YAML data should end up in the hiera/ directory somehow. This
 is the first step in making our repository public (#29387) but also using
 Hiera as a more elaborate inventory system (#30273).

 The idea of switching from LDAP to Hiera for host inventory will
 definitely need to be evaluated more thoroughly before going ahead with
 that part of the conversion, but YAML stuff in Puppet should definitely be
 converted.

 The general goal of this is both to allow for a better inventory system
 but also make it easier for people to get onboarded with Puppet. By using
 community standards like Hiera, we make it easier for new people to get
 familiar with the puppet infrastructures and do things meaningfully.

--

Comment (by anarcat):

 i did more work here. the following macros have now been safely removed:

 {{{
 HOST_STATIC
 HOST_ROLE_PEOPLE
 HOST_ROLE_METRICSBOT
 HOST_ROLE_JABBER_SERVER
 HOST_ROLE_WEBLOG_SOURCE
 HOST_ROLE_WEBLOG_SINK
 }}}

 This also led to the removal of a custom SSH keys generation template
 (`modules/roles/templates/weblog_sink/webstats-authorized_keys.erb`),
 although it hasn't been converted to the native `ssh_authorized_keys`
 because of the format difference between the custom fact we use to export
 the ssh keys and the one expected by the type. This could be fixed in
 another refactoring at some other time.

 Now, I'm working on the `static_*` stuff, which is like `weblog_*` but a
 little more complicated because the config files are not (yet) built with
 config::fragment. The SSH firewall configuration was a little more
 complicated but it's been migrated already. Next up is the authorized_keys
 which should follow the same pattern as the weblog stuff and then the
 config::fragment conversion. There are also corner cases with more sub-
 roles for that one that will need to be taken into account, but those can
 hopefully be converted into class parameters.

 There are now 36 roles left in the `roles` class. There were about 50
 roles, split between `site.pp` and the `roles` class, when I started this,
 about a week ago, so i think it would be fair to assume this first part of
 the conversion will be done in a week or two.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org

Re: [tor-bugs] #30020 [Internal Services/Tor Sysadmin Team]: switch from our custom YAML implementation to Hiera

[Tor Bug Tracker & Wiki]

#30020: switch from our custom YAML implementation to Hiera
-+-
 Reporter:  anarcat  |  Owner:  anarcat
 Type:  project  | Status:
 |  assigned
 Priority:  Medium   |  Milestone:
Component:  Internal Services/Tor Sysadmin Team  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:  #29387   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by anarcat):

 site.pp is now mostly empty. all the `has_role` constructs are gone from
 there.

 those two are gone as well:

 {{{
 HOST_ROLE_BACULA_DIRECTOR
 HOST_ROLE_BACULA_STORAGE
 }}}

 the trickiest part, surprisingly, was the little warning added to the
 motd. i've hacked something together using `update-motd.d` but i'm
 actually quite unhappy about it, because it doesn't display the same way
 that it did before. if the machines were all running buster, this wouldn't
 be a problem anymore because there's /etc/motd.d there, but we're probably
 stuck in stretch for a while.

 since this is only for *three* machines, I think we can afford the little
 ugliness for now.

 {{{
 Linux build-arm-02 4.19.0-0.bpo.4-arm64 #1 SMP Debian 4.19.28-2~bpo9+1
 (2019-03-27) aarch64

  Note that this host is _NOT_ being backed up.  If you care about your
  data, run your own backups.


 This device is for authorized users only.

 %%%

 Welcome to build-arm-02.torproject.org, used for the following services:
 buildbox
 porterbox

  If you use this as a porter/buildbox, you might find
  https://dsa.debian.org/doc/schroot/ helpful.

 %%%

 Last login: Fri Apr 19 20:44:31 2019 from 95.216.141.241
 }}}


 I have also found HOST_TPO which is basically a list of the public IP of
 all TPO hosts, as taken from LDAP
 (`modules/puppetmaster/lib/puppet/parser/functions/allnodeinfo.rb`). So we
 can keep that macro for now until we decide about the overlap between LDAP
 and Hiera. The motd is similarly extracted mostly from stuff in LDAP and
 would benefit from such a refactoring as well.

 Anyways. Next up is the roles file, which has tons more fun stuff like
 this to clear out. :)

 Note that I've had answers to my earlier questions, somehow:

  1. I don't think there's any serious security issues with exported
 resources, they way they're setup. At worst a host might be able to push
 different firewall holes than expected. If we want to fix that issue, we
 can make new defines with hardcoded definitions that, when collected on
 hosts, will only poke the holes that are expected.

  2. it's just a copy-paste historical error, that I've made myself in
 other occasions

  3. no solution to the NRPE `allowed_hosts` problem just yet, but I'm
 tempted to just use a hardcoded variable for now. this is what is used for
 `bacula::bacula_director_address` for example: it's hardcoded to
 `dictyotum.torproject.org` so there's prior art to hardcoding stuff like
 that. of course it would be hardcoded into hiera, not the class name,
 ideally...

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30020 [Internal Services/Tor Sysadmin Team]: switch from our custom YAML implementation to Hiera

[Tor Bug Tracker & Wiki]

#30020: switch from our custom YAML implementation to Hiera
-+-
 Reporter:  anarcat  |  Owner:  anarcat
 Type:  project  | Status:
 |  assigned
 Priority:  Medium   |  Milestone:
Component:  Internal Services/Tor Sysadmin Team  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:  #29387   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by anarcat):

 Another possible solution is to move from LDAP to Hiera for host metadata.
 That is where, after all, Puppet is getting some of those IP addresses
 from and it would be possible to simply do lookups in Hiera for those, if
 it was properly loaded and ordered.

 Another case I found is `roles::weblog_sink` which constructs SSH keys
 from the YAML data. This could be generated from exported resources as
 well, for example with the
 [https://puppet.com/docs/puppet/4.8/type.html#sshauthorizedkey
 ssh_authorized_keys builtin type].

 So in other words, I think this project is doable, but it will require
 refactoring and lots of work.

 In the end, though, we would have one YAML file per host in
 `hiera/nodes/$FQDN.yaml`. This could be made fairly human-readable if we
 make a good template, and be the single source of truth for all
 information about a host including hosting provider, cost and so on,
 solving our inventory problem, (partly) described in #29816.

 I think this is worth it and will make it easier to get people involved in
 Puppet work.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30020 [Internal Services/Tor Sysadmin Team]: switch from our custom YAML implementation to Hiera

[Tor Bug Tracker & Wiki]

#30020: switch from our custom YAML implementation to Hiera
-+-
 Reporter:  anarcat  |  Owner:  anarcat
 Type:  project  | Status:
 |  assigned
 Priority:  Medium   |  Milestone:
Component:  Internal Services/Tor Sysadmin Team  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:  #29387   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by anarcat):

 some more progress, but this time harder stuff: I converted the DNS
 servers to Hiera. this involved splitting some classes and exporting
 resources. in my travels, those are the important HOST_ROLE_ ferm rules
 that I found might be problematic:

 {{{
 HOST_ROLE_BACULA_DIRECTOR
 HOST_ROLE_BACULA_STORAGE
 HOST_ROLE_DIP
 HOST_ROLE_DNS_SECONDARY
 HOST_ROLE_JENKINS
 HOST_ROLE_NAGIOSMASTER
 HOST_ROLE_PUPPETMASTER
 }}}

 I also found `HOST_NETNOD` but I think that might be a static definition.

 `HOST_ROLE_DNS_SECONDARY` is now gone, and replaced by exported
 `ferm::rule` constructs. This works well, but @weasel was somehow worried
 about security issues with exported resources, which I am not sure are
 relevant in this case.

 Another problem is that the ferm` module is setup to ''realize'' the
 virtual `ferm::rule` stuff defined everywhere. This implies that the
 exported resources are '''also''' realized '''locally'''. That's fairly
 harmless, because the host allows itself access to itself, but it's noisy
 and annoying.

 I don't know why `ferm::rule` entries are virtual everywhere, so that's
 something I'd like to explore as well in the future.

 Another problem I found when working on the DNS stuff is that the DNS
 primary does checks on the the DNS secondaries, seemingly through NRPE,
 because it is in the `allowed_hosts` list in the NRPE config. This makes
 it impossible to remove the `dns_primary` role from `local.yaml` for now
 and I'm not sure how to work around that without creating a global
 variable for the DNS primary host, which would be an unfortunate
 regression.

 So two pending questions:

  1. what is the security issue with exported resources? is the current
 pattern used in the bind module and prometheus profile acceptable?

  2. why are `ferm::rule` entries virtual?

  3. how can we export arbitrary IPs in configuration files in Hiera?
 specifically, how do we construct NRPE's `allowed_hosts` list of IPs from
 other hosts?

 My tentative guesses on this are:

  1. impact minor, even if security issue (possibility to manipulate
 firewall rules between nodes)
  2. probably just an oversight?
  3. i feel dirty saying it, but a fancy `sed` Exec exported resource?

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30020 [Internal Services/Tor Sysadmin Team]: switch from our custom YAML implementation to Hiera

[Tor Bug Tracker & Wiki]

#30020: switch from our custom YAML implementation to Hiera
-+-
 Reporter:  anarcat  |  Owner:  anarcat
 Type:  project  | Status:
 |  assigned
 Priority:  Medium   |  Milestone:
Component:  Internal Services/Tor Sysadmin Team  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:  #29387   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by anarcat):

 this has started. most of site.pp has been emptied, with the easy stuff
 first. the hard stuff are hosts where the `$roles` function is actually
 relevant. for example, the following ferm macros are probably actually in
 use:

 HOST_ROLE_PUPPETMASTER HOST_ROLE_DIP HOST_ROLE_JENKINS
 HOST_ROLE_NAGIOSMASTER

 ... and probably more, namely bacula. other classes will refer to the
 $roles or nodeinfo lists explicitely as well and will need to be broken up
 in separate classes that then get properly included. but it's a great
 start and so far no breakage that i know of.

 i documented the impact of the change in site.pp, but it might be good to
 add something to the wiki docs as well.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

[tor-bugs] #30020 [Internal Services/Tor Sysadmin Team]: switch from our custom YAML implementation to Hiera

[Tor Bug Tracker & Wiki]

#30020: switch from our custom YAML implementation to Hiera
-+-
 Reporter:  anarcat  |  Owner:  anarcat
 Type:  project  | Status:  assigned
 Priority:  Medium   |  Milestone:
Component:  Internal Services/Tor Sysadmin   |Version:
  Team   |
 Severity:  Normal   |   Keywords:
Actual Points:   |  Parent ID:  #29387
   Points:   |   Reviewer:
  Sponsor:   |
-+-
 We currently use a custom-made YAML database for assigning roles to
 servers and other metadata. I started using Hiera for some hosts and it
 seems to be working well.

 Hiera is officially supported in Puppet and shipped by default in Puppet 5
 and later. It's the standard way of specifying metadata and class
 parameters for hosts. I suspect it covers most of our needs in terms of
 metadata and should cover most if not all of what we're currently doing
 with the YAML stuff in Puppet.

 We should therefore switch to using Hiera instead of our homegrown
 solution.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs