#30020: switch from our custom YAML implementation to Hiera -------------------------------------------------+------------------------- Reporter: anarcat | Owner: anarcat Type: project | Status: | accepted Priority: Medium | Milestone: Component: Internal Services/Tor Sysadmin Team | Version: Severity: Normal | Resolution: Keywords: | Actual Points: Parent ID: #29387 | Points: Reviewer: | Sponsor: -------------------------------------------------+-------------------------
Comment (by anarcat): grand milestone today: `local.yaml` was removed from the repository, along with `get_role` and `yamlinfo`, which are all now useless. WHOOHOO! == Next step: hoster.yaml the next chunk we need to convert would be, i think, `./modules/torproject_org/misc/hoster.yaml`, which specifies those things: * `netrange`: used to create the `TPO_NET` macro in ferm (unused?) and determine in which `hoster` a given host is (through `whohosts.rb`, which does IP range calculations from the host's IP as seen from LDAP) * `mirror-debian`: used in `torproject_org` class to define the APT mirror for this host * `mirror-debian-security`: unused? * `nameservers`: used to configured upstream forwarders in unbound on each host * `nameservers_break_dnssec` : used to disable unbound forwarding in case of broken upstream DNS, unused * `allow_dns_query`: used to tell unbound to allow other network ranges (ie. generally on this site) to use *this* node as recursive DNS server (if `misc.resolver-recursive` is true, which is the case when the LDAP ip of the host is listed in the hoster's `nameservers` list) Debian.org has hooked hosters.yaml into hiera, and the way they did it is to have one .yaml file per hoster, for example: https://salsa.debian.org/dsa-team/mirror/dsa- puppet/blob/master/hieradata/bytemark.yaml Unfortunately, the `hoster.yaml` is still present on d.o: https://salsa.debian.org/dsa-team/mirror/dsa- puppet/blob/master/modules/puppetmaster/lib/puppet/parser/functions/whohosts.rb there we have the same code as on tpo: {{{ yamlfile = Puppet::Parser::Files.find_file('debian_org/misc/hoster.yaml', compiler.environment) }}} Here the file, which contains more than just ip ranges: https://salsa.debian.org/dsa-team/mirror/dsa- puppet/blob/master/modules/debian_org/files/misc/hoster.yaml So their transition isn't complete, but it matches some of the ideas I had (namely to have one YAML file per hoster). So what I would suggest we do to get rid of the hoster.yaml file is this: 1. convert all the aforementioned variables used in `hoster.yaml` into class variables, defaulting to the values loaded from `hoster.yaml` (ie. `$nodeinfo`) 2. test the variables by overriding them in (e.g.) `hiera/nodes/foo.example.com.yaml` 3. break up the `hoster.yaml` file into multiple smaller files in `hiera/hoster/%{hoster}.yaml` 4. add that path to `hiera.yaml` 5. test that host can load its variables from the `hoster` search path by hardcoding a value by hand in facter 6. create a new YAML variable that gives us a IP range -> hoster mapping 7. create a function that looks through those to guess the hoster for a given IP address 8. use that function to create a fact (through a template, but with a variable defined in the base class) that defines the `$hoster` variable that hiera will use to load the right YAML 9. remove `hoster.yaml` That's a first step. At that stage, `hoster.yaml` is gone, but `$nodeinfo` remains and still might contain host-specific configuration. Those should be extract *out* of the `$nodeinfo` construct and into manifest business logic. And *then* the `nodeinfo.rb` code can be ripped out. There might be a better way to define a hoster per node than guess it with its IP address and drop it as a fact, but I can't think of anything right now. -- Ticket URL: <https://trac.torproject.org/projects/tor/ticket/30020#comment:10> Tor Bug Tracker & Wiki <https://trac.torproject.org/> The Tor Project: anonymity online
_______________________________________________ tor-bugs mailing list tor-bugs@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs