Re: [tor-bugs] #30881 [Internal Services/Tor Sysadmin Team]: answer the opsreportcard questionnaire, AKA the "limoncelli test"

[Tor Bug Tracker & Wiki]

#30881: answer the opsreportcard questionnaire, AKA the "limoncelli test"
-+-
 Reporter:  anarcat  |  Owner:  anarcat
 Type:  task | Status:  closed
 Priority:  Medium   |  Milestone:
Component:  Internal Services/Tor Sysadmin Team  |Version:
 Severity:  Normal   | Resolution:  fixed
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by anarcat):

 i designed a service docs template in here:

 https://help.torproject.org/tsa/howto/template/

 it's quite exhaustive and most documentation don't have all fields, but it
 gives us a good thing to copy-paste from.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30881 [Internal Services/Tor Sysadmin Team]: answer the opsreportcard questionnaire, AKA the "limoncelli test"

[Tor Bug Tracker & Wiki]

#30881: answer the opsreportcard questionnaire, AKA the "limoncelli test"
-+-
 Reporter:  anarcat  |  Owner:  anarcat
 Type:  task | Status:  closed
 Priority:  Medium   |  Milestone:
Component:  Internal Services/Tor Sysadmin Team  |Version:
 Severity:  Normal   | Resolution:  fixed
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-
Changes (by anarcat):

 * status:  assigned => closed
 * resolution:   => fixed


Comment:

 i think this ticket is done insofar as we've answered the questionaire.
 there is still a lot of work to be done to get a 100% score, but that will
 take a long time to achieve, if ever. for now, let's consider this done
 and it can be kept as future reference for quiet times when we want to get
 started on new projects.

 children tickets have been detached from this ticket so it can be closed,
 but they are linked in the summary.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30881 [Internal Services/Tor Sysadmin Team]: answer the opsreportcard questionnaire, AKA the "limoncelli test"

[Tor Bug Tracker & Wiki]

#30881: answer the opsreportcard questionnaire, AKA the "limoncelli test"
-+-
 Reporter:  anarcat  |  Owner:  anarcat
 Type:  task | Status:
 |  assigned
 Priority:  Medium   |  Milestone:
Component:  Internal Services/Tor Sysadmin Team  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by anarcat):

 = Summary

  * Section A: Public Facing Practices: 1.5/3 (50%)
  * Section B: Modern Team Practices: 3.5/7 (50%)
  * Section C: Operational Practices: 0.5/5 (10%)
  * Section D: Automation Practices: 1.5/3 (50%)
  * Section E: Fleet Management Processes: 2.5/4 (63%)
  * Section F: Disaster Preperation Practices: 4/5 (80%)
  * Section G: Security Practices: 0.5/5 (10%)

 = Final score: 14/32 (44%)

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30881 [Internal Services/Tor Sysadmin Team]: answer the opsreportcard questionnaire, AKA the "limoncelli test"

[Tor Bug Tracker & Wiki]

#30881: answer the opsreportcard questionnaire, AKA the "limoncelli test"
-+-
 Reporter:  anarcat  |  Owner:  anarcat
 Type:  task | Status:
 |  assigned
 Priority:  Medium   |  Milestone:
Component:  Internal Services/Tor Sysadmin Team  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by anarcat):

 = Section G: Security Practices

 == 28. Do Desktops/laptops/servers run self-updating, silent, anti-malware
 software?

 No.

 == 29. Do you have a written security policy?

 No. See http://www.sans.org/security-resources/policies/ for an example.

 == 30. Do you submit to periodic security audits?

 No.

 == 31. Can a user's account be disabled on all systems in 1 hour?

 Yes, through LDAP, although some services are not directly hooked into
 LDAP.

 == 32. Can you change all privileged (root) passwords in 1 hour?

 No.

 = Score: 0.5/5

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30881 [Internal Services/Tor Sysadmin Team]: answer the opsreportcard questionnaire, AKA the "limoncelli test"

[Tor Bug Tracker & Wiki]

#30881: answer the opsreportcard questionnaire, AKA the "limoncelli test"
-+-
 Reporter:  anarcat  |  Owner:  anarcat
 Type:  task | Status:
 |  assigned
 Priority:  Medium   |  Milestone:
Component:  Internal Services/Tor Sysadmin Team  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by anarcat):

 = F. Disaster Preparation Practices

 == 23. Can your servers keep operating even if 1 disk dies?

 http://opsreportcard.com/section/23

 Yes, of course we have RAID-1 everywhere, and the new cluster has DRBD on
 top of *that*. The reportcard suggests there are possible exceptions for
 this, but we make none.

 == 24. Is the network core N+1?

 http://opsreportcard.com/section/24

 We generally do not manage our own network and that is delegated upstream,
 so yes, in a way.

 == 25. Are your backups automated?

 http://opsreportcard.com/section/25

 Yes.

 == 26. Are your disaster recovery plans tested periodically?

 http://opsreportcard.com/section/26

 What's a disaster recovery plan?

 == 27. Do machines in your data center have remote power / console access?

 http://opsreportcard.com/section/27

 Yes, mostly.

 == Score: 4/5

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30881 [Internal Services/Tor Sysadmin Team]: answer the opsreportcard questionnaire, AKA the "limoncelli test"

[Tor Bug Tracker & Wiki]

#30881: answer the opsreportcard questionnaire, AKA the "limoncelli test"
-+-
 Reporter:  anarcat  |  Owner:  anarcat
 Type:  task | Status:
 |  assigned
 Priority:  Medium   |  Milestone:
Component:  Internal Services/Tor Sysadmin Team  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by anarcat):

 = Section E: Fleet Management Processes

 == 19. Is there a database of all machines?

 http://opsreportcard.com/section/19

 Yes, but it's somewhat spread around LDAP, Puppet and a spreadsheet.
 There's a ticket open to "improve the inventory" (#30273) which aims at
 solving the problem, possibly with the hope of merging everything in a
 single source of truth (most likely Puppet).

 == 20. Is OS installation automated?

 http://opsreportcard.com/section/19

 Somewhat. New installer scripts have been introduced for our various
 platforms and documentation has been established, but there's some work to
 be done to standardize the process further. See #31239.

 == 21. Can you automatically patch software across your entire fleet?

 http://opsreportcard.com/section/21

 We have a semi-automated process: there's a magic command that can be
 launched manually to perform upgrades over all affected machines,
 requiring approving each similar change manually.

 As for this recommendation:

 > When possible, updates should happen silently. If they require a reboot
 or other interruptions, users should have the ability to delay the update.
 However, there should be a limit; maybe 2 weeks. However the deadline
 should be adjustable so that emergency security fixes can happen sooner.

 ... it's not currently done. See #31957 for followup.

 == 22. Do you have a PC refresh policy?

 http://opsreportcard.com/section/22

 > If you don't have a policy about when PC will be replaced, they'll never
 be replaced. [By "PC" I mean the laptop and desktops that people use, not
 the servers.]

 Strangely, I believe this should also apply to servers, which the report
 card seems to assume are already covered.

 In our case, they are not. There was some work in Brussels to establish
 formal processes to manage the lifetime of systems, see #29304. There is
 also work underway to decommission old machines and replace them with
 newer ones. This crosses over the inventory work (#30272) as well.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30881 [Internal Services/Tor Sysadmin Team]: answer the opsreportcard questionnaire, AKA the "limoncelli test"

[Tor Bug Tracker & Wiki]

#30881: answer the opsreportcard questionnaire, AKA the "limoncelli test"
-+-
 Reporter:  anarcat  |  Owner:  anarcat
 Type:  task | Status:
 |  assigned
 Priority:  Medium   |  Milestone:
Component:  Internal Services/Tor Sysadmin Team  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by anarcat):

 = 4. Do you have a "policy and procedure" wiki?

 http://opsreportcard.com/section/4

 Yes, in [https://help.torproject.org/tsa/ help.tpo]. It might become a
 GitLab wiki in the future, but that's kind of an implementation detail at
 this point. It's good enough for now, but it *is* lacking some
 documentation. In particular:

  * #30880 - backup / restore procedures
  * emergency procedures (disk, machine replacement, intrusion, etc) - see
 https://help.torproject.org/tsa/howto/incident-response/ for now

 Consider separating the documentation in four categories:

  1. tutorials - simple, brainless step-by-step instructions requiring no
 or little technical background
  2. howtos - more in-depth procedure that may require interpretation
  3. reference - how things are built, explaining the complex aspects of
 the setup without going into "how to do things", policy decisions and so
 on
  4. discussion - *why* things are setup this way and *how else* they could
 have been built

 That separation comes from [https://www.divio.com/blog/documentation/ what
 nobody tells you about documentation].

 = 5. Do you have a password safe?

 http://opsreportcard.com/section/5

 Yes, we do.

 = 6. Is your team's code kept in a source code control system?

 http://opsreportcard.com/section/6

 Mostly. There are some ad-hoc scripts here and there, but everything is
 being committed into git and/or Puppet as much as possible.

 = 7. Does your team use a bug-tracking system for their own code?

 Yes, this bug tracker.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30881 [Internal Services/Tor Sysadmin Team]: answer the opsreportcard questionnaire, AKA the "limoncelli test"

[Tor Bug Tracker & Wiki]

#30881: answer the opsreportcard questionnaire, AKA the "limoncelli test"
-+-
 Reporter:  anarcat  |  Owner:  anarcat
 Type:  task | Status:
 |  assigned
 Priority:  Medium   |  Milestone:
Component:  Internal Services/Tor Sysadmin Team  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by anarcat):

 = 3. Does the team record monthly metrics?

 http://opsreportcard.com/section/3

 Somewhat. We now have a Prometheus server that records lots of information
 on the TPA machines, but it doesn't store information beyond one month. It
 also doesn't record more high-level metrics like:

  * how many machines do we have
  * how many support tickets we deal with
  * how many people on staff
  * etc

 The monitoring systems also collect a *lot* of metrics and it might be
 worth creating a dashboard with the most important ones for our purposes,
 to get a bird eye's view of everything.

 Cute dashboard doesn't seem like high priority, but I've created a ticket
 for long-term prometheus storage in #31244 at least, so that we can create
 a dashboard that looks further back in time in the future eventually.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30881 [Internal Services/Tor Sysadmin Team]: answer the opsreportcard questionnaire, AKA the "limoncelli test"

[Tor Bug Tracker & Wiki]

#30881: answer the opsreportcard questionnaire, AKA the "limoncelli test"
-+-
 Reporter:  anarcat  |  Owner:  anarcat
 Type:  task | Status:
 |  assigned
 Priority:  Medium   |  Milestone:
Component:  Internal Services/Tor Sysadmin Team  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by anarcat):

 = 2. Are "the 3 empowering policies" defined and published?

 http://opsreportcard.com/section/2

 Specifically, this is three questions:

 == How do users get help?

 Right now, this is unofficially "open a ticket in Trac", "ping us over IRC
 for small stuff", or "write us an email". This could be made more official
 somewhere.

 == What is an emergency?

 I am not sure this is formally defined.

 == What is supported?

 We have the distinction between systems and service admins. We did
 
[https://trac.torproject.org/projects/tor/wiki/org/meetings/2019Stockholm/Notes/SysadminTeamRoadmapping
 talk in Stockholm] about clarifying that item, so this is worth expanding
 further.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30881 [Internal Services/Tor Sysadmin Team]: answer the opsreportcard questionnaire, AKA the "limoncelli test"

[Tor Bug Tracker & Wiki]

#30881: answer the opsreportcard questionnaire, AKA the "limoncelli test"
-+-
 Reporter:  anarcat  |  Owner:  anarcat
 Type:  task | Status:
 |  assigned
 Priority:  Medium   |  Milestone:
Component:  Internal Services/Tor Sysadmin Team  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by anarcat):

 = Question 1. Are user requests tracked via a ticket system?

 http://opsreportcard.com/section/1

 The answer is: mostly yes. Most requests can be tracked here, in Trac, but
 some requests *do* come by email, on torproject-admin@tpo, and some of
 those *are* a little more difficult to track. Furthermore, that alias gets
 a lot of noise from servers, as root@ aliases are redirected there.

 Because Trac is public, we don't have a good way of tracking requests that
 should be private as well.

 Recommendation, as discussed in Stockholm: start experimenting with
 triaging root@ emails to RT, and possibly the rest of torproject-admin to
 RT as well.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

[tor-bugs] #30881 [Internal Services/Tor Sysadmin Team]: answer the opsreportcard questionnaire, AKA the "limoncelli test"

[Tor Bug Tracker & Wiki]

#30881: answer the opsreportcard questionnaire, AKA the "limoncelli test"
-+-
 Reporter:  anarcat  |  Owner:  anarcat
 Type:  task | Status:  assigned
 Priority:  Medium   |  Milestone:
Component:  Internal Services/Tor Sysadmin   |Version:
  Team   |
 Severity:  Normal   |   Keywords:
Actual Points:   |  Parent ID:
   Points:   |   Reviewer:
  Sponsor:   |
-+-
 Tom Limoncelli is the reknowned author of [https://www.tomontime.com/ Time
 management for sysadmins] and [https://the-sysadmin-book.com/ practice of
 network and system administration], two excellent books I recommend every
 sysadmin reads attentively.

 He made up a [https://everythingsysadmin.com/the-test.pdf 32-question
 test] (PDF, website version on [http://opsreportcard.com/
 opsreportcard.com] or the
 [http://web.archive.org/web/20120827040816/http://everythingsysadmin.com:80
 /the-test.html previous one-page HTML version]) that covers the basic of a
 well-rounded setup. I believe we will get a good score, but going through
 the list will make sure we don't miss anything.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs