rob05c commented on a change in pull request #4628: Add ORT Rewrite Blueprint URL: https://github.com/apache/trafficcontrol/pull/4628#discussion_r406975554
########## File path: blueprints/ort-rewrite-unix-style.md ########## @@ -0,0 +1,210 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> +# ORT Rewrite in UNIX Philosophy + +## Problem Description +ORT is: +- Difficult to maintain. Writing Perl is difficult, and reading it is even more difficult. +- Dangerous to modify. Perl is not compiled, and even validity checks (`perl –c`) fail to verify dynamic runtime errors. This makes it very easy to introduce a bug in seldom-executed areas. +- Untested. Perl ORT has no unit or integration tests. +- Opaque. Nobody really knows everything it does, or when, or why. + +## Proposed Change + +ORT will be rewritten into a series of standalone executables, in the "UNIX Philosophy" + +> 1. Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new "features". +> 2. Expect the output of every program to become the input to another, as yet unknown, program. Don't clutter output with extraneous information. Avoid stringently columnar or binary input formats. Don't insist on interactive input. + +- Each executable should do exactly 1 thing, and if a new "thing" becomes necessary, a new executable will be created. +- The input and output of executables should be text which is easily parseable, so the executables can easily be pipelined (passing the output of one to the input of another), as well as easily read by humans and manipulated by standard Linux/POSIX tools. + +This makes ORT: +- Easier to maintain. Each binary does one thing, is much smaller, and is more obvious. Presumably they’re also written in a language easier to read and write, such as Go. +- Safer to modify. If each component is smaller, it’s more obvious what it does. We also presume the apps will be written with good development practices (such as modularization), with a language which verifies more at compile-time, and with tests. +- Clear and easy for operators to understand what each app does. We assume clean interfaces, and good documentation (ideally in the app itself, via help flags, printing usage when no arguments are received, and/or man pages). + +#### Implementation + +The implementation should adhere to the "UNIX Philosophy," POSIX, Linux Standard Base (LSB), and GNU as much as possible. + +ORT will continue to consist of a single OS package (e.g. RPM), which installs all executables. + +ORT will require the following executables: +- **Aggregator**. This is the “primary application” which will emulate the existing ORT script, and be called by CRON or operators to deploy all configs, as ORT does today. Note this is similar to how git works, and several other common Linux CLI utilities. + This app will have no logic itself, except to call the other executables. + - INPUT: configuration and specification to fetch and emplace config files. + - BEHAVIOR: fetches and places config files + - OUTPUT: success or failure message + +- **Traffic Ops Requestor**. This will fetch data needed from Traffic Ops, such as the Update Pending flag, packages, etc. This should never modify TO data, and should be guaranteed read-only. Any status modifications should go in the Traffic Ops Updater. + - INPUT: Traffic Ops URL and credentials, and data to fetch + - BEHAVIOR: Requests data from Traffic Ops + - OUTPUT: Traffic Ops data requested + - Format is probably multipart/mixed, but format may be different if a better format is determined. Ideal "UNIX Philosophy" format is line-delimited text, but the complexity may preclude that. The more complex and difficult to parse, the further from the "UNIX Philosophy." E.g. multipart/mixed is preferable to JSON. Review comment: IMO JSON on the CLI strongly violates the UNIX Philosophy. Unix apps should return columnar text, for ease of parsing, and to interoperate with other tools. multipart is very not-ideal either; but we need a way to delimit files. You can almost do two newlines, but you at least need the metadata of the file name, and other metadata is really useful too. Which, if you were making a custom format, leads you to something very similar to multipart. JSON is a tree. It's nice that it's a simple language, but trees are drastically more complex than lines. I wish there were something closer to the standard UNIX line-delimiting, but I don't think there is. Any tree format is way more complex to parse, search, hold in memory, etc. It basically makes operating on with the standard UNIX toolset (awk/grep/sed/etc) impossible, without `jq`. Where multipart is at least possible to use the standard tools on, if slightly more complex from the two delimiter levels. You're right, JSON is more popular than multipart. But multipart is really just a delimiter and headers, it's super-easy to parse. And most languages do have libraries for it. It took me maybe an hour to do it in Perl, language I'm very much not an expert in: https://github.com/apache/trafficcontrol/blob/5693451bb898f62d0e14afa60a9af9824d3c8e06/traffic_ops/ort/traffic_ops_ort.pl#L1600 And e.g. Go has libraries for it: https://golang.org/pkg/mime/multipart/ I could not write a JSON parser in an hour. And no, I won't need to. But, it shouldn't even be a possibility. _Whatever_ has to be parsed should be as easy, human readable, and line-delimited as possible. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
