This is an automated email from the ASF dual-hosted git repository. zwoop pushed a commit to branch 9.1.x in repository https://gitbox.apache.org/repos/asf/trafficserver.git
commit ee755b5adeb06437d4a0d03887a9bbe167fe4aa2 Author: Walt Karas <[email protected]> AuthorDate: Tue Feb 2 11:49:57 2021 -0600 Add command line utility to help convert remap plugin usage to ATS9. (#7426) --- doc/appendices/command-line/cvtremappi.en.rst | 69 ++++ tools/cvtremappi | 575 ++++++++++++++++++++++++++ tools/insnew | 67 +++ 3 files changed, 711 insertions(+) diff --git a/doc/appendices/command-line/cvtremappi.en.rst b/doc/appendices/command-line/cvtremappi.en.rst new file mode 100644 index 0000000..264d474 --- /dev/null +++ b/doc/appendices/command-line/cvtremappi.en.rst @@ -0,0 +1,69 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +.. include:: ../../common.defs + +.. _cvtremappi: + +cvtremappi +********** + +Description +=========== + +To help convert your remapping configuration from pre-ATS9 to ATS9 and later. It may be useful if you use any +of the core plugins regex_remap.so, header_rewrite.so or gzip.so. (For this script to work, the python3 +command has to be in your path.) You can specify where your remap configuration file is with the option: + +--filepath FILEPATH + +If this parameter is omitted, it defaults to ``./remap.config`` . The script will make necessary modifications +to this file, and any files it includes with ``.include`` . It will change `@plugin=gzip.so` to its new name, +`@plugin=compress.so` . When regex_remap.so is invoked as the first remap plugin, it will add the parameter +@pparam=pristine . (This makes it work the same as in pre-9 ATS, where the request URL is the pre-remapping +URL for the first plugin for a remap rule.) When `header_rewrite.so` is used as a remap plugin, no changes +are needed in the remap configuration line invoking it. However, changes may be necessary to the +configuration files passed to it as parameters. If a header rewrite configuration file is used for both the +invocation of header rewrite as the first plugin for remap rules, and for other invocations, it may be +necessary to generate two new versions of it. In these cases, the prefix `1st-` is added to file's name, +for the version used with header rewrite as the first plugin. If you prefer that a different prefix be added, +you can specify it with this option: + +--prefix PREFIX + +If you are also using header rewrite as a global plugin, you should also provide the filepath of the global +plugin configuration file with this option: + +--plugin PLUGIN + +(Note that, if the PLUGIN filepath is relative, it should be relative to the directory containing the remap +configuration file, not relative to the directory the script is run from. Note also that, if relative paths +for include files for header rewrite config files appear in the configuration files, they are assumed to be +relative to the directory containing the remap configuration file.) + +Header rewrite previously had some logic that has been eliminated in ATS9. If a line in a header rewrite +configuration file relies on this deprecated logic, an error message will be output to standard error. The +text `ERROR:` will be prepended to the line in the configuration file causing the error. + +The script writes, one per line, a list of the files it is changing or creating to the standard output. But +both new and changed files will be written into entirely new files with the suffix `.new` added to the filepath. +For example, if `remap.config` is changed by the script, it will put the changed version of the file in +`remap.config.new` . This gives you a chance to review the changes the script has made. You can then put the +changed files into effect with the tool script `insnew`. This script reads a list of filepaths, one per line, +from the standard input. For each filepath `FP`, if it specifies an existing file, it will rename it to +`FP.old`. It will then rename the file `FP.new` to `FP`. This second script should be run from the same +current directory as the first script was run from. diff --git a/tools/cvtremappi b/tools/cvtremappi new file mode 100755 index 0000000..290a3b5 --- /dev/null +++ b/tools/cvtremappi @@ -0,0 +1,575 @@ +#!/usr/bin/env python3 + +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Automatically convert remapping configuration for use of header_rewrite.so, regex_remap.so and gzip.so +# plugins from pre-ATS9 to ATS9 and later. (See docs.trafficserver.apache.org, Command Line Utilities +# Appendix.) + +import argparse + +BACKSLASH = "\\" + +parser = argparse.ArgumentParser( + prog="cvt7to9", + description= "Convert remap configuration from ATS7 to ATS9" +) +parser.add_argument("--filepath", default="remap.config", help="path specifier of remap config file") +parser.add_argument("--prefix", default="1st-", help="prefix for new header_rewrite config files") +parser.add_argument( + "--plugin", default=None, help="path specifier (relative to FILEPATH) of (global) plugins config file" +) +args = parser.parse_args() + +import sys + +def eprint(*args, **kwargs): + print(*args, file=sys.stderr, **kwargs) + +# Prefix to add to file relative pathspecs before opening file. +# +pathspec_prefix = None + +import os +import copy + +# Remap or header rewrite config file. +# +class CfgFile: + def __init__(self, param): + if isinstance(param, CfgFile): + # Making copy of header rewrite config file to use when header rewrite is first plugin for remap + # rule. + # + self.changed = True + d = os.path.dirname(param.pathspec) + if (d != "") and (d != "/"): + d += "/" + self.pathspec = d + args.prefix + os.path.basename(param.pathspec) + self.lines = copy.copy(param.lines) + + else: + # Opening existing config file, param is pathspec. + # + self.changed = False + self.pathspec = param + ps = param + if ps[:1] != "/": + ps = pathspec_prefix + ps + try: + fd = open(ps, "r") + except: + eprint(f"fatal error: failure opening {ps} for reading") + sys.exit(1) + + try: + self.lines = fd.readlines() + except: + eprint(f"fatal error: failure reading {ps}") + sys.exit(1) + + try: + fd.close() + except: + eprint(f"fatal error: failure closing {ps}") + sys.exit(1) + + # Write out new version of file if it's contents are new. ".new" is appended for the pathspec to get the + # pathspec of the new file. + # + def close(self): + if self.changed: + ps = self.pathspec + if ps[:1] != "/": + ps = pathspec_prefix + ps + print(ps) + ps += ".new" + try: + fd = open(ps, "w") + except: + eprint(f"fatal error: failure opening {ps} for writing") + sys.exit(1) + + for ln in self.lines: + if ln != None: + try: + fd.write(ln) + except: + eprint(f"fatal error: failure writing {ps}") + sys.exit(1) + + try: + fd.close() + except: + eprint(f"fatal error: failure closing {ps}") + sys.exit(1) + +# A generic object. +# +class Obj: + def __init__(self): + pass + +# A dictionary that is a mapping from pathspecs for header rewrite config files to generic objects. If the +# object has a "first" attribute, it is a list of 2-tuples. The first tuple entry is a reference to the +# the CfgFile object for a remap config file. The second tuple entry is the index into the list of lines, +# with the index of the line where header_rewrite.so is the first remap plugin, and the header rewrite +# config file is a parameter to it. If "other" is an attribute of the generic object, the config file is +# used as a parameter to calls to a global instance of header_rewrite.so, or instances of header_rewrite.so on +# a remap rule as the second or later remap plugin. (The value of the "other" attribute is always None because +# the value doesn't matter.) +# +header_rewrite_cfgs = dict() + +# Read in (global) header rewrite config file pathspecs from plugin.config file. +# +def handle_global(gbl_pathspec): + + if gbl_pathspec[:1] != "/": + gbl_pathspec = pathspec_prefix + gbl_pathspec + + try: + fd = open(gbl_pathspec, "r") + except: + eprint(f"fatal error: failure opening {gbl_pathspec} for reading") + sys.exit(1) + + try: + lines = fd.readlines() + except: + eprint(f"fatal error: failure reading {gbl_pathspec}") + sys.exit(1) + + try: + fd.close() + except: + eprint(f"fatal error: failure closing {gbl_pathspec}") + sys.exit(1) + + ln_num = 0 + while ln_num < len(lines): + ln = lines[ln_num] + + # Join continuation lines to the first line. + # + num_ln_joined = 1 + while ((ln_num + num_ln_joined) < len(lines)) and (ln[-2:] == (BACKSLASH + "\n")): + ln[:-2] += " " + lines[ln_num + num_ln_joined] + num_ln_joined += 1 + + ofst = ln.find("#") + if ofst >= 0: + # Remove comment. + # + ln = ln[:ofst] + + ln = ln.split() + + if (len(ln) > 0) and (ln[0] == "header_rewrite.so"): + for param in ln[1:]: + hr_obj = Obj() + hr_obj.other = None + header_rewrite_cfgs[param] = hr_obj + break + + ln_num += num_ln_joined + +# A list of CfgFile instances for the main remap config file and any include files it contains, directly or +# indirectly. +# +remap_cfgs = [] + +# Handle the remap config file, and call this recursively to handle include files. +# +def handle_remap(filepath): + + def skip_white(a_str): + if a_str[:1].isspace(): + return 1 + elif a_str.startswith(BACKSLASH + "\n"): + return 2 + return 0 + + rc = CfgFile(filepath) + + remap_cfgs.append(rc) + + ln_num = 0 + while ln_num < len(rc.lines): + ln = rc.lines[ln_num] + + # Join continuation lines to the first line, and make them None in the rc.lines array. + # + num_ln_joined = 1 + while ((ln_num + num_ln_joined) < len(rc.lines)) and (ln[-2:] == (BACKSLASH + "\n")): + ln += rc.lines[ln_num + num_ln_joined] + rc.lines[ln_num + num_ln_joined] = None + num_ln_joined += 1 + + rc.lines[ln_num] = ln + + len_content = ln.find("#") + if len_content < 0: + len_content = len(ln) - 1 + + # Only process lines with content. + # + if (len_content > 0) and not ln[:len_content].isspace(): + ofst = ln.find(".include") + if (ofst == 0) or ((ofst > 0) and ln[:ofst].isspace()): + ofst += len(".include") + start_ps = -1 + while ofst < len_content: + sw = skip_white(ln[ofst:]) + if sw == 0: + if start_ps < 0: + start_ps = ofst + ofst += 1 + else: + if start_ps >= 0: + handle_remap(ln[start_ps:ofst]) + start_ps = -1 + ofst += sw + + if start_ps >= 0: + handle_remap(ln[start_ps:len_content]) + else: + # First, gzip.so -> compress.so + # + lnc = ln[:len_content] + lncr = lnc.replace("@plugin=gzip.so", "@plugin=compress.so") + if lncr != lnc: + ln = lncr + ln[len_content:] + len_content = len(lncr) + rc.lines[ln_num] = ln + rc.changed = True + + ofst = ln[:len_content].find("@plugin=") + if ofst >= 0: + # Assuming it's some sort of remap line since it's got @plugin. + # + ofst += len("@plugin=") + + new_ln = None + + if ln[ofst:].startswith("header_rewrite.so"): + ofst += len("header_rewrite.so") + + while ofst < len_content: + ofst2 = ln[ofst:len_content].find("@") + if ofst2 < 0: + break; + + ofst += ofst2 + + if not ln[ofst:].startswith("@pparam="): + break + + ofst += len("@pparam=") + + ofst2 = ofst + while ofst2 < len_content: + sw = skip_white(ln[ofst2:]) + if sw != 0: + break + ofst2 += 1 + + hr_pathspec = ln[ofst:ofst2] + ofst = ofst2 + + if hr_pathspec not in header_rewrite_cfgs: + hr_obj = Obj() + header_rewrite_cfgs[hr_pathspec] = hr_obj; + else: + hr_obj = header_rewrite_cfgs[hr_pathspec] + + if not hasattr(hr_obj, "first"): + hr_obj.first = [] + + hr_obj.first.append((rc, ln_num)) + + elif ln[ofst:].startswith("regex_remap.so"): + ofst += len("regex_remap.so") + + ofst2 = ln[ofst:len_content].find("@plugin=") + + if ofst2 < 0: + ofst2 = len_content + else: + ofst2 += ofst + + if ((ln[ofst:ofst2].find("@pparam=pristine") < 0) and + (ln[ofst:ofst2].find("@pparam=no-pristine") < 0)): + + new_ln = ln[:ofst2] + + if not new_ln[-1].isspace(): + new_ln += " " + + new_ln += "@pparam=pristine" + + if not ln[ofst2].isspace(): + new_ln += " " + + ofst = len(new_ln) + + new_ln += ln[ofst2:] + + else: + ofst = ofst2 + + if new_ln: + rc.lines[ln_num] = new_ln + rc.changed = True + len_content += len(new_ln) - len(ln) + ln = new_ln + + # Handle it if header rewrite is called as a second or later plugin. + # + while ofst < len_content: + ofst2 = ln[ofst:len_content].find("@plugin=header_rewrite.so") + + if ofst2 < 0: + break + + ofst += ofst2 + len("@plugin=header_rewrite.so") + + while ofst < len_content: + ofst2 = ln[ofst:len_content].find("@") + if ofst2 < 0: + break + + ofst += ofst2 + 1 + + if not ln[ofst:].startswith("pparam="): + ofst -= 1 + break + + ofst += len("pparam=") + + ofst2 = ofst + while ofst2 < len_content: + sw = skip_white(ln[ofst2:]) + if sw != 0: + break + ofst2 += 1 + + hr_pathspec = ln[ofst:ofst2] + ofst = ofst2 + + if hr_pathspec not in header_rewrite_cfgs: + hr_obj = Obj() + header_rewrite_cfgs[hr_pathspec] = hr_obj; + else: + hr_obj = header_rewrite_cfgs[hr_pathspec] + + hr_obj.other = None # Only the existence of this attr matters, not its value. + + ln_num += num_ln_joined + +def handle_header_rewrite(pathspec, obj): + + # This class manages changes to a header rewrite config file. + # + class HRCfg: + # 'pathspec' is the key in header_rewrite_cfgs and 'obj' is the generic object value. + # + def __init__(self, pathspec, obj): + self._base = CfgFile(pathspec) + self._obj = obj + + def has_first(self): + return hasattr(self._obj, "first") + + def has_other(self): + return hasattr(self._obj, "other") + + def get_lines(self): + return self._base.lines + + # Assumes has_first is true. + # + def chg_ln_first(self, ln_num, ln): + if not hasattr(self, "_first"): + if self.has_other(): + # Must make two versions of header rewrite config file. + # + self._first = CfgFile(self._base) + else: + self._first = self._base + self._first.changed = True + + self._first.lines[ln_num] = ln + + # Assumes has_other is true. + # + def chg_ln_cmn(self, ln_num, ln): + if hasattr(self, "_first") and not (self._first is self._base): + self._first.lines[ln_num] = ln + + self._base.lines[ln_num] = ln + self._base.changed = True + + def close(self): + self._base.close() + if hasattr(self, "_first") and not (self._first is self._base): + self._first.close() + + old_pparam = "@pparam=" + self._base.pathspec + + # Backpatch remap configuration files with pathspec of new header rewrite config file for call + # to header rewrite as first remap plugin. + # + for pair in self._obj.first: + rc = pair[0] + rc.changed = True + ln = rc.lines[pair[1]] + ofst = ln.find(old_pparam) + len("@pparam=") + rc.lines[pair[1]] = ln[:ofst] + self._first.pathspec + ln[ofst + len(self._base.pathspec):] + + # Returns a pair: length of string before # comment, and a flag indicating if %< (beginning of a variable + # exapansion) was in any of the double-quoted strings. + # + def get_content_len(s): + # States. + # + COPYING = 0 + ESCAPE_COPYING = 1 + IN_QUOTED = 2 + ESCAPE_IN_QUOTED = 3 + PERCENT_IN_QUOTED = 4 + + state = COPYING + ofst = 0 + variable_expansion = False + + while ofst < (len(s) - 1): + if state == COPYING: + if s[ofst] == "#": + # Start of comment. + # + return (ofst, variable_expansion) + elif s[ofst] == '"': + state = IN_QUOTED + else: + if s[ofst] == BACKSLASH: + state = ESCAPE_COPYING + + elif state == IN_QUOTED: + if s[ofst] == BACKSLASH: + state = ESCAPE_IN_QUOTED + elif s[ofst] == '"': + state = COPYING + elif s[ofst] == '%': + state = PERCENT_IN_QUOTED + + elif state == ESCAPE_COPYING: + state = COPYING + + elif state == ESCAPE_IN_QUOTED: + if s[ofst] == '%': + state = PERCENT_IN_QUOTED + else: + state = IN_QUOTED + + elif state == PERCENT_IN_QUOTED: + if s[ofst] == '<': + variable_expansion = True + state = IN_QUOTED + elif s[ofst] == BACKSLASH: + state = ESCAPE_IN_QUOTED + elif s[ofst] == '"': + state = COPYING + elif s[ofst] == '%': + state = PERCENT_IN_QUOTED + else: + state = IN_QUOTED + + ofst += 1 + + return (ofst, variable_expansion) + + hrc = HRCfg(pathspec, obj) + lines = hrc.get_lines() + ln_num = 0 + while ln_num < len(lines): + ln = lines[ln_num] + prepend_error = False + + len_content, variable_expansion = get_content_len(ln) + + if variable_expansion: + eprint( + f"error: {pathspec}, line {ln_num + 1}: variable expansions cannot be automatically converted" + ) + prepend_error = True + + lnc = ln[:len_content] + lnc = lnc.replace("%{CLIENT-IP}", "%{INBOUND:REMOTE-ADDR}") + lnc = lnc.replace("%{INCOMING-PORT}", "%{INBOUND:LOCAL-PORT}") + lnc = lnc.replace("%{PATH}", "%{URL:PATH}") + lnc = lnc.replace("%{QUERY}", "%{URL:QUERY}") + + if hrc.has_other() and (prepend_error or (lnc != ln[:len_content])): + if prepend_error: + hrc.chg_ln_cmn(ln_num, "ERROR: " + lnc + ln[len_content:]) + else: + hrc.chg_ln_cmn(ln_num, lnc + ln[len_content:]) + + if hrc.has_first(): + lnc_save = lnc + ofst = lnc.find("set-destination") + if (ofst >= 0): + if (ofst == 0) or lnc[:ofst].isspace(): + ofst += len("set-destination") + ofst2 = lnc[ofst:].find("PATH") + if (ofst2 > 0) and lnc[ofst:(ofst + ofst2)].isspace(): + eprint( + f"error: {pathspec}, line {ln_num + 1}: the functionality of set-destination PATH" + + " for the first remap plugin does not exist in ATS9" + ) + prepend_error = True + + lnc = lnc.replace("%{URL:", "%{CLIENT-URL:") + if lnc != lnc_save or prepend_error: + if prepend_error: + hrc.chg_ln_first(ln_num, "ERROR: " + lnc + ln[len_content:]) + else: + hrc.chg_ln_first(ln_num, lnc + ln[len_content:]) + + ln_num += 1 + + hrc.close() + +pathspec_prefix = os.path.dirname(args.filepath) +if (pathspec_prefix != "") and (pathspec_prefix != "/"): + pathspec_prefix += "/" + +if not (args.plugin is None): + handle_global(args.plugin) + +handle_remap(os.path.basename(args.filepath)) + +for pathspec in header_rewrite_cfgs: + handle_header_rewrite(pathspec, header_rewrite_cfgs[pathspec]) + +# loop through remap_cfgs and close each one. +for rc in remap_cfgs: + rc.close() + +sys.exit(0) diff --git a/tools/insnew b/tools/insnew new file mode 100755 index 0000000..94b1c8d --- /dev/null +++ b/tools/insnew @@ -0,0 +1,67 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# For each filespec FS given (one per line) on stdin, rename FS (if it exists) to FS.old, then rename FS.new to +# FS. + +if [[ "$*" = "--help" ]] ; then + echo "For each filespec FS read from stdin, rename FS.new to FS. If FS already exists, rename to FS.old" >&2 + exit 0 +fi + +while read F +do +python3 - "$F" << THE_END + +import sys +import os + +def eprint(*args, **kwargs): + print(*args, file=sys.stderr, **kwargs) + +def rename_file(old_path, new_path): + try: + os.rename(old_path, new_path) + except: + eprint(f"fatal error: failure renaming {old_path} to {new_path}\n") + sys.exit(1) + +def rename_to_old(pathspec): + op = pathspec + ".old" + + if os.access(op, os.F_OK): + last = 2 + while os.access(op + ".{}".format(last), os.F_OK): + last += 1 + while last > 2: + rename_file(op + ".{}".format(last - 1), op + ".{}".format(last)) + last -= 1 + rename_file(op, op + ".2") + + rename_file(pathspec, op) + +filespec = sys.argv[1] +if os.access(filespec, os.F_OK): + rename_to_old(filespec) +rename_file(filespec + ".new", filespec) + +sys.exit(0) + +THE_END +done
