This is an automated email from the ASF dual-hosted git repository. cmcfarlen pushed a commit to branch 10.0.x in repository https://gitbox.apache.org/repos/asf/trafficserver.git
commit 4762fe53736951a72e1af4037ea0b7f2d2304a89 Author: Leif Hedstrom <[email protected]> AuthorDate: Sun May 19 19:51:34 2024 -0600 Continued updates to the Cripts docs (#11365) (cherry picked from commit 4bfae93db73be12b2734151ebf0e0ffbe8f7bc26) --- doc/developer-guide/cripts/cripts-bundles.en.rst | 45 +++ .../cripts/cripts-connections.en.rst | 110 +++++ doc/developer-guide/cripts/cripts-crypto.en.rst | 90 +++++ doc/developer-guide/cripts/cripts-headers.en.rst | 8 +- doc/developer-guide/cripts/cripts-matcher.en.rst | 89 ++++ doc/developer-guide/cripts/cripts-misc.en.rst | 187 ++++++++- doc/developer-guide/cripts/cripts-overview.en.rst | 29 +- doc/developer-guide/cripts/cripts-urls.en.rst | 3 + doc/developer-guide/cripts/cripts-variables.en.rst | 37 +- doc/developer-guide/cripts/index.en.rst | 1 + include/cripts/Connections.hpp | 2 +- src/cripts/README.md | 446 --------------------- 12 files changed, 586 insertions(+), 461 deletions(-) diff --git a/doc/developer-guide/cripts/cripts-bundles.en.rst b/doc/developer-guide/cripts/cripts-bundles.en.rst index c50efe31db..d0230c22dc 100644 --- a/doc/developer-guide/cripts/cripts-bundles.en.rst +++ b/doc/developer-guide/cripts/cripts-bundles.en.rst @@ -24,3 +24,48 @@ Bundles ******* + +While developing Cripts, we realized that Cripts often repeat the same, +common patterns of code. To minimize such duplications across 100's or +even 1000's of scripts, we introduced the concept of a bundle. + +A bundle is a collection of functions, classes, and other definitions +turning these common patterns into easily reusable components. A bundle +must be activated in the ``do_create_instance()`` hook of a Cript. This +does *not* exclude doing additional hooks in the Cript itself. + +The following bundles are available in the core today: + +============================ ==================================================================== +Bundle Description +============================ ==================================================================== +``Bundle::Common`` For DSCP and an overridable Cache-Control header. +``Bundle::LogsMetrics`` Log sampling, TCPInfo and per-remap metrics. +============================ ==================================================================== + +This example shows how a Cript would enable both of these bundles with all features: + +.. code-block:: cpp + + #include <cripts/Preamble.hpp> + + #include <cripts/Bundles/Common.hpp> + #include <cripts/Bundles/LogsMetrics.hpp> + + do_create_instance() + { + Bundle::Common::activate().dscp(0x2e) + .cache_control("max-age=3600"); + + Bundle::LogsMetrics::activate().logsample(100) + .tcpinfo(true) + .propstats("example.com"); + } + +.. note:: + You don't have to activate all components of a Bundle, just leave it out if you don't need it. + +.. note:: + The bundles are not enabled by default. You have to explicitly activate them in your Cript, + with the appropriate include directives. This is because the list of Bundles may grow over time, + as well as the build system allowing for custom bundles locally. diff --git a/doc/developer-guide/cripts/cripts-connections.en.rst b/doc/developer-guide/cripts/cripts-connections.en.rst index c0ee65105c..272904fc7b 100644 --- a/doc/developer-guide/cripts/cripts-connections.en.rst +++ b/doc/developer-guide/cripts/cripts-connections.en.rst @@ -24,3 +24,113 @@ Connections *********** + +Cripts will manage all client and server connections, in a set +of objects that it manages for the user. This ownership again +implies that access to these objects must be ``borrowed``. + +.. code-block:: cpp + + do_remap() + { + static Matcher::Range::IP ALLOW_LIST({"192.168.201.0/24", "10.0.0.0/8"}); + borrow conn = Client::Connection::get(); + auto client_ip = conn.ip(); + + if (!ALLOW_LIST.contains(client_ip)) { + // Deny the request (see examples for details) + } + } + +There are two kinds of connections provided by the run-time system: + +======================= ========================================================================= +Connection Object Description +======================= ========================================================================= +``Client::Connection`` The connection from the client to the ATS server. +``Server::Connection`` The connection from ATS to parent or origin server. +======================= ========================================================================= + +As usual, the ``Server::Connection`` object is only available assuming that the request +is a forward proxy request. On cache misses, there is no such connection. + +.. _cripts-connections-variables: + +Connection Variables +==================== + +Both connection objects provide a number of variables that can be accessed. These are: + +======================= ========================================================================= +Variable Description +======================= ========================================================================= +``count`` The number of transactions processed on the connection so far. +``tcpinfo`` A number of TCPinfo related fields (see below). +``geo`` If available (compile time) acess to Geo-IP data (see below). +``congestion`` Configure the congestion algorithm used on the socket. +``pacing`` Configure socket pacing for the connection. +``dscp`` Manage the DSCP value for the connection socket. +``mark`` Manage the Mark value for the connection socket. +======================= ========================================================================= + +For other advanced features, a cript has access to the socket file descriptor, via the ``fd()`` +method of the connection object. + +.. note:: + For pacing, the special value ``Cript::Pacing::Off`` can be used to disable pacing. + +Lets show an example of how one could use these variables: + +.. code-block:: cpp + + do_remap() + { + borrow conn = Client::Connection::get(); + + conn.congestion = "bbrv2"; + conn.pacing = 100; + conn.dscp = 0x2e; + conn.mark = 0x1; + } + +.. _cripts-connections-tcpinfo-variables: + +TCPInfo Variables +================= + +The ``tcpinfo`` variable is a structure that provides access to the TCP information for the connection. + +======================= ========================================================================= +Field Description +======================= ========================================================================= +``rtt`` The round trip time for the connection. +``rto`` Retransmission timeout. +``retrans`` The number of retransmissions. +``snd_cwnd`` The congestion window. +``info`` The *raw* TCP information. +======================= ========================================================================= + +In addition to these convenience fields, the ``tcpinfo`` object provides a method to access the raw +TCP information as well in the ``info`` field. There's also a predefined log format, which can be +accessed via the ``log()`` method. See the ``tcpinfo`` plugin in ATS for details. + +.. _cripts-connections-geo-ip: + +Geo-IP +====== + +If ATS has been built with Geo-IP support, the connection object will provide access to the Geo-IP +data for the connection. The following methods will then be available: + +======================= ========================================================================= +Method Description +======================= ========================================================================= +``ASN()`` The ASN number. +``ASNName()`` The name of the ASN. +``Country()`` Country. +``CountryCode()`` Country code. +======================= ========================================================================= + +.. note:: + All methods return string values. These are methods and not fields, so they must be called as + functions. diff --git a/doc/developer-guide/cripts/cripts-crypto.en.rst b/doc/developer-guide/cripts/cripts-crypto.en.rst new file mode 100644 index 0000000000..77ace8f140 --- /dev/null +++ b/doc/developer-guide/cripts/cripts-crypto.en.rst @@ -0,0 +1,90 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +.. include:: ../../common.defs + +.. highlight:: cpp +.. default-domain:: cpp + +.. _cripts-crypto: + +Crypto +****** + +Sometimes doing cryptographic operations can be useful in a Cript. Cripts provides +a foundation for such tasks, providing hashing, encryption and encoding/decoding. + +.. note:: + This is still very much work in progress, and thus far we've only added the most + basic of cryptographic operations. More will be added as needed. + +.. _cripts-misc-crypto-hash: + +Hash +==== + +================================= =============================================================== +Object Description +================================= =============================================================== +``Crypto::MD5`` MD5 hashing. +``Crypto::SHA256`` SHA256 hashing. +``Crypto::SHA512`` SHA512 hashing. +``Crypto::HMAC::SHA256`` HMAC-SHA256 hashing. +================================= =============================================================== + +These objects all provide a ``encod()`` and ``decode()`` method, to hash and unhash strings. +Examples: + +.. code-block:: cpp + + do_remap() + { + CDebug("SHA256 = {}", Crypto::SHA256::encode("Hello World")); + } + +.. _cripts-misc-crypto-encryption: + +Encryption +========== + +Currently only one encryption object is provides, for AES256. This object provides +``encrypt()`` and ``decrypt()`` methods. A ``hex()`` method is also provided to retreive +the encrypted data as a hex string. For encrypting data in chunks, a ``finalize()`` method +is provided to retrieve the final encrypted data. + +================================= =============================================================== +Object Description +================================= =============================================================== +``Crypto::AES256`` AES256 encryption and decryption. +================================= =============================================================== + +.. _cripts-misc-crypto-encoding: + +Encoding +======== + +Finally, for convenience, Cripts provides a ``Base64`` object for encoding and decoding, as well +as a URL escaping object, ``Escape``. + +================================= =============================================================== +Object Description +================================= =============================================================== +``Crypto::Base64`` Methods for Base64 encoding. +``Crypto::Escape`` Methods for URL escaping. +================================= =============================================================== + +These objects all provide a ``encode()`` and ``decode()`` method, to encode and decode strings. diff --git a/doc/developer-guide/cripts/cripts-headers.en.rst b/doc/developer-guide/cripts/cripts-headers.en.rst index 46fe2b96bf..4e952eaf10 100644 --- a/doc/developer-guide/cripts/cripts-headers.en.rst +++ b/doc/developer-guide/cripts/cripts-headers.en.rst @@ -45,9 +45,11 @@ Header Object Description ``Server::Response`` The server response headers. ===================== =========================================================================== -Note that for all of these headers, except the ``Client::Request``, the headers are not -available until the respective hook is called. For example, the ``Client::Response`` headers -are not available until the response headers are received from the origin server or cache lookup. +.. note:: + + For all of these headers, except the ``Client::Request``, the headers are not + available until the respective hook is called. For example, the ``Client::Response`` headers + are not available until the response headers are received from the origin server or cache lookup. Assigning the empty value (``""``) to a header will remove it from the header list. For example: diff --git a/doc/developer-guide/cripts/cripts-matcher.en.rst b/doc/developer-guide/cripts/cripts-matcher.en.rst index 17876d83ec..0cd5d73336 100644 --- a/doc/developer-guide/cripts/cripts-matcher.en.rst +++ b/doc/developer-guide/cripts/cripts-matcher.en.rst @@ -24,3 +24,92 @@ Matchers ******** + +Cripts supports most common comparisons, ``==``, ``!=``, etc. for many of the +strings and variables that it provides. However, this is sometimes not adequate +for all use cases, so Cripts provides a way to define custom matchers. There are +currently three types of matchers: + +============================ ==================================================================== +Matcher Description +============================ ==================================================================== +``Matcher::Range`` Matching IP addresses against one or many IP ranges. +``Matcher::PCRE`` Matching strings against one or many regular expressions. +``Matcher::List::Method`` Match a request method against a list of methods. +============================ ==================================================================== + +Often you will declare these ranges once, and then use them over and over again. For this purpose, +Cripts allows ranges to be declared ``static``, which means it can optimize the code around the matches. + +Here's an example using the regular expression matcher: + +.. code-block:: cpp + + do_remap() + { + static Matcher::PCRE pcre({"^/([^/]+)/(.*)$", "^(.*)$"}); // Nonsensical ... + + borrow url = Client::URL::get(); + + if (pcre.match(url.path)) { + // Do something with the match + } + } + +.. note:: + + For the IP-range and regular expression matcher, you can specify a single range or regular expression, + it does not have to be declared as a list with the ``{}`` syntax. For both, the single or list arguments + are strings withing ``""``. + +.. _cripts-matchers-functions: + +Matcher Functions +================= + +All matchers have the following functions: + +============================ ==================================================================== +Function Description +============================ ==================================================================== +``match()`` Match the given string against the matcher. +``contains()` Another name for ``match()`` +``add()`` Add another element to the matcher (can not be used with ``static``) +============================ ==================================================================== + +.. _cripts-matchers-pcre: + +Matcher::PCRE +============= + +The PCRE matcher is used to match strings against one or many regular expressions. When a match +is found, a ``Matcher::PCRE::Result`` object is returned. This object has the following functions +to deal with the matched results and the capture groups: + +============================ ==================================================================== +Function Description +============================ ==================================================================== +``matched()`` A boolean indicating if a regex was matched. +``count()`` Returns the number of regex capture groups that are matched. +``matchIX()`` Returns the index of the matched regex capture group. +[] Index Retrives the matched string for the given capture group index. +============================ ==================================================================== + +Lets show an example: + +.. code-block:: cpp + + do_remap() + { + static Matcher::PCRE allow({"^([a-c][^/]*)/?(.*)", "^([g-h][^/]*)/?(.*)"}); + + borrow url = Client::URL::get(); + + auto res = allow.match(url.path); + + if (res.matched()) { + CDebug("Matched: {}", res[1]); + CDebug("Matched: {}", res[2]); + // Now do something with these URLs matching these paths + } + } diff --git a/doc/developer-guide/cripts/cripts-misc.en.rst b/doc/developer-guide/cripts/cripts-misc.en.rst index 4590e9ad8b..d259cec275 100644 --- a/doc/developer-guide/cripts/cripts-misc.en.rst +++ b/doc/developer-guide/cripts/cripts-misc.en.rst @@ -22,5 +22,188 @@ .. _cripts-misc: -Miscelanious -************ +Miscellaneous +************* + +Of course, there's a lot more to Cripts than :ref:`URLs <cripts-urls>`, +:ref:`headers <cripts-headers>` and :ref:`connections <cripts-connections>`. This +chapter in the Cripts documentation covers a variety of topics that don't fit +into the their own chapter. + +.. _cripts-misc-errors: + +Errors +====== + +Often it's useful to be able to abort a client transaction prematurely, and +return an error to the client. Cripts provides a few convenience functions for +making this easy. + +.. note:: + Explicitly forcing an HTTP error overrides any other response status that may have been set. + +========================= ======================================================================= +Function Description +========================= ======================================================================= +``Error::Status::set`` Sets the response to the status code, and force the request to error. +``Error::Reason::set`` Sets an explicit reason message with the status code. **TBD** +========================= ======================================================================= + +Example: + +.. code-block:: cpp + + do_remap() + { + borrow req = Client::Request::get(); + + if (req["X-Header"] == "yes") { + Error::Status::set(403); + } + } + +.. _cripts-misc-transaction: + +ATS transactions are generally hidden within Cripts, but for power users, the +``transaction`` object provides access to the underlying transaction. In this object, +the following functions are available: + +========================= ======================================================================= +Function Description +========================= ======================================================================= +``disableCallback()`` Disables a future callback in this Cript, for this transaction. +``aborted()`` Has the transaction been aborted. +``lookupStatus()`` Returns the cache lookup status for the transaction. +========================= ======================================================================= + +When disabling a callback, use the following names: + +======================================= ========================================================= +Callback Description +======================================= ========================================================= +``Cript::Callback::DO_REMAP`` The ``do_remap()`` hook. +``Cript::Callback::DO_POST_REMAP`` The ``do_post_remap()`` hook. +``Cript::Callback::DO_SEND_RESPONSE`` The ``do_send_response()`` hook. +``Cript::Callback::DO_CACHE_LOOKUP`` The ``do_cache_lookup()`` hook. +``Cript::Callback::DO_SEND_REQUEST`` The ``do_send_request()`` hook. +``Cript::Callback::DO_READ_RESPONSE`` The ``do_read_response()`` hook. +``Cript::Callback::DO_TXN_CLOSE`` The ``do_txn_close()`` hook. +======================================= ========================================================= + +Finally, the ``transaction`` object also provides access to these ATS objects, which can be used +with the lower level ATS APIs: + +========================= ======================================================================= +Object Description +========================= ======================================================================= +``txnp`` The TSHttpTxn pointer. +``ssnp`` TSHttpSsn pointer. +========================= ======================================================================= + +The ``transaction`` object is a global available everywhere, just like the ``instance`` object. +Example usage to turn off a particular hook conditionally: + +.. code-block:: cpp + + do_remap() + { + static borrow req = Client::Request::get(); + + if (req["X-Header"] == "yes") { + transaction.disableCallback(Cript::Callback::DO_READ_RESPONSE); + } + } + +.. note:: + Disabling callbacks like this is an optimization, avoding for the hook to be called at all. + It can be particularly useful when the decision to run the hook is made early in the Cript. + +Time +==== + +Cripts has encapsulated some common time-related functions in the core. At the +moment only the localtime is available, via the ``Time::Local`` object and its +``now()`` method. The ``now()`` method returns the current time as an object +with the following functions: + +===================== =========================================================================== +Function Description +===================== =========================================================================== +``epoch()`` Returns the number of seconds since the Unix epoch (00:00:00 UTC, January 1, 1970). +``year()`` Returns the year. +``month()`` Returns the month (1-12). +``day()`` Returns the day of the month (1-31). +``hour()`` Returns the hour (0-23). +``minute()`` Returns the minute (0-59). +``second()`` Returns the second (0-59). +``weekday()`` Returns the day of the week (0-6, Sunday is 0). +``yearday()`` Returns the day of the year (0-365). +===================== =========================================================================== + +The time as returned by ``now()`` can also be used directly in comparisons with previous or future +times. + +.. _cripts-misc-plugins: + +Plugins +======= + +While Cripts provides a lot of functionality out of the box, there are times +when you want to continue using existing remap plugins conditionally. Cripts +allows you to load and run existing remap plugins from within your Cripts. +This opens up new possibilities for your existing plugins, as you gain the +full power of Cript to decide when to run such plugins. + +Setting up existing remap plugins must be done in the ``do_create_instance()`` +hook. The instanatiated remap plugins must be added to the instance object for the +Cript, using the ``addPlugin()`` method. Here's an example to run the rate limiting +plugin based on the client request headers: + +.. code-block:: cpp + + do_create_instance() + { + instance.addPlugin("my_ratelimit", "rate_limit.so", {"--limit=300", "--error=429"}); + } + + do_remap() + { + static borrow plugin = instance.plugins["my_ratelimit"]; + borrow req = Client::Request::get(); + + if (req["X-Header"] == "yes") { + plugin.runRemap(); + } + } + +.. note:: + The name of the plugin instance, as specified to ``addPlugin()``, must be + unique across all Cripts. + +.. _cripts-misc-files: + +Files +===== + +In same cases, albeit not likely, you may need to read lines of text from A +file. Cripts of course allows this to be done with C or C++ standard file APIs, +but we also provide a few convenience functions to make this easier. + +The ``File`` object encapsulates the common C++ files operations. For convenience, +and being such a common use case, reading a single line from a file is provided +by the ``File::Line::Reader`` object. Some examples: + +.. code-block:: cpp + + do_remap() + { + static const File::Path p1("/tmp/foo"); + static const File::Path p2("/tmp/secret.txt"); + if (File::Status(p1).type() == File::Type::regular) { + resp["X-Foo-Exists"] = "yes"; + } else { + resp["X-Foo-Exists"] = "no"; + } + string secret = File::Line::Reader(p2); + CDebug("Read secret = {}", secret); + } diff --git a/doc/developer-guide/cripts/cripts-overview.en.rst b/doc/developer-guide/cripts/cripts-overview.en.rst index 0a56371f0d..b0eb649768 100644 --- a/doc/developer-guide/cripts/cripts-overview.en.rst +++ b/doc/developer-guide/cripts/cripts-overview.en.rst @@ -47,6 +47,17 @@ Plugins built using this method loads exactly like any other ATS plugin, and can used in the same way. The only difference is that the plugin will be a lot simpler to write and maintain. +.. _cripts-overview-building-additions: + +Building Additions +------------------ + +The ATS build system is already setup such that when building ATS, and therefore Cripts, +you can add additional modules and bundles to the build. This is done by adding the +module or bundle to the ``src/cripts``, ``src/cripts/Bundles``, ``include/cripts`` and +``include/cripts/Bundles`` directories. This has to be done before running the +``cmake`` command in your build process. + .. _cripts-overview-usage: Usage @@ -90,8 +101,11 @@ depending on your preference. The file must be readable by the ATS process. Exam #include <cripts/Epilogue.hpp> -In this example, note that both the Preamble and Epilogue are explicitly included. -This is strictly necessary for the Cript to compile and load correctly. +.. note:: + + Both the Preamble and Epilogue must explicitly be included in every cript, + in the correct order. The Preamble must come first, setting up the environment, + and the Epilogue must come last, initiating the hooks and the plugin. .. _cripts-overview-usage-compiler: @@ -117,13 +131,13 @@ also shown below: # This was done for a Mac, may need to be adjusted for other platforms. ATS_ROOT=/opt/ats CXX=clang++ - CXXFLAGS="-std=c++20 -I/opt/homebrew/include -undefined dynamic_lookup" - STDFLAGS="-shared -fPIC -Wall -Werror -I${ATS_ROOT}/include -L${ATS_ROOT}/lib -lcripts" + OSFLAGS="-I/opt/homebrew/include -undefined dynamic_lookup" + CXXFLAGS="-std=c++20 -O3 -shared -fPIC -Wall -Werror -I${ATS_ROOT}/include -L${ATS_ROOT}/lib -lcripts" SOURCE=$1 DEST=$2 - ${CXX} ${CXXFLAGS} ${STDFLAGS} -o $DEST $SOURCE + ${CXX} ${CXXFLAGS} ${OSFLAGS} -o $DEST $SOURCE The example in the ATS source directory is more complex, as it also provides a basic cache for the shared object files. This is useful for large setups with many Cript files. @@ -215,7 +229,10 @@ In addition to these normal hooks, there are also three hooks that are used for up a cript and remap plugin instance. This is primarily useful when writing traditional remap plugins in Cripts. -**Note:** These hooks are not needed for most Cripts that are used in remap rules. +.. note:: + + These hooks are not needed for most Cripts that are used in remap rules. And you + will only use the instance variables (see below) if you are using these instance hooks. .. _cripts-overview-hooks-do-init: diff --git a/doc/developer-guide/cripts/cripts-urls.en.rst b/doc/developer-guide/cripts/cripts-urls.en.rst index c9877abe36..022d898c52 100644 --- a/doc/developer-guide/cripts/cripts-urls.en.rst +++ b/doc/developer-guide/cripts/cripts-urls.en.rst @@ -67,6 +67,9 @@ Component Description ``query`` The query. =============== ================================================================================= +.. note:: + The path component of all URLs in ATS do **not** include the leading slash! + These components can be accessed and modified as needed. Both the ``path`` and ``query`` are strings, and can be manipulated as such. However, they are both also considered list of their constituent parts, and can be accessed as such. For example, to get the first part of the path, diff --git a/doc/developer-guide/cripts/cripts-variables.en.rst b/doc/developer-guide/cripts/cripts-variables.en.rst index 80e0e994de..06a59ca06f 100644 --- a/doc/developer-guide/cripts/cripts-variables.en.rst +++ b/doc/developer-guide/cripts/cripts-variables.en.rst @@ -95,8 +95,10 @@ In addition to this, there's a number of *matching* features in Cripts, which ca with strings. These are covered in more detail in the :ref:`cripts-matcher` section. Of course, regular comparisons such as ``==`` and ``!=`` are also available. -**Note:** We'll continue to update features of Cripts as we start using it more in production. If you -have any suggestions or requests for strings (or any other data type), please let us know! +.. note:: + + We'll continue to update features of Cripts as we start using it more in production. If you + have any suggestions or requests for strings (or any other data type), please let us know! .. _cripts-variables-configuration: @@ -143,7 +145,7 @@ Variable Description ============================ ==================================================================== All of these are controlled via a boolean value, and can be set to either ``true`` or ``false``, -the same ``.get()`` and ``.set()`` as for configuration variables. As an example, lets randomly +using the same ``.get()`` and ``.set()`` as for configuration variables. As an example, lets randomly turn off logging for some percentage of requests: .. code-block:: cpp @@ -153,3 +155,32 @@ turn off logging for some percentage of requests: control.logging.set(false); // 10% log sampling } } + + +.. _cripts-misc-versions: + +Versions +======== + +Cripts provides a way to get the version of ATS and Cripts at runtime. The +following global variables are available: + +============================ ==================================================================== +Variable Description +============================ ==================================================================== +versions.major The major version of ATS. +versions.minor The minor version of ATS. +versions.patch The patch version of ATS. +============================ ==================================================================== + +.. _cripts-variables-other: + +Other Variables +=============== + +There are a number of other variables that are available in Cripts. They are generally closely +tied to another object, and are therefore documented in various chapters within here. However, +here's a quick list of some of the more common ones: + +- :ref:`cripts-connections-variables` +- :ref:`cripts-connections-tcpinfo-variables` diff --git a/doc/developer-guide/cripts/index.en.rst b/doc/developer-guide/cripts/index.en.rst index 6a61611bd5..155b055deb 100644 --- a/doc/developer-guide/cripts/index.en.rst +++ b/doc/developer-guide/cripts/index.en.rst @@ -31,6 +31,7 @@ Cripts cripts-headers.en cripts-connections.en cripts-matcher.en + cripts-crypto.en cripts-misc.en cripts-bundles.en cripts-examples.en diff --git a/include/cripts/Connections.hpp b/include/cripts/Connections.hpp index 206035e2f7..46c6ae1be7 100644 --- a/include/cripts/Connections.hpp +++ b/include/cripts/Connections.hpp @@ -213,7 +213,7 @@ class ConnBase private: ConnBase *_owner = nullptr; - }; // End class Geo::TcpInfo + }; // End class ConnBase::Geo class TcpInfo { diff --git a/src/cripts/README.md b/src/cripts/README.md deleted file mode 100644 index 07b9dc632e..0000000000 --- a/src/cripts/README.md +++ /dev/null @@ -1,446 +0,0 @@ -# 1. Cripts - -Cripts, C-Scripts, is a set of wrappers and include files, for making simple ATS -plugins easy to generate, modify or create from scratch. A key design here is that -the Code is the Configuration, i.e. the intent really is to have a custom Cript file -for every remap rule in the running system. - -ToDo: This document is not 100% updated with all features, but is at least a starting point. - -- [1. Cripts](#1-cripts) -- [2. Building Cripts](#2-building-cripts) - - [2.1. Building a Cript](#21-building-a-cript) - - [2.2. Lint: Validating a Cript](#22-lint-validating-a-cript) - - [2.3. Clang-tidy](#23-clang-tidy) -- [3. Writing Cripts](#3-writing-cripts) - - [3.1. Data types](#31-data-types) - - [3.2. Hooks](#32-hooks) - - [3.3. Processing and modifying headers](#33-processing-and-modifying-headers) - - [3.4. Processing and modifying URLs](#34-processing-and-modifying-urls) - - [3.4.1. Special case: Cache Key URL](#341-special-case-cache-key-url) - - [3.5. Accessing and modifying connections](#35-accessing-and-modifying-connections) - - [3.6. Overridable configurations](#36-overridable-configurations) - - [3.7. Pattern and identity matching](#37-pattern-and-identity-matching) - - [3.8. Cryptography functions](#38-cryptography-functions) - - [3.9. Various other utilities](#39-various-other-utilities) - - [3.10. Transaction contexts](#310-transaction-contexts) - - [3.10.1. Instance data (parameters)](#3101-instance-data-parameters) - - [3.10.2. Transaction data](#3102-transaction-data) -- [4. Plugins Cript can mimic or replace](#4-plugins-cript-can-mimic-or-replace) - - [4.1. conf\_remap](#41-conf_remap) - - [4.2. cachekey](#42-cachekey) - - [4.3. header\_rewrite](#43-header_rewrite) - - [4.4. regex\_remap](#44-regex_remap) - - [4.5. geoip\_acl and maxmind\_acl](#45-geoip_acl-and-maxmind_acl) - - [4.6. tcpinfo](#46-tcpinfo) - -# 2. Building Cripts - -Cripts needs the `{fmt}` and `PCRE2` libraries and include files. We currently only -build cripts when explicitly enabled, and only using `cmake``. - -## 2.1. Building a Cript - -At the moment, building Cripts needs to be done manually, using cmake and package -config tools. Once built, it can be loaded as any regular remap plugin: - -``` -map https://example.com https://origin.example.com @plugin=cript_test.so -``` - -## 2.2. Lint: Validating a Cript - -TBD: I have the beginning of this tool, will land it later. - -## 2.3. Clang-tidy - -A custom clang-tidy configuration is provided with this patch, and I've run all -code through clang-tidy with this configuration. - -# 3. Writing Cripts - -Cripts follow the same basic model of how ATS splits up transaction processing -into what we call "hooks". A hook is essentially a callback mechanism, where -custom code can be injected into the ATS core via plugins. - -The Cript language itself is essentially C++17, except it imposes some serious, -but important, limitations on what can and can not be used. Albeit we call this -a scripting language, it's truly compiled into regular, reloadable ATS plugins. - -To start off with, we'll show a very basic Cript, to get an idea of what to -expect: - -``` -// The primary include file, this has to always be included -#include <Cript/Preamble.hpp> - -do_send_response() -{ - borrow req = Client::Request::get(); - borrow resp = Client::Response::get(); - - if (req["X-Miles"] != "") { - resp["X-Miles"] = req["X-Miles"]; // Echo back the request header - } -} - -do_remap() -{ - borrow req = Client::Request::get(); - - req["@receipt"] = "PropertyX"; -} - -#include <Cript/Epilogue.hpp> -``` - -Don't worry about the exact details here, we will discuss them all further -down this documentation. There are however two critical pieces here to remember: - -* All Cript's must have the Preamble and Epilogue include's as above -* You can have your own C/C++ function definitions if you like, but the predefined - callbacks (such as `do_remap()`) have fixed names - -## 3.1. Data types - -Cript will make a best-effort to hide data types from the users as much as possible. -As such, it's highly recommended to use the "auto" style when declaring variables. -Cript being C/C++, it's impossible to hide everything, and to optimize integration -with ATS, we have a few quirks. - -For example, there are two types of strings which can be returned, regular `strings` -and something called `string_view`. The latter are immutable representations of -strings that are owned and managed by ATS, and you can and should only use these -when reading values out of say a request or request header. - -These are the typical types that are used by Cripts: - -| Type | Description | -| ----------- | --------------------------------------------- | -| string | This is the common std::string type from C++ | -| string_view | An immutable, ATS owned string representation | -| integer | A signed integer value, 64-bit long | -| float | A signed, floating point value | -| boolean | A `true` or `false` flag | - -## 3.2. Hooks - -To simplify usage, Cript predefines a set of function names, each corresponding -to an ATS hook. These names are set in stone, and will be automatically added -to the ATS core if provided in the Cript. Those hooks not used will not be -called, obviously. - -The current version of Cript supports the following callbacks / hooks, all of which are optional: - -| Callback name | Hook equivalent | Description | -| ------------------ | ------------------------------ | -------------------------------------------------- | -| do_remap() | none | This is the main entry point for all remap plugins | -| do_post_remap() | TS_HTTP_POST_REMAP_HOOK | Called right after the remap rule is done | -| do_send_request() | TS_HTTP_SEND_REQUEST_HDR_HOOK | Called before making an origin request | -| do_read_response() | TS_HTTP_READ_RESPONSE_HDR_HOOK | Called when the origin has a response | -| do_send_response() | TS_HTTP_SEND_RESPONSE_HDR_HOOK | Called before sending a response to client | -| do_txn_close() | TS_HTTP_TXN_CLOSE_HOOK | Called just before the transaction is done | - -Note that not all of these callbacks are triggered for all requests. For example, -upon a cache hit in ATS, `do_send_request()` and `do_read_response()` are not called. -In addition, there are two plugin specific callbacks, which are used when loading -the plugins and instantiating remap rules: - -| Callback name | API equivalent | Description | -| ------------------ | --------------------- | ---------------------------------------------- | -| do_init | TSRemapInit() | Called once when the Cript is loaded | -| do_create_instance | TSRemapNewInstance | Called for every remap rule using the Cript | -| do_delete_instance | TSRemapDeleteInstance | Called to cleanup any instance data from above | - -## 3.3. Processing and modifying headers - -A big part of all Cripts is to read and modify various headers. ATS as it works, -has four distinct headers, which translates into the following four environments: - -| Header name | Description | -| ---------------- | -------------------------------------------------- | -| Client::Request | The clients request header | -| Client::Response | The response header that is sent to the client | -| Server::Request | The request header being sent to the origin server | -| Server::Response | The response header being received from origin | - -Accessing these headers is easy: -``` - borrow client_req = Client::Request::get(); - borrow client_resp = Client::Response::get(); - borrow server_req = Server::Request::get(); - borrow server_resp = Server::Response::get(); -``` - -Note that not all of these are available in every callback; For example, the client -response header is not available to read or write until we are in the hook for -`do_send_response()`. - -The response headers has a couple of additional features specifically for -responses: -``` -borrow client_resp = Client::Response::get(); - -if (resp.status = 220) { - resp.status = 200; -} - -// TBD more stuff here ? -``` - -Similarly, the request headers has a few unique traits as well: - -``` -borrow client_req = Client::Request::get(); - -if (client_req.method == "GET") { - ... -} - -// TBD more stuff here ? -``` - -## 3.4. Processing and modifying URLs - -Similarly to headers, URLs are important in Cripts, in all its various forms. -Currently, Cript supports the following URLs: - -| URL name | Description | -| ---------------- | -------------------------------------------------------------- | -| Client::Pristine | The pristine client request URL, which is immutable | -| Client::URL | The clients request URL, which can be modified | -| Cache::URL | This is a special URL, used internally of ATS as the cache key | - -Getting these URLs follows a similar getter pattern to the headers: -``` -borrow pristine = Pristine::URL::get(); -borrow client = Client::URL::get(); -borrow cache = Cache::URL::get(); -``` - -Within a URL object, you can access all its various components via names following -standard URL naming. - -| Identifier | Description | -| ---------- | --------------------------------------- | -| host | The URL host | -| port | The URL port number | -| path | The URL path | -| query | The URL query parameters (all of them!) | - -Using this is easy (of course): -``` -auto url = Client::URL; - -if (url.host == "www.example.com") { - ... -} -``` - -The `path` components does support indexes, e.g. the first URL path -component ("directory") can be read or written with `path[0]`. - -The query parameter has a set of additional features, which are particularly -useful for modifying the cache key URLs. This includes sorting the list of -query parameters, or adding and removing a query parameter by name. This is -best explained with a real example: - -``` -borrow client_req = Client::Request::get(); -borrow c_url = Cache::URL::get(); - -c_url.query.sort(); // Sorts all query parameter by name first -c_url.query["foo"].erase(); // Removes the foo query parameter, if it exists -c_url.query["bar"] = "fie"; // Adds, or modifies, the bar query param to be "fie" - -if (client_req["X-Miles"]) { - c_url.path += "key_add="; - c_url.path += client_req["X-Miles"]; // Adds this header to the cache key -} -``` - -With this in mind, a URL query path of - -``` -?foo=fum&z=xxx&d=yyy -``` - -gets normalized and modified to - -``` -?bar=fie&d=yyy&z=xxx -``` - -In addition, the cache key URL path is appended with an additional string -extracted from the request headers. - -### 3.4.1. Special case: Cache Key URL - -## 3.5. Accessing and modifying connections - -There are essentially two possible connections being involved with a -transaction: The client connections, and the origin server connection. The -latter will not exist on a cache miss, and should therefore only be used -in callbacks involving origin transactions (*cache misses*). - -Accessing the client connections is easily done with - -``` -borrow conn = Client::Connection::get(); -``` - -TODO: More stuff here, explaining the details - -## 3.6. Overridable configurations -In ATS, some (not all) configurations can be overriden per transaction, or -remap rule. Cripts supports all such configurations, in a way that retains -the naming from `records.config`! Examples: - -``` -proxy.config.http.cache.http.set(0); - -if (proxy.config.http.cache.generation.get() > 0) { - // Do something -} -``` - -All overridable configurations are documented in the [records.config](https://docs.trafficserver.apache.org/en/latest/admin-guide/files/records.config.en.html) -documentation. Look for a tag `Overridable` for each setting. - -## 3.7. Pattern and identity matching - -Several pattern matching features exists in Cripts today: - -| Name | Description | -| ------------------ | ------------------------------------------------------------------------------- | -| Matcher::Range::IP | Match an IP (from e.g. a `Client::Connection`) against a range of IPs | -| Matcher::PCRE | Create a Perl compatible regular expression, to match arbitrary strings against | - -They all follow the same basic concept: - -1. Setup a matcher (ideally `static` for performance) -2. Get the string or identity to match from the transaction -3. Call the `contains()` (or `match()`, they are synonyms) function - -Examples: - -``` -static Matcher::Range::IP allow({"192.168.201.0/24", "17.0.0.0/8"}); -borrow conn = Client::Connection::get(); - -if (allow.contains(conn.ip())) { - CDebug("Client IP allowed: {}", conn.ip().string(24, 64)); - ... -} -``` - -Again, it's important to use the `static` keyword here, which helps the Cripts compiler -to optimize the creation of the Matcher. - -``` -static Matcher::PCRE pcre("^/([^/]+)/(.*)$"); -auto url = Client::URL::get(); -auto res = pcre.match(url.path); - -if (res) { - borrow resp = Client::Response::get() - - resp["X-PCRE-Test"] = format("{} and {}", res[1], res[2]); -} -``` - -## 3.8. Cryptography functions - -## 3.9. Various other utilities - -## 3.10. Transaction contexts -### 3.10.1. Instance data (parameters) - -### 3.10.2. Transaction data - -# 4. Plugins Cript can mimic or replace - -This is a list of existing ATS plugins that properly written and configure Cript -scripts could replace. This will repeat some of the sections -and mentions above, but helps identifying what can be used and when. - -## 4.1. conf_remap - -This plugin can set overridable configuration per remap. Cripts supports that via -the global `proxy.` object. E.g. - -``` -proxy.config.http.cache.http.set(1); - -if (proxy.config.http.cache.http.get() { - //... -} -``` - -## 4.2. cachekey - -A Cript is free to modify the `Cache::URL`` as needed, in either `do_remap()` or -`do_post_remap()`. We recommend the latter to be used for cache-key manipulation. - -``` -do_post_remap() -{ - borrow ckey = Cache::URL::get(); - - ckey.query.sort(); - ckey.path += "entropy"; -} -``` - -## 4.3. header_rewrite - -Almost all features from `header_rewrite` are available, and more flexible, in Cript. -If there's anything missing, please let us know. - -## 4.4. regex_remap - -Cript supports PCRE2, as well as modifying the request URI as you like. For example: - -``` -do_post_remap() -{ - static Matcher::PCRE hosts({"(.*(\\.ogre.\\.com"}); - borrow url = Client::URL::get(); - auto res = hosts.match(url.host); - - if (res) { - url.path.insert(0, res[1]); // Prepend the path with the hostname - } -} -``` - -## 4.5. geoip_acl and maxmind_acl - -Lookups into the geo-location database is done via the connection object - -``` -do_remap() -{ - borrow conn = Client::Connection::get(); - - if (conn.geo.ASN() == 123) { - Error::Status::set(403); - } -} -``` - -## 4.6. tcpinfo - -The existing connection TcpInfo features can do everything this plugin can do, -and more. Feature parity is done with - -``` -do_send_response() -{ - borrow resp = Client::Response::get(); - borrow conn = Client::Connection::get(); - - resp["@TCPInfo"] = conn.tcpinfo.log(); // Can also use format() for more flexibility - } -} -```
