bneradt commented on code in PR #13586: URL: https://github.com/apache/trafficserver/pull/13586#discussion_r3883951946
########## plugins/experimental/abuse_shield/unit_tests/test_ip_data.cc: ########## @@ -0,0 +1,138 @@ +/** @file + + Unit tests for abuse_shield per-rule token buckets. + + @section license License + + Licensed to the Apache Software Foundation (ASF) under one or more contributor license + agreements. See the NOTICE file distributed with this work for additional information regarding + copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software distributed under the License + is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express + or implied. See the License for the specific language governing permissions and limitations under + the License. +*/ + +#include "../ip_data.h" Review Comment: Fixed in bd810d6c4f. Both unit tests now use direct plugin-header includes, and the test target explicitly puts the abuse_shield source directory ahead of the JAx include directory. That also avoids the two config.h files colliding. ########## doc/admin-guide/plugins/jax_fingerprint.en.rst: ########## @@ -122,6 +136,22 @@ Hybrid setup is the best if you: * Need a fingerprint only for specific server names (in TLS SNI extension), and * Need a fingerprint only on specific paths +Fingerprint Registry Export +--------------------------- + +The export registry is an in-process, read-only view stored in a named |TS| +user-argument slot. It is not JSON and does not serialize fingerprints. The +registry contains length-delimited method/value entries plus a magic value, +ABI version, and structure sizes, allowing consumers to reject incompatible +layouts. JAx owns the registry and strings for the lifetime of the connection +or transaction; consumers must not modify them or retain their pointers. + +Connection-based methods such as JA3 and JA4 use a VConn registry. Request- +based methods such as JA4H use a transaction registry and therefore cannot be +consumed at a ClientHello hook. A downstream build can add its own method to +JAx and publish it through the same registry without adding a method-specific +API to ``ts.h``. Review Comment: Updated in bd810d6c4f with the organization and custom JA method wording. ########## plugins/experimental/abuse_shield/UdiTable.h: ########## @@ -0,0 +1,474 @@ +/** @file + + Private fixed-size table used by the abuse_shield plugin. + + @section license License + + Licensed to the Apache Software Foundation (ASF) under one or more contributor license + agreements. See the NOTICE file distributed with this work for additional information regarding + copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software distributed under the License + is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express + or implied. See the License for the specific language governing permissions and limitations under + the License. +*/ + +#pragma once + +#include <algorithm> +#include <atomic> +#include <chrono> +#include <cstdint> +#include <functional> +#include <memory> +#include <mutex> +#include <string> +#include <unordered_map> +#include <vector> + +namespace abuse_shield +{ + +/** A fixed-size hash table using the Udi "King of the Hill" algorithm. + * + * Instantiations of this table track the keys/entities (IPs, URLs, etc.) with + * the highest rates of events (e.g. number of requests, number of errors, etc.). + * + * Key properties: + * - Fixed memory: N slots = bounded memory, no unbounded growth + * - Self-cleaning: No cleanup thread needed, table manages itself + * - Hot tracking: High-score entries naturally stay in the table + * - Simple locking: Single mutex for all operations + * - Safe references: Returns shared_ptr so data survives eviction + * + * @tparam Key The entity type (e.g., IP address, URL) + * @tparam Data User's custom data type to associate with each entry + * @tparam Hash Hash function for keys (defaults to std::hash) + * + * The table owns the key and score for each entry. Users provide only their custom + * Data type which is stored in a shared_ptr for safe access. Review Comment: Fixed in bd810d6c4f. The template documentation now states that Data must be default-constructible, provide a thread-safe bool is_evictable() const method, and return whether an entry may be replaced; the example implements the required method. ########## plugins/experimental/jax_fingerprint/context_map.h: ########## @@ -52,12 +53,22 @@ * to a string literal with static storage duration, so storing the view * is safe). */ -class ContextMap +class ContextMap : public jax_fingerprint::RegistryV1 { public: static constexpr std::size_t MAX_METHODS = JAX_FINGERPRINT_MAX_METHODS; static_assert(MAX_METHODS >= 1, "Must accommodate at least one fingerprinting method"); + ContextMap() + { + magic = jax_fingerprint::REGISTRY_MAGIC; + abi_version = jax_fingerprint::REGISTRY_ABI_VERSION; + struct_size = sizeof(jax_fingerprint::RegistryV1); + entry_size = sizeof(jax_fingerprint::RegistryEntryV1); + entry_count = 0; + entries = _entries.data(); + } Review Comment: Fixed in bd810d6c4f. ContextMap now initializes the ABI header reserved field to zero. ########## plugins/experimental/abuse_shield/abuse_shield.yaml: ########## @@ -0,0 +1,54 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more contributor license +# agreements. See the NOTICE file distributed with this work for additional information regarding +# copyright ownership. Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software distributed under the License +# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express +# or implied. See the License for the specific language governing permissions and limitations under +# the License. + +global: + ip_tracking: + slots: 50000 + blocking: + duration_seconds: 300 + log_interval_sec: 10 + log_file: abuse_shield + # Load matching jax_fingerprint instances before abuse_shield.so and give + # each one this name with --export. + fingerprint_registry: abuse_shield.fingerprints + +rules: + - name: excessive_requests + filter: + max_req_rate: 100 + req_burst_multiplier: 2.0 + action: [log, block, close] + + - name: excessive_connections + filter: + max_conn_rate: 20 + conn_burst_multiplier: 1.5 + action: [log, block, close] + + - name: excessive_h2_errors + filter: + max_h2_error_rate: 10 + h2_burst_multiplier: 1.0 + action: [log, block, close] + + # Values across fingerprint methods use OR logic. Add "block" if the client + # IP should also be denied for the global blocking duration. + - name: blocked_tls_clients + filter: + fingerprints: + JA3: + - "0123456789abcdef0123456789abcdef" + JA4: + - "t13d1516h2_8daaf6152771_02713d6af862" + action: [log, close] Review Comment: Yes. traffic_ctl plugin msg abuse_shield.reload reparses and atomically swaps the rules, so fingerprint method and value additions or removals apply to the next ClientHello. The selected registry name and loaded JAx plugin instances remain startup-only. I clarified the sample comment in bd810d6c4f. ########## doc/admin-guide/plugins/jax_fingerprint.en.rst: ########## @@ -122,6 +136,22 @@ Hybrid setup is the best if you: * Need a fingerprint only for specific server names (in TLS SNI extension), and * Need a fingerprint only on specific paths +Fingerprint Registry Export +--------------------------- + +The export registry is an in-process, read-only view stored in a named |TS| +user-argument slot. It is not JSON and does not serialize fingerprints. The Review Comment: Removed in bd810d6c4f, including the equivalent wording in the abuse_shield documentation. ########## doc/admin-guide/plugins/abuse_shield.en.rst: ########## @@ -0,0 +1,297 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +.. include:: ../../common.defs + +.. _admin-plugins-abuse-shield: + +Abuse Shield Plugin +******************* + +The abuse_shield global plugin provides consolidated abuse protection using +bounded per-client state and token-bucket rate limiting. It supports: + +* per-IP request, connection, and HTTP/2 error token buckets; +* trusted IP ranges and rule-specific rate-limit tiers; +* logging, temporary IP blocking, and connection closing; +* atomic rule reloads through traffic_ctl; and +* optional ClientHello fingerprint matching before the TLS handshake continues. + +Building and Loading +==================== + +Enable the experimental plugin and build it with CMake:: + + cmake -B build -DENABLE_ABUSE_SHIELD=ON + cmake --build build --target abuse_shield + +Add the plugin and its YAML configuration to plugin.config:: + + abuse_shield.so abuse_shield.yaml + +A relative configuration path is resolved from the |TS| configuration +directory. + +Fingerprint rules also require global JAx instances before Abuse Shield. Each +method publishes to the registry named by ``global.fingerprint_registry``:: + + jax_fingerprint.so --method JA3 --export abuse_shield.fingerprints + jax_fingerprint.so --method JA4 --export abuse_shield.fingerprints + abuse_shield.so abuse_shield.yaml + +Configuration +============= + +The following example configures several available abuse controls: + +.. code-block:: yaml + + global: + ip_tracking: + slots: 50000 + blocking: + duration_seconds: 300 + trusted_ips_file: /etc/trafficserver/abuse_shield_trusted.yaml + log_interval_sec: 10 + log_file: abuse_shield + fingerprint_registry: abuse_shield.fingerprints + + rules: + - name: excessive_requests + filter: + max_req_rate: 100 + req_burst_multiplier: 2.0 + action: [log, block, close] + + - name: excessive_connections + filter: + max_conn_rate: 20 + conn_burst_multiplier: 1.5 + action: [log, block, close] + + - name: excessive_h2_errors + filter: + max_h2_error_rate: 10 + action: [log, block, close] + + - name: blocked_tls_clients + filter: + fingerprints: + JA3: + - "238bcebdfa16aa0be417a7f7a80063a9" + - "99c071c5a5e14cc2527c9e8e0dde4a50" + JA4: + - "t13d1516h2_8daaf6152771_02713d6af862" + action: [log, close] + + enabled: true + +Global Settings +--------------- + +===================================== ============================================== +Setting Description +===================================== ============================================== +ip_tracking.slots Slots in each bounded IP table (default 50000) +blocking.duration_seconds Duration of a block action (default 300) +trusted_ips_file Optional YAML file of IP ranges to bypass +log_interval_sec Minimum log interval per IP (default 10) +log_file Optional separate |TS| text log object +fingerprint_registry Named JAx registry used by fingerprint rules +===================================== ============================================== + +Rules are evaluated in file order and the first matching rule wins. + +Rule Filters +------------ + +===================================== ============================================== +Filter Description +===================================== ============================================== +max_req_rate Maximum requests per second +req_burst_multiplier Request bucket capacity multiplier +max_conn_rate Maximum connections per second +conn_burst_multiplier Connection bucket capacity multiplier +max_h2_error_rate Maximum HTTP/2 errors per second +h2_burst_multiplier HTTP/2 error bucket capacity multiplier +rate_limited_ips_file Optional IP list that selects this rule +fingerprints Map of method names to fingerprint sequences +===================================== ============================================== + +Burst multipliers default to 1.0 and must be at least 1.0. Every rule has an +independent token bucket for each configured rate, so thresholds do not depend +on rule order. A rate is exceeded when its token bucket becomes negative. +Excess events add proportional token debt; the rule stops matching only after +the configured rate replenishes that debt. Entries with debt are protected +from bounded-table eviction. + +All configured rate filters in a rule use AND logic. Fingerprint values use OR +logic across both values and methods, and that fingerprint result is ANDed with +the rule's rate filters. Use separate rules when OR behavior is needed between +rate limits. + +ClientHello Fingerprints +------------------------ + +The supplied JAx methods are JA3 and JA4. Their method names are +case-insensitive. JA3 values are validated as 32 hexadecimal characters and +canonicalized to lowercase; JA4 values are validated against the 36-character +JA4 layout. Other method names and values are treated as opaque strings and +matched exactly, allowing downstream JAx builds to publish site-specific +methods. + +Only methods derived entirely from a TLS ClientHello can be used. JA4H is +derived from an HTTP request and is therefore unavailable at the +TS_SSL_CLIENT_HELLO_HOOK where the plugin makes its decision. + +JAx computes each fingerprint and publishes it in a versioned, read-only +registry held in a named VConn user-argument slot. Abuse Shield consumes that +result at its ClientHello hook; it does not link fingerprint algorithms or +recompute their values. The registry is an in-process array of +length-delimited method/value entries, not JSON. Its header contains a magic +value, ABI version, and structure sizes so a consumer can reject an +incompatible layout. JAx owns all registry memory for the VConn lifetime. + +The JAx plugin lines must precede Abuse Shield in :file:`plugin.config` so +their hooks publish values before Abuse Shield evaluates them. A missing or +incompatible named registry is a startup error. A matching close action uses +``TSVConnReenableEx(vconn, TS_EVENT_ERROR)``, stopping processing before +ServerHello and key-exchange work. Review Comment: Updated in bd810d6c4f. The load-order requirement is now an important admonition beginning with the requested dependency wording. ########## tests/gold_tests/pluginTest/abuse_shield/idle_connections.py: ########## @@ -0,0 +1,45 @@ +#!/usr/bin/env python3 +"""Open plain TCP connections without sending HTTP requests.""" + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information regarding +# copyright ownership. The ASF licenses this file to you under the Apache +# License, Version 2.0 (the "License"); you may not use this file except in +# compliance with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + Review Comment: Fixed in bd810d6c4f. The helper now keeps the ASF license immediately after the shebang and places the module docstring after the license block. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
