hussein-awala commented on code in PR #64610:
URL: https://github.com/apache/airflow/pull/64610#discussion_r3564816729
##########
airflow-core/src/airflow/utils/sqlalchemy.py:
##########
@@ -62,6 +62,34 @@ def get_dialect_name(session: Session) -> str | None:
return getattr(bind.dialect, "name", None)
[email protected]
+def apply_regex_query_timeout(session: Session) -> Generator[None, None, None]:
+ """
+ Bound the runtime of a user-supplied regex filter on PostgreSQL, scoped to
the wrapped query.
+
+ Reads the ``[api] regexp_query_timeout`` config (in seconds) and, on
PostgreSQL, sets a
+ transaction-local ``statement_timeout`` (via ``set_config(...,
is_local=True)``) for the
+ duration of the ``with`` block, then resets it to the default so it does
not affect any other
+ statement running later in the same transaction. This is a ReDoS
safeguard: a malicious pattern
+ is aborted instead of pinning a database backend. No-op on non-PostgreSQL
backends and when the
+ configured timeout is ``0`` (which also means regexp filtering is
disabled).
+ """
+ timeout_seconds = conf.getint("api", "regexp_query_timeout")
+ if timeout_seconds <= 0 or get_dialect_name(session) != "postgresql":
Review Comment:
Good call. `apply_regex_query_timeout` now enforces the configured timeout
on MySQL too, via the session `max_execution_time` (in ms), scoped to the
regexp query and restored to the previous value on exit -- mirroring the
PostgreSQL `statement_timeout`. It's a no-op only on SQLite (dev-only, no
server-side regex). So `regexp_query_timeout` now bounds both supported
production backends.
##########
airflow-core/src/airflow/config_templates/config.yml:
##########
@@ -1635,6 +1635,30 @@ api:
type: boolean
example: ~
default: "True"
+ regexp_query_timeout:
+ description: |
+ Timeout, in seconds, for regular-expression based query filters in the
API. This single
+ value both enables the feature and bounds its runtime: a value of
``0`` (the default)
+ disables regexp filtering entirely, and any positive value enables it
while capping how
+ long such a query may run.
+
+ When enabled, it allows additional filtering features that require
database-side regexp
+ matching, such as the ``partition_key_regexp_pattern`` filter on ``GET
/assets/events``
+ and on the Execution API asset-event endpoints. It is disabled by
default for security
+ reasons: a user-supplied regular expression is passed to the
database's own regex engine,
+ which is a Regular expression Denial of Service (ReDoS) vector: a
crafted pattern can
+ force the engine into pathological backtracking and consume database
backend CPU.
+
+ The timeout is the primary runtime mitigation for that risk: on
PostgreSQL it is enforced
+ as a transaction-local ``statement_timeout`` so a pathological pattern
is aborted instead
+ of pinning a database backend (PostgreSQL has no built-in regex step
limit, unlike MySQL's
+ ``regexp_time_limit``). Keep it as low as your legitimate queries
allow. Coupling the
+ on/off switch with the timeout intentionally makes it impossible to
enable regexp
+ filtering without a runtime bound in place.
+ version_added: 3.4.0
+ type: integer
Review Comment:
Done -- `regexp_query_timeout` is now a `float`, so fractional seconds like
`0.5` work. It's read via `conf.getfloat` and converted to milliseconds for
both the PostgreSQL and MySQL limits.
##########
airflow-core/docs/authoring-and-scheduling/assets.rst:
##########
@@ -247,7 +247,19 @@ Inlet asset events can be read with the ``inlet_events``
accessor in the executi
Each value in the ``inlet_events`` mapping is a sequence-like object that
orders past events of a given asset by ``timestamp``, earliest to latest. It
supports most of Python's list interface, so you can use ``[-1]`` to access the
last event, ``[-2:]`` for the last two, etc. The accessor is lazy and only hits
the database when you access items inside it.
-The accessor also supports chaining methods to filter events before fetching
them. For example, to retrieve only events where specific ``extra`` keys match
given values:
+The accessor also supports chaining methods to filter events before fetching
them. For example, to retrieve only events matching a specific partition key
regular expression:
+
+.. code-block:: python
+
+ @task(inlets=[regional_sales])
+ def process_us_sales(*, inlet_events):
+ us_events =
inlet_events[regional_sales].partition_key_regexp_pattern(r"^us\|")
+ for event in us_events:
+ print(event.extra, event.partition_key)
+
+For an exact partition key match, use ``.partition_key(value)`` instead.
Regexp filtering is opt-in: it is enabled only by setting ``[api]
regexp_query_timeout`` to a positive number of seconds, which also bounds the
query runtime; see the config for the security trade-off.
Review Comment:
Done -- linked it with a `:ref:` to `config:api__regexp_query_timeout` (the
repo's config-anchor convention). I didn't find a `:conf:` role registered, so
I used the `:ref:` + `config:` anchor pattern used elsewhere in the docs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
