[
https://issues.apache.org/jira/browse/GROOVY-12123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul King updated GROOVY-12123:
-------------------------------
Description:
h2. Summary
Add an opt-in, thread-free runtime guard against Regular Expression Denial of
Service
(ReDoS / catastrophic backtracking) for Groovy's regex features, offered as a
runtime
helper ({{RegexGuard}}), a scoped annotation ({{@SafeRegex}}), and type-checker
awareness
of both. Everything is opt-in; no existing code changes behaviour.
h2. Motivation
Groovy makes regex frictionless -- {{=~}}, {{==~}}, {{~/.../}},
{{String#matches}},
{{replaceAll}}, {{split}}, and the {{Matcher}} sugar -- which also makes ReDoS
easy to
reach when a pattern (or its input) comes from outside the program: scripts,
build files,
web handlers, {{ConfigSlurper}}, templating.
{{java.util.regex}} has *no native timeout*, so a crafted input against a
backtracking-prone pattern can hang the calling thread indefinitely. Nothing in
Groovy
guards against this today. Note the trust boundary is usually the *input*, not
the
pattern: the textbook ReDoS applies a hardcoded, developer-written validator to
attacker-supplied text, so "don't run untrusted code" is not a sufficient
answer.
Thread interruption does not help: the regex engine never checks the interrupt
flag, so
neither {{@ThreadInterrupt}} nor {{@TimedInterrupt}} can stop a runaway match
-- the hang
is inside a single JDK library call, not in user code where those transforms
insert checks.
JLine 4.3.1 shipped a {{SafeRegex}} utility using the well-known
{{TimeoutCharSequence}}
technique to close four CVEs (GHSA-r2xf-8xr9-62gw, GHSA-2v9w-34q6-wpqx,
GHSA-ph9c-7hw9-vhhw, GHSA-5q95-hrpc-m3w3). We apply the same idea.
h2. Proposal
Bound regex matching by a wall-clock deadline, without a watchdog thread. The
input
{{CharSequence}} is wrapped so that {{charAt()}} checks a deadline and throws
an unchecked
{{RegexTimeoutException}} once it has passed; because the regex engine re-reads
input
characters continuously while backtracking, a runaway match is interrupted with
no extra
thread and no {{Thread.interrupt}} plumbing. The clock is consulted once every
512 reads,
so evaluations finishing in fewer reads never pay for a clock call.
Four ways to apply it, from most explicit to most ergonomic.
*(a) Explicit guarded evaluation* -- guarded equivalents of the match and find
operators,
with a {{long}} millisecond or {{Duration}} timeout:
{code:groovy}
RegexGuard.matches(pattern, input, 200) // -> boolean
RegexGuard.matcher(pattern, input, Duration.ofMillis(200)) // -> Matcher
over guarded input
{code}
*(b) Guarded data* -- attach a single total deadline to the sequence itself.
Wrap
untrusted input once at a trust boundary and pass it around freely; every regex
evaluated
against it shares the one budget, including Groovy's {{CharSequence}} extension
methods
and anything handing the sequence to {{java.util.regex}} untouched:
{code:groovy}
def safe = RegexGuard.guard(untrusted, 200)
safe.findAll(~/\d+/)
{code}
*(c) Ambient scope* -- a per-evaluation timeout for the dynamic extent of a
closure on the
current thread, covering the regex operators and the {{CharSequence}} extension
methods,
including in methods called from the closure:
{code:groovy}
def groups = RegexGuard.guard(200) {
input.findGroups(/(\d+)\.(\d+)\.(\d+)/)
}
{code}
Nested scopes only tighten -- the smaller per-evaluation timeout wins -- and an
explicit
timeout from (a) is likewise capped by an active ambient timeout.
*(d) Scoped annotation ({{@SafeRegex}})* -- an AST transform rewrites the match
({{==~}})
and find ({{=~}}) operators lexically within the annotated scope to guarded
calls, so no
call-site changes are needed. Fits the existing {{@TimedInterrupt}} /
{{@ThreadInterrupt}}
/ {{@ConditionalInterrupt}} family in {{groovy.transform}}:
{code:groovy}
@SafeRegex(millis = 200)
class Handler {
boolean check(String input) {
input ==~ /(a+)+$/ // rewritten -> RegexGuard.matchRegex(input,
/(a+)+$/, 200)
}
}
{code}
The annotation may be placed on a class, method, constructor, field or local
variable
declaration; on a field or local variable only the initializer expression is
guarded. Its
single attribute is {{millis}} (default {{1000}}, must be a positive constant).
A guarded
field initializer runs, as usual, in the constructor -- or in the static
initializer for a
static field, where a timeout surfaces as {{ExceptionInInitializerError}} whose
cause is
the {{RegexTimeoutException}}.
The transform targets the operator-order entry points {{matchRegex(input,
pattern,
millis)}} and {{findRegex(input, pattern, millis)}} rather than
{{matches(pattern, input, millis)}}, so a rewritten expression still evaluates
its
operands left to right exactly as the original operator did.
The guard is on *matching*, not on {{~/.../}} compilation: {{Pattern.compile}}
is not
subject to backtracking and is not guarded.
h3. Naming
|| Concern || Name || Notes ||
| Scoped annotation | {{@SafeRegex}} | in {{groovy.transform}}, alongside
{{@TimedInterrupt}} et al. |
| Runtime helper | {{RegexGuard}} | in {{groovy.util.regex}}; distinct from the
annotation to avoid a clash |
| Timeout exception | {{RegexTimeoutException}} | unchecked; lets callers
distinguish a ReDoS abort from a normal non-match |
| Operator-order entry points | {{matchRegex}} / {{findRegex}} | mirror
{{ScriptBytecodeAdapter}} naming and operand order; used by the transform |
Both {{RegexGuard}} and {{@SafeRegex}} are marked {{@Incubating}}, {{@since
6.0.0}}.
h3. Implementation notes
* The ambient scope in (c) is held in an
{{org.apache.groovy.runtime.async.ScopedLocal}},
which delegates to {{java.lang.ScopedValue}} on JDK 25+ and falls back to a
save-and-restore {{ThreadLocal}} below, so the binding cannot outlive its
closure.
* A process that never installs an ambient guard never touches that scoped
local: a
monotonic static flag short-circuits the lookup, leaving a single cached read
on the
regex path.
* Deadlines are only ever compared as {{System.nanoTime()}} differences, which
stays
correct across wrap-around; a timeout large enough to overflow the deadline
therefore
reads as "effectively no timeout" rather than expiring immediately.
h2. Scope
*In scope*
* Runtime deadline guard ({{TimeoutCharSequence}}-style), thread-free
* {{RegexGuard}} helper: explicit guarded evaluation, guarded data, and ambient
closure scope
* {{@SafeRegex}} scoped annotation rewriting the {{==~}} and {{=~}} operators
* {{RegexTimeoutException}} type
* {{RegexChecker}} (groovy-typecheckers) awareness of guarded evaluation, so
code inside a
{{@SafeRegex}} scope keeps full regex checking after the rewrite
* Documentation in the operator, metaprogramming and type-checker guides
*Out of scope / deferred*
* *Internal hardening of Groovy's own regex call sites* ({{groovysh}}
completion,
templating, builders) -- proposed originally, not implemented here; deferred
to a
follow-up so this ticket stays a purely opt-in, additive change
* Making the guard the default for {{=~}} / {{==~}} (behaviour change +
overhead)
* Rewriting the {{String#matches}} / {{replaceAll}} / {{split}} family of
method calls
within a {{@SafeRegex}} scope -- only the two operators are rewritten; use
{{RegexGuard}} explicitly for the rest
* Static ReDoS pattern detection -- a possible future {{RegexChecker}} extension
* Guarding {{Pattern.compile}}
* Rewriting the regex engine or adopting a non-backtracking (RE2-style) engine
h2. Limitations
* {{@SafeRegex}} rewrites only the *lexically visible* {{==~}} and {{=~}}
operators --
scoped hardening, not whole-program. Regex evaluation via method calls, or
occurring in
code called from the annotated scope, is not rewritten; use {{RegexGuard}}
there, whose
ambient and guarded-data forms do reach both.
* Direct JDK calls bypass the guard. On a {{String}} receiver,
{{matches(String)}},
{{replaceAll(String, String)}}, {{replaceFirst(String, String)}} and
{{split(String)}}
dispatch to {{java.lang.String}} and never enter Groovy's runtime. Their
{{Pattern}}-argument variants are extension methods and are covered.
* The deadline fires only while the engine is reading input characters (the
backtracking
phase) -- true of catastrophic cases, but not a hard guarantee for every
pathological
state. A well-behaved evaluation over a short input may still succeed after
the deadline.
* The clock is consulted every 512 character reads rather than on every read,
so overshoot
past the deadline is bounded by those reads; pathological patterns perform
millions of
reads per second, so in practice the overshoot is negligible.
* The ambient scope is thread-confined: threads spawned within the closure are
not covered
(on JDK 25+ the binding does propagate into {{StructuredTaskScope}} forks).
* Reduces hang risk; it is not a correctness or a data-at-rest control.
h2. Relationship to existing/planned features
|| Layer || Mechanism || Status || Covers ReDoS? ||
| Compile-time validity | {{RegexChecker}} (groovy-typecheckers) | ships | No
-- pattern *syntax* only |
| Compile-time ReDoS lint | possible {{RegexChecker}} extension | idea |
Partially |
| Untrusted-input tracking | GEP-25 {{@Tainted}} -> regex sink | draft |
Identifies risky matches |
| *Runtime guard (this ticket)* | {{RegexGuard}} / {{@SafeRegex}} |
*implemented* | *Yes -- the actual backstop* |
{{RegexChecker}} only validates that a literal pattern compiles; it says
nothing about
backtracking. As part of this ticket it was extended to recognise the guarded
forms, so
the rewrite performed by {{@SafeRegex}} does not blind it: bad regex literals
are still
reported, and matcher group-count inference survives the rewrite.
h2. Ordering with GROOVY-12123
GROOVY-12123 (constant regex hoisting) and this transform touch the same
expressions, so
whichever lands second must be ordering-aware:
* When {{@SafeRegex}} rewrites {{input ==~ /(a+)+$/}} to
{{RegexGuard.matchRegex(input, /(a+)+$/, 200)}}, the hoisting pass must still
recognise
the pattern -- now the *second* argument, not the first -- as a hoistable
constant and
lift it to a {{static final}} field.
* When hoisting runs first and produces a {{static final Pattern P}},
{{@SafeRegex}} must
still recognise a match against {{P}} and wrap the *input* in the deadline
guard.
Desired combined result: the constant {{Pattern}} is compiled once into a
{{static final}}
field *and* each match wraps the input in the deadline guard.
h2. Licensing / reuse
JLine is BSD-3 (ASF category A) and already on the {{groovysh}} classpath
(jline-builtins/jansi), so {{groovysh}} may use JLine's {{SafeRegex}} directly.
For *core*
Groovy the technique is ~20 lines and well-known, so it is reimplemented from
the
technique rather than taking a core dependency on JLine. No code was copied.
h2. References
* JLine 4.3.1 release notes -- SafeRegex / TimeoutCharSequence:
https://github.com/jline/jline3/releases/tag/4.3.1
* CWE-1333: Inefficient Regular Expression Complexity:
https://cwe.mitre.org/data/definitions/1333.html
* OWASP -- Regular expression Denial of Service (ReDoS):
https://owasp.org/www-community/attacks/Regular_expression_Denial_of_Service_-_ReDoS
* Related: GEP-25 (Data-flow Security -- taint tracking); {{RegexChecker}} in
groovy-typecheckers; {{@TimedInterrupt}} / {{@ThreadInterrupt}} /
{{@ConditionalInterrupt}} in groovy.transform
* Relates to: GROOVY-12123 (constant regex hoisting -- compile-once static
final Pattern fields): https://issues.apache.org/jira/browse/GROOVY-12123
* Pull request: https://github.com/apache/groovy/pull/2668
was:
h2. Summary
Hoist effectively-constant regular expressions to synthetic {{private static
final Pattern}}
fields so they are compiled once, rather than recompiled on every evaluation --
most
importantly inside loops. Transparent optimization; no new user-facing syntax.
h2. Motivation
Groovy's {{~/.../}} operator (and the {{=~}} / {{==~}} operators and JDK
conveniences such
as {{String#matches}}) construct/compile a {{Pattern}} each time they are
evaluated. In a
loop, a constant pattern is recompiled every iteration:
{code:groovy}
for (line in lines) {
if (line ==~ /\d{4}-\d{2}-\d{2}/) { ... } // recompiled every iteration
today
}
{code}
{{Pattern.compile}} is not cheap, and the pattern here never changes. Lifting
it to a
compile-once {{static final}} field removes the repeated compilation with no
change in
observable behaviour.
h2. Proposal
Where the compiler can prove a pattern is *effectively constant*, replace the
inline
regex expression with a reference to a synthetic {{private static final
Pattern}} field on
the enclosing class, initialized once. Identical (source + flags) patterns in
the same
class share a single field.
{code:groovy}
// conceptually, the loop above becomes:
private static final Pattern $pat$0 = Pattern.compile('\\d{4}-\\d{2}-\\d{2}')
...
for (line in lines) {
if ($pat$0.matcher(line).matches()) { ... }
}
{code}
This is loop-invariant code motion specialized for regex construction. It is
applied
automatically by the compiler when safe (see _Applicability_ and
_Correctness_), not via
an annotation.
h2. Applicability ("was applicable")
Only patterns with no runtime-dependent parts qualify:
* {{~/literal/}} with no interpolation -- hoistable.
* {{Pattern.compile("literal"[, constantFlags])}} -- hoistable (method-call
form).
* {{~"text ${CONST}"}} where the interpolated hole constant-folds -- hoistable
after folding.
* {{~"text ${var}"}} with a runtime {{var}} -- *not* hoistable; must build each
time.
* Any pattern closing over a captured, non-constant local -- *not* hoistable.
The primary targets are the {{~/.../}} operator and {{Pattern.compile}}.
Rewriting the JDK
convenience methods ({{String#matches}}/{{replaceAll}}/{{split}} with a
constant arg) into a
hoisted-{{Pattern}} form is a larger semantic change (routing through
{{Pattern}}/{{Matcher}})
and is a possible follow-up extension, not the initial cut.
h2. Correctness / caveats
* *Thread-safety is fine.* {{Pattern}} is immutable and thread-safe, so a shared
{{static final}} field is safe. {{Matcher}} is stateful and is *never*
hoisted -- only the
{{Pattern}}, with a fresh {{Matcher}} per match as today.
* *Exception timing.* An *invalid* constant pattern currently throws
{{PatternSyntaxException}}
at the point of evaluation (possibly never, if guarded by a branch). Eager
static-field
init would move that to class initialization
({{ExceptionInInitializerError}}). Mitigation:
only hoist patterns already known valid (the {{RegexChecker}} type checker
validates constant
literals), and/or use a lazy holder so init timing is preserved.
* *Identity.* User code must not rely on each {{~/.../}} yielding a distinct
{{Pattern}}
instance ({{===}}); {{Pattern}} carries no meaningful identity or mutability
contract, so
sharing is observationally equivalent.
h2. Scope
*In scope*
* Hoisting constant {{~/.../}} and {{Pattern.compile(constant[, constFlags])}}
to synthetic
{{static final}} fields, with per-class deduplication of identical patterns.
* Applying inside loops, closures, and script bodies (field on the enclosing
class).
*Out of scope / deferred*
* Rewriting {{String#matches}}/{{replaceAll}}/{{split}} convenience methods
(follow-up).
* Hoisting non-constant patterns or any form of runtime {{Pattern}} cache.
* Cross-class / cross-method caching.
h2. Relates to / must compose with GROOVY-12122
GROOVY-12122 adds a ReDoS runtime guard ({{RegexGuard}} helper, {{@SafeRegex}}
scoped
annotation). The two touch the same regex AST surface and sit on opposite ends
of the same
operation -- this ticket optimizes *Pattern construction*, GROOVY-12122 guards
*matching* --
so they compose, but the transforms must be ordering-aware:
* When {{@SafeRegex}} rewrites {{input ==~ /(a+)+$/}} to
{{RegexGuard.matches(~/(a+)+$/, input, ...)}}, this optimization must still
recognize the
{{~/(a+)+$/}} argument as a hoistable constant and lift it.
* When this optimization runs first and produces a {{static final Pattern P}},
{{@SafeRegex}}
must still recognize a match against {{P}} and wrap the input.
Desired combined result: the constant {{Pattern}} is compiled once into a
{{static final}}
field *and* each match wraps the input in the deadline guard. Whoever
implements the second
of the two should account for the first.
h2. Implementation notes
* A compiler optimization pass / AST transform over resolved expressions; runs
in both
dynamic and {{@CompileStatic}} modes.
* Constant detection reuses the same effectively-constant analysis as constant
folding.
* Synthetic field naming should be collision-free and deduplicated per class.
* Consider gating behind a flag initially (or restricting to
{{RegexChecker}}-validated
patterns) until the exception-timing behaviour is settled.
h2. References
* Related: GROOVY-12122 (ReDoS runtime guard -- RegexGuard / @SafeRegex):
https://issues.apache.org/jira/browse/GROOVY-12122
* {{java.util.regex.Pattern}} -- immutable/thread-safe; {{Matcher}} -- not
thread-safe
* {{RegexChecker}} in groovy-typecheckers (compile-time constant-pattern
validation)
> Hoist constant regexes to static final fields
> ---------------------------------------------
>
> Key: GROOVY-12123
> URL: https://issues.apache.org/jira/browse/GROOVY-12123
> Project: Groovy
> Issue Type: Improvement
> Reporter: Paul King
> Priority: Major
> Fix For: 7.x
>
>
> h2. Summary
> Add an opt-in, thread-free runtime guard against Regular Expression Denial of
> Service
> (ReDoS / catastrophic backtracking) for Groovy's regex features, offered as a
> runtime
> helper ({{RegexGuard}}), a scoped annotation ({{@SafeRegex}}), and
> type-checker awareness
> of both. Everything is opt-in; no existing code changes behaviour.
> h2. Motivation
> Groovy makes regex frictionless -- {{=~}}, {{==~}}, {{~/.../}},
> {{String#matches}},
> {{replaceAll}}, {{split}}, and the {{Matcher}} sugar -- which also makes
> ReDoS easy to
> reach when a pattern (or its input) comes from outside the program: scripts,
> build files,
> web handlers, {{ConfigSlurper}}, templating.
> {{java.util.regex}} has *no native timeout*, so a crafted input against a
> backtracking-prone pattern can hang the calling thread indefinitely. Nothing
> in Groovy
> guards against this today. Note the trust boundary is usually the *input*,
> not the
> pattern: the textbook ReDoS applies a hardcoded, developer-written validator
> to
> attacker-supplied text, so "don't run untrusted code" is not a sufficient
> answer.
> Thread interruption does not help: the regex engine never checks the
> interrupt flag, so
> neither {{@ThreadInterrupt}} nor {{@TimedInterrupt}} can stop a runaway match
> -- the hang
> is inside a single JDK library call, not in user code where those transforms
> insert checks.
> JLine 4.3.1 shipped a {{SafeRegex}} utility using the well-known
> {{TimeoutCharSequence}}
> technique to close four CVEs (GHSA-r2xf-8xr9-62gw, GHSA-2v9w-34q6-wpqx,
> GHSA-ph9c-7hw9-vhhw, GHSA-5q95-hrpc-m3w3). We apply the same idea.
> h2. Proposal
> Bound regex matching by a wall-clock deadline, without a watchdog thread. The
> input
> {{CharSequence}} is wrapped so that {{charAt()}} checks a deadline and throws
> an unchecked
> {{RegexTimeoutException}} once it has passed; because the regex engine
> re-reads input
> characters continuously while backtracking, a runaway match is interrupted
> with no extra
> thread and no {{Thread.interrupt}} plumbing. The clock is consulted once
> every 512 reads,
> so evaluations finishing in fewer reads never pay for a clock call.
> Four ways to apply it, from most explicit to most ergonomic.
> *(a) Explicit guarded evaluation* -- guarded equivalents of the match and
> find operators,
> with a {{long}} millisecond or {{Duration}} timeout:
> {code:groovy}
> RegexGuard.matches(pattern, input, 200) // -> boolean
> RegexGuard.matcher(pattern, input, Duration.ofMillis(200)) // -> Matcher
> over guarded input
> {code}
> *(b) Guarded data* -- attach a single total deadline to the sequence itself.
> Wrap
> untrusted input once at a trust boundary and pass it around freely; every
> regex evaluated
> against it shares the one budget, including Groovy's {{CharSequence}}
> extension methods
> and anything handing the sequence to {{java.util.regex}} untouched:
> {code:groovy}
> def safe = RegexGuard.guard(untrusted, 200)
> safe.findAll(~/\d+/)
> {code}
> *(c) Ambient scope* -- a per-evaluation timeout for the dynamic extent of a
> closure on the
> current thread, covering the regex operators and the {{CharSequence}}
> extension methods,
> including in methods called from the closure:
> {code:groovy}
> def groups = RegexGuard.guard(200) {
> input.findGroups(/(\d+)\.(\d+)\.(\d+)/)
> }
> {code}
> Nested scopes only tighten -- the smaller per-evaluation timeout wins -- and
> an explicit
> timeout from (a) is likewise capped by an active ambient timeout.
> *(d) Scoped annotation ({{@SafeRegex}})* -- an AST transform rewrites the
> match ({{==~}})
> and find ({{=~}}) operators lexically within the annotated scope to guarded
> calls, so no
> call-site changes are needed. Fits the existing {{@TimedInterrupt}} /
> {{@ThreadInterrupt}}
> / {{@ConditionalInterrupt}} family in {{groovy.transform}}:
> {code:groovy}
> @SafeRegex(millis = 200)
> class Handler {
> boolean check(String input) {
> input ==~ /(a+)+$/ // rewritten -> RegexGuard.matchRegex(input,
> /(a+)+$/, 200)
> }
> }
> {code}
> The annotation may be placed on a class, method, constructor, field or local
> variable
> declaration; on a field or local variable only the initializer expression is
> guarded. Its
> single attribute is {{millis}} (default {{1000}}, must be a positive
> constant). A guarded
> field initializer runs, as usual, in the constructor -- or in the static
> initializer for a
> static field, where a timeout surfaces as {{ExceptionInInitializerError}}
> whose cause is
> the {{RegexTimeoutException}}.
> The transform targets the operator-order entry points {{matchRegex(input,
> pattern,
> millis)}} and {{findRegex(input, pattern, millis)}} rather than
> {{matches(pattern, input, millis)}}, so a rewritten expression still
> evaluates its
> operands left to right exactly as the original operator did.
> The guard is on *matching*, not on {{~/.../}} compilation:
> {{Pattern.compile}} is not
> subject to backtracking and is not guarded.
> h3. Naming
> || Concern || Name || Notes ||
> | Scoped annotation | {{@SafeRegex}} | in {{groovy.transform}}, alongside
> {{@TimedInterrupt}} et al. |
> | Runtime helper | {{RegexGuard}} | in {{groovy.util.regex}}; distinct from
> the annotation to avoid a clash |
> | Timeout exception | {{RegexTimeoutException}} | unchecked; lets callers
> distinguish a ReDoS abort from a normal non-match |
> | Operator-order entry points | {{matchRegex}} / {{findRegex}} | mirror
> {{ScriptBytecodeAdapter}} naming and operand order; used by the transform |
> Both {{RegexGuard}} and {{@SafeRegex}} are marked {{@Incubating}}, {{@since
> 6.0.0}}.
> h3. Implementation notes
> * The ambient scope in (c) is held in an
> {{org.apache.groovy.runtime.async.ScopedLocal}},
> which delegates to {{java.lang.ScopedValue}} on JDK 25+ and falls back to a
> save-and-restore {{ThreadLocal}} below, so the binding cannot outlive its
> closure.
> * A process that never installs an ambient guard never touches that scoped
> local: a
> monotonic static flag short-circuits the lookup, leaving a single cached
> read on the
> regex path.
> * Deadlines are only ever compared as {{System.nanoTime()}} differences,
> which stays
> correct across wrap-around; a timeout large enough to overflow the deadline
> therefore
> reads as "effectively no timeout" rather than expiring immediately.
> h2. Scope
> *In scope*
> * Runtime deadline guard ({{TimeoutCharSequence}}-style), thread-free
> * {{RegexGuard}} helper: explicit guarded evaluation, guarded data, and
> ambient closure scope
> * {{@SafeRegex}} scoped annotation rewriting the {{==~}} and {{=~}} operators
> * {{RegexTimeoutException}} type
> * {{RegexChecker}} (groovy-typecheckers) awareness of guarded evaluation, so
> code inside a
> {{@SafeRegex}} scope keeps full regex checking after the rewrite
> * Documentation in the operator, metaprogramming and type-checker guides
> *Out of scope / deferred*
> * *Internal hardening of Groovy's own regex call sites* ({{groovysh}}
> completion,
> templating, builders) -- proposed originally, not implemented here;
> deferred to a
> follow-up so this ticket stays a purely opt-in, additive change
> * Making the guard the default for {{=~}} / {{==~}} (behaviour change +
> overhead)
> * Rewriting the {{String#matches}} / {{replaceAll}} / {{split}} family of
> method calls
> within a {{@SafeRegex}} scope -- only the two operators are rewritten; use
> {{RegexGuard}} explicitly for the rest
> * Static ReDoS pattern detection -- a possible future {{RegexChecker}}
> extension
> * Guarding {{Pattern.compile}}
> * Rewriting the regex engine or adopting a non-backtracking (RE2-style) engine
> h2. Limitations
> * {{@SafeRegex}} rewrites only the *lexically visible* {{==~}} and {{=~}}
> operators --
> scoped hardening, not whole-program. Regex evaluation via method calls, or
> occurring in
> code called from the annotated scope, is not rewritten; use {{RegexGuard}}
> there, whose
> ambient and guarded-data forms do reach both.
> * Direct JDK calls bypass the guard. On a {{String}} receiver,
> {{matches(String)}},
> {{replaceAll(String, String)}}, {{replaceFirst(String, String)}} and
> {{split(String)}}
> dispatch to {{java.lang.String}} and never enter Groovy's runtime. Their
> {{Pattern}}-argument variants are extension methods and are covered.
> * The deadline fires only while the engine is reading input characters (the
> backtracking
> phase) -- true of catastrophic cases, but not a hard guarantee for every
> pathological
> state. A well-behaved evaluation over a short input may still succeed after
> the deadline.
> * The clock is consulted every 512 character reads rather than on every read,
> so overshoot
> past the deadline is bounded by those reads; pathological patterns perform
> millions of
> reads per second, so in practice the overshoot is negligible.
> * The ambient scope is thread-confined: threads spawned within the closure
> are not covered
> (on JDK 25+ the binding does propagate into {{StructuredTaskScope}} forks).
> * Reduces hang risk; it is not a correctness or a data-at-rest control.
> h2. Relationship to existing/planned features
> || Layer || Mechanism || Status || Covers ReDoS? ||
> | Compile-time validity | {{RegexChecker}} (groovy-typecheckers) | ships | No
> -- pattern *syntax* only |
> | Compile-time ReDoS lint | possible {{RegexChecker}} extension | idea |
> Partially |
> | Untrusted-input tracking | GEP-25 {{@Tainted}} -> regex sink | draft |
> Identifies risky matches |
> | *Runtime guard (this ticket)* | {{RegexGuard}} / {{@SafeRegex}} |
> *implemented* | *Yes -- the actual backstop* |
> {{RegexChecker}} only validates that a literal pattern compiles; it says
> nothing about
> backtracking. As part of this ticket it was extended to recognise the guarded
> forms, so
> the rewrite performed by {{@SafeRegex}} does not blind it: bad regex literals
> are still
> reported, and matcher group-count inference survives the rewrite.
> h2. Ordering with GROOVY-12123
> GROOVY-12123 (constant regex hoisting) and this transform touch the same
> expressions, so
> whichever lands second must be ordering-aware:
> * When {{@SafeRegex}} rewrites {{input ==~ /(a+)+$/}} to
> {{RegexGuard.matchRegex(input, /(a+)+$/, 200)}}, the hoisting pass must
> still recognise
> the pattern -- now the *second* argument, not the first -- as a hoistable
> constant and
> lift it to a {{static final}} field.
> * When hoisting runs first and produces a {{static final Pattern P}},
> {{@SafeRegex}} must
> still recognise a match against {{P}} and wrap the *input* in the deadline
> guard.
> Desired combined result: the constant {{Pattern}} is compiled once into a
> {{static final}}
> field *and* each match wraps the input in the deadline guard.
> h2. Licensing / reuse
> JLine is BSD-3 (ASF category A) and already on the {{groovysh}} classpath
> (jline-builtins/jansi), so {{groovysh}} may use JLine's {{SafeRegex}}
> directly. For *core*
> Groovy the technique is ~20 lines and well-known, so it is reimplemented from
> the
> technique rather than taking a core dependency on JLine. No code was copied.
> h2. References
> * JLine 4.3.1 release notes -- SafeRegex / TimeoutCharSequence:
> https://github.com/jline/jline3/releases/tag/4.3.1
> * CWE-1333: Inefficient Regular Expression Complexity:
> https://cwe.mitre.org/data/definitions/1333.html
> * OWASP -- Regular expression Denial of Service (ReDoS):
> https://owasp.org/www-community/attacks/Regular_expression_Denial_of_Service_-_ReDoS
> * Related: GEP-25 (Data-flow Security -- taint tracking); {{RegexChecker}} in
> groovy-typecheckers; {{@TimedInterrupt}} / {{@ThreadInterrupt}} /
> {{@ConditionalInterrupt}} in groovy.transform
> * Relates to: GROOVY-12123 (constant regex hoisting -- compile-once static
> final Pattern fields): https://issues.apache.org/jira/browse/GROOVY-12123
> * Pull request: https://github.com/apache/groovy/pull/2668
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
