https://github.com/python/cpython/commit/25328b3477aaf2f8d739270e5ac87dd704ae0869
commit: 25328b3477aaf2f8d739270e5ac87dd704ae0869
branch: 3.14
author: Lysandros Nikolaou <[email protected]>
committer: lysnikolaou <[email protected]>
date: 2026-10-22T14:50:13-07:00
summary:
[3.14] gh-160268: Fix tokenize on a mismatched closing bracket in an f-string
(GH-160272) (#160336)
Since gh-105042, `tokenize` tolerates mismatched brackets. That also let
a mismatched closing bracket pop the brace that opened an f-string
replacement field, which the f-string state relies on. In `f'{]a'`, this
triggered an assertion failure in _PyLexer_scan_string.
Check the replacement field brace in both modes.
(cherry picked from commit 86758d1950ce2ed2626014b52d26d24c74dee55b)
files:
A
Misc/NEWS.d/next/Core_and_Builtins/2026-10-22-17-37-27.gh-issue-160268.gJ2iDx.rst
M Parser/lexer/lexer.c
diff --git
a/Misc/NEWS.d/next/Core_and_Builtins/2026-10-22-17-37-27.gh-issue-160268.gJ2iDx.rst
b/Misc/NEWS.d/next/Core_and_Builtins/2026-10-22-17-37-27.gh-issue-160268.gJ2iDx.rst
new file mode 100644
index 000000000000000..5132085d21258d2
--- /dev/null
+++
b/Misc/NEWS.d/next/Core_and_Builtins/2026-10-22-17-37-27.gh-issue-160268.gJ2iDx.rst
@@ -0,0 +1,3 @@
+Fix a crash in :mod:`tokenize` when a mismatched closing bracket follows the
+opening brace of an f-string or t-string replacement field, as in ``f'{]a'``.
+It now raises the same "unmatched" error as the parser.
diff --git a/Parser/lexer/lexer.c b/Parser/lexer/lexer.c
index af57deb12cbefb5..9c40b7d66d17f6c 100644
--- a/Parser/lexer/lexer.c
+++ b/Parser/lexer/lexer.c
@@ -1283,9 +1283,9 @@ tok_get_normal_mode(struct tok_state *tok,
tokenizer_mode* current_tok, struct t
if (tok->level > 0) {
tok->level--;
int opening = tok->parenstack[tok->level];
- if (!tok->tok_extra_tokens && !((opening == '(' && c == ')') ||
- (opening == '[' && c == ']') ||
- (opening == '{' && c == '}'))) {
+ if (!((opening == '(' && c == ')') ||
+ (opening == '[' && c == ']') ||
+ (opening == '{' && c == '}'))) {
/* If the opening bracket belongs to an f-string's expression
part (e.g. f"{)}") and the closing bracket is an arbitrary
nested expression, then instead of matching a different
@@ -1299,7 +1299,10 @@ tok_get_normal_mode(struct tok_state *tok,
tokenizer_mode* current_tok, struct t
"%c-string: unmatched '%c'",
TOK_GET_STRING_PREFIX(tok), c));
}
}
- if (tok->parenlinenostack[tok->level] != tok->lineno) {
+ if (tok->tok_extra_tokens) {
+ /* Allow other mismatched brackets. */
+ }
+ else if (tok->parenlinenostack[tok->level] != tok->lineno) {
return MAKE_TOKEN(_PyTokenizer_syntaxerror(tok,
"closing parenthesis '%c' does not match "
"opening parenthesis '%c' on line %d",
_______________________________________________
Python-checkins mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3//lists/python-checkins.python.org
Member address: [email protected]
