Script 'mail_helper' called by obssrc
Hello community,
here is the log from the commit of package python-asttokens for
openSUSE:Factory checked in at 2026-08-21 16:50:16
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Comparing /work/SRC/openSUSE:Factory/python-asttokens (Old)
and /work/SRC/openSUSE:Factory/.python-asttokens.new.1258 (New)
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "python-asttokens"
Fri Aug 21 16:50:16 2026 rev:16 rq:1372222 version:3.0.2
Changes:
--------
--- /work/SRC/openSUSE:Factory/python-asttokens/python-asttokens.changes
2025-11-18 15:30:08.490881748 +0100
+++
/work/SRC/openSUSE:Factory/.python-asttokens.new.1258/python-asttokens.changes
2026-08-21 16:50:56.495883916 +0200
@@ -1,0 +2,10 @@
+Thu Aug 20 09:06:20 UTC 2026 - Dirk Müller <[email protected]>
+
+- update to 3.0.2:
+ * Modernize type annotations
+ * Fix pypy CI by skipping redundant mypy runs on it
+ * Fix IndexError on source mixing lone CR with LF line endings
+ * Release modernization, and publish to PyPI from CI via
+ trusted publishing
+
+-------------------------------------------------------------------
Old:
----
asttokens-3.0.1.tar.gz
New:
----
asttokens-3.0.2.tar.gz
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Other differences:
------------------
++++++ python-asttokens.spec ++++++
--- /var/tmp/diff_new_pack.J3e8DB/_old 2026-08-21 16:50:57.184908662 +0200
+++ /var/tmp/diff_new_pack.J3e8DB/_new 2026-08-21 16:50:57.187908769 +0200
@@ -1,7 +1,7 @@
#
# spec file for package python-asttokens
#
-# Copyright (c) 2025 SUSE LLC and contributors
+# Copyright (c) 2026 SUSE LLC and contributors
# Copyright (c) 2019-2021 Malcolm J Lewis <[email protected]>
#
# All modifications and additions to the file contributed by third parties
@@ -19,7 +19,7 @@
%{?sle15_python_module_pythons}
Name: python-asttokens
-Version: 3.0.1
+Version: 3.0.2
Release: 0
Summary: Annotate AST trees with source code positions
License: Apache-2.0
++++++ asttokens-3.0.1.tar.gz -> asttokens-3.0.2.tar.gz ++++++
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/asttokens-3.0.1/.github/workflows/build-and-test.yml
new/asttokens-3.0.2/.github/workflows/build-and-test.yml
--- old/asttokens-3.0.1/.github/workflows/build-and-test.yml 2025-11-15
17:04:32.000000000 +0100
+++ new/asttokens-3.0.2/.github/workflows/build-and-test.yml 2026-07-12
05:31:23.000000000 +0200
@@ -64,8 +64,10 @@
pip install .[test] 'astroid${{ matrix.astroid-version }}'
- name: Mypy testing
+ # Skip testing mypy on pypy since mypy no longer installs there, and
+ # type-checking on pypy is anyway redundant with CPython jobs.
+ if: ${{ !startsWith(matrix.python-version, 'pypy') }}
run: |
- # Not an exact mypy version, as we need 0.942 for pypy-3.8 support,
but it's not available on 3.5
pip install "mypy>=1.10"
python -m mypy asttokens tests/*.py
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/asttokens-3.0.1/.github/workflows/publish.yml
new/asttokens-3.0.2/.github/workflows/publish.yml
--- old/asttokens-3.0.1/.github/workflows/publish.yml 1970-01-01
01:00:00.000000000 +0100
+++ new/asttokens-3.0.2/.github/workflows/publish.yml 2026-07-12
05:31:23.000000000 +0200
@@ -0,0 +1,22 @@
+name: Publish to PyPI
+
+on:
+ release:
+ types: [published]
+
+jobs:
+ publish:
+ runs-on: ubuntu-latest
+ environment: pypi
+ permissions:
+ id-token: write
+ steps:
+ - uses: actions/checkout@v4
+ with:
+ fetch-depth: 0 # setuptools_scm derives the version from git tags
+ - uses: actions/setup-python@v5
+ with:
+ python-version: "3.x"
+ - run: pip install build
+ - run: python -m build
+ - uses: pypa/gh-action-pypi-publish@release/v1
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/asttokens-3.0.1/Makefile new/asttokens-3.0.2/Makefile
--- old/asttokens-3.0.1/Makefile 2024-11-30 05:46:25.000000000 +0100
+++ new/asttokens-3.0.2/Makefile 1970-01-01 01:00:00.000000000 +0100
@@ -1,21 +0,0 @@
-dist:
- @echo Build python distribution
- @echo "(If no 'build' module, install with 'python -m pip install build
setuptools_scm')"
- python -m build
-
-publish:
- @echo "Publish to PyPI at https://pypi.python.org/pypi/asttokens/";
- @VER=`python -c 'import setuptools_scm;
print(setuptools_scm.get_version())'`; \
- echo "Version in setup.py is $$VER"; \
- echo "Git tag is `git describe --tags`"; \
- echo "Run this manually: twine upload dist/asttokens-$$VER*"
-
-docs:
- @echo Build documentation in docs/_build/html
- source env/bin/activate ; PYTHONPATH=$(abspath .) $(MAKE) -C docs html
-
-clean:
- python setup.py clean
- source env/bin/activate ; PYTHONPATH=$(abspath .) $(MAKE) -C docs clean
-
-.PHONY: dist publish docs clean
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/asttokens-3.0.1/PKG-INFO new/asttokens-3.0.2/PKG-INFO
--- old/asttokens-3.0.1/PKG-INFO 2025-11-15 17:13:26.206192000 +0100
+++ new/asttokens-3.0.2/PKG-INFO 2026-07-12 05:31:31.914345300 +0200
@@ -1,6 +1,6 @@
Metadata-Version: 2.4
Name: asttokens
-Version: 3.0.1
+Version: 3.0.2
Summary: Annotate AST trees with source code positions
Home-page: https://github.com/gristlabs/asttokens
Author: Dmitry Sagalovskiy, Grist Labs
@@ -119,3 +119,23 @@
4. Run tests across all supported interpreters with the ``tox`` command. You
will need to have the interpreters installed separately. We recommend ``pyenv``
for that. Use ``tox -p auto`` to run the tests in parallel.
5. By default certain tests which take a very long time to run are skipped,
but they are run in CI.
These are marked using the ``pytest`` marker ``slow`` and can be run on
their own with ``pytest -m slow`` or as part of the full suite with ``pytest -m
''``.
+
+Documentation
+-------------
+
+Documentation at https://asttokens.readthedocs.io/ is built by Read the Docs
automatically on
+every push, from the sources in ``docs/`` (configured in
``.readthedocs.yaml``). To preview
+changes locally::
+
+ pip install -r docs/requirements.txt
+ sphinx-build -M html docs docs/_build
+
+Release
+-------
+
+The version is derived from the git tag by ``setuptools_scm``; there is no
version to update in
+the source. To release, draft a new release at
https://github.com/gristlabs/asttokens/releases
+with a new tag named ``vX.Y.Z``, and use "Generate release notes" to fill in
the description.
+Publishing the release triggers the ``publish.yml`` workflow, which builds the
package and
+uploads it to PyPI via `trusted publishing
+<https://docs.pypi.org/trusted-publishers/>`_.
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/asttokens-3.0.1/README.rst
new/asttokens-3.0.2/README.rst
--- old/asttokens-3.0.1/README.rst 2025-11-03 06:34:32.000000000 +0100
+++ new/asttokens-3.0.2/README.rst 2026-07-12 05:31:23.000000000 +0200
@@ -80,3 +80,23 @@
4. Run tests across all supported interpreters with the ``tox`` command. You
will need to have the interpreters installed separately. We recommend ``pyenv``
for that. Use ``tox -p auto`` to run the tests in parallel.
5. By default certain tests which take a very long time to run are skipped,
but they are run in CI.
These are marked using the ``pytest`` marker ``slow`` and can be run on
their own with ``pytest -m slow`` or as part of the full suite with ``pytest -m
''``.
+
+Documentation
+-------------
+
+Documentation at https://asttokens.readthedocs.io/ is built by Read the Docs
automatically on
+every push, from the sources in ``docs/`` (configured in
``.readthedocs.yaml``). To preview
+changes locally::
+
+ pip install -r docs/requirements.txt
+ sphinx-build -M html docs docs/_build
+
+Release
+-------
+
+The version is derived from the git tag by ``setuptools_scm``; there is no
version to update in
+the source. To release, draft a new release at
https://github.com/gristlabs/asttokens/releases
+with a new tag named ``vX.Y.Z``, and use "Generate release notes" to fill in
the description.
+Publishing the release triggers the ``publish.yml`` workflow, which builds the
package and
+uploads it to PyPI via `trusted publishing
+<https://docs.pypi.org/trusted-publishers/>`_.
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/asttokens-3.0.1/asttokens/asttokens.py
new/asttokens-3.0.2/asttokens/asttokens.py
--- old/asttokens-3.0.1/asttokens/asttokens.py 2025-11-15 16:46:22.000000000
+0100
+++ new/asttokens-3.0.2/asttokens/asttokens.py 2026-07-12 05:31:23.000000000
+0200
@@ -18,17 +18,14 @@
import sys
import token
from ast import Module
-from typing import Iterable, Iterator, List, Optional, Tuple, Any, cast,
TYPE_CHECKING
+from typing import Iterable, Iterator, List, Optional, Tuple, Any, cast
from .line_numbers import LineNumbers
from .util import (
- Token, match_token, is_non_coding_token, patched_generate_tokens, last_stmt,
+ AstNode, Token, TokenInfo, match_token, is_non_coding_token,
patched_generate_tokens, last_stmt,
annotate_fstring_nodes, generate_tokens, is_module, is_stmt
)
-if TYPE_CHECKING: # pragma: no cover
- from .util import AstNode, TokenInfo
-
class ASTTextBase(metaclass=abc.ABCMeta):
def __init__(self, source_text: str, filename: str) -> None:
@@ -43,8 +40,9 @@
self._line_numbers = LineNumbers(source_text)
@abc.abstractmethod
- def get_text_positions(self, node, padded):
- # type: (AstNode, bool) -> Tuple[Tuple[int, int], Tuple[int, int]]
+ def get_text_positions(
+ self, node: AstNode, padded: bool
+ ) -> Tuple[Tuple[int, int], Tuple[int, int]]:
"""
Returns two ``(lineno, col_offset)`` tuples for the start and end of the
given node.
If the positions can't be determined, or the nodes don't correspond to any
particular text,
@@ -56,8 +54,7 @@
"""
raise NotImplementedError # pragma: no cover
- def get_text_range(self, node, padded=True):
- # type: (AstNode, bool) -> Tuple[int, int]
+ def get_text_range(self, node: AstNode, padded: bool = True) -> Tuple[int,
int]:
"""
Returns the (startpos, endpos) positions in source text corresponding to
the given node.
Returns (0, 0) for nodes (like `Load`) that don't correspond to any
particular text.
@@ -70,8 +67,7 @@
self._line_numbers.line_to_offset(*end),
)
- def get_text(self, node, padded=True):
- # type: (AstNode, bool) -> str
+ def get_text(self, node: AstNode, padded: bool = True) -> str:
"""
Returns the text corresponding to the given node.
Returns '' for nodes (like `Load`) that don't correspond to any particular
text.
@@ -102,9 +98,15 @@
tree created separately.
"""
- def __init__(self, source_text, parse=False, tree=None,
filename='<unknown>', tokens=None):
- # type: (Any, bool, Optional[Module], str, Optional[Iterable[TokenInfo]])
-> None
- super(ASTTokens, self).__init__(source_text, filename)
+ def __init__(
+ self,
+ source_text: Any,
+ parse: bool = False,
+ tree: Optional[Module] = None,
+ filename: str = '<unknown>',
+ tokens: Optional[Iterable[TokenInfo]] = None
+ ) -> None:
+ super().__init__(source_text, filename)
self._tree = ast.parse(source_text, filename) if parse else tree
@@ -119,8 +121,7 @@
if self._tree:
self.mark_tokens(self._tree)
- def mark_tokens(self, root_node):
- # type: (Module) -> None
+ def mark_tokens(self, root_node: Module) -> None:
"""
Given the root of the AST or Astroid tree produced from source_text,
visits all nodes marking
them with token and position information by adding ``.first_token`` and
@@ -131,8 +132,7 @@
from .mark_tokens import MarkTokens # to avoid import loops
MarkTokens(self).visit_tree(root_node)
- def _translate_tokens(self, original_tokens):
- # type: (Iterable[TokenInfo]) -> Iterator[Token]
+ def _translate_tokens(self, original_tokens: Iterable[TokenInfo]) ->
Iterator[Token]:
"""
Translates the given standard library tokens into our own representation.
"""
@@ -143,39 +143,33 @@
self._line_numbers.line_to_offset(end[0], end[1]))
@property
- def text(self):
- # type: () -> str
+ def text(self) -> str:
"""The source code passed into the constructor."""
return self._text
@property
- def tokens(self):
- # type: () -> List[Token]
+ def tokens(self) -> List[Token]:
"""The list of tokens corresponding to the source code from the
constructor."""
return self._tokens
@property
- def tree(self):
- # type: () -> Optional[Module]
+ def tree(self) -> Optional[Module]:
"""The root of the AST tree passed into the constructor or parsed from the
source code."""
return self._tree
@property
- def filename(self):
- # type: () -> str
+ def filename(self) -> str:
"""The filename that was parsed"""
return self._filename
- def get_token_from_offset(self, offset):
- # type: (int) -> Token
+ def get_token_from_offset(self, offset: int) -> Token:
"""
Returns the token containing the given character offset (0-based position
in source text),
or the preceeding token if the position is between tokens.
"""
return self._tokens[bisect.bisect(self._token_offsets, offset) - 1]
- def get_token(self, lineno, col_offset):
- # type: (int, int) -> Token
+ def get_token(self, lineno: int, col_offset: int) -> Token:
"""
Returns the token containing the given (lineno, col_offset) position, or
the preceeding token
if the position is between tokens.
@@ -185,15 +179,13 @@
# but isn't explicit.
return
self.get_token_from_offset(self._line_numbers.line_to_offset(lineno,
col_offset))
- def get_token_from_utf8(self, lineno, col_offset):
- # type: (int, int) -> Token
+ def get_token_from_utf8(self, lineno: int, col_offset: int) -> Token:
"""
Same as get_token(), but interprets col_offset as a UTF8 offset, which is
what `ast` uses.
"""
return self.get_token(lineno, self._line_numbers.from_utf8_col(lineno,
col_offset))
- def next_token(self, tok, include_extra=False):
- # type: (Token, bool) -> Token
+ def next_token(self, tok: Token, include_extra: bool = False) -> Token:
"""
Returns the next token after the given one. If include_extra is True,
includes non-coding
tokens from the tokenize module, such as NL and COMMENT.
@@ -204,8 +196,7 @@
i += 1
return self._tokens[i]
- def prev_token(self, tok, include_extra=False):
- # type: (Token, bool) -> Token
+ def prev_token(self, tok: Token, include_extra: bool = False) -> Token:
"""
Returns the previous token before the given one. If include_extra is True,
includes non-coding
tokens from the tokenize module, such as NL and COMMENT.
@@ -216,8 +207,9 @@
i -= 1
return self._tokens[i]
- def find_token(self, start_token, tok_type, tok_str=None, reverse=False):
- # type: (Token, int, Optional[str], bool) -> Token
+ def find_token(
+ self, start_token: Token, tok_type: int, tok_str: Optional[str] = None,
reverse: bool = False
+ ) -> Token:
"""
Looks for the first token, starting at start_token, that matches tok_type
and, if given, the
token string. Searches backwards if reverse is True. Returns ENDMARKER
token if not found (you
@@ -229,12 +221,12 @@
t = advance(t, include_extra=True)
return t
- def token_range(self,
- first_token, # type: Token
- last_token, # type: Token
- include_extra=False, # type: bool
- ):
- # type: (...) -> Iterator[Token]
+ def token_range(
+ self,
+ first_token: Token,
+ last_token: Token,
+ include_extra: bool = False,
+ ) -> Iterator[Token]:
"""
Yields all tokens in order from first_token through and including
last_token. If
include_extra is True, includes non-coding tokens such as tokenize.NL and
.COMMENT.
@@ -243,16 +235,14 @@
if include_extra or not is_non_coding_token(self._tokens[i].type):
yield self._tokens[i]
- def get_tokens(self, node, include_extra=False):
- # type: (AstNode, bool) -> Iterator[Token]
+ def get_tokens(self, node: AstNode, include_extra: bool = False) ->
Iterator[Token]:
"""
Yields all tokens making up the given node. If include_extra is True,
includes non-coding
tokens such as tokenize.NL and .COMMENT.
"""
return self.token_range(node.first_token, node.last_token,
include_extra=include_extra)
- def get_text_positions(self, node, padded):
- # type: (AstNode, bool) -> Tuple[Tuple[int, int], Tuple[int, int]]
+ def get_text_positions(self, node: AstNode, padded: bool) ->
Tuple[Tuple[int, int], Tuple[int, int]]:
"""
Returns two ``(lineno, col_offset)`` tuples for the start and end of the
given node.
If the positions can't be determined, or the nodes don't correspond to any
particular text,
@@ -287,27 +277,24 @@
which incurs the usual setup cost the first time.
If you want to avoid this, check ``supports_tokenless(node)`` before calling
``get_text*`` methods.
"""
- def __init__(self, source_text, tree=None, filename='<unknown>'):
- # type: (Any, Optional[Module], str) -> None
- super(ASTText, self).__init__(source_text, filename)
+ def __init__(self, source_text: Any, tree: Optional[Module] = None,
filename: str = '<unknown>') -> None:
+ super().__init__(source_text, filename)
self._tree = tree
if self._tree is not None:
annotate_fstring_nodes(self._tree)
- self._asttokens = None # type: Optional[ASTTokens]
+ self._asttokens: Optional[ASTTokens] = None
@property
- def tree(self):
- # type: () -> Module
+ def tree(self) -> Module:
if self._tree is None:
self._tree = ast.parse(self._text, self._filename)
annotate_fstring_nodes(self._tree)
return self._tree
@property
- def asttokens(self):
- # type: () -> ASTTokens
+ def asttokens(self) -> ASTTokens:
if self._asttokens is None:
self._asttokens = ASTTokens(
self._text,
@@ -316,8 +303,9 @@
)
return self._asttokens
- def _get_text_positions_tokenless(self, node, padded):
- # type: (AstNode, bool) -> Tuple[Tuple[int, int], Tuple[int, int]]
+ def _get_text_positions_tokenless(
+ self, node: AstNode, padded: bool
+ ) -> Tuple[Tuple[int, int], Tuple[int, int]]:
"""
Version of ``get_text_positions()`` that doesn't use tokens.
"""
@@ -382,8 +370,7 @@
return start, end
- def get_text_positions(self, node, padded):
- # type: (AstNode, bool) -> Tuple[Tuple[int, int], Tuple[int, int]]
+ def get_text_positions(self, node: AstNode, padded: bool) ->
Tuple[Tuple[int, int], Tuple[int, int]]:
"""
Returns two ``(lineno, col_offset)`` tuples for the start and end of the
given node.
If the positions can't be determined, or the nodes don't correspond to any
particular text,
@@ -405,7 +392,7 @@
# Node types that _get_text_positions_tokenless doesn't support.
# These initial values are missing lineno.
-_unsupported_tokenless_types = ("arguments", "Arguments", "withitem") # type:
Tuple[str, ...]
+_unsupported_tokenless_types: Tuple[str, ...] = ("arguments", "Arguments",
"withitem")
if sys.version_info[:2] == (3, 8):
# _get_text_positions_tokenless works incorrectly for these types due to
bugs in Python 3.8.
_unsupported_tokenless_types += ("arg", "Starred")
@@ -413,8 +400,7 @@
_unsupported_tokenless_types += ("Slice", "ExtSlice", "Index", "keyword")
-def supports_tokenless(node=None):
- # type: (Any) -> bool
+def supports_tokenless(node: Any = None) -> bool:
"""
Returns True if the Python version and the node (if given) are supported by
the ``get_text*`` methods of ``ASTText`` without falling back to
``ASTTokens``.
@@ -440,10 +426,8 @@
and not (
# astroid nodes
not isinstance(node, ast.AST) and node is not None and (
- (
type(node).__name__ == "AssignName"
and type(node.parent).__name__ in ("Arguments", "ExceptHandler")
- )
)
)
and 'pypy' not in sys.version.lower()
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/asttokens-3.0.1/asttokens/line_numbers.py
new/asttokens-3.0.2/asttokens/line_numbers.py
--- old/asttokens-3.0.1/asttokens/line_numbers.py 2025-11-03
07:07:34.000000000 +0100
+++ new/asttokens-3.0.2/asttokens/line_numbers.py 2026-07-12
05:31:23.000000000 +0200
@@ -16,7 +16,10 @@
import re
from typing import Dict, List, Tuple
-_line_start_re = re.compile(r'^', re.M)
+# Matches the end-of-line sequences that Python treats as line boundaries in
source code, i.e.
+# "\r\n", "\r", or "\n". Using this (rather than a plain `re.M` `^`) means we
recognise a lone
+# "\r" as a line separator, matching how the tokenizer and ast module number
lines. See issue #105.
+_line_end_re = re.compile(r'\r\n|\r|\n')
class LineNumbers:
"""
@@ -26,16 +29,15 @@
This class expects unicode for input and stores positions in unicode. But it
supports
translating to and from utf8 offsets, which are used by ast parsing.
"""
- def __init__(self, text):
- # type: (str) -> None
- # A list of character offsets of each line's first character.
- self._line_offsets = [m.start(0) for m in _line_start_re.finditer(text)]
+ def __init__(self, text: str) -> None:
+ # A list of character offsets of each line's first character. The first
line always starts at
+ # offset 0, and each subsequent line starts right after an end-of-line
sequence.
+ self._line_offsets = [0] + [m.end(0) for m in _line_end_re.finditer(text)]
self._text = text
self._text_len = len(text)
- self._utf8_offset_cache = {} # type: Dict[int, List[int]] # maps line num
to list of char offset for each byte in line
+ self._utf8_offset_cache: Dict[int, List[int]] = {} # maps line num to list
of char offset for each byte in line
- def from_utf8_col(self, line, utf8_column):
- # type: (int, int) -> int
+ def from_utf8_col(self, line: int, utf8_column: int) -> int:
"""
Given a 1-based line number and 0-based utf8 column, returns a 0-based
unicode column.
"""
@@ -50,8 +52,7 @@
return offsets[max(0, min(len(offsets)-1, utf8_column))]
- def line_to_offset(self, line, column):
- # type: (int, int) -> int
+ def line_to_offset(self, line: int, column: int) -> int:
"""
Converts 1-based line number and 0-based column to 0-based character
offset into text.
"""
@@ -63,8 +64,7 @@
else:
return min(self._line_offsets[line] + max(0, column), self._text_len)
- def offset_to_line(self, offset):
- # type: (int) -> Tuple[int, int]
+ def offset_to_line(self, offset: int) -> Tuple[int, int]:
"""
Converts 0-based character offset to pair (line, col) of 1-based line and
0-based column
numbers.
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/asttokens-3.0.1/asttokens/mark_tokens.py
new/asttokens-3.0.2/asttokens/mark_tokens.py
--- old/asttokens-3.0.1/asttokens/mark_tokens.py 2025-11-15
17:04:32.000000000 +0100
+++ new/asttokens-3.0.2/asttokens/mark_tokens.py 2026-07-12
05:31:23.000000000 +0200
@@ -22,9 +22,7 @@
from . import util
from .asttokens import ASTTokens
from .astroid_compat import astroid_node_classes as nc, BaseContainer as
AstroidBaseContainer
-
-if TYPE_CHECKING:
- from .util import AstNode
+from .util import AstNode
# Mapping of matching braces. To find a token here, look up token[:2].
@@ -46,19 +44,18 @@
Helper that visits all nodes in the AST tree and assigns .first_token and
.last_token attributes
to each of them. This is the heart of the token-marking logic.
"""
- def __init__(self, code):
- # type: (ASTTokens) -> None
+ def __init__(self, code: ASTTokens) -> None:
self._code = code
self._methods = util.NodeMethods()
- self._iter_children = None # type: Optional[Callable]
+ self._iter_children: Optional[Callable] = None
- def visit_tree(self, node):
- # type: (Module) -> None
+ def visit_tree(self, node: Module) -> None:
self._iter_children = util.iter_children_func(node)
util.visit_tree(node, self._visit_before_children,
self._visit_after_children)
- def _visit_before_children(self, node, parent_token):
- # type: (AstNode, Optional[util.Token]) -> Tuple[Optional[util.Token],
Optional[util.Token]]
+ def _visit_before_children(
+ self, node: AstNode, parent_token: Optional[util.Token]
+ ) -> Tuple[Optional[util.Token], Optional[util.Token]]:
col = getattr(node, 'col_offset', None)
token = self._code.get_token_from_utf8(node.lineno, col) if col is not
None else None
@@ -70,8 +67,9 @@
# parent_token argument. The second value becomes the token argument of
_visit_after_children.
return (token or parent_token, token)
- def _visit_after_children(self, node, parent_token, token):
- # type: (AstNode, Optional[util.Token], Optional[util.Token]) -> None
+ def _visit_after_children(
+ self, node: AstNode, parent_token: Optional[util.Token], token:
Optional[util.Token]
+ ) -> None:
# This processes the node generically first, after all children have been
processed.
# Get the first and last tokens that belong to children. Note how this
doesn't assume that we
@@ -112,8 +110,7 @@
node.first_token = nfirst
node.last_token = nlast
- def _find_last_in_stmt(self, start_token):
- # type: (util.Token) -> util.Token
+ def _find_last_in_stmt(self, start_token: util.Token) -> util.Token:
t = start_token
while (not util.match_token(t, token.NEWLINE) and
not util.match_token(t, token.OP, ';') and
@@ -121,15 +118,16 @@
t = self._code.next_token(t, include_extra=True)
return self._code.prev_token(t)
- def _expand_to_matching_pairs(self, first_token, last_token, node):
- # type: (util.Token, util.Token, AstNode) -> Tuple[util.Token, util.Token]
+ def _expand_to_matching_pairs(
+ self, first_token: util.Token, last_token: util.Token, node: AstNode
+ ) -> Tuple[util.Token, util.Token]:
"""
Scan tokens in [first_token, last_token] range that are between node's
children, and for any
unmatched brackets, adjust first/last tokens to include the closing pair.
"""
# We look for opening parens/braces among non-child tokens (i.e. tokens
between our actual
# child nodes). If we find any closing ones, we match them to the opens.
- to_match_right = [] # type: List[Tuple[int, str]]
+ to_match_right: List[Tuple[int, str]] = []
to_match_left = []
for tok in self._code.token_range(first_token, last_token):
tok_info = tok[:2]
@@ -162,39 +160,43 @@
# Node visitors. Each takes a preliminary first and last tokens, and returns
the adjusted pair
# that will actually be assigned.
- def visit_default(self, node, first_token, last_token):
- # type: (AstNode, util.Token, util.Token) -> Tuple[util.Token, util.Token]
+ def visit_default(
+ self, node: AstNode, first_token: util.Token, last_token: util.Token
+ ) -> Tuple[util.Token, util.Token]:
# pylint: disable=no-self-use
# By default, we don't need to adjust the token we computed earlier.
return (first_token, last_token)
- def handle_comp(self, open_brace, node, first_token, last_token):
- # type: (str, AstNode, util.Token, util.Token) -> Tuple[util.Token,
util.Token]
+ def handle_comp(
+ self, open_brace: str, node: AstNode, first_token: util.Token, last_token:
util.Token
+ ) -> Tuple[util.Token, util.Token]:
# For list/set/dict comprehensions, we only get the token of the first
child, so adjust it to
# include the opening brace (the closing brace will be matched
automatically).
before = self._code.prev_token(first_token)
util.expect_token(before, token.OP, open_brace)
return (before, last_token)
- def visit_comprehension(self,
- node, # type: AstNode
- first_token, # type: util.Token
- last_token, # type: util.Token
- ):
- # type: (...) -> Tuple[util.Token, util.Token]
+ def visit_comprehension(
+ self,
+ node: AstNode,
+ first_token: util.Token,
+ last_token: util.Token,
+ ) -> Tuple[util.Token, util.Token]:
# The 'comprehension' node starts with 'for' but we only get first child;
we search backwards
# to find the 'for' keyword.
first = self._code.find_token(first_token, token.NAME, 'for', reverse=True)
return (first, last_token)
- def visit_if(self, node, first_token, last_token):
- # type: (util.Token, util.Token, util.Token) -> Tuple[util.Token,
util.Token]
+ def visit_if(
+ self, node: util.Token, first_token: util.Token, last_token: util.Token
+ ) -> Tuple[util.Token, util.Token]:
while first_token.string not in ('if', 'elif'):
first_token = self._code.prev_token(first_token)
return first_token, last_token
- def handle_attr(self, node, first_token, last_token):
- # type: (AstNode, util.Token, util.Token) -> Tuple[util.Token, util.Token]
+ def handle_attr(
+ self, node: AstNode, first_token: util.Token, last_token: util.Token
+ ) -> Tuple[util.Token, util.Token]:
# Attribute node has ".attr" (2 tokens) after the last child.
dot = self._code.find_token(last_token, token.OP, '.')
name = self._code.next_token(dot)
@@ -205,8 +207,9 @@
visit_assignattr = handle_attr
visit_delattr = handle_attr
- def handle_def(self, node, first_token, last_token):
- # type: (AstNode, util.Token, util.Token) -> Tuple[util.Token, util.Token]
+ def handle_def(
+ self, node: AstNode, first_token: util.Token, last_token: util.Token
+ ) -> Tuple[util.Token, util.Token]:
# With astroid, nodes that start with a doc-string can have an empty body,
in which case we
# need to adjust the last token to include the doc string.
if not node.body and (getattr(node, 'doc_node', None) or getattr(node,
'doc', None)): # type: ignore[union-attr]
@@ -222,8 +225,9 @@
visit_classdef = handle_def
visit_functiondef = handle_def
- def handle_following_brackets(self, node, last_token, opening_bracket):
- # type: (AstNode, util.Token, str) -> util.Token
+ def handle_following_brackets(
+ self, node: AstNode, last_token: util.Token, opening_bracket: str
+ ) -> util.Token:
# This is for calls and subscripts, which have a pair of brackets
# at the end which may contain no nodes, e.g. foo() or bar[:].
# We look for the opening bracket and then let the matching pair be found
automatically
@@ -235,8 +239,9 @@
last_token = call_start
return last_token
- def visit_call(self, node, first_token, last_token):
- # type: (util.Token, util.Token, util.Token) -> Tuple[util.Token,
util.Token]
+ def visit_call(
+ self, node: util.Token, first_token: util.Token, last_token: util.Token
+ ) -> Tuple[util.Token, util.Token]:
last_token = self.handle_following_brackets(node, last_token, '(')
# Handling a python bug with decorators with empty parens, e.g.
@@ -246,22 +251,24 @@
first_token = self._code.next_token(first_token)
return (first_token, last_token)
- def visit_matchclass(self, node, first_token, last_token):
- # type: (util.Token, util.Token, util.Token) -> Tuple[util.Token,
util.Token]
+ def visit_matchclass(
+ self, node: util.Token, first_token: util.Token, last_token: util.Token
+ ) -> Tuple[util.Token, util.Token]:
last_token = self.handle_following_brackets(node, last_token, '(')
return (first_token, last_token)
- def visit_subscript(self,
- node, # type: AstNode
- first_token, # type: util.Token
- last_token, # type: util.Token
- ):
- # type: (...) -> Tuple[util.Token, util.Token]
+ def visit_subscript(
+ self,
+ node: AstNode,
+ first_token: util.Token,
+ last_token: util.Token,
+ ) -> Tuple[util.Token, util.Token]:
last_token = self.handle_following_brackets(node, last_token, '[')
return (first_token, last_token)
- def visit_slice(self, node, first_token, last_token):
- # type: (AstNode, util.Token, util.Token) -> Tuple[util.Token, util.Token]
+ def visit_slice(
+ self, node: AstNode, first_token: util.Token, last_token: util.Token
+ ) -> Tuple[util.Token, util.Token]:
# consume `:` tokens to the left and right. In Python 3.9, Slice nodes are
# given a col_offset, (and end_col_offset), so this will always start
inside
# the slice, even if it is the empty slice. However, in 3.8 and below, this
@@ -280,8 +287,9 @@
last_token = next_
return (first_token, last_token)
- def handle_bare_tuple(self, node, first_token, last_token):
- # type: (AstNode, util.Token, util.Token) -> Tuple[util.Token, util.Token]
+ def handle_bare_tuple(
+ self, node: AstNode, first_token: util.Token, last_token: util.Token
+ ) -> Tuple[util.Token, util.Token]:
# A bare tuple doesn't include parens; if there is a trailing comma, make
it part of the tuple.
maybe_comma = self._code.next_token(last_token)
if util.match_token(maybe_comma, token.OP, ','):
@@ -289,8 +297,9 @@
return (first_token, last_token)
# In Python3.8 parsed tuples include parentheses when present.
- def handle_tuple_nonempty(self, node, first_token, last_token):
- # type: (AstNode, util.Token, util.Token) -> Tuple[util.Token, util.Token]
+ def handle_tuple_nonempty(
+ self, node: AstNode, first_token: util.Token, last_token: util.Token
+ ) -> Tuple[util.Token, util.Token]:
assert isinstance(node, ast.Tuple) or isinstance(node,
AstroidBaseContainer)
# It's a bare tuple if the first token belongs to the first child. The
first child may
# include extraneous parentheses (which don't create new nodes), so
account for those too.
@@ -302,16 +311,18 @@
return self.handle_bare_tuple(node, first_token, last_token)
return (first_token, last_token)
- def visit_tuple(self, node, first_token, last_token):
- # type: (AstNode, util.Token, util.Token) -> Tuple[util.Token, util.Token]
+ def visit_tuple(
+ self, node: AstNode, first_token: util.Token, last_token: util.Token
+ ) -> Tuple[util.Token, util.Token]:
assert isinstance(node, ast.Tuple) or isinstance(node,
AstroidBaseContainer)
if not node.elts:
# An empty tuple is just "()", and we need no further info.
return (first_token, last_token)
return self.handle_tuple_nonempty(node, first_token, last_token)
- def _gobble_parens(self, first_token, last_token, include_all=False):
- # type: (util.Token, util.Token, bool) -> Tuple[util.Token, util.Token]
+ def _gobble_parens(
+ self, first_token: util.Token, last_token: util.Token, include_all: bool =
False
+ ) -> Tuple[util.Token, util.Token]:
# Expands a range of tokens to include one or all pairs of surrounding
parentheses, and
# returns (first, last) tokens that include these parens.
while first_token.index > 0:
@@ -324,16 +335,17 @@
break
return (first_token, last_token)
- def visit_str(self, node, first_token, last_token):
- # type: (AstNode, util.Token, util.Token) -> Tuple[util.Token, util.Token]
+ def visit_str(
+ self, node: AstNode, first_token: util.Token, last_token: util.Token
+ ) -> Tuple[util.Token, util.Token]:
return self.handle_str(first_token, last_token)
- def visit_joinedstr(self,
- node, # type: AstNode
- first_token, # type: util.Token
- last_token, # type: util.Token
- ):
- # type: (...) -> Tuple[util.Token, util.Token]
+ def visit_joinedstr(
+ self,
+ node: AstNode,
+ first_token: util.Token,
+ last_token: util.Token,
+ ) -> Tuple[util.Token, util.Token]:
if sys.version_info < (3, 12):
# Older versions don't tokenize the contents of f-strings
return self.handle_str(first_token, last_token)
@@ -363,12 +375,14 @@
break
return (first_token, last_token)
- def visit_bytes(self, node, first_token, last_token):
- # type: (AstNode, util.Token, util.Token) -> Tuple[util.Token, util.Token]
+ def visit_bytes(
+ self, node: AstNode, first_token: util.Token, last_token: util.Token
+ ) -> Tuple[util.Token, util.Token]:
return self.handle_str(first_token, last_token)
- def handle_str(self, first_token, last_token):
- # type: (util.Token, util.Token) -> Tuple[util.Token, util.Token]
+ def handle_str(
+ self, first_token: util.Token, last_token: util.Token
+ ) -> Tuple[util.Token, util.Token]:
# Multiple adjacent STRING tokens form a single string.
last = self._code.next_token(last_token)
while util.match_token(last, token.STRING):
@@ -376,13 +390,13 @@
last = self._code.next_token(last_token)
return (first_token, last_token)
- def handle_num(self,
- node, # type: AstNode
- value, # type: Union[complex, int, numbers.Number]
- first_token, # type: util.Token
- last_token, # type: util.Token
- ):
- # type: (...) -> Tuple[util.Token, util.Token]
+ def handle_num(
+ self,
+ node: AstNode,
+ value: Union[complex, int, numbers.Number],
+ first_token: util.Token,
+ last_token: util.Token,
+ ) -> Tuple[util.Token, util.Token]:
# A constant like '-1' gets turned into two tokens; this will skip the '-'.
while util.match_token(last_token, token.OP):
last_token = self._code.next_token(last_token)
@@ -398,14 +412,16 @@
first_token = self._code.prev_token(first_token)
return (first_token, last_token)
- def visit_num(self, node, first_token, last_token):
- # type: (AstNode, util.Token, util.Token) -> Tuple[util.Token, util.Token]
+ def visit_num(
+ self, node: AstNode, first_token: util.Token, last_token: util.Token
+ ) -> Tuple[util.Token, util.Token]:
n = node.n # type: ignore[union-attr] # ast.Num has been removed in
python 3.14
assert isinstance(n, (complex, int, numbers.Number))
return self.handle_num(node, n, first_token, last_token)
- def visit_const(self, node, first_token, last_token):
- # type: (AstNode, util.Token, util.Token) -> Tuple[util.Token, util.Token]
+ def visit_const(
+ self, node: AstNode, first_token: util.Token, last_token: util.Token
+ ) -> Tuple[util.Token, util.Token]:
assert isinstance(node, ast.Constant) or isinstance(node, nc.Const)
if isinstance(node.value, numbers.Number):
return self.handle_num(node, node.value, first_token, last_token)
@@ -415,8 +431,9 @@
visit_constant = visit_const
- def visit_keyword(self, node, first_token, last_token):
- # type: (AstNode, util.Token, util.Token) -> Tuple[util.Token, util.Token]
+ def visit_keyword(
+ self, node: AstNode, first_token: util.Token, last_token: util.Token
+ ) -> Tuple[util.Token, util.Token]:
# Until python 3.9 (https://bugs.python.org/issue40141),
# ast.keyword nodes didn't have line info. Astroid has lineno None.
assert isinstance(node, ast.keyword) or isinstance(node, nc.Keyword)
@@ -427,8 +444,9 @@
first_token = name
return (first_token, last_token)
- def visit_starred(self, node, first_token, last_token):
- # type: (AstNode, util.Token, util.Token) -> Tuple[util.Token, util.Token]
+ def visit_starred(
+ self, node: AstNode, first_token: util.Token, last_token: util.Token
+ ) -> Tuple[util.Token, util.Token]:
# Astroid has 'Starred' nodes (for "foo(*bar)" type args), but they need
to be adjusted.
if not util.match_token(first_token, token.OP, '*'):
star = self._code.prev_token(first_token)
@@ -436,8 +454,9 @@
first_token = star
return (first_token, last_token)
- def visit_assignname(self, node, first_token, last_token):
- # type: (AstNode, util.Token, util.Token) -> Tuple[util.Token, util.Token]
+ def visit_assignname(
+ self, node: AstNode, first_token: util.Token, last_token: util.Token
+ ) -> Tuple[util.Token, util.Token]:
# Astroid may turn 'except' clause into AssignName, but we need to adjust
it.
if util.match_token(first_token, token.NAME, 'except'):
colon = self._code.find_token(last_token, token.OP, ':')
@@ -448,8 +467,9 @@
# but Python < 3.7 doesn't put the col_offset there
# AsyncFunctionDef is slightly different because it might have
# decorators before that, which visit_functiondef handles
- def handle_async(self, node, first_token, last_token):
- # type: (AstNode, util.Token, util.Token) -> Tuple[util.Token, util.Token]
+ def handle_async(
+ self, node: AstNode, first_token: util.Token, last_token: util.Token
+ ) -> Tuple[util.Token, util.Token]:
if not first_token.string == 'async':
first_token = self._code.prev_token(first_token)
return (first_token, last_token)
@@ -457,12 +477,12 @@
visit_asyncfor = handle_async
visit_asyncwith = handle_async
- def visit_asyncfunctiondef(self,
- node, # type: AstNode
- first_token, # type: util.Token
- last_token, # type: util.Token
- ):
- # type: (...) -> Tuple[util.Token, util.Token]
+ def visit_asyncfunctiondef(
+ self,
+ node: AstNode,
+ first_token: util.Token,
+ last_token: util.Token,
+ ) -> Tuple[util.Token, util.Token]:
if util.match_token(first_token, token.NAME, 'def'):
# Include the 'async' token
first_token = self._code.prev_token(first_token)
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/asttokens-3.0.1/asttokens/util.py
new/asttokens-3.0.2/asttokens/util.py
--- old/asttokens-3.0.1/asttokens/util.py 2025-11-15 17:04:32.000000000
+0100
+++ new/asttokens-3.0.2/asttokens/util.py 2026-07-12 05:31:23.000000000
+0200
@@ -15,6 +15,7 @@
import ast
import collections
import io
+import re
import sys
import token
import tokenize
@@ -38,27 +39,18 @@
if TYPE_CHECKING: # pragma: no cover
from .astroid_compat import NodeNG
+else:
+ NodeNG = Any
- # Type class used to expand out the definition of AST to include fields
added by this library
- # It's not actually used for anything other than type checking though!
- class EnhancedAST(AST):
- # Additional attributes set by mark_tokens
- first_token = None # type: Token
- last_token = None # type: Token
- lineno = 0 # type: int
- end_lineno = 0 # type : int
- end_col_offset = 0 # type : int
- AstNode = Union[EnhancedAST, NodeNG]
+TokenInfo = tokenize.TokenInfo
+_lone_cr_re = re.compile(r'\r(?!\n)')
- TokenInfo = tokenize.TokenInfo
-
-def token_repr(tok_type, string):
- # type: (int, Optional[str]) -> str
+def token_repr(tok_type: int, string: Optional[str]) -> str:
"""Returns a human-friendly representation of a token with the given type
and string."""
# repr() prefixes unicode with 'u' on Python2 but not Python3; strip it out
for consistency.
- return '%s:%s' % (token.tok_name[tok_type], repr(string).lstrip('u'))
+ return f"{token.tok_name[tok_type]}:{repr(string).lstrip('u')}"
class Token(collections.namedtuple('Token', 'type string start end line index
startpos endpos')):
@@ -75,50 +67,64 @@
- [6] .startpos Starting character offset into the input text.
- [7] .endpos Ending character offset into the input text.
"""
- def __str__(self):
- # type: () -> str
+ def __str__(self) -> str:
return token_repr(self.type, self.string)
-def match_token(token, tok_type, tok_str=None):
- # type: (Token, int, Optional[str]) -> bool
+# Type class used to expand out the definition of AST to include fields added
by this library
+# It's not actually used for anything other than type checking though!
+class EnhancedAST(AST):
+ # Additional attributes set by mark_tokens
+ first_token: Token = None # type: ignore
+ last_token: Token = None # type: ignore
+ lineno: int = 0
+ end_lineno: int = 0
+ end_col_offset: int = 0
+
+
+AstNode = Union[EnhancedAST, NodeNG]
+
+
+def match_token(token: Token, tok_type: int, tok_str: Optional[str] = None) ->
bool:
"""Returns true if token is of the given type and, if a string is given, has
that string."""
return token.type == tok_type and (tok_str is None or token.string ==
tok_str)
-def expect_token(token, tok_type, tok_str=None):
- # type: (Token, int, Optional[str]) -> None
+def expect_token(token: Token, tok_type: int, tok_str: Optional[str] = None)
-> None:
"""
Verifies that the given token is of the expected type. If tok_str is given,
the token string
is verified too. If the token doesn't match, raises an informative
ValueError.
"""
if not match_token(token, tok_type, tok_str):
- raise ValueError("Expected token %s, got %s on line %s col %s" % (
- token_repr(tok_type, tok_str), str(token),
- token.start[0], token.start[1] + 1))
+ raise ValueError(
+ f"Expected token {token_repr(tok_type, tok_str)}, "
+ f"got {str(token)} on line {token.start[0]} col {token.start[1] + 1}"
+ )
-def is_non_coding_token(token_type):
- # type: (int) -> bool
+def is_non_coding_token(token_type: int) -> bool:
"""
These are considered non-coding tokens, as they don't affect the syntax tree.
"""
return token_type in (token.NL, token.COMMENT, token.ENCODING)
-def generate_tokens(text):
- # type: (str) -> Iterator[TokenInfo]
+def generate_tokens(text: str) -> Iterator[TokenInfo]:
"""
Generates standard library tokens for the given code.
"""
+ # Before Python 3.12, tokenize treats an entire line containing a lone
carriage return as a
+ # non-coding NL token, even though ast.parse treats the carriage return as a
line boundary.
+ # Replacing lone carriage returns is length-preserving, so token offsets
still map to the
+ # original source. Keep CRLF intact because changing its length would shift
later offsets.
+ text = _lone_cr_re.sub('\n', text)
# tokenize.generate_tokens is technically an undocumented API for Python3,
but allows us to use the same API as for
# Python2. See https://stackoverflow.com/a/4952291/328565.
# FIXME: Remove cast once https://github.com/python/typeshed/issues/7003
gets fixed
return tokenize.generate_tokens(cast(Callable[[], str],
io.StringIO(text).readline))
-def iter_children_func(node):
- # type: (AST) -> Callable
+def iter_children_func(node: AST) -> Callable:
"""
Returns a function which yields all direct children of a AST node,
skipping children that are singleton nodes.
@@ -127,8 +133,7 @@
return iter_children_astroid if hasattr(node, 'get_children') else
iter_children_ast
-def iter_children_astroid(node, include_joined_str=False):
- # type: (NodeNG, bool) -> Union[Iterator, List]
+def iter_children_astroid(node: NodeNG, include_joined_str: bool = False) ->
Union[Iterator, list]:
if not include_joined_str and is_joined_str(node):
return []
@@ -139,8 +144,7 @@
issubclass(c, (ast.expr_context, ast.boolop, ast.operator,
ast.unaryop, ast.cmpop))}
-def iter_children_ast(node, include_joined_str=False):
- # type: (AST, bool) -> Iterator[Union[AST, expr]]
+def iter_children_ast(node: AST, include_joined_str: bool = False) ->
Iterator[Union[AST, expr]]:
if not include_joined_str and is_joined_str(node):
return
@@ -169,31 +173,26 @@
# These feel hacky compared to isinstance() but allow us to work with both ast
and astroid nodes
# in the same way, and without even importing astroid.
-def is_expr(node):
- # type: (AstNode) -> bool
+def is_expr(node: AstNode) -> bool:
"""Returns whether node is an expression node."""
return node.__class__.__name__ in expr_class_names
-def is_stmt(node):
- # type: (AstNode) -> bool
+def is_stmt(node: AstNode) -> bool:
"""Returns whether node is a statement node."""
return node.__class__.__name__ in stmt_class_names
-def is_module(node):
- # type: (AstNode) -> bool
+def is_module(node: AstNode) -> bool:
"""Returns whether node is a module node."""
return node.__class__.__name__ == 'Module'
-def is_joined_str(node):
- # type: (AstNode) -> bool
+def is_joined_str(node: AstNode) -> bool:
"""Returns whether node is a JoinedStr node, used to represent f-strings."""
# At the moment, nodes below JoinedStr have wrong line/col info, and trying
to process them only
# leads to errors.
return node.__class__.__name__ == 'JoinedStr'
-def is_expr_stmt(node):
- # type: (AstNode) -> bool
+def is_expr_stmt(node: AstNode) -> bool:
"""Returns whether node is an `Expr` node, which is a statement that is an
expression."""
return node.__class__.__name__ == 'Expr'
@@ -207,26 +206,22 @@
# astroid is not available
pass
-def is_constant(node):
- # type: (AstNode) -> bool
+def is_constant(node: AstNode) -> bool:
"""Returns whether node is a Constant node."""
return isinstance(node, CONSTANT_CLASSES)
-def is_ellipsis(node):
- # type: (AstNode) -> bool
+def is_ellipsis(node: AstNode) -> bool:
"""Returns whether node is an Ellipsis node."""
return is_constant(node) and node.value is Ellipsis # type: ignore
-def is_starred(node):
- # type: (AstNode) -> bool
+def is_starred(node: AstNode) -> bool:
"""Returns whether node is a starred expression node."""
return node.__class__.__name__ == 'Starred'
-def is_slice(node):
- # type: (AstNode) -> bool
+def is_slice(node: AstNode) -> bool:
"""Returns whether node represents a slice, e.g. `1:2` in `x[1:2]`"""
# Before 3.9, a tuple containing a slice is an ExtSlice,
# but this was removed in https://bugs.python.org/issue34822
@@ -239,8 +234,7 @@
)
-def is_empty_astroid_slice(node):
- # type: (AstNode) -> bool
+def is_empty_astroid_slice(node: AstNode) -> bool:
return (
node.__class__.__name__ == "Slice"
and not isinstance(node, ast.AST)
@@ -251,8 +245,11 @@
# Sentinel value used by visit_tree().
_PREVISIT = object()
-def visit_tree(node, previsit, postvisit):
- # type: (Module, Callable[[AstNode, Optional[Token]], Tuple[Optional[Token],
Optional[Token]]], Optional[Callable[[AstNode, Optional[Token],
Optional[Token]], None]]) -> None
+def visit_tree(
+ node: Module,
+ previsit: Callable[[AstNode, Optional[Token]], Tuple[Optional[Token],
Optional[Token]]],
+ postvisit: Optional[Callable[[AstNode, Optional[Token], Optional[Token]],
None]]
+) -> None:
"""
Scans the tree under the node depth-first using an explicit stack. It avoids
implicit recursion
via the function call stack to avoid hitting 'maximum recursion depth
exceeded' error.
@@ -275,7 +272,7 @@
iter_children = iter_children_func(node)
done = set()
ret = None
- stack = [(node, None, _PREVISIT)] # type: List[Tuple[AstNode,
Optional[Token], Union[Optional[Token], object]]]
+ stack: List[Tuple[AstNode, Optional[Token], object]] = [(node, None,
_PREVISIT)]
while stack:
current, par_value, value = stack.pop()
if value is _PREVISIT:
@@ -294,8 +291,7 @@
return ret
-def walk(node, include_joined_str=False):
- # type: (AST, bool) -> Iterator[Union[Module, AstNode]]
+def walk(node: AST, include_joined_str: bool = False) ->
Iterator[Union[Module, AstNode]]:
"""
Recursively yield all descendant nodes in the tree starting at ``node``
(including ``node``
itself), using depth-first pre-order traversal (yieling parents before their
children).
@@ -323,8 +319,7 @@
stack.insert(ins, c)
-def replace(text, replacements):
- # type: (str, List[Tuple[int, int, str]]) -> str
+def replace(text: str, replacements: List[Tuple[int, int, str]]) -> str:
"""
Replaces multiple slices of text with new values. This is a convenience
method for making code
modifications of ranges e.g. as identified by
``ASTTokens.get_text_range(node)``. Replacements is
@@ -347,12 +342,10 @@
"""
Helper to get `visit_{node_type}` methods given a node's class and cache the
results.
"""
- def __init__(self):
- # type: () -> None
- self._cache = {} # type: Dict[Union[ABCMeta, type], Callable[[AstNode,
Token, Token], Tuple[Token, Token]]]
+ def __init__(self) -> None:
+ self._cache: Dict[Union[ABCMeta, type], Callable[[AstNode, Token, Token],
Tuple[Token, Token]]] = {}
- def get(self, obj, cls):
- # type: (Any, Union[ABCMeta, type]) -> Callable
+ def get(self, obj: Any, cls: Union[ABCMeta, type]) -> Callable:
"""
Using the lowercase name of the class as node_type, returns
`obj.visit_{node_type}`,
or `obj.visit_default` if the type-specific method is not found.
@@ -365,8 +358,7 @@
return method
-def patched_generate_tokens(original_tokens):
- # type: (Iterable[TokenInfo]) -> Iterator[TokenInfo]
+def patched_generate_tokens(original_tokens: Iterable[TokenInfo]) ->
Iterator[TokenInfo]:
"""
Fixes tokens yielded by `tokenize.generate_tokens` to handle more
non-ASCII characters in identifiers.
Workaround for https://github.com/python/cpython/issues/68382.
@@ -374,7 +366,7 @@
because it assumes that error tokens are not actually errors.
Combines groups of consecutive NAME, NUMBER, and/or ERRORTOKEN tokens into
a single NAME token.
"""
- group = [] # type: List[tokenize.TokenInfo]
+ group: List[tokenize.TokenInfo] = []
for tok in original_tokens:
if (
tok.type in (tokenize.NAME, tokenize.ERRORTOKEN, tokenize.NUMBER)
@@ -390,8 +382,7 @@
for combined_token in combine_tokens(group):
yield combined_token
-def combine_tokens(group):
- # type: (List[tokenize.TokenInfo]) -> List[tokenize.TokenInfo]
+def combine_tokens(group: List[tokenize.TokenInfo]) ->
List[tokenize.TokenInfo]:
if not any(tok.type == tokenize.ERRORTOKEN for tok in group) or
len({tok.line for tok in group}) != 1:
return group
return [
@@ -405,8 +396,7 @@
]
-def last_stmt(node):
- # type: (AstNode) -> AstNode
+def last_stmt(node: AstNode) -> AstNode:
"""
If the given AST node contains multiple statements, return the last one.
Otherwise, just return the node.
@@ -429,8 +419,7 @@
@lru_cache(maxsize=None)
-def fstring_positions_work():
- # type: () -> bool
+def fstring_positions_work() -> bool:
"""
The positions attached to nodes inside f-string FormattedValues have some
bugs
that were fixed in Python 3.9.7 in
https://github.com/python/cpython/pull/27729.
@@ -460,8 +449,7 @@
)
return positions_are_unique and correct_source_segments
-def annotate_fstring_nodes(tree):
- # type: (ast.AST) -> None
+def annotate_fstring_nodes(tree: ast.AST) -> None:
"""
Add a special attribute `_broken_positions` to nodes inside f-strings
if the lineno/col_offset cannot be trusted.
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/asttokens-3.0.1/asttokens/version.py
new/asttokens-3.0.2/asttokens/version.py
--- old/asttokens-3.0.1/asttokens/version.py 2025-11-15 17:13:25.000000000
+0100
+++ new/asttokens-3.0.2/asttokens/version.py 2026-07-12 05:31:31.000000000
+0200
@@ -1 +1 @@
-__version__ = "3.0.1"
+__version__ = "3.0.2"
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/asttokens-3.0.1/asttokens.egg-info/PKG-INFO
new/asttokens-3.0.2/asttokens.egg-info/PKG-INFO
--- old/asttokens-3.0.1/asttokens.egg-info/PKG-INFO 2025-11-15
17:13:26.000000000 +0100
+++ new/asttokens-3.0.2/asttokens.egg-info/PKG-INFO 2026-07-12
05:31:31.000000000 +0200
@@ -1,6 +1,6 @@
Metadata-Version: 2.4
Name: asttokens
-Version: 3.0.1
+Version: 3.0.2
Summary: Annotate AST trees with source code positions
Home-page: https://github.com/gristlabs/asttokens
Author: Dmitry Sagalovskiy, Grist Labs
@@ -119,3 +119,23 @@
4. Run tests across all supported interpreters with the ``tox`` command. You
will need to have the interpreters installed separately. We recommend ``pyenv``
for that. Use ``tox -p auto`` to run the tests in parallel.
5. By default certain tests which take a very long time to run are skipped,
but they are run in CI.
These are marked using the ``pytest`` marker ``slow`` and can be run on
their own with ``pytest -m slow`` or as part of the full suite with ``pytest -m
''``.
+
+Documentation
+-------------
+
+Documentation at https://asttokens.readthedocs.io/ is built by Read the Docs
automatically on
+every push, from the sources in ``docs/`` (configured in
``.readthedocs.yaml``). To preview
+changes locally::
+
+ pip install -r docs/requirements.txt
+ sphinx-build -M html docs docs/_build
+
+Release
+-------
+
+The version is derived from the git tag by ``setuptools_scm``; there is no
version to update in
+the source. To release, draft a new release at
https://github.com/gristlabs/asttokens/releases
+with a new tag named ``vX.Y.Z``, and use "Generate release notes" to fill in
the description.
+Publishing the release triggers the ``publish.yml`` workflow, which builds the
package and
+uploads it to PyPI via `trusted publishing
+<https://docs.pypi.org/trusted-publishers/>`_.
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/asttokens-3.0.1/asttokens.egg-info/SOURCES.txt
new/asttokens-3.0.2/asttokens.egg-info/SOURCES.txt
--- old/asttokens-3.0.1/asttokens.egg-info/SOURCES.txt 2025-11-15
17:13:26.000000000 +0100
+++ new/asttokens-3.0.2/asttokens.egg-info/SOURCES.txt 2026-07-12
05:31:31.000000000 +0200
@@ -5,13 +5,12 @@
.readthedocs.yaml
LICENSE
MANIFEST.in
-Makefile
README.rst
pyproject.toml
setup.cfg
-setup.py
tox.ini
.github/workflows/build-and-test.yml
+.github/workflows/publish.yml
asttokens/__init__.py
asttokens/astroid_compat.py
asttokens/asttokens.py
@@ -24,8 +23,9 @@
asttokens.egg-info/SOURCES.txt
asttokens.egg-info/dependency_links.txt
asttokens.egg-info/requires.txt
+asttokens.egg-info/scm_file_list.json
+asttokens.egg-info/scm_version.json
asttokens.egg-info/top_level.txt
-docs/Makefile
docs/api-index.rst
docs/conf.py
docs/index.rst
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore'
old/asttokens-3.0.1/asttokens.egg-info/scm_file_list.json
new/asttokens-3.0.2/asttokens.egg-info/scm_file_list.json
--- old/asttokens-3.0.1/asttokens.egg-info/scm_file_list.json 1970-01-01
01:00:00.000000000 +0100
+++ new/asttokens-3.0.2/asttokens.egg-info/scm_file_list.json 2026-07-12
05:31:31.000000000 +0200
@@ -0,0 +1,70 @@
+{
+ "files": [
+ ".coveragerc",
+ "setup.cfg",
+ "tox.ini",
+ "LICENSE",
+ "pyproject.toml",
+ "README.rst",
+ "MANIFEST.in",
+ ".editorconfig",
+ ".pylintrc",
+ ".gitignore",
+ ".readthedocs.yaml",
+ "docs/api-index.rst",
+ "docs/index.rst",
+ "docs/requirements.txt",
+ "docs/requirements.in",
+ "docs/user-guide.rst",
+ "docs/conf.py",
+ "tests/tools.py",
+ "tests/test_line_numbers.py",
+ "tests/test_astroid.py",
+ "tests/__init__.py",
+ "tests/test_tokenless.py",
+ "tests/context.py",
+ "tests/test_mark_tokens.py",
+ "tests/test_asttokens.py",
+ "tests/test_util.py",
+ "tests/testdata/README.md",
+ "tests/testdata/python3/astroid/suppliermodule_test.py",
+ "tests/testdata/python3/astroid/__init__.py",
+ "tests/testdata/python3/astroid/descriptor_crash.py",
+ "tests/testdata/python3/astroid/joined_strings.py",
+ "tests/testdata/python3/astroid/format.py",
+ "tests/testdata/python3/astroid/email.py",
+ "tests/testdata/python3/astroid/clientmodule_test.py",
+ "tests/testdata/python3/astroid/all.py",
+ "tests/testdata/python3/astroid/nonregr.py",
+ "tests/testdata/python3/astroid/recursion.py",
+ "tests/testdata/python3/astroid/notall.py",
+ "tests/testdata/python3/astroid/absimport.py",
+ "tests/testdata/python3/astroid/noendingnewline.py",
+ "tests/testdata/python3/astroid/module2.py",
+ "tests/testdata/python3/astroid/module.py",
+ "tests/testdata/python2/astroid/suppliermodule_test.py",
+ "tests/testdata/python2/astroid/__init__.py",
+ "tests/testdata/python2/astroid/descriptor_crash.py",
+ "tests/testdata/python2/astroid/joined_strings.py",
+ "tests/testdata/python2/astroid/format.py",
+ "tests/testdata/python2/astroid/email.py",
+ "tests/testdata/python2/astroid/clientmodule_test.py",
+ "tests/testdata/python2/astroid/all.py",
+ "tests/testdata/python2/astroid/nonregr.py",
+ "tests/testdata/python2/astroid/recursion.py",
+ "tests/testdata/python2/astroid/notall.py",
+ "tests/testdata/python2/astroid/absimport.py",
+ "tests/testdata/python2/astroid/noendingnewline.py",
+ "tests/testdata/python2/astroid/module2.py",
+ "tests/testdata/python2/astroid/module.py",
+ "asttokens/asttokens.py",
+ "asttokens/py.typed",
+ "asttokens/__init__.py",
+ "asttokens/astroid_compat.py",
+ "asttokens/line_numbers.py",
+ "asttokens/util.py",
+ "asttokens/mark_tokens.py",
+ ".github/workflows/build-and-test.yml",
+ ".github/workflows/publish.yml"
+ ]
+}
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/asttokens-3.0.1/asttokens.egg-info/scm_version.json
new/asttokens-3.0.2/asttokens.egg-info/scm_version.json
--- old/asttokens-3.0.1/asttokens.egg-info/scm_version.json 1970-01-01
01:00:00.000000000 +0100
+++ new/asttokens-3.0.2/asttokens.egg-info/scm_version.json 2026-07-12
05:31:31.000000000 +0200
@@ -0,0 +1,8 @@
+{
+ "tag": "3.0.2",
+ "distance": 0,
+ "node": "gf97a6e1403d9f0e93940a11b507524550a5c3896",
+ "dirty": false,
+ "branch": "HEAD",
+ "node_date": "2026-07-12"
+}
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/asttokens-3.0.1/docs/Makefile
new/asttokens-3.0.2/docs/Makefile
--- old/asttokens-3.0.1/docs/Makefile 2016-12-10 19:00:48.000000000 +0100
+++ new/asttokens-3.0.2/docs/Makefile 1970-01-01 01:00:00.000000000 +0100
@@ -1,20 +0,0 @@
-# Minimal makefile for Sphinx documentation
-#
-
-# You can set these variables from the command line.
-SPHINXOPTS =
-SPHINXBUILD = sphinx-build
-SPHINXPROJ = asttokens
-SOURCEDIR = .
-BUILDDIR = _build
-
-# Put it first so that "make" without argument is like "make help".
-help:
- @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
-
-.PHONY: help Makefile
-
-# Catch-all target: route all unknown targets to Sphinx using the new
-# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
-%: Makefile
- @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
\ No newline at end of file
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/asttokens-3.0.1/setup.py new/asttokens-3.0.2/setup.py
--- old/asttokens-3.0.1/setup.py 2020-02-22 20:10:30.000000000 +0100
+++ new/asttokens-3.0.2/setup.py 1970-01-01 01:00:00.000000000 +0100
@@ -1,11 +0,0 @@
-"""A setuptools based setup module.
-
-See:
-https://packaging.python.org/en/latest/distributing.html
-https://github.com/pypa/sampleproject
-"""
-
-from setuptools import setup
-
-if __name__ == "__main__":
- setup()
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/asttokens-3.0.1/tests/test_line_numbers.py
new/asttokens-3.0.2/tests/test_line_numbers.py
--- old/asttokens-3.0.1/tests/test_line_numbers.py 2024-11-30
02:47:20.000000000 +0100
+++ new/asttokens-3.0.2/tests/test_line_numbers.py 2026-07-12
05:31:23.000000000 +0200
@@ -36,6 +36,15 @@
self.assertEqual(ln.offset_to_line(100), (8, 0))
self.assertEqual(ln.offset_to_line(-100), (1, 0))
+ def test_carriage_returns(self):
+ # A lone "\r" (old-Mac line ending) and "\r\n" (Windows) must both count
as line boundaries,
+ # matching how Python's tokenizer and ast module number source lines. See
issue #105.
+ ln = asttokens.LineNumbers("a\rb\r\nc\n\rd\r\re\r\r\nf")
+ line_offsets = [0, 2, 5, 7, 8, 10, 11, 13, 15]
+ for line, offset in enumerate(line_offsets, 1):
+ self.assertEqual(ln.line_to_offset(line, 0), offset)
+ self.assertEqual(ln.offset_to_line(offset), (line, 0))
+
def test_unicode(self):
ln = asttokens.LineNumbers("фыва\nячсм")
self.assertEqual(ln.line_to_offset(1, 0), 0)
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/asttokens-3.0.1/tests/test_mark_tokens.py
new/asttokens-3.0.2/tests/test_mark_tokens.py
--- old/asttokens-3.0.1/tests/test_mark_tokens.py 2025-11-15
17:04:32.000000000 +0100
+++ new/asttokens-3.0.2/tests/test_mark_tokens.py 2026-07-12
05:31:23.000000000 +0200
@@ -303,6 +303,21 @@
"%s:%s" % ("AssignName" if self.is_astroid_test else "Name",
source.split("=")[0]),
})
+ def test_mixed_line_endings(self):
+ # Test of https://github.com/gristlabs/asttokens/issues/105
+ source = "a=1\rb.c\r\nd=2\n\re.f\r\rg=3\rh.i"
+ atok = self.create_asttokens(source)
+ self.assertEqual(atok.tokens[-1].type, token.ENDMARKER)
+ self.assertEqual(
+ [atok.get_text(stmt) for stmt in atok.tree.body],
+ ["a=1", "b.c", "d=2", "e.f", "g=3", "h.i"],
+ )
+ for node in util.walk(atok.tree):
+ first, last = getattr(node, 'first_token', None), getattr(node,
'last_token', None)
+ if first is not None:
+ self.assertNotEqual(last.type, token.ENDMARKER, node)
+ self.assertLessEqual(last.index, len(atok.tokens) - 1)
+
def test_bytes_smoke(self):
const = 'Const' if self.is_astroid_test else (
'Constant'
@@ -931,4 +946,3 @@
m = self.create_mark_checker(source)
assert 'DictComp' in str(m.view_nodes_at(0, 0))
-
\ No newline at end of file
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/asttokens-3.0.1/tests/test_util.py
new/asttokens-3.0.2/tests/test_util.py
--- old/asttokens-3.0.1/tests/test_util.py 2025-11-03 06:34:32.000000000
+0100
+++ new/asttokens-3.0.2/tests/test_util.py 2026-07-12 05:31:23.000000000
+0200
@@ -114,6 +114,26 @@
asttokens.util.expect_token(tok, token.OP)
+def test_generate_tokens_normalizes_lone_carriage_returns():
+ tokens = list(asttokens.util.generate_tokens("\ry.y\n"))
+ coding_tokens = [tok for tok in tokens if tok.type in (token.NAME, token.OP)]
+
+ assert [(tok.type, tok.string, tok.start, tok.end) for tok in coding_tokens]
== [
+ (token.NAME, "y", (2, 0), (2, 1)),
+ (token.OP, ".", (2, 1), (2, 2)),
+ (token.NAME, "y", (2, 2), (2, 3)),
+ ]
+
+ crlf_tokens = list(asttokens.util.generate_tokens("\r\ny.y\n"))
+ assert crlf_tokens[0].string == "\r\n"
+
+ triple_tokens = list(asttokens.util.generate_tokens('value = """a\rb"""\n'))
+ string_token = next(tok for tok in triple_tokens if tok.type == token.STRING)
+ assert string_token.string == '"""a\nb"""'
+ assert string_token.start == (1, 8)
+ assert string_token.end == (2, 4)
+
+
def test_combine_tokens():
from tokenize import TokenInfo, generate_tokens, ERRORTOKEN, OP, NUMBER,
NAME
from asttokens.util import combine_tokens, patched_generate_tokens
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/asttokens-3.0.1/tox.ini new/asttokens-3.0.2/tox.ini
--- old/asttokens-3.0.1/tox.ini 2025-11-03 06:34:32.000000000 +0100
+++ new/asttokens-3.0.2/tox.ini 2026-07-12 05:31:23.000000000 +0200
@@ -4,7 +4,7 @@
# and then run "tox" from this directory.
[tox]
-envlist = py{38,39,310,311,312,313,py3}
+envlist = py{39,310,311,312,313,314,py3}
[testenv]
commands = pytest {posargs}
