Hi everyone, i finished the first milestone i set for myself - which was
rypthon is python3 syntax and it runs/works. now its time to reshuffle
and land those bits so its can be build on. its visible under
https://github.com/RonnyPfannschmidt/pypy/pull/1 - but expect churn to
come up as i prepare eventual actual prs/a pr stack. a key fun bit was
creating a new assert rewrite hook that is rflow compatible plus plays
into my future plans on having annotated/debuggable assertion as part of
python itself as next steps i want to resuffle the pr a bit to be on
more merge friendly comits that should each carry over a enhancement Im
also preparing updates to the ci docker images, but cot.assert may have
more churn than desirable for a while
i'd appreciate feedback on the order of things and reach some consensus
on how to merge things going forward
-- Ronny
------------------------------
report by opus:
Written by Claude Opus 5.5 via Claude Code for the pypy-dev list; I
prompted it, it did the work, I read it.
*Milestone 1: all of |rpython/| is valid Python 3 syntax*
The first goal is narrow on purpose: every file under |rpython/| parses
on Python 3, while running on Python 2 exactly as before. RPython on a
Python 3 host stays thoroughly broken at this stage: nothing about
|str|/bytes, removed builtins or the flow space's bytecode handling
changes. What the milestone buys is that from then on, Python 3 work is
about behaviour, not syntax, and tools that parse the source (linters,
|ast|, a Python 3 test collector) work on the whole tree.
The milestone is reached on the branch:
https://github.com/RonnyPfannschmidt/pypy/pull/1 (against the fork, for
discussion). Every clean-up commit is valid Python 2 with identical
semantics, so the existing Python 2 CI verifies it, and no call site
gains a |sys.version_info| check.
*What it took*
* Files under|rpython/|that do not parse on Python 3: 221 → 0. The
exceptions are three flow space and annotator fixtures that test
Python 2 constructs, which stay Python 2 on purpose.
* Rewritten: 641 print statements, 360 tuple parameters
in|def|and|lambda|, 174|L|suffixes, 118 octal literals, 17 exec
statements, 5|raise E, v|, 4 backticks,|<>|,|has_key|,
one|ur""|literal, and four list comprehensions over an
unparenthesised tuple.
* |print x,|with a trailing comma needed per-site reading. Python 2
defers the separating space until the next item and drops it after
an item ending in a newline, while the flow space always inserts it
and|end=|resets it. Each site was converted by what follows it, so
output stays identical on CPython and in translated code. The one
exception is|richards.trace()|, which runs only with tracing enabled.
* Some prints parse on both versions but mean different things.|print
>> f, x,|is a tuple expression on Python 3: one test logger
compiled there and silently logged nothing. And|print(a, b)|without
the|print_function|import prints a tuple on Python 2. Both are
fixed, and the second is now counted.
* |rpython/tool/py3distance.py|counts all of this per category,
including the constructs that only mean something different. A CI
job fails a PR if any count grows. It compares against the PR's base
commit, so there is no stored baseline to conflict over.
* One mechanism fix came with it:|pairtype.extendabletype|now skips
the names Python 3 adds to the class namespace (|__qualname__|and,
since 3.13,|__firstlineno__|and friends). With that, all ~275|class
__extend__(X)|sites work on a Python 3 host unchanged.
*Beyond milestone 1 (counted, not started)*
753 uses of removed builtins, 202 |iteritems()|-style calls, 53 imports
of renamed stdlib modules, 13 |__metaclass__| attributes. All of these
parse on Python 3 and only fail when run. After them come
|str|/bytes/unicode and the flow space's bytecode handling. As Matti
pointed out, translating with PyPy as the host stays a hard requirement
throughout.
*Test framework (needed along the way)*
The vendored pytest 2.9.2 and py 1.4.29 at the repo root block any newer
pytest, and a Python 3 host needs one. The branch removes them, depends
on pytest 4.6.11 (the last to support Python 2), and fixes what that
exposed in |rpython/|: yield tests (256 tests were being silently
skipped), module-level skips, fixtures called directly, and a duplicate
option. |rpython/| collects completely under 4.6 (14391 items).
|pypy/| is not ported: its app-level tests rely on |--assert=reinterp|,
which pytest 3.0 removed. For now, CI runs |pypy/objspace| on pytest
2.9.2 from PyPI (2178 passed, same as main), plus a non-blocking job on
4.6 to track the gap. |package.py| still uses |py.path|, so CI installs
the py lib for the packaging step.
Asserts in |rpython/| tests are rewritten by cot-assert
(https://pypi.org/project/cot-assert/). It is a small plugin of mine,
started with Claude Opus 5.5, whose rewritten asserts the flow space
accepts, so failing RPython tests get their explanation back. It is
optional and separable from the rest.
CI: the job that translates and packages PyPy is green. The RPython unit
tests are not fully green yet. Apart from what also fails or flakes on
main (|rvmprof test_same_file|, the weakref-dict flake, rvmprof
|TestEnable| on macOS), a handful fail only on the branch: fdlistdir,
jitlog, vmprof and |call_release_gil| on macOS, a leak check on win64,
and one JIT llgraph test. None of them are in the syntax clean-ups. I am
working through them.
*How it would travel to the py3.x branches*
I trial-merged the branch into main, py3.11 and py3.12:
* |rpython/|is nearly identical on all three (2–3 differing files),
since main is merged into py3.x regularly. The syntax clean-ups
merge without conflicts there. The three|rpython/|test files that do
conflict do so only because main has moved on since the branch
point; I will rebase next.
* The conflicts that matter are in the test infrastructure. The py3.x
branches carry their own changes to the vendored|_pytest|/|py|(for
example "fix pytest assertion rewriting on python 2" and "adapt
pytest.py to python3.12"). Removing the vendored copies on main
would collide with those on every merge.
*A proposed path*
Land milestone 1 on main in small, independent PRs, each rebased on
current main, and let them reach py3.x through the regular merges:
1. The distance tool and its CI check. No code changes, and it stops
new Python-2-only syntax from creeping back in.
2. The|pairtype|fix.
3. The syntax clean-ups, one kind per PR (tuple parameters, octal
literals,|L|literals, exec, raise, print, ...). Each is valid Python
2 with the same meaning, so a reviewer checks one rule, not hundreds
of unrelated edits. The trailing-comma prints come as their own PR,
since those were judged one site at a time.
After that, the test framework as its own discussion, since it is the
one piece that affects the py3.x branches' own pytest changes. Then the
runtime categories, in the same shape.
*Questions*
1. Is "valid Python 3 syntax across|rpython/|" a milestone you are
happy to have on main, given that it changes nothing about running
on Python 3 yet?
2. Does "small PRs on main, propagate through the regular merges" work
for you? Or would you rather land in larger batches, or on the py3.x
branches directly?
3. For the main → py3.x merges: are mechanical sweeps across|rpython/|a
burden, or fine given how close|rpython/|is on both sides? Is there
a time you would prefer they land, for example right after a merge?
4. Pytest: one shared pytest for main and py3.x, or a per-branch
vendored copy for now? Would the pytest3.10 branch
(https://github.com/pypy/pypy/commits/pytest3.10/) be the preferred
base?
5. What PR size is comfortable to review?
6. Should anything outside|rpython/|be in scope yet? I have kept away
from|pypy/|, since it diverges between branches.
Each commit carries a Co-Authored-By trailer naming the model and harness.
_______________________________________________
pypy-dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3//lists/pypy-dev.python.org
Member address: [email protected]
