Author: Carl Friedrich Bolz-Tereick <[email protected]> Branch: extradoc Changeset: r5948:4ee007fa782d Date: 2019-02-14 10:45 +0100 http://bitbucket.org/pypy/extradoc/changeset/4ee007fa782d/
Log: preserve full notes (will extract tasks too) diff --git a/sprintinfo/ddorf2019/planning.txt b/sprintinfo/ddorf2019/planning.txt new file mode 100644 --- /dev/null +++ b/sprintinfo/ddorf2019/planning.txt @@ -0,0 +1,262 @@ +# PyPy sprint + +https://piratenpad.de/p/IkjitZdh +https://hackmd.io/1eT6t4_lReigSw0-4pEyQw + +## People + +Tim +Stefan +Andy +Ronan +Manuel +Alexander +Julian +Carl Friedrich +John +Maciej +Armin +Łukasz +Matti +Anto +Mercurial devs (Wednesday) +some external people + +## Tasks - Mon + +- intro talk (CF, John, Julian, Andy, Stefan) + + + +## Tasks - Tues +- Start the ARM64 backend (Armin, fijal) STARTED+DISCUSSIONS +- Merging regalloc branch (CF, fijal, Armin) +- Set up benchmarking machine (fijal, Matti) +- Triage 3.6 failures and progress on them (Łukasz, Ronan) https://bitbucket.org/pypy/pypy/wiki/Status%20of%20PyPy3%20support%20for%203.6 + - re and csv failures are done + - removed duplicate code + - next: fspath + +- zlib missing methods pypy2/3 (Julian, Anto) RZLIB DONE + - new failing tests: http://buildbot.pypy.org/summary?category=linux64&branch=%3Ctrunk%3E&recentrev=95849:c5c3ad13d149 + - app-level test: when run on top of compiled PyPy, can't call function defined on interp-level + - lib-python test: .flush() sets .stream to nullptr, .copy() calls deflateCopy() with nullptr, it segfaults in RZlibError.fromstream() + - subtle difference: on CPython, z_stream is part of structure and can't become nullptr + - possible fixes: either handle .stream == nullptr in pypy module, or make RZlibError.fromstream() robust agains nullptr + - TESTS fixed? py3 problems +- Fix RevDB (Manuel, Armin around) DONE + +- investigate struct.unpack weirdness (Antocuni, Julian) + +- Look into Windows failures on 3.5 (Manuel, Andy, Ronan around) - NANs are not consistent DONE + +- Windows installer (Andy, Matti) WORKS, needs adding to package.py + + + +## Tasks +- Release 7.0 (Anto) reports version 7.1 :( +- utf-8 progress/merging (Matti looking, Ronan will help) + NEEDS REVIEW. Benchmarks are [here](https://gist.github.com/mattip/05951f44e41224e36f00b62ebb4e2444) and [here](https://gist.github.com/mattip/d25536cd88ffe0361feca7760745a524). `logging.warn`, which is slow on py3.5 has gotten even worse SATURDAY + +- still one failing test in zlib branch untranslated (Ronan) + +- cpyext performance (Tim, Ronan around) Carl will ask Tim what is going on and file an issue + +- math-improvements branch (Alex to start) (REVIEW DONE, needs to be merged into default after the release, more bugs, benchmark on Py3?) (CF to merge into default/py3.5) + +- memoryview/ctypes bug https://bitbucket.org/pypy/pypy/issues/2930 (Matti, Armin around) + +- pypeg (lpeg reimplementation in RPython) (Stefan, Tim around, +?) + +- Tracker gardening + +- CFFI OS X bug (Maciej, Armin around) + +- finish fspath stuff (Łukasz, CF around) + +- look at test_compile.py (Łukasz, CF around) + +- asynio on Windows (Andy) + +- ARM32 build server (David, Matti, Maciek, Armin) + +- Answer Stefan Beyer email on pypy-dev + + +## Tasks Saturday + +* let's look at spectral_norm after regalloc merge +* utf-8 (Armin, CF, Matti, Maciek around) +* progress on ARM64 (Maciek, David, Rodolph) + + +## Discussions Done + +- Think about Windows support - conclusion is to keep muddling along: + + - There is money from SFC - who will do the work? + - Maybe the Microsoft people?????? + - Andy has a first-attempt at a MSI packager + - Try to have green buildbots and not much more? + +- Funding + - SFC - $69,000, $300 in OpenCollective + - Karen (SFC) Seem to be ready for us to part ways. + - We should spend the money. There are no marked funds. + - PLC needs to approve expenses. + - Python3.6/3.7/3.8: Ronan, Carl, Lucasz $30,000, incl. travel, salary + - Next sprint: summer? + - Cpyext: Antonio, Ronan $30,000, incl. travel, salary + - Buildbots: machines and website update $10,000 + - pypy.org $2,000 +- Catching up with 3.6/3.7/3.8 - stop with py3.5 + +- Intro talk (DONE) +- Web Assembly (DONE) +- Collaboration possibilities GraalPython around numpy stack TODAY +- Scientific Stack +- VMProf AFTER LUNCH +- Unstuck Benchmarking - do it!! (decision made) +- Packaging (wheels and conda) situation + +## Discussions +- Infrastructure, testing: Travis, Azure Pipelines (1800 minutes per month) FRI MORNING +- State and Future of PyPy +- apptest-file branch merge (Ronan) +- focused sprint: when, where + +### Webassembly and PyPy + +### pyodide + +Python as an ecosystem is bad suited? + +PyPy should wait for GC support and DOM support. + + +## Cython and GraalPython + +* Problem for GraalPython and PyPy (and numba) is that numpy is a black box +* Other problem: handling expressions as a unit to not need to compute intermediate values +* Numpy discussions around PyIR +* Use (subset of) Cython or Cython-like to make [PyIR](https://docs.google.com/document/d/1jGksgI96LdYQODa9Fca7EttFEGQfNODphVmbCX0DD1k/edit#) real + * Cython is used by numeric people already, they are used to it + * as much Cython as possible + * compilable to something sensible +* Cython is a tiny bit messy, it's a one-pass compiler +* can we make Cython better designed? +* potentially ask Oracle for funding + +## PyPy and the Numeric Stack + +* Important to support well, it's what many people use Python for +* Problems: + * they don't use PyPy yet! + * numpy code tends to not be fast on PyPy + * the big libraries are hard to install + * we don't support wheels + * we don't support conda +* Performance problems: + * cpyext speedups + * long term plans around cython or similar +* maybe building wheels is going to be easier with manylinux2010? +* let's try to build pypy on the right manylinux2010 +* conda doesn't care about us +* they want us to do work? +* but we would need changes in conda and deep understanding on their side + * Open issue with design proposal: https://github.com/conda/conda/issues/6950 +* contrete tasks: + * try to build manylinux2010 pypy + * talk to conda people? + + +## ``VMProf`` + +* Problems with vmprof: + * frontend: graphs are hard to understand + * kills buildbot host + * we need to maintain it + * huge RAM usage + * backend: + * rpython functions not profiled correctly + * what's the scope of vmprof? + * no "customers"/motivation + +* Scope: + * ??? +* Frontend: + * Can we make our life easier by using somebody else's frontend? + * Chrome devtools + * PyCharm + * https://github.com/KDAB/hotspot + +* Backend: + * profiling in production + * make it cheap enough + * make it easy to turn on and off + * make it possible to deal with data in-process + * the vmprof pure python modules is badly implemented + * boring parts: rpython functions + + * steps: + * improve the vmprof module to not suck + * make ``_vmprof`` should always provide reliable data + * make local deployment very simple + +* we should be able to connect source code and traces +* unique identifiers for loops from the JIT + * what happens if the loop is formed before `_vmprof` is on + * just have it always on ;-) + * how to connect pieces of traces to source code? Offets, pieces of loops, code in memory do not always map back to source file + * exposes bugs that probably already exist but we don't care + * blackhole, generated code, tracing, ... + * debug_merge_point is not necesssarily always correct +* when you have correct input you see correct output, but when you do not have correct input do not show garbage + +##### Summary + +* Can we make an improved JitViewer with statistical profiling info? Will it be helpful? + * what about tracing? blackhole? Function repitition? Fix it in `_vmprof` ! + +* Possible architecture: + * _vmprof feeds data to a socket + * separate program reads the data and distills it into a known format + * that is presented in a known frontend + +* Better packaging and API + * callback hook function what to do with the info: socket, file, + * on/off per block, + * on/off generally which would not necessarily add overhead but would allow later on/off on a lower level. + * allow self-reporting of global/per-thread/per-green thread state of the app + +## Discussion: State and Future of PyPy +* Let's try to have team-internal "office hours" Monday 10.00-13.00 to deal with the reduced availability of people +* How can we constructively interact with CPython-dev? + * maybe a PEP? + * what are concrete things that would be good to have in there? + * something about C-API? + * having cpython implementation details marked as such in the regression tests + * have people from other implementations be invited to lang summit +* when PyPy 3.6 is out, we should get other projects to run their tests on PyPy +* How do we grow the community? + * Advertise sprints more widely + * steal some cpython devs? + * move away from bitbucket.org? + * beginner mentoring program + * blog posts, videos + * "PyPy internals for CPython devs" + * twitch streams? + * docs are a mess + * success stories on the web page +* What can we stop doing (brainstorming) + * dropping platforms? + * maintain own fork of pytest + * using less custom infrastructure + +## disjointed benchmarking notes +* ask whether we can use speed.python.org +* run benchmarks weekly +* + +## dummy _______________________________________________ pypy-commit mailing list [email protected] https://mail.python.org/mailman/listinfo/pypy-commit
