On Sun, Apr 23, 2023, 3:43 PM turnbull <turnb...@sk.tsukuba.ac.jp> wrote:
> This isn't a PEP yet: it's a set of requirements. A PEP eventually > needs to > say how to implement the requirements, and even at this "proto-PEP" > stage, > it needs to be plausible that it's implementable. It's on you to > explain > how your very ambitious requirements can be satisfied in Python. > Nobody's > going to ask you for an implementation, but references to related tech > like > HPy and ctypes combined with discussion of how they do or don't meet > your > requirements would be helpful. > > 2023-04-23 21:13 に Evan Greenup via Python-ideas さんは書きました: > > > However there is a lot of limitation in those library [like PyO3]. > > For good reason. Python does not share data structures with those other > languages. CPython's native data structures are a subset of those > implemented > in C -- obviously, because CPython is implemented in C. But other > languages > will implement "mutable extensible memory-safe sequence" (ie, Python's > list) > in different ways (and famously C doesn't provide that!) On the other > hand, > data structures in other languages may have no built-in equivalent in > Python. > The low-level cytpes stdlib module provides the flexibility you want, > but it's > implementation-dependent on the foreign side, and must be. > > > * The project structure is rigid. > > This is a complaint about specific third-party libraries that provide > high- > level wrapping of a fundamentally low-level facility. I would *expect* > the > project structure to be rigid. > > I think you might find it easier to present the proposal convincingly if > you > "build up" from ctypes, instead of "building down" from PyO3. > > > * It is really ridiculous when you want to stick some item on the > > wall. You need to totally redesign this item and manufacture a new > > item to fit the glue you are going to use. > > This is not true of ctypes, which is designed as the thinnest possible > wrapper around other languages (specifically C, but to the extent that > most C > implementations provide facilities for calling FORTRAN and other such > languages, it should be possible to extend ctypes to those languages in > that > way). > > > As a glue language, Python should be designed to glue other native > > programming > > language as a feature of Python programming language itself > > As far as I can see this is not feasible, and vastly overemphasizes > Python's role > as a glue language. Python is a programming language first, and the > business of > the Python programming language is to be Python. Interfacing to other > languages > is going to be more or less painful depending on how closely the > internals are > related, and in general it will be hardware-dependent. > > > * The interface is universal across all variant and version of Python > > implementation > > It took Microsoft 15 versions and a couple of decades to manage this > with just > its own runtime library. Remember, not only do Python internals change > from > version to version, but so do those of other languages. C++ is infamous > for > incompatibility, in fact. It's hard to imagine that the Python side of > the > interface can be completely independent of the other language, when the > whole > point is that the other language has specific features that *Python does > not*. > > As Jelle mentions, a standardized ABI for Python is in process, the > current > iteration being the HPy project. However AIUI the goal is a consistent > ABI > across Python versions, not making construction of FFIs easier. All it > should > do I believe is remove ABI compatibility across Python versions from the > set of > problems an FFI needs to deal with. That's useful, but doesn't remove > any of > the complexity caused by different representations in the target > language. > > > It is a lively data structure with in-memory representation, they are > > unified > > no matter what Python variant is used and what low level native > > language is > > used. > > So you're suggesting an intermediate data representation, likely > requiring two > translations (Python to intermediate and intermediate to target > language) each > time data is to be transferred from Python to a target language and > back. If > HPy succeeds then that ABI can be frozen as both the Python ABI and the > inter- > mediate representation, of course. > Arrow does zero-copy with nested ~Structs + Schema. Have you already considered Apache Arrow? https://arrow.apache.org/ https://github.com/apache/arrow https://arrow.apache.org/ : """ ## What is Arrow? ### Format Apache Arrow defines a language-independent columnar memory format for flat and hierarchical data, organized for efficient analytic operations on modern hardware like CPUs and GPUs. The Arrow memory format also supports zero-copy reads for lightning-fast data access without serialization overhead. Learn more about the design or read the specification. ### Libraries Arrow's libraries implement the format and provide building blocks for a range of use cases, including high performance analytics. Many popular projects use Arrow to ship columnar data efficiently or as the basis for analytic engines. Libraries are available for C, C++, C#, Go, Java, JavaScript, Julia, MATLAB, Python, R, Ruby, and Rust. See how to install and """ However, consider the C++ standard template library. In Python, > everything is > an object, with a consistent handle. Lists and tuples are uniform > sequences of > handles, dicts are uniform handle-to-handle mappings. That's not so in > the C++ > template library. The whole point is to provide individual routines > optimized > to each variant of a data structure based on the template's type > variables. It's > one-many, not one-one, from the point of view of your proposed FFI ABI. > It seems > to me that more than Python's internal representation, which changes > fairly slowly > and is pretty well-documented, the target ABI is more variable, and if > the target > is as low-level as C there is not going to be one because equivalents of > Python > structures will be defined per-project rather than for all C libraries. > > > * This mechanism is transparent to users, there is modules in standard > > library > > to support it. [...] This mechanism provide user with maximum > > flexibility. > > That sounds like ctypes to me. > > > * almost zero-cost abstraction. > > What does that mean? > > > It just make some basic data representation conversion and > > invoke the method in dynamic library. > > Still sounds like ctypes to me. > > So I come back to the theme: what do you want that ctypes doesn't > provide? > https://docs.python.org/3/library/ctypes.html > > _______________________________________________ > Python-ideas mailing list -- python-ideas@python.org > To unsubscribe send an email to python-ideas-le...@python.org > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > Message archived at > https://mail.python.org/archives/list/python-ideas@python.org/message/IEHRGPWLUTB4OYB7OBIAF6OCPUO6S624/ > Code of Conduct: http://python.org/psf/codeofconduct/ >
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/NUUYPJNALGKRI3RWP53TBW63TGPBMBB5/ Code of Conduct: http://python.org/psf/codeofconduct/
