Zerolzj opened a new issue, #9656:
URL: https://github.com/apache/paimon/issues/9656
### Search before asking
- [x] I searched in the issues and found nothing similar.
### Motivation
PyPaimon currently selects readers and writers with built-in `if/elif`
branches. New formats such
as Lance and Mosaic have therefore been integrated directly into the
PyPaimon package. This works
well for formats that the project wants to support and release as built-ins,
but there is no stable
extension boundary for a deployment-specific format or for an embedding
runtime that supplies its
own implementation.
Java Paimon already discovers `FileFormatFactory` implementations. I also
noticed that Java PR
#8292 proposed another runtime-side `FileFormatProvider` and was closed
because an execution engine
could provide that SPI in its own integration. PyPaimon is itself the
SDK/runtime that performs the
format dispatch, so an embedding application cannot add a format without
patching that dispatch.
Would the community consider a small PyPaimon extension contract, or is
built-in-only format
support the intended boundary?
### Solution
A possible contract would have these properties:
1. Table metadata stores only an immutable, language-neutral format
identifier, never a Python
module or class name.
2. A provider declares read and write capabilities independently and
receives public context
objects rather than PyPaimon writer implementation details.
3. Registration is explicit and deterministic. Python package entry points
may be an optional
discovery mechanism, but duplicate identifiers fail rather than depend on
import order.
4. A missing provider fails during planning with a clear diagnostic. It
never falls back to a
different physical format.
5. Packaging and distribution to remote workers remain the responsibility of
the embedding engine;
the table does not persist package locations.
6. The extension API has a small version/capability handshake so a provider
developed against one
PyPaimon release does not silently run with incompatible context objects.
The minimum API could be an explicit registry such as
`register_file_format(identifier, provider)`. Entry-point discovery can be
considered separately
after the packaging work in #8049 is settled.
Questions:
- Is third-party format discovery in scope for PyPaimon?
- If so, would maintainers prefer an explicit registry, Python entry points,
or an embedding-runtime
hook?
- Should the first version expose read-only providers before defining writer
and abort semantics?
- Would this public extension contract require a PIP before a prototype PR?
### Anything else?
Relevant code and discussions:
- PyPaimon reader dispatch:
https://github.com/apache/paimon/blob/master/paimon-python/pypaimon/read/split_read.py
- PyPaimon writer dispatch:
https://github.com/apache/paimon/blob/master/paimon-python/pypaimon/write/writer/data_writer.py
- Lance format discussion: https://github.com/apache/paimon/issues/6739
- PyPaimon packaging discussion: https://github.com/apache/paimon/issues/8049
- Java runtime-side provider discussion:
https://github.com/apache/paimon/pull/8292
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
I can contribute a minimal read-only prototype with an out-of-tree example
provider and tests for
duplicate registration, missing providers, capability checks, and
distributed-worker loading after
the preferred boundary is clear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
