timsaucer opened a new pull request, #1678: URL: https://github.com/apache/datafusion-python/pull/1678
# Which issue does this PR close? Part 2 of 3 in the split of #1672 (part 1: FFI query planner support; part 3: atomic `SessionContext.with_extensions`). Stacked on part 1. # Rationale for this change Supporting a foreign planner surfaced a codec problem: a query can involve three independent native libraries (datafusion-python, a provider library, and a planner library), and each library needs its extension codecs active on the session at the same time. Previously, installing a logical or physical extension codec replaced the prior codec, so the second library's install silently discarded the first — plans then failed later with a confusing decode error. # What changes are included in this PR? - `with_logical_extension_codec` / `with_physical_extension_codec` now prepend to a codec chain instead of replacing the prior codec. The most recently installed codec is consulted first, falling through codec by codec to DataFusion's default codec. A codec signals "not mine" by returning an error. - Encoding runs each codec against a scratch buffer so failed attempts leave no partial bytes, and treats Ok-with-no-bytes (encode by name) as "no opinion" so later codecs still get a chance. - When every codec in the chain fails, the errors are aggregated so the owning codec's diagnostic is not masked by the default codec's generic error. - Fixed a latent bug where installing a codec silently reset `python_udf_inlining` back to enabled. - `docs/source/contributor-guide/ffi.md` gains sections on composable codecs: family-prefix discipline, and that registration order between libraries no longer matters. # Are there any user-facing changes? Behavior change: installing an extension codec now composes with previously installed codecs instead of replacing them. Code that relied on replacement semantics (installing a codec to remove a prior one) is affected; all other usage keeps working and no longer loses earlier codecs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
