dwsmith1983 opened a new pull request, #7686:
URL: https://github.com/apache/datafusion-comet/pull/7686

   ## Which issue does this PR close?
   
   Refs #7573. This is the second of three PRs:
   
   1. #7600: the native `InterleaveBits` kernel.
   2. This PR: a core hook that lets a contrib register an expression serde.
   3. #7597: the Delta serde and expression suite. It uses both and closes 
#7573.
   
   1 and 2 can merge in either order; #7597 goes last.
   
   ## Rationale for this change
   
   Comet has no way to convert an expression class that only a contrib knows 
about. Delta's `InterleaveBits` lives in Delta, so core can't register a serde 
for it without naming a Delta class. The scan side already solves this with the 
`CometScanContrib` ServiceLoader interface. This adds the same kind of hook for 
expressions, so the Delta contrib can register its serde and core stays free of 
Delta classes. Matching the class name in core, the way 
`icebergFunctions.scala` handles Iceberg, was the other option raised in #7573.
   
   ## What changes are included in this PR?
   
   - `CometExpressionContrib`, a trait a contrib implements to return serdes 
keyed by expression class. Implementations are found through `ContribServices`, 
so a default build with no service file sees an empty registry.
   - Core serdes always win. A contrib entry for a class core already handles 
is ignored with a warning, and a class two contribs both claim is dropped from 
both, since ServiceLoader order is unspecified. A contrib whose serde map 
throws, including a `LinkageError` from a jar built against other Comet 
internals, is skipped with a warning.
   - `QueryPlanSerde.serdeFor` looks up core first, then contribs. Expression 
conversion and the `ScalaUDF` guard check use it instead of `exprSerdeMap` 
directly.
   - A shared `ContribServiceFiles` test helper, now used by 
`CometScanContribSuite` as well.
   
   ## How are these changes tested?
   
   `CometExpressionContribSuite` covers the merge rules (an unclaimed class is 
kept, core wins, a class two contribs claim is dropped, a contrib that throws 
or hits a `LinkageError` is skipped), discovery from a service file through a 
child ClassLoader, the `serdeFor` lookup order, and the empty registry on a 
default build. The suite is added to the Linux and macOS PR workflows. 
`CometScanContribSuite` still passes on the shared helper. The end-to-end 
check, a real contrib serde converting `InterleaveBits` natively, is in #7597.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to