chris-twiner commented on PR #48477: URL: https://github.com/apache/spark/pull/48477#issuecomment-2577308912
> Would not the `TransformingEncoder` be enough of a customization point to implement the custom encoders provided by at least `frameless` (I am not familiar enough with the others)? > > I guess it not ideal as it would require a rewrite of that part of the library. But at least from my experiments creating custom encoders by first creating `AgnosticEncoders` is much easier than creating the `ExpressionEncoders` directly. And based on the comments by @hvanhovell it seems that is a better approach for downstream libraries. hihi, per the response above, aside from the extra black box indirection that will not allow optimisation (e.g. constant folding etc.) the transformed type itself must be fully represented by AgnosticEncoders already, that's a significant amount of added spaghetti instead of simply using the ExpressionEncoder already present. Longer term, a full rewrite to agnosticencoder derivation (https://github.com/typelevel/frameless/issues/701) might be possible (TransformingEncoder _could_ be a good implementation for the Injection/Union approach although the block box indirection would still exist it might be tolerable for that). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
