Hi everyone, I’d like to start a vote for the FunctionCatalog design proposal (SPIP).
The proposal is to add a FunctionCatalog interface that can be used to load and list functions for Spark to call. There are interfaces for scalar and aggregate functions. In the discussion we’ve come to consensus and I’ve updated the design doc to match how functions will be called: In addition to produceResult(InternalRow), which is optional, functions can define produceResult methods with arguments that are Spark’s internal data types, like UTF8String. Spark will prefer these methods when calling the UDF using codgen. I’ve also updated the AggregateFunction interface and merged it with the partial aggregate interface because Spark doesn’t support non-partial aggregates. The full SPIP doc is here: https://docs.google.com/document/d/1PLBieHIlxZjmoUB0ERF-VozCRJ0xw2j3qKvUNWpWA2U/edit#heading=h.82w8qxfl2uwl Please vote on the SPIP in the next 72 hours. Once it is approved, I’ll do a final update of the PR and we can merge the API. [ ] +1: Accept the proposal as an official SPIP [ ] +0 [ ] -1: I don’t think this is a good idea because … -- Ryan Blue
