benbellick commented on PR #1399: URL: https://github.com/apache/arrow-go/pull/1399#issuecomment-6926707877
Hi @zeroshade! I agree that removing `substrait-go` types from Arrow's public API would be ideal. But that seems difficult for functions like `ExecuteScalarExpression`, which lets libraries like Iceberg build Substrait expressions and pass them to Arrow for execution. First, I think we should reduce the exposure of `substrait-go`'s API where we can. For example, Arrow probably doesn't need to expose function/type anchor bookkeeping. I put together [a draft exploring that cleanup](https://github.com/benbellick/arrow-go/pull/1). Beyond that, I see a few options: 1. Keep existing execution functions and add new ones for each major version. For example, keep `ExecuteScalarExpression` accepting v8 expressions and add `ExecuteScalarExpressionV9` for v9. We'd maintain some duplicate code, but users could upgrade when they're ready. Iceberg already builds its own expressions, so we wouldn't need to duplicate Arrow's builders for this use case. 2. Move the Arrow/Substrait code into its own independently versioned module. We'd still have breaking changes, but they wouldn't force a major release of all of Arrow. 3. Remove Substrait types from the public API entirely. I'm not sure how to do this cleanly without moving execution elsewhere or giving Arrow its own expression API, which goes against the idea of sharing Substrait's model. We'd want to make any necessary breaking changes together in one major release, with a plan to avoid repeating this in the future. After that, adding execution support for newer Substrait versions wouldn't need to break existing callers. What do you think? Any preference between those three? I think 1 is the simplest, though I'm not sure how much code we'd need to duplicate for each new major version. It would also mean keeping dependencies on multiple `substrait-go` major versions rather than replacing the old one with the new one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
