JerAguilon opened a new issue, #38706: URL: https://github.com/apache/arrow/issues/38706
### Describe the enhancement requested ### Describe the enhancement requested In the past I have made implementations of custom Acero nodes in arrow C++. I vastly prefer wiring the actual DAG in python, and working with pyarrow-style tables/record batches on the output. The API for constructing an Acero `Declaration` looks like: ``` class pyarrow.acero.Declaration(factory_name, ExecNodeOptions options, inputs=None) ``` The issue is that `ExecNodeOptions` is a polymorphic type, and in order to use, say, `ScanNodeOptions` or `HashJoinNodeOptions`, one would need to add bindings in cython. This means that as external users, if we create a custom node, there's no ergonomic way to construct it from pyarrow without forking arrow and adding bindings, which may be untenable if we want to keep with mainline acero. One way to tackle this is by adding wrappers/unwrappers to `arrow::acero::ExecNodeOptions`. And since many options use arrow expressions, it'd be nice to have wrappers/unwrappers for `arrow:compute::Expression`. One challenge is that `_compute` and `_acero` are submodules of pyarrow, which means that the usual approaches of adding wrappers will result in circular dependency issues. We can solve this by creating libraries for submodules. Side note: Another related solution to interopping C++/python is to make a generic options interface, which I described in https://github.com/apache/arrow/issues/38654, but I think this approach would be more powerful/flexible. ### Component(s) C++, Python ### Component(s) C++, Python -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
