lidavidm commented on issue #34016: URL: https://github.com/apache/arrow/issues/34016#issuecomment-1423312385
Just a quick response, but the Python middleware is the one that incurs FFI overhead on every call, not the C++ one. On Wed, Feb 8, 2023, at 17:09, Paul Nienaber wrote: > > > @lidavidm <https://github.com/lidavidm> I've been working with @ravjotbrar <https://github.com/ravjotbrar> and I can't see that implementing the middleware callback contract across an FFI is a good option here. My thinking here: > > • It's an application state management helper, and managing the state behind an FFI other than performance reasons doesn't make a lot of sense—and crossing the FFI extra times is more likely a performance hit than a benefit. > • It would save duplication of the core implementation code, however the FFI definitions would increase complexity of maintenance. > • The middleware callback interface is already in Python (if there's ever a performance argument it might make sense to allow two types of middleware objects where the Flight implementation detects whether there's a pure-C++ one and dynamically wires in the callbacks that way), so using C++ code for this small component would make extension or modification by Python app authors much more difficult. > • Even without the goal of modifying the included middleware implementation, Python app authors could easily want to simply look at the code to understand its behaviour in more detail, and reusing the C++ implementation would again present a significant barrier to users here. > • Arrow is a multi-language set of bindings/SDK. Normally all of the bindings would be written in native languages, which would mean code duplication, but as it stands some interpreted languages are wrapped around the C++ implementation for performance reasons alone. But expecting to reduce code/implementation duplication across multiple supported languages as a goal in and of itself is not usually something to expect in such a multi-language library implementation. > Perhaps I am missing something further about why it would in fact be for the best to use the C++ impl anyways? > > > — > Reply to this email directly, view it on GitHub <https://github.com/apache/arrow/issues/34016#issuecomment-1423303637>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACQB33FSP7D34VMWT2QFG3WWQKRBANCNFSM6AAAAAAUPWGM2I>. > You are receiving this because you were mentioned.Message ID: ***@***.***> > -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
