drin commented on issue #40583: URL: https://github.com/apache/arrow/issues/40583#issuecomment-2007920107
I also want to note that pushing any more operators below the `SkyhookSourceNode` would be done blindly without some type of optimizer making that decision. So, the new operator makes sense when inserted above `Read` operators and that would be the sensible first step. @bkietz would you know how Acero serializes custom operators into substrait? Does it use a message that is only known internally? I have been working with custom substrait messages and `SkyhookSourceNode` is a perfect example of where it seems to make sense to use one (and I have something similar I can use [here](https://github.com/drin/mohair-protocol/blob/mainline/proto/mohair/algebra.proto#L20-L43) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
