joellubi commented on PR #38385: URL: https://github.com/apache/arrow/pull/38385#issuecomment-1879892514
@lidavidm @emkornfield Following up on some thoughts I had on the general approach of this PR. This discussion above regarding splitting up DDL and Write operations got me looking at the Substrait spec to see if analogous message types were defined. It turns out [DDL](https://substrait.io/relations/logical_relations/#ddl-data-definition-language-operator) and [Write](https://substrait.io/relations/logical_relations/#write-operator) have in fact been added to the spec. We discussed this a bit during the last Arrow community call, where I said I would sketch out a Substrait-native alternative to handle the bulk ingestion scenario. If it's a good fit, it could help keep the Flight SQL spec slimmer by having clients send a Substrait plan for bulk ingestion to a Flight SQL server instead of the purpose-built message proposed in this PR. Given the limited support currently available for DDL/Write messages and to an extent Substrait in general, we had the idea to put together some "helpers" to make it easier to interpret a bulk ingestion request on the Flight SQL side without having to implement support for arbitrary Substrait plans. I'll be taking a look at what gaps need to be filled in Substrait implementations in order to facilitate this, as well as the feasibility of the "helper" approach. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
