Vibhatha Lakmal Abeykoon created ARROW-16036:
------------------------------------------------
Summary: Substrait SinkNode Modification for usability
Key: ARROW-16036
URL: https://issues.apache.org/jira/browse/ARROW-16036
Project: Apache Arrow
Issue Type: Improvement
Reporter: Vibhatha Lakmal Abeykoon
Currently, the Substrait implementation uses a `consuming_sink` node as the
end-point to capture the results of an executed query. To further enhance the
ability to use the wide variety of `SinkNode`s in the streaming execution
engine, it is better to evaluate other options to provide usability for various
use cases.
Some of the use cases include consuming a Substrait query plan in R and Python
APIs. And this extends beyond the Arrow APIs where third-party tools willing to
use Substrait can be benefitted from these. At the moment there is active work
on providing R/Python APIs for Substrait consumption
(https://issues.apache.org/jira/browse/ARROW-15779).
A potential modification to provide usability is to take in a
`SinkNodeFactory`-like interface instead of just using the `ConsumingSink`
(current implementation). Such a modification would provide flexibility in
developing a wide array of applications.
This issue is open for discussion.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
