[
https://issues.apache.org/jira/browse/ARROW-15820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17500519#comment-17500519
]
Vibhatha Lakmal Abeykoon commented on ARROW-15820:
--------------------------------------------------
[~westonpace] adding some thoughts to this, I already have reported an issue to
enhance the existing exec-plan examples with the new additions, TableSource and
TableSink. I can work on that PR just after this most probably([C++] Update
ExecPlan example code and documentation with new options). But I am holding
that there are some issues that we may need to solve before that (i.e: [C++]
Simplify creating a scan node, [C++] The write node options shouldn't require a
schema).
> [C++][Doc] Add table_source to streaming_execution.rst & clarify parameter
> name
> -------------------------------------------------------------------------------
>
> Key: ARROW-15820
> URL: https://issues.apache.org/jira/browse/ARROW-15820
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++
> Reporter: Weston Pace
> Assignee: Vibhatha Lakmal Abeykoon
> Priority: Major
>
> Currently the table_source node does not appear in our documentation.
> Also, in {{TableSourceNodeOptions}} we have:
> {noformat}
> // Size of batches to emit from this node
> // If the table is larger the node will emit multiple batches from the
> // the table to be processed in parallel.
> int64_t batch_size;
> {noformat}
> However, when looking into a performance issue today, I realized this
> description is incomplete. In reality we should probably call this parameter
> {{max_batch_size}}.
> Furthermore, we should make it clear that a table with smaller batches will
> emit smaller batches directly (this is a good thing in my case) and will not
> concatenate small batches together into a larger batch.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
