[
https://issues.apache.org/jira/browse/ARROW-13590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17397794#comment-17397794
]
Weston Pace commented on ARROW-13590:
-------------------------------------
I discovered today that the FileSystemDataset::Write method has never applied
back pressure correctly. In Arrow 3.0.0+ the entire dataset will be loaded
into RAM.
> [C++] Ensure dataset writing applies back pressure
> --------------------------------------------------
>
> Key: ARROW-13590
> URL: https://issues.apache.org/jira/browse/ARROW-13590
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++
> Reporter: Weston Pace
> Assignee: Weston Pace
> Priority: Major
> Labels: query-engine
> Fix For: 6.0.0
>
>
> Dataset writing via exec plan (ARROW-13542) does not apply back pressure
> currently and will take up far more RAM than it should when writing a large
> dataset. The node should be applying back pressure. However, the preferred
> back pressure method (via scheduling) will need to wait for ARROW-13576.
> Once those two are finished this can be studied in more detail. Also, the
> vm.dirty_ratio might be experimented with. In theory we should be applying
> our own back pressure and have no need of dirty pages. In practice, it may
> be more work than we want to tackle right now and we just let it do its thing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
