[
https://issues.apache.org/jira/browse/ARROW-16259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17525600#comment-17525600
]
Weston Pace commented on ARROW-16259:
-------------------------------------
Probably the easiest thing to do is when the failure occurs the file writer
should take whatever rows it has reserved from the backpressure and release
them. The only tricky part is that the failure is going to pop up from an I/O
thread and so we need to make sure we're properly guarding access to the
backpressure to avoid race conditions.
> [C++] If an I/O error is encountered while the dataset writer is applying
> backpressure there could be deadlock in rare (artifical?) situations
> ----------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: ARROW-16259
> URL: https://issues.apache.org/jira/browse/ARROW-16259
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++
> Reporter: Weston Pace
> Priority: Minor
>
> When I/O errors occur the dataset writer handles them when the next batch
> would be written (or at teardown time if it was the last batch).
> If the dataset writer has applied backpressure then this doesn't work because
> there is no "next batch".
> This was encountered in the python backpressure test which is a bit
> artificial. To recreate this change the GatingFs::open_output_stream method
> to return None after its unlocked.
> This is fairly minor because it doesn't really happen in practice. It takes
> many concurrent writes to get into backpressure and unless all of the fail
> there will be some that succeed and eventually get the writer out of
> backpressure enough to discover the failure.
> Still, it would be nice to fix this to clean things up.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
