Gengliang Wang created SPARK-61284:
--------------------------------------
Summary: Close the stream writer of a native data source after it
commits or aborts its micro-batch
Key: SPARK-61284
URL: https://issues.apache.org/jira/browse/SPARK-61284
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Gengliang Wang
A batch write of a native data source closes its writer handle right after a
successful commit, and after an abort (NativeDataSourceWriter). A streaming
write does not: NativeDataSourceStreamWriter.commit and abort leave the handle
open, and NativeStreamWriters.register closes it only when the query creates
the stream writer of its next micro-batch, after create_writer of that
micro-batch, or, for the last micro-batch of a query, once the handle is no
longer referenced. A library that releases an exclusive resource in
close_writer, such as a lock on its destination, then fails or blocks at the
second micro-batch.
Close the handle of a stream writer after a successful commit and after an
abort, as for batch writes, so that NativeStreamWriters only closes the stream
writers that never finish, and update the lifecycle that spark_datasource.h
documents.
Found in https://github.com/apache/spark/pull/59520#discussion_r4515939460.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
