HyukjinKwon commented on code in PR #59932:
URL: https://github.com/apache/spark/pull/59932#discussion_r4418775275
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarWrite.scala:
##########
@@ -82,18 +82,24 @@ class ColumnarBatchWrite(createWriter: () =>
ColumnarWriter, schema: StructType)
extends BatchWrite {
private var writer: ColumnarWriter = _
+ // Whether the writer committed the write, after which it is not aborted,
even if Spark fails
+ // afterwards: a native data source relies on it.
+ private var committed = false
Review Comment:
Done in 14a1d3a: the Scaladoc of `ColumnarBatchWrite` says "as any later
failure before the commit succeeds", and `ColumnarWriter` says "Spark does not
abort a write whose commit succeeded".
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarWrite.scala:
##########
@@ -115,6 +121,8 @@ class ColumnarStreamingWrite(
extends StreamingWrite {
private var writer: ColumnarStreamWriter = _
+ // The epochs that the writer committed, which are not aborted, as for a
batch write.
+ private val committedEpochs = scala.collection.mutable.Set.empty[Long]
Review Comment:
Done in 14a1d3a: `ColumnarStreamingWrite` keeps a `private var committed`,
as `ColumnarBatchWrite` does.
##########
sql/core/src/test/resources/native-datasource/test_native_datasource.cc:
##########
@@ -727,6 +776,30 @@ void writeFile(const std::string& path, const std::string&
content) {
std::fclose(file);
}
+// Records a call that the order of the calls of the header rules out. Unlike
the files of the
+// options, a failure to record it is ignored: it may be called outside of
`guarded`, as in the
+// close functions, which cannot fail.
Review Comment:
Done in 14a1d3a: both comments are fixed. `orderViolation` names the checks
at the start of the functions on a handle, and `ORDER_VIOLATIONS_PATH` says
that the suites check it at the end of each test, with `checkingOrderOfCalls`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
