HyukjinKwon commented on code in PR #59932:
URL: https://github.com/apache/spark/pull/59932#discussion_r4469606192
##########
sql/core/src/main/resources/org/apache/spark/sql/datasource/spark_datasource.h:
##########
@@ -281,9 +306,21 @@ typedef struct SparkDsHost {
const char* spark_version;
/*
* Logs a message, a NUL-terminated UTF-8 string, at one of the
SPARK_DS_LOG_* levels: in the log
- * of the driver or of the executor, with the path of the library.
+ * of the driver or of the executor, with the path of the library. Another
level is treated as
+ * SPARK_DS_LOG_INFO.
*/
void (*log)(int32_t level, const char* message);
+ /*
+ * Returns whether the logger of Spark that log writes to is enabled for one
of the
+ * SPARK_DS_LOG_* levels, another level being treated as SPARK_DS_LOG_INFO,
so that the library
+ * only formats the messages that Spark may write, and gets those that Spark
writes at the levels
Review Comment:
Reworded in d4de133: "... so that the library only formats the messages that
Spark may write, and can log at the levels that it would not log by default,
such as debug, when Spark does."
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSource.scala:
##########
@@ -901,7 +964,11 @@ class NativeDataWriter(
// The task, which the library may use until the data writer is closed.
private val task = NativeTask.create(Some(partitionId), Some(taskId))
private val handle: Long = try {
- call(name, "write to")(library.createDataWriter(task, writerState,
epochId))
+ NativeArrow.exportSchema(schema, timeZoneId) { schemaAddress =>
Review Comment:
Added in #59931 (9681e3e), merged here: "create the data writers with the
time zone of their writer" creates a writer under `Asia/Tokyo` and one of its
data writers under `UTC`, and `create_data_writer` of the test library fails
unless it gets the schema that the writer got, with its zone.
##########
sql/core/src/test/resources/native-datasource/test_native_datasource.cc:
##########
@@ -735,6 +826,30 @@ void writeFile(const std::string& path, const std::string&
content) {
std::fclose(file);
}
+// Records a call that the order of the calls of the header rules out. Unlike
the files of the
+// options, a failure to record it is ignored: it is called outside of
`guarded`, by the checks at
Review Comment:
Fixed in d4de133: `abort_data_writer` checks a successful commit before
`guarded`, as `abort` does, and still records the violation and returns success
without removing the file, so every `orderViolation` call runs outside of
`guarded`, as its comment says.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
