juliuszsompolski commented on code in PR #57402:
URL: https://github.com/apache/spark/pull/57402#discussion_r3622989191
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryBaseTable.scala:
##########
@@ -215,6 +215,41 @@ abstract class InMemoryBaseTable(
val commits: ListBuffer[Commit] = ListBuffer[Commit]()
+ protected def commandOutput: Array[InternalRow] = {
+ commits.lastOption.flatMap(_.writeSummary).map {
+ case summary: MergeSummary =>
+ val affectedRows = sumIfKnown(
+ summary.numTargetRowsUpdated(),
+ summary.numTargetRowsDeleted(),
+ summary.numTargetRowsInserted())
+ Array(
+ metricRow("num_affected_rows", affectedRows),
+ metricRow("num_updated_rows", summary.numTargetRowsUpdated()),
+ metricRow("num_deleted_rows", summary.numTargetRowsDeleted()),
+ metricRow("num_inserted_rows", summary.numTargetRowsInserted()))
Review Comment:
nit: I noticed that you have num_copied_rows in update and delete, but not
here. Is that mirroring what e.g. Delta is doing?
##########
sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoTableSuiteBase.scala:
##########
@@ -2512,7 +2524,6 @@ abstract class MergeIntoTableSuiteBase extends
RowLevelOperationSuiteBase
|""".stripMargin
}
-
Review Comment:
nit: unrelated stray but whatever
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala:
##########
@@ -1010,4 +1019,3 @@ private[v2] case class DataWritingSparkTaskResult(
* Sink progress information collected after commit.
*/
private[sql] case class StreamWriterCommitProgress(numOutputRows: Long)
-
Review Comment:
nit: unrelated stray but whatever
##########
sql/catalyst/src/main/java/org/apache/spark/sql/connector/write/RowLevelOperation.java:
##########
@@ -54,6 +55,18 @@ default String description() {
*/
Command command();
+ /**
+ * Returns the schema of the rows produced by this operation.
+ * <p>
+ * Spark calls this method during planning. The returned schema must match
the rows returned by
+ * {@link Write#output()} after the write commits successfully.
+ *
+ * @since 4.3.0
+ */
+ default StructType outputSchema() {
+ return new StructType();
Review Comment:
just checking: was calling a collect() on a command df returning an empty
schema next to the 0 rows before?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
