aokolnychyi commented on code in PR #55518:
URL: https://github.com/apache/spark/pull/55518#discussion_r4428867391
##########
sql/catalyst/src/main/java/org/apache/spark/sql/connector/write/DataWriter.java:
##########
@@ -82,9 +87,67 @@ default void write(T metadata, T record) throws IOException {
write(record);
}
+ /**
+ * Writes one updated or copied record with metadata in the column update
layout.
+ * <p>
+ * When {@link LogicalWriteInfo#columnUpdateSchema()} is present for a
row-level operation that
+ * does not mix in {@link SupportsDelta}, Spark passes updated and copied
records to this method
+ * instead of {@link #write(Object, Object)} if the operation returns a
non-empty
+ * {@link RowLevelOperation#requiredMetadataAttributes()}, and to
+ * {@link #writeColumnUpdate(Object)} otherwise. The record follows
+ * {@link LogicalWriteInfo#columnUpdateSchema()} and the metadata follows
+ * {@link LogicalWriteInfo#metadataSchema()}. Operations that mix in {@link
SupportsDelta} receive
+ * such rows through {@link DeltaWriter#update} and {@link
DeltaWriter#reinsert} instead.
+ * <p>
+ * By default, delegates to {@link #writeColumnUpdate(Object)} and drops the
metadata.
+ * <p>
+ * If this method fails (by throwing an exception), {@link #abort()} will be
called and this
+ * data writer is considered to have been failed.
+ *
+ * @throws IOException if failure happens during disk/network IO like
writing files.
+ * @throws SparkUnsupportedOperationException if neither this method nor
+ * {@link #writeColumnUpdate(Object)} is overridden.
+ *
+ * @since 4.4.0
+ */
+ default void writeColumnUpdate(T metadata, T record) throws IOException {
+ writeColumnUpdate(record);
+ }
+
+ /**
+ * Writes one updated or copied record without metadata in the column update
layout.
+ * <p>
+ * When {@link LogicalWriteInfo#columnUpdateSchema()} is present for a
row-level operation that
+ * does not mix in {@link SupportsDelta}, Spark passes updated and copied
records to this method
+ * instead of {@link #write(Object)} if the operation returns no
+ * {@link RowLevelOperation#requiredMetadataAttributes()}. The record follows
+ * {@link LogicalWriteInfo#columnUpdateSchema()}.
+ * <p>
+ * A writer for such an operation must override this method, unless the
operation returns a
+ * non-empty {@link RowLevelOperation#requiredMetadataAttributes()} and the
writer overrides
+ * {@link #writeColumnUpdate(Object, Object)}.
+ * <p>
+ * If this method fails (by throwing an exception), {@link #abort()} will be
called and this
+ * data writer is considered to have been failed.
+ *
+ * @throws IOException if failure happens during disk/network IO like
writing files.
+ * @throws SparkUnsupportedOperationException if this method is not
overridden.
+ *
+ * @since 4.4.0
+ */
+ default void writeColumnUpdate(T record) throws IOException {
Review Comment:
This also makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
