anew commented on code in PR #58843:
URL: https://github.com/apache/spark/pull/58843#discussion_r4044692956
##########
python/pyspark/pipelines/api.py:
##########
@@ -587,6 +590,19 @@ def create_auto_cdc_flow(
this list will be in the output table.
:param stored_as_scd_type: The SCD type for the target table. 1 (or "1")
and 2 (or "2") are \
supported. When not specified, the server default applies.
+ :param ignore_null_updates: When True, null values in an incoming update
are ignored and the \
+ existing target value is preserved, for every column. Defaults to
False, in which case \
+ nulls overwrite the target. Mutually exclusive with
ignore_null_updates_column_list and \
+ ignore_null_updates_except_column_list.
+ :param ignore_null_updates_column_list: The subset of columns for which
null values in an \
+ incoming update are ignored. This should be a list of column
identifiers without \
+ qualifiers, expressed as either Python strings or PySpark Columns.
Mutually exclusive \
+ with ignore_null_updates and ignore_null_updates_except_column_list.
+ :param ignore_null_updates_except_column_list: The subset of columns for
which null values \
+ in an incoming update overwrite the target; nulls are ignored for all
other columns. \
+ This should be a list of column identifiers without qualifiers,
expressed as either \
+ Python strings or PySpark Columns. Mutually exclusive with
ignore_null_updates and \
+ ignore_null_updates_column_list.
Review Comment:
documented in the ignore_null_updates param: leaving it False (the default)
alongside either column list is a no-op, not a conflict.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
