Jungtaek Lim created SPARK-52065:
------------------------------------
Summary: Produce another plan tree with output columns (name, data
type, nullability) in plan change logging
Key: SPARK-52065
URL: https://issues.apache.org/jira/browse/SPARK-52065
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.1.0
Reporter: Jungtaek Lim
We recently got into very tricky issue (nullability change broke stateful
operator) which required custom debug logging on plan change logging. This is
because of lack of visibility for the output columns, especially their
nullability, in our tree string of the plan.
To address the lack of visibility, we propose to add another tree string which
focuses to produce output columns with data type and nullability.
Ideally, we shouldn't have two different tree strings and just consolidate, but
in many cases, current tree string is long enough so that we had to restrict
the number of fields to show, hence we think it's better to have a separate
tree plan for it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
