[
https://issues.apache.org/jira/browse/SPARK-52065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-52065:
-----------------------------------
Labels: pull-request-available (was: )
> Produce another plan tree with output columns (name, data type, nullability)
> in plan change logging
> ---------------------------------------------------------------------------------------------------
>
> Key: SPARK-52065
> URL: https://issues.apache.org/jira/browse/SPARK-52065
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.1.0
> Reporter: Jungtaek Lim
> Priority: Major
> Labels: pull-request-available
>
> We recently got into very tricky issue (nullability change broke stateful
> operator) which required custom debug logging on plan change logging. This is
> because of lack of visibility for the output columns, especially their
> nullability, in our tree string of the plan.
> To address the lack of visibility, we propose to add another tree string
> which focuses to produce output columns with data type and nullability.
> Ideally, we shouldn't have two different tree strings and just consolidate,
> but in many cases, current tree string is long enough so that we had to
> restrict the number of fields to show, hence we think it's better to have a
> separate tree plan for it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
