[
https://issues.apache.org/jira/browse/SPARK-33138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leanken.Lin updated SPARK-33138:
--------------------------------
Description: Currently, temp view store mapping of temp view name and its
logicalPlan, and permanent view store in HMS stores its origin SQL text. So for
permanent view, when try to refer the permanent view, its SQL text will be
parse-analyze-optimize-plan again with current SQLConf and SparkSession
context, so it might keep changing the SQLConf and context is different each
time. So, in order the unify the behaviors of temp view and permanent view,
propose that we keep its origin SQLText for both temp and permanent view, and
also keep record of the SQLConf when the view was created. Each time we try to
refer the view, we using the Snapshot SQLConf to parse-analyze-optimize-plan
the SQLText, in this way, we can make sure the output of the created view to be
stable.
Environment: (was: Currently, temp view store mapping of temp view name
and its logicalPlan, and permanent view store in HMS stores its origin SQL
text. So for permanent view, when try to refer the permanent view, its SQL text
will be parse-analyze-optimize-plan again with current SQLConf and SparkSession
context, so it might keep changing the SQLConf and context is different each
time. So, in order the unify the behaviors of temp view and permanent view,
propose that we keep its origin SQLText for both temp and permanent view, and
also keep record of the SQLConf when the view was created. Each time we try to
refer the view, we using the Snapshot SQLConf to parse-analyze-optimize-plan
the SQLText, in this way, we can make sure the output of the created view to be
stable.)
> unify temp view and permanent view behaviors
> --------------------------------------------
>
> Key: SPARK-33138
> URL: https://issues.apache.org/jira/browse/SPARK-33138
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.1.0
> Reporter: Leanken.Lin
> Priority: Major
>
> Currently, temp view store mapping of temp view name and its logicalPlan, and
> permanent view store in HMS stores its origin SQL text. So for permanent
> view, when try to refer the permanent view, its SQL text will be
> parse-analyze-optimize-plan again with current SQLConf and SparkSession
> context, so it might keep changing the SQLConf and context is different each
> time. So, in order the unify the behaviors of temp view and permanent view,
> propose that we keep its origin SQLText for both temp and permanent view, and
> also keep record of the SQLConf when the view was created. Each time we try
> to refer the view, we using the Snapshot SQLConf to
> parse-analyze-optimize-plan the SQLText, in this way, we can make sure the
> output of the created view to be stable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
