[
https://issues.apache.org/jira/browse/SPARK-48530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Milicevic updated SPARK-48530:
------------------------------------
Description:
At the moment, variables in SQL scripts are creating session variables. We
don't want this, we want variables to be considered as local (within the
block/compound).
-To achieve this, we probably need to wait for labels support. Once we have it,
we can prepend variable names with labels to make distinction between variables
with the same name and only then reuse session variables mechanism to save
values with such composed names.-
-If the block/compound doesn't have label, we should generate it automatically
(GUID or something similar).-
Labels support is done, so we can use them now for local variables - they need
to be able to be referenced by <labelName>.<varName>.
We cannot reuse session variables for this - we need to implement local
variables per script, with scoping and shadowing.
Session variables need to be able to be accessed by `system.session.<varName>`
at all times from the script.
-Also, variables cannot recover from script failures - if a script fails,
variables won’t be dropped. If we intend to reuse session variables for local
variables, we should think about how to fix this.- We won't be using session
variables, so this shouldn't be a problem, but need to have it in mind because
wrong implementation can cause the same problem.
Use [Session Variables PR|https://github.com/apache/spark/pull/40474/files] as
a reference in which places variable changes need to happen.
was:
At the moment, variables in SQL scripts are creating session variables. We
don't want this, we want variables to be considered as local (within the
block/compound).
-To achieve this, we probably need to wait for labels support. Once we have it,
we can prepend variable names with labels to make distinction between variables
with the same name and only then reuse session variables mechanism to save
values with such composed names.-
-If the block/compound doesn't have label, we should generate it automatically
(GUID or something similar).-
Labels support is done, so we can use them now for local variables - they need
to be able to be referenced by <labelName>.<varName>.
We cannot reuse session variables for this - we need to implement local
variables per script, with scoping and shadowing.
-Also, variables cannot recover from script failures - if a script fails,
variables won’t be dropped. If we intend to reuse session variables for local
variables, we should think about how to fix this.- We won't be using session
variables, so this shouldn't be a problem, but need to have it in mind because
wrong implementation can cause the same problem.
Use [Session Variables PR|https://github.com/apache/spark/pull/40474/files] as
a reference in which places variable changes need to happen.
> [M0] Support for local variables
> --------------------------------
>
> Key: SPARK-48530
> URL: https://issues.apache.org/jira/browse/SPARK-48530
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Affects Versions: 4.0.0
> Reporter: David Milicevic
> Priority: Major
>
> At the moment, variables in SQL scripts are creating session variables. We
> don't want this, we want variables to be considered as local (within the
> block/compound).
>
> -To achieve this, we probably need to wait for labels support. Once we have
> it, we can prepend variable names with labels to make distinction between
> variables with the same name and only then reuse session variables mechanism
> to save values with such composed names.-
> -If the block/compound doesn't have label, we should generate it
> automatically (GUID or something similar).-
>
> Labels support is done, so we can use them now for local variables - they
> need to be able to be referenced by <labelName>.<varName>.
> We cannot reuse session variables for this - we need to implement local
> variables per script, with scoping and shadowing.
>
> Session variables need to be able to be accessed by
> `system.session.<varName>` at all times from the script.
>
> -Also, variables cannot recover from script failures - if a script fails,
> variables won’t be dropped. If we intend to reuse session variables for local
> variables, we should think about how to fix this.- We won't be using session
> variables, so this shouldn't be a problem, but need to have it in mind
> because wrong implementation can cause the same problem.
>
> Use [Session Variables PR|https://github.com/apache/spark/pull/40474/files]
> as a reference in which places variable changes need to happen.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
