[
https://issues.apache.org/jira/browse/SPARK-34246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gengliang Wang updated SPARK-34246:
-----------------------------------
Description:
Add new implicit cast syntax rules in ANSI mode.
In Spark ANSI mode, the type coercion rules are based on the type precedence
lists of the input data types.
As per the section "Type precedence list determination" of "ISO/IEC 9075-2:2011
Information technology — Database languages - SQL — Part 2: Foundation
(SQL/Foundation)", the type precedence lists of primitive
data types are as following:
* Byte: Byte, Short, Int, Long, Decimal, Float, Double
* Short: Short, Int, Long, Decimal, Float, Double
* Int: Int, Long, Decimal, Float, Double
* Long: Long, Decimal, Float, Double
* Decimal: Any wider Numeric type
* Float: Float, Double
* Double: Double
* String: String
* Date: Date, Timestamp
* Timestamp: Timestamp
* Binary: Binary
* Boolean: Boolean
* Interval: Interval
As for complex data types, Spark will determine the precedent list recursively
based on their sub-types.
With the definition of type precedent list, the general type coercion rules are
as following:
* Data type S is allowed to be implicitly cast as type T iff T is in the
precedence list of S
* Comparison is allowed iff the data type precedence list of both sides has at
least one common element. When evaluating the comparison, Spark casts both
sides as the tightest common data type of their precedent lists.
* There should be at least one common data type among all the children's
precedence lists for the following operators. The data type of the operator is
the tightest common precedent data type.
{code:java}
In
Except(odd)
Intersect
Greatest
Least
Union
If
CaseWhen
CreateArray
Array Concat
Sequence
MapConcat
CreateMap
{code}
* For complex types (struct, array, map), Spark recursively looks into the
element type and applies the rules above. If the element nullability is
converted from true to false, add runtime null check to the elements.
was:
Add new implicit cast syntax rules in ANSI mode.
In Spark ANSI mode, the type coercion rules are based on the type precedence
lists of the input data types.
As per the section "Type precedence list determination" of "ISO/IEC 9075-2:2011
Information technology — Database languages - SQL — Part 2: Foundation
(SQL/Foundation)", the type precedence lists of primitive
data types are as following:
* Byte: Byte, Short, Int, Long, Decimal, Float, Double
* Short: Short, Int, Long, Decimal, Float, Double
* Int: Int, Long, Decimal, Float, Double
* Long: Long, Decimal, Float, Double
* Decimal: Any wider Numeric type
* Float: Float, Double
* Double: Double
* String: String
* Date: Date, Timestamp
* Timestamp: Timestamp
* Binary: Binary
* Boolean: Boolean
* Interval: Interval
As for complex data types, Spark will determine the precedent list recursively
based on their sub-types.
With the definition of type precedent list, the general type coercion rules are
as following:
* Data type S is allowed to be implicitly cast as type T iff T is in the
precedence list of S
* Comparison is allowed iff the data type precedence list of both sides has at
least one common element. When evaluating the comparison, Spark casts both
sides as the tightest common data type of their precedent lists.
* There should be at least one common data type among all the children's
precedence lists for the following operators. The data type of the operator is
the tightest common precedent data type.
* In
* Except(odd)
* Intersect
* Greatest
* Least
* Union
* If
* CaseWhen
* CreateArray
* Array Concat
* Sequence
* MapConcat
* CreateMap
* For complex types (struct, array, map), Spark recursively looks into the
element type and applies the rules above. If the element nullability is
converted from true to false, add runtime null check to the elements.
> New implicit cast syntax rules in ANSI mode
> -------------------------------------------
>
> Key: SPARK-34246
> URL: https://issues.apache.org/jira/browse/SPARK-34246
> Project: Spark
> Issue Type: New Feature
> Components: SQL
> Affects Versions: 3.1.2
> Reporter: Gengliang Wang
> Assignee: Gengliang Wang
> Priority: Major
>
> Add new implicit cast syntax rules in ANSI mode.
> In Spark ANSI mode, the type coercion rules are based on the type precedence
> lists of the input data types.
> As per the section "Type precedence list determination" of "ISO/IEC
> 9075-2:2011
> Information technology — Database languages - SQL — Part 2: Foundation
> (SQL/Foundation)", the type precedence lists of primitive
> data types are as following:
> * Byte: Byte, Short, Int, Long, Decimal, Float, Double
> * Short: Short, Int, Long, Decimal, Float, Double
> * Int: Int, Long, Decimal, Float, Double
> * Long: Long, Decimal, Float, Double
> * Decimal: Any wider Numeric type
> * Float: Float, Double
> * Double: Double
> * String: String
> * Date: Date, Timestamp
> * Timestamp: Timestamp
> * Binary: Binary
> * Boolean: Boolean
> * Interval: Interval
> As for complex data types, Spark will determine the precedent list
> recursively based on their sub-types.
> With the definition of type precedent list, the general type coercion rules
> are as following:
> * Data type S is allowed to be implicitly cast as type T iff T is in the
> precedence list of S
> * Comparison is allowed iff the data type precedence list of both sides has
> at least one common element. When evaluating the comparison, Spark casts both
> sides as the tightest common data type of their precedent lists.
> * There should be at least one common data type among all the children's
> precedence lists for the following operators. The data type of the operator
> is the tightest common precedent data type.
> {code:java}
> In
> Except(odd)
> Intersect
> Greatest
> Least
> Union
> If
> CaseWhen
> CreateArray
> Array Concat
> Sequence
> MapConcat
> CreateMap
> {code}
> * For complex types (struct, array, map), Spark recursively looks into the
> element type and applies the rules above. If the element nullability is
> converted from true to false, add runtime null check to the elements.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
