[ 
https://issues.apache.org/jira/browse/SPARK-61931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serge Rielau updated SPARK-61931:
---------------------------------
    Labels: SPIP  (was: )

> SPIP: Add the OBJECT data type
> ------------------------------
>
>                 Key: SPARK-61931
>                 URL: https://issues.apache.org/jira/browse/SPARK-61931
>             Project: Spark
>          Issue Type: New Feature
>          Components: SQL
>    Affects Versions: 5.0.0
>            Reporter: Serge Rielau
>            Priority: Major
>              Labels: SPIP
>
> h1. Q1. What are you trying to do? Articulate your objectives using 
> absolutely no jargon.
> Add a SQL data type, {{{}OBJECT{}}}, for structured objects: ordered groups 
> of typed attributes where attribute names must match for a value to be 
> inserted, assigned, compared, or implicitly coerced.
> Today, STRUCT is Spark's only named group of fields, and it matches fields by 
> position. By default, names are ignored. All of these succeed under ANSI mode:
> {code:sql}
> CREATE TABLE employees (e STRUCT<name STRING, title STRING>) USING parquet;
> INSERT INTO employees SELECT named_struct('salary', 150000, 'is_contractor', 
> true);
> -- e.name = '150000', e.title = 'true'
> SELECT named_struct('a', 1, 'b', 2) = named_struct('b', 1, 'a', 2);
> -- true
> {code}
> OBJECT keeps STRUCT's shape (ordered, typed attributes, field access, 
> nesting) but makes names part of the type. Reordering needs an explicit CAST 
> that remaps by name:
> {code:sql}
> CAST(object('firstname', 'Ada', 'lastname', 'Lovelace')
>      AS OBJECT<lastname STRING, firstname STRING>);
> -- result: lastname = Lovelace, firstname = Ada
> {code}
> Users can declare OBJECT columns, construct values, access attributes, cast 
> between OBJECT and STRUCT, and use {{{}OBJECT_APPEND{}}}, 
> {{{}OBJECT_DELETE{}}}, and {{{}OBJECT_PICK{}}}. The type is gated by 
> {{spark.sql.objectType.enabled}} (default false).
> h1. Q2. What problem is this proposal NOT designed to solve?
> Changing STRUCT. STRUCT keeps its positional semantics.
> Open, semi-structured objects with dynamic keys (Snowflake-style OBJECT). Use 
> VARIANT or MAP for those.
> ANSI SQL structured user-defined types ({{{}CREATE TYPE{}}}), inheritance, or 
> methods.
> A new on-disk format. OBJECT is stored like STRUCT; table metadata records 
> the difference.
> h1. Q3. How is it done today, and what are the limits of current practice?
> By default, STRUCT fields are matched by position for insert, assignment, and 
> comparison. Names are ignored and arity must match. Field types need only be 
> store-assignable for insert and assignment (for example, any atomic type into 
> STRING). For comparison they must have a common type.
> {{INSERT ... BY NAME}} is the existing mitigation. It matches top-level 
> columns and nested fields by name, and it is the clearest evidence of the 
> problem. It falls short in three ways:
> It is per statement. One INSERT without BY NAME silently reverts to 
> positional matching.
> The table owner cannot declare a column as name-safe.
> It covers INSERT and MERGE only. Comparison, CAST, UNION, CTAS, view schemas, 
> and function arguments remain positional.
> Other workarounds are weaker: rebuilding values with {{withField}} / 
> {{dropFields}} / manual projection (verbose, not enforced by the type 
> system), {{MAP<STRING, T>}} (one value type, no per-attribute types), and 
> VARIANT (semi-structured, different type system).
> h1. Q4. What is new in your approach and why do you think it will be 
> successful?
> Names become part of the type. The table owner declares name-safety once. 
> INSERT, comparison, CAST, and set operations follow it without a 
> per-statement clause. STRUCT and BY NAME remain available for existing code.
> A new first-class type, not a flag on STRUCT. CHAR and VARCHAR are StringType 
> subtypes that carry a length, and that required special handling across the 
> engine. OBJECT is its own type, so its rules do not depend on overlays.
> Reuse of STRUCT's machinery. Execution, codegen, and storage reuse STRUCT's 
> representation. New work concentrates on analysis and the type contract, 
> following the path used for TIMESTAMP_NTZ, ANSI intervals, and TIME.
> h1. Q5. Who cares? If you are successful, what difference will it make?
> Users who write nested data where field order changes over time and who today 
> rely on positional matching.
> Platforms and warehouses building on Spark SQL that need name-safe structured 
> values without moving to VARIANT.
> Users migrating from engines with structured object types.
> Success means new workloads choose OBJECT over STRUCT as the default for 
> structured records. STRUCT stays unchanged for positional use.
> h1. Q6. What are the risks?
> ||Risk||Mitigation||
> |Seen as duplicating BY NAME|BY NAME is per statement and DML only. OBJECT is 
> a type contract. See Q3 and Appendix C.|
> |Confused with Snowflake's semi-structured OBJECT|Describe it as a structured 
> object throughout. VARIANT and MAP remain the semi-structured types.|
> |Confused with the JVM {{ObjectType}} in Scala|The programmatic type is 
> {{{}SqlObjectType{}}}. {{ObjectType}} is unchanged.|
> |Implicit STRUCT/OBJECT coercion|None. An explicit CAST is required.|
> |Client or catalog gaps when enabled early|The feature flag defaults to 
> false. Default-on only after the Q8 exit criteria.|
> h1. Q7. How long will it take?
> About 3 months, reusing STRUCT's execution and storage. Rough split: base 
> type; compatibility and casts; functions; persistence and catalogs; clients; 
> docs and golden tests.
> h1. Q8. What are the mid-term and final "exams" to check for success?
> h2. Mid-term (about 6 weeks):
>  * DDL, constructor, field access, and equality work for OBJECT.
>  * Reordered attributes fail implicit assignment and comparison with a clear 
> error.
>  * Explicit reorder CAST and STRUCT <-> OBJECT casts work, with golden tests.
>  * {{{}OBJECT_APPEND{}}}, {{{}OBJECT_DELETE{}}}, and {{OBJECT_PICK}} work, 
> with golden tests.
>  * STRUCT workloads are unchanged when OBJECT is not used.
> h2. Final (about 3 months):
>  * Catalog persistence and file round-trip through the STRUCT physical layout.
>  * Spark Connect and PySpark can create, query, and collect OBJECT columns.
>  * The SQL reference documents the type, casts, and functions.
>  * Default-enablement criteria are agreed with the shepherd.
> h1. Appendix A. Proposed API
> {code:scala}
> case class SqlObjectType(fields: Seq[StructField]) extends DataType
> {code}
> SQL syntax: {{{}OBJECT<firstname STRING, lastname STRING>{}}}. Constructor: 
> {{{}object('firstname', 'Ada', 'lastname', 'Lovelace'){}}}.
> Feature flag: {{spark.sql.objectType.enabled}} (default false).
> h1. Appendix B. Language surface
> ||Area||Intent||
> |Type|Ordered, uniquely named attributes; nested types allowed; {{OBJECT<>}} 
> allowed|
> |Assignment and comparison|Names must match at each position; no implicit 
> reordering|
> |Casts|Explicit by-name remap when name sets match; {{ADD FIELDS}} and 
> {{RENAME FIELDS}} modifiers; explicit STRUCT <-> OBJECT|
> |Functions|{{OBJECT_APPEND(o, k, v [, updateFlag])}} (an update preserves the 
> attribute type), {{{}OBJECT_DELETE{}}}, {{OBJECT_PICK}}|
> |Storage|STRUCT physical layout; OBJECT vs STRUCT recorded in table metadata|
> Detailed rules, error classes, and the full test matrix belong in the design 
> document.
> h1. Appendix C. Rejected designs
>  * INSERT BY NAME only. Keep STRUCT and rely on a per-statement BY NAME 
> clause. Rejected: the default stays positional, the table owner cannot 
> declare name-safety, and comparison, CAST, and set operations stay 
> positional. See Q3.
>  * Metadata flag on StructType (CHAR/VARCHAR style). Rejected: an overlay 
> leaks into every rule that matches on the type, and CHAR/VARCHAR show the 
> cost. A separate DataType keeps the semantics isolated.
>  * VARIANT or MAP. Rejected: MAP requires one value type, and VARIANT is open 
> and untyped. Neither gives a closed, typed attribute list with STRUCT-like 
> execution.
>  * Open, Snowflake-style OBJECT. Rejected: VARIANT already covers open 
> documents. The gap is a static, name-checked product type.
>  * Implicit order-independent assignment. Rejected: implicit reordering makes 
> the result of inserts and unions depend on hidden remapping, and it is 
> invisible in review. An explicit CAST keeps the reorder visible.
>  * Positional {{{}OBJECT_INSERT(o, pos, k, v){}}}. Rejected: attributes are 
> identified by name, not by ordinal position.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to