[
https://issues.apache.org/jira/browse/SPARK-61931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Serge Rielau updated SPARK-61931:
---------------------------------
Labels: SPIP (was: )
> SPIP: Add the OBJECT data type
> ------------------------------
>
> Key: SPARK-61931
> URL: https://issues.apache.org/jira/browse/SPARK-61931
> Project: Spark
> Issue Type: New Feature
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Serge Rielau
> Priority: Major
> Labels: SPIP
>
> h1. Q1. What are you trying to do? Articulate your objectives using
> absolutely no jargon.
> Add a SQL data type, {{{}OBJECT{}}}, for structured objects: ordered groups
> of typed attributes where attribute names must match for a value to be
> inserted, assigned, compared, or implicitly coerced.
> Today, STRUCT is Spark's only named group of fields, and it matches fields by
> position. By default, names are ignored. All of these succeed under ANSI mode:
> {code:sql}
> CREATE TABLE employees (e STRUCT<name STRING, title STRING>) USING parquet;
> INSERT INTO employees SELECT named_struct('salary', 150000, 'is_contractor',
> true);
> -- e.name = '150000', e.title = 'true'
> SELECT named_struct('a', 1, 'b', 2) = named_struct('b', 1, 'a', 2);
> -- true
> {code}
> OBJECT keeps STRUCT's shape (ordered, typed attributes, field access,
> nesting) but makes names part of the type. Reordering needs an explicit CAST
> that remaps by name:
> {code:sql}
> CAST(object('firstname', 'Ada', 'lastname', 'Lovelace')
> AS OBJECT<lastname STRING, firstname STRING>);
> -- result: lastname = Lovelace, firstname = Ada
> {code}
> Users can declare OBJECT columns, construct values, access attributes, cast
> between OBJECT and STRUCT, and use {{{}OBJECT_APPEND{}}},
> {{{}OBJECT_DELETE{}}}, and {{{}OBJECT_PICK{}}}. The type is gated by
> {{spark.sql.objectType.enabled}} (default false).
> h1. Q2. What problem is this proposal NOT designed to solve?
> Changing STRUCT. STRUCT keeps its positional semantics.
> Open, semi-structured objects with dynamic keys (Snowflake-style OBJECT). Use
> VARIANT or MAP for those.
> ANSI SQL structured user-defined types ({{{}CREATE TYPE{}}}), inheritance, or
> methods.
> A new on-disk format. OBJECT is stored like STRUCT; table metadata records
> the difference.
> h1. Q3. How is it done today, and what are the limits of current practice?
> By default, STRUCT fields are matched by position for insert, assignment, and
> comparison. Names are ignored and arity must match. Field types need only be
> store-assignable for insert and assignment (for example, any atomic type into
> STRING). For comparison they must have a common type.
> {{INSERT ... BY NAME}} is the existing mitigation. It matches top-level
> columns and nested fields by name, and it is the clearest evidence of the
> problem. It falls short in three ways:
> It is per statement. One INSERT without BY NAME silently reverts to
> positional matching.
> The table owner cannot declare a column as name-safe.
> It covers INSERT and MERGE only. Comparison, CAST, UNION, CTAS, view schemas,
> and function arguments remain positional.
> Other workarounds are weaker: rebuilding values with {{withField}} /
> {{dropFields}} / manual projection (verbose, not enforced by the type
> system), {{MAP<STRING, T>}} (one value type, no per-attribute types), and
> VARIANT (semi-structured, different type system).
> h1. Q4. What is new in your approach and why do you think it will be
> successful?
> Names become part of the type. The table owner declares name-safety once.
> INSERT, comparison, CAST, and set operations follow it without a
> per-statement clause. STRUCT and BY NAME remain available for existing code.
> A new first-class type, not a flag on STRUCT. CHAR and VARCHAR are StringType
> subtypes that carry a length, and that required special handling across the
> engine. OBJECT is its own type, so its rules do not depend on overlays.
> Reuse of STRUCT's machinery. Execution, codegen, and storage reuse STRUCT's
> representation. New work concentrates on analysis and the type contract,
> following the path used for TIMESTAMP_NTZ, ANSI intervals, and TIME.
> h1. Q5. Who cares? If you are successful, what difference will it make?
> Users who write nested data where field order changes over time and who today
> rely on positional matching.
> Platforms and warehouses building on Spark SQL that need name-safe structured
> values without moving to VARIANT.
> Users migrating from engines with structured object types.
> Success means new workloads choose OBJECT over STRUCT as the default for
> structured records. STRUCT stays unchanged for positional use.
> h1. Q6. What are the risks?
> ||Risk||Mitigation||
> |Seen as duplicating BY NAME|BY NAME is per statement and DML only. OBJECT is
> a type contract. See Q3 and Appendix C.|
> |Confused with Snowflake's semi-structured OBJECT|Describe it as a structured
> object throughout. VARIANT and MAP remain the semi-structured types.|
> |Confused with the JVM {{ObjectType}} in Scala|The programmatic type is
> {{{}SqlObjectType{}}}. {{ObjectType}} is unchanged.|
> |Implicit STRUCT/OBJECT coercion|None. An explicit CAST is required.|
> |Client or catalog gaps when enabled early|The feature flag defaults to
> false. Default-on only after the Q8 exit criteria.|
> h1. Q7. How long will it take?
> About 3 months, reusing STRUCT's execution and storage. Rough split: base
> type; compatibility and casts; functions; persistence and catalogs; clients;
> docs and golden tests.
> h1. Q8. What are the mid-term and final "exams" to check for success?
> h2. Mid-term (about 6 weeks):
> * DDL, constructor, field access, and equality work for OBJECT.
> * Reordered attributes fail implicit assignment and comparison with a clear
> error.
> * Explicit reorder CAST and STRUCT <-> OBJECT casts work, with golden tests.
> * {{{}OBJECT_APPEND{}}}, {{{}OBJECT_DELETE{}}}, and {{OBJECT_PICK}} work,
> with golden tests.
> * STRUCT workloads are unchanged when OBJECT is not used.
> h2. Final (about 3 months):
> * Catalog persistence and file round-trip through the STRUCT physical layout.
> * Spark Connect and PySpark can create, query, and collect OBJECT columns.
> * The SQL reference documents the type, casts, and functions.
> * Default-enablement criteria are agreed with the shepherd.
> h1. Appendix A. Proposed API
> {code:scala}
> case class SqlObjectType(fields: Seq[StructField]) extends DataType
> {code}
> SQL syntax: {{{}OBJECT<firstname STRING, lastname STRING>{}}}. Constructor:
> {{{}object('firstname', 'Ada', 'lastname', 'Lovelace'){}}}.
> Feature flag: {{spark.sql.objectType.enabled}} (default false).
> h1. Appendix B. Language surface
> ||Area||Intent||
> |Type|Ordered, uniquely named attributes; nested types allowed; {{OBJECT<>}}
> allowed|
> |Assignment and comparison|Names must match at each position; no implicit
> reordering|
> |Casts|Explicit by-name remap when name sets match; {{ADD FIELDS}} and
> {{RENAME FIELDS}} modifiers; explicit STRUCT <-> OBJECT|
> |Functions|{{OBJECT_APPEND(o, k, v [, updateFlag])}} (an update preserves the
> attribute type), {{{}OBJECT_DELETE{}}}, {{OBJECT_PICK}}|
> |Storage|STRUCT physical layout; OBJECT vs STRUCT recorded in table metadata|
> Detailed rules, error classes, and the full test matrix belong in the design
> document.
> h1. Appendix C. Rejected designs
> * INSERT BY NAME only. Keep STRUCT and rely on a per-statement BY NAME
> clause. Rejected: the default stays positional, the table owner cannot
> declare name-safety, and comparison, CAST, and set operations stay
> positional. See Q3.
> * Metadata flag on StructType (CHAR/VARCHAR style). Rejected: an overlay
> leaks into every rule that matches on the type, and CHAR/VARCHAR show the
> cost. A separate DataType keeps the semantics isolated.
> * VARIANT or MAP. Rejected: MAP requires one value type, and VARIANT is open
> and untyped. Neither gives a closed, typed attribute list with STRUCT-like
> execution.
> * Open, Snowflake-style OBJECT. Rejected: VARIANT already covers open
> documents. The gap is a static, name-checked product type.
> * Implicit order-independent assignment. Rejected: implicit reordering makes
> the result of inserts and unions depend on hidden remapping, and it is
> invisible in review. An explicit CAST keeps the reorder visible.
> * Positional {{{}OBJECT_INSERT(o, pos, k, v){}}}. Rejected: attributes are
> identified by name, not by ordinal position.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]