[
https://issues.apache.org/jira/browse/SPARK-61931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Serge Rielau updated SPARK-61931:
---------------------------------
Description:
h1. Q1. What are you trying to do? Articulate your objectives using absolutely
no jargon.
Add a SQL data type, {{{}OBJECT{}}}, for structured objects: ordered groups of
typed attributes where attribute names must match for a value to be inserted,
assigned, compared, or implicitly coerced.
Today, STRUCT is Spark's only named group of fields, and it matches fields by
position. By default, names are ignored. All of these succeed under ANSI mode:
{code:sql}
CREATE TABLE employees (e STRUCT<name STRING, title STRING>) USING parquet;
INSERT INTO employees SELECT named_struct('salary', 150000, 'is_contractor',
true);
-- e.name = '150000', e.title = 'true'
SELECT named_struct('a', 1, 'b', 2) = named_struct('b', 1, 'a', 2);
-- true
{code}
OBJECT keeps STRUCT's shape (ordered, typed attributes, field access, nesting)
but makes names part of the type. Reordering needs an explicit CAST that remaps
by name:
{code:sql}
CAST(object('firstname', 'Ada', 'lastname', 'Lovelace')
AS OBJECT<lastname STRING, firstname STRING>);
-- result: lastname = Lovelace, firstname = Ada
{code}
Users can declare OBJECT columns, construct values, access attributes, cast
between OBJECT and STRUCT, and use {{{}OBJECT_APPEND{}}},
{{{}OBJECT_DELETE{}}}, and {{{}OBJECT_PICK{}}}. The type is gated by
{{spark.sql.objectType.enabled}} (default false).
h1. Q2. What problem is this proposal NOT designed to solve?
Changing STRUCT. STRUCT keeps its positional semantics.
Open, semi-structured objects with dynamic keys (Snowflake-style OBJECT). Use
VARIANT or MAP for those.
ANSI SQL structured user-defined types ({{{}CREATE TYPE{}}}), inheritance, or
methods.
A new on-disk format. OBJECT is stored like STRUCT; table metadata records the
difference.
h1. Q3. How is it done today, and what are the limits of current practice?
By default, STRUCT fields are matched by position for insert, assignment, and
comparison. Names are ignored and arity must match. Field types need only be
store-assignable for insert and assignment (for example, any atomic type into
STRING). For comparison they must have a common type.
{{INSERT ... BY NAME}} is the existing mitigation. It matches top-level columns
and nested fields by name, and it is the clearest evidence of the problem. It
falls short in three ways:
It is per statement. One INSERT without BY NAME silently reverts to positional
matching.
The table owner cannot declare a column as name-safe.
It covers INSERT and MERGE only. Comparison, CAST, UNION, CTAS, view schemas,
and function arguments remain positional.
Other workarounds are weaker: rebuilding values with {{withField}} /
{{dropFields}} / manual projection (verbose, not enforced by the type system),
{{MAP<STRING, T>}} (one value type, no per-attribute types), and VARIANT
(semi-structured, different type system).
h1. Q4. What is new in your approach and why do you think it will be successful?
Names become part of the type. The table owner declares name-safety once.
INSERT, comparison, CAST, and set operations follow it without a per-statement
clause. STRUCT and BY NAME remain available for existing code.
A new first-class type, not a flag on STRUCT. CHAR and VARCHAR are StringType
subtypes that carry a length, and that required special handling across the
engine. OBJECT is its own type, so its rules do not depend on overlays.
Reuse of STRUCT's machinery. Execution, codegen, and storage reuse STRUCT's
representation. New work concentrates on analysis and the type contract,
following the path used for TIMESTAMP_NTZ, ANSI intervals, and TIME.
h1. Q5. Who cares? If you are successful, what difference will it make?
Users who write nested data where field order changes over time and who today
rely on positional matching.
Platforms and warehouses building on Spark SQL that need name-safe structured
values without moving to VARIANT.
Users migrating from engines with structured object types.
Success means new workloads choose OBJECT over STRUCT as the default for
structured records. STRUCT stays unchanged for positional use.
h1. Q6. What are the risks?
||Risk||Mitigation||
|Seen as duplicating BY NAME|BY NAME is per statement and DML only. OBJECT is a
type contract. See Q3 and Appendix C.|
|Confused with Snowflake's semi-structured OBJECT|Describe it as a structured
object throughout. VARIANT and MAP remain the semi-structured types.|
|Confused with the JVM {{ObjectType}} in Scala|The programmatic type is
{{{}SqlObjectType{}}}. {{ObjectType}} is unchanged.|
|Implicit STRUCT/OBJECT coercion|None. An explicit CAST is required.|
|Client or catalog gaps when enabled early|The feature flag defaults to false.
Default-on only after the Q8 exit criteria.|
h1. Q7. How long will it take?
About 3 months, reusing STRUCT's execution and storage. Rough split: base type;
compatibility and casts; functions; persistence and catalogs; clients; docs and
golden tests.
h1. Q8. What are the mid-term and final "exams" to check for success?
h2. Mid-term (about 6 weeks):
* DDL, constructor, field access, and equality work for OBJECT.
* Reordered attributes fail implicit assignment and comparison with a clear
error.
* Explicit reorder CAST and STRUCT <-> OBJECT casts work, with golden tests.
* {{{}OBJECT_APPEND{}}}, {{{}OBJECT_DELETE{}}}, and {{OBJECT_PICK}} work, with
golden tests.
* STRUCT workloads are unchanged when OBJECT is not used.
h2. Final (about 3 months):
* Catalog persistence and file round-trip through the STRUCT physical layout.
* Spark Connect and PySpark can create, query, and collect OBJECT columns.
* The SQL reference documents the type, casts, and functions.
* Default-enablement criteria are agreed with the shepherd.
h1. Appendix A. Proposed API
{code:scala}
case class SqlObjectType(fields: Seq[StructField]) extends DataType
{code}
SQL syntax: {{{}OBJECT<firstname STRING, lastname STRING>{}}}. Constructor:
{{{}object('firstname', 'Ada', 'lastname', 'Lovelace'){}}}.
Feature flag: {{spark.sql.objectType.enabled}} (default false).
h1. Appendix B. Language surface
||Area||Intent||
|Type|Ordered, uniquely named attributes; nested types allowed; {{OBJECT<>}}
allowed|
|Assignment and comparison|Names must match at each position; no implicit
reordering|
|Casts|Explicit by-name remap when name sets match; {{ADD FIELDS}} and {{RENAME
FIELDS}} modifiers; explicit STRUCT <-> OBJECT|
|Functions|{{OBJECT_APPEND(o, k, v [, updateFlag])}} (an update preserves the
attribute type), {{{}OBJECT_DELETE{}}}, {{OBJECT_PICK}}|
|Storage|STRUCT physical layout; OBJECT vs STRUCT recorded in table metadata|
Detailed rules, error classes, and the full test matrix belong in the design
document.
h1. Appendix C. Rejected designs
* INSERT BY NAME only. Keep STRUCT and rely on a per-statement BY NAME clause.
Rejected: the default stays positional, the table owner cannot declare
name-safety, and comparison, CAST, and set operations stay positional. See Q3.
* Metadata flag on StructType (CHAR/VARCHAR style). Rejected: an overlay leaks
into every rule that matches on the type, and CHAR/VARCHAR show the cost. A
separate DataType keeps the semantics isolated.
* VARIANT or MAP. Rejected: MAP requires one value type, and VARIANT is open
and untyped. Neither gives a closed, typed attribute list with STRUCT-like
execution.
* Open, Snowflake-style OBJECT. Rejected: VARIANT already covers open
documents. The gap is a static, name-checked product type.
* Implicit order-independent assignment. Rejected: implicit reordering makes
the result of inserts and unions depend on hidden remapping, and it is
invisible in review. An explicit CAST keeps the reorder visible.
* Positional {{{}OBJECT_INSERT(o, pos, k, v){}}}. Rejected: attributes are
identified by name, not by ordinal position.
was:
h2. Q1. What are you trying to do? Articulate your objectives using absolutely
no jargon.
Add a new Spark SQL data type, {*}OBJECT{*}, for {*}structured objects{*}:
named groups of typed attributes where {*}attribute names are part of type
compatibility{*}.
Spark's *STRUCT* is also a named group of fields, but compatibility is
{*}positional{*}. Names are ignored for insert, assignment, and comparison;
field types need only be store-assignable. Under ANSI, this succeeds:
{{CREATE TABLE employees (e STRUCT<name STRING, title STRING>) USING parquet;
INSERT INTO employees
SELECT named_struct('salary', 150000, 'is_contractor', true);
-- stored as: e.name = '150000', e.title = 'true'}}
The names are unrelated, the types differ ({{{}INT{}}} / {{BOOLEAN}} into
{{STRING}} / {{{}STRING{}}}), and the insert still succeeds. The same rule
silently swaps values when names are only reordered:
{{CREATE TABLE pts (s STRUCT<x INT, y INT>) USING parquet;
INSERT INTO pts SELECT named_struct('y', 20, 'x', 10);
-- stored as: s.x = 20, s.y = 10}}
*OBJECT* keeps STRUCT's useful shape (ordered, typed attributes; field access;
nesting) but makes names matter: attributes must align by name for assignment,
comparison, and implicit coercion. Reordering requires an explicit {{CAST}}
that remaps by name. Attribute names within a structured object are unique.
{{CAST(object('firstname', 'Ada', 'lastname', 'Lovelace')
AS OBJECT<lastname STRING, firstname STRING>);
-- \{lastname: Lovelace, firstname: Ada}
object('firstname', 'Ada', 'lastname', 'Lovelace')
= object('lastname', 'Lovelace', 'firstname', 'Ada');
-- analysis error}}
Users can declare OBJECT columns, construct values, access attributes, cast
between OBJECT and STRUCT, and use functions to append, delete, and pick
attributes. The type is gated behind {{spark.sql.objectType.enabled}} (default
false) until coverage is ready.
h2. Q2. What problem is this proposal NOT designed to solve?
This proposal does {*}not{*}:
* Replace or change STRUCT behavior.
* Add Snowflake-style *semi-structured* OBJECT (open key sets /
schema-on-read). Use VARIANT or MAP for that.
* Add ANSI SQL *CREATE TYPE* structured user-defined types, inheritance, or
methods.
* Invent a new on-disk binary format. OBJECT is stored physically like STRUCT;
table metadata distinguishes OBJECT from STRUCT.
h2. Q3. How is it done today, and what are the limits of current practice?
Structured records in Spark SQL are modeled with {*}STRUCT{*}. For nested
insert, assignment, {*}and comparison{*}, Spark resolves struct fields {*}by
position{*}: names are not part of the compatibility check, and field types
need only be ANSI store-assignable (for example any atomic type into
{{{}STRING{}}}). Arity must match; names and exact types need not.
Workarounds when name-safe structured values are needed:
||Workaround||Limitation||
|Rely on STRUCT and careful field ordering|Silent mis-assignment or misleading
comparisons when names and positions disagree|
|Rebuild with {{withField}} / {{dropFields}} / manual projection|Verbose; easy
to get wrong; not enforced by the type system|
|{{MAP<STRING, T>}} for a single value type {{T}}|Homogeneous values only;
loses per-attribute types and a fixed attribute set|
|VARIANT|Semi-structured; different type system and tooling|
There is no first-class Spark SQL type that is STRUCT-shaped but *name-checked*
for assignment and comparison.
h2. Q4. What is new in your approach and why do you think it will be successful?
OBJECT fixes STRUCT's positional hazard while keeping what makes STRUCT useful:
a closed, ordered, typed attribute list; ordinary field access; nesting; and
the same physical execution shape.
It is introduced as a {*}new first-class data type{*}, not as a soft overlay on
STRUCT. That is deliberate: the CHAR/VARCHAR approach of specializing
{{StringType}} without a fully separate type has proven spotty across the
stack. OBJECT follows the playbook used for types such as TIMESTAMP_NTZ, ANSI
intervals, and TIME: its own type in Catalyst, explicit analysis rules, a
feature flag, and end-to-end coverage (SQL, catalogs, clients).
Because execution, codegen, and file layout can reuse STRUCT's representation,
the incremental cost is concentrated where the value is: name-checked
compatibility and a clear SQL contract for structured objects.
h2. Q5. Who cares? If you are successful, what difference will it make?
* *SQL users and pipeline authors* who want structured columns where attribute
names are part of the contract (APIs, nested payloads, migrations where field
order is unstable).
* *Platforms and warehouses building on Spark SQL* that need a stronger
structured type than positional STRUCT without moving to VARIANT.
* *Existing Spark users* who today rebuild structs by hand to avoid positional
surprises.
Success means new workloads prefer *OBJECT* over *STRUCT* as the safer default
for structured objects, while STRUCT remains available and unchanged for
positional use cases and existing code.
h2. Q6. What are the risks?
||Risk||Mitigation||
|Confusion with Snowflake / semi-structured OBJECT|Stress throughout docs and
naming that Spark OBJECT is a {*}structured object{*}: a closed, typed
attribute set. VARIANT and MAP remain the semi-structured options.|
|Confusion with JVM {{ObjectType}} in the Scala API|SQL type name remains
OBJECT; programmatic type uses a distinct name ({{{}SqlObjectType{}}}) so
existing {{ObjectType}} (JVM pass-through) is untouched|
|Accidental STRUCT/OBJECT coercion|No implicit STRUCT ↔ OBJECT coercion;
require explicit CAST|
|Incomplete client / catalog coverage if enabled too early|Feature flag default
false; enable-by-default only after Q8 exit criteria|
h2. Q7. How long will it take?
About {*}3 months{*}, leaning on STRUCT for execution and storage.
Suggested split (JIRA sub-tasks):
# *Base type* — {{{}SqlObjectType{}}}, parser/DDL, constructor, field access,
typeof/describe, feature flag.
# *Compatibility and casts* — name-checked assignability and comparison;
OBJECT↔OBJECT reorder cast; OBJECT↔STRUCT; ADD FIELDS / RENAME FIELDS cast
modifiers.
# *Functions* — {{{}OBJECT_APPEND{}}}, {{{}OBJECT_DELETE{}}},
{{{}OBJECT_PICK{}}}.
# *Persistence and catalogs* — table metadata distinction; file formats via
struct physical type.
# *Clients* — Spark Connect, JDBC/Hive/Thrift mapping, PySpark/DataFrame API.
# *Docs and golden SQL tests*
h2. Q8. What are the mid-term and final "exams" to check for success?
*Mid-term (~6 weeks):*
* OBJECT usable in SQL: DDL, constructor, field access, equality, name-checked
assignment failures for reordered attributes.
* Explicit CAST can reorder by name; STRUCT ↔ OBJECT round-trip with
documented rules.
* Core functions ({{{}OBJECT_APPEND{}}} / {{OBJECT_DELETE}} /
{{{}OBJECT_PICK{}}}) with golden tests.
* No behavioral change to STRUCT workloads when OBJECT is unused.
*Final exam (~3 months):*
* Tables and views preserve OBJECT in catalog metadata; file round-trip via
struct physical encoding.
* Connect and PySpark can create, query, and collect OBJECT columns.
* Documented SQL reference for the type, casts, and functions.
* Exit criteria for considering default enablement: agreed surface implemented
and tested; shepherd sign-off that partial support will not be enabled by
default.
----
h2. Appendix A. Proposed API sketch
Programmatic type in {{org.apache.spark.sql.types}} (name avoids clashing with
existing JVM {{{}ObjectType{}}}):
{{/**
* SQL OBJECT type: a structured object — an ordered product of uniquely
* named attributes. Compatibility is name-checked (names must align by
* position); reordering requires an explicit cast.
*
* SQL / DDL / typeof print as, e.g., object<a INT, b STRING>.
*/
case class SqlObjectType(fields: Array[StructField]) extends DataType \{
override def defaultSize: Int = fields.map(_.dataType.defaultSize).sum
override def asNullable: SqlObjectType = /* fields as nullable */
}}}
SQL syntax:
{{OBJECT<firstname STRING, lastname STRING>
object('firstname', 'Ada', 'lastname', 'Lovelace')}}
Feature flag: {{spark.sql.objectType.enabled}} (default {{{}false{}}}).
----
h2. Appendix B. Language surface (summary only)
||Area||Intent||
|Type|Structured object: closed, ordered attributes; unique names; nested types
allowed; empty {{OBJECT<>}} allowed|
|Compatibility|Name-checked positional assignability and comparison; no
implicit reorder|
|Cast|By-name remap when name sets match; {{ADD FIELDS}} / {{RENAME FIELDS}}
modifiers; explicit STRUCT ↔ OBJECT|
|Functions|{{OBJECT_APPEND(o, k, v [, updateFlag])}} (default updateFlag false;
updates preserve attr type), {{{}OBJECT_DELETE{}}}, {{OBJECT_PICK}}|
|Storage|Physical STRUCT layout; OBJECT vs STRUCT distinguished in table
metadata|
Detailed rules, error classes, and the full SQL test matrix are left to design
docs and implementation reviews so this SPIP stays focused on approval of the
type.
----
h2. Appendix C. Rejected Designs
# *Overlay / Metadata flag on* {{StructType}} *(the CHAR/VARCHAR approach):*
** _Alternative:_ Treat OBJECT as a {{StructType}} carrying a schema metadata
flag or type-tag rather than a separate {{{}DataType{}}}.
** _Why rejected:_ Spark's experience with
{{{}CharType{}}}/{{{}VarcharType{}}} as annotations on {{StringType}} showed
that metadata and type-tag overlays are leak-prone across Catalyst optimizer
rules, expression resolution, codegen, UDF boundaries, and external data
connectors. A distinct first-class {{DataType}} ({{{}SqlObjectType{}}})
provides clean pattern matching, strict isolation, and prevents silent decay
into positional STRUCT semantics.
# *Reusing* {{VARIANT}} *or* {{MAP}} *for structured records:*
** _Alternative:_ Advise users needing name-safe structured fields to use
{{VARIANT}} or {{{}MAP<STRING, T>{}}}.
** _Why rejected:_ {{MAP}} requires homogeneous value types and loses schema
enforcement; {{VARIANT}} is designed for open, semi-structured documents
without static compile-time schemas, rigid per-attribute types, or positional
physical layout. Structured objects require static schema checking, strong
types per attribute, and high-throughput columnar storage/execution identical
to {{{}STRUCT{}}}.
# *Snowflake-style semi-structured* {{OBJECT}} *(open schema-on-read):*
** _Alternative:_ Implement an open, schemaless key-value object type where
attributes are untyped variants.
** _Why rejected:_ Spark's {{VARIANT}} already serves the semi-structured
document use case with high performance. What users lack is a _safe, static,
schema-enforced_ product type where field names protect against positional
insertion accidents.
# {*}Implicit order-independent assignment ({*}{{{}OBJECT<a, b>{}}}
*assignable to* {{OBJECT<b, a>}} *without CAST):*
** _Alternative:_ Allow automatic by-name matching during assignment/insert
even when attribute order differs.
** _Why rejected:_ Creates subtle ambiguities when projecting or unnesting
(e.g. {{o.*}} attribute order becoming non-deterministic or inconsistent with
table definitions). Requiring explicit {{CAST(... AS OBJECT<...>)}} makes
attribute reordering intentional, self-documenting, and visible in code
reviews, while keeping assignment predictable.
# {*}Positional attribute insertion ({*}{{{}OBJECT_INSERT(o, pos, k,
v){}}}{*}):{*}
** _Alternative:_ Model object insertion after {{array_insert}} taking an
integer ordinal offset.
** _Why rejected:_ Users think of object attributes by their logical names,
not by ordinal array indices. Modeling attribute insertion with integer offsets
adds needless complexity and does not match standard SQL/warehouse object
manipulation habits.
----
h2. Appendix D. Naming and Terminology
* *SQL name:* {{OBJECT}} (a structured object, not a semi-structured document).
* *Scala/Java name:* {{SqlObjectType}} (existing {{ObjectType}} in
{{org.apache.spark.sql.types}} remains the JVM expression-pass-through type).
* *Not* Snowflake semi-structured OBJECT; *not* ANSI structured UDTs.
> SPIP: Add the OBJECT data type
> ------------------------------
>
> Key: SPARK-61931
> URL: https://issues.apache.org/jira/browse/SPARK-61931
> Project: Spark
> Issue Type: New Feature
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Serge Rielau
> Priority: Major
>
> h1. Q1. What are you trying to do? Articulate your objectives using
> absolutely no jargon.
> Add a SQL data type, {{{}OBJECT{}}}, for structured objects: ordered groups
> of typed attributes where attribute names must match for a value to be
> inserted, assigned, compared, or implicitly coerced.
> Today, STRUCT is Spark's only named group of fields, and it matches fields by
> position. By default, names are ignored. All of these succeed under ANSI mode:
> {code:sql}
> CREATE TABLE employees (e STRUCT<name STRING, title STRING>) USING parquet;
> INSERT INTO employees SELECT named_struct('salary', 150000, 'is_contractor',
> true);
> -- e.name = '150000', e.title = 'true'
> SELECT named_struct('a', 1, 'b', 2) = named_struct('b', 1, 'a', 2);
> -- true
> {code}
> OBJECT keeps STRUCT's shape (ordered, typed attributes, field access,
> nesting) but makes names part of the type. Reordering needs an explicit CAST
> that remaps by name:
> {code:sql}
> CAST(object('firstname', 'Ada', 'lastname', 'Lovelace')
> AS OBJECT<lastname STRING, firstname STRING>);
> -- result: lastname = Lovelace, firstname = Ada
> {code}
> Users can declare OBJECT columns, construct values, access attributes, cast
> between OBJECT and STRUCT, and use {{{}OBJECT_APPEND{}}},
> {{{}OBJECT_DELETE{}}}, and {{{}OBJECT_PICK{}}}. The type is gated by
> {{spark.sql.objectType.enabled}} (default false).
> h1. Q2. What problem is this proposal NOT designed to solve?
> Changing STRUCT. STRUCT keeps its positional semantics.
> Open, semi-structured objects with dynamic keys (Snowflake-style OBJECT). Use
> VARIANT or MAP for those.
> ANSI SQL structured user-defined types ({{{}CREATE TYPE{}}}), inheritance, or
> methods.
> A new on-disk format. OBJECT is stored like STRUCT; table metadata records
> the difference.
> h1. Q3. How is it done today, and what are the limits of current practice?
> By default, STRUCT fields are matched by position for insert, assignment, and
> comparison. Names are ignored and arity must match. Field types need only be
> store-assignable for insert and assignment (for example, any atomic type into
> STRING). For comparison they must have a common type.
> {{INSERT ... BY NAME}} is the existing mitigation. It matches top-level
> columns and nested fields by name, and it is the clearest evidence of the
> problem. It falls short in three ways:
> It is per statement. One INSERT without BY NAME silently reverts to
> positional matching.
> The table owner cannot declare a column as name-safe.
> It covers INSERT and MERGE only. Comparison, CAST, UNION, CTAS, view schemas,
> and function arguments remain positional.
> Other workarounds are weaker: rebuilding values with {{withField}} /
> {{dropFields}} / manual projection (verbose, not enforced by the type
> system), {{MAP<STRING, T>}} (one value type, no per-attribute types), and
> VARIANT (semi-structured, different type system).
> h1. Q4. What is new in your approach and why do you think it will be
> successful?
> Names become part of the type. The table owner declares name-safety once.
> INSERT, comparison, CAST, and set operations follow it without a
> per-statement clause. STRUCT and BY NAME remain available for existing code.
> A new first-class type, not a flag on STRUCT. CHAR and VARCHAR are StringType
> subtypes that carry a length, and that required special handling across the
> engine. OBJECT is its own type, so its rules do not depend on overlays.
> Reuse of STRUCT's machinery. Execution, codegen, and storage reuse STRUCT's
> representation. New work concentrates on analysis and the type contract,
> following the path used for TIMESTAMP_NTZ, ANSI intervals, and TIME.
> h1. Q5. Who cares? If you are successful, what difference will it make?
> Users who write nested data where field order changes over time and who today
> rely on positional matching.
> Platforms and warehouses building on Spark SQL that need name-safe structured
> values without moving to VARIANT.
> Users migrating from engines with structured object types.
> Success means new workloads choose OBJECT over STRUCT as the default for
> structured records. STRUCT stays unchanged for positional use.
> h1. Q6. What are the risks?
> ||Risk||Mitigation||
> |Seen as duplicating BY NAME|BY NAME is per statement and DML only. OBJECT is
> a type contract. See Q3 and Appendix C.|
> |Confused with Snowflake's semi-structured OBJECT|Describe it as a structured
> object throughout. VARIANT and MAP remain the semi-structured types.|
> |Confused with the JVM {{ObjectType}} in Scala|The programmatic type is
> {{{}SqlObjectType{}}}. {{ObjectType}} is unchanged.|
> |Implicit STRUCT/OBJECT coercion|None. An explicit CAST is required.|
> |Client or catalog gaps when enabled early|The feature flag defaults to
> false. Default-on only after the Q8 exit criteria.|
> h1. Q7. How long will it take?
> About 3 months, reusing STRUCT's execution and storage. Rough split: base
> type; compatibility and casts; functions; persistence and catalogs; clients;
> docs and golden tests.
> h1. Q8. What are the mid-term and final "exams" to check for success?
> h2. Mid-term (about 6 weeks):
> * DDL, constructor, field access, and equality work for OBJECT.
> * Reordered attributes fail implicit assignment and comparison with a clear
> error.
> * Explicit reorder CAST and STRUCT <-> OBJECT casts work, with golden tests.
> * {{{}OBJECT_APPEND{}}}, {{{}OBJECT_DELETE{}}}, and {{OBJECT_PICK}} work,
> with golden tests.
> * STRUCT workloads are unchanged when OBJECT is not used.
> h2. Final (about 3 months):
> * Catalog persistence and file round-trip through the STRUCT physical layout.
> * Spark Connect and PySpark can create, query, and collect OBJECT columns.
> * The SQL reference documents the type, casts, and functions.
> * Default-enablement criteria are agreed with the shepherd.
> h1. Appendix A. Proposed API
> {code:scala}
> case class SqlObjectType(fields: Seq[StructField]) extends DataType
> {code}
> SQL syntax: {{{}OBJECT<firstname STRING, lastname STRING>{}}}. Constructor:
> {{{}object('firstname', 'Ada', 'lastname', 'Lovelace'){}}}.
> Feature flag: {{spark.sql.objectType.enabled}} (default false).
> h1. Appendix B. Language surface
> ||Area||Intent||
> |Type|Ordered, uniquely named attributes; nested types allowed; {{OBJECT<>}}
> allowed|
> |Assignment and comparison|Names must match at each position; no implicit
> reordering|
> |Casts|Explicit by-name remap when name sets match; {{ADD FIELDS}} and
> {{RENAME FIELDS}} modifiers; explicit STRUCT <-> OBJECT|
> |Functions|{{OBJECT_APPEND(o, k, v [, updateFlag])}} (an update preserves the
> attribute type), {{{}OBJECT_DELETE{}}}, {{OBJECT_PICK}}|
> |Storage|STRUCT physical layout; OBJECT vs STRUCT recorded in table metadata|
> Detailed rules, error classes, and the full test matrix belong in the design
> document.
> h1. Appendix C. Rejected designs
> * INSERT BY NAME only. Keep STRUCT and rely on a per-statement BY NAME
> clause. Rejected: the default stays positional, the table owner cannot
> declare name-safety, and comparison, CAST, and set operations stay
> positional. See Q3.
> * Metadata flag on StructType (CHAR/VARCHAR style). Rejected: an overlay
> leaks into every rule that matches on the type, and CHAR/VARCHAR show the
> cost. A separate DataType keeps the semantics isolated.
> * VARIANT or MAP. Rejected: MAP requires one value type, and VARIANT is open
> and untyped. Neither gives a closed, typed attribute list with STRUCT-like
> execution.
> * Open, Snowflake-style OBJECT. Rejected: VARIANT already covers open
> documents. The gap is a static, name-checked product type.
> * Implicit order-independent assignment. Rejected: implicit reordering makes
> the result of inserts and unions depend on hidden remapping, and it is
> invisible in review. An explicit CAST keeps the reorder visible.
> * Positional {{{}OBJECT_INSERT(o, pos, k, v){}}}. Rejected: attributes are
> identified by name, not by ordinal position.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]