[ 
https://issues.apache.org/jira/browse/SPARK-61931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serge Rielau updated SPARK-61931:
---------------------------------
    Description: 
h1. Q1. What are you trying to do? Articulate your objectives using absolutely 
no jargon.

Add a SQL data type, {{{}OBJECT{}}}, for structured objects: ordered groups of 
typed attributes where attribute names must match for a value to be inserted, 
assigned, compared, or implicitly coerced.

Today, STRUCT is Spark's only named group of fields, and it matches fields by 
position. By default, names are ignored. All of these succeed under ANSI mode:
{code:sql}
CREATE TABLE employees (e STRUCT<name STRING, title STRING>) USING parquet;
INSERT INTO employees SELECT named_struct('salary', 150000, 'is_contractor', 
true);
-- e.name = '150000', e.title = 'true'

SELECT named_struct('a', 1, 'b', 2) = named_struct('b', 1, 'a', 2);
-- true
{code}
OBJECT keeps STRUCT's shape (ordered, typed attributes, field access, nesting) 
but makes names part of the type. Reordering needs an explicit CAST that remaps 
by name:
{code:sql}
CAST(object('firstname', 'Ada', 'lastname', 'Lovelace')
     AS OBJECT<lastname STRING, firstname STRING>);
-- result: lastname = Lovelace, firstname = Ada
{code}
Users can declare OBJECT columns, construct values, access attributes, cast 
between OBJECT and STRUCT, and use {{{}OBJECT_APPEND{}}}, 
{{{}OBJECT_DELETE{}}}, and {{{}OBJECT_PICK{}}}. The type is gated by 
{{spark.sql.objectType.enabled}} (default false).
h1. Q2. What problem is this proposal NOT designed to solve?

Changing STRUCT. STRUCT keeps its positional semantics.

Open, semi-structured objects with dynamic keys (Snowflake-style OBJECT). Use 
VARIANT or MAP for those.

ANSI SQL structured user-defined types ({{{}CREATE TYPE{}}}), inheritance, or 
methods.

A new on-disk format. OBJECT is stored like STRUCT; table metadata records the 
difference.
h1. Q3. How is it done today, and what are the limits of current practice?

By default, STRUCT fields are matched by position for insert, assignment, and 
comparison. Names are ignored and arity must match. Field types need only be 
store-assignable for insert and assignment (for example, any atomic type into 
STRING). For comparison they must have a common type.

{{INSERT ... BY NAME}} is the existing mitigation. It matches top-level columns 
and nested fields by name, and it is the clearest evidence of the problem. It 
falls short in three ways:

It is per statement. One INSERT without BY NAME silently reverts to positional 
matching.

The table owner cannot declare a column as name-safe.

It covers INSERT and MERGE only. Comparison, CAST, UNION, CTAS, view schemas, 
and function arguments remain positional.

Other workarounds are weaker: rebuilding values with {{withField}} / 
{{dropFields}} / manual projection (verbose, not enforced by the type system), 
{{MAP<STRING, T>}} (one value type, no per-attribute types), and VARIANT 
(semi-structured, different type system).
h1. Q4. What is new in your approach and why do you think it will be successful?

Names become part of the type. The table owner declares name-safety once. 
INSERT, comparison, CAST, and set operations follow it without a per-statement 
clause. STRUCT and BY NAME remain available for existing code.

A new first-class type, not a flag on STRUCT. CHAR and VARCHAR are StringType 
subtypes that carry a length, and that required special handling across the 
engine. OBJECT is its own type, so its rules do not depend on overlays.

Reuse of STRUCT's machinery. Execution, codegen, and storage reuse STRUCT's 
representation. New work concentrates on analysis and the type contract, 
following the path used for TIMESTAMP_NTZ, ANSI intervals, and TIME.
h1. Q5. Who cares? If you are successful, what difference will it make?

Users who write nested data where field order changes over time and who today 
rely on positional matching.

Platforms and warehouses building on Spark SQL that need name-safe structured 
values without moving to VARIANT.

Users migrating from engines with structured object types.

Success means new workloads choose OBJECT over STRUCT as the default for 
structured records. STRUCT stays unchanged for positional use.
h1. Q6. What are the risks?
||Risk||Mitigation||
|Seen as duplicating BY NAME|BY NAME is per statement and DML only. OBJECT is a 
type contract. See Q3 and Appendix C.|
|Confused with Snowflake's semi-structured OBJECT|Describe it as a structured 
object throughout. VARIANT and MAP remain the semi-structured types.|
|Confused with the JVM {{ObjectType}} in Scala|The programmatic type is 
{{{}SqlObjectType{}}}. {{ObjectType}} is unchanged.|
|Implicit STRUCT/OBJECT coercion|None. An explicit CAST is required.|
|Client or catalog gaps when enabled early|The feature flag defaults to false. 
Default-on only after the Q8 exit criteria.|
h1. Q7. How long will it take?

About 3 months, reusing STRUCT's execution and storage. Rough split: base type; 
compatibility and casts; functions; persistence and catalogs; clients; docs and 
golden tests.
h1. Q8. What are the mid-term and final "exams" to check for success?
h2. Mid-term (about 6 weeks):
 * DDL, constructor, field access, and equality work for OBJECT.
 * Reordered attributes fail implicit assignment and comparison with a clear 
error.
 * Explicit reorder CAST and STRUCT <-> OBJECT casts work, with golden tests.
 * {{{}OBJECT_APPEND{}}}, {{{}OBJECT_DELETE{}}}, and {{OBJECT_PICK}} work, with 
golden tests.
 * STRUCT workloads are unchanged when OBJECT is not used.

h2. Final (about 3 months):
 * Catalog persistence and file round-trip through the STRUCT physical layout.
 * Spark Connect and PySpark can create, query, and collect OBJECT columns.
 * The SQL reference documents the type, casts, and functions.
 * Default-enablement criteria are agreed with the shepherd.

h1. Appendix A. Proposed API
{code:scala}
case class SqlObjectType(fields: Seq[StructField]) extends DataType
{code}
SQL syntax: {{{}OBJECT<firstname STRING, lastname STRING>{}}}. Constructor: 
{{{}object('firstname', 'Ada', 'lastname', 'Lovelace'){}}}.

Feature flag: {{spark.sql.objectType.enabled}} (default false).
h1. Appendix B. Language surface
||Area||Intent||
|Type|Ordered, uniquely named attributes; nested types allowed; {{OBJECT<>}} 
allowed|
|Assignment and comparison|Names must match at each position; no implicit 
reordering|
|Casts|Explicit by-name remap when name sets match; {{ADD FIELDS}} and {{RENAME 
FIELDS}} modifiers; explicit STRUCT <-> OBJECT|
|Functions|{{OBJECT_APPEND(o, k, v [, updateFlag])}} (an update preserves the 
attribute type), {{{}OBJECT_DELETE{}}}, {{OBJECT_PICK}}|
|Storage|STRUCT physical layout; OBJECT vs STRUCT recorded in table metadata|

Detailed rules, error classes, and the full test matrix belong in the design 
document.
h1. Appendix C. Rejected designs
 * INSERT BY NAME only. Keep STRUCT and rely on a per-statement BY NAME clause. 
Rejected: the default stays positional, the table owner cannot declare 
name-safety, and comparison, CAST, and set operations stay positional. See Q3.
 * Metadata flag on StructType (CHAR/VARCHAR style). Rejected: an overlay leaks 
into every rule that matches on the type, and CHAR/VARCHAR show the cost. A 
separate DataType keeps the semantics isolated.
 * VARIANT or MAP. Rejected: MAP requires one value type, and VARIANT is open 
and untyped. Neither gives a closed, typed attribute list with STRUCT-like 
execution.
 * Open, Snowflake-style OBJECT. Rejected: VARIANT already covers open 
documents. The gap is a static, name-checked product type.
 * Implicit order-independent assignment. Rejected: implicit reordering makes 
the result of inserts and unions depend on hidden remapping, and it is 
invisible in review. An explicit CAST keeps the reorder visible.
 * Positional {{{}OBJECT_INSERT(o, pos, k, v){}}}. Rejected: attributes are 
identified by name, not by ordinal position.

  was:
h2. Q1. What are you trying to do? Articulate your objectives using absolutely 
no jargon.

Add a new Spark SQL data type, {*}OBJECT{*}, for {*}structured objects{*}: 
named groups of typed attributes where {*}attribute names are part of type 
compatibility{*}.

Spark's *STRUCT* is also a named group of fields, but compatibility is 
{*}positional{*}. Names are ignored for insert, assignment, and comparison; 
field types need only be store-assignable. Under ANSI, this succeeds:

{{CREATE TABLE employees (e STRUCT<name STRING, title STRING>) USING parquet;
INSERT INTO employees
SELECT named_struct('salary', 150000, 'is_contractor', true);
-- stored as: e.name = '150000', e.title = 'true'}}

The names are unrelated, the types differ ({{{}INT{}}} / {{BOOLEAN}} into 
{{STRING}} / {{{}STRING{}}}), and the insert still succeeds. The same rule 
silently swaps values when names are only reordered:

{{CREATE TABLE pts (s STRUCT<x INT, y INT>) USING parquet;
INSERT INTO pts SELECT named_struct('y', 20, 'x', 10);
-- stored as: s.x = 20, s.y = 10}}

*OBJECT* keeps STRUCT's useful shape (ordered, typed attributes; field access; 
nesting) but makes names matter: attributes must align by name for assignment, 
comparison, and implicit coercion. Reordering requires an explicit {{CAST}} 
that remaps by name. Attribute names within a structured object are unique.

{{CAST(object('firstname', 'Ada', 'lastname', 'Lovelace')
     AS OBJECT<lastname STRING, firstname STRING>);
-- \{lastname: Lovelace, firstname: Ada}

object('firstname', 'Ada', 'lastname', 'Lovelace')
  = object('lastname', 'Lovelace', 'firstname', 'Ada');
-- analysis error}}

Users can declare OBJECT columns, construct values, access attributes, cast 
between OBJECT and STRUCT, and use functions to append, delete, and pick 
attributes. The type is gated behind {{spark.sql.objectType.enabled}} (default 
false) until coverage is ready.
h2. Q2. What problem is this proposal NOT designed to solve?

This proposal does {*}not{*}:
 * Replace or change STRUCT behavior.

 * Add Snowflake-style *semi-structured* OBJECT (open key sets / 
schema-on-read). Use VARIANT or MAP for that.

 * Add ANSI SQL *CREATE TYPE* structured user-defined types, inheritance, or 
methods.

 * Invent a new on-disk binary format. OBJECT is stored physically like STRUCT; 
table metadata distinguishes OBJECT from STRUCT.

h2. Q3. How is it done today, and what are the limits of current practice?

Structured records in Spark SQL are modeled with {*}STRUCT{*}. For nested 
insert, assignment, {*}and comparison{*}, Spark resolves struct fields {*}by 
position{*}: names are not part of the compatibility check, and field types 
need only be ANSI store-assignable (for example any atomic type into 
{{{}STRING{}}}). Arity must match; names and exact types need not.

Workarounds when name-safe structured values are needed:
||Workaround||Limitation||
|Rely on STRUCT and careful field ordering|Silent mis-assignment or misleading 
comparisons when names and positions disagree|
|Rebuild with {{withField}} / {{dropFields}} / manual projection|Verbose; easy 
to get wrong; not enforced by the type system|
|{{MAP<STRING, T>}} for a single value type {{T}}|Homogeneous values only; 
loses per-attribute types and a fixed attribute set|
|VARIANT|Semi-structured; different type system and tooling|

There is no first-class Spark SQL type that is STRUCT-shaped but *name-checked* 
for assignment and comparison.
h2. Q4. What is new in your approach and why do you think it will be successful?

OBJECT fixes STRUCT's positional hazard while keeping what makes STRUCT useful: 
a closed, ordered, typed attribute list; ordinary field access; nesting; and 
the same physical execution shape.

It is introduced as a {*}new first-class data type{*}, not as a soft overlay on 
STRUCT. That is deliberate: the CHAR/VARCHAR approach of specializing 
{{StringType}} without a fully separate type has proven spotty across the 
stack. OBJECT follows the playbook used for types such as TIMESTAMP_NTZ, ANSI 
intervals, and TIME: its own type in Catalyst, explicit analysis rules, a 
feature flag, and end-to-end coverage (SQL, catalogs, clients).

Because execution, codegen, and file layout can reuse STRUCT's representation, 
the incremental cost is concentrated where the value is: name-checked 
compatibility and a clear SQL contract for structured objects.
h2. Q5. Who cares? If you are successful, what difference will it make?
 * *SQL users and pipeline authors* who want structured columns where attribute 
names are part of the contract (APIs, nested payloads, migrations where field 
order is unstable).

 * *Platforms and warehouses building on Spark SQL* that need a stronger 
structured type than positional STRUCT without moving to VARIANT.

 * *Existing Spark users* who today rebuild structs by hand to avoid positional 
surprises.

Success means new workloads prefer *OBJECT* over *STRUCT* as the safer default 
for structured objects, while STRUCT remains available and unchanged for 
positional use cases and existing code.
h2. Q6. What are the risks?
||Risk||Mitigation||
|Confusion with Snowflake / semi-structured OBJECT|Stress throughout docs and 
naming that Spark OBJECT is a {*}structured object{*}: a closed, typed 
attribute set. VARIANT and MAP remain the semi-structured options.|
|Confusion with JVM {{ObjectType}} in the Scala API|SQL type name remains 
OBJECT; programmatic type uses a distinct name ({{{}SqlObjectType{}}}) so 
existing {{ObjectType}} (JVM pass-through) is untouched|
|Accidental STRUCT/OBJECT coercion|No implicit STRUCT ↔ OBJECT coercion; 
require explicit CAST|
|Incomplete client / catalog coverage if enabled too early|Feature flag default 
false; enable-by-default only after Q8 exit criteria|
h2. Q7. How long will it take?

About {*}3 months{*}, leaning on STRUCT for execution and storage.

Suggested split (JIRA sub-tasks):
 # *Base type* — {{{}SqlObjectType{}}}, parser/DDL, constructor, field access, 
typeof/describe, feature flag.

 # *Compatibility and casts* — name-checked assignability and comparison; 
OBJECT↔OBJECT reorder cast; OBJECT↔STRUCT; ADD FIELDS / RENAME FIELDS cast 
modifiers.

 # *Functions* — {{{}OBJECT_APPEND{}}}, {{{}OBJECT_DELETE{}}}, 
{{{}OBJECT_PICK{}}}.

 # *Persistence and catalogs* — table metadata distinction; file formats via 
struct physical type.

 # *Clients* — Spark Connect, JDBC/Hive/Thrift mapping, PySpark/DataFrame API.

 # *Docs and golden SQL tests*

h2. Q8. What are the mid-term and final "exams" to check for success?

*Mid-term (~6 weeks):*
 * OBJECT usable in SQL: DDL, constructor, field access, equality, name-checked 
assignment failures for reordered attributes.

 * Explicit CAST can reorder by name; STRUCT ↔ OBJECT round-trip with 
documented rules.

 * Core functions ({{{}OBJECT_APPEND{}}} / {{OBJECT_DELETE}} / 
{{{}OBJECT_PICK{}}}) with golden tests.

 * No behavioral change to STRUCT workloads when OBJECT is unused.

*Final exam (~3 months):*
 * Tables and views preserve OBJECT in catalog metadata; file round-trip via 
struct physical encoding.

 * Connect and PySpark can create, query, and collect OBJECT columns.

 * Documented SQL reference for the type, casts, and functions.

 * Exit criteria for considering default enablement: agreed surface implemented 
and tested; shepherd sign-off that partial support will not be enabled by 
default.

----
h2. Appendix A. Proposed API sketch

Programmatic type in {{org.apache.spark.sql.types}} (name avoids clashing with 
existing JVM {{{}ObjectType{}}}):

{{/**
 * SQL OBJECT type: a structured object — an ordered product of uniquely
 * named attributes. Compatibility is name-checked (names must align by
 * position); reordering requires an explicit cast.
 *
 * SQL / DDL / typeof print as, e.g., object<a INT, b STRING>.
 */
case class SqlObjectType(fields: Array[StructField]) extends DataType \{
  override def defaultSize: Int = fields.map(_.dataType.defaultSize).sum
  override def asNullable: SqlObjectType = /* fields as nullable */
}}}

SQL syntax:

{{OBJECT<firstname STRING, lastname STRING>
object('firstname', 'Ada', 'lastname', 'Lovelace')}}

Feature flag: {{spark.sql.objectType.enabled}} (default {{{}false{}}}).
----
h2. Appendix B. Language surface (summary only)
||Area||Intent||
|Type|Structured object: closed, ordered attributes; unique names; nested types 
allowed; empty {{OBJECT<>}} allowed|
|Compatibility|Name-checked positional assignability and comparison; no 
implicit reorder|
|Cast|By-name remap when name sets match; {{ADD FIELDS}} / {{RENAME FIELDS}} 
modifiers; explicit STRUCT ↔ OBJECT|
|Functions|{{OBJECT_APPEND(o, k, v [, updateFlag])}} (default updateFlag false; 
updates preserve attr type), {{{}OBJECT_DELETE{}}}, {{OBJECT_PICK}}|
|Storage|Physical STRUCT layout; OBJECT vs STRUCT distinguished in table 
metadata|

Detailed rules, error classes, and the full SQL test matrix are left to design 
docs and implementation reviews so this SPIP stays focused on approval of the 
type.
----
h2. Appendix C. Rejected Designs
 # *Overlay / Metadata flag on* {{StructType}} *(the CHAR/VARCHAR approach):*

 ** _Alternative:_ Treat OBJECT as a {{StructType}} carrying a schema metadata 
flag or type-tag rather than a separate {{{}DataType{}}}.

 ** _Why rejected:_ Spark's experience with 
{{{}CharType{}}}/{{{}VarcharType{}}} as annotations on {{StringType}} showed 
that metadata and type-tag overlays are leak-prone across Catalyst optimizer 
rules, expression resolution, codegen, UDF boundaries, and external data 
connectors. A distinct first-class {{DataType}} ({{{}SqlObjectType{}}}) 
provides clean pattern matching, strict isolation, and prevents silent decay 
into positional STRUCT semantics.

 # *Reusing* {{VARIANT}} *or* {{MAP}} *for structured records:*

 ** _Alternative:_ Advise users needing name-safe structured fields to use 
{{VARIANT}} or {{{}MAP<STRING, T>{}}}.

 ** _Why rejected:_ {{MAP}} requires homogeneous value types and loses schema 
enforcement; {{VARIANT}} is designed for open, semi-structured documents 
without static compile-time schemas, rigid per-attribute types, or positional 
physical layout. Structured objects require static schema checking, strong 
types per attribute, and high-throughput columnar storage/execution identical 
to {{{}STRUCT{}}}.

 # *Snowflake-style semi-structured* {{OBJECT}} *(open schema-on-read):*

 ** _Alternative:_ Implement an open, schemaless key-value object type where 
attributes are untyped variants.

 ** _Why rejected:_ Spark's {{VARIANT}} already serves the semi-structured 
document use case with high performance. What users lack is a _safe, static, 
schema-enforced_ product type where field names protect against positional 
insertion accidents.

 # {*}Implicit order-independent assignment ({*}{{{}OBJECT<a, b>{}}} 
*assignable to* {{OBJECT<b, a>}} *without CAST):*

 ** _Alternative:_ Allow automatic by-name matching during assignment/insert 
even when attribute order differs.

 ** _Why rejected:_ Creates subtle ambiguities when projecting or unnesting 
(e.g. {{o.*}} attribute order becoming non-deterministic or inconsistent with 
table definitions). Requiring explicit {{CAST(... AS OBJECT<...>)}} makes 
attribute reordering intentional, self-documenting, and visible in code 
reviews, while keeping assignment predictable.

 # {*}Positional attribute insertion ({*}{{{}OBJECT_INSERT(o, pos, k, 
v){}}}{*}):{*}

 ** _Alternative:_ Model object insertion after {{array_insert}} taking an 
integer ordinal offset.

 ** _Why rejected:_ Users think of object attributes by their logical names, 
not by ordinal array indices. Modeling attribute insertion with integer offsets 
adds needless complexity and does not match standard SQL/warehouse object 
manipulation habits.

----
h2. Appendix D. Naming and Terminology
 * *SQL name:* {{OBJECT}} (a structured object, not a semi-structured document).

 * *Scala/Java name:* {{SqlObjectType}} (existing {{ObjectType}} in 
{{org.apache.spark.sql.types}} remains the JVM expression-pass-through type).

 * *Not* Snowflake semi-structured OBJECT; *not* ANSI structured UDTs.

 


> SPIP: Add the OBJECT data type
> ------------------------------
>
>                 Key: SPARK-61931
>                 URL: https://issues.apache.org/jira/browse/SPARK-61931
>             Project: Spark
>          Issue Type: New Feature
>          Components: SQL
>    Affects Versions: 5.0.0
>            Reporter: Serge Rielau
>            Priority: Major
>
> h1. Q1. What are you trying to do? Articulate your objectives using 
> absolutely no jargon.
> Add a SQL data type, {{{}OBJECT{}}}, for structured objects: ordered groups 
> of typed attributes where attribute names must match for a value to be 
> inserted, assigned, compared, or implicitly coerced.
> Today, STRUCT is Spark's only named group of fields, and it matches fields by 
> position. By default, names are ignored. All of these succeed under ANSI mode:
> {code:sql}
> CREATE TABLE employees (e STRUCT<name STRING, title STRING>) USING parquet;
> INSERT INTO employees SELECT named_struct('salary', 150000, 'is_contractor', 
> true);
> -- e.name = '150000', e.title = 'true'
> SELECT named_struct('a', 1, 'b', 2) = named_struct('b', 1, 'a', 2);
> -- true
> {code}
> OBJECT keeps STRUCT's shape (ordered, typed attributes, field access, 
> nesting) but makes names part of the type. Reordering needs an explicit CAST 
> that remaps by name:
> {code:sql}
> CAST(object('firstname', 'Ada', 'lastname', 'Lovelace')
>      AS OBJECT<lastname STRING, firstname STRING>);
> -- result: lastname = Lovelace, firstname = Ada
> {code}
> Users can declare OBJECT columns, construct values, access attributes, cast 
> between OBJECT and STRUCT, and use {{{}OBJECT_APPEND{}}}, 
> {{{}OBJECT_DELETE{}}}, and {{{}OBJECT_PICK{}}}. The type is gated by 
> {{spark.sql.objectType.enabled}} (default false).
> h1. Q2. What problem is this proposal NOT designed to solve?
> Changing STRUCT. STRUCT keeps its positional semantics.
> Open, semi-structured objects with dynamic keys (Snowflake-style OBJECT). Use 
> VARIANT or MAP for those.
> ANSI SQL structured user-defined types ({{{}CREATE TYPE{}}}), inheritance, or 
> methods.
> A new on-disk format. OBJECT is stored like STRUCT; table metadata records 
> the difference.
> h1. Q3. How is it done today, and what are the limits of current practice?
> By default, STRUCT fields are matched by position for insert, assignment, and 
> comparison. Names are ignored and arity must match. Field types need only be 
> store-assignable for insert and assignment (for example, any atomic type into 
> STRING). For comparison they must have a common type.
> {{INSERT ... BY NAME}} is the existing mitigation. It matches top-level 
> columns and nested fields by name, and it is the clearest evidence of the 
> problem. It falls short in three ways:
> It is per statement. One INSERT without BY NAME silently reverts to 
> positional matching.
> The table owner cannot declare a column as name-safe.
> It covers INSERT and MERGE only. Comparison, CAST, UNION, CTAS, view schemas, 
> and function arguments remain positional.
> Other workarounds are weaker: rebuilding values with {{withField}} / 
> {{dropFields}} / manual projection (verbose, not enforced by the type 
> system), {{MAP<STRING, T>}} (one value type, no per-attribute types), and 
> VARIANT (semi-structured, different type system).
> h1. Q4. What is new in your approach and why do you think it will be 
> successful?
> Names become part of the type. The table owner declares name-safety once. 
> INSERT, comparison, CAST, and set operations follow it without a 
> per-statement clause. STRUCT and BY NAME remain available for existing code.
> A new first-class type, not a flag on STRUCT. CHAR and VARCHAR are StringType 
> subtypes that carry a length, and that required special handling across the 
> engine. OBJECT is its own type, so its rules do not depend on overlays.
> Reuse of STRUCT's machinery. Execution, codegen, and storage reuse STRUCT's 
> representation. New work concentrates on analysis and the type contract, 
> following the path used for TIMESTAMP_NTZ, ANSI intervals, and TIME.
> h1. Q5. Who cares? If you are successful, what difference will it make?
> Users who write nested data where field order changes over time and who today 
> rely on positional matching.
> Platforms and warehouses building on Spark SQL that need name-safe structured 
> values without moving to VARIANT.
> Users migrating from engines with structured object types.
> Success means new workloads choose OBJECT over STRUCT as the default for 
> structured records. STRUCT stays unchanged for positional use.
> h1. Q6. What are the risks?
> ||Risk||Mitigation||
> |Seen as duplicating BY NAME|BY NAME is per statement and DML only. OBJECT is 
> a type contract. See Q3 and Appendix C.|
> |Confused with Snowflake's semi-structured OBJECT|Describe it as a structured 
> object throughout. VARIANT and MAP remain the semi-structured types.|
> |Confused with the JVM {{ObjectType}} in Scala|The programmatic type is 
> {{{}SqlObjectType{}}}. {{ObjectType}} is unchanged.|
> |Implicit STRUCT/OBJECT coercion|None. An explicit CAST is required.|
> |Client or catalog gaps when enabled early|The feature flag defaults to 
> false. Default-on only after the Q8 exit criteria.|
> h1. Q7. How long will it take?
> About 3 months, reusing STRUCT's execution and storage. Rough split: base 
> type; compatibility and casts; functions; persistence and catalogs; clients; 
> docs and golden tests.
> h1. Q8. What are the mid-term and final "exams" to check for success?
> h2. Mid-term (about 6 weeks):
>  * DDL, constructor, field access, and equality work for OBJECT.
>  * Reordered attributes fail implicit assignment and comparison with a clear 
> error.
>  * Explicit reorder CAST and STRUCT <-> OBJECT casts work, with golden tests.
>  * {{{}OBJECT_APPEND{}}}, {{{}OBJECT_DELETE{}}}, and {{OBJECT_PICK}} work, 
> with golden tests.
>  * STRUCT workloads are unchanged when OBJECT is not used.
> h2. Final (about 3 months):
>  * Catalog persistence and file round-trip through the STRUCT physical layout.
>  * Spark Connect and PySpark can create, query, and collect OBJECT columns.
>  * The SQL reference documents the type, casts, and functions.
>  * Default-enablement criteria are agreed with the shepherd.
> h1. Appendix A. Proposed API
> {code:scala}
> case class SqlObjectType(fields: Seq[StructField]) extends DataType
> {code}
> SQL syntax: {{{}OBJECT<firstname STRING, lastname STRING>{}}}. Constructor: 
> {{{}object('firstname', 'Ada', 'lastname', 'Lovelace'){}}}.
> Feature flag: {{spark.sql.objectType.enabled}} (default false).
> h1. Appendix B. Language surface
> ||Area||Intent||
> |Type|Ordered, uniquely named attributes; nested types allowed; {{OBJECT<>}} 
> allowed|
> |Assignment and comparison|Names must match at each position; no implicit 
> reordering|
> |Casts|Explicit by-name remap when name sets match; {{ADD FIELDS}} and 
> {{RENAME FIELDS}} modifiers; explicit STRUCT <-> OBJECT|
> |Functions|{{OBJECT_APPEND(o, k, v [, updateFlag])}} (an update preserves the 
> attribute type), {{{}OBJECT_DELETE{}}}, {{OBJECT_PICK}}|
> |Storage|STRUCT physical layout; OBJECT vs STRUCT recorded in table metadata|
> Detailed rules, error classes, and the full test matrix belong in the design 
> document.
> h1. Appendix C. Rejected designs
>  * INSERT BY NAME only. Keep STRUCT and rely on a per-statement BY NAME 
> clause. Rejected: the default stays positional, the table owner cannot 
> declare name-safety, and comparison, CAST, and set operations stay 
> positional. See Q3.
>  * Metadata flag on StructType (CHAR/VARCHAR style). Rejected: an overlay 
> leaks into every rule that matches on the type, and CHAR/VARCHAR show the 
> cost. A separate DataType keeps the semantics isolated.
>  * VARIANT or MAP. Rejected: MAP requires one value type, and VARIANT is open 
> and untyped. Neither gives a closed, typed attribute list with STRUCT-like 
> execution.
>  * Open, Snowflake-style OBJECT. Rejected: VARIANT already covers open 
> documents. The gap is a static, name-checked product type.
>  * Implicit order-independent assignment. Rejected: implicit reordering makes 
> the result of inserts and unions depend on hidden remapping, and it is 
> invisible in review. An explicit CAST keeps the reorder visible.
>  * Positional {{{}OBJECT_INSERT(o, pos, k, v){}}}. Rejected: attributes are 
> identified by name, not by ordinal position.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to