Serge Rielau created SPARK-61931:
------------------------------------

             Summary: SPIP: Add the OBJECT data type
                 Key: SPARK-61931
                 URL: https://issues.apache.org/jira/browse/SPARK-61931
             Project: Spark
          Issue Type: New Feature
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Serge Rielau


h2. Q1. What are you trying to do? Articulate your objectives using absolutely 
no jargon.

Add a new Spark SQL data type, {*}OBJECT{*}, for {*}structured objects{*}: 
named groups of typed attributes where {*}attribute names are part of type 
compatibility{*}.

Spark's *STRUCT* is also a named group of fields, but compatibility is 
{*}positional{*}. Names are ignored for insert, assignment, and comparison; 
field types need only be store-assignable. Under ANSI, this succeeds:

{{CREATE TABLE employees (e STRUCT<name STRING, title STRING>) USING parquet;
INSERT INTO employees
SELECT named_struct('salary', 150000, 'is_contractor', true);
-- stored as: e.name = '150000', e.title = 'true'}}

The names are unrelated, the types differ ({{{}INT{}}} / {{BOOLEAN}} into 
{{STRING}} / {{{}STRING{}}}), and the insert still succeeds. The same rule 
silently swaps values when names are only reordered:

{{CREATE TABLE pts (s STRUCT<x INT, y INT>) USING parquet;
INSERT INTO pts SELECT named_struct('y', 20, 'x', 10);
-- stored as: s.x = 20, s.y = 10}}

*OBJECT* keeps STRUCT's useful shape (ordered, typed attributes; field access; 
nesting) but makes names matter: attributes must align by name for assignment, 
comparison, and implicit coercion. Reordering requires an explicit {{CAST}} 
that remaps by name. Attribute names within a structured object are unique.

{{CAST(object('firstname', 'Ada', 'lastname', 'Lovelace')
     AS OBJECT<lastname STRING, firstname STRING>);
-- \{lastname: Lovelace, firstname: Ada}

object('firstname', 'Ada', 'lastname', 'Lovelace')
  = object('lastname', 'Lovelace', 'firstname', 'Ada');
-- analysis error}}

Users can declare OBJECT columns, construct values, access attributes, cast 
between OBJECT and STRUCT, and use functions to append, delete, and pick 
attributes. The type is gated behind {{spark.sql.objectType.enabled}} (default 
false) until coverage is ready.
h2. Q2. What problem is this proposal NOT designed to solve?

This proposal does {*}not{*}:
 * Replace or change STRUCT behavior.

 * Add Snowflake-style *semi-structured* OBJECT (open key sets / 
schema-on-read). Use VARIANT or MAP for that.

 * Add ANSI SQL *CREATE TYPE* structured user-defined types, inheritance, or 
methods.

 * Invent a new on-disk binary format. OBJECT is stored physically like STRUCT; 
table metadata distinguishes OBJECT from STRUCT.

h2. Q3. How is it done today, and what are the limits of current practice?

Structured records in Spark SQL are modeled with {*}STRUCT{*}. For nested 
insert, assignment, {*}and comparison{*}, Spark resolves struct fields {*}by 
position{*}: names are not part of the compatibility check, and field types 
need only be ANSI store-assignable (for example any atomic type into 
{{{}STRING{}}}). Arity must match; names and exact types need not.

Workarounds when name-safe structured values are needed:
||Workaround||Limitation||
|Rely on STRUCT and careful field ordering|Silent mis-assignment or misleading 
comparisons when names and positions disagree|
|Rebuild with {{withField}} / {{dropFields}} / manual projection|Verbose; easy 
to get wrong; not enforced by the type system|
|{{MAP<STRING, T>}} for a single value type {{T}}|Homogeneous values only; 
loses per-attribute types and a fixed attribute set|
|VARIANT|Semi-structured; different type system and tooling|

There is no first-class Spark SQL type that is STRUCT-shaped but *name-checked* 
for assignment and comparison.
h2. Q4. What is new in your approach and why do you think it will be successful?

OBJECT fixes STRUCT's positional hazard while keeping what makes STRUCT useful: 
a closed, ordered, typed attribute list; ordinary field access; nesting; and 
the same physical execution shape.

It is introduced as a {*}new first-class data type{*}, not as a soft overlay on 
STRUCT. That is deliberate: the CHAR/VARCHAR approach of specializing 
{{StringType}} without a fully separate type has proven spotty across the 
stack. OBJECT follows the playbook used for types such as TIMESTAMP_NTZ, ANSI 
intervals, and TIME: its own type in Catalyst, explicit analysis rules, a 
feature flag, and end-to-end coverage (SQL, catalogs, clients).

Because execution, codegen, and file layout can reuse STRUCT's representation, 
the incremental cost is concentrated where the value is: name-checked 
compatibility and a clear SQL contract for structured objects.
h2. Q5. Who cares? If you are successful, what difference will it make?
 * *SQL users and pipeline authors* who want structured columns where attribute 
names are part of the contract (APIs, nested payloads, migrations where field 
order is unstable).

 * *Platforms and warehouses building on Spark SQL* that need a stronger 
structured type than positional STRUCT without moving to VARIANT.

 * *Existing Spark users* who today rebuild structs by hand to avoid positional 
surprises.

Success means new workloads prefer *OBJECT* over *STRUCT* as the safer default 
for structured objects, while STRUCT remains available and unchanged for 
positional use cases and existing code.
h2. Q6. What are the risks?
||Risk||Mitigation||
|Confusion with Snowflake / semi-structured OBJECT|Stress throughout docs and 
naming that Spark OBJECT is a {*}structured object{*}: a closed, typed 
attribute set. VARIANT and MAP remain the semi-structured options.|
|Confusion with JVM {{ObjectType}} in the Scala API|SQL type name remains 
OBJECT; programmatic type uses a distinct name ({{{}SqlObjectType{}}}) so 
existing {{ObjectType}} (JVM pass-through) is untouched|
|Accidental STRUCT/OBJECT coercion|No implicit STRUCT ↔ OBJECT coercion; 
require explicit CAST|
|Incomplete client / catalog coverage if enabled too early|Feature flag default 
false; enable-by-default only after Q8 exit criteria|
h2. Q7. How long will it take?

About {*}3 months{*}, leaning on STRUCT for execution and storage.

Suggested split (JIRA sub-tasks):
 # *Base type* — {{{}SqlObjectType{}}}, parser/DDL, constructor, field access, 
typeof/describe, feature flag.

 # *Compatibility and casts* — name-checked assignability and comparison; 
OBJECT↔OBJECT reorder cast; OBJECT↔STRUCT; ADD FIELDS / RENAME FIELDS cast 
modifiers.

 # *Functions* — {{{}OBJECT_APPEND{}}}, {{{}OBJECT_DELETE{}}}, 
{{{}OBJECT_PICK{}}}.

 # *Persistence and catalogs* — table metadata distinction; file formats via 
struct physical type.

 # *Clients* — Spark Connect, JDBC/Hive/Thrift mapping, PySpark/DataFrame API.

 # *Docs and golden SQL tests*

h2. Q8. What are the mid-term and final "exams" to check for success?

*Mid-term (~6 weeks):*
 * OBJECT usable in SQL: DDL, constructor, field access, equality, name-checked 
assignment failures for reordered attributes.

 * Explicit CAST can reorder by name; STRUCT ↔ OBJECT round-trip with 
documented rules.

 * Core functions ({{{}OBJECT_APPEND{}}} / {{OBJECT_DELETE}} / 
{{{}OBJECT_PICK{}}}) with golden tests.

 * No behavioral change to STRUCT workloads when OBJECT is unused.

*Final exam (~3 months):*
 * Tables and views preserve OBJECT in catalog metadata; file round-trip via 
struct physical encoding.

 * Connect and PySpark can create, query, and collect OBJECT columns.

 * Documented SQL reference for the type, casts, and functions.

 * Exit criteria for considering default enablement: agreed surface implemented 
and tested; shepherd sign-off that partial support will not be enabled by 
default.

----
h2. Appendix A. Proposed API sketch

Programmatic type in {{org.apache.spark.sql.types}} (name avoids clashing with 
existing JVM {{{}ObjectType{}}}):

{{/**
 * SQL OBJECT type: a structured object — an ordered product of uniquely
 * named attributes. Compatibility is name-checked (names must align by
 * position); reordering requires an explicit cast.
 *
 * SQL / DDL / typeof print as, e.g., object<a INT, b STRING>.
 */
case class SqlObjectType(fields: Array[StructField]) extends DataType \{
  override def defaultSize: Int = fields.map(_.dataType.defaultSize).sum
  override def asNullable: SqlObjectType = /* fields as nullable */
}}}

SQL syntax:

{{OBJECT<firstname STRING, lastname STRING>
object('firstname', 'Ada', 'lastname', 'Lovelace')}}

Feature flag: {{spark.sql.objectType.enabled}} (default {{{}false{}}}).
----
h2. Appendix B. Language surface (summary only)
||Area||Intent||
|Type|Structured object: closed, ordered attributes; unique names; nested types 
allowed; empty {{OBJECT<>}} allowed|
|Compatibility|Name-checked positional assignability and comparison; no 
implicit reorder|
|Cast|By-name remap when name sets match; {{ADD FIELDS}} / {{RENAME FIELDS}} 
modifiers; explicit STRUCT ↔ OBJECT|
|Functions|{{OBJECT_APPEND(o, k, v [, updateFlag])}} (default updateFlag false; 
updates preserve attr type), {{{}OBJECT_DELETE{}}}, {{OBJECT_PICK}}|
|Storage|Physical STRUCT layout; OBJECT vs STRUCT distinguished in table 
metadata|

Detailed rules, error classes, and the full SQL test matrix are left to design 
docs and implementation reviews so this SPIP stays focused on approval of the 
type.
----
h2. Appendix C. Rejected Designs
 # *Overlay / Metadata flag on* {{StructType}} *(the CHAR/VARCHAR approach):*

 ** _Alternative:_ Treat OBJECT as a {{StructType}} carrying a schema metadata 
flag or type-tag rather than a separate {{{}DataType{}}}.

 ** _Why rejected:_ Spark's experience with 
{{{}CharType{}}}/{{{}VarcharType{}}} as annotations on {{StringType}} showed 
that metadata and type-tag overlays are leak-prone across Catalyst optimizer 
rules, expression resolution, codegen, UDF boundaries, and external data 
connectors. A distinct first-class {{DataType}} ({{{}SqlObjectType{}}}) 
provides clean pattern matching, strict isolation, and prevents silent decay 
into positional STRUCT semantics.

 # *Reusing* {{VARIANT}} *or* {{MAP}} *for structured records:*

 ** _Alternative:_ Advise users needing name-safe structured fields to use 
{{VARIANT}} or {{{}MAP<STRING, T>{}}}.

 ** _Why rejected:_ {{MAP}} requires homogeneous value types and loses schema 
enforcement; {{VARIANT}} is designed for open, semi-structured documents 
without static compile-time schemas, rigid per-attribute types, or positional 
physical layout. Structured objects require static schema checking, strong 
types per attribute, and high-throughput columnar storage/execution identical 
to {{{}STRUCT{}}}.

 # *Snowflake-style semi-structured* {{OBJECT}} *(open schema-on-read):*

 ** _Alternative:_ Implement an open, schemaless key-value object type where 
attributes are untyped variants.

 ** _Why rejected:_ Spark's {{VARIANT}} already serves the semi-structured 
document use case with high performance. What users lack is a _safe, static, 
schema-enforced_ product type where field names protect against positional 
insertion accidents.

 # {*}Implicit order-independent assignment ({*}{{{}OBJECT<a, b>{}}} 
*assignable to* {{OBJECT<b, a>}} *without CAST):*

 ** _Alternative:_ Allow automatic by-name matching during assignment/insert 
even when attribute order differs.

 ** _Why rejected:_ Creates subtle ambiguities when projecting or unnesting 
(e.g. {{o.*}} attribute order becoming non-deterministic or inconsistent with 
table definitions). Requiring explicit {{CAST(... AS OBJECT<...>)}} makes 
attribute reordering intentional, self-documenting, and visible in code 
reviews, while keeping assignment predictable.

 # {*}Positional attribute insertion ({*}{{{}OBJECT_INSERT(o, pos, k, 
v){}}}{*}):{*}

 ** _Alternative:_ Model object insertion after {{array_insert}} taking an 
integer ordinal offset.

 ** _Why rejected:_ Users think of object attributes by their logical names, 
not by ordinal array indices. Modeling attribute insertion with integer offsets 
adds needless complexity and does not match standard SQL/warehouse object 
manipulation habits.

----
h2. Appendix D. Naming and Terminology
 * *SQL name:* {{OBJECT}} (a structured object, not a semi-structured document).

 * *Scala/Java name:* {{SqlObjectType}} (existing {{ObjectType}} in 
{{org.apache.spark.sql.types}} remains the JVM expression-pass-through type).

 * *Not* Snowflake semi-structured OBJECT; *not* ANSI structured UDTs.

 



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to