Serge Rielau created SPARK-61931:
------------------------------------
Summary: SPIP: Add the OBJECT data type
Key: SPARK-61931
URL: https://issues.apache.org/jira/browse/SPARK-61931
Project: Spark
Issue Type: New Feature
Components: SQL
Affects Versions: 5.0.0
Reporter: Serge Rielau
h2. Q1. What are you trying to do? Articulate your objectives using absolutely
no jargon.
Add a new Spark SQL data type, {*}OBJECT{*}, for {*}structured objects{*}:
named groups of typed attributes where {*}attribute names are part of type
compatibility{*}.
Spark's *STRUCT* is also a named group of fields, but compatibility is
{*}positional{*}. Names are ignored for insert, assignment, and comparison;
field types need only be store-assignable. Under ANSI, this succeeds:
{{CREATE TABLE employees (e STRUCT<name STRING, title STRING>) USING parquet;
INSERT INTO employees
SELECT named_struct('salary', 150000, 'is_contractor', true);
-- stored as: e.name = '150000', e.title = 'true'}}
The names are unrelated, the types differ ({{{}INT{}}} / {{BOOLEAN}} into
{{STRING}} / {{{}STRING{}}}), and the insert still succeeds. The same rule
silently swaps values when names are only reordered:
{{CREATE TABLE pts (s STRUCT<x INT, y INT>) USING parquet;
INSERT INTO pts SELECT named_struct('y', 20, 'x', 10);
-- stored as: s.x = 20, s.y = 10}}
*OBJECT* keeps STRUCT's useful shape (ordered, typed attributes; field access;
nesting) but makes names matter: attributes must align by name for assignment,
comparison, and implicit coercion. Reordering requires an explicit {{CAST}}
that remaps by name. Attribute names within a structured object are unique.
{{CAST(object('firstname', 'Ada', 'lastname', 'Lovelace')
AS OBJECT<lastname STRING, firstname STRING>);
-- \{lastname: Lovelace, firstname: Ada}
object('firstname', 'Ada', 'lastname', 'Lovelace')
= object('lastname', 'Lovelace', 'firstname', 'Ada');
-- analysis error}}
Users can declare OBJECT columns, construct values, access attributes, cast
between OBJECT and STRUCT, and use functions to append, delete, and pick
attributes. The type is gated behind {{spark.sql.objectType.enabled}} (default
false) until coverage is ready.
h2. Q2. What problem is this proposal NOT designed to solve?
This proposal does {*}not{*}:
* Replace or change STRUCT behavior.
* Add Snowflake-style *semi-structured* OBJECT (open key sets /
schema-on-read). Use VARIANT or MAP for that.
* Add ANSI SQL *CREATE TYPE* structured user-defined types, inheritance, or
methods.
* Invent a new on-disk binary format. OBJECT is stored physically like STRUCT;
table metadata distinguishes OBJECT from STRUCT.
h2. Q3. How is it done today, and what are the limits of current practice?
Structured records in Spark SQL are modeled with {*}STRUCT{*}. For nested
insert, assignment, {*}and comparison{*}, Spark resolves struct fields {*}by
position{*}: names are not part of the compatibility check, and field types
need only be ANSI store-assignable (for example any atomic type into
{{{}STRING{}}}). Arity must match; names and exact types need not.
Workarounds when name-safe structured values are needed:
||Workaround||Limitation||
|Rely on STRUCT and careful field ordering|Silent mis-assignment or misleading
comparisons when names and positions disagree|
|Rebuild with {{withField}} / {{dropFields}} / manual projection|Verbose; easy
to get wrong; not enforced by the type system|
|{{MAP<STRING, T>}} for a single value type {{T}}|Homogeneous values only;
loses per-attribute types and a fixed attribute set|
|VARIANT|Semi-structured; different type system and tooling|
There is no first-class Spark SQL type that is STRUCT-shaped but *name-checked*
for assignment and comparison.
h2. Q4. What is new in your approach and why do you think it will be successful?
OBJECT fixes STRUCT's positional hazard while keeping what makes STRUCT useful:
a closed, ordered, typed attribute list; ordinary field access; nesting; and
the same physical execution shape.
It is introduced as a {*}new first-class data type{*}, not as a soft overlay on
STRUCT. That is deliberate: the CHAR/VARCHAR approach of specializing
{{StringType}} without a fully separate type has proven spotty across the
stack. OBJECT follows the playbook used for types such as TIMESTAMP_NTZ, ANSI
intervals, and TIME: its own type in Catalyst, explicit analysis rules, a
feature flag, and end-to-end coverage (SQL, catalogs, clients).
Because execution, codegen, and file layout can reuse STRUCT's representation,
the incremental cost is concentrated where the value is: name-checked
compatibility and a clear SQL contract for structured objects.
h2. Q5. Who cares? If you are successful, what difference will it make?
* *SQL users and pipeline authors* who want structured columns where attribute
names are part of the contract (APIs, nested payloads, migrations where field
order is unstable).
* *Platforms and warehouses building on Spark SQL* that need a stronger
structured type than positional STRUCT without moving to VARIANT.
* *Existing Spark users* who today rebuild structs by hand to avoid positional
surprises.
Success means new workloads prefer *OBJECT* over *STRUCT* as the safer default
for structured objects, while STRUCT remains available and unchanged for
positional use cases and existing code.
h2. Q6. What are the risks?
||Risk||Mitigation||
|Confusion with Snowflake / semi-structured OBJECT|Stress throughout docs and
naming that Spark OBJECT is a {*}structured object{*}: a closed, typed
attribute set. VARIANT and MAP remain the semi-structured options.|
|Confusion with JVM {{ObjectType}} in the Scala API|SQL type name remains
OBJECT; programmatic type uses a distinct name ({{{}SqlObjectType{}}}) so
existing {{ObjectType}} (JVM pass-through) is untouched|
|Accidental STRUCT/OBJECT coercion|No implicit STRUCT ↔ OBJECT coercion;
require explicit CAST|
|Incomplete client / catalog coverage if enabled too early|Feature flag default
false; enable-by-default only after Q8 exit criteria|
h2. Q7. How long will it take?
About {*}3 months{*}, leaning on STRUCT for execution and storage.
Suggested split (JIRA sub-tasks):
# *Base type* — {{{}SqlObjectType{}}}, parser/DDL, constructor, field access,
typeof/describe, feature flag.
# *Compatibility and casts* — name-checked assignability and comparison;
OBJECT↔OBJECT reorder cast; OBJECT↔STRUCT; ADD FIELDS / RENAME FIELDS cast
modifiers.
# *Functions* — {{{}OBJECT_APPEND{}}}, {{{}OBJECT_DELETE{}}},
{{{}OBJECT_PICK{}}}.
# *Persistence and catalogs* — table metadata distinction; file formats via
struct physical type.
# *Clients* — Spark Connect, JDBC/Hive/Thrift mapping, PySpark/DataFrame API.
# *Docs and golden SQL tests*
h2. Q8. What are the mid-term and final "exams" to check for success?
*Mid-term (~6 weeks):*
* OBJECT usable in SQL: DDL, constructor, field access, equality, name-checked
assignment failures for reordered attributes.
* Explicit CAST can reorder by name; STRUCT ↔ OBJECT round-trip with
documented rules.
* Core functions ({{{}OBJECT_APPEND{}}} / {{OBJECT_DELETE}} /
{{{}OBJECT_PICK{}}}) with golden tests.
* No behavioral change to STRUCT workloads when OBJECT is unused.
*Final exam (~3 months):*
* Tables and views preserve OBJECT in catalog metadata; file round-trip via
struct physical encoding.
* Connect and PySpark can create, query, and collect OBJECT columns.
* Documented SQL reference for the type, casts, and functions.
* Exit criteria for considering default enablement: agreed surface implemented
and tested; shepherd sign-off that partial support will not be enabled by
default.
----
h2. Appendix A. Proposed API sketch
Programmatic type in {{org.apache.spark.sql.types}} (name avoids clashing with
existing JVM {{{}ObjectType{}}}):
{{/**
* SQL OBJECT type: a structured object — an ordered product of uniquely
* named attributes. Compatibility is name-checked (names must align by
* position); reordering requires an explicit cast.
*
* SQL / DDL / typeof print as, e.g., object<a INT, b STRING>.
*/
case class SqlObjectType(fields: Array[StructField]) extends DataType \{
override def defaultSize: Int = fields.map(_.dataType.defaultSize).sum
override def asNullable: SqlObjectType = /* fields as nullable */
}}}
SQL syntax:
{{OBJECT<firstname STRING, lastname STRING>
object('firstname', 'Ada', 'lastname', 'Lovelace')}}
Feature flag: {{spark.sql.objectType.enabled}} (default {{{}false{}}}).
----
h2. Appendix B. Language surface (summary only)
||Area||Intent||
|Type|Structured object: closed, ordered attributes; unique names; nested types
allowed; empty {{OBJECT<>}} allowed|
|Compatibility|Name-checked positional assignability and comparison; no
implicit reorder|
|Cast|By-name remap when name sets match; {{ADD FIELDS}} / {{RENAME FIELDS}}
modifiers; explicit STRUCT ↔ OBJECT|
|Functions|{{OBJECT_APPEND(o, k, v [, updateFlag])}} (default updateFlag false;
updates preserve attr type), {{{}OBJECT_DELETE{}}}, {{OBJECT_PICK}}|
|Storage|Physical STRUCT layout; OBJECT vs STRUCT distinguished in table
metadata|
Detailed rules, error classes, and the full SQL test matrix are left to design
docs and implementation reviews so this SPIP stays focused on approval of the
type.
----
h2. Appendix C. Rejected Designs
# *Overlay / Metadata flag on* {{StructType}} *(the CHAR/VARCHAR approach):*
** _Alternative:_ Treat OBJECT as a {{StructType}} carrying a schema metadata
flag or type-tag rather than a separate {{{}DataType{}}}.
** _Why rejected:_ Spark's experience with
{{{}CharType{}}}/{{{}VarcharType{}}} as annotations on {{StringType}} showed
that metadata and type-tag overlays are leak-prone across Catalyst optimizer
rules, expression resolution, codegen, UDF boundaries, and external data
connectors. A distinct first-class {{DataType}} ({{{}SqlObjectType{}}})
provides clean pattern matching, strict isolation, and prevents silent decay
into positional STRUCT semantics.
# *Reusing* {{VARIANT}} *or* {{MAP}} *for structured records:*
** _Alternative:_ Advise users needing name-safe structured fields to use
{{VARIANT}} or {{{}MAP<STRING, T>{}}}.
** _Why rejected:_ {{MAP}} requires homogeneous value types and loses schema
enforcement; {{VARIANT}} is designed for open, semi-structured documents
without static compile-time schemas, rigid per-attribute types, or positional
physical layout. Structured objects require static schema checking, strong
types per attribute, and high-throughput columnar storage/execution identical
to {{{}STRUCT{}}}.
# *Snowflake-style semi-structured* {{OBJECT}} *(open schema-on-read):*
** _Alternative:_ Implement an open, schemaless key-value object type where
attributes are untyped variants.
** _Why rejected:_ Spark's {{VARIANT}} already serves the semi-structured
document use case with high performance. What users lack is a _safe, static,
schema-enforced_ product type where field names protect against positional
insertion accidents.
# {*}Implicit order-independent assignment ({*}{{{}OBJECT<a, b>{}}}
*assignable to* {{OBJECT<b, a>}} *without CAST):*
** _Alternative:_ Allow automatic by-name matching during assignment/insert
even when attribute order differs.
** _Why rejected:_ Creates subtle ambiguities when projecting or unnesting
(e.g. {{o.*}} attribute order becoming non-deterministic or inconsistent with
table definitions). Requiring explicit {{CAST(... AS OBJECT<...>)}} makes
attribute reordering intentional, self-documenting, and visible in code
reviews, while keeping assignment predictable.
# {*}Positional attribute insertion ({*}{{{}OBJECT_INSERT(o, pos, k,
v){}}}{*}):{*}
** _Alternative:_ Model object insertion after {{array_insert}} taking an
integer ordinal offset.
** _Why rejected:_ Users think of object attributes by their logical names,
not by ordinal array indices. Modeling attribute insertion with integer offsets
adds needless complexity and does not match standard SQL/warehouse object
manipulation habits.
----
h2. Appendix D. Naming and Terminology
* *SQL name:* {{OBJECT}} (a structured object, not a semi-structured document).
* *Scala/Java name:* {{SqlObjectType}} (existing {{ObjectType}} in
{{org.apache.spark.sql.types}} remains the JVM expression-pass-through type).
* *Not* Snowflake semi-structured OBJECT; *not* ANSI structured UDTs.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]