[
https://issues.apache.org/jira/browse/SPARK-58820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uroš Bojanić updated SPARK-58820:
---------------------------------
Description:
*Q1. What are you trying to do? Articulate your objectives using absolutely no
jargon.*
Add a new Spark SQL data type *DECFLOAT* that stores decimal numbers with a
flexible decimal point (floating point decimal numbers). Each value keeps up to
a fixed number of significant decimal digits and its own exponent, so one
column can hold both very large integers and very small fractions without
picking a single fixed scale for the whole column.
The type follows the IEEE 754 decimal floating-point formats:
||*SQL type*||*IEEE format*||*Significant digits*||*Storage width*||
|DECFLOAT(16)|decimal64|16|8 bytes|
|DECFLOAT(34)|decimal128|34|16 bytes|
Bare DECFLOAT means DECFLOAT(34).
Unlike DECIMAL(p,s), there is no column-wide scale. Unlike FLOAT / DOUBLE,
arithmetic is done in base 10, so values such as 0.1 are exact.
Because each value carries at most 16 or 34 significant digits, a result
needing more digits is rounded to the format's precision. Rounding is decimal
(base 10), unlike binary rounding in FLOAT/DOUBLE. The v1 fixed default is IEEE
roundTiesToEven / HALF_EVEN (banker's rounding), which differs from Spark's
existing DECIMAL arithmetic (rounds HALF_UP). This can be configurable in
future iterations.
The type also supports IEEE special values: signed zero, +/-Infinity, and quiet
NaN.
*Q2. What problem is this proposal NOT designed to solve?*
* *Arbitrary-precision / unbounded decimals.* Extending Spark's fixed-point
DECIMAL beyond 38 digits, or adding a PostgreSQL-style unbounded NUMERIC, is
out of scope. Those needs are different from IEEE decimal floating point.
* *Wider-than-IEEE formats in v1.* Formats such as a 70-digit / 256-bit
decimal float (sometimes discussed as DECFLOAT(70) / decimal256) are out of
scope for the first delivery. The type and storage designs should not preclude
them later.
* *Replacing* DECIMAL *or* DOUBLE{*}.{*} Existing fixed-point and binary-float
types remain unchanged. DECFLOAT is additive.
* *Non-IEEE decimal-float dialects as the native type.* Engines such as
Snowflake, Oracle, and Teradata expose a decimal float that is not a strict
IEEE width (e.g. 38 digits, no NaN/Inf). Spark's native type targets IEEE 754
decimal64 and decimal128. Mapping those dialects into Spark is a connector /
cast concern, not a second native type.
* *ORC / CSV / JSON as first-class DECFLOAT storage in v1.* The intended
persistence target is Parquet (and table formats that sit on Parquet), via the
standardized logical type (Appendix D). Other formats may round-trip via
existing types (e.g. string or binary) until separately specified.
*Q3. How is it done today, and what are the limits of current practice?*
Spark SQL today offers two numeric families for real numbers:
# DECIMAL(p,s) *(fixed-point).* Precision and scale are fixed for the column.
Spark caps precision at 38. Mixed magnitudes force a tradeoff: many fractional
digits leave few integer digits (and the reverse). Chains of arithmetic expand
precision quickly and often require manual casts.
# FLOAT */* DOUBLE *(binary floating-point).* Wide range, but many common
decimal fractions are not exact. About 15-16 significant decimal digits for
DOUBLE, with binary rounding error.
Workarounds in use today:
||*Workaround*||*Limitation*||
|Map high-precision source decimals to DECIMAL(38,s)|Overflow / out-of-range
errors for large integers; silent rounding of long fractions|
|Cast to DOUBLE|Binary rounding; unsuitable when exact decimal fractions matter|
|Store as STRING|Loses numeric semantics; arithmetic and aggregation must be
rebuilt|
|Emulate with (unscaled DECIMAL, scale INT) structs|Not a first-class type;
poor pushdown, stats, and ecosystem support|
Spark also cannot recognize IEEE decimal floating-point values in Parquet (or
other built-in file sources) as a distinct logical type today, so even when
another system wrote such values, Spark cannot load them as decimals with
per-value exponents.
*Q4. What is new in your approach and why do you think it will be successful?*
The approach is deliberately conventional and layered, following prior Spark
type introductions (TIMESTAMP_NTZ, ANSI intervals, TIME):
# *Standard semantics first.* Align the SQL type with IEEE 754-2019 decimal64
/ decimal128 and with existing SQL DECFLOAT practice (DB2, Firebird, MongoDB),
rather than inventing a Spark-only numeric model. Rounding follows the IEEE
default (roundTiesToEven / HALF_EVEN, as in libbid/DB2). Language-neutral IEEE
contract, shared conformance vectors, testable across implementations.
# *Concrete physical representation.* Internal to engine, represent DECFLOAT
in memory as fixed-width IEEE binary integer decimal (BID) bit patterns
(Appendix A) - computed by a single pure-Java in-tree arithmetic path (POC:
SPARK-59111) with no native/JNI dependency as part of this SPIP. The prototype
ports and validates against libbid conformance vectors; whereas the technicals
of arithmetic kernel are implementation detail to be finalized during actual
code review. Note that the on-disk Parquet encoding is a separate, deferred
decision (Appendix D), independent of this in-memory layout / representation
(e.g. similar to Timestamp nano). This avoids a logical-only typedef over
DECIMAL / STRING that would diverge across implementations.
# *Open storage path.* Collaborate with the
[Parquet|https://docs.google.com/document/d/104397AVUqg_JSlzGBdpa3D98X6Dd-RONIgABn3omBcw/edit?tab=t.0]
community on a standardized decimal-float logical type; the actual physical
representation will be discussed and decided there, and Spark ships no private
on-disk encoding in the meantime (Appendix D).
# *Additive rollout.* Gate the type behind a config during incubation, and
define measurable exit criteria for enabling it by default (Q8) once coverage
matches peer numeric types.
We expect this to succeed because:
* The user-visible type matches what migrators already know from DB2 /
Firebird / MongoDB Decimal128, and closes a clear gap versus warehouses that
already ship a decimal-float type.
* The encoding and the BID arithmetic (compliant with IEEE 754) are
industry-proven, rather than bespoke.
* Spark already has a playbook for introducing a new fractional type
end-to-end (parser → Catalyst → execution → datasources → Connect / PySpark /
JDBC).
*Q5. Who cares? If you are successful, what difference will it make?*
* *Users migrating from PostgreSQL, Oracle, DB2, BigQuery, and similar
systems* who today hit Spark's DECIMAL(38,*) ceiling or lose precision via
DOUBLE / STRING workarounds. Unconstrained or high-precision source numerics
map cleanly to a first-class numeric type.
* *Financial, actuarial, and crypto / fintech workloads* that need exact
decimal fractions and mixed magnitudes in one column (rates, notionals,
micro-quantities and large balances together).
* *Existing Spark users* who need to read or write Parquet datasets produced
by systems that already use IEEE decimal128 / DECFLOAT.
* *The wider storage ecosystem* (Parquet, and eventually Iceberg / Delta
consumers) gains a portable decimal-float logical type rather than
engine-private encodings.
Success means: declare DECFLOAT columns, run SQL arithmetic and aggregations
without leaving the numeric domain, and once the Parquet logical type is
standardized - round-trip values through Parquet with standardized semantics /
encoding.
*Q6. What are the risks?*
||*Risk*||*Mitigation*||
|Parquet / table-format standardization lagging Spark SQL|Land the in-memory
type, compute, and clients first in Spark; make Parquet persistence conditional
on the standardized logical type; ship no private on-disk encoding in the
meantime (Appendix D).|
|Surprising type coercion (DECFLOAT outranking DOUBLE / interacting with
DECIMAL)|Document precedence explicitly; add golden SQL tests; match IEEE / SQL
expectations rather than silent demotion to binary float.|
|Special-value semantics (NaN, Inf) surprising users coming from engines
without them (e.g. some warehouse DECFLOAT dialects)|Document clearly;
DECFLOAT's Inf/NaN mirror DOUBLE's, which Spark already supports.|
|External API choice for Inf/NaN (Java BigDecimal cannot represent
them)|Resolved in Appendix B - Spark-owned JVM value class, decimal.Decimal
(Python), Arrow extension type, DECFLOAT JDBC mapping.|
|Scope creep into arbitrary precision or DECFLOAT(70)|Keep v1 strictly at
precisions 16 and 34; record wider formats as follow-ons.|
|Implementation overhead and ensuring full behavioural consistency in a custom
JVM implementation versus existing libraries|Use a pure Java, in-tree BID
implementation ported from libbid and validated against its conformance vectors
(SPARK-59111), providing reference-grade IEEE correctness without any
native/JNI dependencies.|
*Q7. How long will it take?*
Rough estimate: *on the order of 12-15 months* for feature-complete parity with
peer numeric types. DECFLOAT is larger in scope than prior single-type efforts
(it adds arithmetic kernels and a special-value domain), so TIMESTAMP_NTZ
(SPARK-35662) and ANSI intervals (SPARK-27790) are a lower bound rather than a
direct model. The estimate is baselined against the v1 scope in Appendix C.
Suggested work split (can become JIRA sub-tasks):
# *Base type: ~1 month* DecFloatType, parser / DDL, literals (DECFLOAT '...',
optional DF suffix), casts to/from string and numeric types, etc.
# *Arithmetic and comparison kernels: ~2-3 months* + - * /, unary minus,
comparisons, ordering and equality following Spark's DOUBLE conventions (-0.0 =
0.0, NaN = NaN, NaN sorts last), hashing / grouping-equality canonicalization,
codegen / interpreted execution.
# *Functions and aggregates: ~2 months* Core scalars (abs, sign, floor, ceil,
round, sqrt, isnan, …), DECFLOAT-specific helpers (quantize, same_quantum,
total_order), sum / avg / min / max / count, window variants.
# *Persistence: ~2-3 months* Parquet read/write integration - with logical
annotation, partition values, stats / predicate pushdown, caching / shuffle;
gated on the Parquet RFC (Appendix D).
# *Clients: ~1.5 months* Spark Connect proto, JDBC / Thrift / Hive result
mapping, catalog / information_schema.
# *PySpark / Arrow: ~1.5 months* DataFrame API, pandas / Arrow interchange,
Python UDFs.
# *Docs, golden tests, benchmarks: ~1 month (overlaps)*
Note for delivery: the in-tree BID arithmetic library (POC: SPARK-59111 /
[#58410|https://github.com/apache/spark/pull/58410]) will land together with
its first consumer - the base type and its Catalyst wiring; to be reviewed &
exercised as part of the feature, not as a standalone module. Also, Arrow will
require a parallel extension type for transport (Appendix B).
*Q8. What are the mid-term and final "exams" to check for success?*
*Mid-term (~6-8 months):*
* DecFloatType(16|34) usable in SQL: literals, DDL, casts, arithmetic,
comparisons, basic aggregates.
* Correct IEEE behavior for a representative set of finite values, signed
zeros, Inf, and NaN, and rounding (especially at the precision boundary, e.g.
results exceeding 16/34 digits, ties).
* Round-trip DECFLOAT through at least one client path (e.g. Spark Connect /
Arrow collect); no on-disk file encoding is shipped because durable file
persistence is gated on the standardized Parquet logical type (Appendix D).
* No behavioral change to existing DECIMAL / DOUBLE workloads when the new
type is unused.
*Final exam (~12-15 months):*
* Feature parity with other numeric types for the agreed v1 function set (see
Appendix C sketch).
* Interoperable Parquet read/write against an independent implementation (e.g.
parquet-java ↔ parquet-rs) once the logical type is specified.
* Connect, JDBC, and PySpark can create, query, and collect DECFLOAT columns.
* Documented ANSI / IEEE compliance notes and migration guidance from DECIMAL
/ DOUBLE / string workarounds.
Exit criteria for enabling DECFLOAT by default in Spark:
* the v1 function set (Appendix C) is implemented, with full test coverage
* data-source support: Parquet read/write via the standardized logical type
(Appendix D)
* full client support: Spark Connect, JDBC/Thrift, and PySpark (Arrow) can
create, query, and collect DECFLOAT
* SPIP shepherd to ensure completion in reasonable proposed time bounds,
rather than shipping partial support by default.
was:
*Q1. What are you trying to do? Articulate your objectives using absolutely no
jargon.*
Add a new Spark SQL data type *DECFLOAT* that stores decimal numbers with a
flexible decimal point (floating point decimal numbers). Each value keeps up to
a fixed number of significant decimal digits and its own exponent, so one
column can hold both very large integers and very small fractions without
picking a single fixed scale for the whole column.
The type follows the IEEE 754 decimal floating-point formats:
||*SQL type*||*IEEE format*||*Significant digits*||*Storage width*||
|DECFLOAT(16)|decimal64|16|8 bytes|
|DECFLOAT(34)|decimal128|34|16 bytes|
Bare DECFLOAT means DECFLOAT(34).
Unlike DECIMAL(p,s), there is no column-wide scale. Unlike FLOAT / DOUBLE,
arithmetic is done in base 10, so values such as 0.1 are exact.
Because each value carries at most 16 or 34 significant digits, a result
needing more digits is rounded to the format's precision. Rounding is decimal
(base 10), unlike binary rounding in FLOAT/DOUBLE. The v1 fixed default is IEEE
roundTiesToEven / HALF_EVEN (banker's rounding), which differs from Spark's
existing DECIMAL arithmetic (rounds HALF_UP). This can be configurable in
future iterations.
The type also supports IEEE special values: signed zero, +/-Infinity, and quiet
NaN.
*Q2. What problem is this proposal NOT designed to solve?*
* *Arbitrary-precision / unbounded decimals.* Extending Spark's fixed-point
DECIMAL beyond 38 digits, or adding a PostgreSQL-style unbounded NUMERIC, is
out of scope. Those needs are different from IEEE decimal floating point.
* *Wider-than-IEEE formats in v1.* Formats such as a 70-digit / 256-bit
decimal float (sometimes discussed as DECFLOAT(70) / decimal256) are out of
scope for the first delivery. The type and storage designs should not preclude
them later.
* *Replacing* DECIMAL *or* DOUBLE{*}.{*} Existing fixed-point and binary-float
types remain unchanged. DECFLOAT is additive.
* *Non-IEEE decimal-float dialects as the native type.* Engines such as
Snowflake, Oracle, and Teradata expose a decimal float that is not a strict
IEEE width (e.g. 38 digits, no NaN/Inf). Spark's native type targets IEEE 754
decimal64 and decimal128. Mapping those dialects into Spark is a connector /
cast concern, not a second native type.
* *ORC / CSV / JSON as first-class DECFLOAT storage in v1.* Persistence focus
for v1 is Parquet (and table formats that sit on Parquet). Other formats may
round-trip via string or binary until separately specified.
*Q3. How is it done today, and what are the limits of current practice?*
Spark SQL today offers two numeric families for real numbers:
# DECIMAL(p,s) *(fixed-point).* Precision and scale are fixed for the column.
Spark caps precision at 38. Mixed magnitudes force a tradeoff: many fractional
digits leave few integer digits (and the reverse). Chains of arithmetic expand
precision quickly and often require manual casts.
# FLOAT */* DOUBLE *(binary floating-point).* Wide range, but many common
decimal fractions are not exact. About 15-16 significant decimal digits for
DOUBLE, with binary rounding error.
Workarounds in use today:
||*Workaround*||*Limitation*||
|Map high-precision source decimals to DECIMAL(38,s)|Overflow / out-of-range
errors for large integers; silent rounding of long fractions|
|Cast to DOUBLE|Binary rounding; unsuitable when exact decimal fractions matter|
|Store as STRING|Loses numeric semantics; arithmetic and aggregation must be
rebuilt|
|Emulate with (unscaled DECIMAL, scale INT) structs|Not a first-class type;
poor pushdown, stats, and ecosystem support|
Spark also cannot recognize IEEE decimal floating-point values in Parquet (or
other built-in file sources) as a distinct logical type today, so even when
another system wrote such values, Spark cannot load them as decimals with
per-value exponents.
*Q4. What is new in your approach and why do you think it will be successful?*
The approach is deliberately conventional and layered, following prior Spark
type introductions (TIMESTAMP_NTZ, ANSI intervals, TIME):
# *Standard semantics first.* Align the SQL type with IEEE 754-2019 decimal64
/ decimal128 and with existing SQL DECFLOAT practice (DB2, Firebird, MongoDB).
Prefer IEEE default exception handling (produce Inf/NaN), rather than inventing
a Spark-only numeric model. Same goes for rounding, roundTiesToEven is chosen
because it's the IEEE default and libbid/DB2 behaviour.
# *Concrete physical representation.* Use the IEEE binary integer decimal
(BID) interchange encoding in memory and on disk, with a well-known math
library (Intel Decimal Floating-Point Math Library / libbid) for arithmetic
kernels. This avoids a logical-only typedef over DECIMAL / STRING that would
diverge across implementations.
# *Open storage path.* Propose a
[Parquet|https://docs.google.com/document/d/104397AVUqg_JSlzGBdpa3D98X6Dd-RONIgABn3omBcw/edit?tab=t.0]
logical type for decimal floating point (width-extensible FIXED_LEN_BYTE_ARRAY
annotation; initially decimal64 and decimal128) so Spark, other engines, and
table formats can interoperate without Arrow-footer private conventions.
# *Additive rollout.* Gate the type behind a config during incubation (pattern
used for TIME), then enable by default once coverage matches peer numeric types.
We expect this to succeed because:
* The user-visible type matches what migrators already know from DB2 /
Firebird / MongoDB Decimal128, and closes a clear gap versus warehouses that
already ship a decimal-float type.
* The encoding and math library are industry-proven rather than bespoke.
* Spark already has a playbook for introducing a new atomic type end-to-end
(parser → Catalyst → execution → datasources → Connect / PySpark / JDBC).
*Q5. Who cares? If you are successful, what difference will it make?*
* *Users migrating from PostgreSQL, Oracle, DB2, BigQuery, and similar
systems* who today hit Spark's DECIMAL(38,*) ceiling or lose precision via
DOUBLE / STRING workarounds. Unconstrained or high-precision source numerics
map cleanly to a first-class numeric type.
* *Financial, actuarial, and crypto / fintech workloads* that need exact
decimal fractions and mixed magnitudes in one column (rates, notionals,
micro-quantities and large balances together).
* *Existing Spark users* who need to read or write Parquet datasets produced
by systems that already use IEEE decimal128 / DECFLOAT.
* *The wider storage ecosystem* (Parquet, and eventually Iceberg / Delta
consumers) gains a portable decimal-float logical type rather than
engine-private encodings.
Success means: declare DECFLOAT columns, run SQL arithmetic and aggregations
without leaving the numeric domain, and round-trip values through Parquet with
IEEE semantics preserved (including cohort / quantum where the format retains
it).
*Q6. What are the risks?*
||*Risk*||*Mitigation*||
|Parquet / table-format standardization lagging Spark SQL|Ship Spark SQL type
with a documented on-disk encoding; upstream Parquet logical type in parallel;
avoid relying on undocumented footer keys|
|Surprising type coercion (DECFLOAT outranking DOUBLE / interacting with
DECIMAL)|Document precedence explicitly; add golden SQL tests; match IEEE / SQL
expectations rather than silent demotion to binary float|
|Special-value semantics (NaN, Inf) surprising users coming from engines
without them (e.g. some warehouse DECFLOAT dialects)|Document clearly; default
IEEE non-stop handling; ANSI mode provides fail-fast behavior consistent with
other numeric types.|
|External API choice for Inf/NaN (Java BigDecimal cannot represent them)|Pick
an explicit external type story early (see Appendix B) and test Connect /
PySpark / JDBC thoroughly|
|Scope creep into arbitrary precision or DECFLOAT(70)|Keep v1 strictly at
precisions 16 and 34; record wider formats as follow-ons|
|Implementation overhead and ensuring full behavioural consistency in a custom
JVM implementation versus existing libraries|Implement DECFLOAT via JNI calls
to the existing libbid library (which offers fast, exact math for decimal
floating-point numbers)|
*Q7. How long will it take?*
Rough estimate: *on the order of 9-12 months* for feature-complete parity with
peer numeric types, based on TIME (SPARK-51162), TIMESTAMP_NTZ (SPARK-35662),
and ANSI intervals (SPARK-27790), plus extra time for the math library
integration and Parquet logical type.
Suggested work split (can become JIRA sub-tasks):
# *Base type: ~1 month* DecFloatType, parser / DDL, literals (DECFLOAT '...',
optional DF suffix), casts to/from string and numeric types, etc.
# *Arithmetic and comparison kernels: ~2 months* + - * /, unary minus,
comparisons, ordering (totalOrder for sort), hashing / grouping equality
(canonicalization for =-equal values), codegen / interpreted execution.
# *Functions and aggregates: ~2 months* Core scalars (abs, sign, floor, ceil,
round, sqrt, isnan, …), DECFLOAT-specific helpers (quantize, same_quantum,
total_order), sum / avg / min / max / count, window variants.
# *Persistence: ~3 months* Parquet read/write with logical annotation,
partition values, stats / predicate pushdown, caching / shuffle; coordinate
with Parquet format RFC.
# *Clients: ~1.5 months* Spark Connect proto, JDBC / Thrift / Hive result
mapping, catalog / information_schema.
# *PySpark / Arrow: ~1.5 months* DataFrame API, pandas / Arrow interchange,
Python UDFs.
# *Docs, golden tests, benchmarks: ~1 month (overlaps)*
[OPEN] Confirm estimate after sketching the Parquet dependency and whether
Arrow needs a parallel extension type for transport.
*Q8. What are the mid-term and final "exams" to check for success?*
*Mid-term (~4-5 months):*
* DecFloatType(16|34) usable in SQL: literals, DDL, casts, arithmetic,
comparisons, basic aggregates.
* Correct IEEE behavior for a representative set of finite values, signed
zeros, Inf, and NaN (under both ANSI and non-ANSI modes), and rounding
(especially at the precision boundary, e.g. results exceeding 16/34 digits,
ties).
* Round-trip through at least one built-in file source (Parquet) with a stable
encoding, even if the upstream Parquet annotation is still landing.
* No behavioral change to existing DECIMAL / DOUBLE workloads when the new
type is unused.
*Final exam (~9-12 months):*
* Feature parity with other numeric types for the agreed v1 function set (see
Appendix C sketch).
* Interoperable Parquet read/write against an independent implementation (e.g.
parquet-java ↔ parquet-rs) once the logical type is specified.
* Connect, JDBC, and PySpark can create, query, and collect DECFLOAT columns.
* Documented ANSI / IEEE compliance notes and migration guidance from DECIMAL
/ DOUBLE / string workarounds.
> SPIP: Add the DECFLOAT data type
> --------------------------------
>
> Key: SPARK-58820
> URL: https://issues.apache.org/jira/browse/SPARK-58820
> Project: Spark
> Issue Type: Umbrella
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Uroš Bojanić
> Assignee: Uroš Bojanić
> Priority: Major
> Labels: SPIP
>
> *Q1. What are you trying to do? Articulate your objectives using absolutely
> no jargon.*
> Add a new Spark SQL data type *DECFLOAT* that stores decimal numbers with a
> flexible decimal point (floating point decimal numbers). Each value keeps up
> to a fixed number of significant decimal digits and its own exponent, so one
> column can hold both very large integers and very small fractions without
> picking a single fixed scale for the whole column.
> The type follows the IEEE 754 decimal floating-point formats:
> ||*SQL type*||*IEEE format*||*Significant digits*||*Storage width*||
> |DECFLOAT(16)|decimal64|16|8 bytes|
> |DECFLOAT(34)|decimal128|34|16 bytes|
> Bare DECFLOAT means DECFLOAT(34).
> Unlike DECIMAL(p,s), there is no column-wide scale. Unlike FLOAT / DOUBLE,
> arithmetic is done in base 10, so values such as 0.1 are exact.
> Because each value carries at most 16 or 34 significant digits, a result
> needing more digits is rounded to the format's precision. Rounding is decimal
> (base 10), unlike binary rounding in FLOAT/DOUBLE. The v1 fixed default is
> IEEE roundTiesToEven / HALF_EVEN (banker's rounding), which differs from
> Spark's existing DECIMAL arithmetic (rounds HALF_UP). This can be
> configurable in future iterations.
> The type also supports IEEE special values: signed zero, +/-Infinity, and
> quiet NaN.
> *Q2. What problem is this proposal NOT designed to solve?*
> * *Arbitrary-precision / unbounded decimals.* Extending Spark's fixed-point
> DECIMAL beyond 38 digits, or adding a PostgreSQL-style unbounded NUMERIC, is
> out of scope. Those needs are different from IEEE decimal floating point.
> * *Wider-than-IEEE formats in v1.* Formats such as a 70-digit / 256-bit
> decimal float (sometimes discussed as DECFLOAT(70) / decimal256) are out of
> scope for the first delivery. The type and storage designs should not
> preclude them later.
> * *Replacing* DECIMAL *or* DOUBLE{*}.{*} Existing fixed-point and
> binary-float types remain unchanged. DECFLOAT is additive.
> * *Non-IEEE decimal-float dialects as the native type.* Engines such as
> Snowflake, Oracle, and Teradata expose a decimal float that is not a strict
> IEEE width (e.g. 38 digits, no NaN/Inf). Spark's native type targets IEEE 754
> decimal64 and decimal128. Mapping those dialects into Spark is a connector /
> cast concern, not a second native type.
> * *ORC / CSV / JSON as first-class DECFLOAT storage in v1.* The intended
> persistence target is Parquet (and table formats that sit on Parquet), via
> the standardized logical type (Appendix D). Other formats may round-trip via
> existing types (e.g. string or binary) until separately specified.
> *Q3. How is it done today, and what are the limits of current practice?*
> Spark SQL today offers two numeric families for real numbers:
> # DECIMAL(p,s) *(fixed-point).* Precision and scale are fixed for the
> column. Spark caps precision at 38. Mixed magnitudes force a tradeoff: many
> fractional digits leave few integer digits (and the reverse). Chains of
> arithmetic expand precision quickly and often require manual casts.
> # FLOAT */* DOUBLE *(binary floating-point).* Wide range, but many common
> decimal fractions are not exact. About 15-16 significant decimal digits for
> DOUBLE, with binary rounding error.
> Workarounds in use today:
> ||*Workaround*||*Limitation*||
> |Map high-precision source decimals to DECIMAL(38,s)|Overflow / out-of-range
> errors for large integers; silent rounding of long fractions|
> |Cast to DOUBLE|Binary rounding; unsuitable when exact decimal fractions
> matter|
> |Store as STRING|Loses numeric semantics; arithmetic and aggregation must be
> rebuilt|
> |Emulate with (unscaled DECIMAL, scale INT) structs|Not a first-class type;
> poor pushdown, stats, and ecosystem support|
> Spark also cannot recognize IEEE decimal floating-point values in Parquet (or
> other built-in file sources) as a distinct logical type today, so even when
> another system wrote such values, Spark cannot load them as decimals with
> per-value exponents.
> *Q4. What is new in your approach and why do you think it will be successful?*
> The approach is deliberately conventional and layered, following prior Spark
> type introductions (TIMESTAMP_NTZ, ANSI intervals, TIME):
> # *Standard semantics first.* Align the SQL type with IEEE 754-2019
> decimal64 / decimal128 and with existing SQL DECFLOAT practice (DB2,
> Firebird, MongoDB), rather than inventing a Spark-only numeric model.
> Rounding follows the IEEE default (roundTiesToEven / HALF_EVEN, as in
> libbid/DB2). Language-neutral IEEE contract, shared conformance vectors,
> testable across implementations.
> # *Concrete physical representation.* Internal to engine, represent DECFLOAT
> in memory as fixed-width IEEE binary integer decimal (BID) bit patterns
> (Appendix A) - computed by a single pure-Java in-tree arithmetic path (POC:
> SPARK-59111) with no native/JNI dependency as part of this SPIP. The
> prototype ports and validates against libbid conformance vectors; whereas the
> technicals of arithmetic kernel are implementation detail to be finalized
> during actual code review. Note that the on-disk Parquet encoding is a
> separate, deferred decision (Appendix D), independent of this in-memory
> layout / representation (e.g. similar to Timestamp nano). This avoids a
> logical-only typedef over DECIMAL / STRING that would diverge across
> implementations.
> # *Open storage path.* Collaborate with the
> [Parquet|https://docs.google.com/document/d/104397AVUqg_JSlzGBdpa3D98X6Dd-RONIgABn3omBcw/edit?tab=t.0]
> community on a standardized decimal-float logical type; the actual physical
> representation will be discussed and decided there, and Spark ships no
> private on-disk encoding in the meantime (Appendix D).
> # *Additive rollout.* Gate the type behind a config during incubation, and
> define measurable exit criteria for enabling it by default (Q8) once coverage
> matches peer numeric types.
> We expect this to succeed because:
> * The user-visible type matches what migrators already know from DB2 /
> Firebird / MongoDB Decimal128, and closes a clear gap versus warehouses that
> already ship a decimal-float type.
> * The encoding and the BID arithmetic (compliant with IEEE 754) are
> industry-proven, rather than bespoke.
> * Spark already has a playbook for introducing a new fractional type
> end-to-end (parser → Catalyst → execution → datasources → Connect / PySpark /
> JDBC).
> *Q5. Who cares? If you are successful, what difference will it make?*
> * *Users migrating from PostgreSQL, Oracle, DB2, BigQuery, and similar
> systems* who today hit Spark's DECIMAL(38,*) ceiling or lose precision via
> DOUBLE / STRING workarounds. Unconstrained or high-precision source numerics
> map cleanly to a first-class numeric type.
> * *Financial, actuarial, and crypto / fintech workloads* that need exact
> decimal fractions and mixed magnitudes in one column (rates, notionals,
> micro-quantities and large balances together).
> * *Existing Spark users* who need to read or write Parquet datasets produced
> by systems that already use IEEE decimal128 / DECFLOAT.
> * *The wider storage ecosystem* (Parquet, and eventually Iceberg / Delta
> consumers) gains a portable decimal-float logical type rather than
> engine-private encodings.
> Success means: declare DECFLOAT columns, run SQL arithmetic and aggregations
> without leaving the numeric domain, and once the Parquet logical type is
> standardized - round-trip values through Parquet with standardized semantics
> / encoding.
> *Q6. What are the risks?*
> ||*Risk*||*Mitigation*||
> |Parquet / table-format standardization lagging Spark SQL|Land the in-memory
> type, compute, and clients first in Spark; make Parquet persistence
> conditional on the standardized logical type; ship no private on-disk
> encoding in the meantime (Appendix D).|
> |Surprising type coercion (DECFLOAT outranking DOUBLE / interacting with
> DECIMAL)|Document precedence explicitly; add golden SQL tests; match IEEE /
> SQL expectations rather than silent demotion to binary float.|
> |Special-value semantics (NaN, Inf) surprising users coming from engines
> without them (e.g. some warehouse DECFLOAT dialects)|Document clearly;
> DECFLOAT's Inf/NaN mirror DOUBLE's, which Spark already supports.|
> |External API choice for Inf/NaN (Java BigDecimal cannot represent
> them)|Resolved in Appendix B - Spark-owned JVM value class, decimal.Decimal
> (Python), Arrow extension type, DECFLOAT JDBC mapping.|
> |Scope creep into arbitrary precision or DECFLOAT(70)|Keep v1 strictly at
> precisions 16 and 34; record wider formats as follow-ons.|
> |Implementation overhead and ensuring full behavioural consistency in a
> custom JVM implementation versus existing libraries|Use a pure Java, in-tree
> BID implementation ported from libbid and validated against its conformance
> vectors (SPARK-59111), providing reference-grade IEEE correctness without any
> native/JNI dependencies.|
> *Q7. How long will it take?*
> Rough estimate: *on the order of 12-15 months* for feature-complete parity
> with peer numeric types. DECFLOAT is larger in scope than prior single-type
> efforts (it adds arithmetic kernels and a special-value domain), so
> TIMESTAMP_NTZ (SPARK-35662) and ANSI intervals (SPARK-27790) are a lower
> bound rather than a direct model. The estimate is baselined against the v1
> scope in Appendix C.
> Suggested work split (can become JIRA sub-tasks):
> # *Base type: ~1 month* DecFloatType, parser / DDL, literals (DECFLOAT
> '...', optional DF suffix), casts to/from string and numeric types, etc.
> # *Arithmetic and comparison kernels: ~2-3 months* + - * /, unary minus,
> comparisons, ordering and equality following Spark's DOUBLE conventions (-0.0
> = 0.0, NaN = NaN, NaN sorts last), hashing / grouping-equality
> canonicalization, codegen / interpreted execution.
> # *Functions and aggregates: ~2 months* Core scalars (abs, sign, floor,
> ceil, round, sqrt, isnan, …), DECFLOAT-specific helpers (quantize,
> same_quantum, total_order), sum / avg / min / max / count, window variants.
> # *Persistence: ~2-3 months* Parquet read/write integration - with logical
> annotation, partition values, stats / predicate pushdown, caching / shuffle;
> gated on the Parquet RFC (Appendix D).
> # *Clients: ~1.5 months* Spark Connect proto, JDBC / Thrift / Hive result
> mapping, catalog / information_schema.
> # *PySpark / Arrow: ~1.5 months* DataFrame API, pandas / Arrow interchange,
> Python UDFs.
> # *Docs, golden tests, benchmarks: ~1 month (overlaps)*
> Note for delivery: the in-tree BID arithmetic library (POC: SPARK-59111 /
> [#58410|https://github.com/apache/spark/pull/58410]) will land together with
> its first consumer - the base type and its Catalyst wiring; to be reviewed &
> exercised as part of the feature, not as a standalone module. Also, Arrow
> will require a parallel extension type for transport (Appendix B).
> *Q8. What are the mid-term and final "exams" to check for success?*
> *Mid-term (~6-8 months):*
> * DecFloatType(16|34) usable in SQL: literals, DDL, casts, arithmetic,
> comparisons, basic aggregates.
> * Correct IEEE behavior for a representative set of finite values, signed
> zeros, Inf, and NaN, and rounding (especially at the precision boundary, e.g.
> results exceeding 16/34 digits, ties).
> * Round-trip DECFLOAT through at least one client path (e.g. Spark Connect /
> Arrow collect); no on-disk file encoding is shipped because durable file
> persistence is gated on the standardized Parquet logical type (Appendix D).
> * No behavioral change to existing DECIMAL / DOUBLE workloads when the new
> type is unused.
> *Final exam (~12-15 months):*
> * Feature parity with other numeric types for the agreed v1 function set
> (see Appendix C sketch).
> * Interoperable Parquet read/write against an independent implementation
> (e.g. parquet-java ↔ parquet-rs) once the logical type is specified.
> * Connect, JDBC, and PySpark can create, query, and collect DECFLOAT columns.
> * Documented ANSI / IEEE compliance notes and migration guidance from
> DECIMAL / DOUBLE / string workarounds.
> Exit criteria for enabling DECFLOAT by default in Spark:
> * the v1 function set (Appendix C) is implemented, with full test coverage
> * data-source support: Parquet read/write via the standardized logical type
> (Appendix D)
> * full client support: Spark Connect, JDBC/Thrift, and PySpark (Arrow) can
> create, query, and collect DECFLOAT
> * SPIP shepherd to ensure completion in reasonable proposed time bounds,
> rather than shipping partial support by default.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
