davidm-db opened a new pull request, #56439:
URL: https://github.com/apache/spark/pull/56439
### What changes were proposed in this pull request?
Removes the `spark.sql.types.framework.enabled` configuration and makes the
Types Framework unconditionally active — it is the single path for the data
types it manages (`TimeType` and the nanosecond timestamp types), with no flag
gating it.
Two follow-on adjustments fall out of making the framework unconditional:
- The nanosecond timestamp types (SPARK-56822) gated their preview flag on
the framework flag ("can only be enabled when the framework is enabled"). With
the framework always on that requirement is always satisfied, so it is removed;
the nanosecond types remain gated by their own
`spark.sql.timestampNanosTypes.enabled` at the parser and schema entry points.
- The TIME-type-enabled check that lived on the legacy
serializer/deserializer path (reached only when the framework was off) moves
into the framework's `TimeType` operations, so disabling
`spark.sql.timeType.enabled` still rejects TIME exactly as before.
### Why are the changes needed?
The flag was only ever a release-rollout kill switch — on under tests, off
in production, never intended as a per-session toggle. Every framework dispatch
read the session config, and those dispatches sit on per-row hot paths
(columnar reads, projections, encoder building), where the lookup is a
measurable cost; the only way to make it cheap is to effectively pin it, at
which point it stops being a meaningful toggle. A permanently test-on/prod-off
switch also adds standing surface area — an audit-allowlist exception, and a
divergent legacy path that tests never exercise — for no benefit now that the
framework is validated. Removing it deletes the per-row cost and collapses the
two code paths into one.
### Does this PR introduce _any_ user-facing change?
No. `spark.sql.types.framework.enabled` is internal, defaulted on only under
tests and off in production, and never shipped in a released (GA) version, so
there is no migration concern and no `RemovedConfig` entry is needed (cf.
SPARK-49671, which removed the internal, already-on-by-default
`spark.sql.collation.trim.enabled` without one). Behavior for the managed types
is unchanged, and feature gates such as `spark.sql.timeType.enabled` /
`spark.sql.timestampNanosTypes.enabled` continue to apply.
### How was this patch tested?
Existing encoder, type-conversion, Row-JSON, nanosecond-timestamp, and SQL
config-audit suites pass with the framework now unconditionally enabled
(already the default under tests); suites that toggled the flag were updated to
drop the now-impossible disabled-framework cases. A new `TimeTypeOpsSuite` pins
the relocated gate: building a TIME serializer/deserializer with
`spark.sql.timeType.enabled=false` is rejected with `UNSUPPORTED_TIME_TYPE`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.8 (claude-opus-4-8)
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
