[
https://issues.apache.org/jira/browse/SPARK-57572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-57572:
------------------------------------
Assignee: Shrirang Mhalgi
> Infer the TIME type during CSV and JSON schema inference
> --------------------------------------------------------
>
> Key: SPARK-57572
> URL: https://issues.apache.org/jira/browse/SPARK-57572
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: Max Gekk
> Assignee: Shrirang Mhalgi
> Priority: Major
> Labels: pull-request-available
>
> h2. What
> Infer {{TimeType}} for time-only string values during CSV and JSON schema
> inference (when no
> explicit schema is provided), analogous to the existing Date/Timestamp
> inference.
> h2. Gap
> {{CSVInferSchema}} (and {{JsonInferSchema}}) infer
> DateType/TimestampNTZType/TimestampType but
> never {{TimeType}}. With an explicit schema, TIME read/write already works;
> only auto-inference
> is missing.
> h2. Scope
> * Add a tryParseTime step to the inference type ladder, ordered to avoid
> regressions (a
> time-only string like 12:13:14 must not be misclassified, and existing
> date/timestamp
> inference must be unchanged).
> * Consider gating behind the timeFormat option / a config and the
> spark.sql.timeType.enabled
> flag, given inference ambiguity.
> h2. Acceptance criteria
> * CSV/JSON files with time-only columns infer TimeType (when enabled);
> existing inference is
> unchanged.
> * Tests in CSVInferSchemaSuite / JsonInferSchemaSuite and the CSV/JSON file
> suites.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
