[
https://issues.apache.org/jira/browse/SPARK-57572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18090346#comment-18090346
]
Max Gekk commented on SPARK-57572:
----------------------------------
[~shrirangmhalgi] Sure, go ahead.
> Infer the TIME type during CSV and JSON schema inference
> --------------------------------------------------------
>
> Key: SPARK-57572
> URL: https://issues.apache.org/jira/browse/SPARK-57572
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: Max Gekk
> Priority: Major
>
> h2. What
> Infer {{TimeType}} for time-only string values during CSV and JSON schema
> inference (when no
> explicit schema is provided), analogous to the existing Date/Timestamp
> inference.
> h2. Gap
> {{CSVInferSchema}} (and {{JsonInferSchema}}) infer
> DateType/TimestampNTZType/TimestampType but
> never {{TimeType}}. With an explicit schema, TIME read/write already works;
> only auto-inference
> is missing.
> h2. Scope
> * Add a tryParseTime step to the inference type ladder, ordered to avoid
> regressions (a
> time-only string like 12:13:14 must not be misclassified, and existing
> date/timestamp
> inference must be unchanged).
> * Consider gating behind the timeFormat option / a config and the
> spark.sql.timeType.enabled
> flag, given inference ambiguity.
> h2. Acceptance criteria
> * CSV/JSON files with time-only columns infer TimeType (when enabled);
> existing inference is
> unchanged.
> * Tests in CSVInferSchemaSuite / JsonInferSchemaSuite and the CSV/JSON file
> suites.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
