Babu Mahesh created SPARK-49763:
-----------------------------------
Summary: [CSV Reader] Add Flag to Control Inference of Time-Only
Columns as String or Timestamp During Schema Detection
Key: SPARK-49763
URL: https://issues.apache.org/jira/browse/SPARK-49763
Project: Spark
Issue Type: Task
Components: SQL
Affects Versions: 3.5.2
Reporter: Babu Mahesh
This task involves adding a configurable flag to control the inference of
time-only columns during schema detection in Spark. By default, Spark converts
time-only columns to {{Timestamp}} type, which can lead to unintended behavior
in certain use cases. This new flag will allow users to specify whether
time-only columns should be inferred as {{Timestamp}} or as {{{}String{}}}.
h3. Key Changes:
* Introduce a flag (e.g., {{{}inferStringTypeForTimeOnlyColumn{}}}).
* When the flag is set to {{{}true{}}}, time-only columns will be inferred as
{{{}String{}}}.
* When the flag is set to {{false}} (default), time-only columns will be
inferred as {{{}Timestamp{}}}.
* Update documentation to reflect the new option.
* Ensure backward compatibility by defaulting to the current behavior.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
