cloud-fan commented on code in PR #58175:
URL: https://github.com/apache/spark/pull/58175#discussion_r3901271303
##########
docs/sql-data-sources-csv.md:
##########
@@ -225,7 +225,13 @@ Data source options of CSV can be set via:
<tr>
<td><code>singleVariantColumn</code></td>
<td>(none)</td>
- <td>If specified, the entire CSV record is parsed and stored as a single
column of <code>VariantType</code> with the given column name, instead of being
split into individual fields.</td>
+ <td>If specified, the entire CSV record is parsed and stored as a single
column of <code>VariantType</code> with the given column name, instead of being
split into individual fields. By default, scalar values are type-inferred
inside the variant (for example, <code>"0001"</code> becomes the integer
<code>1</code>). To keep every scalar as a string instead, set
<code>variantRespectInferSchema</code> to <code>true</code> and
<code>inferSchema</code> to <code>false</code>.</td>
+ <td>read</td>
+ </tr>
+ <tr>
+ <td><code>variantRespectInferSchema</code></td>
+ <td>false</td>
+ <td>When reading into a <code>VARIANT</code> (see
<code>singleVariantColumn</code> or a <code>VariantType</code> column in the
schema), controls whether the <code>inferSchema</code> option is honored. When
<code>true</code> and <code>inferSchema</code> is <code>false</code>, scalar
values are preserved as strings inside the variant instead of being inferred as
boolean, long, decimal, date, or timestamp. When <code>false</code> (the
default), scalar types are always inferred regardless of
<code>inferSchema</code>.</td>
Review Comment:
**Nit (P3):** `VariantValueConverter` inference is stateful per field: once
an earlier value falls back to `StringType`, later values use string parsing
without retrying the scalar parsers. For example, a single unsplit column
containing `text` followed by `0001` stores the second value as the string
`"0001"`, so saying scalar types are *always* inferred overstates the behavior.
Could we describe default inference as enabled or best-effort, and keep `0001`
as an illustrative rather than unconditional example?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
