gene-db commented on code in PR #58175:
URL: https://github.com/apache/spark/pull/58175#discussion_r3897455856
##########
docs/sql-data-sources-csv.md:
##########
@@ -225,7 +225,13 @@ Data source options of CSV can be set via:
<tr>
<td><code>singleVariantColumn</code></td>
<td>(none)</td>
- <td>If specified, the entire CSV record is parsed and stored as a single
column of <code>VariantType</code> with the given column name, instead of being
split into individual fields.</td>
+ <td>If specified, the entire CSV record is parsed and stored as a single
column of <code>VariantType</code> with the given column name, instead of being
split into individual fields. By default, scalar values are type-inferred
inside the variant (for example, <code>"0001"</code> becomes the integer
<code>1</code>). To keep every scalar as a string instead, set
<code>variantRespectInferSchema</code> to <code>true</code> and
<code>inferSchema</code> to <code>false</code>.</td>
+ <td>read</td>
+ </tr>
+ <tr>
+ <td><code>variantRespectInferSchema</code></td>
Review Comment:
This seems a bit confusing, since it also seems to involve `inferSchema`.
Can we just have an option like `variantScalarStrings` (or something
similar), so if it is enabled, we always use `string` for scalars? Then, we
don't have to also depend on `inferSchema`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
