This is an automated email from the ASF dual-hosted git repository.
MaxGekk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new b5d141dc8210 [SPARK-57418][DOC] Add singleVariantColumn option to CSV,
JSON, and XML data source options tables
b5d141dc8210 is described below
commit b5d141dc8210236beb8e91fdb642e4fc55770ec0
Author: BRIJ RAJ KISHORE <[email protected]>
AuthorDate: Tue Jun 23 17:38:01 2026 +0200
[SPARK-57418][DOC] Add singleVariantColumn option to CSV, JSON, and XML
data source options tables
### What changes were proposed in this pull request?
Added the missing `singleVariantColumn` option to the Data Source Options
tables in:
- `docs/sql-data-sources-csv.md`
- `docs/sql-data-sources-json.md`
- `docs/sql-data-sources-xml.md`
The option was introduced in Spark 4.1.0 (SPARK-51298 for CSV, also
supported for JSON and XML) but was never documented in the reference tables.
It is defined as a shared constant in `DataSourceOptions.scala` and consumed by
`CSVOptions`, `JSONOptions`, and `XmlOptions`.
### Why are the changes needed?
Users have no way to discover `singleVariantColumn` from the official data
source options reference. The option allows ingesting an entire CSV/JSON/XML
record as a single `VariantType` column instead of parsing it into individual
fields — a key use case for the Variant type introduced in Spark 4.0.
### Does this PR introduce _any_ user-facing change?
No. Documentation only.
### How was this patch tested?
No code change — documentation only. Verified the option is defined in
`DataSourceOptions.scala` (line 78), `CSVOptions.scala` (line 338),
`JSONOptions.scala` (line 215), and `XmlOptions.scala` (line 194).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude (Anthropic)
Closes #56469 from brijrajk/SPARK-57418-singlevariantcolumn-docs.
Authored-by: BRIJ RAJ KISHORE <[email protected]>
Signed-off-by: Max Gekk <[email protected]>
---
docs/sql-data-sources-csv.md | 6 ++++++
docs/sql-data-sources-json.md | 6 ++++++
docs/sql-data-sources-xml.md | 7 +++++++
3 files changed, 19 insertions(+)
diff --git a/docs/sql-data-sources-csv.md b/docs/sql-data-sources-csv.md
index 8008bc562082..a88b4ab14986 100644
--- a/docs/sql-data-sources-csv.md
+++ b/docs/sql-data-sources-csv.md
@@ -216,6 +216,12 @@ Data source options of CSV can be set via:
<td>Allows renaming the new field having malformed string created by
<code>PERMISSIVE</code> mode. This overrides
<code>spark.sql.columnNameOfCorruptRecord</code>.</td>
<td>read</td>
</tr>
+ <tr>
+ <td><code>singleVariantColumn</code></td>
+ <td>(none)</td>
+ <td>If specified, the entire CSV record is parsed and stored as a single
column of <code>VariantType</code> with the given column name, instead of being
split into individual fields.</td>
+ <td>read</td>
+ </tr>
<tr>
<td><code>multiLine</code></td>
<td>false</td>
diff --git a/docs/sql-data-sources-json.md b/docs/sql-data-sources-json.md
index e5c86054c09b..e9449ee17a4d 100644
--- a/docs/sql-data-sources-json.md
+++ b/docs/sql-data-sources-json.md
@@ -207,6 +207,12 @@ Data source options of JSON can be set via:
<td>Allows inferring of <code>TimestampType</code> and
<code>TimestampNTZType</code> from strings that match the timestamp patterns
defined by the <code>timestampFormat</code> and <code>timestampNTZFormat</code>
options respectively.</td>
<td>read</td>
</tr>
+ <tr>
+ <td><code>singleVariantColumn</code></td>
+ <td>(none)</td>
+ <td>If specified, the entire JSON record is parsed and stored as a single
column of <code>VariantType</code> with the given column name, instead of being
split into individual fields.</td>
+ <td>read</td>
+ </tr>
<tr>
<td><code>enableDateTimeParsingFallback</code></td>
<td>Enabled if the time parser policy has legacy settings or if no custom
date or timestamp pattern was provided.</td>
diff --git a/docs/sql-data-sources-xml.md b/docs/sql-data-sources-xml.md
index 949e6239e52b..bdc0c7e066c3 100644
--- a/docs/sql-data-sources-xml.md
+++ b/docs/sql-data-sources-xml.md
@@ -105,6 +105,13 @@ Data source options of XML can be set via:
<td>read</td>
</tr>
+ <tr>
+ <td><code>singleVariantColumn</code></td>
+ <td>(none)</td>
+ <td>If specified, the entire XML record is parsed and stored as a single
column of <code>VariantType</code> with the given column name, instead of being
split into individual fields. When writing, if the schema consists solely of
this column, the Variant value is written directly under the row tag.</td>
+ <td>read/write</td>
+ </tr>
+
<tr>
<td><code>attributePrefix</code></td>
<td><code>_</code></td>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
