cloud-fan commented on code in PR #58175:
URL: https://github.com/apache/spark/pull/58175#discussion_r3864082814
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala:
##########
@@ -3897,6 +3898,104 @@ abstract class CSVSuite
}
}
+ test("csv variant retains scalars as strings when inferSchema is disabled") {
+ withTempPath { path =>
+ val data =
+ """field 1,field2
+ |100,1.1
+ |2000-01-01,2000-01-01 01:02:03
+ |,true
+ |1e9,hello,extra
+ |missing
+ |""".stripMargin
+ Files.write(path.toPath, data.getBytes(StandardCharsets.UTF_8))
+
+ def checkSingleVariant(options: Map[String, String], expected: String*):
Unit = {
+ val allOptions = options ++ Map("singleVariantColumn" -> "v")
Review Comment:
**Non-blocking:**
The documented explicit `VariantType` schema path needs a regression that
distinguishes strings from inferred scalars. This helper always adds
`singleVariantColumn`, so please add an explicit-schema case under
`variantRespectInferSchema=true` and `inferSchema=false` and assert the Variant
type (for example, with `schema_of_variant`); casting a scalar Variant to
string cannot tell `"100"` from `100`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
