LuciferYang commented on code in PR #40332:
URL: https://github.com/apache/spark/pull/40332#discussion_r1129391698
##########
connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/ClientE2ETestSuite.scala:
##########
@@ -644,6 +645,34 @@ class ClientE2ETestSuite extends RemoteSparkSession with
SQLHelper {
.collect()
assert(result sameElements expected)
}
+
+ test("json from Dataset[String]") {
+ val session = spark
+ import session.implicits._
+ val expected = Seq(
+ new GenericRowWithSchema(
+ Array(73, "Shandong", "Kong"),
+ new StructType().add("age", LongType).add("city",
StringType).add("name", StringType)))
+ val ds = Seq("""{"name":"Kong","age":73,"city":'Shandong'}""").toDS()
+ val result = spark.read.option("allowSingleQuotes", "true").json(ds)
+ checkSameResult(expected, result)
Review Comment:
https://github.com/apache/spark/blob/69dd20b5e45c7e3533efbfdc1974f59931c1b781/sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala#L404-L417
From the code of the server side, `userSpecifiedSchema` is an
`Option[StructType]` and default is None, so we can use it without specifying
the`userSpecifiedSchema`? Or is my test case not the correct scenario?
##########
connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/ClientE2ETestSuite.scala:
##########
@@ -644,6 +645,34 @@ class ClientE2ETestSuite extends RemoteSparkSession with
SQLHelper {
.collect()
assert(result sameElements expected)
}
+
+ test("json from Dataset[String]") {
+ val session = spark
+ import session.implicits._
+ val expected = Seq(
+ new GenericRowWithSchema(
+ Array(73, "Shandong", "Kong"),
+ new StructType().add("age", LongType).add("city",
StringType).add("name", StringType)))
+ val ds = Seq("""{"name":"Kong","age":73,"city":'Shandong'}""").toDS()
+ val result = spark.read.option("allowSingleQuotes", "true").json(ds)
+ checkSameResult(expected, result)
Review Comment:
https://github.com/apache/spark/blob/69dd20b5e45c7e3533efbfdc1974f59931c1b781/sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala#L404-L417
From the code of the server side, `userSpecifiedSchema` is an
`Option[StructType]` and default is None, so we can use it without specifying
the`userSpecifiedSchema`? Or is my test case not the correct scenario?
@zhengruifeng
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
