Github user MaxGekk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21299#discussion_r187697424
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonDataSource.scala
---
@@ -99,12 +99,7 @@ object TextInputJsonDataSource extends JsonDataSource {
def inferFromDataset(json: Dataset[String], parsedOptions: JSONOptions):
StructType = {
val sampled: Dataset[String] = JsonUtils.sample(json, parsedOptions)
- val rdd: RDD[InternalRow] = sampled.queryExecution.toRdd
- val rowParser = parsedOptions.encoding.map { enc =>
- CreateJacksonParser.internalRow(enc, _: JsonFactory, _: InternalRow)
- }.getOrElse(CreateJacksonParser.internalRow(_: JsonFactory, _:
InternalRow))
-
- JsonInferSchema.infer(rdd, parsedOptions, rowParser)
+ JsonInferSchema.infer(sampled, parsedOptions,
CreateJacksonParser.string)
--- End diff --
String Reader of JacksonParser can be slower than specialized
[UTF8StreamJsonParser](https://github.com/FasterXML/jackson-core/blob/master/src/main/java/com/fasterxml/jackson/core/json/UTF8StreamJsonParser.java)
used when you pass an array of bytes (even if it spends some time for encoding
detection). You can check that by
[JsonBenchmarks](https://github.com/apache/spark/blob/bd14da6fd5a77cc03efff193a84ffccbe892cc13/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonBenchmarks.scala#L48).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
