imback82 edited a comment on pull request #29328: URL: https://github.com/apache/spark/pull/29328#issuecomment-673180206
> LGTM except one question: do we still need this fix? https://github.com/apache/spark/pull/29328/files#r468330253 Yes, without it, the following tests fail: ``` org.apache.spark.sql.execution.datasources.csv.CSVLegacyTimeParserSuite.SPARK-23846: schema inferring touches less data if samplingRatio < 1.0 org.apache.spark.sql.execution.datasources.csv.CSVv1Suite.SPARK-23846: schema inferring touches less data if samplingRatio < 1.0 org.apache.spark.sql.execution.datasources.csv.CSVv2Suite.SPARK-23846: schema inferring touches less data if samplingRatio < 1.0 org.apache.spark.sql.execution.datasources.json.JsonLegacyTimeParserSuite.SPARK-11544 test pathfilter org.apache.spark.sql.execution.datasources.json.JsonLegacyTimeParserSuite.SPARK-23849: schema inferring touches less data if samplingRatio < 1.0 org.apache.spark.sql.execution.datasources.json.JsonV1Suite.SPARK-11544 test pathfilter org.apache.spark.sql.execution.datasources.json.JsonV1Suite.SPARK-23849: schema inferring touches less data if samplingRatio < 1.0 org.apache.spark.sql.execution.datasources.json.JsonV2Suite.SPARK-11544 test pathfilter org.apache.spark.sql.execution.datasources.json.JsonV2Suite.SPARK-23849: schema inferring touches less data if samplingRatio < 1.0 ``` Btw, this is an existing bug. For example, for `"SPARK-11544 test pathfilter"`, if I change `spark.read.options(extraOptions).json(path)` to `spark.read.options(extraOptions).option("path", path). format("json").load()`, it would fail. The changes in this PR are surfacing the existing issue. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
