Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/20959#discussion_r180943744
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala
---
@@ -1279,4 +1278,68 @@ class CSVSuite extends QueryTest with
SharedSQLContext with SQLTestUtils {
Row("0,2013-111-11 12:13:14") :: Row(null) :: Nil
)
}
+
+ test("SPARK-23846: schema inferring touches less data if samplingRatio <
1.0") {
+ // Set default values for the DataSource parameters to make sure
+ // that whole test file is mapped to only one partition. This will
guarantee
+ // reliable sampling of the input file.
+ withSQLConf(
+ "spark.sql.files.maxPartitionBytes" -> (128 * 1024 * 1024).toString,
+ "spark.sql.files.openCostInBytes" -> (4 * 1024 * 1024).toString
+ )(withTempPath { path =>
+ val rdd = spark.sqlContext.range(0, 100).map {row =>
--- End diff --
Can we actually do
```
val predefinedSample = Set[Long](2, 8, 15, 27, 30, 34, 35, 37, 44,
46,
57, 62, 68, 72)
val value = row.getLong(0)
if (predefinedSample.contains(value)) {
value.toString
} else {
(value.toDouble + 0.1).toString
}
```
outside of `map`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
