mengxr commented on a change in pull request #24483: [SPARK-27588] Binary file
data source fails fast and doesn't attempt to read very large files
URL: https://github.com/apache/spark/pull/24483#discussion_r279522459
##########
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/binaryfile/BinaryFileFormatSuite.scala
##########
@@ -339,4 +341,31 @@ class BinaryFileFormatSuite extends QueryTest with
SharedSQLContext with SQLTest
assert(df.select("LENGTH").first().getLong(0) === content.length,
"column pruning should be case insensitive")
}
+
+ test("fail fast and do not attempt to read if a file is too big") {
+ assert(spark.conf.get(SOURCES_BINARY_FILE_MAX_LENGTH) === Int.MaxValue)
+ withTempPath { file =>
+ val path = file.getPath
+ val content = "123".getBytes
+ Files.write(file.toPath, content, StandardOpenOption.CREATE,
StandardOpenOption.WRITE)
+ def readContent(): DataFrame = {
+ spark.read.format(BINARY_FILE)
+ .load(path)
+ .select(CONTENT)
+ }
+ val expected = Seq(Row(content))
+ QueryTest.checkAnswer(readContent(), expected)
+ withSQLConf(CONF_SOURCES_BINARY_FILE_MAX_LENGTH ->
content.length.toString) {
+ QueryTest.checkAnswer(readContent(), expected)
+ }
+ // Disable read. If the implementation attempts to read, the exception
would be different.
+ file.setReadable(false)
Review comment:
If we still set the max to `content.length`, the test will fail. This is to
ensure we don't even attempt to read the file if the file is too big.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
