Github user CodingCat commented on a diff in the pull request:
https://github.com/apache/spark/pull/20072#discussion_r159474580
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/HadoopFsRelation.scala
---
@@ -82,7 +82,11 @@ case class HadoopFsRelation(
}
}
- override def sizeInBytes: Long = location.sizeInBytes
+ override def sizeInBytes: Long = {
+ val sizeFactor = sqlContext.conf.sizeToMemorySizeFactor
+ (location.sizeInBytes * sizeFactor).toLong
--- End diff --
before the latest commit, there is a safe check
https://github.com/apache/spark/pull/20072/commits/e6065c75015b8a2c0eff9f3c6e1ebfe148b28e65#diff-fcb68cd3c7630f337ce9a3b479b6d0c4R88
However, since sizeFactor is a double, any overflow with positive double
numbers would be capped as Double.PositiveInfinity, and as @wzhfy indicated,
any double number which is larger than Long.MaxValue would return Long.MaxValue
in its toLong method
so it should be safe here
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
