liupc commented on a change in pull request #21950: [SPARK-24914][SQL] Add
configuration to avoid OOM during broadcast join (and other negative side
effects of incorrect table sizing)
URL: https://github.com/apache/spark/pull/21950#discussion_r376188873
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceUtils.scala
##########
@@ -59,4 +60,9 @@ object DataSourceUtils {
val name = path.getName
!(name.startsWith("_") || name.startsWith("."))
}
+
+ def calcDataSize(properties: Map[String, String], sizeInBytes: BigInt):
BigInt = {
+ val factor =
NumberUtils.toInt(properties.get("deserFactor").getOrElse("1"), 1)
Review comment:
As documented, this factor should be a float, and also I think we can use
scala type conversions to simplify the codes(without introducing NumberUtils)
```
hive.stats.deserialization.factor
Default Value:
Hive 0.13 to 2.x.x: (float) 1.0
Hive 3.0.0 and later: (float) 10.0
Added In: Hive 0.13 with HIVE-5921
Default value changed from 1.0 to 10.0 in Hive 3.0
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
