kejiqing created SPARK-23563:
--------------------------------
Summary: make size fo cache in CodeGenerator configable
Key: SPARK-23563
URL: https://issues.apache.org/jira/browse/SPARK-23563
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 2.3.0
Reporter: kejiqing
the cache in class
org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator has a hard cod
maxmunSize 100, current code is:
{code:java}
// scala
private val cache = CacheBuilder.newBuilder()
.maximumSize(100)
.build(
new CacheLoader[CodeAndComment, (GeneratedClass, Int)]() {
override def load(code: CodeAndComment): (GeneratedClass, Int) = {
val startTime = System.nanoTime()
val result = doCompile(code)
val endTime = System.nanoTime()
def timeMs: Double = (endTime - startTime).toDouble / 1000000
CodegenMetrics.METRIC_SOURCE_CODE_SIZE.update(code.body.length)
CodegenMetrics.METRIC_COMPILATION_TIME.update(timeMs.toLong)
logInfo(s"Code generated in $timeMs ms")
result
}
})
{code}
In some specific situation, for example: a long term and spark tasks are
unchanged, the size of cache maximumSize configuration is a better idea.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
