[
https://issues.apache.org/jira/browse/SPARK-23563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-23563.
----------------------------------
Resolution: Incomplete
> make the size fo cache in CodeGenerator configable
> --------------------------------------------------
>
> Key: SPARK-23563
> URL: https://issues.apache.org/jira/browse/SPARK-23563
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.3.0
> Reporter: kejiqing
> Priority: Minor
> Labels: bulk-closed
>
> the cache in class
> org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator has a hard
> cod maxmunSize 100, current code is:
>
> {code:java}
> // scala
> private val cache = CacheBuilder.newBuilder()
> .maximumSize(100)
> .build(
> new CacheLoader[CodeAndComment, (GeneratedClass, Int)]() {
> override def load(code: CodeAndComment): (GeneratedClass, Int) = {
> val startTime = System.nanoTime()
> val result = doCompile(code)
> val endTime = System.nanoTime()
> def timeMs: Double = (endTime - startTime).toDouble / 1000000
> CodegenMetrics.METRIC_SOURCE_CODE_SIZE.update(code.body.length)
> CodegenMetrics.METRIC_COMPILATION_TIME.update(timeMs.toLong)
> logInfo(s"Code generated in $timeMs ms")
> result
> }
> })
> {code}
> In some specific situation, for example: a long term and spark tasks are
> unchanged, the size of cache maximumSize configuration is a better idea.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
