kejiqing created SPARK-23563: -------------------------------- Summary: make size fo cache in CodeGenerator configable Key: SPARK-23563 URL: https://issues.apache.org/jira/browse/SPARK-23563 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 2.3.0 Reporter: kejiqing
the cache in class org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator has a hard cod maxmunSize 100, current code is: {code:java} // scala private val cache = CacheBuilder.newBuilder() .maximumSize(100) .build( new CacheLoader[CodeAndComment, (GeneratedClass, Int)]() { override def load(code: CodeAndComment): (GeneratedClass, Int) = { val startTime = System.nanoTime() val result = doCompile(code) val endTime = System.nanoTime() def timeMs: Double = (endTime - startTime).toDouble / 1000000 CodegenMetrics.METRIC_SOURCE_CODE_SIZE.update(code.body.length) CodegenMetrics.METRIC_COMPILATION_TIME.update(timeMs.toLong) logInfo(s"Code generated in $timeMs ms") result } }) {code} In some specific situation, for example: a long term and spark tasks are unchanged, the size of cache maximumSize configuration is a better idea. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org