Hello, we send a lot of small jobs to Spark (up to 500 in a second). When profiling I see Throwable.getStackTrace() in the top of memory profiler which is caused by SparkContext.getCallSite - this is memory consuming.
we use Java API, I tried to call SparkContext.setCallSite("-") before submitting a job. After that I see less calls of Throwable.getStackTrace() but some of them are still there. I think it is because SparkContext.setCallSite sets thread local variable and in some cases SparkContext.getCallSite is called from a Spark thread. Is there any way to disable CallSite completely? We dont need this information and we want to improve performance by disabling CallSite. Thank You -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/SparkContext-getCallSite-is-in-the-top-of-profiler-by-memory-allocation-tp22716.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org