Marcelo Vanzin created SPARK-13626: -------------------------------------- Summary: SparkConf deprecation log messages are printed multiple times Key: SPARK-13626 URL: https://issues.apache.org/jira/browse/SPARK-13626 Project: Spark Issue Type: Bug Components: Spark Core, SQL Affects Versions: 2.0.0 Reporter: Marcelo Vanzin Priority: Minor
I noticed that if I have a deprecated config in my spark-defaults.conf, I'll see multiple warnings when running, for example, spark-shell. I collected the backtrace from when the messages are printed, and here's a few instances. The first one is the only one I expect to be printed. {noformat} java.lang.Exception: ... at org.apache.spark.SparkConf.<init>(SparkConf.scala:53) at org.apache.spark.repl.Main$.<init>(Main.scala:30) {noformat} The following ones are causing duplicate log messages and we should clean those up: {noformat} java.lang.Exception: at org.apache.spark.SparkConf$$anonfun$logDeprecationWarning$2.apply(SparkConf.scala:682) ... at org.apache.spark.SparkConf.<init>(SparkConf.scala:53) at org.apache.spark.repl.Main$.createSparkContext(Main.scala:82) {noformat} {noformat} java.lang.Exception: at org.apache.spark.SparkConf$$anonfun$logDeprecationWarning$2.apply(SparkConf.scala:682) ... at org.apache.spark.SparkConf.setAll(SparkConf.scala:139) at org.apache.spark.SparkConf.clone(SparkConf.scala:358) at org.apache.spark.SparkContext.<init>(SparkContext.scala:368) at org.apache.spark.repl.Main$.createSparkContext(Main.scala:98) {noformat} There are also a few more caused by the use of {{SparkConf.clone()}}. {noformat} java.lang.Exception: at org.apache.spark.SparkConf$$anonfun$logDeprecationWarning$2.apply(SparkConf.scala:682) ... at org.apache.spark.SparkConf.<init>(SparkConf.scala:59) at org.apache.spark.SparkConf.<init>(SparkConf.scala:53) at org.apache.spark.deploy.SparkHadoopUtil.<init>(SparkHadoopUtil.scala:48) {noformat} {noformat} java.lang.Exception: at org.apache.spark.SparkConf$$anonfun$logDeprecationWarning$2.apply(SparkConf.scala:682) ... at org.apache.spark.SparkConf.<init>(SparkConf.scala:53) at org.apache.spark.sql.hive.client.HiveClientImpl.<init>(HiveClientImpl.scala:93) at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:238) {noformat} {noformat} java.lang.Exception: at org.apache.spark.SparkConf$$anonfun$logDeprecationWarning$2.apply(SparkConf.scala:682) ... at org.apache.spark.SparkConf.<init>(SparkConf.scala:53) at org.apache.spark.sql.hive.client.HiveClientImpl.<init>(HiveClientImpl.scala:93) {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org