[
https://issues.apache.org/jira/browse/SPARK-13626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15179276#comment-15179276
]
Apache Spark commented on SPARK-13626:
--------------------------------------
User 'vanzin' has created a pull request for this issue:
https://github.com/apache/spark/pull/11510
> SparkConf deprecation log messages are printed multiple times
> -------------------------------------------------------------
>
> Key: SPARK-13626
> URL: https://issues.apache.org/jira/browse/SPARK-13626
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, SQL
> Affects Versions: 2.0.0
> Reporter: Marcelo Vanzin
> Priority: Minor
>
> I noticed that if I have a deprecated config in my spark-defaults.conf, I'll
> see multiple warnings when running, for example, spark-shell. I collected the
> backtrace from when the messages are printed, and here's a few instances. The
> first one is the only one I expect to be printed.
> {noformat}
> java.lang.Exception:
> ...
> at org.apache.spark.SparkConf.<init>(SparkConf.scala:53)
> at org.apache.spark.repl.Main$.<init>(Main.scala:30)
> {noformat}
> The following ones are causing duplicate log messages and we should clean
> those up:
> {noformat}
> java.lang.Exception:
> at
> org.apache.spark.SparkConf$$anonfun$logDeprecationWarning$2.apply(SparkConf.scala:682)
> ...
> at org.apache.spark.SparkConf.<init>(SparkConf.scala:53)
> at org.apache.spark.repl.Main$.createSparkContext(Main.scala:82)
> {noformat}
> {noformat}
> java.lang.Exception:
> at
> org.apache.spark.SparkConf$$anonfun$logDeprecationWarning$2.apply(SparkConf.scala:682)
> ...
> at org.apache.spark.SparkConf.setAll(SparkConf.scala:139)
> at org.apache.spark.SparkConf.clone(SparkConf.scala:358)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:368)
> at org.apache.spark.repl.Main$.createSparkContext(Main.scala:98)
> {noformat}
> There are also a few more caused by the use of {{SparkConf.clone()}}.
> {noformat}
> java.lang.Exception:
> at
> org.apache.spark.SparkConf$$anonfun$logDeprecationWarning$2.apply(SparkConf.scala:682)
> ...
> at org.apache.spark.SparkConf.<init>(SparkConf.scala:59)
> at org.apache.spark.SparkConf.<init>(SparkConf.scala:53)
> at
> org.apache.spark.deploy.SparkHadoopUtil.<init>(SparkHadoopUtil.scala:48)
> {noformat}
> {noformat}
> java.lang.Exception:
> at
> org.apache.spark.SparkConf$$anonfun$logDeprecationWarning$2.apply(SparkConf.scala:682)
> ...
> at org.apache.spark.SparkConf.<init>(SparkConf.scala:53)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.<init>(HiveClientImpl.scala:93)
> at
> org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:238)
> {noformat}
> {noformat}
> java.lang.Exception:
> at
> org.apache.spark.SparkConf$$anonfun$logDeprecationWarning$2.apply(SparkConf.scala:682)
> ...
> at org.apache.spark.SparkConf.<init>(SparkConf.scala:53)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.<init>(HiveClientImpl.scala:93)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
