Marcelo Vanzin created SPARK-13626:
--------------------------------------
Summary: SparkConf deprecation log messages are printed multiple
times
Key: SPARK-13626
URL: https://issues.apache.org/jira/browse/SPARK-13626
Project: Spark
Issue Type: Bug
Components: Spark Core, SQL
Affects Versions: 2.0.0
Reporter: Marcelo Vanzin
Priority: Minor
I noticed that if I have a deprecated config in my spark-defaults.conf, I'll
see multiple warnings when running, for example, spark-shell. I collected the
backtrace from when the messages are printed, and here's a few instances. The
first one is the only one I expect to be printed.
{noformat}
java.lang.Exception:
...
at org.apache.spark.SparkConf.<init>(SparkConf.scala:53)
at org.apache.spark.repl.Main$.<init>(Main.scala:30)
{noformat}
The following ones are causing duplicate log messages and we should clean those
up:
{noformat}
java.lang.Exception:
at
org.apache.spark.SparkConf$$anonfun$logDeprecationWarning$2.apply(SparkConf.scala:682)
...
at org.apache.spark.SparkConf.<init>(SparkConf.scala:53)
at org.apache.spark.repl.Main$.createSparkContext(Main.scala:82)
{noformat}
{noformat}
java.lang.Exception:
at
org.apache.spark.SparkConf$$anonfun$logDeprecationWarning$2.apply(SparkConf.scala:682)
...
at org.apache.spark.SparkConf.setAll(SparkConf.scala:139)
at org.apache.spark.SparkConf.clone(SparkConf.scala:358)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:368)
at org.apache.spark.repl.Main$.createSparkContext(Main.scala:98)
{noformat}
There are also a few more caused by the use of {{SparkConf.clone()}}.
{noformat}
java.lang.Exception:
at
org.apache.spark.SparkConf$$anonfun$logDeprecationWarning$2.apply(SparkConf.scala:682)
...
at org.apache.spark.SparkConf.<init>(SparkConf.scala:59)
at org.apache.spark.SparkConf.<init>(SparkConf.scala:53)
at
org.apache.spark.deploy.SparkHadoopUtil.<init>(SparkHadoopUtil.scala:48)
{noformat}
{noformat}
java.lang.Exception:
at
org.apache.spark.SparkConf$$anonfun$logDeprecationWarning$2.apply(SparkConf.scala:682)
...
at org.apache.spark.SparkConf.<init>(SparkConf.scala:53)
at
org.apache.spark.sql.hive.client.HiveClientImpl.<init>(HiveClientImpl.scala:93)
at
org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:238)
{noformat}
{noformat}
java.lang.Exception:
at
org.apache.spark.SparkConf$$anonfun$logDeprecationWarning$2.apply(SparkConf.scala:682)
...
at org.apache.spark.SparkConf.<init>(SparkConf.scala:53)
at
org.apache.spark.sql.hive.client.HiveClientImpl.<init>(HiveClientImpl.scala:93)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
