Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5124#discussion_r27308927
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -474,13 +477,13 @@ class SparkContext(config: SparkConf) extends Logging
with ExecutorAllocationCli
* Spark fair scheduler pool.
*/
def setLocalProperty(key: String, value: String) {
- if (localProperties.get() == null) {
- localProperties.set(new Properties())
+ if (localProperties.get().isEmpty) {
+ localProperties.set(Some(new Properties()))
--- End diff --
> If we can change DeveloperApi, I would like to change all properties =
null in SparkListenerStageSubmitted and SparkListenerJobStart to Option.
A nice advantage of returning an empty properties object rather than using
`Option` is that we don't need to break binary compatibility in these events.
It might be nice to add a note to these events' docstrings to clarify that the
field used to be nullable in earlier versions, since this issue might bite
users who write code against the newest version of Spark's listener API and get
NPEs when they run on older versions.
One gotcha is that the UI's JSONProtocol will continue to deserialize old
event streams such that these events can still contain `null` properties
fields. It wouldn't be too hard to modify the
[propertiesFromJson](https://github.com/hunglin/spark/blob/baea4fd9f6df0af466e51a8f19380f194ec502ae/core/src/main/scala/org/apache/spark/util/JsonProtocol.scala#L839)
and
`https://github.com/hunglin/spark/blob/baea4fd9f6df0af466e51a8f19380f194ec502ae/core/src/main/scala/org/apache/spark/util/JsonProtocol.scala#L411`
to also return empty Properties objects instead of nulls.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
