Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/1723#discussion_r15726777
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/dstream/DStream.scala ---
@@ -106,6 +106,24 @@ abstract class DStream[T: ClassTag] (
/** Return the StreamingContext associated with this DStream */
def context = ssc
+ private[streaming] val RDD_NAME: String = "rddName";
+
+ @transient var name: String = null
+
+ /** Assign a name to this DStream */
+ def setName(_name: String) = {
+ name = _name
+ }
+
+ /* Find the creation callSite */
+ val creationSite = Utils.getCallSite
+
+ /* Store the creation callSite in threadlocal */
+ private[streaming] def setCallSite = {
+ ssc.sparkContext.setLocalProperty(name + Utils.CALL_SITE_SHORT,
creationSite.short)
+ ssc.sparkContext.setLocalProperty(name + Utils.CALL_SITE_LONG,
creationSite.long)
--- End diff --
What is the use of the "name"? You seem to create set name in each DStream,
then set local property using that name, and then the same name and subsequent
property gets read by RDDs generated in the thread. It doesnt seem like that
the "name" gets is even shown as part of the short/long callsite. Only used in
the property name. Wouldnt it have worked without the "name" in the property
name?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
