[
https://issues.apache.org/jira/browse/SPARK-8896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14618764#comment-14618764
]
Aniket Bhatnagar commented on SPARK-8896:
-----------------------------------------
As per the documentation (scala docs), only one SparkContext per JVM is
allowed. However, no such documentation exists for StreamingContext.
SparkContext makes an effort to ensure only one SparkContext instance exists in
the JVM using contextBeingConstructed variable. However, no such effort is made
in StreamingContext. This lead me to believe that multiple StreamingContexts
are allowed in a JVM.
I can understand why only one SparkContext is allowed per JVM (global state, et
al), but I don't think thats true for StreamingContext. There maybe genuine use
cases to have 2 StreamingContext instances using the same SparkContext to
leverage the same workers and have different batch intervals.
> StreamingSource should choose a unique name
> -------------------------------------------
>
> Key: SPARK-8896
> URL: https://issues.apache.org/jira/browse/SPARK-8896
> Project: Spark
> Issue Type: Bug
> Components: Streaming
> Affects Versions: 1.4.0
> Reporter: Aniket Bhatnagar
>
> If 2 instances of StreamingContext are created and run using the same
> SparkContext, it results the following exception in the logs and causes the
> latter StreamingContext's metrics to go unreported.
> [ForkJoinPool-2-worker-7] [info] o.a.s.m.MetricsSystem -Metrics already
> registered
> java.lang.IllegalArgumentException: A metric named
> <AppName>.StreamingMetrics.streaming.lastReceivedBatch_processingEndTime
> already exists
> at
> com.codahale.metrics.MetricRegistry.register(MetricRegistry.java:91)~[metrics-core-3.1.0.jar:3.1.0]
> at
> com.codahale.metrics.MetricRegistry.registerAll(MetricRegistry.java:385)~[metrics-core-3.1.0.jar:3.1.0]
> at
> com.codahale.metrics.MetricRegistry.register(MetricRegistry.java:85)~[metrics-core-3.1.0.jar:3.1.0]
> at
> org.apache.spark.metrics.MetricsSystem.registerSource(MetricsSystem.scala:148)
> ~[spark-core_2.11-1.4.0.jar:1.4.0]
> at
> org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:199)
> [spark-streaming_2.11-1.4.0.jar:1.4.0]
> at
> org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:71)
> [spark-streaming_2.11-1.4.0.jar:1.4.0]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
