[
https://issues.apache.org/jira/browse/SPARK-27647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16835259#comment-16835259
]
bettermouse edited comment on SPARK-27647 at 5/8/19 2:29 AM:
-------------------------------------------------------------
First, in DAGScheduler thread, Field dagScheduler.failedStages is updated.
Second DAGSchedulerSource mainly uses Metrics jar. This method use Gauge.
Metrics should call this method periodically, this is the second thread.
you can see Gauge class.
A gauge metric is an instantaneous reading of a particular value. To instrument
a queue's depth, for example:
final Queue<String> queue = new ConcurrentLinkedQueue<String>();
final Gauge<Integer> queueDepth = new Gauge<Integer>() {
public Integer getValue()
{ return queue.size(); }
};
it uses a ConcurrentLinkedQueue which is thread safe.in my local,I write a
example,
I find this method is called in Metrics thread.
so I think it is not safe.one thread is DAGScheduler thread. on thread is
Metrics thread
to report the message.the both read dagScheduler.failedStages field.
I mean it's not safe to read a variable(dagScheduler.failedStages) in other
thread.
was (Author: bettermouse):
First in DAGScheduler thread, Field dagScheduler.failedStages is updated.
Second DAGSchedulerSource mainly use Metrics jar.This method use Gauge.
Metrics should call this method periodically,this is second thread.
you can see Gauge class.
A gauge metric is an instantaneous reading of a particular value. To instrument
a queue's depth, for example:
final Queue<String> queue = new ConcurrentLinkedQueue<String>();
final Gauge<Integer> queueDepth = new Gauge<Integer>() {
public Integer getValue() {
return queue.size();
}
};
it uses a ConcurrentLinkedQueue which is thread safe.in my local,I write a
example,
I find this method is called in Metrics thread.
so I think it is not safe.one thread is DAGScheduler thread. on thread is
Metrics thread
to report the message.the both read dagScheduler.failedStages field.
> Metric Gauge not threadsafe
> ---------------------------
>
> Key: SPARK-27647
> URL: https://issues.apache.org/jira/browse/SPARK-27647
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.4.2
> Reporter: bettermouse
> Priority: Major
>
> when I read class DAGSchedulerSource,I find some Gauges may be not
> threadSafe.like
> metricRegistry.register(MetricRegistry.name("stage", "failedStages"), new
> Gauge[Int] {
> override def getValue: Int = dagScheduler.failedStages.size
> })
> this method may be called in other thread,but failedStages field is not
> thread safe
> filed runningStages,waitingStages have same problem
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
