Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/18760
@vanzin Hi Vanzin, thanks a lot for your comments.
Firstly answer your question about `Why isn't hold mode just calling
queue.put (blocking) instead of queue.offer (non-blocking)?`
In general scenario of the queue is full, we need a time that hold all
events push into the queue, here I use offer to control the `empty rate` in the
configuration. If here use `put(blocking)`, this will not relief the queue
blocking, and just hanging each post events.
Actually this patch is a internal fix patch for the event dropping problem
in Baidu internal env as I described in
[jira](https://issues.apache.org/jira/browse/SPARK-21560), glad to see
SPARK-18838 has been merged.
Here I had another thought -- how about I port SPARK-18838 to our product
env which solved the event dropping by the current patch, if it works well I
just close this. What do you two think about this? @cloud-fan @vanzin :)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
