Github user Astralidea commented on the issue:
https://github.com/apache/spark/pull/15588
@jerryshao I agree waiting time waste the CPU time, and I have tested
@lw-lin mentioned feasible solution is not work in my environment.
OK, If there is no better solution or advise. I will
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/15588
I think this fix cannot really handle this imbalance receiver allocation
problem, also blindly waste the CPU time.
What @lw-lin mentioned is a feasible solution to wait for executors to be
Github user Astralidea commented on the issue:
https://github.com/apache/spark/pull/15588
@lw-lin
spark.scheduler.minRegisteredResourcesRatio does not work.
The reason it may could be I use mesos and it run executor not through
driver.
but I still need to make sure it have
Github user Astralidea commented on the issue:
https://github.com/apache/spark/pull/15588
@lw-lin
Thanks for you reply. In my private cluster running spark is a little
different. (I start drivr & executor by myself)
I had try maxRegisteredWaitingTime, but I had not try
minRegi
Github user lw-lin commented on the issue:
https://github.com/apache/spark/pull/15588
Spark Streaming would do a very simple dummy job ensure that all slaves
have registered before scheduling the `Receiver`s; please see
https://github.com/apache/spark/blob/v2.0.0/streaming/src/main/sc
Github user Astralidea commented on the issue:
https://github.com/apache/spark/pull/15588
@srowen But in my cluster I tested 10 times. 9 times successed, 1 time
failed.
Why not necessary? receiver balance scheduler affect performance.
If new executor delay add to driver. rece
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/15588
I don't think it's necessarily true that you want to wait for all receivers
to begin processing. This change won't work in any event.
---
If your project is set up for it, you can reply to this emai
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15588
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feat
