Hi, I'm doubtful that dynamic allocation / killing executors work in Standalone and YARN as far as web UI's concerned (perhaps it's just web UI).
I can successfully request as many executors as I want using sc.requestTotalExecutors and they show up nicely in the web UI as ACTIVE, but whenever I request sc.killExecutor(s) I can see the following in YARN logs: INFO YarnAllocator: Received 2 containers from YARN, launching executors on 2 of them. INFO YarnAllocator: Driver requested a total number of 3 executor(s). INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 3 executors. INFO YarnAllocator: Driver requested a total number of 2 executor(s). DEBUG YarnAllocator: Completed 1 containers DEBUG YarnAllocator: Finished processing 1 completed containers. Current running executor count: 2. INFO YarnAllocator: Driver requested a total number of 1 executor(s). INFO YarnAllocator: Will request 1 executor container(s), each with 1 core(s) and 1408 MB memory (including 384 MB of overhead) INFO YarnAllocator: Submitted 1 unlocalized container requests. DEBUG YarnAllocator: Completed 2 containers DEBUG YarnAllocator: Finished processing 2 completed containers. Current running executor count: 0. DEBUG YarnAllocator: Allocated containers: 1. Current executor count: 0. Cluster resources: <memory:5120, vCores:1>. INFO YarnAllocator: Launching container container_1475155700518_0003_01_000006 on host 192.168.65.1 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. INFO YarnAllocator: Driver requested a total number of 0 executor(s). DEBUG YarnAllocator: Completed 1 containers DEBUG YarnAllocator: Finished processing 1 completed containers. Current running executor count: 0. But the web UI shows all the executors as ACTIVE in Status column in Executors tab. Can anyone confirm that dynamic allocation works fine and web UI shows the current status of executors? What other information do you want me to offer to verify it. I'm doubtful that web UI shows what it's supposed to show regarding executors. Pozdrawiam, Jacek Laskowski ---- https://medium.com/@jaceklaskowski/ Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark Follow me at https://twitter.com/jaceklaskowski --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
