Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/21502
@rxin this can reduce the problem since the destroy operation is invoked
when the execution of the query is terminated (in STS), instead of some time
later when the cleaner thread does it. In this way, when dynamic allocation is
enabled, it is very likely that during the delay introduced by waiting for the
cleaner thread to remove the blocks a lot of containers are killed, while if we
destroy it immediately after the query terminates, the containers are very
likely to not have been killed yet, so they can clean up the blocks.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
