Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/16774
Thanks for getting back to this @MLnick. The only difference with daemon
threads is they won't prevent the JVM from shutting down - which is what we
would want in any case. The name is a bit misleading, but the JVM will not
shutdown until all non-daemon threads have finished, so if the main thread is
done and only daemon threads are still running, then the JVM will shutdown
immediately. This is what we would want here because if the thread invoking
CrossValidation is interrupted, then we do not want to wait for all the worker
threads to finish, we want them to stop immediately. It doesn't matter if the
work is incomplete because unless CrossValidation has run it's course the
results are not really useful, so graceful shutdown is not really needed here.
By cleanup, I mean the threadpool should be scoped to the `fit` method, not
the class instance, so any resources wouldn't be hanging around after calling
`fit` once the threadpool went out of scope - you don't really need to cleanup
explicitly. We can still do this without using the factory.
I guess the reason for me using the factory pattern was attempting to be as
flexible as possible and still allow using the `numParallelEval` param with a
custom executor. My thinking was thatcompletley maybe `numParallelEval` is a
user level param and the `ExecutorService` would be configured on some higher
level. Since none of this is in use today and it could be a completely
different when it does, it would make sense to just make the API simple like
you mentioned until it can be better hashed out. So I'll redo this with using
`def setExecutorService(executorService: ExecutorService): Unit` - sorry for
the long-winded response!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
