I don't have a Dask cluster yet, but I'm interested in taking advantage of it for ML tasks. My use case would be bursting a lot of ML jobs into a Dask cluster all at once. >From what I understand, Dask clusters utilize caching to help speed up jobs so I don't know if it makes sense to launch a Dask cluster for every single ML job. Conceivably, I could just have a single Dask worker running 24/7 and when its time to burst k8 could autoscale the Dask workers as more ML jobs are launched into the Dask cluster?
On Fri, Apr 27, 2018 at 10:35 PM Daniel Imberman <[email protected]> wrote: > Hi Kyle, > > So you have a static Dask cluster running your k8s cluster? Is there any > reason you wouldn't just launch the Dask cluster for the job you're running > and then tear it down? I feel like with k8s the elasticity is one of the > main benefits. > > On Fri, Apr 27, 2018 at 12:32 PM Kyle Hamlin <[email protected]> wrote: > > > Hi all, > > > > If I have a Kubernetes cluster running in DCOC and a Dask cluster running > > in that same Kubernetes cluster is it possible/does it makes sense to use > > the KubernetesExecutor to launch tasks into the Dask cluster (these are > ML > > jobs with sklearn)? I feel like there is a bit of inception going on here > > in my mind and I just want to make sure a setup like this makes sense? > > Thanks in advance for anyone's input! > > > -- Kyle Hamlin
