Hi Lance! Yes, we do the same. Specifically, we have multiple DAGs that share access to a Spark cluster through the use of Pools. By setting the pool size to say 4, we remove the possibility of some backfill swamping the Spark cluster. BTW, there were some bugs with over-subscription of pools. It's not a common occurrence, but it has been reported.
-s On Thu, May 19, 2016 at 9:37 PM, Lance Norskog <[email protected]> wrote: > How should we use pools in our dags? > > We do a lot of analytics queries and copying between databases. I've set up > pools for each database instance so that we avoid overloading instances > with queries. Is this the right approach? > > Thanks, > > -- > Lance Norskog > [email protected] > Redwood City, CA >
