hi Chenya > As we know, streaming applications are long-running and need to secure all requested resources before starting to run. In most cases, they do not have a strong need to be queued, ordered, or preempted to wait to obtain or give back their resource.
You are right if the assumption is pure streaming cases, all long-running jobs, and the cluster has sufficient resources for all jobs. Maybe it is fair to say it is not a day 1 challenge. However, in my past experience, this is not always enough and will not be enough. When we operate large-scale Flink jobs, the major issues we were dealing with: resource utilization, resource contention, hot-spot, isolation, etc. We used to have tens of queues per cluster and shared by many users, and jobs have different priorities and high-priority jobs can make room by preempting lower priority ones. We have a customized node-score system in order to distribute pods more efficiently. As you see, resource queues, app-sorting, node-sorting, preemption, all play a role here. Also central job management, scheduling latency/throughput are also important. On K8s and Cloud, it brings more challenges. I guess one thing challenging and also interesting is how to do auto-scaling more efficiently. Sometimes we need a strategy to warm up resources on Cloud in order to fit new jobs in low latency. Most likely the scheduler can give some hints for that. This will be a fun part to explore too. With all being said, I do think a customized scheduler (instead of the pod-level scheduler - default-k8s-scheduler) will be necessary. On Tue, Jan 4, 2022 at 10:18 PM Chenya Zhang <[email protected]> wrote: > Hi Weiwei > > Thanks for sharing. I checked the video and for Alibaba's use case, they > have a mixed cluster for streaming and batch applications running with > Apache Flink. Our use case is different. We only use Apache Flink for > stream processing in physical clusters separate from Spark for batch > processing. > > As we know, streaming applications are long-running and need to secure all > requested resources before starting to run. In most cases, they do not have > a strong need to be queued, ordered, or preempted to wait to obtain or give > back their resource. > > I'm gathering more streaming use case requirements that could not be > satisfied by K8s namespace for resource quota management or other advanced > scheduling needs. Will keep this thread updated. > > Meanwhile, happy to hear more thoughts from you! > > Best, > Chenya > > On Tue, Jan 4, 2022 at 9:20 PM Weiwei Yang <[email protected]> wrote: > > > Hi Chenya > > > > The use case is similar, YK will play a big role there. Lots of features > > are relevant, such as queues, job ordering, user/group ACLs, preemption, > > over-subscription, and performance etc. > > Some of the basic functionalities are available in YK, some more needs to > > be built. > > Please take a look at the slides from the Alibaba Flink team, they have > > shared how they use YK to address their use cases. > > This was presented in ApacheConf: > > https://www.youtube.com/watch?v=4hghJCuZk5M > > > > On Tue, Jan 4, 2022 at 6:35 PM Chenya Zhang <[email protected] > > > > wrote: > > > > > Hey folks, > > > > > > We have some new streaming use cases with Apache Flink that could > > > potentially leverage YuniKorn for resource scheduling. > > > > > > The initial implementation is to use K8s namespace for resource quota > > > management. We are investigating what could be some strong benefits > > > switching to YuniKorn in streaming cases for long-running services. For > > > example: Job queueing, job ordering, resource reservation, user groups > > etc > > > all seem to be more desirable for batch use cases. > > > > > > Any thoughts or suggestions? > > > > > > Thanks, > > > Chenya > > > > > >
