Hi Prateek, Hung would be documenting what we have on Streaming, but that would happen over the next month.
Regarding Yarn, please refer to this document: https://gobblin. readthedocs.io/en/latest/user-guide/Gobblin-on-Yarn/ This talks about the architectural overview, and role of different sub-components. While a lot has changed since this doc was first written, this is still a great starting point. Regarding adding auto-scalability / elasticity component, you might want to look at YarnService, specifically the callbacks. I suspect, it will extend on top of it. Joel / Kadaan had run Gobblin on Yarn for a while before he moved to Gobblin on AWS. He found that elasticity based on worker task queue size was a good signal for scaling up / down in AWS mode. @Joel, Prateek and Deepak are using Gobblin on Yarn and are looking to add elasticity to it based on load. Do you have any thoughts to share? Regards, Abhishek On Thu, Mar 22, 2018 at 7:05 AM, Prateek Gupta <prateek.gup...@myntra.com> wrote: > Hi Abhishek, > > As per our discussion in yesterday's video conference, please share the > design documents for YARN auto-scaling and streaming. > > Thanks, > Prateek Gupta
