Hi Ryan, Thanks for the introduction e-mail and welcome !
While ago I already proposed some potential new features to Livy. Especially I proposed to "decouple" Livy from Spark to be able to interact with different execution engines.
I'm planning to move forward on Livy PoCs. Looking forward to work together. Regards JB On 16/04/2019 17:50, Ryan Skraba wrote:
Hello! I haven't seen many "presentation" emails in the mailing list, but I thought I'd introduce myself and share how we're using Livy in production. I'm Ryan, a developer at Talend (not far from Alexey Romanenko and Jean-Baptiste Onofre), and we've just released Talend Pipeline Designer[1]. To simplify, we're a graphical front end in the cloud for designing data pipelines, and an "agent" on an edge node for running them. For us, Livy is a great match for running pipelines in that agent, and we've been using it for quite some time now with success! I believe we've pushed a few changes back to Livy (thanks Alexey!), but for the most part we're living in a forked repo at some arbitrary commit, with lots of modifications that are *mostly* only interesting to our jobs. I'm pretty excited to hear about the release (belated congratulations!) and I'm currently taking a look at rebasing our work and (1) identifying features that might be useful (e.g. around docker deployment, job log enhancements, some optional job argument preprocessing) and (2) getting off of our fork by isolating our very specific code elsewhere. I also see a LOT of interesting work in the PRs and JIRA, so I'm keenly interested in aligning back with the community! All my best, Ryan [1] For more info: https://www.talend.com/blog/2019/04/16/introducing-pipeline-designer-reinventing-data-integration/
