Hi Jeff,
the idea is to provide kind of agnostic API and being able to leverage
Flink and eventually some Spark "flavors".
With a "Livy API", we can also imagine other engines to support Livy
(I'm thinking about Google Dataflow, etc).
Regards
JB
On 16/04/2019 18:19, Jeff Zhang wrote:
Hi JB,
I notice the ticket of `decoupling` livy from spark. Which alternative
engine do you consider to adopt ?
Jean-Baptiste Onofré <j...@nanthrax.net> 于2019年4月17日周三 上午12:06写道：
Hi Ryan,
Thanks for the introduction e-mail and welcome !
While ago I already proposed some potential new features to Livy.
Especially I proposed to "decouple" Livy from Spark to be able to
interact with different execution engines.
I'm planning to move forward on Livy PoCs.
Looking forward to work together.
Regards
JB
On 16/04/2019 17:50, Ryan Skraba wrote:
Hello! I haven't seen many "presentation" emails in the mailing list,
but
I thought I'd introduce myself and share how we're using Livy in
production.
I'm Ryan, a developer at Talend (not far from Alexey Romanenko and
Jean-Baptiste Onofre), and we've just released Talend Pipeline
Designer[1]. To simplify, we're a graphical front end in the cloud for
designing data pipelines, and an "agent" on an edge node for running
them.
For us, Livy is a great match for running pipelines in that agent, and
we've been using it for quite some time now with success!
I believe we've pushed a few changes back to Livy (thanks Alexey!), but
for
the most part we're living in a forked repo at some arbitrary commit,
with
lots of modifications that are *mostly* only interesting to our jobs.
I'm pretty excited to hear about the release (belated congratulations!)
and
I'm currently taking a look at rebasing our work and (1) identifying
features that might be useful (e.g. around docker deployment, job log
enhancements, some optional job argument preprocessing) and (2) getting
off
of our fork by isolating our very specific code elsewhere.
I also see a LOT of interesting work in the PRs and JIRA, so I'm keenly
interested in aligning back with the community!
All my best, Ryan
[1] For more info:
https://www.talend.com/blog/2019/04/16/introducing-pipeline-designer-reinventing-data-integration/
