I am really interested to try a framework! We are using beam directly for our ETL in production... and it is working fine... We use it with airflow... and so far so good...
https://medium.com/@selsaber/data-etl-using-apache-beam-part-one-48ca1b30b10a But we write the code we need ourselves actually! On Sat, Oct 12, 2019 at 2:18 AM Robert Bradshaw <[email protected]> wrote: > These can be externalized as PTransforms. E.g. the generic ETL > pipeline could just be written > > pipeline > .appy(SomeExtractPTransform()) // aka Source > .apply(SomeTransformPTransform()) > .apply(SomeLoadPTransform()) // aka Sink > > Any and all of these PTransforms may be composite (i.e .composed of > smaller transforms). But perhaps I'm not quite following what you're > trying to say. > > On Fri, Oct 11, 2019 at 11:11 AM Steve973 <[email protected]> wrote: > > > > The real benefit of a good ETL framework is being able to externalize > your extraction and transformation mappings. If I didn't have to write > that part, that would be really cool! > > > > On Fri, Oct 11, 2019 at 1:28 PM Robert Bradshaw <[email protected]> > wrote: > >> > >> I would like to call out that Beam itself can be directly used for > >> ETL, no extra framework required (not to say that both of these > >> frameworks don't provide additional value, e.g. GUI-style construction > >> of pipelines). > >> > >> > >> On Fri, Oct 11, 2019 at 9:29 AM Ryan Skraba <[email protected]> wrote: > >> > > >> > Hello! Talend has a big data ETL product in the cloud called Pipeline > >> > Designer, entirely powered by Beam. There was a talk at Beam Summit > >> > 2018 (https://www.youtube.com/watch?v=1AlEGUtiQek), but unfortunately > >> > the live demo wasn't captured in the video. You can find other videos > >> > of Pipeline Designer online to see if it might fit your needs, and > >> > there is a free trial! Depending on how your work project is > >> > oriented, it may be of interest. > >> > > >> > Best regards, Ryan > >> > > >> > On Fri, Oct 11, 2019 at 12:26 PM Steve973 <[email protected]> wrote: > >> > > > >> > > Thank you for your reply. I will check it out! I'm in the > evaluation phase, especially since I have some time before I have to > implement all of this. > >> > > > >> > > On Fri, Oct 11, 2019 at 3:25 AM Dan <[email protected]> wrote: > >> > >> > >> > >> I'm not sure if this will help but kettle runs on beam too. > >> > >> > >> > >> https://github.com/mattcasters/kettle-beam > >> > >> > >> > >> https://youtu.be/vgpGrQJnqkM > >> > >> > >> > >> Depends on your use case but kettle rocks for etl. > >> > >> > >> > >> Dan > >> > >> > >> > >> Sent from my phone > >> > >> > >> > >> On Thu, 10 Oct 2019, 10:12 pm Steve973, <[email protected]> > wrote: > >> > >>> > >> > >>> Hello, all. I still have not been given the tasking to convert > my work project to use Beam, but it is still something that I am looking to > do in the fairly near future. Our data workflow consists of ingest and > transformation, and I was hoping that there are ETL frameworks that work > well with Beam. Does anyone have some recommendations and maybe some > samples that show how people might use and ETL framework with Beam? > >> > >>> > >> > >>> Thanks in advance and have a great day! > -- Soliman ElSaber Data Engineer www.mindvalley.com
