The real benefit of a good ETL framework is being able to externalize your extraction and transformation mappings. If I didn't have to write that part, that would be really cool!
On Fri, Oct 11, 2019 at 1:28 PM Robert Bradshaw <[email protected]> wrote: > I would like to call out that Beam itself can be directly used for > ETL, no extra framework required (not to say that both of these > frameworks don't provide additional value, e.g. GUI-style construction > of pipelines). > > > On Fri, Oct 11, 2019 at 9:29 AM Ryan Skraba <[email protected]> wrote: > > > > Hello! Talend has a big data ETL product in the cloud called Pipeline > > Designer, entirely powered by Beam. There was a talk at Beam Summit > > 2018 (https://www.youtube.com/watch?v=1AlEGUtiQek), but unfortunately > > the live demo wasn't captured in the video. You can find other videos > > of Pipeline Designer online to see if it might fit your needs, and > > there is a free trial! Depending on how your work project is > > oriented, it may be of interest. > > > > Best regards, Ryan > > > > On Fri, Oct 11, 2019 at 12:26 PM Steve973 <[email protected]> wrote: > > > > > > Thank you for your reply. I will check it out! I'm in the evaluation > phase, especially since I have some time before I have to implement all of > this. > > > > > > On Fri, Oct 11, 2019 at 3:25 AM Dan <[email protected]> wrote: > > >> > > >> I'm not sure if this will help but kettle runs on beam too. > > >> > > >> https://github.com/mattcasters/kettle-beam > > >> > > >> https://youtu.be/vgpGrQJnqkM > > >> > > >> Depends on your use case but kettle rocks for etl. > > >> > > >> Dan > > >> > > >> Sent from my phone > > >> > > >> On Thu, 10 Oct 2019, 10:12 pm Steve973, <[email protected]> wrote: > > >>> > > >>> Hello, all. I still have not been given the tasking to convert my > work project to use Beam, but it is still something that I am looking to do > in the fairly near future. Our data workflow consists of ingest and > transformation, and I was hoping that there are ETL frameworks that work > well with Beam. Does anyone have some recommendations and maybe some > samples that show how people might use and ETL framework with Beam? > > >>> > > >>> Thanks in advance and have a great day! >
