These can be externalized as PTransforms. E.g. the generic ETL
pipeline could just be written
pipeline
.appy(SomeExtractPTransform()) // aka Source
.apply(SomeTransformPTransform())
.apply(SomeLoadPTransform()) // aka Sink
Any and all of these PTransforms may be composite (i.e .composed of
smaller transforms). But perhaps I'm not quite following what you're
trying to say.
On Fri, Oct 11, 2019 at 11:11 AM Steve973 <[email protected]> wrote:
>
> The real benefit of a good ETL framework is being able to externalize your
> extraction and transformation mappings. If I didn't have to write that part,
> that would be really cool!
>
> On Fri, Oct 11, 2019 at 1:28 PM Robert Bradshaw <[email protected]> wrote:
>>
>> I would like to call out that Beam itself can be directly used for
>> ETL, no extra framework required (not to say that both of these
>> frameworks don't provide additional value, e.g. GUI-style construction
>> of pipelines).
>>
>>
>> On Fri, Oct 11, 2019 at 9:29 AM Ryan Skraba <[email protected]> wrote:
>> >
>> > Hello! Talend has a big data ETL product in the cloud called Pipeline
>> > Designer, entirely powered by Beam. There was a talk at Beam Summit
>> > 2018 (https://www.youtube.com/watch?v=1AlEGUtiQek), but unfortunately
>> > the live demo wasn't captured in the video. You can find other videos
>> > of Pipeline Designer online to see if it might fit your needs, and
>> > there is a free trial! Depending on how your work project is
>> > oriented, it may be of interest.
>> >
>> > Best regards, Ryan
>> >
>> > On Fri, Oct 11, 2019 at 12:26 PM Steve973 <[email protected]> wrote:
>> > >
>> > > Thank you for your reply. I will check it out! I'm in the evaluation
>> > > phase, especially since I have some time before I have to implement all
>> > > of this.
>> > >
>> > > On Fri, Oct 11, 2019 at 3:25 AM Dan <[email protected]> wrote:
>> > >>
>> > >> I'm not sure if this will help but kettle runs on beam too.
>> > >>
>> > >> https://github.com/mattcasters/kettle-beam
>> > >>
>> > >> https://youtu.be/vgpGrQJnqkM
>> > >>
>> > >> Depends on your use case but kettle rocks for etl.
>> > >>
>> > >> Dan
>> > >>
>> > >> Sent from my phone
>> > >>
>> > >> On Thu, 10 Oct 2019, 10:12 pm Steve973, <[email protected]> wrote:
>> > >>>
>> > >>> Hello, all. I still have not been given the tasking to convert my
>> > >>> work project to use Beam, but it is still something that I am looking
>> > >>> to do in the fairly near future. Our data workflow consists of ingest
>> > >>> and transformation, and I was hoping that there are ETL frameworks
>> > >>> that work well with Beam. Does anyone have some recommendations and
>> > >>> maybe some samples that show how people might use and ETL framework
>> > >>> with Beam?
>> > >>>
>> > >>> Thanks in advance and have a great day!
