btw Raymond already has a PR up here for this :) https://github.com/apache/incubator-hudi/pull/1440
On Mon, Mar 23, 2020 at 5:32 PM Shiyan Xu <[email protected]> wrote: > Seems like an abstract class would be good enough for generic use? > User can provide a list of `Transformer` then the abstract class just apply > all the way through the list. > The implementation can be minimal for this approach. > > On Mon, Mar 23, 2020 at 4:12 PM Vinoth Chandar <[email protected]> wrote: > > > sg. Filed https://issues.apache.org/jira/browse/HUDI-731 > > > > Someone looking to pick this? :). Its an nice feature to implement, that > > fits a good template.. > > > > ofc we can discuss this more here in parallel > > > > On Mon, Mar 23, 2020 at 8:31 AM FO O <[email protected]> wrote: > > > > > Thank you Vinoth. > > > > > > >"If you are talking about implementing support for chained calling of > > > multiple Transformers, within DeltaStreamer itself" > > > > > > Yes, chained calling support for transformers would be super helpful, > if > > > this discussion can be revived it would be great. > > > > > > I see this useful for folks using DMS transformer and that need some > kind > > > of transformation before the DMS transformer adds the op filed for > > initial > > > load or when loading the CDC. In the meantime, I will create a custom > > > transformer. > > > > > > Thanks again, > > > -F. > > > > > > > > > Vinoth Chandar <[email protected]> escreveu no dia domingo, 22/03/2020 > > > à(s) > > > 20:58: > > > > > > > Hi F, > > > > > > > > The Transformer interface allows you to basically plugin anything > that > > > > takes a DataFrame and returns a transformed DataFrame. Does that > help? > > > > If you are talking about implementing support for chained calling of > > > > multiple Transformers, within DeltaStreamer itself..It has been > > discussed > > > > before. > > > > And we can revive that conversation. > > > > > > > > Thanks > > > > Vinoth > > > > > > > > On Sat, Mar 21, 2020 at 5:14 PM FO O <[email protected]> wrote: > > > > > > > > > Hi team! > > > > > > > > > > My use case would benefit from running a SQL transformer followed > by > > > the > > > > > DMS transformer. > > > > > > > > > > It seems my best options is to create a new transformer that is > based > > > on > > > > > the current DMS transformer and add the additional transformation I > > > need > > > > > (add new columns, concatenate fields). > > > > > > > > > > Wanted to see if there are additional recommendations that I should > > > > > consider instead of this one. > > > > > > > > > > Thank you, > > > > > F > > > > > > > > > > > > > > >
