Hello! +1 on this feature. I see that it is targeted to be shipped with release 0.6.0. I know there are other priorities but if the community could consider shipping this with earlier releases it would be great (my customers don't like me messing up with what EMR provides by default....).
Thank you, James. On 2020/03/26 00:39:41, FO O <[email protected]> wrote: > Thank you folks for the fast response and work. > > Vinoth Chandar <[email protected]> escreveu no dia quarta, 25/03/2020 à(s) > 11:22: > > > btw Raymond already has a PR up here for this :) > > https://github.com/apache/incubator-hudi/pull/1440 > > > > On Mon, Mar 23, 2020 at 5:32 PM Shiyan Xu <[email protected]> > > wrote: > > > > > Seems like an abstract class would be good enough for generic use? > > > User can provide a list of `Transformer` then the abstract class just > > apply > > > all the way through the list. > > > The implementation can be minimal for this approach. > > > > > > On Mon, Mar 23, 2020 at 4:12 PM Vinoth Chandar <[email protected]> > > wrote: > > > > > > > sg. Filed https://issues.apache.org/jira/browse/HUDI-731 > > > > > > > > Someone looking to pick this? :). Its an nice feature to implement, > > that > > > > fits a good template.. > > > > > > > > ofc we can discuss this more here in parallel > > > > > > > > On Mon, Mar 23, 2020 at 8:31 AM FO O <[email protected]> wrote: > > > > > > > > > Thank you Vinoth. > > > > > > > > > > >"If you are talking about implementing support for chained calling > > of > > > > > multiple Transformers, within DeltaStreamer itself" > > > > > > > > > > Yes, chained calling support for transformers would be super helpful, > > > if > > > > > this discussion can be revived it would be great. > > > > > > > > > > I see this useful for folks using DMS transformer and that need some > > > kind > > > > > of transformation before the DMS transformer adds the op filed for > > > > initial > > > > > load or when loading the CDC. In the meantime, I will create a custom > > > > > transformer. > > > > > > > > > > Thanks again, > > > > > -F. > > > > > > > > > > > > > > > Vinoth Chandar <[email protected]> escreveu no dia domingo, > > 22/03/2020 > > > > > à(s) > > > > > 20:58: > > > > > > > > > > > Hi F, > > > > > > > > > > > > The Transformer interface allows you to basically plugin anything > > > that > > > > > > takes a DataFrame and returns a transformed DataFrame. Does that > > > help? > > > > > > If you are talking about implementing support for chained calling > > of > > > > > > multiple Transformers, within DeltaStreamer itself..It has been > > > > discussed > > > > > > before. > > > > > > And we can revive that conversation. > > > > > > > > > > > > Thanks > > > > > > Vinoth > > > > > > > > > > > > On Sat, Mar 21, 2020 at 5:14 PM FO O <[email protected]> wrote: > > > > > > > > > > > > > Hi team! > > > > > > > > > > > > > > My use case would benefit from running a SQL transformer followed > > > by > > > > > the > > > > > > > DMS transformer. > > > > > > > > > > > > > > It seems my best options is to create a new transformer that is > > > based > > > > > on > > > > > > > the current DMS transformer and add the additional > > transformation I > > > > > need > > > > > > > (add new columns, concatenate fields). > > > > > > > > > > > > > > Wanted to see if there are additional recommendations that I > > should > > > > > > > consider instead of this one. > > > > > > > > > > > > > > Thank you, > > > > > > > F > > > > > > > > > > > > > > > > > > > > > > > > > > > >
