hi James, glad that this could be helpful. Would it be possible for you to build the jar off master branch and use it in the meantime? You could install the jars to the EMR nodes during bootstrap.
On Wed, Jul 15, 2020 at 8:11 AM James Walter <[email protected]> wrote: > Hello! +1 on this feature. I see that it is targeted to be shipped with > release 0.6.0. I know there are other priorities but if the community could > consider shipping this with earlier releases it would be great (my > customers don't like me messing up with what EMR provides by default....). > > Thank you, James. > On 2020/03/26 00:39:41, FO O <[email protected]> wrote: > > Thank you folks for the fast response and work. > > > > Vinoth Chandar <[email protected]> escreveu no dia quarta, 25/03/2020 > à(s) > > 11:22: > > > > > btw Raymond already has a PR up here for this :) > > > https://github.com/apache/incubator-hudi/pull/1440 > > > > > > On Mon, Mar 23, 2020 at 5:32 PM Shiyan Xu <[email protected] > > > > > wrote: > > > > > > > Seems like an abstract class would be good enough for generic use? > > > > User can provide a list of `Transformer` then the abstract class just > > > apply > > > > all the way through the list. > > > > The implementation can be minimal for this approach. > > > > > > > > On Mon, Mar 23, 2020 at 4:12 PM Vinoth Chandar <[email protected]> > > > wrote: > > > > > > > > > sg. Filed https://issues.apache.org/jira/browse/HUDI-731 > > > > > > > > > > Someone looking to pick this? :). Its an nice feature to implement, > > > that > > > > > fits a good template.. > > > > > > > > > > ofc we can discuss this more here in parallel > > > > > > > > > > On Mon, Mar 23, 2020 at 8:31 AM FO O <[email protected]> wrote: > > > > > > > > > > > Thank you Vinoth. > > > > > > > > > > > > >"If you are talking about implementing support for chained > calling > > > of > > > > > > multiple Transformers, within DeltaStreamer itself" > > > > > > > > > > > > Yes, chained calling support for transformers would be super > helpful, > > > > if > > > > > > this discussion can be revived it would be great. > > > > > > > > > > > > I see this useful for folks using DMS transformer and that need > some > > > > kind > > > > > > of transformation before the DMS transformer adds the op filed > for > > > > > initial > > > > > > load or when loading the CDC. In the meantime, I will create a > custom > > > > > > transformer. > > > > > > > > > > > > Thanks again, > > > > > > -F. > > > > > > > > > > > > > > > > > > Vinoth Chandar <[email protected]> escreveu no dia domingo, > > > 22/03/2020 > > > > > > à(s) > > > > > > 20:58: > > > > > > > > > > > > > Hi F, > > > > > > > > > > > > > > The Transformer interface allows you to basically plugin > anything > > > > that > > > > > > > takes a DataFrame and returns a transformed DataFrame. Does > that > > > > help? > > > > > > > If you are talking about implementing support for chained > calling > > > of > > > > > > > multiple Transformers, within DeltaStreamer itself..It has been > > > > > discussed > > > > > > > before. > > > > > > > And we can revive that conversation. > > > > > > > > > > > > > > Thanks > > > > > > > Vinoth > > > > > > > > > > > > > > On Sat, Mar 21, 2020 at 5:14 PM FO O <[email protected]> > wrote: > > > > > > > > > > > > > > > Hi team! > > > > > > > > > > > > > > > > My use case would benefit from running a SQL transformer > followed > > > > by > > > > > > the > > > > > > > > DMS transformer. > > > > > > > > > > > > > > > > It seems my best options is to create a new transformer that > is > > > > based > > > > > > on > > > > > > > > the current DMS transformer and add the additional > > > transformation I > > > > > > need > > > > > > > > (add new columns, concatenate fields). > > > > > > > > > > > > > > > > Wanted to see if there are additional recommendations that I > > > should > > > > > > > > consider instead of this one. > > > > > > > > > > > > > > > > Thank you, > > > > > > > > F > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
