Hi Shiyan, Thanks for rasing this thread up again and sharing your thoughts. They are valuable.
Regarding the date-time specific transform, there is an issue[1] that describes this business requirement. Best, Vino Shiyan Xu <[email protected]> 于2020年2月24日周一 上午7:22写道： > Late to the party. :P > > I really favor the idea of built-in support enrichment. It is a very common > case where we want to set datetime fields for partition path. We could have > a built-in support to normalize ISO format / unix timestamp. For example > `HourlyPartitionTransformer` will normalize whatever field user specified > as partition path. Let's say user set `create_ts` as partition path field, > the transfromer will apply change create_ts => _hoodie_partition_path > > > - 2020-02-23T22:41:42.123456789Z => 2020/02/23/22 > - 1582497702.123456789 => 2020/02/23/22 > > Does that make sense? If so, I may file a jira for this. > > As for FilterTransformer or FlatMapTransformer which is designed for > generic purpose, they seem to belong to Spark or Flink's realm. > You can do these 2 transformation with Spark Dataset now. Or once > decoupled from Spark, you'll probably have an abstract Dataset class > to perform engine-agnostic transformation > > My understanding of transformer in HUDI is more specifically purposed, > where the underlying transformation is handled by the actual > processing engine (Spark or Flink) > > > On Tue, Feb 18, 2020 at 11:00 AM Vinoth Chandar <[email protected]> wrote: > > > Thanks Hamid and Vinoyang for the great discussion > > > > On Fri, Feb 14, 2020 at 5:18 AM vino yang <[email protected]> wrote: > > > > > I have filed a Jira issue[1] to track this work. > > > > > > [1]: https://issues.apache.org/jira/browse/HUDI-613 > > > > > > vino yang <[email protected]> 于2020年2月13日周四 下午9:51写道： > > > > > > > Hi hamid, > > > > > > > > Agree with your opinion. > > > > > > > > Let's move forward step by step. > > > > > > > > Will file an issue to track refactor about Transformer. > > > > > > > > Best, > > > > Vino > > > > > > > > hamid pirahesh <[email protected]> 于2020年2月13日周四 下午6:38写道： > > > > > > > >> I think it is a good idea to decouple the transformer from spark so > > > that > > > >> it can be used with other flow engines. > > > >> Once you do that, then it is worth considering a much bigger play > > rather > > > >> than another incremental play. > > > >> Given the scale of Hudi, we need to look at airflow, particularly in > > the > > > >> context of what google is doing with Composer, addressing > autoscaling, > > > >> scheduleing, monitoring, etc. > > > >> You need all of that to manage a serious tetl/elt flow. > > > >> > > > >> On Thu, Feb 6, 2020 at 8:25 PM vino yang <[email protected]> > > wrote: > > > >> > > > >> > Currently, Hudi has a component that has not been widely used: > > > >> Transformer. > > > >> > As we all know, before the original data fell into the data lake, > a > > > very > > > >> > common operation is data preprocessing and ETL. This is also the > > most > > > >> > common use scenario of many computing engines, such as Flink and > > > Spark. > > > >> Now > > > >> > that Hudi has taken advantage of the power of the computing > engine, > > it > > > >> can > > > >> > also naturally take advantage of its ability of data > preprocessing. > > We > > > >> can > > > >> > refactor the Transformer to make it become more flexible. To > > > summarize, > > > >> we > > > >> > can refactor from the following aspects: > > > >> > > > > >> > - Decouple Transformer from Spark > > > >> > - Enrich the Transformer and provide built-in transformer > > > >> > - Support Transformer-chain > > > >> > > > > >> > For the first point, the Transformer interface is tightly coupled > > with > > > >> > Spark in design, and it contains a Spark-specific context. This > > makes > > > it > > > >> > impossible for us to take advantage of the transform capabilities > > > >> provided > > > >> > by other engines (such as Flink) after supporting multiple > engines. > > > >> > Therefore, we need to decouple it from Spark in design. > > > >> > > > > >> > For the second point, we can enhance the Transformer and provide > > some > > > >> > out-of-the-box Transformers, such as FilterTransformer, > > > >> FlatMapTrnasformer, > > > >> > and so on. > > > >> > > > > >> > For the third point, the most common pattern for data processing > is > > > the > > > >> > pipeline model, and the common implementation of the pipeline > model > > is > > > >> the > > > >> > responsibility chain model, which can be compared to the Apache > > > commons > > > >> > chain[1], combining multiple Transformers can make data-processing > > > >> become > > > >> > more flexible and expandable. > > > >> > > > > >> > If we enhance the capabilities of Transformer components, Hudi > will > > > >> provide > > > >> > richer data processing capabilities based on the computing engine. > > > >> > > > > >> > What do you think? > > > >> > > > > >> > Any opinions and feedback are welcome and appreciated. > > > >> > > > > >> > Best, > > > >> > Vino > > > >> > > > > >> > [1]: https://commons.apache.org/proper/commons-chain/ > > > >> > > > > >> > > > > > > > > > >
