Hi Suneel, Thank you for your suggestion, let me clarify.
*The context of this email is that we are evaluating how to implement a Stream Delta writer base on Flink.* About the discussion between me, Taher and Vinay, those are just some trivial details in the preparation of the document, and the discussion is also based on mail. When we don't have the first draft, discussing the details on the mailing list may confuse others and easily deviate from the topic. Our initial plan was to facilitate community discussions and reviews when we had a draft of the documentation available to the community. Best, Vino Suneel Marthi <[email protected]> 于2019年8月2日周五 下午10:37写道： > Please keep all discussions to Mailing lists here - no offline discussions > please. > > On Fri, Aug 2, 2019 at 10:22 AM vino yang <[email protected]> wrote: > > > Hi guys, > > > > Currently, I, Taher and Vinay are working on issue HUDI-184.[1] > > > > As a first step, we are discussing the design doc. > > > > After diving into the code, We listed some relevant classes about the > Spark > > delta writer. > > > > - module: hoodie-utilities > > > > com.uber.hoodie.utilities.deltastreamer.HoodieDeltaStreamer > > com.uber.hoodie.utilities.deltastreamer.DeltaSyncService > > com.uber.hoodie.utilities.deltastreamer.SourceFormatAdapter > > com.uber.hoodie.utilities.schema.SchemaProvider > > com.uber.hoodie.utilities.transform.Transformer > > > > - module: hoodie-client > > > > com.uber.hoodie.HoodieWriteClient (to commit compaction) > > > > > > The fact is *hoodie-utilities* depends on *hoodie-client*, however, > > *hoodie-client* is also not a pure Hudi component, it also depends on > Spark > > lib. > > > > So I propose hoodie should provide a pure hoodie-client and decouple with > > Spark. Then Flink and Spark modules should depend on it. > > > > Moreover, based on the old discussion[2], we all agree that Spark is not > > the only choice for Hudi, it could also be Flink/Beam. > > > > IMO, We should decouple Hudi from Spark at the height of the project, > > including but not limited to module splitting and renaming. > > > > Not sure if this requires a HIP to drive. > > > > We should first listen to the opinions of the community. Any ideas and > > suggestions are welcome and appreciated. > > > > Best, > > Vino > > > > [1]: https://issues.apache.org/jira/browse/HUDI-184?filter=-1 > > [2]: > > > > > https://lists.apache.org/api/source.lua/1533de2d4cd4243fa9e8f8bf057ffd02f2ac0bec7c7539d8f72166ea@%3Cdev.hudi.apache.org%3E > > >
