Hey Vinoth, Thanks. Since you mention you do the same at Uber, is there a use for keeping the log forever? Or is it just more practical to maintain two tables rather than coordinate an off-peak slot to run compaction on the read-optimized view?
Rishan On Wed, May 15, 2019 at 12:05 AM Vinoth Chandar <[email protected]> wrote: > Hi Roshan, > > Good point. Actually the incremental view + either read-optimized/realtime > view can provide similar functionality. > However, I think Minh wanted to keep a log forever. When using just a > single Hudi dataset, once the compactor runs or the cleaning happens, the > log is compacted away. > Does that make sense? > > Thanks > Vinoth > > On Tue, May 14, 2019 at 2:19 AM Roshan Nair (Data Platform) > <[email protected]> wrote: > > > Hi Minh/Vinoth, > > > > I'm curious about what use cases having two tables addresses. I'm > assuming > > here the two tables you mention are the read-optimized (COW) table, and > an > > uncompacted write optimized (MOR) table. > > > > Hudi already provides two views (read-optimized and write-optimized) on > the > > same table, so what use cases require splitting this into two different > > hudi tables? > > > > Roshan > > > > On Tue, May 7, 2019 at 8:58 PM Vinoth Chandar <[email protected]> wrote: > > > > > Thanks for starting the thread, Minh! > > > > > > We do the same thing at Uber actually. Its handy to join these two at > > times > > > and its a common pattern. > > > so curious to know what others think? > > > > > > DeltaStreamer option seems like a good idea. Some implementation > > > considerations on how we configure this second table etc.. > > > but we can figure that out on the PR/JIRA. > > > > > > > Can we update both tables transactionally? This would be a nice > > > property to have. The current 2-job pattern does not support this. > > > It's achievable with some caveats. For e.g, you can write both to > > datasets, > > > then commit the second one only after first one succeeds. If second > > commit > > > fails, then we do restore/rollback first one. Note that some queries > may > > > have already picked up the first commit changes technically speaking > > (race > > > time window will be small). General support for this, needs more work > and > > > overlaying timelines etc... You are welcome to take this on if you are > > > interested. :) > > > > > > > Can we share the Avro logs? This might save some time as well > > > as achieving the transactionality mentioned above but it increases > > > complexity. > > > yes. it would change the core models and design a lot. In some cases, > the > > > logs may not even be the same across these tables. for e.g, if you take > > the > > > HBase data model, you might get new cells out of your change stream, > > which > > > is the raw change log . You can have the snapshot/row table have either > > > cells in the Avro log or full row images, depending on where you want > to > > > pay the cost of merge. let me know what you think. > > > > > > > > > > > > On Mon, May 6, 2019 at 10:19 PM Minh Pham <[email protected]> wrote: > > > > > > > Hi, > > > > > > > > A common pattern that I see is having 1 Kafka topic for data change > > > events > > > > and 2 Hudi ingestion job (1 in insert mode and 1 in upsert mode). > This > > > > creates 2 tables, 1 with all raw data change events and 1 with the > > latest > > > > snapshot of data. > > > > > > > > What do you guys think about adding support for as an option in > > > > DeltaStreamer? > > > > > > > > There are some complications to consider: > > > > - Can we update both tables transactionally? This would be a nice > > > property > > > > to have. The current 2-job pattern does not support this. > > > > - Can we share the Avro logs? This might save some time as well as > > > > achieving the transactionality mentioned above but it increases > > > complexity. > > > > > > > > Best, > > > > Minh > > > > > > > > > >
