I agree that we can merge RFC-46 feature branch to master at the end of this week to give leeway to 0.12.2.
On Tue, Dec 6, 2022 at 4:03 PM Alexey Kudinkin <ale...@onehouse.ai> wrote: > Thanks for bubbling this up, Siva! > > Merge has not happened yet. > > What you're saying makes sense to me -- after merging RFC-46 everyone will > likely have to rebase their changes since there it touches quite a few core > components, > so it'd make it harder to cherry-pick 0.12.2 changes. > > What do other folks think? > > On Tue, Dec 6, 2022 at 2:06 PM Sivabalan <n.siv...@gmail.com> wrote: > > > Hey folks, > > Has this happened already? If not, with 0.12.2 underway, is it > > possible to delay this until end of this week. Not sure if code freeze > for > > 0.12.2 will be this Friday, but atleast we can do our best and ask RM to > > cherry-pick the commits until Friday. Sorry, I also don't want to drag > > RFC-46 anymore, open to hear others thoughts. > > > > On Mon, 5 Dec 2022 at 09:56, Alexey Kudinkin <ale...@onehouse.ai> wrote: > > > > > Hey, everyone! > > > > > > Long-awaited RFC-46 update: It took the crew quite a bit more time than > > > originally anticipated to stabilize and make sure everything is working > > as > > > expected and now we believe it's finally ready for prime time! > > > > > > As such, we're going to execute on a following plan: > > > > > > 1. As called out originally, to move forward with the PR of this > > > magnitude we will be instituting *temporary code-freeze* on master > > > starting at *00:00 (midnight) 12/06 PST* and lasting for *24h. * > > > 2. During that window we're planning to do one final rebase and land > > it > > > on master. > > > 3. We will lift the code-freeze as soon as PR lands, but reserve 24h > > for > > > us to have ample buffer just in case. > > > > > > Let me know if you have any questions or concerns with this plan > > > > > > Alexey, on behalf of RFC-46 team > > > > > > On Wed, Sep 28, 2022 at 9:17 AM Alexey Kudinkin <ale...@onehouse.ai> > > > wrote: > > > > > > > @Ken yes, that's the plan eventually -- to rely on Execution (Query) > > > > Engines to provide their own representation that Hudi will be > handling > > > w/o > > > > any intermediate transitions. If you're curious to learn more i'd > > > encourage > > > > you (and everyone) to check out the RFC-46 itself. > > > > > > > > In the Phase 1 though, we're only focusing on Spark integration for > now > > > > (as proof of concept) and then later on after the new infra is > hardened > > > > enough we can expand it to other engines as well. > > > > > > > > On Tue, Sep 27, 2022 at 7:07 PM Gary Li <yanjia.gary...@gmail.com> > > > wrote: > > > > > > > >> Great work! Really excited about this feature. Kudos to RFC-46 team. > > > >> > > > >> Best, > > > >> Gary > > > >> > > > >> On Wed, Sep 28, 2022 at 7:22 AM Ken Krugler < > > > kkrugler_li...@transpac.com> > > > >> wrote: > > > >> > > > >> > Hi Alexey, > > > >> > > > > >> > Thanks for the update! > > > >> > > > > >> > So for maximum performance when writing to Hudi from a low-level > > > >> > (DataStream, not Table) Flink workflow, we’d be creating RowData > > > >> records? > > > >> > > > > >> > — Ken > > > >> > > > > >> > > > > >> > > On Sep 27, 2022, at 2:08 PM, Alexey Kudinkin < > ale...@onehouse.ai> > > > >> wrote: > > > >> > > > > > >> > > Hello, everyone! > > > >> > > > > > >> > > As you might be aware, community has been very busy at work on > > > RFC-46 > > > >> > > aiming to bring long-awaited cutting edge level of performance > to > > > >> Hudi by > > > >> > > avoiding using Avro as an intermediate representation, instead > > > >> relying on > > > >> > > individual engines to host data in their own formats > (InternalRow > > > for > > > >> > > Spark, RowData for Flink, etc) > > > >> > > > > > >> > > We wanted to share an update in terms of where we are and what > are > > > the > > > >> > next > > > >> > > steps from here: > > > >> > > > > > >> > > - We're very close to completing the work and are already > > > preparing > > > >> to > > > >> > > be landing complete implementation of the Phase 1 of the > RFC-46 > > > >> > currently > > > >> > > being developed in a feature branch > > > >> > > <https://github.com/apache/hudi/tree/release-feature-rfc46> > > > >> > > - To be able to successfully merge the change of such scale, > we > > > will > > > >> > > have to do a *code freeze* for the master branch barring any > > > >> changes to > > > >> > > land before we're able to merge the feature-branch. > > > >> > > - To make sure that this activity doesn't interrupt the 0.12.1 > > > >> release > > > >> > > that is currently in progress we're tentatively planning to > > > schedule > > > >> > this > > > >> > > code-freeze *after* successful finalization of the release > > process > > > >> with > > > >> > > RC branch being cut and validated for release. As of now, > > provided > > > >> RC > > > >> > > candidate will be cut tomorrow on 09/28 we're aiming to > > schedule a > > > >> > merge > > > >> > > attempt somewhere mid to late next week. > > > >> > > - We will follow-up on this thread separately at least *24h* > > > before > > > >> the > > > >> > > scheduled code-freeze with an exact date and time frame for > it. > > > Stay > > > >> > tuned. > > > >> > > > > > >> > > > > > >> > > Alexey, on behalf of the RFC-46 group > > > >> > > > > >> > -------------------------- > > > >> > Ken Krugler > > > >> > http://www.scaleunlimited.com > > > >> > Custom big data solutions > > > >> > Flink, Pinot, Solr, Elasticsearch > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > > > > > > > > > > -- > > Regards, > > -Sivabalan > > >
