Hi Danny, All the planned tickets have landed in master and we are good for cutting 0.10 RC. Please let us know if you see any CI issues with the latest master and we can jump in to do the needful. Thanks for your patience.
thanks, Manoj On Fri, Nov 26, 2021 at 8:07 PM Manoj Govindassamy < manoj.govindass...@gmail.com> wrote: > Hi Danny, > > We have one last PR https://github.com/apache/hudi/pull/4114 to land to > master. We are noticing one test flakiness with this last pending PR. The > same test is consistently passing in the local setup though. We are waiting > for the CI to finish before the merge to master. After this PR we are good > for cutting the 0.10 RC. Will keep you posted on the status. > > thanks, > Manoj > > > > > On Sat, Nov 20, 2021 at 2:10 PM Raymond Xu <xu.shiyan.raym...@gmail.com> > wrote: > >> Hi Danny, I'm good with the timeline. >> >> Cheers, >> Raymond >> >> On Fri, Nov 19, 2021 at 7:34 PM sagar sumit <sagarsumi...@gmail.com> >> wrote: >> >> > Hi Danny, >> > >> > I've added one more blocker: HUDI-2742 >> > <https://issues.apache.org/jira/browse/HUDI-2742> >> > I am also good with the timelines. >> > >> > Regards, >> > Sagar >> > >> > On Sat, Nov 20, 2021 at 8:14 AM Sivabalan <n.siv...@gmail.com> wrote: >> > >> > > Hi Danny, >> > > I am good with the timelines. All my jiras should be completed by >> > > then. >> > > >> > > >> > > On Fri, Nov 19, 2021 at 8:41 PM Y Ethan Guo <ethan.guoyi...@gmail.com >> > >> > > wrote: >> > > >> > > > Hi Danny, >> > > > >> > > > Thanks for summarizing the current progress towards the 0.10.0 >> release. >> > > > I'm good with Nov 26th cutoff. >> > > > >> > > > Regarding my blockers: >> > > > - [HUDI-2332] Implement scheduling of compaction/ clustering for >> Kafka >> > > > Connect (Owner: Ethan Guo) >> > > > PR is up. I'm addressing comments. >> > > > >> > > > - [HUDI-2737] Use earliest instant by default for compaction and >> > > > clustering job (Owner: Ethan Guo) >> > > > PR is up and approved. It's near-landing after fixing CI failures. >> > > > >> > > > - [HUDI-2745] Record count does not match input after compaction is >> > > > scheduled when running Hudi Kafka Connect sink (Owner: Ethan Guo) >> > > > HUDI-2745 is going to be blocked on HUDI-2480, which is going to >> > resolve >> > > > this issue once done. >> > > > >> > > > - [HUDI-2735] Fix archival of commits in Java client for Kafka >> Connect >> > > > (Owner: Ethan Guo) >> > > > This is pending and requires investigation into the archival logic >> > which >> > > is >> > > > not Kafka-connect specific. >> > > > >> > > > Best, >> > > > - Ethan >> > > > >> > > > >> > > > On Fri, Nov 19, 2021 at 4:41 PM Rajesh Mahindra < >> rmahin...@gmail.com> >> > > > wrote: >> > > > >> > > > > Hi Danny, >> > > > > >> > > > > I have the following blockers that have a PR up. I am working on >> a PR >> > > for >> > > > > the Debezium Source. I am fine with Nov 26th as cut off. >> > > > > >> > > > > - [HUDI-2325] Implement and test Hive Sync support for Kafka >> > Connect >> > > > > (Owner: Rajesh Mahindra) >> > > > > - [HUDI-2671] Fix record offset handling in Kafka connect >> > > transaction >> > > > > participant (Owner: Rajesh Mahindra) >> > > > > - [HUDI-2672] Avoid empty commits and rollbacks when there is >> no >> > > event >> > > > > from the topic (Owner: Rajesh Mahindra) >> > > > > >> > > > > ** Pending >> > > > > - [HUDI-1290] Implement Debezium avro source for Delta Streamer >> > > > > >> > > > > Thanks >> > > > > Rajesh >> > > > > >> > > > > >> > > > > On Fri, Nov 19, 2021 at 4:01 PM Udit Mehrotra <udi...@apache.org> >> > > wrote: >> > > > > >> > > > > > Hi Danny, >> > > > > > >> > > > > > I have a blocker as well >> > > > > > https://issues.apache.org/jira/browse/HUDI-2802. Nov 26th cut >> off >> > > date >> > > > > > works fine for me. >> > > > > > >> > > > > > Also, just an update on the above list: HUDI-2641, HUDI-2314, >> > > > > > HUDI-2362 are unblocked/merged. HUDI-2314 and HUDI-2362 can be >> > marked >> > > > > > in the highlights section. We will work on getting some doc >> updates >> > > > > > for the same by next week. >> > > > > > >> > > > > > Thanks, >> > > > > > Udit >> > > > > > >> > > > > > On Fri, Nov 19, 2021 at 3:49 PM Vinoth Chandar < >> vin...@apache.org> >> > > > > wrote: >> > > > > > > >> > > > > > > Hi Danny, >> > > > > > > >> > > > > > > I have one blocker. I plan to complete it by end of next >> week. I >> > am >> > > > > good >> > > > > > > with the prior Nov 26 cutoff. >> > > > > > > Does that work for everyone? >> > > > > > > >> > > > > > > Thanks >> > > > > > > Vinoth >> > > > > > > >> > > > > > > On Fri, Nov 19, 2021 at 12:12 AM Danny Chan < >> > danny0...@apache.org> >> > > > > > wrote: >> > > > > > > >> > > > > > > > Hi Community, >> > > > > > > > >> > > > > > > > As we draw close to doing Hudi 0.10.0 release, I am happy to >> > > share >> > > > a >> > > > > > > > summary of the key features/improvements that would be >> going in >> > > the >> > > > > > release >> > > > > > > > and the current blockers for everyone's visibility. >> > > > > > > > >> > > > > > > > *Highlights* >> > > > > > > > >> > > > > > > > - [HUDI-1290] Implement Debezium avro source for Delta >> > > Streamer >> > > > > > > > - [HUDI-1491] Support partition pruning for MOR snapshot >> > query >> > > > > > > > - [HUDI-1763] DefaultHoodieRecordPayload does not honor >> > > ordering >> > > > > > value >> > > > > > > > when records within multiple log files are merged >> > > > > > > > - [HUDI-1827] Add ORC support in Bootstrap Op >> > > > > > > > - [HUDI-1869] Upgrading Spark3 To 3.1 >> > > > > > > > - [HUDI-2101] support z-order for hudi >> > > > > > > > - [HUDI-2276] Enable Metadata Table by default for both >> > > writers >> > > > > and >> > > > > > > > readers >> > > > > > > > - [HUDI-2581] Analyze metadata size estimate in hudi with >> > > Hfile >> > > > > for >> > > > > > col >> > > > > > > > stats partition >> > > > > > > > - [HUDI-2634] Improve bootstrap performance for very >> large >> > > > tables >> > > > > > > > - [HUDI-2086] redo the logical of mor_incremental_view >> for >> > > hive >> > > > > > > > - [HUDI-2191] Bump flink version to 1.13.1 >> > > > > > > > - [HUDI-2285] Metadata Table Synchronous Design >> > > > > > > > - [HUDI-2316] Support Flink batch upsert >> > > > > > > > - [HUDI-2371] Improve flink streaming reader >> > > > > > > > - [HUDI-2394] [Kafka Connect Mileston 1] Implement kafka >> > > connect >> > > > > for >> > > > > > > > immutable data >> > > > > > > > - [HUDI-2449] Incremental read for Flink >> > > > > > > > - [HUDI-2562] Embedded timeline server on JobManager >> > > > > > > > >> > > > > > > > *Current Blockers* >> > > > > > > > >> > > > > > > > - [HUDI-1856] Upstream changes made in PrestoDB to >> eliminate >> > > > file >> > > > > > > > listing to Trino (Owner: Sagar Sumit) >> > > > > > > > - [HUDI-1912] Presto defaults to GenericHiveRecordCursor >> for >> > > all >> > > > > > Hudi >> > > > > > > > tables (Owner: Sagar Sumit) >> > > > > > > > - [HUDI-1932] Hive Sync should not always update >> > > > > > last_commit_time_sync >> > > > > > > > (Owner: Raymond Xu) >> > > > > > > > - [HUDI-1937] When clustering fail, generating unfinished >> > > > > > replacecommit >> > > > > > > > timeline. (Owner: Sagar Sumit) >> > > > > > > > - [HUDI-2077] Flaky test: TestHoodieDeltaStreamer (Owner: >> > > Sagar >> > > > > > Sumit) >> > > > > > > > - [HUDI-2314] Add DynamoDb based lock provider (Owner: >> > Wenning >> > > > > Ding) >> > > > > > > > - [HUDI-2325] Implement and test Hive Sync support for >> Kafka >> > > > > Connect >> > > > > > > > (Owner: Rajesh Mahindra) >> > > > > > > > - [HUDI-2332] Implement scheduling of compaction/ >> clustering >> > > for >> > > > > > Kafka >> > > > > > > > Connect (Owner: Ethan Guo) >> > > > > > > > - [HUDI-2362] Hudi external configuration file support >> > (Owner: >> > > > > > Wenning >> > > > > > > > Ding) >> > > > > > > > - [HUDI-2409] Using HBase shaded jars in Hudi presto >> bundle >> > > > > (Owner: >> > > > > > > > Sagar Sumit) >> > > > > > > > - [HUDI-2443] KVComparator in HFile for metadata table is >> > tied >> > > > to >> > > > > > HBase >> > > > > > > > version and shading (Owner: Sagar Sumit) >> > > > > > > > - [HUDI-2472] Tests failure follow up when metadata is >> > enabled >> > > > by >> > > > > > > > default (Owner: Manoj Govindassamy) >> > > > > > > > - [HUDI-2475] Rolling Upgrade downgrade story for 0.10 & >> > > > enabling >> > > > > > > > metadata (Owner: Manoj Govindassamy) >> > > > > > > > - [HUDI-2478] Handle failure mid-way during init buckets >> > > (Owner: >> > > > > > Vinoth >> > > > > > > > Chandar) >> > > > > > > > - [HUDI-2480] FileSlice after pending >> compaction-requested >> > > > > > instant-time >> > > > > > > > is ignored by MOR snapshot reader (Owner: Danny Chen) >> > > > > > > > - [HUDI-2488] Support bootstrapping a single or more >> > > partitions >> > > > in >> > > > > > > > metadata table while regular writers and table services >> are >> > in >> > > > > > progress >> > > > > > > > (Owner: Vinoth Chandar) >> > > > > > > > - [HUDI-2527] Flaky test: >> > > > > > > > >> > > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> TestHoodieClientMultiWriter.testMultiWriterWithAsyncTableServicesWithConflict >> > > > > > > > (Owner: sivabalan narayanan) >> > > > > > > > - [HUDI-2559] Ensure unique timestamps are generated for >> > > commit >> > > > > > times >> > > > > > > > with concurrent writers (Owner: sivabalan narayanan) >> > > > > > > > - [HUDI-2593] Virtual keys support for metadata table >> > (Owner: >> > > > > Manoj >> > > > > > > > Govindassamy) >> > > > > > > > - [HUDI-2599] [Performance] Lower parallelism with >> snapshot >> > > > query >> > > > > > on COW >> > > > > > > > tables in Presto (Owner: Sagar Sumit) >> > > > > > > > - [HUDI-2628] Fix Chinese Docs (Owner: Kyle Weller) >> > > > > > > > - [HUDI-2636] Make release notes discoverable (Owner: >> Kyle >> > > > Weller) >> > > > > > > > - [HUDI-2637] Triage all bugs around Multi-writer and >> > certify >> > > > the >> > > > > > tested >> > > > > > > > flows (Owner: sivabalan narayanan) >> > > > > > > > - [HUDI-2641] One inflight commit rolling back other >> > > concurrent >> > > > > > inflight >> > > > > > > > commits causing them to fail (Owner: Udit Mehrotra) >> > > > > > > > - [HUDI-2649] Kick off all the Hive query issues for >> 0.10.0 >> > > > > (Owner: >> > > > > > > > Sagar Sumit) >> > > > > > > > - [HUDI-2666] async compaction failing with timeline >> > > mismatches >> > > > > > between >> > > > > > > > server and client when metadata is enabled (Owner: Manoj >> > > > > > Govindassamy) >> > > > > > > > - [HUDI-2667] Avoid fs.exists() and fs.mkdirs() call to >> > > > partitions >> > > > > > in >> > > > > > > > AbstractTablefileSystemView (Owner: Sagar Sumit) >> > > > > > > > - [HUDI-2671] Fix record offset handling in Kafka connect >> > > > > > transaction >> > > > > > > > participant (Owner: Rajesh Mahindra) >> > > > > > > > - [HUDI-2672] Avoid empty commits and rollbacks when >> there >> > is >> > > no >> > > > > > event >> > > > > > > > from the topic (Owner: Rajesh Mahindra) >> > > > > > > > - [HUDI-2716] Fix InLineFS path conversions for S3FS >> paths >> > > > (Owner: >> > > > > > Manoj >> > > > > > > > Govindassamy) >> > > > > > > > - [HUDI-2725] Add precommit validators doc (Owner: Kyle >> > > Weller) >> > > > > > > > - [HUDI-2731] Clustering should work regardless of >> whether >> > > there >> > > > > are >> > > > > > > > base files (Owner: Sagar Sumit) >> > > > > > > > - [HUDI-2734] Disable metadata by default for flink and >> java >> > > > > (Owner: >> > > > > > > > sivabalan narayanan) >> > > > > > > > - [HUDI-2735] Fix archival of commits in Java client for >> > Kafka >> > > > > > Connect >> > > > > > > > (Owner: Ethan Guo) >> > > > > > > > - [HUDI-2737] Use earliest instant by default for >> compaction >> > > and >> > > > > > > > clustering job (Owner: Ethan Guo) >> > > > > > > > - [HUDI-2741] Validate metadata config for all readers >> > (Owner: >> > > > > Sagar >> > > > > > > > Sumit) >> > > > > > > > - [HUDI-2745] Record count does not match input after >> > > compaction >> > > > > is >> > > > > > > > scheduled when running Hudi Kafka Connect sink (Owner: >> Ethan >> > > > Guo) >> > > > > > > > - [HUDI-2762] Ensure hive can query insert only logs in >> MOR >> > > > > (Owner: >> > > > > > agar >> > > > > > > > Sumit) >> > > > > > > > - [HUDI-2763] Avoid persisting redundant key field in the >> > > > Metadata >> > > > > > table >> > > > > > > > record payload (Owner: Manoj Govindassamy) >> > > > > > > > - [HUDI-2764] Address test failures after enabling >> virtual >> > > keys >> > > > > > support >> > > > > > > > for the metadata table (Owner: Manoj Govindassamy) >> > > > > > > > - [HUDI-2766] Enable marker based rollback by default >> > (Owner: >> > > > > > sivabalan >> > > > > > > > narayanan) >> > > > > > > > - [HUDI-2767] Enable timeline server based marker type as >> > > > default >> > > > > > > > (Owner: sivabalan narayanan) >> > > > > > > > - [HUDI-2770] Update docs for HoodieCompactor >> (compaction) >> > and >> > > > > > > > HoodieClusteringJob (clustering) (Owner: Kyle Weller) >> > > > > > > > >> > > > > > > > >> > > > > > > > Please respond to the thread if you think that I have missed >> > > > > capturing >> > > > > > any >> > > > > > > > of the highlights or blockers for Hudi 0.10.0 release. For >> the >> > > > owners >> > > > > > of >> > > > > > > > these release blockers, can you please provide a specific >> > > timeline >> > > > > you >> > > > > > are >> > > > > > > > willing to commit to for finishing these so we can cut an >> RC ? >> > > > > > > > >> > > > > > > > Thanks, >> > > > > > > > Danny >> > > > > > > > >> > > > > > >> > > > > >> > > > > >> > > > > -- >> > > > > Take Care, >> > > > > Rajesh Mahindra >> > > > > >> > > > >> > > >> > > >> > > -- >> > > Regards, >> > > -Sivabalan >> > > >> > >> >
