Alowator commented on PR #12729: URL: https://github.com/apache/hudi/pull/12729#issuecomment-2667844385
> > @Alowator @cshuo @danny0405 great to see this effort. Do we agree on two separate rfcs? > > Let's make a call and either land this PR to claim 87 or add @Alowator to 88? > > As discussed offline, it's ok to keep the flink optimizing stuff separate, so as to make it possible to land solving avro performance issue in release 1.1. @Alowator Could you briefly outline the scope of RFC-87, so that optimization works about other aspect of Flink integration can start simultaneously, such as reading/compaction..just make sure we are not doing the same things. @cshuo I agree with your point. To ensure we don't duplicate efforts, here’s the plan I propose: 1. RFC-88 will focus solely on defining new abstractions with old avro writers. 2. RFC-87 will be dedicated to performance improvements that leverage those abstractions. Since RFC-88 is big, I can assist by implementing the abstractions for the writer as part of that RFC. Then, I will move forward with implementing Avro elimination in RFC-87. For RFC-87, I can start drafting the design for the performance improvements. However, the actual implementation of this design will require the abstractions from RFC-88 to be completed first. For RFC-87 it does not affect the reader's logic directly. However, conflicts could arise in the compactor logic, for writers which utilize it. These potential conflicts should be clearly outlined in the design of RFC-87. TL;DR: It’s possible to start designing RFC-87 without RFC-88, but full implementation depends on completing the abstractions in RFC-88 first. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
