hangc0276 commented on issue #4621: URL: https://github.com/apache/iceberg/issues/4621#issuecomment-1107756484
> > However, we found current Iceberg Java reader implementation doesn't support read records by order or doesn't support order by operation on reading. > > It sounds like the pulsar reader hope the apache iceberg reader guarantee the record-level writing order semantic. As far as I know, the table format based on DFS files is hard to maintain the record-level order semantic. Currently, the iceberg table format was optimized for batch analysis readers, not for a message queue consumer. > > I think @rdblue and @RussellSpitzer had discussed the approach about dual writing the iceberg table format and message queue, for downstream reader to consume the records with the writing order, but I don't see a public discussion or design document to describe the details. Maybe they can provide more input about this issue. @openinx Thanks for your quick reply. Looking forward Iceberg community to discuss and support this feature. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
