ssdong commented on issue #2818: URL: https://github.com/apache/hudi/issues/2818#issuecomment-820180331
@n3nash Thank you for getting back to me. Let me know if you need extra manpower to help fix `MergeOnReadIncrementalRelation`. :) As for the second issue, thank you for providing extra insights into this, and I understand your point that if the checkpoint has not changed; once there is another "latest" change, it will surely pull that out for me incrementally given it will be on the active timeline again. However, what if the "intermediate" changes are supposed to the "last" changes to the records, and the archival puts them in an _invisible_ state? ๐คAnd will the checkpoint stay the same all the time? Right now, it seems like the relationship between archived/active timeline and incremental pull _requires_ the user to have a carefully controlled incremental pull interval to get all updates before it gets archived, and I wonder if that's easy to achieve? Just throwing a bunch of my thoughts and concerns, and please do let me know what you find regarding commit time provided is way in the past and not in the hudi timeline. ๐ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
