nsivabalan commented on issue #8132: URL: https://github.com/apache/hudi/issues/8132#issuecomment-1528118945
hey @coffee34 : The code that you pointed out where inserts converted to updates is not applicable to bloom and simple index. Those are meant for few other index types. But for these two indexes, inserts will always go into a new base file and only update can go into log files. From the info provided, we could not get any leads and we don't see any other data loss reports from the community. We do have some reports around global index, but those will surface differently. What you are reporting is completely diff. Can you provide us w/ a full backup of ".hoodie" and write configs. but as of now, we can't think of any reason why such huge data loss can occur. unless you changed the record key / partition path config for instance. Do you happen to know any changes to your pipelines around the time where you see data loss. like any config updates. or anything of those sort. If you have a back up of the file slice in question, we can try to copy them locally(on your end) and trigger compaction and debug whats going on. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
