zhannngchen opened a new pull request, #30366: URL: https://github.com/apache/doris/pull/30366
## Proposed changes Issue Number: close #xxx We found in our stress test environment that when performing concurrent partial column updates, data was lost or inconsistent due to not to handle write exceptions of the newly generated segment well in publish phase: 1. when two concurrent partial column update job try to update an identical key at the same time, additional conflict handling needs to be done in the publish phase 2. and the load task with a higher version needs to generate the new rows and write them to the newly generated segmnet, while marking the corresponding key in the original segment for deletion 3. this deletion marker is updated in txn_info->delete_bitmap in real time, when writing the new segment failed, these deletion markers for the original segment are not rolled back 4. and the next time the publish task retries, it will not process these keys because they have already been marked for deletion. ## Further comments If this is a relatively large or complex change, kick off the discussion at [[email protected]](mailto:[email protected]) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
