jon-wei commented on issue #8605: Failed to publish segments because of [java.lang.RuntimeException: Aborting transaction!]. URL: https://github.com/apache/incubator-druid/issues/8605#issuecomment-557738191 @qianmoQ The ingestion system is designed to be resilient to individual task failures (subsequent tasks would pick up from the latest committed offsets and start reading from there, so a task failing shouldn't lead to permanent data loss). If the data that was not ingested by the failed task is picked up by subsequent tasks, then the system is still operating within expected situations, is that consistent with your observations? For this specific error: > Caused by: org.apache.druid.java.util.common.ISE: Failed to publish segments because of [java.lang.RuntimeException: Aborting transaction!]. This indicates that the partition/offsets associated with the segments that the task attempted to publish were inconsistent with the checkpoints stored in metadata (e.g., another task had published data for some of the same partitions while the task that failed was running), so the publish is aborted to prevent data inconsistency issues. The error message is unfortunately not very informative from the task log side and we plan to improve that in the future, but for now to see the full details about the error it's necessary to check the overlord logs. That kind of situation should not happen too often, how often did you observe that kind of task failure? If you can provide overlord log contents showing the transaction failure and the surrounding time window, and the full associated task logs, that would be very helpful for figuring out what you're seeing.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
