imbajin opened a new pull request, #801: URL: https://github.com/apache/hugegraph-toolchain/pull/801
## Purpose of the PR Loader can report success after parse, insert, or checkpoint failures. With batch fallback disabled, rejected batches can also disappear from both the failure files and the already advanced progress. This change makes unsuccessful loads fail visibly and retains the information needed for replay. Reported during [master integration testing](https://github.com/hugegraph/hugegraph-toolchain/pull/47#issuecomment-6064536763).  ## Main Changes - Write failed or stopped batches to failure-data when compensation remains safe, and propagate asynchronous errors. - Preserve the existing fully parsed progress plus failure-data recovery contract; reject unreliable progress after write, flush, cancellation, or checkpoint failures. - Close task admission and drain submitters, running handlers, and cancelled queued tasks before closing their clients and logs. - Attempt every failure-log close, retain suppressed errors and the original runtime exception type/message, and return an unsuccessful process status after cleanup. The shutdown and recovery changes are delivered together as one independent fix: splitting them would leave unsafe intermediate resource and progress behavior. This does not add per-record acknowledgements or a new replay protocol. ```mermaid flowchart LR A[Batch fails without fallback] --> B[Retain the batch in failure-data] B --> C[Drain work and flush logs] C --> D{Progress reliable?} D -- Yes --> E[Save parsed progress] D -- No --> F[Keep old progress] E --> G[Return failure] F --> G ``` ## Verifying these changes - [ ] Trivial rework / code cleanup without any test coverage. (No Need) - [ ] Already covered by existing tests. - [x] Need tests and can be verified as follows: - Targeted task and context regressions pass with failing negative controls for lost compensation, checkpoint refusal, active callbacks, admission races, and cancelled queued tasks. - Real file streams and controlled in-flight work exercise log flushing, cancellation, timeout, and interruption boundaries. - Checkstyle and three independent final reviews pass. - Parse-error compatibility and first-failure cleanup regressions pass; the CLI fixture explicitly fails integer conversion and still checks nonzero exit, valid-row insertion, and retained failure-data. - Original source and packaged CLI/Server recovery execution and current PR CI remain pending. ## Does this PR potentially affect the following parts? - [ ] Nope - [ ] Dependencies (add/update license info) - [ ] Modify configurations - [ ] The public API - [x] Other affects: Loader failure status, shutdown, and existing recovery data ## Documentation Status - [ ] `Doc - TODO` - [ ] `Doc - Done` - [x] `Doc - No Need`: the existing progress and failure-data contract is preserved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
