my-ship-it opened a new pull request, #93: URL: https://github.com/apache/cloudberry-backup/pull/93
## Summary Fixes #92. `gprestore --resize-cluster` (larger-to-smaller) only works with `--jobs 1` today. Any `--jobs > 1` run fails with: ``` ERROR: command error message: cat: .../gpbackup_<c>_<ts>_pipe_..._<oid>_0: No such file or directory ``` This is the same class of bug upstream gpbackup fixed in 1.30.6 (and documented in [Broadcom KB 378496](https://knowledge.broadcom.com/external/article/378496/gprestore-with-resizecluster-will-fail-w.html)); the pre-fix logic is still present in this fork. ## Root cause In `restore/data.go:restoreDataFromTimestamp`, the oid list for resize restore is expanded into `(oid, batch)` pairs in oid-major order: ``` [T1B0, T1B1, T2B0, T2B1, ..., TnB0, TnB1] ``` `CreateInitialSegmentPipes` then preloads only `min(NumConns, len(oidList))` pipes, and `gpbackup_helper`'s `preloadCreatedPipesForRestore` mirrors that count via `--copy-queue-size`. With `NumConns == --jobs` and `batches ≥ 2`, the preload covers only the first `NumConns/batches` tables. But the coordinator's task channel dispatches **one task per table**, and `restoreSingleTableData` iterates the batches sequentially inside each worker. So `NumConns` concurrent workers each immediately open a COPY against oid indices `0, batches, 2*batches, …, (NumConns-1)*batches`. Workers beyond `NumConns/batches` request pipes the helper has not created yet — hence the "No such file or directory" error. With `--jobs 1` the coordinator serializes completely and the helper keeps up, which matches the reporter's observation. ## Fix Preload `NumConns * batches` pipes (clamped to `len(oidList)`), so every concurrent worker's first batch is covered. The helper then rolls its queue forward one pipe per completed batch, and because a worker must finish **all** batches of its current table before pulling the next task from the channel, helper progress stays ahead of worker demand. - `batches == 1` (non-resize or equal-size resize): `NumConns * 1 == NumConns`, behavior unchanged. - `batches > 1`: each of the `NumConns` concurrent workers has its first batch preloaded; subsequent tables are served from the rolling queue. No change to the helper — `preloadCreatedPipesForRestore` and the `if i < len(oidWithBatchList)-*copyQueue` guard work with any queue size. ## Test plan - [x] `go build ./...` passes - [x] `go vet ./restore/...` clean - [x] `gofmt -l restore/data.go` clean - [ ] Manual repro on a 6→4 segment resize restore with `--jobs 8` (pre-patch: errors on every table; post-patch: restore completes) - [ ] CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
