vyrodovalexey opened a new issue, #100:
URL: https://github.com/apache/cloudberry-backup/issues/100
### Apache Cloudberry and cloudberry-backup version
2.1.0
### What happened
gpbackup crashes Cloudberry segment backend (exit 255) on segment-dispatched
commands
### What you think should happen instead
maybe to write dedicated lib to work with cloudberry/greenplum
### How to reproduce
## Reproduction
Against the running `cb-prod` cluster (HA coordinator + 2 primary segments +
2 mirrors), with `mydb` containing distributed tables:
```bash
gpbackup --dbname mydb --backup-dir /tmp/gpb --jobs 1 --no-history
```
Result (reproducible **100% of the time**, even on a freshly created
1000-row database, independent of `--jobs`, `--no-history`,
`--single-data-file`, or the S3 plugin):
```text
gpbackup ... [INFO]:-Starting backup of database mydb
gpbackup ... [CRITICAL]:-ERROR: Error on receive from seg0 10.1.10.236:5432
pid=3646:
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request. (SQLSTATE 58M01)
EXIT=2
```
## Evidence from the segment log (`gpseg0/log`)
```text
... "LOG","00000","server process (PID 3655) exited with exit code 255",
...,"postmaster.c",4264
... "LOG","00000","terminating any other active server processes",
...,"postmaster.c",4000
... "LOG","00000","all server processes terminated; reinitializing",
...,"postmaster.c",4576
... "LOG","00000","database system was not properly shut down; automatic
recovery in progress"
... "LOG","00000","database system is ready"
```
Exit code 255 with *"terminating any other active server processes"* +
*"reinitializing"* is the postmaster's response to a backend that died
abnormally (not a clean `ERROR` return) — a **crash of the segment backend**,
not a SQL-level error.
## The crash trigger sequence (what gpbackup does)
The coordinator log pinpoints the failing statement and the session state
around it:
```text
seg0 ... "SET transaction_isolation TO 'serializable'" (worker sessions;
CB falls back to repeatable read)
seg0 ... server process (PID 3655) exited with exit code 255
seg-1 (coordinator) ... "ERROR","58M01","Error on receive from seg0 ...
server closed the connection unexpectedly"
seg-1 (coordinator) ... "An exception was encountered during the execution
of statement:
SELECT pg_size_pretty(pg_database_size('mydb')) as
dbsize"
```
gpbackup opens its connection, sets the session to serializable (Cloudberry
logs *"serializable isolation requested, falling back to repeatable read until
serializable is supported"*), exports a distributed snapshot, and then
dispatches a series of commands to segments. The first to crash a segment was
the informational `pg_database_size` query. After patching that out (it's
cosmetic — only used for the report's *"database size:"* line), the crash
simply moved to the next segment-dispatched command, and so on through:
- `pg_database_size('mydb')` — report DB-size query
- `LOCK TABLE ... ON SEGMENT`
- `SAVEPOINT`
- `COPY <table> TO PROGRAM ... ON SEGMENT` — the core MPP data-backup
operation
## The discriminating fact (root cause isolation)
The same statements that crash the segment from gpbackup **succeed** from
`psql`:
| Statement | From `psql` (libpq / C) | From gpbackup (Go `database/sql`,
pgx & lib/pq) |
| --- | --- | --- |
| `SELECT pg_database_size('mydb')` | ✅ returns 309 MB | ❌ crashes segment
(exit 255) |
| `SELECT pg_database_size(oid) FROM pg_database` | ✅ returns values for all
DBs | — |
| `BEGIN ISOLATION LEVEL SERIALIZABLE; SELECT pg_database_size('mydb')` | ✅
works | — |
| `COPY ... TO PROGRAM ON SEGMENT` | ✅ (psql path) | ❌ crashes segment |
So the crash correlates with the **client driver / wire-protocol path**, not
the SQL text or the isolation level alone. The Go `database/sql` drivers send
the extended query protocol / specific bind-parameter and
snapshot-synchronization patterns that this Cloudberry build's segment backend
mishandles, leading to the backend dying instead of returning an error.
### Operating System
rockylinux:9.6
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes, I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/cloudberry/blob/main/CODE_OF_CONDUCT.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
