tuhaihe opened a new issue, #1968:
URL: https://github.com/apache/cloudberry/issues/1968
### Apache Cloudberry version
main (89baa48a4f4) — also present in 2.x releases.
### What happened
On a Rocky Linux 10 cluster (Python 3.12), every invocation of `gpsync` /
`gpssh` / `gpssh-exkeys` prints `SyntaxWarning` messages to stderr before it
does any work. It surfaced while installing the Cloudberry backup utilities,
whose `make install` calls `gpsync` to distribute binaries to the segments:
```
$ make install
cp /home/gpadmin/go/bin/gpbackup ... /usr/local/cloudberry-db/bin
/usr/local/cloudberry-db/lib/python/gppylib/util/ssh_utils.py:268:
SyntaxWarning: invalid escape sequence '\ '
'''Escape occurrences of \ and $ as needed and package the line as an
"eval" shell command'''
/usr/local/cloudberry-db/bin/lib/pexpect/pxssh.py:105: SyntaxWarning:
invalid escape sequence '\['
self.UNIQUE_PROMPT = "\[PEXPECT\][\$\#] "
/usr/local/cloudberry-db/bin/lib/pexpect/pxssh.py:109: SyntaxWarning:
invalid escape sequence '\$'
self.PROMPT_SET_SH = "PS1='[PEXPECT]\$ '"
/usr/local/cloudberry-db/bin/lib/pexpect/pxssh.py:110: SyntaxWarning:
invalid escape sequence '\$'
self.PROMPT_SET_CSH = "set prompt='[PEXPECT]\$ '"
Successfully copied gpbackup_helper and gpbackup_s3_plugin to
/usr/local/cloudberry-db on all segments
```
The operation itself succeeds — this is cosmetic — but it makes utility
output
noisy and looks like a failure to anyone reading a deployment log.
Why it appears now: string literals such as `"\["` or `"\$"` contain escape
sequences Python does not recognise. Until 3.11 that was a
`DeprecationWarning`,
hidden by default; **Python 3.12 promoted it to a `SyntaxWarning`, which is
printed** ([What's New in
3.12](https://docs.python.org/3/whatsnew/3.12.html#other-language-changes)).
So the same code is quiet on Rocky 8/9 (Python 3.6/3.9) and noisy on Rocky
10,
Fedora 40+, and Ubuntu 24.10+. The Python docs also state these sequences
**will become a `SyntaxError` in a future release**, so this is not purely
cosmetic in the long run.
Note the warnings can repeat on every run: if `$GPHOME` is root-owned and the
utility runs as `gpadmin`, CPython cannot write `__pycache__`, so the modules
are recompiled — and re-warn — each time.
### What you think should happen instead
`gpsync`/`gpssh` should produce no Python warnings. The literals in question
are
regexes and shell snippets that were meant to be raw strings; they should be
written as such (`r"\[PEXPECT\]..."`), which leaves their value unchanged.
Compiling everything under `gpMgmt/` shows the problem is broader than the
four
warnings above — 41 literals in 16 files:
| File | Lines |
| --- | --- |
| `gpMgmt/bin/gpload.py` | 720, 2530, 2675, 2689 |
| `gpMgmt/bin/gpload_test/gpload/TEST.py` | 322 (×2), 329, 331, 333, 335,
353, 355, 410 |
|
`gpMgmt/bin/gppylib/programs/test/unit/test_cluster_clsrecoversegment_triples.py`
| 297, 304, 332 |
| `gpMgmt/bin/gppylib/test/unit/test_unit_database_segment_guc.py` | 26 |
| `gpMgmt/bin/gppylib/test/unit/test_unit_file_segment_guc.py` | 31 |
| `gpMgmt/bin/gppylib/test/unit/test_unit_gppkg.py` | 72 |
| `gpMgmt/bin/gppylib/test/unit/test_unit_gpsegrecovery.py` | 304, 333 |
| `gpMgmt/bin/gppylib/test/unit/test_unit_gpsegsetuprecovery.py` | 262, 287 |
| `gpMgmt/bin/gppylib/test/unit/test_unit_package.py` | 122 |
| `gpMgmt/bin/gppylib/util/ssh_utils.py` | 268 |
| `gpMgmt/bin/lib/pexpect/pxssh.py` | 105, 109, 110 |
| `gpMgmt/sbin/seg_update_pg_hba.py` | 41 |
| `gpMgmt/test/behave/mgmt_utils/steps/gpstate_utils.py` | 79 |
| `gpMgmt/test/behave/mgmt_utils/steps/mgmt_utils.py` | 683, 908, 2007,
2223, 3346, 3351, 3849, 3980 |
| `gpMgmt/test/behave/mgmt_utils/steps/replication_slots_utils.py` | 27, 90 |
| `gpMgmt/test/behave_utils/utils.py` | 682 |
A further 67 literals in 11 files under `src/` and `contrib/` have the same
problem (`src/backend/gporca/scripts/`, `contrib/try_convert/scripts/`).
Those
are developer scripts rather than shipped utilities, so they are lower
priority.
Two side findings while auditing:
1. `gpMgmt/bin/lib/pexpect` is a vendored copy of **pexpect 3.3** (2013).
Upstream pexpect made these literals raw years ago; refreshing the
vendored
copy — or dropping it in favour of the packaged `pexpect` — would be worth
considering separately.
2. `gpMgmt/test/behave/mgmt_utils/steps/replication_slots_utils.py:31` reads
`export WITH_MIRRORS={with_mirrors} && \A`, in the middle of a run of
shell
line continuations. The stray `A` looks like a keystroke slip: the shell
receives `&& A ./demo_cluster.sh -d && ...` and tries to run a command
named
`A`. Pre-existing, unrelated to the encoding of the escape, and probably
deserves its own fix.
### How to reproduce
On any host with Python 3.12 or newer (Rocky Linux 10, Fedora 40+,
Ubuntu 24.10+):
```
$ gpsync --help # or: gpssh --help
```
Or without a cluster, straight from a source checkout:
```
$ python3 --version
Python 3.12.4
$ python3 -W error::SyntaxWarning -m py_compile \
gpMgmt/bin/gppylib/util/ssh_utils.py gpMgmt/bin/lib/pexpect/pxssh.py
```
To enumerate every occurrence in the tree:
```
$ python3 - <<'PY'
import pathlib, warnings
for p in sorted(pathlib.Path('gpMgmt').rglob('*.py')):
with warnings.catch_warnings(record=True) as w:
warnings.simplefilter('always')
try:
compile(p.read_text(encoding='utf-8', errors='replace'), str(p),
'exec')
except SyntaxError:
continue
for i in w:
if issubclass(i.category, SyntaxWarning):
print(f'{p}:{i.lineno}: {i.message}')
PY
```
### Operating System
Rocky Linux 10 (Python 3.12); reproduced on macOS with Python 3.12.4.
### Anything else
Happens every time on Python 3.12+, never on Python ≤ 3.11.
Workaround for operators until this is fixed:
```
$ PYTHONWARNINGS=ignore::SyntaxWarning make install
```
I am preparing a PR that fixes the 41 literals under `gpMgmt/`, with the
`src/`+`contrib/` remainder as a follow-up.
### Are you willing to submit PR?
- [x] Yes, I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/cloudberry/blob/main/CODE_OF_CONDUCT.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
