avamingli opened a new pull request, #1774:
URL: https://github.com/apache/cloudberry/pull/1774
<!-- Thank you for your contribution to Apache Cloudberry (Incubating)! -->
## Background
Apache Cloudberry today builds cleanly on Linux only. macOS developers
have to spin up a Linux VM or container to compile and run the
project locally, which slows the inner loop on day-to-day work (read
code, change a line, see it run).
This PR brings the macOS host (Intel Mac tested; Apple Silicon
should follow the same pattern) up to first-class buildable status:
- `./configure` succeeds with the full feature set
(PAX, ic-proxy, ORCA, orafce, mapreduce, gpcloud, paxformat,
tap-tests, plperl, plpython3, GSSAPI, PAM, LDAP, LZ4, ZSTD, …).
- `make && make install` complete cleanly.
- `gpAux/gpdemo` produces a 3-pair demo cluster with coordinator +
standby + 3 primaries + 3 mirrors, all in sync.
Tested on macOS 13.x x86_64 with Xcode CommandLineTools (Apple clang
14.0.3) and Homebrew at `/usr/local`.
## Strict guarantee: zero impact on existing Linux/gcc builds
Every change in this PR is gated so the existing Linux build path is
**byte-for-byte unchanged**. Specifically, each modified file falls
into one of these categories:
| Mechanism | Files |
|---|---|
| `PORTNAME=darwin` Makefile gate | `src/backend/Makefile`,
`src/backend/gporca/gporca.mk`, `contrib/pax_storage/Makefile` |
| `if(APPLE) / NOT APPLE / CMAKE_SYSTEM_NAME` cmake gate |
`contrib/pax_storage/CMakeLists.txt`,
`contrib/pax_storage/src/cpp/CMakeLists.txt`,
`contrib/pax_storage/src/cpp/cmake/{pax,pax_format}.cmake` |
| `case $host_os in linux*) ;;` autoconf gate | `configure.ac`, `configure`
(for liburing + Python probe) |
| `src/template/darwin` — loaded only on darwin | (the DLSUFFIX pin) |
| `#ifdef __APPLE__` / `#ifdef __linux__` C preprocessor gate |
`contrib/pax_storage/src/cpp/comm/fast_io.{h,cc}`,
`contrib/pax_storage/src/cpp/storage/local_file_system.cc`,
`contrib/pax_storage/src/cpp/storage/file_system.h` |
| `#if defined(__clang__)` C++ gate (preserves gcc codegen) |
`contrib/pax_storage/src/cpp/storage/columns/pax_delta_encoding.cc` (VLA vs
std::vector) |
| Identical compilation on Linux (zero functional change) |
`src/include/utils/numeric.h` (drop bogus `const` on non-pointer return type),
`contrib/pax_storage/src/cpp/storage/proto/proto_wrappers.h` (defensive
`#undef`/redef of PG macros around protobuf include),
`contrib/pax_storage/src/cpp/storage/proto/protobuf_stream.{h,cc}`
(`google::protobuf::int64` → `int64_t`, equivalent typedef on older protobuf),
`contrib/pax_storage/src/cpp/storage/columns/pax_encoding_utils.{h,cc}`
(`int64_t *` → `int64 *`, identical LP64 type on Linux),
`src/test/regress/GNUmakefile` (`.so` → `$(DLSUFFIX)`, which expands to `.so`
on Linux) |
For every gated change the `else` / non-gated branch retains the
exact pre-PR code (verbatim). For every non-gated change the
generated machine code on Linux gcc is identical.
`make installcheck` on Linux should be entirely unaffected.
## What the changes touch (high level)
Roughly four buckets, **9 atomic commits, 23 files, +280 / −67**.
1. **Two universally-beneficial fixes** for clang strictness that
*happen* to unblock macOS:
- Drop the gcc-only `-Werror -Wextra -Wpedantic` triple from
`gporca.mk` **on darwin only**. clang reports far more
diagnostics than gcc under these flags and breaks the build
for no real safety benefit (per-feature `-Werror=` flags in
the base CXXFLAGS still catch real bugs). Linux retains the
original triple.
- Fix a typo in `init_var_from_str`'s prototype
(`extern const bool` → `extern bool`) that gcc silently
tolerated but clang correctly flags as a conflicting type
against the definition. Identical compilation on gcc;
introduced by upstream PR
*#392 Export numeric interface to public*.
2. **macOS link-time differences** vs Linux:
- macOS `<sys/param.h>` doesn't define `HZ`; provide a 100
fallback so `contrib/interconnect/udp/ic_udpifc.c` compiles.
`#ifndef HZ` makes this a no-op on Linux glibc.
- macOS `ld` rejects unresolved symbols in shared libraries
by default. With `--enable-shared-postgres-backend`, the
`libpostgres.so` recipe deliberately omits `main/main.o`
but other objects reference its symbols (`progname`, …).
Add `-Wl,-undefined,dynamic_lookup` only when
`PORTNAME=darwin` so those resolve at load time.
3. **PG 16 ↔ Cloudberry DLSUFFIX cohesion** on darwin only.
Upstream PG 16 commit `b55f62abb2c "Unify DLSUFFIX on Darwin"`
changed macOS module suffix from `.so` to `.dylib`.
Cloudberry's catalog SQL, cdb_init.d scripts, expected/*.out
files still hard-code `$libdir/foo.so`. When PG sees an
explicit `.so` it does NOT re-append `DLSUFFIX`, so catalog
bootstrap breaks (`FATAL: could not access file "foo"`).
This PR pins `DLSUFFIX=.so` on darwin via `src/template/darwin`
(file loaded only on darwin), plus two small follow-ons that
no-op on Linux:
- `configure`'s Python-shared-library probe gets a `.dylib`
fallback only when `$PORTNAME = darwin` (macOS `libpython`
ships as `.dylib` regardless of module DLSUFFIX).
- `src/test/regress/GNUmakefile` uses `$(DLSUFFIX)` instead
of hardcoded `.so` for install/uninstall lines — on Linux
`$(DLSUFFIX)` expands to `.so`, so this is a textual
no-change after make-time substitution.
4. **PAX portability for non-Linux**. Four focused commits:
- **liburing optional**: `configure` keeps the hard error on
Linux (`case $host_os in linux*) AC_MSG_ERROR ;;`) but
allows missing on non-Linux. PAX has a pread-based
`SyncFastIO` fallback selected at runtime by
`IOUringFastIO::available()`. C++ code wraps the
`IOUringFastIO` class with `#ifdef __linux__`. Linux gets
identical behavior.
- **C++ portability**: `off64_t` typedef
(`#ifdef __APPLE__`-only), `int64_t` ↔ `int64` signature
alignment (identical type on Linux LP64 — same `long int`),
defensive `#undef Min/Max/IsPowerOf2` around protobuf
includes (no-op on Linux: if old protobuf, no abseil
shadowing; if new protobuf, the same fix is needed there
too), VLA → `std::vector` *only under clang* via
`#if defined(__clang__)` (gcc keeps zero-overhead stack
VLA), `google::protobuf::int64` → `int64_t` (typedef
identical for older protobuf, required for v22+).
- **cmake portability**: gcc-only flags moved behind
`if(APPLE)/else()`; `BUILD_GTEST=OFF` only on darwin so
Linux `make pax-unit-test` continues to work; `-luuid` only
on Linux (`libSystem` provides it on macOS); on macOS only,
build `pax` as a `MODULE` (Mach-O bundle) and link with
`-bundle_loader <postgres> -undefined dynamic_lookup` so
backend globals have a single shared instance between
postgres and pax.so; abseil deps via pkg-config on macOS
(Homebrew protobuf v22+ splits them); `@loader_path`
instead of `$ORIGIN` on macOS. Linux uses the `else`
branches in all cases — original behavior verbatim.
- **paxformat**: the standalone reader. On macOS add
`-Wl,-undefined,dynamic_lookup` to defer PG-backend symbol
references; Linux build path unchanged.
## Commits (kept atomic — order matters for builds)
```
Drop -Werror -Wextra -Wpedantic from ORCA C++ build
numeric: drop bogus 'const' qualifier on init_var_from_str
macOS: provide HZ fallback for UDP interconnect
macOS: defer libpostgres.so's undefined symbols to load time
macOS: pin DLSUFFIX=.so for PG 16 compatibility
macOS: make PAX's liburing dependency optional
macOS: PAX C++ portability
macOS: PAX cmake portability
macOS: build the standalone libpaxformat reader
```
## How I verified
Fresh checkout of this branch, on macOS 13.x x86_64 / Apple clang
14.0.3, with these Homebrew packages installed:
```
bison flex pkg-config libxml2 zstd openssl@3 readline apr apr-util
libevent xerces-c icu4c gettext libyaml lz4 perl protobuf libuv krb5
```
```sh
./configure --prefix=$HOME/install/cbdb-mac --enable-debug --enable-cassert \
--enable-pax --enable-ic-proxy --enable-orafce --enable-tap-tests \
--enable-mapreduce --enable-gpcloud \
--with-perl --with-python --with-libxml --with-openssl \
--with-lz4 --with-pam --with-ldap --with-gssapi \
--with-apr-config=$(brew --prefix apr)/bin/apr-1-config \
--with-includes=<homebrew prefixes joined by ':'> \
--with-libs=<homebrew prefixes joined by ':'>
make -j$(sysctl -n hw.ncpu) \
CUSTOM_COPT="-Wno-error=uninitialized
-Wno-error=gnu-variable-sized-type-not-at-end"
make install
ln -sf $PREFIX/cloudberry-env.sh $PREFIX/greenplum_path.sh
cd gpAux/gpdemo
NUM_PRIMARY_MIRROR_PAIRS=3 make create-demo-cluster
```
After which:
- `gp_segment_configuration` shows 1 coordinator + 1 standby +
3 primaries + 3 mirrors, all `status='u'` / `mode='s'`.
- `CREATE TABLE t USING pax` → insert 1000 rows → `SELECT
count(*)` returns 1000 (ORCA plan used).
- `orafce`, `plperl`, `plpython3u` extensions install + work.
- `gpstate -s` reports `Mirror status = Synchronized` for all
three mirrors.
Linux build path was not re-verified in this PR (no Linux machine
handy), but is structurally unchanged — see "Strict guarantee"
section above. Welcome any CI run to confirm.
## Caveats / out of scope
- **PAX's `io_uring` fast path stays Linux-only.** macOS has no
equivalent. On Linux: unchanged. On macOS: PAX falls back to
the existing pread-based `SyncFastIO`.
- **`uuid-ossp` extension** is not enabled in the configure
command above because Homebrew `e2fsprogs` / `ossp-uuid` are
heavy to pull in. Orthogonal to this PR.
- **GSSAPI on macOS requires Homebrew MIT `krb5`** in `PATH` /
`PKG_CONFIG_PATH`; macOS's system Heimdal lacks
`gss_store_cred_into` (used by PG 16). Environment setup
detail, no code change in this PR.
### What does this PR do?
<!-- Brief overview of the changes, including any major features or fixes -->
### Type of Change
- [ ] Bug fix (non-breaking change)
- [ ] New feature (non-breaking change)
- [ ] Breaking change (fix or feature with breaking changes)
- [ ] Documentation update
### Breaking Changes
<!-- Remove if not applicable. If yes, explain impact and migration path -->
### Test Plan
<!-- How did you test these changes? -->
- [ ] Unit tests added/updated
- [ ] Integration tests added/updated
- [ ] Passed `make installcheck`
- [ ] Passed `make -C src/test installcheck-cbdb-parallel`
### Impact
<!-- Remove sections that don't apply -->
**Performance:**
<!-- Any performance implications? -->
**User-facing changes:**
<!-- Any changes visible to users? -->
**Dependencies:**
<!-- New dependencies or version changes? -->
### Checklist
- [ ] Followed [contribution
guide](https://cloudberry.apache.org/contribute/code)
- [ ] Added/updated documentation
- [ ] Reviewed code for security implications
- [ ] Requested review from [cloudberry
committers](https://github.com/orgs/apache/teams/cloudberry-committers)
### Additional Context
<!-- Any other information that would help reviewers? Remove if none -->
### CI Skip Instructions
<!--
To skip CI builds, add the appropriate CI skip identifier to your PR title.
The identifier must:
- Be in square brackets []
- Include the word "ci" and either "skip" or "no"
- Only use for documentation-only changes or when absolutely necessary
-->
---
<!-- Join our community:
- Mailing list:
[[email protected]](https://lists.apache.org/[email protected])
(subscribe: [email protected])
- Discussions: https://github.com/apache/cloudberry/discussions -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
