thejens opened a new issue, #4663: URL: https://github.com/apache/arrow-adbc/issues/4663
Hi! Whilst working with dbt's ADBC integration, me and Claude bumped into this issue. Below report is written by Claude, but I have verified it seems to be a real issue. --- ### What happened? The Rust driver manager opens driver libraries with `RTLD_LAZY`, so a driver whose own dependency symbols cannot be resolved loads **successfully**. The unresolved symbols are bound at first *call* instead, and for a driver linked with `-undefined dynamic_lookup` (flat namespace) they bind to `NULL` — so the process jumps to address `0` and dies with `SIGSEGV`, arbitrarily far from the load site and with no diagnostic. [`DriverLibrary::load_library`](https://github.com/apache/arrow-adbc/blob/00dfb8b9c0f776940781ea42c5b32b6bf3618652/rust/driver_manager/src/search.rs#L280-L324) passes the flags [here](https://github.com/apache/arrow-adbc/blob/00dfb8b9c0f776940781ea42c5b32b6bf3618652/rust/driver_manager/src/search.rs#L320): ```rust libloading::os::unix::Library::open( Some(filename.as_ref()), libloading::os::unix::RTLD_LAZY | libloading::os::unix::RTLD_LOCAL | RTLD_NODELETE, ) ``` The error handling here is fine — the `dlopen` result *is* checked. The problem is purely that `RTLD_LAZY` defers the failure past the point where it could be reported. The same `dlopen` with `RTLD_NOW` returns a precise, actionable error naming the missing symbol. The two differ only in that one flag, on the identical library: | flag | `dlopen` result | outcome | |---|---|---| | `RTLD_NOW` | fails | `symbol not found in flat namespace '_OPENSSL_sk_num'` | | `RTLD_LAZY` | **succeeds** | `SIGSEGV` at `0x0` on first call | This matters more for a driver manager than for a typical `dlopen` caller: the whole point is loading third-party plugins whose link-time hygiene you do not control, and `RTLD_LAZY` converts "this driver is broken" into "your process crashed somewhere unrelated". ### Real-world instance A PostgreSQL driver built with `-undefined dynamic_lookup` on macOS ARM64 leaves the OpenSSL symbols libpq needs unresolved. `ManagedDriver::load_dynamic_from_filename` returns `Ok`, `AdbcDriverInit` is resolved and called, and the process dies at the first `OPENSSL_*` call inside libpq. `lldb` shows `EXC_BAD_ACCESS (code=1, address=0x0)` with `frame #0: 0x0000000000000000` — no symbol, no backtrace, nothing pointing at the driver. Because the crash surfaces at the first *connection attempt* rather than at load, this reads as a bug in whatever was executing at the time. It took a `dlopen` probe outside the process to identify the real cause. ### Stack Trace ``` * thread #2, stop reason = EXC_BAD_ACCESS (code=1, address=0x0) frame #0: 0x0000000000000000 error: memory read failed for 0x0 ``` ### How can we reproduce the bug? Self-contained; no ADBC build required. It isolates the flag choice, which is the whole of the bug. `fake_driver.c` — stands in for any driver linked with `-undefined dynamic_lookup` that left a dependency symbol unresolved: ```c extern int OPENSSL_sk_num(const void *); int AdbcDriverInit(int version, void *raw_driver, void *error) { (void)version; (void)raw_driver; (void)error; return OPENSSL_sk_num(0); /* unresolved at load time */ } ``` `loader.c` — the driver manager's flags vs. strict ones: ```c #include <dlfcn.h> #include <stdio.h> int main(int argc, char **argv) { int lazy = (argc > 2 && argv[2][0] == 'l'); int flags = (lazy ? RTLD_LAZY : RTLD_NOW) | RTLD_LOCAL; printf("dlopen(%s)\n", lazy ? "RTLD_LAZY" : "RTLD_NOW"); void *h = dlopen(argv[1], flags); if (!h) { printf(" -> load error (clean): %s\n", dlerror()); return 0; } printf(" -> load reported SUCCESS\n"); int (*init)(int, void *, void *) = (int (*)(int, void *, void *))dlsym(h, "AdbcDriverInit"); printf(" -> calling AdbcDriverInit ...\n"); fflush(stdout); init(1000000, 0, 0); return 0; } ``` ```bash cc -dynamiclib -o libfake_driver.dylib fake_driver.c \ -Wl,-undefined,dynamic_lookup -Wl,-flat_namespace cc -o loader loader.c # what a strict loader would do ./loader ./libfake_driver.dylib now; echo "exit=$?" # what the driver manager does ./loader ./libfake_driver.dylib lazy; echo "exit=$?" ``` Output: ``` dlopen(RTLD_NOW) -> load error (clean): dlopen(./libfake_driver.dylib, 0x0006): symbol not found in flat namespace '_OPENSSL_sk_num' exit=0 dlopen(RTLD_LAZY) -> load reported SUCCESS -> calling AdbcDriverInit ... exit=139 # SIGSEGV ``` ### Suggested fix Use `RTLD_NOW` in `load_library`, so an unloadable driver is reported as a load error rather than becoming a latent crash. The existing `map_err(libloading_error_to_adbc_error)` then surfaces the linker's own message, which already names the offending symbol. `RTLD_NODELETE` and `RTLD_LOCAL` are unaffected — the reasons documented in the comment above those lines (the Go runtime not honouring `-z nodelete` on macOS) are orthogonal to eager vs. lazy binding. If eager binding is deliberate for startup cost, an alternative is to keep `RTLD_LAZY` but retry with `RTLD_NOW` when a load succeeds, so the diagnostic is at least available — though that pays the cost twice and still leaves the default path crashing. ### Environment/Setup - macOS 15 (Darwin 25.6.0), ARM64 - `adbc_driver_manager` (Rust) 0.22.0 - Also present on `apache/arrow-adbc` `main` at `00dfb8b9c0f776940781ea42c5b32b6bf3618652` — the flags are unchanged there. The `-undefined dynamic_lookup` linkage that triggers it is a property of how a given driver binary was built, not of ADBC itself; the report here is about the driver manager turning that into an undiagnosable crash rather than a load error. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
