caldempsey commented on PR #152:
URL: https://github.com/apache/spark-connect-go/pull/152#issuecomment-3067308565
I looked into the panic `signal 13 received but handler not on signal
stack... fatal error: non-Go code set up signal handler without SA_ONSTACK
flag` that was happening in my application w/ ODBC and Spark Connect loaded
side by side. Turns out it only occurs when running the Databricks ODBC driver
alongside Spark Connect in the same process - run either one alone and
everything works fine.
I added some socket monitoring with `lsof` into the Go app (get its own PID
and pass it through and print stats on a ticker) to see what was going on and
found both libraries maintain their own connections to Databricks through port
443:
```bash
ODBC Connections: 172.17.0.2:42562->44.234.192.45:443
172.17.0.2:42568->44.234.192.45:443
Spark Connect: 172.17.0.2:60452->44.234.192.46:443
172.17.0.2:41742->44.234.192.45:443
```
I _**THINK**_ the issue is becaauuuuse POSIX signals get delivered to the
entire process, not individual sockets. I'm guessing the ODBC driver
(`/opt/simba/spark/lib/64/libsparkodbc_sb64.so`) installs signal handlers when
it loads, but doesn't use the `SA_ONSTACK` flag that Go requires. So when
either library hits a broken pipe, the kernel sends SIGPIPE to the process, the
ODBC handler catches it, and Go panics because the handler isn't set up
correctly for Go's stack management.
With Spark Connect v40's new streaming and context cancellation support,
SIGPIPE happens more often - cancel a context mid-stream and you'll likely
trigger it. The ODBC driver's process-wide signal handler intercepts these
signals regardless of which library's socket actually broke, while Spark
Connect seems to handle the termination gracefully. The 'fix' here is really
sweeping the issue under the rug by adding `signal.Ignore(syscall.SIGPIPE)` at
startup e.g.
```go
package main
func init() {
signal.Ignore(syscall.SIGPIPE)
}
```
When you call signal.Ignore(syscall.SIGPIPE), the entire process ignores
SIGPIPE signals. The kernel simply doesn't deliver them to the Go process at
all. Both libraries already handle EPIPE errors properly through return values,
so they don't need the signal. A C++ dev I know (who I can't mention) said this
is actually the modern way to handle broken pipes - check return values instead
of relying on signals.
So with this, Spark Connect streaming and ODBC can coexist in the same
process without issues. Pretty important for services that need both
traditional ODBC connectivity (faster than gRPC) and Spark Connect's DataFrame
capabilities (easier to use than ODBC).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
