fracasula opened a new issue, #20571: URL: https://github.com/apache/pulsar/issues/20571
### Search before asking - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. ### Version Pulsar version v3.0.0 Kubernetes version: ``` Server Version: version.Info{Major:"1", Minor:"22+", GitVersion:"v1.22.17-eks-0a21954", GitCommit:"cd59ae711d1a42f73ca91ede375c2748d7416174", GitTreeState:"clean", BuildDate:"2023-04-15T00:32:54Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"} ``` Golang Pulsar client version: ``` github.com/apache/pulsar-client-go v0.10.0 ``` ### Minimal reproduce step While consuming our brokers crash with a fatal error (SIGSEGV) and keep on crashing periodically until the backlog is fully consumed. ### What did you expect to see? Normal operation while consuming, perhaps an error report in the logs either server or client but without panics and crashes. ### What did you see instead? Panics in the Golang client: https://ibb.co/tP8yfSV SIGSEGVs in the brokers: ``` # # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x00007f334ad41632, pid=1, tid=524 # # JRE version: OpenJDK Runtime Environment (17.0.5+8) (build 17.0.5+8-Ubuntu-2ubuntu120.04) # Java VM: OpenJDK 64-Bit Server VM (17.0.5+8-Ubuntu-2ubuntu120.04, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64) # Problematic frame: # V [libjvm.so+0xae2632] # # Core dump will be written. Default location: /pulsar/core.1 # # An error report file with more information is saved as: # /pulsar/hs_err_pid1.log # # Compiler replay data is saved as: # /pulsar/replay_pid1.log # # If you would like to submit a bug report, please visit: # Unknown # [error occurred during error reporting (), id 0xb, SIGSEGV (0xb) at pc=0x00007f334b671941] ``` And then the [hs_err_pid1.log](https://file.io/melBufCdoRXJ) file. ### Anything else? We tried leaving that topic alone altogether and redicted both producers and consumers to a brand new topic but it didn't help. The brokers keep on crashing. I think we'll try next to wipe the entire cluster and try again. By the way, this happened in a production environment and we have no way to resume operations. We're thinking about downgrading to Pulsar v2.11.1 as well but looking at the changelogs we can't figure out if upcoming upfixes are applied to both v2 and v3 versions, please shed some light on this. ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
