rastiehaiev opened a new issue, #11148: URL: https://github.com/apache/apisix/issues/11148
### Current Behavior I have an APISIX Gateway 3.6.0 installation with 1 upstream, 1 service, 1 route, and 1 ssl. After upgrading to 3.9.0 the most of the requests to my API end up with SSL error like this (curl verbose response): ``` ... * Send failure: Broken pipe * LibreSSL SSL_write: LibreSSL/3.3.6: error:02FFF020:system library:func(4095):Broken pipe, errno 32 * Failed sending HTTP request * Connection #0 to host example.com left intact curl: (55) getpeername() failed with errno 22: Invalid argument ``` I've managed to reproduce it locally using docker-compose. I've added steps to reproduce and created a small repo with all needed files (docker-compose.yaml, apisix config etc., ssl keys): https://github.com/rastiehaiev/apisix-migration-to-3_9_0-bug. ### Expected Behavior API requests succeed after upgrading APISIX to 3.9.0. ### Error Logs ``` 2024/04/10 16:53:53 [alert] 56#56: *2224 ignoring stale global SSL error (SSL: error:1C800064:Provider routines::bad decrypt) while SSL handshaking, client: MY_IP, server: 0.0.0.0:9443 2024-04-10T16:53:53.994117366Z 2024/04/10 16:53:53 [crit] 56#56: *2224 SSL_do_handshake() failed (SSL: error:1C800064:Provider routines::bad decrypt) while SSL handshaking, client: MY_IP, server: 0.0.0.0:9443 ``` ### Steps to Reproduce 1. Run APISIX Gateway 3.6.0 + etcd using docker-compose. 2. Create upstream, service, route and ssl like this (I used Admin API to create these entities): ``` --- upstreams: - id: api name: api desc: API upstream type: roundrobin scheme: https timeout: connect: 5 read: 120 send: 10 nodes: - host: "httpbin.org" port: 443 weight: 1 services: - id: api-http name: api-http plugins: gzip: min_length: 1100 comp_level: 6 types: - text/plain - application/json vary: true routes: - id: api-http name: api-http uri: /* hosts: - "example.com" service_id: api-http upstream_id: api priority: 100 ssls: - id: ssl cert: "../certs/example.pem" key: "../certs/example.key" snis: - "example.com" ``` 3. Make sure apisix installation works as expected: ``` for i in {1..10}; do curl -k --resolve 'example.com:9443:127.0.0.1' 'https://example.com:9443/get?query=1' -vvv -I || { echo "curl failed at attempt $i"; break; }; done ``` 4. Run `docker-compose down` (and make sure etcd volume is not deleted). 5. Change the version of APISIX image from 3.6.0 to 3.9.0. Execute `docker-compose up -d` (there is some [breaking change in configuration](https://github.com/apache/apisix/blob/master/CHANGELOG.md#change) introduced in 3.9.0 - `enable_http2` should be on `apisix` level - make sure to change it). 6. Try curling your APISIX gateway once again: ``` for i in {1..10}; do curl -k --resolve 'example.com:9443:127.0.0.1' 'https://example.com:9443/get?query=1' -vvv -I || { echo "curl failed at attempt $i"; break; }; done ``` **Expected**: requests succeed. **Actual**: some requests fail with SSL error. To fix this, the entities created in step 2 should be re-created. Once they are re-created, the error disappears. More verbose steps to reproduce are in the repository I created, please take a look: https://github.com/rastiehaiev/apisix-migration-to-3_9_0-bug. ### Environment - APISIX version (run `apisix version`): 3.6.0 - 3.6.9 - Operating system (run `uname -a`): Darwin NAME 23.4.0 Darwin Kernel Version 23.4.0: Fri Mar 15 00:11:05 PDT 2024; root:xnu-10063.101.17~1/RELEASE_X86_64 x86_64 - OpenResty / Nginx version (run `openresty -V` or `nginx -V`): nginx version: openresty/1.25.3.1, built by gcc 10.2.1 20210110 (Debian 10.2.1-6), built with OpenSSL 3.2.0 23 Nov 2023, TLS SNI support enabled - etcd version, if relevant (run `curl http://127.0.0.1:9090/v1/server_info`): bitnami/etcd:3.4.15 - APISIX Dashboard version, if relevant: irrelevant -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
