AlinsRan commented on issue #2708: URL: https://github.com/apache/apisix-ingress-controller/issues/2708#issuecomment-5866160550
@manueljishi Thanks for the detailed write-up. Your root cause is correct. It is also a separate problem from the one this issue was opened for, which is on `apisix-standalone` and has no etcd involved. ### Who is affected This only happens when an etcd-mode deployment upgrades from a 2.0.0 release candidate to 2.0.0 GA or later. The release candidates shipped an ADC version before 0.22.0, and every GA release ships a later one: | apisix-ingress-controller | ADC | |---|---| | 2.0.0-rc1 ~ rc3 | 0.20.0 | | 2.0.0-rc4 ~ rc5 | 0.21.0 | | 2.0.0, 2.0.1, 2.1.0 | 0.23.1 | ADC 0.22.0 changed how a service's upstream is stored in etcd. Before 0.22.0 the upstream is embedded in the service. From 0.22.0 on it is a separate `/apisix/upstreams/<id>` object that the service references through `upstream_id`. Data written by an rc release is in the old format, and a later ADC does not migrate it when it updates a service, which leads to the stale nodes and the 400 you saw. These deployments are not affected: - fresh installs of 2.0.0 GA or later, since everything is written in the new format from the start; - `apisix-standalone` deployments, which have nothing stored in etcd. ### Solution: blue/green upgrade To upgrade from an rc release, do a blue/green switch instead of upgrading in place: 1. Deploy a new APISIX with an empty etcd, or with a new `etcd.prefix`. 2. Point the GatewayProxy at the new APISIX's Admin API. The controller then pushes the full configuration to it. Because etcd is empty, every service goes through ADC's create path, which writes the upstream object before the service, so everything is stored in the new format. 3. Verify the new APISIX, then shift traffic to it gradually. 4. Remove the old APISIX and its etcd. Keep the window between step 2 and step 3 short. The old APISIX stops receiving updates once the GatewayProxy points at the new one. Any configuration that is not managed through Kubernetes resources, such as objects created directly through the Admin API or custom `config.yaml` settings, has to be carried over to the new APISIX separately. ### Chart Agreed. The chart should include an upgrade note about crossing ADC 0.22.0, since most users never set the sidecar version explicitly and a chart upgrade crosses that boundary without any warning. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
