shantanu10 opened a new issue, #2171: URL: https://github.com/apache/apisix-ingress-controller/issues/2171
### Description Hi team, I am facing one issue where I am trying to use a custom plugin with apisix when deployed in composite architecture mode. I followed the approach mentioned here: https://apisix.apache.org/blog/2023/10/18/ingress-apisix/#install-apisix-ingress-controller https://apisix.apache.org/docs/ingress-controller/composite/ and to create a custom plugin I used the approach mentioned here: https://apisix.apache.org/docs/ingress-controller/tutorials/using-custom-plugins/ After this when I tried to deployed the following apisix route resource, its not able to get synced. I checked the ingress controller logs and found these errors: ``` 2024-02-24T21:53:05+08:00 error apisix/apisix_route.go:379 failed to sync ApisixRoute to apisix {"error": "1 error occurred:\n\t* unknown plugin [custom-oauth]\n\n"} 2024-02-24T21:53:05+08:00 warn apisix/apisix_route.go:481 sync ApisixRoute failed, will retry {"object": {"Type":2,"Object":{"Key":"shared/bytebase-shared-test","OldObject":{"Object":{"metadata":{"name":"bytebase-shared-test","namespace":"shared","uid":"5653149f-27cc-4b12-aedc-59d81fd01d9f","resourceVersion":"9445672","generation":1,"creationTimestamp":"2024-02-23T09:52:43Z","annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apisix.apache.org/v2\",\"kind\":\"ApisixRoute\",\"metadata\":{\"annotations\":{},\"name\":\"bytebase-shared-test\",\"namespace\":\"shared\"},\"spec\":{\"http\":[{\"backends\":[{\"serviceName\":\"bytebase-shared\",\"servicePort\":80,\"weight\":10}],\"match\":{\"hosts\":[\"bytebase-test.internal.saas.syfe.com\"],\"paths\":[\"/*\"]},\"name\":\"rule1\"}]}}\n"},"managedFields":[{"manager":"apisix-ingress-controller","operation":"Update","apiVersion":"apisix.apache.org/v2","time":"2024-02-23T09:52:43Z","fieldsType":"FieldsV1","field sV1":{"f:status":{".":{},"f:conditions":{}}},"subresource":"status"},{"manager":"kubectl-client-side-apply","operation":"Update","apiVersion":"apisix.apache.org/v2","time":"2024-02-23T09:52:43Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}}},"f:spec":{".":{},"f:http":{}}}}]},"spec":{"http":[{"name":"rule1","match":{"paths":["/*"],"hosts":["bytebase-test.internal.saas.syfe.com"]},"backends":[{"serviceName":"bytebase-shared","servicePort":80,"weight":10}],"websocket":false,"authentication":{"enable":false,"type":"","keyAuth":{},"jwtAuth":{},"ldapAuth":{}}}]},"status":{"conditions":[{"type":"ResourcesAvailable","status":"True","observedGeneration":1,"lastTransitionTime":null,"reason":"ResourcesSynced","message":"Sync Successfully"}]}}},"GroupVersion":"apisix.apache.org/v2"},"OldObject":null,"Tombstone":null}, "error": "1 error occurred:\n\t* unknown plugin [custom-oauth]\n\n"} ``` To verify whether custom plugin is actually loaded by apisix, I exec into the ingress-controller pod and ran the admin api for plugins (https://apisix.apache.org/docs/apisix/admin-api/#plugin-api). I can see that the plugin was there in the response. I also got the response for the `plugins/{{plugin_id}}` api for my custom plugin (custom-oauth). So I can see that the error is only coming when ingress-controller is trying to sync the apisixRoute with apisix. Somehow it's not able to find the custom plugin. The issue is not happening when I am using any built-in plugin. I really need some help out here. I will be very grateful if you can help here. Thanks in advance. ### Changes I made in the [composite.yaml](https://github.com/apache/apisix-ingress-controller/blob/master/samples/deploy/composite.yaml) ``` apisix: enable_control: true enable_reuseport: true extra_lua_path: /usr/local/apisix/custom_plugins/?.lua # change 1 ... ... plugins: ... ... - ext-plugin-post-resp # priority: -4000 - custom-oauth # change 2 ... ... --- apiVersion: apps/v1 kind: Deployment metadata: name: ingress-apisix-composite-deployment namespace: ingress-apisix spec: replicas: 1 selector: matchLabels: app.kubernetes.io/name: ingress-apisix-composite-deployment template: metadata: labels: app.kubernetes.io/name: ingress-apisix-composite-deployment spec: volumes: - name: apisix-config-yaml-configmap configMap: name: apisix-gw-config.yaml defaultMode: 420 - name: plugin-custom-oauth-config # change 3 configMap: name: custom-oauth-config defaultMode: 0744 ... ... volumeMounts: - name: apisix-config-yaml-configmap mountPath: /usr/local/apisix/conf/config.yaml subPath: config.yaml - mountPath: /usr/local/apisix/custom_plugins/apisix/plugins/custom-oauth.lua # change 4 name: plugin-custom-oauth-config subPath: custom-oauth.lua ... ... --- # Change 5 apiVersion: v1 data: custom-oauth.lua: | -- some required functionalities are provided by apisix.core local core = require("apisix.core") -- define the schema for the Plugin local schema = { type = "object", properties = { body = { description = "custom response to replace the Upstream response with.", type = "string" }, }, required = {"body"}, } local plugin_name = "custom-oauth" local metadata_schema = { name = "custom-oauth", priority = "30000" } -- custom Plugins usually have priority between 1 and 99 -- higher number = higher priority local _M = { version = 0.1, priority = 30000, name = plugin_name, schema = schema, metadata_schema = metadata_schema } -- verify the specification function _M.check_schema(conf) return core.schema.check(schema, conf) end -- run the Plugin in the access phase of the OpenResty lifecycle function _M.access(conf, ctx) return 200, conf.body end return _M kind: ConfigMap metadata: creationTimestamp: null name: custom-oauth-config namespace: ingress-apisix --- # Change 6 apiVersion: apisix.apache.org/v2 kind: ApisixRoute metadata: name: bytebase-shared-test namespace: shared spec: http: - name: rule1 match: hosts: - bytebase-test.internal.saas.syfe.com paths: - /* backends: - serviceName: bytebase-shared servicePort: 80 weight: 10 plugins: - name: "custom-oauth" enable: true config: body: "Hello from your custom Plugin!" ``` ### Environment - APISIX version (run `apisix version`): `3.2.2` - Operating system (run `uname -a`): ```Linux ingress-apisix-composite-deployment-7888c84f64-6chvc 5.10.205-195.807.amzn2.x86_64 apache/apisix#1 SMP Tue Jan 16 18:28:59 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux ``` - OpenResty / Nginx version (run `openresty -V` or `nginx -V`): ``` nginx version: openresty/1.21.4.1 built by gcc 9.3.1 20200408 (Red Hat 9.3.1-2) (GCC) built with OpenSSL 1.1.1s 1 Nov 2022 ``` - etcd version, if relevant (run `curl http://127.0.0.1:9090/v1/server_info`): ``` {"etcdserver":"3.5.0","etcdcluster":"3.5.0"} ``` (Its the version I got by hitting `curl http://127.0.0.1:12379/version` as its a mock etcd service) - APISIX Dashboard version, if relevant: not using - Plugin runner version, for issues related to plugin runners: not using - LuaRocks version, for installation issues (run `luarocks --version`): not using -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
