bzp2010 commented on code in PR #12686:
URL: https://github.com/apache/apisix/pull/12686#discussion_r2741867364
##########
docs/en/latest/plugins/opentelemetry.md:
##########
@@ -153,37 +168,152 @@ In OpenTelemetry collector's log, you should see
information similar to the foll
```text
2024-02-18T17:14:03.825Z info ResourceSpans #0
-Resource SchemaURL:
-Resource attributes:
- -> telemetry.sdk.language: Str(lua)
- -> telemetry.sdk.name: Str(opentelemetry-lua)
- -> telemetry.sdk.version: Str(0.1.1)
- -> hostname: Str(e34673e24631)
- -> service.name: Str(APISIX)
ScopeSpans #0
ScopeSpans SchemaURL:
InstrumentationScope opentelemetry-lua
Span #0
- Trace ID : fbd0a38d4ea4a128ff1a688197bc58b0
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : 905f850f13e32bfb
+ ID : 5a3835b61110d942
+ Name : http_router_match
+ Kind : Internal
+ Start time : 2025-10-24 06:58:04.430430976 +0000 UTC
+ End time : 2025-10-24 06:58:04.431542016 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #1
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : 905f850f13e32bfb
+ ID : 4ab25e2b92f394e1
+ Name : resolve_dns
+ Kind : Internal
+ Start time : 2025-10-24 06:58:04.432521984 +0000 UTC
+ End time : 2025-10-24 06:58:04.44903296 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #2
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : 905f850f13e32bfb
+ ID : 3620c0f05dd2be4f
+ Name : apisix.phase.header_filter
+ Kind : Internal
+ Start time : 2025-10-24 06:58:06.960481024 +0000 UTC
+ End time : 2025-10-24 06:58:06.960510976 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #3
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : 4c5f3476f62a7e8a
+ ID : a9bfad7bb6986e41
+ Name : apisix.phase.body_filter
+ Kind : Internal
+ Start time : 2025-10-24 06:58:06.960579072 +0000 UTC
+ End time : 2025-10-24 06:58:06.96059008 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #4
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : b2994675df6baa83
+ ID : 26705f9c47584a5b
+ Name : apisix.phase.delayed_body_filter.opentelemetry
+ Kind : Internal
+ Start time : 2025-10-24 06:58:06.960613888 +0000 UTC
+ End time : 2025-10-24 06:58:06.960687104 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #5
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : 4c5f3476f62a7e8a
+ ID : b2994675df6baa83
+ Name : apisix.phase.delayed_body_filter
+ Kind : Internal
+ Start time : 2025-10-24 06:58:06.96059904 +0000 UTC
+ End time : 2025-10-24 06:58:06.960692992 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #6
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : 905f850f13e32bfb
+ ID : 4c5f3476f62a7e8a
+ Name : apisix.phase.body_filter
+ Kind : Server
+ Start time : 2025-10-24 06:58:06.96056704 +0000 UTC
+ End time : 2025-10-24 06:58:06.960698112 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #7
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : 2024d73d32cbd81b
+ ID : 223c64fb691a24e8
+ Name : apisix.phase.body_filter
+ Kind : Internal
+ Start time : 2025-10-24 06:58:06.961624064 +0000 UTC
+ End time : 2025-10-24 06:58:06.961635072 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #8
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : fd193dd24c618f60
+ ID : 8729ad6e0d94a23b
+ Name : apisix.phase.delayed_body_filter.opentelemetry
+ Kind : Internal
+ Start time : 2025-10-24 06:58:06.961648896 +0000 UTC
+ End time : 1970-01-01 00:00:00 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #9
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : 2024d73d32cbd81b
+ ID : fd193dd24c618f60
+ Name : apisix.phase.delayed_body_filter
+ Kind : Internal
+ Start time : 2025-10-24 06:58:06.961641984 +0000 UTC
+ End time : 1970-01-01 00:00:00 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #10
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : 905f850f13e32bfb
+ ID : 2024d73d32cbd81b
+ Name : apisix.phase.body_filter
+ Kind : Server
+ Start time : 2025-10-24 06:58:06.960980992 +0000 UTC
+ End time : 1970-01-01 00:00:00 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #11
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : cfb0b4603dc2e385
+ ID : 905f850f13e32bfb
+ Name : apisix.phase.access
+ Kind : Server
+ Start time : 2025-10-24 06:58:04.427932928 +0000 UTC
+ End time : 1970-01-01 00:00:00 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #12
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
Parent ID :
- ID : af3dc7642104748a
- Name : GET /anything
+ ID : cfb0b4603dc2e385
+ Name : GET /headers
Kind : Server
- Start time : 2024-02-18 17:14:03.763244032 +0000 UTC
- End time : 2024-02-18 17:14:03.920229888 +0000 UTC
+ Start time : 2025-10-24 06:58:04.432427008 +0000 UTC
+ End time : 2025-10-24 06:58:06.962299904 +0000 UTC
Status code : Unset
Status message :
Attributes:
-> net.host.name: Str(127.0.0.1)
-> http.method: Str(GET)
-> http.scheme: Str(http)
- -> http.target: Str(/anything)
- -> http.user_agent: Str(curl/7.64.1)
+ -> http.target: Str(/headers)
+ -> http.user_agent: Str(curl/8.16.0)
-> apisix.route_id: Str(otel-tracing-route)
-> apisix.route_name: Empty()
- -> http.route: Str(/anything)
+ -> http.route: Str(/headers)
-> http.status_code: Int(200)
Review Comment:
Ditto. check them again against the conventions
https://opentelemetry.io/docs/specs/semconv/registry/attributes/http/
For previously added attributes that do not comply with the convention, you
may retain them, but please add a new attribute that complies with the
convention.
##########
docs/zh/latest/plugins/opentelemetry.md:
##########
@@ -152,37 +167,152 @@ curl "http://127.0.0.1:9080/anything";
```text
2024-02-18T17:14:03.825Z info ResourceSpans #0
-Resource SchemaURL:
-Resource attributes:
- -> telemetry.sdk.language: Str(lua)
- -> telemetry.sdk.name: Str(opentelemetry-lua)
- -> telemetry.sdk.version: Str(0.1.1)
- -> hostname: Str(e34673e24631)
- -> service.name: Str(APISIX)
ScopeSpans #0
ScopeSpans SchemaURL:
InstrumentationScope opentelemetry-lua
Span #0
- Trace ID : fbd0a38d4ea4a128ff1a688197bc58b0
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : 905f850f13e32bfb
+ ID : 5a3835b61110d942
+ Name : http_router_match
+ Kind : Internal
+ Start time : 2025-10-24 06:58:04.430430976 +0000 UTC
+ End time : 2025-10-24 06:58:04.431542016 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #1
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : 905f850f13e32bfb
+ ID : 4ab25e2b92f394e1
+ Name : resolve_dns
+ Kind : Internal
+ Start time : 2025-10-24 06:58:04.432521984 +0000 UTC
+ End time : 2025-10-24 06:58:04.44903296 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #2
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : 905f850f13e32bfb
+ ID : 3620c0f05dd2be4f
+ Name : apisix.phase.header_filter
+ Kind : Internal
+ Start time : 2025-10-24 06:58:06.960481024 +0000 UTC
+ End time : 2025-10-24 06:58:06.960510976 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #3
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : 4c5f3476f62a7e8a
+ ID : a9bfad7bb6986e41
+ Name : apisix.phase.body_filter
+ Kind : Internal
+ Start time : 2025-10-24 06:58:06.960579072 +0000 UTC
+ End time : 2025-10-24 06:58:06.96059008 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #4
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : b2994675df6baa83
+ ID : 26705f9c47584a5b
+ Name : apisix.phase.delayed_body_filter.opentelemetry
+ Kind : Internal
+ Start time : 2025-10-24 06:58:06.960613888 +0000 UTC
+ End time : 2025-10-24 06:58:06.960687104 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #5
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : 4c5f3476f62a7e8a
+ ID : b2994675df6baa83
+ Name : apisix.phase.delayed_body_filter
+ Kind : Internal
+ Start time : 2025-10-24 06:58:06.96059904 +0000 UTC
+ End time : 2025-10-24 06:58:06.960692992 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #6
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : 905f850f13e32bfb
+ ID : 4c5f3476f62a7e8a
+ Name : apisix.phase.body_filter
+ Kind : Server
+ Start time : 2025-10-24 06:58:06.96056704 +0000 UTC
+ End time : 2025-10-24 06:58:06.960698112 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #7
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : 2024d73d32cbd81b
+ ID : 223c64fb691a24e8
+ Name : apisix.phase.body_filter
+ Kind : Internal
+ Start time : 2025-10-24 06:58:06.961624064 +0000 UTC
+ End time : 2025-10-24 06:58:06.961635072 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #8
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : fd193dd24c618f60
+ ID : 8729ad6e0d94a23b
+ Name : apisix.phase.delayed_body_filter.opentelemetry
+ Kind : Internal
+ Start time : 2025-10-24 06:58:06.961648896 +0000 UTC
+ End time : 1970-01-01 00:00:00 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #9
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : 2024d73d32cbd81b
+ ID : fd193dd24c618f60
+ Name : apisix.phase.delayed_body_filter
+ Kind : Internal
+ Start time : 2025-10-24 06:58:06.961641984 +0000 UTC
+ End time : 1970-01-01 00:00:00 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #10
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : 905f850f13e32bfb
+ ID : 2024d73d32cbd81b
+ Name : apisix.phase.body_filter
+ Kind : Server
+ Start time : 2025-10-24 06:58:06.960980992 +0000 UTC
+ End time : 1970-01-01 00:00:00 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #11
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : cfb0b4603dc2e385
+ ID : 905f850f13e32bfb
+ Name : apisix.phase.access
+ Kind : Server
+ Start time : 2025-10-24 06:58:04.427932928 +0000 UTC
+ End time : 1970-01-01 00:00:00 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #12
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
Parent ID :
- ID : af3dc7642104748a
- Name : GET /anything
+ ID : cfb0b4603dc2e385
+ Name : GET /headers
Kind : Server
- Start time : 2024-02-18 17:14:03.763244032 +0000 UTC
- End time : 2024-02-18 17:14:03.920229888 +0000 UTC
+ Start time : 2025-10-24 06:58:04.432427008 +0000 UTC
+ End time : 2025-10-24 06:58:06.962299904 +0000 UTC
Status code : Unset
Status message :
Attributes:
- -> net.host.name: Str(127.0.0.1)
- -> http.method: Str(GET)
- -> http.scheme: Str(http)
- -> http.target: Str(/anything)
- -> http.user_agent: Str(curl/7.64.1)
- -> apisix.route_id: Str(otel-tracing-route)
- -> apisix.route_name: Empty()
- -> http.route: Str(/anything)
- -> http.status_code: Int(200)
-{"kind": "exporter", "data_type": "traces", "name": "debug"}
+ -> net.host.name: Str(127.0.0.1)
+ -> http.method: Str(GET)
+ -> http.scheme: Str(http)
+ -> http.target: Str(/headers)
+ -> http.user_agent: Str(curl/8.16.0)
+ -> apisix.route_id: Str(otel-tracing-route)
+ -> apisix.route_name: Empty()
+ -> http.route: Str(/headers)
+ -> http.status_code: Int(200)
Review Comment:
ditto
https://opentelemetry.io/docs/specs/semconv/registry/attributes/http/
##########
docs/zh/latest/plugins/opentelemetry.md:
##########
@@ -152,37 +167,152 @@ curl "http://127.0.0.1:9080/anything";
```text
2024-02-18T17:14:03.825Z info ResourceSpans #0
-Resource SchemaURL:
-Resource attributes:
- -> telemetry.sdk.language: Str(lua)
- -> telemetry.sdk.name: Str(opentelemetry-lua)
- -> telemetry.sdk.version: Str(0.1.1)
- -> hostname: Str(e34673e24631)
- -> service.name: Str(APISIX)
ScopeSpans #0
ScopeSpans SchemaURL:
InstrumentationScope opentelemetry-lua
Span #0
- Trace ID : fbd0a38d4ea4a128ff1a688197bc58b0
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : 905f850f13e32bfb
+ ID : 5a3835b61110d942
+ Name : http_router_match
+ Kind : Internal
+ Start time : 2025-10-24 06:58:04.430430976 +0000 UTC
+ End time : 2025-10-24 06:58:04.431542016 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #1
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : 905f850f13e32bfb
+ ID : 4ab25e2b92f394e1
+ Name : resolve_dns
+ Kind : Internal
+ Start time : 2025-10-24 06:58:04.432521984 +0000 UTC
+ End time : 2025-10-24 06:58:04.44903296 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #2
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : 905f850f13e32bfb
+ ID : 3620c0f05dd2be4f
+ Name : apisix.phase.header_filter
+ Kind : Internal
+ Start time : 2025-10-24 06:58:06.960481024 +0000 UTC
+ End time : 2025-10-24 06:58:06.960510976 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #3
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : 4c5f3476f62a7e8a
+ ID : a9bfad7bb6986e41
+ Name : apisix.phase.body_filter
+ Kind : Internal
+ Start time : 2025-10-24 06:58:06.960579072 +0000 UTC
+ End time : 2025-10-24 06:58:06.96059008 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #4
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : b2994675df6baa83
+ ID : 26705f9c47584a5b
+ Name : apisix.phase.delayed_body_filter.opentelemetry
+ Kind : Internal
+ Start time : 2025-10-24 06:58:06.960613888 +0000 UTC
+ End time : 2025-10-24 06:58:06.960687104 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #5
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : 4c5f3476f62a7e8a
+ ID : b2994675df6baa83
+ Name : apisix.phase.delayed_body_filter
+ Kind : Internal
+ Start time : 2025-10-24 06:58:06.96059904 +0000 UTC
+ End time : 2025-10-24 06:58:06.960692992 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #6
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : 905f850f13e32bfb
+ ID : 4c5f3476f62a7e8a
+ Name : apisix.phase.body_filter
+ Kind : Server
+ Start time : 2025-10-24 06:58:06.96056704 +0000 UTC
+ End time : 2025-10-24 06:58:06.960698112 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #7
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : 2024d73d32cbd81b
+ ID : 223c64fb691a24e8
+ Name : apisix.phase.body_filter
+ Kind : Internal
+ Start time : 2025-10-24 06:58:06.961624064 +0000 UTC
+ End time : 2025-10-24 06:58:06.961635072 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #8
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : fd193dd24c618f60
+ ID : 8729ad6e0d94a23b
+ Name : apisix.phase.delayed_body_filter.opentelemetry
+ Kind : Internal
+ Start time : 2025-10-24 06:58:06.961648896 +0000 UTC
+ End time : 1970-01-01 00:00:00 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #9
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : 2024d73d32cbd81b
+ ID : fd193dd24c618f60
+ Name : apisix.phase.delayed_body_filter
+ Kind : Internal
+ Start time : 2025-10-24 06:58:06.961641984 +0000 UTC
+ End time : 1970-01-01 00:00:00 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #10
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : 905f850f13e32bfb
+ ID : 2024d73d32cbd81b
+ Name : apisix.phase.body_filter
+ Kind : Server
+ Start time : 2025-10-24 06:58:06.960980992 +0000 UTC
+ End time : 1970-01-01 00:00:00 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #11
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : cfb0b4603dc2e385
+ ID : 905f850f13e32bfb
+ Name : apisix.phase.access
+ Kind : Server
+ Start time : 2025-10-24 06:58:04.427932928 +0000 UTC
+ End time : 1970-01-01 00:00:00 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #12
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
Parent ID :
- ID : af3dc7642104748a
- Name : GET /anything
+ ID : cfb0b4603dc2e385
+ Name : GET /headers
Kind : Server
- Start time : 2024-02-18 17:14:03.763244032 +0000 UTC
- End time : 2024-02-18 17:14:03.920229888 +0000 UTC
+ Start time : 2025-10-24 06:58:04.432427008 +0000 UTC
+ End time : 2025-10-24 06:58:06.962299904 +0000 UTC
Status code : Unset
Status message :
Attributes:
- -> net.host.name: Str(127.0.0.1)
- -> http.method: Str(GET)
- -> http.scheme: Str(http)
- -> http.target: Str(/anything)
- -> http.user_agent: Str(curl/7.64.1)
- -> apisix.route_id: Str(otel-tracing-route)
- -> apisix.route_name: Empty()
- -> http.route: Str(/anything)
- -> http.status_code: Int(200)
-{"kind": "exporter", "data_type": "traces", "name": "debug"}
+ -> net.host.name: Str(127.0.0.1)
+ -> http.method: Str(GET)
+ -> http.scheme: Str(http)
+ -> http.target: Str(/headers)
+ -> http.user_agent: Str(curl/8.16.0)
+ -> apisix.route_id: Str(otel-tracing-route)
+ -> apisix.route_name: Empty()
+ -> http.route: Str(/headers)
+ -> http.status_code: Int(200)
Review Comment:
ditto
##########
apisix/ssl/router/radixtree_sni.lua:
##########
@@ -177,9 +179,10 @@ function _M.match_and_set(api_ctx, match_only, alt_sni)
-- with it sometimes
core.log.error("failed to find any SSL certificate by SNI: ", sni)
end
+ tracer.finish(api_ctx.ngx_ctx, tracer.status.ERROR, "failed match SNI")
return false
end
-
+ tracer.finish(api_ctx.ngx_ctx)
Review Comment:
Strictly, this API design is still strange, with many internal details
hidden within the stack data structure. For developers, it presents an opaque
data structure.
If any span of code begins with `tracer.start` but fails to conclude with
`tracer.finish`, the entire tracing process will encounter an error. This
unclosed span will encompass all subsequent spans, which is not the intended
behavior.
If we examine OTEL implementations in other major programming languages, we
find that many of them follow the following pattern:
Use the tracer's method to create a span, then obtain an instance of that
span. After internal operations complete, use the span instance's methods to
actively terminate the specified span. The ctx will be used to track the
current span, ensuring that parent-child relationships are correctly maintained
when creating child spans.
```lua
-- context-based
local parent_span = tracer.start(ctx, "parent")
local child_span = tracer.start(ctx, "child")
child_span:end()
parent_span:end()
```
ref:
https://opentelemetry.io/docs/languages/go/instrumentation/#create-nested-spans
ref:
https://opentelemetry.io/docs/languages/php/instrumentation/#create-nested-spans
ref:
https://opentelemetry.io/docs/languages/js/instrumentation/#create-nested-spans
ref: https://opentelemetry.io/docs/languages/java/api/#span
ref:
https://opentelemetry.io/docs/languages/cpp/instrumentation/#create-nested-spans
Alternatively, explicitly pass the parent span instance to the child span to
achieve precise hierarchical control.
```lua
-- pass span directly
local parent_span = tracer.start("parent")
local child_span = tracer.start("child", parent_span)
child_span:end()
parent_span:end()
```
Note that the following Rust code typically terminates spans automatically
via drop calls when the span instance's variable scopes end.
ref:
https://docs.rs/tracing/latest/tracing/span/index.html#span-relationships
None of the OTEL libraries for these languages organize and manage the
entire span stack using only global state. Each one terminates spans by
returning span instances and requiring developers to explicitly call span.end()
when operations conclude.
Our current approach requires any developer writing trace code to be
familiar with the stack mechanism we've created and to have complete clarity
about where their spans appear in the stack, whether parent spans exist,
whether they've closed as expected, or any other state. Otherwise, they cannot
write correct code. Therefore, I disagree with the current API design.
I have already pointed this out in
https://github.com/apache/apisix/pull/12686#discussion_r2471659658, and
@membphis has clearly reiterated this point through images.
##########
apisix/plugins/opentelemetry.lua:
##########
@@ -376,49 +377,101 @@ function _M.rewrite(conf, api_ctx)
ngx_var.opentelemetry_span_id = span_context.span_id
end
+ if not ctx:span():is_recording() and ngx.ctx.tracing then
+ ngx.ctx.tracing.skip = true
+ end
+
api_ctx.otel_context_token = ctx:attach()
-- inject trace context into the headers of upstream HTTP request
trace_context_propagator:inject(ctx, ngx.req)
end
-function _M.delayed_body_filter(conf, api_ctx)
- if api_ctx.otel_context_token and ngx.arg[2] then
- local ctx = context:current()
- ctx:detach(api_ctx.otel_context_token)
- api_ctx.otel_context_token = nil
+local function create_child_span(tracer, parent_span_ctx, spans, span_idx)
+ local span = spans[span_idx]
+ if not span or span.finished then
+ return
+ end
+ span.finished = true
+ local new_span_ctx, new_span = tracer:start(parent_span_ctx, span.name,
+ {
+ kind = span.kind,
+ attributes = span.attributes,
+ })
+ new_span.start_time = span.start_time
+
+ for _, idx in ipairs(span.child_ids or {}) do
+ create_child_span(tracer, new_span_ctx, spans, idx)
+ end
+ if span.status then
+ new_span:set_status(span.status.code, span.status.message)
+ end
+ new_span:finish(span.end_time)
+end
- -- get span from current context
- local span = ctx:span()
- local upstream_status = core.response.get_upstream_status(api_ctx)
- if upstream_status and upstream_status >= 500 then
- span:set_status(span_status.ERROR,
- "upstream response status: " .. upstream_status)
- end
- span:set_attributes(attr.int("http.status_code", upstream_status))
+local function inject_core_spans(root_span_ctx, api_ctx, conf)
+ local tracing = api_ctx.ngx_ctx.tracing
+ if not tracing then
+ return
+ end
- span:finish()
+ local span = root_span_ctx:span()
+
+ local metadata = plugin.plugin_metadata(plugin_name)
+ local plugin_info = metadata.value
+ if span and not span:is_recording() then
+ return
+ end
+ local inject_conf = {
+ sampler = {
+ name = "always_on",
+ options = conf.sampler.options
+ },
+ additional_attributes = conf.additional_attributes,
+ additional_header_prefix_attributes =
conf.additional_header_prefix_attributes
+ }
+ local tracer, err = core.lrucache.plugin_ctx(lrucache, api_ctx, nil,
+ create_tracer_obj,
inject_conf, plugin_info)
+ if not tracer then
+ core.log.error("failed to fetch tracer object: ", err)
+ return
+ end
+
+ if #tracing.spans == 0 then
+ return
+ end
+ span.start_time = tracing.spans[1].start_time
+ for i, _ in ipairs(tracing.spans or {}) do
+ create_child_span(tracer, root_span_ctx, tracing.spans, i)
end
end
--- body_filter maybe not called because of empty http body response
--- so we need to check if the span has finished in log phase
function _M.log(conf, api_ctx)
if api_ctx.otel_context_token then
-- ctx:detach() is not necessary, because of ctx is stored in ngx.ctx
local upstream_status = core.response.get_upstream_status(api_ctx)
-- get span from current context
- local span = context:current():span()
+ local ctx = context:current()
+ local span = ctx:span()
if upstream_status and upstream_status >= 500 then
span:set_status(span_status.ERROR,
"upstream response status: " .. upstream_status)
end
+ inject_core_spans(ctx, api_ctx, conf)
+ span:set_attributes(attr.int("http.status_code", upstream_status))
Review Comment:
Use `http.response.status_code` to comply with the OTEL semantic conventions.
https://opentelemetry.io/docs/specs/semconv/registry/attributes/http/
##########
apisix/plugins/opentelemetry.lua:
##########
@@ -376,49 +377,101 @@ function _M.rewrite(conf, api_ctx)
ngx_var.opentelemetry_span_id = span_context.span_id
end
+ if not ctx:span():is_recording() and ngx.ctx.tracing then
+ ngx.ctx.tracing.skip = true
+ end
+
api_ctx.otel_context_token = ctx:attach()
-- inject trace context into the headers of upstream HTTP request
trace_context_propagator:inject(ctx, ngx.req)
end
-function _M.delayed_body_filter(conf, api_ctx)
- if api_ctx.otel_context_token and ngx.arg[2] then
- local ctx = context:current()
- ctx:detach(api_ctx.otel_context_token)
- api_ctx.otel_context_token = nil
+local function create_child_span(tracer, parent_span_ctx, spans, span_idx)
+ local span = spans[span_idx]
+ if not span or span.finished then
+ return
+ end
+ span.finished = true
+ local new_span_ctx, new_span = tracer:start(parent_span_ctx, span.name,
+ {
+ kind = span.kind,
+ attributes = span.attributes,
+ })
+ new_span.start_time = span.start_time
+
+ for _, idx in ipairs(span.child_ids or {}) do
+ create_child_span(tracer, new_span_ctx, spans, idx)
+ end
+ if span.status then
+ new_span:set_status(span.status.code, span.status.message)
+ end
+ new_span:finish(span.end_time)
+end
- -- get span from current context
- local span = ctx:span()
- local upstream_status = core.response.get_upstream_status(api_ctx)
- if upstream_status and upstream_status >= 500 then
- span:set_status(span_status.ERROR,
- "upstream response status: " .. upstream_status)
- end
- span:set_attributes(attr.int("http.status_code", upstream_status))
+local function inject_core_spans(root_span_ctx, api_ctx, conf)
+ local tracing = api_ctx.ngx_ctx.tracing
+ if not tracing then
+ return
+ end
- span:finish()
+ local span = root_span_ctx:span()
+
+ local metadata = plugin.plugin_metadata(plugin_name)
+ local plugin_info = metadata.value
+ if span and not span:is_recording() then
+ return
+ end
+ local inject_conf = {
+ sampler = {
+ name = "always_on",
+ options = conf.sampler.options
+ },
+ additional_attributes = conf.additional_attributes,
+ additional_header_prefix_attributes =
conf.additional_header_prefix_attributes
+ }
+ local tracer, err = core.lrucache.plugin_ctx(lrucache, api_ctx, nil,
+ create_tracer_obj,
inject_conf, plugin_info)
+ if not tracer then
+ core.log.error("failed to fetch tracer object: ", err)
+ return
+ end
+
+ if #tracing.spans == 0 then
+ return
+ end
+ span.start_time = tracing.spans[1].start_time
+ for i, _ in ipairs(tracing.spans or {}) do
+ create_child_span(tracer, root_span_ctx, tracing.spans, i)
end
end
--- body_filter maybe not called because of empty http body response
--- so we need to check if the span has finished in log phase
function _M.log(conf, api_ctx)
if api_ctx.otel_context_token then
-- ctx:detach() is not necessary, because of ctx is stored in ngx.ctx
local upstream_status = core.response.get_upstream_status(api_ctx)
-- get span from current context
- local span = context:current():span()
+ local ctx = context:current()
+ local span = ctx:span()
if upstream_status and upstream_status >= 500 then
span:set_status(span_status.ERROR,
"upstream response status: " .. upstream_status)
end
+ inject_core_spans(ctx, api_ctx, conf)
+ span:set_attributes(attr.int("http.status_code", upstream_status))
Review Comment:
Pls use `http.response.status_code` to comply with the OTEL semantic
conventions.
https://opentelemetry.io/docs/specs/semconv/registry/attributes/http/
##########
docs/en/latest/plugins/opentelemetry.md:
##########
@@ -153,37 +168,152 @@ In OpenTelemetry collector's log, you should see
information similar to the foll
```text
2024-02-18T17:14:03.825Z info ResourceSpans #0
-Resource SchemaURL:
-Resource attributes:
- -> telemetry.sdk.language: Str(lua)
- -> telemetry.sdk.name: Str(opentelemetry-lua)
- -> telemetry.sdk.version: Str(0.1.1)
- -> hostname: Str(e34673e24631)
- -> service.name: Str(APISIX)
ScopeSpans #0
ScopeSpans SchemaURL:
InstrumentationScope opentelemetry-lua
Span #0
- Trace ID : fbd0a38d4ea4a128ff1a688197bc58b0
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : 905f850f13e32bfb
+ ID : 5a3835b61110d942
+ Name : http_router_match
+ Kind : Internal
+ Start time : 2025-10-24 06:58:04.430430976 +0000 UTC
+ End time : 2025-10-24 06:58:04.431542016 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #1
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : 905f850f13e32bfb
+ ID : 4ab25e2b92f394e1
+ Name : resolve_dns
+ Kind : Internal
+ Start time : 2025-10-24 06:58:04.432521984 +0000 UTC
+ End time : 2025-10-24 06:58:04.44903296 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #2
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : 905f850f13e32bfb
+ ID : 3620c0f05dd2be4f
+ Name : apisix.phase.header_filter
+ Kind : Internal
+ Start time : 2025-10-24 06:58:06.960481024 +0000 UTC
+ End time : 2025-10-24 06:58:06.960510976 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #3
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : 4c5f3476f62a7e8a
+ ID : a9bfad7bb6986e41
+ Name : apisix.phase.body_filter
+ Kind : Internal
+ Start time : 2025-10-24 06:58:06.960579072 +0000 UTC
+ End time : 2025-10-24 06:58:06.96059008 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #4
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : b2994675df6baa83
+ ID : 26705f9c47584a5b
+ Name : apisix.phase.delayed_body_filter.opentelemetry
+ Kind : Internal
+ Start time : 2025-10-24 06:58:06.960613888 +0000 UTC
+ End time : 2025-10-24 06:58:06.960687104 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #5
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : 4c5f3476f62a7e8a
+ ID : b2994675df6baa83
+ Name : apisix.phase.delayed_body_filter
+ Kind : Internal
+ Start time : 2025-10-24 06:58:06.96059904 +0000 UTC
+ End time : 2025-10-24 06:58:06.960692992 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #6
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : 905f850f13e32bfb
+ ID : 4c5f3476f62a7e8a
+ Name : apisix.phase.body_filter
+ Kind : Server
+ Start time : 2025-10-24 06:58:06.96056704 +0000 UTC
+ End time : 2025-10-24 06:58:06.960698112 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #7
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : 2024d73d32cbd81b
+ ID : 223c64fb691a24e8
+ Name : apisix.phase.body_filter
+ Kind : Internal
+ Start time : 2025-10-24 06:58:06.961624064 +0000 UTC
+ End time : 2025-10-24 06:58:06.961635072 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #8
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : fd193dd24c618f60
+ ID : 8729ad6e0d94a23b
+ Name : apisix.phase.delayed_body_filter.opentelemetry
+ Kind : Internal
+ Start time : 2025-10-24 06:58:06.961648896 +0000 UTC
+ End time : 1970-01-01 00:00:00 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #9
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : 2024d73d32cbd81b
+ ID : fd193dd24c618f60
+ Name : apisix.phase.delayed_body_filter
+ Kind : Internal
+ Start time : 2025-10-24 06:58:06.961641984 +0000 UTC
+ End time : 1970-01-01 00:00:00 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #10
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : 905f850f13e32bfb
+ ID : 2024d73d32cbd81b
+ Name : apisix.phase.body_filter
+ Kind : Server
+ Start time : 2025-10-24 06:58:06.960980992 +0000 UTC
+ End time : 1970-01-01 00:00:00 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #11
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
+ Parent ID : cfb0b4603dc2e385
+ ID : 905f850f13e32bfb
+ Name : apisix.phase.access
+ Kind : Server
+ Start time : 2025-10-24 06:58:04.427932928 +0000 UTC
+ End time : 1970-01-01 00:00:00 +0000 UTC
+ Status code : Unset
+ Status message :
+Span #12
+ Trace ID : 95a1644afaaf65e1f0193b1f193b990a
Parent ID :
- ID : af3dc7642104748a
- Name : GET /anything
+ ID : cfb0b4603dc2e385
+ Name : GET /headers
Kind : Server
- Start time : 2024-02-18 17:14:03.763244032 +0000 UTC
- End time : 2024-02-18 17:14:03.920229888 +0000 UTC
+ Start time : 2025-10-24 06:58:04.432427008 +0000 UTC
+ End time : 2025-10-24 06:58:06.962299904 +0000 UTC
Status code : Unset
Status message :
Attributes:
-> net.host.name: Str(127.0.0.1)
-> http.method: Str(GET)
-> http.scheme: Str(http)
- -> http.target: Str(/anything)
- -> http.user_agent: Str(curl/7.64.1)
+ -> http.target: Str(/headers)
+ -> http.user_agent: Str(curl/8.16.0)
-> apisix.route_id: Str(otel-tracing-route)
-> apisix.route_name: Empty()
- -> http.route: Str(/anything)
+ -> http.route: Str(/headers)
-> http.status_code: Int(200)
Review Comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
