janecms opened a new issue #5307:
URL: https://github.com/apache/skywalking/issues/5307
Please answer these questions before submitting your issue.
- Why do you submit this issue?
- [ ] Question or discussion
- - [✔️ ] Bug
- [ ] Requirement
- [ ] Feature or performance improvement
___
### Bug
- Which version of SkyWalking, OS and JRE?
1.spring-cloud-starter-gateway-2.1.5.RELEASE
2.java version "1.8.0_141"
3.skywalking 8.1.0
- What happened?
Use an interface'/user-center/query' of the jmeter pressure measurement
gateway, and the function is provided by the service'user-center'.
Stress test effect: 8 times of ‘More than 300 spans required to create’
error occurred at 10,000 times. As an additional note, the abnormal rate
reached 89% due to too frequent calls.
jmeter report:
**The Error stack detail :**
`WARN 2020-08-13 11:13:12:730 reactor-http-nio-5 TracingContext : More than
300 spans required to create
java.lang.RuntimeException: Shadow tracing context. Thread dump
at
org.apache.skywalking.apm.agent.core.context.TracingContext.isLimitMechanismWorking(TracingContext.java:562)
at
org.apache.skywalking.apm.agent.core.context.TracingContext.createLocalSpan(TracingContext.java:297)
at
org.apache.skywalking.apm.agent.core.context.ContextManager.createLocalSpan(ContextManager.java:102)
at
org.apache.skywalking.apm.plugin.spring.cloud.gateway.v21x.NettyRoutingFilterInterceptor.beforeMethod(NettyRoutingFilterInterceptor.java:40)
at
org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInterWithOverrideArgs.intercept(InstMethodsInterWithOverrideArgs.java:75)
at
org.springframework.cloud.gateway.filter.NettyRoutingFilter.filter(NettyRoutingFilter.java)
at
org.springframework.cloud.gateway.handler.FilteringWebHandler$GatewayFilterAdapter.filter(FilteringWebHandler.java:138)
at
org.springframework.cloud.gateway.filter.OrderedGatewayFilter.filter(OrderedGatewayFilter.java:44)
at
org.springframework.cloud.gateway.handler.FilteringWebHandler$DefaultGatewayFilterChain.lambda$filter$0(FilteringWebHandler.java:118)
at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:44)
at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
at
reactor.core.publisher.MonoPeekTerminal.subscribe(MonoPeekTerminal.java:61)
at reactor.core.publisher.Mono.subscribe(Mono.java:3879)
at reactor.core.publisher.FluxRetryWhen.subscribe(FluxRetryWhen.java:86)
at reactor.core.publisher.MonoRetryWhen.subscribe(MonoRetryWhen.java:50)
at
reactor.core.publisher.MonoRepeatWhen.subscribe(MonoRepeatWhen.java:79)
at
reactor.core.publisher.MonoSourceFlux.subscribe(MonoSourceFlux.java:47)
at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
at
reactor.core.publisher.MonoPeekTerminal.subscribe(MonoPeekTerminal.java:61)
at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
at reactor.core.publisher.Mono.subscribe(Mono.java:3879)
at
reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:172)
at
reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56)
at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
at reactor.core.publisher.MonoPeek.subscribe(MonoPeek.java:71)
at reactor.core.publisher.Mono.subscribe(Mono.java:3879)
at
reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:172)
at
reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56)
at
reactor.core.publisher.MonoPeekFuseable.subscribe(MonoPeekFuseable.java:74)
at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
at
reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:153)
at
reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56)
at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
at
reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150)
at
reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:67)
at
reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:121)
at
reactor.core.publisher.FluxContextStart$ContextStartSubscriber.onNext(FluxContextStart.java:103)
at
reactor.core.publisher.FluxContextStart$ContextStartSubscriber.onNext(FluxContextStart.java:103)
at
reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:287)
at
reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:331)
at
reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1623)
at
reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:160)
at
reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:136)
at
reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:252)
at
reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:136)
at
reactor.netty.channel.FluxReceive.terminateReceiver(FluxReceive.java:427)
at reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:211)
at
reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:369)
at
reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:369)
at
reactor.netty.http.server.HttpServerOperations.onInboundNext(HttpServerOperations.java:496)
at
reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:92)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at
reactor.netty.http.server.HttpTrafficHandler.channelRead(HttpTrafficHandler.java:214)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at
io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
at
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
at
io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Unknown Source)`
Checked the ui trace, it feels a bit strange。
Comparing several red entries, we found:
The endpoint information is consistent: "VNode:
da87aad68dfd479292e868af12e61647.99.15972883737272806";
The component information is different. As shown
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
