aqasem1 opened a new issue, #11307: URL: https://github.com/apache/skywalking/issues/11307
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/skywalking/issues?q=is%3Aissue) and found no similar issues. ### Apache SkyWalking Component UI (apache/skywalking-booster-ui) ### What happened When I tried to select **last 1 day** from the data range, all charts kept loading :  I faced these errors : UI : ``` 2023-09-06 11:06:49,495 com.linecorp.armeria.client.logging.LoggingClient 95 [armeria-common-worker-epoll-2-2] WARN [] - [creqId=f959617a, sreqId=1f27eace, chanId=ae1117b6, laddr=10.200.176.9:33106, raddr=skywalking-skywalking-helm-oap/10.201.44.95:12800][http://skywalking-skywalking-helm-oap:12800/graphql#POST] Request: {startTime=2023-09-06T11:06:39.490Z(1693998399490000), length=588B, duration=321µs(321103ns), scheme=none+h2c, name=POST, headers=[:method=POST, :scheme=http, :path=/graphql, host=dev-skywalking.literatumonline.com, user-agent=Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36, content-length=588, accept=application/json, text/plain, */*, accept-encoding=gzip, accept-language=en-US,en;q=0.9, cdn-loop=cloudflare, cf-connecting-ip=178.20.184.42, cf-ipcountry=JO, cf-ray=80263e6aee1d2c5d-FRA, cf-visitor={"scheme":"https"}, content-type=application/json, origin=https://dev-skywalking.literatumonline.com, priority=u=1 , i, referer=https://dev-skywalking.literatumonline.com/dashboard/GENERAL/Service/c3RhZ2luZzo6QjI1fHxoYW5zZXI=.1/General-Service, sec-ch-ua="Chromium";v="116", "Not)A;Brand";v="24", "Google Chrome";v="116", sec-ch-ua-mobile=?0, sec-ch-ua-platform="Linux", sec-fetch-dest=empty, sec-fetch-mode=cors, sec-fetch-site=same-origin, x-forwarded-for=10.201.75.47, x-forwarded-host=dev-skywalking.literatumonline.com, x-forwarded-port=443, x-forwarded-proto=https, x-forwarded-server=traefik-6b8564dd4b-6zj8m, x-real-ip=10.201.75.47, xcf-bot-score=97, xcf-cert-verified=false, xcf-geoip-country=JO, xcf-geoip-sub-1=JO-AM, xcf-ja3-hash=c623c5ebffe395e777fbffb4b2347257, xcf-threat-score=0, xcf-verified-bot=false, cookie=_rdt_uuid=1681296344939.1ed0056e-0d1c-4d8f-931b-62b3f5452fd4; _hjSessionUser_864760=eyJpZCI6IjViMDliZmY5LWEyNDYtNWUwZC1hOTNmLTkxYmE0N2ZlN2JjZSIsImNyZWF0ZWQiOjE2ODU1MzEyMzM0ODEsImV4aXN0aW5nIjpmYWxzZX0=; _hjSessionUser_3229557=eyJpZCI6IjMwMDYxNTJkLTNmZmUtNWQ2Zi1hZTgwLWIzMThkODA5Yjc4MSIs ImNyZWF0ZWQiOjE2OTAxOTgzNjc1ODcsImV4aXN0aW5nIjp0cnVlfQ==; _hjSessionUser_2304749=eyJpZCI6ImViMWRiOTYwLTY0MmItNTNkZS04MjViLTM5YzE4Y2ZjMGE0YSIsImNyZWF0ZWQiOjE2OTEyNDAyMjkyNzEsImV4aXN0aW5nIjpmYWxzZX0=; AMCV_242B6472541199F70A4C98A6%40AdobeOrg=179643557%7CMCIDTS%7C19587%7CMCMID%7C37157943146741475962215755434207938115%7CMCAAMLH-1692870019%7C6%7CMCAAMB-1692870019%7CRKhpRz8krg2tLO6pguXWp5olkAcUniQYPHaMWWgdJ3xzPWQmdj0y%7CMCOPTOUT-1692272419s%7CNONE%7CvVersion%7C5.5.0; _uetvid=d52438e0f4ad11edb4312b2bbc51c046; AMCV_A5F7776A5245AF770A490D44%40AdobeOrg=-1124106680%7CMCIDTS%7C19587%7CMCMID%7C41422232191461782371464502193362020850%7CMCAAMLH-1692875807%7C6%7CMCAAMB-1692875807%7CRKhpRz8krg2tLO6pguXWp5olkAcUniQYPHaMWWgdJ3xzPWQmdj0y%7CMCOPTOUT-1692278208s%7CNONE%7CMCAID%7CNONE%7CvVersion%7C5.2.0; AMCV_1B6E34B85282A0AC0A490D44%40AdobeOrg=MCMID|45059540498227182811415972770887560710; _ga_9C4T5P6PP6=GS1.1.1692798963.1.0.1692798963.60.0.0; lastRskxRun=1692798967478; rskxRunCookie=0; rCookie=yqtwuy4xdno 9ff14j6sm4gllnssqtc; _ga_QE1SHLYHD2=GS1.1.1693663738.1.1.1693664125.0.0.0; kndctr_1B6E34B85282A0AC0A490D44_AdobeOrg_identity=CiY0NTA1OTU0MDQ5ODIyNzE4MjgxMTQxNTk3Mjc3MDg4NzU2MDcxMFIPCJKD7pGiMRgBKgRJUkwx8AHV9dyspjE=; _gcl_au=1.1.2115466952.1692265225.406042282.1693921591.1693921597; __cf_bm=KCzDQWL4A0HQMGPsgBoQLu42sFrxbH_K4G.ui6VfWRc-1693997755-0-AcVjL5RsnNs08o8/1+JgJxWUAgm2rUvna4MjqgujfIWcMiISDuNmQIvRJWmrpJq8nungPV1d782wRfa8I3/we/A=; cf_clearance=7MGG8yGshNWAn3Snr6Pz_JvBTdV8cJCQlBbPO88wvsk-1693997755-0-1-fba9aa7e.ac483a9a.8d7f3390-0.2.1693997755]} 2023-09-06 11:06:49,495 com.linecorp.armeria.client.logging.LoggingClient 156 [armeria-common-worker-epoll-2-2] WARN [] - [creqId=f959617a, sreqId=1f27eace, chanId=ae1117b6, laddr=10.200.176.9:33106, raddr=skywalking-skywalking-helm-oap/10.201.44.95:12800][http://skywalking-skywalking-helm-oap:12800/graphql#POST] Response: {startTime=2023-09-06T11:06:49.494Z(1693998409494000), length=0B, duration=0ns, totalDuration=10004ms(10004242963ns), cause=com.linecorp.armeria.server.RequestTimeoutException, headers=[:status=0]} com.linecorp.armeria.server.RequestTimeoutException: null at com.linecorp.armeria.server.RequestTimeoutException.get(RequestTimeoutException.java:36) ~[skywalking-webapp.jar:?] at com.linecorp.armeria.internal.common.CancellationScheduler.invokeTask(CancellationScheduler.java:468) ~[skywalking-webapp.jar:?] at com.linecorp.armeria.internal.common.CancellationScheduler.lambda$init0$2(CancellationScheduler.java:123) ~[skywalking-webapp.jar:?] at com.linecorp.armeria.common.DefaultContextAwareRunnable.run(DefaultContextAwareRunnable.java:45) ~[skywalking-webapp.jar:?] at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) ~[skywalking-webapp.jar:?] at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:153) ~[skywalking-webapp.jar:?] at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) [skywalking-webapp.jar:?] at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) [skywalking-webapp.jar:?] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) [skywalking-webapp.jar:?] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:406) [skywalking-webapp.jar:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [skywalking-webapp.jar:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [skywalking-webapp.jar:?] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [skywalking-webapp.jar:?] at java.lang.Thread.run(Unknown Source) [?:?] 2023-09-06 11:06:53,251 com.linecorp.armeria.server.DefaultUnhandledExceptionsReporter 104 [armeria-common-worker-epoll-2-15] WARN [] - Observed 17 exception(s) that didn't reach a LoggingService in the last 10000ms(10000000000ns). Please consider adding a LoggingService as the outermost decorator to get detailed error logs. One of the thrown exceptions: com.linecorp.armeria.server.RequestTimeoutException: null ``` OAP Backend: ``` 2023-09-06 11:02:13,115 com.linecorp.armeria.server.logging.LoggingService 95 [armeria-eventloop-epoll-5-45] WARN [] - [sreqId=b41f3ab4, chanId=29d15757, raddr=10.200.176.9:52056, laddr=10.200.180.23:12800][h2c://skywalking-skywalking-helm-oap-7b7d6dd967-4lj56/graphql#POST] Request: {startTime=2023-09-06T11:01:49.156Z(1693998109156244), length=600B, duration=23958ms(23958859227ns), cause=com.linecorp.armeria.common.ClosedSessionException, scheme=none+h2c, name=graphql, headers=[:method=POST, :scheme=http, :path=/graphql, host=dev-skywalking.literatumonline.com, user-agent=Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36, accept=application/json, text/plain, */*, accept-encoding=gzip, accept-language=en-US,en;q=0.9, cdn-loop=cloudflare, cf-connecting-ip=178.20.184.42, cf-ipcountry=JO, cf-ray=802637537dc730f3-FRA, cf-visitor={"scheme":"https"}, content-type=application/json, origin=https://dev-skywalking.literatumonline.com, prio rity=u=1, i, referer=https://dev-skywalking.literatumonline.com/dashboard/GENERAL/Service/c3RhZ2luZzo6QjI1fHxoYW5zZXI=.1/General-Service, sec-ch-ua="Chromium";v="116", "Not)A;Brand";v="24", "Google Chrome";v="116", sec-ch-ua-mobile=?0, sec-ch-ua-platform="Linux", sec-fetch-dest=empty, sec-fetch-mode=cors, sec-fetch-site=same-origin, x-forwarded-for=10.201.75.215, x-forwarded-host=dev-skywalking.literatumonline.com, x-forwarded-port=443, x-forwarded-proto=https, x-forwarded-server=traefik-6b8564dd4b-6zj8m, x-real-ip=10.201.75.215, xcf-bot-score=97, xcf-cert-verified=false, xcf-geoip-country=JO, xcf-geoip-sub-1=JO-AM, xcf-ja3-hash=f0401ad0e08e9d24acdb39fb71eabaea, xcf-threat-score=0, xcf-verified-bot=false, content-length=600, cookie=_rdt_uuid=1681296344939.1ed0056e-0d1c-4d8f-931b-62b3f5452fd4; _hjSessionUser_864760=eyJpZCI6IjViMDliZmY5LWEyNDYtNWUwZC1hOTNmLTkxYmE0N2ZlN2JjZSIsImNyZWF0ZWQiOjE2ODU1MzEyMzM0ODEsImV4aXN0aW5nIjpmYWxzZX0=; _hjSessionUser_3229557=eyJpZCI6IjMwMDYxNTJkLTNmZmUtNW Q2Zi1hZTgwLWIzMThkODA5Yjc4MSIsImNyZWF0ZWQiOjE2OTAxOTgzNjc1ODcsImV4aXN0aW5nIjp0cnVlfQ==; _hjSessionUser_2304749=eyJpZCI6ImViMWRiOTYwLTY0MmItNTNkZS04MjViLTM5YzE4Y2ZjMGE0YSIsImNyZWF0ZWQiOjE2OTEyNDAyMjkyNzEsImV4aXN0aW5nIjpmYWxzZX0=; AMCV_242B6472541199F70A4C98A6%40AdobeOrg=179643557%7CMCIDTS%7C19587%7CMCMID%7C37157943146741475962215755434207938115%7CMCAAMLH-1692870019%7C6%7CMCAAMB-1692870019%7CRKhpRz8krg2tLO6pguXWp5olkAcUniQYPHaMWWgdJ3xzPWQmdj0y%7CMCOPTOUT-1692272419s%7CNONE%7CvVersion%7C5.5.0; _uetvid=d52438e0f4ad11edb4312b2bbc51c046; AMCV_A5F7776A5245AF770A490D44%40AdobeOrg=-1124106680%7CMCIDTS%7C19587%7CMCMID%7C41422232191461782371464502193362020850%7CMCAAMLH-1692875807%7C6%7CMCAAMB-1692875807%7CRKhpRz8krg2tLO6pguXWp5olkAcUniQYPHaMWWgdJ3xzPWQmdj0y%7CMCOPTOUT-1692278208s%7CNONE%7CMCAID%7CNONE%7CvVersion%7C5.2.0; AMCV_1B6E34B85282A0AC0A490D44%40AdobeOrg=MCMID|45059540498227182811415972770887560710; _ga_9C4T5P6PP6=GS1.1.1692798963.1.0.1692798963.60.0.0; lastRskxRun=1692798967478; rskxRu nCookie=0; rCookie=yqtwuy4xdno9ff14j6sm4gllnssqtc; _ga_QE1SHLYHD2=GS1.1.1693663738.1.1.1693664125.0.0.0; kndctr_1B6E34B85282A0AC0A490D44_AdobeOrg_identity=CiY0NTA1OTU0MDQ5ODIyNzE4MjgxMTQxNTk3Mjc3MDg4NzU2MDcxMFIPCJKD7pGiMRgBKgRJUkwx8AHV9dyspjE=; _gcl_au=1.1.2115466952.1692265225.406042282.1693921591.1693921597; __cf_bm=KCzDQWL4A0HQMGPsgBoQLu42sFrxbH_K4G.ui6VfWRc-1693997755-0-AcVjL5RsnNs08o8/1+JgJxWUAgm2rUvna4MjqgujfIWcMiISDuNmQIvRJWmrpJq8nungPV1d782wRfa8I3/we/A=; cf_clearance=7MGG8yGshNWAn3Snr6Pz_JvBTdV8cJCQlBbPO88wvsk-1693997755-0-1-fba9aa7e.ac483a9a.8d7f3390-0.2.1693997755]} 2023-09-06 11:02:13,118 com.linecorp.armeria.server.logging.LoggingService 95 [armeria-eventloop-epoll-5-45] WARN [] - [sreqId=c342cc92, chanId=29d15757, raddr=10.200.176.9:52056, laddr=10.200.180.23:12800][h2c://skywalking-skywalking-helm-oap-7b7d6dd967-4lj56/internal/l7check#HEAD] Request: {startTime=2023-09-06T11:01:04.599Z(1693998064599972), length=0B, duration=68518ms(68518322789ns), scheme=none+h2c, name=HEAD, headers=[EOS, :method=HEAD, :authority=skywalking-skywalking-helm-oap:12800, :scheme=http, :path=/internal/l7check, if-none-match="healthy", prefer=wait=60, user-agent=armeria/1.24.3]} 2023-09-06 11:02:13,118 com.linecorp.armeria.server.logging.LoggingService 156 [armeria-eventloop-epoll-5-45] WARN [] - [sreqId=c342cc92, chanId=29d15757, raddr=10.200.176.9:52056, laddr=10.200.180.23:12800][h2c://skywalking-skywalking-helm-oap-7b7d6dd967-4lj56/internal/l7check#HEAD] Response: {startTime=2023-09-06T11:01:04.600Z(1693998064600193), length=0B, duration=68518ms(68518119201ns), totalDuration=68518ms(68518341233ns), cause=com.linecorp.armeria.common.ClosedSessionException, headers=[:status=500, content-type=text/plain; charset=utf-8, content-length=47, server=Armeria/1.24.3, date=Wed, 6 Sep 2023 11:02:13 GMT]} com.linecorp.armeria.common.ClosedSessionException: null at com.linecorp.armeria.common.ClosedSessionException.get(ClosedSessionException.java:35) ~[armeria-1.24.3.jar:?] at com.linecorp.armeria.server.HttpServerHandler.cleanup(HttpServerHandler.java:243) [armeria-1.24.3.jar:?] at com.linecorp.armeria.server.HttpServerHandler.channelInactive(HttpServerHandler.java:236) [armeria-1.24.3.jar:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:305) [netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281) [netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274) [netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:411) [netty-codec-4.1.86.Final.jar:4.1.86.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:376) [netty-codec-4.1.86.Final.jar:4.1.86.Final] at io.netty.handler.codec.http2.Http2ConnectionHandler.channelInactive(Http2ConnectionHandler.java:430) [netty-codec-http2-4.1.86.Final.jar:4.1.86.Final] at com.linecorp.armeria.server.Http2ServerConnectionHandler.channelInactive(Http2ServerConnectionHandler.java:103) [armeria-1.24.3.jar:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:305) [netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281) [netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274) [netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.handler.logging.LoggingHandler.channelInactive(LoggingHandler.java:206) [netty-handler-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303) [netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281) [netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274) [netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405) [netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:301) [netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281) [netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901) [netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:813) [netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) [netty-common-4.1.86.Final.jar:4.1.86.Final] at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) [netty-common-4.1.86.Final.jar:4.1.86.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) [netty-common-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:403) [netty-transport-classes-epoll-4.1.86.Final.jar:4.1.86.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.86.Final.jar:4.1.86.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.86.Final.jar:4.1.86.Final] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.86.Final.jar:4.1.86.Final] at java.lang.Thread.run(Unknown Source) [?:?] ``` And there are no errors on elasticsearch logs, My cluster specs : Deployed with helm chart 1 Node UI Version 9.6.0 3 Node elasticsearch ( 4 CPU, 8 RAM) 4 Node OAP (4 CPU , 8 RAM) Version 9.6.0 Around 250 Java agent version 9.0.0 Thanks all ### What you expected to happen Retrieve all data I select it up to 1 month ### How to reproduce try to build SW with many Java agent , and you will face the issue , ### Anything else _No response_ ### Are you willing to submit a pull request to fix on your own? - [ ] Yes I am willing to submit a pull request on my own! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
